The pace of regulatory change is slowing, and investors are eagerly awaiting the latest round of stress test results. But a study from the Office of Financial Research at Columbia University argues that the tests have become too predictable to be useful – instead of preparing for the unexpected, banks may be preparing for requirements known well in advance.

“Whereas the results of stress tests may be predictable, the results of actual shocks to the financial system are not, and herein lies the concern,” write study authors Paul Glasserman and Gowtham Tangirala. “The SCAP worked, in part, by providing new information. To the extent that stress test results become more predictable, they become less informative.”

stress test loss rate distribution

Strong correlations within stress test results

To show how much inter-correlation there is in recent stress test results, Glasserman and Tangirala plot loan loss rate projections for all seven categories of loan under adverse and severely adverse scenarios. They find a simple linear relationship for each bank (though the slope does vary a bit by bank).

“These patterns are puzzling. We would expect to see a more complex relationship between adverse and severely adverse outcomes, reflecting a nonlinear response of bank portfolios to economic shocks,” they write.

adverse v severe loss rates

There are two ways of interpreting the result: either mortgages, credit cards and counterparty risk all react the same way to real world shocks, or the stress test simply doesn’t tell us how their reactions are likely to differ. When you plot the same data sorted by loan type instead of by bank the correlation is just as striking.

adverse v severe loss rates by loan type

Stress test results consistent from year to year

When trying to predict what a given bank’s projected loss rates will be this year, looking at last year’s results is an obvious starting. But Glasserman and Tangirala found that it’s difficult to improve your forecasting from there.

“In some cases, actual charge-offs appear to have some forecasting power,” they write. “However, none of the variables we tested adds much in forecasting stress loss rates compared with using a bank’s prior year’s stress loss rate.”

Again, it’s not surprising that there is a correlation between two years’ stress test results – a company as large as Citigroup or Bank of America can only change so much in a single year. But the idea that no other factor helps you improve your forecast implies that we don’t know how well banks will fare under pressure so much as how good they are at taking tests.

loan loss rates yeay on year