Stress Test Transparency and Model Uncertainty

10 Sep 2019
Read Time 4 mins
Categories :
Bank Capital

The stress tests that are run by the Federal Reserve each year determine whether large banks maintain sufficient capital to remain well-capitalized in a severe economic downturn.  The results of the test are important as they have a direct impact on credit allocation and also support the safety and stability of the economy.  The models, methods and assumptions used in the stress tests are not fully transparent to the public, which makes it difficult to fully understand the results of each year’s tests. 

In this post, we review some recent research suggesting that the outcomes of model-based stress tests are inherently volatile and that different modeling assumptions could lead to significantly different results.  We conclude with a discussion of the importance of transparency when the outcomes of stress tests are model dependent and subject to considerable uncertainty.

The Federal Reserve’s stress testing program is a complex modeling exercise that takes in a variety of data inputs about a bank’s exposures to risk through loans and securities.  The models then are used to estimate the losses the bank would sustain in a severe economic downturn and the resulting impact on capital levels.  A recent research paper by Paul Kupiec of the American Enterprise Institute, “Policy Uncertainty and Bank Stress Testing,” investigates the extent to which the outcome of model-based stress tests depends upon modeling assumptions.  Perhaps unsurprisingly, the paper finds that different models reach different conclusions about bank capital adequacy.  What is surprising, however, is the wide range in results that is generated from modest changes in model design and specification.  Indeed, Kupiec finds that the “results are all over the map.”  The chart below summarizes the key results from Kupiec’s paper, which considers the comparative results of six mainstream, econometric models.  Specifically, the chart shows the number of banks predicted to experience a capital shortfall and the size of the capital shortfall according to each model.

Source: Kupiec, Paul, Policy Uncertainty and Bank Stress Testing (July 2019)

As shown in the figure above, the results vary widely across the six models.  The number of banks experiencing a capital shortfall ranges from zero to six (out of 14 banks considered) while the size of the capital shortfall ranges from $0 to over $900 billion.  Accordingly, which banks are appropriately capitalized depends quite a lot on the specific stress test model employed.

While the results of this research paper are interesting and worth considering, it is important to be clear that these results can’t be directly translated to the potential variability in the Federal Reserve’s stress test models for two reasons.  First, the models employed by Kupiec are relatively simple and parsimonious “top down” models that only use a modest number of highly aggregated inputs, such as  the total value of a bank’s construction loans, while the Federal Reserve makes use of a suite of “bottom up” models that make use of detailed and granular, bank-specific information, such as the value and characteristics of specific construction loans.  Second, while the Federal Reserve has made some information about some of its models public, we simply do not have enough transparency on the design and specification of the Federal Reserve’s stress test models to assess their sensitivity to differing inputs and assumptions. Put more bluntly, we don’t know what we don’t know.

The stress tests are an important part of the Federal Reserve’s bank capital regime.  At the same time, it must be recognized that the tests create winners and losers based on the outcome of stylized, and imperfect statistical models.  In particular, in light of Kupiec’s results, one has to wonder how the results of the test would differ if moderately different modeling choices were employed.   The potential for significant variation in outcomes demonstrated by Kupiec’s research highlights the importance of transparency in the models, methods and assumptions that are used to determine a bank’s capital adequacy. 

The results of the stress tests have important economic consequences that determine credit allocation in the U.S. economy.  As a result, both the public and banks have a right to understand how capital adequacy is being assessed and how those assessments may be sensitive to specific assumptions or modeling choices that are not clearly supported by data and may be subject to considerable uncertainty.  Ultimately, regulators must make some determination about capital adequacy based on imperfect information and foresight.  Nobody has a crystal ball.  At the same time, in light of the significant uncertainties that abound, it is reasonable for banks and the public to have a clear and transparent understanding of the models and assumptions that determine winners and losers.  While the Federal Reserve has recently improved the degree of stress test transparency, more is needed.                           

Forum Updates