Download Entire Document (710 KB) PDF
Benchmarking Quantitative Default Risk Models: A Validation Methodology
March 2000
Many of the world's largest financial institutions have developed advanced quantitative credit risk models
that help to measure, monitor and manage credit risk across their business lines. However, the Basel
Committee on Banking Supervision recently identified credit model validation as one of the most challenging
issues in quantitative credit model development1. In particular, issues of data sufficiency and
model sensitivity analysis were highlighted as was the lack of a consistent and formalized validation
methodology in many institutions.
Because of Moody's leading role in credit risk assessment, Moody's has also been active in developing
and testing quantitative methods that can be used for credit risk management. This article presents a summary
of the approach Moody's used to validate and benchmark a series of popular quantitative default risk
models, including our own model for public companies2. We discuss performance measurement and sampling
techniques, as well as other practical considerations associated with performance evaluation for quantitative
credit risk models. This framework specifically addresses issues of data sparseness and the sensitivity
of models to changing economic conditions. Our model validation approach continues to evolve and is
used extensively for evaluating internal and external quantitative models.
In summary:
-
We describe some of the techniques used at Moody's to benchmark the performance of a number
of corporate default prediction models. This approach uses a combination of statistical and computational
methods to address the data problems that often appear in credit model validation and
to provide an indication of the stability of default models over time.
-
Because we have found that simple statistics (such as the number of defaults correctly predicted)
are insufficient and often inappropriate in the domain of credit models, we have developed the
use of several metrics for evaluating model performance: cumulative accuracy profiles (CAP)
plots, accuracy ratios (AR), conditional information entropy ratios (CIER) and Mutual
Information Entropy (MIE).
-
We demonstrate the validation techniques we describe by benchmarking a variety of popular
credit risk models, including Moody's own default model for public companies, using our proprietary
databases. To our knowledge, it is the first time that such broad analysis has been undertaken
using an extensive comprehensive data set and a consistent methodology based on the information
content of the models.
-
Using Mutual Information Entropy, we are able to demonstrate the amount of additional predictive
information contained in one credit model versus another, which often suggests situations in
which two models can be profitably combined or in which an inferior model can be eliminated.
1 Basel (1999).
2 This work and the details of Moody's model for public companies are described more fully in
Sobehart, Stein, Mikityanskaya and Li (2000).
Download Entire Document (710 KB) PDF
|
