Validation of Credit Models in ECB Templates

This example uses:

Modelscape for MATLAB Modelscape for MATLAB

This example shows how to fill in the European Central Bank (ECB) model validation templates using Modelscape™ in MATLAB®.

ECB has published a suite of model validation templates covering a wide array of credit models, including Probability of Default (PD) models. Find the ECB templates at this address under 'Related Documents'. Find the instructions for filling in the templates in the document Instructions for reporting the validation results of internal models - IRB Pillar I models for credit risk.

Fill in ECB Template Input Data Using Modelscape

The ECB validation templates require three kinds of inputs:

Model and organization metadata - for example, model and institution names, and the start and end dates of the review period
Directly observable quantitative data - for example, the number of customers who defaulted during the observation period
Model performance metrics - for example, metrics that measure the discriminatory power of the model, or credit rating migrations

Use Modelscape function fillValidationTemplate to fill in the last two types of inputs. You can then fill in the model names, identifiers, and other variables in Excel.

fillValidationTemplate(fileName, ...
      "InitialValidationData", initialValidationData, ...
      "InitialPortfolioData", reviewStartPortfolioData, ...
      "TerminalPortfolioData", reviewEndPortfolioData);

fileName is the name of the spreadsheet you fill in. Use the standard format required by ECB: "[LEICode]_[ModelType]_[ModelID]_[ReferenceDate]_[VersionNumber].xlsx”. For more details, see the final paragraph of Section 2.2 of the ECB instructions.
reviewStartPortfolioData and reviewEndPortfolioData are mrm.data.validation.reporting.pd.OperationalData objects with information about the portfolio at the beginning and the end of the current review period. Examples include the model PDs, exposures, and ratings for the customer populations, as well as flags to indicate credit transferals and overrides within the populations.
initialValidationData is an mrm.data.validation.reporting.pd.ValidationData object with the information used in the initial validation of the model, potentially many years before the current review period. Examples of such information include the original model PDs and default indicators for some validation data set.

Set Operational Data

The templates require operational data as a table containing various data with one item per customer or entity, together with the observation date and flags such as whether the model allows technical defaults. In this section, the term name indicates a single obligor such as a customer.

Supply the following data as arrays of equal lengths:

IDs: a string identifier for each name in the data set
PDs: the probabilities of default predicted by the model
Ratings: the ratings assigned to the customers (as ordered categorical variables - see below for more details)
Exposures: the current exposure for each name

Store Ratings as categorical variables with a fixed ascending order. Otherwise, certain data quality checks built into the ECB templates will fail. Define the categorical variables in MATLAB using the following syntax:

ratingLabels = categorical(["CCC", "B", "BB", "BBB", "A", "AA", "AAA"], ...
    ["CCC", "B", "BB", "BBB", "A", "AA", "AAA"], 'Ordinal', true);

While the ECB instructions do not specify a choice of currency, express the exposures in a single reference currency. This is because various validation measures require calculating the total exposures within some classes of names.

In addition to the four vectors, the operational data must contain the following arrays of logicals of the same size as the identifier arrays. These indicate anomalous states of the names and are optional. The OperationalData class will default these values to false if you do not specify them.

The flags determine the correct in-scope sample, so it is important that you set them correctly.

InDefault: set to True if the name currently in default (bullet point 2.3 (c) of the ECB instructions).
TechnicalDefaultIndicators: set to True if the name is in technical default (2.3 (d)).
OtherModel: set to True if a different model is used for the name for rating or regulatory capital purposes (2.3 (e)).
NonDefaultablePositive: set to True if the name is non-defaultable even if it has a positive exposure (p. 10, footnote 18).
Deficient: set to True if the name should be excluded from the sample due to process deficiency (2.5.1 (c)).
Outdated: Does the name have an outdated rating or financial statements (2.5.1 (d)).
Overridden: set to True if the rating has been overridden (2.5.1 (e)).
Transferred: set to True if the name had been transferred to it the rating of some third party (2.5.1 (f)).

Finally, the OperationalData class should contain the following:

OutdatedAllowed, TransferralAllowed, OverridesAllowed, TechnicalDefaultsAllowed: Single logical variables to indicate whether the model and assignment process allow for outdated accounts, rating transferals, overrides, or technical defaults (PD validation template, Sheet 2.0).
Date: The 'as-of' date for the above data.

In practice, it may be convenient to save the one-datum-per-name data as a table and supply the final four logicals and the as-of date separately.

Set Validation Data

Operational data objects contain snapshots of the model predictions and other relevant data at given time points. Validation data objects, on the other hand, carry information about how the status of the names changes over time. Validation metrics assess how these changes relate to the PDs predicted by the model.

Each ValidationData object carries the following data.

ObsStartDate and ObsEndDate: initial and the terminal dates of the observation period.
IDs: string identifier for each name in the data set.
PDs: model PDs as predicted at ObsStartDate.
Exposures: exposures of the names in some appropriate reference currency, as of ObsStartDate.
DefaultIndicators: logical flags indicating the names that have defaulted during the observation period.
InitialRatings and TerminalRatings: rating grades predicted by the model at the beginning and the end of the observation period (as ordered categorical variables). The terminal ratings should have an ascending order again.

In addition, ValidationData uses an augmented set of possible ratings, where the following three extra 'ratings' are allowed (see point 2.5.1 (h) of the ECB instructions):

Default - for names that defaulted during the observation period
Migrated - for non-defaulted names that have moved to a different model during the observation period
Gone - for non-defaulted names that have terminated their business relationship with the credit institution during the observation period

Note that the validation data containers should have data only for the names that have been pre-screened to be in scope for validation. Given operational data containers operationalData1 and operationalData2 for some reference dates, use an MRM helper function to construct the associated validation data:

validationData = mrm.data.validation.reporting.pd.validationDataFromOperationalData(operationalData1, operationalData2);

This not only performs the appropriate screening but also takes care of the augmentation of the terminal rating categories as explained above.