When Should You Perform an ALM Validation or Back-Test Your Models?

Emily Hollis
October 21, 2009 | COMMENTS

A validation in essence is a review of the ALM model's calculations and processes. Validations can be viewed more as an “audit” of the ALM process to ensure accuracy and verify the strength of internal controls. It is a review of the logical and conceptual soundness of a credit union's ALM model. Below are excerpts from the FDIC Supervisory Insights-Compliance Examinations guidelines that offer additional explanation of a validation:

“Validation should not be thought of as a purely mathematical exercise performed by quantitative specialists. It encompasses any activity that assesses how effective a model is operating. Validation procedures focus not only on confirming the appropriateness of model theory and accuracy of program, code, but also tests the integrity of model input, outputs, and reporting.”

“Regulatory review typically focuses on the core components of the bank's governance practices by evaluating model oversight, examining model controls, and reviewing model validation.”

Validation work should be performed by parties completely independent from the model's design and use. They can be performed by an independent model validation group within the credit union, the internal audit department, staff with model expertise from other areas of the credit union, or an external vendor.

The depth and extent of the validation should be consistent with the materiality and complexity of the risk being managed. Validations can also be somewhat costly and are therefore generally not needed for smaller credit unions (less than $100 million in assets). Additionally, they might not be warranted for those credit unions that are fairly simplistic particularly those with short-duration assets and high levels of capital.

On the other hand, for large complex credit unions where the risk profile is high especially those with a high amount of embedded options (i.e., mortgage exposure, especially “innovative” type mortgages or complex investments) they are imperative.

Case in point, strategies are dependent upon the outcome of the ALM reports. If assumptions or modeling techniques are flawed it can be extremely costly, lessening the ability to generate additional income, or placing the credit union in jeopardy of potential losses.

Guidance from the FDIC Supervisory Insights states that the board of directors sets policy for the level of controls and the scope of a validation. Management and staff implement the policies, and internal audit verifies that implementation meets policy requirements. A graph of the responsibilities is depicted below:

A validation should include three general procedures:

• Independent review of the logical and conceptual soundness of the model
• Comparison against other models
• Comparison of model predictions against subsequent real-world events

For those that want additional guidance on what a validation truly entails, the OCC 2000-16 bulletin for model validations can be retrieved via: www.occ.treas.gov/ftp/bulletin/2000-16.txt.

The second part of the discussion is whether to conduct a “back test” or a “variance analysis.”

Back-testing can be very costly. A thorough analysis would entail modeling a prior balance sheet in the current rate environment. It is difficult to perform and might not bring much value because analyses are usually projected with parallel movements in rates, which typically do not occur.

A variance analysis is the identification of material differences between actual and forecasted income statement and balance sheet amounts, and then ascertaining the causes for these differences. The methodology usually includes a direct comparison to the financial statements for a particular forecast period, or by using key financial indicators, such as net interest margin, cost of funds, and asset yields. Variance analyses are generally accepted by auditors and examiners and are much easier to complete.

Emily Hollis is president of ALM First Financial Advisors in Dallas, Texas. Contact Hollis at 800-752-4628 or ehollis@almfirst.com.