The Secret to Automation and Lineage: MathWorks Model Inventory
The heart of any robust model risk management framework is the model inventory. However, many inventory systems are proving unable to keep up with the increased demands from regulators because they are effectively databases with a well-designed interface and lack the traceability and automation needs of a modern system
While linking and drilling down on models, data, and documents is often the starting point for oversight discussions, the areas that are often overlooked are document management, interoperability, workflow automation, and customization through advanced analytics. These key areas provide enhanced cost reduction to the business by accelerating regulatory approval and eliminating inefficiencies.
In this session, you’ll see a live demonstration of the MathWorks Model Inventory, showcasing how several large global banks that adopted this solution addressed key challenges around model lineage, governance, automation, and review.
Published: 5 Oct 2021
Hi. Welcome to my talk. My name's Ian McKenna, and today we're going to be talking about the secret to automation and lineage through the use of The MathWorks model inventory. So here's a picture of a dependency diagram that consists of models, documents, and data sources. And the outputs of the models, here, are fed into other downstream models, which ultimately become aggregated and become a line item on the general ledger of the business, which you see on the right-hand side here.
Now, the challenge that we face is that we have many models which effectively are not siloed, but are instead interconnected. And if a particular model's not working as we might expect, how do we go about determining the effect that this model has on the business, or a particular product?
To further complicate this interconnected ecosystem, models have additional assets such as documents, data sources, scripts findings, and so forth, and all that must be organized and managed. So not only do you need a centralized location for all the information related to models, but you also need a mechanism to be able to manage the versioning of these assets over time. And this really is a key requirement for traceability. That way if a regulator comes up to you and asks, hey, how did your model behave, say, six months ago? Are you able to replicate those results today?
Another common risk is the turnover in organizations. So if a mall developer leaves a company, we need to be confident that we're able to find the information that we need. And I've personally seen situations where that is actually very difficult to do. It can take months to track down a particular model document if key people have left and things are not organized appropriately. So a model inventory system must be robust to these types of situations.
To that end, MathWorks has built a unified system of technologies to address the key business modeling, workflow, and governance needs of the entire model lifecycle, while taking into account the needs of various stakeholders such as quants, regulators, business owners, validation teams, chief risk officer, and so forth.
And the key is to make automation and transparency and traceability first-line citizens here, where we want to be able to integrate the entire solution between both the first and second lines of defense. And we cover this entire workflow wheel, here, that you see on the left-hand side, from development to validation, pre-implementation-- putting those models into production, and eventually monitoring them and recalibrating and sunsetting those models.
And doing so-- providing this framework for this entire model lifecycle-- allows us to achieve three key results-- 1, reduce the cost of our system by eliminating any efficiencies; 2, enhance the communication between the first and second line of defense, effectively providing more of an agile, MRM-type workflow; and 3, accelerate the regulatory approval of your models themselves.
Today we're going to focus on the model inventory component specifically. And there are a couple key business values that we want to extract from this model inventory system. First is that we don't want to reinvent the wheel.
If a team member has already built a model, can we leverage those models when we need to construct new models? That way we don't have to go about building everything from scratch again. So we need to be able to find those models that our team members have built all in one location.
The second is being able to provide automation capabilities around the model inventory. Now, if you embed a model lifecycle or model workflow component to it, that will naturally allow you to prior to that automation so that if a state of a model changes over time, you can do different actions automatically.
And then the third, here, is traceability. So again, these assets are not static. They're dynamic. They change over time. And we need to be able to trace models not only from an upstream or downstream model perspective, but also over time as well.
And so some of the key capabilities we're going to be talking about today is the fact that you need to take those model assets and centralize them in a single location that's accessible through a variety of different mediums, whether it's a local desktop, or it's on an iPhone or tablet. You want those assets to be accessible.
Two is the ability to drill down. So being able to drill down at various levels of granularity is very important to be able to answer the business questions that arise. And then the third one, here, is the customization. Model inventory system needs to be flexible to the needs of the business. That could mean things like, for example, changing the workflows that are required for different types of models, or being able to customize executive dashboards with the key risk indicators that are most important to this particular business.
So having said all that, let's take a look at the MathWorks Model Inventory. So this is the MathWorks Model Inventory. It provides multiple stakeholder views at varying levels of granularity.
So at the top-level view you have aggregated statistics to show how it impacts the bottom line of the business, all the way down to the model-level view, where you have information about the metadata of the model itself, and views all in between those two different extremes. So on the right-hand side, here, you have an at-a-glance view of all the different models and how they're interconnected with each other.
And the nice thing about the model inventory system is that you can quickly create connections and manage these dependencies using visual tools and point-and-click operations like I just did here. So in this case, I just created a connection between these two different models. And likewise, if I wanted to remove connections, it's just as easy.
Now, the nice thing, too, about the model inventory system, here, is that it's bi-directionally linked between the different views. So in this case here, the dependency view-- if I care about a certain model-- like, in this case, the LGD model-- you can see it's highlighted in the Table view, so I can quickly find and filter down those models of interest. And likewise, if I want to take a look at a certain subset of models here in my Table view, it highlights them in the Dependency view so I can quickly identify those models from a visual perspective as well.
Now, let's say that I care about a certain subset of models. I care about models that impact the provisioning of a credit card-- so in this case, these four models up here-- and I don't care about these additional models down here, in which case I want to eliminate the noise of those additional models and focus on just the models of interest. So I can do that by applying focus here, and this shows me just those models. And again, I can quickly take a look at these models here in the Model Table view, and highlight them and see them very quickly from the bi-directional linking.
Now, let's say that I want to take a look at all the models here, and I want to look at specifically the Ledger view on the left-hand side, here, at top-left, that shows me the aggregated view of the different lines of business and what impact that has to the bottom line of my company, here.
So in this case here, let's say that a regulator comes to us and says, why does retail banking, here, have such a high model risk score? Well, if that's the case, we need to be able to drill down and identify those models of interest. So filtering and drilldown is a key aspect of the model inventory, where let's say I care about the retail banking.
In this case here, you see that clicking on this provided a drilldown onto the models of interest. And now let's say, again, I care only about that provisioning model. So I can layer on additional filters very quickly, and now it shows me just those two models here that may be of interest to me. And the Model view down here is, again, the most granular-level view, where I can see all the information associated with the particular model-- what business group was it in?
Is it a proprietary or vendor-based model? What's the tier, or risk score? When was it last validated? Who was the validator? All of that information, here, is right at your fingertips so you can quickly identify that information. And the nice thing, again, about the model inventory is that it's customizable. So if you have different fields that you want to be shown in the view down here, you can easily make those changes within the system.
Now, let's say that I care about a certain model-- let's say, again, this provisioning model. Let's open this up and take a look at the details. So on the right-hand side, here, is basically a one-stop-shop for everything model-related-- so all of your model assets, your model metadata, documents, findings, exceptions. All that information is right here so that you don't have to go to sign into different systems, and then find that data. It's all centralized in one location.
And on top of that, that small inventory can be deployed to a web page. So this entire interface that you're seeing here can be run inside a browser, or on your phone or tablet. So it's a truly centralized location from a single point.
Now, on the right-hand side, here, you see a bunch of different metadata that is in the details, here-- so again, things like days left under waiver, all this information. Again, this metadata can be modified at your leisure, here. And the nice thing, too, about this, I want to point out, is that it is using interoperability with gold standard technologies like GitLab and GitHub to save the models and scripts that you have under version and control systems.
So again, you can take a look back at any point in time to see what the source code of the model was, and how it behaved. And in fact, you can actually run that version directly within the model inventory here. So I can take this and run it within the model review environment, which is another component of The MathWorks Model Risk Management solution that extends the capabilities of the model inventory.
Now, here, let's say that we care about a variety of different models, because we don't necessarily live in a world where everything is built in one language. We live in a polyglot world of many different languages. And so again, the nice things about The MathWorks Model Inventory is that it supports all languages, here. There is no preference for any language. We support all of the ones that you expect-- Python, RSS, Java, C++, C#, and so forth. And you can include other ones, too, here, as well, because again, the options, here, are customizable.
Now, another thing I want to point out here is that these tabs themselves are also customizable as well, depending upon the needs of the business. So for example, Usages tab, we can customize what's associated with this or add a whole other completely new tab to the model assets here. In this case, we have documents here, which are associated with document standards, maybe some proprietary documents, the model documents themselves, validation documents and reports.
All that information here is stored within the model inventory. And we can take these models and then launch them directly from the model inventory system here. So if I want to take a look at this particular document, here is what that document looks like. So in this case here, you can see that the documents being saved and served off of SharePoint.
So again, the nice thing about the model inventory is that it has interconnections between these modern technologies like SharePoint, OneDrive, GitLab, GitHub, and so forth. And so we can leverage that and within the model inventory system so that you don't have to do multiple authentication points. You just have a single sign-on at the beginning, and then everything else just works.
And so here in this case, I've got my document. And again, you can upload any series of documents that you want within the model inventory and save them in the location that makes sense for your business whether that's SharePoint or a map drive or some other system we can interrupt with it.
On top of the documents, we also have information associated with how the model is behaving over time, and if we found some key things of interest, and where would we find them whether they were in the testing or documentation or the actual model itself? Who is assigned to this particular issue? And how long do they have to remedy the problem? Or can we continue to execute this model in production if needed? So you have all that information, again, available to you, which can be customized for the different fields that you need.
And so all this information, again, is stored within the Details tab of the model inventory here. But one other thing I want to point out that's unique to the solution here is the embedded workflow diagram. Now, this is unique in that this provides us a way for tracking the state of a particular model and then also triggering the execution of some custom code.
And so that isn't just things like alerts or emails that are sent to people. But it could be literally anything because-- and you'll see this a little bit later on-- the MathWorks model inventory is actually based on the MATLAB language, meaning that you can use MATLAB and anything that MATLAB interopts with. So including things like Python or R, other languages, you can interopt and use those from within MATLAB too.
So if I want to, say, for example, set up a repository on GitLab automatically when I go from draft to in development, or if I want to trigger an email to a bunch of different people and stakeholders when it goes into the validation state so that everyone is kept in line what's going on, or if I go from invalidation to maybe another preimplementation stage-- I want to automatically build some test documents-- you can do that and trigger any type of custom code that you like here.
And to give you an example of what you can do, let's say that I'm going to trigger some custom code to happen when I go into the submission state here. And so you see here the views of the states that are available are dependent upon the workflow and which direction these things are linked to each other. So in this case here, let's go to submission state.
And now what you see here is a custom form that has been populated to capture some information. It's automatically put in some template information here for me. And so this form and what is being done can be all, again, customized, but it provides an automation point for us. So in this case here, let's say-- I don't know-- we're going to upload a impact analysis report. And this is going to be a to-do that we want to do. And I'm going to assign that to, in this case, myself here.
So now you see here that this has been saved into the system. I can take a look at that. And you see right here that now we've got that upload impact analysis report that was just populated. And so this is one way that I can trigger and automate pieces of my pipeline workflow directly from the model inventory system.
So the model inventory will also automatically send out emails to those stakeholders that are impacted by the assignment of this particular issue, this workflow issue. So let's take a look at a email that was just sent to me since I assigned this to myself. Let's see what that looks like.
So here is the email that was just sent to me that shows me that there has been a new issue assigned to me and that I need to take a look at this in more detail. So again, those alerts and triggers are automatically done for us.
Now, one thing I do want to point out, again, about this workflow diagram is that it is fully customizable, again, using drag-and-drop tools from within the MathWorks model risk management platform. And so this is an example here of how I can easily create those workflows and do that with just, again, point-and-click tools here.
So in this case, say this is development. And I can create continuous connections between these different workflows and create my particular workflow of interest. So again, it's all customizable. And so if I have different types of models that have different types of workflows, I can have a particular workflow for a particular subset of models or documents.
The last thing I wanted to mention before we move on to the next piece here is that the metadata here is available for being modified based on the access level and permissions that you have. So for example, a model developer and a model validator may have different subset of different fields here that they can change and certain fields that may be disabled by default and cannot be changed by them.
Now, another benefit of the MathWorks model inventory is that it has a built-in risk-scoring capability. So let's take a look at what that looks like here. So this is the risk-scoring capability that's built directly into the tool. I'm going to hide the code here so we just take a look at what this might look like here. This is effectively a form view, but it's actually powered by MATLAB code. I just hid the corresponding MATLAB code. So this is just a MATLAB script, which has embedded controls and buttons and dropdown lists and things of that nature.
And so you can leverage the power of MATLAB as an analytics platform to do literally anything. This is more of a traditional-based questionnaire with-- where you may overlay additional stakeholder considerations on what the rating of the particular model should be. But it could be anything. You could use cutting-edge advanced AI and machine learning or NLP or literally anything.
So for example, maybe you have a-- for every model, you have a description of that model. And the description is a couple of sentences around what the model does. And so say you have a model that's associated with capturing the price of oil. And given the pandemic now and the fact that oil prices went negative for a little while, maybe that's a particular thing that you want to filter out and apply a higher risk associated with that because it's an oil-particular model.
And so you could use things like NLP or deep learning or whatever to identify those models of interest and then associate a specific risk score with those using those advanced, cutting-edge tools. And since MATLAB has all the tooling for advanced analytics from reinforcement learning to deep nets and neural nets and all that stuff, you can access that within this script here.
And the other thing I want to point out too is that this customizable script that we have here is, again, linked back into the model inventory data itself. So if I make changes to, say, for example, the risk rating here-- I think the exposure level should be low-- and I apply those modifications here, then when I make those changes, it's going to reflect that back into the model here.
So when I save this, it shows me here, hey, right now the exposure's set to high. But now this is telling me that it should be set to low. Do you want to do that? So if we say OK, you can see now here that the exposure has been set from high to low.
Now, all of this is great, but one of the most important things here is an executive-level view of the details of all of these models and how they're behaving. And that's what the Dashboard view here provides us with. So here in this case, this shows us customizable key risk indicators that we may care about.
So this is just an example. The actual dashboard view could look like anything you want. And in fact, we can also interconnect with external providers like Tableau, Klipfolio, Spotfire, and things of that nature too.
But it provides us with analytics that provide drill down too. So for example, let's say we care about the risk rating of models based on what they were written in. You can see here that I can apply these types of operations and do what-if analysis and stress testing of models here if I wanted to as well and see how they behave and identify models of interest.
And speaking of customizing the models themselves, just to show you what it's like to do a customization of a dashboard like this, let's take a look at that dashboard within the MATLAB system here. And so this is that same dashboard.
And I can see I can make modifications to it like, let's say, for example, I want to put a button on here. And I want to modify the way this button, what it's called. Let's call this Submit, and then apply some additional functionality in here. Maybe I want to put a checkbox here and make modifications to those components. I can make changes to any of the properties here within that particular widget. And I can select from many different options here.
So again, a drag-and-drop way to easily create these dashboards without needing to any type of programming language, and then on top of that, you can take all of this and then behind it create a callback, which will then execute some predefined command or script or anything that you've created on the back end. And that's what it's like to create a dashboard using MATLAB. So again, this entire dashboard, it can be deployed onto a web page so that you don't have to install anything on a local machine as well.
And now the last thing I want to point out about this entire system here is that it's actually logging everything that's happening here. So there's an audit trail of everything that we've been doing here, from creating different components, making edits or modifications to the models themselves, any updates that are happening on the back end for us.
All that information is being stored within the log and audit trail here so that I can go back at any point in time in conjunction with the versioning of things like SharePoint and the version control of source code using things like GitLab and identify how a model behaved and rerun it, that model, at that point in time so that everything is being tracked. I have full traceability, in this case, model lineage.
So in a nutshell, that is the MathWorks model inventory system. Going back to our slides here, just to summarize what we've seen, so, again, the model inventory system here allows us to automate tasks using the built-in workflow components to issue workflow issues to various people and then provides views using customizations to the dashboards themselves as well as the tables, and then all of this has drill-down capabilities so you can drill down and identify those key models of interest.
And because the model inventory itself is connected to a variety of different systems-- or you can have it embedded just all within the model inventory if you don't have systems in place, but most folks do-- this provides a one-stop shop for everything that you need for your model. And the most important thing, of course, is that you are able to, from a regulatory perspective, get those models approved and answer questions that regulators ask you. And because this is effectively like a time machine, you can look back at how a model behaved at any point in time and the associated documents and data and so forth.
And so with that, we have come to the end of our session. I want to thank you again for taking the time today to see how the MathWorks model inventory can streamline your model management practices. Thank you.