Panel Discussion on ModelOps in Quant Finance - MATLAB
Video Player is loading.
Current Time 0:00
Duration 36:07
Loaded: 0.46%
Stream Type LIVE
Remaining Time 36:07
 
1x
  • Chapters
  • descriptions off, selected
  • en (Main), selected
    Video length is 36:07

    Panel Discussion on ModelOps in Quant Finance

    Ben Steiner, Columbia University
    Oleh Khalayim, World Bank
    Matt Barnes, HSBC

    Modeling and analytics can provide tremendous value to organizations across all industries and disciplines, and financial services is no exception. Financial institutions are investing significant time and capital to build modern analytics capabilities. While several of these institutions have teams with skills to build highly complex models, they face significant challenges in moving these models to production and getting buy-in from key stakeholders including business owners, internal compliance teams, and regulators.

    In this session, panelists from Columbia University, World Bank, HSBC, and MathWorks discuss how an agile ModelOps infrastructure can help organizations take their models from design to deployment in a cost-effective and timely manner. By effectively monitoring model performance to determine proactive actions, organizations can satisfy the needs of all stakeholders involved across the model lifecycle.

    Published: 6 Oct 2021

    Take it away, Siddhartha.

    Thanks, Stu. So like Stu said, I'm Siddhartha. I'm in the product marketing/product management side for the computational finance tools here at the MathWorks where I lead some of the responsible AI efforts in finance at the MathWorks.

    So I'm going to be moderating this panel discussion about model ops and quant finance. And what I'm really excited about is the varied backgrounds of our panelists today.

    Matt Barnes from HSBC brings his experience on the technology side. Oleh here from World Bank will bring a different perspective as an economist with extensive experience working with clients all over the world. Ben Steiner has a lot of experience in AI, the history of promoting decision science, MRM, Model Risk Management, at investment firms.

    And Stuart here is going to be a part of the panel as well. And he leads the finance product team here at MathWorks. And so he'll be able to share some valuable insight from working with a number of clients. So when we go around the table, I'll let you introduce yourselves real quick. How about we start with you, Matt?

    Hi, everyone. Pleasure to be here and meet you all. I'm Matt Barnes. So as Siddhartha mentioned, I work for HSBC. I've been at the bank for many years, but most recently, looking after technology for our global risk analytics function.

    We have about 200 people in my organization, and we are responsible for managing a set of platforms that our business use for building, packaging, deploying, and monitoring our risk models, particularly capital models in the wholesale space. And we've been using MATLAB products now for the past four or five years and implementing solutions in operational systems where we're bringing MLOps or ModelOps into the whole process of properly deploying models within our space.

    Thanks, Matt. Oleh?

    Hi, glad to be here. My name is Oleh Kahlayim. I work for the World Bank, and I'm doing results-based finance. And I'm an economist.

    And in my results-based finance program, I work on the last mile, but not MLOps last mile, actually the last mile of bringing electric wire or water pipe to the clients in developing countries across Africa and Asia. And I am a part of ModelOps that Matt would collaborate with, but on the business side.

    So I'm working, for example, with people like Matt at the World Bank. And many of those people are within the World Bank treasury. And together, we bring a MATLAB-based models into operations.

    OK. Hi, first of all, Thank you. Thanks for having us. Thanks, MATLAB. Thanks, Sidd. My name is Ben Steiner. My background is on the buy side. So investment managers and hedge funds.

    Initially, as a model developer on quant using MATLAB, then as a portfolio manager, then a model development team head, and most recently, as chief of staff working to bring a more model-based thinking to a discretionary asset manager. In a nutshell, that's me. And I teach applied modeling at Columbia, as well as an adjunct.

    That's great. Stu.

    Yes, my name is Stuart Kozola. And as Siddharth mentioned, I manage the computational finance area here at the MathWorks. I've been in the role at the MathWorks for about 15 years of variety of different areas from data science, financial mathematics to the deployment into large-scale enterprise systems.

    So I have a varied and variety background in kind of the ModelOps deployment from way back when we first started the area in the computational finance to where we're at today. So it's a pleasure to be here. I think it's a great topic, and I'm looking forward to the conversation with everybody today.

    That's great. Thank you all for the introductions. Before we get started, to the audience, please post questions in the Q&A panel, and we'll try to get to it in the last five minutes or so. I'll keep looking at it as we go along.

    So with that, let's get things off. And let's do that by maybe setting the stage with a discussion about what does ModelOps constitute? And what it means to each one of you. And if you could just speak a little bit to how it's different from MLOps. People tend to use these terms interchangeably. But if there is a difference, I'd like to kind of bring that out. So let's start with you, Ben.

    So on my side, I guess I see two parts of the model development life cycle. There's the bit that happens before the model logic has been defined and the bit that happens after the model logic has been defined. So anything that helps model developers create models faster is good. So things like common access to data, common tooling, shared tooling, shared resources, things that help developers create models faster.

    I would call MLOps, or others would call MLOps now because I think that's the fashionable term. In the old days, it was just called being a quant. I think that's the MLOps parts it.

    And I think the ModelOps part of it is the bit that happens after the model logic has been defined. So things like model governance, model validation, model monitoring, getting the logic into production, deploying things. So in my mind-- and I'm curious of what others think-- I don't think it's a clear cut. I think there's a lot of sort of buzzwords flying around.

    As I joked in the old days, a quant did all of these things. But I think now there's a little bit of a bifurcation between MLOps a bit before the model logic and model ops perhaps a bit after the actual productionlization.

    Great. Oleh?

    To me ModelOps is much broader than machine learning operations, operationalization. And I think I'm an embodiment of that breadth. In order for our people on the technical side to deploy the model, to make the model scalable, to get feedback, they work on improving machine learning, but they also work with the users in their organization, especially those who have interest. And the more people have the interest and are able to be early adopters of the model, the easier it is and the faster it is to adopt the model.

    Got it. So, Matt, I'm curious to hear from you on the technology side. So how do you define ModelOps?

    Yeah, so, I mean, MLOps and ModelOps. I suppose MLOps fits in the space that the modeler actually uses the toolkits, the environments that they use to build the model and the iterative processes that go around it. But ModelOps really, to me, is, once that model has been built, how can we ensure that it can be packaged and deployed down a pipeline of environments with a lot of monitoring in place?

    And that we can ensure that there's a repetitive loop where models can be recalibrated and run operationally so that teams that are running the operational systems can support it when it runs against the data, and that it's got considerations around performance and meet certain non-functional type of requirements. And traditionally, we've never really had the ModelOps part of what we have been supporting in the environment. Very much, the business built their model, and then suddenly had to come along, understand it, and rebuild it in an operational technology taking many months to do so.

    By doing the MLOps side of things-- oh, sorry, the ModelOps, even I get it wrong-- is almost bringing the DevOps culture that application teams have had to do in the past five to 10 years to the building and deployment of models. That's the way I see it from a technology perspective.

    Fantastic description. The way I see it, MLOps is almost a subset of ModelOps which--

    Yeah.

    --right? And it's a small part of it. But ModelOps is-- like Ben put it-- involves a whole lot more. And anything that gets into operationalizing a model-- it could be governance, it could be validation-- anything that's in the loop there fits the criteria. Stu, anything to add on this?

    Sure. So, I'd also give a different perspective slightly. I think, like Matt mentioned, ModelOps is really kind of a spin on the old DevOps. For now, they're modifying the DevOps processes beyond just deploying applications to actually thinking about the model.

    The models in terms of machine learning are now more complicated or require more data, often require online training several times. So it kind of broke the old DevOps pipeline. So now they're redoing the pipeline to take care of these new different models under the AI or machine learning space.

    But I think another way to also think about ModelOps-- and we used to call it taking research into production several years ago here at the MathWorks. But you really have to define ModelOps in terms of the context of the team, right?

    So I think a good example is an Excel spreadsheet, in many cases, is often a model that's managing a business. So it's a model that's in production, but it's not really deployed through a DevOps process, and it might be just on one person's machine where they bring in data, do the calculations, and then they make decisions based upon that, right?

    So ModelOps, I think, is a broad umbrella. It's really trying to talk about the whole process of taking model into production with the full support of an IT team. But we often to with a lot of customers that, they're a two-person team. They don't have a full IT team supporting them, but they still need the ability to adopt ModelOp practices, do model validation, and have compliance around the model because a lot of these models are important to the business do fall under SR11-7, but they're not deployed necessarily into a production environment that's being run with the technology staff behind it.

    So I think modern ModelOps should think about the team size, how the team goes from a person to two people to like an organization supporting it. Like, how do you scale it up into a full-fledged enterprise environment?

    That that's a great point, Stu. And in terms of scaling it up and actually implementing it in the enterprise, whose responsibility should be to own ModelOps, an AI model operationalisation, for example? Should it come from IT? Should it be the head of AI function?

    Also, another question to tack on to that. Should it be enterprise-wide? Should it be centralized, or should it be more federated? Ben, do you have any thoughts on this?

    Sure. Maybe I'll help in the first one, and then I'll defer the [AUDIO OUT] for the second one. I mean, firstly, who should be responsible for ModelOps? And I don't think there's a clear-cut answer. I think it's very much depends on the organization, the size of the organization, the people in the organization.

    So sort of bouncing off what Stu said about a team could be one or two people, or a team could be an enterprise. And so different organizations have different sides. So I'd argue that in some organizations, it shouldn't actually be an IT person or an operations person or a data science person at all, because sort of model-based design and doing things systematically is so integral to the companies or is becoming so integral to companies.

    And so it needs to be someone who sits above all of those. So ModelOps and how we do things and how fast we can do things and how efficiently we can do things is part of the direction that a lot of companies are going in. So who should lead ModelOps? Well, maybe someone who's got an understanding of all those parts.

    So someone who understands the IT part; someone who understands the operational flows; someone who certainly understands the business model of the company and what the business model of the company needs to be tomorrow.

    There was a great quote that Mark Anderson put out years ago about how software is eating the world. And so everyone jumped on this idea that every company is becoming a software company now. But more likely, model-based thinking is eating the world. And doing things systematically is sort of the way things are going.

    So who should lead ModelOps? Maybe someone like a chief operating officer, or maybe someone at a suitably senior level that could actually influence the rest of the organization. So part of it a change management role, I think, in some firms.

    So the risk is if it's punted to a functional role a bit lower down, it becomes very siloed either within IT or within operations or within data science. So certainly, those organizations going through change. It should be sufficiently high up. And that's my personal view.

    That's a fantastic answer. Anyone else? Matt, Oleh, any thoughts on this?

    Well, yeah, I mean, how dare I tell what a model should and shouldn't do. I think I've been burnt with that before. You need to be more efficient, need to introduce model ops. You know, how dare I? And I think you're right, it's got to be something that comes culturally down from above.

    And we are having-- we haven't gone through it in certain areas-- a sea change in terms of how models are developed. And where I've come from in the finance space within the bank is that modelers are just effectively build models. They analyze large amounts of data they come up with predictability based on that data.

    And then once they've done, that they've put forward effectively what they deem to be the model for approval, and then that's it. They go away and do something else. And then it's very much dropped over the fence for someone else to pick up. And the issue with trying to enforce that to change is that the model has got no responsibility in terms of making sure that model is operational, that it's monitored, that it tracks closely to what they say the model should be doing.

    And as a result, they don't really care when you come to try and convert it into its running operation. And they don't really care about the ML ModelOp side of things. So really what you have to do is, in terms of making sure that you have that top-down culture of top-down drive from the organization, is ensuring that the responsibilities of the modeling teams encompasses the full end-to-end cycle as well to say you are responsible for this.

    And as part of that, you can then introduce effectively efficiencies through operational aspects to make that life easier. And we've seen teams now have that accountability, responsibility in modeling space. And as a result, we can introduce these kinds of tolling. I think it's brilliant. Other areas where they don't have that get full gamut responsibility, it's much more difficult.

    Absolutely. And Oleh.

    It seems that it also depends on how much of the core of the business that relies on the model. Or is the model more in step with the overall business model and business strategy? Or is it something very new, very interesting, innovative?

    And for more innovative approaches, I think there's still a case for more federated policy of federated approach to governance and ModelOp and then its mediation. For something that the business is very sure that it's needed, and there is a central push, again, beyond the IT side of business in changing the culture, then the culture will carry it. And then I think the top-down approach and governance and clarity marching orders would be most appropriate.

    And let me give you an example from a completely different field, but I think it's interesting here. So National Institute of Health has a separate stream that is called science of implementation. And the only question that they are working is, why some amazing, interesting, potent drugs do not hit the market and do not save lives of people? Or why the adoption process is so long sometimes?

    So the statistics is that of only 20% of those promised drugs that have gone through prototyping get adopted. And the adoption process can take up seven years. And this is life and death, very often, life and death question.

    And even though they have the stream, they don't have a unified answer. In here, you can push the culture, but the culture also can carry you into model adoption. And there are many, many problems with deployment and inefficiencies of deployment, but I think on the spectrum of how much of a core of the business that depends on the model. And that will also play a role in answering our question.

    That's a fantastic answer, Oleh. And I think what I'm hearing is it's a combination, right? Culture needs to exist, and that needs to be top-down, but technical implementations may be federated, maybe centralized, it depends. And so that usually is the answer.

    So with respect-- we spoke about culture. With respect to the maturity of ModelOps, how much is cultural, or how much is technical related? So what does it mean to go from working in silos to implementing enterprise-wide ModelOps? Matt, do you want to go on first?

    That's a killer question. No. [LAUGHS] What does it take? I mean, it took us five years to get it right for my area. And I think it's got to be both cultural and technology. And you've got to make sure that you do both.

    We actually went in and did the technology first, naively thinking that there was a real desire to change the culture. And actually, it's lots of areas that didn't happen. And it made it difficult then to try and justify some of the investment that we made as a result.

    So what we're now doing for some of our new service implementations is making sure off the bat we get engagement from the modelers so that they're part of the process. So a typical example-- we develop some tools that we thought would be great for the modelers to start using.

    And we trained them up on it, but it was very much lip service that was paid to it because they didn't feel that they were part of the original definition of that toolset. So straight away, it's almost being seen as dumb to them. And really you've got to bring them along as part of the journey off the bat and say, look, we want to implement something that makes your life easier as a modeler.

    This is effectively the objective, and we want to be part of that journey with you. Can you help us understand how you see that happening for your particular use cases? And then we can build solutions around that.

    So you can't think of them in isolation. It's got to be together. And the reason it took five years is because we went down one approach, and that's a revisit again later on. If we went from the offset talking about how we want the modelers to come along with us and be part of that journey, we would have done it in half the time.

    That is great insight. Oleh, you have any thoughts on this? I you work with some clients all over the world. So maybe you can give us an idea, kind of the spectrum of implementations and model ops maturity that you've seen.

    Yes. And for example, I work with utility clients and, say, a major electricity company, a electricity utility in Zambia called Zesco. And they have business needs in deploying model ops they have in their particular topics of business. And they are experts-- would be no different from your experts, from experts at the World Bank.

    But they would still be part of the overall culture of adopting things. And that would determine, again, how the culture would allow or not allow people who are quants who use the models, but who do less in building the model to be part and to carry the project forward.

    On the other hand, the more people like MathWorks are involved, the more, I think, it creates sort of good vibes. And I think there is a space for more outreach for MathWorks and for the community that would include broader people-- a broader set of people, and that, at the end, would help the champions to bring more towards enterprise models.

    That's a great answer, Oleh. And we've seen this happen before, right? Silos exist. Mortgage models might have no awareness about a loan that's been taken out by the same customer. Data exist, but just different models don't know about it.

    So bringing things up to the enterprise, removing things out of the silos can actually have a business impact. So that's been seen. It just might take time for a lot of clients. And a part of this is culture.

    And Ben, real quick-- Ben and Stu, I want to hear from you about if you had a magic wand, and you could change something, and you could change it immediately in the world of model ops in the enterprise, what would you fix? Ben, maybe we'll start with you and then go to Stu.

    Yeah. I guess maybe a 30-second answer. I put the people in the culture ahead of the technology because they're harder to change. Programming people is much harder than programming machines. So if I had a magic wand, I'd want to program people to just get collaboration.

    What does it mean to collaborate with each other efficiently and without wastage? So a sense of just instantaneously a magic wand, make everyone collaborate, and just gel beautifully.

    That's a great one. I think I pretty echo the same thing. I mean, the technology exists to do a lot of automation and make things a lot easier in a variety of different formats. So I don't think the technology isn't there; it's just whether or not it's deployed.

    But I think it's unfortunate when we talk about model ops. The term ModelOps actually kind of silos teams in terms of models, modelers, and operations teams, right? And the same thing with DevOps or AIOps. By the way we talk about it, we're kind of siloing them.

    I think I'd prefer them to use a different term where they integrate both operations and the modeling team together. If you look at the principles of agile, it's about small teams really owning everything about it. So I think an agile team really should own the model development as well as the operations. And those teams should be integrated and focused on making sure they deliver these things at scale and speed for the organization.

    So at the end of the day, I think it comes back to the magic wand of making sure that teams can be coalesced and put together and really collaborate on things. And I think the principles of agile is a good magic wand to kind of throw at them because the foundations are there. If you adopt agile, make the small pizza teams, as you know Amazon and Bezos would say, I think you'd actually go quite a way to really changing how the organization does model operations.

    That is great insights, too. So we're getting towards the end of this session, and I want to get to a question or two. But I guess one last question for all of you. And I'm going to combine a couple of questions that I had in mind.

    So if you could-- in a minute, I'll just go around the table here for this-- talk about any trends, any changes that you've noticed in the last five years in this space. But also take it further and talk about what you expect the future to hold-- where will ModelOps, MLOps, AI in general be in five or 10 years. That would be fantastic. And Matt, how about you go first.

    OK, so I think the biggest change in the, past well, less than five years, probably the last two years, that I've seen that really influencing the direction of where the business is going is the ability to start using the power of external cloud providers. We've been very much traditional financial organization running in our data centers. And we've always had to be very mindful of the constraints that provides us.

    Now, we've been able to push quite hard. And being a highly regulated bank, it is difficult to get now some material data sets into the external cloud. And we're no longer really managing infrastructure as a separate thing that we have to worry about evergreening. And it's not really-- it's an IT problem, it's not a business problem. It's very costly.

    The infrastructure is just brought together with the solution. And it means that we don't have to worry about the whole evergreening side of things. And we can really scale up and we can do elastic compute like we've never done before. So it's almost unbounded in a way.

    And as a result, the way in which we approach how we implement services, it's completely different. And that's the thing that's revolutionizing what we can do within technology. I've got use cases that require me to run models once a month a million compute that I can run in less than 15 minutes.

    If I have to do it on premise, I'd have to have a massive server cluster setup just so I can run it in probably eight hours one night. But I have to keep it up and running all the time, taking air conditioning and cooling. And it's that really I see is changing things from a technology perspective in the last couple of years that's really bringing powers to model development, model execution.

    Ben, how about you?

    Well, I'll mention a couple of things. I mean, the one big thing we've mentioned and touched on already is enterprise collaboration. This idea that over the last five, 10 years, there's been more collaboration and more awareness of enterprise thinking, that will only carry on over the next five years.

    But I think I want to mention three other things about the future and where I think this is going and what things have evolved. Matt mentioned the cloud and the technology benefit of that. But from a third-party supplier, much more stuff has been and will be software as a service.

    So what that means is for an investment firm, we don't need to buy and have the capital investment in third-party products at the outset. Instead, we're subject to either a monthly or quarterly or annual subscription which is an operating expense.

    So from the executive level, it means that we're no longer making huge investments as a capital expense, which is an ongoing-- it's effectively like the car leasing model or the Netflix model. And I think more and more companies are going to adopt that. From our perspective, it's both good and bad, but it certainly changes the calculus a little.

    The second thing I wanted to highlight was certainly modelers have a tendency to focus on algorithms and formulas at the expense of model risk management in some places. Sometimes, model risk management doesn't have the visibility it should, and that's based on-- because when we teach mathematicians and modelers, we put a priority on the formula. We put approach on the algorithm and the programming. We don't give the same status to like when the model can be wrong.

    I can say that's changing because I know I'm teaching that at Columbia now. So I think the next generation of modelers that come through in five or 10 years will have that sort of better balance of the priority of model risk management. And the final thing I'm going to mention is that I think this session with the algorithm and the AI, whether it's deep learning or reinforcement learning or deep reinforcement learning or so whichever the latest greatest algorithm is, is going to sort of subside. It's an obsession with data.

    It's all going to be about the data. The algorithm is going to be sort of second-order importance to the data itself. That's the way I think we go.

    And I absolutely agree with that. There's no AI or machine learning without data. And if you don't focus on the data op side of things then, everything comes tumbling down. Oleh, some quick final thoughts?

    Continuing on this question, I think a great change that I have been observing in the past 10 or five years is that this conversation that we are talking about here in the panel, you can make also if you go for lunch with your colleague who is outside the model office. And this is a great shift.

    It's hard to predict when it will completely change the culture, but it's a very, very powerful trend that people get excited about if people want to understand it. People do want to understand the formula more than they want to understand model risk. But I think that it's very important for us to be able to capitalize on this trend and to bring more people into collaboration and to use it into the user space and to advance through the space and to those who adopt and share feedback, and at the end, create more robust ones.

    Thanks, Oleh. And so we have about a minute before we go into break. So I want to address a couple of questions that have come up. And I'll leave the second question for Stu.

    But one thing I want to ask, which is, although the management of model ops originates as part of the evolution of finance and financial institutions and corporate companies, have models more complex than banks, how do we apply this when they're given the number and specialization of human resources are not comparable? Anyone with any thoughts on this?

    Well, let me just take a first step. So I started talking about utility, electrical utility in a developing country. And those utilities have a lot of data, a lot of very interesting data. A lot of people are helping the financial side of the business. And what are the utility, let's say, issuing bonds or doing some other complex things can really benefits from ModelOps.

    And that way, the utility can be on the same page with the bank and with other lenders and with investors and with hedge funds and what have you. So I think that has definitely outgrown the financial intermediary section.

    Got it. Appreciate that, Oleh. And we do have a couple of other questions, but I'll have Stu answer that in the Q&A panel itself, given that we're a little bit over time.

    But, thank you, Ben. Thank you, Matt. Thank you, Oleh, too, for joining us on this panel. I think this was really eye-opening, an important trend to discuss. And it was great having you here.

    View more related videos