Effective Integration of MATLAB and Simulink into a CI Pipeline

Well, welcome, everyone. My name's Adam Sifounakis. I'm a product manager at MathWorks. I focus on the software developer workflows within MATLAB. So that's everything from continuous integration, to projects, to some of the software developer tools we have in MATLAB, as well as the core language features that support this. I've been at MathWorks about nine years. I spent five of those in the field working with customers like yourselves, primarily focused on aerospace and defense, but I also supported automotive customers. We also have Wensi Jin, the automotive industry manager, on the webinar. And we'll be here for your Q&A after the presentation.

So our presentation today is about effective integration of MATLAB and Simulink into a pipeline. So we're going to start off with an overview of CI-- a very brief overview-- and some of the things to consider, some of the terminology, and some of the setups. We're going talk about some CI use cases and applications of when and where you'd use them, and then give you an example of how to integrate MATLAB and Simulink into your CI pipeline. So here we'll focusing on Jenkins as the example.

And then we'll talk about a couple of the future looking things that we're working on that will help improve your experience with continuous integration. And then I'll have some summary and resources-- actionable resources at the end. So with that, let's hop in.

So what is continuous integration? So it's the practice of automating the building and testing of software. So here, this is a vague diagram that shows up in quite a few places, starting from the middle and going to the top left. It's where you start a plan, design, build, test-- got that continuous integration, continuous delivery section in the middle. And that transitions over to-- from the developed side to the operate side-- so the deploy, the operate, the monitor, and the feedback. And all of that feeds back into the software you're developing, so you're just continuously growing your code, and your features, and your development.

Now, CI-- continuous integration-- or CI/CD-- continuous integration, continuous delivery-- isn't just that middle item. It's actually several of these buckets, everything from the building-- automating the building, the testing, the delivery, and the deployment of your software and your models. Now, there's lots of benefits here-- everything from a consistent, repeatable environment to exercise your code in and test your code to automating testing of the latest changes. So you're always testing what is being developed, and nothing gets left behind.

You're improving the speed at which you develop by testing early and testing often. You're identifying integration issues quickly. Earlier you're identifying, the less expensive they are to fix, before they get deeply embedded in your systems. And you're creating a trace-ready system so that, when you do find an issue, you can find out, where did it come from? How long has it been here? Who added it? They might be the best one to fix it. All of that now gets documented and tracked. This becomes critically important as your organization scales or as your applications become mission-ready or safety-critical.

So it's not just continuous integration. You guys might have heard the term CI/CD, and that could basically translate to continuous integration and continuous delivery, or even continuous deployment. And each one of these is a step further along this route, and it's supporting these industry trends that we're seeing of the automate everything, 100% uptime, and deploy on the fly.

So that deploy on the fly, especially in the automotive world, starts to take that form of those over-the-air updates that now, once it's been built, and tested, and delivered, you can immediately switch the running system to the new software. That's that continuous deployment, going from your organizational-- organization's development directly to the target system. And all of that is now becoming automated.

Now, the other element for continuous integration is, are you doing local continuous integration on a continuous integration server or are you doing it in the cloud? So are you leveraging some kind of CI service? Now, local CI-- this is where all of the machines are located on premise. So you control the hardware. You control the software. A lot of the CI service software, like Jenkins, Bamboo, GitLab, and TeamCity-- some of these are just freely available, so it's one of the benefits here. And you get full code privacy. None of your code ever leaves your organization.

And some of the cons in this area are just it's up to you to install and maintain that hardware. If something breaks, you've got to take care of it. It's harder to scale, because now there's complexity involved in scaling all of these hardware systems. And it's more expensive to scale. You have to buy all of the hardware for this.

Now, the flip side of this is cloud services. That's where everything is taken care of by somebody else. So here we've got Azure DevOps, GitHub Actions, CircleCI, Travis CI are the major players in the space. There's no installation needed, no-- the scaling is really easy because they've got a pool of hardware, and they actually support you. So if something breaks or isn't working, part of the cost here is support from those companies.

But ultimately, those costs can add up, especially as you scale higher and higher. And your code technically isn't fully private now. These organizations try to make sure that they maintain all privacy, but if you're really concerned about your IP going anywhere except your organization, some people get leery in this area. But by and large, we're seeing massive industry trends shifting in this direction.

And then there's that middle bucket, where technically, it's Cloud CI, but you actually have local agents. So the orchestration, the load balancing will happen by the Cloud CI service, and they will distribute the work to machines that you own and that are on prem. And what are the benefits here that you actually have control of that hardware. So if you need to connect it to custom hardware that isn't available, let's say, through GitHub or through CircleCI or Azure, you can do that. So your proprietary hardware, your very custom developed hardware-- that's accessible.

And then your machines can actually connect to your private network, whereas you might not want to open direct cloud access to that private infrastructure. But now you get some of those cons again you have to build and maintain that hardware. But the software tends to be delivered from those companies. So these are the three main approaches to CI. There's everything is local-- you take care and you use your own IT-- to everything is remote, and then that hybrid model of part of it is remote, part of it is internal.

And you can also have cloud agents in that middle bucket as well. So we try to support all of these. So on prem, we've got our standard licensing that takes care of that. That middle bucket-- it's still that standard licensing for the local agents. And if you're trying to migrate to the cloud, and leverage these services, and scale up, we have work in that area, so please reach out to us directly if that's where you're going. And like I said, we're seeing significant players in the industry moving in that direction.

Now, how does CI apply to your workflows? So I'm going to represent this in a pyramid where, as we move up, these are typically the things that take more time, and they're more difficult to do by hand. But as you go up, you're getting more and more value out of this. So starting at the bottom is unit testing. It's the easiest piece to do. Everyone does it, and you just-- you start automating this using continuous integration.

Moving up to integration testing, the earlier, you do integration testing, the earlier you catch issues, and the earlier-- the faster you can iterate without having to re-architect your entire system. Regression testing-- so as you build this out, you want to make sure that it just continues to work as you develop new features, or new models, or new capabilities.

Now, taking it a big step forward is that code generation and functional equivalent verification-- so going from your virtual model to deployable code onto target hardware, making sure that those systems behave the same. And then deeper than that is that static analysis using formal methods. This is often very, very time-consuming to do, but you get a lot of value here, because it's proving the absence-- proving that your code's going to behave the way you expect. Now, all of this can be built into a CI pipeline, so it just runs through this battery of tests to make sure that you catch as many issues as early as possible.

Next bit is talking about some of the myths around when and where you can use virtual modeling or integrating continuous integration. So one of the things that we've heard quite a bit is, I can't do system simulation on a two-week sprint, as organizations are moving to that kind of agile workflow. The reality is you can. So here we're looking at a typical process of you architect the system, develop, elaborate, build your unit-- run your unit test, generate code, integrate code, and system test.

Unfortunately, a lot of people think that they have to go through the whole process, and then they'll determine that they have a requirement problem near the very end. This is really, really too late to do that-- leads to long, highly repetitive cycles, and really delays your delivery, and time to market. By adding in that model integration early and that virtual system test early, you can actually start to iterate really rapidly at the beginning and determine that requirement problem very, very early. So your develop and validate actually shifts left, and now you can really quickly cycle through this, and then move on to implementation, and have far fewer problems here.

Myth number 2-- so demonstrations must be done on product. We hear this all the time. And virtual demonstrations are actually becoming the way to iterate quickly. You can't always have access to the hardware, and being able to do this test early will help you catch-- so here we'll break down the various phases. So looking at the beginning part, it's that validate the architecture. And oftentimes, you'll do that in CI/CD using just tests around your architecture specifically.

And then you move on to validating the requirements and demonstrating the system functionality. Now, here you'll have your battery of simple tests-- simple unit tests or system tests. And this is your main development loop. This is where you determined the biggest blockers in progress. From here you start diving more deeply into the components as you start implementing everything you've developed, and then moving on to demonstrating that the actual code, the code that's deployable to the hardware-- so not just the virtual models-- actually works and adheres to your requirements.

And then you move on to the vehicle demo. Breaking it down into these sprints lets your teams rapidly iterate and actually make progress much faster than they normally would, because now this whole workflow applies to tens or hundreds of teams, and they now integrate more seamlessly between all of these sprints.

So let's talk about integrating MATLAB and Simulink into CI pipelines. So here we're going to focus on Jenkins. So the assumption in this space is you have a machine already. It has MATLAB, and Simulink, and Jenkins installed on it already. Once you have those things installed-- because they're relatively straightforward to install-- we can now start integrating into Jenkins.

So within Jenkins, you can install the MATLAB plugin. Now, you don't need the MATLAB plugin. It just streamlines. It makes it easier to configure your CI system with MATLAB and Simulink. So within Jenkins, they have a plugin manager. You can search for MATLAB, check the checkbox, and hit Install Without Restart. And within a few seconds, the MATLAB plugin becomes available.

MATLAB plugin gives you a few options. So while you are creating a build, we've introduced new options here to make it really easy to configure your system to work with MATLAB and Simulink. So we have little options here that say, hey, use this MATLAB version. So you can tell us either where on the machine to find it or you can actually create a custom list of all of the versions of MATLAB installed on the machine, because you can have any number of MATLABs installed on that machine. So a lot of our customers are actually leveraging that to say, I want to run 2020b, or 2021b, be or 2017a. And this makes it really easy to make it accessible to anyone using Jenkins.

Now, once you actually create a build, you want to actually add a build step. Say, do something here. We've added two options-- Run MATLAB Command or Run MATLAB Tests. Now, here, it just says MATLAB Command, but this is anything MATLAB and Simulink. So we'll have just a little field that you can put whatever MATLAB code you want, and we will make sure that MATLAB is launched and run with these options, and we'll try to capture any outputs from that job.

If you just want to run tests, you can just say Run MATLAB Tests, and what we'll do is we'll automatically search the code base for MATLAB unit test framework-based tests, or Simulink tests-- we'll search for those and run them all for you automatically. So you don't have to tell us where they are. You can actually customize what happens when you run the tests. So here you can specify where your source folder is. You can filter test by folder name or by tag, or you can tell us which test artifacts you want to generate-- so everything from a PDF test report to TAP, JUnit-style, Simulink Test Manager results, or Cobertura coverage artifacts.

And that's it. So install the MATLAB plugin. Tell Jenkins where to find MATLAB. Tell Jenkins what to do. And that could be as easy as just Run MATLAB Tests. And then choose which test results you actually want to see, and we'll generate all of it for you. Now, like I said, I've been saying MATLAB as kind of a shorthand, but it's anything MATLAB and Simulink, and just MathWorks tools. So you can integrate MATLAB, Simulink, Stateflow, Simulink Check, Simulink Test, Simulink Coverage, Polyspace tools-- they actually do some-- a little bit more of that formal verification. And we actually have technical articles, so white papers that show you some of these workflows and walk you through step by step of, how do you integrate very deeply with some of the deep verification validation tools that we have, especially for model-based design?

Now, viewing your build results-- so you can view everything from test results. And we've got a variety of formats that we can generate for TAP, JUnit-style, Simulink Test Manager reports, Cobertura coverage reports-- so code coverage, model coverage. Test coverage history is captured by Jenkins itself, and even any custom reports that you generate or that our tools generate. So here you can see some of the MC/DC coverage for some of the models.

Just talk about cloud-based CI support-- so in the last year, we actually rolled out a free MATLAB and Simulink support for public GitHub repositories. So if you have a code that is publicly available on GitHub and you have a CI build that's on one of the supported platforms and is publicly available, and you don't use any of our excluded products in the service-- so those are the coders and the compilers-- you can have access to MATLAB and Simulink for free in a non-interactive fashion.

So you can actually run your unit tests or your regression tests for free. And the supported platforms are Azure DevOps, GitHub Actions, CircleCI, and Travis CI. So we have a repository that provides the configurations for you, so it's really easy to get started. So it's got a quick start guide, ready-to-use CI configurations, example MATLAB code and tests, and even CI badge examples.

Now, again, this is for publicly available code. So we get a lot of questions, can commercial customers use this free CI free service? Yes and no. So yes, it's not restricted by industry, and you can use this as an exploratory tool. If you're considering moving to the cloud, this is a really cheap and easy way to just try it out. Try out the various services. See what feels more comfortable and would actually be viable to adopt at your organization.

The reason why not to adopt this is the obvious one. You have to share your code publicly. This is of our community service that we're providing. You probably don't want to do that in the commercial space. You don't want to share your IP. But there are alternatives in this area. So our plugins actually support on-premise CI servers, so it doesn't have to be cloud-based for our plugins. Those first two buckets on that CI server diagram that I showed earlier can still leverage the plugins, so you can use an individual license dedicated for that CI server. And like I said, if you're trying to use cloud-based CI for private repositories-- so not a public offering like this one-- please reach out to us. We're doing some work in that area, and we can work with you to see how we can best support you as you transition.

So let's take a look at what we're working on in the future. So we're looking at releasing a CI support package in the near future, and this is going to provide you kind of additional capabilities to help you with CI and model-based design. So the CI support package is going to come with some prebuilt templates for model-based design pipelines. It's going to provide some workflow automation tools to help you streamline how your models are exercised in a CI system.

So with my desktop pre-qualification capabilities-- so you'll be able to leverage CI for your local development before you commit it to the full code base. You can actually pre-qualify that everything that you've made works, and now you want to commit and submit this to your primary developer. And we're going to add additional support for some of the popular CI systems. So Jenkins and GitLab or two that we're going to add even deeper integration for.

Now, model integration and source control-- so this is something that we've heard across the board loud and clear. So our models and some of our other MATLAB artifacts, our binaries-- they don't integrate of well currently. We are looking at direct support in GitHub and CI-- and source control systems to be able to do direct model diff-in in viewing.

So in the future, you'll be able to see one of these files and say, I actually want to of this. And by clicking on the file, it will take you to a view of the model diff from the previous run. So you'll be able to see what has changed and examine that model all without launching MATLAB or Simulink. You won't have to download the artifacts. You want to do any of that. You'll be able to just see it immediately.

And container creation-- so for those of you that are actually building MATLAB and Simulink containers to work in CI systems, today we offer two containers-- the MATLAB container on the MATLAB plus deep learning container. So it has some of the tools to support deep learning. We also offer the capability to build your own container using some of our reference architectures.

In the future, we're looking at streamlining this process. So we're going to be providing a streamlined, fully-scriptable installer that is significantly simpler than what we have today. So it'll be really straightforward. And we're working really, really strongly in the area of shrinking the container sizes. So that's work that's going to be ongoing, but we're looking at just shrinking the size of MATLAB to fit into smaller and smaller containers.

All right, so I went through quite a bit of information relatively quickly, so I just want to summarize a couple of parts of this. So continuous integration helps you develop high-quality software faster. MATLAB and Simulink can support you all the way through all of the stages of CI. We showed how some of this model verification can be done earlier in your pipeline, and then automated through CI.

And then getting started with CI is actually very easy with our plugins and our example codes. Like I said, you don't need the plugins. They just make it a little bit easier. If you'd like to do without the plugins, we can still do that. Definitely reach out to us. We're happy to work with you in this area. So we've got some example links here.

So you'll be getting copies of the slides sometime in the next week. And there'll be links to these code examples-- so the CI configurations, the-- an example of CI with Simulink, and then code coverage using third-party code coverage tools, like Codecov. Here are the links to all the CI plugins that we have, as well as the reference architectures for Amazon Web Services, Google Cloud Platform-- everything you might need for getting connected to these cloud services.

Our documentation-- so if you want to learn more, dive a little bit more deeply, our solutions page has links to pretty much everything I've shown here, but we also have these technical papers that-- they get very technical, and they just walk you through this workflow to help you get up and running and get your system configured to work. And we've got our great documentation and blogs-- so everything from the continuous integration documentation hub to our developer zone, where we have blogs by Andy Campbell and a few other people really diving into some of the nitty gritty for some of this stuff. So if you're really interested, I highly recommend that.

And the big thing here is reach out. One of the key aspects is that you have our support. So reach out to Wensi-- he's our automotive industry manager-- or your current contact at MathWorks. Let them know that you're looking for either model-based design support, continuous integration support, and they will connect you to the right people-- either someone like myself, or Andy, or Wensi, or Si, or Eric. There's quite a few people at MathWorks that are here to support you through this process and actually have deep engagement with you to help transition from your current workflow, current pipeline to this future automated system that everyone's moving to.