CI/CD 2.0: From Scripted Jenkins Pipelines to Process Advisor - MATLAB & Simulink
Video Player is loading.
Current Time 0:00
Duration 21:58
Loaded: 1.40%
Stream Type LIVE
Remaining Time 21:58
 
1x
  • Chapters
  • descriptions off, selected
  • captions off, selected
      Video length is 21:58

      CI/CD 2.0: From Scripted Jenkins Pipelines to Process Advisor

      Martin Römpert, Continental Automotive

      See how Continental uses the new Process Advisor app to rebuild and simplify its CI/CD Jenkins Pipeline, including JFrog Artifactory and Conan. 

      Published: 7 Aug 2023

      Welcome to the presentation CI/CD 2.0, From Scripted Jenkins Pipelines to Process Advisor. Let's have a short look at the agenda for the next 25 minutes. We will start with a brief introduction of the Continental Automotive Technologies' GmbH. After that, I'll describe the motivation for the introduction of Process Advisor for our CI/CD workflow.

      In the next step, we take a look at the history of CI/CD within MATLAB and Simulink within Continental Automotive. And we describe where we started and which customizations of the CI/CD pipeline were necessary over the years. After a short introduction of Process Advisor, we have a look at its use in the context of our internal project, modular software. The end of the talk will be a conclusion and an outlook.

      Most of you will think about tires if they hear the name Continental. And that's not surprising, because Continental tires is something everyone can buy for her or his car. Continental Automotive Technologies is an OEM. So these products are not for end customers but for car manufacturers.

      As you can see, we develop products for autonomous mobility, interior, and safety and motion. Within safety and motion, we have the areas dynamic and comfort, passive and integrated safety, and the area I belong to is breaks-- in detail, the electronic brake systems.

      To get an idea of what we produce, on the bottom right, you can see our actual second generation one-box system the MK C2, which integrates the master cylinder, brake booster, and control systems for ABS, ESC into a single and lightweight module.

      But let's start with the part of the presentation you cannot google and which is hopefully of more interest for you, as my company structure. So for those who are not familiar with the term CI/CD, I copied the definitions from Wikipedia. So the key aspect is that you are doing small changes and short cycles.

      I will show you how you can double the speed of your CI/CD system and cut your maintenance effort in half. So at least I can describe how we at Continental try to achieve that. Now let's dig into the details.

      How did the CI/CD journey start at Continental Automotive? Back in 2017, we set up the first prototype of a CI/CD pipeline based on Jenkins. On the bottom of this slide, you can see how the scripted pipeline looked as we started the rollout. As tools, we used Git as source control system, GitHub to manage the repositories, and Jenkins to automate everything, and of course, MATLAB, Simulink, and Polyspace products to implement the stages. The generated artifacts were stored in JFrog Artifactory.

      The switch from non-AUTOSAR to AUTOSAR Classic was the starting point for heavy customization of this Jenkins pipeline. First of all, there was the need of custom stages to handle AUTOSAR software component and AUTOSAR software composition-specific tasks. As the compositions grew bigger and bigger and the number of software components increased, Jenkins pipeline run took hours to days.

      So the second need was to shorten the CI/CD execution time by implementing a kind of dependency analysis. The goal was to only run the Jenkins stages for the models that changed. In parallel, as the maturity level of some software components was already high, there was the need to introduce maturity-level-dependent actions. For example, Model Advisor checks and the Polyspace Bug Finder rule sets should be more strict for models with high maturity levels.

      As you may expect, all these changes to an already running Jenkins pipeline caused risk of incompatibilities and high effort. But which options do you have to reduce this effort? What is the right alternative to scripted CI/CD?

      This is where Process Advisor comes into play. Let's see how Process Advisor works and have a look at some terms and definitions. Process Advisor allows you to model your build and verification process. You can think about this process as a number of tasks that needs to be done before you are ready to release.

      Process Advisor comes with a set of predefined tasks that allow you to run, for example, your unit tests or generate code from your model. Moreover, it has the possibility to implement custom tasks. The picture on the upper right shows you that every task has inputs and outputs-- the so-called artifacts.

      An example of input artifacts could be the models to test, which are defined using queries and the Model Advisor checks to run. An example of output artifacts are the check results. Every task defines its own assessment to determine if it's passed or failed. To organize multiple tasks, you can define relations, like "depends on" or "runs after," between the tasks.

      On the bottom right, you can see how tasks are visualized in the graphical user interface of Process Advisor on your PC. Here, I need to mention that this GUI provides links to all the artifacts that are related to the process and allow you to jump back and forth between them interactively.

      Dependency information is extracted from the so-called digital thread, a database that tracks dependencies between artifacts and changes to these artifacts. You can see that the Web View task does not depend on the model A only but on some data dictionaries and, of course, the process model itself too.

      Now that these dependencies and when a change happened, you can provide staleness information. In this screenshot, you can see, for example, the second task was successful in the past. You have a checkmark there. But the gray color indicates that some changes caused this result to become stale. The interactivity, digital thread, and staleness information are the big benefits of Process Advisor.

      You can execute Process Advisor on your local machine and on various CI/CD systems. But how should this work? What is the price you need to pay to be independent of your CI/CD system? The answer is quite simple. Bring everything into MATLAB.

      Now, what is everything? In our case, the top four things to bring into MATLAB are listed here. The artifacts handling, where we use JFrog Artifactory was brought into MATLAB by using system calls to the JFrog client executable. The package manager Conan, we use together with JFrog Artifactory, was brought into MATLAB by using system calls to the Conan client executable.

      To handle our repositories, we use GitHub, and we use the GraphQL client for MATLAB to interact with this system. Of course, we need to know the status of the Git repository on our disk to be sure that everything is unchanged. And we use the MATLAB Git client to handle this.

      Here, you can see one of the advantages of the MATLAB Simulink environment. It allows you to interface to almost every system you may find in your process. After this introduction of Process Advisor, let's have a look at our internal project, modular software.

      The goals of our modular software project was to provide a scalable, reusable, and AUTOSAR Classic-compatible environment for model-based software development. On the right side, you can see that the architecture we use to integrate our-- you can see the architecture we use to integrate our final software product.

      The lowest level is the interface level, which defines the input and output interfaces of each software components. And this really means AUTOSAR software components are defined on the unit test level. Interfaces are usually used by multiple components.

      Compositions-- and again, this means AUTOSAR compositions integrate multiple components or multiple compositions and are defined on the levels where the integration testing is done. As you can see, this approach is scalable vertically and horizontally.

      The design decisions we made needed to fit the goals of the architecture of modular software. We decided to use separate Git repositories for interfaces, components, and compositions. The repositories should be managed using GitHub. Until now, this may sound familiar. And the design decisions may be obvious.

      Why we decided to use a package manager and Process Advisor together with CI/CD and to set up the prototype for modular software and how this helps us to solve the issues we discovered using our scripted pipeline will be shown in a short video sequence.

      We pulled the GitHub repository for component A on disk using project from Git. After that, we open the project. And we open Project Advisor-- Process Advisor. As you can see, there is a detailed analysis of the MATLAB project, and the digital thread is updated.

      Now you can see the Process Advisor graphical user interface that shows all tasks that have been defined for that project. This GUI is interactive. And if you hover over the I/O icons, you can see the outputs and inputs and dependencies of each task. Moreover, you can click on all the light blue elements, for example, to open the generated web view.

      The input artifacts for the web view task can also be opened via link. You can run all these tasks via the Run All button or specific tasks via the Play button of the task itself. You can now see the tasks that the actual tasks depend on are highlighted in blue. The Info button shows details about the task.

      I now run the generate web view task. And please have a look at the output that is generated in the command window. We will see the same output later when Process Advisor is running on CI/CD systems. Let's now walk through the tasks quickly.

      The first four tasks are custom tasks that extract version information, pull in Conan packages, and check the status of your repository. You may remember that the Git, GitHub, and Conan communication was pulled into the MATLAB environment to be able to execute it locally and to be independent from the underlying CI/CD system.

      By the way, the implementation of custom tasks solves our first issue because any user is able to add tasks to Process Advisor. And there is no need to involve experts for the CI/CD system. Check modeling standards is a Process Advisor default task to run the Model Advisor checks.

      As you can see, it was customized to run checks for the maturity level MRC 40. This was achieved by adding the task multiple times to the Process Advisor with different names and different rule sets as inputs. The component A, as an X file, was labeled with the MRC 40 label in the MATLAB project.

      And the query that provides the input for the task uses this label to filter the models by maturity level. This solves our issue of implementing maturity-level-dependent actions in a nice and intuitive way.

      The next two tasks are Process Advisor default tasks. And Generate Conan Package for Model is, again, a custom task that takes the model A and its dependencies-- for example, Simulink data dictionaries or reference models and subsystems-- and puts them into a Conan package.

      The version information determined by the version assistant is added to the package. And the package is pushed into JFrog Artifactory. Let's have a look at the JFrog Artifactory and how Conan packages look like in the browser. Here, you can see the nested structure of a Conan package. And you will find your model and the dependent data dictionaries here. We go on with a number of Process Advisor standard tasks and generate two more component packages for reports and code at the end of our process.

      The third of our issues in the past was the missing dependency analysis. I want to show you what happens if I change the model component A. So let me just change the gain block and save the model.

      As you can see, Process Advisor detects the change of a file in the MATLAB project and updates the digital thread. This causes all results that depend on component A to become stale now, which is indicated by the gray icon.

      The incremental build option of Process Advisor will execute outdated tasks only. And so the dependency analysis helps to reduce the time needed on your CI/CD system. Until now, everything was done locally. Let's see how the CI/CD integration works.

      For Jenkins, we use a simple Jenkins file in the root of our project. As you can see, the pipeline code is reduced to a simple open project and run process call. In the console output of Jenkins, you can see the same output that you get executing Process Advisor locally.

      For GitHub Actions, we use a YAML file to trigger the action. You may realize that you have the same simple open project and run process calls here again. In GitHub Actions, it looks like that.

      So unfortunately, this build was not successful. But if you click on the Build icon, you are able to see the same Process Advisor output that was already shown for local execution and execution on Jenkins CI systems.

      So now that we've seen the modular software prototype in more detail, it's time for the conclusion and outlook. So what did we learn during our six years of CI/CD journey? First of all, CI/CD systems are exposed to change.

      A well-documented process management framework helps you a lot to be flexible and compatible. You can use a lot of predefined tasks out of the box as a starting point and implement your own custom tasks and queries. The digital thread helps you to run only these tasks that are outdated and reduce the runtime of your CI/CD pipeline. Nevertheless, you are able to do a full run if needed.

      As mentioned, bringing everything into MATLAB is an additional effort in the beginning. But using the simple commands open project and run process as an entry point for your complete CI/CD workflow makes you independent of your CI/CD system. For example, the transfer from Jenkins to GitHub Actions took us only four hours and was done by setting up the worker and adding the YAML file to the repository that triggers the GitHub action.

      Using a package manager like Conan helped us to generate multiple packages as output artifacts for a component or composition. This and the version and dependency features of Conan allow model integration, code integration, and repo integration. That allows us to handle software components and software compositions in their own Git repositories and to pull in the packages they are depending on.

      Last but not least, the collaboration between MathWorks and Continental helped a lot to get things up and running. So what's next? In our prototyping setup, there's not yet much reuse of Process Advisor tasks and queries. One of the next steps is to create a library of already parameterized and customized tasks and queries. And of course, the custom tasks and queries we defined in the modular software context-- for example, the Conan package handling-- will be part of this library too.

      Some of them will be protected against user changes using p-code. The handling of variants of parameters and algorithms within AUTOSAR Classic is on our road map. MathWorks supports here by extending the functionalities to handle AUTOSAR variation products in the arxml.importer.

      The import of handwritten code and AUTOSAR ARXML files into Simulink for simulation and testing purposes is defined and will be implemented by MathWorks Consulting. MathWorks Consulting is importing us to handle complex device drivers too. Last but not least, we have to scale everything up. That means that we will need to handle more than 500 interfaces, more than 1,000 software components, and more than 100 software compositions.

      This brings me to the end of my presentation. Thank you for your attention. And I'm happy to answer your questions.

      [APPLAUSE]

      Related Products