AI with Model-Based Design: Virtual Sensor Modeling - MATLAB
Video Player is loading.
Current Time 0:00
Duration 35:53
Loaded: 0.11%
Stream Type LIVE
Remaining Time 35:53
 
1x
  • Chapters
  • descriptions off, selected
  • captions off, selected
      Video length is 35:53

      AI with Model-Based Design: Virtual Sensor Modeling

      Overview

      Virtual sensor (also known as soft sensor) modeling is a powerful technique for mimicking the behavior of a physical sensor when the signal of interest cannot be directly measured, or when a physical sensor adds too much cost and complexity to the design. For example, when developing a Battery Management System (BMS) for electrified systems, having an accurate value of the Battery State of Charge (SOC) is a critical design element, and it is challenging to directly measure the SOC. AI-based techniques can be applied as alternatives or supplements to Kalman Filters and other well-known techniques. Another example is using deep learning to estimate motor position for control applications and eliminate the cost of sensor hardware. However, as you integrate AI models into these applications, they must be verified and tested with other parts of the overall system to ensure reliability and safety in operation. The AI models should also satisfy compute requirements when being deployed onto resource-constrained embedded devices.

      The examples of SOC in BMS and motor position estimation demonstrate the integration of AI models into system-level design, the execution of validation and verification activities, and the trade-off management between different deployment objectives. This process involves linking design to requirements and tests for full traceability, simulating the AI model within the larger system, and applying both simulation-based testing and formal neural network verification techniques. It also includes compressing the AI model, evaluating performance, memory, and accuracy tradeoffs, and automatically generating code for processor-in-the-loop (PIL) testing and production deployment on embedded systems, such as an NXP microcontroller board.

      The presented workflow offers an end-to-end solution for designing, training, validating and verifying, compressing, and deploying AI-based virtual sensor models to embedded processors within a single environment.

      Highlights

      • Design and train AI-based virtual sensors using MATLAB
      • Import trained models, such as from TensorFlow, into MATLAB and Simulink
      • Integrate AI models into Simulink for system-level simulation, verification, and simulation-based testing
      • Apply formal verification techniques to assert neural network behavior
      • Compress the AI model for memory footprint reduction and execution speedup
      • Generate library-free C code from AI models and performing PIL tests
      • Profile code performance and evaluate design and model selection tradeoffs

      About the Presenter

      Tianyi Zhu is a product marketing manager at MathWorks focused on Simulink® and AI. He helps engineers worldwide adopt Model-Based Design, integrate AI functionality into embedded systems, and leverage the power of modeling and automation to accelerate development processes. Tianyi holds an M.S. in Engineering Management and M.S. and B.S. degrees in Electrical and Computer Engineering, all from Carnegie Mellon University.

      Recorded: 18 Dec 2024

      Hello, everyone. Welcome to today's webinar on AI. My name is Tianyi. I'm from MathWorks. AI is a hot topic nowadays. It promises good results when it comes to solving tasks that or otherwise difficult or impractical using traditional techniques. Given the advances of AI, there is an increasing need for engineers to integrate AI into the systems that they're building to either enhance or complement the system.

      And today, we'll cover one of the applications of designing AI into your systems, virtual sensor modeling. Before we dive in, let me first introduce myself and go over the agenda for today's session.

      I'm Tianyi, and I'm a product manager at MathWorks, focused on Simulink and AI-related workflows. Today, we'll first have an overview of what it means to integrate AI into model-based design. And then we'll introduce virtual sensors as one of the applications you can implement with model-based design, including what is the virtual sensor and how it has been applied to solve domain-specific problems. And after that, we'll go through the key workflow stages for designing and deploying virtual sensors. And finally, we'll show you additional resources for learning more and how you can quickly get started.

      So first of all, what is model-based design? It is the systematic use of models throughout your development process. At a high level, models are used to develop the system architecture consisting of behavior models and functional specifications, which are associated with specific design components. And these components and systems can represent either the physical system or the algorithms and software that's being used to control or manage that system. As you build the components, you continuously test them, first, through simulation and then adding more rigorous techniques, such as testing, and you integrate these components into your larger systems.

      You might be wondering how exactly does AI fit into the system development workflow with model-based design. There are two main ways of integrating AI into the system models that you're building. AI can be applied to component modeling. There are high-complexity, physical systems that are difficult to model using first principles. And AI-based system identification techniques can be leveraged for characterizing system behavior. Or you have high fidelity physical models created using 3D tools such as FEM, FEA, or CFD tools, and these models are computationally intensive to run.

      In this case, you can apply AI-based reduced order modeling techniques to reduce the complexity, while preserving the model's essential behavior. As a result, we have models that are suitable for running fast system-level simulations at a larger time scale and can also be applied in real-time control design and hardware in the loop testing.

      AI can also be applied to algorithm development, giving advanced perception and sensing capabilities to systems, enhancing traditional techniques when it comes to planning and controls. And today, we'll focus on virtual sensors.

      A virtual sensor can be used to estimate signals of interest that a physical sensor cannot directly measure or when the physical sensor is too costly or complex to implement. Another emerging use case is to provide a backup for the physical sensor in case the physical sensor has a heart failure or becomes unreliable during its operation.

      A well-known example of a virtual sensor is for estimating battery state of charge. This is a quantity that is very hard to measure empirically without elaborate lab instrumentation. What we can do is to collect battery data from the lab and use it to train an AI model for the virtual sensor, which takes a set of inputs, performs inference, and produces an estimated quantity. The advantage of this data-driven approach over a traditional method, such as Kalman filter, is that it does not require or assume any physical model of the battery.

      Another application of virtual sensors is for motor control, such as field-oriented control, FOC. An FOC algorithm requires real-time rotor position feedback to implement speed control, and traditionally, these algorithms rely on physical sensors such as a rotor position sensor. However, a sensorless position estimator, by using virtual sensors, can help increase cost effectiveness of the overall design.

      In this case, we have a shipping example that shows how you can design, train, and test an autoregressive neural network to replace the position sensor. This example shows an alternative approach to data collection. In a battery state-of-charge estimation example, the data is gathered empirically from lab experiments. In this case, the data is generated from a first-principle based simulation model of the motor and being used to train the virtual sensor. You can then simulate and deploy the Simulink model containing the trained neural network to actual embedded hardware and spin the motor.

      These two examples are only the tip of the iceberg when it comes to the potential of integrating virtual sensors in engineered systems. Companies across industries have successfully adopted model-based design for designing and deploying virtual sensors into production systems.

      Gotion developed a neural network based virtual sensor for estimating state of charge of electric vehicle battery packs. The design strikes a balance between performance, memory usage, and accuracy and met all of their deployment objectives. Mercedes simulated a piston pressure sensor with deep neural networks and deployed the quantized network on an automotive electronic control unit.

      Coca-Cola developed a pressure sensor with machine learning for their beverage dispensers. They reduce the footprint of the AI model and deployed it onto an ARM Cortex-M microprocessor. This project helped them eliminate the need to retrofit their dispensers with costly physical sensors. Poclain Hydraulics designed a motor temperature virtual sensor using deep learning and deployed it as an embedded application.

      The main value that Simulink provides for virtual sensor modeling is being a system-level design and simulation platform. As we mentioned before, the virtual sensor does not exist in isolation. It has to be integrated and simulated with other components and the rest of the system.

      An example of a battery model, we have a battery plant model that describes the physical characteristics. We also have the battery management system VMS, which has control algorithms, combining supervisory controls and closed-loop controls for performing functionalities such as cell balancing and thermal management. The virtual sensor model for state of charge estimation is one part of the system that can be plugged right into the VMS for overall testing and verification.

      So when we talk about the virtual sensor workflow, we're actually talking about a more broadly applicable workflow for the design, simulation, and deployment of embedded AI algorithms. And now, let's briefly look at the different stages in this workflow. It starts with data preparation. We have domain-specific toolboxes for pre-processing the data.

      The next step is AI modeling. We provide flexibility of choice. You can either use modeling and training capabilities in MATLAB, or you can import third-party models. After that, you can bring the trained AI models into Simulink for simulation and testing. And once we know the design works well in simulation, we can then get ready for deployment. This includes applying model compression techniques to reduce model size and speed up inference.

      The next step is code generation for embedded targets. The reason that I didn't say it's the last step in this process is because you need to profile the code and determine if the code satisfies all of your deployment objectives. You might need to go back to the model to perform more trade-off analysis and make your changes accordingly. The orange arrow shows where model-based design plays a significant role in this workflow.

      So now, let's go through each stage and use battery state-of-charge estimation as an example to show how you can use MATLAB and Simulink for accelerating virtual sensor development. The very first step in this workflow is data preparation. You need clean and well-formatted training data sets to ensure high quality of the AI model. However, data labeling, cleaning, pre-processing, as well as feature extraction from the data can be a time and labor intensive process.

      Since 2017, we've been investing heavily in building low-code and no-code approaches in MATLAB and other domain specific toolboxes that help you automate this process as much as possible and save time preparing your data. This includes different types of training data, whether it's image data, video data, tabular data, or time series signal data.

      For the purpose of the demo, we're going to use data that was collected experimentally and that has already been preprocessed. And this data includes current, voltage and temperature measurements from the battery, the calculated SOC values for the response, and also, moving average values for current and voltage. These additional variables have been computed to provide the AI model with temporal information about the past. We will need these additional variables for AI models that cannot look or remember at past values. As for how to label the response, it was done by integrating currents. This simple and computationally efficient method is called Coulomb counting.

      After the data is prepared, it's time to consider different options for AI modeling. And today we'll introduce three of them. You can train the model in MATLAB statistics and machine learning framework. You can import models designed and trained with a third-party framework. Or you can train the model using MATLAB's deep learning framework.

      For creating classical machine learning models, such as KNNs and SVMs, the Stats and Machine Learning Toolbox offers low code apps for interactively exploring the data, choosing pre-built algorithms to train and validate the models and comparing results. There are two apps for doing this, Classification Learner app and Regression Learner app.

      We've also built automated machine learning, or AutoML, that runs a wide selection of different models and automatically selects the best model and its associated hyperparameters. And this makes machine learning easy and accessible for everyone, even if you're just getting started.

      For deep learning, that the broader AI community is incredibly active and new models are coming out all the time. Because we support model import from TensorFlow, PyTorch, and other frameworks, you have access to these new models and can work with them directly within MATLAB and Simulink.

      We have the Deep Network Designer app, where you can import and visualize pre-trained models created in other frameworks. This helps you gain a better understanding of how the network was set up in the very first place. Or you can create a model from scratch in MATLAB by using a drag and drop, layer-wise assembly approach, and also analyze and export the model for use later.

      There's also an app called Experiment Manager to help you experiment with different AI models. And here's a short video highlighting the different capabilities within this app. This is where you can explore the different hyperparameters that are associated with your AI models in your experiments.

      You can see some plots to monitor the training progress of the different AI models. You can use MATLAB code to define your own experiments and be able to customize these experiments. And finally, you can keep track of all of your work. And after the training is complete, you can export the model results.

      So after you've built your AI model, whether it's trained in MATLAB or other environments, how do you integrate it into Simulink and run simulations? And the answer is, Simulink blocks specifically designed for integrating AI algorithms.

      And there is a core set of AI blocks which leverage Deep Learning Toolbox, Statistics and Machine Learning Toolbox and System Identification Toolbox. In addition to going through the import process, you can also use code execution blocks to bring Python-based networks directly into Simulink for simulation-only workflows. And there are also other domain-specific toolboxes that include AI blocks, including Audio Toolbox and Computer Vision Toolbox.

      So let's dive into a quick demo on importing a third-party network into Simulink and running simulations. In MATLAB, you can use the App's Gallery to find your favorite apps. In this case, we'll look for the Deep Network Designer app and launch the app from there. This app, as you can see here, enables you to assemble deep neural networks using layers from the library or import networks from other frameworks, like TensorFlow.

      You can simply hit the Import button, and here, you can select the TensorFlow model files from your local file directory. And the app will automatically analyze and convert the model. And once the network has been imported, you can visualize and also inspect the network structure. In this case, we have a multi-layer perceptron model. And what you can do is that you can check each layer's exact specifications.

      And after that, you can use the Export menu to export the model to Simulink. The network information will be automatically saved in associated MAT file, and you can specify where to save this file. And after that's done, you will have a new Simulink model automatically created, and the predict block is inserted into the new model. So this block represents the neural network.

      And for data inputs, you can add other blocks such as from workspace block, and you can visualize simulation outputs using a scope block. And then you interconnect the blocks with signals to complete this very simple model.

      So if you use MATLAB live scripts, you can prepare the input data for simulations by making that data available in a MATLAB-based workspace. And then in from workspace block, you can configure the workspace variables to use for the input data. And then you can do something that's very similar with the ground truth data that you want to visualize in the simulation.

      You can also specify the input data formats so that the Simulink block understands how to process the data. And after that, you're ready to run the simulation. So it's going to take some time for the model to compile, and after the compilation is done and simulation runs to completion, you can double click the scope block to see the simulation results.

      Here, the blue line shows the ground truth data, whereas the yellow line represents the predictions from the AI model. And we can also turn on the legend to have a better distinction between the two signal lines. And overall, we can see that the AI model captures the trends very well, reflecting the battery's discharging and recharging cycles.

      And here is what the whole system simulation looks like after we integrate the AI model into the battery management system, which is the VMs. We see the VMs model on the left and the plant on the right. And when we run this closed-loop, system-level simulation, we note that the extended Kalman filter method, as well as the neural network-based method, perform really well when compared with the ground truth data. However, this isn't the case with results highlighted in yellow from a method called Coulomb counting, which is an overly simplistic method for state of charge estimation.

      So that's it for simulations. When it comes to designing complex systems, such as the VMS, a common challenge that engineers face is managing requirements, design, and tests. They want to whether the AI requirements are implemented, how they are tested, and whether traceability is established between artifacts from different development stages. A model-based design provides a digital thread that helps you link requirements to your design models and to your tests.

      And here, we can see the Requirements Editor, which provides an interface for central requirements management. This interface enables you to visualize, edit, and manage requirements and links to other artifacts. Here, we can see the requirements view, which includes different visual elements such as a hierarchical list of requirements, a list of lower level-supporting sub requirements, the higher level requirement, a text area for textual specifications, links to tests implemented with MATLAB or Simulink, a column showing whether the requirement has been verified by running the associated tests, and also, a right-click menu that allows you to directly execute your tests from the Editor.

      When it comes to testing, I mentioned that you can link tests implemented in either Simulink or MATLAB. For Simulink tests, you can perform baseline testing to assert the accuracy of the AI model. Baseline data is what you collect from the lab, including inputs, which are operating conditions of the battery and actual battery state-of-charge information. With Simulink test, you can compare the ground truth data against the AI-based prediction results that you collect as the simulation output.

      And this is what the baseline testing results look like. In the top plot, you have a ground truth curve in blue, the predictions results in orange, and you can visualize the upper and lower tolerance bounds. In the bottom plot, you can see the differences and how they all fall within the tolerance levels.

      Another important consideration in testing your AI models is the notion of equivalence testing. How do you make sure that your model behavior is consistent as you take it through different development stages? When you go from desktop simulations to software-in-the-loop, processor-in-the-loop, and hardware-in-the-loop simulations? And the answer here is back-to-back testing.

      We can use different simulation modes to run the same model, check and compare the outputs, and make sure that the model behaves in the same way. So that's all about simulation-based verification. These techniques are taken from model-based design and are still applicable to AI-based models.

      Another component of AI verification is using neural network-based formal verification methods. This is an area of active research and development. And we released the Deep Learning Toolbox Verification library to support you with your verification workflows.

      One aspect of the network that you want to verify is its robustness. For example, if there is a perturbation in the inputs of the network, would that cause significant deviations in the output of the network? If there is a deviation happening, can we assert lower and upper bounds of the output deviation? Using a MATLAB test and the estimate network output bounds function in the Verification library, we can assert the robustness of the network against different input perturbation levels.

      Another major safety concern of deploying AI models into production is how will it react to input data in the operational environment that the model might not have seen in training. Especially when the input is significantly different and out of distribution from its training data. Consider a scenario where an AI model trained to classify images of cats and dogs encounters a picture of a bear in its operational environment.

      How can we implement a mechanism to flag such inputs as significantly different from the samples it has seen during the training process? Again, the Deep Learning Toolbox Verification library offers functionality to establish safety guardrails around the AI model, allowing for human intervention or an auxiliary system to cross check results when the AI model sees an unfamiliar input.

      So how does the implementation of out-of-distribution detection look like in Simulink? We have input data, which is fed into the AI component, which predicts the state of charge, or SOC, based on the input. And simultaneously, the input data is processed by the out-of-distribution detector, or OODD, which identifies whether the input is within the expected distribution, which the detector has learned from looking at the distribution of the training data set. The integration of such a detector enables detection of anomalies in the input data, enhancing the overall reliability of AI predictions and dynamic environments.

      And after rigorous AI testing and verification, the next step is deploying the model onto hardware for running inferences. Running large AI models requires a modern data center or compute cluster that has huge pools of memory, high throughput compute, dedicated grid connection, and even backup power generators. These highly scalable and resource intensive clusters investors support massive training and inference of AI models with billions of parameters.

      As we move down to desktops and embedded processors, the memory, performance, and power consumption become more limited, and that imposes constraints on the size and the complexity of AI models that we can run on these resource-limited devices. On one hand, we have limited hardware functionality. On the other hand, we still want to maintain good throughput, keep the model as small as possible, meanwhile, having high accuracy when running inferences with the AI models.

      And now, the question is, how do we reconcile the tension by striking a balance between all of the deployment objectives when it comes to embedded devices? And the answer is model compression, which helps reduce model size and speed up inference. This is essential for bridging the gap when you go from simulation to hardware deployment.

      We categorize compression techniques into two main types, structural and data-type compression. Structural compression involves analyzing the network's architecture to remove insignificant weights and connections. By using pruning and projection techniques, you can reduce complexity of the neural network without compromising its predictive power.

      And data type compression focuses on optimizing data representation, using smaller, more efficient formats for storing network weights. And there are techniques such as quantization and fixed-point conversions that can be applied, leveraging data types like bfloat16 to replace the full 32-bit floating point representation. And this approach effectively halves model size while maintaining accuracy, especially in deep learning models that are resilient to precision loss. Additionally, the industry is increasingly adopting even smaller data types, such as 8-bit integer representations and fixed point formats.

      And there is a wide variety of tools and resources for you to effectively apply compression techniques based on your AI models. We have a documentation page that has an overview of different compression techniques, how they compare with each other, and what kind of networks do these compression techniques support.

      We released new analysis capability in the Deep Network Designer app for showing how much compression you can potentially achieve with different techniques. There is also a Deep Network Quantizer app that walks you through the quantization workflow step by step.

      And here we have an example of compressing an LSTM network using either network projection or bfloat16 data type conversion, or a combination of both. We can see that by progressively applying structural compression and data type compression, we're still maintaining the same level of accuracy for making predictions, but we have significantly reduced the size of the model. And we've also reduced the inference latency, which means that we've sped up the execution of the model.

      And after you've compressed your models, you can automatically generate efficient code from your AI models to target different hardware platforms, whether they are CPUs, GPUs, or microcontrollers, or FPGAs. And you do all of this without having to rewrite the original model. You can configure the code generation process so that you can leverage specific compute libraries provided by different hardware vendors for accelerating computations on their specific devices. Or you can generate library-free code that is portable across devices.

      When it comes to deployment onto hardware devices, one important question to ask is, how do you profile performance of the model as well as the generated code to gain a better understanding of its memory usage and execution speed? You can use the estimate network metrics function in MATLAB to see a ballpark for the total memory usage of your network parameters. For measuring execution time, we have Code Profile Analyzer app from Embedded Coder that helps you get the execution time profiles for benchmarking the code that's being generated from different AI models.

      The next step in this workflow is to perform processor-in-the-loop, or PIL testing. In this stage, the objective is to verify the code behavior on the actual hardware target. We execute the control algorithm as generated code running on an embedded processor, and perform a closed-loop simulation with the plant model that is simulated in Simulink on the desktop. In this example, we'll deploy the entire battery management software, which includes the virtual sensor component for estimating battery state of charge.

      And here's a quick demo to show how this works in action. So what's happening is that there is an I/O connection between Simulink running on the desktop and the processor board that you can see here on the top left corner. Once you've configured your hardware and model to run in PIL mode, you can hit Run SIL/PIL button, and this will kick off code generation, and it will deploy the code onto the microcontroller.

      And once ready, simulation starts. And what you're seeing here is the VMS running on the processor and communicating back with the plant that's being modeled in Simulink. So you're seeing the Simulink model running and communicating with real hardware.

      And after you've wrapped up the simulation process, you can get a Code Execution Profile report where you get to see all the execution metrics for the model, how long each task takes, and you'll be able to compare different AI models and select which is the best to take to production.

      And speaking of model selection, there is a wide variety of models, both AI-based and non-AI-based, that you can choose from for building virtual sensor applications, each with its pros and cons. The first step is to evaluate whether AI is the best fit for your specific use case. For instance, in virtual sensor applications, Kalman filters are really popular, but they require a physical model of the component, which might not be feasible for you to build or obtain.

      For AI-based techniques, we have classical machine learning methods that are typically faster to train and run, smaller in size, and easier to interpret and understand, but their accuracy might be compromised, whereas a deep-learning based method might be more accurate. But it's also more computationally intensive to train and run, bigger in size, and more or less a black box approach when it comes to interpretability. No matter which model you end up using for production purposes, MATLAB and Simulink offer tools for you to explore the trade offs of these different options and maximize model performance, whether it's AI based or not.

      There are many other use cases when it comes to integrating AI into engineered systems, for example, using AI for object detection, anomaly detection, reduced-order modeling, reinforcement learning, and learning based control. We have examples, user stories, and webinars on all of these topics.

      You might wonder why use MATLAB and Simulink for virtual sensor modeling or more generally, for embedded AI. Our tools not only enable you to train AI models, but also integrate and simulate them in complex, multi-domain engineered systems. This integration helps you understand how AI components function within your larger designs.

      Whether you train AI models with MATLAB or import them from TensorFlow and PyTorch, our tools facilitate the use of different models in your projects. We focus on engineering applications like virtual sensors, reduce-order modeling, visual inspection, predictive maintenance, among many others, and we provide lots of tools and resources specifically designed for engineers working in specific domains, such as easy-to-use apps that guide you through each design stage. We have shipping examples related to different industries across aerospace, automotive, medical devices, industrial automation, among many others.

      And last but not the least, MATLAB and Simulink offer strong support for hardware deployment of AI models with capabilities around model compression, verification, and cross-platform code generation. It's really powerful to deploy the same model to different hardware platforms. And we work closely with different hardware vendors to make sure that all of the devices are well supported.

      As we wrap up this session, I want to show some additional resources. First, we have overview content, including a web page and a webinar on the broader topic of AI with model-based design. Next, you can find a hands-on documentation example on battery state-of-charge estimation that covers many workflow elements that we went through today, from requirements definition all the way through to code generation.

      Finally, there is another virtual sensor example focused on rotor position estimation for motor control applications. We also have a webinar on this topic, as well as a shipping example with MATLAB scripts and Simulink models that you can try out yourself.

      And now, let's open up the floor for questions.

      View more related videos