Scenario-Based Cosimulation of Autonomous Systems Using RoadRunner and CarMaker

So [INAUDIBLE] has explained how we can create different scenarios and variant scenarios from the seed scenario. So that will be an input for our co-simulation framework here. So before going to the co-simulation framework, I would like to talk on the complexity of the vehicle systems. As you see, the level of autonomous driving automation increases the complexity of the vehicle system traffic environment, and dynamic driving behaviors are increased.

So this poses us a new challenge to create a different validation methodology for testing these applications. We from Bosch Global Software Technologies, in collaboration with MathWorks, developed a modular and scalable platform that allows the [INAUDIBLE] loop simulation. So our integrated tool chain consists of fastest, which is an [INAUDIBLE] in-house developer tool. So this generates test scripts for a variety of automation tools like ECU test and robot test framework. So that combines the test management and automation tool for us.

We also have a carmaker and virtual test drive. In this, we are creating different scenarios and simulating with high fidelity vehicle dynamics. Instead of MATLAB, we are collaborating with third party simulator tools like ETAS COSYM and then DSpace, where we are integrating our virtual issues for our testing purpose. And finally, we have MathWorks RoadRunner for the creation of effective test scenarios.

So before developing the core simulation framework, we had a reference implementation for both HiL and SiL applications. Primarily for the HiL applications, we used a radar ECU that is integrated with the real time PC, and then the scenarios are simulated using the virtual test drive. But when it comes to the SiL, we solely depended on the car maker side because here there is no real issue present. We are validating the virtual ECUs generated out of different C++ and different source codes, basically.

But then why co-simulation is required? If you see the reference implementation previously, we faced few challenges on this, so primarily, the inclusion of unexpected objects. We want different objects to be presented in the scenario to have a complete scenario in place. Basically, I guess, previous talk, if you see the virtual simulation part. So we need to create different virtual scenarios before going for a real world test driving.

And we also faced some compatibility issue with the scenarios. We developed almost 700 plus scenarios in the virtual test drive for almost 10 different ADAS features. But then we are unable to reuse that in the Carmaker because there is a compatibility issue between the two tools. But when it comes to the RoadRunner, we actually solved that by importing those scenarios into the RoadRunner and exporting it to the Carmaker specific platform.

So the probable solutions would be running different scenarios, considering different properties, and then adapting the scenario to the particular simulator, where we need the co-simulation approach possible, and then reproduce as much as realistic behavior. Yeah, let's go to the scenario simulation. If you see this particular slide, you have two scenarios. One is in Carmaker. Another is running in RoadRunner. This one's running right. Yeah, it's the same scenario created in and generated in RoadRunner. We basically exported this into the Carmaker format and running it in Carmaker.

But if you see, these two scenarios are not synchronized. They are running with their own driving maneuvers. So RoadRunner has its own driving maneuver, and Carmaker has its own maneuver. We are trying to do a co simulation with a synchronized approach. Just remember this scenario because we are doing a co simulation for this only. Yeah. So scenario based validation provides an efficient way of virtual testing. So that allows increased development, helps in the development, and then decreased on-road testing.

So we take scenarios, which are generated from recorded data, and the variant scenarios generated from the seed scenario, basically. And then that has been given to our automation tool. This tool connects to the RoadRunner, which, in turn, generates a scenario compatible for the Carmaker. The connection between our automation tool and RoadRunner is using the scenario simulation engine that supports the gRPC protocol. We use the C++ and Python for this. Both are being used. And it connects to the Carmaker using their internal APIs. That's called APO communication.

And internally, RoadRunner and Carmaker are connected via middleware, where the co-simulation synchronization and the data exchange is happening with the same protocols, gRPC protocol, and then the APO communication. So this is for the co-simulation, but how do we validate the ADAS's use? To the same middleware our ADAS ECU is being connected via an interface. Here I'm talking about the virtual ECU, not the real ECU.

We didn't stopped over there. We also integrated system of system ECUs, like the braking ECU, steering ECU, and then the engine ECU. We already tested few scenarios with this. All of these four issues, along with the rest of the simulations are connected to the middleware, and they are running fine with the scenarios generated from the RoadRunner. So the key part of this co simulation is how the data is being exchanged, and what exactly is the data.

Carmaker provides a multi-body simulation. It has the various sensor models. It has high fidelity vehicle dynamics, and it also has different objects in the scenarios possible. So the objects inside, or the traffic actors inside the Carmaker is defined by the RoadRunner. So RoadRunner basically controls the traffic actors in the Carmaker, and the Carmaker eco vehicle data is being exported into RoadRunner. This allows an unplanned and realistic way of creating the scenarios.

And then we can have these examples, like if you see the congestion, or a dense traffic conditions, or slow or fast moving traffic, and then sudden lane changes, which you cannot-- you have to define each traffic active maneuvers in the Carmaker itself. But then with this co-simulation approach, you have a normal scenario in place, and then all the traffic actors, all these unplanned events, are controlled by RoadRunner itself. So we developed a client application. This is written in C++. It is connected to the RoadRunner using gRPC protocol, and then to the Carmaker using APO communication.

So there are certain steps what client will do. The first is it will initialize the SSE of the RoadRunner, and it will download the scenario to get what are the actors present in, or the traffic elements present in the RoadRunner scenario because this is the key part where it has to export all these traffic actors into Carmaker side. That way, what we do is, we will have the same scenario and same number of traffic elements in both RoadRunner and Carmaker.

And then after the first step, it will also initialize the Carmaker and get the ego vehicle position because this has to be fed into the RoadRunner in the initialized step only. And it also needs the time step because we need all these things to happen in the same simulation step. So that's why we will get the time step from the RoadRunner, and also from the Carmaker. So our client will exchange this time step along with the traffic actors between these two tools. So it follows like this. The data exchange happens between RoadRunner and Carmaker in every single step.

Along with that, we also need the feedback from the virtual ECUs. So Carmaker sensor data, whatever the object sensor, or the lane sensor, or the camera sensor, radar sensor, all these object data list and the lane list will be given from Carmaker to the virtual ECU by the client itself. And then the feedback of the brake, or steering values, or an acceleration values, that has been fed back to the Carmaker via the same client application.

So in short, if you see these two scenarios, we will send the RoadRunner timestep to the client, and also the Carmaker time step to the client. The data is exchanged between them for the time synchronization. RoadRunner will send the traffic position details. Carmaker gets the traffic position details. Carmaker sends the ego vehicle position, and then RoadRunner gets the ego position in the same step. So these all three things will happen in every simulation step synchronously.

So let's see the example. So if you remember the previous video where the two scenarios are not synchronized, are now being synchronized in this video. So the scenario will start from the Carmaker that, intern, starts the RoadRunner from Carmaker itself. I hope that's visible. You can see the turnover there where the ego vehicle is following through the traffic elements.

So this is simply the co-simulation between the two tools using one sample scenario we created. But you can create number of scenarios in the RoadRunner, and you can export this into Carmaker. And then you can run the co-simulation with many number of scenarios as possible. So if I go into the details of how the actor positions are being updated and how exactly the time is also being synchronized, you see there is an option in RoadRunner. If you are familiar with RoadRunner, there is an option to pause the simulation, and you can step forward simple, like in Simulink how you are stopping the simulation and going each step.

So RoadRunner also provides that possibility. So for each step forward, I'm not sure whether the details are visible. But for each step forward, you can see the traffic positions get updated, both in RoadRunner and then Carmaker, the same way the ego vehicle positions are also will get updated for the particular time step.

Yeah, so this is the IPG movie from the Carmaker, and then on the right side, you are seeing the RoadRunner scenario. So for each step forward, if you clearly observe, the camera angle will change. Yeah, so this is with the actor position updates. I guess this is clearly visible. What you are seeing here on the command line is the traffic data. There are three actors present in that scenario. The xy positions of all the traffic actors being printed here, and the same has been reflected on the Carmaker side.

Along with the traffic actor updates, we also seeing the time synchronization. This is, I think, more clearer. There is an ego vehicle. These are the traffic vehicles. And then this is the time. In Carmaker it's 20.8 seconds, and in RoadRunner, also the same 20.8 seconds. So for each step, everything will get updated. Yeah, I hope it's clear. But after this co-simulation, we tried to validate this. We have taken MATLAB AB controller as an ADAS ECU, and we try to validate that.

We thought of different approaches how we can validate this co-simulation because we want to compare the results of each approach. So the first approach we followed is, we are running the client as a standalone application. We are not using any middleware here. So the client is connected to the RoadRunner and Carmaker. But the ADAS ECU is imported into the Carmaker. Carmaker provides an option to import the virtual ECUs as an FMU. So we generated an FMU from the Simulink and imported into the Carmaker.

So the Carmaker will send that object sensor data everything to the ADAS ECU, and RoadRunner will send this to the client. Client will send the co-simulation traffic data to the Carmaker. So that's how we are closing the loop between all these four tools, four blocks. So the ADAS ECU basically receives the object sensor data and the details like XYZ positions, and the velocity of each vehicle, the traffic vehicles. And it also receives the ego vehicle data, like the same XYZ positions, and the velocity of it. In turn, the feedback, like acceleration and braking, because as we are validating the feature, it sends the braking and the acceleration feature feedback to the Carmaker.

So Carmaker vehicle, or the ego vehicle inside the Carmaker, will receive that and it will try to apply the brakes whenever the scenario condition is satisfied. So this is the sample video we tried to capture. So we created a scenario. So there is a standstill object just for the reference purpose. There is a standstill object the velocity of the traffic vehicle is zero. So the ego vehicle, when it tries to approach the traffic element, it will break.

You can see both the scenarios got stopped. The vehicle stopped in both the scenarios. I'm sorry. Yeah, so this is the FMU inside the Carmaker. I'm not sure whether the details are visible. But it takes four inputs and it gives six outputs as the feedback, six outputs the IP status, all those things which are required for our testing purpose. There is a second approach we followed. This is using the ETAS COSYM as a middleware. So instead of client, now the client code has been integrated into the middleware. Along with that, the ADAS ECU is integrated into it using the FMU interface. Now the FMU is imported into the COSYM.

But the difference between the previous and this approach is there is a virtual CAN network being placed here, so ETAS VNet we are using. So now the object sensor details, everything will be transferred via CAN, not with simple variables. And even the outputs like acceleration and braking, they are also coming from the CAN, and middleware will receive that. So that will send on the synchronized the RoadRunner and Carmaker application.

So the same thing will happen here. It receives the object sensor data, traffic data, and it also receives the ego vehicle data. It sends back the acceleration and brake values. So these are the images we captured for the above scenario. So if there is an object approaching our vehicle, we can see that this is the lab car, E experiment, experiment, environment in person, the traffic vehicles. An object is detected at the car, and the collision alert has been given by the FMU. This is the second approach we followed.

For the automation purpose, we used robot test framework. This is the script for that. So this will connect to the RoadRunner and Carmaker. It will automatically inbox both the tools. It will start the simulations, and it will finally, after the simulation ends, it will give us the results. Yeah, so the only difference is we are running the Carmaker in headless mode. So that means you cannot see how the scenario is running inside the Carmaker. It runs in the background. But we can see the RoadRunner scenario here. Once the scenario is completed, we can see the test is passed.

Yeah, this is how the report will be present in the road test framework. Yeah, then basically the car velocity and the brake values from the ECU. So there is a third approach we want to follow to validate this ADAS ECU using the co-simulation. So that is using MATLAB Simulink as an integration platform. So the same client code what we'd written using C++, we are trying to import that as an FMU inside the MATLAB itself. And the same may be controller. Instead of having an FMU or something, we develop the model inside the MATLAB itself, Simulink itself, and we can talk with the RoadRunner directly.

So now that ADAS ECU will be there in the Simulink. This would be our future focus. We have not tried yet, but this we will try later. So if you see the existing simulation tools, like the virtual test drive, or Carmaker, Carla, they generally focus on single vehicle based operation, like the ego vehicle if you concentrate. You define the maneuvers, you put the sensors, you keep the vehicle dynamics, everything integrated to that. But what about the traffic participants, like traffic actors? You need an unexpected events to be performed by the traffic vehicles.

So that's where we tried to develop a solution where the co-simulation of RoadRunner and Carmaker achieves these unrealistic and unachievable scenarios. So the key benefits what we observed is interactive traffic simulation using realistic environment, and then accurate traffic conditions, which are close to the realistic again, and then synchronized co-simulation for VnV automated driving. And finally, there is a complex scenario validation. Whatever you saw the seed scenarios with big trucks coming into place, all those are complex scenarios for us.

So those scenarios we can easily validate because the traffic participants are controlled by the RoadRunner here. Yeah, so with this, I'll end my presentation. Thank you.

I think that is very impressive. Any questions?

Yeah, so you said that you're using the RoadRunner for providing the actor specific dynamic data, right, for Carmaker simulation. But in Carmaker also, we can write commands, right, in test manager to make the actors dynamic and provide that dynamic actors data for the test model. So what is the specific advantage of co-simulating over with RoadRunner? Why it is not can be done only in Carmaker?

Yeah, I understand your question. There is an option in Carmaker itself, something called stochastic traffic generator, where you will generate multiple traffic options. And then there is an unexpected maneuvers you can define. But all those are predefined maneuvers that you see in Carmaker. So Carmaker already knows how the vehicle is going on the road, and in which way it will turn, and all. But then what happens with the RoadRunner is, if you do Carmaker traffic objects, there is an option called user defined motion.

So you can define the motion of the traffic elements via the external client. So that means there is no control of the Carmaker over traffic participants anymore. So the benefit is you can achieve unexpected scenarios. There is no predetermined termination here. Predefined maneuver is also not there. All the maneuvers are defined by external clients. So if you have a recorded data, all are unexpected. You don't know from which way or which point the vehicle will come.

So Carmaker shouldn't know this if you want to achieve in realistic scenarios in place. So RoadRunner will create a scenario out of that realistic recorded data, and then it will control the traffic participants. That position and the speed of that particular traffic vehicles we are transferring it to Carmaker using that user defined motion. So the ego vehicle will have all that unachievable, and then the unexpected events in place.

OK, and how do you pass on that to Carmaker, via some kind of interface?

Yeah, that's where the road and client using gRPC protocol will get the details from RoadRunner. And using this client, there is an EPU communication in Carmaker. So we'll pass that information to the Carmaker using that APIs.

Thank you.

Hello. I had similar question like him. So that's answered partially. So I'll twist the question. Then why not only RoadRunner? Why do we have to still use Carmaker or a dependency on that?

OK.

Why the need of co-simulation here? Because we also have scenario builder in RoadRunner, as explained by [INAUDIBLE].

Correct. So partially, I already answered. That's where you see this unexpected behavior between the co-simulation, how it is possible. But the other point is, you should have the sensors' data, which should read the traffic, and it should detect within that certain ranges. You might have physical sensors being connected to the Carmaker or VTD. And you should pass this information to the ECU. RoadRunner cannot do that, actually, in this current tool.

So this co-simulation, what will happen is, the scenarios are completely running in both the tools, but the data of the sensors needs to be transferred to the virtual ECU, or any ECU. So that can be done by Carmaker sensors only. Yeah, and one more point is the vehicle dynamics. You see, you need a physics based sensor models and all, so the vehicle dynamics like the steering, or the braking, whatever it is. So you need to integrate that with your ego vehicle directly. So that can be done with your environment simulator.

Any other questions? Good. Thank you.

Thank you.