Virtual World Generation for BMW Driving Simulation

So today, I want to talk to you about how we use RoadRunner at BMW to generate virtual environments. We will start slowly with a short intro video.

[MUSIC PLAYING] I took the car tonight with some dynamite.

[VOCALIZING]

I like it, like it like that. I'll make you say yeah. I like it, like it like that. I'll make you say yeah. I like it, like it like that. I'll make you say yeah. I'll make you say yeah. Aha.

All of these lights k-k-k-k-kill my pain. Kill my pain. Taking me high, leaving me entertained.

[VOCALIZING]

Like it! Like it! Like it!

I'll make you say yeah.

[VOCALIZING]

i Know the video was quite a bit fast. So as a summary, some of the key facts that we've seen in the video-- yeah, we operate the most advanced and diversified driving simulation center worldwide. That's something we at BMW are quite proud of. So you will see every occasion I can-- I have to mention this, you'll see me mention it.

Yeah, we have 14 simulators and usability labs. It's important to say that we test the drivers' perception on certain functionalities-- so what we call driver in the loop. We don't usually do massive simulation for training ADAS models or something. So it's always about how the driver perceives a certain new functionality.

Yeah, we're 14 simulators. We can have up to 100 study participants per day. You can see that we have different kinds of-- different sizes of simulators. On the bottom, we have our big simulators, the high fidelity and the high dynamic simulator. They run on a track system with a dome on top of it. And the dome, you have a 360-degrees projection system.

And on the upper floor, we have several smaller simulators with a limited range of motion-- with a limited range of motion system. Some also have 360 degrees LED walls, 270 degrees, 180. So we have different sizes of simulators for different use cases.

Speaking of use cases, that's something we aim to do with our simulation center, is to provide BMW the perfect simulation tools for each step of the vehicle development-- so from the early concept phase to the final function validation phase.

But of course, we have some main use cases. Or the main use cases that we have are-- yeah, ADAS use cases, so if there's a new ADAS functionality you want to test how the driver perceives the new functionality, if he or she can understand it correctly. We have driving dynamics use cases. So if we develop a new vehicle dynamics model, we also want to test it in simulation. And of course, also UI and UX concepts is a big part of what we do.

Here are a few images or impressions that I've brought with me. Yeah, on the upper right-hand side, you can see one of the LED walls. As you can see, in each simulator, we have a full BMW in there. So the study participant basically, yeah, goes into a real car, has everything around him. It's basically the real interior of the car.

The pedals and the steering wheels are connected to the simulation. So you basically drive the real car. But around you, you have the virtual simulation. Yeah, that's really good because then basically, the study participant has a higher immersion and feels more engaged to interact with the virtual scene.

OK, enough of the hardware. We are here for the software, right? So for you to understand why we need a RoadRunner, you basically also have to understand our simulation architecture.

On the one hand side, we have our simulation software. It's called Spider. It's a BMW in-house development. It has been developed for over 20 years. It basically does all the logic and physics calculations. So it calculates all the traffic, vehicles, pedestrians, cyclists, you name it, but also vehicle dynamics, and basically everything that needs to be calculated.

Yeah, we support a bunch of open standards like OpenDrive OpenCRG, Open Simulation Interface. And yeah, obviously, we use BMW vehicle dynamics. But we also support MATLAB Simulink, which we mainly use to inject our ADAS functionalities that we've developed with Simulink.

On the right-hand side, we have the visualization, which we've programmed in Unreal using Unreal Engine. For those of you who don't know Unreal Engine, it's a state of the art game engine. We use it because it has a out of the box support for multicluster systems.

So for example, in our 360-degrees projection system, we have 50 nodes to render the pictures. And the pictures have to be homogeneous. They can't have tiering in between them. So that's something that Unreal Engine provides out of the box, which is pretty cool.

Unreal Engine also has a high baseline for visual fidelity and performance and also has cool tools to generate content like Quixel Megascans for foliage and buildings and MetaHumans for pedestrians or human models, in general.

So the question is, how do we connect these two software? We use Open Simulation Interface. It's a open ASAM, open standard for communicating between simulation softwares.

All right, so coming from the setup, if we talk about environment, we have-- on the Spider side, we use OpenDrive to describe the map or the environment that we are driving on. For those of you who don't know OpenDrive, it's basically a XML format to describe roads, buildings, surroundings, and basically the whole environment in an XML file. So that's how Spider or our simulation software knows where the roads are and how the traffic can move.

But on the visualization side, we obviously need 3D content, like FBX format, glTF, et cetera, so that we can build a 3D world. And that's one of our biggest challenges that we have, is that these two files-- or these two formats have to match. So you can't have a 3D world that doesn't match with the OpenDrive, because then your simulation software will drive somewhere outside of the road. And you can't see it, basically.

So that's where RoadRunner comes in. RoadRunner has the capability to import and export OpenDrive but also export 3D content. So that's basically exactly what we need. We model our roads in RoadRunner. And then we can use it in our simulation.

So now we have the road. But the environment doesn't only consist of roads. You have buildings. You have trees. You have landscapes, mountains, whatever around them. So that's why we've developed our own 3D environment generator, using a state of the art game industry pipeline.

We use Houdini, which is the game and film industry-leading tool for procedural or procedurally generating content. And yeah. And the output goes into the Unreal Editor, where we do-- or we place the assets. But I'll go into detail a bit later.

So now you have the big picture. I want to first dive into a smaller part, which is this red circle here-- CRG. CRG isn't very familiar, I think, or isn't that popular or familiar in the automotive industry.

OpenCRG is basically also a open standard to describe road surfaces. So yeah, you basically have files that describe, with very high precision, how your road surface look like. And we use the CRG files to test our driving dynamic systems or driving dynamics models.

And fortunately, RoadRunner has a tool to edit these CRGs. So we use RoadRunner to create and edit real and synthetic CRG files. As I said, it's for our driving dynamic simulations.

And with such a visual editor, we can iterate really quickly over use cases. So if the colleagues want another surface or want it to place somewhere else, then it is right now, we can do it in minutes, basically. And before that, without an editor, it was really tedious to do that.

So yeah, this was a bit apart from the big picture, just something I wanted to mention because it's a really cool feature in RoadRunner. Yeah, so now that we go back to the big picture, I want to dive a bit into each of these blocks, let's say.

So let's start with RoadRunner. RoadRunner-- we use RoadRunner for the main reason that RoadRunner is there, to create roads. So in this example, we've modeled-- or we've remodeled a real existing junction in order to test our traffic participants. So in this case, we had a recording of the junction and wanted to see if our traffic participants behaved the same way as in the recording.

So that's why we created or remodeled this junction. And then you can export it to OpenDrive. And we could use it in our simulation and have it in our graphics. Everything fine. You can test. All good. So this is, yeah, the basic use case, let's say, that probably everyone knows of. But you can also use RoadRunner to do more complex stuff.

For example, we've used RoadRunner to import synthetically generated height maps. We wanted to have a mountain environment like in the Alps so that you can basically drive around serpentines and so on. So we've created a height map, which you can, with a few tips-- with a few tricks, you can import into RoadRunner. And then you can basically model your roads so that they smoothly fit into your mountain range. And then you have awesome roads to drive on.

So yeah, another use case that we did is that we basically color coded areas in RoadRunner. In this case, we've modeled a city. And for example, we've color coded green with a park area. Red should be skyscrapers. And maybe yellow should be some tiny buildings so that we can automatically create the buildings or the props that need to be in this area automatically in the next step.

Yeah. Here, we also could just export the OpenDrive and road network geometry to use in our simulation. But obviously, as you can imagine, these kind of things are pretty time intensive to do because you have to model everything by hand. So that's something we only do for special scenarios and one-shot projects.

So going a step further, now we have the roads. We go into Houdini to procedurally generate our environment. So what we do is we take the output files from RoadRunner-- so OpenDrive and the 3D model-- and put it into Houdini. Houdini can spawn the assets, for example, based on the area type that we've defined in the previous step, like with the green area and the purple area, and so on.

And it also can utilize the OpenDrive information so that it knows where the roads are-- or how the roads are shaped so that, for example, in this case, they can-- it can scatter these alley trees automatically apart or on the side of the road. So yeah, in the best case, you basically just import your OpenDrive and 3D environment-- or 3D file.

And then Houdini can generate the surroundings for you. Yeah. So as an output from Houdini, we basically get a point cloud or several point clouds that define where in the scene which asset is placed. And that's what we import into Unreal Engine.

So we import this point cloud. And then we can say, OK, for this point cloud, we just place-- for every point, we place a tree. And for every point which should be a building, we place a building. And then in the end, yeah, we can have a city or whatever you want to create or produced automatically. So the assets that are in Unreal are just placed onto the positions that Houdini calculated.

And important is that we use the road-- or the road geometry we still use from RoadRunner. So we don't change anything from-- like in the geometry. But what we have to do is we have to switch the materials to better-looking ones, let's say, so that we have a material that reacts to the weather. So if it's snowing, if it's raining, you have rain puddles and so on. So that's the last step that we do in order to create our environment.

Yeah, and maybe more to say, if you're more interested into this workflow, I can recommend the Unreal Engine 5 Tech Talk, where they've created this Matrix demo. They really dive into every step really deeply. It goes for over two hours, I guess. So if you're interested into the details, I can recommend that Tech Talk.

So as you can imagine, this kind of tool chain brings a few challenges. So the first challenge I've already mentioned is that it's very time intensive and cost intensive because everything has to be-- so at least the roads, they have to be modeled by hand. So the larger environment is, the more time you have to spend on modeling the road network.

For us, it's only feasible for studies where we have really long lead times. As you can imagine, like I said, in the beginning, we have 100 participants per day. So it's around 50 studies a year. Yeah, you can't have this long of process for every study.

So that's why we are working with the RoadRunner team together. And there's already an approach for a script-based automated road generator, which we are really happy about and which is being further developed.

The second part is that the tool pipeline itself is pretty complex. Houdini is not a tool that every beginner can use. So you need to learn it.

And the more tools you have, the more you have the risk of compatibility issues. So our hope is that in the future, at some point, RoadRunner will have the capability to have maybe a rule-based procedural generator for all kinds of 3D assets. But yeah, unfortunately, it's still not the case. But maybe. Let's see what the future holds.

Yeah, so to summarize how we use RoadRunner at BMW, the main use case for us is that we have a tool to create a road network, where the OpenDrive file and the 3D geometry is consistent. So that's the most important part for us. It's great that there is a OpenCRG editor, where we can basically iterate over our driving dynamics use cases faster and create new scenarios.

The creation of simple road networks is pretty straightforward. And that's what we do pretty regularly. But yeah, larger road networks is really time and cost consuming. So yeah, and in the end, we still need a game industry tool chain to create all the rest of the environment so that we can achieve the requirements that we need for visual fidelity.

Yeah. So thank you for attention. If you have any questions, then I'm happy to answer them.

[APPLAUSE]