Design 3D Scenes for Automated Driving Simulation with RoadRunner

Hi, I'm Peter Fryscak, a product manager at MathWorks. I'm here to introduce RoadRunner, which is our 3D road editor designed to create scenes for autonomous and ADAS simulators.

To start off, I'd like to show you a quick video illustrating some of the key features. RoadRunner is a visual editor designed to create 3D roads very quickly. As you can see, we have tools to carve into the roads, to create slip roads, to adjust the size of objects in the scene, to change banking, and to essentially quickly change and edit all of the visual information in the scene, for example, these road markings. And we can then adjust their characteristics.

Adding 3D objects is very easy. And we even have some probabilistic tools. Signs can be US or international. And in addition, we have a Unicode highway sign editor.

Here we are creating an assembly of various parts. So we're essentially adding signals to this scene. And then we'll have phasing and timing at this intersection that is connected to the lights.

Before we go into the demonstrations, I'd like to motivate our discussion. Why do you need a 3D road editor in the first place? In my opinion, it's because the autonomous industry has been growing the kinds of tests. Originally, we had mainly mechanical tests, and I mean things like testing if your brakes, which stop the vehicle, in time to avoid hitting an obstacle. And we've been adding more artificial intelligence tests, which are, for example, can I navigate through a complex city?

Urban areas are extremely complex and varied. And because the industry stakes are so high, you want to make sure that you have a tool that can adequately represent these areas for simulation. RoadRunner has a very robust data model that can handle even the most complex situations.

To make a good simulator model, you have to have more than just the road paint and the geometric objects, like the islands and curbs in the scene. You also need the semantic data. And by semantic data, I mean the additional data that a human or autonomous driver has to know. For example, if I'm in a lane and there's a lane to my left, can I drive in it or is that reserved for oncoming traffic? Can I make a U-turn?

And this can get really complicated. For example, if you look at the image in the bottom right, here we have a roundabout surrounded by five other roundabouts, all left-hand driving because it's in Swindon, England. What you need is a tool that accurately represents the real world visually, geometrically, and semantically.

RoadRunner is a MathWorks product, but completely independent from other MathWorks products like MATLAB and Simulink. In this slide, you see the three products that are part of the family-- RoadRunner that you already saw a short video about and then Asset Library, which is a collection of hundreds of 3D objects, and a new product, which we call Scene Builder. Scene Builder imports HD maps, converts them into RoadRunner roads, and then exports them out to simulators.

At this point, I'd like to give you a demo, which will also illustrate some of our new features. This is the user interface of RoadRunner and a work area, some panels, like the Library browser, where you put your assets. And then across the top, we have all of these tools in the toolbars.

So we'll start off with the Road Plan tool. And I'll just make a couple of clicks in the scene. And you see that I've already created a 3D road. And it has elements like the gutter and the sidewalk.

The next tool I would like to illustrate is our Lane Add tool. So I'm just going to right click, and I've added a new lane. I go to the Lane Marking tool. I'll click on this marking and just hit the Delete key.

And then from here, I'm going to go to our Lane Width tool. And I'm just going to grab and pull this whole section over. Or if I want to be more precise, I can go into the interview panel and simply select an exact number.

At this point, I'd like to show you the Lane Carve tool, which is very simple to use. I just right click and drag. I'll do the same thing on this side. And then inside the Lane Tool, I'm going to select these end sections and just delete. And this is one of the really nice things about RoadRunner, that it has these sort of procedural tools, for example, bringing that sidewalk over when we did that deletion, that automatically engage. But you still have full control over your editing.

The next tool is the Lane Chop tool. And I'm just going to put one here and here. And then I'll go back into our Lane tool, and I'll select each of these. And I'll change them from driving to shoulder. And this middle section, I'm going to change into parking.

So we have some excellent tools for importing various formats, including open drive. But what I'd like to show you right now is our OpenDRIVE Export Preview. So this will export the whole scene out as OpenDRIVE and re-import it. And what you see here is that we have these colored sections in a section for the parking.

Now, since we're talking about parking, I'm going to go into our Parking tool. And you can see that spaces have been automatically laid out. And the next thing that I'm going to do is go into Markings and back to Parking. And I'm just going to take this single solid line and I'm making those side edge markings.

And then if I go into Stencils, I'll look for a marking that looks something like a cross. Here's one. And I'll attach that to the side nodes and also to the corners. And then if I look at it, you can see we've created a nice little parking area. Now, if like me you're from California, this is an atypical thing for parking on the side of the street. I'll just change the angle. And this is a much more likely use case.

So the next thing that I would like to do is go back into the Road Plan tool. And we're going to go into Road Styles. And I'll pick another style. And I'll just show you how to create intersections.

So here, I'm just clicking twice again. This draws the new style of the road. And then I can go back into my original road and simply drag it. And you'll see that it automatically creates this 3D intersection here, which then I can go back into and extend all the way across the road. And you'll see that that has now made a full intersection on that side as well.

The next thing to show is the Maneuver Road tool, where if I click on an intersection, you see that we algorithmically and heuristically figure out which roads are connected to which. This is all fully editable. And then I'll pick another road type, and I'll simply add this road here. So that you see if I go back to the Maneuver Road Tool, those new connections have also been created.

One of the nice things about RoadRunner is we have a full undo stack. So I'm going to hit Control Z. And you can as well if you need to get rid of some work that you previously did and want to change.

Now, I'll use the same road type to just cross again. Then I'll zoom in a little bit on that. And then here, I'm going to direct your attention to this bottom left panel, where I can raise the height of something. And again, I can go into the side panel and make it more precise.

And then from here, I'm going to go to our Road Carve tool. And that's like the Lane Carve, but it works across the entire road. So I'm just defining that section.

And the next thing I'll do is I'll go into our Bridge tool. I'll click on that bit. And I'll turn it into a bridge. So that defines this underside section, but also has some other effects, which will be easily seen once I go into our Surface tool and add a few surfaces. So you see that the terrain has been automatically formed where the bridge is connected.

Now, the Surface tool has a couple other ways of operating. It doesn't have to only connect point to point. I can also make a curve. Or if I hold down the Control key, it will make these sort of sharper angles.

So what if you're looking for a road style that's different from the one or the group of them that you already have in this window? Well, you can make your own. So I'm just going to right click twice. This creates a road.

And as we did before, I'm going to go into Lane Add. And I'll add a line on either side.

Then, we'll go back to Markings I'm just going to drag this dashed single white line onto both of those. And let's say I like this. So I go to the Cross Section tool. I right click. I create a road style. And now, I have a new road style. So if I go to the Road Plan tool with my new road style, it uses that.

And then here, I'll show you our assemblies. So these are like what you saw in the video. They're groups of objects that are put together. In this case, we have this lights and armature.

And then I'll go into Signalize. So we have a tool, click on the intersection, Auto Signalize. In this case, I'm going to choose a four-way permitted left. And I'm going to select that prop. So you see that I've got signalize intersection with phases. In this first phase, all the north-south routes can go, as well as the right turns. And then, the light will change here and here. And now, I have the east-west and their right turns.

And so the last thing I'll do is I'll simply take this road that we've created. And like you were saying before, I'm just going to extend that to make this intersection. And at this point, what you typically do is save it. So I'm going to save this scene as my Test Scene. And then you would go out to export it to the simulator of your choice.

RoadRunner connects to many of the world's leading simulators through standard formats. So typically, we're exporting two types of information. The first is a geometric mesh, for example, the road and the curbs and the objects in the scene. And the second type of file is the semantic information. And very typically that comes out as OpenDRIVE.

Many customers have told me that they like RoadRunner, especially when they have two groups using different simulators inside of their organization. For example, let's say you have one group working with CARLA and another working with IPG CarMaker. So you create the scene once in RoadRunner, and you can use it with both.

Here, we're showing the various export formats that we have. The triangulated mesh formats, like FBX are at the top. Then you have the semantic data formats in the middle. These include OpenDRIVE, but also Baidu Apollo and a human readable GeoJSON format. We also have specialized exporters for Unity and Unreal that come with plug-ins. And the plug-ins take care of things like the collision mesh and making sure that textures are exported out correctly to those game engines. And then in addition, we have specific exporters for CARLA, VTD and Metamoto.

We also have very good support for geographic data. Some of that data is for a visual reference. And some, like elevation, can affect the roads and other things in your scene. And I'll show that in just a moment.

So here, we have some geographic data. I'll start off by importing this aerial imagery. And Roadrunner under the hood takes care of all the projections. And since this had geographic information in it, the latitude and longitude is set. We generally set up a small work area, because some of the file types like point cloud can be quite dense, but that can be extended. We'll just apply those world changes to reload the data. And so now we have that extended area of the imagery, and I'm just going to drag and drop elevation.

So here, I'll show you another way that you can create roads. I'm going to start off with a very simple type. So from my Road Styles, I'll just click on this. I'll go into my Road Plan tool, clicking along the center line. And now, I'm going to use the Road Width tool to simply drag. And this is more for illustration than for exactness.

And now, in the Lane Split tool, I can simply right click everywhere that I see a lane. And then as we've shown before, I can go into Markings. And, for example, I saw a couple of places where we're going to use this dashed marking. And if I go into my scene inside of the Marking tool, I can click and shift click on these. And then when I drag over, it gets applied automatically three at a time.

And here, I'm just going to bring in that solid yellow, or actually double yellow here. And what I'll point your attention to if I take this road section, you'll see here in the bottom left, there's a blue line. That represents the elevation. And the red line represents our road. So we're going to use the Project Road tool to project those. And then I'll do that again. I just grab, let's say, an end of this road and pull it forward. And then he used that same tool to re-project.

Now, going back to our example, I can also bring in the point cloud. This is millions and millions of data points. And we bring them in originally gray, but I can also recolor them by the image or by classification or intensity, whatever data is in the data set. And I think you can see how this would be useful if you wanted to put down trees to represent this area.

So one thing I'd like to mention is that the US Geological Survey has excellent and comprehensive data for most of the United States. That's where I got the example data that I used in the previous demonstration. You can find elevation and aerial imagery for pretty much all of the United States. And sometimes LiDAR is also available, especially along the coasts.

RoadRunner has some nice facilities as well for bringing in 3D objects. We support many 3D object types, like FBX and GLTF and OBJ. So if you already have these kinds of objects, you should be able to just drag and drop them into RoadRunner. And the same thing is true of 2D object types, like SVG and PNG.

If you would prefer to already have a premade set, then we sell the Asset Library, which is a complete collection of hundreds of these types of objects that are ready to be used in your scene. So now, I'll show you a quick demonstration that highlights some of the objects in the Asset Library.

So here's the scene that we saved earlier, and now we're going to add some assets. So I've gone into the folder that has a lot of our trees. And what I'll do is I'll just pick one of these randomly. And I'll drop it into the scene. And then if I hit the F key, it will frame on that object.

So in addition to adding props individually, we have this very nice facility called the Prop Set. And what the Prop Set does is it simply sets up these objects that you add to it in proportion, so 1:1:2 in this case. So if I have the Prop Set selected and then I add a line, we're going to randomly add trees from that Prop Set in that proportion.

If I go into the Prop Set again, I can change the proportion. Now, I'm going to drastically reduce the type of that tree that we have. And I can go in and change the spacing, for example, and the position variation. And like all of the other sorts of objects in RoadRunner, I can simply click and add points and change curvature.

Similarly, if I can take this polygon, I can create a set like that. Or I can drag and drop a set that I had originally or even take an individual and drop it into that set. And then I can change density upwards or downwards to quickly create this sort of forested area.

We have some nice building props. So I'll just take this one and randomly drop it here, again, hitting F to frame. And then I can do things like go into my props for signals. I know that I've got some posts. I'll just drag and drop this into the scene.

And now, I'll show you some of our signs. So we have a very large collection of signs and also for various countries. So I'm just going to choose something here that's semi-random and show you that when I drag it onto the object, it'll automatically click to that reference curve. And then, of course, I can do things like rotate it.

We also have some really nice structures in the Asset Library for doing things like damage to the roads. So as an example, I can take cracks and simply place them, placing cracks in the scene with the Marking tool. And let's see, I've got a linear crack here. I can just draw that line. And if we zoom in a little bit, you'll see that individual crack.

I can also get some oil paths that I'll place in the scene here. And I'll pick up some kind of an asphalt patch. And I'll just put that into the scene to illustrate that we also can control the sort index. So right now, we have the asphalt on top. And now we have the paint on top. And now the asphalt entirely on top of everything.

In addition to this, we also have some really nice extrusions. For example, here, we've made that bridge. I'm just going to quickly add a bridge railing type. And I'll just drag that into the scene and place it on the bridge. And I'll do the same on this side. So I think you can see how the prop library really can help flesh out a scene very quickly.

The next thing I'd like to show you is our new scene builder tool, which automatically generates roads from high definition maps. So here, I have a KML file, which is a single point. I'm it's going to drag and drop that into my scene. And since it has a geographic location, it automatically sets the location of my scene. I could have easily gone into Latitude and Longitude and just typed it in. And here to give you a sense of what I'm going to be bringing in, I'm just loading a little bit of imagery. And then I can turn that off.

And the next thing that I'm going to do is reach out to an online service. So to do that, I use our new Scene Builder tool. And in this case, I am grabbing from Here, HD Live Map. So we're using their API. And it brings in this large area of high definition map data.

I should point out that you will need a license with any of the HD map providers. For example, if you're not sure who to reach out to at Here, then please contact us. We'll give you an email a little bit later in the presentation. And we'd be happy to direct your inquiry.

Now what I'm going to do is simply box select an area. And then I'm going to go to our Build Roads tool, which will build the selected area, or rather convert it from an HD map form into RoadRunner roads. And here we see the results. So a very complex geometry of this interchange that we brought in just a few seconds. And I think you'll agree that if you were to build something like this by hand, it would take tens to hundreds of hours.

What I'd like to show you is that this data that we just brought in is Roadrunner road data now. So, for example, I can do some of those operations that I showed you earlier, like just dragging and dropping a marking. So, for example, I can go into the Lane Width tool. I can change a width at any place. I can even add a data point if I want. So I think you can see how being able to reach out to these data services and pull in roads automatically is a really nice supplement to the kinds of synthetic creation that we were showing earlier.

At this point, I'd like to quickly recapitulate. So I introduced RoadRunner and the Asset Library and Scene Builder. And I also talked a little bit about how it exports to external simulators. But those of you who are already MathWorks clients are probably wondering, how does it interact with MATLAB and Simulink? And that's what I'd like to show you next.

And the answer is it's pretty similar to what we showed for all the other simulators. We'll export an FBX file that will be sent to Unreal Engine along with the geometry. And then, we'll export an OpenDRIVE file with the semantics. That will go into driving scenario. And everything gets connected up in Simulink.

So in this example, I have a couple of sensors. So we have a camera sensor up here in the top left. And in the bottom left, we have a LiDAR sensor. This is a very simple terrain that was created using RoadRunner. And here, everything is running and coordinated by Simulink.

At this point, we're hoping that some of you are interested in learning more about RoadRunner and potentially in becoming new customers. The best way to reach us is the email at the bottom of the screen, automated-driving@mathworks.com. And we're eager to speak with you and to learn about your use cases.

RoadRunner was originally created by a company called VectorZero that was later acquired by MathWorks. If your company had a VectorZero license, then you have been provided with an upgraded MathWorks license. And that has many, many, many more features. So if you're still using an old VectorZero license, please also contact us at the email below and we'll help you find the upgrade.

There is also a class of customers that have MathWorks Enterprise licenses. And Enterprise licenses include MATLAB and Simulink and many of the other programs from MathWorks, including RoadRunner. So if you're an Enterprise customer, you do have access to RoadRunner. And again, if you don't already know how to get it, then just email us and we'll help you.

And lastly, over 1,300 universities around the world have access to MATLAB, Simulink, and RoadRunner. So if you are in a university or if you're collaborating with a university, then chances are that students and faculty have access to RoadRunner. And again, if you email us, we'd be happy to help you.

Thank you very much for your time and attention. And we hope that we'll be hearing from you soon.