Engineers developing perception algorithms for ADAS or fully automated driving generally have a wealth of input data to work with. But this data, coming from many different types of sensors, including radar, LiDAR, cameras, and lane detectors, initially raises more questions than it answers. How do you make sense of the data, bring it to life? How do you reconcile conflicting inputs from different sensors? And once you’ve developed an algorithm, how do you evaluate the results it’s producing?
Here’s a guide to features and capabilities in MATLAB® and Automated Driving System Toolbox™ that can help you address these questions. We’ll focus on four key tasks: visualizing vehicle sensor data, labeling ground truth, fusing data from multiple sensors, and synthesizing sensor data to test tracking and fusion algorithms.
Visualizing Vehicle Sensor Data
Making sense of the data is the primary challenge in the early stages of perception system development. Sensors deliver their output in different formats and at different rates. For example, cameras provide images in 3D matrices, LiDAR provides a list of points, and embedded or intelligent cameras provide object lists with details on vehicles, lanes, and other objects. The disparate nature of these outputs makes it difficult to see the overall picture (Figure 1).
At this early stage, we need to know exactly how the sensors are representing the environment around the vehicle. The best type of visualization to use for this is a bird’s-eye plot because it allows us to visualize all the data from the different sensors in one place.
To create birds-eye plots we use the visualization tools in MATLAB and Automated Driving System Toolbox. We then add more detail to the views with these objects:
coverageAreaPlotter, which displays the sensor coverage area
detectionPlotter, which displays lists of objects detected by vision, radar, and LiDAR sensors
laneBoundaryPlotter, which overlays lane detections onto images
We now have accurate visualizations of sensor coverage, detections, and lane boundaries (Figure 2).
Automating Ground Truth Labeling
Ground truth is required to train object detectors using machine learning or deep learning techniques. It is also essential to evaluate existing detection algorithms. Establishing ground truth is often a labor-intensive process, requiring labels to be inserted into a video manually, frame by frame. The Ground Truth Labeler app in Automated Driving System Toolbox includes computer vision algorithms to accelerate the process of labeling ground truth. The app has three key features (Figure 3):
- The vehicle detector automates the detection and labeling of vehicles in keyframes by using an aggregate channel feature (ACF).
- The temporal interpolator labels the objects detected in all frames between selected keyframes.
- The point tracker uses a version of the Kanade-Lucas-Tomasi (KLT) algorithm to track regions of interest across frames.
- The add algorithm lets you add custom algorithms and facilitates iterative development of object detectors.
Fusing Data from Multiple Sensors
Virtually every perception system uses input from several complementary sensors. Reconciling data from these sensors can be challenging because each one is likely to give a slightly different result—for example, the vision detector might report that a vehicle is in one location, while the radar detector shows that same vehicle in a nearby but distinctly different location.
multiObjectTracker in Automated Driving System Toolbox tracks and fuses detections. A common application is to fuse radar and vision detections and improve the estimated position of surrounding vehicles (Figure 4).
Synthesizing Sensor Data to Generate Test Scenarios
Some test scenarios, such as imminent collisions, are too dangerous to execute in a real vehicle, while others may require overcast skies or other specific weather conditions. We can address this challenge by synthesizing object-level sensor data to generate scenarios that include roads, vehicles, and pedestrians as virtual objects. We can use this synthetic data to test our tracking and sensor fusion algorithm. (Figure 5).
Using Vehicle Data in Perception Systems
Visualizing, fusing, and synthesizing vehicle data lays the groundwork for developing object detection algorithms. When we are ready to deploy the MATLAB algorithms, we can use MATLAB Coder™ to generate portable, ANSI/ISO compliant C/C++ code for integration into our embedded environment.