Automate Testing for Highway Lane Following Controls and Sensor Fusion

This example uses:

This example shows how to integrate the sensor fusion and controls components of a highway lane following application. It also shows how to assess the functionality of this integration by defining scenarios based on requirements, automating testing of components, and generating code for those components.

Introduction

Sensor fusion and lane following controller components are the two fundamental components of a highway lane following application. The sensor fusion component fuses information from a camera and a radar sensor to detect vehicles and their tracks. The lane following controller generates control signals for vehicle dynamics. It uses perception-based lane detections to provide the steering angle for lateral control and acceleration for longitudinal control. Simulating and testing the interactions between these two components using metrics for different test scenarios is important. Automatically running these simulations enables regression testing to verify system-level functionality.

This example enables integration and simulation of these two components using a system-level test bench model. It shows how to automate testing against multiple scenarios using Simulink Test™. The scenarios are based on system-level requirements. Additionally, it also shows how you can verify the generated code using software-in-the-loop (SIL) simulation. In this example, you:

Review requirements — The requirements describe system-level test conditions. Simulation test scenarios are created to represent these conditions.
Explore the test bench model — The model contains sensor fusion, controls, vehicle dynamics, sensors, and metrics to assess functionality. The metric assessments integrate the test bench model with Simulink Test for the automated testing.
Disable runtime visualizations — Runtime visualizations are disabled to reduce the execution time for automated testing.
Automate testing — A test manager is configured to simulate each test scenario, assess success criteria, and report results. The results are explored dynamically in the test manager and exported to a PDF for external reviewers.
Automate testing with generated code — The sensor fusion, decision logic, and controls components are configured to generate C++ code. You run automated testing on the generated code to verify expected behavior.
Automate testing in parallel — Overall execution time for running the tests is reduced using parallel computing on a multicore computer.

Testing the integration of the sensor fusion and controls algorithms requires a photorealistic simulation environment. In this example, you enable system-level simulation through integration with the Unreal Engine® from Epic Games®.

if ~ispc
    error(['This example is supported only on Microsoft', char(174), ' Windows', char(174), '.'])
end

Review Requirements

To explore the test bench model, load the highway lane following controls and sensor fuision project.

openProject("HLFControlsSensorFusion");

Requirements Toolbox™ enables you to author, analyze, and manage requirements within Simulink. This example contains 10 test scenarios, with high-level testing requirements defined for each scenario. Open the requirement set.

open('HLFControlsWithSensorFusionTestRequirements.slreqx')

Alternatively, you can also open the file from the Requirements tab of the Requirements Manager app in Simulink.

Each row in this file specifies the requirements in textual and graphical formats for testing the lane-following system for a test scenario. The scenarios with the scenario_LF_ prefix enable you to test lane-detection and lane-following algorithms without obstruction by other vehicles. The scenarios with the scenario_LFACC_ prefix enable you to test lane-detection, lane-following, and adaptive cruise control (ACC) behavior with other vehicles on the road.

scenario_LF_01_Straight_RightLane — Straight road scenario with the ego vehicle in the right lane.
scenario_LF_02_Straight_LeftLane — Straight road scenario with the ego vehicle in the left lane.
scenario_LF_03_Curve_LeftLane — Curved road scenario with the ego vehicle in the left lane.
scenario_LF_04_Curve_RightLane — Curved road scenario with the ego vehicle in the right lane.
scenario_LFACC_01_Curve_DecelTarget — Curved road scenario with a decelerating lead vehicle in the ego lane.
scenario_LFACC_02_Curve_AutoRetarget — Curved road scenario with changing lead vehicles in the ego lane. This scenario tests the ability of the ego vehicle to retarget to a new lead vehicle while driving along a curve.
scenario_LFACC_03_Curve_StopnGo — Curved road scenario with a lead vehicle slowing down in the ego lane.
scenario_LFACC_04_Curve_CutInOut — Curved road scenario with a lead car cutting into the ego lane.
scenario_LFACC_05_Curve_CutInOut_TooClose — Curved road scenario with a lead car cutting aggressively into the ego lane.
scenario_LFACC_06_Straight_StopandGoLeadCar — Straight road scenario with a lead vehicle that breaks down in the ego lane.

Explore Test Bench Model

Open test bench model.

open_system('HLFControlsWithSensorFusionTestBench');

The test bench model contains these subsystems:

Simulation 3D Scenario — Subsystem that specifies the road, vehicles, vision detection generator, and radar sensors used for simulation.
Forward Vehicle Sensor Fusion — Subsystem that fuses the detections of vehicles in front of the ego vehicle that were obtained from the vision and radar sensors.
Lane Following Decision Logic — Algorithm model that specifies the lateral and longitudinal decision logic and provides lane center information and the most important object (MIO) related information to the controller.
Lane Following Controller — Algorithm model that specifies the controller.
Vehicle Dynamics — Subsystem that specifies the dynamic model for the ego vehicle
Metrics Assessment — Subsystem that assesses system-level behavior.

The Simulation 3D Scenario subsystem configures the road network, sets vehicle positions, and synthesizes sensors. Open the Simulation 3D Scenario subsystem.

open_system("HLFControlsWithSensorFusionTestBench/Simulation 3D Scenario")

The scene and road network are specified by these parts of the subsystem:

The Simulation 3D Scene Configuration block has the SceneName parameter set to Curved road.
The Scenario Reader block is configured to use a driving scenario that contains a road network.

The vehicle positions are specified by these parts of the subsystem:

The Ego input port controls the position of the ego vehicle, which is specified by the Simulation 3D Vehicle with Ground Following 1 block.
The Vehicle To World block converts actor poses from the coordinates of the ego vehicle to the world coordinates.
The Scenario Reader block outputs actor poses, which control the position of the target vehicles. These vehicles are specified by the other Simulation 3D Vehicle with Ground Following blocks.
The Cuboid To 3D Simulation block converts the ego pose coordinate system (with respect to below the center of the vehicle rear axle) to the 3D simulation coordinate system (with respect to below the vehicle center).

The sensors attached to the ego vehicle are specified by these parts of the subsystem:

The Simulation 3D Vision Detection Generator block is attached to the ego vehicle to detect vehicles and lanes in the 3D simulation environment. The block derives vehicle detections from simulated actor poses that are based on cuboid representations of the actors in the scenario.
The Delete Velocity From Vision block packs vision detections to match the bus used by the vision processing algorithm. To do so, the block replaces the measurement noise and disables the velocity measurement.
The Pack Lanes Truth block packs the lane ground truth information into the lanes bus structure, which the Lane Following Decision Logic reference model requires.
The Simulation 3D Probabilistic Radar block is attached to the ego vehicle to detect vehicles in the 3D Simulation environment.

The Forward Vehicle Sensor Fusion, Lane Following Decision Logic, Lane Following Controller, Vehicle Dynamics, and Metrics Assessment subsystems are based on the subsystems used in the Highway Lane Following example.

In this example, the focus is on automating the simulation runs for this test bench model using Simulink Test for the different test scenarios. The Metrics Assessment subsystem enables integration of system-level metric evaluations with Simulink Test. This subsystem uses Check Static Range (Simulink) blocks for this integration. Open the Metrics Assessment subsystem.

open_system('HLFControlsWithSensorFusionTestBench/Metrics Assessment');

In this example, four metrics are used to assess the lane-following system.

Verify Lateral Deviation — This block verifies that the lateral deviation from the center line of the lane is within prescribed thresholds for the corresponding scenario. Define the thresholds when you author the test scenario.
Verify In Lane — This block verifies that the ego vehicle is following one of the lanes on the road throughout the simulation.
Verify Time gap — This block verifies that the time gap between the ego vehicle and the lead vehicle is more than 0.8 seconds. The time gap between the two vehicles is defined as the ratio of the calculated headway distance to the ego vehicle velocity.
Verify No Collision — This block verifies that the ego vehicle does not collide with the lead vehicle at any point during the simulation.

Disable Runtime Visualizations

The system-level test bench model opens a Unreal Engine simulation window for visualizing the scenario. This window is not required when the tests are automated.

Configure the Simulation 3D Scene Configuration block to run the Unreal Engine in headless mode, where the 3D simulation window is disabled.

blk = 'HLFControlsWithSensorFusionTestBench/Simulation 3D Scenario/Simulation 3D Scene Configuration';
set_param(blk,'EnableWindow','off');

Automate Testing

The Test Manager is configured to automate the testing of the lane-following application. Open the HLFControlsWithSensorFusionMetricAssessments.mldatx test file in the Test Manager.

sltestmgr;
sltest.testmanager.load('HLFControlsWithSensorFusionMetricAssessments.mldatx');

Observe the populated test cases that were authored previously in this file. These tests are configured to run the model.

Each test case uses the POST-LOAD callback to run the setup script with appropriate inputs. After the simulation of the test case, the Test Manager invokes helperGenerateFilesForLaneFollowingWithSensorFusionReport from the CLEAN-UP callback to generate the plots.

Run and Explore Results for Single Test Scenario

To test the system-level model with the scenario_LFACC_03_Curve_StopnGo test scenario from Simulink Test, use this code:

testFile = sltest.testmanager.load('HLFControlsWithSensorFusionMetricAssessments.mldatx');
testSuite = getTestSuiteByName(testFile,'Test Scenarios');
testCase = getTestCaseByName(testSuite,'scenario_LFACC_03_Curve_StopnGo');
resultObj = run(testCase);

To generate a report after the simulation, use this code:

sltest.testmanager.report(resultObj,'Report.pdf',...,
'Title','Highway Lane Following Controls with Sensor Fusion',...
'IncludeMATLABFigures',true,...
'IncludeErrorMessages',true,...
'IncludeTestResults',0,'LaunchReport',true);

Examine the Report.pdf. Observe that the Test environment section shows the platform on which the test is run and the MATLAB® version used for testing. The Summary section shows the outcome of the test and duration of the simulation in seconds. The Results section shows pass/fail results based on the assessment criteria. This section also shows the plots logged from the helperGenerateFilesForLaneFollowingWithSensorFusionReport function.

Run and Explore Results for All Test Scenarios

You can simulate the system for all the tests by using sltest.testmanager.run. Alternatively, you can simulate the system by clicking Play in the Test Manager app.

When the test simulations are complete, the results for all the tests can be viewed in the Results and Artifacts tab of the Test Manager. For each test case, the Check Static Range (Simulink) blocks in the model are associated with the Test Manager to visualize overall pass/fail results.

You can find the generated report in the current working directory. This report contains a detailed summary of pass/fail statuses and plots for each test case.

Verify Test Status in Requirements Editor

Open the Requirements Editor and select Display. Then, select Verification Status to see a verification status summary for each requirement. Green and red bars indicate the pass/fail status of simulation results for each test.

Automate Testing with Generated Code

The HLFControlsWithSensorFusionTestBench model enables integrated testing of the Forward Vehicle Sensor Fusion, Lane Following Decision Logic, and Lane Following Controller components. Regression testing of these components through SIL verification is often helpful. If you have a Simulink Coder™ license, then you can generate code for these components. This workflow lets you verify that the generated code produces expected results that match the system-level requirements throughout simulation.

Set Forward Vehicle Sensor Fusion to run in software-in-the-loop mode.

model = 'HLFControlsWithSensorFusionTestBench/Forward Vehicle Sensor Fusion';
set_param(model,'SimulationMode','Software-in-the-loop');

Set Lane Following Decision Logic to run in software-in-the-loop mode.

model = 'HLFControlsWithSensorFusionTestBench/Lane Following Decision Logic';
set_param(model,'SimulationMode','Software-in-the-loop');

Set Lane Following Controller to run in software-in-the-loop mode.

model = 'HLFControlsWithSensorFusionTestBench/Lane Following Controller';
set_param(model,'SimulationMode','Software-in-the-loop');

Now, run sltest.testmanager.run to simulate the system for all the test scenarios. After tests are complete, review the plots and results in the generated report.

Automate Testing in Parallel

If you have a Parallel Computing Toolbox™ license, then you can configure Test Manager to execute tests in parallel using a parallel pool. To run tests in parallel, save the models after disabling the runtime visualizations using save_system('HLFControlsWithSensorFusionTestBench'). Test Manager uses the default Parallel Computing Toolbox cluster and executes tests only on the local machine. Running tests in parallel can speed up execution and decrease the amount of time it takes to get test results. For more information on how to configure tests in parallel from the Test Manager, see Run Tests Using Parallel Execution (Simulink Test).