Automate Testing for Highway Lane Following
This example shows how to assess the functionality of a lane-following application by defining scenarios based on requirements, automating testing of components and the generated code for those components. The components include lane-detection, sensor fusion, decision logic, and controls. This example builds on the Highway Lane Following example.
A highway lane-following system steers a vehicle to travel within a marked lane. It also maintains a set velocity or safe distance from a preceding vehicle in the same lane. The system typically includes lane detection, sensor fusion, decision logic, and controls components. System-level simulation is a common technique for assessing functionality of the integrated components. Simulations are configured to test scenarios based on system requirements. Automatically running these simulations enables regression testing to verify system-level functionality.
The Highway Lane Following example showed how to simulate a system-level model for lane-following. This example shows how to automate testing that model against multiple scenarios using Simulink Test™. The scenarios are based on system-level requirements. In this example, you will:
Review requirements: The requirements describe system-level test conditions. Simulation test scenarios are created to represent these conditions.
Review the test bench model: Review the system-level lane-following test bench model that contains metric assessments. These metric assessments integrate the test bench model with Simulink Test for the automated testing.
Disable runtime visualizations: Runtime visualizations are disabled to reduce execution time for the automated testing.
Automate testing: A test manager is configured to simulate each test scenario, assess success criteria, and report results. The results are explored dynamically in the test manager and exported to a PDF for external reviewers.
Automate testing with generated code: The lane detection, sensor fusion, decision logic, and controls components are configured to generate C++ code. The automated testing is run on the generated code to verify expected behavior.
Automate testing in parallel: Overall execution time for running the tests is reduced using parallel computing on a multi-core computer.
Testing the system-level model requires a photorealistic simulation environment. In this example, you enable system-level simulation through integration with the Unreal Engine from Epic Games®. The 3D simulation environment requires a Windows® 64-bit platform.
if ~ispc error("The 3D simulation environment requires a Windows 64-bit platform"); end
To ensure reproducibility of the simulation results, set the random seed.
Requirements Toolbox™ lets you author, analyze, and manage requirements within Simulink. This example contains ten test scenarios, with high-level testing requirements defined for each scenario. Open the requirement set.
To explore the test requirements and test bench model, open a working copy of the project example files. MATLAB copies the files to an example folder so that you can edit them. The
TestAutomation folder contains the files that enables the automate testing.
addpath(fullfile(matlabroot, 'toolbox', 'driving', 'drivingdemos')); helperDrivingProjectSetup('HighwayLaneFollowing.zip', 'workDir', pwd);
Alternatively, you can also open the file from the Requirements tab of the Requirements Manager app in Simulink.
Each row in this file specifies the requirements in textual and graphical formats for testing the lane-following system for a test scenario. The scenarios with the scenario_LF_ prefix enable you to test lane-detection and lane-following algorithms without obstruction by other vehicles. The scenarios with the scenario_LFACC_ prefix enable you to test lane-detection, lane-following, and ACC behavior with other vehicles on the road.
scenario_LF_01_Straight_RightLane— Straight road scenario with ego vehicle in right lane.
scenario_LF_02_Straight_LeftLane— Straight road scenario with ego vehicle in left lane.
scenario_LF_03_Curve_LeftLane— Curved road scenario with ego vehicle in left lane.
scenario_LF_04_Curve_RightLane— Curved road scenario with ego vehicle in right lane.
scenario_LFACC_01_Curve_DecelTarget— Curved road scenario with a decelerating lead vehicle in ego lane.
scenario_LFACC_02_Curve_AutoRetarget— Curved road scenario with changing lead vehicles in ego lane. This scenario tests the ability of the ego vehicle to retarget to a new lead vehicle while driving along a curve.
scenario_LFACC_03_Curve_StopnGo— Curved road scenario with a lead vehicle slowing down in ego lane.
scenario_LFACC_04_Curve_CutInOut— Curved road scenario with a fast moving car in the adjacent lane cuts into the ego lane and cuts out from ego lane.
scenario_LFACC_05_Curve_CutInOut_TooClose— Curved road scenario with a fast moving car in the adjacent lane cuts into the ego lane and cuts out from ego lane aggressively.
scenario_LFACC_06_Straight_StopandGoLeadCar— Straight road scenario with a lead vehicle that breaks down in ego lane.
These requirements are implemented as test scenarios with the same names as the scenarios used in the
Review Test Bench Model
This example reuses the
HighwayLaneFollowingTestBench model from the Highway Lane Following example. Open the test bench model.
This test bench model has Simulation 3D Scenario, Lane Marker Detector, Vehicle Detector, Forward Vehicle Sensor Fusion, Lane Following Decision Logic and Lane Following Controller and Vehicle Dynamics components.
This test bench model is configured using the
helperSLHighwayLaneFollowingSetup script. This setup script takes
scenarioName as input.
scenarioName can be any one of the previously described test scenarios. To run the setup script, use code:
scenarioName = "scenario_LFACC_03_Curve_StopnGo"; helperSLHighwayLaneFollowingSetup("scenarioFcnName",scenarioName);
You can now simulate the model and visualize the results. For more details on the analysis of the simulation results and the design of individual components in the test bench model, see the Highway Lane Following example.
In this example, the focus is more on automating the simulation runs for this test bench model using Simulink Test for the different test scenarios. The Metrics Assessment subsystem enables integration of system-level metric evaluations with Simulink Test. This subsystem uses Check Static Range (Simulink) blocks for this integration. Open the Metrics Assessment subsystem.
In this example, four metrics are used to assess the lane-following system.
Verify Lateral Deviation: Verifies that the lateral deviation from the centerline of the lane is within prescribed thresholds for the corresponding scenario. Prescribed thresholds are defined while authoring the test scenario.
Verify In Lane: Verifies that the ego vehicle is following one of the lanes on the road throughout the simulation.
Verify Time gap: Verifies that the time gap between the ego vehicle and the lead vehicle is above 0.8 seconds. The time gap between the two vehicles is defined as the ratio of the calculated headway distance to the ego vehicle velocity.
Verify No Collision: Verifies that the ego vehicle does not collide with the lead vehicle at any point during the simulation.
Disable Runtime Visualizations
The system-level test bench model visualizes intermediate outputs during the simulation for the analysis of different components in the model. These visualizations are not required when the tests are automated. You can reduce execution time for the automated testing by disabling them.
Disable runtime visualizations for the Lane Marker Detector subsystem.
load_system('LaneMarkerDetector'); blk = 'LaneMarkerDetector/Lane Marker Detector'; set_param(blk,'EnableDisplays','off');
Disable runtime visualizations for the Vehicle Detector subsystem.
load_system('VisionVehicleDetector'); blk = 'VisionVehicleDetector/Vision Vehicle Detector/ACF/ACF'; set_param(blk,'EnableDisplay','off');
Configure the Simulation 3D Scene Configuration block to run the Unreal Engine in headless mode, where the 3D simulation window is disabled.
blk = ['HighwayLaneFollowingTestBench/Simulation 3D Scenario/', ... 'Simulation 3D Scene Configuration']; set_param(blk,'EnableWindow','off');
The Test Manager is configured to automate the testing of the lane-following application. Open the
HighwayLaneFollowingTestAssessments.mldatx test file in the Test Manager.
sltestmgr; testFile = sltest.testmanager.load('HighwayLaneFollowingTestAssessments.mldatx');
Observe the populated test cases that were authored previously in this file. Each test case is linked to the corresponding requirement in the Requirements Editor for traceability. Each test case uses the
POST-LOAD callback to run the setup script with appropriate inputs and to configure the output video file name. After the simulation of the test case, it invokes
helperGenerateFilesForLaneFollowingReport from the
CLEAN-UP callback to generate the plots explained in the Highway Lane Following example.
Run and explore results for a single test scenario:
To reduce command-window output, turn off the MPC update messages.
To test the system-level model with the
scenario_LFACC_03_Curve_StopnGo test scenario from Simulink Test, use this code:
testSuite = getTestSuiteByName(testFile,'Test Scenarios'); testCase = getTestCaseByName(testSuite,'scenario_LFACC_03_Curve_StopnGo'); resultObj = run(testCase);
To generate a report after the simulation, use this code:
sltest.testmanager.report(resultObj,'Report.pdf',..., 'Title','Highway Lane Following',... 'IncludeMATLABFigures',true,... 'IncludeErrorMessages',true,... 'IncludeTestResults',0,'LaunchReport',true);
Examine the report
Report.pdf. Observe that the Test environment section shows the platform on which the test is run and the MATLAB® version used for testing. The Summary section shows the outcome of the test and duration of the simulation in seconds. The Results section shows pass/fail results based on the assessment criteria. This section also shows the plots logged from the
Run and explore results for all test scenarios:
You can simulate the system for all the tests by using
sltest.testmanager.run. Alternatively, you can simulate the system by clicking Play in the Test Manager app.
After completion of the test simulations, the results for all the tests can be viewed in the Results and Artifacts tab of the Test Manager. For each test case, the Check Static Range (Simulink) blocks in the model are associated with the Test Manager to visualize overall pass/fail results.
You can find the generated report in current working directory. This report contains a detailed summary of pass/fail statuses and plots for each test case.
Verify test status in Requirements Editor:
Open the Requirements Editor and select Display. Then, select Verification Status to see a verification status summary for each requirement. Green and red bars indicate the pass/fail status of simulation results for each test.
Automate Testing with Generated Code
HighwayLaneFollowingTestBench model enables integrated testing of Lane Marker Detector, Vehicle Detector, Forward Vehicle Sensor Fusion, Lane Following Decision Logic, and Lane Following Controller components. It is often helpful to perform regression testing of these components through software-in-the-loop (SIL) verification. If you have Embedded Coder™ Simulink Coder™ license, then you can generate code for these components. This workflow lets you verify that the generated code produces expected results that match the system-level requirements throughout simulation.
Set Lane Marker Detector to run in Software-in-the-loop mode.
model = 'HighwayLaneFollowingTestBench/Lane Marker Detector'; set_param(model,'SimulationMode','Software-in-the-loop');
Set Vehicle Detector to run in Software-in-the-loop mode.
model = 'HighwayLaneFollowingTestBench/Vehicle Detector'; set_param(model,'SimulationMode','Software-in-the-loop');
Set Forward Vehicle Sensor Fusion to run in Software-in-the-loop mode.
model = 'HighwayLaneFollowingTestBench/Forward Vehicle Sensor Fusion'; set_param(model,'SimulationMode','Software-in-the-loop');
Set Lane Following Decision Logic to run in Software-in-the-loop mode.
model = 'HighwayLaneFollowingTestBench/Lane Following Decision Logic'; set_param(model,'SimulationMode','Software-in-the-loop');
Set Lane Following Controller to run in Software-in-the-loop mode.
model = 'HighwayLaneFollowingTestBench/Lane Following Controller'; set_param(model,'SimulationMode','Software-in-the-loop');
sltest.testmanager.run to simulate the system for all the test scenarios. After the completion of tests, review the plots and results in the generated report.
Enable the MPC update messages again.
Automate Testing in Parallel
If you have a Parallel Computing Toolbox™ license, then you can configure Test Manager to execute tests in parallel using a parallel pool. To run tests in parallel, save the models after disabling the runtime visualizations using
save_system('HighwayLaneFollowingTestBench'). Test Manager uses the default Parallel Computing Toolbox cluster and executes tests only on the local machine. Running tests in parallel can speed up execution and decrease the amount of time it takes to get test results. For more information on how to configure tests in parallel from the Test Manager, see Run Tests Using Parallel Execution (Simulink Test).