Main Content

Process Test Results with Custom Scripts

Testing your model often requires assessing conditions that ensure a test is valid, in addition to verifying model behavior. MATLAB® Unit Test provides a framework for such assessments. In Simulink® Test™, you can use the test case custom criteria to author specific assessments, and include MATLAB Unit Test qualifications in your script.

Custom criteria apply as post-simulation criteria to the simulation output. See Simulink.SimulationOutput. If you require run-time verifications, use a verify() statement in a Test Assessment or Test Sequence block. See Assess Model Simulation Using verify Statements.

MATLAB Testing Framework

A custom criteria script is a method of test, which is a matlab.unittest test case object. To enable the function, in the test case Custom Criteria section of the Test Manager, select function customCriteria(test). Inside the function, enter the custom criteria script in the embedded MATLAB editor.

The embedded MATLAB editor lists properties of test. Create test assessments using MATLAB Unit Test qualifications. Custom criteria supports verification and assertion type qualifications. See Table of Verifications, Assertions, and Other Qualifications. Verifications and assertions operate differently when custom criteria are evaluated:

  • Verifications – Other assessments are evaluated when verifications fail. Diagnostics appear in the results. Use verifications for general assessments, such as checking simulation against expected outputs.

    Example: test.verifyEqual(lastValue,0)

  • Assertions – The custom criteria script stops evaluating when an assertion fails. Diagnostics appear in the results. Use assertions for conditions that render the criteria invalid.

    Example: test.assertEqual(lastValue,0).

Define a Custom Criteria Script

This example shows how to create a custom criteria script for an autopilot test case.

  1. Open the test file.

  2. In the Test Browser, select AutopilotTestFile > Basic Design Test Cases > Requirement 1.3 Test. In the test case, expand the Custom Criteria section.

  3. Enable the custom criteria script by selecting function customCriteria(test).

  4. In the embedded MATLAB editor, enter the following script. The script gets the final value of the signals Phi and APEng, and verifies that the final values equal 0. Signals_Req1_3 is one of the test requirements and sltest_simout is the simulation output.

    % Get the last values
    lastPhi = test.sltest_simout.get...
    lastAPEng = test.sltest_simout.get...
    % Verify the last values equal 0
       ['Final Phi value: ',num2str(lastPhi),'.']);
       ['Final APEng value: ',num2str(lastAPEng),'.']);
  5. Run the test case.

  6. In the Results and Artifacts pane, expand the Custom Criteria Result. Both criteria pass.

    Custom criteria results

Reuse Custom Criteria and Debug Using Breakpoints

In addition to authoring criteria scripts in the embedded MATLAB editor, you can author custom criteria in a standalone function, and call the function from the test case. Using a standalone function allows you

  • To reuse the custom criteria in multiple test cases.

  • To set breakpoints in the criteria script for debugging.

  • To investigate the simulation output using the command line.

In this example, you add a breakpoint to a custom criteria script. You run the test case, list the properties of the test object at the command line, and call the custom criteria from the test case.

Call Custom Criteria Script from the Test Case

  1. Navigate to the folder containing the criteria function.

  2. Open the custom criteria script


    % This is a custom criteria function for a Simulink Test test case.
    % The function gets the last values of Phi and APEng from the 
    % Requirements 1.3 test case in the test file AutopilotTestFile.
    function sltestCheckFinalRollRefValues(test)
    % Get the last values
    lastPhi = test.sltest_simout.get...
    lastAPEng = test.sltest_simout.get...
    % Verify the last values equal 0
       ['Final Phi value: ',num2str(lastPhi),'.']);
       ['Final APEng value: ',num2str(lastAPEng),'.']);
  3. Open the test file


  4. In the embedded MATLAB editor under Custom Criteria, enter the function call to the custom criteria:


Set Breakpoints and List test Properties

  1. On line 8 of sltestCheckFinalRollRefValues.m, set a breakpoint by clicking the dash to the right of the line number.

  2. In the Test Manager, run the test case.

    The command window displays a debugging prompt.

  3. Enter test at the command prompt to display the properties of the STMCustomCriteria object. The properties contain characteristics and simulation data output of the test case.

    test = 
      STMCustomCriteria with properties:
                  TestResult: [1×1 sltest.testmanager.TestCaseResult]
                      sltest_simout: [1×1 Simulink.SimulationOutput]
             sltest_testCase: [1×1 sltest.testmanager.TestCase]
               sltest_bdroot: {'RollReference_Requirement1_3'}
                  sltest_sut: {'RollAutopilotMdlRef/Roll Reference'}
            sltest_isharness: 1
        sltest_iterationName: ''

    The property sltest_simout contains the simulation data. To view the data PhiRef, enter

    ans = 
      Package: Simulink.SimulationData
      struct with fields:
                  Name: 'PhiRef'
        PropagatedName: ''
             BlockPath: [1×1 Simulink.SimulationData.BlockPath]
              PortType: 'outport'
             PortIndex: 1
                Values: [1×1 timeseries]
  4. In the MATLAB editor, click Continue to continue running the custom criteria script.

  5. In the Results and Artifacts pane, expand the Custom Criteria Result. Both criteria pass.

  6. To reuse the script in another test case, call the function from the test case custom criteria.

Custom Criteria Programmatic Interface Example

This example shows how to set and get custom criteria using the programmatic interface.

Before running this example, temporarily disable warnings that result from verification failures.

warning off Stateflow:Runtime:TestVerificationFailed;
warning off Stateflow:cdr:VerifyDangerousComparison;

Load a Test File and Get Test Case Object

tf = sltest.testmanager.load('AutopilotTestFile.mldatx');

ts = getTestSuiteByName(tf,'Basic Design Test Cases');

tc = getTestCaseByName(ts,'Requirement 1.3 Test');

Create the Custom Criteria Object and Set Criteria

Create the custom criteria object.

tcCriteria = getCustomCriteria(tc)
tcCriteria = 
  CustomCriteria with properties:

     Enabled: 0
    Callback: '% Return value: customCriteria...'

Create the custom criteria expression. This script gets the last value of the signal Phi and verifies that it equals 0.

criteria = ...
    sprintf(['lastPhi = test.SimOut.get(''Signals_Req1_3'')',...
	'test.verifyEqual(lastPhi,0,[''Final: '',num2str(lastPhi),''.'']);'])
criteria = 
    'lastPhi = test.SimOut.get('Signals_Req1_3').get('Phi').Values.Data(end);
     test.verifyEqual(lastPhi,0,['Final: ',num2str(lastPhi),'.']);'

Set and enable the criteria.

tcCriteria.Callback = criteria;
tcCriteria.Enabled = true;

Run the Test Case and Get the Results

Run the test case.

tcResultSet = run(tc);

Get the test case results.

tcResult = getTestCaseResults(tcResultSet);

Get the custom criteria result.

ccResult = getCustomCriteriaResult(tcResult)
ccResult = 
  CustomCriteriaResult with properties:

             Outcome: Failed
    DiagnosticRecord: [1x1 sltest.testmanager.DiagnosticRecord]

Restore warnings from verification failures.

warning on Stateflow:Runtime:TestVerificationFailed;
warning on Stateflow:cdr:VerifyDangerousComparison;

Related Topics