verifyEqualsBaseline

Verify value is equal to baseline data

Since R2024b

Syntax

verifyEqualsBaseline(testCase,actual,baseline)

verifyEqualsBaseline(testCase,actual,baseline,diagnostic)

verifyEqualsBaseline(___,Name=Value)

Description

verifyEqualsBaseline(testCase,actual,baseline) verifies that actual is strictly equal to the baseline data represented by baseline. The method compares values in the same way as the verifyEqual method.

You can use the verifyEqualsBaseline method within a test class that specifies baseline parameters. If the qualification fails, the testing framework provides you with options to either create new baseline data or update the existing baseline data using the actual value. For more information about baseline testing, see Create Baseline Tests for MATLAB Code.

verifyEqualsBaseline(testCase,actual,baseline,diagnostic) associates the diagnostic information in diagnostic with the qualification.

verifyEqualsBaseline(___,Name=Value) specifies options using one or more name-value arguments in addition to other input argument combinations in previous syntaxes. For example, verifyEqualsBaseline(testCase,actual,baseline,RelTol=0.01) verifies that the difference between the corresponding elements of the actual and baseline numeric arrays is within 1%.

Input Arguments

testCase — Test case

matlab.unittest.qualifications.Verifiable object

Test case, specified as a matlab.unittest.qualifications.Verifiable object. Because the matlab.unittest.TestCase class subclasses matlab.unittest.qualifications.Verifiable and inherits its methods, testCase is typically a matlab.unittest.TestCase object.

actual — Value to test

any value

Value to test, specified as a value of any data type. To create or update the baseline data, the testing framework must be able to save and load actual. For more information about the save and load operations, see Default Save and Load Process for Objects.

baseline — Representation of baseline data

matlabtest.baselines.MATFileBaseline object

Representation of the baseline data, specified as a matlabtest.baselines.MATFileBaseline object.

diagnostic — Diagnostic information to display

string array | character array | function handle | array of matlab.automation.diagnostics.Diagnostic objects

Diagnostic information to display when the qualification passes or fails, specified as a string array, character array, function handle, or array of matlab.automation.diagnostics.Diagnostic objects.

Depending on the test runner configuration, the testing framework can display diagnostics when the qualification passes or fails. By default, the framework displays diagnostics only when the qualification fails. You can override the default behavior by customizing the test runner. For example, use a DiagnosticsOutputPlugin instance to display both failing and passing event diagnostics.

Example: "My Custom Diagnostic"

Example: @dir

Name-Value Arguments

Specify optional pairs of arguments as Name1=Value1,...,NameN=ValueN, where Name is the argument name and Value is the corresponding value. Name-value arguments must appear after other arguments, but the order of the pairs does not matter.

Example: verifyEqualsBaseline(testCase,actual,baseline,RelTol=0.01) verifies that the difference between the corresponding elements of the actual and baseline numeric arrays is within 1%.

AbsTol — Absolute tolerance

nonnegative real scalar

Absolute tolerance when comparing numeric arrays in a baseline test, specified as a nonnegative real scalar. For an absolute tolerance to be satisfied by an actual array A and baseline array B, abs(B-A) <= AbsTol must be true.

Data Types: double | single | int8 | int16 | int32 | int64 | uint8 | uint16 | uint32 | uint64

RelTol — Relative tolerance

nonnegative real scalar

Relative tolerance when comparing numeric arrays in a baseline test, specified as a nonnegative real scalar. For a relative tolerance to be satisfied by an actual array A and baseline array B, abs(B-A) <= RelTol.*abs(B) must be true.

Data Types: double | single | int8 | int16 | int32 | int64 | uint8 | uint16 | uint32 | uint64

Examples

Create Simple Baseline Test

Create a baseline test to test a function whose output is not straightforward to verify through automated testing. Run the test to create baseline data in a MAT file from the "known good" function output. Run the test a second time to test the function output against the created baseline data.

First, manually verify that your function behaves as expected. For example, you can inspect the function output in the Command Window. For illustrative purposes, this example uses the magic function. In practice, you work with user-defined code.

M = magic(5)

M =

    17    24     1     8    15
    23     5     7    14    16
     4     6    13    20    22
    10    12    19    21     3
    11    18    25     2     9

To create a baseline test for your function, you must define a baseline and then associate it with a test using a test class. Create the ExampleTest test class in a file named ExampleTest.m in your current folder. Then, complete the test class code by following these steps:

1. In a properties block with the TestParameter attribute, add the baseline property to define a baseline.

2. Set the baseline property using the matlabtest.parameters.matfileBaseline function. The function creates a baseline parameter that defines the data in the testdata.mat file as baseline data.

3. In a methods block with the Test attribute, add the parameterized baselineTest method that accepts baseline as an input.

4. Implement the baselineTest method by specifying the actual value and then using the verifyEqualsBaseline method to test the actual value against the baseline data.

classdef ExampleTest < matlab.unittest.TestCase
    properties (TestParameter)        
        baseline = matlabtest.parameters.matfileBaseline("testdata.mat")
    end

    methods (Test)
        function baselineTest(testCase,baseline)
            actual = magic(5);
            testCase.verifyEqualsBaseline(actual,baseline)
        end
    end
end

Run the test. The test fails because the testdata.mat file does not yet exist and there is no baseline data to compare the actual value against. However, the test diagnostic includes a hyperlink that enables you to create baseline data from the actual value that the testing framework records.

result = runtests("ExampleTest");

Running ExampleTest

================================================================================
Verification failed in ExampleTest/baselineTest(baseline=testdata.mat(data)).
    ---------------------
    Framework Diagnostic:
    ---------------------
    verifyEqualsBaseline failed.
    --> An error occurred when loading baseline data: 
    --> MATLAB:MatFile:NoFile
    --> Cannot access 'data' because 'C:\work\testdata.mat' does not exist.
    --> Create baseline from recorded test data
    ------------------
    Stack Information:
    ------------------
    ...
    In C:\work\ExampleTest.m (ExampleTest.baselineTest) at 9
================================================================================
.
Done ExampleTest
__________

Failure Summary:

     Name                                                 Failed  Incomplete  Reason(s)
    ==================================================================================================
     ExampleTest/baselineTest(baseline=testdata.mat(data))    X                 Failed by verification.

Because you already inspected the actual value before creating the baseline test, approve it as baseline data by clicking Create baseline from recorded test data. The framework creates the testdata.mat file including the actual value as baseline data and displays a dialog box to confirm the successful operation. Now that baseline data exists for your test, you can verify that your function continues to produce the same output by running the baseline test. If the function output changes, for instance, due to code refactoring, the baseline test signals the change through a qualification failure. The test passes only if the function output remains the same as the baseline data in the MAT file.

Run the test again to test the function output against the baseline data. In this case, the test passes because the function has not changed since baseline data creation.

result = runtests("ExampleTest");

Running ExampleTest
.
Done ExampleTest
__________

Tips

• verifyEqualsBaseline is a convenience method. For example, verifyEqualsBaseline(testCase,actual,baseline) is functionally equivalent to the following code.

  import matlabtest.constraints.EqualsBaseline
  testCase.verifyThat(actual,EqualsBaseline(baseline))

  Similarly, verifyEqualsBaseline(testCase,actual,baseline,AbsTol=abstol,RelTol=reltol) is functionally equivalent to the following code.

  import matlabtest.constraints.EqualsBaseline
  import matlab.unittest.constraints.AbsoluteTolerance
  import matlab.unittest.constraints.RelativeTolerance
  testCase.verifyThat(actual,EqualsBaseline(baseline, ...
      Within=AbsoluteTolerance(abstol) | RelativeTolerance(reltol)))

  More functionality is available when using the EqualsBaseline constraint directly with verifyThat.

• Use verification qualifications to produce and record failures without throwing an exception. Because verifications do not throw exceptions, the test content runs to completion even when verification failures occur. Typically, verifications are the primary qualification for a test because they typically do not require an early exit from the test. Use other qualification types to test for violation of preconditions or incorrect test setup:

  • Use assumption qualifications to make the test environment meet preconditions that otherwise do not result in a test failure. Assumption failures result in filtered tests, and the testing framework marks the tests as Incomplete. For more information, see matlab.unittest.qualifications.Assumable.

  • Use assertion qualifications when the failure condition invalidates the remainder of the current test content but does not prevent execution of subsequent tests. A failure at the assertion point renders the current test as Failed and Incomplete. For more information, see matlab.unittest.qualifications.Assertable.

  • Use fatal assertion qualifications to stop the test session upon failure. These qualifications are useful when the failure is so fundamental that continuing testing does not make sense. Fatal assertion qualifications are also useful when fixture teardown does not restore the environment state, and stopping testing and starting a fresh session is preferable. For more information, see matlab.unittest.qualifications.FatalAssertable.

Version History

Introduced in R2024b