matlab.perftest.TimeExperiment.limitingSamplingError

Class: matlab.perftest.TimeExperiment
Namespace: matlab.perftest

Construct time experiment for specified margin of error and confidence level

Syntax

experiment = matlab.perftest.TimeExperiment.limitingSamplingError

experiment = matlab.perftest.TimeExperiment.limitingSamplingError(Name,Value)

Description

experiment = matlab.perftest.TimeExperiment.limitingSamplingError constructs a time experiment for each test suite element, with the specified statistical objectives (such as margin of error and confidence level). This method returns an instance of FrequentistTimeExperiment. This syntax uses the following defaults to determine the number of sample measurements.

• Number of warm-up measurements: 5

• Minimum number of samples: 4

• Maximum number of samples collected in the event other statistical objectives are not met: 256

• Objective relative margin of error for samples: 0.05 (5%)

• Confidence level for samples to be within relative margin of error: 0.95 (95%)

experiment = matlab.perftest.TimeExperiment.limitingSamplingError(Name,Value) constructs a time experiment with additional options specified by one or more name-value arguments. Use this syntax to override the defaults listed above.

Input Arguments

Name-Value Arguments

Specify optional pairs of arguments as Name1=Value1,...,NameN=ValueN, where Name is the argument name and Value is the corresponding value. Name-value arguments must appear after other arguments, but the order of the pairs does not matter.

Before R2021a, use commas to separate each name and value, and enclose Name in quotes.

Example: experiment = matlab.perftest.TimeExperiment.limitingSamplingError('RelativeMarginOfError',0.12,'MaxSamples',100) creates a time experiment that collects sample measurements until samples have a relative margin of error of 12%, or until it collects 100 measurements.

NumWarmups — Number of warm-up measurements

5 (default) | nonnegative integer

Number of warm-up measurements, specified as a nonnegative integer. NumWarmups defines the number of times that the testing framework runs the test code to warm it up.

MinSamples — Minimum number of samples

4 (default) | positive integer

Minimum number of sample measurements, specified as a positive integer. The value defines the minimum number of times that the framework exercises the test code after any warm-up runs. The framework exercises the test code at least MinSamples times, regardless of whether the experiment meets the statistical objectives.

MaxSamples — Maximum number of samples

256 (default) | positive integer

Maximum number of sample measurements, specified as a positive integer. The value defines the maximum number of times that the framework exercises the test code after NumWarmups. If the experiment does not meet the statistical objectives, the framework collects up to MaxSamples.

RelativeMarginOfError — Objective relative margin of error for samples

0.05 (default) | positive number

Objective relative margin of error for samples, specified as a positive number.

The framework calculates the relative margin of error for a sample X using the equation

$$\text{relMoE}=\frac{\text{T}\ast \text{std(X)}}{\text{mean(X)}\ast \sqrt{\text{length(X)}}}$$

where T is the T-score from Student's T distribution using the specified ConfidenceLevel and length(X)-1 degrees of freedom.

ConfidenceLevel — Confidence level for samples to be within relative margin of error

0.95 (default) | number between 0 and 1

Confidence level for the samples to be within the relative margin of error, specified as a number from 0 through 1.

Examples

Performance Test with Variable Number of Measurements

In your current folder, create a class-based test, preallocationTest.m, that compares different methods of preallocation.

classdef preallocationTest < matlab.perftest.TestCase
    methods (Test)
        function testOnes(testCase)
            x = ones(1,1e7);
        end
        function testIndexingWithVariable(testCase)
            id = 1:1e7;
            x(id) = 1;
        end
        function testIndexingOnLHS(testCase)
            x(1:1e7) = 1;
        end
        function testForLoop(testCase)
            for i=1:1e7
                x(i) = 1;
            end
        end
    end
end

Create a test suite.

suite = testsuite("preallocationTest");

Construct a time experiment with a variable number of sample measurements, and run the tests.

import matlab.perftest.TimeExperiment
experiment = TimeExperiment.limitingSamplingError;
results = run(experiment,suite);

Running preallocationTest
.......... .......... .......... ......
Done preallocationTest
__________

View the test activity for the first test. For this test, the performance testing framework collected five warm-up measurements (the default), and four sample measurements. After four sample measurements, the framework satisfied the default statistical objectives.

results(1).TestActivity

ans =

  9×12 table

               Name               Passed    Failed    Incomplete    MeasuredTime    Objective         Timestamp             Host        Platform                 Version                            TestResult                         RunIdentifier            
    __________________________    ______    ______    __________    ____________    _________    ____________________    ___________    ________    __________________________________    ______________________________    ____________________________________

    preallocationTest/testOnes    true      false       false         0.016621       warmup      12-Oct-2022 16:19:12    MY-HOSTNAME     win64      9.14.0.2078117 (R2023a) Prerelease    1×1 matlab.unittest.TestResult    a04b1262-f4c3-4af5-b68e-ae54a6ae24b1
    preallocationTest/testOnes    true      false       false         0.016783       warmup      12-Oct-2022 16:19:12    MY-HOSTNAME     win64      9.14.0.2078117 (R2023a) Prerelease    1×1 matlab.unittest.TestResult    a04b1262-f4c3-4af5-b68e-ae54a6ae24b1
    preallocationTest/testOnes    true      false       false         0.016685       warmup      12-Oct-2022 16:19:12    MY-HOSTNAME     win64      9.14.0.2078117 (R2023a) Prerelease    1×1 matlab.unittest.TestResult    a04b1262-f4c3-4af5-b68e-ae54a6ae24b1
    preallocationTest/testOnes    true      false       false         0.017241       warmup      12-Oct-2022 16:19:12    MY-HOSTNAME     win64      9.14.0.2078117 (R2023a) Prerelease    1×1 matlab.unittest.TestResult    a04b1262-f4c3-4af5-b68e-ae54a6ae24b1
    preallocationTest/testOnes    true      false       false         0.017496       warmup      12-Oct-2022 16:19:12    MY-HOSTNAME     win64      9.14.0.2078117 (R2023a) Prerelease    1×1 matlab.unittest.TestResult    a04b1262-f4c3-4af5-b68e-ae54a6ae24b1
    preallocationTest/testOnes    true      false       false         0.016733       sample      12-Oct-2022 16:19:12    MY-HOSTNAME     win64      9.14.0.2078117 (R2023a) Prerelease    1×1 matlab.unittest.TestResult    a04b1262-f4c3-4af5-b68e-ae54a6ae24b1
    preallocationTest/testOnes    true      false       false         0.016654       sample      12-Oct-2022 16:19:12    MY-HOSTNAME     win64      9.14.0.2078117 (R2023a) Prerelease    1×1 matlab.unittest.TestResult    a04b1262-f4c3-4af5-b68e-ae54a6ae24b1
    preallocationTest/testOnes    true      false       false         0.016602       sample      12-Oct-2022 16:19:12    MY-HOSTNAME     win64      9.14.0.2078117 (R2023a) Prerelease    1×1 matlab.unittest.TestResult    a04b1262-f4c3-4af5-b68e-ae54a6ae24b1
    preallocationTest/testOnes    true      false       false         0.017102       sample      12-Oct-2022 16:19:12    MY-HOSTNAME     win64      9.14.0.2078117 (R2023a) Prerelease    1×1 matlab.unittest.TestResult    a04b1262-f4c3-4af5-b68e-ae54a6ae24b1

Construct a time experiment that collects two warm-up measurements and runs the tests a variable number of times to reach a sample mean with a 10% relative margin of error within a 90% confidence level.

experiment = TimeExperiment.limitingSamplingError("NumWarmups",2, ...
    "RelativeMarginOfError",0.10,"ConfidenceLevel",0.90);
results = run(experiment,suite);

Running preallocationTest
.......... .......... ....
Done preallocationTest
__________

View the test activity for the first test. For this test, the framework collected two warm-up measurements and four sample measurements. After four sample measurements, the framework satisfied the specified statistical objectives.

results(1).TestActivity

ans =

  6×12 table

               Name               Passed    Failed    Incomplete    MeasuredTime    Objective         Timestamp             Host        Platform                 Version                            TestResult                         RunIdentifier            
    __________________________    ______    ______    __________    ____________    _________    ____________________    ___________    ________    __________________________________    ______________________________    ____________________________________

    preallocationTest/testOnes    true      false       false         0.016576       warmup      12-Oct-2022 16:23:04    MY-HOSTNAME     win64      9.14.0.2078117 (R2023a) Prerelease    1×1 matlab.unittest.TestResult    932317a8-bb61-4cba-945e-082f38eb0ced
    preallocationTest/testOnes    true      false       false         0.016709       warmup      12-Oct-2022 16:23:04    MY-HOSTNAME     win64      9.14.0.2078117 (R2023a) Prerelease    1×1 matlab.unittest.TestResult    932317a8-bb61-4cba-945e-082f38eb0ced
    preallocationTest/testOnes    true      false       false         0.016672       sample      12-Oct-2022 16:23:04    MY-HOSTNAME     win64      9.14.0.2078117 (R2023a) Prerelease    1×1 matlab.unittest.TestResult    932317a8-bb61-4cba-945e-082f38eb0ced
    preallocationTest/testOnes    true      false       false         0.017231       sample      12-Oct-2022 16:23:04    MY-HOSTNAME     win64      9.14.0.2078117 (R2023a) Prerelease    1×1 matlab.unittest.TestResult    932317a8-bb61-4cba-945e-082f38eb0ced
    preallocationTest/testOnes    true      false       false         0.017646       sample      12-Oct-2022 16:23:04    MY-HOSTNAME     win64      9.14.0.2078117 (R2023a) Prerelease    1×1 matlab.unittest.TestResult    932317a8-bb61-4cba-945e-082f38eb0ced
    preallocationTest/testOnes    true      false       false         0.016836       sample      12-Oct-2022 16:23:04    MY-HOSTNAME     win64      9.14.0.2078117 (R2023a) Prerelease    1×1 matlab.unittest.TestResult    932317a8-bb61-4cba-945e-082f38eb0ced

Version History

Introduced in R2016a

R2023a: Default number of warm-up measurements has increased

The default number of times that the framework exercises the test code to warm it up in a frequentist time experiment has increased from four to five. This change results in typically fewer samples required to meet the objective relative margin of error.

If you want to use the previous default value, explicitly specify the value in your code. This table shows an example of how to update code that runs tests using four warm-up measurements.

Before	After
import matlab.perftest.TimeExperiment experiment = TimeExperiment.limitingSamplingError; results = run(experiment,mySuite);	import matlab.perftest.TimeExperiment experiment = TimeExperiment.limitingSamplingError("NumWarmups",4); results = run(experiment,mySuite);