How to write a composite contraint or comparator to be usable in unittest framework

I'm struggeling with a supposedly simple task:

For a given example class

classdef SimpleClass
    properties
        some_string (1,1) string = "some_string"
        some_vague_number (1,1) double = 0
        some_precise_number (1,1) double = pi()
    end
end

I want to write a constraint that checks "similarity" of two instances of the class. The strings should be equal and I want to parametrize the tolerances for each property separately. Clearly i could verify the conditions one after another but I would have to repeat that for each qualification, which is not good, when the class changes.

In short, I would like to have a class that combines a sequence of Comparators (pluggable into IsEqualTo(...'Using'...) ) or a constraint that uses the tolerances for checking and the diagnostic reporting thereof.

I have already made some attempts that don't work:

• Derive a class SimpleClassComparator from IsEqualTo and pass an array of PublicPropertyComparator, each just looking at one field, to the base class constructor. But the the tests always pass.
• Derive a class from contraint and implement the abstract methods as a sequence of my nested constraints for each field. That had some other problems and cannot be used for containers.

I can provide code for my approaches but firstly I would like to know:

What is the right way to implement a custom Comparator for the unittest framework?

[Jan Tusch]

Here is the code of my first approach:

runtests('test_simple_class')

produces a 'Pass' !

[per isakson]

BUMP

I tried to understand what's going on by debug stepping through the code. The size of dbstack was 36 when I found a real "low-level" comparison. I found comparisons of property names, but not propery values.

Proposal: replace

some_string (1,1) string = "some_string"

by

some_string (1,1) string = "factory_string_value"

Have you found the answer?

[Jan Tusch]

Sorry Per,

I don't get the point of your message.

Could you explain in more detail, how that is of any help to solve my problem?

[per isakson]

I'm amazed of the sheer size of the code of the Matlab unit testing framework. The Myth tells that the first unit testing framework was designed and implemented during the flight back from Switzerland to the US.

I tried to step through the code to find the cause of your problem. At the dbstack depth of 36 it was hard to stay concentrated and keep track of a variable, the name and value of which are identical. Thus, I proposed you to change the value.

[Jan Tusch]

Thanks, for making that clear and thanks for your efforts.

I totally agree, that understanding the workflow of test execution is a nightmare. To make things worse, the interfaces are undocumented, hidden or even sealed with a lot of side-effects and magic in the details.

By now I don't think that there is a straightforward way of doing what I want by just assembling framework components and glue them togethher in a customized comparator.

Maybe the "framework" is not as mature as it appears at first sight.

[David Hruska]

Hi Jan and others,

I was going to suggest writing a custom comparator subclass, but it seems you've already tried this and run into issues. Could you expand on what those issues were? Also, could you clarify how you'd like your constraint to work with containers? Are you saying that you'd like to compare two SimpleClass instances stored inside cell arrays, for instance?

As you've found, comparing values can be trickier than you might expect. For this reason, the Comparator interface isn't documented for extension (it may change in future releases as the framework evolves). We'd definitely want to address these challenges in writing a custom comparator before documenting the interface.

Answer 1

[Lawrence]

A bit of a late reply, but here's a solution for anyone still needing one.

Probably implementing a custom constraint is simpler than implementing a custom comparator. You would follow the steps here: https://www.mathworks.com/help/matlab/matlab_prog/create-custom-constraint.html. I'm not sure what you mean by "That had some other problems and cannot be used for containers", but the code below works with the SimpleClass you provided.

In short, the constructor of the constraint would take an expected value and two additional tolerance inputs, all of which are stored to member variables. Then you would have a helper function to do the comparisons.

classdef similarTo < matlab.unittest.constraints.Constraint
    properties (SetAccess = immutable)
        expected
        large_tolerance
        small_tolerance
    end
    methods
        function constraint = similarTo(expected,large_tolerance,small_tolerance)
            constraint.expected = expected;
            constraint.large_tolerance = large_tolerance;
            constraint.small_tolerance = small_tolerance;
        end
        function bool = satisfiedBy(constraint,actual)
            bool = constraint.customSimilarTo(actual);
        end
        function diag = getDiagnosticFor(constraint,actual)
            import matlab.unittest.diagnostics.StringDiagnostic
            if constraint.customSimilarTo(actual)
                diag = StringDiagnostic('similarTo has passed.');
            else
                diag = StringDiagnostic(sprintf(['similarTo has failed.\n'...
                    'Actual string: %s, Expected string: %s\n'...
                    'Actual vagueNum: %d, Expected vagueNum: %d, Precision: %d\n'...
                    'Actual preciseNum: %d, Expected preciseNum: %d, Precision: %d'],...
                    actual.some_string, constraint.expected.some_string,...
                    actual.some_vague_number, constraint.expected.some_vague_number, constraint.large_tolerance...
                    actual.some_precise_number, constraint.expected.some_precise_number, constraint.small_tolerance));
            end
        end
    end
    
    methods (Access = private)
        function bool = customSimilarTo(constraint, actual)
            bool = strcmp(actual.some_string, expected.some_string)...
                && abs(actual.some_vague_number - constraint.expected.some_vague_number) < constraint.large_tolerance...
                && abs(actual.some_precise_number - constraint.expected.some_precise_number) < constraint.small_tolerance;
        end
    end
end

You then use this in your unit test:

verifyThat(testCase, actual, similarTo(expected, large_tolerance, small_tolerance))

Some customizations you could do would be baking the tolerances into the constraint class, either within similarTo or by sub-classing from it, and splitting the conditions in customSimilarTo into separate helper functions, so you can have a more granular diagnostic in getDiagnosticFor.

If by "cannot be used for containers" you are talking about arrays and cell arrays, you can combine the custom similarTo constraint with matlab's EveryElementOf and EveryCellOf without any additional modification.

https://www.mathworks.com/help/matlab/ref/matlab.unittest.constraints.everyelementof-class.html

https://www.mathworks.com/help/matlab/ref/matlab.unittest.constraints.everycellof-class.html

[Lawrence]

For those who don't need a re-usable custom constraint but just want to combine constraints by hand, many accept the boolean operations |, &, and ~.

https://www.mathworks.com/help/matlab/ref/matlab.unittest.constraints.booleanconstraint-class.html

verifyThat(testCase,[], matlab.unittest.constraints.IsEmpty |...
    matlab.unittest.constraints.IsEqualTo(0));

[Lawrence]

Here's a shorter class from the one in my answer above which makes use of MATLAB's built-in constraint primitives:

classdef similarTo < matlab.unittest.constraints.AndConstraint
    methods
        function constraint = similarTo(expected,large_tolerance,small_tolerance)
            import matlab.unittest.constraints.IsEqualTo
            import matlab.unittest.constraints.AbsoluteTolerance
            import matlab.unittest.constraints.AndConstraint
            import matlab.unittest.constraints.PublicPropertyComparator
            import matlab.unittest.constraints.StringComparator
            import matlab.unittest.constraints.NumericComparator
            props = properties(expected);
            [e-mail](...
                IsEqualTo(expected,...
                    'Using',PublicPropertyComparator(StringComparator,'IgnoringProperties',props(~strcmp(props,'some_string')))...
                    ),...
                AndConstraint(IsEqualTo(expected,...
                    'Within',AbsoluteTolerance(large_tolerance),...
                    'Using',PublicPropertyComparator(NumericComparator,'IgnoringProperties',props(~strcmp(props,'some_vague_number')))...
                    ),...
                IsEqualTo(expected,...
                    'Within',AbsoluteTolerance(small_tolerance),...
                    'Using',PublicPropertyComparator(NumericComparator,'IgnoringProperties',props(~strcmp(props,'some_precise_number')))...
                )))
        end
    end
end

I'll admit it's a bit ugly at first glance, but it allows you to avoid having to define satisfiedBy or getDiagnosticFor or modify them if you modify the class. The only shortcoming I can see is that NumericComparator and StringComparator don't have a class-insensitive option, so 'asdf' compared to "asdf" will fail, as will double(3) compared to uint8(3). The hand-rolled constraint above allows you to have these compare equal.

[Jan Tusch]

That looks like reasonable effort for adapting constraints for a class and keep tolerances configurable. Thanks alot for your code sample and the good explanation.