Unit tests fail on "even" runs, but pass on "odd" runs (alternating pass/fail)

I have some unit tests that I wrote for code that is doing some very minor image manipulation (combining several small images into a larger image). When I first run the unit tests, I noticed that three out of four of them failed on the line where they are reading an image from a directory (fails with an index out of bounds error). However, if I run it a second time, they all pass. When I was writing the code as well, I noticed that whenever I set a breakpoint in my code, I'd have to run the unit tests twice because the first time it would run through the tests without hitting any breakpoints.

I have my repo organized like this

src/
    /* source code .m files are in here */
    
unit_tests/
    images/
        squares/
            - img1.png
            - img2.png
            ...
            - imgn.png
    - unit_tests.m

And I have a line in my setup (inside the unit tests) to add paths that I generate:

function tests = unit_tests()
    addpath(genpath('..'));    
    tests = functiontests(localfunctions);
end

At first, I thought that perhaps the unit tests were executing too quickly and it was running the unit tests before the addpath/genpath functions had finished, so I added a pause statement and re-ran the tests, but I had the same issue only this time it would wait for the requisite number of seconds before going ahead and failing. If I run it again, no problem - all my tests pass.

Here's the output of running the unit tests twice (I started on an "odd" run):

>> runtests('unit_tests\unit_tests.m')
Running unit_tests
.......
Done unit_tests
__________


ans = 

  1×7 TestResult array with properties:

    Name
    Passed
    Failed
    Incomplete
    Duration
    Details

Totals:
   7 Passed, 0 Failed, 0 Incomplete.
   0.39979 seconds testing time.

>> runtests('unit_tests')
Running unit_tests
..
================================================================================
Error occurred in unit_tests/testCompositeImage_2x2 and it did not run to completion.

    ---------
    Error ID:
    ---------
    'MATLAB:badsubscript'

    --------------
    Error Details:
    --------------
    Index exceeds array bounds.
    
    Error in unit_tests>testCompositeImage_2x2 (line 47)
        img = imread([squares(1).folder filesep squares(1).name]); % all same size
================================================================================
.
================================================================================
Error occurred in unit_tests/testCompositeImage_2x3 and it did not run to completion.

    ---------
    Error ID:
    ---------
    'MATLAB:badsubscript'

    --------------
    Error Details:
    --------------
    Index exceeds array bounds.
    
    Error in unit_tests>testCompositeImage_2x3 (line 71)
        img = imread([squares(1).folder filesep squares(1).name]); % all same size
================================================================================
.
================================================================================
Error occurred in unit_tests/testCompositeImage_3x2 and it did not run to completion.

    ---------
    Error ID:
    ---------
    'MATLAB:badsubscript'

    --------------
    Error Details:
    --------------
    Index exceeds array bounds.
    
    Error in unit_tests>testCompositeImage_3x2 (line 95)
        img = imread([squares(1).folder filesep squares(1).name]); % all same size
================================================================================
...
Done unit_tests
__________

Failure Summary:

     Name                               Failed  Incomplete  Reason(s)
    ==================================================================
     unit_tests/testCompositeImage_2x2    X         X       Errored.
    ------------------------------------------------------------------
     unit_tests/testCompositeImage_2x3    X         X       Errored.
    ------------------------------------------------------------------
     unit_tests/testCompositeImage_3x2    X         X       Errored.
    

ans = 

  1×7 TestResult array with properties:

    Name
    Passed
    Failed
    Incomplete
    Duration
    Details

Totals:
   4 Passed, 3 Failed (rerun), 3 Incomplete.
   0.0040965 seconds testing time.

The fact that it's failing on basically the first line because it's failing to read anything from the folder leads me to suspect that even when the other 4 tests supposedly pass, they are in fact not running at all. I am completely at a loss as to why this is happening; I am using vanilla matlab (R2018a) and don't have anything fancy going on.

[Geoff Hayes]

matquest - why (in the above output) do you call

runtests('unit_tests\unit_tests.m')

first and then on the second run call as

runtests('unit_tests')

? How is this input used by the runtests function?

[matquest]

I actually just noticed this and it appears to explain the bizarre output. I am just pressing the "Run Tests" button in the matlab editor tab. I have no idea why, but when I press it, it alternates between running the command:

runtests('unit_tests\unit_tests.m')

and

runtests('unit_tests')

Do you have any ideas as to why it might be doing that?

Answer 1

[matquest]

Leaving this here in case someone else runs into the same issue.

Turns out something about the line:

addpath(genpath('..'));

causes the matlab GUI to go into a weird state where pressing the "Run Tests" button alternates between calling runtests('unit_tests\unit_tests.m'); and runtests('unit_tests'); which in turn cases the tests to alternately pass and fail (run/not run). If anyone knows why this is the case or has any insights into using relative paths in matlab, I would be very interested to learn more.

Replacing that line with the following seems to resolve the issue:

pth = fullfile(fileparts(fileparts(mfilename('fullpath'))),'src'); % goes up one dir and into 'src' => ../src
paths = regexp(genpath(pth), ';', 'split');
for idx = 1:length(paths) - 1 % last element is empty
    addpath(paths{idx});
end

Restart matlab and the changes should take effect.

Answer 2

[Steven Lord]

You add a directory (or multiple directories) to your path but you never restore the path to the state it was in prior to the test running. Since your test is a function-based test you should create setup and teardown functions. The setup function will run before your test does to store the path and add directories to it. The teardown function will run after your test does and restore the path to its original value afterwards. See this documentation page for a description of how to do this.

When or if you decide to convert your tests (or write new tests) as class-based tests you could use a fixture to handle the setup and teardown once it has been applied (with applyFixture) to your test object.

[matquest]

I see. I added a setupOnce function which only includes that call and the tests now run correctly. I did see in the documentation an example for changing directories where the original path is stored in a temp variable and then restored during teardown and I think I will do this for the path.

However, I am still curious as to why this happens. I naively assumed that any changes made within a function would be cleaned up automatically as soon as the function returns. Could you elaborate on how this happens? It appears (based on running some tests and saving the value in path) that path is a global variable whose value is preserved between calls/runs of the unit tests. Which lead me to look into what happens duing the genpath function (as it seemed that it should return the same set of paths between each run). But when I stepped into the genpath function during an erroneous call, I can see that it in fact goes up two directories (equivalent to '../..') and thus give the path some erroneous paths (and yet I cannot see why this should be a problem as the path variable still contains the desired directories - namely, src and unit_tests/images/squares/). This appears to happen in the dir() function, which seems to be a compiled/encrypted file. Could you describe what happens in the dir() function?

Also of note: this behavior does not occur if I run the tests either from outside of the repository or from within the unit_tests directory itself.

[Steven Lord]

Not all changes that you make to the environment from inside the function are reversed or restored when the function exits. You have to clean up many changes that you make or use tools (like the test infrastructure's fixtures or onCleanup objects) that do it for you.

Variables you create in a function's workspace, unless you return them as output arguments or do a few other special things with them, are destroyed when the function exits. This is the nature of a function workspace.

Directories you add to or remove from the path are not removed from or added to the path respectively.

Figure windows you create inside the function are not closed.

Files you've created inside the function are not deleted.

If you change to a different directory with cd, MATLAB will not cd back to the original directory.

[matquest]

Is there anything besides the path variable which could explain this behavior? Or has a similar "global" attribute? I just tried resetting the path variable manually between runs (as theoretically this should force it back into the "initial" state) and I am still seeing this strange alternating effect.

[Sean de Wolski]

I'm almost certainly willing to bet it's a path issue. I'm also almost certainly willing to bet that using a project to manage the path will fix it.

[matquest]

@Sean de Wolski - Projects appear to have been added in R2019a based on the release notes, so I'm not sure if that's an option for me.

Answer 3

[Sean de Wolski]

You should convert your code files and tests into a MATLAB Project which manages the path for you. Then as long as the project is open (and there's a ProjectFixture in the test framework to ensure this) then everything you need is on the path. Additionally, projects make it easy to run tests because you can run all tests in a project and the test framework makes sure everything is set up.