Why eval seems to be faster than other alternatives in this example?!

Hi

I know that using eval is highly discouraged, but for the below test case it seems to be faster than other options. Can you help me explain this and let me know which alternative you prefer?

struct1.data = 2*ones(1e3);
struct2.data = 4*ones(1e3);
struct(1).data = 2*ones(1e3);
struct(2).data = 4*ones(1e3);
selected_struct_str = 'struct1';
selected_struct_num = 1;

num_iter = 10;

tic
for i = 1:num_iter
    test1 = doWork1(struct1, struct2, selected_struct_str);
end
toc

tic
for i = 1:num_iter
    test2 = doWork2(struct1, struct2, selected_struct_num);
end
toc

tic
for i = 1:num_iter
    test3 = doWork3(struct, selected_struct_num);
end
toc

function data = doWork1(struct1, struct2, selected_struct)
    data = eval([selected_struct '.data']);
end

function data = doWork2(struct1, struct2, selected_struct_num)
    switch selected_struct_num
        case 1
            data = struct1.data;
        case 2
            data = struct2.data;
    end
end

function data = doWork3(struct, selected_struct_num)
    data = struct(selected_struct_num).data;
end

I am getting the following output:

>> eval_test
Elapsed time is 0.000418 seconds.
Elapsed time is 0.001516 seconds.
Elapsed time is 0.001594 seconds.
>> eval_test
Elapsed time is 0.000384 seconds.
Elapsed time is 0.001027 seconds.
Elapsed time is 0.001186 seconds.
>> eval_test
Elapsed time is 0.000558 seconds.
Elapsed time is 0.001280 seconds.
Elapsed time is 0.001304 seconds.
>> eval_test
Elapsed time is 0.000641 seconds.
Elapsed time is 0.001754 seconds.
Elapsed time is 0.001802 seconds.
>> eval_test
Elapsed time is 0.000357 seconds.
Elapsed time is 0.001117 seconds.
Elapsed time is 0.001430 seconds.
>> eval_test
Elapsed time is 0.000533 seconds.
Elapsed time is 0.001231 seconds.
Elapsed time is 0.001234 seconds.

[Walter Roberson]

Possibly the JIT is optimizing away calls???

[Bruno Luong]

I put your script in function to make sure JIT optimization works correctly (in script JIT might not work), and increase the num_iter to 1000 for better average. Here is my result (R2019b, Windows PC).

>> test
Elapsed time is 0.015345 seconds. % eval
Elapsed time is 0.000555 seconds. % struct switch/case
Elapsed time is 0.000928 seconds. % switch array

So EVAL is 28 times slower than struct switch/case and 16 times slower than switch array.

function test

struct1.data = 2*ones(1e3);
struct2.data = 4*ones(1e3);
struct(1).data = 2*ones(1e3);
struct(2).data = 4*ones(1e3);
selected_struct_str = 'struct1';
selected_struct_num = 1;
num_iter = 1000;
tic
for i = 1:num_iter
    test1 = doWork1(struct1, struct2, selected_struct_str);
end
toc
tic
for i = 1:num_iter
    test2 = doWork2(struct1, struct2, selected_struct_num);
end
toc
tic
for i = 1:num_iter
    test3 = doWork3(struct, selected_struct_num);
end
toc

end

function data = doWork1(struct1, struct2, selected_struct)
data = eval([selected_struct '.data']);
end

function data = doWork2(struct1, struct2, selected_struct_num)
switch selected_struct_num
    case 1
        data = struct1.data;
    case 2
        data = struct2.data;
end
end

function data = doWork3(struct, selected_struct_num)
data = struct(selected_struct_num).data;
end

[Stephen Cobeldick]

"...for the below test case it seems to be faster than other options."

Those tests are probably too short to be reliable. The MATLAB documentation states that "Sometimes programs run too fast for tic and toc to provide useful data. If your code is faster than 1/10 second, consider measuring it running in a loop, and then average to find the time for a single run." This advice is repeated on these pages:

https://www.mathworks.com/help/matlab/matlab_prog/measure-performance-of-your-program.html#bt1lr5a

https://www.mathworks.com/help/matlab/ref/tic.html?s_tid=srchtitle#bswc7ww-3

Answer 1

[per isakson]

Testing is tricky

>> Untitled5
Elapsed time is 0.167926 seconds.
Elapsed time is 0.007457 seconds.
Elapsed time is 0.010043 seconds.

where Untitled5 is

%%
struct1.data = 2*ones(2);
struct2.data = 4*ones(2);
struct(1).data = 2*ones(2);
struct(2).data = 4*ones(2);
selected_struct_str = 'struct1';
selected_struct_num = 1;
num_iter = 1e4;     %    <<<<<<<<<<<<<<<<<<
tic
for i = 1:num_iter
    test1 = doWork1(struct1, struct2, selected_struct_str);
end
toc
tic
for i = 1:num_iter
    test2 = doWork2(struct1, struct2, selected_struct_num);
end
toc
tic
for i = 1:num_iter
    test3 = doWork3(struct, selected_struct_num);
end
toc
function data = doWork1(struct1, struct2, selected_struct)
    data = eval([selected_struct '.data']);
    data = rand()*data;            %    <<<<<<<<<<<<<<<<<<
end
function data = doWork2(struct1, struct2, selected_struct_num)
    switch selected_struct_num
        case 1
            data = struct1.data;
            data = rand()*data;    %    <<<<<<<<<<<<<<<<<<
        case 2
            data = struct2.data;
            data = rand()*data;    %    <<<<<<<<<<<<<<<<<<
    end
end
function data = doWork3(struct, selected_struct_num)
    data = struct(selected_struct_num).data;
    data = rand()*data;            %    <<<<<<<<<<<<<<<<<<
end

ADDENDUM

I converted the script to a function and left doWork-functions as subfunction. The elapse times didn't change much

>> Untitled5
Elapsed time is 0.171473 seconds.
Elapsed time is 0.005879 seconds.
Elapsed time is 0.010672 seconds.
>> Untitled5
Elapsed time is 0.172317 seconds.
Elapsed time is 0.006429 seconds.
Elapsed time is 0.010051 seconds.
>> 

/R2018b

[Hesham]

Thanks for your answer.

You increased num_iter signifcantly. I got the same effect on my machine.

Can we conclude that eval is faster for small num_iter, but it is slower for larger num_iter?

The question "Why?" is not answered yet.

[Bjorn Gustavsson]

Have you made sure that your test-functions are already JIT-prepared first time you call your test-script? If so that overhead-time would be a significantly larger effect for 10 iterations than for 1e4 iteration. Also for a more accurate test, you should chose selected_struct randomly since "a very clever JIT" would figure out what is done in the loops doesn't change, and you want to test for the run-time of doWork-functions.

[per isakson]

I increased num_iter to improve the precision of the timing.

The real difference in our tests is the line

data = rand()*data;

which forces Matlab to do some work, not only copy data, which the JIT (just in time compiler) is smart enough to avoid. I decreased the size of data to avoid letting the multiplication dominate. Mathworks have given the JIT a fancy name, which I forgotten.

Mathworks keeps the inner working of the JIT secret, hence no answer to "Why?". However, see Documentation on JIT compiler?

[Bruno Luong]

"Can we conclude that eval is faster for small num_iter, but it is slower for larger num_iter?"

No we rather conclude that the test on small num_iter is not reliable.

[Guillaume]

In any case, in a real world case the eval test case would involve constructing the structure name with something like sprintf('test%d', idx) which will absolutely kill performance compared to the much simpler test(idx).