I'm not sure why it may of slowed down, but I can say that for me it was much faster to use readtable with opts. In my case I was only reading a single cell located in the same place in many sheets.
opts = spreadsheetImportOptions('Sheet',1,'DataRange','B3:B3');%I put opts outside of my for loop
a = readtable(filetoread(i),opts,'useexcel',false);
[~,~,tempCellArray] = xlsread(filetoread(i),1,'b3:b3','basic');
I did the above operations on 131 Files. readtable took 124s and xlsread took 716s