How does MATLAB deal with wanted linear dependencies?

Dear Community,

I am using the function fitlm for solving a regression problem. I am using the 'modelspec'-option 'quadratic' which includes all products of pairs of distinct predictors like:

My Question is about the last term.

I guess if I want to use these Interaction MATLAB adds a extra column to X, but this makes X linear dependend.

So how does MATLAB deal with these Interactions, which I want in my model but which make my Designmatrix linear dependend?

I mean it gets rid of unwanted linear dependencies automatically, but I am asking about a wanted Interaction .

Thank you in advance!

Chris

Answer 1

[Bjorn Gustavsson]

That's not the problem. What you should look at is how "wellconditioned" your model-matrix X is. Look at this example:

[x,y] = meshgrid(-3.7:0.31:2.7,-4.1:0.29:2);
M = [ones(size(x(:))),x(:),y(:),x(:).^2,y(:).^2,x(:).*y(:)];
cond(M)
% 
% ans =
%
%   13.9913

So that model matrix M is not that poorly conditioned. What you really should look at is the singular values of M:

[U,S,V] = svd(M);
diag(S)
%
%ans =
%
%  169.9684
%   98.1878
%   82.1093
%   28.9911
%   23.1045
%   12.1482

However if we shift the x and y-points problems arise:

x = x + 321;
y = y + 567;
M = [ones(size(x(:))),x(:),y(:),x(:).^2,y(:).^2,x(:).*y(:)];
cond(M)
% ans =
%
%   4.6934e+10
[U,S,V] = svd(M);
log10(diag(S))
%
%ans =
%
%    6.9145
%    4.5290
%    1.8519
%    1.5992
%   -1.0903
%   -3.7570

So here you see that the matrix M becomes rather poorly conditioned and the smallest components of the eigenvalue-spectra is now smaller and much smaller than 1 (this means trouble, in the general/typical this leads to noise amplification)

HTH

[Christian]

Hi and thank you a lot for your detailled answer!

I am afraid i cannot understand it completely. From your answer i understand, that X should be well conditioned, so that small changes in X don´t cause big changes in S. I read before that MATLAB uses SVD to solve the problem somehow, but i don´t know how...

I don´t get the connection between your answer and my question, but I am sure thats just because i am not skilled enough.

If the condition number is too high, the model matrix X is close to deficient rank, right? But how do linear dependencies relate to this? I guess they cause a high condition number (Inf) and cause the problem, that X^T*X cannot be inverted (least squares method).

But the "fitlm"-function makes it possible to include these interactions . I need them aswell in my equation, but i have no idea how MATLAB solves it. People will ask me, but i am afraid i have no clue :S .

Would you please explain it again to someone like me, who isn´t as wellconditioned as someone who´s doing regression should be? ;)

Thank you in advance! This helps a lot!

Chris

[Bjorn Gustavsson]

As I showed in my little example, the matrix M (or X in your notation) does not become that poorly conditioned when you add the x(:).*y(:) column to it - that's not a linear combination of x and y. If you try this scary example instead:

M = [ones(size(x(:))),x(:),y(:),x(:).^2,y(:).^2,x(:).*y(:)];
M2 = [ones(size(x(:))),x(:),y(:),x(:).^2,y(:).^2,x(:)+y(:)];  
cond(M)
%
% ans =
%
%    13.9913
%
%
cond(M2)
%
%ans =
%
%   2.3730e+16

There you get a poorly conditioned problem.

Remember that the main point of interest is whether your is ill-conditioned or not, check that, if it isn't you're not in trouble yet. If it is ill-conditioned then you have to start working. First thing is to centre and scale your x and y data:

x_p = (x-mean(x(:)))/std(x(:));
y_p = (y-mean(y(:)))/std(y(:));
M3 = [ones(size(x_p(:))),x_p(:),y_p(:),x_p(:).^2,y_p(:).^2,x_p(:).*y_p(:)];
cond(M3)
%
% ans =
%
%    4.0024

This centering is helpful even for the nice-case example. For this you obviously also have to keep track of the variable transformation since your fitted model will be fitted as a 2-D polynomial in x_p and y_p and not your original x and y.

If the centering doesn't help then you might fit for too many parameters, and you have to accept that your data doesn't allow you to estimate that much. You might turn to the idea that you could try to iteratively search for what parameters to fit for (maybe it is enough to fit for a subset of all 6). There are tools for that as well, but that might be for later.

[Christian]

Thank you a lot for your detailed answer. I got it now and I didn't get a problem with a I'll conditioned matrix since I used sigmoid-standardizaion already :)

Thank you again for your help! Best regards

Chris