33 views (last 30 days)

Show older comments

I am experiencing sign differences between my computations for principal component analysis using the pca function and the svd function. Consider this matrix:

X = magic(4)

X =

16 2 3 13

5 11 10 8

9 7 6 12

4 14 15 1

coeff = pca(X) %this will automatically save only the first few principal components

coeff =

0.5000 0.6708 0.4458

-0.5000 -0.2236 0.3573

-0.5000 0.2236 0.6229

0.5000 -0.6708 0.5344

Now when I do svd, I get the following results.

x0 = bsxfun(@minus,X,mean(X,1)); %first center the data

[~,~,v] = svd(x0)

v =

-0.5000 0.6708 -0.4458 -0.3182

0.5000 -0.2236 -0.3573 -0.7565

0.5000 0.2236 -0.6229 0.5586

-0.5000 -0.6708 -0.5344 0.1202

So I get essentially negatives of the first and third principal component. Does anyone know why this is?

Jonathan Chien
on 30 Aug 2020

Edited: Jonathan Chien
on 17 Feb 2021

What we're seeing here is a sign indeterminancy problem that is often intrinsic to methods such as singular value decomposition or eigendecomposition and that can in some cases impact downstream statistical analysis and interpretation. Briefly, the signs of a given pair of corresponding singular vectors are mathematically indeterminant, as both they and their reflections can satisfy the factorization A = UΣV*. The sign is thus essentially arbitrarily assigned during the implementation of algorithms that carry out SVD, with different SVD algorithms capable of producing different results from the same inputs. In some applications, this doesn't matter, but in other cases, it can pose nontrivial problems (for more on this, see this paper).

Within the pca.m function, MATLAB thus enforces a convention such that the largest component (by absolute magnitude) in each right singular vector/eigenvector will be positive (there are other possible conventions as well that also resolve this ambiguity). The following code accomplishes this in lines 429 to 437 of the pca.m function in version R2019b. However, the built-in svd function does not seem to make any such correction (consider this excerpt from the svd documentation: Different machines and releases of MATLAB can produce different singular vectors that are still numerically accurate. Corresponding columns in U and V can flip their signs, since this does not affect the value of the expression A = U*S*V'.), hence the difference you observed. At any rate, if you apply the following convention to the matrix V returned by svd, you should observe results identical to the coeff matrix returned by pca, inclusive of sign. Also note that since principal components (in keeping with the convention of both Hotelling's original 1933 paper and the nearly universal convention in the statistical literature, I use the term "principal components" to refer to the derived empirical orthogonal variables (whose elements are often called scores) defined wrt the eigenvectors, not to the eigenvectors themselves) can be calculated from the singular vectors, either as A*V or as U*S, you should make sure to apply the convention before calculating the principal components, or apply it directly to the principal components afterwards (as is done below).

% Enforce a sign convention on the coefficients -- the largest element in

% each column will have a positive sign.

[~,maxind] = max(abs(coeff), [], 1);

[d1, d2] = size(coeff);

colsign = sign(coeff(maxind + (0:d1:(d2-1)*d1)));

coeff = bsxfun(@times, coeff, colsign);

if nargout > 1

score = bsxfun(@times, score, colsign); % scores = score

end

Christine Tobler
on 1 Sep 2020

I'm not sure why this happens. As mentioned by others, it doesn't matter what sign these outputs have. You could try stepping through the code of pca.m in the debugger, to see what they do differently before / after the call to the SVD.

Note that the signs returned by the SVD can also change based on which machine MATLAB runs on / which MATLAB version is used (and some other machine configuration-related settings can influence this, e.g. number of threads).

John D'Errico
on 26 Aug 2016

SO? Eigenvectors are not unique down to a sign difference. WTP?

John D'Errico
on 27 Aug 2016

Probably correct? I know I am correct in this respect. (You are not even close to the first person to trip over this issue. Not even the 1000'th person.)

There may still be many subtle differences in codes, even if both of them use svd in the core. If the matrix may be subtly different, even down in the least significant bits, then you may get different results in those signs.

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!