Warning: Equation is badly conditioned. Remove repeated data points or try centering and scaling.
16 views (last 30 days)
Show older comments
I have a data set, FR, atttached. I am trying to plot the confidence levels for the same but I am encountering a warning message :"Warning: Equation is badly conditioned. Remove repeated data points or try centering and scaling.". I dont know what is the issue as well as how to solve it. Any kind of help will be very useful. I am getting a graph like this (as shown below) but I think this is not correct. Please help me.
%%% this is the code I am using
%Create fit and confidence interval
FitVec = fit(freq,alpha,'poly9')
pRvecConf = predint(FitVec,freq,0.95,'observation','off');
%get the fitting values
fitY=feval(FitVec,freq);
%multiply the confidence interval with sqrt(xVec(i)).^2 .*xVec'
ci=(fitY-pRvecConf(:,1));
%get the weighted confidence interval
Conf_new=[fitY-ci,fitY+ci];
%Plot
figure
plot(FitVec,freq,alpha)
hold on
plot(freq,Conf_new,'m--')
legend('Data','Fitted curve','Confidence','Location','se')
xlabel('Length')
ylabel('Strength')
7 Comments
Accepted Answer
Bruno Luong
on 5 Nov 2024
Edited: Bruno Luong
on 6 Nov 2024
I don't use MATLAB native FIT, I can't get it works correctly to fit your data, even with smoothingspline model, and predictive interval predint with spline model as input is not suported.
I rather use my file exchange BSFK here https://www.mathworks.com/matlabcentral/fileexchange/25872-free-knot-spline-approximation
data = readmatrix('https://www.mathworks.com/matlabcentral/answers/uploaded_files/1803220/FR.txt');
freq = data(:,1);
alpha = data(:,2);
options = struct('knotremoval_factor', 1, 'lambda', 2e-7);
pp = BSFK(freq, alpha, [], [], [], options);
fitY=ppval(pp,freq);
dY = (alpha-fitY);
nu = 33;
sigma = movstd(dY,nu); % sqrt(sum(dY.^2) / (n-1));
level = 0.98;
normconf = erfinv(level)*sqrt(2); % tinv((1+level)/2,nu-1) is better choice?
ci = sigma * normconf;
fi=linspace(freq(1),freq(end),400);
%Plot
close all
figure
grid on
hold on
xconf = [freq; freq(end:-1:1)] ;
yconf = [fitY+ci; fitY(end:-1:1)-ci(end:-1:1)];
p = fill(xconf,yconf,'r');
p.FaceColor = [0.8 0.8 0.8];
p.EdgeColor = 'none';
h1 = plot(freq, alpha,'.k');
h2 = plot(fi,ppval(pp,fi),'r','LineWidth',2);
legend([h1 h2 p], 'Data','Fitted curve', ...
sprintf('%g%% Confidence', level*100),'Location','se')
xlabel('Length')
ylabel('Strength')
5 Comments
Bruno Luong
on 6 Nov 2024
Edited: Bruno Luong
on 6 Nov 2024
i gave a link to download the BSFK file exchange; please read my post again
More Answers (3)
John D'Errico
on 5 Nov 2024
Edited: John D'Errico
on 5 Nov 2024
Do NOT use polynomials to model data like this In fact, do not use high order polynomials, pretty much ever.
FR = load('FR.txt');
x = FR(:,1);
y = FR(:,2);
plot(x,y,'-')
UGH.
Next, polynomials abhor a singularity, and it looks like you may have a singularity at x==0, or at least something very close to a singularity. No finite order standard polynomial ever has a singularity in it. (Don't talk about rational polynomials, which are completely different animals, but also significantly more difficult to work with.)
Next, your curve has a deep, sharp drop in it. Polynomials will never have anything in them that does this. These are things that simply do not match the expected behavior of a polynomial model.
Yes, I know, polynomials are easy to use. Easy to fit. And so people tend to use them at will. And if a low order fit is not good enough, just use a higher order fit. Heck, we learned about Taylor/Maclaurin series in some long forgotten class. And they are just polynomials, right?
I'm sorrry, but this always works out poorly for the person wanting a fit for their data.
You might decide to use a regression or smoothing spline, or you might decide to use a more general smoothing tool, or time series methods. There are many things, that with some amount of effort, you might do here. But not a polynomial.
4 Comments
John D'Errico
on 5 Nov 2024
@Bruno Luong I think the issue is the need for the nice pretty confidence intervals. I doubt NURBS does that. Easy enough to build the code for a basic regression spline, then get confidence intervals.
Bruno Luong
on 5 Nov 2024
MATLAB fit with smoothingspline does not supportt confidence. Though it is linear regression.
I also play with smoothing parameters of FIT, I did not understand what does it do.
Image Analyst
on 5 Nov 2024
What if you just scanned your data with movmean to get the local mean, and movstd to get the local standard deviation, and then set confidence levels 2 or 3 SDs away from the mean? If the SD array is noisier than you want, you could smooth that. Would that work for you? I'm not sure why you think you need a polynomial model in the first place. You can get a good idea of the range of data jst by doing it numerically on your actual data.
% Demo by Image Analyst.
% Initialization steps.
clc; % Clear the command window.
fprintf('Beginning to run %s.m ...\n', mfilename);
close all; % Close all figures (except those of imtool.)
clearvars;
workspace; % Make sure the workspace panel is showing.
format long g;
format compact;
fontSize = 16;
lineWidth = 2;
markerSize = 30;
FR = load('FR.txt');
x = FR(:,1);
y = FR(:,2);
subplot(3, 2, 1);
plot(x,y,'-')
grid on;
localMeans = movmean(y, 5);
subplot(3, 2, 3);
plot(localMeans, 'b-')
title('Local means')
grid on;
localSD = movstd(y, 5);
subplot(3, 2, 5);
plot(x, localSD, 'b-')
title('Local SD')
grid on;
% Compute upper level
upperCL = localMeans + 3 * localSD;
lowerCL = localMeans - 3 * localSD;
subplot(3, 2, [2,4,6]);
plot(x, y, 'b-')
hold on;
plot(x, upperCL, 'r-')
plot(x, lowerCL, 'r-')
title('Data with Confidence Limits in red')
grid on;
3 Comments
Image Analyst
on 6 Nov 2024
Thanks @Ron. If an Answer solves your original question, then could you always click the Vote icon (thumbs up) to award the answerer with "reputation points" for their efforts in helping you? They'd appreciate it. Thanks in advance. 🙂 Note: you can only accept one answer (so pick the best one) but you can click the "Vote" icon for as many Answers as you want. Voting for an answer will also award reputation points.
For full details on how to earn reputation points see: https://www.mathworks.com/matlabcentral/answers/help?s_tid=al_priv#reputation
Matt J
on 5 Nov 2024
Edited: Matt J
on 5 Nov 2024
9th order polynomials do tend to be badly conditioned. Try using a lower order, or perhaps a spline model.
3 Comments
Bruno Luong
on 5 Nov 2024
@Ron Your confidence shall be meaning less if you can't correctly fit the data.
See Also
Categories
Find more on Get Started with Curve Fitting Toolbox in Help Center and File Exchange
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!