How to fit a general curve on different datasets ?

Hi,

I have different datasets that I would like to fit together in order to create a general equation.

Yet I have fitted each dataset independently as : y_i=a_i + b_i * log10(x)

%% Example of one dataset
x= [1 10 100]';
y= [165 145 124]';
fo = fitoptions('Method','NonlinearLeastSquares',...
               'Lower',[1],...
               'Upper',[200],...
               'StartPoint',[180 180]);
ft = fittype('a + b*log10(x)','dependent',{'y'},'independent',{'x'},'coefficients',{'a','b'},'options',fo);
myfit=fit(x(:),y(:),ft);
plot(x,y, 'ko', 'LineWidth',1.5, 'MarkerSize',7)
hold on
plot(myfit, 'k--')

But now I would like to test this thing : y_i = a_i + b * log10(x) with a general b for all the dataset. So I need to fit this new model into my whole dataset to get an estimation of b that match with every dataset.

Does anyone as an idea on how to do this in matlab ?

Thank you very much for your help.

Answer 1

[Jeff Miller]

You need to set up separate independent variables for each dataset, something like this:

%% Example of two datasets (3 scores each)
i1= [1 1 1 0 0 0]; % 1 indicates member of 1st data set
i2= [0 0 0 1 1 1]; % 1 indicates member of 2nd data set
x= [1 10 100 1 10 100]';  % concatenated x's for both data sets
y= [165 145 124 155 135 114]'; % concatenated y's for both data sets

% Then fit a model like this
'a1*i1 + a2*i2 + b*log10(x)'

But I don't think the 'fit' function will do that--at least not for more than 2 datasets. You could do it with regress, at least if you make a new independent variable:

log10x = log10(x);
% Then fit 
'a1*i1 + a2*i2 + b*log10x'

MATLAB probably has some other nonlinear fitting routines that would also work, using the same basic indicator variable trick.

[yj]

Yes the fit functions didn't want to do such model...

[Jeff Miller]

Right, but regress will do it.

i1= [1 1 1 0 0 0]'; % 1 indicates member of 1st data set
i2= [0 0 0 1 1 1]'; % 1 indicates member of 2nd data set
x= [1 10 100 1 10 100]';  % concatenated x's for both data sets
y= [165 145 124 155 135 114]'; % concatenated y's for both data sets
log10x = log10(x);
regress(y,[i1 i2 log10x])

The first two outputs are a1 and a2, and the last output is the common b.

[yj]

Thank you for your answer, it actually worked for all my dataset, I have all my ai and one common b. I appologize if I ask another stupid question, but here my common b is the same value as the mean between the different bi that I previously calculated (separately) for each dataset. Is it right ?

[John D'Errico]

I don't think it a stupid question, but indicates that you are at least thinking about the results. A good start.

The overall value for b is not exactly the average of all the individual slopes in general. HOWEVER, the average of those slopes will often be a good approximation to the overall slope.

In certain circumstances, if the data was exactly balanced, with the same x values for all distinct data sets, in that rare set of circumstances, then the average will probably be the same, but only in that circumstance.

[Jeff Miller]

A small simulation shows that John is exactly right:

% Simulation with equal x's in all groups, but random y's

i1= [1 1 1 0 0 0]'; % 1 indicates member of 1st data set
i2= [0 0 0 1 1 1]'; % 1 indicates member of 2nd data set
x = [1 10 100 1 10 100]';  % concatenated x's for both data sets
log10x = log10(x);

NTries = 10;
for i=1:NTries
    y1 = randn(3,1);
    a1 = regress(y1,[i1(1:3) log10x(1:3)]);
    y2 = randn(3,1);
    a2 = regress(y2,[i2(4:6) log10x(4:6)]);
    y = [y1; y2];
    a = regress(y,[i1 i2 log10x]);
    dif = abs(a(3) - (a1(2) + a2(2))/2)
end
% The resulting difs are all miniscule, so in this case the slope of the
% combined data set matches the average of the individual slopes.

% Simulation with different x's in all groups, but random y's
for i=1:NTries
    x = randn(6,1).^2;
    log10x = log10(x);
    y1 = randn(3,1);
    a1 = regress(y1,[i1(1:3) log10x(1:3)]);
    y2 = randn(3,1);
    a2 = regress(y2,[i2(4:6) log10x(4:6)]);
    y = [y1; y2];
    a = regress(y,[i1 i2 log10x]);
    dif2 = abs(a(3) - (a1(2) + a2(2))/2)
end
% The resulting dif2s are much larger, so in this case the slope of the
% combined data set need not match the average of the individual slopes.

[yj]

Yes thank you very much for your answers !!

Answer 2

[Jon]

If I am understanding what you are trying to do correctly, it seems like you should be able to concatenate all of your individual x values into one overall x vector, and similarly concatenate all of your y values into one overall y vector and then perform the curve fit on the combined set.

[Jon]

Did this answer your question?

[yj]

I did concatenate ma datasets, but then I had a unique dataset and thus unique coefficients a and b for the related fitted curve. What I wanted to do is to fit each dataset independantly and get a global fit in order to have a_i that depends on each dataset, and a global b that applies for every dataset. Would it be a loop function or something like this ?

[Image Analyst]

Yes, if you want each set of fitting parameters separately you could put the fitting into a loop and store the fit parameters in an array indexed by the loop iterator variable.

[Jon]

I did not initially understand the problem you are trying to solve. Now I get that you want to fit an equation that possibly has different values of "a" for each individual data set but a common value of b. Conceptually, this suggests to me that there should be an additional, numerical parameter value (e.g. voltage, pressure, temperature, valve position ...), which characterizes each of the individual tests. Let's call this parameter, z, so we have have a vector z, the same length as x and y (the overall concatenated vectors) with the values of the test parameter corresponding to each experiment in the overall concatenated data set (so it repeats for the points that fall within one test). The simplest possibility would be that your test dependent parameter "a" depends linearly on z, so we would have a = c*(z - z0) where z0 is a nominal value for z. Putting it all together you want to fit y = c*(z - z0) + b*log10(x), or expanding y = cz + b*log10(x) + d, where d = -c*z0. You can solve this as a linear regression to solve for c,b, and d by putting it into the form:

[y(1);y(2);..y(n)] = [z(1) log10(x(1) 1;z(2) log10(x(2)) 1; ... z(n) log10(x(n)) 1]*[c;b;d]

which has the form y = A*p, where p are the unknown parameters c,b, and d you can solve this using

p = A\y;

If you needed the values of "a" you could then compute them afterwards using a = c*(z-c0)

Note that in the above x,y,and z are the concatenated values for all of the tests. In particular z will have repeated values, as it will be the same for each individual test.

[Jon]

Did this answer your question?

[yj]

Sorry for the delay of my response. I am still trying to do what you suggest, but without results yet !

[Jon]

Ok good luck

[John D'Errico]

Actually, this is the incorrect answer, since it will also use a common value for a.

[Jon]

I don't think that the approach I outlined will result in a common value of a, but I'm glad that you found a solution. (The approach I outlined may be buried under the older comments, you may not have seen it.)