Best fitting curve for highly variable data

I am given a matrix of 55 vectors with 7 different values each. Each vector must be analized value by value to find the next possible number.

e.g.

55 sets of x1 + x2 + x3 + x4 + x5 + x6 + x7 with different values of x1, x2,...,x7 in each set.

All values of x1 are arranged as a single vector and the interpolated for the 56th value. Given the size of the vector, a polynomial is unreasonable, and spline function does not seem to fit reasonable data (red x in image).

 load('file.txt') %load file
 valotto = file'; %transpose matrix to analyze by vector
 %open('valotto')
 cice = size (valotto); %read size of matrix
 quant = cice(1); %number of vectos (width of matrix)
 len = cice(2); % find number of values per vector (length of vectors)
 I = linspace(1,len,len); %Assign values of 1 to a matrix for x vector

 xp = I; %x values for interpolating
 nextguess = len+1; %next interpolating

 for i = 1:quant
    yp = valotto(i,:); %y values for intepolating
    if i==1
        fprintf('SPLINE\n')
        spline(xp,yp);
    end

 value = spline(xp,yp,nextguess);
 fprintf('value %1.d is %3.f \n', i, value)

 figure(i)
 plot(xp,yp,'ok',xp,yp,'--g',nextguess,value,'rx')

 end

[Kirby Fears]

The spline function is useful for interpolating new points that fall inside your training set, but not reliable for extrapolating new observations outside of your training set. E.g. if you wanted to interpolate a Y value for an X value of 5.5, the red X would fall along the green line at a value around 39 or so.

To extrapolate forward, you need a predictive model. Linear regression is a simple starting point. Check out the link below for a discussion of what Matlab can do.

https://www.mathworks.com/discovery/predictive-modeling.html

Answer 1

[John D'Errico]

You call it highly variable data, but I'd just call it very low signal to noise. In fact, virtually no signal at all.

INTERPOLATING spline fits are a terrible choice here. Worse, interpolating splines are an obscenely bad way to extrapolate. Expect complete crap from such a prediction. No surprise, that is what you got.

Your data is simply not sufficient to estimate any kind of high order model, spline or polynomial, or anything else. You might be able to find sufficient signal in that data to estimate a straight line, and only barely distinguish it statistically from a constant function. So that is where I would start.

Use a tool like polyfit, (or regress if you have the stats toolbox) to estimate a model. Since you have provided no data, only a picture I cannot help much beyond this. You will fit a linear polynomial, then test to see if the linear coefficient confidence intervals include zero. If they do, then at best, you have a constant model of the process. In either case, trivial to extrapolate.

[Alexander Holman]

The values of x are increasing by 1 with the values of y oscillating as shown, these being real natural numbers.

When I tried polyfit, MATLAB kept saying the system was badly conditioned most likely due to the size of the vector. Any other suggestions? Thanks.