scatter data interpolation from text files in matlab

I am going to use scatteredInterpolant for interpolation of missing data. My data consist on text files name from Output_00 to Output_23 in a folder. Each text file consist on three columns, first is latitude, second is longitude and third is temperature. -9999.0000 value in temperature column representing NaN or missing data. This value appear randomly (some times appear in start, middle or bottom in file.

I want to make a code which read each text file from the folder and using scatteredInterpolant, interpolate 3rd column of each text file and then save this interpolated text file with output_interpol_00.text. It means there will be 24 more text files which will be interpolated

I have prepared this code. But it is manually entry requirement which is time consuming.I want to use looping or other to energize my code and instead on manully it run for all 24 text files in folder

% Load the data
data = load('Output_00.txt');
% separate the data columns, just to make the code clear
Lat = data(:,1); % Column 1 is Latitude
Lon = data(:,2); % Column 2 is Longitude
Tmp = data(:,3); % Column 3 is Temperature
% Find the "good" data points
good_temp = find(Tmp > -9999.000);
% creating vector 
T = scatteredInterpolant(Lat(good_temp),Lon(good_temp),Tmp(good_temp),'linear');
% find the "bad" data points
bad_temp = find(Tmp == -9999.000);
% use the interpolation object to interpolate temperature values
interp_values = T(Lat(bad_temp),Lon(bad_temp));
% replace the bad values with the interpolated values
Tmp(bad_temp) = interp_values;
interpolant_output=[Lat';Lon';Tmp']';
fid = fopen('interpot_linear_00.txt','wt');
for ii = 1:size(interpolant_output,1)
    fprintf(fid,'%.3f\t',interpolant_output(ii,:));
    fprintf(fid,'\n');
end
 fclose(fid)

Please also check it whether it is correct or not??

I have attached one text file of my data set. Please check it

Thanks for this kind assistance

[Muhammad Usman Saleem]

Once with assistance of Stephon, i prepared this code which was running according to my all requirements expect than scatteredInterpolant. Please also check it may modify it if you can.

metC = {'linear','cubic','next','pchip','previous','spline','v5cubic','nearest'};
    % Read the file data:
S = dir('Output_*.txt');
N = sort({S.name});
nmf = numel(N);
nmr = size(load(N{1},'-ascii'),1);
mat = zeros(nmr,3,nmf);
for k = 1:nmf
    mat(:,:,k) = load(N{k},'-ascii');
end
tmp = 0==diff(mat(:,1:2,:),1,3);
assert(all(tmp(:)),'First columns do not match')
% Rearrange:
[VC,NA,IC] = unique(mat(:,1,1));
[VR,NA,IR] = unique(mat(:,2,1));
out = reshape(mat(:,3,:),numel(VR),numel(VC),nmf);
% Detect -9999:
idx = out<(-9998);
% Interpolate:
vec = 1:nmf;% 24 times
for m = 1:numel(metC)% the length of techniques
    metS = metC{m} % the method
for r = 1:numel(VR)% 19 times
    for c = 1:numel(VC)% 17 times
        idy = squeeze(idx(r,c,:)).'; % removing dimension 1
        if any(idy)
            Xold = vec(~idy);
            Yold = squeeze(out(r,c,~idy)).';
            Xnew = vec(idy);
            out(r,c,idy) = interp1(Xold,Yold,Xnew,metS,'extrap');
        end
    end
end
% Write to file:
pressures = [1000, 925, 850, 700, 600, 500, 400, 300, 250, 200, 150, 100, 70, 50, 30, 20, 15, 10, 7, 5, 3, 2, 1.5, 1] ;
for k = 1:nmf
    fprintf('Export Interpolation. %d -> p=%.1f hpa.\n', k-1, pressures(k) );
  str = sprintf('Interpolated_%s_%d.txt', metS, k-1);
    fid = fopen(str,'wt');
    fprintf(fid,' \t \t \n');
    tmp = out(:,:,k);
    fprintf(fid,'%.3f\t%.3f\t%.3f\n',[mat(:,1:2),tmp(:)].');
    fclose(fid);
end
end

[Walter Roberson]

What assistance are you looking for at this time? The major question you seemed to have was about processing the files automatically, and the code you posted here does that.

[Muhammad Usman Saleem]

Dear Walter;

there are two codes i have shared in this question

1. In first code, i have tried my effort in order to processes on the single text file. It is very time consuming to write each line for each text file(24 in no) also when i want to implement every new technique i have to again write this whole code? It will be more keen work and time consuming I want to make time saving code, which read all text file and interpolate their 3rd column -9999.000 value as mentioned in my code. Then save each text file with this format inter_method_00.text. while method stand for method and 00 stand for output_oo.text file
2. Look the same idea i above mentioned is done by @Stephon, but this code runing perfrectly for Interpolate1 not for scatteredInterpolant there is just little modification in my second code for my requirements. Please read my question and give an assistance for my requirements.

Thanks in advance for this kind assistance

Answer 1

[Walter Roberson]

S = dir('Output_*.txt');
N = sort({S.name});

for K = 1 : length(N)

    infile = N{K};
    whichfile = sscanf(infile, 'Output_%c%c');
    outfile = ['interpot_linear_' whichfile '.txt'];

   % Load the data

    data = load(infile);

    % separate the data columns, just to make the code clear
    Lat = data(:,1); % Column 1 is Latitude
    Lon = data(:,2); % Column 2 is Longitude
    Tmp = data(:,3); % Column 3 is Temperature
    % Find the "good" data points
    good_temp = find(Tmp > -9999.000);
    % creating vector 
    T = scatteredInterpolant(Lat(good_temp),Lon(good_temp),Tmp(good_temp),'linear');
    % find the "bad" data points
    bad_temp = find(Tmp == -9999.000);
    % use the interpolation object to interpolate temperature values
    interp_values = T(Lat(bad_temp),Lon(bad_temp));
    % replace the bad values with the interpolated values
    Tmp(bad_temp) = interp_values;
    interpolant_output=[Lat';Lon';Tmp']';
    fid = fopen(outfile,'wt');
    for ii = 1:size(interpolant_output,1)
        fprintf(fid,'%.3f\t',interpolant_output(ii,:));
        fprintf(fid,'\n');
    end
    fclose(fid)

end

[Muhammad Usman Saleem]

I got that error

Attempt to reference field of non-structure array.

I have also attached more text files of my data set

[Walter Roberson]

I corrected the code. Please remember to include complete error messages: that makes it much faster to isolate problems.

[Muhammad Usman Saleem]

Sorry for my inconvenient.Next time, i shall care about posting error.

1. I am getting

ans =

     0

ans =

     0

ans =

     0

ans =

     0

ans =

     0

ans =

     0

ans =

     0

ans =

     0

ans =

     0

ans =

     0

ans =

     0

ans =

     0

ans =

     0

ans =

     0

ans =

     0

ans =

     0

ans =

     0

ans =

     0

ans =

     0

ans =

     0

ans =

     0

ans =

     0

ans =

     0

ans =

     0

In command window. Please check where it is coming?

2. I want to apply, 'linear','nearest','natural' separately.Please modify this code in such a way that, it run for each technique and save each text file with method name( as your kind code is doing). After running that code, i will have 96 text files in my folder

3. Please check for any kind of bug, if remains. Because my all data sets will interpolate through this code.

4. and in last always thanks for this kind assistance

[Walter Roberson]

Change

fclose(fid)

to

fclose(fid);

Why would you end up with 96 files? You would have:

• 12 input files
• 12 outputs for linear
• 12 outputs for natural
• 12 outputs for nearest

Total: 48 files.

Where would the other 48 come from?

[Muhammad Usman Saleem]

Please check it my text files (input files) are 24 with name output_00.text to output_23.text

then

24 input files

24 output from linear

24 outputs form natural

24 outputs form nearest

total files= 96

Please modify this code to my both requiremnts. I shall be very thankful to you on this kind favor

[Walter Roberson]

methods = {'linear', 'natural', 'nearest'};
for midx = 1 :length(methods) 
method = methods{midx};
... 
end

And modify your code to use the variable name method in place of the hard-coded reference to linear. The technique is the same as for the variable I named whichfile, using [] to put strings together

[Muhammad Usman Saleem]

in this manner??

S = dir('Output_*.txt');
N = sort({S.name});
for K = 1 : length(N)
methods = {'linear', 'natural', 'nearest'};
for midx = 1 :length(methods) 
method = methods{midx};
    infile = N{K};
    whichfile = sscanf(infile, 'Output_%c%c');
    outfile = ['interpot_linear_' whichfile '.txt'];

     % Load the data

      data = load(infile);

      % separate the data columns, just to make the code clear
      Lat = data(:,1); % Column 1 is Latitude
      Lon = data(:,2); % Column 2 is Longitude
      Tmp = data(:,3); % Column 3 is Temperature
      % Find the "good" data points
      good_temp = find(Tmp > -9999.000);
      % creating vector 
      T = scatteredInterpolant(Lat(good_temp),Lon(good_temp),Tmp(good_temp),method);
      % find the "bad" data points
      bad_temp = find(Tmp == -9999.000);
      % use the interpolation object to interpolate temperature values
      interp_values = T(Lat(bad_temp),Lon(bad_temp));
      % replace the bad values with the interpolated values
      Tmp(bad_temp) = interp_values;
      interpolant_output=[Lat';Lon';Tmp']';
      fid = fopen(outfile,'wt');
      for ii = 1:size(interpolant_output,1)
          fprintf(fid,'%.3f\t',interpolant_output(ii,:));
          fprintf(fid,'\n');
      end
      fclose(fid);

end
end

i do not sure whether i am correct or not? totally confuse from last two lines of your kind comment?

[Walter Roberson]

methods = {'linear', 'natural', 'nearest'};

S = dir('Output_*.txt');
N = sort({S.name});

for K = 1 : length(N)
  infile = N{K};
  whichfile = sscanf(infile, 'Output_%c%c');

    % Load the data
    data = load(infile);

    % separate the data columns, just to make the code clear
    Lat = data(:,1); % Column 1 is Latitude
    Lon = data(:,2); % Column 2 is Longitude
    Tmp = data(:,3); % Column 3 is Temperature

    % Find the "good" data points
    good_temp = find(Tmp > -9999.000);

    % find the "bad" data points
    bad_temp = find(Tmp == -9999.000);

    for midx = 1 :length(methods) 
      method = methods{midx};
      outfile = ['interpot_' method '_' whichfile '.txt'];

      % creating vector 
      T = scatteredInterpolant(Lat(good_temp), Lon(good_temp), Tmp(good_temp), method);

      % use the interpolation object to interpolate temperature values
      interp_values = T(Lat(bad_temp), Lon(bad_temp));
      % replace the bad values with the interpolated values
      Tmp(bad_temp) = interp_values;

      interpolant_output = [Lat';Lon';Tmp']';

      fid = fopen(outfile,'wt');
      for ii = 1:size(interpolant_output,1)
        fprintf(fid,'%.3f\t',interpolant_output(ii,:));
        fprintf(fid,'\n');
      end
      fclose(fid);

    end    %methods

end      %files

[Muhammad Usman Saleem]

i try to extrapolate with the same technique as T doing. That why i replace

T = scatteredInterpolant(Lat(good_temp), Lon(good_temp), Tmp(good_temp), method,'extrap');

but getting that error

>> scater_interpolatoin
Error using scatteredInterpolant
Invalid extrapolation type specified.

Error in scater_interpolatoin (line 30)
      T = scatteredInterpolant(Lat(good_temp), Lon(good_temp), Tmp(good_temp),
      method,'extrap');

[Muhammad Usman Saleem]

thanks walter. Is this possible this code can be change to grideddata interpolation. But its working remains same?

[Walter Roberson]

No, in order to use griddata() you need the input to be a grid, which requires that all samples be present (even if they are nan.) That would not fit in with your coding requirement that only the good data be passed to the interpolant. It would therefore not be possible for the working to remain the same.

[Muhammad Usman Saleem]

i got from your above comment that, griddata interpolation will not suited for me as my data set is not grided. I am confuse with this word as my latitude and longitude vectors are grided and continous. Only NaN value in temperature column appear as randomly. Will still this technique not work?

[Walter Roberson]

You are only submitting goodsamp to the interpolant so you are interpolating over an incomplete grid. If you were to griddata with the whole grid you would be passing in the bad samples as well and then when you tried to interpolate at those points you would be given back the bad values.

You never actually said why you are doing all of this, why you think it would be good to try the three different method values. I speculate that your overall task would be handled well by using John D'Errico's file exchange contribution inpaint_nans

[Muhammad Usman Saleem]

@walter thanks for your kind assistance. Is this possible i apply IDW and kringing interpolation over my data set? I know matlab has not such build in tools we have to make new codes?

[Walter Roberson]

I am not familiar with those. Looking around I see that there are a few different kriging toolboxes available for MATLAB.

[Muhammad Usman Saleem]

And there are also some IDW tools are available for Matlab. Matlab must lunch this functions for these interpolations.

[Walter Roberson]

I am not sure what you are trying to say there.

scatteredInterpolant does not invoke those kriging or IDW tools, and is does not happen to be customizable to invoke them, so you would need to alter your code to invoke the external tools.

[Muhammad Usman Saleem]

thanks walter. Always remains happy