how to delete nan values from array columns without losing data from other columns

i have a huge file from a lab expremient

every run gives me 4 columns and every 2 out of the 4 start with nan then 2 nan vars and end with nan and 3 nan vars i want to delet the nan values without losing any data so that when i try to make graphs from it every graph will have enough inputs and neither of them get lost

i am espisally troubled by the second 2 columns of every 4 and how to deal with them

[Torsten]

NaN values have the advantage that they are not plotted. Thus if plotting something is your only concern, you should leave everything as is.

[dareen]

its just that it fails to plot becuase the number of nan values differ from one to another or posibbly i am not writing the code write

[Rik]

What code are you using to plot?

The problem is that Matlab does not allow holes in arrays, so you can't remove those NaN values without removing a row/column. You can get close to the effect with a cell array (which can contain an empty array), but most plotting functions can't handle cell inputs.

[dareen]

i think i have a much more fundemental problem i am not sure what to do to plot my goal , my goal is to get something as x(t)=a(t) sin(w*t+phi)

i have x(t) and (at) i have t and v(t) i know that i can get the rest using cftool but in regards to how to so do not really ,i tried to deal with the problem by starting to take values from 3 entery from row i to row j in a certain column , and so on for the 4 first columns and i remove nan values from the buttom and take the bottom -4 so that i would not have diffrent lengths for the function ,

emp=rmmissing(data2{:,1:4}); % Remove NaN from col 1,2,3 4

t=data2(3:end-4,1) x=data2(3:end-4,2) v=data2(3:end-4,3) a=data2(3:end-4,4)

but now stuck with the implementing

here when i tried to turn this into a function

[num_rows, num_columns] = size(data2);

function rebuildtable (data2,num_rows,num_columns)

emp=rmmissing(data2{:,1:4:end}); % Remove NaN from col 1,2,3 4

t=data2(3:end-4,1:4:end) x=data2(3:end-4,2:4:end) v=data2(3:end-4,3:4:end) a=data2(3:end-4,4:4:end)

end

but now stuck with the implementing

of the funciton above in order to get w and phi using cftool

this did not really work so i backed down and reached a new way

[dareen]

@Rik

here is my complete attempt

data=readtable('exp21.xlsx')

data2=readtable('expr2lab.csv')

data1= table2array(data)

data2=table2array(data2)

num_columns = size(data2, 2);

for q=17:4:num_columns

temp=rmmissing(data2(:,q:q+3));

t = temp(:,1);

x = temp(:,2);

v = temp(:,3);

a = temp(:,4);

[t x v a];

cftool(t,x)

model = @(a, w, phi, t) a .* sin(w .* t + phi);

end

[Mathieu NOE]

we could probably be more efficient if you would also share the data

also I am not sure how v(t) is used (or not ) in your equation x(t)=a(t) sin(w*t+phi) or what is it for ?

[dareen]

v is for this section where i have a force slowing the masses down ,

the graphes i made are for both sections without distingusing between them but mostly the section with this solutuion has 2 written next to it in the data without it has 1 or nothing

[Mathieu NOE]

ok, so you want to fit a exponential decaying sine wave to your data ... that was not very clear at the beginning

first I wanted to have a look at your data , at you have many (34) runs and I was not sure what was in your data

now , which runs are the best to use ? probably not the first ones

data= readmatrix('expr2lab.csv'); % or  readtable or whatever
runs = size(data,2)/4;

for ck = 1:runs
    ind_cols = (1:4)+(ck-1)*4; % Time (s) 	Position (m) 	Velocity (m/s) 	Acceleration (m/s²) 

    data_this_run = data(:,ind_cols);
    [m,n] = size(data_this_run);
    Time = data_this_run(:,1);
    Position = data_this_run(:,2);
    Velocity = data_this_run(:,3);
    Acceleration = data_this_run(:,4);

    figure(ck)
    subplot(3,1,1),plot(Time,Position)
    title(['Run # : ' num2str(ck)]);
    ylabel('Position');
    subplot(3,1,2),plot(Time,Velocity)
    ylabel('Velocity');
    subplot(3,1,3),plot(Time,Acceleration)
    ylabel('Acceleration');

end

[dareen]

@Mathieu NOE

thank you a lot cant express how thankful i am i searched for something like this for hours , as for which runs do i think are the best 12 , 16 ,17,20,22,24,29,30,31,34,33,11

i tried to clean the data up a bit that it will be easier for reading so it has one when i attached some extra friction called with sails and one before adding the extra friction called no sails , and if possible a question about the code from what i understood the first line changes the number of columns taken every time by 4, then we are dealing with for columns at a time [m,n] = size(data_this_run); this line i assume just for checking the size and was not used after, this not sure what it does figure(ck)

[Mathieu NOE]

glad I could help a bit , but this was just a starter

ok, let's focus on the selected runs (for both files I presume ?)

these 2 lines are indeed just to extract a block of 4 contiguous columns , and we simply shift by a factor 4 when we change to the next run :

ind_cols = (1:4)+(ck-1)*4;

data_this_run = data(:,ind_cols);

and yes this line is not anymore of any use (I used it for another purpose that I removed afterwards) : [m,n] = size(data_this_run);

I'll come back soon with a new code

[dareen]

i appreciate your time and help ,sorry that the question is lengthy and not written in the most clear form, yeah for both files ,i tried to seprate them since each should yield some intersting observations ,= i was not sure if i should also keep the accelration and velocity since i am not sure implementation wise what to do , if any of the data before is needed in the later sections then we can work on the original file and i will try to learn and implement it in the the tided up versions and add whats needed , as for what i am trying to reach i firstly had another file of data from excel that helped find the spring constant and i ploted angular velocity intialy as a function of mass and now the new angular velocity when the system is osillating and slowing down as a function of mass here what i did for now i tried to copperate cftool since i thougth maybe there i can the sin and cos values yet failed mesriably not sure how to get the data from normal plots either

Answer 1

[Mathieu NOE]

ok, so this is now the main dishes....

have first tried this code on the first data file , will adapt to your new files as soon as possible

have fun !

data= readmatrix('expr2lab.csv'); % or  readtable or whatever

runs = [12 , 16 ,17,20,22,24,29,30,31,34,33,11]; % selection of best runs

%% main loop

for ck = 1:numel(runs)
    k = runs(ck);
    ind_cols = (1:4)+(k-1)*4; % Time (s) 	Position (m) 	Velocity (m/s) 	Acceleration (m/s²) 
    data_this_run = data(:,ind_cols);
    [m,n] = size(data_this_run);
    Time = data_this_run(:,1);
    Position = data_this_run(:,2);
    Velocity = data_this_run(:,3);
    Acceleration = data_this_run(:,4);
    
    %% fit on acceleration data 
    % remove all NaN's first
    ind = isnan(Time) | isnan(Acceleration);
    Time(ind) = [];
    Position(ind) = [];
    Velocity(ind) = [];
    Acceleration(ind) = [];
    % select valid data between first max peak and end of data 
    [v,indm] = max(Acceleration); % search for the first positive max peak
    nn = numel(Time);
    [b,yf] = exp_decay_sinus_fit(Time(indm:nn),Acceleration(indm:nn));
    % b array contains 5 coefficients according to this equation (model)
    % y = b(1).*exp(b(2).*x).*(sin(2*pi*b(3)*x + b(4))) + b(5)
    eq_str = [ ' y =  ' num2str(b(1),'%.2f'),'* exp(-' num2str(-b(2),'%.2f') ' t * sin(2*pi*' num2str(b(3),'%.2f') ' t + ' num2str(b(4),'%.2f') ' ) + (' num2str(b(5),'%.2f') ')'];
    figure(ck)
    subplot(3,1,1),plot(Time,Position)
    title(['Run # : ' num2str(k)]);
    ylabel('Position');
    subplot(3,1,2),plot(Time,Velocity)
    ylabel('Velocity');
    subplot(3,1,3),plot(Time,Acceleration,Time(indm:nn),yf);
    ylabel('Acceleration');
    legend('data',eq_str);

end

%%%%%%%%%%%%%%% functions %%%%%%%%%%%%%%%%%%%%%%

function [B,yf] = exp_decay_sinus_fit(t,y)

    [yu,indm] = max(y);
    yl = min(y);
    yr = (yu-yl);                                                                   % Range of y
    yz = y-yu+(yr/2);
    
    % zero crossing are performed on first 1/3 of data (better signal to
    % noise ratio)
    n = round(numel(y)/3);
    yz = yz(indm:n); % extract from first major peak to end of 30% of data
    
    zt = t(yz(:) .* circshift(yz(:),[1 0]) <= 0);                                   % Find zero-crossings
    per = 2*mean(diff(zt));                                                         % Estimate period
    freq = 1/per;                                                                   % Estimate frequency
    
    % initial phase estimate 
    tmax = t(indm);
    phase_init = mod(-2*pi*freq*tmax + pi/2,2*pi);                                  % initial phase estimate 

    ym = mean(y);                                                                   % Estimate DC value (offset)
    fit = @(b,x)  b(1).*exp(b(2).*x).*(sin(2*pi*b(3)*x + b(4))) + b(5);             % Objective Function to fit
    fcn = @(b) norm(fit(b,t) - y);                                                  % Least-Squares cost function
    B = fminsearch(fcn, [yr; -0.1;  freq;  phase_init;  ym]);                       % Minimise Least-Squares
    if B(4)<0 % complement negative phase with 2pi
        B(4) = B(4) + 2*pi;
    end
    yf = fit(B,t);

end

[dareen]

wow looks really great i will try to understand the code and the fucntions ,but certainly sheds light on how i did not know what to implement i thought i should reach a fit on the x(t) but the x(t ) is a solution for the accleration so i was supposed to to think of fitting the a(t) graph i did not even grasp that so the folders i sent where cleaned to contation x(t) and t without accelration

[dareen]

so if i can ask a couple of questions to clear things up , the goal is to fit a(t) from there i can get the constants for every run such as omega and so on , since we know what solution works for a(t) we ask the app to do an approximation that fits the expected results

[dareen]

and once again thank you for your time and effort i am learning a lot

[Mathieu NOE]

as always, my pleasure !