opening a 1GB .dat file in matlab.

Hi, I have a data file (i dont know if its binary or ascii). But it has extension test.dat and I can open it in excel 2010. In excel it says could not open full file and displays 1048576 rows (maximum number of rows) and 20 coloumns. While opening in excel I choose comma delimited option and excel could open it in 1048576 X 20 cells. The first row is text (like srno, V1, V2, etc) rest of the cells are just floating point nos.

--------------------contents of TestFile.txt--------------------------

Y[mm], Freq[GHz], t[ns], P1_LSB[V], P2_LSB[V], P3_LSB[V], P4_LSB[V], P5_LSB[V], P6_LSB[V], P7_LSB[V], P8_LSB[V], P9_LSB[V], P1_USB[V], P2_USB[V], P3_USB[V],P4_USB[V],P5_USB[V], P6_USB[V], P7_USB[V],P8_USB[V],P9_USB[V], REF_LSB[V], REF_USB[V]
-1.639999,8.000000,-2.500000e+01,2.613065e-01,0.000000e+00,6.834170e-01,2.211055e-01,-8.844220e-01,1.206030e-01,-1.165829e+00,6.432160e-01,3.618090e-01,-3.015075e-01,-8.040200e-01,-1.407035e-01,-5.025125e-01,-8.241205e-01,-1.226130e+00,1.407035e-01,1.005025e-01,2.412060e-01,3.417085e-01,5.025125e-01
-1.639999,8.000000,-2.492187e+01,2.613065e-01,-4.020100e-02,7.035175e-01,2.814070e-01,-9.045225e-01,1.005025e-01,-1.165829e+00,6.231155e-01,3.216080e-01,-3.216080e-01,-8.241205e-01,-1.206030e-01,-5.226130e-01,-8.241205e-01,-1.226130e+00,1.407035e-01,1.407035e-01,2.613065e-01,3.618090e-01,5.025125e-01
-1.639999,8.000000,-2.484375e+01,2.412060e-01,-1.005025e-01,7.236180e-01,3.417085e-01,-9.045225e-01,8.040200e-02,-1.145728e+00,6.030150e-01,2.814070e-01,-3.216080e-01,-8.241205e-01,-1.206030e-01,-5.427135e-01,-8.442210e-01,-1.226130e+00,1.407035e-01,1.608040e-01,2.613065e-01,3.819095e-01,5.025125e-01
-1.639999,8.000000,-2.476562e+01,2.412060e-01,-1.407035e-01,7.437185e-01,3.819095e-01,-9.246230e-01,6.030150e-02,-1.145728e+00,5.829145e-01,2.211055e-01,-3.417085e-01,-8.442210e-01,-1.005025e-01,-5.628140e-01,-8.442210e-01,-1.226130e+00,1.608040e-01,1.809045e-01,2.613065e-01,4.020100e-01,5.025125e-01

---------------------------------------------------

If I use load in matlab, it says

>>load data.dat
??? Error using ==> load
Unknown text on line number 1 of
ASCII file C:\Users\Documents\MATLAB\data.dat
"Y[mm]"."

Pls tell me how to open the file or read the data into matrix so that I can do further processing the data.

tks

[Jan]

This is an ASCII file with 1 header line.

Answer 1

[Walter Roberson]

textscan() with HeaderLines 1, Delimiter of ',', CollectOutput true, and format of

repmat('%f', 1, 13)

[strunack]

Sorry cannot do this in matlab, dont find the correct format to use collectoutput

[Walter Roberson]

fmt = repmat('%f', 1, 13);
FID = fopen(FileName, 'r');
C = textscan(FID, fmt, 'HeaderLines', 1, 'Delimiter', ',', 'CollectOutput', 1);
fclose(FID);
M = C{1};

[strunack]

This hangs the PC but works, takes a long time. The format I am getting is not in the right way. In the mean time I tried this

[fid,message] = fopen('data.dat','r');
nLines = 0;
while (fgets(fid) ~= -1),
  nLines = nLines+1;
end
fclose(fid);
disp(nLines)

to find out number of coloumns.

then I used

A=dlmread('data.dat',',',[1 0 6903623130 23]); size(A)

it gives the format. But takes too much time.

Thank you for your help.

[Walter Roberson]

Looks like I miscounted, and the 13 I indicated for "fmt" should be 23.

[strunack]

Ok ok I got it. tks. For the huge time it takes, I think I will try reading it part by part.

[Walter Roberson]

You can give textscan() an optional parameter indicating the number of times to apply the format (and thus, for your purposes, the number of lines to read in at a time.)

Answer 2

[Jan]

Please give us a chance to identify the file type.

FID = fopen(FileName, 'r');
if FID == -1, error('Cannot open file for reading.'); end
C = cell(1,5);
for ii = 1:5
  C{ii} = fgetl(FID);
end
fclose(FID);

FID = fopen(fullfile(tempdir, 'TestFile.txt'), 'w');
if FID == -1, error('Cannot open file for writing.'); end
fprintf(FID, '%s\n', C{:});
fclose(FID);

Now copy and paste the contents of the first 5 lines taken from TestFile.txt. Please insert it in the original question, neither as comment nor as answerm because it is an essential part of the question.