Reading data from file with breaks

I'm trying to read values from some very large files and store them as a matrix. The issues is as well as having headers at the top, there are also headers repeated every 10k lines or so. For example

header 1, header 2, header 3;

1 2 3;

%(9998 rows of numbers)

4 5 6;

header 1, header 2, header 3;

7 8 9

dlmread doesn't work, and I've tried textscan and fread, which both sort of do what I want but not quite. Thanks in advance.

[Thorsten]

It is easier to answer such questions if you provide a sample file.

[Timothy Devenport]

Sample uploaded

Answer 1

[Star Strider]

The textscan function will work, but you have to use fseek to restart it each time it hits the interim text line every 10K or so lines.

This is an archived code snippet to illustrate the technique:

fidi = fopen('S21.txt');
D1 = textscan(fidi, '%f %f', 'HeaderLines',2, 'Delimiter','\n', 'CollectOutput',1);
D1{:}([1:5 end-4:end],:)        % Diagnostic Look
fseek(fidi,0,0);                % Position Start Of Second Part Of File
D2 = textscan(fidi, '%f %f', 'HeaderLines',2, 'Delimiter','\n', 'CollectOutput',1);
D2{:}([1:5 end-4:end],:)        % Diagnostic Look

The ‘Diagnostic Look’ lines verify that the file is importing correctly. They are not important for the code and you can delete them.

[Timothy Devenport]

That produced this error: Subscript indices must either be real positive integers or logicals.

Error in Analysisatt1 (line 19) D1{:}([1:5 end-4:end],:) % Diagnostic Look

How do I fix it?

Also, will that code work for skipping more than 1 line?

[Star Strider]

I would have to have your file to attempt textscan code specifically for it. I intend this as an illustration on restarting textscan each time it encounters text, and then stops, assuming the number of header lines and the following numeric data do not change each time. (My code worked for the file it was written for.)

[Timothy Devenport]

Ok, here is a cut down version of one of the files.

[Star Strider]

This works on the file you provided, and should work on your entire file:

fidi = fopen('Timothy Devenport five_2_x.txt');
k1 = 1;
D{k1} = textscan(fidi, '%f%f%f', 'HeaderLines',5, 'Delimiter','\t', 'CollectOutput',1);
while ~feof(fidi)
    k1 = k1 + 1;
    D{k1} = textscan(fidi, '%f%f%f', 'HeaderLines',3, 'Delimiter','\t', 'CollectOutput',1);
end
% View Imported Data & Correct Addressing Format: 
D1 = D{1}{1}([1:5 end-4:end],:)     % Diagnostic Look
D2 = D{2}{1}([1:5 end-4:end],:)     % Diagnostic Look

The code assumes more than one ‘text break’ in the file, and that except for the first section (with 5 header lines), all the rest have 3. The cell array does not require that all sections of the file have the same number of entries. It simply cycles through all of them until it encounters and ‘end-of-file’ indicator, then stops.

The ‘Diagnostic Look’ lines let you verify that the data imported correctly, and also provide the correct way to get the data from the cell array.

[Timothy Devenport]

This appears to be on the right lines, but I'm not sure it's working quite right. It seems to cut down the data files to just 30 cells in D1 and D2, and that's clearly incorrect, as there should be hundreds of cells.

I'm wondering if it's skipping the header lines, reading a line, then skipping the next lines (as if they're headers) and reading a line and so on...

[Star Strider]

The ‘D1’ and ‘D3’ lines are there only to verify that the code is reading the file correctly. They display the first 5 and last 5 lines in each segment to be sure they were imported correctly. They are otherwise not part of the actual code, so you can delete them.

When I run the code I posted, then add these two lines:

SizeD1 = size(D{1}{1})
SizeD2 = size(D{2}{1})

I get:

SizeD1 =

     200     3

SizeD2 =

     335     3

I didn’t count the lines manually, but it seems to be working as it should, at least with the file you provided.

While I’m thinking about it, this is the way to concatenate them if you want to:

Dcat = [D{1}{1}; D{2}{1}];

producing a (535x3) double array with the file you provided.