What is maximum size of line that can be read by fgetl/fgets?

Hello everyone,

I am using array of structures to read a text file that contains many lines of variable lengths. I am getting an error when MATLAB reads the longest line. The error is

Error using fgets Invalid file identifier. Use fopen to generate a valid file identifier.

Error in fgetl (line 33) [tline,lt] = fgets(fid);

Error in ReadRefGPSData (line 18) ASCII(i).tline= fgetl(fid);

Where i in ASCII(i).tline= fgetl(fid); is the index or line number of the longest line.

Can anyone tell me is there any maximum limit of fgetl/fgets(as inside the fgetl MATLAB function fgets is used)? If yes, what is the otherway around, to read a long line in two halves. I cannot break line in two or more shorter lines, as it is generated by system.

Thanks in advance.

[José-Luis]

Maybe it is not due to the line length. What's in the aggravating line?

[Naveed Ahmed]

Thank you for the comments.

The section of file that contains the unusually long line is shown below:

#RAWGPSWORDA,COM1,0,40.5,FINESTEERING,1903,198017.471,00000000,9b16,9554;28,82bd86dc*f953d131 #RAWGPSWORDA,COM1,0,40.5,FINESTEERING,1903,198017.468,00000000,9b16,9554;11,83c240b0*8883bb4f #RAWGPSWORDA,COM1,0,40.5,FINESTEERING,1903,198017.476,00000000,9b16,9554;8,e83c41c*f1d4946a #RAWEPHEMA,COM1,0,40.5,SATTIME,1903,198000.000,00000000,97b7,9554;11,1903,201600,8b0dbc4074a7dbd0000e8b0ed9ea257b2a6588dbe60f313800fff0aa4ca6,8b0dbc4075290fff8b44aa12da365cff6d0875a26b0b02a10d538f313852,8b0dbc4075afff977f127d51ffd8248f73ec1edb3e95eab6ff99930f0902*01c8f7af #RAWGPSWORDA,COM1,0,40.5,FINESTEERING,1903,198017.467,00000000,9b16,9554;1,713f8484*d023a902

#RAWEPHEMA (before #RAWGPSWORDA) is all in the same line.

It is important to mention here that the MATLAB is reading correctly the (shorter)lines before this line.

Answer 1

[KSSV]

There is no limit as such to read the lines according to my experience. Have you tried opening the file using fopen?

fid = fopen('yourfile.txt','r') ;
tline = fgetl(fid) ;

[Naveed Ahmed]

Thank you for the answer.

Yes, I tried but in vain.

The section of file that contains the unusually long line is shown below:

#RAWGPSWORDA,COM1,0,40.5,FINESTEERING,1903,198017.471,00000000,9b16,9554;28,82bd86dc*f953d131 #RAWGPSWORDA,COM1,0,40.5,FINESTEERING,1903,198017.468,00000000,9b16,9554;11,83c240b0*8883bb4f #RAWGPSWORDA,COM1,0,40.5,FINESTEERING,1903,198017.476,00000000,9b16,9554;8,e83c41c*f1d4946a #RAWEPHEMA,COM1,0,40.5,SATTIME,1903,198000.000,00000000,97b7,9554;11,1903,201600,8b0dbc4074a7dbd0000e8b0ed9ea257b2a6588dbe60f313800fff0aa4ca6,8b0dbc4075290fff8b44aa12da365cff6d0875a26b0b02a10d538f313852,8b0dbc4075afff977f127d51ffd8248f73ec1edb3e95eab6ff99930f0902*01c8f7af #RAWGPSWORDA,COM1,0,40.5,FINESTEERING,1903,198017.467,00000000,9b16,9554;1,713f8484*d023a902

#RAWEPHEMA (before #RAWGPSWORDA) is all in the same line.

It is important to mention here that the MATLAB is reading correctly the (shorter)lines before this line.

Answer 2

[Steven Lord]

That error isn't related to the length of the lines in the file; MATLAB didn't get that far. The file was not opened successfully. Call fopen with two output arguments. Before using the first of those output arguments, the file identifier, check if it is -1. If it is that means fopen was not able to open it successfully and the second output from the fopen call will be a message that hopefully provides an explanation as to why it wasn't able to open the file.

[Naveed Ahmed]

Thanks for your answer,

I have verified the file identifier is not -1 as the section of file before the line is being read impeccably. Please see my comments to other responders for more detail.

Answer 3

[Walter Roberson]

My tests indicate that in R2016a on OS-X, fgetl() can read lines whose lengths exceed 6/10 gigabytes. I expect that it would be able to handle lines up to the point where it runs out of memory or hits the limit you have configured on array size.

However, beyond about 1/4 gigabytes it gets pretty slow; I have had it working on reading 1.2 gigabytes for several minutes now, and the time it is taking leaves me wondering whether the internals are growing the array dynamically with lots of copying (rather than, for example, scanning ahead to determine how far away the line terminator is, allocating that much memory, and then going back to read in the line -- an algorithm you can only use on block structured inputs that you can seek backwards in.)

[Walter Roberson]

Okay, so I have checked, and besides the obvious limit of "amount of memory available", there is a more subtle limit.

fgetl() is implemented by calling fgets(), similar to

[data, line_terminator] = fgets(FileID);

On my system, that worked fine on a line of total length 1207959544, producing a data variable with twice that many bytes (because each char at the MATLAB level is 2 bytes.)

Then fgetl does the equivalent of

data = data(1:end-length(line_terminator))

and that was hitting the limit I had configured in preferences for maximum array size.

I was puzzled as to why it was complaining about needing a 9 Gb array to do the above operation, when the output should be no more than 1207959544*2 bytes, about 2 1/4 Gb. I was thinking that perhaps it was failing to take into account that char are smaller than double and was over-counting the space requirement.

But then I realized that the 9 Gb was being required to generate the elements in the input vector, 1:end-length(line_terminator) which are being generated as double

So that is the more subtle limit: your lines are limited to the number of characters equal to 1/8th of your configured largest array.

However, you will not get an error about invalid file ID if you encounter this: you will get an error about array being too large.