error in tall array tables due to missing data in end column of end row

I have a csv file which looks something like this:

DATA_START
02/16/18 01:03:32.11,,4,
02/16/18 01:03:32.22,,4,
02/16/18 01:03:32.33,,5,
02/16/18 01:03:32.44,,5,
02/16/18 01:03:32.11,,0.4,
02/16/18 01:03:32.01,4,4,
02/16/18 01:03:32.02,,4,
02/16/18 01:03:32.03,,3,
02/16/18 01:03:32.04,7,0,

I have noticed that when I use data store and tall arrays (tt = tall(ds)), I get an error and tall array looks like:

tt =

M×4 tall table

    date_time    Var2    Var3    Var4
    _________    ____    ____    ____

    ?            ?       ?       ?   
    ?            ?       ?       ?   
    ?            ?       ?       ?   
    :            :       :       :
    :            :       :       :

I have noticed that if I add a variable at the end row of the end column, this problem goes away, but as I am dealing with a lot of data files (some of which are too large to open) I need to be able to resolve this issue without having to manually change the file. Any idea how I can fix this?

[Benjamin Kraus]

What error do you get? What code are you using to load the CSV file to a datastore?

[minomi]

I've created datastore using:

ds = datastore(data_file_location);

and tall array using:

 tt = tall(ds)

the error is:

Warning: Error encountered during preview of tall array 'tt'. Attempting to gather 'tt' will probably result in an error. The error encountered was:
The data in file: 'C:\Users\foldername\test1.csv' does not
appear to be tabular, with the same number of fields in each row and in each column. Verify the Text Format and Advanced Text Format Properties.

Answer 1

[Edric Ellis]

It sounds like you haven't got the format quite right for your datastore. I suspect you would encounter the same error if you called readall on your datastore. To work with datastore, you need to ensure that all the files are in the same format.

There's lots of information about how to set up the datastore to read tabular text (such as .csv) in the documentation.

[minomi]

I am not sure if I understand your reply. Do you mean that I should apply

ttds = tabularTextDatastore(location)

to the data file and that puts the data in a format that is acceptable to datastore? If that is what you meant, I did that and it didn't work. I have used

 ds.SelectedFormats(1) = {'%{MM/dd/yy HH:mm:ss.SS}D'}

to alter the format of the data time values. Perhaps you are indicating that I should be doing something similar to the last column?

[Edric Ellis]

For me, using the example data that you show in the question, then simply writing

ds = datastore('data.csv');
t = tall(ds)

results in a working tall table. Please could you supply an example of the data that doesn't work.

[Chandani Madnani]

There could be a mismatch because of missing values in the data. Treat 'NA' values as missing data so that they are replaced with NaN values

ds = datastore('data.csv','TreatAsMissing', 'NA') t=tall(ds)

Does this resolve the issue ?

[minomi]

@Edric Ellis this is the example of the data file that doesn't work for me and hence my question posted here. Can I ask which version of Matlab you are using? I cannot think of any reason why the same code and data file would for you but not me ...

[minomi]

@Chandani Madnani I still get the same error . Can I ask if you get the same errors with this data file?

[minomi]

I just copied and pasted the file into a new gedit document and that seemed to get rid of the error. Previously I was using Notepad. Any idea why this might be? How can I get rid of this problem without having to open the files?