What is the best way to add empty rows to an existing table, to pre-allocate for more data?

Let's say I have a table that has 100 rows of ten variables and now I know I am going to need another 100 rows (or perhaps 100,000) in this table. To avoid loads of copying, I should preallocate these rows before populating them. The variables can be of various different types. I tried doing the following without success:

T{101:200,:} = missing;

I also tried:

T(101:200,:) = cell(100, width(T));

"Conversion to char from cell is not possible."

I just want the equivalent of T.addemptyrows(100). Or, if Matlab had an emptyrow(T) function, I could do:

T(101:200,:) = emptyrow(T)

Here's a toy example to play with:

T = table(["One"; "Two"; "Three"], [1; 2; 3], ['the'; 'cat'; 'sat'], {{1 2 3}; {4 5 6}; {7 8 9}})
T{4:10,:} = missing

Error using {}

Unable to perform assignment because value of type 'missing' is not convertible to 'char'.

Caused by:

Error using char

Conversion to char from missing is not possible.

If I remove the char variable, the same happens with cell.

I considered that I could take the first line of the table and duplicate it 100 times, then overwrite it one line at a time, but this is very inefficient, because cells could contain large amounts of data when all I really want is an empty variable. Additionally, this could hide errors since cells could appear to be populated with viable data when they wouldn't have been, and also could pose serious privacy and security risks in some scenarios.

I could, presumably, write a function that loops through the variable types of each and every variable in the existing table, and then creates a new table of the approriate size and variable types, and then copies the variable names from the first table, and then joins the two tables together, but that is a mess. I also worry about clobbering existing meta data or column data (since several variables could be in one column) and other issues I'm not even aware of. Is there not a more elegant solution? This must be a common thing to need to do. Thanks!

Answer 1

[Matt J]

T = table(["One"; "Two"; "Three"], [1; 2; 3], ['the'; 'cat'; 'sat'], {{1 2 3}; {4 5 6}; {7 8 9}})

T = 3×4 table
     Var1      Var2    Var3       Var4   
    _______    ____    ____    __________

    "One"       1      the     {1×3 cell}
    "Two"       2      cat     {1×3 cell}
    "Three"     3      sat     {1×3 cell}

newlength=10;

T(newlength+1,:)=T(1,:);
T(end,:)=[]

T = 10×4 table
      Var1       Var2    Var3        Var4    
    _________    ____    ____    ____________

    "One"         1      the     {1×3 cell  }
    "Two"         2      cat     {1×3 cell  }
    "Three"       3      sat     {1×3 cell  }
    <missing>     0              {0×0 double}
    <missing>     0              {0×0 double}
    <missing>     0              {0×0 double}
    <missing>     0              {0×0 double}
    <missing>     0              {0×0 double}
    <missing>     0              {0×0 double}
    <missing>     0              {0×0 double}

[Leon]

This achieves what I was looking for and does so without suppressing a warning; I could use this for my code. Thanks. One obvious disadvantage is that if the first row had cells with a lot of data, the unnecessary copying of it to the last line (that is then removed) could be substantial. This might not be so relevant when planning to add hundreds of lines, but imagine a table of two rows where each row contains cells amounting to a few hundred megabytes each and you want to add just three more rows. Apart from avoiding the warning, I wonder if there is an advantage to copying the first line to generate the last line, compared to the way I found and mentioned under Voss's example, which was:

T{end+n,1} = 0  % n is number of rows to add

The 0 strangely enough works for char, cell, string and double at least. For char, it conveniently turns into a char array of spaces of the required length. Is there any class for which 0 wouldn't work?

One minor disadvantage of this 0 method is that if the first variable is a string, the last row will have it as "0" rather than <missing>, which of course wouldn't matter for overwriting. If <missing> were required, your idea of adding an extra line and deleting it could be used.

[Matt J]

This achieves what I was looking for and does so without suppressing a warning

And avoids the need to parse the different data types of the table columns.

but imagine a table of two rows where each row contains cells amounting to a few hundred megabytes each and you want to add just three more rows.

Copying cells doesn't result in any new memory consumption. It's all done with pointers.

[Leon]

"Copying cells doesn't result in any new memory consumption. It's all done with pointers."

That's great, I didn't realise that. So your solution is very good. Thanks!

[Matt J]

You're welcome, but if it resolves your question, please Accept-click the answer.

Answer 2

[Voss]

T = table(["One"; "Two"; "Three"], [1; 2; 3], ['the'; 'cat'; 'sat'], {{1 2 3}; {4 5 6}; {7 8 9}})

T = 3×4 table
     Var1      Var2    Var3       Var4   
    _______    ____    ____    __________

    "One"       1      the     {1×3 cell}
    "Two"       2      cat     {1×3 cell}
    "Three"     3      sat     {1×3 cell}

T{end+100,1} = missing % add 100 rows

Warning: The assignment added rows to the table, but did not assign values to all of the table's existing variables. Those variables are extended with rows containing default values.

T = 103×4 table
      Var1       Var2    Var3        Var4    
    _________    ____    ____    ____________

    "One"         1      the     {1×3 cell  }
    "Two"         2      cat     {1×3 cell  }
    "Three"       3      sat     {1×3 cell  }
    <missing>     0              {0×0 double}
    <missing>     0              {0×0 double}
    <missing>     0              {0×0 double}
    <missing>     0              {0×0 double}
    <missing>     0              {0×0 double}
    <missing>     0              {0×0 double}
    <missing>     0              {0×0 double}
    <missing>     0              {0×0 double}
    <missing>     0              {0×0 double}
    <missing>     0              {0×0 double}
    <missing>     0              {0×0 double}
    <missing>     0              {0×0 double}
    <missing>     0              {0×0 double}

[Voss]

T = table(["One"; "Two"; "Three"], [1; 2; 3], ['the'; 'cat'; 'sat'], {{1 2 3}; {4 5 6}; {7 8 9}})

T = 3×4 table
     Var1      Var2    Var3       Var4   
    _______    ____    ____    __________

    "One"       1      the     {1×3 cell}
    "Two"       2      cat     {1×3 cell}
    "Three"     3      sat     {1×3 cell}

T{10,1} = missing % extend to the 10th row

Warning: The assignment added rows to the table, but did not assign values to all of the table's existing variables. Those variables are extended with rows containing default values.

T = 10×4 table
      Var1       Var2    Var3        Var4    
    _________    ____    ____    ____________

    "One"         1      the     {1×3 cell  }
    "Two"         2      cat     {1×3 cell  }
    "Three"       3      sat     {1×3 cell  }
    <missing>     0              {0×0 double}
    <missing>     0              {0×0 double}
    <missing>     0              {0×0 double}
    <missing>     0              {0×0 double}
    <missing>     0              {0×0 double}
    <missing>     0              {0×0 double}
    <missing>     0              {0×0 double}

[Leon]

Thanks, this is a great improvement. I forgot that if you write to a later row, any intermediate rows will be filled with empty values. Is there any more general solution, for example if the table might only have columns that don't support the <missing> value, or if we don't know what any of the column variable types will be? I mean, I could check the type of the first column and do various things based on it's type, but that feels very hacky. Also, I would like to avoid the error message if possible. I added to my original question, by the way.

[Matt J]

for example if the table might only have columns that don't support the <missing> value

Such as what?

Also, I would like to avoid the error message if possible

Warning, not error. You can use the warning command to turn off specific warnings.

[Leon]

char and cell, perhaps others too.

[Matt J]

But these do support a missing value, as you can see from Voss' example.

[Leon]

No, actually it doesn't show that. For example, if the first variable is char:

T = table(['One..'; 'Two..'; 'Three'], [1; 2; 3], ['the'; 'cat'; 'sat'], {{1 2 3}; {4 5 6}; {7 8 9}})
T{end+5,1} = missing

Produces:

Error using {}

Unable to perform assignment because value of type 'missing' is not convertible to 'char'.

Caused by:

Error using char

Conversion to char from missing is not possible.

If the first column is char, then I would have to use:

T = table(['One..'; 'Two..'; 'Three'], [1; 2; 3], ['the'; 'cat'; 'sat'], {{1 2 3}; {4 5 6}; {7 8 9}}, ["One"; "Two"; "Three"])
T{end+5,1} = blanks(length(T{1,1}))

Voss's example is very helpful, but it relies on the first column being of a type that supports <missing>, or finding and using another column that happens to support it. I could check the type of the first variable and do different things based on what type it is, but, like I said, this is hacky and it's disappointing if there really is no elegant way to achieve this.

In terms of checking the first column, is it only char and cell that do not support <missing> or are there others to check for? How would I even find this out, I wonder.

Actually, using 0 rather than <missing> seems to work for char, so maybe that will work for all classes of variable?:

T = table(['One..'; 'Two..'; 'Three'], [1; 2; 3], ['the'; 'cat'; 'sat'], {{1 2 3}; {4 5 6}; {7 8 9}}, ["One"; "Two"; "Three"])
T{end+2,1} = 0

Regarding the warning in Voss's example, is there a special comment that can be added to that one line to suppress the warning?

[Voss]

"is there a special comment that can be added to that one line to suppress the warning?"

Not a comment, but a command. The warning function can be used to turn on and off individual warning messages by ID. To get the ID you can use the lastwarn function.

Demonstration: First, generate the warning in question:

T = table(["One"; "Two"; "Three"], [1; 2; 3], ['the'; 'cat'; 'sat'], {{1 2 3}; {4 5 6}; {7 8 9}});
T{end+100,1} = missing; % warning is issued

Warning: The assignment added rows to the table, but did not assign values to all of the table's existing variables. Those variables are extended with rows containing default values.

Get the warning ID using lastwarn:

[~,wid] = lastwarn()

wid = 'MATLAB:table:RowsAddedExistingVars'

Turn the warning off:

warning('off',wid)

Verify that it worked by trying to generate the same warning again:

T{end+100,1} = missing; % no warning this time

Now that you know the relevant warning ID, in your code you can turn off the warning before any are generated:

warning('off','MATLAB:table:RowsAddedExistingVars')

[Leon]

Thanks re. info about lastwarn and warning.