Data extraction from txt file with changing number of columns

2 views (last 30 days)
Hi everyone,
I am running simulations in a software where the output is essentially a massive amount of data in .txt files. I am trying to extract and plot the data using matlab. This is as far as I have made it, with the trial data set for 10 Monte Carlo runs. The problem is that the number of Monte Carlo runs vary so the output which is in coloumns that I will attach below varies as well. Is there maybe an alternative way to extract the data thats more proficient than what I am doing right now? or is there maybe a loop that could make it faster?
clc; close all; clear all;
%import data from DAMAGE
fid = fopen('All-In-19-QUICK_EVAL.txt');
%file 1 - lifetime
lifetime = fscanf(fid, '%s %s %s %s', [1 4]);
lifetimedata = fscanf(fid,'\n %f %f',[2 24]);
proportion = lifetimedata(1:2:39);
countlifetime = lifetimedata(2:2:40);
plot(proportion, countlifetime,'b');
title('Lifetime Proportion')
xlabel('proportion [.1%]'); ylabel('count'); xlim([0 1]);
%file 2 - altitude
altituded = fscanf(fid, '\n %s %s %s', [1 3])
altitudedata = fscanf(fid, '\n %f %f', [2 36]);
altitude = altitudedata(1:2:72);
countaltitude = altitudedata(2:2:72);
% figure(2)
% plot(altitude, countaltitude,'b');
% title('Altitude');
% xlabel('altitude [km]'); ylabel('count'); xlim([0 2000]);
%file 3 - cumulative number of collisions - this section is my main problem
file3 = fscanf(fid, '\n %s %s %s', [1 30]);
file3data = fscanf(fid, '\n %f %f %f', [14 1800])
year3 = file3data(1:14:1414);
run1in3 = file3data(2:14:1414);
run2in3 = file3data(3:14:1414);
run3in3 = file3data(4:14:1414);
run4in3 = file3data(5:14:1414);
run5in3 = file3data(6:14:1414);
run6in3 = file3data(7:14:1414);
run7in3 = file3data(8:14:1414);
run8in3 = file3data(9:14:1414);
run9in3 = file3data(10:14:1414);
run10in3 = file3data(11:14:1414);
mean3 = file3data(12:14:1414);
negStandardDev3 = file3data(13:14:1414);
posStandardDev4 = file3data(14:14:1414);
i have attached the .txt this evaluates. NOTE: code is only extracting data from the first 3 data blocks, ideally i would want a way to extract all data.
  1 Comment
dpb on 3 Jul 2022
It is easy enough to read each section; the problem is they left no trail of bread crumbs through the forest/file to use in reading the file as for number of variables, number of observations, etc., etc., etc., ... so you've got to parse the file to find all of those things --
I'd probably start a file like this by reading the whole thing into a string array and finding the empty lines as the sections -- then you'll need a corollary database of the section types and what is the unique section heading that can be used to distinguish which section you're reading.
It's all doable and not terribly difficult, but the tedium factor is immense in working through all the details.

Sign in to comment.

Answers (1)

per isakson
per isakson on 22 Jul 2022
Edited: per isakson on 22 Jul 2022
The file All-In-19-QUICK_EVAL.txt contains 73 sections. Each section
  • starts with two consequtive empty rows followed by
  • in the final 27 sections a simulation case ID, e.g. 2028_86, followed by
  • a descriptive header row followed by
  • a column header row followed by
  • a block of numerical data
I noticed that
  • neither the header nor the column header provides names, which are legal in Matlab
  • not all section types have unique headers, e.g. more than one section type has the header, Number of Objects > 10 cm
  • there are nine (not ten) unique simulation case ID
I created a function to read and parse All-In-19-QUICK_EVAL.txt. It's a first attempt. The data of each section is stored in one structure. (73 structures in total.) The names of the structures are modified versions of the headers and the names of the fields are modified versions of the column headers. (That's simple to implement.) To create dynamically named structures I use the function, matfile. Thus acknowledging: TUTORIAL: Why Variables Should Not Be Named Dynamically (eval).
avdb_3( 'All-In-19-QUICK_EVAL.txt', 'test.mat' )
load test.mat
Name Size Bytes Class Attributes Altitude 1x1 1240 struct CumulativeNumberOfCatastrophicCollisions 1x1 6120 struct CumulativeNumberOfCatastrophicCollisionsWithoutConstellation 1x1 6120 struct CumulativeNumberOfCatastrophicCollisionsWithoutSentinel 1x1 6120 struct CumulativeNumberOfCollisions 1x1 6120 struct CumulativeNumberOfCollisionsWithoutConstellation 1x1 6120 struct CumulativeNumberOfCollisionsWithoutSentinel 1x1 6120 struct CumulativeNumberOfCriticalCollisions 1x1 6120 struct CumulativeNumberOfDamagingCollisions 1x1 6120 struct CumulativeNumberOfDamagingCollisionsWithoutConstellation 1x1 6120 struct CumulativeNumberOfDamagingCollisionsWithoutSentinel 1x1 6120 struct LifetimeProportion 1x1 656 struct NumberOfCatastrophicCollisionsByAltitude 1x1 4040 struct NumberOfCatastrophicCollisionsByAltitudeWithoutConstellation 1x1 4040 struct NumberOfCollisionFragmentsGt10Cm 1x1 108600 struct NumberOfCollisionFragmentsGt10CmWithoutConstellation 1x1 108600 struct NumberOfCollisionFragmentsGt10CmWithoutSentinel 1x1 108600 struct NumberOfCollisionFragmentsGt1Cm 1x1 108600 struct NumberOfDeadObjectsGt10Cm 1x1 108600 struct NumberOfDeadObjectsGt10CmWithoutConstellation 1x1 108600 struct NumberOfDeadObjectsGt10CmWithoutSentinel 1x1 108600 struct NumberOfExplosionFragmentsGt10Cm 1x1 108600 struct NumberOfExplosionFragmentsGt1Cm 1x1 108600 struct NumberOfIntactsGt10Cm 1x1 108600 struct NumberOfIntactsGt10CmWithoutConstellation 1x1 108600 struct NumberOfIntactsGt10CmWithoutSentinel 1x1 108600 struct NumberOfIntactsGt1Cm 1x1 108600 struct NumberOfNEWCollisionFragmentsGt10Cm 1x1 108600 struct NumberOfNEWCollisionFragmentsGt10CmWithoutConstellation 1x1 108600 struct NumberOfNEWCollisionFragmentsGt10CmWithoutSentinel 1x1 108600 struct NumberOfNon_CatastrophicCollisionsByAltitude 1x1 4040 struct NumberOfNon_CatastrophicCollisionsByAltitudeWithoutConstellatio 1x1 4040 struct NumberOfNon_CatastrophicCollisionsByAltitudeWithoutSentinel 1x1 4040 struct NumberOfOLDCollisionFragmentsGt10Cm 1x1 108600 struct NumberOfObjectsGt10Cm 1x1 10024 struct NumberOfObjectsGt10CmWithoutConstellation 1x1 108600 struct NumberOfObjectsGt10CmWithoutConstellation_1 1x1 108600 struct NumberOfObjectsGt10CmWithoutSentinel 1x1 108600 struct NumberOfObjectsGt10CmWithoutSentinel_1 1x1 108600 struct NumberOfObjectsGt10Cm_1 1x1 108600 struct NumberOfObjectsGt10Cm_2 1x1 108600 struct NumberOfObjectsGt1Cm 1x1 6120 struct SpatialDensity_ObjectsGt10Cm 1x1 106520 struct SpatialDensity_ObjectsGt10CmWithoutConstellation 1x1 106520 struct SpatialDensity_ObjectsGt10CmWithoutConstellation_1 1x1 106520 struct SpatialDensity_ObjectsGt10CmWithoutConstellation_2 1x1 106520 struct SpatialDensity_ObjectsGt10CmWithoutConstellation_3 1x1 106520 struct SpatialDensity_ObjectsGt10CmWithoutConstellation_4 1x1 106520 struct SpatialDensity_ObjectsGt10CmWithoutConstellation_5 1x1 106520 struct SpatialDensity_ObjectsGt10CmWithoutConstellation_6 1x1 106520 struct SpatialDensity_ObjectsGt10CmWithoutConstellation_7 1x1 106520 struct SpatialDensity_ObjectsGt10CmWithoutConstellation_8 1x1 106520 struct SpatialDensity_ObjectsGt10CmWithoutConstellation_9 1x1 106520 struct SpatialDensity_ObjectsGt10CmWithoutSentinel 1x1 106520 struct SpatialDensity_ObjectsGt10CmWithoutSentinel_1 1x1 106520 struct SpatialDensity_ObjectsGt10CmWithoutSentinel_2 1x1 106520 struct SpatialDensity_ObjectsGt10CmWithoutSentinel_3 1x1 106520 struct SpatialDensity_ObjectsGt10CmWithoutSentinel_4 1x1 106520 struct SpatialDensity_ObjectsGt10CmWithoutSentinel_5 1x1 106520 struct SpatialDensity_ObjectsGt10CmWithoutSentinel_6 1x1 106520 struct SpatialDensity_ObjectsGt10CmWithoutSentinel_7 1x1 106520 struct SpatialDensity_ObjectsGt10CmWithoutSentinel_8 1x1 106520 struct SpatialDensity_ObjectsGt10CmWithoutSentinel_9 1x1 106520 struct SpatialDensity_ObjectsGt10Cm_1 1x1 106520 struct SpatialDensity_ObjectsGt10Cm_2 1x1 106520 struct SpatialDensity_ObjectsGt10Cm_3 1x1 106520 struct SpatialDensity_ObjectsGt10Cm_4 1x1 106520 struct SpatialDensity_ObjectsGt10Cm_5 1x1 106520 struct SpatialDensity_ObjectsGt10Cm_6 1x1 106520 struct SpatialDensity_ObjectsGt10Cm_7 1x1 106520 struct SpatialDensity_ObjectsGt10Cm_8 1x1 106520 struct SpatialDensity_ObjectsGt10Cm_9 1x1 106520 struct SpatialDensity_ObjectsGt1Cm 1x1 106520 struct ans 1x49 98 char cmdout 1x33 66 char
With help of tab-completion and copy&paste it's possible to use these names.
plot( LifetimeProportion.Proportion, LifetimeProportion.Count,'b');
title('Lifetime Proportion')
xlabel('proportion [.1%]'); ylabel('count'); xlim([0 1]);




Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!