Combine data from text files into 1?

I have equipment that gives us data at 3 frequency ranges. Each range (High, Mid, Low), has its own text file with 7 headers and 3 columns of data. How would I combine these 3 data sets so that they go from Lows to Highs and have no headers? I added the first handful of rows for each frequency as an example.

High:

Test Type:     SE
Test Point Information:       H  HF
Fiber Gain dB  External attenuation dB 
!      50           0  
Test Parameters -    Start Frequency:          Stop Frequency:   RBW:           Number of Points:  
!    200000000      1000000000.00001          30          200  
Frequency                          Amplitude                                 SE
200000000                          -95.7444746348                            105.8548329235
201624084.182                          -94.8430797684                            105.586601757
203261356.611                          -105.026149057                            113.9170141066
204911924.381                          -98.2910069978                            110.6246375614
206575895.456                          -104.618037004                            114.6189605781

Mid:

Test Type:     SE
Test Point Information:       H  MF
Fiber Gain dB  External attenuation dB 
!      50           0  
Test Parameters -    Start Frequency:          Stop Frequency:   RBW:           Number of Points:  
!    20000000      198852014.790596          30          394  
Frequency                          Amplitude                                 SE
20000000                          -88.182304421                            120.9731205046
20115461.26                          -94.3640015779                            127.6628331846
20231589.085                          -95.81268609200001                            129.5194587399
20348387.324                          -98.3275216973                            131.9908467909

Low:

Test Type:     SE
Test Point Information:       H  LF
Fiber Gain dB  External attenuation dB 
!      50           0  
Test Parameters -    Start Frequency:          Stop Frequency:   RBW:           Number of Points:  
!    10000      19654172          30          200  
Frequency                          Amplitude                                 SE
10000                          -73.7473367895                            32.2434412276
10387                          -77.5007700307                            36.59746393959999
10790                          -76.8728977036                            36.36316745000001
11208                          -76.2866404235                            36.27402701850001
11642                          -83.5528747158                            43.83786599699999

Answer 1

[dpb]

Exact details for the easiest way to handle the three files would depend on how they're named/saved so as to keep the sets associated together; somewhere you'll want to have an ID to identify who's who in the zoo, besides...

But, the details of that aside, with only three it's probably just as simple to simply write something such as

root='TheRootDataDirLocation';
dL=dir(fullfile(root,'*LOW*.txt'));     % salt to suit wildcard to match naming convention
dM=dir(fullfile(root,'*MID*.txt'));
dH=dir(fullfile(root,'*HIGH*.txt'));
for i=1:numel(dl)                       % there must be same number Lo, Mid, High -- add error check first
  tD=readtable(fullfile(dL(i).folder,dL(i).name),'numheaderlines',6,'readvariablenames',1);
  tD=[tD;readtable(fullfile(dM(i).folder,dM(i).name),'numheaderlines',6,'readvariablenames',1)];
  tD=[tD;readtable(fullfile(dH(i).folder,dH(i).name),'numheaderlines',6,'readvariablenames',1)];
  % do whatever with each set here before going on...
  writetable(tD,fullfile(root,'NEEDNAMINGCONVENTIONIDENTIFYEACHCASEHERE.'))
end

You could use readmatrix instead and skip all seven headerlines; the table form above may be handy to have the variable names for reference; your choice.

[Tyler Lawson]

This is awesome! I feel like its almost perfect! The only issue I am running into is that I need to be able to open a dialog box and select the 3 different files or have a dialog box open for each frequency and select the file I want and then it can put them together. The reason for that is I have anywhere from a few hundred to a few thousand of each frequency and I need to grab the 3 that go together. I can't write them into the code each time :)

Thank you so much though! My code is almost complete and I am super pumped for it to be done!

[dpb]

Well, doing that manually for hundreds or thousands of datasets won't be fun for the user, either.

That's why I wrote the preamble above about needing a way to be able to characterize the files wanted to select them with some wildcard pattern given a key word/phrase/something...

But, you've not shared what the naming system is to have any way to know what the most user friendly/least user beligerent method might be...

[Tyler Lawson]

Oh I understand what you are saying!

Here are what the files will look like

S121 Brooktrout - 2 HHF N-U-06 NOM.txt

S123 Brooktrout - 2 HMF N-U-06 NOM.txt

S125 Brooktrout - 2 HLF N-U-06 NOM.txt

Thanks for all the help! This is the first time I have used MatLab to select multiple files and do something with them. My experience is limited but I almost have a code that pulls data from equipment and changes the date file type (for example, .929 or .820) and then makes them text files. Then if I can get the 3 frequencies to be put into file, the code graphs the newly longer files with correct colors and legend and all that jazz!

[dpb]

So what is the key that those three go together -- the numbering in the beginning is different although there is the one text string? Is there a requirement there of odd and separated by 2 or are those fixed or random and just happen to have that pattern here?

You need some magic lookup table that the user can refer to to identify what it is they want and then the way to match that to the actual files. I presume the middle HXF is the ID for the three frequency ranges, but it's only with the "Brooktrout-2" and the last section that are unique. But the "N-U-06 NOM" part doesn't look to be enough to be unique for thousands of runs with only a 2-digit sequence number. Let's hope they didn't let that be free form.

Gonna' need more definition of "who's who in the zoo" to be able to define anything very handy...

[Tyler Lawson]

So the number at the beginning is a "shot" number that the software uses to track and save each time the equipment takes data. Then the first "H" is orientation so horizontal versus vertical. The second and third number are the frequency. The rest of the title is all user input that can be changed prior to saving each "shot". For each location, which is what the N-U-06 is, it will have anywhere from 2 to 8 files saved and then another location like N-U-07 will have data. Below is what the data from N-U-06 looked like. Sometimes there are some extra files that are from a diagnostic as well but those will act just like these where I need to select the 3 files in each orientation in order to cover the whole range of frequencies.

Thank you so much for all your help! You are way to nice to spend the time to help me understand what the code is looking for and helping me learn the more nested and deeper parts of this code!

[dpb]

OK, that helps some, at least. I'd then probably provide the user an input text box to enter the location of interest(*) and a radio button to select H,V, and then filter the files on that location and orientation and give them then the file selection dialog with those that match that description.

What about the "Brooktrout -2"? Is that some other possible frontend screening for a set that could also be a dropdown or a text input the user will know what they're interested in?

What would be useful it appears would be to create a database that has the various pieces of metadata buried in these filenames as actual data variables that could then be selected from by value.

You might attach a chunk of a dir() listing as a .mat or .txt file so could manipulate them; not much can do with the image other than look at it.

(*) Or, you could possibly parse out that substring and use it to populate a dropdown list as well although if there are hundreds of locations, that's also pretty overwhelming to have to sort through.

[Tyler Lawson]

The Brooktrout - 2 is user input into the equipment software. That's just the location we took this particular data but its all up to whoever is on the PC and using the software when collecting the data.

Here is another example at another location that someone else was at (not me) and this is how they had their naming setup.

I initially wanted the code to just ask for the LOW, MID, and HIGH files for the Horizontal direction, put them into a table and remove the 7 header lines and then add LOW + MID + HIGH so they are ordered correctly and then it can ask and do the same for the vertical direction, but I struggled to get the txt files to read into MatLab correctly which was annoying.

[dpb]

That, then, is going to be VERY hard to generalize without a consistent naming pattern; the previous idea of a global database then becomes all the more needed to be able to standardize those.

Is the "S" sequence number globally unique or does it start over, too, every time the instrument is redeployed?

The other alternative if the filestamps on these files doesn't get modified, would be to screen by date ranges as well, perhaps?

[Tyler Lawson]

Oh I figured by using something like uigetfile I could grab the 3 files I wanted put together and then have it read into a table but it seems like thats not as easy as it seems huh :)

The S is always there... it can be a few other options but it never is used and wouldn't be put into this code if it was anything other than the S. The # after the S starts over at 001 when the software is opened. It can be changed manually in case of a mistake made by the user but otherwise it just counts up after each time a file is saved.

[Tyler Lawson]

The original code you gave me is actually super close to working after messing around a little more with it since I first tried.

Is there a way to have this:

dL=dir(fullfile(root,'*HLF*.txt'));

dM=dir(fullfile(root,'*HMF*.txt'));

dH=dir(fullfile(root,'*HHF*.txt'));

look for both the HLF and something like a NOM or AON as these are options as well for different type os tests but they look the same to this current code and it wants to grab more than just the 3 files I want it to grab. See the image below for more details on what I mean. I need the AON ones put together and the FTN ones put together and then horizontal and vertical ones put togther so that I have 4 final files at the end of it all :)

[dpb]

Sure, that's simple enough...

dL=dir(fullfile(root,'*HLF*NOM.txt'));

will require "NOM" as the last three characters of the filename, yes.

Unfortunately by the available dir() wildcard patterns you can't get much more granularity than that; you could add in the location code something like

location=input('Please enter a location string: ')

so you end up with something like

location='N-L-09';

and then could write

orientation='H';
frequency='L';
type='NOM';
wildcard="*"+orientation+frequency+"F*"+location+type+".txt";
dL=dir(fullfile(root,wildcard));

To get the whole group, you would have to repeat for each of the three frequences and then combine the three listings.

The alternative is to let the frequency also be a wildcard character "?" -- that might work; I've not tried to see if does, not having an actual text set of filenames to work on.

Alternatively, of course, one can make the dir search very broad and then use regular expressions on the resulting name array and get much more selective than can do with the wildcards.

[Tyler Lawson]

I figured it out! Using your help and your code I was able to make it run through the different test types and create a single file for each of them and graph it. It asks for some input in order to name the files correctly but it works!

THANK YOU SO MUCH!

I do have one last question for you though! I have a plot being made and it brings up a dialog box in order to save it where the user wants. However, the plot still opens. Is there a way to surpress the plot so it doesn't open everytime the code is run and just saves them instead?

[dpb]

Wherever you create the figure (or, if you're letting the figure be created automagically, then do the creation yourself first instead) write

hF=figure('Visible','off');

then carry on as before.

If you want to have a way for the user to inspect one, then

hF.Visible='on';

will show the figure...