Getting MD5 hash of files with "bad" names (Windows, NTFS)
Show older comments
I tryed several variants of getting MD5 hash of files with "bad" names (e.g. 'C:\wrongname.' (with trailing dots); 'C:\wrongname ' (with trailing spaces); (path)names with accents like 'é', 'é' which occur in French, German, Hungarian; containing various forms of dashes (–) etc.).
None is working, except Var5, but is too slow (calculates a 10Mb file within 10 min).
Can you suggest any working variant, or how to speed up Var 5?
OS: Windows 10, File system: NTFS, Matlab 2015a
% Var 1 ---------
% dirinfo(i).name - is string containing full pathname (e.g. 'C:\myfolder\myfile.ext')
Opt.Format = 'HEX'; Opt.Method = 'MD5'; Opt.Input='file';
hash(i) = DataHash(['\\?\' dirinfo(i).name], Opt); % ERROR - not working with "bad" names
% DataHash.m - https://www.mathworks.com/matlabcentral/fileexchange/31272-datahash
%Var 2 ----------
hash(i)=mMD5(['\\?\' dirinfo(i).name]); % fast, works on 'C:\wrongname.' (with ending dots), 'C:\wrongname ' (with ending spaces), but do NOT works with file names (or pathes) with accents like 'é', é'
% (mMD5.c, see https://www.mathworks.com/matlabcentral/fileexchange/7919-md5-in-matlab)
% Var 3 ---------
mddigest = java.security.MessageDigest.getInstance('MD5');
bufsize = 8192;
[fid,errmsg] = fopen(['\\?\' dirinfo(i).name]); % ERROR here - matlab fopen don't understand "bad" names
if fid>=3 % if success
while ~feof(fid)
[currData,len] = fread(fid, bufsize, '*uint8');
if ~isempty(currData)
mddigest.update(currData, 0, len);
end
end
fclose(fid);
hash(i) = reshape(dec2hex(typecast(mddigest.digest(),'uint8'))',1,[]);
else
disp('can't open file');
end
% Var 4 ---------
file = java.io.File(['\\?\' dirinfo(i).name]);
digestream = java.security.DigestInputStream(file,mddigest);
file_bytes = typecast(org.apache.commons.io.FileUtils.readFileToByteArray(file),'uint8'); % ERROR: out of memory if BIG file
if ~isempty(file_bytes)
mddigest.update(file_bytes, 0, numel(file_bytes));
end
hash(i) = reshape(dec2hex(typecast(mddigest.digest(),'uint8'))',1,[]);
% Var 5 --------
mddigest = java.security.MessageDigest.getInstance('MD5');
filestream = java.io.FileInputStream(java.io.File(['\\?\' dirinfo(i).name]));
digestream = java.security.DigestInputStream(filestream,mddigest);
while(digestream.read() ~= -1), end % TOO LONG - never goes out this cycle
hash(i)=reshape(dec2hex(typecast(mddigest.digest(),'uint8'))',1,[]);
Answers (2)
Do not end a file or directory name with a space or a period. Although the underlying file system may support such names, the Windows shell and user interface does not.
As you've found out, matlab fopen does not support it. .Net (which you can directly from matlab) also does not. From your testing, it looks like Java does not handle it properly either.
The only way you can access such file is directly through the win32 api, e.g. with CreateFile. So if you really need to handle such paths you'll have to resort to mex.
But really, the best fix would be to fix the tool that creates these files in the first place so that it doesn't use bad filenames. It's not just matlab that can't handle them, it's also most backup tools, file transfer tools, etc.
As for path names with accents, there does not appear to be any problem there. Matlab (R2017a tested) handles them fine.
8 Comments
Guillaume
on 17 Oct 2017
No the tool to fix is the one that creates files ending with dots and spaces. Microsoft are not going to change the win32 api and you'll find plenty of other programs that cannot cope with these files.
Yes, you can work around the problem by using the '\\?\' syntax with some api (java it seems, and parts of win32), but you're still papering over the problem that these files shouldn't exist.
+1 Fixing the file names is the correct solution. Anything else will just be a total waste of time (as the duration of the current discussion already proves).
Sometimes creating hacks can be useful, when it creates some useful functionality that cannot be achieved efficiently using other methods. However in this case there is a much more efficient solution: naming the files by following Microsoft's advice. All of this time wasted on badly named files just takes you away from doing whatever data analysis that you are probably trying to do.
As Guillaume pointed out, using these badly named files will be unpredictable with any tool that you will try to use, so this is a pointless fight with no end.
Jan
on 17 Oct 2017
+1 It is the best way to provide properly formatted input files. Insisting on using R2015a and Windows 10 and trailing dots and spaces and dates in the far future and unicode file names is simply messed up. There are too many obstacles before the actual problem can be solved.
Fixing the Windows API is not an option, because it is not broken. The behavior is well defined and documented. The method which creates your data set should be fixed to consider the well known limitations.
Guillaume
on 17 Oct 2017
Files ending with dots or spaces are in the realm of undefined behaviour. The underlying file system allows them and parts of the Win32 API (and I repeat: only parts of it) but a great number of other APIs don't. As I said, the whole .Net API (at one point supposedly the future of Windows) does not.
So I'm not sure what's reliable about a program that creates files that are guaranteed to cause problems for the majority of other programs.
In any case, this is not a matlab issue. This is a Microsoft issue. And I don't think microsoft documents which of their functions work or not with files ending with dots and spaces. So all that can be done is use the syntax \\?\ to bypass the checks in the current API and pass the path unprocessed to the underlying API and hope for the best. In my view, a good way to introduce less reliability.
By the way I'm not sure how you can say that Microsoft is a supposedly authoritative source. They are authoritative. The page I linked (link fixed, an extra : got in there) is not mere advice. It is the rules you have to follow for a program to work properly on Windows.
@bbb_bbb: Reliable programs work on well formatted input. There is no coding style which can change the old rule:
Garbage in, garbage out.
There are many limitation in file names, such that e.g. backup systems will fail: At first the old limitation to 260 characters for the path and for the file name also. Using \\?\ you can expand the limit to 32,767 characters - many, but limited. Then special characters > < ? | : " \ / * and forbidden file names: CON, PRN, AUX, NUL, COM1, COM2, COM3, COM4, COM5, COM6, COM7, COM8, COM9, LPT1, LPT2, LPT3, LPT4, LPT5, LPT6, LPT7, LPT8, LPT9. Such names can come e.g. from extracting folders, which have been created under linux, which has less limitations. Backup systems are influenced by the underlying file systems, and I have seen bugs concerning the limits of fat16, fat32, NTFS, AFS and BTRFS. Sometimes hard or soft links let the system crash, sometimes the Alternate Data Streams of NTFS. Some programs crashed during the change of the day-light-saving time, or during the extra second.
Any software has a limited reliability only, as any mechanical tool has a limited applicability also. "Reliable" does not mean that you can process everything you want and always get what you expect, but it means, that the software works correctly inside the specified limits and with inputs inside the specifications.
Therefore the Windows API does work reliably, because it is well documented, that trailing dots are not handled. Your input files violate the specifications. Then asking for modifying the Windows API and the tools I offer for free is the wrong approach. Even if you would get the MD5 hashes, you cannot copy, move or process these files with standard software, e.g. the Windows Explorer or any backup tool.
If you now insist on using such file names, you act like using a drilling machine to beat a nail into the wall.
We have suggested several work-arounds and explained clearly, that the reliable way to solve the problem is not to process weird inputs by even more weird code, but to remove the source of the problem by fixing the program, which creates the input data. Your ironic question "do not you want to create reliable programs?" is ignorant. I think it is you, who does "not understand the true deepness of the problem".
I think, that the problem occurs in
% dirinfo(i).name - is string containing full pathname (e.g. 'C:\myfile.ext')
and
['\\?\' dirinfo(i).name]
already. I see no reason to assume, that Matlab or any of the other functions "does not understand bad names". Please post the complete error messages instead of the rough description 'ERROR - not working with "bad" names', 'ERROR here - matlab fopen don't understand "bad" names'.
Do you obtain dirinfo by the dir command? Then try:
File = ['\\?\', fullfile(dirinfo(i).folder, dirinfo(i).name)];
exist(File, 'file')
disp(['\\?\' dirinfo(i).name])
exist(['\\?\' dirinfo(i).name], 'file')
What do you get as output?
[EDITED, Walter is right: The Windows Command shell suffers from trailing dots and spaces.]
18 Comments
Walter Roberson
on 16 Oct 2017
"Names with trailing dots or spaces are not "bad" in any way at all."
Microsoft says not to use files whose name ends with dots, that individual file systems might support them but that the command shells do not.
bbb_bbb
on 16 Oct 2017
Jan
on 16 Oct 2017
Thanks Walter, you are right. The Windows Explorer cannot create such files or folder, but it works in a command shell:
mkdir \\?\C:\Users\Jan\asd.
A folder with the name 'asd.' is created, but you cannot create such a name in the Windows Explorer or enter this folder there.
@bbb_bbb: What about renaming the files?
Walter Roberson
on 16 Oct 2017
Which MATLAB version are you using? The change to include the field named "folder" was about two releases ago.
Walter Roberson
on 16 Oct 2017
Is there are particular reason the use \\?\ for this?
Walter Roberson
on 16 Oct 2017
However, you could copy them one by one to a different known-good file name, such as in tempdir(), and MD5 that name. Furthermore you appear to have a way to detect that the failure will occur, so you could handle those ones specially.
Sorry bbb_bbb, I'm not asking for fun, but I try to help you. Are you really sure, that the output is "\\?\C:\mypath\wrongname."? Then you are using strange folder and file names beside the trailing dot. I assume, the output is created manually and you use "wrongname." as a place holder. Please don't do this, because this can shadow the information, which actually contains the problem.
I asked for the error messages also. Please answer such questions to make it as easy as possible to help you.
I do not have problems under R2009a and Win7:
file = fullfile('\\?\', tempdir, 'asd.');
fid = fopen(file, 'w');
if fid == -1, error('Cannot open file: %s', file); end
fwrite(fid, 'hello', 'char');
fclose(fid);
Opt.Input = 'file';
DataHash(file, Opt)
ans =
5d41402abc4b2a76b9719d911017c592
Does this work with your R2015a/Win10 also?
I do not see the problem with renaming the 100 bad file names. This is done with a few lines of code and solved in a minute. movefile can rename such files:
movefile(file, [file, 'txt']);
You have files with a date of 27 July 2079? This is a really strange data set.
Perhaps the file names do not end with a dot in opposite to your assumptions. How did you get the list of file names? Could they contain non-printable unicode characters? The output is at least suspicious:
File = ['\\?\', dirinfo(i).name]
exist(File, 'file')
\\?\C:\mypath\wrongname.
ans = 0
This means that there is no such file. Check for unicode characters by converting the characters to double:
disp(dirinfo(i).name)
double(dirinfo(i).name)
Please with the real output, not some created pseudo-data.
This code [...] actually creates file named 'asd' , not 'asd.
No, not on my R2009b/Windows 7 computer: Here the file "asd." is created exactly as expected - with the dot. I will try it later on a R2016b system.
As far as I understand, on your R2015a/Win10 system a file called "asd" is created. Did you check this carefully? Do not trust the display of the Windows Explorer, which is obviously flawed in this point.
After the successful creation, fopen() can open the files also, as well as GetMD5.c and DataHash can process them directly.
E.g. L = dir(fullfile(tempdir, 'asd*')) finds the file, replies the correct name 'asd.', but the wrong size 0 Bytes and empty datenum. But
L = dir(fullfile('\\?\', tempdir, 'asd*'))
works correctly.
In the Windows Command Shell this works also:
mkdir \\?\C:\Temp\folder.
I do see the name in the Windows Explorer with a dot also and can access the folder from Matlab.
On my computer there is no "true deepness of the problem", but no problem at all. I still do not see any evidence, that dirinfo(i).name contains the correct names include the path and the \\?\ tag. You do not answer my corresponding question, what this replies:
disp(dirinfo(i).name)
double(dirinfo(i).name)
It would explain all your observations directly, if this is not the correct file name. When you do not post, how you have created it and its contents, the "assumption for simplicity", that might be the actual and only problem. Maybe there is another, but what's wrong with excluding this?! It would help to find out the different between out systems, which let the code fail on your computer, but not on mine.
Why do you assume, that dirinfo(i).name contains the correct file names?
You have 4 options:
- Either your code contains another bug and you find and fix it
- Or there is really a problem with processing the files and you can find and fix it
- Or you rename the files and process them normally
- or you give up.
Equivalently for the file dates: Either fix the source of the strange dates, or fix the dates by a small tool (FEX->Get/SetFileTime), or use strange .NET tools, which are currently suspected to provide wrong file names.
And what to do with files which have similar names
('asd', 'asd.', 'asd ')?
You did not provide enough information to answer this. So it is your decision.
If you ignore my questions for clarifications, I cannot help you further with narrowing down the source of the problem.
Good luck.
bbb_bbb
on 16 Oct 2017
Matlab 2009a/Win 7 creates "asd.", Matlab 2016b/Win7 creates "asd". So downgrading would be an option, but Win10 is supported since 2015a only, see https://www.mathworks.com/matlabcentral/answers/223444-is-matlab-compatible-with-windows-10.
movefile('asd.', 'asd') fails in R2016b:
Error using movefile
The filename, directory name, or volume label syntax is
incorrect.
My conclusion: Rename the files:
Folder = '\\?\C:\myfolder\'
FileList = dir(fullfile(Folder, '*.*'));
FileName = {FileList.name};
NewName = regexprep(FileName, '\.$', '_dot_');
NewName = regexprep(NewName, '\s$', '_space_');
toRename = find(~strcmp(FileName, NewName));
for k = toRename
FileRename(fullfile(Folder, FileName{k}), ...
fullfile(Folder, NewName{k}));
end
Now trailing dots are replaced by '_dot_', and trailing white space characters by '_space_'. Adjust this like you want.
[EDITED], GetMD5.c from the FileExchange is updated and you find a pre-compiled MEX file also now. It processes '\\?\C:\myfolder\asd.' directly.
27 July 2079, trailing dots and spaces, Unicode characters - this seems to be a never ending story with a deeply disturbed data set. I will look, if I can find a reliable way to use unicode file names in GetMD5.c, but it looks more and more like fiddling. I recommend to follow Guillaume's advice.
Thank you for repairing GetMD5. It now processes files with trailing
dots well.
This function did not had any problems with the file names, but the compilation failed, because you used the weak LCC32 compiler.
file = '\\?\C:\1\Chuck Berry - Fresh Berry - 1965\A.jpg';
tmp = fullfile(tempdir, 'tempfile');
FileRename(file, tmp); % Handles Unicode and trailing dots
Hash = GetMD5(tmp, 'file');
FileRename(tmp, file);
But then you might come with the next restriction: Renaming of files work on the same disk only. And the disk or file cannot be write protected, such that it does not work on a DVD. Or if the file name has more than 32767 characters. Or contains a \0 character.
You cannot process garbage reliably.
Walter Roberson
on 19 Oct 2017
It is not yet clear to me that copyfile() to tempname() is not an option for files detected to be a problem based upon exist() .
bbb_bbb
on 19 Oct 2017
'It violates the rule of good programming: "don't change original data unless necessary."'
No it doesn't, because the rule "fix bugs where they occur" exactly makes this change "necessary". Your filenames are outside of those specified to work correctly with Windows. Solution: change them so that they are suitable for Windows. Not only that, but filenames should not contain data at all, or at most only some very high-level meta-data, so changing the name should make no difference to your data.
"Renaming is not quite good decision"
Ensuring that the names are created correctly in the first place would be the best decision. Anything else will ultimately just waste more of your time. Like this discussion already proves.
"because it is very time-consumpting, e.g. if there are many files in a "bad" folder or files themselves are big-sized."
Nope, it would only take a few minutes with the right tool. I do this all the time with quite large files from our test department, to store the files systematically. It doesn't change the data at all.
It is not clear to me what the problem is. Why are you letting something as trivial as filenames get in the way of doing your work?
Categories
Find more on Data Import and Export in Help Center and File Exchange
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!
