Jump to content

How to purge deleted entries from the MFT.


Alan_B

Recommended Posts

Wipe Free space is NOT the answer.

 

A Macrium Reflect "Intelligent Copy" partition Image captures for example10,000 files in used space,

and exclude empty space which could for example have 100,000 deleted files,

BUT

it includes the MFT which remembers 10,000 files PLUS 100,000 deleted files.

 

When I restore that image backup to a clean empty partition,

then I have 10,000 files, but the MFT has 10,000 file real names and is bloated with another 100,000 "deleted" files which CANNOT be recovered.

 

A file gets lost deleted and I wish to recover so I try Data Recovery tools to search for it ;

PROBLEM :-

Recuva (or anything else) finds 100,000 "deleted" files in the MFT

That is a might big haystack to search for one needle.

 

This is why I would like to purge deleted entries from the MFT before I ever accidentally delete something.

The best from CCleaner Wipe is that each deleted MFT entry is changed to a weird zz.zz.z.z..zzz.z sequence,

which is hardly helpful if the file I lost should also start "zzz..."

 

Regards

Alan

 

 

Link to comment
Share on other sites

Alan, are you asking how to purge the MFT of non-used entries?

 

If you're looking to clear out the MFT you need to compact and compress it back to what it would be like when a new disk is formatted, but of course keeping the existing valid-in-use entries. The MFT only grows, NTFS/Windows never shrinks it on its own.

 

When you delete a file you mark a spot (in the mft) as being open) and when you create a new file, that spot (or others nearby) can be used to store the new location information. In the meantime, in that spot rests the previous filename of what you just deleted, and some of the location data!

 

Let's say you format a disk fresh. It now has only 10 MFT entries (an arbitrarily low number I made up). And as you add your music collection it goes to 5,000 entries. Then 23,456 entries with your photo collection. And up and up. If you decide to delete your music collection the count still remains at 23,456. Those 5,000 music spots would now get filled with CCleaner's zzz.z.z..zzz.zz files.

 

And since they were made and deleted, the zzz.z.z..zzz.zz entries are remant spots earmarked for re-use when you start creating files again. What used to be validfilename-1.mp3 validfilename-2.mp3 and so on and so forth are now zzz.z.z..zzz.zz. (after drive wipe).

 

What you need to do is remove any MFT entries that are un-used, by way of concatenating the branches of the MFT's B+ tree's structure and reconnecting the structure together over the open spots - those zzz.z.z..zzz.zz entries otherwise known as recently deleted entries. And as a result you have no filler space where stuff is hiding. And in the process you're shrinking the MFT down to size as a natural result.

 

The plain english analogy is compacting the dead air space out of something until only real substance remains. When CCleaner's drive wipe is used, it only fills the holes with unidentifiable grey "stuff". It does nothing to remove the holes and old content.

 

I hope that makes sense to you?

Link to comment
Share on other sites

:) Please don't tease me :)

 

Thanks - you know exactly what I need to know  - now tell me how to do it please. :wub:

 

I have just searched for "Compact MFT" and found two topics requesting this without success on the Mydefrag forum,

and also found that $49-95 may buy this from Paragon.

 

I am hoping for a free solution.

 

Regards

Alan

 

 

 

 

Link to comment
Share on other sites

Hello Alan - I did a Google search using the phrase "purge entries for deleted files from MFT" and came up with three possibilities for you to consider. 

 

1.) BCWipe by Jetico, $39.95 USD. Go to this link:https://www.jetico.com/products/personal-privacy/bcwipe  Click the "Features" tab, then scroll down. I think this describes what you want to do:

 

Wipe MFT Records and Directory Entries

Prevent recovery when you delete files. The file system records the names and attributes of files to a special area of your disk drive (so called 'directory entries' for FAT and MFT for NTFS). When a file is deleted, the corresponding directory entry is modified by the file system which makes it invisible to Windows and to you. However, most of the information still exists and the name and attributes can be restored using any recovery utility. BCWipe shreds directory entries and MFT so that the information can never be recovered.

 

2.) Directory Snoop by Briggs Software, also $39.95 USD. Here is the link: https://www.briggsoft.com/dsnoop.htm

Scroll down the page and find the Flash Demo for "NTFS Filename Purge", then watch the demo. It appears that this software can be used to filter then delete MFT entries based on user defined parameters, so caution is needed. As long as you get the filters right, everything is fine. Get it wrong and you wreck the MFT beyond repair. Your call if you want to consider using this one. 

 

3.) Eraser by Joel Low, freeware. Here is the link to the homepage: http://eraser.heidi.ie/  And here is the download page: http://eraser.heidi.ie/download.php

I'm not too sure about this one being able to do what you want. I downloaded it but haven't had the time to install it and try it out. I included it based on the developer's statement found in this link: http://eraser.heidi.ie/forum/viewtopic.php?f=2&t=8864  Look for Joel's reply:

 

"Eraser's erase features are non-destructive for data you want to keep. So although we do clean the MFT, we only erase entries which already are deleted. MFT erasure comes as part of doing an unused space erase. There's no option to turn it off or on at this point."

Start every day with a smile and get it over with. - W.C. Fields

Link to comment
Share on other sites

Thanks for the information.

 

I only knew of Eraser as a complete disk eraser and had not realized it could erase free space.

 

I will clone a well used partition onto a VHD and try Eraser on that, and post back.

 

Many thanks

Alan

Link to comment
Share on other sites

  • Moderators

If I have read Alan's post correctly then he wants the live records in the MFT grouped consecutively, thus removing all the free records from the MFT. I don't think that any of the above software does this, or does any more than CC's Wipe MFT.

 

The records in the MFT are numbered in ascending sequence, and this number is used to access the records, so you can see the problem with compaction. I've always thought that the complexity of the MFT would prevent compaction, but Paragon apparently has some success with its File Manager (I think that's what it's called). Considering the links between MFT records, extension records, external index clusters, referbacks to folders, bitmaps etc I wouldn't touch any software that promised to compact the MFT with a barge pole. Not unless I had a pretty good explanation of how it works.

Link to comment
Share on other sites

  • Moderators

I will clone a well used partition onto a VHD and try Eraser on that, and post back.

 

There's Eraser Portable, that way if you don't like it there's no installation. Remember that Eraser isn't released by the original developer anymore and back then when he did manage it people really loved the tool, I personally won't use it anymore.

Edited by Andavari
Link to comment
Share on other sites

@Augeas

I do not care about how the MFT is grouped or modified.

 

My concern is that if I lose a file in the 10 GB partition D:\,

Recuva will search and find a total of 15803 files, of which only 1280 will be ignored because they have not been deleted.

 

I periodically backup the entire D:\., after which I might create and then delete 140 more files.

If a file is lost (or should not have been deleted) then Recuva may find it somewhere in the middle of 14,000 other deleted files.

It would be much easier and quicker to find it if there were only 140 other deleted files.

By purging the deleted file names from the MFT after creating a backup the 14,000 old deleted files would no longer appear.

 

CCleaner Erase Free Space etc may solve the problem,

unless the required file is named "zebedee" in which case it would appear somewhere in the middle of 14,000 anonymized z.zz..z.z

 

@Andavari

Thanks, I do prefer Portable where available.

 

Regards

Alan

 

 

 

 

 

Link to comment
Share on other sites

  • Moderators

@Augeas

I do not care about how the MFT is grouped or modified.

 

Well, that's your prerogative. I think that if I were to download a free application written by somebody I've never heard of that made structural changes to the most critical metafile in an undocumented and proprietary file system, I would care. It's good that you are fond of backups.

 

Some of the mentioned software might serve your purpose, in as much that an MFT 'clean' would overwrite deleted records with zero length files (CC uses around 600-byte files). Then Recuva, with Show Zero Length Files unchecked, would not display these files. Doesn't compact the MFT though. I don't know which of the mentioned software does the zero byte overwrite.

Link to comment
Share on other sites

Thanks for all the replies.

 

I prepared a VHD for testing Eraser Portable v5.8.8.1

 

I ticked the top option "Unused space on drive" and browse to K-130MB-K (K:)
 

@Augeas

Thanks for the warning.

I understand that Defraggler etc. does its "magic" by using Microsoft publish API's,

so if Microsoft do not publish MFT API's or even information about MFT structure then I share your concern.

 

Regardless of the danger to a zero cost VHD drive K:\ I have decided to honor my promise to test Erase and post back the result.

 

I downloaded and used Eraser Portable 5.8.8.1.

 

I clicked the green RUN Arrow and am told I am about to erase all selected data,
AND AM WARNED THAT FILES CANNOT BE RECOVERED AFTER ERASING.

 

That warning proves that Eraser is defective and I cannot trust it.

Either it has given me a spurious and irrelevant warning,

or it is about to delete more than unused space,

AND IT MIGHT ALSO MISUNDERSTAND MY INTENDED TARGET DRIVE.

 

I am happy for Eraser to destroy my test VHD,

but decided to spare System C:\ from any possibility of "collateral damage by Friendly Fire".

 

I will therefore give up hoping for an ideal purge of the MFT.

 

Regards

Alan

 

Link to comment
Share on other sites

I have a solution that fits my present needs - Freeware FastCopy.

http://ipmsg.org/tools/fastcopy.html.en

 

FastCopy runs under Windows and is able to copy everything I want from one drive to another, but excludes the wretched $MFT and other $MetaData files,

and as each file is copied across,

the destination $MFT is updated by Windows with the relevant information.

 

It even correctly duplicates Reparse Points,

unlike Freeware Portable Teracopy v2.27 which mistakenly copies the contents of the Reparse Point destination.

 

Other benefits of FastCopy over Teracopy are that when the duplicated file is read back for hash check-sum verification against the source,

it actually reads back the file that was written to the destination and NOT the intermediate copy to RAM which Windows used as a write cache.

 

I am disappointed that this is not an in-situ solution that simply re-writes a small $MFT but requires the writing of all the contents of one partition to a new partition,

but thankful that I am not having to buy another 1000 GB HDD to temporarily hold this new partition.

All I am afflicted with is a 10 GB partition with thousands of real files and the ghostly remnants of a gazillion browser cache files which Firefox accumulated and deleted.

 

Regards

Alan

 

Link to comment
Share on other sites

Hello Alan_B - I'm glad to see you found something that works for your situation. I tried out Eraser on my Windows 8.1 system by creating a small 20GB partition, loading 50 files onto it, doing a wipe, deleting all but one of the files, then doing a second wipe. Using Recuva, I could not find any of the files afterwards, so I can only assume it works.

 

Which leads me to these questions: How do you go about examining the MFT directly? Is there some software utility that allows you to do this? If there is, does it display the information using a table or graphic of some sort that's easy to understand? Or does it spit out a bunch of hexadecimal gobbledygook that's beyond my understanding? I'm just curious about this, that's all.    

Start every day with a smile and get it over with. - W.C. Fields

Link to comment
Share on other sites

WizTree shows the names and sizes of all $Metadata files including $MFT and $MFTMirr.

 

If you check the Defraggker Search TAB option "Include no-fragmented files" and "Filename contains:" and specify "$MFT"

You will be shown the names and sizes of both $MFT and $MFTMirr.,

and if you select one or both then their positions in the file clusters will be shown.

 

When I select the square in front of the $MFT

System.ServiceModel.ni.dll    1    23913984    C:\Windows\assembly\NativeImages_v2.0.50727_64\System.ServiceModel\0b51b0626d95de7446d132c73edd77cc\    
 

There is freeware available for Hexadecimal displaying and even editing the contents of file clusters if you stipulate the LBN address.

It may also be able to find the start address if you give it a file name.

If it cannot accept and search for the very special and generally inaccessible name $MFT,

it may be able to accept and search for whatever file precedes it, such as System.ServiceModel.ni.dll,

and this will give you a starting point.

 

N.B.  I do not recommend that you alter the content of C:\$MFT, :wacko:

But would really enjoy hearing about the consequences if you attempt that :D :D

Link to comment
Share on other sites

  • Moderators

Good old Recuva will show the size and cluster allocations of all files, if you have Show undeleted files checked.

 

I use WinHex and the slightly more user friendly HxDen to poke around in the $MFT and other files. Don't forget little-endian numerical values. There is a utility from Sysinternals that produces a csv list of the MFT's contents, but I wouldn't say that it was very helpful.

 

If you read to the end of the Eraser forum link above, the more revealing quote from the developer is:

 

'It took me around 1 week to implement the directory entry cleaning for FAT since the specifications were rather readily available. In the same amount of time I did not even implement proper parsing of the NTFS MFT... which would give a hint on what scale NTFS is on.'

 

I think that puts it in context.

 

You can edit the MFT with gay abandon using a hex editor, but NFTS will back out your edits a few seconds later.

Link to comment
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
×
×
  • Create New...

Important Information

By using this site, you agree to our Terms of Use.