15 replies to this topic

[Francis Favorini]

Folks,

I deleted a bunch of files (about 14.8 million). Turns out I need about 700 thousand of them back. They are all in a single directory tree. Nothing has been written to the disk since. Scanning the MFT should find everything. The problem I am having is that no software I've tried so far can deal with that many deleted files without running out of memory or hanging, including Recuva.

System has RAID array with 2.75 TB, dual Xeon 3.6 GHz, 3 GB RAM. Recuva went all the way through to the end of Stage 1, reporting 14,840,569 files, 100%, 0 seconds left. Then it just sat there. Left it overnight, no change. It was using about 2.8 GB of RAM and occasionally a 1% of CPU. Pagefile was up to 22 GB. Finally had to kill the process. Maddening.

Is there some way I can tell Recuva only to process files in the directory tree that I care about (say F:\data\important\stuff) during Stage 1 (and 2)? Would it actually have finished if I had left it? Any way to tell what it was doing?

Any other programs out there that deal with memory in a better way?

Thanks,
Fran

[marmite]

I should imagine most programs struggle with those sorts of numbers

The easier way to tackle it might seem to be to try and do it in smaller chunks, as you've already suggested. Though the problem I assume that you've encountered with the folder scan is that you might at first think that "one folder" seems a smaller place to search ... but files in that folder could be all over the place so you still have to initially scan the whole partition in order to filter out matches on "your folder".

Did you actually try that though? Since I assume it would "filter as it went along" you might still only get a relatively small result set. Unless of course by "single directory tree" you are also saying "single folder" ... in which case the whole folder argument goes out of the window as a way of splitting the files into chucks ..... and you need to try something else .....

These guys also do various bits of recovery software. I've used their stuff and their Active@ File Recovery can work by specifying various attributes; file sizes / dates (created, modified, accessed, deleted) and filename extensions/wildcards. So you could try that for piecemeal recovery too - not everything at once. Again, it will always mean scanning a whole partition though. Their stuff isn't free like Recuva, but you could run the scan on the trial version and at least see if it finished.

[Francis Favorini]

Note, I am not doing a deep scan, so Recuva should only be looking in the MFT for entries marked deleted, right? If so, there should (at least in theory) be a way to specify a path instead of just a drive to return results for. The MFT still includes the names of all deleted files and directories, as I understand it. Apparently, Recuva is storing some bits of info for each file it finds in the MFT marked as deleted. That's the problem, since there are 14 million of them. If it would only store that info for the files in the path (i.e., the whole directory subtree) that I want, it should have enough memory ("only" 700K files). Filtering the list of files after it is built is no good, since it takes too much memory (apparently) to build the whole list.

I thought of trying to restore some files, even those I don't care about, to reduce the total number of deleted files. I am worried that this might cause some blocks on the disk to get written to, however. I suppose in theory, all it should have to do is unset the deleted bit in the MFT, but I'm not exactly sure how this works. If I instead restore the (unwanted) files to a different disk, it won't help, since they will still be marked deleted on the original disk. What would help is if there were some way to ignore those unwanted files.

Also, there is no way using attributes, dates, or filenames to get just the files I want. The only way is to specify the base directory (and yes, I want the whole subtree, not just one directory).

-Fran

[Augeas]

Fran, you might want to try cancelling Recuva stage 2. I always do this and it has no discernable effect on the results. The list is still produced and files can be recovered. Just chop it as soon as stage 1 has completed. (Your list is of course way out of my league.)

Recovering files creates a new file, so the old deleted entry in the MFT still remains (the new file uses a 'random' deleted MFT entry). The new file will overwrite something if placed on the same disk - hence the warning message. I think it's safe to say that you must recover (if you get to that happy state) to another volume.

Although you can filter the initial scan, as far as I know filtering just limits the display, the full scan is always done whatever the filter says, so you won't save much if any memory by filtering. Selecting a particular disk, if you can, would of course ease the problem.

[marmite]

Francis Favorini, on Sep 19 2009, 01:54 AM, said:

Also, there is no way using attributes, dates, or filenames to get just the files I want. The only way is to specify the base directory (and yes, I want the whole subtree, not just one directory).

So the files you want are across multiple folders (directories) then - one folder plus sub-folders. I take it from what you've said that you don't know the individual folder (directory) names - in order to do a bit at a time?

Augeas that's an interesting little gem about cancelling stage 2. I've just been reading the Recuva documentation and Stage 2 only assesses "recoverability potential" - superfluous for the OP since the files are known to be intact. So that would save a helluva lot of time!

[[ Going off on a complete tangent for a sec (and not aimed at you Fran) the Recuva "Technical Information" section should be compulsory reading for anyone on these forums with deletion / recovery problems ... very misunderstood area! ]]

Francis Favorini, on Sep 19 2009, 01:54 AM, said:

... there should (at least in theory) be a way to specify a path instead of just a drive to return results for.

Sorry Fran just to confirm; you have tried the path option in Recuva? I had originally assumed the answer to that question was "yes, but you wanted to limit Stage 1 and 2 as well as the results".

I did a test recovering a single folder containing 340,000 deleted files - though obviously I couldn't simulate your 15,000,000 total deletions. Cancelling Recuva after Stage 1 caused no issues and the file list displayed without problem. I also started the recovery off and it didn't seem to have any problem handling the list size (it would have taken longer than I was prepared to wait for it to finish so I cancelled it). But I'm thinking that if you can identify the right folder(s) then you may be okay.

Incidentally "Active@ File Recovery" has a tree view pane like Windows Explorer, so if you have forgotten or don't know what the folder names are, that might be a useful way of identifying the folder names to feed into Recuva.

And to the Recuva devs ... how a tree view ?

Augeas, on Sep 19 2009, 08:17 AM, said:

I think it's safe to say that you must recover (if you get to that happy state) to another volume.

Absolutely wot Augeas says!

As an aside, is your RAID a stripe or a mirror? If it was a mirror I was wondering out of curiosity whether you were going to take one disk out as a "back up".

[Augeas]

marmite, on Sep 19 2009, 10:19 AM, said:

And to the Recuva devs ... how a tree view ?

Options/View Mode/Tree View.

[marmite]

Augeas said:

Options/View Mode/Tree View.

Ah!!! Thank you

[Francis Favorini]

Augeas, on Sep 19 2009, 03:17 AM, said:

Fran, you might want to try cancelling Recuva stage 2. I always do this and it has no discernable effect on the results. The list is still produced and files can be recovered. Just chop it as soon as stage 1 has completed. (Your list is of course way out of my league.)

Tried cancelling when it said Stage 1, 100% complete, but it just sat there and did nothing. Had to kill the process. Perhaps I should've waited longer than 15 minutes. It seems that it's using the page file heavily at this point, so it's very slow. Does anyone know what Recuva's actually doing at this point (before Stage 2 starts)?

Augeas, on Sep 19 2009, 03:17 AM, said:

Recovering files creates a new file, so the old deleted entry in the MFT still remains (the new file uses a 'random' deleted MFT entry). The new file will overwrite something if placed on the same disk - hence the warning message. I think it's safe to say that you must recover (if you get to that happy state) to another volume.

I figured as much.

Augeas, on Sep 19 2009, 03:17 AM, said:

Although you can filter the initial scan, as far as I know filtering just limits the display, the full scan is always done whatever the filter says, so you won't save much if any memory by filtering. Selecting a particular disk, if you can, would of course ease the problem.

Yeah, I have selected the correct disk. I just wish it would not save the info for the directories I don't care about. That would make the memory issue go away. So close...

[Francis Favorini]

marmite, on Sep 19 2009, 05:19 AM, said:

So the files you want are across multiple folders (directories) then - one folder plus sub-folders. I take it from what you've said that you don't know the individual folder (directory) names - in order to do a bit at a time?

Yes, one folder including all of its subfolders. The subfolders do not have unique names across the filesystem. It doesn't seem there is any way to restore things a bit at a time anyhow. If I undelete any particular file, it will be restored to another drive, leaving the original drive in exactly the same state. Hence, a subsequent scan will have exactly the same memory issues.

marmite, on Sep 19 2009, 05:19 AM, said:

Augeas that's an interesting little gem about cancelling stage 2. I've just been reading the Recuva documentation and Stage 2 only assesses "recoverability potential" - superfluous for the OP since the files are known to be intact. So that would save a helluva lot of time!

Yes, I would be happy to cancel Stage 2, but it never gets there. It just stays at the end of Stage 1. I suppose I could let it run for a few days and see what happens.

marmite, on Sep 19 2009, 05:19 AM, said:

Sorry Fran just to confirm; you have tried the path option in Recuva? I had originally assumed the answer to that question was "yes, but you wanted to limit Stage 1 and 2 as well as the results".

Yeah, I put in the path to the files I want. Didn't seem to change the memory usage or hanging at the end of Stage 1. (When I say hang, it may be that it's just really, really slow because of the huge amount of virtual memory it's using.)

marmite, on Sep 19 2009, 05:19 AM, said:

I did a test recovering a single folder containing 340,000 deleted files - though obviously I couldn't simulate your 15,000,000 total deletions. Cancelling Recuva after Stage 1 caused no issues and the file list displayed without problem.

When you clicked Cancel, did the dialog box say it was on Stage 2 or Stage 1 with 100% complete? Mine stays on Stage 1 with 100% complete. It may be trying to sort the list or something, which is taking forever.

marmite, on Sep 19 2009, 05:19 AM, said:

As an aside, is your RAID a stripe or a mirror? If it was a mirror I was wondering out of curiosity whether you were going to take one disk out as a "back up".

It's RAID 5.

Thanks for any help you can think of.

[Francis Favorini]

So I've been running another program (PC Inspector File Recovery) all weekend, and it has only used 400K of RAM, but it's only about 37% done with the scan. It says "Retrieving existing files and directories", which makes me worry a bit that it isn't scanning deleted files yet. Not sure if I want to wait for this to finish or try something else (like Active@ File Recovery). I suppose I could run Active@ scan while the other one is still scanning, since both should only be reading from the disk.

[Francis Favorini]

For completeness, here are the other programs I've tried:

EASEUS Deleted File Recovery
Free Undelete
Glary Undelete
Pandora
PCInspector File Recovery
Undelete & Unerase RecoverFiles
Restoration
TestDisk & PhotoRec
UndeletePlus
Uneraser

They all either crashed at some point, complained the disk was corrupt (it isn't), or were taking so long it didn't seem likely they would work.

[Augeas]

For info, Stage 2 has to be running to be cancelled, but it doesn't look as if this will ever happen anyway.

As I assume from the system size and data volume the data must have some value. Have you contacted a data recovery company for advice? I think that they would be best equipped to recover this amount of data. It will cost, but we don't know the cost of not recovering it. It doesn't really look as if a self-recovery is going to work, and your disk(s), or the data on them, is possibly deteriorating as each day passes.

[marmite]

Francis Favorini, on Sep 21 2009, 04:43 PM, said:

When you clicked Cancel, did the dialog box say it was on Stage 2 or Stage 1 with 100% complete?

It had finished Stage one and was on Stage 2. Looks like it's the other 14+ million that are blowing it!

Maybe Augeas is right - time to consider specialist assistance? It ain't gonna come cheap is it.

At least you know what you're doing now isn't harming the data - so you can exhaust reasonable possibilities first.

If you do decide to try Active@, the time for a one-pass "quick scan" (which should be sufficient for files that aren't overwritten) extrapolated from the number of files on my system partition should give you a scan time of 90-odd minutes (probably slower than Recuva, before it hung?). I'm sure it's not going to be as simple as that, especially based on your experiences so far; but because it's one-pass you should be able to gauge progress. The downside of course is that you can't recover files of any size with the trial version; so if you went ahead and you got grief at the recovery stage you'd be stung there.

[MrRon]

We've tested Recuva with up to 10million files and it worked fine, although it was a little slow. But this was on a fairly high spec test machine. There's no real reason why it would not handle the extra 4 million if the spec was up to it.
We will re-test and report back.

MrRon

[Francis Favorini]

MrRon, on Sep 28 2009, 03:06 AM, said:

We've tested Recuva with up to 10million files and it worked fine, although it was a little slow. But this was on a fairly high spec test machine. There's no real reason why it would not handle the extra 4 million if the spec was up to it.
We will re-test and report back.

OK, cool. Specs for the machine in question are dual Xeon 3.6 GHz, 3 GB RAM. OS is 64-bit Windows 2003 R2 SP2. If you can tell me how much RAM I need, I will try to steal some from other machines and see if it works.

Also, if there were a way to set a filter so that resources are not kept around for files that don't match the filter, that would reduce the amount of memory needed. In my can, all of the files start with a known path (E:\Data\Some\Dir\). Most of the deleted files are in other directory trees, so storing their info is not necessary. The above discussion goes into more detail. Let me know if this is not clear.

Thanks,
Fran

[mr don]

Francis Favorini, on Sep 18 2009, 09:17 PM, said:

Folks,

I deleted a bunch of files (about 14.8 million). Turns out I need about 700 thousand of them back. They are all in a single directory tree. Nothing has been written to the disk since. Scanning the MFT should find everything. The problem I am having is that no software I've tried so far can deal with that many deleted files without running out of memory or hanging, including Recuva.

System has RAID array with 2.75 TB, dual Xeon 3.6 GHz, 3 GB RAM. Recuva went all the way through to the end of Stage 1, reporting 14,840,569 files, 100%, 0 seconds left. Then it just sat there. Left it overnight, no change. It was using about 2.8 GB of RAM and occasionally a 1% of CPU. Pagefile was up to 22 GB. Finally had to kill the process. Maddening.

Is there some way I can tell Recuva only to process files in the directory tree that I care about (say F:\data\important\stuff) during Stage 1 (and 2)? Would it actually have finished if I had left it? Any way to tell what it was doing?

Any other programs out there that deal with memory in a better way?

Thanks,
Fran

Handy Recovery undeletes files, including those from password protected user accounts on XP & Vista. It is payware though, but if you have the full version, it works really well. It structures the results exactly the same as you would have seen the folder trees if they had not been deleted.

You can restore using folder structure on there as well as preview pics before recovery. In addition, you can drill down & right click a folder to recover & only select that folder if you wish. Or, select multiple folders. It can also scan for deleted partitions & try to find those as well.

It is one of the best programs I have tried so far, although Recuva does keep improving. It would be nice to see Recuva implement folder recovery in the future, as well as partition recovery, or at least the option to scan for deleted partitions, then recover files from deleted partitions, even if it did not restore them!

http://www.handyrecovery.com/ is the website for it. If you want to buy it, you will not regret it, I assure you.