high CPU, low IO, slow defrag

[spongman]

I have a large drive (1.5TB) with lots of small files on it. Defraggler is using a lot of CPU (effectively pegging one core), and it's taking over 1 second to defrag each file.

 

I tried running the profiler on it, but since you guys stripped the symbols (why?), I can't give you anything interesting to go on... It seems to be spending a lot of time in EnterCriticlSection...

[redhawk]

Slow file transfer with high CPU usage / interrupts could indicate the IDE controller is operating in PIO mode (not DMA) or the hard drive is pending failure.

 

Check your drive's SMART status using Speccy (or HD Tune) and see if anything looks suspicious i.e. Reallocated Sector Count > 0.

 

Richard S.

[spongman]

Slow file transfer with high CPU usage / interrupts could indicate the IDE controller is operating in PIO mode (not DMA) or the hard drive is pending failure.

 

Check your drive's SMART status using Speccy (or HD Tune) and see if anything looks suspicious i.e. Reallocated Sector Count > 0.

 

Richard S.

 

nope, the drive is definitely operating correctly.

 

i ran procmon.exe and it seems that one of defraggler's threads that is pegging the CPU is doing a GET_VOLUME_BITMAP after every file. i think this is related...

 

i'm going to run it under xperf to see if i can get anything more useful.

 

ok, i can't get anything helpful out of xPerf due to lack of symbols, care to share the PDB?

 

but it seems there are two threads which basically take turns pegging the core. almost all of the CPU time is spent inside the defraggler64.exe code, ie. it's not in kernel or other DLLs.

[spongman]

but it seems there are two threads which basically take turns pegging the core. almost all of the CPU time is spent inside the defraggler64.exe code, ie. it's not in kernel or other DLLs.

 

ok, one thread seems to spend a considerable amount of time looping through the items in the 'file types' dialog. it looks like it's doing this in an extremely inefficient manner: the 'lock xadd' in the following loop is causing the CPU to purge cache & sync in a tight loop. It looks like this is getting called for every file on the disk, for every file type in the types dialog. is that right? it would seem that moving the lock to a higher level would be a good place to start.

 

(defraggler64.exe is loaded at 0x13f510000)

 

000000013F55C7D0  mov         rdx,qword ptr [rbx]  
000000013F55C7D3  sub         rdx,18h  
000000013F55C7D7  or          eax,0FFFFFFFFh  
000000013F55C7DA  lock xadd   dword ptr [rdx+10h],eax  
000000013F55C7DF  add         eax,0FFFFFFFFh  
000000013F55C7E2  test        eax,eax  
000000013F55C7E4  jg          000000013F55C7EF  
000000013F55C7E6  mov         rcx,qword ptr [rdx]  
000000013F55C7E9  mov         rax,qword ptr [rcx]  
000000013F55C7EC  call        qword ptr [rax+8]  
000000013F55C7EF  add         rbx,10h  
000000013F55C7F3  cmp         rbx,rsi  
000000013F55C7F6  jne         000000013F55C7D0  

 

the other thread that's pegging the CPU is also looping through this list of file types. although it's a different piece of code. but it's also got the dreaded lock in a tight loop:

 

 

000000013F63E819  jne         000000013F63E826  
000000013F63E81B  lea         r12,[rdi-18h]  
000000013F63E81F  lock add    dword ptr [rdi-8],1  
000000013F63E824  jmp         000000013F63E866  

[spongman]

ok, here's a repro.

 

on a large drive with a lot of small, fragmented files, the following uses approximately 0 (zero) CPU:

 

df64.exe e:

 

the following pegs 1 core:

 

df64.exe e: /Large .arj 0 /Large .tar 0 /Large .rar 0 /Large .zip 0 /Large .cab 0 /Large .tar.gz 0 /Large .tgz 0 /Large .arj 0 /Large .lzh 0 /Large .7z 0 /Large .gzip 0 /Large .gz 0 /Large .uue 0 /Large .bz2 0 /Large .img 0 /Large .mig 0 /Large .mp3 0 /Large .wav 0 /Large .aiff 0 /Large .rm 0 /Large .ra 0 /Large .flac 0 /Large .bin 0 /Large .iso 0 /Large .mdf 0 /Large .dmg 0 /Large .vxd 0 /Large .avi 0 /Large .mkv 0 /Large .mpg 0 /Large .mov 0 /Large .mp4 0 /Large .m4v 0 /Large .mpeg 0 /Large .wmv 0 /Large .bup 0 /Large .ifo 0 /Large .vob 0 /Large .msi 0 /Large .msp 0 /Large .msu 0 /Large .wim 0 

[Kuba]

ok, here's a repro.

 

on a large drive with a lot of small, fragmented files, the following uses approximately 0 (zero) CPU:

 

df64.exe e:

 

the following pegs 1 core:

 

df64.exe e: /Large .arj 0 /Large .tar 0 /Large .rar 0 /Large .zip 0 /Large .cab 0 /Large .tar.gz 0 /Large .tgz 0 /Large .arj 0 /Large .lzh 0 /Large .7z 0 /Large .gzip 0 /Large .gz 0 /Large .uue 0 /Large .bz2 0 /Large .img 0 /Large .mig 0 /Large .mp3 0 /Large .wav 0 /Large .aiff 0 /Large .rm 0 /Large .ra 0 /Large .flac 0 /Large .bin 0 /Large .iso 0 /Large .mdf 0 /Large .dmg 0 /Large .vxd 0 /Large .avi 0 /Large .mkv 0 /Large .mpg 0 /Large .mov 0 /Large .mp4 0 /Large .m4v 0 /Large .mpeg 0 /Large .wmv 0 /Large .bup 0 /Large .ifo 0 /Large .vob 0 /Large .msi 0 /Large .msp 0 /Large .msu 0 /Large .wim 0 

This guy should get free Defrag support for life Nailed it, I guess.

[romanoff]

Hi,

 

thanks for all information. We will look into this.

 

Best regards

Romanoff

[SteveE]

Having the "move large files to the end" also seems to make it more CPU intensive ("pegging" one core).

 

edit: or maybe not

[WiZARD]

I have the same problem, high cpu usage, slow defrag.

(it was fast for 11%, then it slowed down, and started using 100% cpu)

[spongman]

Hi,

 

thanks for all information. We will look into this.

 

Best regards

Romanoff

 

did this ever get fixed? i'm still seeing very high CPU consistently. I'm wondering if it's still due to all those InterlockedIncrements?

 

it looks like you're iterating over a vector<shared_ptr<>>, very bad... you need to optimize for the common case, not the rare one...

[Nergal]

Spongman as this is an old thread I just want to make sure that you are seeing this in the most recent version of defraggler

[spongman]

Spongman as this is an old thread I just want to make sure that you are seeing this in the most recent version of defraggler

yes, still seeing it.

 

please try the two commands I gave in post #5 and compare CPU usage...

[Mineralwasser]

I'm also having the same problem with the most recent release of defraggler - are there any news on this? Defraggler estimates "> 1 day" for defragmentation (on a 500 GB drive)....

[properjob70]

yep, this user is seeing unwarranted slowness since v 2.09.391 was released although the option to "quick defrag" seems to operate at more like the speed of the old release.

[Mineralwasser]

I am not sure if this happens since the 2.09 release only.

But this is a definitly a bug which should be fixed - i've never seen a defrag application which is cpu-bound instead of io-bound.

[spongman]

i'm STILL seeing this (see comment #5). i also see a bunch of high CPU comments on the forums, i'm pretty sure this remains the issue.

 

any word on a fix for this?

[inspector]

Sorry for wake up an old thread, but I have the same problem (high cpu usage, 1 core / really low defrag speed, few KB/sec or 1 file/1-3 sec). I'm using the latest version.

Any chance to fix it?

[Alan_B]

Sorry for wake up an old thread, but I have the same problem (high cpu usage, 1 core / really low defrag speed, few KB/sec or 1 file/1-3 sec). I'm using the latest version.

Any chance to fix it?

Many people do not have a problem.

 

If half a dozen people posted system specifications then common factors might become obvious for the developers to investigate, e.g. :-

Operating System;

x86 / x64;

System Drive / Secondary Drive / External drive;

Make of HDD.

 

Hint, Hint

[inspector]

Win 7 SP1 Ulti x64 (Full updated)

Intel Core 2 Quad @ 2.83GHz / 8GB DDR2 @ 1066MHz

Sys hdd : WD 1TB Caviar Blue / Sec hdd : 2x WD 640GB Caviar Blue @ Raid 0

 

The problem is system independent, at least hardware independent..

While defragging a partition I monitored it with HDTune and I saw something strange, after each file (from 10KB small to 1GB big) there is a 2-3 sec pause. While defragging small files ( <10MB) defraggler use 1 core of my cpu..

 

Why 3 sec pause after each file? Why use 1 cpu core while defragging small files ( <10MB) ?

Definitely there is a big problem in program's algorythm.. Please fix it, it is a good program.

 

Edit: I forgot the images

 

 

[mta]

are you running the same commands as in post #5 ?

does the fact you are running HDTune while defragging skew the result ?

what else is running ?

could your RAID 0 be adding an overhead as the data blocks span 2 drives ?

does the same issue happen in Safe Mode ?

 

for testing purposes, have you created another partition, copied some files across and ran DF on that, or on another PC for that matter ?

 

if it was still a big problem, I suspect there would be many more posts, so in the absence of those, you have to treat it as something particular to your setup.

 

so begins the process of elimination....

[eL_PuSHeR]

I have also noticed that DF is very much slow when defragging a lot of small files than other defraggers. I thought it was that way though.

 

I don't know if this is a bug or not but DF definitely needs a speed improvement in the defragging algorithm department.

[login123]

OS = win xp pro 32 bit

CPU = AMD A8-5500, quad core, 3.2 GHz

HDD = Seagate Barracuda 1TB 7200 RPM 64MB Cache SATA 6.0Gb/s 3.5, no RAID

RAM = 8 GB ddr3

 

One other 1 tb HDD, has win 7 but is not visible to this OS when win xp is running.

 

Not many files, approx 25,700 small files

Time for individual files varies, apparently by file size

Seconds per file = 1, 3, 12, 13, maybe a few seconds more for some

No delay between each file

 

At start, Defraggler estimates > 1 day,

After 38 minutes estimate = 5.00 hrs

After 42 minutes estimate = 4.00 hrs

After 43 minutes estimate = 3.50 hrs

After 44 minutes estimate = 2.25 hrs

After 45 minutes estimate = 2.00 hrs

After 51 minutes estimate = 0.50 hrs

After 54 minutes estimate = 10 minutes

After 55 minutes estimate = 5 minutes

After 58 minutes estimate = 2 minutes

After 58 minutes = done

actual time = about 58 minutes

 

There's my data. Analyze away.

 

Comment, I never met an app that did well at estimating completion time.

Don't know why. Some backup programs predict 4 days when they start but they finish in a bout 30-40 min.

 

Edited for clarity.

[eL_PuSHeR]

Offtopic:

 

You are running WinXP x86 WITH 8GB RAM?!!!

 

Your Operating System is probably "viewing" 3.5 GB at best. All the rest is lost.

[login123]

Offtopic:

 

You are running WinXP x86 WITH 8GB RAM?!!!

 

Your Operating System is probably "viewing" 3.5 GB at best. All the rest is lost.

 

Yes. Good catch.

I hadn't even thought about that, just remembered that 8 gb is installed.

Its a dual boot setup. The computer came with win 7 on the factory HDD and 8 GB RAM.

I put win xp on a second HDD, and as you say, win xp recognizes about 3.3 gb RAM.

 

Fwiw, not off topic at all imho, you were double checking.

[Alan_B]

It would be exceptionally wasteful for the computer to be supplied with 8 GB of RAM in a 32 bit system.

Even Windows 7 could not use the top half.

 

You have a friend here that is willing to find a good home for your spare RAM :D

 

Is it possible that you were supplied with 64 bit hardware and possibly 64 bit Windows 7,

but for some reason your installed Windows XP is only 32 bit ?