Arch Linux

Please read this before reporting a bug:

Do NOT report bugs when a package is just outdated, or it is in the AUR. Use the 'flag out of date' link on the package page, or the Mailing List.

REPEAT: Do NOT report bugs for outdated packages!

FS#7778 - xfs_fsr silently corrupts files while or after(?) defragmenting.

Attached to Project: Arch Linux
Opened by Felix (thetrivialstuff) - Saturday, 11 August 2007, 05:20 GMT
Last edited by Tobias Powalowski (tpowa) - Tuesday, 21 August 2007, 19:56 GMT
Task Type Bug Report
Category Kernel
Status Closed
Assigned To Tom Killian (tomk)
Architecture i686
Severity Critical
Priority Normal
Reported Version 2007.08 Don't Panic
Due in Version Undecided
Due Date Undecided
Percent Complete 100%
Votes 0
Private No


Files written to during or after the defragmentation run seem most at risk. Blocks of files get zeroed out (filled with null bytes). There are no reports of the corruption in any system logs, or when accessing the files, and xfs_check and xfs_repair both report that the filesystem is completely healthy.

Originally, I could not prove the cause of the corruption, which happened to my home partition, but I have now had the time to set up a virtual machine (since I didn't want to do this to my real system again if I could help it), and I was able to reproduce the bug.

It might or might not be related to o_direct issues, and it might or might not be xfs_fsr's fault, but xfs_fsr is definitely involved. I really don't know enough about kernel code etc. to look into this.

Additional info:
* package version(s)

xfsdump 2.2.45-1

$ uname -a
Linux myhost 2.6.22-ck #1 SMP PREEMPT Sun Jul 22 21:53:24 IST 2007 i686 Intel(R) Pentium(R) 4 CPU 3.20GHz GenuineIntel GNU/Linux
(this is the same on my virtual machine and on the real one -- that's a dual core CPU, if that's relevant.)

* config and/or log files etc.
A complete virtual machine image is available, if desired. For obvious reasons (size), I'm not going to attach it ;)

Steps to reproduce:
1. Fragment an XFS filesystem; preferably the fragmentation should affect a large file (> 30MB), but I'm not sure if this really increases the likelihood of this happening. In order to cause fragmentation, I ran the 'makefiles' script (attached) while simultaneously running the test.c program (attached). I would let the test.c program run for a few seconds, kill it, move the output file, run it again, etc. until the volume was full.

2. Remove the test.c output files so that there is some free space on the volume for xfs_fsr to work.

3. Run the 'makefiles' script again, and while that's running, start xfs_fsr on the filesystem in question. It may make the bug more likely to occur if you repeatedly and rudely press ctrl+c and restart xfs_fsr -- this is what I did during the first incident, because I had mistyped some command line options and it wasn't defragging what I wanted.

4. Let xfs_fsr finish its run of the filesystem.

5. sync. I don't know if this is needed or not, but since I suspect cache/filesystem incoherency might be involved, try it.

5. Run the program in conjunction with find, looking through files you know to be text -- the files generated by 'makefiles' are a good place to start, but the corruption won't necessarily happen there. For example:

find -mount -type f -exec -s 00 '{}' \;

If you're running it in an area where there might be lots of binary files that are supposed to contain null characters, increasing the number of consecutive nulls in the search string might cut down on false positives:

find -mount -type f -exec -s 000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 '{}' \; > nulls

(Spurious side note -- this will match every Microsoft Office document in existence ;) )

In my virtual machine test case, none of the output files from makefiles got corrupted, but I deliberately ran it on /var so that all the text files in /var/log would be bait -- and it worked; pacman.log got a hole of 512 null bytes punched into it.

Some other things of note:

- on my real system, the corrupted volume was a software RAID-1. On the test virtual machine, my test volume was not RAID.

- the virtual machine and my real machine are using different IDE controllers --
real: Intel Corporation 82801EB/ER (ICH5/ICH5R) IDE Controller (rev 02)
virtual: Intel Corporation 82371AB/EB/MB PIIX4 IDE (rev 01)
I think that means it's not the IDE controller's fault, or any drivers related to it?

- I have not been able to reproduce the issue by any means not involving xfs_fsr -- that includes very large file copies with and without the o_direct test program, with and without the hard drives under other stress, and over 100GB of data that I copied from one hard drive to another and then checksummed -- that copy was perfect.

- At least two other people have experienced similar things with xfs_fsr; however, their kernels were vastly different from mine --
for prior discussions on this. In the former case it was an o_direct issue on a 2.4 kernel and recompiling it with an older GCC was able to fix things. (This is what initially inspired me to find out more about o_direct, and in light of everything that's happened to me I now side with Linus in the debate about its worth ;) .)

- I first noticed the corruption in a mailbox file, in an area of the file that had not existed when the xfs_fsr run was done (about 2 hours had elapsed since it finished). Does this mean xfs_fsr is reallocating data and/or free space without informing the kernel, which later tries to write data to what was formerly the correct place?

- I have run a few passes of memtest on this box and found no problems. I will be doing an exhaustive run tonight (I didn't have time earlier), but I do not suspect memory issues as this was formerly a gaming machine and has very good quality components.

- I've also exhaustively tested the hard drives for faults and found no problems (but if this was a hard drive issue, something ought to show up in messages.log and nothing has. Hard drives usually complain loudly when data corruption is their fault. Also, remember that in one case this happened on a RAID1 array.)

I hope all this is at least somewhat helpful in figuring out what causes this...

This task depends upon

Closed by  Tobias Powalowski (tpowa)
Tuesday, 21 August 2007, 19:56 GMT
Reason for closing:  Won't fix
Comment by Felix (thetrivialstuff) - Saturday, 11 August 2007, 06:20 GMT
Update -- this is now much more likely to be a problem specific to the CK kernel. I was unable to reproduce the bug when I booted my virtual machine into 2.6.22, and was able to reproduce it first try when running CK again. This time it corrupted two files, also in /var/log, when running xfs_fsr on /var.

Appearences of bug with ck kernel: 2/2 trials (3/3 if you count my /home mishap)
Appearences of bug with standard kernel: 0/4 trials (the last one of which I was particularly nasty to xfs_fsr)


PS: Sorry about breaking the template up there with my long string of zeros.. I'd go back & edit it if I could figure out how.
Comment by Roman Kyrylych (Romashka) - Saturday, 11 August 2007, 13:05 GMT
could you please try it with new kernel26ck
Comment by Felix (thetrivialstuff) - Saturday, 11 August 2007, 15:54 GMT
Hi -- bug is still present under new kernel26ck; got two corrupted files on this run. I was actually under the impression that kernel26ck was at its last version? :)

What got changed?

Update: I ran the memtest as promised and there were 0 errors after 7 complete test rounds (about 7 or 8 hours continuous testing).
Comment by Tobias Powalowski (tpowa) - Tuesday, 21 August 2007, 19:56 GMT
closing this one now ck patchset is dead