Arch Linux

Description:
Files written to during or after the defragmentation run seem most at risk. Blocks of files get zeroed out (filled with null bytes). There are no reports of the corruption in any system logs, or when accessing the files, and xfs_check and xfs_repair both report that the filesystem is completely healthy.

Originally, I could not prove the cause of the corruption, which happened to my home partition, but I have now had the time to set up a virtual machine (since I didn't want to do this to my real system again if I could help it), and I was able to reproduce the bug.

It might or might not be related to o_direct issues, and it might or might not be xfs_fsr's fault, but xfs_fsr is definitely involved. I really don't know enough about kernel code etc. to look into this.

Additional info:
* package version(s)

xfsdump 2.2.45-1
kernel26ck 2.6.22.1.ck1-2

$ uname -a
Linux myhost 2.6.22-ck #1 SMP PREEMPT Sun Jul 22 21:53:24 IST 2007 i686 Intel(R) Pentium(R) 4 CPU 3.20GHz GenuineIntel GNU/Linux
(this is the same on my virtual machine and on the real one -- that's a dual core CPU, if that's relevant.)

* config and/or log files etc.
A complete virtual machine image is available, if desired. For obvious reasons (size), I'm not going to attach it ;)

Steps to reproduce:
1. Fragment an XFS filesystem; preferably the fragmentation should affect a large file (> 30MB), but I'm not sure if this really increases the likelihood of this happening. In order to cause fragmentation, I ran the 'makefiles' script (attached) while simultaneously running the test.c program (attached). I would let the test.c program run for a few seconds, kill it, move the output file, run it again, etc. until the volume was full.

2. Remove the test.c output files so that there is some free space on the volume for xfs_fsr to work.

3. Run the 'makefiles' script again, and while that's running, start xfs_fsr on the filesystem in question. It may make the bug more likely to occur if you repeatedly and rudely press ctrl+c and restart xfs_fsr -- this is what I did during the first incident, because I had mistyped some command line options and it wasn't defragging what I wanted.

4. Let xfs_fsr finish its run of the filesystem.

5. sync. I don't know if this is needed or not, but since I suspect cache/filesystem incoherency might be involved, try it.

5. Run the bingrep.pl program in conjunction with find, looking through files you know to be text -- the files generated by 'makefiles' are a good place to start, but the corruption won't necessarily happen there. For example:

find -mount -type f -exec bingrep.pl -s 00 '{}' \;

If you're running it in an area where there might be lots of binary files that are supposed to contain null characters, increasing the number of consecutive nulls in the search string might cut down on false positives:

find -mount -type f -exec bingrep.pl -s 000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 '{}' \; > nulls

(Spurious side note -- this will match every Microsoft Office document in existence ;) )

In my virtual machine test case, none of the output files from makefiles got corrupted, but I deliberately ran it on /var so that all the text files in /var/log would be bait -- and it worked; pacman.log got a hole of 512 null bytes punched into it.

Some other things of note:

- on my real system, the corrupted volume was a software RAID-1. On the test virtual machine, my test volume was not RAID.

- the virtual machine and my real machine are using different IDE controllers --
real: Intel Corporation 82801EB/ER (ICH5/ICH5R) IDE Controller (rev 02)
virtual: Intel Corporation 82371AB/EB/MB PIIX4 IDE (rev 01)
I think that means it's not the IDE controller's fault, or any drivers related to it?

- I have not been able to reproduce the issue by any means not involving xfs_fsr -- that includes very large file copies with and without the o_direct test program, with and without the hard drives under other stress, and over 100GB of data that I copied from one hard drive to another and then checksummed -- that copy was perfect.

- At least two other people have experienced similar things with xfs_fsr; however, their kernels were vastly different from mine --
http://osdir.com/ml/file-systems.xfs.general/2003-04/msg00070.html
and
http://oss.sgi.com/archives/xfs/2007-06/msg00336.html
for prior discussions on this. In the former case it was an o_direct issue on a 2.4 kernel and recompiling it with an older GCC was able to fix things. (This is what initially inspired me to find out more about o_direct, and in light of everything that's happened to me I now side with Linus in the debate about its worth ;) .)

- I first noticed the corruption in a mailbox file, in an area of the file that had not existed when the xfs_fsr run was done (about 2 hours had elapsed since it finished). Does this mean xfs_fsr is reallocating data and/or free space without informing the kernel, which later tries to write data to what was formerly the correct place?

- I have run a few passes of memtest on this box and found no problems. I will be doing an exhaustive run tonight (I didn't have time earlier), but I do not suspect memory issues as this was formerly a gaming machine and has very good quality components.

- I've also exhaustively tested the hard drives for faults and found no problems (but if this was a hard drive issue, something ought to show up in messages.log and nothing has. Hard drives usually complain loudly when data corruption is their fault. Also, remember that in one case this happened on a RAID1 array.)

I hope all this is at least somewhat helpful in figuring out what causes this...

~Felix.