[NBLUG/talk] Missing ReiserFS superblock! (the plot thickens...)

[Lincoln Peters]

My new hard drive arrived in the mail today.  I can now attempt to  
back up the corrupted ReiserFS filesystem to it, so I can recover if  
I accidentally destroy it while trying to recover it.

I immediately ran into problems, though.  The usb-storage module  
completely failed to recognize the drive when I plugged it in.  I  
guess I've gone too long since my last kernel upgrade even TRYING to  
use a USB mass storage device.  But I knew that the drive itself  
works, because I had no difficulty reading it on my laptop (a MacBook  
Pro, running Mac OS 10.4.7).  To work around this, I decided to boot  
the from a Knoppix DVD.  Knoppix had no difficulty recognizing the  
drive.

Next, I repartitioned the disk and created a gigantic ext2  
filesystem.  My plan is to copy the entire RAID array to a file on  
the disk, using compression (i.e. gzip) if needed.  Since I want to  
make the process as simple as possible (I'm not quite sure what I'm  
doing here!), I aimed for the data on the external disk to be  
completely unencrypted, even though the data on the array is (as per  
dm-crypt).  If I run "shred" on the disk when I'm done, then there  
should be really no chance of sensitive data remaining on the drive,  
and when I use it for regular backups, *then* I'll encrypt it.

Once this was done, I ran into another problem: the dm-crypt tools  
are not present on my Knoppix DVD!  So while I can't seem to use the  
disk while booted from the (still working) Debian OS on the hard  
disk, I can't seem to read the contents of the RAID array while  
booted from Knoppix, either!  This required a more creative  
solution.  I found that the source tree I'd originally used to build  
my system's custom kernel was still on the root filesystem (in root's  
home directory), so I re-examined the configuration to see if I could  
figure out what was wrong.  Sure enough, I found that while I had  
enabled the usb-storage module, I had *disabled* all of its sub- 
options.  This led me to suspect that the Knoppix CD had one or more  
of the sub-options enabled, that allowed it to access the drive, so I  
tried recompiling with all the sub-options enabled (except verbose  
debugging).  Then I waited for it to recompile.  Finally, I install  
and reboot.

Still no luck.  Then I took a closer look at the output of lsmod and  
found that the "echi-hcd" module wasn't loaded, and when I loaded it,  
*then* it recognized the drive!

Finally, I tried to back up the array.  And then things got REALLY  
interesting.  I created an ext2 filesystem on the drive, mounted it  
at /mnt, and issued the following command:

dd_rescue /dev/md1 /mnt/home.img

The command returned almost immediately, with no error.  And yet the  
file it generated was completely empty!  At this point, I suspected  
that the real problem might lie somewhere within the RAID software,  
rather than ReiserFS.  I used mdadm to check the status of the drive,  
and found that it was active, but degraded (apparently it failed to  
locate one of the three disks when initializing--this is a problem  
I'd been working on prior to the meltdown).  I thought that the  
failure might then have something to do with the absence of that one  
disk (which was present and working, but mdadm just failed to  
recognize), so I tried hot-adding it.

The results of the hot-add were surprising.  Usually, when I hot-add  
a disk, there is a flurry of disk activity as the newly-added disk is  
built (although the computer remains usable).  No such flurry of  
activity.  I check the status of the array (as per "mdadm --detail / 
dev/md1"), and the third disk was still shown as "removed", but the  
disk I had hot-added (in reality the same disk) was shown as a  
"spare".  I've never seen it do that before!

I scoured the mdadm manpage, and can't find anything that seems to  
have any effect on the situation.


Anyone have any idea what could possibly be going on here, or if  
nothing else, how to find out?


--
Lincoln "The DiskBuster" Peters
<petersl at sonoma.edu>

Take an astronaut to launch.