General NAS-Central Forums

Welcome to the NAS community
It is currently Sun Nov 19, 2017 2:19 pm

All times are UTC




Post new topic Reply to topic  [ 36 posts ]  Go to page Previous  1, 2, 3  Next
Author Message
PostPosted: Sat May 16, 2009 12:43 pm 
Offline

Joined: Fri Feb 06, 2009 7:49 pm
Posts: 16
Hi wim

Sorry for the long delay since my last response. I've been quite busy. Thank you for posting your first sectors of the partitions. Let me explain what we're looking at here and why.

You observed that the partitions were of type "SFS" in fdisk. Partition types are stored in the partition map, and are meant to suggest what type of partition it is, but don't really tell you anything about the filesystem or data on the partition. For example, on a typical GNU/Linux system, all of the data partitions use partition type 0x83, "Linux", regardless of what filesystem those partitions use (ext2, ext3, ext4, reiserfs, xfs, btrfs, etc.) If you look at a list of partition types, you'll see that some correspord to file systems, some correspond to operating systems, and some correspond to particular functions rather than filesystems (for example 0x82, Linux swap). In the case of your partition type, 0x42 "SFS", that is Peter Gutmann's "Secure Filesystem", one of the early encrypted filesystems for DOS, which I highly doubt is the fs used for your disk. (You *didn't* encrypt your disk, did you? If you did, then we're in for a whole world of hurt coming up, because recovering encrypted disks when there's no backup can be a real bear...) More likely, the disk label is 0x42 "SFS" because "SFS" also stands for "San File System", which doesn't really tell us anything, either, since "San File System" is a class of filesystems, and not a particular one.

Anyway, that's not really relevant. What is relevant is the hexdumps of the first block of each partition that you got. The first one looks like a PBR, and the second looks like a big bunch of zeroes. The reason I wanted to see this is that fdisk listed three partitions in the PBR, but they didn't go to the end of the partition, and I wasn't sure where the data partitions tarted (here I'm making some assumptions based on what I saw on the last EBD I looked at; it may well be that the one you're working with is very, very different.

Anyway, if we look at your PBR, we see it's mostly empty, except for 0x1be - 0x1fd, which is where the actual partition map on an MBR or a PBR is (most of an MBR is boot code; obviously a PBR doesn't need this.), and the last two bytes, 0x55aa, which is the signature for an MBR/PBR. Anyway, your partition table is this:

0002 0100 82fe 3f0f 3f00 0000 92eb 0300
00 02-01-00 82 fe-3f-0f 0000003f 0003eb92
Not Bootable
First block: Head 2 Sector 0 Cylinder 1
Partition Type 0x82 (Usually Linux Swap)
Last block: Head 254 Sector 15 Cylinder 63
First sector: 63
Number of blocks: 256914

0000 0110 05fe 3f10 d1eb 0300 c13e 0000
00 00-01-10 05 fe-3f-10 0003ebd1 00003ec1
First Block: Head 0 Sector 0 Cylinder 65
Partition Type 0x05 (Extended)
Last Block: Head 254 Sector 15 Cylinder 515
First sector: 256977
Number of blocks: 16065

0000 0000 0000 0000 0000 0000 0000 0000
No partition 3

0000 0000 0000 0000 0000 0000 0000 0000
No partition 4

Uhmm, so it looks like we need to see what the disk looks like starting at sector 256977. If what I remember about Extended partitions in an MBR is right, then there will be *another* partition table there. What is throwing me off is the fact that the extended partition (sda1p5) is showing up as having size zero (start and end at cylinder 17) but maybe it's because the entry for the second partition in the PBR says it's only one cylinder long, and it's expecting the whole partition 5 to be contained within partition 2? (My shop an Apple-only shop, so the partition table types we usually work with are APM and GUID, which haven't been hacked to death to support big disks and many partitions the way MBR has.)

Anyway, let's see what we get at cylinder 16 and beyond...

Code:
ddrescue /dev/sda1 /2tbdisk/test.img -i 256977b -o 0 -s 48195b


This will dump a file containing sectors 16, 17, and 18. It will be about 24MB, so too big to post a full hexdump. It wil probably be mostly zeroes so if you could compress it and upload here as an attachment I'd like to see it.


Top
 Profile  
 
PostPosted: Mon May 18, 2009 3:26 pm 
Offline

Joined: Thu May 07, 2009 12:52 pm
Posts: 19
no, i didn't encrypt the disk. yes, there was another partition table there as shown below
the file dump is attached, the archive has a password incase there is sensitive data contained in there, i will PM you the password
Code:
00 01 01 10 83 FE 3F 10 3F 00 00 00 82 3E 00 00
00 00 01 11 05 FE 3F 11 92 2A 04 00 C1 3E 00 00

edit: i am having difficulties getting this forum to accept file attachments, so i have uploaded the test archive here instead


Top
 Profile  
 
PostPosted: Tue May 19, 2009 12:43 pm 
Offline

Joined: Fri Feb 06, 2009 7:49 pm
Posts: 16
Another extended partition table, fun!

OK, the stuff in the dump you sent me looks like it's all OS stuff... Can you do another dump like before, but this time the offset will be 530019 blocks, so

Code:
ddrescue /dev/sda1 /2tbdisk/test2.img -i 530019b -o 0 -s 48195b


Also, please don't rely on password-protected archives to protect data posted on a public forum. It doesn't really matter in this case because the data you sent me only contained part of the operating system for the EBD, but if there is a potential for exposure of truly sensitive data, you should be aware that password-protected archives can be cracked with enough effort.

If there isn't anything on the drive that you're too worried about, you can continue sending password-protected archives. If you would like to protect the file more securely, you can encrypt it to me using my public PGP key. You can use GPG on GNU/Linux (I believe it's already installed on URR as "gpg", but I could be wrong) or you can install GPG on Windows from gnupg.org. You won't need to set up your own key if all you want to do is to encrypt something for me to see; I just won't be able to verify that it came from you (not really a concern here). If you do decide to set up your own key, you'll want to do it on a primary machine, and not on the live CD, because the key will be lost when you reboot.

If you want to compress the file, you should compress it before you encrypt.

My public key is:
Code:
-----BEGIN PGP PUBLIC KEY BLOCK-----
Version: GnuPG v1.4.9 (GNU/Linux)

mQGiBEkfV/4RBACVBILQb/sxe81tyJxN4ZsMd7Lq2xXavwiAiIfmoGXcWYsOUfli
XsMq2++UWsXWmqLvlrfieWyz+QQ2yXd9zkETH+nKMsBkkojTca+dcOyG5S4UHowg
J4Hu8mWLo6iTBRgzB4dBs1DcKHgW+cFoZD++Lv8Cg3vLhELv/kNO7aziSwCgylGl
rqnZv9sn3/E9YrNJyeB6W7UD/3NQxygB2sqcEkih7kd7ARsFRx9VJXS+vTngiPlH
BMJzmFyxNpNrDMfsHKrmLP1LzPf+FtQrt5tev/wu7EjPF7DkjyZ+sd3mxKhLRy3H
tKYiChHueszUvOixRDl1LWwsv2tDI2HlkMtRckZK6e5lpczYltXt21WwrBJPglbC
vBySA/9pranfqFWnSEQazsuFxqI4G3Z5KCiznwX8SeLs8XniaYx2dlqQyYeYkCEK
TsL9yjPYc8XqOV8Ds7hZX0sYP8dA7e/yDdD+I/TV5v2MK3waohEsUFFIql3EjIgT
KGP3fSB4mDIaPIiUEt7wOm8JSyRDv3Z5kFctKVuma2Vh8M1apLRERGFuaWVsIERh
ZGFwICgiRGFuaWwgTWlraGFpbG92aWNoIFNodGFub3NuaW1heWt5IikgPGNoaXRh
eXVAbWFpbC5ydT6IZgQTEQIAJgUCSR9YlwIbAwUJCWYBgAYLCQgHAwIEFQIIAwQW
AgMBAh4BAheAAAoJEDm5/VG/jt7T3JgAoJ+dOu0KDxm+TWYNEqHgIcOxMO4pAJ97
Zo87Ks4SG2HwQz+SfJVD76pJHbQlRGFuaWVsIERhZGFwIDxkYW5pZWwuZGFkYXBA
Z21haWwuY29tPohmBBMRAgAmBQJJH1f+AhsDBQkJZgGABgsJCAcDAgQVAggDBBYC
AwECHgECF4AACgkQObn9Ub+O3tOuPgCfVGjLw8f6lRJQvsL25J5OCudCHYsAoJnY
E5qBz5cSjogcMu3JU02dOfV5tCBEYW5pZWwgRGFkYXAgPGRhbmllbEBkaWdzb2Mu
Y29tPohmBBMRAgAmBQJJH1gmAhsDBQkJZgGABgsJCAcDAgQVAggDBBYCAwECHgEC
F4AACgkQObn9Ub+O3tN0MgCgxPqKI42merzDs7veflwZMk0NL9oAoLxCPJQo7RnD
WkS3z5pLlCkmMuf+tCVEYW5pZWwgRGFkYXAgPGRhbmllbEBtYXJpZWFuZGRhbi5j
b20+iGYEExECACYFAkl7P+UCGwMFCQlmAYAGCwkIBwMCBBUCCAMEFgIDAQIeAQIX
gAAKCRA5uf1Rv47e007qAJ9Zq0RZ6XqkJiuXh2VcFHLgyFbgmACcD998yuyxXpSr
A+Idd/G6d/UjIly0H0RhbmllbCBEYWRhcCA8ZGFuaWVsQGRhZGFwLm5ldD6IZgQT
EQIAJgUCSXtADgIbAwUJCWYBgAYLCQgHAwIEFQIIAwQWAgMBAh4BAheAAAoJEDm5
/VG/jt7T4GMAnjE0FIe7tsfozBGa1q4zSolGpkZ1AJ9MIUEQzW/XZz2Lsjp+9h3C
xr9kBLkCDQRJH1gDEAgAs3Ih8g/iiHEQWqdUlSwFRqV10kGYd9ZZ/Nb3hzuxjGRg
ra7S2AwE8YVqlYW0ZFQMqjl0lDHSFvbDaV39S+zQUbeL3MlrLwZ0dT6jpJPzRi8B
43eU1bhes7GH0V35e6e/0R4EXRs+jPlTMsMlTBAQEZUetUhmiAseM8hhXUc4/6yr
dk6XT77ia2dUUEA89D5kDdeQGALg3WTYflW0JgRKkmK8l1HbcsPJye57+tVbti0z
ctB8y144CEE3mN83nYbeeZCokcNKojuqB75jY5mNiXjEVHlh2WutFyYlFDTjDnUA
Q9mozCgNYQClcPk8XqteZSYG80gUp9/GO9hzp2vyrwADBQf+Niyu48wVO8rBK4eT
nrvyt+iT+IQEch6ssCLq6QTQBGyLsEnaMDaGy4ZBiRJ9kgVR7sFb/1Ta/VX8u1tf
xN+ghjFVEGSulRsNmbFqRexlD6DhT8nC49lMvWh2c/+aP/5nYzeIrILL8N0wjfut
nuQOsw+I1DPx2fmEHVbG5b3W8/CHMdghu2ieAjsYz2UilCHfRoRA8rYzH3xu+2VK
HYGGUv8o38aTImc+QOKZXZMxKuU1qXk6Vzsc5DgG+k1OwSlkB5vHfN/SsO4+3wsM
n4PbCeGE5o2FlCK0iIL3s8SzfUGC1jelihAYLTHGTVztDb4hqHs1Q1vBx0GsQAkZ
BzYiDohPBBgRAgAPBQJJH1gDAhsMBQkJZgGAAAoJEDm5/VG/jt7TxrAAoK1iZENF
MsmqoqsBsEgS+6AaIAyfAKCtUtEtVEsPSlUVO9ZYqgNqNyjoKA==
=w88T
-----END PGP PUBLIC KEY BLOCK-----


Copy and paste the key into a text file, and you'll be able to import it into any recent version of GPG on any platform.

You can also get my public key from several keyservers, including pgp.mit.edu and keyserver.pgp.com under the ID 0xBF8EDED3. The fingerprint for this key is 4F56 9E1D 42B7 93B5 9C13 6576 39B9 FD51 BF8E DED3, although if somebody has compromised this forum for the purposes of feeding you a false PGP key (man-in-the-middle attack), then they will also be able to change the verification information. If you have any reason to suspect that somebody has compromised the delivery of my public key to you, then PM me and I will provide alternate methods of verification.


Top
 Profile  
 
PostPosted: Wed May 20, 2009 2:02 am 
Offline

Joined: Thu May 07, 2009 12:52 pm
Posts: 19
wow, you can find out a lot about someone from their public key block! such as rqnZv9sn3/E9YrNJyeB6W7UD means they speak a bit of russian, vBySA/9pranfqFWnSEQazsuFxq means they played some trombone as a child, and /VG/jt7T4GMAnjE0FIe7tsfozBGa1q4zSolGpkZ1AJ9MIUEQzW means they're getting married to a beautiful gal called marie in 59 days :P congrats dan!! :D :lol:

ok, i executed the command you said but there were bad sectors and ddrescue made a lot of clicking sounds from the disk and spammed the screen with a lot of error messages. it eventually finished, but i thought a log file might be useful so i instead executed
Code:
ddrescue /dev/sdb1 /wd/test2.img /wd/test2.txt -i 530019b -o 0 -s 48195b

(fyi, disk 1 is b now because i'm not bothering to unplug the windows drive anymore, and i'm mounting the 2tbdisk as 'wd' because its shorter and typing in qwerty is awkward and uncomfortable for me (by the way, is there any easy way to map the keyboard back to dvorak whilst using the live cd?))
i checked the output img file from this and the previous time, they were bit-for-bit identical copies.
i have put the log file and the .img file into a archive test2.7z, which compressed from 24 meg down to only 272k this time, and then copied your geek code block, including the ----BEGIN---- and ----END----- stuff into a key.txt file, downloaded gnupg for windows stuff, and executed
Code:
gpg --import key.txt
gpg -r "Daniel Dadap" -o test2e.7z -e test2.7z

and it seemed to work (at least it made a test2e.7z file which is also 272k and can not be opened as a 7z file..). though i was not really sure if what i was doing was correct so if you are unable to decrypt the file please give me some guidance and i'll try again. here is the file


Top
 Profile  
 
PostPosted: Thu May 21, 2009 12:29 pm 
Offline

Joined: Fri Feb 06, 2009 7:49 pm
Posts: 16
wim wrote:
wow, you can find out a lot about someone from their public key block!

To be fair, that information wasn't in the actual key... you found out all of that by following up on information in the key. :) Thanks for the warm wishes -- I hope you understand now why I sometimes take a couple of days to reply. (Putting on a wedding is a LOT of work! Especially when we both already have day jobs!)
Quote:
ok, i executed the command you said but there were bad sectors and ddrescue made a lot of clicking sounds from the disk and spammed the screen with a lot of error messages.


Yes, it's in the same place as before. Luckily, this was inside the one of the OS partitions, as we had hoped earlier.

Quote:
(fyi, disk 1 is b now because i'm not bothering to unplug the windows drive anymore, and i'm mounting the 2tbdisk as 'wd' because its shorter and typing in qwerty is awkward and uncomfortable for me (by the way, is there any easy way to map the keyboard back to dvorak whilst using the live cd?))

Yes, /wd is a much better mountpoint name. :)

I use Dvorak myself as well, though since my job takes me to random other people's computers to fix them, I am still very used to typing in QWERTY. I taught myself Dvorak one afternoon writing a book report or something in junior high school -- what would you call it in Australia? Intermediate school? Secondary school? Anyway, I changed my keyboard layout without physically moving the keys, printed out a diagram of the layout and stuck it on the wall, and started typing. It went very slowly and awkwardly for the first half an hour or so, but started coming naturally after that.

Anyway, I usually set up my keyboard mapping at install time, so I'm not 100% sure this will work, but it should... Basically, do "sudo dpkg-reconfigure console-data" and it should guide you through selecting a new keymap. If you don't have the "console-data" package, by now you've probably learned that the way to install software on Debian and its offspring is by typing "sudo aptitude install console-data". I tested this on my system, which is Debian and not a LiveCD, but it should also work on Ubuntu, with the only caveats for a LiveCD being that 1) the "console-data" package may not be installed by default and 2) you'll have to do this every time you boot, since obviously you can't make permanent changes to the OS configuration. In the case of doing something like changing a keymap, this can actually be a boon, since if you foul something up horribly (a very bad thing when you're talking about your keymap in a text-only environment) all you need to do is reboot and it will go back to default. :)

Also, I'm fairly sure a standard Ubuntu CD gives you a menu when you boot into it that, among other things, allows you to select a keyboard layout. It might be that the Rescue Remix doesn't have the same boot menu, but if it does, this would be a better way of changing the keymap, since it doesn't require you to type "sudo dpkg-reconfigure console-data" in QWERTY.

Quote:
i checked the output img file from this and the previous time, they were bit-for-bit identical copies.

I hope you did this with a checksum, and not by inspecting file contents!!!

Quote:
i ... copied your geek code block

No you didn't, because I didn't give you my Geek Code... :) (I haven't updated my Geek Code in a while... maybe it's time to do a new one, or maybe I should wait until after a wedding since I'm pretty sure that's one of the things on the code.)

Quote:
i was not really sure if what i was doing was correct

It was perfect. Now you can send me secret messages any time you want. Of course, if you want to send real secret messages in the future, you'll probably want to have your own pair of keys, so that when you encrypt the data, you can encrypt it to your recipient or recipients, and also to yourself. Then you can destroy the unencrypted original (make sure to actually erase the file and not just delete it!) and if you need to recover it you can decrypt it with your own key. If the cyber-apocalypse comes, then you'll want to start signing everything, too, and making sure everything you read is also signed. You'll want to make sure you have lots of keys you trust from people you trust. And never trust a key unless you have physically received it or its fingerprint directly from a person you trust. Think man-in-the-middle attack, which will probably become common in the cyber-apocalypse. (I'm half kidding about the cyber-apocalypse stuff. But I'm also half NOT KIDDING. Build a web of trust with all your geeky friends!)

Anyway, now that that is all out of the way, it looks like we found our last partition table, at offset 0x7e0000. I must have undershot in my offset for the last imaging run by about 8MB... or overshot it by some unknown number. I don't really feel like double-checking the math now.

Code:
0001 0122 83fe 3f7c 3f00 0000 5c4e 1600
00 01-01-22 83 fe-3f-7e 0000003f 00164e5c

From the 0x165e5c, we can see that this is the largest partition so far... at just a tad under 750MB... There are no further partitions defined in the EBR.

Let's do two things. Let's check to see if we have the start of our data partition after the end of the last OS partition, and let's grab the MBR of disk 1 and inspect it manually. (We'll also check disk 2's MBR.)

Code:
ddrescue /dev/sdb1 /wd/test3.img -i 2000000b -o -s 134217728
ddrescue /dev/sdb /wd/disk1mbr.img -s 1b
ddrescue /dev/sdc /wd/disk2mbr.img -s 1b

In the first imaging command, I'm deliberately undershooting my calculated offset by about 4MB, and grabbing a whole 128MiB in the image (I think we're getting close, but I'd hate to be thrown off course by another unexpected EBR.) In the last, you'll obviously want to substitute sdc with the actual device name of your second EBD disk.


Top
 Profile  
 
PostPosted: Thu May 21, 2009 2:59 pm 
Offline

Joined: Thu May 07, 2009 12:52 pm
Posts: 19
is it likely that the ebd drive died in the first place due to bad sectors appearing in the OS partition?
will it be possible to revive the ebd by marking the bad sectors and reinstalling the OS somehow? i have seen there are images of the relevant stuff on these forums.
if you learnt dvorak in half an hour, you are superman. it took me more like half a month before i could type in a chat window without getting frustrated.
i did not do it with a checksum, i did it with a handy little util called copycleaner which i believe checks byte for byte..
oh, i see now that geek code block is actually a real thing..(!) i was just calling the public key block a geeky thing to have !! :lol:

i assumed you meant to have -o 0 as a missing argument in the command, and i made the test3.img file .. it's 128 meg but only compresses to 123 meg so i am not able to upload it. however here are the mbrs, take them with a grain of salt though i suspect that windows disk manager may have stomped over this part of the disk when i tried to 'import foreign disks' and assign drive letters. i have bought a new ebd today and its mbrs are different, and its partition tables are like you expected previously (1, 2, 5, 6, 7, 8, 9 for disk 1 and nothing on disk 2). it has seagate drives in it when my old one had samsungs.

anyway here are the mbr for the old disks

Image
Image

and incidentally i think i may have found an XFS file header (the start of the data partition?) in my backup image of disk 1. it looks like this:
Image

the same thing exists in the test3.img file at 0x003EFC00 (i guess it is corresponds to the same part of the disk)
i am very tempted to copy the partition tables over from the new disk to the old disk and see if that reveals any useful information for me. but i'll wait for your next reply before i do anything brash like that

ps, when is the cyber-apocalypse?


Top
 Profile  
 
PostPosted: Thu May 21, 2009 3:48 pm 
Offline

Joined: Fri Feb 06, 2009 7:49 pm
Posts: 16
wim wrote:
oh, i see now that geek code block is actually a real thing..(!) i was just calling the public key block a geeky thing to have !! :lol:

Heh, I assumed that you were familiar with Geek Codes and that "Geek Code" was a Freudian slip since the Geek Code has a very similar BEGIN/END marking to a PGP block. :)
Quote:
i assumed you meant to have -o 0 as a missing argument in the command, and i made the test3.img file .. it's 128 meg but only compresses to 123 meg so i am not able to upload it. however here are the mbrs, take them with a grain of salt though i suspect that windows disk manager may have stomped over this part of the disk when i tried to 'import foreign disks' and assign drive letters. i have bought a new ebd today and its mbrs are different, and its partition tables are like you expected previously (1, 2, 5, 6, 7, 8, 9 for disk 1 and nothing on disk 2). it has seagate drives in it when my old one had samsungs.

Yes, I must have missed the '0' key, thanks for catching that. I guess you are right about importing foreign disks trashing the MBRs... I was really hoping that it wouldn't have been the case; after all, who would be so bold as to rewrite a partition table without being asked to do so, but there you have it.
Quote:
and incidentally i think i may have found an XFS file header (the start of the data partition?) in my backup image of disk 1. it looks like this:

Yes! Of course! This is a much simpler and more elegant solution than my hopping through all of the partition tables. Sometimes I overthink things a little bit.
Quote:
the same thing exists in the test3.img file at 0x003EFC00 (i guess it is corresponds to the same part of the disk)

Yes, it is the same. 63 * 512 + 2000000 * 512 + 0x3efc00 = 0x3d487a00. The 63 blocks is the offset of sda1 from sda, the 2000000 blocks is the offset from our imaging operation, and 0x3efc00 is the offset within the image file. It's also the amount I undershot the offset, just in case I was wrong about the last partition table. It's good to know that we would have come up with the same answer if we had followed through with my Rube Goldberg madness.
Quote:
i am very tempted to copy the partition tables over from the new disk to the old disk and see if that reveals any useful information for me. but i'll wait for your next reply before i do anything brash like that

That is very tempting, but you said yourself the disks are from different manufacturers, and therefore may be of slightly different sizes. "500GB" generally means "5,000,000,000 or more bytes" in the storage industry. How much the "or more" ends up being depends on the platters. Let's stick with the game plan for now.
Code:
rm /wd/ebd.img (if it exists)
ddrescue /dev/sdb /wd/ebd.img /wd/ebd1.log -i 2008125b -o 0
ls -l /wd/ebd.img
ddrescue /dev/sdc /wd/ebd.img /wd/ebd2.log -i 0 -o size_of_ebd.img

Notice that we're applying the offset (2008125 is 0x3d487a00 / 0x200 in decimal) to the whole disk /dev/sdb. We're also going to use the whole disk /dev/sdc instead of /dev/sdc1 (assuming sdc is your second disk) because we don't trust the partition. After all that is done, try mounting the image. Save the size of the image after the first disk is done somewhere, in case we're wrong about ignoring the partition on the second disk. That way we can avoid re-imaging the first disk.
Quote:
ps, when is the cyber-apocalypse?

It could happen at any time, so the most important thing is to be ready for it when it comes! (Again, half kidding, but also half NOT KIDDING.)


Top
 Profile  
 
PostPosted: Thu May 21, 2009 3:51 pm 
Offline

Joined: Fri Feb 06, 2009 7:49 pm
Posts: 16
danek wrote:
Code:
rm /wd/ebd.img (if it exists)
ddrescue /dev/sdb /wd/ebd.img /wd/ebd1.log -i 2008125b -o 0
ls -l /wd/ebd.img
ddrescue /dev/sdc /wd/ebd.img /wd/ebd2.log -i 0 -o size_of_ebd.img


Make sure that you enter size_of_ebd.img in bytes. The output of ls -l should give you size in bytes. Don't postfix it with 'b' like the offset for /dev/sdb unless you divide it by 512 first. You've probably figured this out by now, but I wanted to be sure.


Top
 Profile  
 
PostPosted: Fri May 22, 2009 2:05 pm 
Offline

Joined: Thu May 07, 2009 12:52 pm
Posts: 19
ok, i have tried theses things and made some progress but have not recovered all the data, here's what happened..
creating the ebd.img all worked ok, and there were no read errors (i guess because it was all on the good part of the disk). the image even mounted. but the files and directories could not be read properly, it said the structure was damaged or something i don't recall the exact message. so i went and umounted /ebd and ran xfs_repair, which seemed to be doing a lot of stuff and printed lots to the screen, and finished after a couple of minutes. then some of my directories and files were readable.. but heaps of it was in a dir called lost+found and not in the correct location. and the filenames and subdirectory names in there seem to be random numbers :( there are >3000 folders and >100000 files in lost+found so renaming them manually is out of the question. also, after i went "cp -R /ebd /wd" with the ebd.img mounted at /ebd, i only get about 360 gig worth of data when i was expecting more like 800 gig. what could i try next ?

i did have a couple of questions too.. when we go "ddrescue /dev/sdb /wd/ebd.img /wd/ebd1.log -i 2008125b -o 0" to image the data part of the first disk, does that just go from the start of the XFS header to the end of the disk? are we assuming the data partition ends at the end of the disk, and that there are no other partitions living inbetween 2008125b and end? and then when we image the second disk, appending the whole thing, is that really correct ? i'm fairly sure for example the junk in the 2nd disk MBR doesn't really belong in the middle of the ebd.img file. is there a way to create the img file which will not break the structure of the file system, and hopefully not lose files and directories like seems to have been happening?

also, will i have to extract the data from both the source drives again since xfs_repair has done who knows what to the ebd.img file?


Top
 Profile  
 
PostPosted: Sat May 23, 2009 10:46 am 
Offline

Joined: Thu May 07, 2009 12:52 pm
Posts: 19
here is something else weird i have noticed today. on my new 2tbdisk, i have the image file (~1tb) and the recovered data (~360 gig). but my drive has only free space 271 gig.. i have checked the hidden system files (RECYCLER, etc) and there's negligible space used there. so where has my other 321 541 926 912 bytes gone..?? :?

Image


Top
 Profile  
 
PostPosted: Sun May 24, 2009 2:15 pm 
Offline

Joined: Fri Feb 06, 2009 7:49 pm
Posts: 16
wim wrote:
some of my directories and files were readable.. but heaps of it was in a dir called lost+found and not in the correct location. and the filenames and subdirectory names in there seem to be random numbers :( there are >3000 folders and >100000 files in lost+found so renaming them manually is out of the question. also, after i went "cp -R /ebd /wd" with the ebd.img mounted at /ebd, i only get about 360 gig worth of data when i was expecting more like 800 gig. what could i try next ?

Yes, any files whose directory entries couldn't be found or recovered will be put into lost+found and named by their inode numbers. I suspect that the offset for the second disk isn't 100% right, and we may need to get a little more creative here... As for the folder being small, you mentioned in your later post that there seemed to be some missing space... can you check whether lost+found is being accounted for in Windows' sizes of everything?
Quote:
i did have a couple of questions too.. when we go "ddrescue /dev/sdb /wd/ebd.img /wd/ebd1.log -i 2008125b -o 0" to image the data part of the first disk, does that just go from the start of the XFS header to the end of the disk?

Yes, that's exactly what happens. XFS is a little bit unlike many other file systems in that its header starts at the very beginning of the partition. Usually a little bit of space is reserved for boot code. That's why we start at the XFS header itself and not slightly before it, as we might for another filesystem. We go to the end of the disk because we assume that the EBD case spans the two whole disks together before doing anything else, a assumption I'm no longer comfortable with.
Quote:
are we assuming the data partition ends at the end of the disk, and that there are no other partitions living inbetween 2008125b and end? and then when we image the second disk, appending the whole thing, is that really correct ? i'm fairly sure for example the junk in the 2nd disk MBR doesn't really belong in the middle of the ebd.img file. is there a way to create the img file which will not break the structure of the file system, and hopefully not lose files and directories like seems to have been happening?

The data partition spans both disks. On the EBD that I saw, and on the new one that you purchased, primary partition 1 contains the EBD's OS. (Primary partition 1 contains extended partitions 5-9.) Primary partition 2 contains data. The second disk is the missing second half of primary partition 2. It might not be right to concatenate the whole disk 2 to the parttion from disk 1; I assumed this at first from my experience with other LaCie Big Disks (not EBD) where the two disks were simply spanned together, and in the event of a failed enclosure, one could simply concatenate images of disk one and disk 2 and create a perfectly working filesystem. I now believe that the errors in the filesystem from my earlier EBD recovery, which I assumed to be the result of unclean unmounts as the enclosure was failing, were actually the result of having the second disk concatenated with the wrong offset. The customer owning that particular EBD did not have a lot of data on the disk, and so a minority of files was sent to lost+found, making it a manageable task to send them back to their proper places.
Anyway, I have a solution in mind; read below for what we'll be trying.
Quote:
also, will i have to extract the data from both the source drives again since xfs_repair has done who knows what to the ebd.img file?

Yes, that would be the best thing. Since we have a mystery of missing space, it would probably be best to format the disk, if we can't account for the lost capacity. If you have room somewhere, you may want to first make a copy of the data you've recovered so far, just in case something catastrophic happens to one of your EBD disks in the meantime.

I did some light googling and couldn't find a decent reference to the structure of the XFS header blocks themselves, so I did a little bit of reverse engineering by creating image files of varying sizes and formatting them as XFS volumes. XFS seems to put copies of its header block on the disk at various places, with at least one being located exactly halfway through the volume, and possibibly others throughout. At offset 0x55 there is a big-endian number apparently denoting the interval at which the blocks are spaced, and at offset 0x5b a number apparently denoting the number of copies. These numbers may actually start at 0x54 and 0x5a, respectively, since there is a byte of leading zeroes preceding the numbers.

On the block that you found, the number 0x7452040000 is at 0x55 and 0x20 is at 0x5b. If we multiply these numbers we get a huge number, almost 16GB in fact, so this is probably wrong. However, I also noticed that in your XFS block you have the number 0x10 at 0x53, while in the filesystems I created, I have 0x01 at that offset. This may be some kind of multiplier, or maybe your numbers are shifted by a nybble. Whatever the case, if we accordingly divide your number by 16, we get 0xe8a4080000, or 99184400384, which is quite close to the size of your image at 999187564032. The difference is 3163648 bytes, or 6179 blocks. So if my analysis of the XFS block is correct, your image is 3163648 bytes bigger than it should be.

We do expect an XFS header block at byte 0x7452040000, or 499592200192 in the final reconstructed image, so we should check to make sure that we get one after correcting for the offset. Did you save the number you got as the size of your image of the first disk, starting at the XFS header and ending at the end of the disk? Subtract that number from 49959200192. This will be the offset on the second disk that we would expect a copy of the block if there is no offset between the two disks. I would make an image starting at this offset, about 8MB in size, and check to see if we find it at our expected offset of 3163648.
Code:
ddrescue /dev/sdc /wd/disk2test.img -i (49949200192-size_of_your_disk1_img) -o 0 -s 8388608

Open this in a hex editor and see if you find "XFSB" at 0x304600. If you do, then
Code:
ddrescue /dev/sdb /wd/ebd.img /wd/ebd1.log -i 2008125b -o 0
ddrescue /dev/sdc /wd/ebd.img /wd/ebd2.log -i 3163648 -o size_of_ebd.img

Then try mounting. If it looks like you need to run xfs_repair, then run it with the "-n" option (you should also probably be running it with "-f".) -n means "no repair" and will just print all the stuff that it would do. You'll want to redirect output to a log file, so we can see what it's actually trying to do.
Code:
xfs_repair -nf /wd/ebd.img > xfsrepair.log 2> xfsrepairerr.log

If you don't find "XFSB" at the offset we're expecting, let me know if and where you do find it.

Good luck!


Top
 Profile  
 
PostPosted: Mon May 25, 2009 4:14 pm 
Offline

Joined: Thu May 07, 2009 12:52 pm
Posts: 19
you must be checking that i'm paying attention.. first you say we expect an XFSB at 499592200192 in the image, then you say to subtract the size from 49959200192, then you give 49949200192 in the command! which of these three numbers did you mean? ;) i'll assume you meant 0x7452040000 = 499592200192. the filesize of my image the first time, of entire disk sdb starting from input offset 2008125b, was 499079702016 bytes. 499592200192 - 499079702016 = 512498176. i checked on the second disk at 512498176 + 3163648 = 515661824 = 1EBC6000 and did not find an XFSB thing there..that block looked like this. however, i did not bother to boot up in the live cd and do the ddrescue thing since i am able to open these physical disks directly in HxD which is much faster and easier. the nearest XFSB thing is the first one on disk 2 at 0x1EB80000. by the way, that's block 1006592 just like you asked in the original post, to answer your question a few months late.. :) it is all zeros up to there block 1006592 aswell, apart from the rubbish that windows disk manager thing has wrote into blocks 0 and 6. on disk 2, there are no more XFSB things at least up until 0x1418D4D70, thats about 5 gig in, i didn't bother searching further than that.

so, in summary, the only 2 XFSB things i have found are at 0x3D487A00 on disk 1 and at 0x1EB80000 on disk 2. i don't know what the blocks starting with XAGF and XAGI in the screenshots mean.

i have another question.. when i was going over one of your previous posts and you deciphered 2 partition table entries like so:
Quote:
0002 0100 82fe 3f0f 3f00 0000 92eb 0300
00 02-01-00 82 fe-3f-0f 0000003f 0003eb92
Not Bootable
First block: Head 2 Sector 0 Cylinder 1
Partition Type 0x82 (Usually Linux Swap)
Last block: Head 254 Sector 15 Cylinder 63
First sector: 63
Number of blocks: 256914

0000 0110 05fe 3f10 d1eb 0300 c13e 0000
00 00-01-10 05 fe-3f-10 0003ebd1 00003ec1
First Block: Head 0 Sector 0 Cylinder 65
Partition Type 0x05 (Extended)
Last Block: Head 254 Sector 15 Cylinder 515
First sector: 256977
Number of blocks: 16065

could you explain how you arrived at the head sector cylinder locations for the second entry? if i apply the logic of the first entry, i thought i would get the first block at H0 S16 C1 and the last block at H254 S16 C63 but maybe i'm making some wrong assumptions or something..


Top
 Profile  
 
PostPosted: Tue May 26, 2009 10:44 am 
Offline

Joined: Thu May 07, 2009 12:52 pm
Posts: 19
danek wrote:
I did some light googling and couldn't find a decent reference to the structure of the XFS header blocks themselves

this is the best thing i could find: http://xfs.org/index.php/XFS_FAQ, and from there i found this.


Top
 Profile  
 
PostPosted: Tue May 26, 2009 3:31 pm 
Offline

Joined: Thu May 07, 2009 12:52 pm
Posts: 19
ok dan i have been able to spend a good amount of time working on this tonight.. it seems you are correct that you get these XFS header blocks spaced evenly throughout the volume.
the XAGF thingy is "Allocation Group Free space block" and allows XFS to quickly find free space near a given block or of a given size.
the XAGI thingy is for Allocation Group Inode management
but the XFSB superblock thingy seems to tells us the most useful information.. here is my analysis of the primary superblock (the first one on disk 1 at 0x3D487A00)

Code:
58 46 53 42 00 00 10 00 00 00 00 00 0E 8A 40 80
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
65 66 EB 5C BF DE 11 D3 8C 5B 85 9E 42 64 49 8E
00 00 00 00 08 00 00 04 00 00 00 00 00 00 00 80
00 00 00 00 00 00 00 81 00 00 00 00 00 00 00 82
00 00 00 10 00 74 52 04 00 00 00 20 00 00 00 00
00 00 80 00 30 84 02 00 01 00 00 10 00 00 00 00
00 00 00 00 00 00 00 00 0C 09 08 04 17 00 00 19
00 00 00 00 00 04 B0 C0 00 00 00 00 00 00 18 ED
00 00 00 00 04 47 0C 2A 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 02 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00


58 46 53 42
__uint32_t  sb_magicnum = "XFSB"

00 00 10 00
__uint32_t  sb_blocksize = 4096 bytes

00 00 00 00 0E 8A 40 80
xfs_drfsbno_t  sb_dblocks = 243941504 blocks (= 999184400384 bytes)

00 00 00 00 00 00 00 00
xfs_drfsbno_t  sb_rblocks = 0 blocks

00 00 00 00 00 00 00 00
xfs_drtbno_t  sb_rextents = 0 extents

65 66 EB 5C BF DE 11 D3 8C 5B 85 9E 42 64 49 8E
uuid_t  sb_uuid;   

00 00 00 00 08 00 00 04
xfs_dfsbno_t  sb_logstart;   

00 00 00 00 00 00 00 80
xfs_ino_t  sb_rootino = 128 (for a 4KB block size)

00 00 00 00 00 00 00 81
xfs_ino_t  sb_rbmino = 129

00 00 00 00 00 00 00 82
xfs_ino_t  sb_rsumino = 130

00 00 00 10
xfs_agblock_t  sb_rextsize = 16 blocks

00 74 52 04
xfs_agblock_t  sb_agblocks = 7623172 blocks allocation group size

00 00 00 20
xfs_agnumber_t  sb_agcount = 32 allocation groups total

00 00 00 00
xfs_extlen_t  sb_rbmblocks = 0 blocks

00 00 80 00
xfs_extlen_t  sb_logblocks = 32768 blocks for journaling log

30 84
__uint16_t  sb_versionnum;   

02 00
__uint16_t  sb_sectsize = 512 bytes underlying disk sector size

01 00
__uint16_t  sb_inodesize = 256 (2 inodes per sector)

00 10
__uint16_t  sb_inopblock = 16 = 4096/256

00 00 00 00 00 00 00 00 00 00 00 00
char  sb_fname[12] = 12 null chars (no filesystem name given)

0C
__uint8_t  sb_blocklog = 12 = lb(sb_blocksize) = log_2(4096)

09
__uint8_t  sb_sectlog = 9 = lb(sb_sectsize) = log_2(512)

08
__uint8_t  sb_inodelog = 8 = lb(sb_inodesize) = log_2(256)

04
__uint8_t  sb_inopblog = 4 = lb(sb_inopblock) = log_2(16)

17
__uint8_t  sb_agblklog = 23 = ceil(lb(sb_agblocks)) = ceil(log_2(7623172))

00
__uint8_t  sb_rextslog;   

00
__uint8_t  sb_inprogress = false (true for disk 2 header!!?)

19
__uint8_t  sb_imax_pct = 25% of filesystem space can be used for inodes


00 00 00 00 00 04 B0 C0
__uint64_t  sb_icount = 307392 total inodes

00 00 00 00 00 00 18 ED
__uint64_t  sb_ifree = 6381 free inodes

00 00 00 00 04 47 0C 2A
__uint64_t  sb_fdblocks = 71765034 free data blocks (about 274 gig free)

00 00 00 00 00 00 00 00
__uint64_t  sb_frextents;   

00 00 00 00 00 00 00 00
xfs_ino_t  sb_uquotino;   

00 00 00 00 00 00 00 00
xfs_ino_t  sb_gquotino;   

00 00 00 00 00 00 00 02
__uint16_t  sb_qflags;   

rest of this stuff is zeros...

__uint8_t  sb_flags;   
__uint8_t  sb_shared_vn;   
xfs_extlen_t  sb_inoalignmt;   
__uint32_t  sb_unit;   
__uint32_t  sb_width;   
__uint8_t  sb_dirblklog;   
__uint8_t  sb_logsectlog;   
__uint16_t  sb_logsectsize;   
__uint32_t  sb_logsunit;   
__uint32_t  sb_features2;



according to the pdf, each allocation block should start with a superblock.
so should find another XFSB thingy at "k*sb_agblocks" away from the first one for k = 0,1,2...
the first one is at 0x3D487A00 and sb_agblocks = 7623172 blocks with 4096 byte block size
so look for another superblock at 0x3D487A00 + 7623172*4096 = 0x78268BA00 and sure enough there was one
this is about 30 gig into disk 1. i'll find the rest of the superblocks and work out how to concatenate disk 2 on so as to preserve equal spacing between superblocks!


Top
 Profile  
 
PostPosted: Thu May 28, 2009 1:17 am 
Offline

Joined: Fri Feb 06, 2009 7:49 pm
Posts: 16
wim wrote:
you must be checking that i'm paying attention.. first you say we expect an XFSB at 499592200192 in the image, then you say to subtract the size from 49959200192, then you give 49949200192 in the command! which of these three numbers did you mean? ;) i'll assume you meant 0x7452040000 = 499592200192. the filesize of my image the first time, of entire disk sdb starting from input offset 2008125b, was 499079702016 bytes. 499592200192 - 499079702016 = 512498176. i checked on the second disk at 512498176 + 3163648 = 515661824 = 1EBC6000 and did not find an XFSB thing there..that block looked like this. however, i did not bother to boot up in the live cd and do the ddrescue thing since i am able to open these physical disks directly in HxD which is much faster and easier. the nearest XFSB thing is the first one on disk 2 at 0x1EB80000. by the way, that's block 1006592 just like you asked in the original post, to answer your question a few months late.. :) it is all zeros up to there block 1006592 aswell, apart from the rubbish that windows disk manager thing has wrote into blocks 1 and 7. on disk 2, there are no more XFSB things at least up until 0x1418D4D70, thats about 5 gig in, i didn't bother searching further than that.

The "49949200192" in the command I gave was part of "49949200192-size_of_your_disk_img". The "-" was meant to be a minus sign. This is in line with the "subtract the size" instruction. When I said we expect to see it at that offset, I meant that offset in the final reconstructed image (which we don't have yet; we need to know what we're expecting so that we can properly reconstruct the image.) So, to answer your question, I meant all three of those things. :)

So we found our XFSB 286720 bytes earlier than expected. This probably means that the spanning that happens does not go all the way to the end of the disk. I would just change the offset for reading in the second disk, and also set a size cap so that our final image ends up being the right size...
Code:
ddrescue /dev/sdb /wd/ebd.img /wd/ebd1.log -i 2008125b -o 0
ddrescue /dev/sdc /wd/ebd.img /wd/ebd2.log -i 2876928 -o 499079702016 -s 500104698368

If I'm right in the assumptions I'm making here (there's quite a few of them) this should rebuild the image well enough to mount. Again, if it doesn't mount on the first try, run xfs_repair with the -n option and redirect the output to a log.
Quote:
could you explain how you arrived at the head sector cylinder locations for the second entry? if i apply the logic of the first entry, i thought i would get the first block at H0 S16 C1 and the last block at H254 S16 C63 but maybe i'm making some wrong assumptions or something..

Yes, the method of storing those addresses is a little weird. They smooshed a six-bit number and a ten-bit number into two bytes, and it's all little endian, which is why it looks so funny.

The first byte is the head number. The first six bits of the second byte are the sector number. The third byte is the most significant eight bits of ten of the cylinder number. The last two bits of the second byte are the least significant two bits of ten of the cylinder number. Make sense?

02 01 00 = 00000010 000000 01 <-> 00000000 = 00000010 000000 0000000001 = 0 10 1 = H2 S0 C1
fe 3f 0f = 11111110 001111 11 <-> 00001111 = 11111110 001111 0000111111 = 11111110 1111 111111 = H254 S15 C63
00 01 10 = 00000000 000000 01 <-> 00010000 = 00000000 000000 000100001 = 0 0 100001 = H0 S0 C65
fe 3f 10 = 11111110 001111 11 <-> 00010000 = 11111110 001111 0001000011 = 11111110 1111 1000011 = H254 S15 C67

It looks like I did make a mistake with the cylinder number for the last block on the second entry, which I previously noted as 515 instead of 67... It looks like I accidentally took the 10 and read it as 80 (moving the 1 from the least significant bit in the nybble to the most significant).


Top
 Profile  
 
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 36 posts ]  Go to page Previous  1, 2, 3  Next

All times are UTC


Who is online

Users browsing this forum: No registered users and 6 guests


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
Powered by phpBB® Forum Software © phpBB Group