StorCenter ix4-200d drives seen as new - need data recover

Arjan
Posts: 13
Joined: Wed Sep 24, 2014 7:06 pm

StorCenter ix4-200d drives seen as new - need data recover

Post by Arjan » Wed Sep 24, 2014 7:17 pm

Hey All,

As my StorCenter ix4-200d has been running for quite some time now, I decided to buy a new one. As if the Iomega had heard what I had planned, it quit working on the day the new device arrived... resulting in 3 out of 4 disks showing as 'newly inserted'.

I managed to gain ssh access to the drives using the http://192.168.1.1/support.html link and enable ssh and follow mos of this topic: viewtopic.php?f=251&t=2101
Unfortunately no real solution came out of this... so trying to blow in some new life into this... I see all partitions, but how the heck do I mount the filesystem on the Iomega and recover my data... :S

Code: Select all

cat /proc/partitions
show

Code: Select all

major minor  #blocks  name

   7     0     261573 loop0
   7     1       5120 loop1
   7     2        464 loop2
   8     0  976762584 sda
   8     1    2040254 sda1
   8     2  974722329 sda2
   8    16  976762584 sdb
   8    17    2040254 sdb1
   8    18  974722329 sdb2
   8    32  976762584 sdc
   8    33    2040254 sdc1
   8    34  974722329 sdc2
   8    48  976762584 sdd
   8    49    2040254 sdd1
   8    50  974722329 sdd2
  31     0        640 mtdblock0
  31     1         64 mtdblock1
  31     2       2192 mtdblock2
  31     3       2192 mtdblock3
  31     4      32768 mtdblock4
   9     0    2040128 md0
   9     1 2924166528 md1
 253     0 2924165120 dm-0
Hope to get some helpfull responses. There's a database on this NAS which I need to recover. Yes... I know... but it's a heritage from someone who left a long time ago and never had time/opportunity to find out which data was where.

Cheers!

Mijzelf
Posts: 6199
Joined: Mon Jun 16, 2008 10:45 am

Re: StorCenter ix4-200d drives seen as new - need data recov

Post by Mijzelf » Thu Sep 25, 2014 9:26 am

I guess you didn't tell the nas to initialize the 'newly inserted' disks?

Meanwhile I know a bit more about Iomega/EMC nasses in general. The basic disk layout can be found here. Your md1 array is assembled, according to /proc/partitions, and there is also a volume group dm-0, which is inside it. lvdisplay should show if there is still a logical volume inside. If there is none, try lvscan first.

The internal logical volume should be mounted, if it isn't already. Have a look at 'cat /proc/mounts'.

Arjan
Posts: 13
Joined: Wed Sep 24, 2014 7:06 pm

Re: StorCenter ix4-200d drives seen as new - need data recov

Post by Arjan » Fri Sep 26, 2014 7:41 am

Mijzelf wrote:I guess you didn't tell the nas to initialize the 'newly inserted' disks?

Meanwhile I know a bit more about Iomega/EMC nasses in general. The basic disk layout can be found here. Your md1 array is assembled, according to /proc/partitions, and there is also a volume group dm-0, which is inside it. lvdisplay should show if there is still a logical volume inside. If there is none, try lvscan first.

The internal logical volume should be mounted, if it isn't already. Have a look at 'cat /proc/mounts'.
Hey Mijzelf,

Thanks for the reply! Of course I have not yet intialized the 'newly inserted' drives.
I have rebooted the device, and see there's a different /proc/partitions (i copied as I didnt have acces @ home)

Code: Select all

major minor  #blocks  name

   7     0     263465 loop0
   7     1       5120 loop1
   7     2        464 loop2
   8     0  976762584 sda
   8     1    2040223 sda1
   8    16  976762584 sdb
   8    17    2040254 sdb1
   8    18  974722329 sdb2
   8    32  976762584 sdc
   8    33    2040254 sdc1
   8    34  974722329 sdc2
   8    48  976762584 sdd
   8    49    2040254 sdd1
   8    50  974722329 sdd2
  31     0        640 mtdblock0
  31     1         64 mtdblock1
  31     2       2192 mtdblock2
  31     3       2192 mtdblock3
  31     4      32768 mtdblock4
   9     0    2040128 md0
Here's the /proc/mounts

Code: Select all

rootfs / rootfs rw 0 0
/dev/root.old /initrd ext2 rw 0 0
none / tmpfs rw 0 0
/dev/md0 /boot ext2 rw,noatime 0 0
/dev/loop0 /mnt/apps ext2 ro 0 0
/dev/loop1 /etc ext2 rw,noatime 0 0
/dev/loop2 /oem cramfs ro 0 0
proc /proc proc rw 0 0
none /proc/bus/usb usbfs rw 0 0
none /proc/fs/nfsd nfsd rw 0 0
none /sys sysfs rw 0 0
devpts /dev/pts devpts rw 0 0
tmpfs /mnt/apps/lib/init/rw tmpfs rw,nosuid 0 0
tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0

Code: Select all

lvscan -v
    Finding all logical volumes
  No volume groups found
Thanks!

Mijzelf
Posts: 6199
Joined: Mon Jun 16, 2008 10:45 am

Re: StorCenter ix4-200d drives seen as new - need data recov

Post by Mijzelf » Fri Sep 26, 2014 8:14 am

I have rebooted the device, and see there's a different /proc/partitions
Do you mean the system is degrading (fast)?

Now the data raid array isn't assembled anymore. And sda2 is lacking. Can you post the output of

Code: Select all

cat /proc/mdstat
and

Code: Select all

mdadm --examine /dev/sd[abcd]2

Arjan
Posts: 13
Joined: Wed Sep 24, 2014 7:06 pm

Re: StorCenter ix4-200d drives seen as new - need data recov

Post by Arjan » Fri Sep 26, 2014 10:52 am

Hey Mijzelf,

No... I posted initial output and quicklooked. It looked the same...

Output of cat /proc/mdstat:

Code: Select all

Personalities : [linear] [raid0] [raid1] [raid10] [raid6] [raid5] [raid4] 
md1 : inactive sdb2[1] sdd2[3](S) sdc2[2]
      2924166528 blocks super 1.0
       
md0 : active raid1 sda1[0] sdd1[3] sdc1[2] sdb1[1]
      2040128 blocks [4/4] [UUUU]
      
unused devices: <none>
mdadm --examine

Code: Select all

mdadm: cannot open /dev/sda2: No such device or address

/dev/sdb2:
          Magic : a92b4efc
        Version : 1.0
    Feature Map : 0x0
     Array UUID : ef425eb8:07a58e7e:f78faf4a:bc7d80d9
           Name : storage:1
  Creation Time : Wed Sep  8 23:22:31 2010
     Raid Level : raid5
   Raid Devices : 4

 Avail Dev Size : 1949444384 (929.57 GiB 998.12 GB)
     Array Size : 5848333056 (2788.70 GiB 2994.35 GB)
  Used Dev Size : 1949444352 (929.57 GiB 998.12 GB)
   Super Offset : 1949444640 sectors
          State : clean
    Device UUID : 5c7a3b36:56ca8f3e:028e08a6:85810bf3

    Update Time : Fri Sep 19 07:16:18 2014
       Checksum : bc37e2ae - correct
         Events : 13500

         Layout : left-symmetric
     Chunk Size : 64K

    Array Slot : 1 (empty, 1, 2)
   Array State : _Uu_

/dev/sdc2:
          Magic : a92b4efc
        Version : 1.0
    Feature Map : 0x0
     Array UUID : ef425eb8:07a58e7e:f78faf4a:bc7d80d9
           Name : storage:1
  Creation Time : Wed Sep  8 23:22:31 2010
     Raid Level : raid5
   Raid Devices : 4

 Avail Dev Size : 1949444384 (929.57 GiB 998.12 GB)
     Array Size : 5848333056 (2788.70 GiB 2994.35 GB)
  Used Dev Size : 1949444352 (929.57 GiB 998.12 GB)
   Super Offset : 1949444640 sectors
          State : clean
    Device UUID : 9616f4b2:141a613b:6794ac24:2c294595

    Update Time : Fri Sep 19 07:16:18 2014
       Checksum : 569f7cb3 - correct
         Events : 13500

         Layout : left-symmetric
     Chunk Size : 64K

    Array Slot : 2 (empty, 1, 2)
   Array State : _uU_

/dev/sdd2:
          Magic : a92b4efc
        Version : 1.0
    Feature Map : 0x0
     Array UUID : ef425eb8:07a58e7e:f78faf4a:bc7d80d9
           Name : storage:1
  Creation Time : Wed Sep  8 23:22:31 2010
     Raid Level : raid5
   Raid Devices : 4

 Avail Dev Size : 1949444384 (929.57 GiB 998.12 GB)
     Array Size : 5848333056 (2788.70 GiB 2994.35 GB)
  Used Dev Size : 1949444352 (929.57 GiB 998.12 GB)
   Super Offset : 1949444640 sectors
          State : clean
    Device UUID : e9a1963b:81500f2d:d6c2f979:196b5f02

    Update Time : Fri Sep 19 07:16:18 2014
       Checksum : 9357aecf - correct
         Events : 13500

         Layout : left-symmetric
     Chunk Size : 64K

    Array Slot : 3 (empty, 1, 2)
   Array State : _uu_
It does indeed look like there's no /dev/sda2, but all four drives are present.
This is the output from the web interface:

Disk has been replaced with a disk containing data from another system. Disk 4 (ST31000520AS, CC38 ) 932 GB
Disk has been replaced with a disk containing data from another system. Disk 3 (ST31000520AS, CC38 ) 932 GB
Disk has been replaced with a disk containing data from another system. Disk 2 (ST31000520AS, CC38 ) 932 GB
Disk is operating normally. Disk 1 (ST31000520AS, CC38 ) 932 GB

Mijzelf
Posts: 6199
Joined: Mon Jun 16, 2008 10:45 am

Re: StorCenter ix4-200d drives seen as new - need data recov

Post by Mijzelf » Fri Sep 26, 2014 11:44 am

I wonder why the box concluded that 3 new disks were inserted.

Anyway, your raid1 md0 is alive on 4 partitions, so I guess you have no serious damage on one of the disks.

The md1 array is degraded. sda2 is missing, but it should be able to assemble it with only 3 members, due to redundency. But for some reason sdd2 is marked as 'spare'. Your initial post showed a 3TB raid array, so I guess there shouldn't be a spare. (A non-redundant array with a hot spare is a bit silly).

It is possible to enforce sdd2 back in the array (assuming there a no bad sectors on strategic places), but I think we should first try to get sda2 back. That should be easy, just copy the partition table of another disk:

Code: Select all

su
dd if=/dev/sdb of=/dev/sda bs=512 count=1
After that, reboot, and repost the /proc/mdstat and mdadm outputs.

BTW, does dmesg show any I/O errors?

Arjan
Posts: 13
Joined: Wed Sep 24, 2014 7:06 pm

Re: StorCenter ix4-200d drives seen as new - need data recov

Post by Arjan » Fri Sep 26, 2014 12:06 pm

Mijzelf wrote:I wonder why the box concluded that 3 new disks were inserted.

Anyway, your raid1 md0 is alive on 4 partitions, so I guess you have no serious damage on one of the disks.

The md1 array is degraded. sda2 is missing, but it should be able to assemble it with only 3 members, due to redundency. But for some reason sdd2 is marked as 'spare'. Your initial post showed a 3TB raid array, so I guess there shouldn't be a spare. (A non-redundant array with a hot spare is a bit silly).

It is possible to enforce sdd2 back in the array (assuming there a no bad sectors on strategic places), but I think we should first try to get sda2 back. That should be easy, just copy the partition table of another disk:

Code: Select all

su
dd if=/dev/sdb of=/dev/sda bs=512 count=1
After that, reboot, and repost the /proc/mdstat and mdadm outputs.

BTW, does dmesg show any I/O errors?
Thanks for the quick replies man! Really, really appreciate this!

Okay, Running the dd if=/dev/sdb of=/dev/sda bs=512 count=1

Code: Select all

1+0 records in
1+0 records out
512 bytes (512 B) copied, 8.5955e-05 s, 6.0 MB/s
Reboot the device. Done.
Now there's another issue... The NAS aint booting anymore and shows image like in the attached jpg... :roll:
You do not have the required permissions to view the files attached to this post.

Arjan
Posts: 13
Joined: Wed Sep 24, 2014 7:06 pm

Re: StorCenter ix4-200d drives seen as new - need data recov

Post by Arjan » Fri Sep 26, 2014 12:12 pm

It's a firmware update thingy... did we break it by issueing the dd command?
I can't imagine. We only copied the MBR, containing the partition table, and 400 bytes bootcode. But the bootcode part is not in use.

It is possible that now the data partition is assembled, and it is doing some delayed maintenance?

A google brought me here, and it seems this picture is the result of a reset button press.
Press and hold the reset button on the rear panel of the unit

Power on your ix2-200 or x4-200d

Continue holding the reset button while powering on the device

After about 1 minute, a picture of a USB flash drive will show on the front LCD display. At this point release the reset button.
(USB flash drive? My first thought was an empty battery).

So, is it possible that reset button is pressed?

Mijzelf
Posts: 6199
Joined: Mon Jun 16, 2008 10:45 am

Re: StorCenter ix4-200d drives seen as new - need data recov

Post by Mijzelf » Fri Sep 26, 2014 12:48 pm

Oops. I edited you message, instead of replying. Sorry.

Arjan
Posts: 13
Joined: Wed Sep 24, 2014 7:06 pm

Re: StorCenter ix4-200d drives seen as new - need data recov

Post by Arjan » Fri Sep 26, 2014 12:57 pm

Editing, the favor of being Admin :)

No, I did not press reset. I issued "reboot" from the commandline.
Neither I think it is delayed maintenance, as it has rebooted several times already...
It keeps rebooting after a while on the 'update' screen.

Any thoughts?

Arjan
Posts: 13
Joined: Wed Sep 24, 2014 7:06 pm

Re: StorCenter ix4-200d drives seen as new - need data recov

Post by Arjan » Fri Sep 26, 2014 1:26 pm

Hey Mijzelf,

Think I am going to fine myself a box that fits the drives and install a Ubuntu in it. Then connect the drives and see if I can rebuild md1 and recover some data.
The Iomega is in an ininite firmware loop now...

Feel free to add additional options ;)

Arjan
Posts: 13
Joined: Wed Sep 24, 2014 7:06 pm

Re: StorCenter ix4-200d drives seen as new - need data recov

Post by Arjan » Fri Sep 26, 2014 3:24 pm

Hey Mijzelf,

So... I installed a ubuntu machine.
cat /proc/partitions:
cat /proc/partitions

Code: Select all

major minor  #blocks  name

   8        0  117220824 sda
   8        1     104391 sda1
   8        2    2096482 sda2
   8        3  115017367 sda3
   8       32  976762584 sdc
   8       33    2040254 sdc1
   8       34  974722329 sdc2
   8       48  976762584 sdd
   8       49    2040254 sdd1
   8       50  974722329 sdd2
   8       16  976762584 sdb
   8       17    2040254 sdb1
   8       18  974722329 sdb2
   8       64  976762584 sde
   8       65    2040254 sde1
   8       66  974722329 sde2
did a mdadm --stop /dev/md0

then mdadm --assemble --scan -v --force
md0 and md1 were made but cannot mount the md1

Code: Select all

mount /dev/md1 /mnt/md1
mount: /dev/md1: can't read superblock
Will have to let it rest for the weekend... and dig through some backup locations... I am now at the point to give up and toss this bugger out the window!

Mijzelf
Posts: 6199
Joined: Mon Jun 16, 2008 10:45 am

Re: StorCenter ix4-200d drives seen as new - need data recov

Post by Mijzelf » Fri Sep 26, 2014 3:32 pm

/dev/md1 cannot be mounted. It contains a 'volume group' and a 'logical volume'. (No, no idea why). So you have to

Code: Select all

vgscan
vgdisplay
The internal volume can be mounted. I hope.

What is the output of

Code: Select all

cat /proc/mdstat
now?

Arjan
Posts: 13
Joined: Wed Sep 24, 2014 7:06 pm

Re: StorCenter ix4-200d drives seen as new - need data recov

Post by Arjan » Mon Sep 29, 2014 7:42 am

Goodmorning :)

Here's the output

Code: Select all

root@ubuntutest:~# vgscan
  Reading all physical volumes.  This may take a while...
  No volume groups found

Code: Select all

root@ubuntutest:~# vgdisplay
  No volume groups found

Code: Select all

root@ubuntutest:~# cat /proc/mdstat
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] 
md1 : inactive sdb2[1](S) sdc2[0](S) sdd2[2](S) sde2[3](S)
      3898888768 blocks super 1.0
       
md0 : active raid1 sdb1[1] sdd1[2] sde1[3]
      2040128 blocks [4/3] [_UUU]
      
unused devices: <none>
no LV's on the system

::edit::

Code: Select all

fdisk -l

Disk /dev/sda: 120.0 GB, 120034123776 bytes
255 heads, 63 sectors/track, 14593 cylinders, total 234441648 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x000170cb

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1   *          63      208844      104391   83  Linux
/dev/sda2          208845     4401809     2096482+  82  Linux swap / Solaris
/dev/sda3         4401810   234436544   115017367+  83  Linux

Disk /dev/sdd: 1000.2 GB, 1000204886016 bytes
255 heads, 63 sectors/track, 121601 cylinders, total 1953525168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x000ecf9f

   Device Boot      Start         End      Blocks   Id  System
/dev/sdd1               1     4080509     2040254+  83  Linux
/dev/sdd2         4080510  1953525167   974722329   83  Linux

Disk /dev/sdc: 1000.2 GB, 1000204886016 bytes
255 heads, 63 sectors/track, 121601 cylinders, total 1953525168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x0007d232

   Device Boot      Start         End      Blocks   Id  System
/dev/sdc1               1     4080509     2040254+  83  Linux
/dev/sdc2         4080510  1953525167   974722329   83  Linux

Disk /dev/sdb: 1000.2 GB, 1000204886016 bytes
255 heads, 63 sectors/track, 121601 cylinders, total 1953525168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x0007d232

   Device Boot      Start         End      Blocks   Id  System
/dev/sdb1               1     4080509     2040254+  83  Linux
/dev/sdb2         4080510  1953525167   974722329   83  Linux

Disk /dev/sde: 1000.2 GB, 1000204886016 bytes
255 heads, 63 sectors/track, 121601 cylinders, total 1953525168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00067f9f

   Device Boot      Start         End      Blocks   Id  System
/dev/sde1               1     4080509     2040254+  83  Linux
/dev/sde2         4080510  1953525167   974722329   83  Linux

Disk /dev/md0: 2089 MB, 2089091072 bytes
2 heads, 4 sectors/track, 510032 cylinders, total 4080256 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

Disk /dev/md0 doesn't contain a valid partition table

Mijzelf
Posts: 6199
Joined: Mon Jun 16, 2008 10:45 am

Re: StorCenter ix4-200d drives seen as new - need data recov

Post by Mijzelf » Mon Sep 29, 2014 8:20 am

md1 : inactive
md1 is inactive. That is strange, as it has 4 members. You can try to force it:

Code: Select all

mdadm --stop /dev/md1
mdadm --assemble /dev/md1 /dev/sd[bcde]2 --run

Post Reply