Everything looks to work fine but red light blinking

StorCenter ix2-200 Network Storage
Mijzelf
Posts: 6224
Joined: Mon Jun 16, 2008 10:45 am

Re: Everything looks to work fine but red light blinking

Post by Mijzelf » Tue Jan 07, 2014 8:59 pm

'Cannot open /dev/sdb2 for write' is (or can be) something different as '/dev/sdb2 is readonly'. I guess mdadm needs exclusive write access, to be not disturbed, and to be sure it won't disturb anything else.
Seeing the dmesg log, it looks to me that somehow sdb2 is still being accessed as member of the raid array, so an exclusive lock is impossible.

That's why I asked to reboot. I can imagine the raid member lingers somehow, but I guess it won't be touched on a re-assemble after a reboot.

eyp
Posts: 44
Joined: Thu Aug 09, 2012 11:15 am

Re: Everything looks to work fine but red light blinking

Post by eyp » Tue Jan 07, 2014 9:03 pm

Then I'm going to reboot it now.

eyp
Posts: 44
Joined: Thu Aug 09, 2012 11:15 am

Re: Everything looks to work fine but red light blinking

Post by eyp » Tue Jan 07, 2014 9:31 pm

After rebooting, the system doen't look any good :(
cat /proc/mdstat
Personalities : [linear] [raid0] [raid1] [raid6] [raid5] [raid4]
md1 : inactive sda2[2]
1932533488 blocks super 1.0

md0 : active raid1 sda1[1]
20980800 blocks [2/1] [_U]

unused devices: <none>
The other array is missing a device, I don't understand why. And the first one (md1) is totally inactive :?

Mijzelf
Posts: 6224
Joined: Mon Jun 16, 2008 10:45 am

Re: Everything looks to work fine but red light blinking

Post by Mijzelf » Tue Jan 07, 2014 9:33 pm

What does

Code: Select all

cat /proc/partitions
say?

eyp
Posts: 44
Joined: Thu Aug 09, 2012 11:15 am

Re: Everything looks to work fine but red light blinking

Post by eyp » Tue Jan 07, 2014 9:34 pm

sbd is missing:
cat /proc/partitions
major minor #blocks name

7 0 604505 loop0
7 1 8192 loop1
7 2 204 loop2
8 0 1953514584 sda
8 1 20980888 sda1
8 2 1932533625 sda2
31 0 768 mtdblock0
31 1 16 mtdblock1
31 2 3072 mtdblock2
31 3 3072 mtdblock3
31 4 32768 mtdblock4
9 0 20980800 md0
253 0 4194304 dm-0
253 1 16785408 dm-1

Mijzelf
Posts: 6224
Joined: Mon Jun 16, 2008 10:45 am

Re: Everything looks to work fine but red light blinking

Post by Mijzelf » Tue Jan 07, 2014 9:39 pm

OK. So meanwhile either sdb died completely, or the sata port did. (Or the power supply for the 2nd disk, ...)

Do you have the possibility to test that disk on another device?

And what does

Code: Select all

mdadm --examine /dev/sda2
say? Do you have any data on the system?

eyp
Posts: 44
Joined: Thu Aug 09, 2012 11:15 am

Re: Everything looks to work fine but red light blinking

Post by eyp » Tue Jan 07, 2014 9:46 pm

Now I can't test it on another device, I must buy a usb-hdbox to put the disk in. Anyway:
mdadm --examine /dev/sda2
/dev/sda2:
Magic : a92b4efc
Version : 1.0
Feature Map : 0x2
Array UUID : cba54300:251e4d89:9dbf101c:f04b7735
Name : ix2-200-D8S0EN:1 (local to host ix2-200-D8S0EN)
Creation Time : Fri Jun 8 03:34:59 2012
Raid Level : raid1
Raid Devices : 2

Avail Dev Size : 3865066976 (1843.01 GiB 1978.91 GB)
Array Size : 3865066976 (1843.01 GiB 1978.91 GB)
Super Offset : 3865067232 sectors
Recovery Offset : 970690816 sectors
State : clean
Device UUID : e33b3e1d:43d248f6:f10ae7a2:daeb92ac

Update Time : Tue Jan 7 22:18:54 2014
Checksum : 32dfee19 - correct
Events : 1280087


Array Slot : 2 (0, failed, 1)
Array State : uU 1 failed
How can I know which of the two disks is /dev/sdb?. Something bad could happen if I remove the disks from the box? should I do anything before I do it? or before I put them back in again?

Furthermore it seems that I've lost all the data, there's nothing at /mnt/pools/A/A0 but a Torrent directory.

This is the status of the arrays:
mdadm --detail /dev/md1
/dev/md1:
Version : 01.00
Creation Time : Fri Jun 8 03:34:59 2012
Raid Level : raid1
Used Dev Size : 1932533488 (1843.01 GiB 1978.91 GB)
Raid Devices : 2
Total Devices : 1
Preferred Minor : 1
Persistence : Superblock is persistent

Update Time : Tue Jan 7 22:18:54 2014
State : active, degraded, Not Started
Active Devices : 0
Working Devices : 1
Failed Devices : 0
Spare Devices : 1

Number Major Minor RaidDevice State
0 0 0 0 removed
2 8 2 1 spare rebuilding /dev/sda2
mdadm --detail /dev/md0
/dev/md0:
Version : 00.90
Creation Time : Fri Jun 8 03:23:38 2012
Raid Level : raid1
Array Size : 20980800 (20.01 GiB 21.48 GB)
Used Dev Size : 20980800 (20.01 GiB 21.48 GB)
Raid Devices : 2
Total Devices : 1
Preferred Minor : 0
Persistence : Superblock is persistent

Update Time : Tue Jan 7 22:30:00 2014
State : clean, degraded
Active Devices : 1
Working Devices : 1
Failed Devices : 0
Spare Devices : 0

UUID : bb7db4dd:fdee1f62:a266f1a9:a2250747
Events : 0.5940848

Number Major Minor RaidDevice State
0 0 0 0 removed
1 8 1 1 active sync /dev/sda1

eyp
Posts: 44
Joined: Thu Aug 09, 2012 11:15 am

Re: Everything looks to work fine but red light blinking

Post by eyp » Tue Jan 07, 2014 10:21 pm

As I can't do nothing, I've stopped md1, but it can't run again.
root@ix2-200-D8S0EN:/# mdadm --stop /dev/md1
mdadm: stopped /dev/md1
root@ix2-200-D8S0EN:/# mdadm --run /dev/md1
mdadm: failed to run array /dev/md1: Invalid argument
Then I've tried to create again the array, but it has been unsuccesful
root@ix2-200-D8S0EN:/# mdadm --create --verbose /dev/md1 --metadata 1 --level=mirror --raid-devices=2 /dev/sda2 /dev/sdb2
mdadm: /dev/sda2 appears to be part of a raid array:
level=raid1 devices=2 ctime=Fri Jun 8 03:34:59 2012
mdadm: Cannot open /dev/sdb2: No such device or address
mdadm: create aborted
So I've created without /dev/sdb2
root@ix2-200-D8S0EN:/# mdadm --create --verbose /dev/md1 --level=mirror --raid-devices=2 /dev/sda2 missing
mdadm: /dev/sda2 appears to be part of a raid array:
level=raid1 devices=2 ctime=Fri Jun 8 03:34:59 2012
mdadm: size set to 1932533504K
Continue creating array? y
mdadm: array /dev/md1 started.

Mijzelf
Posts: 6224
Joined: Mon Jun 16, 2008 10:45 am

Re: Everything looks to work fine but red light blinking

Post by Mijzelf » Wed Jan 08, 2014 9:29 am

How can I know which of the two disks is /dev/sdb?. Something bad could happen if I remove the disks from the box? should I do anything before I do it? or before I put them back in again?
Try and error. And no, it's no problem if you pull the wrong one. As the box stores it's history only on disk, it won't 'remember' that you pulled it. (Just don't forget to shutdown and unpower before pulling or putting back a disk.)
Furthermore it seems that I've lost all the data, there's nothing at /mnt/pools/A/A0 but a Torrent directory.
Your data is on md1, which is inactive. Beats me why. Your mdadm --examine and .. --detail shows that the array is healty, yet degraded, but not active. Maybe a timing issue? Maybe the dead disk took that amount of kernel time that the array wasn't put active?
As I can't do nothing, I've stopped md1, but it can't run again.
The command for running is --assemble.
Then I've tried to create again the array, but it has been unsuccesful
Ouch. That was not a good idea. If for some reason the orginal array was created with non-default parameters, you have overwritten part of the payload.
But maybe it's fine, and does the data just come back after a restart.

eyp
Posts: 44
Joined: Thu Aug 09, 2012 11:15 am

Re: Everything looks to work fine but red light blinking

Post by eyp » Wed Jan 08, 2014 9:49 am

Thank you Mijzelf, I'll do some work this afternoon with the array.

One question. If I buy another 2TB disk, which steps must I do to get the array working and reconstructed again?

Mijzelf
Posts: 6224
Joined: Mon Jun 16, 2008 10:45 am

Re: Everything looks to work fine but red light blinking

Post by Mijzelf » Wed Jan 08, 2014 1:38 pm

eyp wrote:If I buy another 2TB disk, which steps must I do to get the array working and reconstructed again?
In theory you should just put an empty disk in, in combination with the degraded one, and the system should rebuild. Maybe after confirmation in the webinterface.

eyp
Posts: 44
Joined: Thu Aug 09, 2012 11:15 am

Re: Everything looks to work fine but red light blinking

Post by eyp » Wed Jan 08, 2014 7:00 pm

Fortunately now I have all the data, I don't know what happened but at least I can access everything again, but the other HD as you said might died.

Is it possible to put the healthy disk on another linux NAS of another brand, then add another healthy disk and rebuild the array?, or everything is very dependent on the firmware of the Iomega device?

Mijzelf
Posts: 6224
Joined: Mon Jun 16, 2008 10:45 am

Re: Everything looks to work fine but red light blinking

Post by Mijzelf » Wed Jan 08, 2014 9:45 pm

The disk contains two standard md raid containers. You can add an empty disk on any Linux system. Clone the partition table, assemble (NOT create) the raid arrays, and add the new partitons.
However, after rebuilding you can't easily mount the raid arrays, as they contain Logical Volumes, which first have to be initialized.

jambon
Posts: 7
Joined: Sun Mar 24, 2013 4:02 am

Re: Everything looks to work fine but red light blinking

Post by jambon » Fri Jun 10, 2016 12:51 pm

Thanks for this post guys, I manage to make it work thanks to the first page of this post (luck me I haven't to go through the rest)
The faulty disk was the HDD1 in the ix2 slot. I had this kind of issue before but unplug and replug it was rebuilding the disk automatically but not this time.

Just want to share what I did in case someone (might be me again) got same issue:

Code: Select all

root@ix2-2:/ $ dmesg -c >/dev/null
root@ix2-2:/ $ mdadm --manage /dev/md1 --add /dev/sdb2
root@ix2-2:/ $ dmesg
results show some results

Code: Select all

Buffer I/O error on device sdb2, logical block 3865020506
Buffer I/O error on device sdb2, logical block 3865020507
Buffer I/O error on device sdb2, logical block 3865020508
Buffer I/O error on device sdb2, logical block 3865020509
I decided to follow advise on the post and set to zero the 100 blocks from 3865020500 to 3865020600

Code: Select all

root@ix2-2:/ $ dd if=/dev/zero of=/dev/sdb2 seek=3865020500 count=100
after this command, it started rebuilding

Code: Select all

root@ix2-2:/ $ cat /proc/mdstat
Personalities : [linear] [raid0] [raid1] [raid10] [raid6] [raid5] [raid4]
md1 : active raid1 sdb2[2] sda2[1]
      1932510124 blocks super 1.0 [2/1] [_U]
      [>....................]  recovery =  0.2% (4464512/1932510124) finish=241.6min speed=132985K/sec

md0 : active raid1 sdb1[0] sda1[1]
      20971456 blocks [2/2] [UU]

unused devices: <none>
I wait a whole night for the rebuild to finish but the red light was still blinking
I decided to reboot the ix2 and all good after that.

Post Reply