Everything looks to work fine but red light blinking

[eyp]

Well, the recovery has stopped suddenly without finishing (I guess), but now I can check the details of the device:

mdadm --examine /dev/sdb2
/dev/sdb2:
Magic : a92b4efc
Version : 1.0
Feature Map : 0x2
Array UUID : cba54300:251e4d89:9dbf101c:f04b7735
Name : ix2-200-D8S0EN:1 (local to host ix2-200-D8S0EN)
Creation Time : Fri Jun 8 03:34:59 2012
Raid Level : raid1
Raid Devices : 2

Avail Dev Size : 3865066976 (1843.01 GiB 1978.91 GB)
Array Size : 3865066976 (1843.01 GiB 1978.91 GB)
Super Offset : 3865067232 sectors
Recovery Offset : 970693888 sectors
State : clean
Device UUID : e33b3e1d:43d248f6:f10ae7a2:daeb92ac

Update Time : Sat Jan 4 16:14:48 2014
Checksum : 32dbaffe - correct
Events : 1280018


Array Slot : 2 (0, failed, 1)
Array State : uU 1 failed

But the Array Slot says 'failed'. Should I mark it as failed, remove it and re-add it againg to get the array to reconstruct the array?

The array detail is:

mdadm --detail /dev/md1
/dev/md1:
Version : 01.00
Creation Time : Fri Jun 8 03:34:59 2012
Raid Level : raid1
Array Size : 1932533488 (1843.01 GiB 1978.91 GB)
Used Dev Size : 3865066976 (3686.02 GiB 3957.83 GB)
Raid Devices : 2
Total Devices : 2
Preferred Minor : 1
Persistence : Superblock is persistent

Update Time : Sat Jan 4 16:14:48 2014
State : clean, degraded
Active Devices : 1
Working Devices : 2
Failed Devices : 0
Spare Devices : 1

Name : ix2-200-D8S0EN:1 (local to host ix2-200-D8S0EN)
UUID : cba54300:251e4d89:9dbf101c:f04b7735
Events : 1280018

Number Major Minor RaidDevice State
0 8 2 0 active sync /dev/sda2
2 8 18 1 spare rebuilding /dev/sdb2

Well, after mark it as failed, it has started the rebuilding again:

mdadm --manage /dev/md1 --fail /dev/sdb2
mdadm: set /dev/sdb2 faulty in /dev/md1
root@ix2-200-D8S0EN:/# cat /proc/mdstat
Personalities : [linear] [raid0] [raid1] [raid6] [raid5] [raid4]
md1 : active raid1 sdb2[2] sda2[0]
1932533488 blocks super 1.0 [2/1] [U_]
[>....................] recovery = 0.0% (873536/1932533488) finish=368.5min speed=87353K/sec

md0 : active raid1 sda1[0] sdb1[1]
20980800 blocks [2/2] [UU]

[eyp]

Na, the recovery finishes (not at 100%), then the device becomes 'spare' again.

[Mijzelf]

Did you have a look in dmesg if still I/O errors are occurring? And did you look if smartctl is available?

[eyp]

dmesg doesn't say anythig:

dmesg -c >/dev/null
root@ix2-200-D8S0EN:/# mdadm --manage /dev/md1 --add /dev/sdb2
mdadm: Cannot open /dev/sdb2: Device or resource busy
root@ix2-200-D8S0EN:/# dmesg
root@ix2-200-D8S0EN:/# dmesg

And yes, smartctl is available, but I don't know how to use it.

[Mijzelf]

dmesg shows the kernel log buffer. You can call it any time and see the kernel logs of the last minuttes/hours. 'dmesg -c' clears the buffer. In this case you could run

Code: Select all

dmesg |  grep "I/O"

while the rebuild is running, to see if I/O errors are occurring.

For smartctl you'll have to use

Code: Select all

smartctl --all /dev/sdb

or maybe

Code: Select all

smartctl -d marvell --all /dev/sdb

[eyp]

with the marvell option it does something, but I don't understand anything

smartctl -d marvell --all /dev/sdb
smartctl 5.40 2010-10-16 r3189 [arm-none-linux-gnueabi] (local build)
Copyright (C) 2002-10 by Bruce Allen, http://smartmontools.sourceforge.net

=== START OF INFORMATION SECTION ===
Device Model: ST2000DM001-9YN164
Serial Number: S1E08YN5
Firmware Version: CC4B
User Capacity: 2,000,398,934,016 bytes
Device is: Not in smartctl database [for details use: -P showall]
ATA Version is: 8
ATA Standard is: ATA-8-ACS revision 4
Local Time is: Sun Jan 5 18:01:28 2014 CET
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
See vendor-specific Attribute list for marginal Attributes.

General SMART Values:
Offline data collection status: (0x00) Offline data collection activity
was never started.
Auto Offline Data Collection: Disabled.
Self-test execution status: ( 0) The previous self-test routine completed
without error or no self-test has ever
been run.
Total time to complete Offline
data collection: ( 584) seconds.
Offline data collection
capabilities: (0x73) SMART execute Offline immediate.
Auto Offline data collection on/off support.
Suspend Offline collection upon new
command.
No Offline surface scan supported.
Self-test supported.
Conveyance Self-test supported.
Selective Self-test supported.
SMART capabilities: (0x0003) Saves SMART data before entering
power-saving mode.
Supports SMART auto save timer.
Error logging capability: (0x01) Error logging supported.
General Purpose Logging supported.
Short self-test routine
recommended polling time: ( 1) minutes.
Extended self-test routine
recommended polling time: ( 225) minutes.
Conveyance self-test routine
recommended polling time: ( 2) minutes.
SCT capabilities: (0x3085) SCT Status supported.

SMART Attributes Data Structure revision number: 10
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x000f 117 099 006 Pre-fail Always - 138070846
3 Spin_Up_Time 0x0003 094 094 000 Pre-fail Always - 0
4 Start_Stop_Count 0x0032 099 099 020 Old_age Always - 1265
5 Reallocated_Sector_Ct 0x0033 100 100 036 Pre-fail Always - 0
7 Seek_Error_Rate 0x000f 081 060 030 Pre-fail Always - 130340756
9 Power_On_Hours 0x0032 086 086 000 Old_age Always - 12339
10 Spin_Retry_Count 0x0013 100 100 097 Pre-fail Always - 0
12 Power_Cycle_Count 0x0032 100 100 020 Old_age Always - 15
183 Runtime_Bad_Block 0x0032 100 100 000 Old_age Always - 0
184 End-to-End_Error 0x0032 100 100 099 Old_age Always - 0
187 Reported_Uncorrect 0x0032 099 099 000 Old_age Always - 1
188 Command_Timeout 0x0032 100 099 000 Old_age Always - 4295032833
189 High_Fly_Writes 0x003a 100 100 000 Old_age Always - 0
190 Airflow_Temperature_Cel 0x0022 058 042 045 Old_age Always In_the_past 42 (26 142 53 39)
191 G-Sense_Error_Rate 0x0032 100 100 000 Old_age Always - 0
192 Power-Off_Retract_Count 0x0032 100 100 000 Old_age Always - 14
193 Load_Cycle_Count 0x0032 085 085 000 Old_age Always - 31793
194 Temperature_Celsius 0x0022 042 058 000 Old_age Always - 42 (0 20 0 0)
197 Current_Pending_Sector 0x0012 100 100 000 Old_age Always - 0
198 Offline_Uncorrectable 0x0010 100 100 000 Old_age Offline - 0
199 UDMA_CRC_Error_Count 0x003e 200 200 000 Old_age Always - 0
240 Head_Flying_Hours 0x0000 100 253 000 Old_age Offline - 192521909050365
241 Total_LBAs_Written 0x0000 100 253 000 Old_age Offline - 3344956179065
242 Total_LBAs_Read 0x0000 100 253 000 Old_age Offline - 264098957423622

SMART Error Log Version: 1
ATA Error Count: 3792 (device log contains only the most recent five errors)
CR = Command Register [HEX]
FR = Features Register [HEX]
SC = Sector Count Register [HEX]
SN = Sector Number Register [HEX]
CL = Cylinder Low Register [HEX]
CH = Cylinder High Register [HEX]
DH = Device/Head Register [HEX]
DC = Device Command Register [HEX]
ER = Error register [HEX]
ST = Status register [HEX]
Powered_Up_Time is measured from power on, and printed as
DDd+hh:mm:SS.sss where DD=days, hh=hours, mm=minutes,
SS=sec, and sss=millisec. It "wraps" after 49.710 days.

Error 3792 occurred at disk power-on lifetime: 12309 hours (512 days + 21 hours)
When the command that caused the error occurred, the device was active or idle.

After command completion occurred, registers were:
ER ST SC SN CL CH DH
-- -- -- -- -- -- --
40 51 00 ff ff ff 0f

Commands leading to the command that caused the error were:
CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name
-- -- -- -- -- -- -- -- ---------------- --------------------
60 00 06 ff ff ff 4f 00 16:26:25.281 READ FPDMA QUEUED
2f 00 01 10 00 00 00 00 16:26:25.095 READ LOG EXT
60 00 06 ff ff ff 4f 00 16:26:24.464 READ FPDMA QUEUED
2f 00 01 10 00 00 00 00 16:26:24.279 READ LOG EXT
60 00 06 ff ff ff 4f 00 16:26:23.647 READ FPDMA QUEUED

Error 3791 occurred at disk power-on lifetime: 12309 hours (512 days + 21 hours)
When the command that caused the error occurred, the device was active or idle.

After command completion occurred, registers were:
ER ST SC SN CL CH DH
-- -- -- -- -- -- --
40 51 00 ff ff ff 0f

Commands leading to the command that caused the error were:
CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name
-- -- -- -- -- -- -- -- ---------------- --------------------
60 00 06 ff ff ff 4f 00 16:26:25.281 READ FPDMA QUEUED
2f 00 01 10 00 00 00 00 16:26:25.095 READ LOG EXT
60 00 06 ff ff ff 4f 00 16:26:24.464 READ FPDMA QUEUED
2f 00 01 10 00 00 00 00 16:26:24.279 READ LOG EXT
60 00 06 ff ff ff 4f 00 16:26:23.647 READ FPDMA QUEUED

Error 3790 occurred at disk power-on lifetime: 12309 hours (512 days + 21 hours)
When the command that caused the error occurred, the device was active or idle.

After command completion occurred, registers were:
ER ST SC SN CL CH DH
-- -- -- -- -- -- --
40 51 00 ff ff ff 0f

Commands leading to the command that caused the error were:
CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name
-- -- -- -- -- -- -- -- ---------------- --------------------
60 00 06 ff ff ff 4f 00 16:26:24.464 READ FPDMA QUEUED
2f 00 01 10 00 00 00 00 16:26:24.279 READ LOG EXT
60 00 06 ff ff ff 4f 00 16:26:23.647 READ FPDMA QUEUED
2f 00 01 10 00 00 00 00 16:26:23.462 READ LOG EXT
60 00 06 ff ff ff 4f 00 16:26:22.856 READ FPDMA QUEUED

Error 3789 occurred at disk power-on lifetime: 12309 hours (512 days + 21 hours)
When the command that caused the error occurred, the device was active or idle.

After command completion occurred, registers were:
ER ST SC SN CL CH DH
-- -- -- -- -- -- --
40 51 00 ff ff ff 0f

Commands leading to the command that caused the error were:
CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name
-- -- -- -- -- -- -- -- ---------------- --------------------
60 00 06 ff ff ff 4f 00 16:26:24.464 READ FPDMA QUEUED
2f 00 01 10 00 00 00 00 16:26:24.279 READ LOG EXT
60 00 06 ff ff ff 4f 00 16:26:23.647 READ FPDMA QUEUED
2f 00 01 10 00 00 00 00 16:26:23.462 READ LOG EXT
60 00 06 ff ff ff 4f 00 16:26:22.856 READ FPDMA QUEUED

Error 3788 occurred at disk power-on lifetime: 12309 hours (512 days + 21 hours)
When the command that caused the error occurred, the device was active or idle.

After command completion occurred, registers were:
ER ST SC SN CL CH DH
-- -- -- -- -- -- --
40 51 00 ff ff ff 0f

Commands leading to the command that caused the error were:
CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name
-- -- -- -- -- -- -- -- ---------------- --------------------
60 00 06 ff ff ff 4f 00 16:26:23.647 READ FPDMA QUEUED
2f 00 01 10 00 00 00 00 16:26:23.462 READ LOG EXT
60 00 06 ff ff ff 4f 00 16:26:22.856 READ FPDMA QUEUED
2f 00 01 10 00 00 00 00 16:26:22.670 READ LOG EXT
60 00 06 ff ff ff 4f 00 16:26:22.039 READ FPDMA QUEUED

SMART Self-test log structure revision number 1
No self-tests have been logged. [To run self-tests, use: smartctl -t]


SMART Selective self-test log data structure revision number 1
SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS
1 0 0 Not_testing
2 0 0 Not_testing
3 0 0 Not_testing
4 0 0 Not_testing
5 0 0 Not_testing
Selective self-test flags (0x0):
After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

[Mijzelf]

Actually this doesn't look bad.

197 Current_Pending_Sector 0x0012 100 100 000 Old_age Always - 0

The number of sectors from which the firmware is aware they are wrong. They can only be exchange after a write operation.

5 Reallocated_Sector_Ct 0x0033 100 100 036 Pre-fail Always - 0

The number of sectors which are already replaced this way.

199 UDMA_CRC_Error_Count 0x003e 200 200 000 Old_age Always - 0

Number of errors cause by bad cabling.
(in all this cases the zero at the end is the actual value.

SMART overall-health self-assessment test result: PASSED

The big total. The disk is healthy.

The bad news is that then I can't explain where the bad sectors are gone. You 'resetted' them by dd'ing zeros, but the replace count is still zero.
You didn't exchange the disks, did you? In that case we are looking to the wrong disk.

[eyp]

The bad news is that then I can't explain where the bad sectors are gone. You 'resetted' them by dd'ing zeros, but the replace count is still zero.
You didn't exchange the disks, did you? In that case we are looking to the wrong disk.

No, I've never touch the disks. But I don't understand why do you say it's the wrong
disk, /dev/sdb2 was the disk that failed so /dev/sdb might be the problematic disk

mdadm --detail /dev/md1
/dev/md1:
Version : 01.00
Creation Time : Fri Jun 8 03:34:59 2012
Raid Level : raid1
Array Size : 1932533488 (1843.01 GiB 1978.91 GB)
Used Dev Size : 3865066976 (3686.02 GiB 3957.83 GB)
Raid Devices : 2
Total Devices : 2
Preferred Minor : 1
Persistence : Superblock is persistent

Update Time : Sun Jan 5 15:10:02 2014
State : clean, degraded
Active Devices : 1
Working Devices : 2
Failed Devices : 0
Spare Devices : 1

Name : ix2-200-D8S0EN:1 (local to host ix2-200-D8S0EN)
UUID : cba54300:251e4d89:9dbf101c:f04b7735
Events : 1280050

Number Major Minor RaidDevice State
0 8 2 0 active sync /dev/sda2
2 8 18 1 spare rebuilding /dev/sdb2

Last mdadm --detail says that /dev/sdb2 is the 'spare' device, sda2 is active. Isn't there a way
to tell the array to rebuild or sync the spare device? or something else?

I'm starting to think that NAS devices are very fragile, and complex to manage.

[Mijzelf]

But I don't understand why do you say it's the wrong disk, /dev/sdb2 was the disk that failed

I said that if you swapped the disks, then we are looking to the wrong one. Or at least I meant to say that.

Last mdadm --detail says that /dev/sdb2 is the 'spare' device, sda2 is active. Isn't there a way to tell the array to rebuild or sync the spare device? or something else?

Not that I'm aware of. There are some tricks like growing the array to span 3 devices, but in that case the array will still be degraded, and so the red led will stay on.

Maybe you should start over, and do it proper:

Code: Select all

mdadm --manage /dev/md1 --remove /dev/sdb2
mdadm --zero-superblock /dev/sdb2
mdadm --manage /dev/md1 --add /dev/sdb2

I'm starting to think that NAS devices are very fragile

Your firmware is based on Debian, and Debian is supposed to be rock-stable. The GUI put on top of it can be fragile, but that has nothing to do with mdadm.

and complex to manage

Well, the manual says: If the led is red, pull the defective disk, and exchange it by a new one. That's not complex. It gets complex if you want to do it another way.

[eyp]

Yes, I know that mdadm it isn't the problem, and that Debian is a very strong distro, I was only whining a little bit

Well, I'm trying to remove the device, but I can't because it's busy. So I've marked it as failed then removed it, but now it seems that it's mounted as readonly

root@ix2-200-D8S0EN:/# mdadm --manage /dev/md1 --remove /dev/sdb2
mdadm: hot remove failed for /dev/sdb2: Device or resource busy
root@ix2-200-D8S0EN:/# mdadm --manage /dev/md1 --fail /dev/sdb2
mdadm: set /dev/sdb2 faulty in /dev/md1
root@ix2-200-D8S0EN:/# mdadm --manage /dev/md1 --remove /dev/sdb2
mdadm: hot removed /dev/sdb2
root@ix2-200-D8S0EN:/# mdadm --zero-superblock /dev/sdb2
mdadm: Couldn't open /dev/sdb2 for write - not zeroing

[Mijzelf]

Code: Select all

dmesg -c
mdadm --zero-superblock /dev/sdb2
dmesg

[eyp]

Code: Select all

dmesg -c
mdadm --zero-superblock /dev/sdb2
dmesg

Thank you, I'll do it this afternoon when I come back home.

[eyp]

Ummh, something is wrong with the device :S

dmesg -c
ng I/O to offline device
I/O error in filesystem ("dm-2") meta-data dev dm-2 block 0x39980840 ("xfs_trans_read_buf") error 5 buf count 8192
xfs_imap_to_bp: xfs_trans_read_buf()returned an error 5 on dm-2. Returning error.
sd 0:0:0:0: rejecting I/O to offline device
I/O error in filesystem ("dm-2") meta-data dev dm-2 block 0x39980840 ("xfs_trans_read_buf") error 5 buf count 8192
xfs_imap_to_bp: xfs_trans_read_buf()returned an error 5 on dm-2. Returning error.
sd 0:0:0:0: rejecting I/O to offline device
I/O error in filesystem ("dm-2") meta-data dev dm-2 block 0x39980840 ("xfs_trans_read_buf") error 5 buf count 8192
xfs_imap_to_bp: xfs_trans_read_buf()returned an error 5 on dm-2. Returning error.
sd 0:0:0:0: rejecting I/O to offline device
I/O error in filesystem ("dm-2") meta-data dev dm-2 block 0x9ba8 ("xfs_trans_read_buf") error 5 buf count 4096
sd 0:0:0:0: rejecting I/O to offline device
I/O error in filesystem ("dm-2") meta-data dev dm-2 block 0x3d8770a0 ("xfs_trans_read_buf") error 5 buf count 8192
xfs_imap_to_bp: xfs_trans_read_buf()returned an error 5 on dm-2. Returning error.
sd 0:0:0:0: rejecting I/O to offline device
I/O error in filesystem ("dm-2") meta-data dev dm-2 block 0x39980840 ("xfs_trans_read_buf") error 5 buf count 8192
xfs_imap_to_bp: xfs_trans_read_buf()returned an error 5 on dm-2. Returning error.
sd 0:0:0:0: rejecting I/O to offline device
I/O error in filesystem ("dm-2") meta-data dev dm-2 block 0x4c40 ("xfs_trans_read_buf") error 5 buf count 8192
xfs_imap_to_bp: xfs_trans_read_buf()returned an error 5 on dm-2. Returning error.
sd 0:0:0:0: rejecting I/O to offline device
I/O error in filesystem ("dm-2") meta-data dev dm-2 block 0x403f6310 ("xfs_trans_read_buf") error 5 buf count 8192
xfs_imap_to_bp: xfs_trans_read_buf()returned an error 5 on dm-2. Returning error.
sd 0:0:0:0: rejecting I/O to offline device
I/O error in filesystem ("dm-2") meta-data dev dm-2 block 0x9ba8 ("xfs_trans_read_buf") error 5 buf count 4096
sd 0:0:0:0: rejecting I/O to offline device
I/O error in filesystem ("dm-2") meta-data dev dm-2 block 0x9ba8 ("xfs_trans_read_buf") error 5 buf count 4096
sd 0:0:0:0: rejecting I/O to offline device
I/O error in filesystem ("dm-2") meta-data dev dm-2 block 0x39980840 ("xfs_trans_read_buf") error 5 buf count 8192
xfs_imap_to_bp: xfs_trans_read_buf()returned an error 5 on dm-2. Returning error.
sd 0:0:0:0: rejecting I/O to offline device
I/O error in filesystem ("dm-2") meta-data dev dm-2 block 0x39980840 ("xfs_trans_read_buf") error 5 buf count 8192
xfs_imap_to_bp: xfs_trans_read_buf()returned an error 5 on dm-2. Returning error.
sd 0:0:0:0: rejecting I/O to offline device
I/O error in filesystem ("dm-2") meta-data dev dm-2 block 0x9ba8 ("xfs_trans_read_buf") error 5 buf count 4096
sd 0:0:0:0: rejecting I/O to offline device
I/O error in filesystem ("dm-2") meta-data dev dm-2 block 0x39980840 ("xfs_trans_read_buf") error 5 buf count 8192
xfs_imap_to_bp: xfs_trans_read_buf()returned an error 5 on dm-2. Returning error.
sd 0:0:0:0: rejecting I/O to offline device
I/O error in filesystem ("dm-2") meta-data dev dm-2 block 0x3d8770a0 ("xfs_trans_read_buf") error 5 buf count 8192
xfs_imap_to_bp: xfs_trans_read_buf()returned an error 5 on dm-2. Returning error.
sd 0:0:0:0: rejecting I/O to offline device
I/O error in filesystem ("dm-2") meta-data dev dm-2 block 0x39980840 ("xfs_trans_read_buf") error 5 buf count 8192
xfs_imap_to_bp: xfs_trans_read_buf()returned an error 5 on dm-2. Returning error.
sd 0:0:0:0: rejecting I/O to offline device
I/O error in filesystem ("dm-2") meta-data dev dm-2 block 0x9ba8 ("xfs_trans_read_buf") error 5 buf count 4096
sd 0:0:0:0: rejecting I/O to offline device
I/O error in filesystem ("dm-2") meta-data dev dm-2 block 0x4c40 ("xfs_trans_read_buf") error 5 buf count 8192
xfs_imap_to_bp: xfs_trans_read_buf()returned an error 5 on dm-2. Returning error.
sd 0:0:0:0: rejecting I/O to offline device
I/O error in filesystem ("dm-2") meta-data dev dm-2 block 0x403f6310 ("xfs_trans_read_buf") error 5 buf count 8192
xfs_imap_to_bp: xfs_trans_read_buf()returned an error 5 on dm-2. Returning error.
sd 0:0:0:0: rejecting I/O to offline device
I/O error in filesystem ("dm-2") meta-data dev dm-2 block 0x9ba8 ("xfs_trans_read_buf") error 5 buf count 4096
sd 0:0:0:0: rejecting I/O to offline device
I/O error in filesystem ("dm-2") meta-data dev dm-2 block 0x39980840 ("xfs_trans_read_buf") error 5 buf count 8192
xfs_imap_to_bp: xfs_trans_read_buf()returned an error 5 on dm-2. Returning error.
sd 0:0:0:0: rejecting I/O to offline device
I/O error in filesystem ("dm-2") meta-data dev dm-2 block 0x9ba8 ("xfs_trans_read_buf") error 5 buf count 4096
sd 0:0:0:0: rejecting I/O to offline device
I/O error in filesystem ("dm-2") meta-data dev dm-2 block 0x39980840 ("xfs_trans_read_buf") error 5 buf count 8192
xfs_imap_to_bp: xfs_trans_read_buf()returned an error 5 on dm-2. Returning error.
sd 0:0:0:0: rejecting I/O to offline device
I/O error in filesystem ("dm-2") meta-data dev dm-2 block 0x39980840 ("xfs_trans_read_buf") error 5 buf count 8192
xfs_imap_to_bp: xfs_trans_read_buf()returned an error 5 on dm-2. Returning error.
sd 0:0:0:0: rejecting I/O to offline device
I/O error in filesystem ("dm-2") meta-data dev dm-2 block 0x9ba8 ("xfs_trans_read_buf") error 5 buf count 4096
sd 0:0:0:0: rejecting I/O to offline device
I/O error in filesystem ("dm-2") meta-data dev dm-2 block 0x3d8770a0 ("xfs_trans_read_buf") error 5 buf count 8192
xfs_imap_to_bp: xfs_trans_read_buf()returned an error 5 on dm-2. Returning error.
sd 0:0:0:0: rejecting I/O to offline device
I/O error in filesystem ("dm-2") meta-data dev dm-2 block 0x39980840 ("xfs_trans_read_buf") error 5 buf count 8192
xfs_imap_to_bp: xfs_trans_read_buf()returned an error 5 on dm-2. Returning error.
sd 0:0:0:0: rejecting I/O to offline device
I/O error in filesystem ("dm-2") meta-data dev dm-2 block 0x9ba8 ("xfs_trans_read_buf") error 5 buf count 4096
sd 0:0:0:0: rejecting I/O to offline device
I/O error in filesystem ("dm-2") meta-data dev dm-2 block 0x4c40 ("xfs_trans_read_buf") error 5 buf count 8192
xfs_imap_to_bp: xfs_trans_read_buf()returned an error 5 on dm-2. Returning error.
sd 0:0:0:0: rejecting I/O to offline device
I/O error in filesystem ("dm-2") meta-data dev dm-2 block 0x403f6310 ("xfs_trans_read_buf") error 5 buf count 8192
xfs_imap_to_bp: xfs_trans_read_buf()returned an error 5 on dm-2. Returning error.
sd 0:0:0:0: rejecting I/O to offline device
I/O error in filesystem ("dm-2") meta-data dev dm-2 block 0x9ba8 ("xfs_trans_read_buf") error 5 buf count 4096
sd 0:0:0:0: rejecting I/O to offline device
I/O error in filesystem ("dm-2") meta-data dev dm-2 block 0x39980840 ("xfs_trans_read_buf") error 5 buf count 8192
xfs_imap_to_bp: xfs_trans_read_buf()returned an error 5 on dm-2. Returning error.
sd 0:0:0:0: rejecting I/O to offline device
I/O error in filesystem ("dm-2") meta-data dev dm-2 block 0x39980840 ("xfs_trans_read_buf") error 5 buf count 8192
xfs_imap_to_bp: xfs_trans_read_buf()returned an error 5 on dm-2. Returning error.
sd 0:0:0:0: rejecting I/O to offline device
I/O error in filesystem ("dm-2") meta-data dev dm-2 block 0x9ba8 ("xfs_trans_read_buf") error 5 buf count 4096
sd 0:0:0:0: rejecting I/O to offline device
I/O error in filesystem ("dm-2") meta-data dev dm-2 block 0x39980840 ("xfs_trans_read_buf") error 5 buf count 8192
xfs_imap_to_bp: xfs_trans_read_buf()returned an error 5 on dm-2. Returning error.
sd 0:0:0:0: rejecting I/O to offline device
I/O error in filesystem ("dm-2") meta-data dev dm-2 block 0x9ba8 ("xfs_trans_read_buf") error 5 buf count 4096
sd 0:0:0:0: rejecting I/O to offline device
I/O error in filesystem ("dm-2") meta-data dev dm-2 block 0x3d8770a0 ("xfs_trans_read_buf") error 5 buf count 8192
xfs_imap_to_bp: xfs_trans_read_buf()returned an error 5 on dm-2. Returning error.
sd 0:0:0:0: rejecting I/O to offline device
I/O error in filesystem ("dm-2") meta-data dev dm-2 block 0x39980840 ("xfs_trans_read_buf") error 5 buf count 8192
xfs_imap_to_bp: xfs_trans_read_buf()returned an error 5 on dm-2. Returning error.
sd 0:0:0:0: rejecting I/O to offline device
I/O error in filesystem ("dm-2") meta-data dev dm-2 block 0x4c40 ("xfs_trans_read_buf") error 5 buf count 8192
xfs_imap_to_bp: xfs_trans_read_buf()returned an error 5 on dm-2. Returning error.
sd 0:0:0:0: rejecting I/O to offline device
I/O error in filesystem ("dm-2") meta-data dev dm-2 block 0x403f6310 ("xfs_trans_read_buf") error 5 buf count 8192
xfs_imap_to_bp: xfs_trans_read_buf()returned an error 5 on dm-2. Returning error.
sd 0:0:0:0: rejecting I/O to offline device
I/O error in filesystem ("dm-2") meta-data dev dm-2 block 0x9ba8 ("xfs_trans_read_buf") error 5 buf count 4096
sd 0:0:0:0: rejecting I/O to offline device
I/O error in filesystem ("dm-2") meta-data dev dm-2 block 0x39980840 ("xfs_trans_read_buf") error 5 buf count 8192
xfs_imap_to_bp: xfs_trans_read_buf()returned an error 5 on dm-2. Returning error.
sd 0:0:0:0: rejecting I/O to offline device
I/O error in filesystem ("dm-2") meta-data dev dm-2 block 0x9ba8 ("xfs_trans_read_buf") error 5 buf count 4096
sd 0:0:0:0: rejecting I/O to offline device
I/O error in filesystem ("dm-2") meta-data dev dm-2 block 0x39980840 ("xfs_trans_read_buf") error 5 buf count 8192
xfs_imap_to_bp: xfs_trans_read_buf()returned an error 5 on dm-2. Returning error.
sd 0:0:0:0: rejecting I/O to offline device
I/O error in filesystem ("dm-2") meta-data dev dm-2 block 0x9ba8 ("xfs_trans_read_buf") error 5 buf count 4096
sd 0:0:0:0: rejecting I/O to offline device
I/O error in filesystem ("dm-2") meta-data dev dm-2 block 0x39980840 ("xfs_trans_read_buf") error 5 buf count 8192
xfs_imap_to_bp: xfs_trans_read_buf()returned an error 5 on dm-2. Returning error.
sd 0:0:0:0: rejecting I/O to offline device
I/O error in filesystem ("dm-2") meta-data dev dm-2 block 0x3d8770a0 ("xfs_trans_read_buf") error 5 buf count 8192
xfs_imap_to_bp: xfs_trans_read_buf()returned an error 5 on dm-2. Returning error.
sd 0:0:0:0: rejecting I/O to offline device
I/O error in filesystem ("dm-2") meta-data dev dm-2 block 0x39980840 ("xfs_trans_read_buf") error 5 buf count 8192
xfs_imap_to_bp: xfs_trans_read_buf()returned an error 5 on dm-2. Returning error.
sd 0:0:0:0: rejecting I/O to offline device
I/O error in filesystem ("dm-2") meta-data dev dm-2 block 0x4c40 ("xfs_trans_read_buf") error 5 buf count 8192
xfs_imap_to_bp: xfs_trans_read_buf()returned an error 5 on dm-2. Returning error.
sd 0:0:0:0: rejecting I/O to offline device
I/O error in filesystem ("dm-2") meta-data dev dm-2 block 0x403f6310 ("xfs_trans_read_buf") error 5 buf count 8192
xfs_imap_to_bp: xfs_trans_read_buf()returned an error 5 on dm-2. Returning error.
sd 0:0:0:0: rejecting I/O to offline device
I/O error in filesystem ("dm-2") meta-data dev dm-2 block 0x9ba8 ("xfs_trans_read_buf") error 5 buf count 4096
sd 0:0:0:0: rejecting I/O to offline device
I/O error in filesystem ("dm-2") meta-data dev dm-2 block 0x9ba8 ("xfs_trans_read_buf") error 5 buf count 4096
sd 0:0:0:0: rejecting I/O to offline device
I/O error in filesystem ("dm-2") meta-data dev dm-2 block 0x39980840 ("xfs_trans_read_buf") error 5 buf count 8192
xfs_imap_to_bp: xfs_trans_read_buf()returned an error 5 on dm-2. Returning error.
sd 0:0:0:0: rejecting I/O to offline device
I/O error in filesystem ("dm-2") meta-data dev dm-2 block 0x9ba8 ("xfs_trans_read_buf") error 5 buf count 4096
sd 0:0:0:0: rejecting I/O to offline device
I/O error in filesystem ("dm-2") meta-data dev dm-2 block 0x39980840 ("xfs_trans_read_buf") error 5 buf count 8192
xfs_imap_to_bp: xfs_trans_read_buf()returned an error 5 on dm-2. Returning error.
sd 0:0:0:0: rejecting I/O to offline device
I/O error in filesystem ("dm-2") meta-data dev dm-2 block 0x39980840 ("xfs_trans_read_buf") error 5 buf count 8192
xfs_imap_to_bp: xfs_trans_read_buf()returned an error 5 on dm-2. Returning error.
sd 0:0:0:0: rejecting I/O to offline device
I/O error in filesystem ("dm-2") meta-data dev dm-2 block 0x3d8770a0 ("xfs_trans_read_buf") error 5 buf count 8192
xfs_imap_to_bp: xfs_trans_read_buf()returned an error 5 on dm-2. Returning error.
sd 0:0:0:0: rejecting I/O to offline device
I/O error in filesystem ("dm-2") meta-data dev dm-2 block 0x39980840 ("xfs_trans_read_buf") error 5 buf count 8192
xfs_imap_to_bp: xfs_trans_read_buf()returned an error 5 on dm-2. Returning error.
sd 0:0:0:0: rejecting I/O to offline device
I/O error in filesystem ("dm-2") meta-data dev dm-2 block 0x9ba8 ("xfs_trans_read_buf") error 5 buf count 4096
sd 0:0:0:0: rejecting I/O to offline device
I/O error in filesystem ("dm-2") meta-data dev dm-2 block 0x4c40 ("xfs_trans_read_buf") error 5 buf count 8192
xfs_imap_to_bp: xfs_trans_read_buf()returned an error 5 on dm-2. Returning error.
sd 0:0:0:0: rejecting I/O to offline device
I/O error in filesystem ("dm-2") meta-data dev dm-2 block 0x403f6310 ("xfs_trans_read_buf") error 5 buf count 8192
xfs_imap_to_bp: xfs_trans_read_buf()returned an error 5 on dm-2. Returning error.
sd 0:0:0:0: rejecting I/O to offline device
I/O error in filesystem ("dm-2") meta-data dev dm-2 block 0x9ba8 ("xfs_trans_read_buf") error 5 buf count 4096
sd 0:0:0:0: rejecting I/O to offline device
I/O error in filesystem ("dm-2") meta-data dev dm-2 block 0x9ba8 ("xfs_trans_read_buf") error 5 buf count 4096
sd 0:0:0:0: rejecting I/O to offline device
I/O error in filesystem ("dm-2") meta-data dev dm-2 block 0x39980840 ("xfs_trans_read_buf") error 5 buf count 8192
xfs_imap_to_bp: xfs_trans_read_buf()returned an error 5 on dm-2. Returning error.
sd 0:0:0:0: rejecting I/O to offline device
I/O error in filesystem ("dm-2") meta-data dev dm-2 block 0x39980840 ("xfs_trans_read_buf") error 5 buf count 8192
xfs_imap_to_bp: xfs_trans_read_buf()returned an error 5 on dm-2. Returning error.
sd 0:0:0:0: rejecting I/O to offline device
I/O error in filesystem ("dm-2") meta-data dev dm-2 block 0x39980840 ("xfs_trans_read_buf") error 5 buf count 8192
xfs_imap_to_bp: xfs_trans_read_buf()returned an error 5 on dm-2. Returning error.
sd 0:0:0:0: rejecting I/O to offline device
I/O error in filesystem ("dm-2") meta-data dev dm-2 block 0x3d8770a0 ("xfs_trans_read_buf") error 5 buf count 8192
xfs_imap_to_bp: xfs_trans_read_buf()returned an error 5 on dm-2. Returning error.
sd 0:0:0:0: rejecting I/O to offline device
I/O error in filesystem ("dm-2") meta-data dev dm-2 block 0x39980840 ("xfs_trans_read_buf") error 5 buf count 8192
xfs_imap_to_bp: xfs_trans_read_buf()returned an error 5 on dm-2. Returning error.
sd 0:0:0:0: rejecting I/O to offline device
I/O error in filesystem ("dm-2") meta-data dev dm-2 block 0x9ba8 ("xfs_trans_read_buf") error 5 buf count 4096
sd 0:0:0:0: rejecting I/O to offline device
I/O error in filesystem ("dm-2") meta-data dev dm-2 block 0x4c40 ("xfs_trans_read_buf") error 5 buf count 8192
xfs_imap_to_bp: xfs_trans_read_buf()returned an error 5 on dm-2. Returning error.
sd 0:0:0:0: rejecting I/O to offline device
I/O error in filesystem ("dm-2") meta-data dev dm-2 block 0x403f6310 ("xfs_trans_read_buf") error 5 buf count 8192
xfs_imap_to_bp: xfs_trans_read_buf()returned an error 5 on dm-2. Returning error.
sd 0:0:0:0: rejecting I/O to offline device
I/O error in filesystem ("dm-2") meta-data dev dm-2 block 0x9ba8 ("xfs_trans_read_buf") error 5 buf count 4096
root@ix2-200-D8S0EN:/# mdadm --zero-superblock /dev/sdb2
mdadm: Couldn't open /dev/sdb2 for write - not zeroing
root@ix2-200-D8S0EN:/# dmesg

[Mijzelf]

I wonder if somehow /dev/sdb2 is still in the array. You can have a look with 'cat /proc/mdstat'. If it is, you can reboot, and then I guess a removed device can no longer be part of the array?

I'm not sure if those I/O errors are actually hardware errors. The error seems to originate in dm-2, which is a (virtual) raid array, and the error is that I/O to an offline device is rejected.

The partition is removed from the array, so it's offline. But somehow the filesystem seems still to be trying to read it. (Seeing the xfs errors). The filesystem doesn't know about the raidarray, so it seems there is still a layer which is redirecting requests to sdb2?

[eyp]

Well, I marked the device as failed then removed it as you told me two or three posts above. Then I tried to do the zero-superblock but it can't be done because it says it can't write on a readonly device. Should I add the device to the array again then try to do 'dmesg -c'?