General NAS-Central Forums

Welcome to the NAS community
It is currently Wed Nov 22, 2017 5:35 am

All times are UTC




Post new topic Reply to topic  [ 9 posts ] 
Author Message
PostPosted: Fri Jul 04, 2014 9:11 am 
Offline

Joined: Fri Jul 04, 2014 8:07 am
Posts: 6
Hi guys,
After upgrading to 2.2.9.3, volume are unavailable so all data are innaccessible. Looking into the dashboard , I guess my data are still there :
- 5 x 1TB , RAID 5 = clean, Capacity 3.6TB , Used = 3.4TB, Available = 222.5 GB

Have installed the custom firmware (plugout.net) to enter backdoor SSH , trying some stuff , but nothing. The /etc/mdmadm.conf (in the box) contains only :
Code:
ARRAY /dev/md4 UUID=5377505f:edc99769:e915500f:f060d051
ARRAY /dev/md3 UUID=af6f0a16:e35692f0:5d387271:3a77b090
ARRAY /dev/md0 UUID=00850252:a1a28dbd:0ef56721:390a19a6
ARRAY /dev/md1 UUID=0330003f:38a3fb9f:5edd0f53:e99b498f
ARRAY /dev/md2 UUID=75c2a7b5:4cb8d472:980a9269:853fbb9d

I decided to pull out the disks and put them into a computer having more than 5 SATA slots.
The disks are arranged in their disk number and SATA slot number.
Booting with ubuntu USB stick.
After 'mdadm --assemble ' looking with GParted , see /dev/md4 are xfs, and capacity/used/available are seem to be correct.
But when I mount it, the command hangs on (did not do nothing) , can't do even 'mdadm --stop /dev/md4'

Here are what I did :
Code:
root@ubuntu:/home/ubuntu# mdadm --examine --scan --config=partitions --verbose
ARRAY /dev/md4 level=raid5 num-devices=5 UUID=5377505f:edc99769:e915500f:f060d051
   devices=/dev/sdb2,/dev/sdd2,/dev/sde2,/dev/sdc2,/dev/sda2
ARRAY /dev/md3 level=raid1 num-devices=5 UUID=af6f0a16:e35692f0:5d387271:3a77b090
   devices=/dev/sdb5,/dev/sdd5,/dev/sde5,/dev/sdc5,/dev/sda5
ARRAY /dev/md0 level=raid1 num-devices=5 UUID=00850252:a1a28dbd:0ef56721:390a19a6
   devices=/dev/sdb7,/dev/sdd7,/dev/sde7,/dev/sdc7,/dev/sda7
ARRAY /dev/md1 level=raid1 num-devices=5 UUID=0330003f:38a3fb9f:5edd0f53:e99b498f
   devices=/dev/sdb8,/dev/sdd8,/dev/sde8,/dev/sdc8,/dev/sda8
ARRAY /dev/md2 level=raid1 num-devices=5 UUID=75c2a7b5:4cb8d472:980a9269:853fbb9d
   devices=/dev/sdb9,/dev/sdd9,/dev/sde9,/dev/sdc9,/dev/sda9

root@ubuntu:/home/ubuntu# cat /proc/mdstat
Personalities :
unused devices: <none>

In attachement :
- the result of 'fdisk -l' before assembly
- mdadm --examine /dev/sd*2 , and the cat /proc/partitions

Assemble following the RaidDevice number
Code:
root@ubuntu:/home/ubuntu# mdadm --assemble /dev/md4 /dev/sda2 /dev/sdc2 /dev/sdb2 /dev/sdd2 /dev/sde2 --run --verbose
mdadm: looking for devices for /dev/md4
mdadm: /dev/sda2 is identified as a member of /dev/md4, slot 0.
mdadm: /dev/sdc2 is identified as a member of /dev/md4, slot 1.
mdadm: /dev/sdb2 is identified as a member of /dev/md4, slot 2.
mdadm: /dev/sdd2 is identified as a member of /dev/md4, slot 3.
mdadm: /dev/sde2 is identified as a member of /dev/md4, slot 4.
mdadm: added /dev/sdc2 to /dev/md4 as 1
mdadm: added /dev/sdb2 to /dev/md4 as 2
mdadm: added /dev/sdd2 to /dev/md4 as 3
mdadm: added /dev/sde2 to /dev/md4 as 4
mdadm: added /dev/sda2 to /dev/md4 as 0
mdadm: /dev/md4 has been started with 5 drives.


Code:
root@ubuntu:/home/ubuntu# fdisk -l
....
....
   Device Boot      Start         End      Blocks   Id  System
/dev/sdj1   *        2048     7821311     3909632    c  W95 FAT32 (LBA)

Disk /dev/md4: 3992.6 GB, 3992583405568 bytes
2 heads, 4 sectors/track, 974751808 cylinders, total 7798014464 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 65536 bytes / 262144 bytes
Disk identifier: 0x00000000

Disk /dev/md4 doesn't contain a valid partition table

Why this last line :?:
Attempt to mount anyway :
Code:
root@ubuntu:/home/ubuntu# mkdir /tmp/nas1
root@ubuntu:/home/ubuntu# mount -o ro /dev/md4 /tmp/nas1

The last command still hangs even if I left it more hours, abd it still the same if I do :
Code:
mdadm --assemble /dev/md4 /dev/sdb2 /dev/sdd2 /dev/sde2 /dev/sdc2 /dev/sda2 --run --verbose

Did I missed out something , or what's wrong
I noticed to just the "Minor" and the "RaidDevice" number betwwen /dev/sdc2 and /dev/sdb2
Code:
      Number   Major   Minor   RaidDevice State
this     1       8       34        1      active sync   /dev/sdc2
   0     0       8        2        0      active sync   /dev/sda2
   1     1       8       34        1      active sync   /dev/sdc2
   2     2       8       18        2      active sync   /dev/sdb2
   3     3       8       50        3      active sync   /dev/sdd2
   4     4       8       66        4      active sync   /dev/sde2

Looking for your precious helps

Best regards,


Top
 Profile  
 
PostPosted: Fri Jul 04, 2014 11:41 am 
Offline

Joined: Mon Jun 16, 2008 10:45 am
Posts: 6073
zzandry wrote:
Code:
Disk /dev/md4 doesn't contain a valid partition table

Why this last line :?:
Because it hasn't. fdisk is a partition table editor, and you ask it to show the partition table of all blockdevices. Well, the raid array doesn't contain a partition table, the filesystem starts at sector 0 (or 1).
Quote:
The last command still hangs even if I left it more hours
Maybe you can open a 2nd terminal, and have a look if dmesg shows any appending information.
Quote:
Did I missed out something
AFAICS not. Your commands are correct.
Quote:
I noticed to just the "Minor" and the "RaidDevice" number betwwen /dev/sdc2 and /dev/sdb2
You mean the minors have another sequence than the raiddevices do? That's normal. The Lacie uses a strange way to name the disks. But it doesn't matter. The role in the raid array is stored in the raid header, on the disk itself. So the actual sequence is only important if you have to forcibly re-create a degraded array.


Top
 Profile  
 
PostPosted: Fri Jul 04, 2014 1:10 pm 
Offline

Joined: Fri Jul 04, 2014 8:07 am
Posts: 6
Thanks a lot, here are the dmesg
After mdadm assemble
Code:
[  504.228255] md: md4 stopped.
[  504.229452] md: bind<sdc2>
[  504.229601] md: bind<sdb2>
[  504.229746] md: bind<sdd2>
[  504.229891] md: bind<sde2>
[  504.230032] md: bind<sda2>
[  504.240931] async_tx: api initialized (async)
[  504.251155] xor: measuring software checksum speed
[  504.288023]    pIII_sse  :  9940.000 MB/sec
[  504.328011]    prefetch64-sse: 11380.000 MB/sec
[  504.328015] xor: using function: prefetch64-sse (11380.000 MB/sec)
[  504.412027] raid6: mmxx1     3416 MB/s
[  504.480017] raid6: mmxx2     3906 MB/s
[  504.548029] raid6: sse1x1    2290 MB/s
[  504.616031] raid6: sse1x2    3190 MB/s
[  504.684021] raid6: sse2x1    4240 MB/s
[  504.752010] raid6: sse2x2    5940 MB/s
[  504.752013] raid6: using algorithm sse2x2 (5940 MB/s)
[  504.752015] raid6: using ssse3x1 recovery algorithm
[  504.776392] md: raid6 personality registered for level 6
[  504.776396] md: raid5 personality registered for level 5
[  504.776397] md: raid4 personality registered for level 4
[  504.776985] md/raid:md4: device sda2 operational as raid disk 0
[  504.776989] md/raid:md4: device sde2 operational as raid disk 4
[  504.776991] md/raid:md4: device sdd2 operational as raid disk 3
[  504.776993] md/raid:md4: device sdb2 operational as raid disk 2
[  504.776994] md/raid:md4: device sdc2 operational as raid disk 1
[  504.777455] md/raid:md4: allocated 0kB
[  504.777508] md/raid:md4: raid level 5 active with 5 out of 5 devices, algorithm 2
[  504.777510] RAID conf printout:
[  504.777512]  --- level:5 rd:5 wd:5
[  504.777514]  disk 0, o:1, dev:sda2
[  504.777515]  disk 1, o:1, dev:sdc2
[  504.777517]  disk 2, o:1, dev:sdb2
[  504.777519]  disk 3, o:1, dev:sdd2
[  504.777520]  disk 4, o:1, dev:sde2
[  504.777559] md4: detected capacity change from 0 to 3992583405568
[  504.778899]  md4: unknown partition table

After 'mount'
Code:
[  577.029308] SGI XFS with ACLs, security attributes, realtime, large block/inode numbers, no debug enabled
[  577.035558] XFS (md4): Mounting Filesystem
[  577.215325] XFS (md4): Starting recovery (logdev: internal)
[  577.437840] XFS: Internal error XFS_WANT_CORRUPTED_GOTO at line 1706 of file /build/buildd/linux-3.13.0/fs/xfs/xfs_alloc.c.  Caller 0xfc5f77aa
[  577.437848] CPU: 1 PID: 7126 Comm: mount Not tainted 3.13.0-24-generic #46-Ubuntu
[  577.437850] Hardware name: System manufacturer System Product Name/P5Q TURBO, BIOS 0210    02/20/2009
[  577.437853]  00000000 00000000 f4d83c68 c164b873 c0040000 f4d83c8c fc5dd2bb 00000000
[  577.437861]  00000008 fc652fc0 fc650d84 000006aa fc653e5c fc5f77aa f4d83cec fc5f66e3
[  577.437867]  fc653e5c 000006aa fc5f77aa 00000016 00000000 00000001 00000000 c0046000
[  577.437874] Call Trace:
[  577.437883]  [<c164b873>] dump_stack+0x41/0x52
[  577.437911]  [<fc5dd2bb>] xfs_error_report+0x4b/0x50 [xfs]
[  577.437938]  [<fc5f77aa>] ? xfs_free_extent+0xca/0x100 [xfs]
[  577.437964]  [<fc5f66e3>] xfs_free_ag_extent+0x593/0x6f0 [xfs]
[  577.437991]  [<fc5f77aa>] ? xfs_free_extent+0xca/0x100 [xfs]
[  577.438011]  [<fc5f77aa>] xfs_free_extent+0xca/0x100 [xfs]
[  577.438035]  [<fc6299f7>] xlog_recover_process_efi+0x1b7/0x1f0 [xfs]
[  577.438039]  [<c1090eec>] ? wake_up_bit+0x1c/0x20
[  577.438063]  [<fc62b713>] xlog_recover_process_efis.isra.26+0x63/0xa0 [xfs]
[  577.438087]  [<fc62eadd>] xlog_recover_finish+0x1d/0xb0 [xfs]
[  577.438111]  [<fc6332fb>] xfs_log_mount_finish+0x2b/0x40 [xfs]
[  577.438130]  [<fc5ed344>] xfs_mountfs+0x484/0x6f0 [xfs]
[  577.438149]  [<fc5f04a3>] xfs_fs_fill_super+0x253/0x2c0 [xfs]
[  577.438168]  [<fc5f0250>] ? xfs_parseargs+0xb20/0xb20 [xfs]
[  577.438172]  [<c1179094>] mount_bdev+0x174/0x1b0
[  577.438191]  [<fc5f0250>] ? xfs_parseargs+0xb20/0xb20 [xfs]
[  577.438195]  [<c12ee936>] ? ida_get_new_above+0x1a6/0x200
[  577.438197]  [<c12ee936>] ? ida_get_new_above+0x1a6/0x200
[  577.438216]  [<fc5ee61f>] xfs_fs_mount+0x1f/0x30 [xfs]
[  577.438235]  [<fc5f0250>] ? xfs_parseargs+0xb20/0xb20 [xfs]
[  577.438237]  [<c1179901>] mount_fs+0x31/0x170
[  577.438241]  [<c113d1df>] ? __alloc_percpu+0xf/0x20
[  577.438244]  [<c1191743>] ? alloc_vfsmnt+0xb3/0x160
[  577.438247]  [<c1191879>] vfs_kern_mount+0x49/0xd0
[  577.438250]  [<c1193960>] do_mount+0x1e0/0x8e0
[  577.438253]  [<c1138f39>] ? strndup_user+0x39/0xd0
[  577.438256]  [<c11942fc>] SyS_mount+0x7c/0xb0
[  577.438260]  [<c1659b4d>] sysenter_do_call+0x12/0x28
[  577.438272] XFS (md4): Failed to recover EFIs
[  577.438274] XFS (md4): log mount finish failed

Failed to recover EFIS :!: :?:
And the rest are a loop retry of this
Code:
[  723.860060] INFO: task mount:7126 blocked for more than 120 seconds.
[  723.860067]       Not tainted 3.13.0-24-generic #46-Ubuntu
[  723.860069] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[  723.860071] mount           D f4d83d64     0  7126   5289 0x00000000
[  723.860077]  f4d83dc4 00000082 c1080318 f4d83d64 c1080c15 c1a87540 f4ce0cf0 f4dd40b0
[  723.860085]  c1a87540 f4ce0cf0 f6f96780 f588f540 f5881540 f4dd40b0 f5881540 f5881540
[  723.860091]  f4d83d94 f4dd40b0 00000246 f4dd40b0 f4d83dc0 c10829b0 00000000 c1a87540
[  723.860098] Call Trace:
[  723.860107]  [<c1080318>] ? resched_task+0x48/0x60
[  723.860111]  [<c1080c15>] ? check_preempt_curr+0x65/0x90
[  723.860115]  [<c10829b0>] ? try_to_wake_up+0x150/0x240
[  723.860120]  [<c164f433>] schedule+0x23/0x60
[  723.860158]  [<fc638b37>] xfs_ail_push_all_sync+0x97/0xd0 [xfs]
[  723.860163]  [<c1091220>] ? prepare_to_wait_event+0xd0/0xd0
[  723.860197]  [<fc63510d>] xfs_log_quiesce+0x2d/0x60 [xfs]
[  723.860221]  [<fc635150>] xfs_log_unmount+0x10/0x30 [xfs]
[  723.860240]  [<fc5ed451>] xfs_mountfs+0x591/0x6f0 [xfs]
[  723.860259]  [<fc5f04a3>] xfs_fs_fill_super+0x253/0x2c0 [xfs]
[  723.860277]  [<fc5f0250>] ? xfs_parseargs+0xb20/0xb20 [xfs]
[  723.860281]  [<c1179094>] mount_bdev+0x174/0x1b0
[  723.860300]  [<fc5f0250>] ? xfs_parseargs+0xb20/0xb20 [xfs]
[  723.860303]  [<c12ee936>] ? ida_get_new_above+0x1a6/0x200
[  723.860306]  [<c12ee936>] ? ida_get_new_above+0x1a6/0x200
[  723.860325]  [<fc5ee61f>] xfs_fs_mount+0x1f/0x30 [xfs]
[  723.860343]  [<fc5f0250>] ? xfs_parseargs+0xb20/0xb20 [xfs]
[  723.860346]  [<c1179901>] mount_fs+0x31/0x170
[  723.860350]  [<c113d1df>] ? __alloc_percpu+0xf/0x20
[  723.860353]  [<c1191743>] ? alloc_vfsmnt+0xb3/0x160
[  723.860356]  [<c1191879>] vfs_kern_mount+0x49/0xd0
[  723.860359]  [<c1193960>] do_mount+0x1e0/0x8e0
[  723.860362]  [<c1138f39>] ? strndup_user+0x39/0xd0
[  723.860365]  [<c11942fc>] SyS_mount+0x7c/0xb0
[  723.860368]  [<c1659b4d>] sysenter_do_call+0x12/0x28

I guess that's why I have a trouble to reboot or shutdown after trying to mount
Quote:
The Lacie uses a strange way to name the disks. But it doesn't matter. The role in the raid array is stored in the raid header, on the disk itself

You are right, even trying to interchange the disk slot , it's the same


Top
 Profile  
 
PostPosted: Fri Jul 04, 2014 2:35 pm 
Offline

Joined: Mon Jun 16, 2008 10:45 am
Posts: 6073
Quote:
XFS: Internal error XFS_WANT_CORRUPTED_GOTO at line 1706
Apparently the filesystem is corrupted, or it's an incompatibility in the xfs implementation of the different kernels.
Don't know if xfs has little/big endian issues, in that case you could retry this using the plugout software. If it's a version incompatibility you can try to find a live distro with a kernel version which (roughly) equals the Lacie kernel.

Or you can try xfs_repair, but it wouldn't be my first choice, as it's supposed to write to the disk. Ough!


Top
 Profile  
 
PostPosted: Fri Jul 04, 2014 4:01 pm 
Offline

Joined: Fri Jul 04, 2014 8:07 am
Posts: 6
I think it's a filesystem is corrupted.
In fact, the history is :
- At 2:50 am, one UPS 10KVA was faulty , the 5Big has just one power supply causing it to down (with power button remain to ON and I didn't mind )
- At this time there were a running backup storing in its
- Power returns at 7 am, I verified what backup are concerned (without knowing that the 5Big was down), there are nothing to panic
- Arround 13 pm, the UPS was faulty again, and after power return I saw that the lacie led blink (as it was starting)
- Noticed on lacie's log : 13:20 RAID 5 clean
- Begin the trouble: the 5big lost domain, with a difficulty to rejoin, I took this opportunity to upgrade firmware (2.2.9.2 to 2.2.9.3) , rejoin domain, etc... all seem ok.
- Enter to lacie file brower, attempting to delete one incomplete backup file, suddenly it showed an error like file not found and all shares disappear. :o
Quote:
Don't know if xfs has little/big endian issues, in that case you could retry this using the plugout software. If it's a version incompatibility you can try to find a live distro with a kernel version which (roughly) equals the Lacie kernel.
Or you can try xfs_repair, but it wouldn't be my first choice, as it's supposed to write to the disk. Ough!

Meanwhile your reply, decided to reboot and I've tried xfs_repair in no modify mode:
Code:
root@ubuntu:/home/ubuntu# xfs_repair -n /dev/md4
....
....
directory flags set on non-directory inode 2097143966
, would fix bad flags.
directory flags set on non-directory inode 2097143967
, would fix bad flags.
No modify flag set, skipping phase 5
Phase 6 - check inode connectivity...
        - traversing filesystem ...
entry "SR_G_Drive040.v2i" in directory inode 1610613018 points to free inode 1620348166, would junk entry
bad hash table for directory inode 1610613018 (no data entry): would rebuild
        - traversal finished ...
        - moving disconnected inodes to lost+found ...
disconnected inode 2949182198, would move to lost+found
Phase 7 - verify link counts...
would have reset inode 2949182198 nlinks from 0 to 1
No modify flag set, skipping filesystem flush and exiting.

It seems to find out something, now I'm hesitating between repair this into the 5Big or externally !
Lets' back to the 5big , wait and see

All the best


Top
 Profile  
 
PostPosted: Fri Jul 04, 2014 4:28 pm 
Offline

Joined: Fri Jul 04, 2014 8:07 am
Posts: 6
In the 5big box, dmesg shows :
Code:
[   64.180000] Starting XFS recovery on filesystem: md4 (logdev: internal)
[   64.390000] XFS internal error XFS_WANT_CORRUPTED_GOTO at line 1733 of file /home/sbs/devs/workspace/5bignetwork2_release/system/components/linux/source/fs/xfs/xfs_alloc.c.  Caller 0xc0226bf4
[   64.390000] [<c0033ee0>] (unwind_backtrace+0x0/0xe4) from [<c022643c>] (xfs_free_ag_extent+0x500/0x768)
[   64.390000] [<c022643c>] (xfs_free_ag_extent+0x500/0x768) from [<c0226bf4>] (xfs_free_extent+0xb0/0xcc)
[   64.390000] [<c0226bf4>] (xfs_free_extent+0xb0/0xcc) from [<c025eddc>] (xlog_recover_process_efi+0x160/0x1d0)
[   64.390000] [<c025eddc>] (xlog_recover_process_efi+0x160/0x1d0) from [<c025eea4>] (xlog_recover_process_efis+0x58/0xa0)
[   64.390000] [<c025eea4>] (xlog_recover_process_efis+0x58/0xa0) from [<c0263804>] (xlog_recover_finish+0x24/0xc0)
[   64.390000] [<c0263804>] (xlog_recover_finish+0x24/0xc0) from [<c0266290>] (xfs_mountfs+0x518/0x67c)
[   64.390000] [<c0266290>] (xfs_mountfs+0x518/0x67c) from [<c027ba4c>] (xfs_fs_fill_super+0x154/0x2cc)
[   64.390000] [<c027ba4c>] (xfs_fs_fill_super+0x154/0x2cc) from [<c00b1950>] (get_sb_bdev+0x104/0x15c)
[   64.390000] [<c00b1950>] (get_sb_bdev+0x104/0x15c) from [<c0279bb8>] (xfs_fs_get_sb+0x24/0x30)
[   64.390000] [<c0279bb8>] (xfs_fs_get_sb+0x24/0x30) from [<c00b149c>] (vfs_kern_mount+0x58/0xb0)
[   64.390000] [<c00b149c>] (vfs_kern_mount+0x58/0xb0) from [<c00b1540>] (do_kern_mount+0x3c/0xe0)
[   64.390000] [<c00b1540>] (do_kern_mount+0x3c/0xe0) from [<c00c4f54>] (do_mount+0x61c/0x688)
[   64.390000] [<c00c4f54>] (do_mount+0x61c/0x688) from [<c00c504c>] (sys_mount+0x8c/0xd4)
[   64.390000] [<c00c504c>] (sys_mount+0x8c/0xd4) from [<c002ddc0>] (ret_fast_syscall+0x0/0x2c)
[   64.390000] Failed to recover EFIs on filesystem: md4
[   64.390000] XFS: log mount finish failed

I think the last thing I can try is doing xfs_repair :!:

What's the safe : in the 5Big or externally ??

Thanks a lot


Top
 Profile  
 
PostPosted: Fri Jul 04, 2014 5:42 pm 
Offline

Joined: Mon Jun 16, 2008 10:45 am
Posts: 6073
As the 5Big2 and your Ubuntu box show roughly the same behavior, I'd go for the latest and greatest version of xfs_repair, on that Ubuntu box.

Having said that, if that data is really valuable, you should first backup the whole filesystem. These days the whole array fits on a single disk.


Top
 Profile  
 
PostPosted: Fri Jul 04, 2014 7:35 pm 
Offline

Joined: Fri Jul 04, 2014 8:07 am
Posts: 6
Yes right, I will do these backup.


Top
 Profile  
 
PostPosted: Mon Jul 07, 2014 7:07 am 
Offline

Joined: Fri Jul 04, 2014 8:07 am
Posts: 6
Hi ,
Thank you Mijzelf for the assistance .
It's ok now, xfs_repair did the trick

All the best


Top
 Profile  
 
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 9 posts ] 

All times are UTC


Who is online

Users browsing this forum: No registered users and 8 guests


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
cron
Powered by phpBB® Forum Software © phpBB Group