ix2-ng recovery usb

StorCenter ix2-200 Network Storage
Mijzelf
Posts: 6206
Joined: Mon Jun 16, 2008 10:45 am

Re: ix2-ng recovery usb

Post by Mijzelf » Wed Jun 08, 2016 2:13 pm

amr42 wrote:What would you suggest I do as next steps?
Have a look at the serial bootlog, it might tell where it fails to start the webserver.

And I would expect you to be able to get a shell on serial.

amr42
Posts: 28
Joined: Mon Jan 16, 2012 9:34 pm

Re: ix2-ng recovery usb

Post by amr42 » Wed Jun 08, 2016 5:44 pm

Mijzelf wrote:
amr42 wrote:What would you suggest I do as next steps?
Have a look at the serial bootlog, it might tell where it fails to start the webserver.
Serial bootlog attached here
Mijzelf wrote:And I would expect you to be able to get a shell on serial.
Yes, login available, but I'm not sure what to check at this point (obviously I can login with root/soho)

Mijzelf
Posts: 6206
Joined: Mon Jun 16, 2008 10:45 am

Re: ix2-ng recovery usb

Post by Mijzelf » Wed Jun 08, 2016 6:53 pm

This cannot be your current bootlog.
amr42 wrote:Reminder - IX2-DL now with 2 fresh 3TB WD Red drives....

Code: Select all

Integrated Sata device found
[0 0 0]: Enable DMA mode (6)
  Device 0 @ 0 0:
Model: SAMSUNG HD103SI                          Firm: 1AG01118 Ser#: S1XGJ90Z425975      
            Type: Hard Disk
            Supports 48-bit addressing
            Capacity: 953869.7 MB = 931.5 GB (1953525168 x 512)
[0 1 0]: Enable DMA mode (6)
  Device 1 @ 0 1:
Model: SAMSUNG HD103SI                          Firm: 1AQ10001 Ser#: S2SDJ9JBB01167      
            Type: Hard Disk
            Supports 48-bit addressing
            Capacity: 953869.7 MB = 931.5 GB (1953525168 x 512)

amr42
Posts: 28
Joined: Mon Jan 16, 2012 9:34 pm

Re: ix2-ng recovery usb

Post by amr42 » Wed Jun 08, 2016 10:15 pm

Mijzelf wrote:This cannot be your current bootlog.
amr42 wrote:Reminder - IX2-DL now with 2 fresh 3TB WD Red drives....
My bad! This drive has 2 x 1TB Samsung drives. I had another unit which needed some TLC that has 3TB WD Red drives...

So boot log IS correct, just what I say sometimes may not be.... :)

Apologies!

Mijzelf
Posts: 6206
Joined: Mon Jun 16, 2008 10:45 am

Re: ix2-ng recovery usb

Post by Mijzelf » Thu Jun 09, 2016 5:36 pm

I see. Well, I *think* here is the problem:

Code: Select all

attempt to access beyond end of device
loop0: rw=0, want=1412902, limit=1399648
EXT2-fs error (device loop0): ext2_get_inode: unable to read inode block - inode=35526, block=706450
Remounting filesystem read-only
cp: can't stat 'sysroot/mnt/apps/run/motd.dynamic': Input/output error
When I'm not mistaken loop0 is a loopdevice on 'apps', a file which can be found in /boot. Apparently this file is smaller than the filesystem which is supposed to be inside it.

Using 'losetup -a' you can see which file is associated with loop0. If that is 'apps', it should be the same 'apps' which can be found at the imager stick. If it's smaller, then there is your problem.

amr42
Posts: 28
Joined: Mon Jan 16, 2012 9:34 pm

Re: ix2-ng recovery usb

Post by amr42 » Thu Jun 09, 2016 9:16 pm

Mijzelf wrote:Using 'losetup -a' you can see which file is associated with loop0. If that is 'apps', it should be the same 'apps' which can be found at the imager stick. If it's smaller, then there is your problem.
Yes you are correct - it is the apps file.
losetup -a shows:

Code: Select all

/dev/loop0: [000c]:66 (/sysroot/boot/images/apps)
/dev/loop1: [000c]:67 (sysroot/boot/images/config)
but it's like there is no file system structure anywhere. A simple ls in the / directory does this:

Code: Select all

root@storage0000:/# ls
attempt to access beyond end of device
loop0: rw=0, want=1429140, limit=1399648
EXT2-fs error (device loop0): ext2_get_inode: unable to read inode block - inode
=35921, block=714569
Remounting filesystem read-only
attempt to access beyond end of device
loop0: rw=0, want=1429140, limit=1399648
EXT2-fs error (device loop0): ext2_get_inode: unable to read inode block - inode
=35921, block=714569
Remounting filesystem read-only
ls: error while loading shared libraries: libattr.so.1: cannot open shared objec
t file: Input/output error
root@storage0000:/#  
But df does show filesystems:

Code: Select all

Filesystem        1K-blocks   Used Available Use% Mounted on
rootfs                51200   2908     48292   6% /
/dev/root.old         11339   3202      8137  29% /initrd
none                  51200   2908     48292   6% /
ubi0:boot            949848 363732    581276  39% /boot
/dev/md0_vg/BFDlv   4128448   8188   3910548   1% /mnt/boot
/dev/loop0           717467 643359     74108  90% /mnt/apps
/dev/loop1             7657    433      6815   6% /mnt/etc
none                   7657    433      6815   6% /etc
tmpfs                 24776     16     24760   1% /run
tmpfs                  5120      0      5120   0% /run/lock
tmpfs                 49540      0     49540   0% /run/shm
Any ideas?

Mijzelf
Posts: 6206
Joined: Mon Jun 16, 2008 10:45 am

Re: ix2-ng recovery usb

Post by Mijzelf » Fri Jun 10, 2016 8:10 am

Can you post the output of

Code: Select all

cat /proc/partitions
cat /proc/mounts
cat /proc/mdstat
cat /proc/cmdline
vgdisplay
ls -l /sysroot/boot/images/

amr42
Posts: 28
Joined: Mon Jan 16, 2012 9:34 pm

Re: ix2-ng recovery usb

Post by amr42 » Tue Jun 14, 2016 9:58 pm

Surely...

Code: Select all

cat /proc/partitions
major minor  #blocks  name

   7        0     699824 loop0
   7        1       8192 loop1
   8        0  976762584 sda
   8        1   20971520 sda1
   8       16  976762584 sdb
   8       17   20971520 sdb1
  31        0        512 mtdblock0
  31        1        128 mtdblock1
  31        2        128 mtdblock2
  31        3       3072 mtdblock3
  31        4       4096 mtdblock4
  31        5    1040384 mtdblock5
  31        6    1048576 mtdblock6
   8       32     976896 sdc
   8       33     976828 sdc1
   9        0   20955008 md0
 253        0    4194304 dm-0

Code: Select all

cat /proc/mounts
rootfs / rootfs rw 0 0
/dev/root.old /initrd ext2 rw,relatime,errors=continue 0 0
none / tmpfs rw,relatime,size=51200k,nr_inodes=30949 0 0
ubi0:boot /boot ubifs rw,sync,noatime 0 0
/dev/md0_vg/BFDlv /mnt/boot ext2 rw,noatime,errors=continue 0 0
/dev/loop0 /mnt/apps ext2 ro,relatime 0 0
/dev/loop1 /mnt/etc ext2 rw,sync,noatime 0 0
none /etc unionfs rw,sync,noatime,dirs=/mnt/etc=rw:/mnt/apps/etc=ro 0 0
proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0
none /proc/bus/usb usbfs rw,relatime 0 0
none /proc/fs/nfsd nfsd rw,relatime 0 0
none /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0
devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620 0 0
tmpfs /run tmpfs rw,nosuid,noexec,relatime,size=24776k,mode=755 0 0
tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0
tmpfs /run/shm tmpfs rw,nosuid,nodev,noexec,relatime,size=49540k 0 0

Code: Select all

cat /proc/mdstat
Personalities : [linear] [raid0] [raid1] [raid10] [raid6] [raid5] [raid4] 
md0 : active raid1 sda1[0] sdb1[1]
      20955008 blocks super 1.2 [2/2] [UU]
                                                                        
unused devices: <none>

Code: Select all

cat /proc/cmdline
console=ttyS0,115200 root=/dev/ram0 :::DB88FXX81:eth0:none mtdparts=nand_mtd:0x8
0000@0(uboot),0x20000@0xa0000(env),0x20000@0xc0000(env2),0x300000@0x100000(uImag
e),0x400000@0x400000(initrd),0x3f800000@0x800000(boot),1024m@0x0(flash)

Code: Select all

vgdisplay
  --- Volume group ---
   VG Name               md0_vg
   System ID             
   Format                lvm2
   Metadata Areas        1
   Metadata Sequence No  2
   VG Access             read/write
   VG Status             resizable
   MAX LV                0
   Cur LV                1
   Open LV               1
   Max PV                0
   Cur PV                1
   Act PV                1
   VG Size               19.98 GiB
   PE Size               4.00 MiB
   Total PE              5115
   Alloc PE / Size       1024 / 4.00 GiB
   Free  PE / Size       4091 / 15.98 GiB
   VG UUID               nlWdSJ-MnDH-ohnB-ohox-LVJU-PpQb-pMLPER
 

Code: Select all

ls -l /sysroot/boot/images/

attempt to access beyond end of device
loop0: rw=0, want=1429140, limit=1399648
EXT2-fs error (device loop0): ext2_get_inode: unable to read inode block - inode=35921, block=714569
Remounting filesystem read-only
attempt to access beyond end of device
loop0: rw=0, want=1429140, limit=1399648
EXT2-fs error (device loop0): ext2_get_inode: unable to read inode block - inode=35921, block=714569
Remounting filesystem read-only
ls: error while loading shared libraries: libattr.so.1: cannot open shared object file: Input/output error


Mijzelf
Posts: 6206
Joined: Mon Jun 16, 2008 10:45 am

Re: ix2-ng recovery usb

Post by Mijzelf » Wed Jun 15, 2016 6:34 pm

Complicated. There is nothing mounted on /sysroot or /sysroot/boot. So I think /sysroot/boot is somehow a symlink to either /boot, which is the mountpoint of /dev/mtdblock5 (ubi:boot), or /mnt/boot, which is the mountpoint of /dev/md0_vg/BFDlv. According to your serial bootlog, /mnt/boot cannot contain apps which is 699824kB:

Code: Select all

/dev/md0_vg/BFDlv was not cleanly unmounted, check forced.
<snip>
       18511 blocks used (1.77%, out of 1048576)
As /md0_vg/BFDlv is about 4GB in size, a 'block' is apparently 4kB, which means the filesystem contains less then 80000kB.

So 'apps' is on flash. But there is something wrong with that, the boot partition seems to be full:

Code: Select all

UBI device number 0, total 8123 LEBs (1048061952 bytes, 999.5 MiB), available 0 LEBs (0 bytes), LEB size 129024 bytes (126.0 KiB)
I must admit that I don't know how to solve that. I guess the filesystem is corrupted somehow, as it should have some free space. But AFAIK you can't simply put a new filesystem on it.
As 'ls' doesn't work, due to corrupted libs, it's not simple to find out what's on the partition.

But maybe there is a trick. The original rootfs will have a version of busybox, which has 'ls' build-in. According to your 'mounts' the initrd is in /initrd. So *maybe* you can run busybox:

Code: Select all

LD_LIBRARY_PATH=/initrd/lib /initrd/bin/busybox
If that works, you can have a look at the contents of /boot:

Code: Select all

LD_LIBRARY_PATH=/initrd/lib /initrd/bin/busybox ls -l /boot/
LD_LIBRARY_PATH=/initrd/lib /initrd/bin/busybox ls -l /boot/images

amr42
Posts: 28
Joined: Mon Jan 16, 2012 9:34 pm

Re: ix2-ng recovery usb

Post by amr42 » Wed Jun 15, 2016 8:53 pm

Look like the filenames in these directories are corrupt... In my console (doesn't survive the paste below I see an <escape> character before each '[0' or '[1' ).
Mijzelf wrote:

Code: Select all


LD_LIBRARY_PATH=/initrd/lib /initrd/bin/busybox ls -l /boot/ 
                  
-rw-r--r--    1 root     root        927590 Jun 15 21:03 [0;0mSystem.map-2.6.31.8[0m
drwxr-xr-x    2 root     root           352 Jun  4 18:16 [1;34mimages[0m
    
LD_LIBRARY_PATH=/initrd/lib /initrd/bin/busybox ls -l /boot/images

-rwxr-xr-x    1 root     root     716619776 Jun  4 18:16 [1;32mapps[0m
-rwxr-xr-x    1 root     root       8388608 Jun  4 18:16 [1;32mconfig[0m
-rwxr-xr-x    1 root     root             0 Jun  4 18:16 [1;32moem[0m
       
Or maybe busybox ls format shows additional info, or color ... :)

Mijzelf
Posts: 6206
Joined: Mon Jun 16, 2008 10:45 am

Re: ix2-ng recovery usb

Post by Mijzelf » Thu Jun 16, 2016 7:30 pm

I think that busybox outputs colors. At least loop0 is associated with a file 'apps', which isn't available if it isn't color.

The flash partition doesn't contain enough data to be full. Maybe a hidden file? Try 'ls -la'

amr42
Posts: 28
Joined: Mon Jan 16, 2012 9:34 pm

Re: ix2-ng recovery usb

Post by amr42 » Sun Jun 19, 2016 3:46 pm

Mijzelf wrote:I think that busybox outputs colors. At least loop0 is associated with a file 'apps', which isn't available if it isn't color.

The flash partition doesn't contain enough data to be full. Maybe a hidden file? Try 'ls -la'
Unfortunately, no other files are shown with the ls -la option (just the '.' and '..' directories)....

Next?

Mijzelf
Posts: 6206
Joined: Mon Jun 16, 2008 10:45 am

Re: ix2-ng recovery usb

Post by Mijzelf » Tue Jun 28, 2016 6:27 pm

I'm a bit out of tricks. That ubi partition seems to have not enough space to contain the necessary files, but I don't see why. Creating a new ubifs on that partition is not straight forward, as you need to specify several low level properties of the flash memory. http://free-electrons.com/blog/creating ... fs-images/

You can try to use the imager key again, and catch the serial log, to see if there is some error message which can help us further.

Post Reply