Moving files created full disk

Post Reply
verdian
Posts: 5
Joined: Tue Apr 15, 2014 7:16 am

Moving files created full disk

Post by verdian » Tue Apr 15, 2014 7:27 am

Hello, long time reader, first time poster here.
Yesterday my NS2 stopped working and I had to do a factory reset. I had it rooted and decided it was time to do an update of the firmware and root it again.
Because of the factory restore all files on it were moved to the /shares/MyShare/Recovery folder.
Before the recovery I had about 100Gb free space left. Then I logged in with SSH to move my files to the proper location again and started to use the mv command to move a bunch of folders and files to the correct shares ("mv /shares/MyShare/Recovery/Openshare/ /shares/Openshare/ -r" for example).
Now every time my NAS tells me after a bunch of files there is no disk space left to create the files.
I have no clue what i'm doing wrong, or how to recover the files to the correct location.

I am assuming it has to do with some files being placed on a different partition... but I'm not sure how to solve this.
Anyone can tell me how to recover this mess?
Thanks in advance.

Mijzelf
Posts: 6197
Joined: Mon Jun 16, 2008 10:45 am

Re: Moving files created full disk

Post by Mijzelf » Tue Apr 15, 2014 11:09 am

Can you post the output of

Code: Select all

df
and

Code: Select all

cat /proc/mounts

verdian
Posts: 5
Joined: Tue Apr 15, 2014 7:16 am

Re: Moving files created full disk

Post by verdian » Tue Apr 15, 2014 11:13 am

Wow thanks for the fast reply.
Here is the output:

Code: Select all

Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/sda7                 7683      6061      1226  83% /oldroot
udev                     10240        60     10180   1% /dev
none                    126880        40    126840   0% /dev/shm
/dev/sda8               837912    334472    460876  42% /oldroot/var/original
/dev/sda9               861728    386512    431444  47% /oldroot/snapshots
unionfs                 861728    386512    431444  47% /
/dev/sda9               861728    386512    431444  47% /var
/dev/sda9               861728    386512    431444  47% /tmp
/dev/sda9               861728    386512    431444  47% /bin
/dev/sda9               861728    386512    431444  47% /sbin
/dev/sda9               861728    386512    431444  47% /lib
/dev/sda9               861728    386512    431444  47% /usr
/dev/sda9               861728    386512    431444  47% /www
/dev/sda2            1951372832 1920940812  30432020  98% /media/internal_6
/dev/sda2            1951372832 1920940812  30432020  98% /lacie/tmp
/dev/sda2            1951372832 1920940812  30432020  98% /lacie/var
/dev/sda2            1951372832 1920940812  30432020  98% /lacie/torrent_dir
/dev/sda2            1951372832 1920940812  30432020  98% /lacie/autoupdate
/dev/sda2            1951372832 1920940812  30432020  98% /lacie/afp_db
/dev/sda2            1951372832 1920940812  30432020  98% /shares/MyShare
/dev/sda2            1951372832 1920940812  30432020  98% /shares/OpenShare
/dev/sda2            1951372832 1920940812  30432020  98% /shares/StarSoft
tmpfs                   126880       596    126284   0% /tmp/samba
and

Code: Select all

rootfs / rootfs rw 0 0
/dev/sda7 /oldroot ext3 ro,relatime,errors=continue,data=writeback 0 0
none /oldroot/proc proc rw,relatime 0 0
none /oldroot/sys sysfs rw,relatime 0 0
udev /dev tmpfs rw,relatime,size=10240k,mode=755 0 0
devpts /dev/pts devpts rw,relatime,mode=600 0 0
none /dev/shm tmpfs rw,relatime 0 0
/dev/sda8 /oldroot/var/original ext3 ro,relatime,errors=continue,data=writeback                                                                                         0 0
/dev/sda9 /oldroot/snapshots ext3 rw,relatime,errors=continue,data=writeback 0 0
unionfs / unionfs rw,relatime,dirs=/oldroot/snapshots/snaps/00=rw:/oldroot/var/o                                                                                        riginal=ro 0 0
proc /proc proc rw,relatime 0 0
sysfs /sys sysfs rw,relatime 0 0
/dev/sda9 /var ext3 rw,relatime,errors=continue,data=writeback 0 0
/dev/sda9 /tmp ext3 rw,relatime,errors=continue,data=writeback 0 0
/dev/sda9 /bin ext3 rw,relatime,errors=continue,data=writeback 0 0
/dev/sda9 /sbin ext3 rw,relatime,errors=continue,data=writeback 0 0
/dev/sda9 /lib ext3 rw,relatime,errors=continue,data=writeback 0 0
/dev/sda9 /usr ext3 rw,relatime,errors=continue,data=writeback 0 0
/dev/sda9 /www ext3 rw,relatime,errors=continue,data=writeback 0 0
usbfs /proc/bus/usb usbfs rw,relatime,devgid=85,devmode=664 0 0
/dev/sda2 /media/internal_6 xfs rw,nosuid,nodev,relatime,attr2,allocsize=4096k,u                                                                                        srquota,prjquota,grpquota 0 0
/dev/sda2 /lacie/tmp xfs rw,nosuid,nodev,relatime,attr2,allocsize=4096k,usrquota                                                                                        ,prjquota,grpquota 0 0
/dev/sda2 /lacie/var xfs rw,nosuid,nodev,relatime,attr2,allocsize=4096k,usrquota                                                                                        ,prjquota,grpquota 0 0
/dev/sda2 /lacie/torrent_dir xfs rw,nosuid,nodev,relatime,attr2,allocsize=4096k,                                                                                        usrquota,prjquota,grpquota 0 0
/dev/sda2 /lacie/autoupdate xfs rw,nosuid,nodev,relatime,attr2,allocsize=4096k,u                                                                                        srquota,prjquota,grpquota 0 0
/dev/sda2 /lacie/afp_db xfs rw,nosuid,nodev,relatime,attr2,allocsize=4096k,usrqu                                                                                        ota,prjquota,grpquota 0 0
/dev/sda2 /shares/MyShare xfs rw,nosuid,nodev,relatime,attr2,allocsize=4096k,usr                                                                                        quota,prjquota,grpquota 0 0
/dev/sda2 /shares/OpenShare xfs rw,nosuid,nodev,relatime,attr2,allocsize=4096k,u                                                                                        srquota,prjquota,grpquota 0 0
/dev/sda2 /shares/StarSoft xfs rw,nosuid,nodev,relatime,attr2,allocsize=4096k,us                                                                                        rquota,prjquota,grpquota 0 0
tmpfs /tmp/samba tmpfs rw,relatime 0 0

Mijzelf
Posts: 6197
Joined: Mon Jun 16, 2008 10:45 am

Re: Moving files created full disk

Post by Mijzelf » Tue Apr 15, 2014 11:36 am

OK, as you can see in the mounts list, the two shares are actually on the same partition/filesystem, sda2. But for some strange reason Lacie decided to bindmount the shares on different directories, which means that from the view of mv the files are on different filesystems.
That means that a move is actually a copy-and-delete action. As the disk has 'only' 30GB free, this means that the move will fail on a file bigger than that.

You can try to move on a lower level, so that it's only an administrative move. According to the mounts list, /dev/sda2 is first mounted on /media/internal_6. So have a look if you can find something like /media/internal_6/shares/MyShare. Maybe the '/shares/' directory has a different name.
If you have found that, apply

Code: Select all

mkdir -p /media/internal_6/shares/Openshare/moved
mv /media/internal_6/shares/MyShare/Recovery/Openshare/* /media/internal_6/shares/Openshare/moved/
(where of course /shares/ has to be replaced). This should move your files instantly.
I used the /moved/ directory because you already party moved the files, and mv won't merge. When this is done, you can move the files from /moved/ to / using samba or something like that.

verdian
Posts: 5
Joined: Tue Apr 15, 2014 7:16 am

Re: Moving files created full disk

Post by verdian » Tue Apr 15, 2014 1:05 pm

I did what you wrote, only difference is that the internal_6 folder contains 3 folders (named 1,2,3) which correspond with the users on the NAS it seems. Folder 1 contained the Recovery folder, and folder 3 is the one that contains the files of the StarSoft user.
The process just completed and gave the same problem.
The move failed because the drive ran out of free space.
I don't have any massive files on there, maybe the biggest might be 10Gb or so, but that's it.
Here is the output from the df command when i do it now (where the disk is full)

Code: Select all

Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/sda7                 7683      6061      1226  83% /oldroot
udev                     10240        60     10180   1% /dev
none                    126880        40    126840   0% /dev/shm
/dev/sda8               837912    334472    460876  42% /oldroot/var/original
/dev/sda9               861728    386616    431340  47% /oldroot/snapshots
unionfs                 861728    386616    431340  47% /
/dev/sda9               861728    386616    431340  47% /var
/dev/sda9               861728    386616    431340  47% /tmp
/dev/sda9               861728    386616    431340  47% /bin
/dev/sda9               861728    386616    431340  47% /sbin
/dev/sda9               861728    386616    431340  47% /lib
/dev/sda9               861728    386616    431340  47% /usr
/dev/sda9               861728    386616    431340  47% /www
/dev/sda2            1951372832 1951372812        20 100% /media/internal_6
/dev/sda2            1951372832 1951372812        20 100% /lacie/tmp
/dev/sda2            1951372832 1951372812        20 100% /lacie/var
/dev/sda2            1951372832 1951372812        20 100% /lacie/torrent_dir
/dev/sda2            1951372832 1951372812        20 100% /lacie/autoupdate
/dev/sda2            1951372832 1951372812        20 100% /lacie/afp_db
/dev/sda2            1951372832 1951372812        20 100% /shares/MyShare
/dev/sda2            1951372832 1951372812        20 100% /shares/OpenShare
/dev/sda2            1951372832 1951372812        20 100% /shares/StarSoft
tmpfs                   126880       596    126284   0% /tmp/samba
[edit]
I just found out the files are actually not moved, but remain on the original location... Is that because of the command failing halfway and not deleting the original files because of it?
Many files that are copied are 0 bytes big, so obviously something went wrong.

Mijzelf
Posts: 6197
Joined: Mon Jun 16, 2008 10:45 am

Re: Moving files created full disk

Post by Mijzelf » Tue Apr 15, 2014 3:36 pm

Is that because of the command failing halfway and not deleting the original files because of it?
mv isn't supposed to copy-and-delete, when moving on the same filesystem. It just generates a new directory entry on the target directory, and removes the one on the source.
Many files that are copied are 0 bytes big
As the filesize is a property of the file, and not of the directory entry, this means those are not hardlinks to the orginal files.

I am beginning to think you have a filesystem problem. Does

Code: Select all

dmesg | tail -n 50
show any filesystem errors?

verdian
Posts: 5
Joined: Tue Apr 15, 2014 7:16 am

Re: Moving files created full disk

Post by verdian » Tue Apr 15, 2014 4:29 pm

I don't think so.
I posted the output of the command here: http://codebin.org/view/f8d58fb6
But the only failure i see is something about page allocation error.
I started to move the files with small batches manually over to the new location, that seems to work fine. Guess there is a problem with the move command which will only delete the files after the whole command has succeeded. Which obviously causes the disk to run out of space and fail, thus leaving the files in the original location.
It will take some time for this to be done, especially when i reach my movie collection, but I see this as the only way to move the files back to the original location, other than renaming the share and creating a symlink in the new location.
If you can look at the attached output and see if there is something wrong it would be great.
I will continue moving my files manually and if I run into more trouble I will let you know.
Thanks allot for the help already.

Mijzelf
Posts: 6197
Joined: Mon Jun 16, 2008 10:45 am

Re: Moving files created full disk

Post by Mijzelf » Tue Apr 15, 2014 7:07 pm

That dmesg listing doesn't look good to me. The stackdump shouldn't be there, and it's not only one, there are 3 of them, in these few lines. Alas I can't find any documentation about 'sys_lgetxattr'. But I wouldn't be surprised if it has to do with your problems.
It might be a good idea to check the filesystem.

Code: Select all

xfs_check /dev/sda2
To do so, the filesystem has to be unmounted, or remounted ro. Unmounting can be difficult, but remounting ro might just work:

Code: Select all

mount -o remount,ro /dev/sda2

verdian
Posts: 5
Joined: Tue Apr 15, 2014 7:16 am

Re: Moving files created full disk

Post by verdian » Tue Apr 15, 2014 9:12 pm

That doesnt sound good.
I tried to unmount the partition, but can't get it done, it shows the following error

Code: Select all

mount: mounting /dev/sda2 on /shares/MyShare failed: Device or resource busy
Also running xfs_check obviously wont work as you said without unmounting it.

Code: Select all

xfs_check: /dev/sda2 contains a mounted and writable filesystem

fatal error -- couldn't initialize XFS library
Running now badblocks on sda2. Will see if that gives something...

Mijzelf
Posts: 6197
Joined: Mon Jun 16, 2008 10:45 am

Re: Moving files created full disk

Post by Mijzelf » Wed Apr 16, 2014 5:39 am

If you have the possibility, you can take the disk out, connect it to a Linux PC, and do the check there.

Post Reply