Backup data from HMNHD to another 1TB HD via USB

Post Reply
azk015
Posts: 3
Joined: Mon Jul 11, 2011 5:39 pm

Backup data from HMNHD to another 1TB HD via USB

Post by azk015 » Mon Jul 11, 2011 6:16 pm

1TB version of HMNDH. Firmware 2.0.64, have telnet / root access thanks to this forum.
I am pretty noob to linux so the answer may be pretty obvious but I don't know it.
Also searched the forum for something similar and couldn't find the answer.

I have almost 700GB worth of data on my HMNHD and have no backup of it. Bought a 1.5 TB USB external HD and formatted it to FAT32. I am able to connect this external HD to HMNHD via USB and backup data manually from HMNHD. (Files with size < 4GB)

Questions I have are:
1. Is it possible to use the external HD with NTFS partition (instead on FAT32) and be able to backup files with size > 4GB (HMNHD does not see the external HD automatically as it sees it when it is FAT32)
2. Is there a way to do only incremental backup (Copy only new / changed files)? (Currently I do delete all and copy over everything which takes hours).
3. If the incremental thing is possible, can I set it to run periodically? (I can leave the external HD always on and connected) or trigger the backup when external HD is connected? (This way I don't need to leave the external HD always on)

Thanks.

Mijzelf
Posts: 6226
Joined: Mon Jun 16, 2008 10:45 am

Re: Backup data from HMNHD to another 1TB HD via USB

Post by Mijzelf » Tue Jul 12, 2011 9:20 am

And answers:
1) Possibly. ntfs-3g seems to be installed on the NAS. However, I never tried it, I don't have any NTFS disks.
The procedure would be:
  • plugin the disk
  • use dmesg (or cat /proc/partitions) to find out the device name
  • manually mount it:

    Code: Select all

    mkdir /media/mountpoint
    mount -t ntfs-3g /dev/devicename /media/mountpoint
    or maybe

    Code: Select all

    mount.ntfs-3g /dev/devicename /media/mountpoint
    (you need to be root for this)
This won't give you very high write speeds though. Technically it would be better to use ext3, or xfs. But I understand this has other drawbacks.
2) Yes. rsync can do that. It's not a part of the firmware, but it can be installed using apt-get. You could also use rsnapshot (which uses rsync internally) to create 'snapshots'. You'll have complete snapshots taken at different times, while it uses hardly more diskspace than an incremental backup. (The trick is the use of hardlinks, a file which is available in several snapshots, is in reality only once on the disk, but it has different entries.) I don't know if rsnapshot works on NTFS, it will certainly not work on FAT.
3) Periodically actions can be arranged using cron. When you want something executed daily, put a script in /etc/cron.daily and restart cron:

Code: Select all

/etc/init.d/cron restart
If you want it weekly, use /etc/cron.weekly, etc.
More complex schemes are also possible, but then you'll have to edit /etc/crontab.
Action on a disk insertion is certainly possible, but I don't know an easy way. There are several ways:
  • use dbus
  • write a script which monitors /sys/block
  • write a script which periodically uses blkid to find out if a certain disk is inserted
  • ...
I suppose it should also be possible to hook hotplug, but I don't know how.

azk015
Posts: 3
Joined: Mon Jul 11, 2011 5:39 pm

Re: Backup data from HMNHD to another 1TB HD via USB

Post by azk015 » Fri Jul 15, 2011 6:22 pm

Thanks for your quick reply. Finally got the time to try out these things.
1. For some reason with entire 1.5 TB external drive with single NTFS partition, it did not auto-mount the volume when I plugged in the hard drive but when I had a smaller NTFS partition on same external HD (500GB), it was auto-mounted when I plugged it in the USB. (I guess using ntfs-3g)

2. I was able to install rsync using apt-get and got the right combination of parameteres to get it working.

3. For your "speed comment" I did some tests.
Made 3 partitions on my external HD NTFS, FAT32 and ext3 and copied ~ 150MB of data to each partition using rsync.

Here are the transfer speeds I got.
NTFS 990142.30 bytes/sec
ext3 3077469.71 bytes/sec
FAT32 3133935.01 bytes/sec

NTFS was the slowest, FAT32 was faster than ext3 but then next time I ran rsync again with no changes at source, it still copied everything again. So I thin right now ext3 seems to be best option.

My question is 3077469.71 bytes/sec (~ 2.94 MB / S) look right? I was expecting much transfer speed when it is connected directly with USB. (This comes out to be ~10GB /hr. No wonder the rsync I started yesterday night for my media folder ~ 500GB is still running this morning and will go on for 2 more days??)

Is rsync slower than cp? The destination was empty during the tests so I think using rsync and using copy (cp) would give me same transfer speed (??).

I haven't looked at cron yet, but I got webmin installed and working so I can schedule the jobs using it's UI (which I think internally schedules a cron job) but that's for later when I have a script to rsync all my folders together.

Mijzelf
Posts: 6226
Joined: Mon Jun 16, 2008 10:45 am

Re: Backup data from HMNHD to another 1TB HD via USB

Post by Mijzelf » Sat Jul 16, 2011 7:49 am

My question is 3077469.71 bytes/sec (~ 2.94 MB / S) look right?
(How did you measure this?)
I did some measurements on my own box:
Raw read performance on HD:

Code: Select all

root@Iomega-09681e:/nethdd/Isaac/iomega.gpl# dd if=/dev/sda1 of=/dev/null bs=16k
257040+0 records in
257040+0 records out
4211343360 bytes (4.2 GB) copied, 133.716 s, 31.5 MB/s
Raw write speed on USB HD:

Code: Select all

root@Iomega-09681e:/mnt# dd if=/dev/zero of=/dev/sdb2 bs=16k
dd: writing `/dev/sdb2': No space left on device
14559+0 records in
14558+0 records out
238533120 bytes (239 MB) copied, 18.7744 s, 12.7 MB/s
Copying a big directory:

Code: Select all

root@Iomega-09681e:/nethdd/Mijzelf/iomega.gpl# time cp -a * /mnt/sdb1/tmp/waste
real	6m39.153s
user	0m13.550s
sys	3m55.340s
Size of big directory:

Code: Select all

root@Iomega-09681e:/nethdd/Mijzelf/iomega.gpl# du -B 1024 -c .
<snip>
2847424	total
So copying to a (ext3) USB device does about 7.1MB/sec.
FAT32 was faster than ext3 but then next time I ran rsync again with no changes at source, it still copied everything again. So I thin right now ext3 seems to be best option.
I suppose you mean rsync on FAT started all over again, but on ext3 it only 'filled the holes'? Interesting. I suppose this is due to poor timestamp metadata on FAT.
Is rsync slower than cp?
Yes. On the first run it's slower, AFAIK it does some checksumming on the files. Further it uses a client/server setup, even with local copies, so there is some extra overhead. So for a first run you might prefer 'cp -a' (don't forget the -a!). Or find out how to influence this rsync behaviour using one of the zillion options. Or just wait a little longer. It's only once.
(which I think internally schedules a cron job)
And so it does.

markarthur
Posts: 5
Joined: Wed Jan 25, 2012 4:28 pm

Re: Backup data from HMNHD to another 1TB HD via USB

Post by markarthur » Wed Jan 25, 2012 4:54 pm

This is the great Iomega’s HMNHD Edition, which is available in 1TB and another one is of 2TB versions, is one of the best new breed of NAS devices. Usually iomega chooses the lengthy name for its products to show the breadth of the product. In terms of the user experience, it's one of the more elegant NAS device which anybody has ever seen with which you can backup data to another Hard Drive of large limit.

Post Reply