Finally a backup for our Kodi media centre

Submitted by charm on Thu, 09/09/2021 - 18:44
The top process in htop is an rsync session copying from one 8TB drive to another 8TB drive

Backups are important. I know backups are important. But for the past 2 years we've gone without a backup for the KODI media centre in our living room. Why haven't I backed up our server when I know it's important? In one word: cost. My budget for hardware is pretty limited. This is the main reason our media centre isn't running a new Ryzen processor with an m.2 drive. It's the reason why when I wanted to upgrade my own desktop (from an old Gigabyte motherboard and an A8-5600K APU) I instead opted for an Atermiter X79 Turbo motherboard and E5-1650 CPU from AliExpress.

That all changed thanks to an early birthday gift, a 4TB WD RED drive, from one of my brothers. Like a bad brother I went and returned the drive (still in the shrinkwrap) and got a credit then paid an extra $145CDN to buy an 8TB Seagate Ironwolf drive to match the 8TB Ironwolf I already had in the media centre. I considered using the 4TB drive in my desktop, but the 8TB Seagate Ironwolf was on sale. The first 8TB drive was purchased back when the 8TB Ironwolf NAS (non-pro) was $365CDN, this sale made it approximately $100 off what the drive originally listed for.

The new 8TB Seagate Ironwolf hard drive - finally a backup!
The new 8TB Seagate Ironwolf hard drive - finally a backup!

Unlike my first drive, the SATA controller recognized the second 8TB drive right away. I used:

dmesg | grep sd

To list the drives recognized by the system. In my case sda is the 500GB SSD, sdb is /dev/media, and sdc is the new 8TB backup drive. I then partitioned the drive using cfdisk:

sudo cfdisk /dev/sdc

I created a single 8TB partition the same as I did with the first 8TB drive. Then I used mkfs.ext4 to format the partition on /dev/sdc1 as ext4:

sudo mkfs.ext4 /dev/sdc1

With the drive partitioned and formatted it was time to add it so that Xubuntu Linux would recognize it on startup. Linux distributions use /etc/fstab to recognize drives to be mounted at startup. I used blkid to list the drives and partitions along with their universally unique identifiers (UUIDs). Typing:

sudo blkid

results in a list of drives/partitions and their UUID. For example:

/dev/sda5: UUID="65ad3476-2704-4a74-bdd9-5c8f92c4e4c9" TYPE="ext4" PARTUUID="443ebe12-05"
/dev/sda1: UUID="39E6-6E63" TYPE="vfat" PARTUUID="443ebe12-01"
/dev/sdb1: LABEL="storage" UUID="8b353bdf-9481-4779-a2c9-59e430ef0596" TYPE="ext4" PARTLABEL="storage" PARTUUID="854dde24-0336-4731-96d1-0d8ae5a4969e"
/dev/sdc1: UUID="eb0fec23-0e93-42b7-ae7b-0d2607265d1e" TYPE="ext4" PARTUUID="cd8cf00f-2dfc-204d-a4d4-08d94fb0f60b"

I now had the UUID for /dev/sdc1, but before I could add it to /etc/fstab I needed a place to mount the drive (in Windows terms think drive e:, f:, g:). Our media drive is mounted as /mnt/media. I decided /mnt was a good place to put the backup drive mount point as well, calling it BACKUP:

sudo mkdir /mnt/BACKUP
sudo chown -R linuxuser.linuxuser /mnt/BACKUP

The first command above makes a mount point/directory in /mnt called BACKUP. The second command above changes permission on the drive so that my username and group (both linuxuser) have permission to create and delete files and directories in that /mnt/BACKUP directory. (But I cannot delete files in /mnt, the directory above) With the mount point created it was now time to assign the drive to the mount point in /etc/fstab. Our /etc/fstab now looks like this:

UUID=65ad3476-2704-4a74-bdd9-5c8f92c4e4c9 /               ext4    errors=remount-ro 0       1
UUID=8b353bdf-9481-4779-a2c9-59e430ef0596 /mnt/media      ext4    errors=remount-ro 0       1
UUID=eb0fec23-0e93-42b7-ae7b-0d2607265d1e /mnt/BACKUP     ext4    errors=remount-ro 0       1

# /boot/efi was on /dev/sda1 during installation
UUID=39E6-6E63  /boot/efi       vfat    umask=0077      0       1
/swapfile                                 none            swap    sw              0       0

The /mnt/BACKUP is the third line in the above example. With /etc/fstab saved it was time to reboot so the /mnt/BACKUP could be recognized. On reboot I ran df -hH to see the mounted directories (for the purpose of keeping this short I've removed temporary directories - tempfs):

Filesystem      Size  Used Avail Use% Mounted on
/dev/sda5       491G   46G  421G  10% /
/dev/sda1       536M  4.1k  536M   1% /boot/efi
/dev/sdb1       8.0T  1.4T  6.3T  18% /mnt/media
/dev/sdc1       8.0T  744G  6.8T  10% /mnt/BACKUP

The keen eyed might notice /mnt/BACKUP is 10% full. I actually ran this command after I started backing up the /mnt/media drive to it. To back up the drive I used the rsync command. One problem: I needed to exclude the directory /mnt/media/Movies/Blu-ray/Uncompressed since some of the files in it are files that I hadn't compressed yet. (I normally run a command-line version of handbrake to compress my media). The rsync command I typed was as follows:

nohup rsync -av --exclude '/mnt/media/Movies/Blu-ray/Uncompressed' /mnt/media /mnt/BACKUP &

I'll unpack this command a bit since there are a few things going on. I started the command with nohup because I was logged in to our media centre from another computer using secure shell (SSH). The nohup command lets the command run even after I log out of the SSH session. The meat of the command is:

rsync -av --exclude '/mnt/media/Movies/Blu-ray/Uncompressed' /mnt/media /mnt/BACKUP

This copies the files and directories in /mnt/media to /mnt/BACKUP, but it excludes (doesn't copy) anything in /mnt/media/Movies/Blu-ray/Uncompressed. The last part of the command, the & symbol, just lets me continue to use the SSH session. Without the & symbol the rsync would run, but I couldn't use the command prompt since rsync would be open in the session. I could open another session, but the & just nicely tucks the rsync command into the background letting me use the ssh session to type more commands.

It's now 46 minutes into the command and just over 950GB of the 1.4TB currently on /mnt/media has been copied to /mnt/BACKUP. This works out to approximately 20.65GB/minute being copied.

One last bit. I decided to schedule periodic backups every Sunday at 4am in the morning. To accomplish this I used:

crontab -e

Crontab opens up in the vi editor by default. Once finished editing the file you'll need to use :wq! to write and quit the crontab file. In my crontab I had as follows:

0 4 * * 7 rsync -av --exclude '/mnt/media/Movies/Blu-ray/Uncompressed' /mnt/media /mnt/BACKUP

I chose to back up only on Sunday to allow a full week for me to add new files. I know not to add any files Sunday at 4am. If memory serves me correctly the next time rsync runs it will only copy what's changed since the last backup. So unless I add a lot of new large files the process should complete pretty quickly.