Local Cloud

Following on the heels of the server configuration, we are now in deep with this project, no turning back and no slowing down. Keep on keeping on.

Linux Client side

Here is where the actual backup magic will happen. We need a backup script and a way to schedule the backup script.

Installation

• For my clients, I have one Dell Latitude E7240 running EndeavourOS, which is the current build we are focusing on and one Dell Latitude E6440 running Antergos (legacy build which has not switched over to EndeavourOS). For all intents and purposes, the systems are the same, both based on Arch Linux.

SSH Key Creation, Sharing, and Connection

• On the client, create a key, send it over to server, and make sure the server is a trusted system for the client.
  1. The default RSA type is created with ssh-keygen
     • Type in a password to encrypt private key.
  2. To send over the public SSH key from client to server, you have two options:
     1. Sneakernet, literally put the public key on a USB drive and bring it over to the server (copy into /home/serveruser1/.ssh.
     2. From the client, ssh-copy-id which will send the public key if it is stored in the default location (~/.ssh/).
  3. Let us make sure the client recognizes the server. I run my client system with two users as well, one admin and one non sudo user for my every day activities. As such:
     • Make an SSH connection from your normal user account (clientuser1):
       1. ssh serveruser1@Server_IP_Address
          • The authenticity of host 'Server_IP_Address (Server_IP_Address)' can't be established.
          • …key fingerprint is YourServerKeyFingerPrint
          • Are you sure you want to continue connecting (yes/no/[fingerprint])?
       2. When prompted if you trust the connection, type yes.
       3. This connection is stored under ~/.ssh/known_hosts.
     • Now make an SSH connection from your admin account, but not as the admin directly, instead use sudo from that account. (clientadmin)
       1. sudo ssh serveruser1@Server_IP_Address
          • The authenticity of host 'Server_IP_Address (Server_IP_Address)' can't be established.
          • …key fingerprint is YourServerKeyFingerPrint
          • Are you sure you want to continue connecting (yes/no/[fingerprint])?
       2. This message should appear:
       3. When prompted if you trust the connection, type yes.
       4. This connection is stored under /root/.ssh/known_hosts
          • I have found that since my anacron scripts execute under the system, not a user, the backup has failed unless the system itself saves my server as a known host.

How to Access Encrypted SSH Key

• We need a way of invoking the encrypted private key. After all, most users do not want to type in their password to decrypt their private key with every connection, which is particularly tough to do for scheduled scripts.
  1. Install and configure Keychain.
     • sudo pacman -S keychain
  2. To invoke, edit your .bashrc file{1} -- nano /home/clientuser1/.bashrc
     • eval `keychain --agents ssh --eval /home/clientuser1/.ssh/NameOfKey`
     • When you open a terminal window, you are prompted to unlock your key by typing in private key password created in step 1 of client side configuration. Until the computer is rebooted or shutdown and powered on again (suspend, hibernate, and logout should all be fine), or you manually lock access, the key will be available to our configured backup script and any other SSH commands.
  3. To somewhat automate the process of unlocking at boot, I create a startup item to launch a terminal window at first login. Again, as I am working in XFCE, the details will vary slightly depending on your desktop environment
     • Launch "Session and Startup from the XFCE" menu.
     • Navigate to Application Autostart.
     • Click Add and then fill in:
       1. Name: E.g. "Launch Terminal"
       2. Description: E.g. SSH Unlock
       3. Command: /bin/bash
       4. Trigger: on login

{1} Not .bash_profile, as .bashrc is sourced by login and non-login shells.

The Rsync Backup Script

• We need an rsync backup script, after all, no script, no backup! Luckily rsync is preinstalled on EndeavourOS so our server and client are good to go.
  1. Create a backup script! Let's assume we want to backup our user directory on our client computer, over SSH to a mount point on our server of course, but we want archived/deleted files to be placed in time stamped folders instead of completely removed.
     • In the text editor of your choice, type this command out. Editing the information with your own as needed.
     • rsync -abvh -e ssh --backup-dir=/media/serveruser1/Backup_DATA/old_`date +%F_%H-%M-%S` --delete /home/clientuser1/ serveruser1@ServerIPAddress:/media/serveruser1/Backup_DATA/ClientUser1_Backup
       • Let's break down that command shall we?
       • -a stands for archive and is actually a grouping of -rlptgoD options.
       • -b stands for backup, notice I specified a backup directory to coincide with this option.
       • -v stands for verbose, in other words when you execute the script it will show everything that is happening.
       • -h stands for human readable progress, it gives output number in a human readable format.
       • -e specifies the remote shell to use, ssh in our case, hence the SSH following
       • --backup-dir shows the directory on our server and makes date stamped folders in this format old_YYYY-MM-DD_HH-MM-SS, so old_2019-10-07_20-40-53 stands for October 7, 2019 at 10:40PM and 53 seconds.
       • --delete tells our backup to delete files not found on the host, so the main backup folder on the server matches the user folder on the client computer. Which is why I decided to setup the "changed/deleted items folder".
       • Next we have the desired directory we want to sync on my client computer. Trailing slash on the source matters.
         1. With a trailing slash instructs rsync to copy everything in the directory.
         2. No trailing slash instructs rsync to copy the directory itself in addition to its contents.
       • Followed by the destination directory on the server. Notice two things about the destination
         1. We have the desired user account name and server IP followed by the actual destination directory.
         2. There is no trailing slash. A trailing slash has no effect on the destination, in contrast to the source, so we can omit without worry.
  2. Before adding the necessary information to the script so the SSH key will be accessible on demand, let us make sure the rsync command actually works correctly.
     • To do a dry run, copy and paste the script we just built into a terminal windows and append -n to it. For instance you can start with rsync -abvhn followed by the rest of the script.
     • A dry run will literally just test the script without sending any files to the destination.
     • If you have not unlocked your key via the terminal, the script will obviously fail.
  3. Did it work? Fantastic! Take a moment to applaud your hard work as we have accomplished quite a bit.
  4. Let us make the script executable (assuming we have the following directory location and script name for our backup script).
     • chmod +x /home/clientuser1/Documents/Backup/users_remote_backup{2}
  5. Now we will finish the script by giving it access to our private key. Again, in the text editor of your choice add the following to our existing script.

#!/bin/bash

# Make sure you can log in to remote server without a password

eval `keychain --noask --eval /home/clientuser1/.ssh/NameOfKey`

source /home/clientuser1/.keychain/ComputerHostName-sh; rsync -abvh -e ssh --backup-dir=/media/serveruser1/Backup_DATA/old_`date +%F_%H-%M-%S` --delete /home/clientuser1/ serveruser1@ServerIPAddress:/media/serveruser1/Backup_DATA/ClientUser1_Backup

So what does all the gibberish mean? We are telling the script where the private key is located and the source of our keychain variable so it can be unlocked, followed by the actually rsync script.

{2} Whether you end your backup script with .sh or if you leave it bare will depend on how you execute the script.

• If you place your backup script in the /etc/cron.daily, /etc/cron.weekly, or /etc/cron.monthly directories, anacrontab will use run-parts to execute the script. Do not append your script with .sh if placed in those directories as run-parts will error out.
• If you add a dedicated line in anacrontab to launch your script, then appending the backup script with .sh is suggested.

Scheduling an Rsync Backup Script

• Okay, so the problem with regular cron is that it runs at set intervals. If a backup is missed because a system is offline (like when a laptop is sleeping), nothing happens until the next scheduled backup. There are a few different ways to handle asynchronous scheduling on Linux, but I have chosen anacron which is installed via the cronie package.
  1. Cronie is not installed by default, so let us rectify its absence, with an invitation to our party -- sudo pacman -S cronie
     • As with other services, such as openssh, we need to enable and start the cronie service with systemd.
       • sudo systemctl enable cronie.service
       • sudo systemctl start cronie.service
  2. There are a few different ways to schedule our script with cronie. We can run our script daily, weekly, or monthly. Also, you can store your script in different places, but the default location for each is:
     • For daily, /etc/cron.daily/
     • For weekly, /etc/cron.weekly/
     • For monthly, /etc/cron.monthly/
       • Since we probably want our backup to run daily, let us copy the script to the daily directory.{3} From your admin account (that pesky sudo is needed again)
         • sudo cp /home/clientuser1/Documents/Backup/users_remote_backup /etc/cron.daily/
           • Which tells us to copy the script we made to the cronie folder for daily job processing.
     • Alternately you can store your script where ever you want, but the /etc/anacrontab file needs to be edited to add a line for your script.
       • sudo /etc/anacrontab
       • @daily 10 mybackup.daily /bin/bash /home/clientuser1/Documents/Backup/users_remote_backup.sh{4}
         • Which means execute our script daily, with a 10 minute delay to start job, bash is the command to use, and our script is located at that location.
     • Other reasons you might want to edit /etc/anacrontab,
       • Perhaps you want to change the hours the backup runs (default is 3-22 or 3am to 10pm)
       • The length, if any of the random delay to execute the backup.
  3. Okay, we have everything needed to schedule your backup script, but what happens if you are on battery, which is common for laptops. No problem, while anacron defaults to only running when the system is connected to power, we can tweak the setting to run on battery. There are a few methods to accomplish this task, but my preferred method is as follows.
     • In your clientadmin account, first backup the 0anacron file cp
     • sudo nano etc/cron.hourly/0anacron
       • Comment out the second part of the file starting under # Do not run jobs when on battery power
         • Yes, you could delete this section, but commenting out means you can change your mind later, then again, we made a backup for this config file too, just like we have every step of the way?
         • Notice that every time cronie is updated, the installer rewrites this file. We could try locking it so it does not get overwritten, or perhaps set up a notification for when it gets overwritten, but otherwise you have to edit it back every time cronie is updated or your backups will revert back to only running when your system is connect to a power source.

{3} This is the method I use, copying the backup script into /etc/cron.daily/ and allowing run-parts to execute.

{4} As described in footnote 2, run-parts does not like files ending in . or .sh, but since we are adding a specific script in this instance, you can leave .sh appended to your script.