Once I needed to make several mirrors in different countries for files that were often downloaded by users, so I made two scripts for synchronization.
The first server could connect to the main one via SSH using keys, so I used rsync:
#!/bin/bash
rsync -avh -e "ssh -p 22" --log-file=/srv/logs/$(date +%Y%m%d)_rsync_mirror.log user@ixnfo.com:/var/www/downloads/file.dmg /var/www/mirror/file.dmg
# Clear old logs
find /srv/logs/ -type d -mtime +90 -exec rm -rfv {} \;
And added a script to /etc/crontab so that it runs every 3 hours in 30 minutes:
30 */3 * * * root /srv/scripts/rsync_mirror.sh > /dev/null 2>&1
On the second server, it was impossible to connect to the main one via SSH, so I decided to simply check the file date using the main link and if the file is newer than on the mirror, then download it (the main part of the script is taken here https://gist.github.com/jacksgt/56dd2dfc115fde19ccf0):
#!/bin/bash
# Exit codes:
# -1 = local file up to date, nothing to do
# 0 = updated local file, successfull
# 1 (and above) = failure
REMOTE_FILE="https://ixnfo.com/testfile";
LOCAL_FILE="/var/www/mirror2/testfile";
REMOTE_DATE=$( date -d "$( curl -Is $REMOTE_FILE | grep 'Modified' | cut -d',' -f2 )" "+%s" );
if [[ ! "$REMOTE_DATE" =~ [[:digit:]] ]]; then
exit 1;
fi
if [[ -e "$LOCAL_FILE" ]]; then
LOCAL_DATE=$( stat "$LOCAL_FILE" -c "%Y" );
else
LOCAL_DATE=0;
touch "$LOCAL_FILE" >> /dev/null 2>&1;
if [ $? -eq 1 ]; then
exit 1;
fi
fi
# Check if we actually got only a digit
if [[ ! "$LOCAL_DATE" =~ [[:digit:]] ]]; then
exit 1;
fi
# Check if local file is younger than remote file
if [ $LOCAL_DATE -gt $REMOTE_DATE ]; then
exit -1;
else
curl -s "$REMOTE_FILE" -o "$LOCAL_FILE";
fi
exit 0;
For the test, you can create an empty file of several gigabytes to make sure that the script downloads the file from the main server only when it has changed:
dd if=/dev/zero of=testfile bs=4G count=1
See also my articles:
Installing and using rsync on Linux
How to compile cURL on Ubuntu