File mirroring script

Once I needed to make several mirrors in different countries for files that were often downloaded by users, so I made two scripts for synchronization.

The first server could connect to the main one via SSH using keys, so I used rsync:

rsync -avh -e "ssh -p 22" --log-file=/srv/logs/$(date +%Y%m%d)_rsync_mirror.log /var/www/mirror/file.dmg
# Clear old logs
find /srv/logs/ -type d -mtime +90 -exec rm -rfv {} \;

And added a script to /etc/crontab so that it runs every 3 hours in 30 minutes:

30 */3 * * * root /srv/scripts/ > /dev/null 2>&1

On the second server, it was impossible to connect to the main one via SSH, so I decided to simply check the file date using the main link and if the file is newer than on the mirror, then download it (the main part of the script is taken here


# Exit codes:
# -1 = local file up to date, nothing to do
# 0 = updated local file, successfull
# 1 (and above) = failure


REMOTE_DATE=$( date -d "$( curl -Is $REMOTE_FILE | grep 'Modified' | cut -d',' -f2 )" "+%s" );

if [[ ! "$REMOTE_DATE" =~ [[:digit:]] ]]; then
        exit 1;

if [[ -e "$LOCAL_FILE" ]]; then
        LOCAL_DATE=$( stat "$LOCAL_FILE" -c "%Y" );
        touch "$LOCAL_FILE" >> /dev/null 2>&1;
        if [ $? -eq 1 ]; then
                exit 1;

# Check if we actually got only a digit
if [[ ! "$LOCAL_DATE" =~ [[:digit:]] ]]; then
        exit 1;

# Check if local file is younger than remote file
if [ $LOCAL_DATE -gt $REMOTE_DATE ]; then
        exit -1;
        curl -s "$REMOTE_FILE" -o "$LOCAL_FILE";

exit 0;

For the test, you can create an empty file of several gigabytes to make sure that the script downloads the file from the main server only when it has changed:

dd if=/dev/zero of=testfile  bs=4G  count=1

See also my articles:
Installing and using rsync on Linux
How to compile cURL on Ubuntu

Leave a comment

Leave a Reply