Backup using encryption and gmail

Today I was a bit bored, so I wrote a script to backup up my personal svn repository. I use my svn-server for my personal projects and to keep a history of my personal documents. So I don’t want to loose the data in my repository.

The repository is already on a RAID1, but I don’t think this is enough. I use Google Apps for hosting my mail (it’s GMail) and this gives me more disk-space then I could use on ordinary emails, so I’ve decided to utilize this for my backups. Both because I’ve the space and because I assume Google has a better backup system then anything I could afford.

I’ve written a script that compresses and encrypts to content of a directory, the archive created is then emailed to my backup email account.

This script is set to run once a day, but the email is only sent if something has changed since the last time it’s sent to my email. This is to save space.

I encrypt the archive because email isn’t all that private. The data isn’t sensitive, but I would still like to minimize the risk of it falling into other peoples hands.

The script splits the backup into 20MB pieces because of the limits on the gmail-smtp.

Well enough text, here is my script:

#!/bin/bash

mkdir -p ${HOME}/tmp

DATE=`date +%Y%m%d-%H:%M:%S`
BACKUP_DIR="svn" # Name of directory to backup, NOT PATH.
PARENT_DIR="/home" # Path to the parent dir for the backup directory

BASE_FILENAME="${HOME}/tmp/svn_backup" # Base path for the working files for this backup
NEW_TAR_FILENAME="${BASE_FILENAME}.tar" # Working path to the tar archive
OLD_TAR_FILENAME="${BASE_FILENAME}_old.tar" # Path to the old tar archive, this will be used to see if changes have happend since last time.
NEW_LZMA_FILENAME="${BASE_FILENAME}-${DATE}.tar.lzma" # Path to the lzma-compress archive of the backup.
NEW_GPG_FILENAME="${NEW_LZMA_FILENAME}.gpg" # Path to the gpg encrypted file containing the back

EMAIL="target@domain.com" # Target email for our backup

SPLIT_SIZE="20MB" # How big do we want the pieces to be, I choose 20MB since that's the maximum file size gmail allows.

# Change dir to the parent dir.
cd ${PARENT_DIR}
echo "Generating tar-archive"
tar cp ${BACKUP_DIR} > ${NEW_TAR_FILENAME}

if [ -f ${OLD_TAR_FILENAME} ]; then
	# if an old copy of the archive exist, check if the new one is different
	echo "Old tar-archive exists."
	diff -q ${NEW_TAR_FILENAME} ${OLD_TAR_FILENAME} && \
		echo "They are the same, nothing has changed stopping." && \
		rm -f "${NEW_TAR_FILENAME}" && exit 0
fi

echo "Compressing backup with lzma: `date`"
lzma -z -9 - < ${NEW_TAR_FILENAME} > ${NEW_LZMA_FILENAME}

echo "Encrypting backup with gpg: `date`"
gpg -r "m_abs@mabs.dk" -e - < ${NEW_LZMA_FILENAME} > ${NEW_GPG_FILENAME}
rm -v ${NEW_LZMA_FILENAME}

echo "Splitting encrypted file: `date`"
split ${NEW_GPG_FILENAME} -b 20MB "${NEW_GPG_FILENAME}_"
rm -v ${NEW_GPG_FILENAME}

# How many pieces is there? This is need because I want to write it in the email.
c=0;
for filename in ${NEW_GPG_FILENAME}_[a-z]*; do
	let "c=${c}+1";
done

# Sent every piece of this backup to the target email.
i=0;
for filename in ${NEW_GPG_FILENAME}_[a-z]*; do
	let "i=${i}+1";
	echo "Sending email ${i}: `date`"
	echo "SVN backup - ${DATE}" | mutt -s "SVN backup part ${i}/${c} - ${DATE}" ${EMAIL} -a ${filename}
	rm -v ${filename}
done

mv -v ${NEW_TAR_FILENAME} ${OLD_TAR_FILENAME}
exit 0

Save it to a file, place the file1 in /etc/cron.daily and make it executable.

In order for this script to work I needed a few packages installed on my Gentoo installation, mutt and ssmtp.

I’m gmails smtp-server to sent the emails, I’ve created a special email-account for this so that I don’t have to use my everyday email account for this. This is my /etc/ssmtp/ssmtp.conf

# The person who gets all mail for userids < 1000
# Make this empty to disable rewriting.
root=postmaster

# The place where the mail goes. The actual machine name is required
# no MX records are consulted. Commonly mailhosts are named mail.domain.com
# The example will fit if you are in domain.com and your mailhub is so named.
mailhub=mail

root=sender@domain.com
mailhub=smtp.gmail.com:587
rewriteDomain=
hostname=sender@domain.com
UseSTARTTLS=YES
AuthUser=sender@domain.com
AuthPass=password
FromLineOverride=YES

To encrypt in this script I use GPG and a self generated key.
To generate the key run the command "gpg --gen-key" and follow the wizard. And export the public key from the machine you created it on and import it on the server you are going to run the script.
Exporting the key:

host# gpg --list-keys
/home/user/.gnupg/pubring.gpg
-----------------------------
pub   1024D/112A2538 2009-04-21
uid                  Firstname Lastname 
sub   4096g/AEFA304A 2009-04-21
host# gpg --export "Firstname Lastname " > key.pub

Once you have transfered the key to the server import it with and set trust-level to ultimate on this key, or the backup script will stop and wait for you to answer if you really trust the key.

server# gpg --import key.pub
server# gpg --edit-key user@domain.com
command> trust
Please decide how far you trust this user to correctly verify other users' keys
(by looking at passports, checking fingerprints from different sources, etc.)

  1 = I don't know or won't say
  2 = I do NOT trust
  3 = I trust marginally
  4 = I trust fully
  5 = I trust ultimately
  m = back to the main menu

Your decision? 5
Do you really want to set this key to ultimate trust? (y/N) y
Command> quit

Restoring the backup:
Get the files from the email, this can take a while if there are many pieces.

host# basename=svn_backup-20090421-20:20:33.tar.lzma

host#  for filename in $basename_[a-z]*; do
cat $filename >> $basename
done

host# lzma -d < ${basename} | tar cvp

This should do the trick. The reason I use a for loop and don't just "cat $basename*>>$basename" is that with danish locale "aa" is perceive like å which is the last letter in the danish alphabet and this messes up the file order for cat.

  1. remember to name the file to the run-parts specification, see man run-parts []

Reintroducing a little sanity in working with JavaScript

When I first started working with JavaScript (JS),

I came from a background working with type-safe languages like Java and C++.

Working with JavaScript wasn’t as much fun as working with C++ and Java. JavaScript has a very poor image and there are several problems with the language which can lead to very bad coding.

One of the main problems I had with JavaScript was the lack of encapsulation of data, the API especially the DOM-API is rather horrible and the browser compatibility can be a mess. Jquery fixes both the problem with DOM-API and the problem with cross-browser development by wrapping is up in a much more developer-friendly API.

Some of the things I learned from working with the JS-library jQuery, was how to exploit JS’s features to get the encapsulation required to make my code a lot cleaner.

Around the same time I started using jQuery1,

I also started using jslint2 to increase the code quality and reduce stupid mistakes.

Jslint is a tool that looks for problems in JS source code. On their website you can copy & paste source code into a form to check the code or you can use a standalone version3

The benefit from using a tool like jslint is that you find problems with the code, you might not find testing it in a browser or would that a long time to find. Jslint is mean and might hurt your feelings, but your code will be much better if you follow the advise it gives you.

In your source file, you can add options to the jslint parser. These are the options I normally use:

/*jslint undef: true, browser: true */
/*global jQuery, $*/

The first line is options for jslint undef: true will give a warning if a variable or function is used that hasn’t been defined or is used before it’s defined. browser: true, tells jslint that this script is meant to run in a web-browser and therefor allows variables the browser provides.

The second line tells jslint which global variable are available in this case jQuery and $ because I intend to use jQuery.

To encapsulate my code I use this structure:

/*jslint undef: true, browser: true */
/*global jQuery, $*/

( function( $ ) {
	// Private variables

	// Private functions

	// Public functions
} )( jQuery );

What happens here is that I create an anonymous function which is called immediate with a reference to the jQuery object. This function can only get called once.
Inside this function I’ve three sections:

  1. Private variables
  2. Private functions
  3. Public functions

You’ll notice this is much like a class definition is Java or C++, which is my intension to mimic.4
In the first two sections I put variables and functions, that can only be reached from within the anonymous outer function.

I’ll now write a very simple plug-in to jQuery using this structure.

/*jslint undef: true, browser: true */
/*global jQuery, $*/

( function( $ ) {
	// Private variables
	var enabled = false;

	// Private functions
	var switchPageBackgroundColor = function( ) {
		$( document.body ).css( {
			"background-color" : "#ABBEEC",
		} );
		enabled = true;
	};

	var switchOfPageBackgroundColor = function( ) {
		$( document.body ).css( {
			"background-color" : ""
		} );
		enabled = false;
	};

	// Public functions
	$.switchPageBackgroundColor = function( ) {
		if ( !enabled ) {
			switchPageBackgroundColor( );
		}
	};

	$.switchOfPageBackgroundColor = function( ) {
		if ( enabled ) {
			switchOfPageBackgroundColor( );
		}
	};
} )( jQuery );

This code adds two public functions to the jQuery object switchPageBackgroundColor and switchOfPageBackgroundColor. These two functions are little more then wrappers to the private functions, but it’s impossible to call the private functions directly from outside of the anonymous function.

But wait!!! This code has a bug that breaks the code in IE but not in Firefox, can you spot this error?

If you run this code through jslint you should get this error:

Lint at line 11 character 44: Extra comma.
"background-color" : "#ABBEEC",

You might be able to find such an error on your own, but imagine you maintained a large website with 20k+ lines of JS code and you introduced such a bug. How would you find it?

When I worked for ScioSphere I created strict guidelines for the JS source code which I were responsible for. The first and most important rule in those guidelines were “Nothing gets committed or deployed without passing the jslint test”.

Code without the error:

/*jslint undef: true, browser: true */
/*global jQuery, $*/

( function( $ ) {
	// Private variables
	var enabled = false;

	// Private functions
	var switchPageBackgroundColor = function( ) {
		$( document.body ).css( {
			"background-color" : "#ABBEEC"
		} );
		enabled = true;
	};

	var switchOfPageBackgroundColor = function( ) {
		$( document.body ).css( {
			"background-color" : ""
		} );
		enabled = false;
	};

	// Public functions
	$.switchPageBackgroundColor = function( ) {
		if ( !enabled ) {
			switchPageBackgroundColor( );
		}
	};

	$.switchOfPageBackgroundColor = function( ) {
		if ( enabled ) {
			switchOfPageBackgroundColor( );
		}
	};
} )( jQuery );
  1. http://jquery.com []
  2. http://www.jslint.com/ []
  3. http://www.jslint.com/rhino/index.html or http://www.jslint.com/wsh/index.html []
  4. If you want to make objects from this function: Name the function and remove the implicit call to it ( jQuery ). Then you can call it with new and create an instance of it. []

Unexpected benefits from even small contributions to Open Source

Once long ago I had a problem with Kopete1. The problem was that the notification dialog got in the way and it didn’t hide it self, if the user didn’t react to it. This was especially a problem while I was playing a game, since the dialog got on top and covered the game.

After a while I decided to fix the problem myself and I took the time to look into Kopete’s notification code and wrote a simple patch, which added a timeout which hid the dialog and added a settings interface to set the timeout delay.

I submitted the patch to the developers of Kopete, after a while one of them got back to me. His name was Jan Ritzerfeld, he improved my patch which were added to Kopete in KDE3.5. Something I’m extremely proud of :).

This is my original bug report about this: http://bugs.kde.org/show_bug.cgi?id=108967

When I made this patch, I was looking my first job after finishing my education as a “datamatiker” and my job hunt wasn’t going all that well for me.

In 2006, I got called in for a job interview with a company called Sapio Systems. For this interview I got a theoretical problem to solve and my solution was to be presented at the job interview.

At the interview we talked about my education, the few projects I had been involved with since my graduation, the time I used helping other Linux users online, my patch for Kopete and finally my solution to the problem I’d been given.

Well, I had misunderstood the problem and my solution was wrong. During the interview I was so nervous that I sold my skills short and having solved the problem wrong, I was sure that I had completely blown my chances at getting this job.

To my great surprise they called me back and I got a second chance which meant a week to solve a practical assignment, writing a small application which looked for unknown words in a text file and a web-interface for it.

This task I handled much better and they gave me my first real job as a developer and I had the job for almost three years. Which gave me a lot of experience and I had a lot of fun developing ScioSphere.

For a long time I didn’t understand why I got this important second chance. I recently found out they were very impressed that I had code in Kopete and thought that I’d sold my skills short at the interview.

I hadn’t actually thought that this patch was all that important and had mostly mentioned it because I had so little to show, but without this contribution I don’t think I’d got that second chance to prove myself.

The lesson I think can be learned from this is that unexpected great rewards can come from small contributions.

  1. KDE’s instant messaging client http://kopete.kde.org/ []