Code, Camera, Action

Update: Aperture 3 now supports the Lumix LX3, G1 and GH1 raw formats. However, there are some problems with DNG support for these cameras. The trade-offs in this post are still valid, but before you decide on a DNG workflow, take a look at the work-arounds for getting certain DNGs working in Aperture 3.

Mac OS X and Aperture don't yet support the RW2 raw format produced by recent Panasonic cameras like the DMC-G1 or LX3 or the Olympus E-P1. That leaves Mac users with a couple of options:

• Shoot JPEG. The quality is pretty good, and the G1's automatic settings get it right much of the time.
• Shoot RAW and convert with SilkyPix or RAW Developer, software packages that support RAW from these cameras.  SilkyPix, while it comes with Panasonic cameras, has an unusual interface (export settings get saved in the "cloakroom"). RAW Developer seems great, but is by no means free. Neither program is designed to manage a photo library.
• Buy the latest version of Adobe Lightroom, which handles these latest raw formats just fine
• Shoot RAW and convert to Adobe's DNG format -- which Aperture and iPhoto can read mostly

For now, I am committed to Aperture.  And I prefer to shoot RAW. So if, like me, you prefer to use Aperture to manage your photos, this workflow can help.

As I've written before (in Adobe DNG 1.3 spec now does RAW for the Panasonic Lumix G1), the state of OS X is that for now, DNGs work fine from adapted manual lenses. While the DNG converters have been updated to support shots taken with the Panasonic and Olympus lenses, OS X doesn't understand the new stuff those lenses add.  In practice, this means you can only work with raw shots made with legacy lenses.  One day Aperture and OS X will be updated to understand the new stuff -- either in RW2 format or in DNG. Until that sunny day, here's what I do:

1. With the kit lens (or any other micro 4/3 lens -- if I had others), I shoot JPEG
2. With manual legacy lenses (I have a couple of Voigtlanders), I shoot RAW and convert to DNG
3. Import these DNG and JPEG images into Aperture

Automating Conversion and Import

This may sound complicated, but at download time, it's really not, because I've automated the process.  Here's how you can, too. For that last import step, I use Image Capture to pull the photos off of the camera and call an Automator workflow that does the conversion.  

I've put the code for the the workflow below. It requires two helper applications:

• Adobe DNG converter turns your RW2 files into DNGs. The originals are still there, if you want them.
• exiftool, which compensates for a date bug. Aperture and DNG converter can't agree on the date and time your photos were taken. Exiftool removes date info from the DNGs so that Aperture can do the right thing.

These programs get called by the import workflow, which looks like this in Automator:

Here's the code for the shell script action that does the conversion:

Simply paste that code into a perl shell script workflow box, just like in the screenshot above. For the import action, feel free to choose import into Aperture (like I do) or iPhoto. Either one will work.

Last, save the workflow as an Image Capture plugin, and you'll be all set. Image Capture will let you pick where to download the images, and your new workflow will appear in the list of automatic tasks. (Restart Image Capture if you don't see it right away.)

Or Download the Whole Workflow

Update: Don't want to mess around with creating a workflow to do this? Why not just download mine. Save it in your home directory to ~/Library/Workflows/Applications/Image Capture, where Image Capture will find it. Restart Image Capture if necessary.

So there it is. You'll be shooting full quality when you can, or JPEG when you must -- and not complicating your workflow.

Click here to download:

RW2_to_DNG_to_Aperture.zip (235 KB)

Loading mentions Retweet

The disadvantage is that somebody needs a FriendFeed login to leave a comment.  Because FF accepts FaceBook, Google or Twitter logins, though, that may not be such a big hurdle

Querying FriendFeed

FriendFeed's API let's you pull a JSON version of posts and comments with a single query.

http://friendfeed.com/api/feed/url?url=[your url here]

This gives us every FriendFeed item that refers to a particular URL.  (The current post, in this case.)  Those results come back in JSON format, where we can act on them in script.  For convenience, FF let's us name a JavaScript function that will be called when we get the data.  I've called mine ffDisplay, as you can see.

callback=ffDisplay

To get this to work, we put the url and callback parameter together in a script tag, like this:

<script type="text/javascript" src="http://friendfeed.com/api/feed/url?url={{ page.url }}&amp;callback=ffDisplay" defer="true"></script>

Then it's up to us to format these the way we want.  To do this, we'll need our ffDisplay function.  This simply walks the structure of the JSON results, as documented by the FriendFeed API.  We also need a place on the page to put the comments.  This script puts comments inside a div with an id of "ff_comments".

As the comment on lines 8 and 9 notes, there is one tricky thing here.  The url value in the JSON is either the URL we want (the one for the FriendFeed discussion) or the blog post's URL (which we don't want).  In our case, anything that's not a FriendFeed URL points to the original post.  To link to the FriendFeed discusion, we combine the userid and post id into an undocumented FriendFeed URL that works just fine.  I hope this behavior doesn't break in the future.

So there we go.  FriendFeed comments on page, courtesy of the FriendFeed API and JavaScript.

One last plug for the FriendFeed API.  JavaScript just touches the surface of that API.  

If you're using WordPress or MovableType to run your blog, there are other options.  Look for the FriendFeed Comments plugin for WordPress or an import plugin for MovableType.  But if you're limited to JavaScript by something like Jekyll or Blogger, then this JavaScript method just might work for you.

Let me know how it goes in the (FriendFeed) comments.

Loading mentions Retweet

Git is a great (and fashionable) source control management program. GitHub takes the pain out of hosting Git for a group — for both open-source and private projects. How to best use them together? The Way to Happiness has these 4 steps:

1. Fork the project on GitHub, and clone your version. This gives you your own copy to work with.
2. Make a branch (or frequent branches) for what you’re working on. Some folks systematize this with issue names and bug numbers — like “RedoPostFormatting-234” — which is probably a good idea for projects with lots of collaborators.
3. If you reserve certain branch names, like “master” and “staging” for production and pre-production code, you can merge a project’s code back into your tree more easily.
4. Or use the github-gem command line tool to grab a copy of somebody else’s code when needed. Here’s the salient snippet from the readme:

Let’s say you just forked `github-gem` on GitHub from defunkt.

$ github clone YOU/github-gem
$ cd github-gem
$ github pull defunkt

This will setup a remote and branch for defunkt’s repository at master. In this case, a ‘defunkt/master’ branch.

If defunkt makes some changes you want, simply `github pull defunkt`. This will leave you in the ‘defunkt/master’ branch after pulling changes from defunkt’s remote. After confirming that defunkt’s changes were what you wanted, run `git checkout master` and then `git merge defunkt/master` to merge defunkt’s changes into your own master branch.

Loading mentions Retweet

The folks at GitHub have taken the pain out of centralized Git hosting. Even better, they’ve built in tools that make distributed version control easy to use for group projects. But how to set up your repositories and branches to avoid merge pain? Here are two good options.

Making Merge Manageable

Setting up your Git repositories for open source projects at github has great tips for getting started — and for structuring your branches for group work. There are a couple of take-aways here:

1. Encourage contributors to track established branch names, like “master” and “edge”. At the office we use “master”, “staging” and “dev”.
2. This makes it much easier for developers to pull in changes from the official repository in a simple, fast-forward way
3. Develop on a separate branch (or branches). You might want to name that branch with your github id, to help folks keep it straight.

Michael also provides a solid how-to, so if “pull request” sounds weird to you, start here.

Another Way: Branch away and let github-gem straighten it out

The folks at GitHub have a Ruby command-line tool to manage this without the fancy branch naming conventions. You fork, and the tool helps keep everybody’s master branches separate. It also speeds merging from these masters.

See defunkt’s Github-gem for the code and a README that explains the tool’s awesomeness.

Github-gem assumes that master is where you’ll want to pull from. This matches what most folks do — unless they’ve benefited from the foresight of the Inoshi folks.

Whichever strategy you choose, Github’s network diagram is your visual to-do list. It shows who committed stuff that’s different than what’s on your branch. Say hello to the Network graph visualizer has the details.

Loading mentions Retweet

Both Apache 2.2 and PHP 5 come with every installation of Mac OS X 10.5 Leopard. MySQL does not. There are a couple of tricks for installing and/or turning them on —and thereby attaining Development Happiness.

Here’s how we configure our development machines at the studio, where our PHP work is primarily Drupaland we use several virtual hosts.

Install MySQL

Not hard, but a little long for this post. Take a look at our instructions for installing MySQL on Leopard.

Configure Apache for PHP and Virtual Hosts

Though Leopard ships with PHP, it’s not turned on by default. Let’s change that, and also turn on the virtual hosts that we’ll use for our various Drupal installs.

• Edit Apache’s configuration file to make the following two changes:
  

  
  
  sudo vi /etc/apache2/httpd.conf
  
  

• Uncomment line 114 so it looks like this:
  

  
  
  LoadModule php5_module        libexec/apache2/libphp5.so
  
  

• Uncomment line 461 so it looks like this:
  

  
  
  Include /private/etc/apache2/extra/httpd-vhosts.conf
  
  

Add your Virtual Hosts

Edit /etc/apache2/extra/httpd-vhosts.conf to add your configuration. See our sample configuration for ideas.

NameVirtualHost *:80
<VirtualHost *:80>
    ServerAdmin eric@localhost
    DocumentRoot "/Users/eric/Sites/agc"
    ServerName agc.local
    RewriteLog /private/var/log/apache2/rewrite_log
    RewriteLogLevel 0
</VirtualHost>

A couple items of note:

• For development, we put our virtual hosts under users’ ~/Sites folder. That way there are no permissions issues to prevent Apache from serving the code — or from deploying fresh.
• You’ll also have to put agc.local into DNS.

Add the new virtual hostnames to /etc/hosts

Edit /etc/hosts to add a couple of nicknames for localhost:

127.0.0.1 localhost agc.local giving.local givingedit.local miyo.local

Start Apache

• Start System Preferences from the Apple menu.
• Go to the Sharing tab
• Click the checkbox by “Web Sharing”

Done. Let’s test everything by starting Safari and visiting http://agc.local

Loading mentions Retweet

MySQL now provides compiled binaries for Mac OSX 10.5 Leopard. But the startup applet doesn’t work properly yet on Leopard.

So here’s how to install MySQL on Leopard and start it using launchd — the standard Leopard way. These instructions should work for both Tiger (10.4) and Leopard (10.5).

• Download MySQL for your OS and architecture (PowerPC or Intel; 10.4 Tiger or 10.5 Leopard)
• Install MySQL from the package in the dmg — something like mysql-5.0.51a-osx10.5-x86_64.pkg
• DO NOT install the startup item package — it doesn’t work on Leopard (as of version 5.0.51a).
• DO NOT install the preference pane — it doesn’t work on Leopard (as of version 5.0.51a, though a fix is in the works).

Starting MySQL

Since neither the preference pane nor startup item work on Leopard, let’s use launchd to start MySQL instead.

• Copy com.mysql.mysqld.plist to /Library/LaunchDaemons
• To start MySQL, use the launchctl command:
  

  
  
  sudo launchctl load -w /Library/LaunchDaemons/com.mysql.mysqld.plist
  
  

• Stop mysqld with an unload
  

  
  
  sudo launchctl unload -w /Library/LaunchDaemons/com.mysql.mysqld.plist
  
  

The -w in either of these commands leaves mysqld running (and restarts it as needed) until you turn it off with the other launchctl command.

For convenience, you may want to create an alias for these commands in .bashrc

alias start_mysql='sudo launchctl load -w /Library/LaunchDaemons/com.mysql.mysqld.plist'
alias stop_mysql='sudo launchctl unload -w /Library/LaunchDaemons/com.mysql.mysqld.plist'

One other convenience — put mysql and mysqldump into your path:

cd /usr/local
sudo ln -s mysql-5.0.51a-osx10.5-x86_64 mysql
cd bin
sudo ln -s ../mysql/bin/mysql .
sudo ln -s ../mysql/bin/mysqldump .

Done. So we’re ready to fire it up and connect:

sudo launchctl load -w /Library/LaunchDaemons/com.mysql.mysqld.plist
mysql -u root -p

Loading mentions Retweet

So you want to use Git for distributed version control. For Mac OS X 10.5 there are a couple of options for installing Git.

• Fink packages an almost up-to-date version, named simply git
• MacPorts packages an up-to-date version asgit-core

Either of these methods, though, installs lots of dependencies. Which requires lots of compile time. And lots of disk space — especially for the MacPorts version.

So I prefer to compile Git myself — and use Fink for dependencies. Here’s how I do it.

Install Fink by following Fink’s installation instructions.

fink install gettext-tools expat1-shlibs

From here the instructions in Git’s INSTALL file work just fine:

curl -O http://kernel.org/pub/software/scm/git/git-1.5.4.5.tar.gz
tar zxf git-1.5.4.5.tar.gz
cd git-1.5.4.5
make prefix=/usr/local all 
sudo make prefix=/usr/local install

Now for the manpages.

Rather than build the docs ourselves (which requires asciidoc and a host of dependencies), I prefer to download them precompiled from Git and manually copy them into /usr/local/share/man.

curl -O http://kernel.org/pub/software/scm/git/git-manpages-1.5.4.5.tar.gz
sudo tar zxf git-manpages-1.5.4.5.tar.gz  -C /usr/local/share/man

(Replace the version numbers here with what’s right for the version of Git you’re installing.)

Now we’re ready to try this out, we’ll ask what version we just installed, then show the manpage:

git --version
man git

Done. Happy versioning!

Update

Sean Santry has an even simpler way to avoid the gettext-tools dependency — by telling make that we don’t have msgfmt at all.

NO_MSGFMT=yes make prefix=/usr/local all
sudo make prefix=/usr/local install

Using this tip you can build Git without Fink altogether!

Comments on the Original Post

You can also just install a complete OS X package from http://code.google.com/p/git-osx-installer/. It’s only 3.6MB and spares you the manual install

Pieter de Bie 4 Apr 2008, 10:31 AM

Loading mentions Retweet

So you want to use Git the latest hotness in distributed version control. It builds easily using the instructions in the INSTALL file.

But on Centos 5 (or Red Hat Enterprise Linux 5) there are some dependencies to install first:

sudo apt-get install gettext-devel expat-devel curl-devel zlib-devel openssl-devel

From here the instructions in Git’s INSTALL file work just fine:

wget http://kernel.org/pub/software/scm/git/git-1.5.4.5.tar.gz
tar zxvf git-1.5.4.5.tar.gz
cd git-1.5.4.5
make prefix=/usr/local all 
sudo make prefix=/usr/local install

Now for the manpages.

Rather than build the docs ourselves (which requires asciidoc and a host of dependencies), I prefer to download them precompiled from Git and manually copy them into /usr/local/share/man.

wget http://kernel.org/pub/software/scm/git/git-manpages-1.5.4.5.tar.gz
cd /usr/local/share/man
sudo tar zxvf ~/git-manpages-1.5.4.5.tar.gz

(Replace the version numbers here with what’s right for the version of Git you’re installing.)

Now we’re ready to try this out, we’ll ask what version we just installed, then show the manpage:

git --version
man git

Done. Happy versioning!

Postscript

This would of course be easier of using rpm-build. If anybody has experience with a good RPM spec file for git — or using a Fedora Core src rpm — please let us know in the comments.

The good folks at Git also provide pre-compiled RPMs.

Loading mentions Retweet

JungleDisk is another great entrant in the online backup business. Unlike Mozy, our favorite for Macs and PCs, JungleDisk also runs on Linux. And, since version 1.30a, it ships a command-line version.

Which means you can give your Linux servers reliable online backup — stored on Amazon’s S3.

These instructions cover version 1.50 — which has a different config file and a number of settings than previous versions. If you’re still using an older version, see the instructions for version 1.30 or 1.40 that we published a while back.

Meanwhile, let’s get down to business with version 1.50. Our server install has three parts:

• JungleDisk command line program, which makes S3 look like a WebDav server
• Fuse, which makes WebDav look like a regular Linux filesystem
• rsync, the standard Linux power copy program

You’ve got rsync, and the other two aren’t much harder to install.

1. Install Fuse

1. Install Fuse. On Ubuntu Dapper this is easy once you enable the Universe repository. Later versions of Ubuntu have fuse-utils in the regular repositories.
   

   	sudo apt-get install fuse-utils
   	sudo modprobe fuse
   

2. To keep JungleDisk happy, we’ll load the fuse kernel module automatically on boot.
   

   	sudo echo “fuse” >> /etc/modules
   

2. Install JungleDisk

1. Download JungleDisk, and untar it. Since version 1.30a, the Linux package comes with both GUI and command-line versions.
   • jungledisk is the command-line version we’ll use here
   • junglediskmonitor is the GUI version. This version makes it easy to sign up for S3, and it writes the config file for you automatically — but you’ve got to have X11 installed to run it :(.
2. Sign up for an Amazon S3 account. junglediskmonitor can help you do this — if you’ve got X-windows installed. The junglediskmonitor also creates a settings file.
3. Or if you install JungleDisk on a Mac or Windows PC you can copy the config from that install. (On a Mac, the config can be found here: ~/Library/Preferences/jungledisk-settings.ini)
4. You could also use a different machine to run the USB-key version of JungleDisk, which lets you run the graphical setup on a Mac or PC and writes the settings back to the USB drive.
5. Or you can make your own ~/.jungledisk/jungledisk-settings.ini. You’ll need to put your Amazon S3 access and secret keys and your username in place of the*<removed>* text below:

1 LoginUsername=
2 LoginPassword=
3 AccessKeyID=<removed>
4 SecretKey=<removed>
5 Bucket=default
6 CacheDirectory=/tmp/.cache/
7 ListenPort=2667
8 CacheCheckInterval=120
9 AsyncOperations=1
10 Encrypt=0
11 ProxyServer=
12 EncryptionKey=
13 DecryptionKeys=
14 MaxCacheSize=50
15 MapDrive=J
16 UseSSL=1
17 RetryCount=3
18 FastCopy=1
19 WebAccess=0
20 LogDuration=0
21 ArchiveFlag=5
22 ArchiveDuration=60
23 MaxArchiveSize=100
24 PasswordPrompt=0
25 UploadResume=0
26 DeltaUpdates=0
27 PlusFallback=1
28 LargeWindows=0
29 License=

You can download a sample config file without the line numbers. A couple of lines will need editing to fit your circumstances.

Lines 3 and 4 are your Amazon S3 credentials. Copy and paste them from Amazon’s website. (Click the button labled “Your Web Services Account”, then choose “AWS Access Identifiers” to see these.)

Line 6: The directory where JungleDisk will write to while its processing. This needs to exist and be writable by the user who will run jungledisk.

Lines 1, 2 and 29 can be left blank (as they are here) for your 30-day trial of JungleDisk. Once you buy the software, however, you’ll want to fill them in. Lines 1 and 2 are the email address and password you used to make your JungleDisk purchase.

Line 29 is a license code generated by junglediskmonitor. I’m not aware of a way to produce this in the command-line client. The Mac OS X version of JungleDisk keeps the license in the keychain, where Keychain Access can show it. If anybody knows where to find this in Windows, please let us know in the comments.

3. Actually Try it Out

Time to try it all out.

1. Get ready to mount the jungledisk drive (first time)
   

   
   
   	sudo mkdir /mnt/s3
   
   

2. Start jungledisk (/mnt/s3 is the mount point you just created)
   

   
   
   	./jungledisk/jungledisk /mnt/s3
   
   

3. Actually copy something over to S3
   

   
   
   	echo “Here’s a file to try” > atestfile.txt
   
   	sudo cp atestfile.txt /mnt/s3
   
   

4. Rsync Backups

If you don’t love rsync yet, you’re gonna. Can you say “Differential Copy algorithm”? Rsync only copies the changed stuff. And that’s what makes over-the-net backups workable.

Here’s how to back up the whole enchilada (or at least all of your server’s /home directories).

rsync -a --inplace --bwlimit=50 /home /mnt/s3

Notice that the rsync command adds a bandwidth limit (—bwlimit=50) to the inplace flags that JungleDisk recommends. Without this, you may find that jungledisk opens more files than the kernel will allow (at least in Ubuntu).

5. Automated Backups

And here’s the shell script we use to actually do the backup. Line 5 here shows that we’ve got one more choice here — when to start jungledisk. It could be done in line 5 here, but because it’s a background process we want to leave running, we could start it on bootup by adding the jungledisk command to the end of /etc/rc.local.

/usr/local/bin/backup-jd

1  #!/bin/sh
2  ### Backs up office data to Jungledisk using rsync
3  LOGFILE=/var/log/backup-jd.log
4  ## Start in rc.local or here
5  /usr/local/bin/jungledisk /mnt/s3
6  echo "`date +"%F %R"`: Start backup-jd" >> $LOGFILE
7  rsync -a --inplace --bwlimit=50 /home /mnt/s3
8  echo "`date +"%F %R"`: Finish backup-jd" >> $LOGFILE

Updates & Revisions

8/14/07: Originally published.

8/19/07: Add line numbers and a better explanation of the JungleDisk config file, plus a version for download.

4/18/08: Updated for JungleDisk version 1.50. The original post is here for anyone still using JungleDisk 1.30 or 1.40.

1 Comment on the Original Post

Loading mentions Retweet

JungleDisk is another great entrant in the online backup business. Unlike Mozy, our favorite for Macs and PCs, JungleDisk also runs on Linux. And, since version 1.30a, it ships a command-line version.

Which means you can give your Linux servers reliable online backup — stored on Amazon’s S3.

Here’s more about how JungleDisk works.

Meanwhile, let’s get down to business. Our server install has three parts:

• JungleDisk command line program, which makes S3 look like a WebDav server
• DavFS, which makes WebDav look like a regular Linux filesystem
• rsync, the standard Linux power copy program

You’ve got rsync, and the other two aren’t much harder to install.

1. Install DavFS

1. Install davfs2. On Ubuntu Dapper or later this is easy once you enable the Universe repository.
   

   
   
   	sudo apt-get install davfs2
   
   

2. To keep JungleDisk happy, we’ll add this line to the bottom of /etc/davfs2/davfs2.conf:
   

   
   
   	kernel_fs fuse
   
   

   
   That tells davfs to use the newer Fuse backend instead of Coda, which avoids a JungleDisk bug.
3. For other linux distributions, see Chris’s instructions for installing davfs2 from source

2. Install JungleDisk

1. Download JungleDisk, and untar it. Since version 1.30a, the Linux package comes with both GUI and command-line versions.
   • jungledisk is the command-line version we’ll use here
   • junglediskmonitor is the GUI version. This version makes it easy to sign up for S3, and it writes the config file for you automatically — but you’ve got to have X11 installed to run it :(.
2. Sign up for an Amazon S3 account. junglediskmonitor can help you do this — if you’ve got X-windows installed. The junglediskmonitor also creates a settings file.
3. Or if you install JungleDisk on a Mac or Windows PC you can copy the config from that install. (On a Mac, the config can be found here: ~/Library/Preferences/jungledisk-settings.ini)
4. Or you can make your own ~/.jungledisk/jungledisk-settings.ini. You’ll need to put your Amazon S3 access and secret keys and your username in place of the*<removed>* text below:

	
 1 LoginUsername=
 2 LoginPassword=
 3 AccessKeyID=*<removed>*
 4 SecretKey=*<removed>*
 5 Bucket=default
 6 CacheDirectory=/home/*<removed>*/.jungledisk/cache/
 7 ListenPort=2667
 8 CacheCheckInterval=120
 9 AsyncOperations=1
10 Encrypt=1
11 ProxyServer=
12 EncryptionKey=PROTECTED:
13 DecryptionKeys=PROTECTED:
14 MaxCacheSize=1000
15 MapDrive=
16 UseSSL=0
17 RetryCount=3
18 FastCopy=1
19 WebAccess=0
20 LogDuration=30
21 License=

You can download a sample config file without the line numbers. A couple of lines will need editing to fit your circumstances.

Lines 3 and 4 are your Amazon S3 credentials. Copy and paste them from Amazon’s website. (Click the button labled “Your Web Services Account”, then choose “AWS Access Identifiers” to see these.)

Line 6: The directory where JungleDisk will write to while its processing. This needs to exist and be writable by the user who will run jungledisk. Most folks should just put their Linux username in place of *<removed>*.

Lines 1, 2 and 21 can be left blank (as they are here) for your 30-day trial of JungleDisk. Once you buy the software, however, you’ll want to fill them in. Lines 1 and 2 are the email address and password you used to make your JungleDisk purchase.

Line 21 is a license code generated by junglediskmonitor. I’m not aware of a way to produce this in the command-line client. The Mac OS X version of JungleDisk keeps the license in the keychain, where Keychain Access can show it. If anybody knows where to find this in Windows, please let us know in the comments.

3. Actually Try it Out

Time to try it all out.

1. Start jungledisk
   

   
   
   	./jungledisk/jungledisk
   
   

2. Mount the jungledisk drive
   

   
   
   	sudo mkdir /mnt/s3
   
   	sudo mount -t davfs -o nolocks,noaskauth http://localhost:2667 /mnt/s3
   
   

3. Actually copy something over to S3
   

   
   
   	echo “Here’s a file to try” > atestfile.txt
   
   	sudo cp atestfile.txt /mnt/s3
   
   

4. Rsync Backups

If you don’t love rsync yet, you’re gonna. Can you say “Differential Copy algorithm”? Rsync only copies the changed stuff. And that’s what makes over-the-net backups workable.

Here’s how to back up the whole enchilada (or at least all of your server’s /home directories).

rsync -r --inplace --size-only --bwlimit=50 /home /mnt/s3

Notice that the rsync command adds a bandwidth limit (—bwlimit=50) to the inplace and size-only flags that webDAV and JungleDisk recommend. Without this, you’ll find that jungledisk opens more files than the kernel will allow (at least in Ubuntu).

5. Automated Backups

To schedule this stuff automatically, we’ll need to add a couple of lines to some of our config files. First we add a line to the end of fstab that lets our backup user mount our jungledrive (use sudo to carefully add this line).

/etc/fstab

http://localhost:2667  /mnt/s3  davfs   user,nolocks,noaskauth   0   0

And here’s the shell script we use to actually do the backup. Line 5 here shows that we’ve got one more choice here — when to start jungledisk. It could be done in line 5 here, but because it’s a background process we want to leave running, we could start it on bootup by adding the jungledisk command to the end of /etc/rc.local.

/usr/local/bin/backup-jd

1  #!/bin/sh
2  ### Backs up office data to Jungledisk using rsync
3  LOGFILE=/var/log/backup-jd.log
4  ## Start in rc.local or here
5  #/usr/local/bin/jungledisk
6  mount /mnt/s3
7  echo "`date +"%F %R"`: Start backup-jd" >> $LOGFILE
8  rsync -r --inplace --size-only --bwlimit=50 /home /mnt/s3
9  echo "`date +"%F %R"`: Finish backup-jd" >> $LOGFILE
10 umount /mnt/s3

Updates & Revisions

8/19/07: Add line numbers and a better explanation of the JungleDisk config file, plus a version for download.

Loading mentions Retweet