Python and AWS Cookbook Available

I recently completed a short book for O'Reilly called "Python and AWS Cookbook".  It's a collection of recipes for solving common problems in Amazon Web Services.  The solutions are all in Python and, of course, use boto heavily.  The focus of this book is EC2 and S3 although there are a couple of quick detours into IAM and SNS.  Many of the examples also work with Eucalyptus so I have included some information about using boto with Eucalyptus as well as with Google Cloud Storage for some of the S3-related recipes.

You can get a hardcopy of the book but if you buy the e-book, you get free updates and I am expecting to do quite a few updates.  Many of the recipes came from problems people have posted on the boto users group or on the boto IRC channel but I'm sure there are lots of other areas where additional example code would be useful.  If you have specific requests, let me know.  Depending on the response, I might also do additional cookbooks that focus on other services.

The bird on the cover is a Sand Grouse.  I lobbied heavily for a Honey Badger but to no avail.

Does Python Scale?

I wonder how many times I've been asked that question over the years.  Often, it's not even in the form of a question (Sorry, Mr. Trebek) but rather stated emphatically; "Python doesn't scale".  This can be the start of long, heated discussions involving Global Interpreter Locks, interpreters vs. compilers, dynamic vs. static typing, etc.  These discussions rarely end satisfactorily for any of the parties involved.  And rarely are any opinions changed as a result.

So, does Python scale?

Well, YouTube is written mostly in Python.  DropBox is written almost entirely in Python.  Reddit.  Quora.  Disqus.  FriendFeed.  These are huge sites, handling gazillions of hits a day.  They are written in Python.  Therefore, Python scales.

Yeah, but what about that web app I wrote that one time.  Hosted on a cheapo, oversubscribed VPS, running straight CGI talking to a remote MySQL database running in a virtual machine on my Macbook Air.  That thing fell over like a drunken sailor when I invited a few of my friends to go check it out.  So, yeah.  Forget what I said before.  Obviously Python doesn't scale.

The truth is, it's the wrong question.  The stuff that allows Dropbox to store a million files every 15 minutes has little to do with Python just as the things that caused my feeble web app to fail had little to do with Python.  It has to do with the overall architecture of the application.  How databases are sharded, how loosely or tightly components have been coupled, how you monitor, and how you react to the data your monitoring is providing you.  And lots of other stuff.  But you have to deal with those issues no matter what language you write the system in.

No reasonable choice of computer language is going to guarantee your success or your failure.  So pick the one you are most productive in and focus on properly architecting your app.  That scales.

Accessing the Eucalyptus Community Cloud with boto

The Eucalyptus Community Cloud (ECC) is a great resource that allows you to try out a real cloud computing system without installing any software or incurring any costs.  It's a sandbox environment that is maintained by Eucalyptus Systems to allow people to testdrive Eucalyptus software and experiment with cloud computing.

To access the ECC, you need to sign up following the instructions here.  Once you are signed up, you will be able to download a zip file containing the necessary credentials for accessing the ECC.  If you unzip that file somewhere on your local filesystem you will find, among other things, a file called eucarc.  The contents of that file will look something like this:



To get things to work seamlessly in boto, you need to copy a few pieces of information from the eucarc file to your boto config file, which is normally found in ~/.boto.  Here's the info you need to add.  The actual values, of course, should be the ones from your own eucarc file.



Notice that the values needed for eucalyptus_host and walrus_host are just the hostname or ip address of the server as specified in the EC2_HOST and S3_HOST variables.  You don't have to include the port number or the http prefix.  Having edited your boto config file, you can now easily access the ECC services in boto.


This example assumes you are using the latest version of boto from github or the release candidate for version 2.1 of boto.