Creating an EBS-backed AMI from an S3-backed AMI

The recent introduction of Boot From EBS for EC2 opens up a lot of new possibilities.  But there are some bootstrapping issues to deal with.  There aren't many EBS-backed AMI's available yet and, given the rather complex process involved in porting them, it may take a while for them to show up.  This article will walk through the process of converting a popular S3-based AMI to an EBS-backed AMI.  I don't guarantee that this is the best process and I certainly wouldn't recommend that anyone use the resulting EBS-backed AMI for anything other than testing and further development, but it puts a stake in the ground regarding a potential process.  I'm sure I will hear about the shortcomings and possible improvements!

As a starting point, I'm going to use one of Eric Hammond's excellent Ubuntu AMI's.  In particular, I'm going to use:

ami-eef61587 alestic-64/ubuntu-9.04-jaunty-base-64-20091011.manifest.xml

This same basic process should work for other Linux-based AMI's.  The first thing I need to do is fire up an new instance of this AMI.  In addition, I'm going to create a new 10GB EBS volume that will serve as the prototype for my EBS-based AMI.  Here's how I do that using boto:

>>> import boto
>>> c = boto.connect_ec2()
>>> c.get_all_images(['ami-eef61587'])
[Image:ami-eef61587]
>>> img = _[0]
>>> img.run(key_name='cloudright', security_groups=['test1'], instance_type='m1.large')
Reservation:r-0369c96b
>>> inst = _.instances[0]
>>> inst.update()
u'pending'
>>> inst.update()
u'running'
>>> inst.placement
u'us-east-1b'
>>> v = c.create_volume(10, inst.placement)
>>> v.attach(inst.id, '/dev/sdh')
u'attaching'
>>> inst.public_dns_name
u'ec2-67-202-30-28.compute-1.amazonaws.com'

So, at this point I have a new EC2 instance up and running using the S3-based AMI and a new 10GB EBS volume attached to that instance.  Now, I need to login to that new instance and do a bit of work.

$ ssh -i ~/.ssh/cloudright root@ec2-67-202-30-28.compute-1.amazonaws.com
...
root@domU-12-31-39-02-31-51:~# apt-get update
...
root@domU-12-31-39-02-31-51:~# apt-get -y upgrade
...
root@domU-12-31-39-02-31-51:~# apt-get -y install cpipe
...
root@domU-12-31-39-02-31-51:~# mkfs -t ext3 /dev/sdh
mke2fs 1.41.4 (27-Jan-2009)
/dev/sdh is entire device, not just one partition!
Proceed anyway? (y,n) y
...
root@domU-12-31-39-02-31-51:~# mkdir /ebs
root@domU-12-31-39-02-31-51:~# mount -t ext3 /dev/sdh /ebs
root@domU-12-31-39-02-31-51:~# tar cpS / | cpipe -vt -b 1024 | gzip -c | tar zxpS -C /ebs
....
root@domU-12-31-39-02-31-51:~# umount /ebs

So, basically I have ssh'ed into the new instance, run apt-get update and apt-get upgrade to install all of the latest patches, formatted the EBS volume as an EXT3 filesystem, mounted that filesystem as /ebs and then copied the entire contents of the current root volume over to the EBS volume.  Then I unmount the EBS volume.  Now, let's go back to my Python session running on my local machine.

>>> v.detach()
u'detaching'
>>> v.create_snapshot('Initial snapshot for EBS-backed 64-bit Ubuntu 9.04 AMI.')
Snapshot:snap-023ca66b
>>> from boto.ec2.blockdevicemapping import EBSBlockDeviceType, BlockDeviceMapping
>>> ebs = EBSBlockDeviceType()
>>> ebs.snapshot_id = 'snap-023ca66b'
>>> block_map = BlockDeviceMapping()
>>> block_map['/dev/sda1'] = ebs
>>> c.register_image('MG-Ubuntu-9.04-EBS-20091204', 'Testing the creation of EBS-backed Ubuntu AMI.',
architecture='x86_64', kernel_id=img.kernel_id,
ramdisk_id=img.ramdisk_id,
root_device_name='/dev/sda1', block_device_map=block_map)
u'ami-f002e099'

So, here we are detaching the EBS volume and then creating a snapshot of that volume.  Then, we need to import some data structures that will allow us to register a new EBS-based image.  The first data structure is the EBSBlockDeviceType.  There are a number of available fields in that object but the only one we need to worry about is the snapshot_id.  This tells EC2 that when someone wants to start up a new instance of our AMI, EC2 needs to start by creating a new EBS volume from this snapshot.  The second data structure is the BlockDeviceMapping.  It is actually a subclass of a Python dictionary and behaves as you would expect.  We need to add an entry that maps the device name of our root volume (in this case /dev/sda1) to the EBS snapshot.  Finally, we call register_image to create the new AMI.  We pass in a name, a description, the architecture, the kernel and ramdisk (we are just referring to the same ones used by the original S3-backed AMI), the name of our root device (/dev/sda1) and the block device mapping we just created.  EC2 returns with a new AMI id and we can then use that to start new EC2 instances.  Just to verify, let's start up a new instance based on our EBS-backed AMI:

>>> c.run_instances('ami-f002e099', key_name='cloudright', security_groups=['test1'], instance_type='m1.large')
Reservation:r-f175d599
>>> inst2 = _.instances[0]
>>> inst2.update()
u'pending'
>>> inst2.update()
u'running'
>>> inst2.public_dns_name
u'ec2-75-101-218-5.compute-1.amazonaws.com'

Now let's SSH into our new EBS-based instance and make sure everything is okay.

jobs:~ mitch$ ssh -i ~/.ssh/cloudright.pem root@ec2-75-101-218-5.compute-1.amazonaws.com
...
root@domU-12-31-39-06-E1-62:~# df
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/sda1             10321208    837672   8959248   9% /
tmpfs                  3935948         0   3935948   0% /lib/init/rw
varrun                 3935948        40   3935908   1% /var/run
varlock                3935948         0   3935948   0% /var/lock
udev                   3935948        72   3935876   1% /dev
tmpfs                  3935948         0   3935948   0% /dev/shm
root@domU-12-31-39-06-E1-62:~# 

I have made this AMI public and available in the us-east-1 region.  Feel free to fire it up and play around with it but be aware that none of the careful testing that accompanies Eric's or Canonical's AMI releases has happened here so it is for illustrative purposes only.