One-step provisioning of web apps on the Joyent Public Cloud

This relatively brief walkthrough is an interactive Python session that shows how the py-smartdc library can be used to:

  • explore the Joyent Public Cloud offerings,
  • interact with Joyent's SmartDataCenter API that is capable of managing both public and private clouds,
  • provision a basic SmartMachine,
  • install, configure and launch a web app with a single Python command,
  • manage selected parts of your cloud with the help of machine metadata.

Some of the terminology used is derived from the underlying API, so if you wish to go into more detail about the Joyent cloud, I suggest examining their API and wiki, or, for a reference on the Python wrapper, see its official documentation.

In order to follow along with this tutorial, you should be able to know how to use pip to install the library and its dependencies (primarily PyCrypto and Requests):

pip install smartdc

You also require an active Joyent Public Cloud account with at least one uploaded SSH public key and with the ability to spend $0.03 on a machine that will be provisioned and torn down.

Aside: this tutorial and walkthrough was generated using the IPython notebook and refined and verified through actual usage.

Setup

In [1]:
 
from smartdc import DataCenter

The most important class in the library that you will interact with is the DataCenter. It is a hub for connections with the API, and offers abstractions for querying the API and for finding and creating new machines.

When running this tutorial yourself, please include your own username and key name in the below arguments.

In [2]:
 
sdc = DataCenter(location='eu-ams-1', key_id='/username/keys/keyname')

One of the easiest things you can do is query the API for all of the datacenters available from the account credentials you entered. As the Smart Data Center product evolves, there will be more and more public and private clouds that conform to this API. Currently, there is one other public cloud like this that I am aware of: Telefónica's InstantServers, which are more or less just like the current Joyent Public Cloud offering, but with datacenters (as of this writing) in London and Madrid.

Joyent runs datacenters in the Netherlands, Virginia, Nevada, and California, respectively:

In [3]:
 
sdc.datacenters()
Out[3]:
{u'eu-ams-1': u'https://eu-ams-1.api.joyentcloud.com',
 u'us-east-1': u'https://us-east-1.api.joyentcloud.com',
 u'us-sw-1': u'https://us-sw-1.api.joyentcloud.com',
 u'us-west-1': u'https://us-west-1.api.joyentcloud.com'}

Each DataCenter object has access to all of the other datacenters within the same cloud (as unified by the user credentials). So from a European datacenter, I can instantiate another DataCenter object, and start querying that:

In [4]:
 
west = sdc.datacenter('us-west-1')

And that object retains the configuration that the original DataCenter object was established with.

The REST API has some simple affordances we can tap into, just to be sure the connection is working well:

In [5]:
 
west.api()
Out[5]:
{u'endpoints': [u'GET    /:login/keys',
  u'GET    /:login/keys/:key',
  u'POST   /:login/keys',
  u'DELETE /:login/keys/:key',
  u'GET    /:login/datacenters',
  u'GET    /:login/datacenters/:name',
  u'GET    /:login/datasets',
  u'GET    /:login/datasets/:id',
  u'GET    /:login/packages',
  u'GET    /:login/packages/:package',
  u'GET    /:login/machines',
  u'GET    /:login/machines/:id',
  u'POST   /:login/machines',
  u'POST   /:login/machines/:id?action=stop',
  u'POST   /:login/machines/:id?action=start',
  u'POST   /:login/machines/:id?action=reboot',
  u'POST   /:login/machines/:id?action=resize',
  u'POST   /:login/machines/:id/snapshots',
  u'POST   /:login/machines/:id/snapshots/:name',
  u'GET    /:login/machines/:id/snapshots',
  u'GET    /:login/machines/:id/snapshots/:name',
  u'DELETE /:login/machines/:id/snapshots/:name',
  u'POST   /:login/machines/:id/metadata',
  u'GET    /:login/machines/:id/metadata',
  u'DELETE /:login/machines/:id/metadata/:key',
  u'DELETE /:login/machines/:id/metadata',
  u'POST   /:login/machines/:id/tags',
  u'GET    /:login/machines/:id/tags',
  u'GET    /:login/machines/:id/tags/:tag',
  u'DELETE /:login/machines/:id/tags/:tag',
  u'DELETE /:login/machines/:id/tags',
  u'DELETE /:login/machines/:id',
  u'GET    /:login/analytics',
  u'GET    /:login/analytics/instrumentations',
  u'GET    /:login/analytics/instrumentations/:id',
  u'GET    /:login/analytics/instrumentations/:id/value/raw',
  u'GET    /:login/analytics/instrumentations/:id/value/heatmap/image',
  u'GET    /:login/analytics/instrumentations/:id/value/heatmap/details',
  u'POST   /:login/analytics/instrumentations',
  u'DELETE /:login/analytics/instrumentations/:id']}

Incidentally, the py-smartdc library covers all of those endpoints except for the analytics ones. I simply hadn't yet found a use case for the analytics. However, if you're keen to use them from Python, please contact me (hello@atl.me).

Exploring machine image datasets and resource size packages

Let's take a look at what what machines we might provision, and let's turn on the verbose switch so we can debug which RESTful endpoints we hit, and when.

In [6]:
 
east = DataCenter(location='us-east-1', key_id='/username/keys/keyname', verbose=True)

The packages() method allows for a regexp to be run on the names, so we can look for "Small"ish packages.

In [7]:
 
east.packages('Small')
2013-01-18T15:14:21.464264	GET	https://us-east-1.api.joyentcloud.com/my/packages
Out[7]:
[{u'default': False,
  u'disk': 15360,
  u'memory': 512,
  u'name': u'Extra Small 512 MB',
  u'swap': 1024,
  u'vcpus': 1},
 {u'default': True,
  u'disk': 30720,
  u'memory': 1024,
  u'name': u'Small 1GB',
  u'swap': 2048,
  u'vcpus': 1}]

Similarly, the datasets() method allows for regexps to be run on the "description" and "urn" fields by default...

In [8]:
 
east.datasets('sdc:sdc:')
2013-01-18T15:14:27.440315	GET	https://us-east-1.api.joyentcloud.com/my/datasets
Out[8]:
[{u'created': u'2012-02-14T05:20:52+00:00',
  u'default': False,
  u'description': u'Fedora 14 64-bit image with just essential packages installed. Ideal for users who are comfortable with setting up their own environment and tools.',
  u'id': u'3f8a3d02-43e4-11e1-9565-7f82a075e289',
  u'name': u'fedora-14',
  u'os': u'linux',
  u'requirements': {},
  u'type': u'virtualmachine',
  u'urn': u'sdc:sdc:fedora-14:1.0.1',
  u'version': u'1.0.1'},
 {u'created': u'2012-02-14T03:54:01+00:00',
  u'default': False,
  u'description': u'Node.js git-deploy PaaS dataset',
  u'id': u'f953e97e-4991-11e1-9ea4-27c6e7e8afda',
  u'name': u'nodejs',
  u'os': u'smartos',
  u'requirements': {},
  u'type': u'smartmachine',
  u'urn': u'sdc:sdc:nodejs:1.3.3',
  u'version': u'1.3.3'},
 {u'created': u'2012-02-14T05:53:49+00:00',
  u'default': False,
  u'description': u'CentOS 5.7 64-bit image with just essential packages installed. Ideal for users who are comfortable with setting up their own environment and tools.',
  u'id': u'988c2f4e-4314-11e1-8dc3-2bc6d58f4be2',
  u'name': u'centos-5.7',
  u'os': u'linux',
  u'requirements': {},
  u'type': u'virtualmachine',
  u'urn': u'sdc:sdc:centos-5.7:1.2.1',
  u'version': u'1.2.1'},
 {u'created': u'2012-02-22T18:27:32+00:00',
  u'default': False,
  u'description': u'Ubuntu 10.04 64-bit image with just essential packages installed. Ideal for users who are comfortable with setting up their own environment and tools.',
  u'id': u'71101322-43a5-11e1-8f01-cf2a3031a7f4',
  u'name': u'ubuntu-10.04',
  u'os': u'linux',
  u'requirements': {},
  u'type': u'virtualmachine',
  u'urn': u'sdc:sdc:ubuntu-10.04:1.0.1',
  u'version': u'1.0.1'},
 {u'created': u'2012-10-01T00:07:01+00:00',
  u'default': False,
  u'description': u'CentOS 5.7 64-bit image with just essential packages installed. Ideal for users who are comfortable with setting up their own environment and tools.',
  u'id': u'2539f6de-0b5a-11e2-b647-fb08c3503fb2',
  u'name': u'centos-5.7',
  u'os': u'linux',
  u'requirements': {},
  u'type': u'virtualmachine',
  u'urn': u'sdc:sdc:centos-5.7:1.3.0',
  u'version': u'1.3.0'},
 {u'created': u'2012-10-03T22:16:53+00:00',
  u'default': False,
  u'description': u'CentOS 6 64-bit image with just essential packages installed. Ideal for users who are comfortable with setting up their own environment and tools.',
  u'id': u'8700b668-0da4-11e2-bde4-17221283a2f4',
  u'name': u'centos-6',
  u'os': u'linux',
  u'requirements': {},
  u'type': u'virtualmachine',
  u'urn': u'sdc:sdc:centos-6:1.3.0',
  u'version': u'1.3.0'},
 {u'created': u'2012-02-15T20:04:18+00:00',
  u'default': False,
  u'description': u'CentOS 6 64-bit image with just essential packages installed. Ideal for users who are comfortable with setting up their own environment and tools.',
  u'id': u'e4cd7b9e-4330-11e1-81cf-3bb50a972bda',
  u'name': u'centos-6',
  u'os': u'linux',
  u'requirements': {},
  u'type': u'virtualmachine',
  u'urn': u'sdc:sdc:centos-6:1.0.1',
  u'version': u'1.0.1'},
 {u'created': u'2012-02-14T05:21:53+00:00',
  u'default': False,
  u'description': u'Debian 6.03 64-bit image with just essential packages installed. Ideal for users who are comfortable with setting up their own environment and tools.',
  u'id': u'e6ac6784-44b3-11e1-8555-87c3dd87aafe',
  u'name': u'debian-6.03',
  u'os': u'linux',
  u'requirements': {},
  u'type': u'virtualmachine',
  u'urn': u'sdc:sdc:debian-6.03:1.0.0',
  u'version': u'1.0.0'},
 {u'created': u'2013-01-03T22:57:52+00:00',
  u'default': False,
  u'description': u'Debian 6.0.6 64-bit image with just essential packages installed. Ideal for users who are comfortable with setting up their own environment and tools.',
  u'id': u'46ecf60e-52c8-11e2-b212-9b51fc749547',
  u'name': u'debian-6.0.6',
  u'os': u'linux',
  u'requirements': {},
  u'type': u'virtualmachine',
  u'urn': u'sdc:sdc:debian-6.0.6:2.3.1',
  u'version': u'2.3.1'},
 {u'created': u'2013-01-03T16:54:01+00:00',
  u'default': False,
  u'description': u'64-bit SmartOS image pre-configured to run Open Source Riak, a Dynamo-inspired key/value store from Basho.',
  u'id': u'fb6ef720-55c1-11e2-b6de-bf6869fd8d41',
  u'name': u'riak',
  u'os': u'smartos',
  u'requirements': {},
  u'type': u'smartmachine',
  u'urn': u'sdc:sdc:riak:1.7.0',
  u'version': u'1.7.0'},
 {u'created': u'2013-01-03T15:49:50+00:00',
  u'default': False,
  u'description': u'Base image with core packages preinstalled',
  u'id': u'fdea06b0-3f24-11e2-ac50-0b645575ce9d',
  u'name': u'base64',
  u'os': u'smartos',
  u'requirements': {},
  u'type': u'smartmachine',
  u'urn': u'sdc:sdc:base64:1.8.4',
  u'version': u'1.8.4'},
 {u'created': u'2013-01-03T15:49:15+00:00',
  u'default': False,
  u'description': u'Base image with core packages preinstalled',
  u'id': u'84cb7edc-3f22-11e2-8a2a-3f2a7b148699',
  u'name': u'base',
  u'os': u'smartos',
  u'requirements': {},
  u'type': u'smartmachine',
  u'urn': u'sdc:sdc:base:1.8.4',
  u'version': u'1.8.4'},
 {u'created': u'2012-10-12T22:38:15+00:00',
  u'default': False,
  u'description': u'64-bit SmartOS image with MongoDB 2.2.0 pre-installed and pre-configured for maximum performance and scalability.',
  u'id': u'b00acc20-14ab-11e2-85ae-4f03a066e93e',
  u'name': u'mongodb',
  u'os': u'smartos',
  u'requirements': {u'min_memory': 1024},
  u'type': u'smartmachine',
  u'urn': u'sdc:sdc:mongodb:1.4.0',
  u'version': u'1.4.0'},
 {u'created': u'2012-10-11T14:41:44+00:00',
  u'default': False,
  u'description': u'A SmartOS image with Node.js, npm, MongoDB and other useful packages pre-installed. Ideal for Node.js applications.',
  u'id': u'1fc068b0-13b0-11e2-9f4e-2f3f6a96d9bc',
  u'name': u'nodejs',
  u'os': u'smartos',
  u'requirements': {},
  u'type': u'smartmachine',
  u'urn': u'sdc:sdc:nodejs:1.4.0',
  u'version': u'1.4.0'},
 {u'created': u'2012-09-25T14:16:18+00:00',
  u'default': False,
  u'description': u'A SmartOS image with just essential packages installed. Ideal for users who are comfortable with setting up their own environment and tools.',
  u'id': u'55330ab4-066f-11e2-bd0f-434f2462fada',
  u'name': u'base',
  u'os': u'smartos',
  u'requirements': {},
  u'type': u'smartmachine',
  u'urn': u'sdc:sdc:base:1.8.1',
  u'version': u'1.8.1'},
 {u'created': u'2012-09-25T21:38:18+00:00',
  u'default': False,
  u'description': u'A SmartOS 64-bit image with just essential packages installed. Ideal for users who are comfortable with setting up their own environment and tools.',
  u'id': u'60a3b1fa-0674-11e2-abf5-cb82934a8e24',
  u'name': u'base64',
  u'os': u'smartos',
  u'requirements': {},
  u'type': u'smartmachine',
  u'urn': u'sdc:sdc:base64:1.8.1',
  u'version': u'1.8.1'},
 {u'created': u'2012-09-21T22:59:06+00:00',
  u'default': False,
  u'description': u'SmartOS image pre-configured and optimized as a Percona 5.5 64-bit MySQL server with Quickbackup, Handlersocket, and Sphinx Plug-in',
  u'id': u'dc1a8b5e-043c-11e2-9d94-0f3fcb2b0c6d',
  u'name': u'percona',
  u'os': u'smartos',
  u'requirements': {},
  u'type': u'smartmachine',
  u'urn': u'sdc:sdc:percona:1.6.0',
  u'version': u'1.6.0'},
 {u'created': u'2012-08-30T21:38:33+00:00',
  u'default': False,
  u'description': u'A SmartOS 64-bit image optimized for web development. Everything you need to build a web stack \u2014 such as Ruby, Apache, and MySQL \u2014 is installed and ready to use.',
  u'id': u'a0f8cf30-f2ea-11e1-8a51-5793736be67c',
  u'name': u'standard64',
  u'os': u'smartos',
  u'requirements': {},
  u'type': u'smartmachine',
  u'urn': u'sdc:sdc:standard64:1.0.7',
  u'version': u'1.0.7'},
 {u'created': u'2012-08-30T21:30:56+00:00',
  u'default': True,
  u'description': u'A SmartOS image optimized for web development. Everything you need to build a web stack \u2014 such as Ruby, Apache, and MySQL \u2014 is installed and ready to use.',
  u'id': u'3390ca7c-f2e7-11e1-8818-c36e0b12e58b',
  u'name': u'standard',
  u'os': u'smartos',
  u'requirements': {},
  u'type': u'smartmachine',
  u'urn': u'sdc:sdc:standard:1.0.7',
  u'version': u'1.0.7'},
 {u'created': u'2012-08-29T00:28:42+00:00',
  u'default': False,
  u'description': u'A SmartOS 64-bit image with just essential packages installed. Ideal for users who are comfortable with setting up their own environment and tools.',
  u'id': u'e8c41d40-f161-11e1-b839-a3631c115653',
  u'name': u'base64',
  u'os': u'smartos',
  u'requirements': {},
  u'type': u'smartmachine',
  u'urn': u'sdc:sdc:base64:1.7.2',
  u'version': u'1.7.2'},
 {u'created': u'2012-08-29T00:24:31+00:00',
  u'default': False,
  u'description': u'A SmartOS image with just essential packages installed. Ideal for users who are comfortable with setting up their own environment and tools.',
  u'id': u'9012a9c2-f15d-11e1-a33a-afaec53ebde9',
  u'name': u'base',
  u'os': u'smartos',
  u'requirements': {},
  u'type': u'smartmachine',
  u'urn': u'sdc:sdc:base:1.7.2',
  u'version': u'1.7.2'},
 {u'created': u'2012-07-02T16:39:53+00:00',
  u'default': False,
  u'description': u'A SmartOS image preconfigured to run Open Source Riak, a Dynamo-inspired key/value store from Basho.',
  u'id': u'399e09fc-c448-11e1-b3c8-a3d517dfbb07',
  u'name': u'riak',
  u'os': u'smartos',
  u'requirements': {},
  u'type': u'smartmachine',
  u'urn': u'sdc:sdc:riak:1.6.1',
  u'version': u'1.6.1'},
 {u'created': u'2012-05-25T14:37:02+00:00',
  u'default': False,
  u'description': u'Generic multi-purpose 64-bit SmartMachine. This image is deprecated and will be replaced by the 64 bit SmartOS Standard image (base64).',
  u'id': u'f9e4be48-9466-11e1-bc41-9f993f5dff36',
  u'name': u'smartos64',
  u'os': u'smartos',
  u'requirements': {},
  u'type': u'smartmachine',
  u'urn': u'sdc:sdc:smartos64:1.6.3',
  u'version': u'1.6.3'},
 {u'created': u'2012-08-29T19:27:47+00:00',
  u'default': False,
  u'description': u'Generic multi-purpose SmartMachine. This image is deprecated and will be replaced by the SmartOS Base image (base).',
  u'id': u'01b2c898-945f-11e1-a523-af1afbe22822',
  u'name': u'smartos',
  u'os': u'smartos',
  u'requirements': {},
  u'type': u'smartmachine',
  u'urn': u'sdc:sdc:smartos:1.6.3',
  u'version': u'1.6.3'},
 {u'created': u'2012-03-02T15:30:58+00:00',
  u'default': False,
  u'description': u'Generic multi-purpose 64-bit SmartMachine. This image is deprecated and will be replaced by the 64 bit SmartOS Standard image (standard64).',
  u'id': u'f4bc70ca-5e2c-11e1-8380-fb28785857cb',
  u'name': u'smartosplus64',
  u'os': u'smartos',
  u'requirements': {},
  u'type': u'smartmachine',
  u'urn': u'sdc:sdc:smartosplus64:3.1.0',
  u'version': u'3.1.0'},
 {u'created': u'2012-03-02T15:24:10+00:00',
  u'default': False,
  u'description': u'Generic multi-purpose SmartMachine. This image is deprecated and will be replaced by the SmartOS Standard image (standard).',
  u'id': u'a963d5d0-5e29-11e1-a4d7-a31977b1e6dd',
  u'name': u'smartosplus',
  u'os': u'smartos',
  u'requirements': {},
  u'type': u'smartmachine',
  u'urn': u'sdc:sdc:smartosplus:3.1.0',
  u'version': u'3.1.0'},
 {u'created': u'2012-03-02T15:20:17+00:00',
  u'default': False,
  u'description': u'Generic multi-purpose 64-bit SmartMachine. This image is deprecated and will be replaced by the 64 bit SmartOS Standard image (base64).',
  u'id': u'31bc4dbe-5e06-11e1-907c-5bed6b255fd1',
  u'name': u'smartos64',
  u'os': u'smartos',
  u'requirements': {},
  u'type': u'smartmachine',
  u'urn': u'sdc:sdc:smartos64:1.5.4',
  u'version': u'1.5.4'},
 {u'created': u'2012-03-02T15:16:12+00:00',
  u'default': False,
  u'description': u'Generic multi-purpose SmartMachine. This image is deprecated and will be replaced by the SmartOS Base image (base).',
  u'id': u'489754f2-5e01-11e1-8ff8-f770c2116b0d',
  u'name': u'smartos',
  u'os': u'smartos',
  u'requirements': {},
  u'type': u'smartmachine',
  u'urn': u'sdc:sdc:smartos:1.5.4',
  u'version': u'1.5.4'},
 {u'created': u'2012-02-13T19:24:17+00:00',
  u'default': False,
  u'description': u'Percona SmartMachine',
  u'id': u'a9380908-ea0e-11e0-aeee-4ba794c83c33',
  u'name': u'percona',
  u'os': u'smartos',
  u'requirements': {},
  u'type': u'smartmachine',
  u'urn': u'sdc:sdc:percona:1.0.7',
  u'version': u'1.0.7'}]

...but that default field can be overridden, if we, say, are looking for images created in 2013:

In [9]:
 
east.datasets('^2013', ['created'])
2013-01-18T15:14:34.866238	GET	https://us-east-1.api.joyentcloud.com/my/datasets
Out[9]:
[{u'created': u'2013-01-03T22:57:52+00:00',
  u'default': False,
  u'description': u'Debian 6.0.6 64-bit image with just essential packages installed. Ideal for users who are comfortable with setting up their own environment and tools.',
  u'id': u'46ecf60e-52c8-11e2-b212-9b51fc749547',
  u'name': u'debian-6.0.6',
  u'os': u'linux',
  u'requirements': {},
  u'type': u'virtualmachine',
  u'urn': u'sdc:sdc:debian-6.0.6:2.3.1',
  u'version': u'2.3.1'},
 {u'created': u'2013-01-03T16:54:01+00:00',
  u'default': False,
  u'description': u'64-bit SmartOS image pre-configured to run Open Source Riak, a Dynamo-inspired key/value store from Basho.',
  u'id': u'fb6ef720-55c1-11e2-b6de-bf6869fd8d41',
  u'name': u'riak',
  u'os': u'smartos',
  u'requirements': {},
  u'type': u'smartmachine',
  u'urn': u'sdc:sdc:riak:1.7.0',
  u'version': u'1.7.0'},
 {u'created': u'2013-01-03T17:13:38+00:00',
  u'default': False,
  u'description': u'64-bit SmartOS image pre-configured to run the Enterprise DS version of Riak, a Dynamo-inspired key/value store from Basho.',
  u'id': u'1bf1cac0-55c4-11e2-8182-c73dfa2ad700',
  u'name': u'riakeds',
  u'os': u'smartos',
  u'requirements': {},
  u'type': u'smartmachine',
  u'urn': u'local:admin:riakeds:1.7.0',
  u'version': u'1.7.0'},
 {u'created': u'2013-01-03T15:49:50+00:00',
  u'default': False,
  u'description': u'Base image with core packages preinstalled',
  u'id': u'fdea06b0-3f24-11e2-ac50-0b645575ce9d',
  u'name': u'base64',
  u'os': u'smartos',
  u'requirements': {},
  u'type': u'smartmachine',
  u'urn': u'sdc:sdc:base64:1.8.4',
  u'version': u'1.8.4'},
 {u'created': u'2013-01-03T15:49:15+00:00',
  u'default': False,
  u'description': u'Base image with core packages preinstalled',
  u'id': u'84cb7edc-3f22-11e2-8a2a-3f2a7b148699',
  u'name': u'base',
  u'os': u'smartos',
  u'requirements': {},
  u'type': u'smartmachine',
  u'urn': u'sdc:sdc:base:1.8.4',
  u'version': u'1.8.4'}]

These methods are nothing earth-shatteringly novel, just simple local filtering in the interest of ease-of-use, typically with interactive exploration in mind.

Something I've found useful for the automation case is to take advantage of an undocumented feature in the REST API: SDC auto-resolves incomplete URNs, so if I want the latest and greatest version of a mchine image (and trust that my other configuration processes won't break it), I can refer to the URN prefix:

In [10]:
 
east.dataset('sdc:sdc:riak:')
2013-01-18T15:14:42.359173	GET	https://us-east-1.api.joyentcloud.com/my/datasets/sdc:sdc:riak:
Out[10]:
{u'created': u'2013-01-03T16:54:01+00:00',
 u'default': False,
 u'description': u'64-bit SmartOS image pre-configured to run Open Source Riak, a Dynamo-inspired key/value store from Basho.',
 u'id': u'fb6ef720-55c1-11e2-b6de-bf6869fd8d41',
 u'name': u'riak',
 u'os': u'smartos',
 u'requirements': {},
 u'type': u'smartmachine',
 u'urn': u'sdc:sdc:riak:1.7.0',
 u'version': u'1.7.0'}

So, when provisioning machines, the dict with full information as well as the identifier are equally valid as parameters. The dict describing a dataset is equivalent to its URN, and the dict describing a package is equivalent to passing its name as a parameter.

Provisioning a SmartMachine, installing an application

In [11]:
 
xsmall = east.packages('512')[0]
xsmall
2013-01-18T15:14:47.178284	GET	https://us-east-1.api.joyentcloud.com/my/packages
Out[11]:
{u'default': False,
 u'disk': 15360,
 u'memory': 512,
 u'name': u'Extra Small 512 MB',
 u'swap': 1024,
 u'vcpus': 1}
In [12]:
 
standard = east.dataset('sdc:sdc:standard:')
standard
2013-01-18T15:14:49.673999	GET	https://us-east-1.api.joyentcloud.com/my/datasets/sdc:sdc:standard:
Out[12]:
{u'created': u'2012-08-30T21:30:56+00:00',
 u'default': True,
 u'description': u'A SmartOS image optimized for web development. Everything you need to build a web stack \u2014 such as Ruby, Apache, and MySQL \u2014 is installed and ready to use.',
 u'id': u'3390ca7c-f2e7-11e1-8818-c36e0b12e58b',
 u'name': u'standard',
 u'os': u'smartos',
 u'requirements': {},
 u'type': u'smartmachine',
 u'urn': u'sdc:sdc:standard:1.0.7',
 u'version': u'1.0.7'}

We want to add tags to the machine we'll be provisioning. Machine tags allow the Smart DataCenter API to find the machine, querying amongst all machines in a given datacenter. For illustration purposes, we'll replicate the same dictionary into the metadata for the machine. Machine metadata exposes key-value pairs inside the machine, allowing for some introspection from the machine itself about its role.

In [13]:
 
tags = {'type': 'trac',
        'project': 'sdc-demo'}

Finally, we want to perform some fast provisioning of an application. The boot script functionality added to SmartDataCenter is incredibly powerful for those purposes: it will run an arbitrary shell script on startup. It will timeout after 60 seconds, so it's not built for long-running processes. However, a lot can be done in that time. For example, here's a script that:

  • Installs trac and its dependencies from pkgsrc
  • Configures a default base trac environment
  • Reconfigures SMF to make trac listen on the public IP address
  • Restarts the trac service

It is hosted as a gist, so you are encouraged to examine the script yourself before following along. The boot_script argument currently takes a local filename, so we'll grab it from gist and stick it into a local temp file (which has a nice side-effect of allowing us to inspect the script before doing anything else with it):

In [14]:
 
from urllib2 import urlopen
from contextlib import closing
 
TEMPFILE = '/tmp/boot_script'
 
with closing(urlopen('https://gist.github.com/raw/4555239/trac-init.sh')) as input:
    with file(TEMPFILE, 'w') as output:
        for line in input:
            output.write(line)
            print line.strip()
#!/usr/bin/sh

# Install trac
/opt/local/bin/pkgin -y in trac

# set up the base directory (set as default by the SMF manifest)
/opt/local/bin/mkdir /trac
/opt/local/bin/chown www:www /trac

# configure a minimal base environment
/opt/local/bin/sudo -u www /opt/local/bin/trac-admin /trac/test initenv "Test environment" sqlite:db/trac.db

## The following is definitely not recommended for production:
## running a wiki with no access controls is just asking for trouble!

# change trac to listen on the public IP address:
PUBLIC=$(/usr/sbin/ifconfig -a | /opt/local/bin/grep -A 1 net0 | /opt/local/bin/awk '/inet/ { print $2 }');
/usr/sbin/svccfg -s trac:default setprop config/listen_ip=astring: $PUBLIC
/usr/sbin/svcadm refresh trac:default
/usr/sbin/svcadm restart trac:default

With this, we're ready to instantiate an Extra Small, SmartOS Standard instance in the us-east-1 datacenter with some tags to index the machine and a boot script to demonstrate the installation, configuration, and launch of a web app.

In [15]:
 
my_trac_demo = east.create_machine(name="provisioning-demo", package=xsmall, dataset=standard, tags=tags, metadata=tags, boot_script=TEMPFILE)
2013-01-18T15:15:01.343895	POST	https://us-east-1.api.joyentcloud.com/my/machines

We need to wait for the machine to provision. As of this writing, it's taking much longer than I've been accustomed. Perhaps the $0.03/hour instances are popular. :)

In [16]:
 
my_trac_demo.poll_until('running', interval=10)
2013-01-18T15:15:13.246104	GET	https://us-east-1.api.joyentcloud.com/my/machines/e7925afd-9adf-44cf-a23e-c7179f770cd3
2013-01-18T15:15:24.465004	GET	https://us-east-1.api.joyentcloud.com/my/machines/e7925afd-9adf-44cf-a23e-c7179f770cd3
2013-01-18T15:15:35.686114	GET	https://us-east-1.api.joyentcloud.com/my/machines/e7925afd-9adf-44cf-a23e-c7179f770cd3
2013-01-18T15:15:46.813774	GET	https://us-east-1.api.joyentcloud.com/my/machines/e7925afd-9adf-44cf-a23e-c7179f770cd3
2013-01-18T15:15:58.333309	GET	https://us-east-1.api.joyentcloud.com/my/machines/e7925afd-9adf-44cf-a23e-c7179f770cd3
2013-01-18T15:16:09.502814	GET	https://us-east-1.api.joyentcloud.com/my/machines/e7925afd-9adf-44cf-a23e-c7179f770cd3
2013-01-18T15:16:20.659734	GET	https://us-east-1.api.joyentcloud.com/my/machines/e7925afd-9adf-44cf-a23e-c7179f770cd3
2013-01-18T15:16:31.796816	GET	https://us-east-1.api.joyentcloud.com/my/machines/e7925afd-9adf-44cf-a23e-c7179f770cd3

After this call returns, we can wait for a few seconds for the boot script to run, and then open a web browser on our brand new trac instance.

In [17]:
 
import webbrowser
 
webbrowser.open("http://%s:9001/test" % my_trac_demo.public_ips[0])
Out[17]:
True

Once you see Trac's front page (you may need to reload after several seconds as the boot script does its work), you can continue. At this point, I like to take a machine snapshot as a checkpoint to a known state. These are remarkably fast, and "cheap" in the sense of taking up only the disk space unique to the snapshot (i.e., the amount of data your SmartMachine has changed since the snapshot was taken).

In [18]:
 
my_trac_demo.create_snapshot("post-install")
2013-01-18T15:17:08.247562	POST	https://us-east-1.api.joyentcloud.com/my/machines/e7925afd-9adf-44cf-a23e-c7179f770cd3/snapshots
Out[18]:
<smartdc.machine.Snapshot: <post-install> on <Machine: provisioning-demo>>

Cleaning up

Now, if your goal was to install trac, then I suggest you stop here, and perhaps spend some time configuring the installation to work with a MySQL or PostgreSQL database installed with the image, password protect the instance, make it listen to localhost only, and/or proxy via something like nginx, which is also available in the Standard SmartMachine image.

However, because this costs some trivial-but-non-zero amount of money, let's go ahead and shut the machine down.

In [19]:
 
demo_machines = east.machines(tags={'project': 'sdc-demo'})
demo_machines
2013-01-18T15:17:30.579795	GET	https://us-east-1.api.joyentcloud.com/my/machines
Out[19]:
[<smartdc.machine.Machine: <provisioning-demo> in <DataCenter: us-east-1>>]

One of the reasons we entered machine tags was so we could recall the instance via the API. The demo_machines list should return a single machine if you're following along with this example line by line, but while debugging this tutorial, for example, a handful of instances were created. So you may well use this pattern a lot.

[For a slightly more sophisticated and efficient way of parallelizing operations on multiple machines, see the "advanced example" in the py-smartdc documentation's tutorial for how to run similar operations as below by using a thread pool.]

In [20]:
 
[m.stop() for m in demo_machines]
2013-01-18T15:17:36.162121	POST	https://us-east-1.api.joyentcloud.com/my/machines/e7925afd-9adf-44cf-a23e-c7179f770cd3
Out[20]:
[None]

Calling delete before the machine reaches the stopped state means that the delete will fail, so ordinarily we poll until the machine is stopped.

In [21]:
 
[m.poll_until('stopped') for m in demo_machines]
2013-01-18T15:17:40.434580	GET	https://us-east-1.api.joyentcloud.com/my/machines/e7925afd-9adf-44cf-a23e-c7179f770cd3
2013-01-18T15:17:43.586652	GET	https://us-east-1.api.joyentcloud.com/my/machines/e7925afd-9adf-44cf-a23e-c7179f770cd3
2013-01-18T15:17:46.728334	GET	https://us-east-1.api.joyentcloud.com/my/machines/e7925afd-9adf-44cf-a23e-c7179f770cd3
2013-01-18T15:17:49.871453	GET	https://us-east-1.api.joyentcloud.com/my/machines/e7925afd-9adf-44cf-a23e-c7179f770cd3
2013-01-18T15:17:52.994264	GET	https://us-east-1.api.joyentcloud.com/my/machines/e7925afd-9adf-44cf-a23e-c7179f770cd3
Out[21]:
[None]
In [22]:
 
[m.delete() for m in demo_machines]
2013-01-18T15:17:54.108202	DELETE	https://us-east-1.api.joyentcloud.com/my/machines/e7925afd-9adf-44cf-a23e-c7179f770cd3
Out[22]:
[None]

One last look to ensure none of the demo machines are still provisioned:

In [23]:
 
east.machines(tags={'project': 'sdc-demo'})
2013-01-18T15:18:01.022509	GET	https://us-east-1.api.joyentcloud.com/my/machines
Out[23]:
[]

At this point we've seen basic usage of the Python SmartDataCenter API wrapper to programmatically interact with SmartDataCenter's RESTful API. After simple exploration of the API's offerings and filtering through the different machine image datasets and resource packages, we saw the installation, configuration, and verification of a web app through the inclusion of a simple boot script. We also cleaned up after ourselves to ensure that we wouldn't be billed extra for extra unused resources.

Further along in the series will be to explore some of the capabilities of IPython for easy parallel computation. In order to get there, we will examine how to build and manage its installation so that provisioning IPython engines is as simple as trac was done here.