In mid-April 2011 OpenStack released Cactus, the third release of the project NASA and Rackspace started in July 2010 under the Apache 2.0 licence. The significant technical addition is live migration in Nova: a virtual machine moves from one compute node to another without being shut down. The rest is consolidation, and that is the part that says most to anyone wanting to know whether this holds up in a real environment. I am writing these notes at the end of the month, once the first architectural and operational material worth reasoning about has appeared.
Context
The Open Source Infrastructure-as-a-Service (IaaS) space in 2011 is not empty. Eucalyptus has been around since 2008, with Amazon EC2/S3-compatible APIs and a GPL licence; OpenNebula comes from the European HPC world; Cloud.com ships CloudStack with a more enterprise-oriented angle. What sets OpenStack apart is its release model: a six-monthly cadence, public code, several vendors at the table — Austin in October 2010, Bexar in February 2011, now Cactus.
The problem driving interest in private cloud is concrete. Anyone with regulatory constraints, data-sovereignty requirements, or simply a hardware estate already paid for cannot replicate the AWS model in-house without writing a lot of code. OpenStack tries to supply the pieces: Nova for compute, Swift for object storage, Glance for the image registry.
Nova’s architecture
Nova is a set of Python daemons that talk asynchronously over a message queue and share state through an SQL database. The structure, described in detail by Ken Pepple, is worth understanding before deciding whether to adopt it.
The entry point is nova-api: it receives REST requests, applies policy (quotas, for instance) and queues actions. The nova-scheduler takes an instance request off the queue and decides which compute host should run it. The nova-compute daemon is the worker that actually creates and tears down VMs, turning messages into hypervisor calls. Alongside these sit nova-volume, which manages persistent volumes with a model close to Amazon EBS, and nova-network, which configures bridges and firewall rules.
Underneath, two pieces of infrastructure: an AMQP (Advanced Message Queuing Protocol) queue, currently on RabbitMQ, and an SQL database holding build-time and run-time state, with SQLite3, MySQL and PostgreSQL. The daemons do not talk to each other directly: they publish actions on the queue and read shared state from the database. It is this choice that makes the system horizontally scalable, because I can start more nova-compute workers on different nodes without touching the communication topology.
On the hypervisor front, Cactus adds LXC through libvirt and VMware/vSphere ESX/ESXi 4.1, on top of the XenServer and KVM support already present. Nova’s REST API follows the EC2 style: a deliberate choice, to remove friction for people coming from Amazon.
The critical point: networking
The most delicate part, and in my view the easiest to underestimate, is nova-network. In the documentation of this period there are three models, chosen with the --network_manager option.
FlatManager creates neither bridges nor VLANs: the administrator prepares by hand the bridges specified when networks are created with nova-manage, and that bridge must exist on every compute host. FlatDHCPManager adds DHCP management. VlanManager — the default, if you specify nothing — creates a managed VLAN per project, assigns each project its own subnet, and runs a DHCP server per subnet.
In practice, choosing a network manager drags in requirements on the physical network — VLAN support on the switches, bridges configured identically across all nodes — that need to be verified up front, not once the deployment is already running. At this stage networking lives inside Nova: there is not yet a separate, independent service to handle it.
Implications
For anyone evaluating a private cloud right now, Cactus gains ground on two concrete fronts. Nova’s live migration makes it possible to maintain a compute host without stopping the workloads running on it, which previously called for a downtime window. The static serving added to Swift serves content directly from object storage, with an index.html in container listings, and in simple cases removes a web server out front.
On the organisational side, the list of corporate members that have joined — HP, Dell, IBM, Intel, Canonical, Red Hat, SUSE, Cisco, among others — says the multi-vendor model is more than a stated intention. When you pick an infrastructure technology on a horizon of years, the breadth of contributors weighs as much as the features: that is where the odds of finding the project still maintained a few years out are decided.
Limits
Open problems remain, and it is fair to name them. There is still no unified identity service: authentication and authorisation between components are fragmented, and it shows when you want coherent access control over Nova, Swift and Glance together. Networking, as noted, sits inside Nova with its constraints on the hardware. Operational documentation is uneven: some of the most useful material for a real deployment comes from third-party guides — like the ones that have just appeared for Ubuntu 11.04 — rather than the official manuals.
To take it to production the question is not whether OpenStack works — Cactus works — but how much glue code and operational competence it takes to move it from a demo to real load. In mid-2011 the answer is still “a fair amount”, and that needs to be budgeted for.
https://wiki.openstack.org/wiki/ReleaseNotes/Cactus https://www.openstack.org/blog/openstack-announces-cactus-release/ https://ken.pepple.info/openstack/2011/04/22/openstack-nova-architecture/ https://cssoss.wordpress.com/2011/04/28/openstack-beginners-guide-for-ubuntu-11-04-network-management/
Cover image: Modular data center container labelled “Nebula” at NASA Ames Research Center, home of the cloud infrastructure that spawned… — photo by Gretchen Curtis, CC BY-SA 3.0 — https://commons.wikimedia.org/wiki/File:NASA.Nebula.cloud.container.JPG