Dogfooding - Pt. 2

This is a continuation of the first post; you should read that first!

This is part 2 of the “Dogfooding Juju” series that I’m doing. This time, I want to go into a little bit of detail about the Warren charm and how I wound up structuring it. As I mentioned in the previous post, there are perhaps more elegant ways to do this, but I found the documentation to be lacking in ways that prevented me from dedicating relatively scant free time to the task. Instead, I wound up following the path that I’ve followed before on the job with several of the charms we use for our own projects.

How a Charm Works

One can think of a charm as a deployment solution. In a lot of ways, it follows the same path as many other dev-ops solutions out there, such as Ansible, Chef, or Puppet. In fact, one can use any one of those solutions (or more than one, if one is so inclined) in managing what happens during the deployment of a charm into a service, as many charms do. Charms are meant to be biased, best-practice solutions that install a service and describe the way that service relates to other services in the Juju environment.

A charm is, at its core, primarily built around two concepts: configuration and hooks. Configuration describes the way the charm is built and how it can interact with other services, while hooks describe how the service responds to state changes, both internally and externally. There is a third bit, which we won’t get into here, as it’s not relevant, but is worth mentioning as part of the charm, which is “actions”. Actions are code within the service that responds to requests, either from the user or from the environment, through Juju itself.

A lot of (digital) ink has been spilled on charm building, so I’m not going to go too far in depth beyond explaining how this works with the Warren charm. Please feel free to check out the extensive docs if you’re interested in diving deeper.

Configuration

Configuration primarily happens in two files used by the charm: metadata.yaml and config.yaml. metadata.yaml contains information about the charm, who wrote it, and how it connects to various other charms within the environment, while config.yaml contains all of the configuration options that a charm may use during execution of any of its numerous hooks.

Anyone who is familiar with packaging software in any way will be familiar with the way these two files work. You can specify the name of the charm, the authors, a short description, and some tags in the metadata.yaml file. Additionally, if this charm relies on other services, they will be defined in the interfaces section. config.yaml is basically a schema describing the configuration options that the charm uses. For each option, a type, a description, and a default may be provided.

Hooks

Hooks are where all of the action takes place in a charm. There are a few main hooks, and then several which depend on the state of the environment. The main hooks are:

• install - work that needs to take place as the charm is first being installed.
• start - actions that take place as the charm is moving from installed to started states.
• stop - actions that take place as the charm is moving from started to stop or dying states.
• config-changed - actions that take place when any configuration value has changed.

The other hooks you might encounter are relation hooks. These are fired as the state of relations to the charm change. They come in four types, each of which includes the name of the relation interface as part of it:

• *-relation-joined - fired when the two services first start talking to each other.
• *-relation-changed - fired when some aspect of the relation is changed, such as data about that relation is changed.
• *-relation-departed - fired when a relation is removed by the user.
• *-relation-broken - fired when the relation is broken between the two services for some reason, such as one service being removed.

The Structure of the Warren Charm

The Warren charm is basically typical, as far as charms go. It has its own metadata and config files, as well as a full collection of hooks.

Configuration

metadata.yaml

The metadata.yaml file contains a lot of basics that will be familiar at a glance. Name, summary, maintainer, description, tags, these are all pretty straight forward. Of note, however, are the subordinate element, which declares whether or not this service will be subordinate to another (a topic for a later date), and the provides/requires elements, which describe how this service can relate to others.

Provides describes the interface that this service will expose to others within the Juju environment. Of particular note (mostly because the others haven’t been implemented yet) is the website interface, which provides a means of hosting content over HTTP/S. This will be used by the haproxy charm, which will provide load balancing over this interface.

Requires describes the interfaces that this service needs other charms to provide within the environment in order to run fully. In this case, this means Mongo via the mongodb charm, and ElasticSearch via the eponymous charm.

name: warren-charm
summary: Warren is a networked content-sharing site.
maintainer: Madison Scott-Clary 
description: |
  Warren is a networked content-sharing site, allowing users to not only post
  their creations, but link them together into a web of their works, and the
  works of others.  It manages each post as an abstract entity and uses content
  types to render those abstract types into something viewable within a
  browser.
tags:
  - social
  - cms
  - applications
subordinate: false
provides:
  website:
    interface: http
  nrpe-external-master:
    interface: nrpe-external-master
    scope: container
  local-monitors:
    interface: local-monitors
    scope: container
requires:
  mongodb:
    interface: mongodb
  elasticsearch:
    interface: elasticsearch

config.yaml

Our configuration values for this charm are also pretty straight-forward. You can see that we have options for an SMTP server, which will be used for sending notification emails, two keys which are used for encrypting session data, the database name, the port to listen on, and the source. Source is interesting because it’s structured to allow various different ways to fetch the source for building Warren. Since this is a thin charm (that is, it does not include any of the source for Warren itself), the charm will have to figure out how to fetch the source as required. We’ve provided a few ways of specifying that, all of which interface with Git: one can specify a branch name, a tag name, or a commit SHA.

options:
  smtp-server:
    default: smtp.example.com
    description: Address for the SMTP server for sending emails from Warren
    type: string
  session-auth-key:
    default: CHANGEME--------
    description: Session authentication key (16 or 32 bytes)
    type: string
  session-encryption-key:
    default: CHANGEME--------
    description: Session encryption key (16, 32, or 64 bytes)
    type: string
  mongo-db:
    default: warren
    description: The mongo database name
    type: string
  listen_port:
    default: 3000
    description: The port to listen on
    type: int
  source:
    default: "branch:master"
    description: A string containing a "branch:", "tag:", or "commit:" followed
      by a branch name, a tag name, or a commit, respectively
    type: string

Hooks

This is where the meat of the charm lives. Hooks are executable bits of code within the /hooks directory of the charm, each named appropriately. That is, there is an executable file in /hooks named install, one named start, and so on for all of the hooks that will be fired for our service. As is standard practice for this type of charm, I actually have all of the code in one file, hooks.py, and all of the hooks files are simply symlinked to point to that file.

I’m not going to go too in depth here, nor post the entire file, which you can look at yourself, but simply outline the way the hooks are called. Future posts may go more in depth as to how things work on a more atomic level.

First is our install hook, as shown by the decorator. This one takes care of some initial work that needs to be done to get the service up and running. It updates all packages, ensures dependencies (such as golang, git, and bzr), adds a user which will be used to run the service, makes source and build directories, and installs the source for Warren.

@hooks.hook('install')
def install():
    '''Install required packages, user, and warren source.'''
    apt_get_update()
    ensure_packages(*dependencies)    
    host.adduser(owner)
    prep_installation()
    install_from_source()

The stop hook is similarly simple. It stops the Warren service and deletes the upstart file for starting it.

@hooks.hook('stop')
def stop():
    '''Stop the warren service.'''
    log('Stopping service...')
    host.service_stop(system_service)
    if upstart_conf:
        unlink_if_exists(upstart_conf)

Here’s where the fun begins. As is standard practice for several charms, many hooks should behave in the same way. This was put to me by Kapil Thangavelu as, “There should only be a config-changed hook, and everything else is subordinate to that.” This means that all or most relation hooks, the config-changed hook, and the start hook should basically be the same.

Below, we’ve decorated the main hook method will most of our relation hooks, start, and config-changed. The work this does is fairly straight forward. It fetches the source and updates to the specified version if necessary, writes the Warren config file, writes the upstart file, opens or closes ports as necessary, and restarts the service.

@hooks.hook('start')
@hooks.hook('config-changed')
@hooks.hook('mongodb-relation-joined')
@hooks.hook('mongodb-relation-departed')
@hooks.hook('mongodb-relation-broken')
@hooks.hook('mongodb-relation-changed')
@hooks.hook('elasticsearch-relation-joined')
@hooks.hook('elasticsearch-relation-departed')
@hooks.hook('elasticsearch-relation-broken')
@hooks.hook('elasticsearch-relation-changed')
def main_hook():
    '''Main hook functionality
    On most hooks, we simply need to write config files, work with hooks, and
    restart.  If the source has changed, we'll additionally need to rebuild.
    '''
    if config.changed('source'):
        log('Source changed; rebuilding...')
        install_from_source()
    write_init_file()
    write_config_file()
    manage_ports()
    restart()

In our case, the haproxy hooks take a little bit more work, however. The haproxy service requires a bit of information from us: the hostname for this unit of the Warren service, and the port on which it is listening. For each website relation on this service, we simply send (using relation_set) those data to the remote service.

@hooks.hook('website-relation-joined')
@hooks.hook('website-relation-departed')
@hooks.hook('website-relation-broken')
@hooks.hook('website-relation-changed')
def website_relation_hook():
    '''Notify all website relations of our address and port.'''
    for relation_id in relations.get('website', {}).keys():
        private_address = hookenv.unit_private_ip()
        hookenv.relation_set(
            relation_id=relation_id,
            relation_settings={'hostname': private_address, 'port': config['listen_port']})

How are the hooks run? Simple. When the hooks.py file is called, we pass all the work on to the charmhelpers library, which will decide which decorated hook methods to call:

if __name__ == "__main__":
    hooks.execute(sys.argv)

The Good

There’s just so much to be said for having a repeatable, debuggable (I’ll get into juju debug-hooks at some point, promise!) means of deploying a service. With this layout for a charm, it’s easy to see what hook does what, and is fairly easy to organize your code around that. The configuration files are in a familiar and readable format (I’m looking at you, countless *.pom files), and the python charmhelpers package keeps our hooks fairly simple.

The Bad

I’ll be totally honest and say that a lot of the work that I did on this charm came from observing the ways other charms were built, not by reading documentation. I don’t mean to harp on this, but I simply had no other path forward for creating my charm, there wasn’t much to read. Again, this is something I’ll be focusing on helping along, myself.

My other problems stem from the issues involved with this path forward and may be mitigated by utilizing the new services framework.

The hooks.py file is big, but there are enough hooks and enough code repetition that it wouldn’t necessarily make sense to have it any other way. There are a few other charms that have gotten big enough to divide the deployment strategies into several different files and classes (notably the Juju GUI charm) in sensible ways. In the case of Warren, though there weren’t obvious break points, and yet the file still feels relatively long.

What’s next

In the next post, I’d like to go more in depth on the process of developing a charm. That means going into debug-hooks, juju ssh, and a few other commands that are useful for developing and debugging a charm.