In the previous part, we started to nail down what it is that we’re going to do to try and accomplish our task. We’re going to write a charm - a package that represents a way to deploy a piece of software repeatably to the cloud - which does all that’s needed to host a WSGI app. It will be able to talk to several different services and use any WSGI server. If you haven’t yet, check that post out first!
“Hey! I think I have deployment all figured out for Honeycomb!” I blurted out in the Guild chat, sharing the link to the previous post.
Blank stares, then finally, “You wrote about the furry website you’re building for work1..?”
Point. But hey, furries make the internet go2.
Anyhow, I digress. The goal of Honeycomb is not to be a furry website but a writing one, and besides, none of the guild really cared about deployment strategies; that’s something for me to worry about. And as a developer with a fascination with the Ops side of things, I do care about deployment strategies.
Chat aside, Juju is a delightful step in making things much easier for us DevOpsy critters, because it allows us to write general and repeatable solutions that can be used within a ecosystem of other solutions to accomplish a specific goal. In our case, we want to be able to deploy any WSGI compliant application that can connect to a variety of other services and be hosted in the cloud.
These solutions, charms, are what we’ll need to have to get our stack up and running, so let’s get started in making our
Bootstrapping the charm
As Juju is a Canonical offering (and I’m a Canonical employee), I’ll be working in Ubuntu, so if you’re following along, you’ll need that running on a machine or VM at your disposal (you’ll need 15.10 Wily or 16.04 Xenial, to be specific). Once you’ve got your system up and running, installation of juju and the tools we need is fairly easy:
sudo apt install juju lxd charm-tools
This will install just about everything we need: we’ll get juju itself, of course, as well as LXD, which is used for local and development environments, and
charm-tools, which contains a few applications we’ll be using for creating, building, and testing our charm.
Getting our charm set up after this point is fairly easy, but it does require deciding on how you’re going to work with your charm. I have a work directory in which I keep all of my projects, and when developing a charm, it’s often easiest to have all of your charm related work in its own directory structure. With that in mind, I’m going to set up the following directory structure for working with
~ └── work └── charms ├── builds ├── interfaces └── layers
We’ll get into what layers are in just a second and interfaces in a later article, but for now, that’s the basic structure that one needs to get started with building a charm. This is because the various charm tools expect to find a certain structure when they do their work. This is all controlled through environment variables:
export JUJU_REPOSITORY=$HOME/work/charms export LAYER_PATH=$JUJU_REPOSITORY/layers export INTERFACE_PATH=$JUJU_REPOSITORY/interfaces # or, in fish: set -x JUJU_REPOSITORY ~/work/charms set -x LAYER_PATH $JUJU_REPOSITORY/layers set -x INTERFACE_PATH $JUJU_REPOSITORY/interfaces # then: mkdir -p $LAYER_PATH $INTERFACE_PATH
Now that we’ve got our basic directory structure in place, we can start to actually build out our charm. Thankfully, the charm tools give us a way to do so with a simple command:
# First, get to the layer directory, where we'll be working on our charm makyo@corrin:~$ cd $LAYER_PATH makyo@corrin:~/work/charms/layers$ charm create wsgi-app INFO: Using default charm template (reactive-python). To select a different template, use the -t option. INFO: Generating charm for wsgi-app in ./wsgi-app INFO: No wsgi-app in apt cache; creating an empty charm instead. Cloning into '/tmp/tmp52ugnq'... remote: Counting objects: 27, done. remote: Total 27 (delta 0), reused 0 (delta 0), pack-reused 27 Unpacking objects: 100% (27/27), done. Checking connectivity... done.
charm create will do a few things, as shown above, but in short, it will create a
wsgi-app directory for us, and populate it with a basic template upon which we’ll build our charm. This winds up giving you a directory like so:
wsgi-app ├── config.yaml ├── icon.svg ├── layer.yaml ├── metadata.yaml ├── reactive │ └── wsgi_app.py ├── README.ex └── tests ├── 00-setup └── 10-deploy
There’s a lot going on here, so it’s time we take a step back and talk about what goes into building a charm, and the two important concepts to learn about: hooks and layers.
Charms, as deployment solutions, are fundamentally reactive. When you deploy a charm, it goes through several stages, and at each stage, the charm has a chance to react to what’s going on during the deployment process. The same with other things going on around the charm: when configuration values are changed, or a relation (a means of communication between two services) is added or removed or changed, the charm can react to that as well.
As with many other reactive frameworks, we call these sorts of reactions ‘hooks’. For example, when you first deploy a charm with juju, the charm receives several hooks from juju once the machine on which it is to be deployed is up and running. It will start with the
install hook, which is usually when the charm has a chance to install all of its software. After that, it will receive the
config-changed hook, which is when the charm will learn about all of the configuration options that the user has specified and can react accordingly. Finally, it will receive a
start hook, which is when any long-running processes such as servers will be started.
stop is another hook, as well, of course, wherein one might back-up any charm data.
The hook-based lifecycle of a charm looks something like the following3:
In reality, most of the work that is done within the charm happens during the
config-changed hook, while
install is used only for installing ancillary software and
start is often just ignored. This is because that is the hook which will always occur after startup and any configuration changes (such as when the user runs
juju set). This way, you can just restart all tasks during
config-changed, as this will cause config files changed during the hook to be reloaded and so on.
We’ll get more into relations in a later part, but for now, it’s enough to understand that relations are, primarily, ways in which one service can expose information to another. Contrary to their names or how they appear in visualizations, they’re not means or representations of communication between services. Rather, one can think of them as the juju controller allowing the two services to acknowledge that they exist to each other. In our instance, this will mean that, when a relation is created between
wsgi-app and PostGres,
wsgi-app will receive connection information for the PostGres database server, and networking will be restructured such that the two instances can talk to each other.
Relations follow their own cycle of hooks, through creation, change, and deletion, but again, we’ll be diving into them later - they’re a topic of their own!
Charms can be written in most any language that’s available on a base ubuntu server image, such as bash or python, or any language installed during the charm hook, which covers just about everything. This works by having a hook directory in the charm with an executable file named for each hook:
hooks ├── config-changed ├── hook.template ├── install ├── leader-elected ├── leader-settings-changed ├── nrpe-external-master-relation-broken ├── nrpe-external-master-relation-changed ├── nrpe-external-master-relation-departed ├── nrpe-external-master-relation-joined ├── start ├── stop ├── update-status ├── upgrade-charm ├── website-relation-broken ├── website-relation-changed ├── website-relation-departed └── website-relation-joined
Many charms are still written by hand in this fashion, as it’s quite easy to do. In most cases, all code lives in, for instance, a
hooks.py file with all hook files being symlinks to that. Pertinent functions are run through the use of
@hook decorators, such as
Although you can always write charms more directly like this, one winds up writing a lot of boilerplate code. If you’ve written charms before, you’ll be well versed in this, and if you haven’t, you may find yourself stealing from other charmers more often than not.
In the spirit of Don’t Repeat Yourself, there’s another way of writing charms: layers.
Layers are composable packages that allow one to include functionality in the final charm without having to write it yourself. They’re the libraries that the charm ecosystem really needed. As yet, they only exist for python charms, but as we’re charming up python applications, we should be good to go in using them!
So what can one do with layers? Well, quite a bit. Say one needs to install some software on installation - Honeycomb requires the use of
pandoc, for instance, which allows writers to upload many different file types - in which case one require the
apt layer, which will allow one to specify in one’s layer configuration what one expects to have installed on the machine by the time we get to running our service. Similarly, we may want to use a git layer which will fetch our website repository as specified in a configuration and use that to deploy our application.
Hooks respond to what is happening in the juju environment and are the basis of how charms work, while layers compose oft-repeated bits of functionality into a charm and are the basis of making it easier to conceptualize and compartmentalize different actions that a charm may make.
Sort of. For the most part. Ish.
Layers also introduce a bit of complexity in that they offer hook-like functionality through a state mechanism. For instance, an apache layer may have a state
apache.available that is set when Apache is up and running and ready to serve your page. That layer may call
set_state('apache.available'), which will mean that your charm will get notified through a state change. If you need to do something when Apache is made available, you can set up a function to do so based on a decorator:
You can think of it like so:
- Hooks are fired when something changes in Juju
- Layers compose repeatable functionality into libraries and set state, and
- State changes happen when something happens within the charm itself as one of the hooks is called.
There’s a lot going on here, and I’ve already thrown a lot of words at it, with a few more posts to go. I learn best by seeing how things are implemented, though, so let’s get onto the charm and see how this all fits together.
wsgi-app as a layer
With the steps outlined above, we created a
wsgi-app charm, right?
Well, not actually. We created a
wsgi-app layer, which can be built into a charm. What we need to do is start working on what the layer will do, so that when we run
charm build in a bit, we’ll wind up with a complete charm that serves up a WSGI application.
Here’s what we need
wsgi-app to do:
- Install some python stuff - we’ll need
pip, for instance, to grab our frameworks and dependencies
- Fetch our source repository and set some configuration values such as our application secret
- Expose relation endpoints for:
- a WSGI server such as
- a database such as PostGres or MySQL/MariaDB
- Apache for proxying and load balancing
- a WSGI server such as
Remember, though, that we’re going to save relations for a later step, so let’s just focus on the first two steps.
For installing stuff, our best bet is to use the
apt layer. Remember that layers act as libraries; in this case, we’re aiming to install the library that will let us install packages through apt.
This is a fairly simple thing to do, if perhaps a little magical: open
layer.yaml; you’ll see the boilerplate code, which looks like so:
includes: ['layer:basic'] # if you have any interfaces, add them here
Adding a layer is as simple as adding it to that list. Add
layer:apt so that you wind up with the following YAML file:
includes: - layer:basic - layer:apt
Good! That was easy! Of course, we can also tell the apt layer what to install here rather than doing so programmatically (such as
pip, perhaps even more down the line), so our
layer.yaml will look like the following:
includes: - layer:basic - layer:apt options: apt: packages: - python-pip
Now we can start testing things out, though lets stick with just the build step for now. When we run
charm build, our layers will be composed together and hopefully we’ll wind up with a built charm in our build dir!
makyo@corrin:~/work/charms/layers/wsgi-app$ charm build build: Composing into /home/makyo/work/charms build: Destination charm directory: /home/makyo/work/charms/trusty/wsgi-app fatal: Not a git repository (or any of the parent directories): .git bzr: ERROR: Not a branch: "/home/makyo/work/charms/layers/wsgi-app/". build: Please add a `repo` key to your layer.yaml, with a url from which your layer can be cloned. build: Processing layer: layer:basic build: Processing layer: layer:apt build: Processing layer: wsgi-app
Well that looks…promising! There’s a lot of green, at least. There’s a few warnings we can address, though. I hadn’t initialized the charm as a git repo yet, because it was created through
charm create, so let’s do that now:
makyo@corrin:~/work/charms/layers/wsgi-app$ git init . Initialized empty Git repository in /home/makyo/work/charms/layers/wsgi-app/.git/ makyo@corrin:~/work/charms/layers/wsgi-app$ git remote add origin email@example.com:makyo/wsgi-app.git makyo@corrin:~/work/charms/layers/wsgi-app$ git pull origin master remote: Counting objects: 5, done. remote: Compressing objects: 100% (4/4), done. remote: Total 5 (delta 0), reused 0 (delta 0), pack-reused 0 Unpacking objects: 100% (5/5), done. From github.com:makyo/wsgi-app * branch master -> FETCH_HEAD * [new branch] master -> origin/master makyo@corrin:~/work/charms/layers/wsgi-app$ mv README.ex README.md
And update our
layers.yaml to add the repo key as suggested:
includes: - layer:basic - layer:apt repo: https://github.com/makyo/wsgi-app.git options: apt: packages: - python-pip
Now, when we run
charm build, we should get a cleaner result:
makyo@corrin:~/work/charms/layers/wsgi-app$ charm build build: Composing into /home/makyo/work/charms build: Destination charm directory: /home/makyo/work/charms/trusty/wsgi-app build: Processing layer: layer:basic build: Processing layer: layer:apt build: Processing layer: wsgi-app
Perfect! We’re on a roll.
Our next step will be to clean up some of the code created by bootstrap. For starters, we can make
metadata.yaml agree with reality as best we can for now - we’ll leave the relations section as is until we get to that point - and to specify which series we want the charm to support (trusty and xenial, in our case):
name: wsgi-app summary: charm for running any WSGI application maintainer: Madison Scott-Clary <Madison.Scott-Clary@canonical.com> description: | WSGI applications such as Django or Flask applications are web applications written in python. This charm allows those applications to talk to a database, a web server, a search engine, and a cache. tags: - misc - cms - app-servers series: - trusty - xenial subordinate: false provides: provides-relation: interface: interface-name requires: requires-relation: interface: interface-name peers: peer-relation: interface: interface-name
We’ll also need to update
config.yaml for some configuration settings that
layer:apt requires, and to get rid of the placeholders. Our resulting file should look like this:
options: install_sources: type: string default: '' description: | Any additional sources (such as PPAs) from which to install packages install_keys: type: string default: '' description: | Any additional GPG keys required for sources added by install_sources extra_packages: type: string default: '' description: | Any additional packages to install on the charm's machine
Again, while we might come up with more configuration values in the future, this will do for now. Running
charm build again gives us another successful output, though it’s worth noting that, since we changed the charm from a single series to a multi-series charm, it is now built into
It’s getting late and that was a lot of information to dump, so I’ll pause here for now. We still have a ways to go - I’ve got at least three future posts lined up for this project - but we’re getting closer with each step. We’ve started work on our
wsgi-app layer, which, when built, produces a functional skeleton of the charm we eventually want.
Here are some resources for the time being:
- Source code for the
wsgi-applayer as it was at the point reached at the end of this article.
- Documentation on charm layers
- List of charm layers and interfaces
- Documentation on charm hooks
As always, views are my own, not my employer’s. ↩
Seriously. After students (54%), nearly 10% of furries are employed in the tech industry, making it the most common occupation within the subculture. (The Furry Poll, conducted March-December 2015, n=11831) ↩