Softwire Blog

Charity Breakfast Club

29 April 2016, by

Every month the wonderful Kitchen Team at Softwire London cook a different culinary themed breakfast to raise money for the charity Refuge. Refuge was founded in 1971 when it opened the first women’s refuge in the UK. It is now the biggest provider of refuges and services to support families in need. They provide vital support and facilities for families escaping domestic abuse. Please go to their website for lots more information including individual stories and monthly newsletter.

How it works:

The Kitchen Team (Helen, Dom and Massimo), decide on a menu and calculate the ingredients costs. We then add £5 to the price all of which is donated to charity. Softwire employees then sign up for breakfast and can top up their donations if they are feeling generous or think that the breakfast was deserving of an additional donation.

Last month’s #1 CBC inspiration came from Massimo and his love of Columbian cooking and raised an incredible £600! We cooked Perico – cherry tomatoes, spring onions and rice fried with eggs and served it with Arepa – homemade grilled corn breads.

This month’s #2 CBC is inspired by several Softwire employees who have requested American style breakfast pancakes and raised an even better; £700! We cooked over 50 breakfasts in total, which included homemade pancakes served with a choice of berries or crispy bacon and topped with maple syrup, fruit compote and yoghurt.

charit 1    charity breakfast (2)

Great job – thanks Kitchen Team!

Tips for managing technical people – How to both learn and deliver

25 April 2016, by

Galvanizing the geeksThe following is an excerpt from my new book, “Galvanizing the Geeks – Tips for Managing Technical People”. You can buy the full book on my website, here.

Being able to learn is a key part of being happy. It is also widely cited as a good reason to take a job – books such as the cheesily named The Start-up Of You by LinkedIn founder Reid Hoffman advise you to consider how much you will learn as a key criterion in deciding whether or not to take a job. My favourite maxim (probably because it involves a rhyme, as all good maxims do) is from Never Eat Alone by Keith Ferrazzi: ‘Learn in your twenties, earn in your thirties’.

(more…)

Tips for managing technical people – Embrace Failure

25 April 2016, by

Galvanizing the geeksThe following is an excerpt from my new book, “Galvanizing the Geeks – Tips for Managing Technical People”. You can buy the full book on my website, here.

Whatever we do, whether it’s software development, building cars or policing a city, we are all trying to do it well. We are all trying to succeed. We set our sights on the end point, and aim towards it. We do everything in our power to avoid failure. We carefully manage a list of risks so we can deal with the unexpected and stop it from throwing us off course.

Some organisations go further. You’ll have heard the expression ‘failure is not an option’. This seems like a reasonable way to go about business, surely?

(more…)

Speed Coding 2016 – Q3

20 April 2016, by

Here’s the third of our speed coding questions.

(more…)

Speed Coding 2016 – Q2

13 April 2016, by

Here’s question 2 from our recent speed coding competition. See how quickly you can solve it.

(more…)

Speed Coding 2016 – Q1

6 April 2016, by

Here’s the first question from our speed coding competition 2016. We’ve already held the event, so there are no prizes, but if you want to play along at home, see how quickly you can solve the challenge.

(more…)

Improving Open-Source Deployment with Docker

6 April 2016, by

Open-source software has a lot going for it, but easy of use is not typically at the top of the list. Fortunately, it’s rarely a problem; as developers much of the open-source code we use is in simple tools and libraries, and most of the core interactions we have these are managed by package managers, which have focused on building a convenient usable layer to manage this for any project.

That’s not the case for other domains though, especially non-trivial standalone applications. There are a lot of popular open-source tools that follow this model, and require you to install and run them in an environment providing all their core dependencies. WordPress (which runs this blog) is a good example, along with apps like Discourse (a forum we use for internal discussion). These provide great value and they’re great tools, but setup isn’t easy, often involves many manual steps following sometimes painful documentation, and typically then fails because of some inexplicable idiosyncrasy of the server you’re using.

Staytus is another good example of this. Staytus is an open-source web application that provides a status site for your product, aiming to be a beautiful, usable, easy to manage tool that companies can drop into place to give their customers information on how their system is doing. Take a look at their demo site to see it in action.

Staytus, like many other tools in this domain, isn’t effortless to set up though. You have to install the right version of Ruby and all their ruby dependencies (still an annoyingly fiddly process on Windows especially, if you’re not already using Ruby elsewhere), install Node, install, configure and prepare a MySQL server, configure Staytus to glue this all together, and then hook the Staytus startup commands into whatever service running tool you want to use. None of this is cripplingly difficult, but it’s all friction that gets in the way of putting Staytus into users’ hands.

I found Staytus recently, while looking out for exciting new open-source projects, and decided this needed fixing. I wanted to try it out, but all the above was hassle. It would be better if you could run a single command, have this all done for you, and get a server up straight away. I’d also been hankering to have a closer look at Docker, which felt like a great fit for this, so I dived in.

The steps

So, we want to provide a single command you can run which downloads, installs, configures and starts a working Staytus server. To do that, we need a few things:

  • A working Ruby 2.1 environment
  • All the required Ruby dependencies
  • Node.js (for Rails’s JS asset uglification)
  • A configured MySQL server
  • Staytus configuration files, including the MySQL server details
  • A startup script that prepares the database, and starts the service

Automating this with Docker

Docker lets us define an immutable machine image, and provides extremely fast and convenient mechanisms to share, update, and use these images.

In practice you can treat this like an incredibly good virtual machine management system. In reality under the hood the details are quite different – it is providing isolated containers for systems, but through process isolation within a single operating system, rather than totally independent machines – but while those differences power the benefits, they doesn’t really need to affect how you think about the basics of using Docker in practice.

I’m not going to go into the details of Docker in great depth here, I’m just going to look at an example real-world use, at a high-level. If you’re really interested, take a look at their introductory video, or read through their excellent documentation.

What we need to do is define a recipe for an image of a machine that is configured and ready to run Staytus, following the steps above. We can then build this into an actual runnable machine image, and hopefully then just start it to immediately have that machine ping into existence.

Dockerfile

To start with we need the recipe for such a machine. That recipe (the Dockerfile), and the startup script it needs (a simple bash script) are below.

It’s important to note that while this is a very effective & working approach, there are parts of this that are more practical than they are Docker Best Practice. We’ll talk about that later (see ‘Caveats’).

# Start from the standard pre-prepared Ruby image
FROM ruby
MAINTAINER Tim Perry <[email protected]>

USER root

# Run all these commands inside the image
RUN apt-get update && \
    export DEBIAN_FRONTEND=noninteractive && \
    # Set MySQL password to temp-pw - reset to random password later
    echo mysql-server mysql-server/root_password password temp-pw \
      | debconf-set-selections && \
    echo mysql-server mysql-server/root_password_again password temp-pw \
      | debconf-set-selections && \   
    # Install MySQL for data, node as the JS engine for uglifier
    apt-get install -y mysql-server nodejs
    
# Copy the current directory (the Staytus codebase) into the image
COPY . /opt/staytus

# Inside that directory in the image, install our dependencies
RUN cd /opt/staytus && \
    bundle install --deployment --without development:test

# When you run this image, run docker-start.sh
ENTRYPOINT /opt/staytus/docker-start.sh

# Persist the MySQL DB to an external volume
# This means it can be independent of the life of the container
VOLUME /var/lib/mysql

# Persist copies of other relevant files (config, custom themes).
# Contents of this are copied to the relevant places when the container starts
VOLUME /opt/staytus/persisted

EXPOSE 5000

With this saved as Dockerfile inside the root of the Staytus codebase, we can then run docker build . to build (or rebuild) an image following this locally.

An interesting consideration when writing these Dockerfiles is image invalidation. Docker builds intermediate images for each command here, and rebuilding an image only reruns the steps that have been invalidated, using as many from its cache as possible. That means that by writing the Dockerfile as above rebuilding a new image with changes to the Staytus codebase is very cheap; the Ruby, Node and MySQL installation and setup phases are all cached, and we just take that image, copy the new code in, and pull down the dependencies the current codebase specifies. We only rerun the parts from COPY . /opt/staytus down. Small tweaks like this make iterating on your Docker image much easier.

Take a look at this article about working with the Docker build cache if you’re interested in this (and don’t forget to look at Docker’s best practices guide generally)

docker-start.sh

That Dockerfile installs everything required, copies the codebase into the image, and tells Docker to run the ‘docker-start.sh’ script when the image is started as a container.

To actually use this, we need a docker-start.sh script, to manage service startup process. That full content of that is below.

Note that this script includes some further database setup that could have been done above, at image definition time. That’s done here instead, to ensure the DB password is randomized for each container not baked into the published image, so we don’t end up with Staytus images all over the internet running databases with identical default passwords. Docker doesn’t obviate the need for good security practices!

#!/bin/bash
/etc/init.d/mysql start # Start MySQL as a background service

cd /opt/staytus

# Configure DB with random password, if not already configured
if [ ! -f /opt/staytus/persisted/config/database.yml ]; then
  export RANDOM_PASSWORD=`openssl rand -base64 32`

  mysqladmin -u root -ptemp-pw password $RANDOM_PASSWORD
  echo "CREATE DATABASE staytus CHARSET utf8 COLLATE utf8_unicode_ci" | mysql -u root -p$RANDOM_PASSWORD

  cp config/database.example.yml config/database.yml
  sed -i "s/username:.*/username: root/" config/database.yml
  sed -i "s|password:.*|password: $RANDOM_PASSWORD|" config/database.yml

  # Copy the config to persist it, and later copy back on each start, to persist this config 
  # without persisting all of /config (which is mostly app code)
  mkdir /opt/staytus/persisted/config
  cp config/database.yml /opt/staytus/persisted/config/database.yml

  # On the first run only, run the staytus:install task to setup the DB schema.
  bundle exec rake staytus:build staytus:install
else
  # If it's not the first run:

  # Use the previously saved config from the persisted volume
  cp /opt/staytus/persisted/config/database.yml config/database.yml

  # The DB should already be configured. Check if there are any migrations to run though:
  bundle exec rake staytus:build staytus:upgrade
fi

# Start the Staytus service
bundle exec foreman start

Putting this to use

With this written, you can check out the Staytus codebase, run docker build . to build an image of Staytus, and run docker run -d -p 0.0.0.0:80:5000 [built-image-id] to instantly start a container with that image, listening locally on port 80.

For end users, that a lot easier than all the previous setup we had! There’s still a little more we can do though. Having done this, we can publish that image to Docker Hub, and users no longer need to check out the codebase at all.

The full setup now, from a blank slate, is:

  • Install Docker (a single standard platform-specific installer)
  • Run docker run -d -p 0.0.0.0:80:5000 --name=staytus adamcooke/staytus
  • Browse to http://localhost:80

(Note the ‘adamcooke/staytus‘ part; that’s the published image name)

This is drastically easier than following all the original steps by hand, and very hard to do wrong!

I wrote this all up, contributed this back to Staytus itself in July last year, Adam Cooke (the maintainer of Staytus) merged in and published the resulting image, and Staytus is now ready and available for quick easy use. Give it a go!

Caveats

Some is this is not exactly how things should be done in Docker land – there’s more than a few concessions to short-term practicality – but this does work very nicely, and provides exactly the benefits we’re looking for.

The key Docker rule that’s not being follow here is that we’ve put two processes (Staytus and its MySQL server) into a single image, to run as a single container. Instead, we should run two containers (a Staytus container, and a totally standard MySQL container) and link them together, typically using Docker Compose. At the time though Docker Compose wasn’t yet recommended for production use, and to this day moving to this model still makes it a little harder for users to get set up and running that it would be with the one image. There’s ongoing work to finish that up now though, and Staytus is likely to evolve further in that direction soon.