Softwire Blog

Speed Coding 2016 – Q4 Solution

3 August 2016, by

This post explains the solutions to Question 4 of our speed coding 2016 competition. If you haven’t yet read the question, head back and have a go now.

(more…)

Speed Coding 2016 – Q3 Solution

27 July 2016, by

This post explains the solutions to Question 3 of our speed coding 2016 competition. If you haven’t yet read the question, head back and have a go now.

(more…)

Speed Coding 2016 – Q2 Solution

20 July 2016, by

This post explains the solutions to Question 2 of our speed coding 2016 competition. If you haven’t yet read the question, head back and have a go now.

(more…)

Speed Coding 2016 – Q1 Solution

13 July 2016, by

This post explains the solutions to Question 1 of our speed coding 2016 competition. If you haven’t yet read the question, head back and have a go now.

(more…)

Speed Coding 2016 – Q3

20 April 2016, by

Here’s the third of our speed coding questions.

(more…)

Speed Coding 2016 – Q2

13 April 2016, by

Here’s question 2 from our recent speed coding competition. See how quickly you can solve it.

(more…)

Speed Coding 2016 – Q1

6 April 2016, by

Here’s the first question from our speed coding competition 2016. We’ve already held the event, so there are no prizes, but if you want to play along at home, see how quickly you can solve the challenge.

(more…)

Improving Open-Source Deployment with Docker

6 April 2016, by

Open-source software has a lot going for it, but easy of use is not typically at the top of the list. Fortunately, it’s rarely a problem; as developers much of the open-source code we use is in simple tools and libraries, and most of the core interactions we have these are managed by package managers, which have focused on building a convenient usable layer to manage this for any project.

That’s not the case for other domains though, especially non-trivial standalone applications. There are a lot of popular open-source tools that follow this model, and require you to install and run them in an environment providing all their core dependencies. WordPress (which runs this blog) is a good example, along with apps like Discourse (a forum we use for internal discussion). These provide great value and they’re great tools, but setup isn’t easy, often involves many manual steps following sometimes painful documentation, and typically then fails because of some inexplicable idiosyncrasy of the server you’re using.

Staytus is another good example of this. Staytus is an open-source web application that provides a status site for your product, aiming to be a beautiful, usable, easy to manage tool that companies can drop into place to give their customers information on how their system is doing. Take a look at their demo site to see it in action.

Staytus, like many other tools in this domain, isn’t effortless to set up though. You have to install the right version of Ruby and all their ruby dependencies (still an annoyingly fiddly process on Windows especially, if you’re not already using Ruby elsewhere), install Node, install, configure and prepare a MySQL server, configure Staytus to glue this all together, and then hook the Staytus startup commands into whatever service running tool you want to use. None of this is cripplingly difficult, but it’s all friction that gets in the way of putting Staytus into users’ hands.

I found Staytus recently, while looking out for exciting new open-source projects, and decided this needed fixing. I wanted to try it out, but all the above was hassle. It would be better if you could run a single command, have this all done for you, and get a server up straight away. I’d also been hankering to have a closer look at Docker, which felt like a great fit for this, so I dived in.

The steps

So, we want to provide a single command you can run which downloads, installs, configures and starts a working Staytus server. To do that, we need a few things:

  • A working Ruby 2.1 environment
  • All the required Ruby dependencies
  • Node.js (for Rails’s JS asset uglification)
  • A configured MySQL server
  • Staytus configuration files, including the MySQL server details
  • A startup script that prepares the database, and starts the service

Automating this with Docker

Docker lets us define an immutable machine image, and provides extremely fast and convenient mechanisms to share, update, and use these images.

In practice you can treat this like an incredibly good virtual machine management system. In reality under the hood the details are quite different – it is providing isolated containers for systems, but through process isolation within a single operating system, rather than totally independent machines – but while those differences power the benefits, they doesn’t really need to affect how you think about the basics of using Docker in practice.

I’m not going to go into the details of Docker in great depth here, I’m just going to look at an example real-world use, at a high-level. If you’re really interested, take a look at their introductory video, or read through their excellent documentation.

What we need to do is define a recipe for an image of a machine that is configured and ready to run Staytus, following the steps above. We can then build this into an actual runnable machine image, and hopefully then just start it to immediately have that machine ping into existence.

Dockerfile

To start with we need the recipe for such a machine. That recipe (the Dockerfile), and the startup script it needs (a simple bash script) are below.

It’s important to note that while this is a very effective & working approach, there are parts of this that are more practical than they are Docker Best Practice. We’ll talk about that later (see ‘Caveats’).

# Start from the standard pre-prepared Ruby image
FROM ruby
MAINTAINER Tim Perry <[email protected]>

USER root

# Run all these commands inside the image
RUN apt-get update && \
    export DEBIAN_FRONTEND=noninteractive && \
    # Set MySQL password to temp-pw - reset to random password later
    echo mysql-server mysql-server/root_password password temp-pw \
      | debconf-set-selections && \
    echo mysql-server mysql-server/root_password_again password temp-pw \
      | debconf-set-selections && \   
    # Install MySQL for data, node as the JS engine for uglifier
    apt-get install -y mysql-server nodejs
    
# Copy the current directory (the Staytus codebase) into the image
COPY . /opt/staytus

# Inside that directory in the image, install our dependencies
RUN cd /opt/staytus && \
    bundle install --deployment --without development:test

# When you run this image, run docker-start.sh
ENTRYPOINT /opt/staytus/docker-start.sh

# Persist the MySQL DB to an external volume
# This means it can be independent of the life of the container
VOLUME /var/lib/mysql

# Persist copies of other relevant files (config, custom themes).
# Contents of this are copied to the relevant places when the container starts
VOLUME /opt/staytus/persisted

EXPOSE 5000

With this saved as Dockerfile inside the root of the Staytus codebase, we can then run docker build . to build (or rebuild) an image following this locally.

An interesting consideration when writing these Dockerfiles is image invalidation. Docker builds intermediate images for each command here, and rebuilding an image only reruns the steps that have been invalidated, using as many from its cache as possible. That means that by writing the Dockerfile as above rebuilding a new image with changes to the Staytus codebase is very cheap; the Ruby, Node and MySQL installation and setup phases are all cached, and we just take that image, copy the new code in, and pull down the dependencies the current codebase specifies. We only rerun the parts from COPY . /opt/staytus down. Small tweaks like this make iterating on your Docker image much easier.

Take a look at this article about working with the Docker build cache if you’re interested in this (and don’t forget to look at Docker’s best practices guide generally)

docker-start.sh

That Dockerfile installs everything required, copies the codebase into the image, and tells Docker to run the ‘docker-start.sh’ script when the image is started as a container.

To actually use this, we need a docker-start.sh script, to manage service startup process. That full content of that is below.

Note that this script includes some further database setup that could have been done above, at image definition time. That’s done here instead, to ensure the DB password is randomized for each container not baked into the published image, so we don’t end up with Staytus images all over the internet running databases with identical default passwords. Docker doesn’t obviate the need for good security practices!

#!/bin/bash
/etc/init.d/mysql start # Start MySQL as a background service

cd /opt/staytus

# Configure DB with random password, if not already configured
if [ ! -f /opt/staytus/persisted/config/database.yml ]; then
  export RANDOM_PASSWORD=`openssl rand -base64 32`

  mysqladmin -u root -ptemp-pw password $RANDOM_PASSWORD
  echo "CREATE DATABASE staytus CHARSET utf8 COLLATE utf8_unicode_ci" | mysql -u root -p$RANDOM_PASSWORD

  cp config/database.example.yml config/database.yml
  sed -i "s/username:.*/username: root/" config/database.yml
  sed -i "s|password:.*|password: $RANDOM_PASSWORD|" config/database.yml

  # Copy the config to persist it, and later copy back on each start, to persist this config 
  # without persisting all of /config (which is mostly app code)
  mkdir /opt/staytus/persisted/config
  cp config/database.yml /opt/staytus/persisted/config/database.yml

  # On the first run only, run the staytus:install task to setup the DB schema.
  bundle exec rake staytus:build staytus:install
else
  # If it's not the first run:

  # Use the previously saved config from the persisted volume
  cp /opt/staytus/persisted/config/database.yml config/database.yml

  # The DB should already be configured. Check if there are any migrations to run though:
  bundle exec rake staytus:build staytus:upgrade
fi

# Start the Staytus service
bundle exec foreman start

Putting this to use

With this written, you can check out the Staytus codebase, run docker build . to build an image of Staytus, and run docker run -d -p 0.0.0.0:80:5000 [built-image-id] to instantly start a container with that image, listening locally on port 80.

For end users, that a lot easier than all the previous setup we had! There’s still a little more we can do though. Having done this, we can publish that image to Docker Hub, and users no longer need to check out the codebase at all.

The full setup now, from a blank slate, is:

  • Install Docker (a single standard platform-specific installer)
  • Run docker run -d -p 0.0.0.0:80:5000 --name=staytus adamcooke/staytus
  • Browse to http://localhost:80

(Note the ‘adamcooke/staytus‘ part; that’s the published image name)

This is drastically easier than following all the original steps by hand, and very hard to do wrong!

I wrote this all up, contributed this back to Staytus itself in July last year, Adam Cooke (the maintainer of Staytus) merged in and published the resulting image, and Staytus is now ready and available for quick easy use. Give it a go!

Caveats

Some is this is not exactly how things should be done in Docker land – there’s more than a few concessions to short-term practicality – but this does work very nicely, and provides exactly the benefits we’re looking for.

The key Docker rule that’s not being follow here is that we’ve put two processes (Staytus and its MySQL server) into a single image, to run as a single container. Instead, we should run two containers (a Staytus container, and a totally standard MySQL container) and link them together, typically using Docker Compose. At the time though Docker Compose wasn’t yet recommended for production use, and to this day moving to this model still makes it a little harder for users to get set up and running that it would be with the one image. There’s ongoing work to finish that up now though, and Staytus is likely to evolve further in that direction soon.

Speed Coding 2016 Intro

30 March 2016, by

In case you didn’t hear through our twitter or facebook accounts, we recently held our 2016 speed coding competition. Questions were devised by the last competition’s runner-up Rupert Wood, with help from John Ginger.

The format of the evening was:

Question 1 – 20 minutes

Question 2 – 30 minutes

Question 3 – 40 minutes

Pizza – 30 minutes

Question 4 – 1 hour

In the upcoming series of posts, we’ll be releasing the questions, quickest answers as well as some hints and tips on interesting techniques that people took in their solutions.

Your Build Smells: Nervous Pushes

6 January 2016, by

This post is the second post in a series looking at Build Smells – signs of major issues in team build processes – and the things we can do to try and solve them. You might also be interested in the first post on painful merges, and do watch this space or follow us on twitter to see more of these as we publish them.

Nervous Pushes

Actually pushing code frequently is important for lots of reasons. As we covered in our previous post, sharing code earlier with your team can significantly reduce the time you spend fighting your version control systems, but even outside that, pushing code brings you closer to actually releasing useful code, and generally getting anything useful done.

Anything that stops you pushing code is something that’s slowing and hurting your team, but unfortunately it’s easy to get into a situation where your build does exactly that. If pushing code normally means something breaks then not only do you lose the time required to investigate and fix the issue (if there even is one), but you also lose the focus and concentration you had up until that point, as you have to context switch out of whatever new task you’d picked up in the meantime back to the fallout from your last one.

All this time and effort is something we humans instinctively avoid, and very quickly you’ll find if pushing code creates hassle, then people will push code less often. At this point, your build is making your team worse at development. This is a build smell.

Confidence

All of this fundamentally undermines your confidence in your build. If you have confidence in your build then you can write code, quickly find out whether it works (and quickly deal with it if it doesn’t), and iterate at high speed to code that actually solves your problem. Confidence lets you go fast, focusing on each task in hand sequentially and quickly moving toward success there, rather than having to keep one distracted eye nervously watching everything else to see what’ll break next.

This is one of the underlying components of flow, the much-discussed feeling of complete focus and being ‘in the zone’ that lets you zoom through development tasks with speed and control. This only works if you can be quickly confident in whether your code works as you write it; you need to be able to keep moving forwards without distractions and uncertainty, have your mind full with only what you’re building, and concentrate on your goals. Without confidence in your build, that’s not possible.

When confidence really falls apart you end up assuming the build is wrong more than it’s right and this gets much worse; you lose more than just the required time to fix issues, but you normalize build failures, start to ignore them, and lose the ability to deal effectively with actual failures when they come up. Once you start automatically just rerunning the build a couple of times when it fails then fixing the build starts to take even longer still, this problem gets even worse, and the whole situation quickly spirals.

If we can set up our builds to give us confidence instead, then we can build better software, faster (and enjoy ourselves more). If your build doesn’t give you this, then you have a problem to fix.

How can we fix this?

This all revolves around the core quality-checking steps of your build, which I mean to include anything that could really fail because of potentially incorrect code changes: automated tests, linting and style checks, and even compilation phases.

To get confidence your build, these steps need to give you accuracy, thoroughness, speed and simplicity.

Accuracy

If the code actually works, the build should pass.

The easiest way to totally undermine confidence is to have a build that fails half the time. As soon as this becomes normal, it becomes very difficult to quickly or easily get confidence in the code you write, and your development speed (and quality) drops significantly. There’s a few things that we’ve found particularly effective for helping here:

  • Reduce test dependencies – The most common cause of inaccuracy is intermittent issues with something outside your codebase, be that external systems, time and timezones, or the current operating system. Dependencies on any of these will bite you eventually, and working to remove or limit those really helps. Setup and teardown dependencies within tests, and take care about the assumptions you make about your environment.
  • Keep environments identical – Builds that fail only on the build server are hard to trust, frustrating, and difficult to fix. Try and keep environments consistent to avoid that, at the very least make it possible to create an environment locally that’s identical to CI, for debugging. There’s lots of great tools nowadays to support this; Docker and Vagrant are both good options, for example.
  • Have zero tolerance for inaccuracy – Lack of accuracy will snowball; once you’ve got one test that randomly fails more will join it without you noticing, and the frustrations of this make it difficult for people to invest in test quality generally, making it worse. You have to disproportionately fight to beat these issues when they happen, or they’ll get far worse.

Thoroughness

If the build passes, the code should actually work.

Passing tests need to be something that gives you good confidence in the quality of the codebase, so you’ve got passing tests you can move forwards and focus on the next problem, with nagging doubts that you’ve broken something and not noticed.

  • Monitor Coverage (a bit) – Code coverage is an oft-rejected metric, and strict adherence to arbitrary goals can be harmful, but overall coverage trends and patterns reliably contain useful data. It’s very useful to know that coverage is going down recently, or that one area is much better covered than another. Measure (and examine) these patterns to get the real value, and keep your team focused on thoroughness.
  • Ensure every potential risk is covered – The key things here is to not skip testing things because it’s hard. Sometimes you can’t usefully unit test something, but that doesn’t mean you should skip testing it. Add thorough integration tests instead, and ensure every potential risk gets caught somewhere.
  • Test what might fail – Overtesting can bite you too, increasing maintenance cost, and sapping morale and team investment in good testing. If there are no reasons you can think of why a test could fail, it’s not useful (even Kent Beck agrees), and you should potentially move the test up to a higher level. Instead of writing tests for every simple getter, test them as part of the code using them.

Speed

Builds should run fast enough to let you quickly know their result.

Speed powers confidence, allowing you to quickly check how you’re doing and iterate. Time spent waiting for builds is typically unproductive time, but also creates painful context switching: If you get pulled back to your previous change three hours later when the build fails then you’re no longer in the best place to fix it, and you’ve lost any focus on what you moved onto next too.

  • Quick feedback – Effective team flow and the ability to move quickly depends on a quick feedback cycle. Waiting 4 hours to know whether your change worked makes it very hard to quickly iterate on changes. Anything you can do to reduce the time to feedback makes this much easier, and helps you and your team develop faster and better. Tools like Wallaby (for JS) and NCrunch (for .Net) are fantastic for this.
  • Watchable unit tests – Unit tests that you are prepared to look at while they run (so 10 seconds max, unless they’re somehow fantastic to watch) are tests that you’ll be much happier to run all the time. Longer, and your mind will wander – taking your focus with it – and resulting in you running the tests far less. Fight to keep unit tests running fast.
  • Smoke test suites – You won’t manage to get integration or system test suites down to this time. Once you’ve got a 4 hour system test suite though, you can be sure that nobody on the team is running it before every commit. You can make that easier by picking a very small representative set of these to use as smoke tests, which are quick enough that people can run them as they work. You can also set these up as a early build step in CI, to ensure major problems are caught quickly in CI (and to avoid waiting 4 hours to discover that 100% of your tests failed).

Simplicity

Builds should be easy enough to check locally and fix that people actually do.

People are lazy, and that includes you, me, and everybody on your team. If running your build takes work, they’ll avoid it. Keeping the build simple and easy to use encourages people to actually go and use it, rather than avoiding it as an obstacle on their road to getting things done.

  • One click – If your build takes a series of actions, people will forget one, do them wrong, or skip to ‘only the important ones’. It’s typically easier than you’d expect to get it down to a single click (or simple command line step), and providing that kind of simple interfaces makes it drastically more likely than you’d expect that people will use it.
  • Zero setup – Manual setup wastes time, and you’ll do it wrong anyway. It also You’ll also have big problems when the setup changes, you have to migrate users from setup A to setup B, and you start finding tests that only pass in one setup. Again, this is often easier than you’d expect (especially with tools like Docker/Vagrant and Puppet/Ansible/Chef), and has big value.
  • Easy debugging – Eventually builds will fail, and making it easy to work out why will make people fix them more quickly and more effectively. Have Selenium take screenshots at the end of failing tests, make sure your build and CI setup provides detailed info on what exactly failed, have any logs from the system under test easily accessible, and ensure it’s easy to run individual tests standalone locally.

Put all this together, and you can put your project in a state where your tests are easy and quick to run, and reliably give you a good clear idea of whether your system works. All of this powers confidence, and makes it much easier to quickly and successfully build high-quality software every day.

Hopefully that’s an interesting insight into the kind of issues we see in projects, and some of our approaches to solving them!

We do encourage all our teams to find and play with their own solutions for problems they face, and we studiously avoid dogmatically sticking to any one approach, so I should note that these are just an example of a range of practices we’ve found effective in different cases. Don’t blindly follow advice from the internet just because it sounds convincing! Take a look at the concrete problems your team in facing, and consider whether some of the approaches and practices here might help you to effectively solve your real issues.

Let us know your thoughts all this below, or on twitter! In the next post, coming soon, we’ll take a look at the next smell in the series: Rare Releases.

Older posts »