Why enterprises can be disrupted
25 May 2017, by Andy Patterson
Software engineering is the foundation that allows new consumer services, efficient back-office systems, and market-disrupting products to be created. As with all foundations, it’s important to get the basic principles correct, and we believe that these principals are not only technical but creative. There are so many possible implementations of software applications that an environment which stimulates creative and innovative solutions will deliver the best results.
If we start with the humble software engineer, we can look at what motivates them to do their job effectively and create software to make a difference. A large body of research exists (summarised neatly in Drive) to demonstrate that extrinsic motivators (reward and punishment) work well for simple rule-based tasks, but work very poorly for creative or complex tasks. For these types of task, we can measure the presence (or lack of) three types of intrinsic motivation:
- Autonomy: the urge and capability to direct our own work and make our own decisions
- Mastery: the desire to master our tools and improve our skills
- Purpose: the need to do something with meaning, and with clear direction
These motivators provide us with a language to discuss how to create an environment that is productive and efficient, producing brilliant software; conversely, it provides terms to explain why companies can have environments that are stagnant and inefficient, leading to software that no one wants to use.
We start with the hardest motivator: autonomy. It might seem confusing that autonomy is important – isn’t the job of a software engineer to implement something that the business has decided is necessary? Allowing engineers to dictate their own workloads is only a single example of autonomy (and, incidentally, one that Google and others have used to great effect). Indeed, most engineers are happy to admit that UX or business analysis are not their areas of expertise. However, autonomy is much deeper than this – it’s about giving engineer the freedom to decide how to implement what’s required (what processes or frameworks to use), and the responsibility for delivering working software to end users. Teams with high autonomy are more efficient, as they’re empowered to remove processes that slow development effort down, and free to adopt better tools or frameworks as they become available.
One of the reasons that agile practices (and by extension, practices like Continuous Delivery and DevOps) have been so successful is that they push autonomy down to the team level, focus on cross-functional teams, and encourage continuous change. This isn’t to say that agile methods such as scrum or kanban are the only methods for achieving autonomy: they’re just a particularly well-known and effective one.
For software engineers, mastery of a toolset, skill, or framework is immensely satisfying. However, Mastery is more than perfecting one skill: it’s heavily reliant on continuous improvement. It should be obvious that technology is a rapidly-changing industry, and that adopting new tools and skills as they become available is the key to success.
Mastery of technical improvements
By now, the analogy of technical debt is well known and addresses one aspect of mastery (the desire to improve the code being worked on). Long-running codebases with low technical debt are strongly correlated with products that are reliable, high-quality, and respond predictably to change requests – Google is an excellent example of a company that invests heavily in reducing debt. However, technical debt is an imperfect analogy as it is unquantifiable, and must be measured by its side effects: slow, unreliable delivery, and disenfranchised, unmotivated developers. Business that focus solely on putting features in front of customers win in the short term, but over the long term incur sufficient debt that delivering even simple features takes a huge effort.
Mastery of personal development
While developers will generally self-improve over time, this can be greatly accelerated with the right support and processes, including formal and informal training. These two approaches differ in what they aim to achieve, and personal development is normally most effective when they’re combined. Training courses and conferences aim to teach a specific skill, or give a breadth of knowledge about the industry. They also provide opportunities to network and engage with companies working on similar problems. By contrast, coaching and mentoring are much more individual, helping developers improve their strengths and mitigate weaknesses. These approaches can deliver huge benefits to teams and organisations:
- Ensuring that knowledge is spread across the team, removing single points of failure and improving consistency
- Improving skills and abilities
- Allowing individuals to take on more responsibility, resulting in more capable teams
The rate at which software development changes is phenomenal; the tools and technologies for most new projects have changed in the last 5 years, and are radically different to those 10 years ago. The NodeJs ecosystem is the canonical example; just over 5 years old, NodeJs and npm now have over 4 million active developers, and are used by thousands of companies.
Mastery deals less with each individual change, and is more concerned with the process of adapting to changes over time. It’s important to determine where best to invest in changing technology, but as we saw from autonomy, the correct place to make this decision is within an engineering team.
Given that most software engineering is product or service-focussed, providing a clear backlog and direction of work gives a purpose to both individual and team efforts. In addition, a frequent and well-defined release schedule provides a visible goal to focus on. A purpose can be highly effective when combined with the goals of Mastery; to not only deliver software, but to get better at the process of delivering software. Teams with a strong sense of purpose are more likely to succeed, and less likely to get distracted and solve unrelated problems.
Scaling and disruption
Autonomy, mastery, and purpose are useful tools as they allow us to examine teams (or individuals) separately, and suggest targeted improvements. Some teams may have a strong sense of purpose, but are held back by poor tooling, whereas other teams may be able to make wide-reaching decisions but lack a clear goal. The ability to address the needs of each team in isolation allows this approach to scale. There are obvious technical challenges with scaling development work across multiple teams (particularly communication overhead made famous by the Mythical Man Month), but teams that are motivated to succeed will overcome these problems for themselves.
Looking at the world through the lens of intrinsic motivation it’s possible to see why a startup can deliver a disruptive product, but an industry behemoth can struggle to update its flagship products and services. The startup’s developers have autonomy (freedom from constraining structure and processes), mastery (time and money to improve skills and tooling), and purpose (the vision of the startup). Large organisations can be disruptive – Apple, Amazon, and Google regularly deliver ground-breaking products – but to do so requires continual effort and focus.
 The perils of ignoring software development: http://www.mckinsey.com/industries/high-tech/our-insights/the-perils-of-ignoring-software-development
 Drive: the surprising truth about what motivates us: http://www.worldcat.org/title/drive-the-surprising-truth-about-what-motivates-us/oclc/311778265
 Google’s “20 percent time” in action: https://googleblog.blogspot.co.uk/2006/05/googles-20-percent-time-in-action.html
 Searching for Build Debt http://static.googleusercontent.com/media/research.google.com/en/us/pubs/archive/37755.pdf