Perspectives

What is data engineering and how can it help your business?

This article was updated on 12th January 2026.

What is data engineering?

Data engineering helps make data more useful and accessible for consumers of data. To do so, data engineering must source, transform and analyse data from each system.

Data engineering can be split into two halves: testing, and defining interfaces between data systems and source/ client applications. It’s about abstracting data and presenting it in relevant business terms that can be aggregated to derive additional value and insight.

The question is, how do we arrange business data in a way that makes it useful to analysts for machine learning and reporting? And can we use governance practices to make sure data storage adheres to software privacy policies?

Focusing on governance

Over 15 years ago, governance was not a consideration. Users now expect their data and privacy to be protected. An example of positive governance is traceability, which is the ability to be able to follow the data back to its original source.

Regulators are there to protect user data, but it does present huge engineering challenges.

If somebody questions a piece of data, being able to say where it comes from is key. We saw this in the handling of COVID data. For example, journalists or fact checkers will dig into numbers and find no source for the information.

Government Statistical departments are under a lot of pressure to produce figures in these scenarios. It’s important to be able to set up an internal system to capture information that can help trace back data to the source .

Importance of GDPR

GDPR, adopted by the European Parliament in April 2016, gave service or product users the right to have data removed or anonymised. However, if their data is lifted from 10 separate applications, removing or anonymising it can be very difficult — and difficult for data engineers to prove they have done it well.

Service requests and governed data engineering approaches make those challenges easier to overcome. Sensible change control processes make sure you’re adhering to regulations and any other privacy policies you may have, whilst also being able to distribute the data from your central data store.

Quality assurance

A common oversight and risk to having high quality and complete data is missing fields. This is where data isn’t present in final outputs for analytics and is difficult to retrofit.

Quality in this context lies in the accuracy, validity, completeness, and consistency of your data. By tracking data quality, a business can pinpoint potential issues harming quality and ensure that shared data is fit to be used for a given purpose.

This is particularly relevant in relational data models, where domain-specific language is designed for managing data held in relational databases, or for stream processing.

You can improve and maintain data quality by focusing on ‘quality assurance’ as part of your data governance — first, define Quality Assurance (QA) metrics and then perform regular QA audits. You can also appoint roles such as data owners, data stewards and data custodians within your organisation and establish proper processes to ensure high data quality.

Storage

The explosion in data engineering has been driven by advancements in storage — it’s faster, cheaper and is suddenly viable to collect and access in vast amounts.

Companies like Netflix, which have millions of customers, are collecting and analysing data that wouldn’t have been possible to store 10 years ago. The companies are solving problems from first principles and building strategies that are trickling into the wider development ecosystem.

Consider the three S model when looking at the ideal ways of storing your data:

Security
To maintain the robustness of your business data, you need to make sure there is a traceable source, it meets security criteria and that it’s accessible to the right people. You’ll need to protect confidential data from unauthorized in-office staff and external suppliers you work with.
Scalability
Every business wants to grow and you need your data infrastructure to be capable of doing so, whilst being cost-effective. It’s worth looking for a storage solution that includes the ability to scale up as your business grows.
Segregating domains
Well-segregated business domains have multiple silos for your data, helping to separate content into separate portals to set different access control for each type of content.

Learn more about how you can make the most of your data

“Data engineering is about thinking upfront about what you need to be able to do with data you have collected. Thinking about where it’s going to come from and how much of it there is lets you plan exciting things — be that machine learning, reporting or simply consuming data from web sites or other platforms.”
Jonathon Artus, Director, Client Portfolio

If you’re interested in learning about our data engineering services or the work we’ve carried out, we recommend starting with our collaboration with Google DeepMind for Moorfields Eye hospital to create an AI system that helps clinicians fast-track patients with serious eye diseases. Read the case study

Jonathon Artus

Director, Client Portfolio

02 September 2022

to our monthly newsletter for our latest expert content.

About the AuthorJonathon Artus

Jon looks after a broad portfolio of Softwire’s clients. He has a particular focus on the financial services sector, drawing on his extensive expertise to help shape and deliver technological solutions in areas from regulatory compliance through to digital transformation. Jon originally trained as an accountant at Deloitte and worked at UBS and KPMG before joining Softwire in 2014. He’s spent more than 15 years working with banks, insurers and other financial organisations to develop technical solutions to some of their biggest challenges. Over this time, Jon has worked as a technical lead, consultant and programme manager on everything from real-time trading platforms to complex regulatory reporting systems. He’s also overseen the implementation of automated software testing and DevOps strategies in large organisations. In his spare time, Jon plays the piano and organ and gets involved with the social side of Softwire, conducting our choir, and writing and acting in our annual pantomime. He also built an escape room in one of our meeting rooms, constructing a full multimedia experience with a sound and lighting rig, and specially built electronics, powered by a custom microservices framework written in Python.