Skip to content
Home>Our work>Delivering the foundations of a cutting-edge data infrastructure for EMR Settlement Ltd
EMR Settlement Limited

Delivering the foundations of a cutting-edge data infrastructure for EMR Settlement Ltd

Rapid discovery, design, validation, and delivery of a key end-to-end data pipeline supporting the future of the UK low-carbon energy market

The challenge

EMR Settlement Ltd (EMRS) is a subsidiary of Elexon Ltd. It is the settlement services provider on behalf of the Low Carbon Contracts Company (LCCC).

EMR Settlement asked Softwire to recommend a suite of data engineering tools that would support their business as it moves into the cloud, and build the foundations of a platform to support their ambitious roadmap.

EMRS was concerned about the limitations of its siloed data architecture, which lacked flexibility to adapt to upcoming industry-wide changes and didn’t meet modern expectations around data analytics and governance.

Our solution

We worked with EMRS to design and build a cloud-native Data Lake and unified analytics platform as part of a wider Azure migration roadmap.

Our team of cloud and data experts worked closely with EMRS teams to understand BAU processes and pain-points with its current data architecture. We also talked to EMRS’s Design Team to understand their long-term cloud roadmap and ambitions around data analytics. After considering both short-term needs and long-term goals, we made detailed recommendations for migrating data infrastructure into Azure, including actionable first steps.

Next, we analysed data interfaces between existing applications and identified key technical barriers preventing analytics across multiple systems in real time. We  presented recommendations on an Azure-native architecture to consolidate data exchange through Azure Data Lake Storage (Gen2) and enable live multi-system analytics via PowerBI.

We prepared options to facilitate an informed discussion about Azure data engineering, including approaches to data ingestion, pipeline orchestration, ETL tasks, and real-time event handling. We also evaluated cloud-native and open-source technologies thoroughly against EMRS’s specific needs.

After considering the options, we proposed a cloud data architecture to strike a perfect balance between cost, scale, and observability. Our proposed architecture made the most of off-the-shelf tooling while keeping the door open for future data integrations.

The result

We built a proof-of-concept solution in Azure, re-implementing industry reporting  to successfully demonstrate the end-to-end feasibility of the technologies recommended.

Our solution established and de-risked design patterns that are now being re-used in EMRS’s ongoing cloud migration. EMRS now has the foundations of cutting-edge data infrastructure built around core data technologies:

  • Azure Data Lake Storage (Gen2) with medallion architecture for data storage
  • Apache Airflow for data flow orchestration and ETL
  • Azure Databricks for data science and regulatory reporting
  • PowerBI for data visualisation and dashboards

With our proof of concept, EMRS realised a secure data infrastructure that would enable application re-platforming to commence.

Connecting self-service reporting and analytics tools, Databricks and PowerBI, directly to the data lake has been a great benefit to EMRS, with access controlled via Azure RBAC. Connecting directly avoided data duplication, while still allowing authorised data scientists and analysts access to explore real-time data and the ability to build custom reports across multiple, previously separate datasets without the need for developer involvement.

“It was a positive experience working with the Softwire team. EMRS has found that Softwire is able to pick up the necessary domain knowledge quickly and efficiently, this has meant that EMRS feels that Softwire understands our vision and architecture landscape. Softwire has also been able to provide potential future solutions

EMRS is excited to begin its architecture transformation journey with Softwire as a trusted and chosen Supplier”

Neha Sinha, EMRS Design Manager

EMR Settlement Limited

Azure Data Lake Storage (Gen2), Apache, Airflow, Azure Databricks, Azure Data Factory, PowerBI, Python, Azure Kubernetes Service, Helm, Azure Functions

The project
We rapidly assessed a suite of legacy systems and architectures, before designing and delivering a proof-of-concept suite of data tooling to support the re-platforming of EMRS legacy systems. This provided the information they needed to develop a cloud migration roadmap.

The results
We built a fully functioning end-to-end slice through one of EMRS primary data flows. Our eight-week proof of concept of the new data platform provided EMRS with real-world evidence of its suitability for rollout across the board, allowing them to push ahead with their ambitious roadmap and move their data workflows to the cloud.