Delivering the foundations of a cutting-edge data infrastructure for EMR Settlement Ltd
Rapid discovery, design, validation, and delivery of a key end-to-end data pipeline supporting the future of the UK low-carbon energy market
The challenge
EMR Settlement Ltd (EMRS) is a subsidiary of Elexon Ltd. It is the settlement services provider on behalf of the Low Carbon Contracts Company (LCCC).
EMR Settlement asked Softwire to recommend a suite of data engineering tools that would support their business as it moves into the cloud, and build the foundations of a platform to support their ambitious roadmap.
EMRS was concerned about the limitations of its siloed data architecture, which lacked flexibility to adapt to upcoming industry-wide changes and didn’t meet modern expectations around data analytics and governance.
Our solution
We worked with EMRS to design and build a cloud-native Data Lake and unified analytics platform as part of a wider Azure migration roadmap.
Our team of cloud and data experts worked closely with EMRS teams to understand BAU processes and pain-points with its current data architecture. We also talked to EMRS’s Design Team to understand their long-term cloud roadmap and ambitions around data analytics. After considering both short-term needs and long-term goals, we made detailed recommendations for migrating data infrastructure into Azure, including actionable first steps.
Next, we analysed data interfaces between existing applications and identified key technical barriers preventing analytics across multiple systems in real time. We presented recommendations on an Azure-native architecture to consolidate data exchange through Azure Data Lake Storage (Gen2) and enable live multi-system analytics via PowerBI.
We prepared options to facilitate an informed discussion about Azure data engineering, including approaches to data ingestion, pipeline orchestration, ETL tasks, and real-time event handling. We also evaluated cloud-native and open-source technologies thoroughly against EMRS’s specific needs.
After considering the options, we proposed a cloud data architecture to strike a perfect balance between cost, scale, and observability. Our proposed architecture made the most of off-the-shelf tooling while keeping the door open for future data integrations.
The result
We built a proof-of-concept solution in Azure, re-implementing industry reporting to successfully demonstrate the end-to-end feasibility of the technologies recommended.
Our solution established and de-risked design patterns that are now being re-used in EMRS’s ongoing cloud migration. EMRS now has the foundations of cutting-edge data infrastructure built around core data technologies:
- Azure Data Lake Storage (Gen2) with medallion architecture for data storage
- Apache Airflow for data flow orchestration and ETL
- Azure Databricks for data science and regulatory reporting
- PowerBI for data visualisation and dashboards
With our proof of concept, EMRS realised a secure data infrastructure that would enable application re-platforming to commence.
Connecting self-service reporting and analytics tools, Databricks and PowerBI, directly to the data lake has been a great benefit to EMRS, with access controlled via Azure RBAC. Connecting directly avoided data duplication, while still allowing authorised data scientists and analysts access to explore real-time data and the ability to build custom reports across multiple, previously separate datasets without the need for developer involvement.
“It was a positive experience working with the Softwire team. EMRS has found that Softwire is able to pick up the necessary domain knowledge quickly and efficiently, this has meant that EMRS feels that Softwire understands our vision and architecture landscape. Softwire has also been able to provide potential future solutions.
EMRS is excited to begin its architecture transformation journey with Softwire as a trusted and chosen Supplier”
Neha Sinha, EMRS Design Manager