Laying a leading edge data pipeline at the heart of the UK electricity market

Our team of cloud and data experts worked closely with EMRS teams to understand BAU processes and pain-points with its current data architecture. We also talked to EMRS’s Design Team to understand their long-term cloud roadmap and ambitions around data analytics. After considering both short-term needs and long-term goals, we made detailed recommendations for migrating data infrastructure into Azure, including actionable first steps.

We prepared options to facilitate an informed discussion about Azure data engineering, including approaches to data ingestion, pipeline orchestration, ETL tasks, and real-time event handling. We also evaluated cloud-native and open-source technologies thoroughly against EMRS’s specific needs.

After considering the options, we proposed a cloud data architecture to strike a perfect balance between cost, scale, and observability. Our proposed architecture made the most of off-the-shelf tooling while keeping the door open for future data integrations.

We built a proof-of-concept solution in Azure, re-implementing industry reporting to successfully demonstrate the end-to-end feasibility of the technologies recommended.

Our solution established and de-risked design patterns that are now being re-used in EMRS’s ongoing cloud migration. EMRS now has the foundations of cutting-edge data infrastructure built around core data technologies:

With our proof of concept, EMRS realised a secure data infrastructure that would enable application re-platforming to commence.

“It was a positive experience working with the Softwire team. EMRS has found that Softwire is able to pick up the necessary domain knowledge quickly and efficiently, this has meant that EMRS feels that Softwire understands our vision and architecture landscape. Softwire has also been able to provide potential future solutions. ”