Lead Data Engineer

We are seeking an experienced and hands-on Lead Data Engineer to design, build and evolve secure, scalable data platforms for clients across the UK Public Sector and Utilities sectors.

You will work closely with architects, engineers and client stakeholders to deliver modern cloud-based data solutions that support operational, analytical and AI-driven use cases. The role combines technical leadership with hands-on engineering, helping clients modernise their data estates while ensuring solutions are secure, reliable and scalable.

Working across multiple projects, you will lead the development of batch and real-time data pipelines, support architectural decision-making, and help establish engineering best practices across delivery teams.

Strong experience with real-time streaming technologies is essential.

Key Responsibilities

• Lead the design, development and deployment of secure, scalable data pipelines and data platform components.
• Design and implement batch and real-time streaming solutions using modern event-driven architectures.
• Own technical design decisions across projects, balancing scalability, security, performance and delivery requirements.
• Lead technical workshops, discovery sessions and architecture discussions with client stakeholders.
• Assess existing client data platforms and design practical approaches for modernisation, migration and integration.
• Build and maintain data ingestion, transformation and integration capabilities across a variety of data sources.
• Develop cloud-native data platforms using AWS services and open-source technologies.
• Ensure data quality, governance, lineage and security requirements are embedded throughout the data lifecycle.
• Drive engineering best practices including code reviews, automated testing, CI/CD and Infrastructure as Code.
• Support multiple client engagements, helping delivery teams maintain consistent engineering quality and standards.
• Mentor and support engineers across project teams, providing technical guidance and leadership.
• Perform data modelling, performance tuning and root cause analysis to resolve complex data and platform issues.

Essential Skills & Experience

Real-Time Streaming & Data Engineering

• Strong commercial experience designing and implementing real-time streaming data platforms.
• Hands-on experience with technologies such as Apache Kafka, AWS Kinesis, Apache Flink, Spark Streaming, Apache Pulsar, or similar event-driven technologies.
• Experience building high-volume, low-latency, resilient data pipelines in production environments.
• Strong understanding of modern data engineering patterns, including batch, streaming and hybrid architectures.

Data Engineering & Databases

• Extensive experience developing and operating enterprise-scale data platforms.
• Strong SQL skills and experience designing data models for analytical and operational workloads.
• Proficiency in Python or other modern programming languages used for data engineering.
• Experience configuring, tuning and optimising relational and NoSQL databases.

Cloud & Software Engineering

• Strong experience working with AWS cloud services, including EC2, RDS, Lambda, Athena, Glue, S3 and IAM.
• Experience working within Agile delivery environments.
• Strong understanding of Test-Driven Development (TDD) and automated testing practices.
• Experience implementing CI/CD pipelines and engineering automation.
• Strong understanding of Infrastructure as Code principles.
• Experience using version control systems such as Git.

Regulated Environments

• Experience delivering technology solutions within UK Public Sector, Utilities, Energy, Water, Financial Services or other highly regulated industries.
• Understanding of security, governance and compliance considerations within regulated environments.

Desirable Skills

• Experience with Terraform.
• Experience with Docker and Kubernetes.
• Knowledge of Ansible and Packer.
• Linux and Windows administration experience.
• Experience supporting AI, machine learning or advanced analytics platforms.
• Experience working on programmes requiring UK Government security clearance.

What We're Looking For

• Strong technical leadership and problem-solving skills.
• Ability to communicate effectively with both technical and non-technical stakeholders.
• Experience working in complex, regulated environments.
• A collaborative approach and willingness to support and mentor others.
• A focus on delivering practical solutions that create value for clients.
• A passion for building reliable, scalable and secure data platforms.

Education

Bachelor’s degree in computer science, Software Engineering, Data Engineering, Mathematics or a related technical discipline, or equivalent practical experience.