Lead Data Platform Engineer

Location

UK / Remote

Clearance

Depending on the client assignment, UK Government Security Check (SC) clearance may be required. Where SC clearance is required, candidates who do not currently hold clearance may be supported through the application process, subject to eligibility requirements, including having lived and worked in the UK for a minimum of 5 years.

The Role

We are looking for a Lead Data Platform Engineer to provide hands-on technical leadership in designing, building and evolving secure, scalable and reliable data platforms for clients across the public sector and utilities.

You will work across cloud-native and distributed data platforms, helping engineering teams build reliable platform capabilities that support batch, real-time streaming, analytical and operational workloads.

The role combines hands-on engineering with technical leadership. You will contribute to platform architecture and design decisions, establish engineering standards, solve complex technical challenges and work closely with architects, engineers and client stakeholders.

Strong experience with cloud platforms, distributed systems and data platform engineering is essential.

Key Responsibilities

• Lead the design, development and evolution of secure, scalable data platform services and components.
• Design and implement platform capabilities supporting batch, real-time streaming and hybrid workloads.
• Build and optimise distributed data services using technologies such as Kafka, Kinesis, Spark, Flink or equivalent.
• Contribute to platform architecture and technical design decisions, balancing scalability, security, performance and maintainability.
• Work closely with architects and engineers to establish reusable platform patterns, standards and engineering practices.
• Develop cloud-native data platform capabilities using AWS and/or Azure.
• Support high availability, fault tolerance, resilience and low-latency performance across platform services.
• Establish and maintain approaches to data modelling, schema management, data quality and data integrity.
• Drive engineering best practices including CI/CD, automated testing, TDD, observability and Infrastructure as Code.
• Support platform monitoring, troubleshooting, performance optimisation and root cause analysis.
• Contribute to resilience, disaster recovery and operational readiness activities.
• Work with DevOps and Platform Engineering teams to improve deployment and operational processes.
• Support technical discovery and solution design with client stakeholders.
• Provide technical guidance, mentoring and support to Senior Data and Platform Engineers.
• Identify technical risks, dependencies and opportunities for continuous improvement.

Essential Technical Skills & Experience

• Strong commercial experience in Data Platform Engineering, Data Engineering, Software Engineering or a similar technical discipline, with experience operating at Lead level.
• Experience designing and building cloud-native, distributed data platforms.
• Strong experience with Java and/or Python or equivalent modern programming languages.
• Experience with streaming and event-driven architectures, including Kafka, Kinesis or equivalent.
• Experience with large-scale data processing technologies such as Spark, Flink, Kafka Streams or equivalent.
• Strong experience working with AWS and/or Azure cloud environments.
• Experience with AWS services such as S3, Lambda, Glue, RDS, Athena, EC2 and IAM is desirable.
• Strong understanding of data pipeline architecture, orchestration and optimisation.
• Experience with relational and NoSQL databases, including data modelling and performance tuning.
• Experience implementing CI/CD, automated testing, observability and platform reliability practices.
• Strong understanding of DevOps and Infrastructure as Code.
• Experience with GitHub/GitLab or equivalent source control platforms.
• Experience working in Agile and/or DevOps environments.
• Desirable: Terraform, Docker, Kubernetes, Ansible, Packer, Elasticsearch/OpenSearch, Jenkins, GitLab CI, Python, Bash or Linux.
• Experience working within public sector, utilities or other regulated environments is advantageous.

Technical Leadership

As a Lead Data Platform Engineer, you will provide technical leadership within delivery teams and platform engineering activities.

You will:

• Lead the technical delivery of data platform capabilities and services.
• Contribute to platform architecture, technical design and technology decisions.
• Establish reusable engineering patterns and promote consistent technical standards.
• Support and mentor Senior Data and Platform Engineers.
• Work closely with architects, developers, DevOps and client stakeholders.
• Identify technical debt, platform risks and opportunities for improvement.
• Promote secure, scalable, observable and maintainable engineering practices.
• Support teams in balancing technical quality with delivery timelines and client outcomes.
• Contribute to platform resilience, operational readiness and continuous improvement.

Experience & Qualifications

A Bachelor's degree or equivalent experience in Computer Science, Software Engineering, Data Engineering or a related technical discipline is desirable.

You should have 5+ years of relevant professional experience across Data Engineering, Software Engineering, Platform Engineering or a similar technical discipline.

For a Lead-level appointment, you should demonstrate experience taking ownership of complex platform engineering solutions, technical decisions and mentoring or guiding other engineers.

Security Clearance

Where required by the client assignment, UK Government Security Check (SC) clearance may be required.

Candidates who do not currently hold SC clearance may be supported through the clearance process, subject to eligibility requirements.

About You

You are a hands-on platform engineer who understands how to build reliable data infrastructure that works in production.

You enjoy solving complex problems across cloud, distributed systems, streaming, data platforms and infrastructure, and can translate architectural requirements into practical engineering solutions.

You are comfortable moving between hands-on development, technical design and stakeholder discussions, while helping other engineers improve their technical capability.

You care about reliability, scalability, security, observability and maintainability, not simply getting a platform into production.

Experience working within public sector, utilities or other complex and regulated environments would be highly advantageous.