Principal Data Platform Engineer

Principal Engineer – Data Platform

Location: Remote (UK-based; Security Clearance Required)

A UK Government Security Check (SC) clearance is required for this role. If you don’t currently hold SC clearance, we can support you in applying, provided you have lived and worked in the UK for a minimum of 5 years.

As a Principal Data Platform Engineer, you will define and drive the technical vision, architecture, and evolution of data platforms for a government client. You will operate at a systems and organisational level, shaping how data is ingested, processed, governed, and consumed across multiple teams and services.

This role goes beyond hands-on engineering — it requires deep technical authority, cross-team influence, and strategic decision-making to ensure platforms are secure, scalable, resilient, and aligned to long-term business outcomes.

Key Responsibilities

• Define and own the technical strategy for data platform architecture, balancing short-term delivery with long-term scalability and maintainability.
• Lead the design of complex, distributed data systems, including batch and real-time processing at scale.
• Establish platform-wide standards, patterns, and best practices for data engineering, reliability, and security.
• Act as a technical authority and mentor, guiding senior and lead engineers across teams.
• Drive architecture decisions across multiple squads, ensuring alignment with enterprise and government standards.
• Partner with architects, product leaders, and stakeholders to translate ambiguous requirements into scalable technical solutions.
• Ensure high availability, fault tolerance, and low-latency performance across critical data services.
• Champion engineering excellence, including TDD, BDD, observability, and continuous delivery practices.
• Oversee data modelling standards and governance, ensuring consistency, lineage, and integrity across systems.
• Lead cross-team incident analysis and root cause resolution, driving systemic improvements.
• Evaluate and introduce new technologies, tools, and frameworks to evolve platform capabilities.
• Provide technical leadership in client engagements, influencing decisions and managing senior stakeholder expectations.
• Ensure operational excellence, including resilience planning, cost optimisation, and platform sustainability.

Required Experience & Competencies

Non-Technical Competencies

• Operating effectively in high ambiguity and incomplete problem spaces
• Influencing senior stakeholders and cross-functional teams
• Navigating complex client environments and competing priorities
• Leading through technical authority rather than hierarchy
• Driving alignment across multiple teams and domains
• Continuously evaluating and adopting emerging technologies strategically

Technical Competencies

• Extensive experience designing and building large-scale, distributed data platforms
• Deep expertise in Java-based systems and modern backend architectures
• Strong experience with streaming and event-driven architectures (e.g., Kafka, Kinesis)
• Experience with large-scale data processing frameworks (e.g., Spark, Flink, Kafka Streams)
• Expertise in cloud-native architectures, ideally on AWS (including Lambda, S3, managed services)
• Strong understanding of data pipeline design, orchestration, and optimisation
• Experience with data modelling at scale, including schema design and evolution
• Expertise in relational and NoSQL database design and performance tuning
• Proven track record of implementing CI/CD, observability, and platform reliability practices
• Strong experience with DevOps and Infrastructure as Code (IaC)
• Ability to define and enforce engineering standards across teams

Desirable Skills

• Advanced expertise in data modelling strategies and trade-offs across systems
• Experience with search and indexing platforms (Elasticsearch, OpenSearch, Solr)
• Strong knowledge of CI/CD ecosystems (e.g., Jenkins, GitLab CI, Drone)
• Experience with infrastructure automation tools (Terraform, Ansible, Packer)
• Exposure to data governance, security, and compliance frameworks in regulated environments

Qualifications

• Degree in Computer Science or a related field (or equivalent experience)
• Typically 10+ years of experience in software/data engineering, with significant time spent in architecture or principal-level roles