Lead Data Engineer

Who we are

CreateFuture is fast becoming the UK’s most recognisable digital consultancy, with years of experience building digital products and services for major organisations whilst putting our people first. We have offices in the centre of Edinburgh, Leeds, Manchester, and London as well as remote employees located throughout the country. 

We are a team of creators - whether that’s code, project plans, go to market strategies, culture initiatives, marketing campaigns, large language models or people policies. And together, with our clients, we create the future. This has seen us collaborate and partner across a multitude of industries and sectors, with the likes of PayPal, adidas, Natwest, FanDuel and Money Saving Expert, to name just a few.

Our reputation as a partner determined to deliver high-quality, robust and thoughtful products has enabled us to scale to over 500 people in the last couple of years, and it is our amazing people - along with the safe, supportive and friendly culture we have built - that makes CreateFuture a great place to work. Don’t just take our word for it though, we have been recognised by Best Workplaces UK multiple years in a row - across a number of categories - and our employee exit rate is astonishingly low. 

Join us on our journey… Let’s create something awesome, together, today.

About the role and team:

We're looking for an experienced Lead Data Engineer to join our growing Data Practice, specialising in iGaming. You'll work as part of a multidisciplinary team delivering data solutions for some of the world's largest betting and gaming operators — organisations processing millions of events per day across sportsbook, casino, and exchange platforms.

This is a role for someone who understands the pressures of the iGaming data landscape: sub-second odds feeds, high-stakes data integrity requirements, complex regulatory reporting, and large-scale platform migrations from legacy on-premises estates to cloud-native architectures on AWS.

As a Lead, you'll shape technical direction, mentor engineers, and act as a trusted partner to client data and engineering teams.

What you'll be doing

• Leading large-scale data migrations from on-premises or legacy platforms to cloud-native architectures on AWS — including Lakehouse designs using Databricks or Snowflake
• Designing and building low-latency data pipelines that process real-time betting events, odds updates, and market data at high throughput
• Implementing robust data integrity controls — validation, reconciliation, and auditability — across betting, trading, and customer data domains
• Architecting streaming and micro-batch solutions using AWS Kinesis, Apache Kafka, or MSK for real-time sportsbook and in-play data feeds
• Working with AWS platform services including S3, Glue, Lambda, Athena, Redshift, Lake Formation, Step Functions, and MWAA
• Supporting regulatory and compliance reporting — ensuring data pipelines meet the auditability and lineage requirements of the UK Gambling Commission and international regulators
• Leading and mentoring a team of data engineers, setting standards for code quality, testing, and documentation
• Contributing to technical proposals, pre-sales scoping, and architecture reviews with iGaming clients

We'd love to talk to you if...

You know the iGaming data landscape

• You've worked in or closely with sportsbook, exchange, or casino platforms — and understand the data models behind markets, selections, bets, and settlements
• You've built or maintained low-latency data feeds — odds ingestion, in-play event streams, price change feeds — where milliseconds matter
• You understand the data integrity challenges specific to betting: reconciliation between trading and settlement systems, detection of data drift, and auditability for regulatory purposes
• You've delivered or contributed to large-scale data migrations — moving high-volume transactional or event data from legacy estates to the cloud without losing fidelity or uptime

You're strong on AWS data engineering

• AWS is your primary cloud — you're confident across Glue, Lambda, Athena, S3, Redshift, RDS, Kinesis, MSK, and Step Functions
• You've built real-time or near-real-time pipelines using streaming frameworks — Kinesis Data Streams, Kafka, Flink, or Spark Streaming
• You understand AWS cost models and how to optimise high-throughput workloads — Spot compute, reserved capacity, and right-sizing for event-driven architectures
• AWS certifications are a bonus — Solutions Architect or Data Analytics Specialty

You know your tools

• Solid experience with Python, SQL, and Spark — and pipeline tools such as dbt or Airflow
• Comfortable with data quality frameworks and testing — Great Expectations, dbt tests, or custom validation layers
• Experience with infrastructure-as-code (Terraform, AWS CDK) is a plus
• Familiar with broader cloud platforms — Azure (Data Factory, Synapse) or GCP (Dataflow, BigQuery) — is beneficial but not essential

You thrive in a consultancy setting

• You've led or mentored data engineers and are comfortable owning technical direction on a complex engagement
• You can navigate the politics of a large operator — multiple teams, competing priorities, and legacy systems that can't just be switched off
• You balance hands-on delivery with client engagement, and know when to push back on scope and when to be pragmatic

Tech snapshot