Data & AI Engineer - Databricks

Aiimi is a British technology and services company on a mission to turn enterprise data from a source of chaos and risk into a trusted, strategic asset. Through our Automated Data Governance platform and expert consultants, we help our customers get business value – from national security and critical infrastructure providers to leading global brands and government agencies.
 
We are looking for a Data & AI Engineer to help clients build trusted data products and evaluate AI-enabled solutions for their business. The role is centred on Databricks and combines strong data engineering with hands-on exploration of capabilities such as Databricks Apps and Genie.
You’ll build the data foundations that make useful AI experiences possible, then work with clients and colleagues to test ideas using Databricks’ evolving features. You’ll bring the engineering judgement to build well-structured prototypes, assess how they perform with real data, and explain what it would take to develop a promising idea further.
You’ll work with client teams, data engineers and AI specialists to understand business problems and turn them into practical solutions.

The Skills You'll Bring

Essential:  

• Strong data engineering experience, including designing and building data pipelines and transformations.
• Strong Python and SQL skills, with practical experience using PySpark or Spark.
• Hands-on experience with Databricks and its data engineering capabilities.
• Experience working with lakehouse technologies such as Delta Lake.
• Experience designing data solutions with attention to quality, structure, performance and access.
• Hands-on experience with Databricks Apps, Genie or other emerging Databricks capabilities.
• Practical understanding of generative AI, including its strengths, limitations and common failure modes.
• Ability to build and assess prototypes, interpret results and communicate what you learn.
• Strong technical judgement and the ability to work independently with clients and delivery teams.
• Clear communication skills, including the ability to explain technical choices to non-specialists.

Desirable: 

• Experience building interactive applications or data experiences within Databricks.
• Familiarity with Unity Catalog and Databricks data governance.
• Experience with Azure or Microsoft data platforms.
• Familiarity with Power Platform, Power Automate or other low-code tools.
• Experience connecting data and AI capabilities to business workflows.
• Experience with generative AI applications, retrieval or natural-language access to data.

Key Responsibilities

• Design and build data pipelines and transformations on Databricks using Python, SQL and PySpark.
• Develop reliable data foundations for analytics and AI, working with technologies such as Spark and Delta Lake.
• Explore Databricks capabilities, including Apps and Genie, and identify where they could address real business needs.
• Build working prototypes that combine well-prepared data with Databricks’ AI and application features.
• Work with business users to identify useful questions, workflows and scenarios to test.
• Evaluate prototypes using realistic examples, identify limitations and explain what affects the results.
• Investigate how data quality, structure, metadata and access permissions affect the usefulness of AI features.
• Recommend practical next steps for promising ideas, including the data, engineering and platform work needed.
• Apply strong engineering practices so prototypes are clear, maintainable and easy to build on.
• Explain technical findings to clients and colleagues and share reusable patterns across Aiimi.
• Provide technical guidance and support the development of other engineers.

Aiimi Benefits

• 25 days annual leave (plus bank holidays)
• Flexible working options – hybrid
• Mental health and wellbeing support, including access to counselling
• Annual wellbeing allowance (e.g. personal training, fitness, wellness apps)
• Up to 10% of your salary in employee benefits, including critical illness cover, life insurance, and private healthcare (post-probation)
• Generous company pension contribution
• Ongoing professional development and training opportunities