Data Integration Principal

Job Title: Data Engineer Lead

Location: London, United Kingdom 

Experience Level: 5+ years in Data Engineering

About the Role

We are seeking a highly skilled Data Engineer Lead to drive a critical data migration project. The role will focus on migrating data from Oracle systems into structured file outputs stored in a SharePoint location. You will be responsible for designing and implementing reusable, scalable ETL pipelines in Azure Data Factory (ADF), ensuring data integrity, performance, and reliability.

This role requires strong leadership, technical expertise, and hands-on ability to guide design, implementation, testing, scheduling, and troubleshooting throughout the migration lifecycle.

Key Responsibilities

• Lead the end-to-end data migration process from Oracle system into SharePoint-hosted file storage.
• Design, build, and maintain reusable ADF pipelines for scalable and efficient ETL processes.
• Define and implement scheduling strategies to ensure timely and consistent data delivery.
• Develop monitoring, alerting, and logging mechanisms to support operational excellence.
• Collaborate with business and technical stakeholders to gather requirements and validate migration objectives.
• Conduct data reconciliation and validation between source and target systems to ensure accuracy and completeness.
• Lead testing and debugging activities, including unit testing, integration testing, and system testing of ETL workflows.
• Provide technical guidance and mentorship to junior engineers on ETL best practices and migration design.
• Ensure compliance with data governance, security, and privacy standards throughout the migration.

Required Skills & Qualifications

• Proven experience as a Data Engineer Lead or Senior Data Engineer on migration or integration projects.
• Hands-on expertise with Azure Data Factory (ADF) for ETL/ELT design and implementation.
• Strong experience in data migration from Oracle.
• Proficiency in SQL and experience with large-scale data processing.
• Experience writing outputs to file-based storage (xBase , CSV, Parquet, JSON, etc.) and integrating with SharePoint.
• Strong understanding of scheduling, orchestration, and monitoring in ADF or similar tools.
• Knowledge of data testing strategies including reconciliation, validation, and defect tracking.
• Solid debugging and problem-solving skills to identify and resolve ETL issues.
• Excellent communication skills and ability to work with cross-functional teams.

Good to Have

• Microsoft certification in Azure ecosystem
• Experience with Azure ecosystem tools (Databricks, Synapse, Key Vault, Logic Apps).
• Knowledge of DevOps practices (CI/CD for data pipelines, Git integration).
• Familiarity with data governance frameworks and compliance requirements.
• Leadership experience in managing small teams or mentoring junior engineers.