Data Engineer

Key Responsibilities

• Design, develop, and support batch and incremental data ingestion pipelines into the data lake.
• Develop and optimize complex SQL queries, data transformations, and ETL workflows.
• Build, maintain, and monitor Airflow DAGs for orchestration and workflow automation.
• Analyze and troubleshoot data quality, data reconciliation, and pipeline execution issues.
• Perform end-to-end data lineage analysis and trace data from source systems to downstream consumers.
• Debug data discrepancies by investigating source systems, transformation logic, and target datasets.
• Support data migration and modernization initiatives from legacy platforms to cloud-based data platforms.
• Work closely with business stakeholders, architects, and upstream/downstream teams to understand data requirements.
• Validate data completeness, accuracy, and consistency across multiple systems.
• Participate in production support activities and resolve data-related incidents within SLA timelines.
• Develop reusable data engineering components and recommend process improvements.

Preferred Candidate Profile

• Strong analytical and problem-solving skills.
• Ability to independently investigate and resolve data issues.
• Good communication and stakeholder management skills.
• Experience in cloud data lake or warehouse environments.
• Exposure to customer, transactional, or enterprise-scale datasets is preferred.

Screening Focus Areas

1. Advanced SQL/PLSQL
2. ETL & Data Pipeline Design
3. BigQuery Concepts
4. Data Debugging & Root Cause Analysis
5. Airflow DAG Development
6. Python Basics
7. Data Migration & Data Validation
8. Source-to-Target Traceability and Data Lineage

Must Have: SQL/PLSQL, BigQuery/Cloud Data Warehouse, ETL, Data Analysis & Debugging
Good to Have: Airflow DAG Development, Python, GCP Services

Kafka, Pub/Sub optional

Required Skills

Strong Expertise

• SQL / PL/SQL
  • Complex joins, CTEs, analytical functions
  • Query optimization and performance tuning
  • Data modeling and database concepts
• BigQuery
  • Query development and optimization
  • Partitioning and clustering concepts
  • Data loading and transformation techniques
• ETL/Data Engineering
  • Data ingestion patterns (Batch and Incremental)
  • Data transformations and validations
  • Data quality checks and reconciliation

Good to Have

• Apache Airflow
  • Developing and troubleshooting DAGs
  • Scheduling, monitoring, and dependency management
• Python
  • Basic to intermediate scripting
  • Data processing and automation
• Cloud Platforms (Preferably GCP)
  • Cloud Storage
  • BigQuery
  • Composer/Airflow