Lead Data Engineer

Position Overview

Job Responsibilities:

Design, develop, and maintain scalable ETL/ELT pipelines using Databricks,
PySpark, and SQL.
● Integrate data from multiple sources, including databases, Amazon S3, files, and REST APIs.
● Build data pipelines with Databricks Unity Catalog.
● Implement business logic, data transformations, and dimensional data models.
● Create, schedule, monitor, and optimize Databricks Jobs and Workflows.
● Design and manage Delta Lake tables using Medallion Architecture (Bronze, Silver,Gold).
● Ensure data quality through validations, error handling, logging, and monitoring.
● Optimize Spark workloads for performance, scalability, and reliability.
● Collaborate with cross-functional teams to deliver production-ready data solutions.

Basic Qualification:

Strong expertise in Python, PySpark, and Advanced SQL.
● Hands-on experience with the Databricks Lakehouse Platform.
● Good understanding of Unity Catalog, Delta Lake, Databricks Workflows/Jobs,
Clusters, Notebooks, Repos, and Medallion Architecture.
● Experience integrating with REST APIs for data ingestion and data export.
● Strong knowledge of ETL/ELT development, batch processing, incremental loading,
and data transformation.
● Experience with data modeling (Star Schema, Snowflake Schema, Fact & Dimension
tables, SCD concepts).
● Understanding of data warehousing concepts and best practices.
● Experience working with structured and semi-structured data (CSV, JSON, Parquet,
Delta).
● Knowledge of partitioning, file optimization, Spark performance tuning, and query
optimization.
● Experience with Git and CI/CD best practices

Preferred Qualifications:

• 9+ years of experience in Data Engineering, including 3+ years of hands-on experience with Databricks.
• Prior experience in a Lead Data Engineer / Technical Lead role, with experience guiding engineers and driving technical decisions.
• Strong hands-on experience with Databricks, Apache Spark, and SQL.
• Experience designing, developing, and optimizing ETL/ELT data pipelines.
•  Experience with Auto Loader, Spark Declarative pipelines, Kafka, Airflow, or dbt is a plus. 
• Databricks certification is an added advantage. Give me Jd for lead role