Databricks Engineer

Role: Databricks Engineer 

Experience: 9-12 Years 

Location: ALL EXL Locations 

Work Mode: Hybrid 

Key Role and Responsibilities: 

 Design, build, and maintain Databricks workspaces, clusters, and compute pools across development, testing, and production environments. 

 Configure and manage Unity Catalog for data governance, fine-grained access control, permissions, metadata management, and data lineage. 

 Optimize Databricks cluster configurations, including instance types, auto-scaling, spot/preemptible nodes, and compute pools to improve performance and reduce costs. 

 Implement workspace best practices, including folder structures, access controls, secret management using Databricks Secrets, Azure Key Vault, or AWS Secrets Manager. 

 Create, schedule, and manage Databricks Jobs, Workflows, and multi-task job orchestration with dependency management. 

 Design and implement Delta Lake tables using partitioning, Z-Ordering, OPTIMIZE, VACUUM, and file compaction techniques. 

 Build and maintain Medallion Architecture (Bronze, Silver, and Gold layers) for scalable and governed data lakehouse solutions. 

 Develop Delta Live Tables (DLT) pipelines with built-in data quality expectations for reliable ETL/ELT processing. 

 Manage schema evolution, table versioning, Time Travel, and Change Data Feed (CDF) to support incremental data processing. 

 Design and implement lakehouse architectures integrating Delta Lake with cloud storage and external systems such as Azure Data Lake Storage (ADLS), Kafka, Event Hubs, and Kinesis. 

 Develop scalable batch and real-time data pipelines using PySpark, Spark SQL, Structured Streaming, and Delta Lake. 

 Build streaming ingestion pipelines from Kafka, Azure Event Hubs, and other streaming platforms into Delta tables. 
 Optimize PySpark applications using broadcast joins, Adaptive Query Execution (AQE), dynamic partition pruning, caching, and Photon Engine.

 Develop reusable transformation frameworks, utility libraries, and pipeline templates to improve engineering productivity and standardization. 

 Implement robust error handling, retry mechanisms, logging, monitoring, and dead-letter queue (DLQ) patterns for production-grade pipelines. 

 Set up and manage MLflow experiment tracking, model registry, and model lifecycle management. 

 Support machine learning workloads by enabling scalable model training, inference, and GPU-based compute environments. 

 Develop feature engineering pipelines using Databricks Feature Store to create reusable and versioned machine learning features. 

 Enable Generative AI solutions, including Retrieval-Augmented Generation (RAG), vector search, LLM fine-tuning, and Mosaic AI capabilities. 

 Implement MLOps best practices, including model versioning, model deployment, A/B testing, and Databricks Model Serving. 

 Integrate Databricks with Azure Data Lake Storage (ADLS) and other cloud-native services. 

 Develop and maintain CI/CD pipelines using Azure DevOps, GitHub Actions, or GitLab CI for Databricks notebooks, jobs, and workflows. 

 Automate Databricks infrastructure deployment using Databricks Asset Bundles (DABs), Terraform, and Infrastructure-as-Code (IaC) practices. 

 Build and manage data ingestion frameworks using Auto Loader, COPY INTO, and third party integration tools such as Fivetran, dbt, and Airbyte. 

 Monitor pipeline execution, cluster utilization, system performance, and cloud costs using Databricks system tables and cloud monitoring tools. 

 Implement row-level security, column-level masking, dynamic views, and governance policies using Unity Catalog. 

 Enforce data quality through Delta Live Tables expectations and Great Expectations frameworks. 

 Perform query optimization, execution plan analysis, caching strategies, and performance tuning to improve workload efficiency. 

 Maintain enterprise data cataloging, metadata management, and end-to-end data lineage. 

 Prepare technical documentation, architecture diagrams, operational runbooks, and standard operating procedures for Databricks platform and data engineering solutions. 

Qualifications for Candidates 

 Bachelor’s or master’s degree in computer science, Information Technology, Data Engineering, or related field. 

 4+ years of total experience in data engineering or software engineering. 

 3+ years of dedicated hands-on experience with the Databricks platform in production environments. 

 Strong background in big data engineering, cloud data platforms, and distributed computing. 

 Deep expertise in Databricks Workspaces, Clusters, Jobs, Workflows, and Repos.

 Proficiency with Unity Catalog — metastore setup, catalog/schema/table management, access controls, and data lineage. 

 Hands-on experience with Delta Live Tables (DLT) — pipeline development, expectations, and monitoring. 

 Strong command of Delta Lake internals — transaction log, ACID guarantees, file layout, and optimization techniques. 

 Experience with Databricks SQL Warehouses, SQL Analytics, and dashboard creation. 

 Knowledge of Databricks Photon engine, serverless compute, and cost optimization strategies. 

 4+ years of PySpark development — Dataframe, Datasets, Spark SQL, RDD operations. 

 Expert-level SQL — window functions, lateral joins, CTEs, recursive queries, and analytical functions. 

 Experience with Spark performance tuning — AQE, query plans (EXPLAIN), partitioning, and caching. 

 Proficiency with Python for pipeline development, utilities, and automation. 

 Hands-on experience with at least one: Azure (ADLS Gen2, ADF, Azure Databricks), AWS (S3, EMR, Glue, AWS Databricks), or GCP (GCS, BigQuery, Dataproc).  Experience with cloud networking for Databricks: VNet/VPC injection, private endpoints, and firewall configurations. 

 Familiarity with IAM roles, managed identities, and service principal authentication for Databricks. 

 (Nice to Have): 

 Working knowledge of MLflow — experiment tracking, model registry, and deployment. 

 Experience supporting ML pipelines on Databricks for training, evaluation, and serving. 

Education: UG - Any

Graduate in Computer Science, Data Science, or related field. 9-12 years of experience in data engineering or related field.

 Design, build, and maintain Databricks workspaces, clusters, and compute pools across development, testing, and production environments. 

 Configure and manage Unity Catalog for data governance, fine-grained access control, permissions, metadata management, and data lineage. 

 Optimize Databricks cluster configurations, including instance types, auto-scaling, spot/preemptible nodes, and compute pools to improve performance and reduce costs. 

 Implement workspace best practices, including folder structures, access controls, secret management using Databricks Secrets, Azure Key Vault, or AWS Secrets Manager. 

 Create, schedule, and manage Databricks Jobs, Workflows, and multi-task job orchestration with dependency management. 

 Design and implement Delta Lake tables using partitioning, Z-Ordering, OPTIMIZE, VACUUM, and file compaction techniques. 

 Build and maintain Medallion Architecture (Bronze, Silver, and Gold layers) for scalable and governed data lakehouse solutions. 

 Develop Delta Live Tables (DLT) pipelines with built-in data quality expectations for reliable ETL/ELT processing. 

 Manage schema evolution, table versioning, Time Travel, and Change Data Feed (CDF) to support incremental data processing. 

 Design and implement lakehouse architectures integrating Delta Lake with cloud storage and external systems such as Azure Data Lake Storage (ADLS), Kafka, Event Hubs, and Kinesis. 

 Develop scalable batch and real-time data pipelines using PySpark, Spark SQL, Structured Streaming, and Delta Lake. 

 Build streaming ingestion pipelines from Kafka, Azure Event Hubs, and other streaming platforms into Delta tables. 
 Optimize PySpark applications using broadcast joins, Adaptive Query Execution (AQE), dynamic partition pruning, caching, and Photon Engine.

 Develop reusable transformation frameworks, utility libraries, and pipeline templates to improve engineering productivity and standardization. 

 Implement robust error handling, retry mechanisms, logging, monitoring, and dead-letter queue (DLQ) patterns for production-grade pipelines. 

 Set up and manage MLflow experiment tracking, model registry, and model lifecycle management. 

 Support machine learning workloads by enabling scalable model training, inference, and GPU-based compute environments. 

 Develop feature engineering pipelines using Databricks Feature Store to create reusable and versioned machine learning features. 

 Enable Generative AI solutions, including Retrieval-Augmented Generation (RAG), vector search, LLM fine-tuning, and Mosaic AI capabilities. 

 Implement MLOps best practices, including model versioning, model deployment, A/B testing, and Databricks Model Serving. 

 Integrate Databricks with Azure Data Lake Storage (ADLS) and other cloud-native services. 

 Develop and maintain CI/CD pipelines using Azure DevOps, GitHub Actions, or GitLab CI for Databricks notebooks, jobs, and workflows. 

 Automate Databricks infrastructure deployment using Databricks Asset Bundles (DABs), Terraform, and Infrastructure-as-Code (IaC) practices. 

 Build and manage data ingestion frameworks using Auto Loader, COPY INTO, and third party integration tools such as Fivetran, dbt, and Airbyte. 

 Monitor pipeline execution, cluster utilization, system performance, and cloud costs using Databricks system tables and cloud monitoring tools. 

 Implement row-level security, column-level masking, dynamic views, and governance policies using Unity Catalog. 

 Enforce data quality through Delta Live Tables expectations and Great Expectations frameworks. 

 Perform query optimization, execution plan analysis, caching strategies, and performance tuning to improve workload efficiency. 

 Maintain enterprise data cataloging, metadata management, and end-to-end data lineage. 

 Prepare technical documentation, architecture diagrams, operational runbooks, and standard operating procedures for Databricks platform and data engineering solutions.