Data and Analytics – Lead

We are looking for a highly experienced Azure Databricks (ADB) Lead with 10+ years of experience in data engineering, cloud platforms, and big data processing. The ideal candidate will have expertise in Azure Data Services, Databricks, Spark, and data pipelines to design, develop, and optimize large-scale data solutions.

Experience- 10+ years

Location- Kochi/Chennai/Coimbatore

 Key Responsibilities:

 • Lead the design, architecture, and implementation of Azure Databricks solutions for data processing and analytics.

• Develop and optimize ETL/ELT pipelines using Apache Spark, Python (PySpark), Scala, and SQL.

• Work closely with data engineers, data scientists, and business stakeholders to ensure efficient data workflows.

• Implement data governance, security, and compliance best practices in Databricks and Azure Data Lake.

• Optimize performance tuning, cost management, and scalability of Databricks workloads.

• Automate CI/CD pipelines for Databricks notebooks using DevOps tools like Azure DevOps, Terraform, or GitHub Actions.

• Troubleshoot and resolve issues related to big data processing, Spark jobs, and cloud storage.

• Stay updated with Azure and Databricks best practices and mentor junior team members.

• Required Skills & Experience: 

10+ years of experience in data engineering, big data, or cloud-based data solutions.

• Strong hands-on experience with Azure Databricks, Apache Spark, and Delta Lake.

• Expertise in Python (PySpark), Scala, SQL, and data transformation techniques.

• Deep understanding of Azure Data Services (Azure Data Lake, Azure Synapse, Azure Data Factory, etc.).

• Experience with data modeling, ETL/ELT design, and distributed data processing.

• Proficiency in DevOps practices, including CI/CD pipelines, Infrastructure as Code (IaC), and version control (Git).

•Familiarity with MLflow, Unity Catalog, and Databricks SQL is a plus. • Strong analytical, problem-solving, and communication skills.

Preferred Qualifications: 

• Databricks certifications (e.g., Databricks Certified Data Engineer Associate/Professional).

• Experience with containerization (Docker, Kubernetes) and orchestration tools.

• Knowledge of real-time data streaming using Kafka, Event Hubs, or similar technologies.

• Exposure to Machine Learning (ML) workflows in Databricks is a plus.

What we provide:

• Competitive Salary

• Performance based Variable Pay

• Health, Vision, Dental, and Life Insurance

• Comprehensive Employee Wellness Plan

• Retirement plan

• Paid Holidays and Vacation Time Accrual

• Encouraging and highly energetic environment

• Personal and Professional Growth Opportunities