Data Engineering Assistant Manager

Here's an overview of what it typically involves, along with the qualifications and skills needed

Qualifications:

• Experience: 4 - 6 years of experience in data engineering, ETL development, and cloud-based data platforms.
• Technical Skills: 
  • Proficiency in Python, PySpark and SQL
  • Strong expertise in Databricks Data Engineering capabilities, including: Databricks Lakehouse Architecture, ETL/ELT Design and Development, Delta Lake, Data Modeling, Batch and Streaming Data Pipelines, Spark Optimization and Performance Tuning
  • Experience in designing and implementing: Medallion Architecture, Data Quality Frameworks, Data Governance Controls, Metadata and Lineage Solutions
  • Experience with orchestration and workflow management tools such as Databricks Workflows, Azure Data Factory, or equivalent.
  • Experience implementing CI/CD pipelines, Git-based source control, automated testing, and deployment automation.
  • Strong understanding of data warehousing, dimensional modeling, and modern Lakehouse architectures. 
  • Exposure to Databricks Mosaic AI, Vector Search, MLflow, feature engineering, or GenAI data preparation workflows (Preferred).
  • Familiarity with data preparation for RAG, LLM, and AI/ML use cases (Preferred).
  • Experience with infrastructure-as-code and containerization technologies (Preferred).
• Certifications:  Relevant certifications in data engineering, such as Databricks Certified Data Engineer Associate/Professional, or similar. (Preferred not mandatory)
• Problem-Solving:  Strong analytical and problem-solving skills, with a proven ability to design scalable data solutions and resolve complex data engineering challenges.
• Communication:  Excellent communication skills with experience working directly with clients and business stakeholders to deliver data and analytics solutions.
• Stakeholder Management: Experience engaging with business stakeholders to understand data and analytics requirements and translate them into scalable data engineering solutions.

Role and Responsibilities:

• Data Pipeline Development: Design, develop, and maintain scalable batch and streaming ETL pipelines using Databricks, PySpark, Python, and SQL.
• Lakehouse Implementation: Build and manage data solutions leveraging Delta Lake and medallion architecture principles to support trusted and reusable data products.
• Data Integration: Integrate data from enterprise applications, databases, APIs, files, and cloud platforms to enable unified analytics and reporting.
• Data Modeling & Transformation: Develop efficient data models and transformation frameworks that support business intelligence, advanced analytics, and AI/ML use cases.
• Platform Optimization: Monitor and optimize Databricks workloads, cluster configurations, and data processing performance to ensure scalability and cost efficiency.
• Data Quality & Governance: Implement data quality checks, monitoring, lineage, security controls, and governance practices in alignment with enterprise standards.
• DevOps & Automation: Contribute to CI/CD implementation, automated deployments, testing frameworks, and release management for data engineering solutions.
• Cross-functional Collaboration: Partner with business stakeholders, data architects, analytics teams, and AI engineers to deliver data products that support business outcomes.
• AI & Analytics Enablement: Enable downstream AI, machine learning, and Generative AI initiatives by building reliable, high-quality, and governed data foundations.
• Continuous Improvement & Innovation: Stay current with advancements in Databricks, cloud technologies, and modern data engineering practices to continuously enhance platform capabilities.