Software Engineer 2 – Data Engineer

Role Overview

We are looking for a hands-on SDE-II – Data Engineer to join InMobi’s Demand-Side Platform (DSP) team. In this role, you will build, maintain, and optimise scalable data pipelines that support programmatic advertising, campaign analytics, audience targeting, and attribution.

You will work with batch and streaming data, distributed processing frameworks, cloud data platforms, and analytical storage systems to transform large volumes of data into reliable and actionable insights. You will collaborate with backend, product, and ML engineering teams to develop efficient data solutions that support DSP operations.

This role is ideal for an engineer with strong fundamentals in Python, SQL, and distributed data processing who is keen to solve data engineering challenges involving high-volume event streams, performance optimisation, and data reliability.

Key Responsibilities

1. Data Engineering and Pipeline Development

• Develop, maintain, and optimise end-to-end batch and streaming data pipelines for processing events such as bid requests, impressions, clicks, and conversion signals.

• Build data ingestion and transformation workflows using Python, SQL, Apache Spark/PySpark, and relevant data engineering frameworks.

• Work with data lakehouse architectures and contribute to ingestion, storage, transformation, and serving layers.

• Implement data quality checks, schema validation, error handling, and monitoring to ensure data accuracy and reliability.

• Support data modelling and query optimisation for analytical and reporting workloads.

• Contribute to pipeline performance improvements, resource utilisation, and efficient compute and storage usage.

2. Distributed Processing and Streaming

• Work with distributed data processing technologies such as Apache Spark, PySpark, and Apache Kafka.

• Develop and support streaming workflows for processing event-driven data, with attention to data freshness, reliability, and processing efficiency.

• Implement mechanisms for handling duplicate, delayed, or failed events, including retries, checkpointing, and recovery where applicable.

• Investigate production issues and troubleshoot data pipeline failures, performance bottlenecks, and data quality discrepancies.

• Contribute to the adoption and effective use of technologies such as Apache Flink, Spark Structured Streaming, and open table formats where relevant to the platform.

3. Data Storage and Analytics

• Work with cloud-based data platforms and storage services to support scalable data processing and analytics.

• Develop and optimise SQL queries, data transformations, partitioning strategies, and table structures.

• Contribute to analytical data stores such as StarRocks, ClickHouse, or similar technologies, based on project requirements.

• Work with data lake table formats such as Apache Iceberg, Delta Lake, or Apache Hudi, where applicable.

• Support data modelling, metadata management, access controls, and data retention practices in line with established engineering standards.

4. Collaboration and Engineering Practices

• Collaborate with backend engineers, ML engineers, and other stakeholders to understand data requirements and deliver reliable data solutions.

• Work with workflow orchestration tools such as Apache Airflow or equivalent to schedule, monitor, and manage data pipelines.

• Participate in code reviews, technical discussions, testing, and deployment activities.

• Follow engineering best practices for version control, CI/CD, observability, documentation, and production readiness.

• Contribute to continuous improvement by identifying opportunities to enhance pipeline reliability, scalability, performance, and cost efficiency.

Required Skills and Qualifications

Experience and Education

• 2–5 years of experience in software engineering, data engineering, or big data development.

• Bachelor's or Master's degree in Computer Science, Engineering, or a related technical field.

• Experience developing and maintaining data pipelines in a production environment.

• Experience handling large datasets or distributed data processing workloads is preferred.

• Exposure to ad-tech, programmatic advertising, real-time bidding (RTB), or event-driven systems is an added advantage.

Core Data Engineering Skills

• Strong programming skills in Python.

• Good command of SQL, including joins, aggregations, data transformations, and query optimisation.

• Hands-on experience with Apache Spark/PySpark or a comparable distributed data processing framework.

• Experience building and maintaining ETL/ELT pipelines.

• Understanding of batch processing and streaming data concepts.

• Exposure to workflow orchestration tools such as Apache Airflow, Prefect, or Dagster.

• Understanding of data modelling, partitioning, schema evolution, and data quality practices.

Streaming, Cloud, and Data Platform Skills

• Familiarity with Apache Kafka or similar messaging and streaming technologies.

• Exposure to AWS, GCP, or Azure data services.

• Familiarity with data lakes, lakehouse architectures, or analytical databases.

• Exposure to Apache Iceberg, Delta Lake, Apache Hudi, StarRocks, or ClickHouse is an advantage.

• Understanding of monitoring, logging, error handling, and production troubleshooting.

Preferred Skills

• Experience processing high-volume event data or building near-real-time data pipelines.

• Exposure to ad-tech datasets such as bid requests, impressions, clicks, conversions, campaign performance, or attribution data.

• Familiarity with Java or Scala in addition to Python.

• Basic understanding of REST APIs, microservices, and event-driven architectures.

• Exposure to Docker, Kubernetes, and CI/CD tools is a plus.

• Experience collaborating with ML teams on feature engineering or training data pipelines is an advantage.