Inactive
I

Intercontinental Exchange

Runs exchanges, data and mortgage technology

Senior Data Engineer

Data EngineeringPosted on Oct 1
Full-timeHyderabad, Telangana, IndiaNo salary listedIn-person

About the Job

Requirements

  • At least three years of production experience running Apache Airflow, including knowledge of scheduler internals, executor types, XComs, Connections, Variables, pools, and the TaskFlow API.
  • Experience deploying and operating Airflow on Kubernetes using KubernetesExecutor or CeleryKubernetesExecutor, including pod templates, resource tuning, persistent volume claims, Helm chart management, and upgrades.
  • Strong Python skills for DAG authoring, including dynamic task mapping, custom operators and sensors, cross-DAG triggers, parameterised pipelines, and test coverage.
  • Demonstrated experience building batch and incremental ETL and ELT pipelines at scale, with transformation logic, data quality validation, and lineage capture.
  • A solid understanding of lakehouse design, medallion models, table formats, and reliable delivery of data to downstream consumers.
  • Hands-on knowledge of Apache Iceberg, including snapshot isolation, partition evolution, hidden partitioning, time-travel queries, and managing Iceberg tables in a catalog environment.
  • Practical Databricks experience with cluster management, Delta Lake, Workflows or Jobs, and Unity Catalog.
  • Confident ability to build dbt projects, including models, sources, tests, snapshots, seeds, and macros; integrate dbt with Airflow; document projects; and implement meaningful freshness checks.
  • Strong engineering-grade Python skills, including writing structured, tested, and packaged code; familiarity with pandas, PySpark, SQLAlchemy, and Pydantic.

Responsibilities

  • Own Apache Airflow end-to-end by building DAGs for multi-step ingestion, transformation, validation, and delivery workflows, ensuring idempotency, safe backfills, SLA alerting, dynamic task mapping, and thorough testing.
  • Review other engineers' Airflow DAGs and improve team standards.
  • Operate Airflow on Kubernetes using KubernetesExecutor or CeleryKubernetesExecutor, manage Helm upgrades, tune pod templates and resource limits, and address scheduler performance issues.
  • Build production-ready ETL and ELT pipelines across APIs, message queues, databases, and object storage, making them fault-tolerant, incremental where appropriate, and straightforward to debug.
  • Build data quality controls into pipelines, including schema validation, row-count reconciliation, anomaly detection, freshness, and accuracy checks.
  • Design and maintain an Apache Iceberg lakehouse architecture using a Bronze, Silver, and Gold medallion structure, addressing schema evolution, partitioning strategies, and time-travel queries.
  • Use Databricks to manage cluster compute, Delta Lake workflows, Unity Catalog, and scheduled jobs, maintaining cost efficiency and usability.
  • Build and maintain dbt projects alongside Airflow, including structured and documented models, tests, snapshots, and macros, and ensure test failures and freshness checks are actionable.
  • Develop reusable platform components, including pipeline templates, custom Airflow operators, and utility libraries.
  • Partner with data scientists and ML engineers to deliver feature pipelines and training datasets with the freshness and reproducibility their models require.
  • Instrument pipelines with metrics, dashboards, and alerts; own runbooks and participate in on-call coverage.
  • Participate in architecture discussions, tooling evaluations, and mentoring.

Desired Qualifications

  • Experience with Databricks Asset Bundles or MLflow.
  • Experience with near-real-time ingestion using Apache Kafka, Kafka Connect, or Flink, including landing data in lakehouse storage.
  • PySpark proficiency for large-scale distributed processing, including memory management, shuffle optimisation, and partition sizing.
  • Platform knowledge of Kubernetes and OpenShift, including Helm, RBAC, and namespace management.
  • Familiarity with data contract approaches, catalogue tools such as DataHub or Collibra, and quality frameworks such as Great Expectations or Soda.
  • Experience with managed data services on AWS, Azure, or GCP, including S3, Glue, ADLS, Synapse, GCS, or Dataproc.
  • Ability to provision data infrastructure using Terraform or Pulumi.
  • Understanding of feature stores and experience building reliable pipelines for model training and inference.

About the Company

Intercontinental Exchange

Runs exchanges, data and mortgage technology

Intercontinental Exchange runs financial marketplaces, clearing houses and data services for banks, investors and businesses that trade or finance assets. Its New York Stock Exchange lets companies raise capital and gives investors a place to buy and sell listed shares. ICE also sells market pricing and analytics, while its mortgage software supports lenders from loan applications through closing and servicing. Trading fees, listing fees and subscriptions pay for these services, and clearing houses manage the risk that one side of a trade fails to pay. Founded in 2000 to digitize energy trading, Intercontinental Exchange is headquartered in Atlanta, Georgia.

Company size
10,001+
Company stage
IPO
Headquarters
Atlanta, GA
Founded
2000

Recently Posted Jobs

Explore more opportunities at Intercontinental Exchange

See more jobs