Photon

Builds custom software and AI for big companies

Data Engineer

Data EngineeringConfirmed live in the last 24 hours

Full-timeIndiaNo salary listedIn-person

About the Job

Requirements

Experience with Apache Airflow, Python, PySpark, Hive or Trino, SQL, and Snowflake.

Responsibilities

Develop and maintain data pipelines, ELT processes, and workflow orchestration using Apache Airflow, Python, PySpark, Hive or Trino, and Snowflake to ensure efficient and reliable data delivery.
Design and implement custom connectors to ingest structured and unstructured data from diverse sources and document formats into the platform.
Collaborate with cross-functional teams to gather requirements, understand data needs, and translate them into technical solutions.
Design and implement data continuous integration and continuous delivery pipelines for automated data integration, transformation, and deployment.
Monitor and troubleshoot data pipelines, proactively identifying and resolving issues related to data ingestion, transformation, and loading.
Conduct data validation and testing to ensure data accuracy, consistency, and compliance.
Stay current with emerging technologies and best practices in data engineering.
Document data workflows, processes, and technical specifications to support knowledge sharing and data governance.

Must Needed Skills: Apache Airflow, Python, PySpark ,Hive(Trino), SQL and Snowflake

Develop and maintain data pipelines, ELT processes, and workflow orchestration using Apache Airflow, Python, PySpark ,Hive(Trino) and Snowflake to ensure the efficient and reliable delivery of data.
Design and implement custom connectors to facilitate the ingestion of diverse data sources into our platform, including structured and unstructured data from various document formats .
Collaborate closely with cross-functional teams to gather requirements, understand data needs, and translate them into technical solutions.
Design and implement data CI/CD pipelines to enable automated and efficient data integration, transformation, and deployment processes.
Monitor and troubleshoot data pipelines, proactively identifying and resolving issues related to data ingestion, transformation, and loading.
Conduct data validation and testing to ensure the accuracy, consistency, and compliance of data.
Stay up-to-date with emerging technologies and best practices in data engineering.
Document data workflows, processes, and technical specifications to facilitate knowledge sharing and ensure data governance.

About the Company

Photon

Builds custom software and AI for big companies

Photon designs, builds, and runs custom software for large companies, combining an in-house creative agency with its own engineering teams. Its flagship client work includes rebuilding Samsung's US online store as a set of smaller linked services and creating Olive Garden's mobile app, which lets diners join the waitlist from their phones instead of waiting at the door. Clients hire Photon's teams to plan and code their apps, websites, and AI tools and to keep them running afterward, paying for the engineers and designers assigned to each project. Photon is privately held and based in London, with its US arm in Dallas and engineering centers in Chennai and Bengaluru.

Company size: 5,001-10,000
Company stage: Not listed
Total funding: Not listed
Headquarters: London, United Kingdom
Founded: Not listed

Recently Posted Jobs

Explore more opportunities at Photon

See more jobs