Data Engineer

Job Summary:

Key Responsibilities:

• Develop and maintain scalable, robust data pipelines using Scala and big data technologies.
• Work with large datasets from multiple sources to ingest, transform, and make data available for analytics and reporting.
• Collaborate with Data Scientists, Analysts, and other engineers to understand data requirements and deliver efficient solutions.
• Optimize ETL jobs for performance and cost.
• Ensure data quality, governance, and consistency across all environments.
• Monitor production jobs, troubleshoot issues, and ensure system reliability.
• Implement best practices for data engineering, including code reviews, testing, and documentation.

Required Qualifications:

• Bachelor’s or Master’s degree in Computer Science, Engineering, or related field.
• 6+ years of experience as a Data Engineer or in a similar role.
• Strong skills in PySpark and SQL (experience with functional programming is a plus).
• Hands-on experience with big data tools like Apache Spark, Kafka, Hadoop, or Hive.
• Proficiency in building ETL pipelines and working with structured and unstructured data.
• Experience with cloud platforms (e.g., AWS, Azure, GCP) and their data services.
• Familiarity with version control systems (e.g., Git), CI/CD, and DevOps practices.
• Solid understanding of data warehousing and data modeling concepts.

Preferred Qualifications:

• Knowledge of SQL and database systems such as Postgres, Redshift, or Snowflake.
• Exposure to containerization technologies like Docker and orchestration tools like Airflow or Kubernetes.