Senior Software Engineer
Core Data
Posted on 3/29/2023
INACTIVE
Locations
Remote
Experience Level
Entry
Junior
Mid
Senior
Expert
Desired Skills
Apache Hive
Apache Spark
BigQuery
Apache Kafka
Data Analysis
Data Structures & Algorithms
Docker
Airflow
Postgres
Redshift
Snowflake
SQL
Terraform
Apache Flink
Kubernetes
Python
Datadog
Requirements
- Design and build our Dimensional Data Warehouse to support complex data analysis and reporting requirements
- Build a reliable Data Platform with Data Quality checks and observability to ensure high data integrity and availability
- Research and explore new methods to optimize our data processing, storage, and retrieval to support efficient data analytics and insights
- Develop and maintain ETL pipelines that transform and process petabytes of structured and unstructured data at scale to enable data-driven decision-making
- Build reliable internal tools for the Data Platform that enable data scientists and machine learning engineers to efficiently develop, test, and deploy their models and algorithms
- Collaborate cross-functionally with data scientists, machine learning engineers, and product managers to design and implement new data models and internal tools that help Data Scientists accelerate their innovation
- Continuously monitor and evaluate the performance of the Data Platform, and recommend improvements to optimize cost, scalability, and reliability
- Bachelor's degree (or equivalent) in Computer Science or a related field
- 5+ years of experience building distributed system architecture, from whiteboard to production
- Strong programming skills in Python, and SQL or SparkSQL
- Versatility. Experience across the entire spectrum of data engineering, including:
- Data stores (e.g., BigQuery, Snowflake, RedShift, Athena, Hive, Postgres)
- Data pipeline and workflow orchestration tools (e.g., Airflow, DBT, Luigi, Azkaban, Storm)
- Data processing technologies and streaming workflows (e.g., Spark, Kafka, Flink)
- Deployment and monitoring infrastructure in public cloud platforms (e.g., Docker, Terraform, Kubernetes, Datadog)
- Loading, querying, and transforming large data sets