Data Engineer

Responsibilities

• Work collaboratively with research scientists and software engineers on software development for a range of different robotic platforms.
• Develop and maintain our data storage solutions and data pipelines in cloud and on-premise infrastructure.
• Use Python and Terraform to develop and scale cloud-native data stores.
• Build event- and batch-driven ingestion systems for machine learning and R&D.
• Write and maintain user guides for internally developed tools.
• Create and use systems to clean, integrate, or fuse datasets to produce data products.
• Establish and monitor data integrity and quality through visualization, profiling, and statistical tools.
• Perform updates, migrations, and administration tasks for data systems.
• Develop and implement data governance and data retention strategies.

Requirements

• BS/MS in computer science, robotics, or equivalent experience.
• 6+ years of experience in a data engineering, software engineering, DevOps, or MLOps role.
• Strong experience building event-driven data ingestion systems.
• Strong experience with distributed data/computing tools, such as Spark, Ray, EMR, Dataproc, Dask, or Pandas on Spark.
• Strong experience with ETL design and implementations in the context of large, multimodal, distributed datasets.
• Strong experience with workflow orchestration tools, such as Airflow, Argo Workflows, Cloud Composer, MWAA, Step Functions, or Prefect.
• Demonstrated experience building containerized applications using tools and frameworks such as Docker, Docker-compose, Podman, or OCI.
• Demonstrated experience with schema management and schema evolution.
• Demonstrated experience with databases and data storage solutions, such as Google Cloud Storage (GCS), S3, BigQuery, NoSQL and/or SQL.
• Experience with container orchestration tools, such as Kubernetes, GKE, EKS, or AKS.
• Experience with UNIX/Linux including basic commands and shell scripting.

Bonus (Not Required)

• Associate- or Professional-level GCP certifications.
• 3+ years of experience working on time-series data and streaming applications.
• 3+ years of experience with NoSQL implementation such as Mongo, Cassandra, DynamoDB, Datastore, or BigTable.
• 3+ years of experience working with on-prem compute and storage appliances.
• 3+ years of experience with data streaming tools, such as Kafka, Flink, Kinesis, Beam, Spark Streaming, or Dataflow.
• 2+ years of experience customizing package managers or build tools, such as Make, Poetry, or Bazel.
• 2+ years of experience with Infrastructure as Code tools such as Terraform, Go CDK, or AWS CDK.
• 2+ years of experience using data quality tools, such as great-expectations, or Cerberus.