Senior Data Engineer

Mission Summary

We are seeking a highly skilled and experienced Senior Data Engineer to play a critical role in evaluating and improving the performance of our autonomous vehicle (AV) technology. This position demands a strong software development background with expertise in Python, coupled with a deep understanding of data engineering principles and the ability to design and build robust data pipelines. A key focus of this role will be the development and implementation of metrics and data infrastructure to effectively measure, analyze, and report on the performance of our AV systems. The ideal candidate will be passionate about autonomous vehicles and possess the technical acumen to transform complex AV data into actionable insights.

What You’ll Be Doing

• Design and Development of AV Performance Metrics: Collaborate with cross-functional teams, including systems engineers, software engineers, AI scientists, control engineers, simulation engineers, and validation teams to understand their needs and define, design, and implement key metrics to evaluate various aspects of autonomous vehicle performance (e.g., lane change capability performance, oscillation and braking metrics).
• Software Development for Metrics and Analysis: Leverage strong Python programming skills to develop framework, libraries and tools for calculating, aggregating, and visualizing AV performance metrics from large datasets.
• Building Scalable Data Pipelines for AV Data: Design, develop, and maintain efficient and scalable data pipelines to ingest, process, transform, and store vast amounts of data generated by autonomous vehicles (e.g., sensor data, simulation data, logs, annotations).
• Data Modeling and Storage for AV Metrics: Design and implement data models and schemas optimized for storing and querying AV performance metrics, considering factors like granularity, temporal aspects, and dimensionality.
• Data Quality and Validation for AV Data: Implement rigorous data quality checks and validation processes to ensure the accuracy, reliability, and integrity of the data used for performance evaluation.
• Performance Optimization of AV Data Infrastructure: Identify and implement performance optimizations for data pipelines and storage solutions to handle the high volume and velocity of AV data.
• Cloud Infrastructure for AV Data: Design, build, and manage data infrastructure on cloud platforms (e.g., AWS, GCP) specifically tailored for AV data processing and storage.
• Automation and Infrastructure as Code: Implement automation for data pipeline deployments, infrastructure provisioning, and metric calculation processes using Infrastructure as Code (IaC) principles and tools.
• Documentation and Knowledge Sharing: Create clear and concise documentation for data pipelines, metrics definitions, and analysis tools. Share knowledge and best practices with the wider engineering team.
• Staying Current with AV and Data Engineering Technologies: Continuously learn and evaluate new technologies and methodologies relevant to autonomous vehicles and data engineering.

What We're Looking For

• Minimum of 5-7 years of experience in data engineering roles.
• Bachelor's degree in Computer Science, Engineering, Robotics, or a related field. A Master's degree is a plus.
• Expert-level proficiency in Python and strong understanding of software development principles.
• Hands-on experience with one or more data pipeline orchestration tools.
• Experience with distributed data processing frameworks.
• Strong understanding of relational databases and data warehousing concepts.
• Experience working with cloud data services on AWS, Azure, or GCP.
• Proven ability to design and implement effective data models for complex data.
• Deep understanding of ETL and ELT concepts and best practices.
• Proficient with Git and related version control workflows.
• Strong analytical and problem-solving skills, particularly in the context of complex system performance evaluation.
• Excellent communication and collaboration skills, with the ability to effectively communicate technical concepts to diverse audiences.
• Ability to work independently and as part of a multidisciplinary team.
• Strong attention to detail and a commitment to data quality and accuracy.
•  
• Comfortable working in a fast-paced, continuous delivery environment.

Bonus Points (not required)

• Experience working with autonomous vehicle sensor data (e.g., LiDAR, camera, radar).
• Familiarity with simulation environments and data generated from AV simulations.
• Experience with data visualization tools (e.g., Looker) for presenting performance metrics.
• Knowledge of statistical analysis and machine learning techniques relevant to performance evaluation.
• Experience with real-time data processing and streaming technologies.
• Familiarity with data annotation pipelines and tools.
• Familiarity with Ray and Ray clusters for scaling Python applications and AI/ML tasks.

Success Metrics

• Successful design and implementation of key autonomous vehicle performance metrics.
• Development of robust and scalable data pipelines for AV data processing and metric calculation.
• Improvement in the efficiency and effectiveness of AV performance evaluation processes.
• Timely and accurate delivery of data and insights to AV development teams.
• Contribution to the overall advancement and safety of our autonomous vehicle technology.

This is a unique opportunity to make a meaningful impact at Motional by enhancing our data platform and accelerating development speed for our industry-leading autonomous vehicle metrics and capabilities ecosystem. We take pride in having some of the brightest and most dedicated talent in the world on our team. If you're innovative, driven, and eager to tackle challenges, we’d love to have you join us!