AI Software Evaluation Engineer - Siri AI Quality

Summary

Play a part in the next revolution in human-computer interaction and redefine mobile computing through voice interaction! Apple's Siri AI Quality team is looking for a passionate and skilled AI Software Evaluation Engineer to join our mission of delivering the highest-quality AI experiences to millions of users worldwide. We own the end-to-end Siri pre-ship evaluation dataset and the synthetic personas that power it. In this role, you will combine agentic engineering, data analytics and AI evaluation experience to help us catch quality issues early, automate repetitive eval operations, and expand coverage as Siri's features and platforms grow.

Minimum Qualifications

5+ years of industry experience in software engineering, quality engineering, data engineering or ML engineering. 2+ years of hands-on experience building LLM-based applications or agents, including tool use, multi-step workflows and prompt engineering. Experience designing and running AI or LLM evaluation: building evaluation datasets, defining metrics and rubrics, LLM-as-judge or automated evaluators, and analyzing failures. Strong data analytics skills, including SQL and Python, and experience turning large datasets into clear insights and dashboards. Strong programming skills in Python and experience with automation and CI/CD. Strong communication and collaboration skills, with the ability to influence cross-functional partners and work across teams.

Preferred Qualifications

Experience with agent frameworks and developer tooling, such as MCP servers, skills or plugins, and agentic testing. Experience with voice assistants, conversational AI or multi-turn dialogue evaluation. Experience with data schemas, validation frameworks, and structured data formats (e.g. JSON). Experience with data visualization and BI tools (e.g. Tableau) and with data warehouses (e.g. Snowflake). Familiarity with coverage analysis, traffic or taxonomy analysis, and root-cause analysis of flaky or regressing tests. Comfortable working in a fast-paced, collaborative environment with shifting priorities.

Description

In this role, you will build the systems that evaluate Siri's AI experiences and turn the results into action. Your work will span multiple areas of focus: * Evaluation data insights. Investigate why test scenarios regress and which issues recur. Turn failure and bug data into root-cause findings and recommendations that product, engineering and partner teams act on. * Coverage analysis. Find gaps across app intents, features and product areas, and partner with domain owners to close them. Build and report on dataset quality, issue trends and coverage. * Agentic scenario generation, triage and auto-fix. Design agentic workflows that propose, ground, validate, triage and nominate new evaluation scenarios with minimal manual effort. Extend automated screening, triage and fix tooling to new domains and device platforms. You will work with engineering, product, evaluator, persona and data-platform teams to define quality standards and deliver evaluation systems that guide Siri's development. If you are a self-motivated engineer with a strong analytical mind, a bias toward automation and a desire to improve a product used by millions, we encourage you to apply.