AI/LLM Quality Engineer

Summary

We are looking for an exceptional Software Quality Engineer with hands-on experience and a strong technical background to elevate the quality bar for Siri features and intelligent services in Mainland China. In this role, you will champion product quality across the entire lifecycle, blend traditional QA rigor with cutting-edge Generative AI and LLM evaluation methodologies, and build innovative automation tools that push testing, evaluation, and operational efficiency to the next level.

Minimum Qualifications

BS/MS in Computer Science, Computer Engineering, or equivalent practical experience. Strong proficiency in at least one modern programming or scripting language (e.g., Python, Swift, Java, Go), with solid fundamentals in data structures, databases, SQL, and OS environments (Unix/Linux/macOS). Hands-on experience architecting and implementing test automation frameworks for client-side applications, backend services, and end-to-end integration flows. Experience with CI/CD pipelines, containerization, and modern cloud/distributed infrastructure. Solid understanding of Generative AI / LLM concepts, with proven ability to build tools, scripts, or workflows utilizing model APIs. Strong analytical, logical, and debugging skills: capable of breaking down complex, ambiguous problems into testable hypotheses and investigating issues across diverse technical stacks. High adaptability and resilience to navigate rapidly shifting priorities in a fast-paced environment. Strong communication and interpersonal skills, with the ability to collaborate effectively across multi-functional and geographically distributed teams. Professional working proficiency in both written and spoken English and Mandarin.

Preferred Qualifications

LLM-Based Lifecycle QA & Evaluation Pipelines: Proven track record in establishing an LLM-driven, full-lifecycle quality assurance system and automated evaluation pipelines (e.g., automated test case generation, offline model evaluation, self-healing E2E test automation, continuous online monitoring, and A/B experimentation quality assessment). Model Quality Metrics & Bad Case Closed-Loop: Demonstrated ability to define model quality metrics and dashboards; systematically collect, categorize, and analyze bad cases (distinguishing hallucinations, tool-invocation errors, retrieval failures, and prompt deficiencies); and partner closely with cross functional teams to drive closed-loop fixes. Evaluation & Benchmark Design: Experience designing evaluation schemes, curated benchmark datasets, adversarial/negative samples, and edge cases across conversational AI, RAG, tool-use/agents, and multimodal scenarios. AI Productivity Tooling: Hands-on experience developing lightweight AI-driven developer productivity tools (e.g., agentic workflows, LLM-assisted code or test generation). Safety & Compliance: Experience in content safety, compliance, privacy, and responsible AI risk evaluation (e.g., bias, harmful content generation). Apple Ecosystem: Experience testing on Apple platforms (iOS, iPadOS, macOS, watchOS and etc). Soft Skills & Mindset: Self-motivated and proactive, with demonstrated creative and critical thinking capabilities. Language Skills: Fluency or proficiency in Cantonese is a strong plus.

Description

As a Siri Quality Engineer, you will work at the intersection of AI, speech/language technologies, and customer experience. Your key responsibilities include:

Responsibilities

End-to-End Quality Ownership: Drive test strategies, comprehensive test plans, design reviews, and quality execution to ensure new Siri capabilities and services ship with exceptional quality to delight our customers. LLM & AI Evaluation: Design rigorous evaluation schemes, curated benchmark datasets, adversarial/negative samples, and edge cases across conversational AI, RAG, tool-use/agents, and multimodal scenarios. AI-Empowered Tooling & Automation: Leverage Generative AI and modern automation technologies to develop internal tools, intelligent test generation pipelines, and self-healing test frameworks to boost operational velocity and daily engineering productivity. Quality Metrics & Bad Case Closed-Loop: Define model quality metrics and monitoring dashboards. Systematically collect, categorize, and diagnose failure modes (e.g., hallucinations, tool invocation errors, retrieval failures, prompt deficiencies) and partner with modeling and product teams to drive continuous improvement. Deep Troubleshooting & Root-Cause Analysis: Investigate and troubleshoot complex, ambiguous issues across multiple layers, including client applications, cloud backend services, APIs, telemetry/logs, and AI model behaviors. Safety & Compliance: Support evaluations for content safety, privacy, bias, and responsible AI risks to maintain high standards for user trust and compliance. Cross-Functional Collaboration: Partner closely with engineering, product, localization, and cross-site teams to instill quality best practices throughout the software development lifecycle.