A

Apple

Designs phones, computers and their operating systems

ML Engineer - Automated Evaluation and Adversarial Design

Natural Language Processing & LLMsConfirmed live in the last 24 hours
Full-timeNo salary listedIn-person

About the Job

Requirements

  • A bachelor's degree in Computer Science, Machine Learning, Statistics, or a related field is required.
  • At least 4 years of experience building or significantly extending ML evaluation systems, including designing evaluation benchmarks or quality assessment frameworks for sequential or multi-step AI outputs, is required.
  • Experience independently defining evaluation architecture and methodology for AI or ML systems, including evaluation approaches where the unit of analysis is a conversation or session rather than a single output, is required.
  • Experience designing adversarial or red-teaming test methodologies for ML models or AI-powered features, including adversarial scenarios targeting failures across multi-turn interactions, is required.
  • Experience with Python and machine learning frameworks such as PyTorch or TensorFlow in production or near-production settings is required.
  • A track record of owning technical direction for evaluation efforts across multiple features or product areas is required.

Responsibilities

  • Define and own the automated evaluation approach for AI features, translating qualitative notions of quality into measurable, reproducible assessments across single-turn and multi-turn agentic experiences.
  • Build adversarial test suites targeting known and emerging model failure modes, including productivity application workflow edge cases and conversation-level failures such as context loss, instruction forgetting, and cascading errors across multi-step tasks.
  • Develop and execute stress test protocols validating minimum performance thresholds under atypical input conditions, including extended conversation lengths, adversarial mid-conversation topic shifts, and complex tool-use sequences.
  • Ensure ongoing alignment between automated and human evaluation methods by identifying and resolving systematic disagreements.
  • Collaborate with engineering partners to integrate evaluation into development and release workflows.
  • Scale adversarial test case generation and stress test execution using automation, including programmatic generation of multi-turn conversation scenarios and agent interaction traces.
  • Influence model and feature quality decisions by communicating evaluation findings and readiness assessments to cross-functional partners.

Desired Qualifications

  • Experience evaluating user-facing AI features in consumer applications and connecting technical metrics to user-perceived quality.
  • Familiarity with productivity software or creative tools and the ability to assess output quality from a user workflow perspective.
  • Experience aligning automated and human evaluation methods, including inter-annotator agreement analysis and bias detection.
  • A track record of designing evaluation systems that scale across multiple features or product areas without bespoke solutions for each.
  • Experience evaluating API-based and custom-trained AI systems.
  • Ability to communicate evaluation findings and readiness assessments to cross-functional partners.
  • Experience using automation to scale evaluation data generation and analysis.
  • Experience building evaluation pipelines for conversational AI, dialogue systems, or agentic workflows, including turn-level and session-level automated scoring.
  • Familiarity with agent orchestration frameworks such as LangChain, LangGraph, CrewAI, and AutoGen, and observability tools such as LangSmith, Braintrust, and Arize, including instrumenting and evaluating multi-step agent runs.
  • Experience designing adversarial tests for tool-use reliability, function-calling accuracy, or agent planning quality.
  • A graduate degree in a relevant field.

About the Company

Apple

Designs phones, computers and their operating systems

Apple designs and sells consumer electronics, from iPhone smartphones and Mac computers to tablets and wearable devices. Its operating systems manage apps and built-in functions for communication and work, while iCloud stores files and synchronizes data across devices. Customers buy additional digital services through subscriptions such as Apple Music and Apple TV, and developers distribute software through the App Store. Apple earns money from hardware sales, subscriptions and digital transactions, selling directly to customers and through retailers and mobile carriers around the world. It was founded in 1976 and has its headquarters in Cupertino, California.

Company size
10,001+
Company stage
IPO
Headquarters
Cupertino, CA
Founded
1976

Recently Posted Jobs

Explore more opportunities at Apple

See more jobs