A

AGI, Inc.

Advanced AI for natural language understanding

Research Engineer - Evals

AI ResearchConfirmed live in the last 24 hours
Full-timeSan Francisco, CA, USA

About the Job

Requirements

  • Build and maintain evaluation systems for model capability, agentic behavior, on-device performance, and end-user experience.
  • Measure non-deterministic systems, including agent evaluation, tool use, long-horizon tasks, and multilingual behavior.
  • Push back on metrics that are being gamed without disrupting the team.
  • Provide a link to an evaluation, benchmark, or measurement system built and explain what decision it changed.
  • Work in person in San Francisco.

Responsibilities

  • Own evaluation suites that gate every model and agent release, covering capability, behavior, regressions, and human-rated rubrics.
  • Own dashboards and tooling that accelerate researcher experiment loops and support leadership decisions.
  • Define the release bar and determine what is ready to ship.
  • Support research by ensuring that measurements reflect intended goals.
  • Assist product engineers by instrumenting real-user behavior on real devices.
  • Support partnerships by translating improvement measurements into standards an original equipment manufacturer partner can use.
  • Audit existing evaluations, identify noise and gaps, and fix evaluation deficiencies.
  • Stand up new agentic, on-device, or behavioral evaluation surfaces and use their output to inform decisions.
  • Catch regressions before release and shape the research roadmap through evaluation results.
  • Evaluate on-device performance trade-offs, real-user evaluation, and quality assurance for artificial intelligence at original equipment manufacturer scale.

About the Company

AGI, Inc.

Advanced AI for natural language understanding

AGI, Inc. develops and deploys highly capable artificial intelligence systems focused on natural language understanding. Its product is a software system that processes language using deep learning models to understand and answer questions, aiming to perform at human-level accuracy on standard NLU benchmarks such as SQuAD. Unlike many AI developers that target broad capabilities, AGI, Inc. centers on advancing NLU performance on established datasets, with the goal of solving core language understanding challenges. The company measures its progress by achieving human-level results on benchmark datasets and applying its technology to real-world language tasks.

Company size
11-50
Company stage
Not listed
Total funding
Not listed
Headquarters
Austin, TX
Founded
2021

Recently Posted Jobs

Explore more opportunities at AGI, Inc.

See more jobs