A

Artificial Analysis

Independent AI model and API benchmarks

Member of Technical Staff - Language Model Evaluations

Natural Language Processing & LLMsConfirmed live in the last 24 hours
Full-timeSan Francisco, CA, USA

About the Job

Requirements

  • At least 3 years of relevant professional experience across industry or research.
  • Strong analytical and critical thinking skills.
  • Strong Python skills, with hands-on experience running evaluation harnesses and building datasets.
  • Deep familiarity with the language model evaluation landscape, including major benchmarks and their failure modes, contamination, preference-based methods, and agentic evaluation.
  • Strong statistical grounding and the ability to distinguish signal from noise in results.
  • Demonstrable interest in and knowledge of frontier artificial intelligence, including informed opinions about the direction of AI.

Responsibilities

  • Design and ship next-generation frontier evaluations across reasoning, knowledge, coding, agentic capability, and other capabilities.
  • Construct evaluation datasets, harnesses, and scoring systems for benchmarks, engineered for contamination resistance and repeatability at frontier scale.
  • Contribute evaluations to future versions of the Artificial Analysis Intelligence Index and other platform areas.
  • Produce reports, indexes, and data visualizations about language model progress.
  • Benchmark leading AI labs' pre-release and newly launched models while working directly with their research teams.
  • Run the evaluation suite across frontier model releases and own the integrity of the results.
  • Use cutting-edge AI tools in an AI-native workflow to increase leverage in AI benchmarking.

About the Company

Artificial Analysis

Independent AI model and API benchmarks

Independent benchmarking and analysis of AI models and API hosting providers, with benchmarks and leaderboards across intelligence, coding agents, image/speech/video, and inference metrics to help teams choose the right model and provider. It publishes data-driven reports like Intelligence Index releases, capability indexes, and trend analyses, packaging benchmarks as insights and data products for AI buyers. Unlike others, it does not host or sell models, focusing on unbiased evaluation and transparent metrics. The goal is to help AI buyers understand the landscape and select AI models and API providers that fit their use case.

Company size
51-200
Company stage
Seed
Total funding
$250K
Headquarters
Newark, DE
Founded
2024

Recently Posted Jobs

Explore more opportunities at Artificial Analysis

See more jobs