A

Arena Intelligence

Crowdsourced LLM evaluation and comparison platform

Machine Learning Engineer

Applied Machine LearningPosted on Nov 14
Full-timeCalifornia, USANo salary listedHybrid

About the Job

Requirements

  • Strong programming skills and ability to work across a typical recommendation-system or large language model stack.
  • Experience in deep learning, language models, or reward model training.
  • Experience working with large language models for fine-tuning, prompt engineering, and function calling.
  • At least 4 years of industry experience or relevant project experience.
  • Solid understanding of statistics and tools and methodologies for evaluating uncertainty in a way specific to the product being shipped.

Responsibilities

  • Architect and build core modeling for data and evaluation products.
  • Own the full-stack data, model-training, and evaluation pipelines.
  • Help grow a culture of feedback and rapid product iteration while building new features as a tight-knit team.
  • Conduct research into state-of-the-art evaluation methods and contribute to the long-term vision for a centralized, scalable evaluation platform.

About the Company

Arena Intelligence

Crowdsourced LLM evaluation and comparison platform

Arena Intelligence provides a crowdsourced, open platform for evaluating large language models (LLMs) by running head-to-head comparisons of two anonymous models on user prompts. Users vote on the better response and, after voting, model identities are revealed; the results generate large-scale preference data used to compute Elo ratings published on a public leaderboard. Originating from LMSYS and UC Berkeley’s Sky Computing Lab as Chatbot Arena, the project evolved into a company and rebranded to Arena, with $100 million seed funding in 2024 to scale the platform. The underlying framework, FastChat, remains open source, including the user interface and serving backend. Arena differentiates itself by providing a transparent, real-world benchmarking method that aggregates community-driven judgments and offers analytics to AI labs and enterprises, aiming to guide model development with observable performance data.

Company size
51-200
Company stage
Series B
Total funding
$450M
Headquarters
San Francisco, CA
Founded
2025

Recently Posted Jobs

Explore more opportunities at Arena Intelligence

See more jobs