M

Magic

AI-powered code generation and collaboration

Member of Technical Staff - Evals

Natural Language Processing & LLMsPosted on Nov 11
Full-timeSan Francisco, CA, USA

About the Job

Requirements

  • Experience developing model evaluations, benchmarks, or machine-learning experiments.
  • Strong programming skills and experience working with machine-learning systems.
  • A track record of owning technical projects end-to-end.
  • Strong understanding of post-training techniques and their impact on model performance.
  • Ability to analyze model performance and interpret experimental results.
  • Ability to determine whether a measurement captures the behavior it claims to measure.

Responsibilities

  • Design and validate evaluations and benchmarks across model training and deployment.
  • Investigate model behavior and identify gaps in existing evaluations.
  • Measure dataset quality and identify opportunities to improve training data.
  • Improve evaluation correctness, reproducibility, and reliability.
  • Audit and improve public benchmarks, research papers, and open-source implementations.
  • Define metrics that accurately reflect model quality.
  • Build tools that make evaluation results accessible and useful across the company.

About the Company

Magic

AI-powered code generation and collaboration

Magic.dev focuses on building AI-powered coding tools that act as a virtual coworker for developers. It creates proprietary models that understand and generate code, enabling collaborative coding tasks rather than simple assistance. The product works through subscription or licensing, allowing clients to integrate the AI into their own systems and workflows after training on large datasets with extensive R&D. What sets Magic.dev apart is its emphasis on collaboration with developers as a true coding partner, backed by frontier-scale models, heavy data collection, and a scalable infrastructure designed for enterprise needs. The company’s goal is to advance AI in software development by enabling its virtual coworker to generate, optimize, and co-create code with human programmers.

Company size
51-200
Company stage
Late Stage VC
Total funding
$465M
Headquarters
San Francisco, CA
Founded
2022

Recently Posted Jobs

Explore more opportunities at Magic

See more jobs