S

Substrate Bio

Autonomous AI-driven wet labs for biological experimentation

Member of Technical Staff - Evaluations

Natural Language Processing & LLMsConfirmed live in the last 24 hours
Full-timeNo salary listed

About the Job

Requirements

  • Experience building evaluations, benchmarks, or task suites for language models or agents, including scoring code.
  • Experience with long-horizon agent tasks using tools such as search, databases, and code execution.
  • Ability to detect reward hacking, grader exploits, data contamination, label leakage, and other sources of misleading scores.
  • Knowledge of statistics for noisy results, including repeats, uncertainty, and determining whether runs differ meaningfully.
  • Experience with model-based grading and rubric design, including calibrating large language model judges against expert ratings.
  • Hands-on experience with evaluation frameworks such as Harbor, Inspect, or Prime Intellect's verifiers, and with building reproducible evaluations.
  • Ability to communicate clearly about evaluation design, results, and limitations.
  • Strong Python skills and daily use of coding agents.

Responsibilities

  • Own evaluation of the company's models and agents, including the biological models used as tools and the agents' scientific planning and execution.
  • Work with scientists to turn workflows into benchmarks for long-horizon scientific tasks, write correctness and data-support checks, and run internal benchmarks comparing models and harnesses.
  • Design robust benchmarks that remain challenging as models improve, protect them against leakage, memorized answers, and gaming, and report results with their uncertainty.
  • Investigate unexpected result shifts and identify whether their cause is the model, harness, or evaluation.
  • Own the data used for training and evaluation by sourcing external datasets, preparing experimental data, and building tools to help researchers understand model failures.
  • Account for noisy and evolving scientific data in evaluations, ensuring they are reproducible and well documented.
  • Work closely with the RL, biological machine learning, and infrastructure engineers, as well as scientists.

Desired Qualifications

  • Experience evaluating models on biological or other experimental data.
  • Experience building or contributing to public benchmarks or evaluation libraries.
  • Familiarity with how agentic benchmarks such as SWE-bench Verified or Terminal-Bench are built and how they can be exploited.
  • Experience running agent evaluations in containers or sandboxes, in parallel and at volume.
  • Experience with inference engines such as vLLM or SGLang for serving evaluated models.

About the Company

Substrate Bio

Autonomous AI-driven wet labs for biological experimentation

Substrate Bio operates AI-native data-production infrastructure for biology, connecting computational models with real-world experimentation. The company runs a network of autonomous, robotics-powered wet labs, starting in London with a second planned for San Francisco, running standardized experiments with full provenance, metadata, and quality control. Researchers access the labs via API, submitting experiment parameters and receiving structured data and metadata, while foundation models help design assay selections, cell systems, and gene panels. What sets Substrate Bio apart is its closed feedback loop, where better data improves its models and better models design better experiments. The goal is to make biological experimentation as programmable and scalable as computation itself.

Company size
Not listed
Company stage
Not listed
Total funding
Not listed
Headquarters
Not listed
Founded
Not listed

Recently Posted Jobs

Explore more opportunities at Substrate Bio

See more jobs