H

HUD

AI agent evaluation platform and benchmarks

Delivery Engineer

Program ManagementConfirmed live in the last 24 hours
Full-time

About the Job

Requirements

  • Experience owning complex technical programs in AI/ML, data operations, support engineering, applied research, forward-deployed engineering, or a similarly cross-functional environment.
  • Strong quantitative and qualitative judgment about data, including moving between aggregate metrics and individual examples to assess whether a dataset or evaluation is useful, reliable, and representative.
  • Experience maintaining and improving complex processes, especially within project delivery or data contexts.
  • Ability to bring structure to ambiguous problems, maintain momentum as requirements change, and make sound tradeoffs without waiting for a perfect specification.
  • Strong written and verbal communication skills, including the ability to turn messy technical information into clear decisions, owners, and next steps.

Responsibilities

  • Own data and evaluation programs end-to-end, from initial scoping and requirements gathering through production, quality assurance, delivery, and retrospective.
  • Translate ambiguous requests from AI labs and internal teams into clear specifications such as milestones, owners, dependencies, and acceptance criteria.
  • Maintain and improve data quality procedures using quantitative signals and qualitative inspection, including analyzing task coverage, difficulty, and reward distributions.
  • Maintain and improve delivery pipelines at scale, including managing external vendors throughout the delivery lifecycle.
  • Streamline the logistics behind data production and evaluation programs by improving workflows, documentation, automation, and cross-functional handoffs.
  • Partner with research, platform, and marketplace teams to translate learnings and feedback into product improvements.

Desired Qualifications

  • Experience with RLHF, reinforcement-learning environments, agent evaluations, human-in-the-loop data pipelines, annotation, or expert data collection.
  • Experience supporting frontier AI labs, technical enterprise customers, or research teams with urgent and evolving requirements.
  • Early-stage startup experience and ability to work independently in fast-paced environments.

About the Company

HUD

AI agent evaluation platform and benchmarks

HUD provides an evaluation platform for AI agents that perform computer-use tasks. It offers an interface that connects to HUD evaluation environments, allowing users to run benchmarks across hundreds of environments and thousands of tasks. Users can integrate their agents using various adapters, and interact with the system through an asynchronous API for efficient experiments. The platform then collects telemetry and benchmarking results to inform agent improvements. HUD differentiates itself by concentrating on evaluating and improving knowledge-work agents across a wide range of tasks and environments, rather than just building models, and by enabling scalable, tool-enabled testing via adapters and an asynchronous workflow. The company’s goal is to help organizations and developers assess, compare, and enhance their AI agents so they perform better in real-world knowledge-work scenarios.

Company size
11-50
Company stage
Seed
Total funding
$130K
Headquarters
San Francisco, CA
Founded
2025

Recently Posted Jobs

Explore more opportunities at HUD

See more jobs