Machine Learning Engineer — Agentic Evals

Responsibilities     •    Design and implement agentic benchmark tasks on machine learning topics (e.g., fine-tuning, reinforcement learning, evaluation pipelines).     •    Tasks must be challenging for frontier LLMs (e.g., GPT-5.6, Claude Fable 5) while remaining verifiably solvable. Target solve rates will be discussed.     •    Tasks must be specified using the Harbor framework, including a containerized environment, task instructions, and an automated verifier.     •    Tasks must be realistic implementation tasks situated in complex codebases - not toy problems or single-file exercises.     •    Deliver, for each task:      ◦    A working, reproducible environment (Dockerfile + dependencies)      ◦    Clear task instructions as they would be presented to the model      ◦    An automated verifier/grader that is robust to shortcuts and reward hacking      ◦    A reference solution demonstrating the task is solvable as specified      ◦    Brief documentation (task rationale, difficulty calibration notes, known edge cases)      ◦    Test tasks end-to-end against at least one frontier model agent and report observed behavior/solve rates. Scope & Engagement     •    Number of tasks: 1–5 (exact number to be discussed).     •    Compensation: $70-100/hr     •    Timeline: 1 week     •    Further task details, style guides, and examples will be provided upon onboarding. Requirements     •    Strong machine learning background, especially LLM post-training (SFT, RLHF, PPO/GRPO, LoRA)     •    Strong Python and PyTorch proficiency     •    Experience creating reinforcement learning environments or automated evaluations     •    Comfort with Docker and containerized development workflows     •    Ability to write robust automated test/verification code Preferences     •    Experience with the Tinker SDK/API     •    Experience with Harbor, Terminal-Bench, SWE-bench-style task authoring, or similar agentic eval frameworks     •    Previous contributions to public or private benchmarks     •    Experience running or evaluating LLM agents (scaffolds, harnesses, tool use)