L

LoiZéro

Director of Evaluations

Natural Language Processing & LLMsConfirmed live in the last 24 hours
Full-timeMontreal, QC, CanadaNo salary listed

About the Job

Requirements

  • A Master's degree, Doctorate, or equivalent in Machine Learning, computer science, or a closely related field is required.
  • More than 10 years of experience in Machine Learning, including at least 5 years in a leadership role building or advancing technical teams working on real-world ML products.
  • Practical expertise designing and executing large-scale evaluations of LLMs or other frontier ML systems for capabilities, safety, and robustness against attacks.
  • Proven experience creating evaluation datasets, benchmarks, or interactive environments, including for safety-related properties such as honesty, sycophancy, refusal behaviors, and robustness against attacks.
  • Strong written and verbal communication skills, including the ability to explain technical findings to non-technical audiences such as executives, funders, and policymakers.
  • Ability to work in a fast-paced, research-focused environment with substantial ambiguity and bring structure without slowing its pace.

Responsibilities

  • Define LawZero's evaluation strategy and roadmap, prioritizing what should be measured and when, in close coordination with research and product teams.
  • Build the Evaluations team during the first 3 to 6 months, growing it to approximately 8 to 10 specialists in research, engineering, dataset and benchmark design, and red-teaming.
  • Lead the team independently from the main research and product tracks to avoid conflicts of interest, including designing novel benchmarks for fair comparisons between Scientist AI and frontier LLMs.
  • Oversee the design and creation of datasets, tasks, and virtual or interactive environments to measure Scientist AI's capabilities, safety (including honesty and goal orientation), applicability, causal mechanisms, and adversarial attack detection.
  • Lead evaluations of Scientist AI when deployed as a guardrail around frontier models, including assessing compliance with harm specifications, detection and blocking of harmful responses, explanation of decisions, and resistance to jailbreaks, prompt injection, and data poisoning.
  • Establish and lead automated and manual red-teaming programs, internally and with external providers, to test Scientist AI as both a general-purpose model and a guardrail.
  • Lead development of internal tools and infrastructure for large-scale evaluations, automating and standardizing the pipeline where possible.
  • Directly support research and product teams with their internal evaluation and benchmarking needs as needed to remove blockers and accelerate processes.
  • Lead public communication of LawZero's evaluation results through model and system cards, technical reports, peer-reviewed publications, and blog posts to build trust with the AI safety community.
  • Represent LawZero externally on AI safety evaluation and measurement, including with AI safety institutes, research collaborators, and funders.

Desired Qualifications

  • Experience leading red-teaming exercises (automated, manual, or both) and working with external evaluation or red-teaming partners.
  • Experience collaborating with third-party partners to create benchmarks and datasets.
  • Experience contributing to publication of open-source datasets, benchmarks, or evaluation tools.
  • Familiarity with current AI safety policy and standards work, including UK AISI, CAISI, NIST, and the EU AI Act.
  • Experience contributing to or coordinating with external safety institutes, funders, or government bodies.

About the Company

LoiZéro

Company size
Not listed
Company stage
Not listed
Total funding
Not listed
Headquarters
Not listed
Founded
Not listed

Recently Posted Jobs

Explore more opportunities at LoiZéro

See more jobs