P

Prosus

Operates technology-led consumer internet ecosystems

Senior Machine Learning Engineer - Evals

Natural Language Processing & LLMsPosted on Nov 16
Full-timeAmsterdam, NetherlandsNo salary listed

About the Job

Requirements

  • At least five years of machine-learning or software-engineering experience, including substantial hands-on work building and evaluating machine-learning systems.
  • Experience owning evaluation frameworks, benchmarks, or model-quality systems that informed training or deployment decisions.
  • Strong Python skills and experience with PyTorch, Transformers, or comparable machine-learning libraries, alongside reliable production software development.
  • Practical knowledge of experimental design and statistics, including sampling, uncertainty, repeated trials, and comparisons across model versions.
  • Experience running inference or evaluation workloads at scale and debugging failures across models, data, scoring code, and infrastructure.
  • Clear communication and the ability to turn ambiguous results into defensible recommendations.

Responsibilities

  • Design evaluations with scientists and domain experts by translating intended capabilities into representative tasks, datasets, and success criteria that cover current applications and capabilities desired in future models.
  • Curate evaluation data and calibrate task difficulty so evaluations are neither saturated nor out of reach and distinguish models of different quality. Work with domain experts on sourcing, annotation, and review within data-provenance, permitted-use, and privacy requirements.
  • Own development and operation of the evaluation platform across training checkpoints, deployed models, and external baselines. Support distributed execution, reproducible configurations, trace inspection, and result comparison while managing runtime and cost.
  • Develop programmatic checks, outcome-based scoring, and large-language-model-based graders. Calibrate automated judgments against expert reviews, investigate disagreements, and test whether graders reward superficial answers or reject valid solutions.
  • Analyze model outputs and agent trajectories, including tool calls, environment state, and task outcomes. Distinguish model failures from evaluation problems, quantify uncertainty, and explain changes between versions.
  • Maintain versioned tasks, datasets, graders, and environments with protected held-out tests and contamination checks. Expand coverage as models improve, retaining useful regression tests and adding harder tasks when existing tests no longer distinguish performance.
  • Prepare commerce benchmarks for public release on ProLLM, including public mirror sets and evaluation code where data can be shared.
  • Integrate evaluations into training and release workflows with dashboards and regression checks that support concrete decisions. Work with portfolio teams to establish whether offline improvements correspond to better outcomes in online experiments.
  • Build tools and agents that help generate candidate tasks, run evaluations, and investigate failures, validating their outputs before adoption. Review contributions and mentor engineers so teams can extend the suite consistently.

Desired Qualifications

  • Experience with language-model training or post-training.
  • Experience with agent or gaming environments.
  • Experience with human evaluation, large-language-model-as-judge calibration, recommendation metrics, or online experimentation.
  • Contributions to evaluation frameworks, public benchmarks, or research.

About the Company

Prosus

Operates technology-led consumer internet ecosystems

Prosus is a global consumer internet group that builds technology-led lifestyle ecosystems across Europe, Latin America, and India, owning platforms like iFood, OLX, PayU, Swiggy, Meesho, and Just Eat Takeaway.com in 100+ markets. Its products are a network of apps and websites that connect consumers, merchants, and partners, powered by hands-on operations, investments, and AI-enabled technology to scale services. The company differentiates itself through its global footprint, diversified portfolio, and cross-market capabilities that support rapid growth across multiple sectors. Its goal is to expand its ecosystem to more markets, deepen engagement on its platforms, and create value by scaling trusted digital services for consumers, retailers, and partners.

Company size
10,001+
Company stage
IPO
Headquarters
Amsterdam, Netherlands
Founded
2015

Recently Posted Jobs

Explore more opportunities at Prosus

See more jobs