Machine Learning Engineer
Raydar
Machine Learning Engineer
Raydar
AI-powered recruiting agency for tech roles
Raydar · New York, United States · Full-time · Posted 2026-10-26
Salary: USD 220,000–300,000
Workplace: on_site
About the company
Our client is a technology company.
The role
Raydar is recruiting for this role on behalf of our client. Design, build and run evaluation systems that measure how machine learning models perform as they change over time. Work across defining metrics, building data pipelines and automated judging, and simulating test environments, then turn the findings into concrete model improvements.
What you'll do
- Define evaluation metrics and success criteria for model behavior that shifts over time, and keep them current as methods evolve.
- Build evaluation pipelines covering data handling, labeling, versioning, reruns and automated judging.
- Read model traces and results to find real failure points and propose fixes.
- Create simulated users and environments to test models at scale.
- Own evaluation work from the initial question through pipeline, experiments and written findings.
What we're looking for
- Experience designing, building and running complex evaluations or simulators for reinforcement learning post-training, such as LLM-as-judge setups, RL environments or simulations.
- Peer-reviewed research at leading machine learning venues, or independently released open-source projects of similar caliber.
- Production-quality Python, including testing, maintenance and a regular release cadence.
- Working knowledge of PyTorch.
Bonus points
- Hands-on time creating artificial test agents or mock user populations for assessing or tuning language models.
- Advanced study in a numerate discipline such as computing or applied maths is a plus.
- Track record of contributing to open-source projects in the machine learning research community.
- Experience with personalization, memory or identity modeling for language models.
- Experience with data pipeline tooling and SQL.
Compensation and benefits
- Salary: $220K - $300K
- Equity
- Health, dental and vision coverage
- Retirement plan
- Flexible time off
Location and work model
- New York, NY, United States
- On-site, 5 days in-office per week
- Full-time
Powered by Workable
Raydar is an AI-powered talent acquisition firm that serves tech-focused organizations on a contingency basis, charging client companies only after a successful hire. It uses proprietary AI to identify a large pool of candidates and conducts a short 15-minute AI-driven interview (recorded and transcribed) before human recruiters review materials, provide interview prep, and assist with offers and negotiations. What sets it apart is the blend of automated sourcing and screening with hands-on human oversight and tailored client support. Its goal is to help organizations hire more efficiently by speeding up sourcing while maintaining thorough evaluation and negotiation assistance.
Explore more opportunities at Raydar