Data Scientist - AI Quality & Evaluation

About the Role

We're looking for a Data Scientist to own the quality, reliability, and trustworthiness of our clinical AI outputs. You'll build the systems that ensure our AI "knows what it doesn't know" — developing evaluation frameworks, calibrated confidence scoring, and automated quality assurance that physicians can actually trust.

What You'll Do

• Design and implement automated evaluation pipelines that assess AI output quality, accuracy, and safety at scale

• Develop uncertainty quantification systems where confidence scores meaningfully correlate with accuracy

• Build comprehensive evaluation frameworks combining automated assessment with clinician-validated test cases

• Implement feedback loops that continuously improve model outputs based on validation signals

• Establish scalable quality gates that catch errors before they reach end users

• Contribute to model alignment and fine-tuning efforts

Qualifications

Required

• Strong foundation in deep learning frameworks (PyTorch) and LLM architectures

• Experience with model evaluation, benchmarking, and quality metrics

• Proficiency in Python and modern ML development tools

• Strong statistical foundations

• Ability to read, implement, and extend research papers

• Excellent communication skills

Preferred

• Master's degree in Computer Science, Machine Learning, Statistics, or related quantitative field (PhD preferred)

• Publications in top ML/AI venues (NeurIPS, ICML, ICLR, ACL)

• Experience with RLHF, DPO, or preference optimization techniques

• Background in healthcare AI or regulated industries

• Experience building evaluation systems for production LLM applications