Applied AI Lead

At Microsoft AI, our Health team is on a mission to help millions of users better understand and proactively manage their health and wellbeing. We're responsible for ensuring that Microsoft AI's models and services are useful, trusted and safe across diverse customer health journeys. 

What "Applied AI" means at Microsoft AI 

We turn frontier models into products people can trust with their health. We build rigorous, health-specific evals and use them to drive real product decisions. We master orchestration, from harness and context engineering to blending different model classes and families and applying state-of-the-art techniques. And we bring deep, bleeding-edge AI expertise that uplevels the wider team and helps shape the product and engineering roadmap. 

The role 

We are looking for an Applied AI Lead to join our engineering team. This is a hands-on leadership role: you will set the technical direction for this work in the health domain, while growing and developing the engineers who build it. You will be predominantly focused on building Copilot Health, acting as a key bridge between the latest research and product and playing a pivotal role in establishing Copilot as the leader in safe, informative, trustworthy and useful health information. 

You'll bring very strong proficiency in designing, building and running LLM evaluations, and in LLM orchestration: building agentic, multi-step systems that combine prompting, tool use and retrieval to deliver reliable results in production. 

Lead the team 

• Lead, mentor and grow a team of Applied AI Engineers, fostering a collaborative, inclusive and high-performing environment where engineers do the best work of their careers. 

• Stay deeply hands-on. Set the technical bar through code and design reviews, lead by example on the hardest problems, and remain a credible technical authority on evals and LLM systems. 

• Co-own the roadmap. Partner with product leads to qualify and size new opportunities, co-author the product roadmap, and lead the architecture and development of new products and features from 0 to 1. 

• Own delivery. Plan and prioritise the team’s roadmap, balance a strong bias towards shipping and learning with a high-quality bar, and ensure the reliability of what reaches production. 

Set the technical direction on evaluation and orchestration 

• Define the evaluation strategy. Design and oversee evaluation systems that test LLM capabilities in the health domain, including internal benchmarking and regression testing that capture model accuracy, safety and utility - and make sure results are interpreted and clearly communicated to stakeholders. 

• Architect LLM orchestration. Guide the design of agentic, multi-step systems that combine prompt / context engineering, tool use and retrieval, and champion best practices for building and deploying them reliably at scale. 

• Run and direct experiments to determine how different prompting and orchestration techniques affect results on internal and industry benchmarks, and turn those findings into product improvements. 

• Invest in tooling. Improve the internal tooling used to implement, run and analyse evaluations, and the data pipelines - dataset sourcing, curation and synthesis - that feed them. 

Required 

• Bachelor’s or higher degree in Computer Science or a related technical discipline, AND significant Python programming experience / machine learning research. 

• Very strong proficiency with LLM evaluations - demonstrated experience designing, building and running eval pipelines, curating and synthesising datasets, designing automated analyses, and explaining results to internal stakeholders. 

• Very strong proficiency with LLM orchestration - deep, hands-on experience building with and around LLMs, including prompt / context engineering, tool use, harness engineering, retrieval and agentic, multi-step systems, and building tools to analyse and understand their performance. 

• Proven engineering leadership - 8+ years of software engineering experience, including at least 3 years leading technical teams or projects as a tech lead and/or people manager - mentoring and developing engineers, and guiding a group to deliver high-quality results. Formal people-management experience is preferred; a strong technical lead ready to step into a TLM role will be considered. 

• 0-to-1 experience with a bias towards shipping and learning while balancing a high-quality bar. 

• Experience collaborating in cross-functional teams, working through ambiguity to deliver high-quality results, and a proven ability to contribute to a positive, inclusive work environment that fosters knowledge sharing and growth. 

Preferred 

• Experience in healthcare technology, or experience in the health domain. 

• Experience with data engineering - handling text dataset sourcing, curation and processing tasks at scale. 

• Experience translating cutting-edge research into shipped products in a fast-paced, startup-like environment. 

• Passionate about conversational AI and its deployment. 

• Demonstrated written and verbal communication skills, with the ability to work closely with cross-functional teams including product managers, designers and other engineers. 

• Passion for learning new technologies and staying up to date with industry trends, best practices and emerging patterns in AI.