Regulatory Compliance GxP AI Model Consultant Senior

At EY, you’ll have the chance to build a career as unique as you are, with the global scale, support, inclusive culture and technology to become the best version of you. And we’re counting on your unique voice and perspective to help EY become even better, too. Join us and build an exceptional experience for yourself, and a better working world for all. 

EY- GDS Consulting – Enterprise Risk (ER) – Regulatory Compliance –Senior – GxP AI/Model Consultant

As part of our EY-ER- Regulatory Compliance team, you will help the clients by understanding their business needs and delivering solutions in accordance with the EY guidelines & methodologies. As a Regulatory Compliance senior actively establish, maintain and strengthen internal and external relationships. In line with our commitment to quality, you’ll consistently drive projects to completion with high quality deliverables, achieve operational efficiency, proactively identify and raise risks with the client as well as EY senior management team and lead internal initiatives.

The opportunity

We are seeking a highly experienced and detail‑oriented AI Model Evaluation Consultant with strong expertise in working alongside AI/ML models, Generative AI (GenAI) systems, and multi‑agent workflows within GxP‑regulated and quality‑critical environments. The ideal candidate will ensure the reliability, fairness, robustness, and regulatory compliance of AI solutions while engaging directly with cross‑functional teams in designing, testing, and maintaining production‑ready, inspection‑ready AI systems.

This role is responsible for leading AI model consultation and development activities across the full model lifecycle, including model integration, training, testing, deployment, and post‑deployment monitoring, in alignment with GxP principles, risk‑based validation, and data integrity expectations. The consultant will play a key role in embedding Responsible AI, quality‑by‑design, and governance‑by‑design practices, enabling the safe, trustworthy, and explainable use of AI and agent‑based solutions in regulated life sciences and pharmaceutical contexts.

Your key responsibilities

• Work on development of AI/ML and Agentic AI strategies by providing analysis, validation insights, and readiness assessments.
• Perform AI/ML model integration, evaluation and validation across development, training, testing, and deployment phases.
• Collaborate with data scientists and engineers to identify model improvement opportunities and risk mitigations.
• Design and execute testing strategies for AI agents and multi‑agent workflows, covering orchestration logic, tool invocation, and output validation.
• Collaborate with data scientists and engineers to identify model improvement opportunities and risk mitigations.
• Document evaluation results, limitations, risks, and acceptance criteria in a clear, audit‑ready manner.
• Develop or review validation plans, user and functional requirements, risk assessments, requirements traceability matrices, test protocols, test evidence, deviation records, and validation summary reports.
• Establish acceptance criteria for data quality and model performance, including accuracy, sensitivity, specificity, precision, recall, F1 score, calibration, error rates, and other measures appropriate to the use case.
• Execute or independently review functional, integration, regression, negative, boundary, security-role, audit-trail, data-reconciliation, and user-acceptance testing for AI-enabled workflows.
• Challenge model reliability through reproducibility, robustness, bias, subgroup-performance, explainability, stress, and edge-case testing, with clear documentation of limitations and residual risk.
• Define production monitoring for model performance, data and concept drift, anomalous outputs, override patterns, human review, complaints, incidents, and other signals that may trigger investigation or revalidation.
• Evaluate proposed model updates, retraining, prompt changes, knowledge-base changes, infrastructure changes, and vendor releases through formal change control and impact assessment.
• Ability to perform root cause analysis for hallucinations, inaccurate outputs, agent failures, latency issues, and unexpected model behaviour.
• Assess model behavior across datasets, edge cases, and failure scenarios to identify bias, drift, and instability.
• Understand ML Fundamentals - Strong grasp of classification, regression, clustering, anomaly detection, and model performance metrics (Precision, Recall, F1, AUC).
• Evaluate LLM and GenAI Systems - Experience measuring accuracy, completeness, relevance, groundedness, faithfulness, consistency, and task completion.
• Analyze Drift and Model Performance - Ability to identify and investigate data drift, concept drift, model drift, and performance degradation in production environments.
• Assess RAG-Based Solutions - Understand retrieval pipelines and be able to evaluate retrieval quality, citation accuracy, and response grounding.
• Work with AI Observability Platforms - Experience with AI monitoring, tracing, evaluation logging, dashboards, and observability tools such as Langfuse, LangSmith, Arize, WhyLabs, MLflow, or similar platforms.
• Use LangChain, LangGraph, and Langfuse for agent workflows, observability, traceability, and evaluation logging.
• Implement and use AI observability capabilities for tracing, evaluation logging, dashboards, alerting, failure analysis, and audit-ready evidence across model and agent workflows.

Qualifications:

• Bachelor’s or master’s degree in Life Sciences, Engineering, or related field.
• 5+ years of experience in AI/ML, GenAI, software testing, validation, or quality engineering roles.
• Experience creating user stories, test scripts, validation plans, or AI solution prototypes.
• Exposure to cloud ecosystems such as Azure, AWS, or GCP.
• Consulting or client‑facing experience preferred.
• Excellent communication, documentation, and stakeholder management skills.

Must-Have Skills And Attributes

• Strong understanding of ML models, LLMs, GenAI workflows, agentic systems, and evaluation techniques.
• Strong foundation in machine learning concepts and model evaluation techniques.
• Hands‑on experience with: Precision, Recall, Confusion Matrix, ROC/AUC, Supervised and Unsupervised ML models
• Practical experience with LangChain, LangGraph, and Langfuse.
• Experience evaluating GenAI / LLM‑based applications and agents.
• Strong documentation, analytical, and stakeholder communication skills.
• Experience designing test cases, user stories, and acceptance criteria for AI/ML‑based systems.
• Ability to validate AI outputs, evaluate accuracy, detect bias, and assess model reliability under varied conditions.
• Hands‑on skills in building prototypes, test harnesses, and demo environments.
• Strong stakeholder management, analytical thinking, and structured communication skills.
• Familiarity with version control, prompt engineering fundamentals, and AI‑assisted testing tools is preferred.
• Understanding of data integrity principles and electronic records/e-signatures compliance
• Experience in client-facing roles, managing expectations and delivering solutions
• Strong communication and presentation skills
• Ability to troubleshoot application issues and recommend solutions
• Strong teamwork and collaboration mindset

Good-to-Have Skills And Attributes

• Exposure to business development activities and client engagement strategies
• Ability to create innovative insights and contribute to thought leadership
• Understanding of market trends and competitor landscape
• Experience in process optimization and operational efficiency initiatives
• Ability to align technology with business transformation goals
• High attention to detail and analytical thinking
• Adaptability to changing regulatory and business environments
• Strong problem-solving and decision-making abilities

What we look for

• A Team of people with commercial acumen, technical experience and enthusiasm to learn new things in this fast-moving environment with consulting skills.
• An opportunity to be a part of market-leading, multi-disciplinary team of 1400 + professionals, in the only integrated global transaction business worldwide.
• Opportunities to work with EY Consulting practices globally with leading businesses across a range of industries

What working at EY offers

At EY, we’re dedicated to helping our clients, from start–ups to Fortune 500 companies — and the work we do with them is as varied as they are.

You get to work with inspiring and meaningful projects. Our focus is education and coaching alongside practical experience to ensure your personal development. We value our employees and you will be able to control your own development with an individual progression plan. You will quickly grow into a responsible role with challenging and stimulating assignments. Moreover, you will be part of an interdisciplinary environment that emphasizes high quality and knowledge exchange. Plus, we offer:

• Support, coaching and feedback from some of the most engaging colleagues around
• Opportunities to develop new skills and progress your career
• The freedom and flexibility to handle your role in a way that’s right for you

EY | Building a better working world 

 
EY exists to build a better working world, helping to create long-term value for clients, people and society and build trust in the capital markets.  

 
Enabled by data and technology, diverse EY teams in over 150 countries provide trust through assurance and help clients grow, transform and operate.  

 
Working across assurance, consulting, law, strategy, tax and transactions, EY teams ask better questions to find new answers for the complex issues facing our world today.