Generative AI Tester – Senior Associate

Line of Service

Advisory

Industry/Sector

Not Applicable

Specialism

Data, Analytics & AI

Management Level

Senior Associate

Job Description & Summary

At PwC, our people in data and analytics engineering focus on leveraging advanced technologies and techniques to design and develop robust data solutions for clients. They play a crucial role in transforming raw data into actionable insights, enabling informed decision-making and driving business growth.

In data engineering at PwC, you will focus on designing and building data infrastructure and systems to enable efficient data processing and analysis. You will be responsible for developing and implementing data pipelines, data integration, and data transformation solutions.

Years of Experience: Candidates with 4+ years of hands on experience

Responsibilities: 

- Design, develop, and maintain test automation frameworks for AI and GenAI projects. 

- Evaluate generative AI systems for adherence to Responsible AI standards, including fairness, transparency, and ethical considerations. Conduct bias detection and mitigation testing, ensuring equitable performance across demographic groups.Assess system robustness, explainability, and user safety. 

- Develop ground truth datasets for evaluating generative outputs across diverse scenarios. Automate test case creation and result evaluation for iterative AI model refinement. 

- Implement and customize frameworks like RAGAS for thorough evaluation of retrieval-augmented generation (RAG) systems. Analyze model performance on metrics such as accuracy, relevance, coherence, and response completeness. 

- Collaborate with developers, product owners, and stakeholders to define test automation requirements and strategies. 

- Create and execute test cases, scripts, and scenarios for functional, integration, and regression testing. 

- Perform API test automation to ensure the seamless integration of AI and GenAI solutions. 

- Utilize Azure Test Automation (or equivalent framework) to automate test cases and validate system functionality. 

- Work with Azure Kubernetes to deploy and manage test environments. 

- Collaborate with the development team to identify and resolve defects and issues. 

- Ensure test coverage and traceability and provide test reports and metrics to stakeholders. 

- Collaborate with Agile teams to integrate test automation into the CI/CD pipeline using Azure DevOps. 

- Stay up to date with industry best practices and advancements in test automation and AI technologies. 

Required Skills: 

- Strong experience in python programming. 

- Experience working on AI/GenAI projects is essential. 

- Experience in LLM Evaluation, Ground truth preparation and GenAI Solution evaluation. 

- Strong experience with Azure Test Automation (or equivalent framework) for test automation. 

- Proficiency in Azure Kubernetes for test environment management. 

- Experience with API test automation using tools such as Postman or RestAssured. 

- Familiarity with Azure DevOps for test management and CI/CD integration. 

- Strong understanding of software testing principles, methodologies, and best practices. 

Professional and Educational Background:

• BE / B.Tech / MCA / M.Sc / M.E / M.Tech / MBA

- Excellent problem-solving and analytical skills, with the ability to identify and troubleshoot issues. 

- Strong communication and collaboration skills, with the ability to work effectively in cross-functional teams. 

- Ability to work in a fast-paced, dynamic environment and manage multiple priorities. 

Education (if blank, degree and/or field of study not specified)

Degrees/Field of Study required:Degrees/Field of Study preferred:

Certifications (if blank, certifications not specified)

Required Skills

API Testing, Azure Automation, Azure Kubernetes Service (AKS), Generative AI, Large Language Models (LLMs)

Optional Skills

Accepting Feedback, Accepting Feedback, Active Listening, Agile Scalability, Amazon Web Services (AWS), Analytical Thinking, Apache Hadoop, Azure Data Factory, Communication, Creativity, Data Anonymization, Database Administration, Database Management System (DBMS), Database Optimization, Database Security Best Practices, Data Engineering, Data Engineering Platforms, Data Infrastructure, Data Integration, Data Lake, Data Modeling, Data Pipeline, Data Quality, Data Transformation, Data Validation {+ 18 more}

Desired Languages (If blank, desired languages not specified)

Travel Requirements

Not Specified

Available for Work Visa Sponsorship?

No

Government Clearance Required?

No

Job Posting End Date