D

Delphi Consulting

Senior Quality Assurance Engineer

Test Automation EngineeringConfirmed live in the last 24 hours
Full-timeIndia

About the Job

Requirements

  • Seven to ten years of experience in Software QA or Test Engineering.
  • Three to four years of experience leading QA teams or delivery pods.
  • Hands-on experience testing AI or Generative AI applications in production environments.
  • Experience working in regulated domains such as Finance, Healthcare, Insurance, or similar industries.
  • Functional testing, exploratory testing, requirements traceability, and test design techniques.
  • API testing using REST, GraphQL, Postman/Newman, RestAssured, and Requests.
  • Backend and database testing across SQL Server, MySQL, Oracle, PostgreSQL, and ETL/data pipelines.
  • Automation frameworks using Selenium, Playwright, PyTest, and Appium.
  • Performance testing using JMeter, k6, or Locust.
  • Strong programming expertise in Python or Java.
  • Knowledge of object-oriented programming concepts, design patterns, data structures, and unit testing frameworks.
  • Ability to build utilities, custom assertions, command-line interface tools, and reusable automation components.
  • Knowledge of LLM platforms including OpenAI, Anthropic, Azure OpenAI, AWS Bedrock, and Vertex AI.
  • Experience with orchestration frameworks such as LangChain, LlamaIndex, and Semantic Kernel.
  • Understanding of multi-agent systems, memory handling, routing, tool calling, and guardrails.
  • Expertise in RAG evaluation, vector databases, embeddings, retrieval quality, and hallucination testing.
  • Familiarity with AI evaluation platforms such as RAGAS, DeepEval, Galileo, Arize Phoenix, Patronus, Okareo, or similar tools.
  • Experience validating OCR systems and Speech-to-Text solutions using WER/CER and degradation testing methodologies.
  • Experience with Azure DevOps, Jenkins, JIRA/Xray or Zephyr, and GitHub, GitLab, or Bitbucket.
  • Knowledge of CI/CD pipelines, regression dashboards, and quality gate implementation.
  • Experience in regulated industry environments such as Banking, Financial Services, Healthcare, Insurance, or Life Sciences.
  • Understanding of data privacy, PII/PHI handling, explainability, and AI governance practices.
  • Bachelor’s degree in Computer Science, Engineering, or a related field is preferred; a master’s degree is preferred.
  • Strong leadership and mentoring capabilities; excellent verbal and written communication; client-centric and solution-oriented approach; stakeholder management and collaboration; ownership and proactive problem-solving; ability to work in fast-paced consulting environments; interest in emerging AI testing methodologies and tools.

Responsibilities

  • Own end-to-end quality assurance for AI/GenAI engagements, including test strategy, planning, execution, defect management, reporting, and exit criteria.
  • Lead and mentor QA pods of three to six engineers across automation frameworks, AI testing practices, and delivery execution.
  • Design and execute functional, API, backend, integration, regression, performance, and AI evaluation test cases.
  • Build and maintain scalable automation frameworks for web, mobile, and API testing.
  • Drive continuous quality improvements through reusable accelerators, frameworks, datasets, and evaluation tools.
  • Define AI-specific test coverage including prompt validation, hallucination detection, groundedness checks, retrieval quality, tool-call correctness, latency, bias, and safety testing.
  • Design and execute testing strategies for RAG architectures, multi-agent workflows, OCR systems, and Speech-to-Text applications.
  • Build synthetic datasets covering edge cases, adversarial scenarios, persona-based workflows, and multimodal inputs.
  • Implement LLM-as-a-Judge evaluation pipelines using customizable scoring rubrics and multi-model evaluations.
  • Produce AI quality scorecards and benchmarking reports across accuracy, relevancy, faithfulness, and safety metrics.
  • Integrate AI evaluation and automation testing into Jenkins and Azure DevOps CI/CD pipelines.
  • Build regression dashboards, quality gates, and automated reporting frameworks.
  • Perform API testing, backend validation, database testing, and event-driven workflow validation.
  • Conduct load and performance testing using performance engineering tools.
  • Support cross-browser, parallel, and scalable automation execution environments.
  • Act as the client-facing QA lead for enterprise engagements.
  • Conduct quality reviews, risk assessments, and stakeholder reporting sessions.
  • Translate technical quality metrics into business-focused outcomes and recommendations.
  • Collaborate with engineering, AI/ML, product, and leadership teams to support successful delivery outcomes.

Desired Qualifications

  • Exposure to Power Platform tools including Power Apps, Power BI, and Power Automate.
  • Certifications such as ISTQB Advanced, Certified Agile Tester, AWS/Azure AI Certifications, or equivalent.
  • Experience with adversarial testing and responsible AI frameworks such as NIST AI RMF or ISO 42001.
  • Exposure to MLOps/LLMOps observability and production monitoring.
  • Contributions to QA or AI communities through blogs, speaking sessions, or open-source initiatives.

About the Company

Company size
Not listed
Company stage
Not listed
Total funding
Not listed
Headquarters
Not listed
Founded
Not listed

Recently Posted Jobs

Explore more opportunities at Delphi Consulting

See more jobs