AI QA Engineer

About Us:

At apexanalytix, we’re lifelong innovators! Since the date of our founding nearly four decades ago we’ve been consistently growing, profitable, and delivering the best procure-to-pay solutions to the world.   We’re the perfect balance of established company and start-up.  You will find a unique home here. 

And you’ll recognize the names of our clients. Most of them are on The Global 2000. They trust us to give them the latest in controls, audit and analytics software every day.  Industry analysts consistently rank us as a top supplier management solution, and you’ll be helping build that reputation.

Read more about apexanalytix - https://www.apexanalytix.com/about/

Quick Take

As an AI QA Engineer, you’ll drive the quality, reliability, and safety of the AI solutions we deliver to our Global 2000 clients, including GenAI and agentic applications, RAG pipelines, and ML models used across our products. This role is for an engineer with a solid data science/AI background who understands how models behave and is skilled in LLM evaluation, end-to-end AI testing, and ML model validation, helping us ship production-grade AI with confidence.

The Work

·        Own the test strategy for our AI products, covering LLM features, agentic workflows, RAG pipelines, and ML models.

·        Write and execute test plans, cases, and reports; manage the defect lifecycle; and perform exploratory testing of AI features.

·        Define acceptance criteria and quality metrics for non-deterministic outputs (accuracy, groundedness, hallucination rate, relevance, cost) using LLM-as-a-judge and human evaluation.

·        Build and maintain golden datasets, ground-truth labels, and regression suites with subject-matter experts, reflecting real scenarios and edge cases.

·        Develop automated evaluation pipelines (DeepEval, Ragas, promptfoo, LangSmith) that run in CI/CD on every prompt, model, retrieval, or code change.

·        Test agentic behavior—multi-step planning, tool calling, integrations, error handling, and fallbacks—to verify agents’ complete tasks correctly and stay within guardrails.

·        Validate ML models and data pipelines for data quality, segment performance, bias/fairness, and drift, using statistical methods to compare versions.

·        Perform adversarial and safety testing: prompt injection, jailbreaks, PII leakage, and compliance checks.

·        Monitor production AI quality through observability tooling, triage degradations, lead root-cause analysis, and partner with Data Science, AI, and engineering teams on fixes and release sign-off.

The Must-Haves

·        B.S./M.S. in Computer Science, Data Science, ML/AI, Statistics, or related field.

·        2–3 years in AI/ML, data science, or AI-focused QA, including 1+ year of hands-on test automation, ideally for production ML systems.

·        Working knowledge of ML model building, evaluation metrics, and LLM behavior (prompting, context, RAG).

·        Experience evaluating GenAI applications (RAG pipelines, agentic workflows, function calling) with DeepEval, Ragas, or similar.

·        Strong Python with PyTest, API testing, and CI/CD integration.

·        Grounding in statistics and experimentation (sampling, significance testing, model A/B comparison), plus SQL and data validation tools such as Great Expectations or Evidently.

·        Experience testing document and image analysis solutions that extract insights from unstructured data.

·        Ability to define pass/fail criteria and rubrics for non-deterministic outputs and translate business requirements into measurable quality standards.

·        Clear written and verbal communication with technical and business stakeholders across time zones.

The Good-to-Haves

·        Synthetic test data generation, including privacy-safe datasets, edge cases, and adversarial scenarios.

·        Docker, Kubernetes, and MLOps tooling for deploying and testing AI applications.

·        Familiarity with AI risk and governance frameworks (NIST AI RMF, ISO/IEC 42001, EU AI Act) and responsible AI practices (OWASP Top 10 for LLMs, bias testing, PII protection).

·        Working knowledge of LangChain, LangGraph, Hugging Face, or similar, plus LLM APIs (Anthropic, OpenAI, open-source models).

·        UI and end-to-end test automation (Playwright, Selenium) for AI-driven interfaces.

Over the years, we’ve discovered that the most effective and successful associates at apexanalytix are people who have a specific combination of values, skills, and behaviors that we call “The apex Way”. Read more about The apex Way - https://www.apexanalytix.com/careers/

 
Benefits

At apexanalytix we know that our associates are the reason behind our successes. We truly value you as an associate and part of our professional family. Our goal is to offer the very best benefits possible to you and your loved ones. When it comes to benefits, whether for yourself or your family the most important aspect is choice. And we get that. apexanalytix offers competitive benefits for the countries that we serve, in addition to our BeWell@apex initiative that encourages employees’ growth in six key wellness areas: Emotional, Physical, Community, Financial, Social, and Intelligence.

With resources such as a strong Mentor Program, Internal Training Portal, plus Education, Tuition, and Certification Assistance, we provide tools for our associates to grow and develop.