N

Newfold Digital

Provides customized web presence solutions

Software Development Engineer in Test

Test Automation EngineeringConfirmed live in the last 24 hours
Full-timeNo salary listedRemote

About the Job

Requirements

  • Strong programming experience in Python.
  • A strong understanding of software testing and quality engineering principles.
  • Hands-on experience with AI/LLM testing, prompt engineering, context engineering, AI evaluations, RAG systems, and AI agents.
  • Experience with CI/CD pipelines, Docker and Kubernetes, cloud platforms (AWS, Azure, OCI, or GCP), API testing and automation, microservices testing, and UI automation.
  • Experience with SQL and data validation.
  • Strong debugging, analytical, and troubleshooting skills.
  • Experience working in Agile/Scrum environments.
  • A Bachelor’s or Master’s degree in Computer Science, Engineering, AI/ML, or a related field is preferred; equivalent hands-on experience and demonstrable projects are valued.

Responsibilities

  • Design, validate, and optimise testing strategies for AI-powered applications, copilots, chatbots, RAG pipelines, and autonomous AI agents.
  • Develop AI evaluation frameworks for hallucination detection, groundedness, toxicity and bias validation, response quality, context relevance, prompt regression testing, determinism and consistency, and safety and guardrail validation.
  • Build automated validation pipelines for LLM outputs using structured assertions, semantic scoring, embeddings, and evaluation metrics.
  • Create synthetic and benchmark datasets for AI testing scenarios.
  • Test multi-agent orchestration workflows and agent memory/context retention.
  • Validate MCP integrations and tool interoperability.
  • Work on context engineering, prompt engineering, and prompt optimisation strategies.
  • Evaluate latency, token usage, model cost optimisation, throughput, and scalability.
  • Develop and maintain scalable UI/API automation frameworks.
  • Automate testing for REST APIs, GraphQL APIs, event-driven systems, streaming systems, webhooks, microservices, and MFE architectures.
  • Build reusable automation libraries, utilities, and AI-assisted testing workflows.
  • Integrate automation into CI/CD pipelines and quality gates.
  • Validate RAG architectures, including chunking strategies, embedding quality, retrieval accuracy, re-ranking, and vector search relevance.
  • Build or validate AI agents using AI frameworks and SDKs.
  • Validate AI workflows involving tool calling, function calling, agent memory, agent planning and execution, multi-modal AI systems, and structured outputs.
  • Build and maintain quality workflows integrated with CI/CD platforms.
  • Work with Docker and Kubernetes environments.
  • Perform cloud-based testing and deployments across AWS, Azure, OCI, and GCP.
  • Validate distributed systems reliability and resiliency.
  • Monitor AI system behaviour using observability platforms and logging/tracing tools.
  • Contribute to shift-left and quality engineering best practices.

Desired Qualifications

  • Experience with Robot Framework.
  • Experience with Playwright.
  • Knowledge of LangChain, LangGraph, LlamaIndex, CrewAI, Semantic Kernel, and MCP.
  • Hands-on experience with Cursor, GitHub Copilot, Claude, Gemini, and OpenAI APIs.
  • Experience with vector databases and embeddings.
  • Exposure to MLOps workflows and ML lifecycle testing.
  • Knowledge of performance testing and security testing.
  • Experience validating AI governance/compliance requirements.
  • Experience with contract testing, chaos testing, or resilience engineering.
  • Familiarity with accessibility testing and cross-browser/device testing.
  • Exposure to synthetic monitoring and production observability.
  • Experience building AI tools and side projects, prototypes, proofs of concept, or open-source contributions.
  • GitHub repositories showcasing AI/automation projects; personally or professionally built AI agents or copilots; contributions to open-source automation or AI ecosystems; blogs, talks, workshops, or technical content on AI/testing; experience integrating AI into SDLC or QE workflows; or experience with autonomous testing systems or self-healing automation.
  • AI red teaming; LLM security testing (prompt injection, jailbreaks, and data leakage); agent reliability and safety testing; AI cost optimisation; distributed systems testing; test data engineering; feature flag testing; AI observability and telemetry; reinforcement learning feedback workflows; human-in-the-loop evaluation systems; knowledge graph integrations; and multi-modal AI testing (vision, audio, and text).

About the Company

Newfold Digital

Provides customized web presence solutions

Newfold Digital helps small- and medium-sized businesses build and manage their online presence by offering a broad portfolio of web presence tools and personalized support. Customers pick hosting, domains, content management, security, email, and related services, then combine them into a customized package with setup help and ongoing guidance. It stands out by pairing a wide product suite with tailored support and customization, instead of one-size-fits-all solutions. Its goal is to deliver measurable value for customers by helping them create a successful online presence through practical, customized products and hands-on help.

Company size
1,001-5,000
Company stage
Not listed
Total funding
Not listed
Headquarters
Jacksonville, FL
Founded
2021

Recently Posted Jobs

Explore more opportunities at Newfold Digital

See more jobs