Software Development Engineer in Test 3 – Model Evaluation

The Role

We are looking for a dedicated ML Model Evals & Benchmarking Engineer to bridge the gap between machine learning research and high-performance production systems. In this role, you will own the end-to-end evaluation lifecycle, designing rigorous offline and online evaluation frameworks, defining robust model performance metrics, and architecting the scalable infrastructure needed to benchmark throughput, latency, resource utilization, and statistical accuracy across diverse ML/AI models.

How You’ll Make an Impact:

• Own model quality before production: Establish standard evaluation frameworks using statistical metrics (e.g., Precision, Recall, F1-Score, ROC-AUC) tailored to specific business domains. Build validation frameworks for computer vision models (preferably satellite or geospatial). Catch regressions in statistical metrics before they reach customers.

• Build version-aware model regression: Set up pipelines that compare inference outputs across model versions on curated golden datasets. Every release should come with a clear, data-backed go/no-go signal.

• Make data quality a first-class gate: Validate training and evaluation datasets for label quality, class balance, geographic and seasonal coverage, leakage, and drift. Ensure models are tested on data that reflects real-world conditions.

• Validate inference at scale: Design, build, and scale distributed evaluation pipelines and testing harnesses capable of running large-scale batch and real-time model benchmarks.

• Harden the platform end-to-end: Benchmark model inference servers for throughput, latency (p50, p99), and hardware efficiency (GPU/CPU/Memory footprints) under heavy concurrent loads.
  
  
• Software Engineering & Scripting: Advanced proficiency in Python (NumPy, Pandas, Scikit-learn, PyTorch/TensorFlow evaluation utilities) for building custom metric calculators and test harnesses.
  
  
• Distributed Compute & Execution: Experience orchestrating heavy evaluation jobs using frameworks like Ray, Apache Spark, or Kubeflow.

• Raise the bar for the team: Strengthen the shared automation frameworks and set standards for maintainable, non-flaky tests. Show how tools like Cursor and Claude can speed up test authoring, triage, and reporting while staying under human review.

• Influence release decisions: This is a high-visibility role. Your quality signals will directly shape what ships across Aidash’s product lines

What we’re looking for:

We are looking for a senior QA engineer with deep backend automation experience who can validate ML models and backend systems end-to-end, and who actively uses agentic AI development tools (e.g., Cursor, Claude) to accelerate test design, automation, debugging, and reporting — without compromising fundamentals, coverage, or engineering quality.

• 8–10 years in software testing and test automation, with strong ownership of backend system quality

• Experience testing ML applications (model/service behavior, data quality, inference pipelines, evaluation/regression of model-backed features)

• Broader stack familiarity: observability (logs/metrics/traces), cloud (AWS), contract testing, data-pipeline validation

• Experience improving shared automation frameworks and raising team standards for maintainable tests

• Solid grasp of testing fundamentals and best practices (test design, risk-based coverage, isolation, maintainability, flaky-test control, clear defect/RCA discipline)

• Strong coding ability in Python, with willingness to learn the other (or additional languages) as needed.

• Good understanding of backend system design and how services interact in distributed systems

• Practical exposure to databases — SQL and/or NoSQL (at least one is mandatory)

• Working knowledge of application deployment and underlying orchestration (e.g., MLflow, Kubeflow, containers, Kubernetes, service rollout/health concepts)

• CI/CD pipelines — must have (Jenkins/GitHub Actions/Bitbucket Pipelines or equivalent): triggering suites, reading failures, improving pipeline reliability

• Performance testing — must have (JMeter/k6/Locust or equivalent): load/benchmark basics, interpreting results, flagging SLA/risk

• Exposure to agentic development tools such as Cursor and Claude (or similar) for coding, test authoring, log/triage assistance, and documentation — used with human review and quality gates

We are proud to be an equal-opportunity employer. We are committed to embracing diversity and inclusion in our hiring practices, and we promote a work environment where everyone, from any race, color, religion, sex, sexual orientation, gender identity, or national origin, can do their best work.