Software Engineer (AI Quality & Reliability)
Taxo
Software Engineer (AI Quality & Reliability)
Taxo
AI-powered healthcare admin automation platform
Software Engineer (AI Quality & Reliability)
Location: New York City | In-office
“Leading healthcare into the agentic era.”
Every day, patients call their clinic to schedule an appointment, ask a question, or request a prescription refill. Behind each conversation is a busy healthcare team coordinating people, information, and systems.
Taxo builds Voice AI agents that take on this administrative work. Our agents connect with EHR and practice management systems to help teams handle patient intake, complex scheduling, and refill requests, with reliable workflows and clear handoffs to people.
As our Software Engineer (AI Quality & Reliability), you’ll build the automated evaluation and regression systems that show whether Taxo is ready to release. You’ll turn product acceptance criteria and real failure modes into repeatable tests across voice agents, application workflows, and integrations. This is a software engineering role focused on test infrastructure, independent quality evidence, and reliable releases.
Build the Evaluation System: Own shared evaluation frameworks, test harnesses, and versioned datasets using synthetic or appropriately authorized, privacy-safe data. Make runs repeatable and results easy to investigate.
Test the Whole Workflow: Build automated end-to-end regression coverage across conversations, tool calls, application behavior, and integration boundaries, including failures, recovery, and human escalation.
Measure AI Behavior: Evaluate task completion, tool-use correctness, latency, and conversation quality. Combine deterministic checks with calibrated model-assisted scoring and appropriate human review.
Make Releases Reviewable: Produce release-readiness evidence, maintain quality dashboards, and coordinate quality gates with product and engineering. Surface regressions, coverage gaps, and unresolved risks before launch.
Turn Failures into Tests: Reproduce defects, isolate failure patterns, and add durable regression cases. Engineering service owners retain component tests, production instrumentation, on-call response, and fixes; you independently verify the results.
Improve the Test System: Integrate automated checks into CI/CD, reduce flaky tests, and keep test environments and fixtures useful as the product evolves. Partner with domain reviewers on healthcare-specific cases and escalate questions that require clinical judgment.
The Test Engineer: Strong software engineering skills in Python or TypeScript and experience building maintainable test automation, reusable fixtures, and CI pipelines.
The AI Evaluator: Practical experience evaluating LLM-powered applications, agents, or voice systems, including nondeterministic behavior and the limits of automated scoring.
The Systems Thinker: Experience testing APIs, asynchronous workflows, and distributed systems, with the ability to trace a failure across service boundaries.
The Investigator: A habit of turning ambiguous failures into reproducible cases, actionable evidence, and reliable regression coverage.
The Quality Partner: Clear communication about risk and release readiness, with sound judgment around sensitive data and human review.
Bonus: Experience with streaming audio, telephony, healthcare workflows, EHR integrations, or reliability testing in privacy-sensitive products.
Healthcare teams should be able to spend more time on patients and less time on administration. Help us build the products and company that make that possible.
🧠 Practical AI Work: Build evaluation systems for real-world AI behavior across healthcare workflows.
🏥 Impact That Matters: Help catch failures before they disrupt patients or clinic teams.
🛠️ Technical Ownership: Shape shared evaluation and test infrastructure that improves every release.
Base salary: $130,000–$170,000 USD annually
Equity: 0.10%–0.25%
Benefits: Health coverage
Taxo is a small, fast, high-conviction team. We care deeply about the problem, move with urgency, and take ownership of the work. We collaborate closely, make practical decisions, and keep improving through what we learn from customers.
If you want to build the engineering systems that make healthcare AI more dependable, we’d love to hear from you.
Taxo.ai provides AI-powered automation for healthcare administration, serving providers and payers. Its Apex engine sits on top of existing large language models to extract data from EHRs and unstructured sources like PDFs and faxes, then automates workflows such as patient intake, benefits verification, prior authorizations, medical coding, claim submissions, and denials/appeals for providers, as well as claim intake, eligibility verification, adjudication, denial management, and analytics for payers. It differentiates itself with a model-agnostic meta-layer, HIPAA and SOC2 compliance, and explainable, auditable AI decisions that reference source documents, along with a claimed 98% pre-authorization rate. The company aims to greatly reduce administrative burden and processing costs in healthcare by speeding up claims handling and improving efficiency, with growth supported by seed funding from Y Combinator, General Catalyst, and Character.
Explore more opportunities at Taxo