Product Engineer - Agent Experience

The Role

We are hiring a Product Engineer to own the quality of our Agents end to end. Carly delivers personalized benefits guidance to employees, grounded in plan designs, coverage rules, and each employee's context. Ava is the agentic intelligence that benefits teams use to run their benefits operations. In this role, you will review evals, collect qualitative and quantitative feedback, define what a high-quality outcome looks like, ship improvements, and measure their impact.

The role combines product judgment with hands-on implementation. You will own decisions such as whether a problem calls for a prompt change, a new skill, or a code change; whether an evaluation is rigorous enough to trust; and whether a new model is ready for production. You will then implement those changes yourself across evaluations, prompts, skills, model configuration, and our TypeScript codebase, using AI development tools as a core part of your workflow, and partner with our agent engineers on deeper changes to retrieval, tooling, or the platform.

You will join our Agent Experience team, embedded with agent engineering and working closely with product leadership, design, and Customer Success. We welcome candidates from either direction: product-minded engineers, and engineering-minded product managers who are already building. We place more weight on what you have shipped and how quickly you learn than on years of experience, and we are open to candidates early in their careers who show high agency, curiosity, and sound judgment.

Success in this role means evaluation results consistently translate into shipped improvements, model and prompt changes are validated against representative customer tasks, recurring failures become regression cases, and the team can clearly explain how each change affects usefulness, reliability, latency, and cost.

What You'll Do

• Define what good looks like for our agents. Review production evals, customer feedback, usage patterns, and evaluation results. Find where an answer or action falls short, including cases where it's factually correct but still doesn't help the person asking.
• Build evaluations the team can trust. Turn customer needs into representative scenarios, clear rubrics, and regression checks. Test whether automated scores agree with human judgment, and use the results to decide what to fix first.
• Ship improvements yourself. Update prompts, skills, and agent behavior. Build prototypes and targeted product changes, and contribute reviewed code. Partner with engineers when the problem sits in retrieval, tools, or the underlying platform.
• Make the call on new models. Compare models against real tasks for quality, safety, latency, and cost. Decide with engineering whether we upgrade, run controlled rollouts, inspect production results, and follow through on regressions.
• Own the feedback loop with customers. Work with CX and product teammates to understand what people are trying to get done. Turn recurring friction into product changes, then check whether those changes improve task completion, engagement, and trust.
• Use AI tools as a core part of how you work. Build with AI coding tools and agents every day, and be ready to explain how you checked what they produced.
• Keep the team current. Try new AI tools and approaches, form a view on where they help, and share what you learn. Turn the experiments that work into changes that hold up in real use, with clear evidence and appropriate guardrails.

What We're Looking For

• Experience shipping digital products in product management, product engineering, software engineering, or a similarly hands-on role. You can point to customer outcomes you personally improved.
• Strong product judgment. You can look at an agent interaction, explain what the user needed, and tell a useful result from one that only looks good in a demo or scores well.
• Active, practical use of AI coding tools and agents. You've built prototypes, tools, or product changes with them and can explain how you checked their behavior and handled mistakes.
• Enough coding fluency to read an unfamiliar codebase, make targeted changes, use version control, and contribute reviewed code. You're comfortable learning our TypeScript stack.
• Comfort with data and experiments: querying or scripting an analysis, defining metrics that mean something, and recognizing when a sample or automated evaluation gives false confidence
• Systems thinking and customer empathy. You spot where a skill, an automation, or a workflow change would remove friction, and you can talk with customers about it.
• High agency in an ambiguous environment. You pick a useful next step, ship, learn quickly, and communicate clearly with engineers, designers, and customer-facing teams.

Nice to Have

• Experience with LLM evaluations, prompt and context design, or agent tools and orchestration
• Running controlled experiments on a live product
• Working knowledge of SQL and TypeScript
• Familiarity with benefits, healthcare, HR technology, or enterprise software

What to Expect from Us

Interview Process:

• Initial screen with our recruiters
• Video Interview with our Engineering Manager
• Technical interview focused on real-world problem solving (no whiteboard algorithms)
• On-site interview loop:
• Team behavioral interview (60 min)
• Session with our Engineering Manager (60 min)
• Final conversation with our CTO (30 min)

The base salary for this role is $115,000–$165,000. Where you land in that range depends on your experience, your skills, and how well your background fits what the role needs.

We also offer:

• Equity: Stock options, so you share in what we’re building
• Health coverage: Medical, dental, and vision, with employee premiums covered by Avante
• Unlimited PTO: Take the time you need to rest and recharge, plus paid company holidays
• Retirement: 401(k)
• Commuter benefit: $150/month toward however you prefer to get to work

Avante is an Equal Opportunity Employer. We’re building a diverse, inclusive team and welcome candidates of all backgrounds.