Introducing Integration-Bench: an AI integration benchmark for real API work. Ramie R and Pasan Gimhana August 19, 2026 Can AI coding agents build integrations that stay correct against a live API? HeyMilo AI gave AI models the same 50 integration tasks against live APIs, then graded them from what the vendor actually recorded: request logs, final state, and whether the data ended up correct. The best model scored 67.7 out of 100. The weakest scored 15.1. Eight tasks scored zero for every model. Today HeyMilo AI is releasing Integration-Bench, the benchmark that produced those numbers. Why a passing test suite proves nothing here. Most coding benchmarks ask an agent to edit a repository until a local test suite goes green. Integration work does not fit inside that setup, because the hard part lives in another company's system. When HeyMilo AI connect HeyMilo to an applicant tracking system, writing the HTTP request is the easy part. The hard part is keeping a correct view of what happened: whether a write committed, whether an event was already processed, and whether a retry created duplicates. A vendor can document pages as zero-indexed while the live API starts at one, or return HTTP 200 with an error buried in the body. Clean code can pass every local test and still corrupt a customer's pipeline. That is why HeyMilo AI built the benchmark around a live vendor instead of a mock. How a task works. Each task gives the agent a ticket, a starter repository, and the documentation an ordinary integrator would get. The agent builds a connector and runs it against one of 15 fictional vendor services, each with its own auth flow, pagination, webhook behavior, rate limits, and failure modes. Some documentation is accurate. Some is stale. The vendors are invented on purpose, so there is no SDK or forum thread for a model to recall. After the agent finishes, a trusted verifier outside the workspace checks final state, submitted output, and the vendor's traffic log. The question is whether the data ended up correct, not whether the code looks plausible. The leaderboard. Claude Fable 5 leads at 67.7 under Claude Code, with GPT-5.6 Sol on Codex and DeepSeek V4 Pro on OpenCode close behind. Every model ran at its top reasoning effort, so these are not under-powered runs. The suite average across 850 graded outputs is 55.4. What matters most is the ceiling. No model cleared 68 out of 100 on work a competent integration engineer finishes routinely. Analysis. Three patterns explain most of the missing points, and none of them are about writing code. Webhooks broke everything. The difference is who controls timing. Polling connectors decide when to ask what changed. Webhook connectors receive events whenever the vendor sends them, and have to verify authenticity, handle duplicates, tolerate reordering, tell an accepted write from a committed one, and reconcile events that never arrived. Miss any one of those and an otherwise solid connector still fails. Here is a concrete example. The agent verified the signature, wrote the candidate update, and returned 200. That first pass looked clean. When the vendor redelivered the same event a few minutes later, the connector treated it as new work and created a second record. The code looked finished. The vendor log showed the duplicate. Eight tasks no model solved. Eight of the 50 tasks scored zero for every model, across all 17. Each one required coordinating delayed state, event delivery, and systems that disagreed about what had happened. Syntax and framework knowledge were rarely the deciding factor. Availability is part of capability. Gemini 3.7 Flash scored 64.0 on the 28 tasks its provider completed, which would have placed it near the front. Provider failures killed the other 22, dropping it to 35.9 across all 50. HeyMilo AI report both numbers, because an agent that cannot finish does not ship the integration. What this means. Models were able to mostly write the connector code, as they authenticated, paginated, and handled the happy path. However, even the leading model still left 32 points on the table, and those misses showed up outside the repository: a duplicate created on redelivery, a watermark that skipped a row, a write treated as committed before the vendor had committed it. The hard part is verification. Event-driven code is not especially hard to write. What is hard is knowing whether it worked, because the evidence lives in another system's state and traffic long after the function returns. Idempotency, reconciliation, and recovery belong in the definition of correctness once a second system is involved. That is how HeyMilo AI build ATS integrations at HeyMilo, and Integration-Bench is how HeyMilo AI check ourselves. Because the environment is replayable, HeyMilo AI can change a prompt, a tool, or a page of documentation and measure whether the integration actually got better. Get Integration-Bench. The full benchmark, leaderboard, and technical report will publish shortly, including the complete model results and the scoring methodology behind them. To see how this shapes the integrations inside HeyMilo, book a demo and bring the requirement you expect to break first. Integration-Bench measures whether AI coding agents can build integrations that survive contact with a live API. * HeyMilo AI ran models across the same 50 tasks: 850 graded model-task outputs. * The best model scored 67.7 out of 100. The weakest scored 15.1. * Webhook tasks averaged 20 while polling tasks averaged 58. * Eight tasks scored zero for every model. * The full leaderboard and methodology land in the technical report. HeyMilo's engineering and research leads, writing on agent evaluation, integration reliability, and how HeyMilo AI measure whether AI systems can do real work. Found this helpful? Share it with your network See how HeyMilo connects to your ATS. Book a demo and bring your messiest integration requirement. HeyMilo AI will walk through how the connector behaves when your data is not clean. Recruiters in the loop. Guides, insights and hiring tips for modern recruiters.

HeyMilo AI, an agentic recruiting platform for high-volume hiring, has raised $6 million in funding led by Category Ventures, with participation from Canaan Partners, Alumni Ventures and ERA. The company has screened over one million candidates for clients including Randstad, WilsonHCG and Neo Financial. Founded in 2023, HeyMilo builds structured screening directly into job applications through voice, video, phone, SMS and resume agents. Recruiters receive rubric-scored results and transcripts integrated into their existing applicant tracking systems. The startup is launching three new agent types: AI Scenario Assessment, which tests candidates' ability to use AI tools; Candidate Recommendations, which resurfaces previously screened applicants; and Notetaker, which captures structured notes from recruiter-led interviews. The platform aims to address the growing mismatch between AI-powered candidate applications and traditional recruiting processes.