Head of Field Engineering

Case study: Aurasell builds an 8B model for extracting information from websites outperforming Sonnet 4.5 by 8%. With Oumi's technology, Aurasell's custom AI beat Anthropic's Sonnet 4.5 at extracting information from webpages by 8% in coverage and 12% in groundedness By Stefan Webb Table of contents Problem. Aurasell is an AI-first CRM platform whose core offering includes a research agent that builds custom value pyramids for any target customer. The agent surfaces corporate objectives, strategic initiatives, and business challenges by running a fixed set of web search queries, then passes the results to an LLM that extracts and organizes the relevant details into a structured output. Two metrics define success for this task: groundedness, meaning every claim in the output must be supported by the source material, and coverage, meaning the model should capture all relevant information present in the search results. Their existing system relied on Sonnet 4.5, which achieved acceptable quality but posed significant challenges for scale. As Aurasell's customer base grew, the cost and latency of running a large frontier model on every research request became unsustainable. Solution. Oumi partnered with Aurasell to build a custom fine-tuned model optimized specifically for their information extraction task. Since Aurasell was unable to share example outputs or their full system prompt, the team took a ground-up approach to data generation and evaluation. Using the list of search queries provided by Aurasell, Oumi generated web search results for approximately 3,000 companies. A large frontier model was then used to produce 2,000 high-quality output summaries as training data. To measure performance, two custom LLM judges were created: a groundedness judge to detect hallucinations and unsupported claims, and a coverage judge to penalize summaries that missed key information. Iteration on both the evaluation framework and the training approach proved critical. Early judges measuring conciseness and clarity inadvertently rewarded shorter, less useful summaries. Replacing these with the coverage metric corrected that bias. Outcome. The final 8B fine-tuned model, built on Qwen3 8B, outperformed Sonnet 4.5 on both key metrics (see table above). The custom model also approached Opus-level groundedness while surpassing it in coverage - all at a fraction of the cost and latency. The bottom line: By moving to an 8B model, Aurasell can now run their research agent at scale, delivering faster results to sellers without compromising on quality. What's next. With Oumi's technology, Aurasell were able to build a small custom AI model that solved pressing business needs. Theirs, however, is just one use case. Many other enterprises are discovering the benefits of custom AI models and the ease and economy of model development that the Oumi Platform makes possible. Why not try it out today and see for yourself? You only need to come with your prompt and the Oumi Agent will take it from there!

Case study: Ada builds a real-time guardrail for their agents with 50% fewer false positives. With Oumi's technology, Ada's custom guardrail model beat GPT-4.1 Mini by 4% in accuracy while slashing latency and false positives rate By Stefan Webb Table of contents Problem. Ada builds AI-powered customer service agents deployed across industries including fintech, e-commerce, SaaS, and travel. For every deployment, each action the agent takes must strictly align with the customer's operating policies - any deviation risks damaging the customer relationship. To enforce this, Ada runs a real-time adherence check: for every proposed agent action, a guardrail model evaluates the action against the customer's standard operating procedure (SOP) and classifies it as "adherent" or "non-adherent." Because this check runs live during customer conversations, latency is critical. Ada's existing guardrail relied on GPT-4.1 Mini. While reasonably accurate at 87.5%, the model introduced unacceptable latency for real-time use. Ada also wanted greater ownership and control over the model powering this critical system. Solution. Ada partnered with Oumi to develop a custom fine-tuned guardrail model optimized for policy adherence classification. Since Ada had only around 20 labeled examples initially, the team built a fully synthetic data pipeline. Using Oumi's platform, they generated 250 SOPs spanning Ada's priority domains, varying attributes like complexity and tool count to ensure diversity. From these, they synthesized thousands of agent conversations with controlled variation across customer clarity, conversation length, and adherence outcomes. This data was used to fine-tune Qwen3 4B, a compact language model well-suited to the latency requirements of Ada's real-time pipeline. Outcome. The fine-tuned 4B-parameter model outperformed GPT-4.1 Mini across all key metrics (see table above). The bottom line: By moving from a general-purpose hosted model to a purpose-built fine-tuned LLM, Ada gained better accuracy, lower latency, and full ownership of a core component in their AI agent infrastructure. What's next. With Oumi's technology, Ada were able to build a small custom AI model that solved pressing business needs. Theirs, however, is just one use case. Many other enterprises are discovering the benefits of custom AI models and the ease and economy of model development that the Oumi Platform makes possible. Why not try it out today and see for yourself? You only need come with your task prompt and the Oumi Agent takes it from there!

Case study: DMG achieves 6% higher quality and 100x lower costs for invoice validation. With Oumi's technology, DMG's custom AI beat GPT5.2 by 6% accuracy and 6% validity at 100x lower cost By Stefan Webb Table of contents Problem. Divisions Maintenance Group (DMG) coordinates facility maintenance across thousands of properties, managing a shared pool of contractors - plumbers, electricians, handymen - who submit invoices for reimbursement after completing jobs. Each invoice must be validated on two dimensions: validity (correct formatting and categorization) and appropriateness (reasonable charges given the job scope, profession, and pricing norms). At scale, manual review was labor-intensive, slow, and expensive, and DMG's existing automated approaches were underperforming - validity accuracy sat at just 72%, with no reliable baseline for appropriateness at all. Solution. DMG partnered with Oumi to fine-tune an ultra-small language model (Qwen3-0.6B) purpose-built for invoice classification. The small footprint was critical: DMG's volume demands and interest in edge deployment ruled out large proprietary models for production use. "Every job we handle is bespoke - even the same HVAC unit breaking down twice runs differently. I'm convinced our future is to have our own fine-tuned models. The results have only gotten better." - Kumar Srinivasan, Chief Product Officer The core challenge was data. DMG provided ~10k unlabeled invoice examples covering only plumbing, but the model needed to generalize across plumbing, electrical, and handyman domains. Oumi built a synthesis recipe to generate labeled training data, including entirely new examples for the missing domains. Random real invoice samples were leveraged as few-shot formatting examples within each synthesis prompt to increase sample diversity. Outcome. The small, specialized, fine-tuned 0.6B model developed with Oumi's technology delivered dramatic improvements across both tasks over the pre-fine-tuned model, improving on validity from 72% to 99%, and on appropriateness from 52% to 91%. Importantly, it also exceeded the large, general-purpose frontier GPT5.2 by 6% on both metrics (see table above). For context, Kimi K2 (a much larger model) achieved only marginally higher agreement at 95% and 87%. The bottom line: A sub-1B fine-tuned model exceeded frontier-scale performance on this task, enabling DMG to move toward fully automated invoice processing with high reliability. What's next. With Oumi's technology, DMG were able to build a small custom AI model that solved pressing business needs. Theirs, however, is just one use case. Many other enterprises are discovering the benefits of custom AI models and the ease and economy of model development that the Oumi Platform makes possible. Why not try it out today and see for yourself? You only need to come with your prompt and the Oumi Agent will take it from there!