Software Engineer

NVIDIA ships a router that outlasts the model beside it. Models & Agents A 30 billion parameter open model and an open source router ship the same day. The router is the release enterprises will find harder to replace. By Shashi Bellamkonda · August 11, 2026 faster output speed, Nemotron 3.5 Lightning vs. class peers (NVIDIA, 2026) lower cost on LangChain's Deep Agents tasks via NeMo Switchyard (LangChain, 2026) cost reduction on Ramp's SWE-Bench suite via NeMo Switchyard (Ramp, 2026) NVIDIA released a small open model and an open source router the same day. Ten partner companies published specific cost and latency numbers for the router, not the model. That router is the layer enterprises will find harder to replace. Thirty billion parameters is the small release this week. NVIDIA's more consequential move sits one layer up, in the software that decides which model touches each request before the model itself does any work. One model among several, by design. NVIDIA expanded its Nemotron 3 family on August 11 with Nemotron 3.5 Lightning, a 30-billion-parameter mixture-of-experts model built for the narrow, repeated tasks that always-on agents run all day (NVIDIA, 2026). The company frames it as one piece inside a system of models. A frontier reasoning model, Nemotron 3 Ultra or a comparable model such as GPT-5.6, plans and orchestrates a workflow, while Lightning executes specific steps inside it: code review, tool calls, security alert triage, billing questions (NVIDIA, 2026). Lightning delivers up to four times faster output and 30% faster agentic task completion than other models in its size class, by NVIDIA's own PinchBench results (NVIDIA, 2026). CrowdStrike is customizing it for cybersecurity workloads. Harvey is pairing it with its Trajectory product for legal research. CodeRabbit built a routing layer on top of it with Baseten for code review (NVIDIA, 2026). The model ships open. Enterprises can post-train it on their own data through NVIDIA NeMo and run it locally, on NVIDIA RTX workstations, NVIDIA DGX Spark systems, or edge devices they already own (NVIDIA, 2026). The router ships with its own adoption numbers. NeMo Switchyard is the second release from the same day: an open source library that routes each step of an agent workflow to whichever model handles it best, tuned for cost, latency, or quality depending on what an enterprise sets (NVIDIA, 2026). NVIDIA's internal benchmarks show it holding frontier-level accuracy while cutting task completion cost to close to a third of running Opus 4.8 alone (NVIDIA, 2026). NVIDIA's GitHub repository still marks Switchyard pre-alpha, a status sitting next to five partners already reporting production or near-production numbers. That gap between the label and the results is worth factoring into any deployment timeline. Several partners published their own numbers alongside the release: * Boomi hit 100% domain-routing accuracy across five routing tests, sent 59% of traffic to a fine-tuned model running five times faster, and cut later-turn latency by 21% (Boomi, 2026). * LangChain cut cost 74% across 145 multi-turn Deep Agents tasks by sending only 7% of calls to a frontier model, at a 6% accuracy tradeoff (LangChain, 2026). * Ramp matched frontier model performance on its SWE-Bench suite while cutting cost 58% and runtime 33% (Ramp, 2026). * Cognition folded a staged router into Devin Desktop, holding near-frontier performance on FrontierCode Main while cutting mean cost 28% (Cognition, 2026). * Classmethod reported an early 27% cost reduction running opencode and Fireworks workloads through the router (Classmethod, 2026). Every figure above comes from a named company running production or near-production workloads. A router is harder to rip out than a model. Kong is shipping Switchyard natively inside Kong AI Gateway. LiteLLM is adding it as a plugin to its proxy layer, the piece that sits between an application and every model call it makes. LangChain built its own routing benchmark on top of it. Swapping a model checkpoint changes an API endpoint and a few configuration lines. Swapping a routing library means re-tuning the cost, latency, and quality tradeoffs across every agent workflow built on top of it, then reworking whatever gateway or proxy the enterprise wired it into. Swapping a model checkpoint changes an endpoint. Swapping a routing library means re-tuning cost and latency across every workflow built on top of it. Ramp, Cognition, and Boomi adopted more than a model this week. They adopted the layer that decides which model runs next, every time, for every agent they operate. That is a genuine efficiency win today, and a routing dependency an enterprise will carry longer than the model that shipped alongside it. Nemotron 3.5 Lightning is the visible release. NeMo Switchyard, shipped the same day, is the piece still running inside enterprise gateways after this model gets replaced. What happens next. Does NeMo Switchyard's routing logic stay open enough that an enterprise can swap NVIDIA's default algorithm without losing the Kong and LiteLLM integrations already built around it? Does the router's default configuration route evenly among competing frontier models over time, or does cost tuning start to favor Nemotron placements as usage scales? Boomi and Ramp published exact percentages this week. Do the next round of adopters match those numbers with the same specificity, or do vaguer results start showing up in six months? CIO/CTO Viability Question NeMo Switchyard is open source, but once it is wired into a gateway or proxy layer, who actually decides which model handles each agent request, and what would it cost to change that decision in twelve months? Boomi. "Why Open Model Routing Matters." Boomi Blog, 2026, boomi.com. Classmethod. "NVIDIA NeMo Switchyard, First Touch." Classmethod Developers Blog, 2026, classmethod.jp. Cognition. "FrontierCode Main." Cognition Blog, 2026, cognition.com. LangChain. "Switchyard Agent Routing Benchmark." LangChain Blog, 2026, langchain.com. NVIDIA. "NVIDIA Nemotron 3.5 Lightning and NeMo Switchyard Deliver Faster, Smarter, More Efficient Agentic AI." NVIDIA Blog, 11 Aug. 2026, blogs.nvidia.com. Ramp. "Ramp SWE-Bench." Ramp Labs, 2026, labs.ramp.com. About Shashi Bellamkonda Shashi Bellamkonda is a Principal Research Director at Info-Tech Research Group, where he covers marketing technology, artificial intelligence platforms, customer experience, and collaboration tools. He is an Entrepreneur in Residence at Stony Brook Univerisity's College of Business. He was an Adjunct Professor at Georgetown University for over 14 years, host of the Talking Headless Show on LinkedIn Live, and brings more than 25 years of technology marketing experience to his research and advisory work. Disclaimer: This blog reflects my personal views only. Content does not represent the views of my employer, Info-Tech Research Group. AI tools may have been used for brevity, structure, or research support. Please independently verify any information before relying on it.

Bolna Open Source: build vs. Buy for voice AI in 2026. Dhiraj · 7 August 2026 · Updated 7 August 2026 Founder of Bolti, writing about voice AI for Indian businesses. Bolti, a voice AI platform for phone agents, provides production-grade telephony infrastructure starting with a ₹6/min pay-as-you-go pricing model and 50 free minutes. If you are exploring "bolna open source" frameworks to build, deploy, or customize conversational voice bots, you need to understand how open-source LLMs translate to real-time phone calls. While open-source frameworks give you code-level control, running a voice pipeline in production requires solving critical latency and telephony challenges. This guide explains how to evaluate open-source voice architectures, when to build with open-source frameworks, and how to use open-weights models like DeepSeek and Llama 4 inside a managed, low-latency environment. What is the architecture of a bolna open source voice agent? An open-source voice agent must orchestrate three distinct pipelines with sub-second latency to feel like a natural human conversation. To build a voice agent using open-source tools, your architecture must continuously run this loop: * Speech-to-Text (STT): Transcribes the caller's incoming audio stream. * Large Language Model (LLM): Processes the transcript, maintains conversation state, and generates the next text response. * Text-to-Speech (TTS): Synthesizes the generated text response back into an audio stream. In a fully self-hosted or open-source stack, you have to manually configure, host, and optimize each component. If your LLM takes more than 1.5 seconds to respond, the caller will experience awkward silences and likely hang up. The voice pipeline components. To achieve natural conversation, your open-source setup needs more than just basic transcription and synthesis. You must integrate: * Voice Activity Detection (VAD): To instantly detect when the caller starts and stops speaking. * Turn Detection: To decide exactly when it is the agent's turn to speak. * Interruption Handling: To immediately stop the agent's audio stream the moment the caller speaks mid-sentence. * Telephony Noise Cancellation: To strip out background noise and line static before the audio hits your STT model. How do you run open source LLMs for voice? Running open-weights models for voice requires highly optimized inference engines because voice is completely unforgiving of latency delays. While closed APIs are easy to plug in, open-weights models on dedicated infrastructure can be 5-10x cheaper at scale. They also keep your prompts off general APIs and allow you to use domain-specific fine-tunes. Instead of managing complex GPU clusters, vLLM instances, and cold starts yourself, Bolti lets you run open-source LLMs as first-class providers via its integration with Baseten. This gives you the cost and data-control benefits of open source without the infrastructure overhead. Supported open-source models. You can select these production-ready open-weights models directly from your agent settings: * DeepSeek-V3.1 (671B MoE): Excellent for strong reasoning, complex tool-calling, and multilingual conversations. This is often the best balance of quality and cost for voice. * Llama-4-Maverick-17B (128E MoE): Optimized for ultra-low latency conversational agents. It has a very long context window (1M tokens) and excellent instruction-following capabilities. * Qwen3-235B-A22B: The highest-quality open model in the suite, best used when complex reasoning capability is more important than raw speed. By leveraging Baseten's infrastructure through Bolti, you get sub-second cold starts, speculative decoding, and FP8 weight optimizations out of the box. You only pay for active GPU time, meaning your agents scale to zero during off-hours. Build vs. Buy: Should you self-host your voice stack? Deciding whether to build your entire stack with open-source libraries or use a managed platform depends on your engineering bandwidth and scale. Here is a direct comparison of self-hosting an open-source voice agent versus using Bolti's managed infrastructure: | Feature / Requirement | Self-Hosted Open Source Stack | Managed Platform (Bolti) | | Inference Latency | High setup effort to optimize TTFT under 150ms | Sub-second turn-taking built-in | | Interruption Handling | Requires complex custom websockets & audio buffering | Native, real-time interruption handling | | Telephony Integration | Manual SIP trunking and WebRTC configurations | BYOC (Twilio, Exotel, Plivo) or native numbers | | Scaling | Must manage autoscaling GPU clusters | Unlimited concurrent calls per agent | | Pricing | High upfront infrastructure & engineering cost | ₹6/minute pay-as-you-go | If you have a dedicated DevOps team capable of tuning low-level WebSockets, optimizing CUDA kernels, and managing SIP trunks, building on a raw open-source framework is viable. For teams that want to focus on business logic, prompts, and integrations, a managed platform is significantly faster to deploy. How to implement open source models in your business workflows. Once you have selected your model, you can deploy it across various business workflows. For example, you can use these open-source models to run automated HR screening funnels. With Bolti, you can build an HR screening agent that uses open-weights models to call candidates, conduct initial interviews, and parse resumes. Here is how you set up a workflow using an open-weights model: * Create a Role: Define the job description and custom questions in your dashboard. * Configure your Agent: Select your preferred open-source model (like Llama 4 or DeepSeek) and link your system prompt using template variables like {{candidate_name}} and {{jd_text}}. * Upload Candidates: Upload resumes to automatically parse details and initiate outbound calls. * Review Results: View structured call transcripts and analysis generated by your open-source model. Set up your first open source voice agent. Deploying open-weights models for real-time voice calls does not require managing complex GPU pipelines or writing low-level websocket code. With Bolti, you can configure a production-ready agent running DeepSeek or Llama 4 in under 10 minutes. Sign up today to get 50 free minutes of call time, or scale your production calling infrastructure with its transparent pricing plans starting at just ₹6/minute. Frequently asked questions. Does Bolti support open-source LLMs? What is the pricing for running open-source models on Bolti? Can I bring my own SIP trunk when using open-source models? How does Bolti handle interruptions with open-source models?