Developer Relations

Run and route AI workloads in your own cloud with Chalk Model Gateway. Sai Atmakuri Engineering Manager October 7, 2026 Production applications rarely use just one model. As teams add providers and models, model selection, fallbacks, spending limits, and provider-specific APIs become application logic. Chalk is introducing Chalk Model Gateway, an OpenAI-compatible model gateway that runs inside your Chalk deployment, in your own cloud. Chalk Model Gateway does what you'd expect from a gateway: routing, fallback, budgets, rate limits, and tracing. But, managed gateways put another company's infrastructure between your application and your models. Self-hosted gateways give your team another service to operate. Chalk Model Gateway gives you the benefits of both without either tradeoff: it runs inside your existing Chalk deployment, in your cloud. With Chalk Model Gateway: * No third-party gateway sits between your agents and the models they call. * Your provider keys stay in your environment. * If you serve open-weight models on Chalk, those requests don't leave your cloud at all. For teams with sensitive data control requirements, Chalk Model Gateway, which is deployed in your environment, can simplify security and compliance reviews. Production agents need more than a model endpoint. A few years ago, most teams building agents were validating the idea with a single model. That's changing. An a16z survey of 100 CIOs at Global 2000 companies found that 81% now use three or more model families, up from 68% less than a year earlier. As agents move into production, model choice becomes a systems problem. Wiring every provider straight into application code breaks down fast. * What happens when a provider rolls back a model or a model is unavailable? * Which requests actually need the most expensive model? * How much can each team, agent, or workflow spend? * How do you switch providers without rewriting application code? * Which model handled a particular request? * What happens when you introduce a different model and quality changes? A gateway answers these in one place. Meet Chalk Model Gateway. Your application talks to one endpoint. The gateway handles where the request goes. You bring your own keys for the providers you already use: OpenAI, Anthropic, Jev, Google Vertex, AWS Bedrock, and more. You can also route to the endpoints for any open-weight models you serve yourself with vLLM. From there, you can put policy around model traffic: * Fallbacks. Define an ordered list of backup models. If the primary model is unavailable or returns an error, the gateway tries the next one before your agent sees a failure. * Budgets and rate limits. Control token usage and request volume across API keys, providers, models, and shared usage pools. Set daily, weekly, or monthly budgets alongside tokens-per-minute, requests-per-minute, and concurrency limits. * Judge-based routing. Route by task type with a policy you configure: a local heuristic difficulty classifier, or a remote judge backed by any LLM or System One model. Start any new policy in shadow mode to see which requests it would downgrade before it touches production traffic. * Open-model routing. Serve open-weight models on GPUs in your own cloud with Chalk Compute and route to them through the same gateway as your frontier providers. * Tracing. Every request gets a trace, with controls over sampling and content retention. Errors are always exported. For example, you can easily point your existing OpenAI SDK at your Chalk host: from openai import OpenAI client = OpenAI( base_url="https://<your-chalk-api-host>/v1/router", api_key="<YOUR_ROUTER_API_KEY>", default_headers={"X-Chalk-Env-Id": "<env-id>"},) resp = client.chat.completions.create( model="openai/gpt-6-luna", messages=[{"role": "user", "content": "Hello"}],) print(resp.choices[0].message.content) Don't send every request to your most expensive model. Routing lets you pay for the capability each request needs. For years, the easiest strategy was to pick the strongest model and send everything to it. That made sense when model quality was changing quickly and volume was relatively low. At production scale, the economics are different. Here is what a typical agent workload looks like at list price, blended at three input tokens for every output token: Model. Input ($/M tokens). Output ($/M tokens). Blended, 3:1 ($/M tokens). * Frontier hosted model * Smaller hosted model * Lower-cost, efficient hosted model * Self-hosted, open-weight, fine-tuned model * 10-15% of frontier hosted model costs [1] [1] An enterprise social-dating application was running a moderation workload on OpenAI models. Their analysis found that switching to a fine-tuned, open-weight model hosted on Chalk would reduce inference costs to as little as 10-15% of their current spend, for an estimated $150k+ in annual savings. At one billion tokens per month, sending every request to a frontier model would cost roughly $8,000 at these list prices. Now imagine a routing policy that sends 70% of tokens to a lower-cost, efficient hosted model and escalates the remaining 30% to the frontier model. The cost falls to roughly $2,770, a 65% reduction. The gateway turns model choice into a policy you can change, instead of logic hard-coded into your application. The economics of open models can be compelling. But cheaper inference isn't enough. You need to know which requests can safely use a smaller model. With Chalk Model Gateway, start any new routing policy in shadow mode to see which requests it would downgrade before you commit production traffic to it. You can measure the tradeoff by routing to a different model, evaluating and observing, adjusting, and rerouting again. One stack for production agents, in your cloud. Chalk Model Gateway is one part of Chalk's platform for production AI. The pieces work together: * Your data and cloud. Chalk runs in your environment and provides the live context and features your agents and models need. The router is in your environment, so you can configure your application to ensure data doesn't leave your cloud. * Your models. Use frontier APIs with your own provider keys, or run open-weight models on your own GPUs with Chalk Compute. * Your routing policies. Use Chalk Model Gateway to decide which model handles each request based on cost, capability, availability, and policy. These systems increasingly depend on each other. All three operate within the same Chalk environment, so you can move from context to inference to routing without stitching together another set of services. The right model depends on the request, and the right decision depends on the context available to that model. Chalk's Model Gateway sits alongside the rest of Chalk's production AI infrastructure: live context and features, model serving, compute, evaluation, and now routing. The model sets cost, the context sets quality, and the routing policy decides which model sees which request. If you are building agentic workflows and want reliable, cost-aware model access without sending traffic outside your cloud, talk with the Chalk team today and check out the Model Gateway docs.

AIC26: what agent teams asked Chalk. Director of Content October 5, 2026 Chalk spent last week at The AI Conference 2026 in San Francisco meeting with teams building across the AI maturity spectrum. Recent AI entrants wanted to hit the ground running with plug-and-play agent builders to support their products. Companies already using AI in production approached Chalk with different concerns: they asked Chalk about accuracy, affordability, and safety under concurrent loads. The questions changed depending on where teams were in their AI journey. But once teams reached production, the concerns were consistent: context, cost, and control. Common ways agents can fail. Teams in production brought up four common deployment pain points: * Quality degrades and agents hallucinate. This happens when agents try to reason on outdated features. Chalk's real-time context engine ensures that agents only act on what's true at decision time. * Agents miss context or retrieve the wrong information. Sometimes agents have too much information to reason over instead of guidance about what data matters to make a decision. * Costs spike when agents repeat tool calls or waste reasoning loops. For teams working at scale, wasted agent reasoning loops are too expensive. * Agents access tools or data they shouldn't. Heavily-regulated industries require safeguards so they can test and deploy agents that enforce data permissions and user privacy. These are the problems Chalk built Chalk to solve. Production AI needs real-time inference, supported by one definition of your data and temporal consistency. Scores of AI teams made their way to Chalk's booth at The AI Conference 2026. Signaling production readiness. The most interesting conversations weren't about whether agents can work. They were about how to make them work reliably within an organization's specific requirements, constraints, and guardrails. They asked whether Chalk delivers context at prompt assembly or through a tool call, how Chalk would deploy to an on-prem stack, and how low-latency context reduces wasted reasoning on private models. Discussions are moving beyond the inference architecture that has been top of mind for the last few years. Teams are more interested in control than agent deployment or performance. They want to know who owns the data, selects the models, governs the context, and ultimately stays accountable for an agent's decisions. What an agent stack needs. Attendees were curious about managing agent context and governance in one system. Conversations drifted toward the same set of production features: * Real-time context delivered at decision time * MCP tool-call governance * Evals that run on the same data production sees (or would've seen) * Observability that goes beyond traces Its Co-Founder, Elliot Marx, dove into how Chalk built these features during his session at Pier 48. He shared why most AI failures that look like model or prompt problems are actually context problems with agents deciding confidently on stale data, and evals that can't predict production performance. A full house stayed to watch Elliot share an end-to-end workflow for shipping agents with a context layer and deploying them in a sandbox. Chalk Co-Founder Elliot Marx shared a framework for improving agent accuracy and evals. Own your intelligence. The AI maturity curve is getting shorter. Newcomers joining the fray daily show how much interest there is in expanding the use cases for models and agents. After they get set up, production AI teams turn their focus to building infrastructure that improves performance and cost-efficiency. That's where Chalk can help. Elliot will host a live session on October 8 that will show you how to turn data into actionable context for your agents at decision time. Own your intelligence with a single system that powers testing, evals, and production, so you can deploy agents with confidence.