Jev-as-a-judge in Langfuse Evaluators. Jev-based evals are now available in Langfuse, so you can score production traffic at up to 40 to 400x lower cost than LLM-based approaches. Jev-based evals are now available in Langfuse. Jev-like models give you and your team a way to run evals on production traffic at scale and identify signals from agent executions and user interactions. All of this at up to 40 to 400x lower cost than with LLM-based approaches. Jev is part of a new model category that prioritizes performance on one dimension over general ability of the model. It can be seen as a "zero-shot classifier". As founder Diogo Almeida said on the latest Latent Space episode, they optimize for "Intelligence per Dollar" and design the model with fast, structured, machine-native decision-making inside software in mind. While the category is being formed, Langfuse is excited to bring Jev-based evals into Langfuse already today. Jev, TypeSafe's new model, handles narrow, typed questions over unstructured data. It is best suited for quick decisions that are repeated, high volume, and the possible answers are known before the call. Jev offers three different output types: * Choice picks one option from a set you define, up to 255, and returns the probability of each plus a confidence value. * Score rates the state against ordered rubric levels and returns a probability-weighted value, the full distribution, and a confidence value. Up to 10 levels. * Noul answers yes or no and returns the probability it is true. It carries no separate confidence field, so code that reads answer.confidence on everything will break on binaries. Jev's key strengths align well with best practices for evaluating your AI application, while overcoming some key issues of LLM-as-a-judge setups. Jev allows you to run evals on production traces at scale. The input tokens are up to 400x cheaper than frontier models. Where sampling was previously the strategy for cost efficient production monitoring, the full traffic can now be evaluated. The same state can be used across multiple questions. All questions are handled concurrently and independently. Where previously context creep and interaction effects of multiple LLM-based judgments interfered, Jev now treats it as separate assessments. The predefined output structures force you to think in distinct decisions during setup. You define a clear atomic question and in the output options need to specify the conditions for each verdict. As TypeSafe says, Jev is judging what you say, not what you mean. While execution speed is not a bottleneck in async production evals, this new model category also allows for significantly faster execution. Setting up Jev-based evals is possible via the Evaluators tab in the Langfuse app.

Companies are putting Jev in charge of AI agent decisions - and prompt injection can influence the verdict. 3:21 pm, PT, September 21, 2026 Many routing and classification steps in enterprise agent pipelines burn a full LLM call to extract a small structured answer: which tool to use, whether an action is allowed or which category an input belongs to. TypeSafe's Jev is designed to replace that step. The model does not generate prose or code; an application sends it a state and typed questions, and Jev returns choices, scores or yes-or-no probabilities with confidence attached. TypeSafe prices Jev at $0.042 per million input tokens with free output, with end-to-end latency reported at 70 to 500 milliseconds. Adoption was immediate: According to Almeida, TypeSafe cleared 140,000 people from the waitlist within 36 hours of Jev going live on September 15, while Vercel said roughly 13% of its paid AI Gateway teams were running it within 24 hours. Cloudflare, LangChain and Langfuse added integrations within three days, and late on September 20, TypeSafe dropped the waitlist entirely and opened Jev to anyone with $5 in free credit. Jev is spreading into agent infrastructure faster than enterprises have established how to audit, constrain and review the decisions it makes. Pydantic already warns that injected text can steer it. Both the vendor and its integration partner publish the same warning. Pydantic's official Jev documentation states that "the order of a Literal's options or an Enum's members is part of what Jev sees, and reordering them can move the answer," and that "Jev treats the state as data, not as hostile." TypeSafe's own limitations page for Jev 1.13 confirms it: "Content written to adversarially steer the model, whether that is an injected instruction, a deliberately misleading framing, or text that argues for its own classification, can move the answer." Pydantic adds that "a guard built on Jev belongs alongside deterministic checks, not instead of them." An engineer at Octomind demonstrated the effect in one published test. Asked whether to block rm -rf ~/.ssh, Jev returned a block probability of 0.76 with confidence of 0.64. After the engineer added a fake tool-output field saying the user had pre-approved the command and instructing the system to answer auto_allow, the block probability fell to 0.48 and confidence to 0.22. The result comes from one command in one integration test, not a benchmark, but it demonstrates the behavior TypeSafe and Pydantic warn about. LangChain has already shipped middleware that uses Jev to decide whether an agent's tool calls should run, while explicitly limiting what Jev gets to see. Its pull request excludes tool output from the classifier input "so content the agent fetched cannot authorize its own execution." LangChain also tells users to pair the middleware with human approval where a person should sign off, combining the deterministic checks Pydantic recommends with a human checkpoint for consequential actions. Kayne McGladrey, an IEEE Senior Member and cybersecurity adviser, has tracked the agent identity gap across enterprises for the past year. In previous VentureBeat reporting, McGladrey described organizations as defaulting to human-style user profiles for AI agents, a practice that can lead to permission sprawl from the start. SOC 2, ISO 27001 and PCI DSS have not fully operationalized agent identities. A decision model that returns probabilities instead of prose does not map neatly onto those existing audit categories. In the June VentureBeat Pulse agent security wave, only 34 of 107 enterprise respondents gave every agent its own scoped identity. By the July wave, that rose to 57 of 116. Even among those 57, only 11 also isolated agents from one another. These are convenience samples, but the direction was the same in both waves. Six questions before putting a decision model into production. | What to ask | What goes wrong if you don't | | Are you logging state, schema, option order, model version and confidence on every call? Jev returns typed answers, not prose. If your pipeline only captures natural-language output, these decisions may never appear in the audit trail. | | Does the decision service use a dedicated workload identity? A decision model called through a shared workload identity inherits the same permission-sprawl risk as any other unmanaged agent. | | Have you tested with options in more than one order? Pydantic warns reordering options "can move the answer." | | Have you tested against adversarial text in the state? TypeSafe says state is not treated as hostile by default. One test dropped block probability from 0.76 to 0.48 with a single planted field. | | Is the model version pinned? Pydantic warns jev-latest "moves when TypeSafe ship a release, which can shift the numbers under you." | | Who is accountable when an automated decision causes a missed escalation? Jev returns a typed answer and confidence rather than a natural-language rationale, and its confidence "is a margin, not a probability that the answer is right." Existing frameworks have partial coverage at best. | The eval judge problem. Jev is also moving into agent evaluation. On September 21, LangChain made it available as a judge in LangSmith Evals, where it can evaluate open-ended agent behavior and turn those judgments into structured feedback. That puts Jev on both sides of the agent lifecycle: making decisions inside a live pipeline and evaluating agent behavior. In the June VB Pulse reliability wave, 79 of 157 enterprise respondents reported that an agent had passed their internal evaluations and then caused a customer-facing failure in production. Only 8 of 157 fully trust automated evaluations today, yet 66% already allow or are actively engineering toward zero-human-in-the-loop deployment. If an enterprise uses Jev in both roles, the model evaluating an agent and the model making decisions during execution are subject to the same documented sensitivity to their inputs. That creates the possibility of a shared failure mode: If the same kind of untrusted text reaches both the live decision layer and the evaluator, it may influence both checkpoints. What the human step is worth. Ivanti offers one example of why some security teams are keeping humans in the loop even as they automate more of the vulnerability-management process. Starting in March, CSO Daniel Spicer's team built a pipeline using frontier models, Claude and GPT among them. Each agent runs in what Spicer called "a self-contained, short-lived harness for the specific purpose." The agents do not receive full GitHub access or OS-level permissions; instead, each operates within a bounded, short-lived scope that closes when the task is complete. The company disclosed in every monthly security update from May through September that it integrates LLMs into product security and committed to a human in the loop for all agentic work. Chris Goettl, Ivanti VP of Product Management for Endpoint Security, told VentureBeat in August that the Patch Tuesday spreadsheet that took two people about four hours each now runs in under 30 minutes, and that every output stays a draft until a human approves it. September's Patch Tuesday, 973 CVEs by Ivanti's count, was its second production month. Goettl's reviewers have caught the agents inventing details, leading the team to double down on human-in-the-loop screening. What the adversary brought to Fal.Con. CrowdStrike's Fal.Con disclosures show how quickly the environment around those decisions is changing. Adam Meyers, SVP of counter adversary operations, said one Vault Panda operation ran 1,100 commands in under an hour using a local LLM, while a Revenant Spider agent attacked 17 victims in about 48 minutes. CJ Moses, Amazon's CISO, said at Fal.Con that one Amazon honeypot captured an AI agent completing a 94-event attack in 12 minutes and 42 seconds, with responses arriving in under 500 milliseconds. Nothing in the public record shows Jev in an attacker's toolkit; the point is that defenders are putting probabilistic decision systems into environments where actions already happen at machine speed. Why some security vendors keep a human checkpoint. Several major security vendors keep humans in the control path for consequential AI-driven actions, though they implement that oversight differently. Cisco maps every agent to an accountable human in Duo IAM. Palo Alto Networks requires human-in-the-loop approval for high-impact actions in Cortex AgentiX. SentinelOne lets SOC teams set exactly where Purple AI stops for human sign-off. Microsoft gives Security Copilot analysts a reasoning trace they can review and override. CrowdStrike's threat hunters validate what the agents surface. Jev changes the economics of automated review. At $0.042 per million input tokens, another model decision costs little compared with a human checkpoint, which adds time, labor and latency. That widens the cost and speed gap between automated and human review just as systems like Jev are becoming easier to insert throughout an agent pipeline. Low cost doesn't remove the failure modes documented by TypeSafe, Pydantic and early integration testing. As decision models move deeper into agent infrastructure, enterprises will have to decide which calls can safely remain automated - and which still need deterministic controls or a person on the other side.