Adding VLMs to your auto-labeling pipeline with FiftyOne Agentic Labeling. Jul 30, 2026 Talk to an AI expert. Building high-performing AI models requires robust, accurate data. Yet getting raw data into a labeled, model-ready state remains the largest operational hurdle in most AI pipelines, and auto-labeling is how teams try to close that gap. According to the 2026 State of Visual and Physical AI report, 78% of organizations expect annotation spending to either increase or remain flat over the next year. While adoption of auto-labeling is on the rise, most annotation pipelines still struggle to automate the specialized labeling tasks that matter most in production. The bottleneck isn't common objects. It's domain-specific concepts. Proprietary defects, rare edge cases, and application-specific objects that off-the-shelf models were never trained to recognize still require significant manual effort, and building a custom model for every one of them isn't always feasible. Recent advances in vision-language models (VLMs) offer an alternative. Instead of training a specialized model from scratch, VLMs can learn domain-specific labeling tasks from natural language instructions and a handful of examples, dramatically lowering the barrier to automating annotation. Today Voxel51 is introducing Agentic Labeling, a new capability in FiftyOne Annotation that lets teams rapidly experiment with vision-language models (VLMs) to scale high-quality labeling. Auto-labeling tradeoffs in off-the-shelf and custom models. With the growing volume and expanding modalities of data required for physical AI, manual labeling has become incredibly resource and cost intensive. Lowering these costs requires deploying AI-assisted workflows to make the pipeline scale more efficiently. Today's auto-labeling workflows are generally organized into two approaches: * Off-the-shelf models can easily detect common objects like industrial components, road lanes, and basic anatomy, but quickly fall apart when asked to identify proprietary manufacturing defects, unusual highway debris, or specialized medical findings. * Custom auto-labeling models solve these domain-specific problems, but only after teams invest in collecting thousands of manually annotated examples for training. Both approaches force a tradeoff. Off-the-shelf models are fast to deploy, but too generic for the long tail. Custom models perform better on proprietary labels, but require extensive manual labeling upfront. Vision language models bridge the gap on auto-labeling. Vision-language models offer an alternative. VLMs are pretrained on massive, web-scale image-text pairs, which gives them the kind of transferable visual vocabulary that narrow, task-specific models don't have. * Zero-shot VLMs outperform traditional AI models at recognizing specialized objects, even without being trained on any domain-specific data. * Research on few-shot adaptation shows that giving a VLM a small number of labeled examples, rather than thousands, is often enough to meaningfully improve its performance on a specialized task. In other words, VLMs let teams get much closer to custom-model accuracy without paying the custom-model data tax, making them a practical starting point any time you need labels for a specialized object or edge case your existing models weren't built to handle. FiftyOne Agentic Labeling, the fastest way to label with VLMs. Vision-language models have become remarkably capable at generating labels, but getting consistent results requires experimentation. Research on VLM adaptation consistently points to the same failure modes: models struggle to tell apart fine-grained, visually similar categories without enough guidance, and small changes to a prompt can swing accuracy significantly. From prompt to prediction: configuring a classification agent in FiftyOne's Agentic Labeling to label the malaria-100 dataset. Powered by VLMs, FiftyOne Agentic Labeling provides a no-code workflow for rapidly iterating on prompts and visual examples to refine label quality. * Author a Labeling Agent: Describe what you want labeled in natural language and optionally add positive and negative visual samples to sharpen the agent's understanding. * Preview results: The VLM generates a preview of outputs on a handful of samples so you can see where the model aligns with the desired ontology and where additional guidance is needed. * Iterate: Continue to refine the prompt or examples and rerun until the labels align with your annotation guidelines. When you're satisfied, you save the configuration as a reusable Labeling Agent. * Scale to your dataset. Select the saved agent and the dataset or view you want labeled. The system runs the agent as a background task using connected compute, returning a labeled baseline across your full dataset. Agentic Labeling gives you a faster path from raw data to a labeled baseline - without writing a single line of code or waiting on a full annotation cycle to see if your approach is working. Get started with Agentic Labeling. As models mature, MLEs are playing an increasingly central role in annotation - not just consuming labeled data, but defining what gets labeled, automating the first pass with AI, identifying where models fail, and closing the loop between evaluation and the next annotation cycle. The Agentic Labeling flywheel: curate data to select what's labeled next, annotate with AI-assisted and human-in-the-loop workflows, then evaluate to surface model failures that inform the next curation pass. That shift also changes what teams need from their data. Early-stage teams need volume and labels to get a model off the ground. Teams with models already in production face a different problem: their models are failing on edge cases and new scenarios, and more data of the same kind won't fix it. In its survey of 700+ MLEs, the majority of teams expect to need less data to achieve the same model performance over the next year - a sign that coverage and quality are displacing data volume as the primary constraint. The question is no longer how to label more. It's which data to label next. 57% expect to need less labeled data each year to achieve the same model performance. FiftyOne Annotation is built for teams that need to move faster from raw data to training-ready labels. Because it lives within FiftyOne's broader platform, curation and model evaluation are just one step away. Model failures surface the right data to label, and newly labeled data informs the next evaluation cycle. * Agentic Labeling: Generate an initial labeled dataset from natural language prompts - no code required. * Smart data selection: Identify the samples most likely to improve model performance before a single label is created * Intelligent review: Surface annotation mistakes in bulk using embeddings and mistakenness scores, before errors reach training * Annotation workflows and project management: configure multi-stage pipelines with role-aware permissions and rejection loops, and coordinate teams with task assignment and per-stage progress tracking Agentic Labeling is an enterprise feature within the broader suite for FiftyOne Annotation. Get in touch to learn more. Frequently asked questions about auto-labeling. Discover more insights and tips to boost your visual and physical AI workflows.

Voxel51 uses AI processing to identify and track objects and activities through video clips.