How Sigma's Data platform team manages the semantic layer with dbt and Dagster. Matt Senick Senior Analytics Engineer September 30, 2026 Snowflake has Semantic Views. dbt has a metric layer. Sigma has data models. Define ARR in all three by hand and you will eventually get three answers, usually in the same meeting. That problem got worse when AI showed up. A person squints at two dashboards and knows something is off. An agent picks one and answers with confidence. At dbt Summit this year, I gave a breakout session with Dagster on how the internal data team at Sigma handles it: Sigma Computing treat the semantic layer as code, build it in dbt, and deploy it with Dagster in the same job that builds its tables. This post walks through that setup and what it powers in Sigma. Why the semantic layer matters more with AI. The semantic layer has been around since the days of OLAP cubes. It keeps coming back under new names because the underlying problem never goes away. Every surface in the company needs to agree on what a customer is, how revenue is counted, and which join is the right one. Large language models raised the stakes. Its semantic layer now does three jobs at once: it carries governance rules like PII restrictions and SQL formatting instructions, it holds the one definition of every metric and relationship, and it gives agents the synonyms, verified queries, and model context that generic text-to-SQL is missing. The goal is to hand an LLM everything it needs in the most condensed form possible. A well-built semantic model does that better than any prompt. It is also auditable, which matters the first time someone asks where an agent's number came from. What goes into a semantic model. Every vendor's semantic model shares the same core: tables, relationships, facts, dimensions, and metrics. The newer additions are what make it useful to AI. Comments describe intent, SQL generation instructions constrain how queries get written, and categorization gives agents synonyms and business context. Snowflake, dbt, and Sigma each have their own version of this structure. Sigma Computing split its semantic models by business function: Sales, Marketing, Customer Success, Renewals, Product, Engineering, Technical Support, Finance, and Partnerships. A CS analyst prepping a QBR works against the Customer Success model, not a 400-column monolith with three fields named arr. Sigma Computing wrote about this domain split in more detail: How Sigma's data platform team manages Snowflake Semantic Views. Managing the semantic layer as code in dbt. Some BI tools only let you build a semantic model by clicking through a UI. That works until you need to roll back a change, review a diff, or let two people work in parallel. Modern semantic layers are moving to code. Code gives you version control, pull requests, and rollbacks. It also turns out coding agents are far better at reasoning over a YAML file and a SQL model than over a sequence of screenshots. When Sigma Computing ask a coding agent to add a dimension to the Renewals model, it reads the definition, makes the change, and opens a pull request. Nobody has to describe where the button is. Sigma supports this directly through data models as code. Sigma Computing define Sigma data models inside the dbt project with SQL and Jinja, map each one to specific warehouse databases, schemas, and tables, and explicitly declare which columns business users can see. 3 dbt packages Sigma Computing use. Three open-source dbt packages turn each semantic object into a dbt node: dbt-sigma-data-models for Sigma data models synced through the Sigma REST API, dbt_semantic_view for Snowflake Semantic Views, and dbt-cortex-agent for Cortex Agents like J.A.K.E., its internal Cortex Agent. Because every object is a node, it sits in the dbt DAG with full lineage back to the tables it depends on. Its project follows a standard medallion architecture. Bronze holds raw ingested source tables, silver holds cleaned and conformed assets, and gold holds business-level marts. On top of that Sigma Computing added a fourth layer, semantic/, that holds every semantic object: Continuously deploying the semantic layer with Dagster. A semantic layer defined in code still has to reach Sigma. If that step is manual, the code and the tool drift apart the first week someone is on vacation. Sigma Computing deploy every semantic object in the same Dagster job that builds its dbt tables, so a merge to main updates the tables and the data models together. Dagster orchestrates everything its team builds: 63 jobs and 2,883 dbt models across six data regions. The semantic layer is one more asset in that graph. It gets no special deployment process and no separate schedule. The orchestration pattern. The dbt half of the job is familiar. Dagster downloads the last production manifest from S3 and runs dbt build -s state:modified, so only changed models and their dependents rebuild. After the build, two steps run in parallel: the updated manifest goes back to S3 for the next run, and a Sigma sync step pushes every changed data model definition to Sigma through the Sigma API. The Sigma step is the new part. It syncs the warehouse connection so new tables are visible, then syncs the data model objects themselves. A new metric merged at 10 a.m. is in the Customer Success data model before the CSM's afternoon renewal call. Nobody opens Sigma to click anything. What the semantic layer powers in Sigma. Nobody maintains a semantic layer for its own sake. For Sigma Computing, the payoff increasingly comes from agents. Sigma Agents. Sigma Agents are where the semantic layer does operational work. An agent can be triggered by hand, embedded in a chat interface, or left running as an autonomous background workflow, and in every case it queries the semantic layer directly and writes results back to the warehouse under the same governance as everything else. Model Context Protocol (MCP) extends its reach to external tools and custom REST APIs. A renewals manager can hand an agent a list of at-risk accounts and get back a summary per account, built on the same ARR numbers finance reports to the board. Sigma Assistant. Sigma Assistant is the open-ended side. Anyone can ask a question in natural language and get live tabular results with source attribution back to the data model that answered it. When a question turns into real analysis, the assistant can spin up a workbook and keep going. Outside Sigma, the same semantic layer backs its Snowflake Cortex Agents and Cortex Search Services. Its coding agents read it too. A semantic model is the most condensed description of the business Sigma Computing has, which makes it the first file an agent should open before touching a pipeline. Where this leaves Sigma Computing. Today, Sigma Computing manage its core Sigma Data Models within this process and integrate it within its teams development lifecycle. A metric changes in a pull request, a reviewer approves it, and every workbook, agent, and assistant answer in Sigma picks it up on the next deployment. Nobody on the data team is in the deployment step anymore. The merge is the deployment. See it in your own warehouse. If you want to run your semantic layer as code and deploy it straight into Sigma, request a demo or start a free trial. Faq. What is a semantic layer? A semantic layer is a shared set of definitions for metrics, dimensions, and relationships that sits between the warehouse and the tools that query it. It makes every dashboard, agent, and query use the same business logic. Why manage the semantic layer in dbt? dbt gives semantic definitions the same version control, code review, CI, and lineage as the models they depend on. A change to a metric becomes a reviewable diff instead of an untracked edit in a UI. How does Dagster deploy Sigma data models? Its Dagster job runs dbt build against modified models, then runs a Sigma sync step that pushes changed data model definitions through the Sigma API. Tables and data models update in the same run after every merge. Do I need Dagster to manage Sigma data models as code? No. Sigma data models as code works with any orchestrator that can run dbt and call the Sigma API. Sigma Computing use Dagster because it already orchestrates the rest of its pipelines. How does the semantic layer improve AI answers in Sigma? Sigma Agents and Sigma Assistant query governed data models instead of guessing at raw tables. Metric definitions, synonyms, and instructions in the semantic layer keep answers consistent with every other report in the company.

Prefect just bought Dagster, another big Airflow rival - and it's not a data pipeline story. What began as a rivalry over data workflows is becoming a bet on how AI agents should be run, governed and measured. Prefect, best known for the eponymous open-source data pipeline and workflow orchestrator, has announced that it's acquiring Dagster, one of the biggest alternatives to Apache Airflow alongside Prefect itself. The deal, which has yet to formally close, folds two of the most established Airflow challengers into a single company. Dagster, the open-source orchestrator, and Dagster+, the fully managed cloud incarnation, will keep their existing names, pricing, and product roadmaps, with some 40 people from the Dagster team moving over to Prefect as part of the deal. In a LinkedIn post published on Monday, Prefect founder and CEO Jeremiah Lowin explains that the acquisition is a bet on the various components an AI agent needs to run reliably: clearly defined goals, paired with the flexibility to improvise when its own decisions send it down a path nobody scripted. Dagster, according to Lowin, handles the goal-setting side by defining and tracking outcomes; Prefect handles the improvising by running the work itself; and FastMCP, Prefect's own tool for connecting agents to outside systems, controls what those agents are allowed to touchd along the way. For Lowin, the acquisition is all about creating scale. Two become one. That "scale" pitch comes some seven years after Prefect and Dagster competed, both emerging as challengers to Airflow, the default tool most teams used to schedule and run their data pipelines. Prefect set out to simplify and make production workflow orchestration more reliable for Python developers. Dagster, for its part, focused on making data pipelines easier to define, understand, and verify, placing greater emphasis on what a pipeline should produce rather than simply running a sequence of tasks. "For years, Prefect and Dagster have raised the bar for each other and our category. That competition produced two exceptional products and two of the strongest open-source communities in the data ecosystem." "For years, Prefect and Dagster have raised the bar for each other and our category," Lowin writes. "That competition produced two exceptional products and two of the strongest open-source communities in the data ecosystem." The strengths each company built for running pipelines are now being stretched to cover something bigger, with Prefect betting those same strengths are what agentic workloads will need next. The AI transition. Both companies have been repositioning around AI agents for a while. Back in 2024, Prefect 3.0 landed with an explicit focus on supporting agentic workflows. A year later, Dagster debuted Components, pitched as a way to build pipelines using reusable, YAML-based building blocks rather than hand-written Python. Shortly after, Dagster went further with Compass, a tool that lets analysts query data using plain-language prompts sent through Slack instead of writing SQL. None of that matches the reach of what Prefect built with FastMCP. The framework works with the Model Context Protocol (MCP), the open standard Anthropic released in late 2024, which lets AI models discover and call external tools and data, allowing developers to build MCP servers by writing plain Python functions instead of hand-coding the protocol directly. Prefect shipped FastMCP the same month that MCP itself launched, and Anthropic later adopted it as the official Python SDK for MCP. The deal has consequences beyond product strategy, though, starting with the people who built Dagster in the first place. Nick Schrock, who founded Dagster and moved from CEO to CTO in 2022 to make way for Pete Hunt, is named alongside Hunt in the official acquisition announcement as staying on to "serve as strategic advisors to Prefect" while remaining "active in the open-source community." However, Schrock's own blog post, published on Monday, reads far more like a clean break from Dagster. "I want to share that I'll be moving on from the project and company," Schrock writes. Schrock's departure closes one chapter for Dagster. The acquisition itself, however, opens another, with Prefect betting that the ideas both companies spent years developing for data orchestration will prove just as valuable as AI agents move into production. Landen Bailey, principal data engineer at ecommerce platform Redo, took to LinkedIn to reflect on the pairing - for years, he writes, Prefect and Dagster were the two "hipsters" companies turned to whenever they wanted a break from Airflow. Together, he argues, they now stand a better chance against the incumbent. "The potential this gives Prefect to mount a real challenge against Airflow's marketshare will be fun to watch," Bailey writes. "Individually, each tool struggled to stand up to the incumbent giant, but now there's a real chance to make a dent."