Staff Sales Engineer

DBT Labs targets costs with data engine, adds AI agents. The new capabilities, revealed amid news that the vendor is merging with Fivetran, include an updated engine that lowers data transformation expenses and task-specific agents. DBT Labs on Tuesday unveiled an updated version of its Fusion engine aimed at improving data transformation workload performance to help customers control spending as they build and maintain AI and analytics applications. In addition, the vendor unveiled DBT Agents, a set of agentic AI-powered capabilities built into the DBT Labs platform to help users carry out tasks such as discovering relevant data and monitoring data quality. The Fusion update and DBT Agents were revealed during Coalesce, DBT Labs' annual user conference in Las Vegas. On Monday, one day before unveiling its newest product offerings, the vendor reached an agreement to merge with Fivetran, a data integration specialist and longtime business partner. Fusion, first introduced in May, is in preview for workloads on Amazon Redshift, Databricks, Google BigQuery and Snowflake. Meanwhile, the first four DBT Agents are generally available through DBT Labs' remote Model Context Protocol server. With AI applications requiring massive amounts of data to reduce errors and deliver outputs that can be trusted, data workloads are much bigger than in the past, when proprietary data was mainly used to inform reports and BI dashboards. As a result, the Fusion update targeting workload performance is a valuable addition for the vendor's users, according to William McKnight, president of McKnight Consulting Group. "The Fusion engine will be a highly significant addition for DBT users, driving major improvements in both efficiency and strategic capability, especially regarding the adoption of AI," he said. "The urgency around cost optimization and improved performance is driven by the necessity to scale analytics and govern data reliably to build the foundation for AI systems." Given that the DBT Agents contribute to more efficient analysis and governance, they're also important additions for DBT Labs customers, McKnight continued. "DBT's AI agents... utilize the platform's structured context to speed up the analytics development process, enhance data governance and produce reliable AI-driven results while maintaining high quality standards, all of which are valuable," he said. Based in Philadelphia, DBT Labs is a data transformation specialist providing tools that enable developers and data engineers to cleanse and validate data so it can be used to develop AI and analytics applications. New capabilities. Nearly three years after OpenAI's November 2022 launch of ChatGPT marked a substantial improvement in generative AI (GenAI) technology and sparked a sudden increase in enterprises investing in AI development, interest in building AI tools continues to rise. However, the platforms many enterprises currently use to develop AI applications are often the same ones they used when their development focus was on building simple reports and dashboards. Because these platforms weren't built to handle AI workloads, which are much larger than previous workloads, they are performing poorly, resulting in skyrocketing costs for users. GenAI has the potential to make workers better informed and business processes more efficient, which is why so many enterprises began prioritizing AI after ChatGPT's launch. Now, development has evolved from building GenAI chatbots to AI agents, which are applications programmed with contextual awareness and reasoning capabilities that enable them to autonomously perform specific tasks. DBT Fusion is an underlying engine for data transformation workloads designed for the demands of AI development, according to Tristan Handy, founder and CEO of DBT Labs. The DBT Fusion update adds state-aware orchestration, which reduces unnecessary compute costs by ensuring that AI pipelines only run models that have changed when training and updating applications. In addition, the engine enables data and AI teams to hone AI pipelines by setting data freshness requirements that the state-aware orchestration upholds while determining the most efficient job execution path. Beyond state-aware orchestration, DBT Fusion now enables users to build DBT-powered pipelines that create and manage Apache Iceberg tables in Databricks and Snowflake, run workloads on-premises and in local clouds for more control than is possible in public clouds, and integrate semantic definitions and data lineage to improve data quality. "The Fusion engine came from a realization that we'd reached the limits of what we could do with the original DBT Core codebase," Handy said. Customer feedback also played a role in DBT Labs' initial decision to develop a new engine and subsequently update it, he continued, noting that users always want faster performance and lower costs. But the primary impetus for building DBT Fusion was understanding that the vendor's original architecture, built in 2016, no longer met the needs of modern workloads. "Our real motivation was recognizing that we couldn't just iterate our way into the future, especially with how quickly things are changing with AI and open standards like Iceberg," Handy said. Because the Fusion improves upon DBT Labs' previous engine and now adds capabilities that help customers control costs, DBT Fusion has potential value for customers, noted Donald Farmer, founder and principal of TreeHive Strategy. "Cloud compute costs are top of the agenda for many CIOs, largely due to the increased scale of AI, analytics and data workloads," he said. "So, this is a timely announcement, especially as DBT has been criticized by users for slow execution on larger projects." State-aware orchestration, meanwhile, is a logical way for DBT Labs to address cost control and improve performance, Farmer continued. "The state-aware orchestration certainly does make sense," he said. "There was a lot of manual work going into this process previously, so users should be happy with this if it is done well." While DBT Fusion aims to improve workload performance and cost efficiency, the DBT Agents target faster development and decision-making. They include the following: * Developer agent to assist developers and engineers. * Discovery agent to discover the right data for informing applications. * Observability agent to monitor data quality, identify the root causes of errors and changes, and propose fixes. * Analyst agent to answer questions about models, jobs and metrics. Many vendors are now adding agents to their platforms, so DBT Labs is not alone in doing so, Farmer noted. However, because data transformation involves more routine work than some other data management processes, agents are welcome additions for the vendor's users. "DBT may have better use cases for agents than many platforms," Farmer said. "A lot of the busy work involved in running DBT is mundane and repetitive, not to mention error-prone. Those are exactly the tasks agents should be taking over." McKnight called out the Developer and Observability agents as perhaps the most significant for users. "The Developer agent stands out by automating tasks like code authoring, refactoring and validation, thereby boosting velocity while maintaining quality and trust," he said. "The Observability agent is significant with its automation of problem identification and solution proposal, which reduces manual remediation work and supports governed AI and trustworthy data infrastructure." Next steps. The pending merger between DBT Labs and Fivetran is part of an ongoing consolidation in data management and analytics. Platform vendors such as Databricks and Snowflake have been buying specialists to add capabilities. Similarly, Qlik and Fivetran of late have expanded by acquiring smaller vendors with platforms that address specific needs. Meanwhile, Informatica, a longtime independent data management specialist, is under agreement to be acquired by Salesforce. Joining forces with Fivetran is aimed at combining complementary capabilities to provide customers with a more full-featured platform, according to Handy. "When the deal closes and our companies join forces, we can become [an] end-to-end [extract, transform and load] solution at scale," he said. "We already have thousands of joint customers... who rely on Fivetran and DBT as the best-of-breed combination. Our goal is to bring that together in a unified solution." From a product development perspective, DBT Labs plans to refine Fusion as more developers adopt the new engine and continue expanding its open ecosystem through integrations, Handy continued. McKnight suggested that DBT Labs could best serve the needs of current users and perhaps attract new ones by improving its data quality capabilities and doing more to integrate its AI agents with existing tools. "DBT could enhance its platform by improving data lineage visualization and supporting more data sources," he said. "Additionally, DBT could deepen the capabilities of its AI agents, integrating them more tightly with DBT Insights to accelerate insight generation and broaden the use of the DBT MCP server to ensure trusted, governed context for external AI systems." Eric Avidon is a senior news writer for Informa TechTarget and a journalist with more than 25 years of experience. He covers analytics and data management.

Data startups Fivetran and dbt Labs are merging in an all-stock deal, forming a data infrastructure company with nearly $600 million in annual revenue. The merger is based on a stock exchange ratio tied to revenues and growth rates. Fivetran was valued at $5.6 billion in 2021, and dbt Labs at $4.2 billion in 2022. Fivetran CEO George Fraser will lead the new entity, while dbt Labs CEO Tristan Handy will become co-founder and president. The merger highlights consolidation in the data tooling market.