Senior Solution Architect

Teaching AI how to refinery. honeycomb.io released a new tool for its MCP server which helps your AIs understand Refinery, and how Honeycomb handles sampling. In this blog, I'll walk you through the new feature and show how it can help you run Refinery. | February 17, 2026 March 11, 2025 The engineer's guide to sampling. At the beginning of February, honeycomb.io released v3.1 of Refinery, its advanced, tail-based sampling solution. The new version comes with more performance enhancements, bug fixes, and a few new pieces of telemetry. In tandem with the 3.1 release, honeycomb.io also released a new tool for its MCP server which helps your AIs understand Refinery, and how Honeycomb handles sampling. With this new tool, you can ask Canvas or your favorite AI questions, like: "What are my sampling rules doing?" "Is my refinery configured correctly?" or "Am I achieving my goal sample rate?" In this blog, I'll walk you through the new feature and show how it can help you run Refinery. Explore honeycomb telemetry pipeline. Make cost control feel safe, smart, and scalable. How honeycomb.io found a bug in its own sampling rules. A great example of how useful this feature can be is how it found an issue in its own sampling rules during development. Once honeycomb.io had written the static content about Refinery, I wanted to test it using an AI assistant. What better way to test than to ask questions about its own sampling? honeycomb.io generate a lot of telemetry from its ingest pipeline, so honeycomb.io use Refinery to sample the data. honeycomb.io use a RulesBasedSampler with seven different rules. One of its rules is set up to ensure that all its end-to-end test data (which runs roughly every minute in production) is all kept. - Name: E2E Data Conditions: - Field: service.name Operator: = Value: shepherd - Field: app.environment.id Operator: = Value: 49427 SampleRate: 1 Some very astute Refinery readers may already see the issue, but at the time, I didn't. While testing, I asked an AI assistant that was hooked up to Honeycomb's MCP server, Look at my production_us1_refinery_values.yaml production refinery rules. What do they do? Use the honeycomb mcp to learn about Refinery first. It used the new Refinery tool to learn about Refinery, review the rules, and give a detailed answer. For this story, the important part of the response was: Sampling Rules Breakdown 1. E2E Data Rule (lines 128-137) - Keeps: 100% of traces (SampleRate: 1) - When: shepherd service AND app.environment.id = "49427" (string comparison) - Purpose: Preserve all E2E test traces for the test environment Again, readers who have hit this issue before may see that the AI already knows something really important about app.environment.id. But again, I didn't see it. This is all I needed to test; the ability for an AI to accurately review and critique Refinery rules was its goal, and goal achieved! But I've been going through my own journey learning how to use AI, and I was curious (and skeptical) if it would be able to apply its knowledge to the data itself. So, I asked a follow-up question: Based on your understanding of the rules, is the data in shepherd dataset being sampled correctly? Also look in the refinery-metrics dataset. Use the honeycomb mcp server to review this data. The AI used the Honeycomb MCP to query data and assess if it was sampling correctly based on the rules it had reviewed. Eventually, it returned an unexpected answer: CRITICAL ISSUE: E2E Data Not Sampled Correctly | Problem: E2E environment traces (app.environment.id = 49427) are matching the wrong rule. - Expected: Should match "E2E Data" rule (lines 128-136) with SampleRate: 1 (100% retention) - Actual: Matching "DeterministicSampler for pro plus/enterprise handle\_batched\_event" rule with SampleRate: 10 - Impact: E2E data is being sampled at 1:10 instead of keeping all traces Since rules are evaluated in order and the E2E rule comes FIRST, it should match before the Pro/Enterprise rule. The fact that it's not matching suggests: 1. Possible cause: The app.environment.id field may not exist or not be set to 49427 on all spans in E2E traces 2. Rule evaluation: Without Scope: span on the E2E rule, it uses Scope: trace (default), meaning conditions can match across different spans - but if app.environment.id doesn't exist anywhere in the trace, the rule fails I was very skeptical, as this rule has been in place for a long time. Surely, this was just another case of AI getting it wrong, right? I told it to try again: This would be pretty serious. Rerun your analysis to confirm your math is correct. Since the e2e rule comes first it should take priority. Test your hypothesis that app.environment.id does not exist. While it ran, I went and searched the data myself and found the same result the AI had: the E2E Data rule was never being hit. I couldn't think of why that would be happening. At this point, my AI assistant was done thinking and returned the answer for me. I'm including the entire output this time because it did such a good job of supporting its claim. FOUND THE BUG\! Looking at the YAML, I can now provide my final analysis. The data confirms: Confirmed Issue 388 E2E traces exist that match BOTH conditions (service.name = shepherd AND app.environment.id = 49427), but 0 traces match the E2E rule. Root Cause The most likely issue is that the integer value 49427 on line 135 needs to be quoted as a string in Refinery's configuration: Value: "49427" \# Should be a string, not an integer Why This Matters In Refinery's rule evaluation, field values may need to be strings regardless of the underlying Honeycomb field type. Even though app.environment.id is stored as an integer in Honeycomb, Refinery may perform string-based comparison during rule evaluation. Evidence 1. Working rules: The Pro/Enterprise rule uses Value: handle\_batched\_event (string) and Value: shepherd (string) - both work 2. Failing rules: The E2E rule uses Value: 49427 (unquoted integer) - doesn't match 3. Boolean example: The API requests rule uses Value: true with Datatype: bool - also doesn't match (similar type issue) Impact Assessment - Current State: \~388 E2E traces/hour are being sampled at 1:10 instead of 1:1 - Data Loss: \~90% of E2E test traces are being dropped - Severity: HIGH - E2E monitoring is significantly degraded Recommendation Change line 135 from: Value: 49427 To: Value: "49427" This should be tested in a non-production environment first to verify it fixes the rule matching. I couldn't believe I had missed this bug. This is a well-known footgun in Refinery that is easy to get wrong. Back in Refinery 1.20, honeycomb.io even added support for Datatype so that users could explicitly set the expected type to use when comparing! It's hard to figure out, though (hence the defensive Datatype feature), as it requires a deep understanding of both your rules and your data. In this case, when honeycomb.io set these rules up a long time ago, honeycomb.io made this mistake and never checked if honeycomb.io were sampling correctly (whoops). Conclusion. This experience really opened my eyes on how good AI can be when pattern-matching correctly. It also highlights how flexible they can be when given the proper tools. honeycomb.io even found a second issue in its Refinery configuration by doing the same kind of "teach" and then "ask" flow. With the power of Honeycomb's MCP, all your high-cardinality, high-dimensionality data is readily available for your favorite AI assistant to be your pattern-matcher. Be curious, ask questions; you might be surprised what answers you'll receive.

Honeycomb launches new Private Cloud offering to address security, compliance, and cost concerns. Latest release also includes revamped Metrics capabilities and Canvas in GA, providing secure, unified, and intelligent observability to aid enterprises in their adoption of AI. AI-Powered observability: business value insights for technology leaders featuring IDC. Discover how organizations using Honeycomb achieve measurable results, including faster issue resolution, proactive problem identification, and better cross-team collaboration. SAN FRANCISCO, November 19, 2025 - Honeycomb, the creators of observability, today announced a series of new product advancements: Honeycomb Private Cloud, Honeycomb Metrics, and Canvas, a part of Honeycomb Intelligence. Together, these new capabilities bolster Honeycomb's enterprise portfolio, combining security, performance, and agentic experiences to help organizations deliver resilient software at scale. Enterprises are rapidly increasing their adoption of private and hybrid clouds for better control, residency, and security. Observability is an essential function for teams who need deep visibility and rapid debugging across cloud-native multi-service architectures, regardless of deployment location. With Honeycomb Private Cloud, organizations get a dedicated AWS infrastructure built to meet stringent security, data residency, and regulatory requirements - all without sacrificing performance or developer experience. By providing complete management over data and environment isolation, this offering is especially well-suited for customers in highly regulated industries like finance and healthcare, who need to confidently access the lighting-fast query performance and intuitive workflows that define Honeycomb's platform. "With the rise of AI, enterprises can no longer just monitor uptime or latency, they must understand how systems are performing in real time," said Thomas Squeo, CTO, Americas at Thoughtworks. "Observability is now a business imperative, and Thoughtworks is thrilled to be Honeycomb's preferred partner for implementing Private Cloud. We believe this offering will unlock a new chapter of observability for the enterprise, particularly as organizations strive to effectively manage their complex data and development requirements." Customers are able to select self-managed or Honeycomb-managed options. Additionally, this new Bring Your Own Cloud (BYOC) option offers a cost-effective solution for organizations that want to keep their data in their own cloud accounts and control costs over time using their existing AWS discounts. With the introduction of overhauled and improved Metrics, Honeycomb is expanding its observability platform to include native support for standard OpenTelemetry metrics. This builds on Honeycomb's longstanding support for custom metrics through events, giving teams a unified way to explore both high-level signals and detailed event data in one place. With Metrics, users can now bring in gauges, counters, and histograms to track trends, monitor system health, and detect performance changes over time. All of this happens within the same intuitive Honeycomb experience. Unlike traditional monitoring tools that separate system data from application behavior, Honeycomb's unified model allows engineers to seamlessly connect what is happening in their infrastructure to why it is happening in their applications. This new capability gives teams faster insights, deeper context, and a clearer path from detection to resolution, making it easier than ever to understand how their systems and users interact. "Full time-series metrics are the industry standard for most developers and site reliability engineers," said Graham Siener, VP of Product at Honeycomb. "Our new Metrics capabilities meet customers where they are, combining open-standard metrics methodologies with our industry-leading tracing data to better determine the context around critical issues. The Honeycomb Intelligence platform, which includes our improved Metrics, is purpose-built for teams of every size and provides a platform for success that organizations can use to navigate their AI journey." Also launching today for General Availability is Canvas, Honeycomb's AI-guided dashboard that blends natural language investigation with interactive notebooks for collaborative debugging. "Writing code is no longer a bottleneck with AI, but quality is suffering and organizations are deploying software they may not fully understand, which makes observability more important than ever," said James Governor, Co-founder of RedMonk, the developer-focused analyst company. "Honeycomb is responding by making natural language a native interface for its strengths in troubleshooting, debugging, and problem resolution, integrating AI across the platform." With Canvas, engineers can ask questions in plain English and watch as Honeycomb autonomously explores telemetry data, surfaces anomalies, and visualizes findings in real time. Canvas runs multiple queries, comparisons, and BubbleUp analyses automatically, producing dynamic charts and trace visualizations that evolve as the investigation progresses. As part of Honeycomb Intelligence, Canvas works seamlessly with MCP Server and Anomaly Detection, bringing AI-driven investigation directly into the developer workflow rather than bolting it on as an afterthought. With these new offerings, Honeycomb further extends its leadership in helping enterprises achieve faster resolution, tighter control, and deeper understanding across distributed systems. To see Honeycomb in action, including Honeycomb Intelligence and Metrics, find honeycomb.io at AWS Re:Invent (booth 1378) on December 1-4, 2025. To learn more or get started, visit https://www.honeycomb.io/get-a-demo. About Honeycomb Honeycomb is the future proofed observability platform that enables engineering teams to find and solve problems they couldn't before. It unifies telemetry, returns fast queries, integrates with AI agents, and reveals issues others miss in your ever-evolving tech stack. Honeycomb's unique event-based pricing model provides a predictable way to ingest high-cardinality telemetry without penalties or hidden costs. Learn more at www.honeycomb.io and follow honeycomb.io on LinkedIn. Media Contact: Kaileigh Higgins [email protected]

How honeycomb.io saved 70% of CPU and 60% of memory in Refinery's Go code, no Rust required. The engineer's guide to sampling. honeycomb.io has just released Refinery 3.0, a performance-focused update which significantly improves Refinery's CPU and memory efficiency. Refinery has a big job: it performs dynamic, consistent tail-based sampling that maintains proportions across key fields, adjusts to changes in throughput, and reports accurate sampling rates. The traffic patterns it handles are challenging, with long or large traces requiring it to hold lots of information in memory, while sudden volume spikes leave little time for infrastructure to scale up - all in a package that people want to run as cheaply as possible, since one of the primary use cases for sampling is cost control. When you're spending money to save money, you always want to spend less. Version 3.0 is a big advance in that direction. When honeycomb.io upgraded its internal Refinery cluster, total CPU usage dropped by 70%, while RAM use dropped by 60%: With an improvement like this, honeycomb.io can downsize this 72-node cluster by half - a meaningful savings - while still keeping more headroom than before. If you're a Refinery user, hopefully so can you. How did honeycomb.io pull off such a big change? The code's all in this merge, but I'll cover the basics here. Like many programming languages, Go is capable of being very fast under the right circumstances (working with bounded quantities of strongly typed data), and very slow under the wrong ones. Unfortunately, Refinery's job of handling customer-defined trace spans is very close to the wrong one. Historically, honeycomb.io followed the standard approach and fully de-serialized every span that came in through the API. Since there's no fixed schema, the fields went into a big map[string]any - hundreds of heap allocations, pointers everywhere. It was simple and effective, but it was also expensive. Compounding this cost, in a cluster configuration, the majority of spans are handled twice since they're redirected from the receiving node to the node which "owns" the relevant trace. Here's what a profile of a typical clustered Refinery looked like: There's a lot going on here, but you can see almost a quarter of CPU time going to garbage collection. Digging further reveals that a lot of the leaf nodes are ultimately some form of malloc. In total, 50% of all CPU time in this process is allocation-related, all in order to hold onto span data as it waits for a sampling decision - after which, in most cases, it's simply thrown away without being sent to Honeycomb! There's also a lot of overhead just for setting up data structures which honeycomb.io is hardly going to use. The actual refining, which is the reason honeycomb.io is doing all of this, all happens in the collect loop, a mere 12% of time in this profile. Even that is mostly internal metrics instrumentation rather than the core decision-making algorithms. The best way to make all this de-serialization (and, eventually, re-serialization) fast is to not to do it at all. Refinery only ever looks at a handful of fields in any given span, the rest is just cargo. And it's very possible to extract only the fields you need from a serialized blob, as in the simplified example below. These two benchmarks demonstrate de-serializing into a map, and its new selective approach, where honeycomb.io pull out any fields Refinery needs, then hang onto the serialized data for re-transmission. I'm using MessagePack here because that's Refinery's native format, with a low-level serialization API provided by the tinylib/msgp library. You can see that selective involves much more code, but it's hard to argue with the results: This is a very simple scenario, and it's common for real spans to have hundreds or even thousands of fields, which in the old version meant much longer parsing times and thousands of distinct allocations per span. Instead, Refinery 3.0 keeps the serialized data, retaining it in a format which is much more compact than the web of headers and pointers created for a fully realized map. This more compact data is the main reason for Refinery's improved memory footprint. Of course, Refinery supports three other types of input data besides its native MessagePack (libhoney/json, otlp/proto, otlp/json). To handle the others, Refinery now transcodes those formats directly to serial MessagePack, binary-to-binary, again extracting any useful fields along the way. This code is even more voluminous than the selective extraction from MessagePack illustrated above, but it avoids an expensive additional step of translation from generated protobuf data structures into its own. To add icing to this cake, honeycomb.io also optimized its metrics instrumentation, implemented pools to re-use large buffers, and (coming soon as a minor version update) parallelized the core decision loop to scale across many CPUs. Notably, there are no clever algorithms or language tricks at play here. honeycomb.io didn't have to rewrite it in Rust. All honeycomb.io has done is reimagine which work this process really needs to do, and focus on only doing that. New to Honeycomb? Get your free account today. Get access to distributed tracing, BubbleUp, triggers, and more.