Today, Vantage is launching Ephemeral Workload Monitoring, providing customers with the ability to configure custom polling intervals as short as 5 seconds for the Vantage Kubernetes Agent.

It is a well-known fact that different model families can use different tokenizers. However, there has been limited analysis on how the process of “tokenization” itself varies across these tokenizers. Do all tokenizers result in the same number of tokens for a given input text? If not, how different are the generated tokens? How significant are the differences?In this article, we explore these questions and examine the practical implications of tokenization variability. We present a comparative story of two frontier model families: OpenAI’s ChatGPT vs Anthropic’s Claude. Although their advertised “cost-per-token” figures are highly competitive, experiments reveal that Anthropic models can be 20–30% more expensive than GPT models.API Pricing — Claude 3.5 Sonnet vs GPT-4oAs of June 2024, the pricing structure for these two advanced frontier models is highly competitive. Both Anthropic’s Claude 3.5 Sonnet and OpenAI’s GPT-4o have identical costs for output tokens, while Claude 3.5 Sonnet offers a 40% lower cost for input tokens.Source: VantageThe hidden “tokenizer inefficiency”Despite lower input token rates of the Anthropic model, we observed that the total costs of running experiments (on a given set of fixed prompts) with GPT-4o is much cheaper when compared to Claude Sonnet-3.5.Why?The Anthropic tokenizer tends to break down the same input into more tokens compared to OpenAI’s tokenizer