Join our daily and weekly newsletters for the latest updates and exclusive content on industry-leading AI coverage. Learn More. Advances in large language models (LLMs) have lowered the barriers to creating machine learning applications. With simple instructions and prompt engineering techniques, you can get an LLM to perform tasks that would have otherwise required training custom machine learning models. This is especially useful for companies that don’t have in-house machine learning talent and infrastructure, or product managers and software engineers who want to create their own AI-powered products.However, the benefits of easy-to-use models are not without tradeoffs. Without a systematic approach to keeping track of the performance of LLMs in their applications, enterprises can end up getting mixed and unstable results. Public benchmarks vs custom evalsThe current popular way to evaluate LLMs is to measure their performance on general benchmarks such as MMLU, MATH and GPQA

Braintrust, an AI evaluation and monitoring startup, raised $36 million in Series A funding led by Andreessen Horowitz, with participation from Datadog and Databricks Ventures. The funding aims to enhance AI accuracy for clients like Notion and Zapier by enabling continuous experimentation and real-time monitoring. This investment will help Braintrust expand its customer base, including companies like Airtable, Instacart, and Stripe, and develop tools to address AI inaccuracies.