Manager – Technical Delivery

Introducing BigLaw Bench: research. BLB: Research helps Harvey AI identify how foundation models are improving on research tasks, enabling investigation of ways to improve AI-based legal research. The second of its major BigLaw Bench (BLB) expansions this quarter is BLB: Research. This dataset focused on hard agentic legal research problems. Working with its data partner Snorkel AI, a leader in creating complex expert data for frontier AI, Harvey AI identified a series of US Case Law research problems that leading models are currently unable to solve - even when provided with search tools like web search. The purpose of BLB: Research is twofold. First, to identify how foundation models are improving on research tasks and the failure modes that continue to cause them to return unsatisfactory answers. Second, to investigate ways to improve AI-based legal research through agentic systems and access to non-public data in addition to foundation model capabilities. By enabling Harvey AI to both identify the best model and to build better infrastructure for those models, BLB: Research helps Harvey AI build deeper and more accurate research capabilities for its customers. Benchmarking beyond models. BLB: Research is its first benchmark that requires models to operate end-to-end over more than documents. Models must use tools to perform searches in order to identify relevant research context and provide a grounded, cited response. While historically Harvey AI had measured search capabilities independently, evolutions in agent building and model deployment led Harvey AI to combine search and response into a unified benchmark. The main driver of this change is the growing consensus on search as the primary means for providing grounded model responses across the industry. On the research side, as models are increasingly trained to use search rather than training data for up-to-date knowledge, benchmarking their capabilities without search unrealistically undersells what they can do. "Building a holistic search benchmark allows us to measure performance in the way that tracks how research capabilities are being developed and how our customers expect them to be delivered." Search is also becoming table stakes in practice, with customers relying on knowledge sources and expecting pin-cited sources as the default for research-driven use cases. Building a holistic search benchmark allows Harvey AI to measure performance in the way that tracks how research capabilities are being developed and how its customers expect them to be delivered. Building realistic complexity. Given its ubiquity as a paradigm, models are, perhaps unsurprisingly, very good at search. They can write booleans and web searches and have learned to use search to explore, understand, and refine their research. A primary objective for BLB: Research was to validate these baseline capabilities, as well as identify areas in which the models were still not capable of turning their search capabilities into meaningful research outcomes for lawyers. In short, Harvey AI wanted to create a hard benchmark. A good way to make benchmarks hard is to make them esoteric. This is unfortunately also a great way to make a benchmark useless. While Harvey AI could show the models fail at a number of tasks, if succeeding at those tasks wouldn't actually help a legal professional, that failure is beside the point. Accordingly, Harvey AI needed to ensure that tasks were difficult and realistic. Realistic tasks are both intrinsically useful and represent a research pattern that is generally useful, such as identifying the best case to cite for a proposition, drafting a research memo, or planning a claim or defense. From these realistic task types Harvey AI then had to find those that were hard. Harvey AI did this not by predicting what should be hard but by actually finding the frontier of model capabilities. First, Harvey AI identified what failure looked like on a structured rubric: How poorly must a model perform before its answer is unhelpful to a legal professional? Harvey AI found that many partly correct answers are still helpful. For example, slightly-off analysis with good citations to principal cases still gives users a research head start. In general, Harvey AI found that model answers become unhelpful once the model completes less than 60% of the required task criteria. On these tasks, models are often missing critical reasoning junctures, making wrong turns in their research, or providing answers that are too surface level to be useful. Here are some research tasks that Harvey will use to benchmark models' research capabilities: Other practice areas included in the benchmark are: Intellectual Property, Commercial Litigation, Constitutional Law, Regulatory, Employment & Labor, Health Law & Life Sciences, Tax, Tort, Real Property, Media and Technology, Immigration, and Family Law. What's next for BLB: Research. Sifting through troves of documents to find answers, narratives, and trends is one of the most common tasks in legal practice. The same search capabilities that make models good at case law research also underpin the ability to search EDGAR, an investigation corpus, a deal room, or even a law firm's own internal knowledge. BLB: Research will build towards measuring all these capabilities and finding the right tools for enabling models to move lawyers from sifting to solving. The next step-change in search capabilities can come from various sources: model capabilities, unique data sources, or the tools to enable the former to interact with the latter. BLB: Research will allow Harvey AI to track and design innovations in all of these spaces, and convert them into the best possible research results for its customers.

Accenture has invested in Snorkel AI through Accenture Ventures to enhance AI deployments in the financial services sector. The collaboration aims to create industry-specific solutions for training AI models at scale. Snorkel, originating from Stanford AI Lab, specializes in programmatic data development. Recently, Snorkel raised $100 million in a Series D round, valuing it at $1.3 billion. This investment aligns with Accenture's contract with NatWest Group for AI-driven banking transformation.