The Mosaic research team at Databricks developed the new TAO method.

OLMo LLM AI2The Allen Institute for AI created the Open Language Model, or OLMo, which is an open-source large language model with the aim of advancing the science of language models through open research. It marks a major milestone in the evolution of large language models.Unlike current open large language models like Llama and Mistral, which might limit access to their training data, architectures, or evaluation methodologies, OLMo stands out by providing full access to its pre-training data, training code, model weights, and evaluation suite. This openness is aimed at empowering academics and researchers to collectively study and advance the field of language modeling.OLMo represents a collaborative effort to advance the science of language models. The developers behind the LLM have a mission to empower academics and researchers by providing access to training code, models, and evaluation code necessary for open research.OLMo's architecture is built on AI2’s Dolma dataset, which features a three trillion-token open corpus. It includes full model weights for four model variants at the 7B scale, each trained to at least 2T tokens. OLMo's innovative aspects include its training approaches, size, and the diversity of data it was trained on