Summary
On February 24, 2023, Meta announced LLaMA, a family of four large language models of 7 billion to 65 billion parameters, and offered the weights to researchers under a noncommercial license granted case by case. Meta reported that the 13-billion-parameter model outperformed OpenAI's much larger GPT-3 on most benchmarks. On March 3 the weights were posted as a torrent on 4chan and spread across the internet, which put a model close to the state of the art into anyone's hands and made LLaMA an early test case in the public argument over whether powerful model weights should be released.
What Happened
Meta's AI research group announced LLaMA (Large Language Model Meta AI) in a blog post and a research paper. The models came in 7B, 13B, 33B and 65B parameter sizes; the two largest were trained on 1.4 trillion tokens and the smallest on one trillion, using public data in the 20 languages with the most speakers. Meta argued that smaller models trained on more data were easier for researchers without large computing budgets to retrain, study and fine-tune. To "maintain integrity and prevent misuse," Meta released the models under a noncommercial license focused on research, with access granted case by case to academic researchers, government, civil society and industry research labs. Mark Zuckerberg announced the release on Facebook, saying Meta was "committed to this open model of research."
The research paper claimed that LLaMA-13B outperformed the 175-billion-parameter GPT-3 on most benchmarks, and that LLaMA-65B was competitive with DeepMind's Chinchilla-70B and Google's PaLM-540B. LLaMA was a base model without the conversational fine-tuning of ChatGPT, so it was a research tool rather than a product.
About a week after Meta began approving access requests, a downloadable torrent of the weights was posted on 4chan on March 3, 2023, and spread through AI communities. The Verge reported that researchers who compared the leaked files with the official release confirmed they matched. Meta's Joelle Pineau said in a statement that "some have tried to circumvent the approval process." On June 6, 2023, Senators Richard Blumenthal and Josh Hawley wrote to Mark Zuckerberg questioning whether Meta had assessed the risks of releasing LLaMA and asking how many people had received access before the leak.
Why It Matters
Meta offered outside researchers weights that it said matched or beat much larger closed models, and the leak made that access effectively universal. Because LLaMA-13B could run on a single data-center GPU, people well outside large labs could now run, study and fine-tune a model close to the state of the art, which Shawn Presser, one of the researchers who helped spread the leaked files, told The Verge was "a huge leap." Meta's next release, Llama 2 in July 2023, was licensed for commercial use, so the research-only gate of February lasted five months.
The leak became an early reference point in the open-versus-closed debate. Critics quoted by The Verge called open release "a terrible idea" that would bring personalized spam and phishing; supporters argued that wide access was needed to find and fix the problems of language models. The senators' letter brought the question to Congress, and it raised a point that later policy debates about open-weight models kept returning to: once weights are out, no access policy can call them back.
§ How to read the metadata
- Landmark
- Fundamentally alters the trajectory; 2–5 per year.
- Major
- Meaningfully shifts the landscape; 2–4 per month.
- Notable
- Worth documenting; significance can be upgraded later.
- Confidence
- High = primary sources corroborate. Medium = credible secondary only. Low = provisional. Disputed = credible sources disagree.
- Contestation
- Uncontested = no formal challenge. Contested = at least one challenge filed and not resolved by a correction. Corrected = a challenge was upheld and the entry corrected; see its revision history. Superseded = replaced by a later entry. Unresolved = dispute still open.