Emergent Abilities of Large Language Models: Real or Metaphor?

This episode examines the 2022 survey "Emergent Abilities of Large Language Models," which argues that some few-shot abilities sit at chance level in small models and then jump sharply in larger ones. It sets up the contrast with smooth scaling laws for cross-entropy loss. It also traces where "emergence" comes from, namely Anderson's "More Is Different" and the loose phase-transition metaphor, and it shows that compute is only a proxy for scale, since data, architecture and training objective all shift where an ability appears. The discussion goes through the evidence: curves stitched together from separately trained models such as GPT-3, LaMDA, Gopher and PaLM, with only a few points per family. Examples include BIG-Bench arithmetic, TruthfulQA, MMLU, and chain-of-thought prompting on GSM8K, which helps only past roughly 10^23 FLOPs. Listeners get a skeptical look at whether the paper actually demonstrates unpredictability, or only builds it into the definition of emergence.

Sources:

1. Emergent Abilities of Large Language Models — Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, William Fedus, 2022

http://arxiv.org/abs/2206.07682

2. Emergent Abilities of Large Language Models — Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, William Fedus, 2022

https://scholar.google.com/scholar?q=Emergent+Abilities+of+Large+Language+Models

3. Are Emergent Abilities of Large Language Models a Mirage? — Rylan Schaeffer, Brando Miranda, Sanmi Koyejo, 2023

https://scholar.google.com/scholar?q=Are+Emergent+Abilities+of+Large+Language+Models+a+Mirage%3F

4. Predictability and Surprise in Large Generative Models — Deep Ganguli, Danny Hernandez, Liane Lovitt, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Nelson Elhage, Sheer El Showk, Stanislav Fort, Zac Hatfield-Dodds, Tom Henighan, Scott Johnston, Andy Jones, Nicholas Joseph, Jackson Kernian, Shauna Kravec, Ben Mann, Neel Nanda, Kamal Ndousse, Catherine Olsson, Daniela Amodei, Tom Brown, Jared Kaplan, Sam McCandlish, Christopher Olah, Dario Amodei, Jack Clark, 2022

https://scholar.google.com/scholar?q=Predictability+and+Surprise+in+Large+Generative+Models

5. Beyond t