BABILong: Testing LLM Long-Context Reasoning in a Haystack

This episode examines BABILong, a NeurIPS 2024 benchmark that asks how much of a language model's advertised 128K-plus context window it can use for reasoning. It argues that the popular needle-in-a-haystack test mostly measures lookup, which strong models pass almost perfectly. BABILong hides the sentences of the synthetic bAbI reasoning tasks (chaining facts, counting, temporal ordering) inside long passages of PG19 book text. Answering then requires finding and combining several scattered facts, and the benchmark can be generated at any length, up to millions of tokens, without risk of training-set contamination. The discussion defines "effective context size" as the longest input at which accuracy stays above 85%, and it flags that this threshold is a judgment call. It also sets up comparisons between long-context LLMs, retrieval-augmented generation, and small recurrent-memory and state-space models. The hosts note that some of the winning models are trained on the task and others are not, so listeners get both the benchmark's design and a caution about reading its results.

Sources:

1. BABILong: Testing the Limits of LLMs with Long Context Reasoning-in-a-Haystack — Yuri Kuratov, Aydar Bulatov, Petr Anokhin, Ivan Rodkin, Dmitry Sorokin, Artyom Sorokin, Mikhail Burtsev, 2024

http://arxiv.org/abs/2406.10149

2. Towards AI-Complete Question Answering: A Set of Prerequisite Toy Tasks (bAbI) — Jason Weston, Antoine Bordes, Sumit Chopra, Alexander M. Rush, Bart van Merriënboer, Armand Joulin, Tomas Mikolov, 2015 (ICLR 2016)

https://scholar.google.com/scholar?q=Towards+AI-Complete+Question+Answering%3A+A+Set+of+Prerequisite+Toy+Tasks+%28bAbI%29

3. Lost in the Middle: How Language Models Use Long Contexts — Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, Percy Liang, 2023 (TACL 2024)

https://scholar.google.com/scholar?q=Lost+in+the+Middle%3A+How+Language+Models+Use+Long+Contexts

4. RULER: What's the Real Context Size of Your Long-Context Language Models? — Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, Yang Zhang, Boris Ginsburg (NVIDIA), 2024 (COLM 2024)

https://scholar.google.com/scholar?q=RULER%3A+What%27s+the+Real+Context+Size+of+Your+Long-Context+Language+Models%3F

5. NoLiMa: Long-Context Evaluation Beyond Literal Matching — Ali Modarressi, Hanieh Deilamsalehy, Franck Dernoncourt, Trung Bui, Ryan A. Rossi, Seunghyun Yoon, Hinrich Schütze (Adobe Research / LMU), 2025