LoCoMo Benchmark: Do LLM Agents Remember Very Long Conversations?

This episode examines LoCoMo, a benchmark for testing whether LLM agents remember across very long conversations. The conversations average about 9,200 tokens, 19 sessions and 305 turns, roughly nine times longer than earlier datasets such as Multi-Session Chat. The episode covers how the conversations are generated: two LLM agents get personas and a temporal event graph of up to 25 causally linked life events. Human annotators then edited about 15 percent of turns, which raises questions about how much of the data was verified. It compares three ways a model can handle history: truncated context, long-context models, and retrieval-augmented generation. It also walks through the question-answering categories (single-hop, multi-hop, temporal, open-domain, adversarial) and F1 scoring. The paper's headline finding is that every approach falls well short of human performance. The episode also previews how later memory products such as MemGPT, Mem0, and Zep use the benchmark, and the leaderboard disputes that followed.

Sources:

1. Evaluating Very Long-Term Conversational Memory of LLM Agents — Adyasha Maharana, Dong-Ho Lee, Sergey Tulyakov, Mohit Bansal, Francesco Barbieri, Yuwei Fang, 2024

http://arxiv.org/abs/2402.17753

2. Evaluating Very Long-Term Conversational Memory of LLM Agents (LoCoMo) — Adyasha Maharana, Dong-Ho Lee, Sergey Tulyakov, Mohit Bansal, Francesco Barbieri, Yuwei Fang, 2024 (ACL 2024)

https://scholar.google.com/scholar?q=Evaluating+Very+Long-Term+Conversational+Memory+of+LLM+Agents+%28LoCoMo%29

3. Recursively Summarizing Enables Long-Term Dialogue Memory in Large Language Models — Qingyue Wang, Liang Ding, Yanan Cao, Zhiliang Tian, Shi Wang, Dacheng Tao, Li Guo, 2023

https://scholar.google.com/scholar?q=Recursively+Summarizing+Enables+Long-Term+Dialogue+Memory+in+Large+Language+Models

4. MemGPT: Towards LLMs as Operating Systems — Charles Packer, Sarah Wooders, Kevin Lin, Vivian Fang, Shishir G. Patil, Ion Stoica, Joseph E. Gonzalez, 2023

https://scholar.google.com/scholar?q=MemGPT%3A+Towards+LLMs+as+Operating+Systems

5. Generative Agents: Interactive Simulacra of Human Behavior — Joon Sung Park, Joseph C. O'Brien, Carrie J. Cai, Meredith Ringel Morris, Percy Liang, Michael S. Bernstein, 2023

https://scholar.google.com/scholar?q=Generative+Agents%3A+Interactive+Simulacra+of+Human+Behavior