Automatic Curriculum Learning Through Value Disagreement in Reinforcement Learning

This episode examines "Automatic Curriculum Learning through Value Disagreement," a NeurIPS 2020 paper proposing that a reinforcement learning agent can choose its own training goals. It walks through the background: goal-conditioned RL with sparse rewards, Hindsight Experience Replay as an implicit curriculum, and earlier explicit approaches like Goal GAN and learning-progress teachers, which found that plain uniform sampling is a surprisingly strong baseline. The core idea is to train an ensemble of Q-functions and sample goals in proportion to the standard deviation of their predictions. The reasoning is that the ensemble agrees on easy goals and on hopeless ones, but splits on goals at the frontier of competence. The discussion explains the method as a simple loop: draw candidate goals, score each by ensemble disagreement, and sample one. It needs no adversary or success classifier, and the policy learner never sees the ensemble. Throughout, the hosts treat the disagreement-marks-the-frontier hypothesis as a claim to test rather than accept, which makes this a useful look at what a cheap uncertainty signal can and can't tell a learning agent.

Sources:

1. Automatic Curriculum Learning Through Value Disagreement in Reinforcement Learning

https://proceedings.neurips.cc/paper/2020/file/566f0ea4f6c2e947f36795c8f58ba901-Paper.pdf

2. Curriculum Learning — Yoshua Bengio, Jérôme Louradour, Ronan Collobert, Jason Weston, 2009

https://scholar.google.com/scholar?q=Curriculum+Learning

3. Automatic Goal Generation for Reinforcement Learning Agents (Goal GAN) — Carlos Florensa, David Held, Xinyang Geng, Pieter Abbeel, 2018

https://scholar.google.com/scholar?q=Automatic+Goal+Generation+for+Reinforcement+Learning+Agents+%28Goal+GAN%29

4. Teacher-Student Curriculum Learning — Tambet Matiisen, Avital Oliver, Taco Cohen, John Schulman, 2017 (IEEE TNNLS 2019)

https://scholar.google.com/scholar?q=Teacher-Student+Curriculum+Learning

5. Automatic Curriculum Learning For Deep RL: A Short Survey — Rémy Portelas, Cédric Colas, Lilian Weng, Katja Hofmann, Pierre-Yves Oudeyer, 2020

https://scholar.google.com/scholar?q=Automatic+Curriculum+Learning+For+Deep+RL%3A+A+Short+Survey