Code Predicts, Jev Judges: Separating Simulation from Evaluation

This episode examines a paper on a closed, API-only action-selection model (Jev) that returns a probability for each option in a single call, with no reasoning text. The paper's question is which decisions such a model can be trusted with. It separates "evaluation" decisions, where the answer can be read from the input and options, from "simulation" decisions, where the right option depends on an unstated prediction such as an opponent's move or a required subgoal like washing a knife before placing it. The discussion works through the paper's test setups, random three-by-three payoff games with iterated dominance depth and the ALFWorld household text game. It also covers the "lure" idea: a fixed surface shortcut that sometimes points to the wrong option, which separates real competence from luck. The paper's proposed fix is to have code do the predicting and the model do the judging. Listeners get a clear framework for why a model can ace trap questions yet fail a simple household step, and for what that means for building cheap, fast agent loops.

Sources:

1. Code Owns the Simulation, Jev Owns the Evaluation — Yaodong Yang, Hongyao Tang, Yi Ma, Xingyu Fan, Weixun Wang, Jinpeng Li, Tianpei Yang, 2026

http://arxiv.org/abs/2610.01834

2. Thinking, Fast and Slow (book) / Dual-Process Theories of Higher Cognition: Advancing the Debate — Daniel Kahneman (2011); Jonathan St. B. T. Evans and Keith E. Stanovich (2013), 2011 / 2013

https://scholar.google.com/scholar?q=Thinking%2C+Fast+and+Slow+%28book%29+%2F+Dual-Process+Theories+of+Higher+Cognition%3A+Advancing+the+Debate

3. Cognitive Reflection and Decision Making (the Cognitive Reflection Test); Human-like intuitive behavior and reasoning biases emerged in large language models but disappeared in ChatGPT — Shane Frederick (2005); Thilo Hagendorff, Sarah Fabi, Michal Kosinski (2023), 2005 / 2023

https://scholar.google.com/scholar?q=Cognitive+Reflection+and+Decision+Making+%28the+Cognitive+Reflection+Test%29%3B+Human-like+intuitive+behavior+and+reasoning+biases+emerged+in+large+language+models+but+disappeared+in+ChatGPT

4. Do As I Can, Not As I Say: Grounding Language in Robotic Affordances (SayCan) — Michael Ahn, Anthony Brohan, et al. (Google Robotics / Everyday Robots), 2022

https://scholar.google.com/scholar?q=Do+As+I+Can%2C+Not+As+I+Say%3A+Grounding+Language+in+Robotic+Affordances+%28SayCan%29

5. SwiftSage: A Generative Agent with Fast and Slow Thinking for Complex Interactive Tasks — Bill Yuchen Lin