“Character Training for Risk-Averse Agents” by arav-dhoot, Punya Syon Pandey, Jamie Johnson, Daniel Tan, Elliott Thornley, David Africa

<p> Tl;Dr:</p><ul> <li value="1">We might be able to reduce harm from misaligned AI if it was sufficiently risk-averse, as it could be paid to cooperate rather than rebel (Thornley &amp; MacAskill 2026).</li><li value="2">We think character training might be a good way to do this.</li><li value="3">We use a short spec/constitution to character train Gemma and Qwen to be risk averse.</li><li value="4">This performs better than baselines out of distribution, and doesn't require labelled data.</li><li value="5">In terms of results:<ul> <li value="1">We find that token budget and model family are very influential here. &nbsp;</li><li value="2">We also find that writing the constitution declaratively ("I am...") works better than procedurally ("Whenever X, I...").</li><li value="3">We didn't find that increasing the number of examples worked well.</li><li value="4">We also find that preference for myopic reward also increased, which is worrying.</li></ul></li><li value="6">We think that character training is a viable and exciting way to improve the alignment of models, and that risk-aversion may be a good target here. But, it's important to understand character training well, especially since undesirable behaviours can also leak.</li></ul><p><strong> Introduction</strong></p><p> Thornley and MacAskill (2026) argue that paying misaligned AIs to cooperate is unaffordable if they are risk-neutral. A sufficiently risk-averse agent, by contrast, has steeply diminishing marginal [...]</p> <p>---</p><p><strong>Outline:</strong></p><p>(01:36) Introduction</p><p>(02:56) Approach</p><p>(04:32) Results</p><p>(07:49) Discussion</p> <p>---</p>

<p><b>First published:</b><br/>

October 16th, 2026 </p>

<p><b>Source:</b><br/>

<a href="https://www.lesswrong.com/posts/HXkQaWpuaxG6fMkTm/character-training-for-risk-averse-agents?utm_source=TYPE_III_AUDIO&utm_medium=Podcast&utm_content=Source+URL+in+episode+description&utm_campaign=ai_narration" rel="noopener noreferrer" target="_blank">https://www.lesswrong.com/posts/HXkQaWpuaxG6fMkTm/character-training-for-risk-averse-agents</a> </p>

<p>---</p>

<p>Narrated by <a href="https://type3.audio/?utm_source=TYPE_III_AUDIO&utm_medium=Podcast&utm_content=Narrated+by+TYPE+III+AUDIO&utm_term=lesswrong&utm_campaign=ai_narration" rel="noopener noreferrer" target="_blank">TYPE III AUDIO</a>.</p>

<p>---</p><div style="max-width: 100%";><p><strong>Images from the article:</strong></p><a href="https://res.cloudinary.com/lesswrong-2-0/image/