How People Inc. Turned Bot Blocking Into a Business, With Jonathan Roberts

Blocking the bots is step one. People Inc.'s Jonathan Roberts on what comes next: the companies reselling publisher content, who sets the price, and why AI is spending its money in the wrong place.

Pete Pachal talks with Jonathan Roberts, chief innovation officer at People Inc. (People, InStyle, Investopedia), about blocking every bot by default and allowing 38 through, the disguises blocked bots wear, why robots.txt is a chocolate teapot, the "picks and shovels" companies taking content for free and selling it for money, the 18 Napsters of content with no Spotify, serving AI companies Markdown instead of HTML, who gets to set the price, per crawl versus per use, why models without reference are fiction writers, why inputs now matter more than the model, TIME and Mobian's ads for bots, and why automating the chore should never mean automating the choice.

Chapters

0:00 Why the demand for information moved upstream

2:30 Jonathan Roberts, welcome back to the Media Copilot

2:36 Can smaller publishers really block bots?

4:29 Yes, and the future is hungrier for content than ever

6:47 Why robots.txt is a chocolate teapot

8:17 The CDNs are the new front line

9:48 Block everything, allow 38

10:30 Blocked bots put on disguises

12:11 Botnets, hijacked smart TVs and the FBI