Product Mastery Now for Product Managers, Leaders, and Innovators

← Product Mastery Now for Product Managers, Leaders, and Innovators9 nov

617: Test your gut, don’t trust it –the scientific mindset for product teams – with Stefan Thomke, PhD

617: Test your gut, don’t trust it –the scientific mindset for product teams – with Stefan Thomke, PhD9 nov

Experiments to inform product decisions

Watch on YouTube

TLDR

Stefan Thomke, professor at Harvard Business School, and author of The Scientific Mindset: How Thinking Scientifically Drives Better Business Decisions, shares five elements of a scientific mindset, four types of uncertainty in innovation, and three tests of a good hypothesis. Learn why good product decisions are rooted in testing more and assuming less.

Introduction

When Stefan Thomke studied almost 22,000 online experiments (mostly A/B tests) run by more than 1,300 companies, only about one in five changes made a measurable difference. Half of the changes made things worse. If experienced teams can’t predict the effect of a small change, how confident should any of us be about a creating a new product?

Stefan is the William Barclay Harding Professor of Business Administration at Harvard Business School, where he has studied experimentation and innovation for more than three decades. He started as an engineer, learning experimental design to solve a production problem in a semiconductor plant. He is the author of Experimentation Matters and Experimentation Works. His new book, The Scientific Mindset: How Thinking Scientifically Drives Better Business Decisions, completes the trilogy.

Stefan was on the show in 2020 to explain how to run product experiments. This time we look at the thinking that decides which experiments are worth running and what you do with the answers. You’ll learn how to find the assumptions hiding in a new product idea using four kinds of uncertainty, how to turn an assumption into a strong hypothesis, and why “customers want this feature” isn’t yet a hypothesis.

Summary of Concepts Discussed for Product Managers

Five Elements of a Scientific Mindset

Stefan opens with a question: What did you do the last time you didn’t know what to do? He quotes Richard Feynman: “The first principle is that you must not fool yourself, and you’re the easiest person to fool.” The human brain is a belief engine. We see connections between unrelated things and confuse correlation with causation. To overcome this, we need to think scientifically about decisions.

Stefan identifies five elements of a scientific mindset that apply to business decisions: