Ryan Lucht, Senior Technical Advocate at Datadog and Marketing Lead at Eppo, widely regarded as one of the sharpest voices in modern experimentation talks about some misconceptions of experimentaiton. Ryan’s knowledge and technical depth, contrarian thinking, and cultural pragmatism has made him a leading figure in the experimentation community.
Ryan shares the origin story that hooked him into A/B testing — including the now-legendary experiment in which tripling the product price tripled revenue. From there, the conversation unfolds into an exploration of the ideas that have shaped his philosophy over the years.
Why some experiments behave like “Black Swans” — rare, extreme, massively impactful outliers that reshape your business.
The groundbreaking academic research showing that experiments follow a fat-tailed distribution, and why this changes everything.
Why companies should radically decrease the cost of experimentation, and how Amazon and Netflix operationalized exactly that.
How small and mid-sized companies can also reduce cost per experiment — even with limited traffic.
The hidden cultural blockers that silently kill experimentation programs.
Why win rate is the wrong metric — and what to use instead.
The real story behind “spaghetti testing,” the myth of best practices, and the surprising ways teams find breakthrough ideas.
Why democratizing experimentation starts with a designer who “just wants to try a button test.”
How confidence intervals (not p-values) and sequential testing can make experimentation safer and more accessible for everyone.
The mindset shift required to build an experimentation culture where people love being wrong.