Book · 2020
Trustworthy Online Controlled Experiments
by Ron Kohavi
The A/B testing manual that Google, Microsoft, Facebook, and LinkedIn all point at when someone asks how to run experiments correctly.
What does Reddit think of Trustworthy Online Controlled Experiments?
All 20 Reddit mentions of Trustworthy Online Controlled Experiments live in r/datascience, and the community treats it as a reference rather than a reading experience. Nobody gushes; nobody trashes it. It appears repeatedly in canonical reading lists alongside Causal Inference: the Mixtape and Mostly Harmless Econometrics, occupying the applied-experimentation slot that the other two don't fill. One commenter flagged it as "heavily recommended by pretty much all FAANGs" for A/B testing, and that framing stuck. The site experimentguide.com is cited as a free companion, which is either generous or a sign the book does the heavy lifting online. The ↑76 quote adds a useful caveat: even owning this book won't substitute for direct professional experience when hiring managers screen candidates.
Community feedback & reader fit
Themes
- · A/B testing methodology
- · causal inference in industry
- · experiment design and validity
- · data science career preparation
- · applied statistics at tech companies
Common praise
- + Gets cited by Google, Microsoft, Facebook, and LinkedIn as the go-to applied resource on online experiments.
- + Fills the practical gap that purely academic causal-inference texts leave open.
- + The companion site experimentguide.com lets you get the core material without paying, which r/datascience repeatedly links.
- + Shows up in every serious data science reading list in the sub, often alongside Mostly Harmless Econometrics as the applied counterpart.
Common criticism
- − Owning the book does not substitute for hands-on experimentation experience — a ↑76 comment makes that explicit when discussing hiring.
- − All 20 mentions are list appearances, not worked discussions, which suggests readers recommend it more than they actually finish it.
Who it's for
Data scientists preparing for roles at companies that run experiments at scale — this is the book hiring managers at the large tech platforms implicitly expect you to know. If your current job involves any A/B testing, even informally, it gives the framework to do it without embarrassing yourself. Pair it with Causal Inference: the Mixtape for the econometric foundations the book assumes. Pure analysts who will never ship a feature don't need it; product data scientists who will absolutely do.
Mentions over time
Top subreddits
Which Reddit comments matter for Trustworthy Online Controlled Experiments?
Top-upvoted quotes across the subreddits where this book is mentioned. Click through to read the full thread.
“The Bible is technically a series of books that form a cohesive narrative. In that sense, here is my Bible of Data Science roughly divided into a classical stats OT and a more modern ML NT: **The Law** - The mathematical foundations [Statistical Inference](
“Trustworthy Online Controlled Experiments is the best applied resource on the subject IMO. But keep in mind that in the current environment, hiring managers can and will prioritize candidates with direct prof…
“A list of books based on recommendations here, fivebooks.com, or on LI from those I follow. These are listed in no particular order and are not meant to be the only "valid" recommendations. Just ones on my reading list based on my areas of interest. Feel free to recommend others. **Data Science, br…
“Some of my favorite resources that I can vouch for: * Cracking the Coding Interview is great for approaching Python questions * LeetCode is the GOAT when it comes to interactive python coding practice * Chip Hyun's free Machine Learning Interviews [Github Book](
“For A/B testing there's a really solid free course recommended by Facebook, Google, Microsoft, and LinkedIn: A more comprehensive approach is this book (heavily recommended by pretty much all faangs…
“Trustworthy online controlled experiments
“I prefer to keep one good book per topic. One such book for AB testing that i sometimes page through is: Trustworthy Online Controlled Experiments (Not a part of my actual job, but since I want to move to product analytics some day)
“In my last role, I defined a lot of the experimentation framework the company ended up adopting. How did I get here? Solid background in inferential statistics (b/c I did a PhD in biostats). However, academic/medical stats is very different and you kind of need to orient yourself to tech experime…
Convinced? Pick up Trustworthy Online Controlled Experiments
What else does r/datascience read?
Other books mentioned in the same sub, ranked. Shared-sub overlap with this title breaks ties.
Hands-On Machine Learning with Scikit-Learn, Keras, and TensorFlow
Aurélien Géron
Aurélien Géron's end-to-end ML textbook: build a working neural net with Scikit-Learn, Keras, and TensorFlow before you finish the first half.
Designing Data-Intensive Applications
Martin Kleppmann
A backend engineer's field guide to the tradeoffs behind every database, queue, and distributed system you will ever touch.
Clean Code
Robert C. Martin
The book 536 Reddit threads cite when arguing about naming variables — revered by beginners, argued over by seniors.
Fluent Python
Luciano Ramalho
The Python book r/Python recommends when you're done with the tutorials and need to know what the language actually does.
The Pragmatic Programmer
Andy Hunt
Two veterans hand you a checklist for the craft: don't write code you don't need, own your tools, and fix the broken window before someone else does.
Pattern Recognition and Machine Learning
Christopher M. Bishop
Bishop's 2006 Bayesian ML textbook: 26 of its 39 Reddit mentions come from r/MachineLearning, where it sits permanently on the serious-math shelf.
Trustworthy Online Controlled Experiments — frequently asked
What does Reddit think of Trustworthy Online Controlled Experiments?+
Broadly positive but quiet. All 20 mentions are in r/datascience, and the sub lists it repeatedly without debating it much. The ↑76 comment calls it "the best applied resource on the subject," and multiple commenters note it is recommended by FAANG-tier companies. Seventeen unique commenters cited it over seven years, which is steady rather than viral.
Is Trustworthy Online Controlled Experiments worth buying if the experimentguide.com site is free?+
Depends on how you read. r/datascience links experimentguide.com as a free companion, so the core material is accessible without spending. The book likely earns its price for someone doing a structured read-through rather than reference lookups. No commenter has argued the paid version adds something the site lacks, though none have argued it doesn't either.
Should I read Trustworthy Online Controlled Experiments before a data science interview?+
Yes for methodology fluency, but a ↑76 r/datascience comment is direct that hiring managers still prioritize candidates with hands-on experimentation experience over those who have only read about it. The book signals the right vocabulary and framework; it doesn't replace the resume line.
How does Trustworthy Online Controlled Experiments compare to Causal Inference: the Mixtape?+
They get recommended together in r/datascience, not as alternatives but as complements. The Mixtape and Mostly Harmless Econometrics supply the econometric theory; this book covers applied experiment design at tech companies. If you want to understand why an experiment was valid, read both.