Book · 2020
Trustworthy Online Controlled Experiments
by Ron Kohavi
A data scientist keeps one book on A/B testing on their desk for a job they don't have yet.
What does Reddit think of Trustworthy Online Controlled Experiments?
r/datascience is the only subreddit that mentions this book in the tracked corpus. It comes up as a specific, repeatable answer: the same title named again across different threads, not one option buried in a long list. One comment calls it flatly "the best applied resource on the subject" (↑76), then immediately complicates the endorsement: hiring managers currently prioritize candidates with direct professional experience over people who just read the right books. Another comment frames it as "heavily recommended by pretty much all faangs," pairing it with a free Udacity course as the fuller path into A/B testing (↑28). The most specific mention (↑23) is almost incidental: a data scientist who doesn't do experimentation in their current role keeps this as their one book on the topic, explicitly because they want to move into product analytics eventually. That's the pattern across this sample: people not currently doing this work, reading it anyway, on purpose.
Community feedback & reader fit
Themes
- · The default answer to 'which A/B testing book' in r/datascience
- · Recommended despite not being required for the reader's current job
- · FAANG-endorsed as a specific, checkable credibility marker
- · One book per topic, chosen deliberately over interview-prep material
Common praise
- + Called "the best applied resource on the subject" outright, not hedged (↑76).
- + Grouped with material "heavily recommended by pretty much all faangs," a specific, checkable claim instead of vague praise (↑28).
- + Kept as someone's single reference book for a role they don't have yet, a different kind of endorsement than a class assignment (↑23).
Common criticism
- − The same top comment praising the book (↑76) warns that reading it won't outweigh a hiring manager's preference for candidates with direct experimentation experience.
- − Every mention in this sample comes from one subreddit, so there's nothing here on how it lands with people outside data science specifically.
- − The bare mention (↑25) is just the title with no elaboration, which limits how much this sample can say about what readers found useful inside it.
Who it's for
Want in on A/B testing but don't currently touch it in your job? That's the exact position of the r/datascience commenter who keeps this as their one book on the subject while aiming at product analytics (↑23). Pair it with the free Udacity A/B testing course for the fuller on-ramp described in the same thread (↑28). Chasing a title alone won't move a hiring manager, per the top comment here: expect this book to sharpen your thinking, not replace direct experimentation experience on a resume (↑76).
Mentions over time
Top subreddits
Which Reddit comments matter for Trustworthy Online Controlled Experiments?
The most relevant excerpts across the subreddits where this book is mentioned — opinionated, argued takes first, then top-upvoted mentions. Click through to read the full thread.
“The Bible is technically a series of books that form a cohesive narrative. In that sense, here is my Bible of Data Science roughly divided into a classical stats OT and a more modern ML NT: **The Law** - The mathematical foundations [Statistical Inference](
“Trustworthy Online Controlled Experiments is the best applied resource on the subject IMO. But keep in mind that in the current environment, hiring managers can and will prioritize candidates with direct prof…
“A list of books based on recommendations here, fivebooks.com, or on LI from those I follow. These are listed in no particular order and are not meant to be the only "valid" recommendations. Just ones on my reading list based on my areas of interest. Feel free to recommend others. **Data Science, br…
“Some of my favorite resources that I can vouch for: * Cracking the Coding Interview is great for approaching Python questions * LeetCode is the GOAT when it comes to interactive python coding practice * Chip Hyun's free Machine Learning Interviews [Github Book](
“For A/B testing there's a really solid free course recommended by Facebook, Google, Microsoft, and LinkedIn: A more comprehensive approach is this book (heavily recommended by pretty much all faangs…
“Trustworthy online controlled experiments
“I prefer to keep one good book per topic. One such book for AB testing that i sometimes page through is: Trustworthy Online Controlled Experiments (Not a part of my actual job, but since I want to move to product analytics some day)
“In my last role, I defined a lot of the experimentation framework the company ended up adopting. How did I get here? Solid background in inferential statistics (b/c I did a PhD in biostats). However, academic/medical stats is very different and you kind of need to orient yourself to tech experime…
“Reasonable books to start you with: * "Trustworthy Online Controlled Experiments" by Kohavi et al. * "Statistical Methods in Online A/B Testing" by Georgi Georgiev Also for a more holistic view consider: "Optimal Design of Experiments: A Case Study Approach" by Goos & Jones.
“Hi, I’m Vetted AI Bot! I researched the **Trustworthy Online Controlled Experiments** and I thought you might find the following analysis helpful. **Users liked:** * Book provides practical advice for running experiments (backed by 10 comments) * Book covers wide range of topics related to experime…
“Been running A/B tests on the job for 8 years. I guess I first learned the ropes from my first boss and then learned more about best practices and creating a good test & learn culture from my current team. But also the book *Trustworthy Online Controlled Experiments* is a good place to start.
“Trustworthy Online Controlled Experiments by Diane Tang, Ron Kohavi, and Ya Xu. That is the bible of A/B testing
“I appreciate the informative post, thanks. I've been meaning to read Trustworthy Online Controlled Experiments for months now if not a year, I even have a copy and skimmed the first chapter... I think that you being person number 947 to speak highly of it might be what pushes me over the edge, haha!
Convinced? Pick up Trustworthy Online Controlled Experiments
What else does r/datascience read?
Other books mentioned in the same sub, ranked. Shared-sub overlap with this title breaks ties.
Trustworthy Online Controlled Experiments — frequently asked
Is Trustworthy Online Controlled Experiments the best book on A/B testing, according to Reddit?+
The top comment in this sample says so directly: "the best applied resource on the subject" (↑76), posted in r/datascience. The same comment cautions that hiring managers currently weigh direct professional experimentation experience more heavily than reading credentials, so the book's value is framed as preparation, not a substitute.
Do you need to work in experimentation to benefit from Trustworthy Online Controlled Experiments?+
No, based on the clearest example here. One r/datascience commenter (↑23) keeps it as their single book on the topic despite it not being part of their current job, specifically because they're aiming to move into product analytics later.
Is Trustworthy Online Controlled Experiments actually used at big tech companies?+
One comment in this sample claims it's "heavily recommended by pretty much all faangs" (↑28), pairing it with a free Udacity A/B testing course as a fuller learning path. That's one commenter's characterization, not a company-by-company confirmation.