Paul Rosania

Hi! I'm Paul. I work at Slack.
You should follow me at @ptr.
I can also be reached at paul@rosania.org.

A/B Testing Will Save Your Career

July 21, 2012

A company is the sum of hundreds of decisions made daily about product, marketing, hiring, etc. The magnitude of each decision varies, but it's rare to find one decision (or several) that is single-handedly responsible for your business' success or failure. What matters is the average: on the whole, does your business get more decisions right than wrong?

It's tempting to conclude that the best company is the one whose people have the best natural intuition. These people, blessed with divine judgment, will carve market-winning products directly out of stone, while their competitors misfire and fall behind.

In such a scenario, people with below-average intuition must be ruthlessly culled in order to maintain above-market performance. As an employee of this company, each decision you make carries the weight of your future career on its shoulders. (Better not fuck up!)

And yet, we all know intuition is fallible, mistakes inevitable. So how do you chart success for your career?

One option is to be lucky. (If you're lucky, you can stop reading here.)

Another option is to embrace A/B testing. A/B testing directs the conversation away from luck and intuition, and toward data-driven learning. In an A/B testing culture, your intuition is not your most prized asset. Rather, your value comes from your ability to brainstorm, define and execute experiments. Instead of focusing on being right, and approaching decisions with dread, you can focus on learning as much as possible by rapidly exploring different solutions to the areas of your business and product you are working on.

One benefit of A/B testing is never making decisions that are quantitatively poor. But there's a long term benefit as well: A/B testing focuses you and your team on the process of learning and building organizational intuition. It enables people to propose ideas that may sound silly or dumb without fear of retribution. And it enables people to gracefully accept defeat when their ideas are rejected by data. The company benefits greatly, as everyone iterates and learns more quickly.

As an individual contributor, you are ultimately judged by your impact. If you use your own judgment to make decisions, you will be graded on your intuition. But if you focus on learning via testing, you will be graded not on intuition, but by the velocity of your experimentation. Unless you're lucky, testing affords you much more control of the outcome.