Here's a truth I've discovered after coaching countless leaders and teams: They often get stuck in their comfort zones, even when experimenting.
Experiments are the fastest way to expose and reduce risk by quickly testing the most critical assumptions.
However, many begin to run the same experiment type even when it doesn't test their riskiest assumption. They are trying to rapidly learn if the customer will do what the team believes - their hypothesis.
The hypothesis should guide the experiment design.
The hypothesis is nothing more than a restatement of the team's critical assumption that makes it crystal clear what they are testing for and how they will measure whether the team's hypothesis has been validated.
Let's look at different experiment types(see the image below for definitions):
- Smoke Test or Landing Page
- A/B Test
- Mechanical Turk/Wizard of Oz
- Concierge
- Dry-Wallet
- High Hurdle
- Imposter Judo
- Prototype
**The Million-Dollar Questions**
Before launching any experiment, teams and their leaders ask:
- Are we crystal clear on our riskiest assumption? (And are we testing it?)
- Are we targeting the right customers? (Real ones, not just convenient stand-ins)
- Are we tracking specific customer behaviors? (Vague observations won't cut it)
- Can we measure success? (If not, we're just guessing)
- Are we testing one thing at a time? (Multiple variables = muddy results)
- Could we run this faster and simpler? (Speed of learning beats perfection)
Here's the bottom line: Effective experimentation isn't about complexity - it's about clarity.
You're essentially asking, "What do we believe customers will do, what experiment can we run to test that, and how many need to do it before we're confident we're on the right track, and how will we measure the results?"
Remember, the goal isn't to run any experiment; it's to mitigate risk by testing critical assumptions to learn fast and iterate faster.
So next time you and the team plan an experiment, step back and ask: Are we testing what matters? The answers might surprise you - and that's exactly the point.
Written By: Pam Krengel