The Secrets of Experimentation

What they don't tell you about experimentation is that it's actually two distinctly different behaviors that are often called by the same name: exploring and testing. To experiment effectively, you need to be comfortable engaging in both modes of behaviors and - critically - you need to know which one you're doing at any given time.

The two modes of behavior each have their own purpose. Exploring is about gathering and generating insights. Testing is about validation - figuring out whether those insights actually hold water. The two are linked together: you generate insights to test those insights, and those tests are one of the ways that you generate new insights. In exploration mode, you're often pushing into an unknown terrain and asking questions that start like, "I wonder what will happen if..." or "I wonder how..." Notably, you don't necessarily have a hypothesis yet. Exploration mode is what people are referring to when they talk about "experimenting with drugs." In testing mode, you're trying to constrain the space you're exploring in order to make a clear declaration of fact, "If x, then y." Testing mode is what people are referring to when they talk about science experiments.

In early stage startups in particular, the two modes of behavior roughly correspond to the two highest value forms of experimentation you need to engage in: you have to know that you have something that people really want (desirability) and that you can actually deliver (feasibility)1 Understanding desirability relies heavily on doing good exploration, where insights discovered are accelerated through testing. Meanwhile, being able to deliver real value comes from relentless testing that is compounded by the exploration that lets a team see new and alternate ways to deliver value. The ability to do both allows an early stage team to move efficiently from insight to concept to proof of concept to MVP.

For early stage products, understanding desirability won't emerge from a survey or some sort of quantitative analysis of a market. At best, that tells you that a market exists, but it doesn't tell you that you can access that market. Understanding desirability at an early stage is actually an equation:

(User + Problem + Scenario) + Insight = Solution.

Desirability means knowing what is currently true and having a strong idea of what could provide a meaningful change. Getting at desirability for that first prototype is more an art than a science, less about a traditional experiment and more about building up strong insights around a specific user group, a problem they have, and a scenario in which they face it. In the early stages, you test desirability by quickly testing insights. When you have enough insight that you think you can build something, then find the quickest, cheapest way to build it, ship it, and test it with the market. At this stage, it's not about having a product yet - it's about having a solid understanding of what people need. A good desirability experiment is anything that helps you build that understanding, whether an in depth needfinding interview, reactions to a low fidelity prototype, A/B tests of product language on a landing page, or structured co-creation with users. What ties all of these together is that they move beyond just ideas into something that engages with users.

Establishing feasibility is more of a science (though it isn't without some artistry as well). Feasibility is often not just one single thing but a multitude of factors that need to align...but, there are often 1 or 2 factors that are the most critical upon which everything else hangs. These are the insights that emerged out of the desirability exploration process - the things you absolutely need to be able to deliver on to provide real value in the market. These drive everything - they are what will differentiate your product from everything else that already exists2.** These tend to be the areas where you are playing to your unique advantage - either something you understand about the market that no one else does, and/or something that you can do that no one else can do. I usually refer to these as critical success metrics, the things that must be true and the way that they must be true in order for your product to succeed.

There's a scene in the movie There's Something About Mary that illustrates a product person who knows his critical success metrics: he's the 7 minute abs guy. He's got a program that tones your abs with just 7 minutes a day of activity. His critical success metric is his program's ability to tone your abs, and his threshold of success is that it has to work in 7 minutes because his competitor is the 8 minute abs guy3. This is glib, but this is where you need to get to: not just what you're measuring but what level of performance you think it needs to reach in order to succeed. Sometimes this is self evident, sometimes you're going to discover it based on how users respond in your feasibility tests. If you can't nail your critical success metric, you are going to be seriously restricted in your ability to deliver desirable value to the user.

What makes early stage product so devilishly difficult is that your team has to actually be solid at both exploring and testing, and it's very rare to find a singular individual who is sufficiently strong in both areas. That's why it's a good thing that product is a team sport. Know where you gravitate to: are you naturally inclined toward one or the other mode? If so, balance yourself out with someone who inclines the other way. Are you a middle of the road generalist? If so, find people who have strength in the mode out of which you need to work. But also, regardless of inclination these are learnable behaviors... Let me make one concrete recommendation for each mode

• If you're trying to improve as an explorer, start to regularly generate and capture observations and collect them in one place. Use 3 different colors of notes, and use a different color for observations about Users, their Problems, and the Problem Context. Make it a habit to start organizing them in different ways - sometimes just organize them with all of the colors together, sometimes group them around common themes (and new themes since the last time) - and then generate some insights that you could test (and then test them).
• If you're trying to improve as a tester, make a regular practice of writing up the big idea you've come up with from your insights and asking, "what would need to be true for this to work?" and "what are the 1-2 critical success metrics for this idea?" And then before you go and build the whole idea, build a smaller prototype that would let you just test the feasibility of your critical success metrics.

Oh, one more secret: I've framed all of this around early stage product because it's essential for early stage product teams, but...these behaviors are valuable throughout the product lifecycle.