Designing Strong Experiments

Thanks to the Author Tendayi Viki & Franziska Beeler

When testing our business ideas, choosing the right experiment is just the beginning. After we have chosen our experiment, it’s important that we spend some time designing it well. Well designed experiments can further strengthen the evidence we get, which will increase our confidence in making decisions. Two of the most important elements for designing sound experiments are selecting the right participants and designing a well-crafted artefact.

In order to reduce the risk and uncertainty of new ideas, it’s important to test them by running experiments. However, the experiments we run must be well designed so they can produce strong evidence. The starting point for designing strong experiments is being explicit about what we are trying to learn. A clear and well stated hypothesis that is testable and precise is at the heart of everything. With a clear hypothesis in hand, we can then think about how to design experiments that will provide strong evidence for what we are trying to learn.

So how do we make sure that we are designing experiments that provide strong evidence? In this post we highlight two key factors; selecting the right participants and designing a well-crafted artefact.

Right Participants

One challenge we have noticed is that innovation teams test their ideas with vaguely defined customer segments. Using such a broad sample of participants will produce weak evidence. Ask yourself, which customer segment are you really targeting with your business idea? This might seem obvious, but if your business is targeting young people, over 18 years old who are still at university, then this is who you should focus on. You have to...

Well Crafted Artefact

The second factor to consider when designing strong experiments is a well-crafted artefact. It is important to recognize that an experiment and the artefact that is used to run that experiment are not the same thing. If we are running an A/B test, the two versions of the landing page we create are the artefact. In a customer interview, the artefact is the interview guide we will use and the questions we plan to ask. The quality of...

Design for Strong Evidence

When we choose the experiments to run, we need to consider the fact that different types of experiments produce evidence of different strength. For example, evidence from customer interviews can be considered weaker than evidence from A/B tests. This is because customer interviews measure what people say, whereas A/B tests measure what people do.

However, it is possible to...

To continue reading click here...