Validation

Validation

One of our key themes is validation of assumptions. There must be a relentless quest to get our most important questions answered as cheaply and as quickly as possible. The phrase “If we build it, they will come” only works in the fantasy of the movies. Entrepreneurship is about risk. Successful entrepreneurship is about intelligently mitigating those risks. The key to mitigating those risks is to validate our assumptions. Throughout we will pursue a process of:

  • What is our assumption?
  • What are the consequences of our assumption being wrong?
  • How do we validate our assumption?
  • What is our response?

Assumptions

There are many, many assumptions that we make when launching into something new. Many of these assumptions are based on our experience and judgement. In many cases this is wrong. If we are not entering into uncharted territory, then we have nothing to offer the marketplace. We must repeatedly clarify our assumptions.

Consequences

Judgement comes in evaluating the consequences of being wrong. We may assume that all users like blue and we therefore use a blue theme in our user interface. In the early stages of a startup, this simply does not matter. If we are wrong, we change a few lines of code and the problem is solved. The consequences are minimal because the repair takes little effort. On the other hand, an assumption that wireless networks will be pervasive with high bandwidth and low latency can be devastating. If we build a technology that depends upon this assumption and it turns out to not be true, we end up with a very expensive product that is not commercially viable.

If we do not make the judgement call about consequences then we will spend enormous amounts of time validating before accomplishing anything. If we do not validate the important assumptions we increase our risk of failure. However, if we are inventing the technology we should have a pretty good idea of the consequences of choice.

Experiment

How do we verify that our assumption is true? We don’t. All we can do is reduce the probability that it is not true. However, for most assumptions, there are many ways to quickly get answers that reduce the risk tremendously. Suppose our assumption was pervasive, low latency wireless. We can quickly find 10 prospective customers, take our laptop for visit and run a few ping tests. We have replaced assumption with experience. We still have an outstanding assumption that our 10 samples accurately reflect our prospective customer base, but the risk of this assumption being wrong is much lower. If wireless is still our most important assumption, then we can test more places and even apply some statistics to solidify our probability of being wrong.

The talent here comes in finding ways to cheaply and rapidly perform experiments that reduce risk. Remember that the goal of validation is to avoid expending resources unnecessarily. Spending enormous resources on validation can also be a problem. “What is the fastest way we can validate or refute this assumption” should be our mantra.

Response

Lastly there is the question of how we respond to our experiment. If 10 out of 10 customers pass the ping test well within the limits we think we need, then we smile broadly and move on to our next problem. If 9 out of 10 customers fail the ping test then we are glad we have not invested a lot in that assumption. We now must redesign our solution, seek a different business opportunity or accept the burden of upgrading our customer’s wireless as part of product deployment.

The challenge comes when 3 out of 10 fail the test. Should we go on, knowing that 7 out of 10 are still potential customers? With a mixed result do we need to test more to be sure that 70% success is really valid?

In many cases we learned other things about our customers as part of the test. If we paid attention to all that was going on we may have learned that those 3 were all preparing for a major wireless upgrade in a year. We may have identified a common configuration problem that we can solve.

Summary

The important thing is that we regularly replace assumptions with experience. We are less concerned with proving truth in the sense of a controlled experiment than we are with replacing uncertainty with experience.