Choosing the best hypotheses

Many problems revolve around choosing the best hypothesis - the one that best explains some observations. This is a cornerstone of science and medicine, and is the day-to-day reality of intelligence agencies. Choosing the best hypothesis is a form of decision making. But there are some differences from traditional decision making problems as I discussed in an earlier blog.

I want to compare two approaches to hypothesis-driven thinking. The first is in a paper by Jeanne Liedtka, of the University of Virginia, Darden School of Business, "Using Hypothesis-Driven Thinking in Strategy Consulting". The second is Richards Heuer's work for the CIA called Analysis of Competing Hypotheses (ACH) (Chapter 8 of his book "The Psychology of Intelligence Analysis.")

Liedtka tunes her process to consulting and states "The traditional decision-making process that we are most familiar with in business involves a linear method of thinking in which the problem is defined, a comprehensive range of alternative solutions is generated and evaluated, and the optimal one is selected. In contrast, the hypothesis driven approach, associated with the scientific method selects the most promising hypothetical solution early in the process and seeks to confirm or refute it" She goes on to out line the following steps:

1. Define the problem


2. Gather preliminary data that allow construction of initial hypotheses about the causes of the problem


3. Develop a set of competing hypotheses about the causes


4. Select the most promising


5. Identify analyses to study these


6. Collect data and test the hypotheses


7. Reformulate as needed

Admittedly, I have paraphrased and shortened the description of these steps, but I have not lost what the paper says to do. When I read this, I get stuck on items 3 and 4. In step 3, what is the difference between "developing a set" and the linear method decried in the quotation? In Set 4, how should you "select the most promising"?

Heuer on the other hand suggests that you should:

1. Identify possible hypotheses


2. Make a list of significant evidence for/against


3. Prepare a Hypothesis X Evidence matrix


4. Refine matrix. Delete evidence and arguments that have no diagnosticity


5. Draw tentative conclusions about relative likelihoods. Try to disprove hypotheses


6. Analyze sensitivity to critical evidential items


7. Report conclusions.Identify milestones for future observations

These two processes are very similar. Where Liedtka, in the quotation beats up on developing multiple hypotheses, her method encourages more than one to be tested at a time. Heuer, on the other hand begins with "identify multiple hypotheses". He never states that this list needs to be complete, just that there needs to be more than one.

My research into how people develop products has shown, that designers that only develop a single alternative for a problem usually get into trouble because they have nothing to compare their efforts to and no secondary course of action, if the first idea hits a road block. When hypothesis driven, it is somewhat the same. You can develop a single hypotheses and test it. If it fails, you can develop a second one, maybe even using some of the information developed during the first effort. However, this is fragile as you may be headed in the wrong direction.

A better approach is more like what Heuer suggests. Develop multiple hypotheses and then use measures (i.e evidence) to try to disprove them. Heuer never says that the the list of hypotheses needs to be extensive or complete, just multiple.

One additional thought- as evidence is generally uncertain then you just take great care in using it as sufficient to eliminate a hypothesis. This is why multiple evidence is generally best. The good news is this is the way we naturally operate. For example, I was approached last week by a man with a business idea close to something I was already interest in. My initial hypotheses about him (in retrospect) were, 1) he had a good idea, was credible and I should invest time with him, and 2) he was a flake and I should waste little time with him. I began with hypothesis #1 and began to collect evidence by talking with him. After a reasonable time, I decided he passed the "sniff test" and modified my hypotheses so I could dig deeper into the business proposition. However, I have not yet totally eliminated hypothesis #2.