Making Robust Decisions

Risk Management as a decision issue

noreply@blogger.com (David G. Ullman) — Wed, 08 Oct 2008 16:40:00 +0000

The term “risk management” has been around for a long time in financial, technical and medical practice. It is a term that is very loosely used and I want to dive in with a decision-centric view just to further muddy the waters.

A good place to begin is with a formal definition of “risk”. If you enter “risk definition” into Google you will get over twenty-five definitions; some are redundant, and there is little consistency. Regardless of the definition, risk traditionally amounts to answering three questions:

What can go wrong? Some event.
How likely is it to happen? Probability that the event happens based past statistics, an analytical model of the event, or best guess based on experience.
What are the consequences? Money, time, and possibly even lives are wasted.

The above set of questions describes event risk. They are the basis of technical risk evaluation. However, most managers are concerned with decision risk rather than event risk. For decision risk, the questions managers really want answered are:

What can go wrong if I choose alternative X?
How likely is it?
What is the impact?

(Brian Seitz of Microsoft articulated these as cleanly as shown here)

Note that these are the same three questions as with event risk, just slightly tweaked. Regardless of whether focused on an event or a decision, risk management, by definition needs to be effort directed at some or all of these questions.

The first question raises the issue of how we know our choice was poor. In the book Why Decisions Fail, the definition of a poor decision is one that had no positive impact after two years. There are other measures of a failed decision. For example, our time is up and no satisfactory alternative has been developed. Or not everyone on the team agrees with the choice made and some team members feel disenfranchised. More often, the result of a poor choice is not known until much later. For example, if we choose a bad restaurant, we will not know until we have eaten there. Or if an engineer makes a poor decision on what material to use for a product, this may not be evident until the customers have used the product for a number of years.

One thing is consistent in this discussion: Without uncertainty there is no risk. A corollary is that the more uncertainty, the higher the risk of making a poor decision. “What can go wrong?” is that one or more of the criteria are not satisfied. “How likely is it?” is directly dependent on our certainty during the alternative evaluation. We may know from past experience or data that the probability of something failing is XX%. But, this probability may be compounded by other uncertainties such as lack of knowledge, disagreement amongst team members or incomplete data. And “What is the impact?” is that the alternative chosen no longer is as good as it was originally thought.

In order to manage risk during decision making:

You must address risk during decision making, not as a task to complete after you have selected an alternative. This is because decision risk is a measure of your lack of knowledge, as well as other uncertainties you need to consider when you make a decision.
There is risk associated with every feature of every alternative. Traditional risk assessment separately addresses financial risk, performance risk, and schedule risk. But when including risk as a part of the decision-making process, you must integrate the uncertainty inherent in all the features at once, because it is the combination of them that drives your decision.
You can get a good assessment of uncertainty, and thus risk, by fusing the evaluations of all the members of your team.

More on Criteria Development

noreply@blogger.com (David G. Ullman) — Fri, 19 Sep 2008 15:26:00 +0000

In my blog of Sept 10, I noted that my newsletter statements in "Robust Criteria for Robust Decisions" had started an interesting conversation with Ralph Keeney. In the newsletter I had stated:

Research has shown:
1. The more effort put into understanding the criteria early in the process, the better the decision
2. Too little effort is generally put into understanding criteria.

He asked for references, and I provided, in my blog what weak evidence I had. He in turn sent me an interesting paper titled “Generating Objectives: Can Decision Makers Articulate What They Want?” (Management Science Vol 54 No 1, Jan 2008, pp 56-70). In this paper, Keeney and his colleagues present the results of a series of experiments designed to address how well people can list the objectives (i.e. criteria) they used in making decisions on real problems. In summary, Keeney and company concluded that people commonly undertake important decisions without considering many of the most important criteria. It seems that they generate only the objectives that are cued by their incomplete representation of the problem. In other words, as people work to understand a problem, by reading a problem statement, talking with others, hearing a news cast, etc, they build a mental model of the situation and base their criteria on this model.

The implications of this are:

Decision making is guided by whatever incomplete set of objectives is made salient
Criteria generation can be improved by helping decision makers develop a broader understanding of the problem through:

Time - One of the studies in the paper showed that addressing the problem at multiple points in time increases the number of criteria identified.
Multiple perspectives - Although not in the study, multiple perspectives (i.e. a team approach) can help develop the broader understanding
Templates - External aids can help in generating the broader understanding.

What follows are some thoughts on these three criteria development crutches

Time
My research in the 1980s focused on how engineers design products. In these studies my students and I video taped engineers solving simple but realistic design problems. We observed how engineers repeatedly returned to the problem statement as their mental model of the situation evolved. Before these experiments I tried to force students in my design classes to develop criteria first and then alternatives. The rationale was that, if you have a solution (or a set of solutions) in mind, then the criteria will evolve to match them. More recently, I have come to believe that criteria and alternatives co-evolve as the understanding is developed. That is not to say that you should just dive in. I now teach Quality Function Deployment (QFD) first but treat it as a living document. Also I have the students work in teams on all projects as that brings multiple perspectives.

Perspectives
One criticism of virtually all the research on decision making (including my own) is that it has been on individual decision makers. I comment on this in an earlier blog. The reality is that at work and to a great degree at home, we all solve problems with others. Either we bounce ideas off of each other or are on teams. In these situations the multiple perspectives help flesh out understanding and criteria. Large teams can actually reverse the situation. In many of my consulting jobs I see teams from multiple groups in an organization working to winnow down the criteria that are important for each individual group into a single, shared understanding. This is almost the antithesis of Keeney’s study.

Templates
The idea of using templates or other criteria crutches is one that we have tried to incorporate into Accord, our decision support software. Currently there are templates for about six different generic problems (e.g. Concept selection, Portfolio evaluation, Proposal selection, Vendor selection, Job candidate selection). However, many decisions in business are unique and developing a template for these problems is not possible. The paper that started this string “Robust Criteria for Robust Decisions” was an effort to address those problems that don’t allow for templates.

The upshot of the dialog with Ralph Keeney is that I will be doing some experiments this fall that address the two points I made initially. We will see what evolves to help frame decision problems.

Robust Criteria for Robust Decisions

noreply@blogger.com (David G. Ullman) — Wed, 10 Sep 2008 20:18:00 +0000

I just wrote a newsletter that appear on my web site on how to develop criteria and titled "Robust Criteria for Robust Decisions" In it I state:

Research has shown:

The more effort put into understanding the criteria early in the process, the better the decision

Too little effort is generally put into understanding criteria.

After sending this to my mailing list, I received a message from Ralph Keeny (A major thinker in decision making for the last 20+ years) asking " I am very interested in both of these issues, and I believe that each are true. However, research addressing these issues is not so easy to come by. Hence, if it is not too inconvenient, I would be pleased to receive references of the research referred to in the statements. Thank you very much."

This is what I wrote back.

Thanks for the note. I agree that data supporting these two contentions are hard to come by. By far the best I have seen is from a German PhD dissertation that used protocol studies of mechanical engineers similar to those I did in the late 1980s: Thinking Methods and Procedures in Mechanical Design, Dissertation, Dylla, N., Technical University of Munich, 1991, in German. From Dylla’s data, I wrote the following and developed the plot (note this is from page 142 in Making Robust Decisions, some copies of which have the wrong plot for the figure)

The experimenters measured the amount of time each of the six engineers spent developing criteria. This included reading the given criteria, rereading them, and refining them. Then a team of professional engineers evaluated the technical quality of each design. Part of the evaluation concerned how well the final designs met the criteria, and part was more objective— evaluating the elegance of the solution. The evaluation team scored each of the six designs on a scale of 0 to 100. As figure 6.1 shows, there is a significant relationship between the percentage of time spent analyzing the goals of the problem and the technical quality of the result. The engineers who spent around 7% of their time understanding and developing criteria had a 60% better solution than those who spent 2–3% of their time developing criteria. I don’t mean to imply that 7% is an adequate time for working on the criteria; this particular experiment involved a simple, crafted problem and just one decision maker. The engineers didn’t spend all their criteria time at the beginning of the task. In fact, the successful engineers worked hard to refine the criteria at the beginning and then revisited and refined them many times during the course of the experiment. This result should come as no surprise: a prime measure of the success of a decision is how well the results meet the criteria. In general, the time you spend up front to clarify the problem (understand the criteria) saves time and many headaches later.

Admittedly, I have taken some license that a better mechanical design is analogous to a better decision. I don’t think the leap is very great however as deign is repetitive decision making.

The second point is based partailly on Dylla’s finding (4 of the 6 engineers might have done better had they put in more time on criteria) and partially on the studies done for the book Why Decisions Fail, by Paul Nutt, Berrett-Koelher, 2004. One of his three decision blunders is “Decision makers base many decisions on premature commitments.” Premature commitment implies that to little time is spent on one or all of the following 1) developing alternative courses of action, 2) developing criteria, 3) evaluating alternatives relative to criteria or 4) managing the decision making strategy. He never breaks this down, but on page 167 he compares the success of four different evaluation tactics: analytical, bargaining, subjective and judgment. Paraphrasing Nutt: In an analytical evaluation, data is gathered and inferences made from analytical tools. In judgment there are no specifics. Thus, analytical methods require more effort on the measures, i.e. the criteria than does judgment. He found a decision adoption rate of 64-75% when analytical methods were used versus 36% -47% for judgment. Unfortunately, Nutt never really addresses the evaluation details and wraps criteria development in with evaluation as many authors do.

All pretty weak stuff. To add useless anecdotal “data”, I see companies do a very poor job of defining criteria for making decisions. One let an RVP with 60+ specs. After reading the 15+ proposals these specs enabled them to separate them into two piles, acceptable and not acceptable. The specs were not really what they needed to make the decision amongst the “acceptable” proposals. They then needed to spend additional time determining what their criteria were for finding the best amongst the acceptable.

Do you have any references that might add to this?

Why pairwise comparisons are a waste of time for finding criteria importance

noreply@blogger.com (David G. Ullman) — Thu, 04 Sep 2008 15:20:00 +0000

Many people use pairwise comparisons during their decision making efforts. First, this method may be used to determine the relative importance (the weightings) of the criteria. Second, pairwise comparisons are sometimes used to evaluate the alternatives relative to the criteria. BOTH OF THESE ARE A WASTE OF TIME!! Don’t get me wrong, I think pairwise comparisons can be helpful, just that there are faster ways of getting virtually the same results. In this brief note I will only tackle why you shouldn't bother for finding importance.

Through his books and companies, Tom Saaty has popularized pairwise comparisons as a part of his Analytic Hierarchy[1] (AHP) and Analytic Network[2] Processes In the Analytic Network Process book, on pages 26- 31, Saaty gives the example of using eight criteria to help select a house (e.g. Size of House, Transportation, Neighborhood, etc). His method requires that all the criteria be compared to each other one pair at a time to find the most important for each comparison. Further, a dominance factor is given to the better of the pair relating how much more important one factor is to another. If working with a team, they need to come to agreement on the dominance factors (I will come back to this point later). For the 8 criteria, there are 28 comparisons and the need to judge 28 dominance factors. In general, for N factors there are (N-1) + (N-2) +…. 1 comparisons.

For the example in his book (where the numbers represent the 8 criteria (i.e. factors)) a matrix of the pairwise comparisons is a shown below. Note that the opposite entries are just reciprocals of each other. Criterion 1 is 5 times as important as criterion 2 and so criterion 2 is 1/5 as important as criterion 1.

The Priority Vector is reduction of the values (using an eigenvector analysis) to develop the relative weightings - the importance of each criterion.

Compare this to a method proposed by Ward Edwards, one of the fathers of modern decision theory. He suggested that asking decision makers to weigh criteria is so fraught with error that it is easier, and no less accurate, just to ask them to rank the criteria and then automatically set the weights according to the ranking[3]. This “error” is exasperated when there are multiple constituencies represented in the organization.

Find the rank order the criteria, write each criterion on a sticky note and arrange them on a wall or desk, and reorder with the most important on top. This is best done in a pairwise fashion by selecting the criteria two at a time and asking, “If an alternative could meet only one of these, which criteria would I choose?” Then, move the chosen one to the top and the other to the bottom of the arrangement.

You can convert the ranking to a weighting by using the table below. This table shows the Rank Order Centroid (ROC) method developed by Edwards and shows the weights for up to 12 criteria.

Weights based on ROC method

If you have more than 12 criteria, you can use the equation wk= (1/K) ∑ (1/ i ) as i goes from k (the number of the criterion, with 1 being the highest weighted and K being the lowest) to K (the number of criteria). This equation was used to generate the values in the table. The values for 8 criteria are shaded in the table above and plotted below compared to the pairwise method.

The results of the two methods are shown on the bar chart below.

The Mean Absolute Error between the two (the sum of the absolute differences between the two) is, on average, 2%. Other examples I have tried have even had less error. Considering that there is no right answer and that one change in pairwise comparisons can change the results. The difference between the two comes at the expense of a major difference in the amount of effort. Twenty eight comparisons and assignments of priorities versus simply rank ordering the criteria.

Now, adherents of pairwise comparisons can argue that the method also computes consistency, a measure of how well the many dominance factors agree with each other. I believe this is of little importance as the dominance factors are just averages across a committee of stakeholders who are trying to quantify subjective values. In other words, the uncertainty in the pairwise scoring is so high due to averaging and quantifying subjective values that consistency in the dominance factors is just noise. Consistency analysis gives false comfort that the matrix is consistent when the numbers themselves are very uncertain.

Based on the above arguments, I believe pairwise comparisons are a waste of time. I prefer to allow all the stake holders to rank order and use the resulting inconsistent weighting factors in my analysis. Thus, I honor all party’s values and use them all in downstream analysis. More on this in another note.

[1] Decision Making for leaders: The Analytic Hierarchy Process for Decisions in a complex World, Thomas L. Saaty,RSW Publications, Pittsburgh PA, 1999
[2] The Analytic Network Process, Thomas L. Saaty, RWS Publications, 1996.
[3] Edwards, Ward and F. Hutton Barron, "SMARTS and SMARTER: Improved Simple Methods for Multi-attribute Utility Measurement" and F. Hutton Barron and Bruce Barrett, "Decision Quality Using Ranked and Partially Ranked Attribute Weights."

Are two heads better than one?

noreply@blogger.com (David G. Ullman) — Mon, 01 Sep 2008 12:57:00 +0000

I was listening to NPR the other day and heard the word “constult”. This word is in Oxford English Dictionary (I checked). It is a verb that means “To play the fool together.” Both examples in the OED are from the 17th century:

1630 J. TAYLOR (Water P.) World's eighth Wonder Wks. II. 67/1 Some English Gentlemen with him consulted And he as nat'rally with them constulted. 1659 GAUDEN Slight Healers (1660) 91 What do they meet, and sit, and consult (or rather constult) together?

So, this brings up the question (posed on the Car Talk web site); “Do two people who don’t know what they are talking about know more or less than one?” This is worth asking since most hard decisions are plagued by a lack of knowledge. In fact, this was this question (not so well articulated) that led me to start thinking about decision making in the mid 1990s.

When I am designing a new product, I don’t know much about the new details of it. Designing is learning. Often I will enlist others to help on the functions and features I know the least about. Are these colleagues consulting or constulting, and if the latter, why do I seek their help?

I think I know the answer. If I know absolutely nothing about something and my response is a wild guess, then the probability of my being right is 50%. If another person is added to my team and they too know nothing then she is truly constulting, together we know nothing and the probability is still 50%. However, if I know a little and what I know is correct, then my probability is greater than 50%. And, if I add a colleague and she knows a little then the probability of us together being correct is greater still.

As a numerical example (taken from “Making Robust Decisions” page 231), say that I know enough that the probability that I am right is 68%. If another person is added to the team and they independently also have a probability of being correct of 68%, then our fused probability is 82%. If there are three of us, it raises to 91%. The theory behind these numbers is in the book, but they do make intuitive sense.

This also supports the examples in the book “Wisdom of Crowds” (a good read). Obviously there are some limitations. What if members of the team disagree on the answer? What about Group Think? What is the level of knowledge is less than estimated? All of these can lead to constulting.

So to answer the question:

Two people who know nothing know as much as one person who knows nothing.
Two people who know a little independently (no group think or peer influence) and their meager knowledge is correct, know more than one person. This is how we get to the wisdom of crowds. If their knowledge is conflicting, they still know more, just that answer is still in doubt.
Two people who know a little and they have influenced each other through peer pressure (think alpha males), group think or other mechanism, or one or more of them knows less than they think they do; together they may actually know less and be constulting.

The goals then are to know when you know nothing and how to avoid the pitfalls that lead to constulting. I have been struggling with these goals for about ten years.

Psychology Research Misses the Reasons for Decision Making Teams

noreply@blogger.com (David G. Ullman) — Sun, 17 Aug 2008 15:50:00 +0000

I am reading “Straight Choices: The Psychology of Decision Making” by Benjamin Newell, David Lagnado and David Shanks (Psychology Press 2006). It is a very good summary of decision making from the viewpoint of cognitive psychology. It makes some muddy topics very clear. However, it totally fails in Chapter 14, Group Decision Making. This isn’t the authors’ fault – all work on team or group decision-making misses the main points. I will get to these in a moment.

First, some background. I attempted to address this topic in Chapter 5 of “Making Robust Decisions”, cleverly titled “Teams Don’t Make Decisions, But…” The title reflects the problem with the topic. In business and technology there is always one person who signs off on a project to move it forward. This manager is the ultimate “decision maker”. But, if this person is good at what they do, for complex problems they have a team that is studying the problem and advising them about what choice to make. On this team, some of the people know about some of the information, they all have different fields of expertise and knowledge. For complex problems, no one person can know all about all of the important features of all the alternatives. Further, they bring a range of viewpoints about what is important.

So, is the manager the decision maker or the team? Depends how you draw the line around the term “decision”. If it is an event, then the manager is the decision maker. If decision-making is a process, then it is the team. I see decision-making as a process. The cognitive psychologists seem to see a decision as the event.

In “Straight Choices”, the authors summarize research on group decision making. All of the studies they sight are for very simple problems, not for problems with distributed knowledge. In other words, the toy problems the psychologists have used to study teams do not reflect what happens in business and technology. They tend to focus on the "right" answer to simple problems with a known solution. This why Chapter 14 was such a let down.

From my reading, the two main reasons to use a team are:

Obtain the best information when no one person can know all that is needed to be known
Build stakeholder buy-in and accountability

These goals mean that you need to:

Build an environment and a team strategy that fosters communication of the right information and buy-in
Suppress the effect of differences in cognitive styles (“suppress” is not the right word, but you want to level the playing field so the alphas don’t dominate, the closers don’t stop things too soon, the wafflers don’t lead to team paralysis and so forth.)
Guard against group think
Help build a shared understanding

Of the six topics itemized above, Chapter 14 only addresses Group Think. It comes close to talking about a shared understanding when it discusses consensus, but consensus is not important for a decision buy-in. (Quoting Margaret Thatcher, "To me, consensus seems to be the process of abandoning all beliefs, principles, values and policies. So it is something in which no one believes and to which no one objects."). The two main goals are never discussed. It seems the cognitive research has focused on getting the “right” answers to toy problems. Most real decisions have no right answer, so the psychologists are asking the wrong questions.

In the authors’ defense, they end the chapter with “Research on group decision making is both appealing and frustrating”. In spite of this frustration, decisions are made by teams every day and I believe the key to robust team decisions is in the six items listed above. These are discussed at length in Making Robust Decisions. I just wish there was more good research on each of them so you don’t have to take my word for it.

Decision making maturity

noreply@blogger.com (David G. Ullman) — Fri, 08 Aug 2008 12:59:00 +0000

I just returned from an ASME (American Society of Mechanical Engineers) meeting in NYC. During my time there I discussed the concept of decision making maturity with three different groups and thought it worth writing about. Best to do in context with my career.

When I was trained as an engineer, I focused on how components and assemblies were shaped, manufactured and functioned. This element-centric view of the world is not at all unique to engineers. Businesses focus on documents, e.g. POs, recepts, memos, contracts, etc)

By the 1980s I became fascinated with the process of developing the elements. This process-centric view is recent. Sure engineers have studied and developed chemical and manufacturing processes for years, but the concept of product design processes and business processes is recent. As evidence of this, in 1990, I wrote an engineering text book about how to progress from customer need to produced product. I debated long and hard about what to title it. I finally landed on "The Mechanical Design Process", a title that proved to be right on the mark (note that its 4th edition will be out in January 2009). The use of the word "process" in the title was problematic because it was only beginning to be used to discuss product evolution.

In the early 1990s my research was about how to capture and manage the evolution of products, the rationale for form and function. This was process-centric, but not getting anywhere. In about 1995, it dawned on me that "design is the evolution of information, punctuated by decisions". That started me on a decision-centric view of the business and technical worlds.

My maturity from element through process to decision is being taken by industry. While in NYC I met with a PLM (Product Lifecycle Management) guru. PLM grew out of CAD (Computer Aided Design) which was all about parts and assemblies - element-centric. PLmis currently focused on the process, i.e. the lifecycle of products. I have been trying to sell decision-centric capabilities into PLM since 2001 with no success. The push back has always been "we aren't ready for that yet" and they weren't. Now there is the beginning of interest. The product development industry is maturing through process to realize that business and technology progress is punctuated by decisions and it is the quality of those decisions that determine the product and business success.

Further, when I talked about this decision-centric view of the world five years ago to industires, audiences had no idea what I was talking about. Now I get good awareness and it is building. There is yet hope for "evolution punctuated by decisions".

Choosing the best hypotheses

noreply@blogger.com (David G. Ullman) — Mon, 21 Jul 2008 17:10:00 +0000

Many problems revolve around choosing the best hypothesis - the one that best explains some observations. This is a cornerstone of science and medicine, and is the day-to-day reality of intelligence agencies. Choosing the best hypothesis is a form of decision making. But there are some differences from traditional decision making problems as I discussed in an earlier blog.

I want to compare two approaches to hypothesis-driven thinking. The first is in a paper by Jeanne Liedtka, of the University of Virginia, Darden School of Business, "Using Hypothesis-Driven Thinking in Strategy Consulting". The second is Richards Heuer's work for the CIA called Analysis of Competing Hypotheses (ACH) (Chapter 8 of his book "The Psychology of Intelligence Analysis.")

Liedtka tunes her process to consulting and states "The traditional decision-making process that we are most familiar with in business involves a linear method of thinking in which the problem is defined, a comprehensive range of alternative solutions is generated and evaluated, and the optimal one is selected. In contrast, the hypothesis driven approach, associated with the scientific method selects the most promising hypothetical solution early in the process and seeks to confirm or refute it" She goes on to out line the following steps:

Define the problem

Gather preliminary data that allow construction of initial hypotheses about the causes of the problem

Develop a set of competing hypotheses about the causes

Select the most promising

Identify analyses to study these

Collect data and test the hypotheses

Reformulate as needed

Admittedly, I have paraphrased and shortened the description of these steps, but I have not lost what the paper says to do. When I read this, I get stuck on items 3 and 4. In step 3, what is the difference between "developing a set" and the linear method decried in the quotation? In Set 4, how should you "select the most promising"?

Heuer on the other hand suggests that you should:

Identify possible hypotheses

Make a list of significant evidence for/against

Prepare a Hypothesis X Evidence matrix

Refine matrix. Delete evidence and arguments that have no diagnosticity

Draw tentative conclusions about relative likelihoods. Try to disprove hypotheses

Analyze sensitivity to critical evidential items

Report conclusions.Identify milestones for future observations

These two processes are very similar. Where Liedtka, in the quotation beats up on developing multiple hypotheses, her method encourages more than one to be tested at a time. Heuer, on the other hand begins with "identify multiple hypotheses". He never states that this list needs to be complete, just that there needs to be more than one.

My research into how people develop products has shown, that designers that only develop a single alternative for a problem usually get into trouble because they have nothing to compare their efforts to and no secondary course of action, if the first idea hits a road block. When hypothesis driven, it is somewhat the same. You can develop a single hypotheses and test it. If it fails, you can develop a second one, maybe even using some of the information developed during the first effort. However, this is fragile as you may be headed in the wrong direction.

A better approach is more like what Heuer suggests. Develop multiple hypotheses and then use measures (i.e evidence) to try to disprove them. Heuer never says that the the list of hypotheses needs to be extensive or complete, just multiple.

One additional thought- as evidence is generally uncertain then you just take great care in using it as sufficient to eliminate a hypothesis. This is why multiple evidence is generally best. The good news is this is the way we naturally operate. For example, I was approached last week by a man with a business idea close to something I was already interest in. My initial hypotheses about him (in retrospect) were, 1) he had a good idea, was credible and I should invest time with him, and 2) he was a flake and I should waste little time with him. I began with hypothesis #1 and began to collect evidence by talking with him. After a reasonable time, I decided he passed the "sniff test" and modified my hypotheses so I could dig deeper into the business proposition. However, I have not yet totally eliminated hypothesis #2.

Event Risk vs Decision Risk

noreply@blogger.com (David G. Ullman) — Sat, 12 Jul 2008 23:31:00 +0000

I was just gave a presentation at the IG's (Inspector General's) office. They are concerned about the risks involved when they make decisions. What they don't realize is that there are two kinds of risk they have to worry about, event risk and decision risk.

Event risk is what most people mean they talk about "risk". It is the expected value of an event, its undesirable consequences and probability of its occurrence. Determining risk amounts to answering:

What can go wrong? –An event occurs that may have bad consequences
How likely is it? – Probability dependent on past statistics and model results
What are the consequences? –Money, time and possibly lives are wasted

NASA and others have entire handbooks on assessing event risk.

During decision-making, risks are inherent in uncertain knowledge, information and models. Uncertainty creates the risk that a poor decision will be made. This doesn't say that the alternative chosen will fail, that is even risk. Drawing analogy to event risk, decision risk focuses on:

What can go wrong? – A poor choice is made
How likely is it? – Probability dependent on uncertain knowledge, and the fusion of the team’s interpretation of information and models
What are the consequences? –Money, time and possibly lives are wasted

One problem the IG wants to address is selecting new employees. Clearly the risk here is a decision risk - they want to ensure that they don't make a poor hiring choice. They also want to manage their portfolio of projects. Here the risk that the project can go wrong affects the risk that they make a poor decision. The higher the event risk associated with an option, the higher the decision risk may be.

Both types of risk are based on probabilities. However, traditional probability methods (often called frequentist methods) are good fro event risk, but are not capable of managing knowledge uncertainty. Rather, Bayesian probability methods are specifically designed to integrate accumulating, uncertain, incomplete and conflicting knowledge.

Can I convince the IG folks of this? We will see.

Evidence Based Decisions

noreply@blogger.com (David G. Ullman) — Mon, 07 Jul 2008 17:51:00 +0000

I have begun to work on intelligence agency decision-making and, in doing so, realized there were two different types. I have never seen this decision-making dichotomy written up anywhere and cant find it in the literature (any literature).

The first decision-making formulation is the standard process where the goal is to choose the best alternative from a list. This is facilitated by comparing each alternative to a set criteria. Each is measured relative to each criterion and its success in meeting it is combined (either formally or informally) to find the overall satisfaction with the alternative. For example, say you are buying a new car. One criteria is that the car should accelerate for 0 to 60 in some fast time and a second is that it should get better than xx miles/gallon. Information with which to evaluate each of these may come from different sources. I may actually measure the acceleration of one car but rely on the test figure in a magazine for another. In this way the evaluations for acceleration are independent from one alternative to the next. The mathematics for this is referred to as Multi-Attribute Utility Theory (MAUT). Methods based on MAUT that support decision-making are the decision matrix, Pugh's method, Expert Choice and Accord.

The second formulation is what is seen in medical and system diagnosis, and business and government intelligence. This has not been formalized as best I can tell. Here the goal is to choose the most likely hypothesis. A hypothesis is like an alternative. The big difference here is that , as each piece of evidence is gathered and analyzed, it either supports, denys or has no import on each of the hypotheses. This differs from the first formulation in that a single evaluation potentially adds information to all the hypotheses (instead of independent). For example, supposed we want to determine Iran's nuclear intentions. Hypotheses include: 1) generate power only, 2) develop a nuclear weapon for ground delivery, or 3) develop a nuclear weapon for air delivery. A piece of evidence, say a satellite photo of activity at an airbase, contributes information to all three of the hypotheses.

The standard MAUT formulation does not support this second type of decision-making and neither do the tools listed above. An aside, the medical profession uses the the term "evidence based" to mean making clinical data available to practitioners in a usable manner so that the doctor trying to figure out why your finger is rotting off has all the best clinical data on which to base her diagnosis. This does not really support choosing which of the hypothetical diseases you actually have, but supplies information on which to evaluate the evidence.

Situations like what I have just described can be modeled with methods like Bayes Nets, but there is no known method for facilitating the process as with the decision matirx, Pugh's and Accord. I have spent months working on this and have found little in the literature. Do you have any leads?

What is decision making, anyway!

noreply@blogger.com (David G. Ullman) — Thu, 03 Jul 2008 12:33:00 +0000

I track the term "decision making". Like most terms the more you think about it, the more you realize that it is used rather loosely. The word "decision" itself is really a noun but is often used in its verb form. As a noun, it is the result of a process (to be discussed in a minute) that is a call to action. I have decided to marry Adele. I have decided to walk through the doorway. I have decided that, indeed grass is green.

When people use the term "decision" they often mean the process of making a decision. I am making a decision about which restaurant to eat at. The team is working on that decision. We often qualify the activity of choosing a course of action by using the term "decision-making".

Decision-making implies that a choice is to be made - i.e. there must be two or more alternative courses of action. However, often the process is about justification for a single course of action and the activity is focused on convincing someone that that this is right course to take. (think politics and decision making processes used in attacking the Bay of Pigs and Iraq).

I came across another curious use of "decision-making". I found a web site titled "Test Your Decision Making" that has you do a short and very simple exercise to guess the rule that governs the sequence 2,4,6. This is on the Yale University web site. The answer is simple (it is a sequence) and the explanation centers on "To test a hypothesis, try to disprove it." I frankly found the discussion confusing, but it at least it was about testing hypotheses (potential courses of action) in order to choose the best. However, the point about testing each one until I found a counter example seems daunting for anything other than simple, toy problems like the one presented.

Most estimates suck

noreply@blogger.com (David G. Ullman) — Tue, 01 Jul 2008 13:55:00 +0000

The rebuilding of the World Trade Center (WTC) is behind schedule and over budget. According to the executive director of the Port Authority, Christopher Ward "The schedule and cost estimates for the rebuilding effort that have been communicated to the public are not realistic," (from AP article). I can only say "Duh!"

All decisions are based on estimates of the murky future. In most public works projects honest estimates would never get them funded. It is much easier to get something approved with optimistic time and cost estimates and then, once its started ask for more. However, even if you are not playing politics, making estimates is hard work that is not well recognized for its impact of organizations

Let's get this straight. Even if you are trying to be as accurate and honest as possible, making an estimate is a highly uncertain undertaking. And, most decisions are based on many, interdependent estimates. In a simple experiment, I asked hundreds of people to estimate the length of time needed to clean some dishes. I gave them a detailed list of the dishes, and a photo of them, the sink, and cleaning materials. I asked them to estimate how long to clean the dishes. Depending on how I worded the question, the average time estimated was anywhere from 17 to 32 minutes and standard deviation was as high 10 minutes. In other words, asking for an estimate for this simple, daily task isn't a whole lot better than using a random number generator set to give an estimate in the range of 10 -40 min.

If this simple estimate is so bad, think of what happens with complexity, or for tasks that have not been done before.

What is to be done then. The only solution I know is to use methods that account for uncertainty. For planning , the old PERT system made an effort at this (much improved method is Critical Chain). For decision making, I have been pushing people for years to include uncertainty in their decisions making process.

In decision making, it is not only the estimates that are uncertain, it is the targets that you are trying to achieve that are uncertain. More on this in a later blog.

Back to the World Trade Center, I believe that if there had been a non-political effort at making decisions, and uncertainty had been included in the estimates, the plan would have been far less grandiose. However, the WTC should be grandiose, shouldn't it? This leads me to believe that honest estimates may not be the bast for all situations. I would hate to run or work for a business with that in mind.

Decision Anchoring

noreply@blogger.com (David G. Ullman) — Tue, 24 Jun 2008 13:28:00 +0000

A nice summary of how results can be anchored just appeared on the web. This human behavior has great implications in business and technical decisions.

Anchoring sets a biased context for estimation. It is a cognitive limitation that affects the quality of our decisions. Anchoring occurs, for example, when a manager asks for an estimate with something like: "I don’t see how we could commit more than $10,000 to this." $10,000 now becomes the anchor point. This stated amount biases all the following estimates that are generated.

Anchoring can happen in subtle ways. Let’s say you are bidding on a project and you have been led to believe that the customer has a ceiling of, say, $10,000. You are now anchored to this value and will make decisions to try to force your project to fit it. This only seems logical, but it has interesting effects. First, the amount of work you propose will be descoped to fit the budget. But people always are optimistic, so, if you get the job, you will still have more work than money. Then begins the dance of working more for less money (overtime), further descoping or asking for more money. This dance is further discussed on pages 82-85 in Making Robust Decisions.

To demonstrate anchoring, I gave a group of people a simple estimation problem - asking them how long it would take them to wash a list of dishes. I described the dishes in detail, how dirty they were, and what "wash" meant. The mean estimate was 32 min with a standard deviation of 10 min. I then asked another group of subjects to estimate how long it would take to clean the dishes exactly as before, but this time I added "Your partner has told you that the kitchen needs to be clean in 15 minutes." This anchoring resulted in a new distribution with a mean of 18 minutes and a standard deviation of 6 min.

Think of the implications on decision making. All decisions are based on best estimates of past performance, assessments of the current situation, and visions of the future. Every one of these can be clouded by anchoring. You cant totally avoid it. However, you can be aware of how you word your need for information and consciously try not to anchor estimates on which decisions will be based.

Cognitive problems in making decisions

noreply@blogger.com (David G. Ullman) — Tue, 17 Jun 2008 16:00:00 +0000

I have been reading "How Doctors Think" by Jerome Groopman. It is one of two books with the same title. Basically, reading this will make you scared to go to the doctor as doctors are human and make lousy decisions, just like all the rest of us. He has a good chapter titled "The Uncertainty of the Expert" which focuses on my favorite topic, decision making with uncertain information. All of Groopman's comments come from cognitive psychology and apply to business and technical decisions as well.

The key point in the chapter is captured in one sentence about half way in: "Physicians, like everyone else, display certain psychological characteristics when they act in the face of uncertainty". Uncertainty arises from 1) incomplete mastery of the available knowledge, 2) limitations of the knowledge, 3) difficulty distinguishing between the 1 and 2, and 4) (he leaves this one out) the variability of the nature of things.

In the face of uncertainty we all tend to:

Focus on the positive rather than the negative (except engineers who are pessimists by nature).
Ignore uncertainty. This is very evident in how we all talk about technical issue in terms of single values. Luckily we have terms like "about", "near to"
Go with what's been done before even if it is based on an unknown and unproven orthodoxy. Of course risk aversion is good and in medicine often leads to the correct diagnosis, whenever the problem is by-the-books. However, in business, technology this aversion can lead to being swamped by the competition.
Have a confirmation bias. This means that we look for support for our favorite alternative or hypothesis, at the expense of work on other possible options and discounting the negative (see item 1).

In the medical profession these characteristics are supported by the lack of time, risk aversion and an old-boys-club attitude. It is interesting watching the decision making on the TV program House. Here an arrogant, troubled doctor who hates the establishment diagnoses rare conditions and displays items 1, 2, and 3. He is really good at #3. But, most of us are.

Bayesian Brain and Bayesian Teams

noreply@blogger.com (David G. Ullman) — Thu, 12 Jun 2008 12:40:00 +0000

An interesting article "Is This a Unified Theory of the Brain?" appeared in the New Scientist (full text) on May 28th. This is basically a discussion of the work and theories of neuroscientist Karl Friston of the University College in London. His work is based on an earlier theories that the brain makes decisions by trying to make sense out of the uncertainties in the outside world. Basically, you make hypotheses about reality and compare sensory inputs to them, updating the hypotheses and the belief in them as you gather more information. In essence you are constantly updating the probabilities that the hypotheses are true. In fact the wiring in the brain is continuously changing to is suppress prediction errors.

What makes this really fascinating is that 1) this is Bayesian updating and 2) this applies to team decision making also. The first observation has spawned the Bayesian Brain camp of neuroscientists. They believe we are all Bayesian thinkers, updating the probabilities that your hypotheses are correct as you solve small problems (e.g. if I turn the knob, the door will open) and large ones (e.g. If I choose to study the Bayesian brain, I will better understand team decision-making).

This brings us to applying what the neuroscientists are doing on the individual, to what happens in a team making business or technical decisions. These decisions are generally about what courses of action to take to address a current situation (obscured by its immediacy) or the future (clouded by uncertainty). A team that is functioning well will develop alternative courses of action or hypotheses and then gather an communicate information to increase its belief that one is better than the other, or to update the options based on the new information. No different than a single brain, just much more complicated.

What makes it harder is not only the communication of information amongst the team members (a clear focus of Information Management, Business Intelligence and the Webex's of the world), but developing a shared vision of the information. This is not to imply that everyone needs to understand all the information, but that there is some common understanding of the important bits. This is a topic I beat on in Chapter 4 of Making Robust Decisions "Team Don't Make Decisions, But...."

When we were first developing Accord software, an effort to support the team decision making process, we assumed that we could help teams by making what occurs inside one person's head transparent for the entire team. Maybe we should become neuro-scioscientists and study "Bayesian Team Dynamics".

Decision Making for KM and BI

noreply@blogger.com (David G. Ullman) — Wed, 11 Jun 2008 12:23:00 +0000

A June 10th article "Knowledge Management and Business Intelligence" tries to tease apart KM and BI. In the piece the author, Richard Herschel, refers to Gartner's definitions of the two terms:

BI= a set of all technologies that gather and analyze data to improve decision making
KM = a systematic process of finding, selecting, organizing, distilling and presenting information in a way that improves an employee's comprehension in a specific area of interest Specific knowledge management activities help focus the organization on acquiring, storing and utilizing knowledge for such things as problem solving, dynamic learning, strategic planning and decision making.

What has always amazed me about these and other fields such as Analysis of Alternatives (AOA), Data Mining and even most Decision Support Systems (DSS) is that they are all about developing and managing information to support and improve decision making, yet none of them actually support the decision making process.

The decision making process requires 1) framing the problem, 2) evaluating the alternatives, 3)fusing the evaluation and 4) deciding what to do next. All of the methods listed above help gather and organize information that is vital to good decisions, but all stop short. Decision making requires more than the availability of database information. In fact, as mentioned in the article "up to 80% of business information is not quantitative or structured in a way that can be captured in a relational database". This non-quantitative information is the basis on which many business and technical decisions are made.

Two cases support this. First, a manufacturer of rocket engines uses our Accord software because many of their key early decisions are qualitative and Accord can manage fusing the qualitative and quantitative evaluations. Second, a Fortune 100 company received 20 proposals in response to an RFP for a piece of electronic equipment. In the RFP were over 60 quantitative specifications for size, functionality, reliability, etc. When they reviewed the proposals, they sorted them into two piles, those that met the specs and those that didn't. Then they began to use the unstated, usually qualitative measures to differentiate the acceptable proposals so they could decide how to award the contract.

The point is, most best practices focus on generating and managing information so decisions can be made. They don't spend enough effort on framing, fusing and managing what to do next. Framing and evaluation fusion are the social interactions occur that develop buy-in, accountability and robust decisions. It is these areas that are the hard part, that are not covered in school and are not well supported.

Stuck in 'Analysis Paralysis'?

noreply@blogger.com (David G. Ullman) — Sun, 08 Jun 2008 15:55:00 +0000

Unresolved decisions can be very damaging to an organization. You know something needs to be done but can't decide what to do. Deliberation continues among your staff without an actionable conclusion. You have 'analysis paralysis'. Your operation is not moving forward, but you're using resources and your competition is not standing still.

Symptoms of 'Stuck' decision making include the following:

Holding more than 3 meetings on a single issue
Continually gathering more information
Endlessly discussing and deliberating but not coming to a conclusion

'Stuck' deliberation is often resolved at the twelfth hour and by a manager who may be forced to act arbitrarily, if not irrationally. For any organization this is a highly risky way of operating. Here are some pointers for avoiding analysis paralysis:

Build a Shared Understanding: Information about the alternatives and criteria in a decision are understood only from each team member's own perspective. Team members may not have a complete picture of the situation, nor the knowledge or time to develop a full, long-term view. The key here is to set up environments that support the sharing of pertinent information needed for the decision.

Work to separate Goals from Importance: Separate what is to be achieved (i.e. goals, targets) from how important it is to achieve it. It is easier to agree on goals than what is important.Evaluation uncertainty may swamp out the differences in importance, but only if this goal/importance separation is made explicit.

Acknowledge and Manage Uncertainty: The future is always uncertain. The extent of this uncertainty needs to be identified as objectively as possible during decision-making. Engineers and financial analysts in particular are prone to giving single, deterministic values for information that is really a distribution. Push back on them to find the distribution, even if it is in terms like "very sure", "about" or "sort-of". Early in the development of a system all estimates are uncertain and need to be managed as such.Information that is highly uncertain needs to be discounted or its uncertainty reduced if-and-only-if it is important (see item 2)

Develop Multiple Alternatives: Always consider multiple courses of action that can be itemized. If the choice is to do A or "do nothing", then make an effort to develop alternatives B and C. Develop methods within your organization that encourage creative options.Find ways to help the champions of each idea compare and contrast their alternatives with others.

Define a Decision-making Strategy: Make sure there is an agreed to decision-making strategy. Decision-making by stirring and restirring existing information is not beneficial.

Build collaboration: Collaboration means that all stakeholders' opinions are heard during deliberation. Then, even those whose first choice is not chosen will more likely buy in to the outcome. This also gives the whole team a sense of accountability for the final decision.

Be Aware of Diminishing Analytical Returns: Analysis is expensive and is likely to postpone resolution. Over-analysis is the risk-averse activity of trying to drive out all uncertainty. When the fidelity of simulation is greater than the uncertainty of the information on which the simulation is based, time and money are being wasted.

Reuse History: Work toward learning from past decisions. Evaluating your success requires keeping track of past choices; the actions as well as the results.

Find a Platform to manage and fuse uncertain team evaluations: Use proven methods and tools that help your organization reduce risk and avoid deliberative quagmires.

A Justification is Not a Decision

noreply@blogger.com (David G. Ullman) — Sun, 08 Jun 2008 15:33:00 +0000

There are two kinds of decision-making: justification and selection. Justification occurs when the result is a foregone conclusion — the choice is made in advance of any argument or consideration. This often happens when the boss already has his favorite option in mind and wants information to "prove" that it is the right choice. I have seen this during my career in product design and the nation saw it in President Bush's decisions about Iraq. In Maureen Dowd's NY times Op-Ed on June 1 2008, "Cult of Deception" she says that "our president is a one-man refutation of Malcolm Gladwell's best seller Blink, about the value of trusting your gut."

In an earlier blog, I discussed Blink and how trusting your gut can be the wrong approach for complex decisions, because you can't include sufficient information or study alternative courses of action. However, managers that get that warm fuzzy feeling when they know the best course of action, before they have sufficient information, can be dangerous. Of course, they may be right. If they usually are, then they clearly have sufficient information and a good decision making style. But if these justifications often end up with later fire-fighting, then justification is not working in lieu of decision-making.

Actually, the CIA has a method that is supposed to short circuit justification-thinking. It is called Analysis of Competing Hypotheses and is in a downloadable book titled "Psychology of Intelligence Analysis." Basically, ACH prescribes the following steps:

Identify the possible hypotheses to be considered.
List the significant evidence and assumptions for and against each hypothesis.
Draw tentative conclusions about the relative likelihood of each hypothesis.
Analyze sensitivity of the conclusion to critical items of evidence.
Identify future observations that would confirm one of the hypotheses or eliminate others.

This differs some from the process I develop in Making Robust Decisions, but both begin with basic fact that — YOU MUST HAVE ALTERNATIVES TO MAKE A DECISION. I put this in bold because it seems evident that many Washington decision makers don't follow this basic truth. In fact, many business leaders don't follow this either. In the book Why Decisions Fail, Paul Nutt gives three basic blunders. One of these, "Premature Commitments" is the the same as the justifications we are discussing here.

So what do you do to stop this kind of ineffective decision-making. The only thing to do is to set up an environment that forces multiple decisions to be considered. Sometimes this is difficult from below, but if you are a manager, insist on multiple alternatives to consider.

Gut decisions don’t work well in many situations

noreply@blogger.com (David G. Ullman) — Sat, 07 Jun 2008 15:53:00 +0000

There has been a recent spate of best-selling books that support the notion that you should go with your gut when making a decision (e.g. Blink: The Power of Thinking Without Thinking by Malcolm Gladwell, Gut Feelings by Gerd Gigerenzer and Gary Klein's The Power of Intuition: How to Use Your Gut Feelings to Make Better Decisions at Work).

The authors of these books state that for many decisions there is no need to formally identify the issue, develop multiple alternatives, and itemize criteria — just go with your intuition. Of course, they are right — for some decisions. We make decisions in two very different ways. Sometimes we reach conclusions unconsciously — our minds quickly and silently sorting through the available information and drawing an immediate judgment. This may be in a blink — so quickly and so far below our level of awareness that we may have no consciousness of where our conclusions came from. If we are trying to decide what to do in an emergency, how to walk through a doorway, or many other simple or no-time-to-think situations then this is the right approach.

One study of decision-making methods (see Payne, Bettman, and Johnson, The Adaptive Decision Maker) found that

People use a variety of cognitive strategies, dependent on task and context factors.
Given that people have limited cognitive abilities, strategy selection is a compromise between accuracy and the desire to minimize cognitive effort.
People are opportunistic; they change their strategies on the fly.

The study indicated that intuition follows a strategy in which we compare the first alternative solution for the situation to the most important criterion. If our evaluation shows that the alternative satisfies that criterion, we accept it and move on. If it doesn’t, we generate another alternative and try again. If there is more than one alternative that satisfies the most important criteria, we move to the next most important criterion to try to zero in on the one to choose. Of course, this strategy may be modified if we are out time, there are too many alternatives, or there are others involved.

Research I did in the early 1990s showed that designers who pursued their first idea generally ended up with concepts that needed much patching to make them work, if they could be fixed at all. Another study of design engineers showed that the quality of design results linearly improved with the time and effort spent on developing alternatives and criteria (this work is in an obscure German dissertation but the results can be read in a paper I wrote "The Ideal Engineering Decision Support System."