More on Criteria Development

In my blog of Sept 10, I noted that my newsletter statements in "Robust Criteria for Robust Decisions" had started an interesting conversation with Ralph Keeney. In the newsletter I had stated:

Research has shown:
1. The more effort put into understanding the criteria early in the process, the better the decision
2. Too little effort is generally put into understanding criteria.

He asked for references, and I provided, in my blog what weak evidence I had. He in turn sent me an interesting paper titled “Generating Objectives: Can Decision Makers Articulate What They Want?” (Management Science Vol 54 No 1, Jan 2008, pp 56-70). In this paper, Keeney and his colleagues present the results of a series of experiments designed to address how well people can list the objectives (i.e. criteria) they used in making decisions on real problems. In summary, Keeney and company concluded that people commonly undertake important decisions without considering many of the most important criteria. It seems that they generate only the objectives that are cued by their incomplete representation of the problem. In other words, as people work to understand a problem, by reading a problem statement, talking with others, hearing a news cast, etc, they build a mental model of the situation and base their criteria on this model.

The implications of this are:

• Decision making is guided by whatever incomplete set of objectives is made salient
• Criteria generation can be improved by helping decision makers develop a broader understanding of the problem through:
• Time - One of the studies in the paper showed that addressing the problem at multiple points in time increases the number of criteria identified.
• Multiple perspectives - Although not in the study, multiple perspectives (i.e. a team approach) can help develop the broader understanding
• Templates - External aids can help in generating the broader understanding.
  

What follows are some thoughts on these three criteria development crutches

Time
My research in the 1980s focused on how engineers design products. In these studies my students and I video taped engineers solving simple but realistic design problems. We observed how engineers repeatedly returned to the problem statement as their mental model of the situation evolved. Before these experiments I tried to force students in my design classes to develop criteria first and then alternatives. The rationale was that, if you have a solution (or a set of solutions) in mind, then the criteria will evolve to match them. More recently, I have come to believe that criteria and alternatives co-evolve as the understanding is developed. That is not to say that you should just dive in. I now teach Quality Function Deployment (QFD) first but treat it as a living document. Also I have the students work in teams on all projects as that brings multiple perspectives.

Perspectives
One criticism of virtually all the research on decision making (including my own) is that it has been on individual decision makers. I comment on this in an earlier blog. The reality is that at work and to a great degree at home, we all solve problems with others. Either we bounce ideas off of each other or are on teams. In these situations the multiple perspectives help flesh out understanding and criteria. Large teams can actually reverse the situation. In many of my consulting jobs I see teams from multiple groups in an organization working to winnow down the criteria that are important for each individual group into a single, shared understanding. This is almost the antithesis of Keeney’s study.

Templates
The idea of using templates or other criteria crutches is one that we have tried to incorporate into Accord, our decision support software. Currently there are templates for about six different generic problems (e.g. Concept selection, Portfolio evaluation, Proposal selection, Vendor selection, Job candidate selection). However, many decisions in business are unique and developing a template for these problems is not possible. The paper that started this string “Robust Criteria for Robust Decisions” was an effort to address those problems that don’t allow for templates.

The upshot of the dialog with Ralph Keeney is that I will be doing some experiments this fall that address the two points I made initially. We will see what evolves to help frame decision problems.

Robust Criteria for Robust Decisions

I just wrote a newsletter that appear on my web site on how to develop criteria and titled "Robust Criteria for Robust Decisions" In it I state:

Research has shown:

1. The more effort put into understanding the criteria early in the process, the better the decision


2. Too little effort is generally put into understanding criteria.

After sending this to my mailing list, I received a message from Ralph Keeny (A major thinker in decision making for the last 20+ years) asking " I am very interested in both of these issues, and I believe that each are true. However, research addressing these issues is not so easy to come by. Hence, if it is not too inconvenient, I would be pleased to receive references of the research referred to in the statements. Thank you very much."


This is what I wrote back.


Thanks for the note. I agree that data supporting these two contentions are hard to come by. By far the best I have seen is from a German PhD dissertation that used protocol studies of mechanical engineers similar to those I did in the late 1980s: Thinking Methods and Procedures in Mechanical Design, Dissertation, Dylla, N., Technical University of Munich, 1991, in German. From Dylla’s data, I wrote the following and developed the plot (note this is from page 142 in Making Robust Decisions, some copies of which have the wrong plot for the figure)

The experimenters measured the amount of time each of the six engineers spent developing criteria. This included reading the given criteria, rereading them, and refining them. Then a team of professional engineers evaluated the technical quality of each design. Part of the evaluation concerned how well the final designs met the criteria, and part was more objective— evaluating the elegance of the solution. The evaluation team scored each of the six designs on a scale of 0 to 100. As figure 6.1 shows, there is a significant relationship between the percentage of time spent analyzing the goals of the problem and the technical quality of the result. The engineers who spent around 7% of their time understanding and developing criteria had a 60% better solution than those who spent 2–3% of their time developing criteria. I don’t mean to imply that 7% is an adequate time for working on the criteria; this particular experiment involved a simple, crafted problem and just one decision maker. The engineers didn’t spend all their criteria time at the beginning of the task. In fact, the successful engineers worked hard to refine the criteria at the beginning and then revisited and refined them many times during the course of the experiment. This result should come as no surprise: a prime measure of the success of a decision is how well the results meet the criteria. In general, the time you spend up front to clarify the problem (understand the criteria) saves time and many headaches later.







Admittedly, I have taken some license that a better mechanical design is analogous to a better decision. I don’t think the leap is very great however as deign is repetitive decision making.

The second point is based partailly on Dylla’s finding (4 of the 6 engineers might have done better had they put in more time on criteria) and partially on the studies done for the book Why Decisions Fail, by Paul Nutt, Berrett-Koelher, 2004. One of his three decision blunders is “Decision makers base many decisions on premature commitments.” Premature commitment implies that to little time is spent on one or all of the following 1) developing alternative courses of action, 2) developing criteria, 3) evaluating alternatives relative to criteria or 4) managing the decision making strategy. He never breaks this down, but on page 167 he compares the success of four different evaluation tactics: analytical, bargaining, subjective and judgment. Paraphrasing Nutt: In an analytical evaluation, data is gathered and inferences made from analytical tools. In judgment there are no specifics. Thus, analytical methods require more effort on the measures, i.e. the criteria than does judgment. He found a decision adoption rate of 64-75% when analytical methods were used versus 36% -47% for judgment. Unfortunately, Nutt never really addresses the evaluation details and wraps criteria development in with evaluation as many authors do.

All pretty weak stuff. To add useless anecdotal “data”, I see companies do a very poor job of defining criteria for making decisions. One let an RVP with 60+ specs. After reading the 15+ proposals these specs enabled them to separate them into two piles, acceptable and not acceptable. The specs were not really what they needed to make the decision amongst the “acceptable” proposals. They then needed to spend additional time determining what their criteria were for finding the best amongst the acceptable.

Do you have any references that might add to this?