In my blog of Sept 10, I noted that my newsletter statements in "Robust Criteria for Robust Decisions" had started an interesting conversation with Ralph Keeney. In the newsletter I had stated:

Research has shown:
1. The more effort put into understanding the criteria early in the process, the better the decision
2. Too little effort is generally put into understanding criteria.

He asked for references, and I provided, in my blog what weak evidence I had. He in turn sent me an interesting paper titled “Generating Objectives: Can Decision Makers Articulate What They Want?” (Management Science Vol 54 No 1, Jan 2008, pp 56-70). In this paper, Keeney and his colleagues present the results of a series of experiments designed to address how well people can list the objectives (i.e. criteria) they used in making decisions on real problems. In summary, Keeney and company concluded that people commonly undertake important decisions without considering many of the most important criteria. It seems that they generate only the objectives that are cued by their incomplete representation of the problem. In other words, as people work to understand a problem, by reading a problem statement, talking with others, hearing a news cast, etc, they build a mental model of the situation and base their criteria on this model.

The implications of this are:

• Decision making is guided by whatever incomplete set of objectives is made salient
• Criteria generation can be improved by helping decision makers develop a broader understanding of the problem through:
• Time - One of the studies in the paper showed that addressing the problem at multiple points in time increases the number of criteria identified.
• Multiple perspectives - Although not in the study, multiple perspectives (i.e. a team approach) can help develop the broader understanding
• Templates - External aids can help in generating the broader understanding.
  

What follows are some thoughts on these three criteria development crutches

Time
My research in the 1980s focused on how engineers design products. In these studies my students and I video taped engineers solving simple but realistic design problems. We observed how engineers repeatedly returned to the problem statement as their mental model of the situation evolved. Before these experiments I tried to force students in my design classes to develop criteria first and then alternatives. The rationale was that, if you have a solution (or a set of solutions) in mind, then the criteria will evolve to match them. More recently, I have come to believe that criteria and alternatives co-evolve as the understanding is developed. That is not to say that you should just dive in. I now teach Quality Function Deployment (QFD) first but treat it as a living document. Also I have the students work in teams on all projects as that brings multiple perspectives.

Perspectives
One criticism of virtually all the research on decision making (including my own) is that it has been on individual decision makers. I comment on this in an earlier blog. The reality is that at work and to a great degree at home, we all solve problems with others. Either we bounce ideas off of each other or are on teams. In these situations the multiple perspectives help flesh out understanding and criteria. Large teams can actually reverse the situation. In many of my consulting jobs I see teams from multiple groups in an organization working to winnow down the criteria that are important for each individual group into a single, shared understanding. This is almost the antithesis of Keeney’s study.

Templates
The idea of using templates or other criteria crutches is one that we have tried to incorporate into Accord, our decision support software. Currently there are templates for about six different generic problems (e.g. Concept selection, Portfolio evaluation, Proposal selection, Vendor selection, Job candidate selection). However, many decisions in business are unique and developing a template for these problems is not possible. The paper that started this string “Robust Criteria for Robust Decisions” was an effort to address those problems that don’t allow for templates.

The upshot of the dialog with Ralph Keeney is that I will be doing some experiments this fall that address the two points I made initially. We will see what evolves to help frame decision problems.