Decision-Management for the New Economy

The new economy forces new practices in making critical business decisions. Cost overruns, late to market, and rescoping all lead to inefficiencies that can no longer be tolerated. One approach to improve performance in these areas is to ensure that your decision-making processes lead to the best choices in an efficient manner every time.

Symptoms of poor decision practice are:

• Decisions take too long – some are discussed, shelved, discussed again a year later, with no resolution
• Meetings end with no clear direction forward – decisions aren’t made and actions not taken
• Firefighting dominates useful work – with some fires clearly caused by poor earlier decisions
• Projects championed by the strong dominate what is best for the organization
• Decisions come unstuck – you decide what to do next, everyone agrees, and then something different happens
• Decisions are made without using all of available information and you know it
• Risk is ignored or padded over – all decisions are based on uncertain information and thus are risky.

It is estimated that half of all decisions fail. By “fail” I mean that some time after the “decision” was made, there is no evidence that any effort went into making it, i.e. nothing changed. The only evidence that any effort was put into the decision was the expenditure of time and money. A decision is a call to action and if no useful action occurs, the time and effort that went into making it was wasted, or worse.

The risk of making a poor decision, one that isn’t actionable and doesn’t stick, can be reduced through decision-management. Decision management is a process whose goal is to use the available uncertain, incomplete, conflicting, and evolving information to make the best possible choices with known expected risks, within time and resources available. Some of the basics of decision management are:

• Decision-management begins with assuring that everyone is addressing the same issue. This isn’t a given. I have been in many meetings where everyone thought they were addressing the same topic, but weren’t. Even if generally on topic, everyone is making different assumptions and these must be made explicit. In watching the debates on bailouts and stimulus packages in early February 2009, it was clear that the issue is not well understood or agreed to.
• To make a decision, there must be more than one alternative. If there is only one alternative and rejecting it is not an option, then the effort is for justification, not decision making. Decision-management helps develop alternative courses of action. The Wall Street bailout program of late 2008 seems to be an instance of rubber stamping a single alternative.
• A major component of decision-management is developing a clear picture of how you will know if you have a good decision - what should be measured – what is important and to whom is it important. Determining stakeholders’ values before diving in too deep is critical if you want a result that everyone can buy into. The lack of clarity with the stimulus packages being debated in Washington seems to hinge on a clear picture of what is to be measured (e.g. jobs created, home foreclosures saved, etc.). Good decision-management helps to develop measures, and at the same time, honors differences of opinion about which measures are important. This might be helpful in Washington.
• All decisions are based on estimates that are uncertain and yet we do a very poor job of taking this uncertainty into account. Past performance may be known (if it was documented), the present is obscured by its immediacy, and the future is just the best guess. In other words, very little is known with certainty. Adding uncertainty at the end of a decision making process with a “what-if” discussion is too late. Decision-management helps identify and capture the information uncertainty at the beginning of the process in an effort to produce a robust decision, one that is as immune as possible to the uncertainty.
• A key part of decision management is a process to fuse together all of the information is a timely and useful manner. Sometimes, there is need for a fast decision and sometimes, if the stakes are high and time is available, more effort needs to be placed on issue, alternative, measure, and estimate development. Good decision-management controls the time spent versus the depth of analysis to reach a decision.

In the 1990s most organizations began to review their processes to make them more efficient, agile and manageable. Making processes more efficient is mandatory in the new economy. However, many find that as they refine their processes, inefficiencies occur with poor decisions. In other words, efficient processes need efficient decision-management. Good decision-management is possible in most organizations and can alleviate many of the symptoms itemized at the beginning of this article.

Decision-making maturity

I just spent a day judging a contest of pre-teens making design decisions. I learned a lot about decision-making. My day was in support of the US First Lego League (FLL) local competition. The theme this year’s event was “Climate Connections Challenge”. As part of the event the 10 -14 year-old students built robots out of Legos to compete in a simulated world. They also had to make a presentation on climate change and their community. Finally, they prove they could work as a team.

This year I was the chief Teamwork judge. To show teamwork each team (between 4 and 8 students) was given a simple task and 5 minutes to solve it. On entering the room, I welcomed them and had them gather around a table. I then put on the table a thin plywood disk and a small bag of large sized paper clips. I read them directions which were something like: “You are to build a structure that will hold the disk off the table. You can bend the paper clips in any form you like. The paper clips must be connected in some way. You can not use the plastic bag. You can not harm the plywood disk. You have 5 minutes. Go!”

Two other judges and I then observed them solving the problem. We were not interested in the solution, but in the team-work displayed during the solution. We had rubrics to guide our assessment and had time to ask questions at the end.

It dawned on me part way through the day that I had 21 samples of conceptual design decision-making by pre-teens to observe. This became clear to me too late to allow me to take formal data, but I did make the following anecdotal observations. In each, I juxtapose them to what I have observed in adult decision makers.

• As soon as I stopped reading, the kids grabbed the clips and started bending and talking at the same time. One team was so eager I feared for my safety as I jumped out of the way (slight exaggeration). There was no planning or reflection by any of the teams! I did an experiment in the late 1980s where I gave 6 professional engineers a design problem and five hours to work it. I video taped the sessions. Most of these mature engineers read the requirement over several times before beginning to generate ideas. One subject however dove right in like these pre-teens. He pursued his first and only idea and proceeded to patch on it (see next item), never leading to a reasonable solution of the problem. My conclusion from these experiences is that it takes maturity to do the up front work necessary to make decisions.
• Once the kids jumped in, they patched their way to a solution. Patching means to try different ideas until you stumble onto a solution (I discuss this further in The Mechanical Design Process). What is bad about patching is that it is usually random with no structured method to explore the design space. I saw the same with the engineer described above and other immature designers.
• When they asked clarification questions we only responded with “Read the problem description”. Few did (less than 1/3). They took our response to mean “No”. to whatever was asked. I was surprised at this and after the first few times, I made a big deal of laying down the problem description (in big font on green paper) in the same location as the wooden disk and paper clips. Only one team (out of 21) reread the description together to clarify their issue after we prompted them. My generalization of this is that, for the most part people don’t use all the information available to them.
• On the whole most of the teams did a good job of including most of the members on their team. This is a credit to FLL and the mentors who worked with the teams. I have had many college level teams that were not as inclusive.

By the way, there were three classes of solutions, 1) bend the clips so they sat on the table and the disk was supported by upward extending wire legs (most did this with great patching), 2) bend the clips so that they fastened onto the disk with downward extending legs (only saw this done once successfully), and 3) lay two clips flat on the table and set the disk on top. The last was the most elegant and simple. One team discovered this 40 sec into the five minutes. To see more teamwork, I quickly said “Great, now make the support as tall as possible” (whew!). Most teams never found this solution, but with their dive right in approach and random patching, I am not surprised.

Decision Thinking – its time is coming

I was trained as a mechanical engineer. I know how to model systems, take data and develop an understanding for physical things. Most (nearly all) technical university courses are about how to analyze things. In my case these were physical things, but my education could have been in any engineering discipline, in business or in the sciences, and the courses still would have been primarily “thing” focused.

In the 1980s I was teaching mechanical engineering design at a university and began to appreciate that things come into being through a process. My thinking moved from being thing focused to being process focused. Process thinking was not new to some fields. In fact, in engineering there are control processes, chemical processes, fluid thermal processes etc. But these are nothing but “process” things. What I became interested in was the process of developing these things.

In 1990 I began work on a text book for mechanical engineers so that they could study the process of how things mature from need to a final, working object. I agonized over the title as the term “process” was not really a part of the engineering lexicon except for process things. I finally chose the title “The Mechanical Design Process” and the book was published in 1992. This title turned out to be a good choice and the 4th edition of the book is out in January 2009.

When I began to research the design process in about 1984, my goal was to understand how objects evolve with eye toward developing methods and tools to better support this evolution. I don’t mean CAD and solid modeling tools as these are representation tools for what is being designed, not tools that actually support the design process. In fact, I have argued in a paper I wrote in 1990 that CAD can be detrimental to the design process ( Ullman and Wood, 1990). Since writing this paper, CAD has evolved into solid modeling which is much better, but the arguments in the paper still hold.

Anyway, I was especially interested in developing a tool that could record the evolution and rationale for product evolution. This design rationale system would be able to capture how a product came into being and could be reused, queried and vetted to form a permanent record of its birth, life and death. When I began, I focused on the evolution of the assemblies, parts, and features. I quickly realized that this wouldn’t work because these things evolve during the design process and thus, focusing on them missed all the birthing – all the interesting, creative engineering.

My thinking turned to capturing the process through which things evolved. Process thinking encapsulates thing thinking as the process is about the evolving things. This shift in my thinking coincided with the text book mentioned above. However, it became evident by the mid 90s that process thinking, although much better than thing thinking, was not the best way to develop a design rationale system. What became evident was that decisions are the punctuation marks in the process and that my approach had to make yet another shift, one to decision thinking.

Thus, by the late 1990s my thinking had matured from thing, to process, to the decisions made during the process to develop things. Specifically, I wanted to understand and support the decision-making process. My research showed that on macro level, decisions were made at gates (in a stage-gate process) a countably few times during the evolution of a product. On the micro level – the cognitive level - they occur about 1 decision per minute (see Stauffer,and Ullman 1991). Somewhere between these extremes there is much need for decision making support.

Before we go on, a definition of decision thinking - decision thinking is focusing on the decision-making process used during technical or business development. “Focusing” implies understanding and supporting individual decision makers and teams of people making decisions when information is incomplete, evolving and conflicting so that the decisions are robust.

I am not alone in this evolution in thinking. First, when I chose the tile “The Mechanical Design Process”, the word “process” was problematic as it was not commonly used in industry. Now, the product development industry makes good use of process thinking. Then, when I started talking about decision thinking in the late 1990s few in industry knew what I was talking about. Now there is much evidence that companies and the government are beginning to realize the importance of decision-making in their processes. My evidence for this is not firm, but many of my contacts seem to understand what I am talking about and few did five years ago, and the number of hits on the Robust Decisions web site continues to climb. Their thinking is maturing through the process to the decisions made during the process.

Second, the CAD industry matured into PLM (Product Lifecycle Management) during the 1990s. Where CAD and solid modeling is about things, PLM is about processes that manage things and document things. I have tried to interest the PLM vendors in decision thinking for about eight years. Initially they told me there was no customer pull (see the previous paragraph). Recently, I have gotten their attention. Now that they have the process under control, they too are maturing toward decision thinking.

Ullman, D.G., S. Wood, D. Craig, "The Importance of Drawing in the Mechanical Design Process," Computers and Graphics, Special Issue on Features and Geometric Reasoning, Vol. 14, No. 2, 1990, pp. 263-274.

Stauffer, L.A., D.G. Ullman, "Fundamental Processes of Mechanical Designers Based on Empirical Data," Journal of Engineering Design, Vol. 2, No. 2, 1991, pp. 113-126.

Risk Management as a decision issue

The term “risk management” has been around for a long time in financial, technical and medical practice. It is a term that is very loosely used and I want to dive in with a decision-centric view just to further muddy the waters.

A good place to begin is with a formal definition of “risk”. If you enter “risk definition” into Google you will get over twenty-five definitions; some are redundant, and there is little consistency. Regardless of the definition, risk traditionally amounts to answering three questions:

1. What can go wrong? Some event.
2. How likely is it to happen? Probability that the event happens based past statistics, an analytical model of the event, or best guess based on experience.
3. What are the consequences? Money, time, and possibly even lives are wasted.

The above set of questions describes event risk. They are the basis of technical risk evaluation. However, most managers are concerned with decision risk rather than event risk. For decision risk, the questions managers really want answered are:

1. What can go wrong if I choose alternative X?
2. How likely is it?
3. What is the impact?
   

(Brian Seitz of Microsoft articulated these as cleanly as shown here)

Note that these are the same three questions as with event risk, just slightly tweaked. Regardless of whether focused on an event or a decision, risk management, by definition needs to be effort directed at some or all of these questions.

The first question raises the issue of how we know our choice was poor. In the book Why Decisions Fail, the definition of a poor decision is one that had no positive impact after two years. There are other measures of a failed decision. For example, our time is up and no satisfactory alternative has been developed. Or not everyone on the team agrees with the choice made and some team members feel disenfranchised. More often, the result of a poor choice is not known until much later. For example, if we choose a bad restaurant, we will not know until we have eaten there. Or if an engineer makes a poor decision on what material to use for a product, this may not be evident until the customers have used the product for a number of years.

One thing is consistent in this discussion: Without uncertainty there is no risk. A corollary is that the more uncertainty, the higher the risk of making a poor decision. “What can go wrong?” is that one or more of the criteria are not satisfied. “How likely is it?” is directly dependent on our certainty during the alternative evaluation. We may know from past experience or data that the probability of something failing is XX%. But, this probability may be compounded by other uncertainties such as lack of knowledge, disagreement amongst team members or incomplete data. And “What is the impact?” is that the alternative chosen no longer is as good as it was originally thought.

In order to manage risk during decision making:

• You must address risk during decision making, not as a task to complete after you have selected an alternative. This is because decision risk is a measure of your lack of knowledge, as well as other uncertainties you need to consider when you make a decision.
• There is risk associated with every feature of every alternative. Traditional risk assessment separately addresses financial risk, performance risk, and schedule risk. But when including risk as a part of the decision-making process, you must integrate the uncertainty inherent in all the features at once, because it is the combination of them that drives your decision.
• You can get a good assessment of uncertainty, and thus risk, by fusing the evaluations of all the members of your team.

More on Criteria Development

In my blog of Sept 10, I noted that my newsletter statements in "Robust Criteria for Robust Decisions" had started an interesting conversation with Ralph Keeney. In the newsletter I had stated:

Research has shown:
1. The more effort put into understanding the criteria early in the process, the better the decision
2. Too little effort is generally put into understanding criteria.

He asked for references, and I provided, in my blog what weak evidence I had. He in turn sent me an interesting paper titled “Generating Objectives: Can Decision Makers Articulate What They Want?” (Management Science Vol 54 No 1, Jan 2008, pp 56-70). In this paper, Keeney and his colleagues present the results of a series of experiments designed to address how well people can list the objectives (i.e. criteria) they used in making decisions on real problems. In summary, Keeney and company concluded that people commonly undertake important decisions without considering many of the most important criteria. It seems that they generate only the objectives that are cued by their incomplete representation of the problem. In other words, as people work to understand a problem, by reading a problem statement, talking with others, hearing a news cast, etc, they build a mental model of the situation and base their criteria on this model.

The implications of this are:

• Decision making is guided by whatever incomplete set of objectives is made salient
• Criteria generation can be improved by helping decision makers develop a broader understanding of the problem through:
• Time - One of the studies in the paper showed that addressing the problem at multiple points in time increases the number of criteria identified.
• Multiple perspectives - Although not in the study, multiple perspectives (i.e. a team approach) can help develop the broader understanding
• Templates - External aids can help in generating the broader understanding.
  

What follows are some thoughts on these three criteria development crutches

Time
My research in the 1980s focused on how engineers design products. In these studies my students and I video taped engineers solving simple but realistic design problems. We observed how engineers repeatedly returned to the problem statement as their mental model of the situation evolved. Before these experiments I tried to force students in my design classes to develop criteria first and then alternatives. The rationale was that, if you have a solution (or a set of solutions) in mind, then the criteria will evolve to match them. More recently, I have come to believe that criteria and alternatives co-evolve as the understanding is developed. That is not to say that you should just dive in. I now teach Quality Function Deployment (QFD) first but treat it as a living document. Also I have the students work in teams on all projects as that brings multiple perspectives.

Perspectives
One criticism of virtually all the research on decision making (including my own) is that it has been on individual decision makers. I comment on this in an earlier blog. The reality is that at work and to a great degree at home, we all solve problems with others. Either we bounce ideas off of each other or are on teams. In these situations the multiple perspectives help flesh out understanding and criteria. Large teams can actually reverse the situation. In many of my consulting jobs I see teams from multiple groups in an organization working to winnow down the criteria that are important for each individual group into a single, shared understanding. This is almost the antithesis of Keeney’s study.

Templates
The idea of using templates or other criteria crutches is one that we have tried to incorporate into Accord, our decision support software. Currently there are templates for about six different generic problems (e.g. Concept selection, Portfolio evaluation, Proposal selection, Vendor selection, Job candidate selection). However, many decisions in business are unique and developing a template for these problems is not possible. The paper that started this string “Robust Criteria for Robust Decisions” was an effort to address those problems that don’t allow for templates.

The upshot of the dialog with Ralph Keeney is that I will be doing some experiments this fall that address the two points I made initially. We will see what evolves to help frame decision problems.

Robust Criteria for Robust Decisions

I just wrote a newsletter that appear on my web site on how to develop criteria and titled "Robust Criteria for Robust Decisions" In it I state:

Research has shown:

1. The more effort put into understanding the criteria early in the process, the better the decision


2. Too little effort is generally put into understanding criteria.

After sending this to my mailing list, I received a message from Ralph Keeny (A major thinker in decision making for the last 20+ years) asking " I am very interested in both of these issues, and I believe that each are true. However, research addressing these issues is not so easy to come by. Hence, if it is not too inconvenient, I would be pleased to receive references of the research referred to in the statements. Thank you very much."


This is what I wrote back.


Thanks for the note. I agree that data supporting these two contentions are hard to come by. By far the best I have seen is from a German PhD dissertation that used protocol studies of mechanical engineers similar to those I did in the late 1980s: Thinking Methods and Procedures in Mechanical Design, Dissertation, Dylla, N., Technical University of Munich, 1991, in German. From Dylla’s data, I wrote the following and developed the plot (note this is from page 142 in Making Robust Decisions, some copies of which have the wrong plot for the figure)

The experimenters measured the amount of time each of the six engineers spent developing criteria. This included reading the given criteria, rereading them, and refining them. Then a team of professional engineers evaluated the technical quality of each design. Part of the evaluation concerned how well the final designs met the criteria, and part was more objective— evaluating the elegance of the solution. The evaluation team scored each of the six designs on a scale of 0 to 100. As figure 6.1 shows, there is a significant relationship between the percentage of time spent analyzing the goals of the problem and the technical quality of the result. The engineers who spent around 7% of their time understanding and developing criteria had a 60% better solution than those who spent 2–3% of their time developing criteria. I don’t mean to imply that 7% is an adequate time for working on the criteria; this particular experiment involved a simple, crafted problem and just one decision maker. The engineers didn’t spend all their criteria time at the beginning of the task. In fact, the successful engineers worked hard to refine the criteria at the beginning and then revisited and refined them many times during the course of the experiment. This result should come as no surprise: a prime measure of the success of a decision is how well the results meet the criteria. In general, the time you spend up front to clarify the problem (understand the criteria) saves time and many headaches later.







Admittedly, I have taken some license that a better mechanical design is analogous to a better decision. I don’t think the leap is very great however as deign is repetitive decision making.

The second point is based partailly on Dylla’s finding (4 of the 6 engineers might have done better had they put in more time on criteria) and partially on the studies done for the book Why Decisions Fail, by Paul Nutt, Berrett-Koelher, 2004. One of his three decision blunders is “Decision makers base many decisions on premature commitments.” Premature commitment implies that to little time is spent on one or all of the following 1) developing alternative courses of action, 2) developing criteria, 3) evaluating alternatives relative to criteria or 4) managing the decision making strategy. He never breaks this down, but on page 167 he compares the success of four different evaluation tactics: analytical, bargaining, subjective and judgment. Paraphrasing Nutt: In an analytical evaluation, data is gathered and inferences made from analytical tools. In judgment there are no specifics. Thus, analytical methods require more effort on the measures, i.e. the criteria than does judgment. He found a decision adoption rate of 64-75% when analytical methods were used versus 36% -47% for judgment. Unfortunately, Nutt never really addresses the evaluation details and wraps criteria development in with evaluation as many authors do.

All pretty weak stuff. To add useless anecdotal “data”, I see companies do a very poor job of defining criteria for making decisions. One let an RVP with 60+ specs. After reading the 15+ proposals these specs enabled them to separate them into two piles, acceptable and not acceptable. The specs were not really what they needed to make the decision amongst the “acceptable” proposals. They then needed to spend additional time determining what their criteria were for finding the best amongst the acceptable.

Do you have any references that might add to this?

Why pairwise comparisons are a waste of time for finding criteria importance

Many people use pairwise comparisons during their decision making efforts. First, this method may be used to determine the relative importance (the weightings) of the criteria. Second, pairwise comparisons are sometimes used to evaluate the alternatives relative to the criteria. BOTH OF THESE ARE A WASTE OF TIME!! Don’t get me wrong, I think pairwise comparisons can be helpful, just that there are faster ways of getting virtually the same results. In this brief note I will only tackle why you shouldn't bother for finding importance.

Through his books and companies, Tom Saaty has popularized pairwise comparisons as a part of his Analytic Hierarchy[1] (AHP) and Analytic Network[2] Processes In the Analytic Network Process book, on pages 26- 31, Saaty gives the example of using eight criteria to help select a house (e.g. Size of House, Transportation, Neighborhood, etc). His method requires that all the criteria be compared to each other one pair at a time to find the most important for each comparison. Further, a dominance factor is given to the better of the pair relating how much more important one factor is to another. If working with a team, they need to come to agreement on the dominance factors (I will come back to this point later). For the 8 criteria, there are 28 comparisons and the need to judge 28 dominance factors. In general, for N factors there are (N-1) + (N-2) +…. 1 comparisons.

For the example in his book (where the numbers represent the 8 criteria (i.e. factors)) a matrix of the pairwise comparisons is a shown below. Note that the opposite entries are just reciprocals of each other. Criterion 1 is 5 times as important as criterion 2 and so criterion 2 is 1/5 as important as criterion 1.


The Priority Vector is reduction of the values (using an eigenvector analysis) to develop the relative weightings - the importance of each criterion.

Compare this to a method proposed by Ward Edwards, one of the fathers of modern decision theory. He suggested that asking decision makers to weigh criteria is so fraught with error that it is easier, and no less accurate, just to ask them to rank the criteria and then automatically set the weights according to the ranking[3]. This “error” is exasperated when there are multiple constituencies represented in the organization.

Find the rank order the criteria, write each criterion on a sticky note and arrange them on a wall or desk, and reorder with the most important on top. This is best done in a pairwise fashion by selecting the criteria two at a time and asking, “If an alternative could meet only one of these, which criteria would I choose?” Then, move the chosen one to the top and the other to the bottom of the arrangement.

You can convert the ranking to a weighting by using the table below. This table shows the Rank Order Centroid (ROC) method developed by Edwards and shows the weights for up to 12 criteria.


Weights based on ROC method

If you have more than 12 criteria, you can use the equation wk= (1/K) ∑ (1/ i ) as i goes from k (the number of the criterion, with 1 being the highest weighted and K being the lowest) to K (the number of criteria). This equation was used to generate the values in the table. The values for 8 criteria are shaded in the table above and plotted below compared to the pairwise method.

The results of the two methods are shown on the bar chart below.


The Mean Absolute Error between the two (the sum of the absolute differences between the two) is, on average, 2%. Other examples I have tried have even had less error. Considering that there is no right answer and that one change in pairwise comparisons can change the results. The difference between the two comes at the expense of a major difference in the amount of effort. Twenty eight comparisons and assignments of priorities versus simply rank ordering the criteria.

Now, adherents of pairwise comparisons can argue that the method also computes consistency, a measure of how well the many dominance factors agree with each other. I believe this is of little importance as the dominance factors are just averages across a committee of stakeholders who are trying to quantify subjective values. In other words, the uncertainty in the pairwise scoring is so high due to averaging and quantifying subjective values that consistency in the dominance factors is just noise. Consistency analysis gives false comfort that the matrix is consistent when the numbers themselves are very uncertain.

Based on the above arguments, I believe pairwise comparisons are a waste of time. I prefer to allow all the stake holders to rank order and use the resulting inconsistent weighting factors in my analysis. Thus, I honor all party’s values and use them all in downstream analysis. More on this in another note.

[1] Decision Making for leaders: The Analytic Hierarchy Process for Decisions in a complex World, Thomas L. Saaty,RSW Publications, Pittsburgh PA, 1999
[2] The Analytic Network Process, Thomas L. Saaty, RWS Publications, 1996.
[3] Edwards, Ward and F. Hutton Barron, "SMARTS and SMARTER: Improved Simple Methods for Multi-attribute Utility Measurement" and F. Hutton Barron and Bruce Barrett, "Decision Quality Using Ranked and Partially Ranked Attribute Weights."