Robust Criteria for Robust Decisions

Decision-Making Resources > Decision Expert Newsletter > Robust Criteria for Robust Decisions

Selecting the proper criteria to improve the decision-making process

The goal of this short article is to present some methods for developing a good set of decision-making criteria. Since a decision is only as good as the criteria on which it is based, it is worth spending time consciously developing criteria before effort is put into making a decision.

Criteria measure how well the alternatives resolve an issue. Often criteria are captured in project goals and requirements, but not always. Research has shown:

The more effort put into understanding the criteria early in the process, the better the decision, and
Too little effort is generally put into understanding criteria.

Thus, the goal of this article is to present methods for generating criteria early in the decision-making process and give ideas about what criteria developing "effort" can lead to better decisions. Before presenting these methods, a little background is needed.

Criteria are needed to determine how well an alternative meets the need and to reveal differences between the alternatives. Let's look at a simplistic example. When I go to a store to buy a bicycle, I do not tell the dealer that one of my criteria for choosing my new ride is that it should have two wheels. All bikes have two wheels, so this criterion doesn't help me choose a bicycle or discriminate between the different models. If the dealer is clever, she will help me develop criteria to choose and discriminate among the bikes in the store.

Two examples from my consulting work will help illustrate the concept of discrimination. First, a company that required a custom piece of electronic equipment sent a request for proposals (RFP) to its vendors. In it were listed 60+ requirements for physical size, power required, signal conditioning, etc. When the company's decision makers received the proposals, they filtered them based on the requirements. Those that didn't meet the requirements were eliminated from consideration. Then came the hard part: figuring out how to discriminate among the remaining acceptable proposals. Unfortunately, the true "discriminating criteria," those that could help in selecting the best from among acceptable alternatives, had not been well articulated in the initial set of requirements. Thus, there were no criteria to use to choose among the best proposals.

A second company, one that develops inkjet printers, formed a team to select the ink chemistry for a new delivery system. The team was composed of eighteen people from six different functions in the organization, (e.g., ink chemistry, manufacturing, and image quality). Altogether, they had over 100 requirements that had to be met before they could approve a new ink chemistry. Most of these filtered out inks that couldn't meet their basic needs. Only a few were used to discriminate between the strongest candidates. Developing the discriminating criteria based on the initial list of requirements took team effort, but helped the team develop a shared understanding of the problem and, ultimately, make a decision.

There has been a big push in the last 10-15 years to collect and manage the requirements or specifications for products and needed business decisions.

These are generally of the form, "a feature must meet a target". A feature (also called an "attribute" or "parameter") is a characteristic of the alternatives that may be important to consider. A target is the stated or unstated goal for the feature. Often it is recommended that the specifications be labeled as "must" or "want". A better classification is that when used as criteria, these requirements are either "filters" or "discriminators". Most requirements or specifications are; therefore, filter criteria that eliminate alternatives from consideration (as in the examples above). It is the discriminating criteria that more directly support decision-making. Furthermore, most discriminating criteria are trade-off criteria — in other words, you might have to give up some of x to get sufficient y. (For more detail on trades offs, see a published paper on the topic. Using our bicycle purchasing example above, after the clerk works with me for a few minutes, we ascertain that I want a road bike that weighs less than 14kg (30 lbs) and costs less than $1000. She then shows me a $1,100 bike that weighs 13kg and a $900 — 15kg bike. I must trade off cost against weight if I am to choose one of these bikes.

One useful way to manage trade-offs is to set a delighted (actual target) value and a disgusted (threshold) value. Most methods only develop a single target, but defining two targets from the beginning leads to a better understanding of sensitivity among alternatives. Say for example that money is a major constraint and my delighted cost is $900 and my threshold value is $1000. I am not as critical about weight, so I will be delighted at 14kg and disgusted at 17 kg. The choice between the two bikes is clearer now.

One additional note, criteria and their targets may be qualitative or quantitative. Although good practice says to measure everything, it takes work, time and knowledge to measure most attributes. In fact, the reason the discriminating criteria are often not made explicit is that they are qualitative and are only realized after the quantitative, filtering criteria are applied. Even the most refined technical disciplines usually use qualitative measures in their decision-making.

To help find the discriminating criteria, here are four different methods that are easy to apply:

1. Discriminating criteria from requirements or specifications

A good starting point is to If you already have a set of requirements or specifications for products and needed business decisions. The following steps should help:

Step 1: Label each requirement as a filter (F) or a discriminating (D) criterion. To do this, ask the question of each "Do all alternatives have to meet this requirement?" If the answer is yes, it is a filter (F), if no then it may be a discriminator (D).
Step 2: Of the remaining requirements, ask, "Can success in meeting this requirement be traded off to meet another requirement?" If the answer is yes, then it may be a discriminator (D).
Step 3: Of the remaining, unlabeled requirements, revisit Steps 1 and 2 and see if they can be reformulated to either filter or discriminate.
Step 4: Review the discriminator list and ask "What measures are still missing?" (try using the Pro/Con Evaluation method below).
Step 5: Try to reduce the list of discriminating criteria to <10. Most choices amongst the best few alternatives come down to just a few discriminating criteria. See the Criteria for Criteria discussion at the end of this paper for some ideas.

2. Discriminating criteria from the issue description

Often the issue (e.g. the statement or question requiring a choice to be made) is poorly defined. If you ask each member of a team or each stakeholder to describe the situation needing a decision, you may get a wide variety of statements, or fairly good agreement with varying caveats. One method to manage this situation is to have each person write down the issue and then work communally to reduce it to a single sentence or question. The goal is agreement on the one sentence before accepting it as the true issue. However, as the issue evolves, note all the caveats. For example, I want a bicycle that is light, good for road trips, one that is inexpensive and looks fast. The issue is "Choose a bicycle". The criteria (at least my initial set of them) focus on weight, roadability, cost and looks. Each item in the list is a potential discriminating criterion.

Step 1: Have each member of the team or stakeholders write down what they think the issue is.
Step 2: Extract all the features (the caveats) mentioned by each.
Step 3: Work to reduce the issue statement to a single sentence and the features mentioned to a list of discriminating criteria.
Step 4: Use the Criteria for Criteria below to refine the list.

3. Discriminating criteria from Pro/Con Evaluations

In a 1772 letter[1] to his nephew, Joseph Priestly (the discoverer of Oxygen), Benjamin Franklin explained how he made decisions using a Pro/Con analysis in situations when he had a choice between two alternatives: Do this, or do something else (including nothing). Franklin's letter reduces to five steps for making a decision:

Make two columns on a sheet of paper and label one "Pros" and the other "Cons."
Fill in the columns with all the Pros and Cons of an alternative.
Estimate the importance of each Pro and Con.
Eliminate Pros and Cons this way:
- When two are of about equal importance, cross them both out and
- Find other importance equalities of Pros and Cons (e.g. the importance of two pros equals three cons) and then strike them out.
When one or the other column becomes dominant, then "come to the determination accordingly."

You can extend the idea of using Pro/Con lists to include more than two alternatives, but the balancing step quickly becomes complex. Still, NASA frequently uses this approach to help experts evaluate multiple project proposals at once. For each proposal, the experts list his/her Pros and Cons. They then informally balance the Pros and Cons to differentiate among the alternatives.

It is important to note that each Pro or Con is an application of a criterion to a single alternative. What is fragile here is that, in general, only the most glaring pros or cons are listed for each alternative. This means that the assessment may be very uneven and trade-offs not made very clearly. For example, weight may be important when considering one bike, but maybe not for another, as some other measure is a more glaring pro or con. However, when two bikes are both close to being acceptable choices, it is necessary to trade off amongst all the discriminating criteria and a Pro/Con list does make this obvious.

To formalize the Pro/Con process and make it useful for discovering discriminating criteria, try the following:

Step 1: Have each member of the team record Pros and Cons for each alternative (at least their favorites).
(Step 1 optional): For selected pairs of alternatives record the Pros and Cons. This pair-wise comparison can help tease out the Pros and Cons.
Step 2: Identify and list the features being measured in each Pro/Con statement. Read the statement and find the clause that describes the feature or attribute that is the focus of the statement.
Step 3: Use affinity methods to organize the features. Try to reduce the total number to <10.
Step 4: For each measure develop the delighted and disgusted targets. One method to do this is to review the alternatives and, for each feature, list the best value (if quantitative) or the best description (if quantitative). These are the delighted values. Do the same for the disgusted. Do not set your targets outside the rage of delighted — disgusted unless you have convincing proof that new alternatives can perform in the expanded range.
Step 5: Apply Criteria of Criteria — see below

4. Discriminating Criteria from SWOT Analysis

SWOT analyses are often used to support business decisions. SWOT stands for Strengths, Weaknesses, Opportunities and Threats. This method gives a refined structure to listing the Pros and Cons. Replace Steps 1 and 2 in the Pro/con with:

Step 1: Have each member of the team record SWOTs for each alternative.
Step 2: Identify and list the features being measured in each SWOT statement.

Criteria for Criteria

Regardless of which method you use, apply the following criteria for criteria to help refine your criterion list:

Does this criterion discriminate? When various alternatives are measured relative to this criterion, will the results differ? If they do then this criterion discriminates. If not, then the criterion will not help in differentiating amongst the alternatives.

	A	B	C
1	↑	↓	↓
2	↑	↑	↓
3	↑	↑	↓

Is this criterion independent of the other criteria? If two criteria are dependent then you may be measuring the same thing twice. A test for independence is; regardless of the alternative considered, does a positive response to two criteria always result in the same change in satisfaction for the alternatives? If the answer is yes, then they are dependent. For example, three alternatives A, B and C are being evaluated versus three criteria, 1, 2 and 3. Say that A satisfies criterion 1, and B and C do not. Both A and B satisfy 2 and 3 and C does not. Criterion 1 is differentiating, but 2 and 3 may be dependent as they provide the same measure on the alternatives. A single, reasonable counterexample to the question (one alternative whose satisfaction relative to the two criteria differs from the others] is sufficient to show independence.
Does this criterion measure one thing? If a criterion is measuring more than one feature, it should be broken into multiple criteria. However, multiple measures can be combined into a single parameter that measures a significant feature, and this new parameter can be considered as a single measure. For example, in physics, engineering, and economics there are many dimensionless parameters, combinations of multiple measures that are very useful and make good criteria [e.g. NPV, ROI, Mach number, aspect ratio, decibel].
Is the criterion universal? A universal criterion characterizes an attribute of all the proposed alternatives. If a criterion only applies to some of the alternatives then it is not universal. If not universal, then either it is a poor criterion or the alternatives have features that are not consistent with the issue being addressed.
Is the criterion positive? Criteria should be stated such that a "yes" response to it indicates a good feature of the alternative.
Is that criterion important to some of the stakeholders? Generally, most criteria filter out weak alternatives. Only a few are important enough to discriminate between acceptable alternatives. If the criterion is not amongst the 10 most important then it should be used as a filter.
In this short article, I have developed some methods for helping to find criteria for robust decisions. What is important to realize is that the requirements often developed for a project or product do not necessarily give a complete and sufficient set of criteria for making decisions. Hopefully the methods in this article will help build the foundation for good decisions.

Please send us your comments or suggestions.

[1] This letter is reprinted in Making Robust Decisions pg 42.

Click below to:

Learn more about the book Making Robust Decisions and how it can help produce optimal decisions.
Download a free trial version of the NEW Accord 2.5 software.
Have a Robust Decisions representative contact you.
Read the latest decision making blog posts.

Are you ready to improve the decision-making in your organization? Robust Decisions offers a full range of products and services that allow you to benefit by making the best decision every time. Learn more about how to build a strong foundation that enables individuals, teams and entire organizations to consistently make the best possible decisions. Request more information or call 541.754.3609.

Newsletter Archive

Learn More

Decision-Maker Blog

» view all posts