Effective Project Reviews

I just returned from facilitating a one day project review for a government agency. The project was a three year effort by a major aerospace contractor, it cost over $13M and was designed to develop a new technology. I knew nothing about the technology (still don’t), but I was hired by the agency to make the review process rational and useful for future projects. I will share the steps I used with you.

Step 1: Develop project targets. In order to have any usefulness, this must happen before the project begins. I use the term “targets” in the broadest sense, as will be discussed. In this case, the project was quite technical and specific targets for the measurable behavior of the resulting system were established about 3 years ago. I was not part of this process, but if I had been, we could have avoided some later review problems (explained below).

There are three important parts to this step.

What Features? What is important to measure should be addressed up front. Each feature measured should help answer the question “How will we know if the project is successful?” This means you need to understand what success means from the beginning in terms of functional performance, cost and time. Reflect on some recent projects or government actions (e.g. invasion of Iraq) and see if this question can be clearly answered. For the project I facilitated, there were seven clearly defined performance measures for the success.

How measurable? Not all features are measurable. In about 1893 Lord Kelvin said: “When you cannot measure it…your knowledge is of meager and unsatisfactory kind”. In 1956 Frank Knight (professor of four future Nobel Memorial Prize winners in economics) appended Lord Kelvin with: “Oh well, if you cannot measure, measure anyhow”. In the book Making Robust Decisions I spend much time discussing the differences between quantitative (Kelvinesque) and qualitative (Knightlike) criteria. For the project at hand, the targets were all quantitative with numerical targets were set. But, as we will see in a moment, they were not all realistic.

What are both the target and the threshold? It is always best to set two targets – an ideal target and a threshold. The ideal target is the level you would like to achieve if everything goes right, and the threshold is the level you might achieve if nothing goes right. The “target” must be realistic. If it is set beyond what has been previously achieved, then there had better be good evidence for this expected improvement. Setting two “targets” is a good idea whether the criterion is measurable or not. Even if a target is qualitative (Yes or No) then what “Yes” means and what “No” means should be defined ahead of time. For the facilitation job only a single “target” was set for each of seven measures. Some of these targets proved unrealistic, and could not be achieved in spite of the contractor’s best effort. If both a threshold and a target had been set, then at least the level of success could have been evaluated

Step 2: Develop Review Plan: Project reviews can happen at any time in the project. The review I facilitated was a final review, all work was finished. Besides when planning to hold reviews, it is important to identify who is going to do the review. For this case, the reviewers were all Subject Matter Experts (SEMs) flown in from around the country. They represented potential customers for the technology. Thus, the review served not only as an opportunity to measure how well the contractor completed the work, but also to identify future uses for the technology. Finally, the question of where to hold the review is becoming a more important issue. With the increase in capability of virtual meetings and web conferencing, and with increasing travel costs, sometimes “virtual” meeting are best.

Step 3: Measure How Well and How Certain: A quandary facing the government agency that sponsored this project was how to evaluate the project. The problem was that some of the targets were not reached (and no thresholds were set). So, on the face of it, the contractor had failed. But, they had advanced the sate-of-the-art significantly; just some of the targets were not realistic. To accommodate for this, the sponsor decided to treat the evaluations as qualitative and we made use of Belief Maps to help with the assessment.


A Belief Map is a simple grid on which an evaluator puts a single dot for each criterion to reflect how well the effort met the criterion versus how-certain they are of their assessment. In this project, all such assessments were made on paper, independently by each individual SEM, the sheets were collected and the data input into the Accord™ software for analysis. A sample sheet for one of the criteria is shown. It was blank (no dots on the Belief Map) when handed out. Here it is shown with the dots as submitted by seven SEMs.

The use of Belief Maps can help in managing:

• Review material is often incomplete, and may present evolving and uncertain information
• Reviewers have differing levels of understanding about material
• Some reviewers are well prepared, others are not
• Some reviewers have dominant personalities
• Some reviewers leave early or conduct other business

Step 4 Analyze Results to Guide Future Work: In just looking at the completed Belief Maps, much can be learned. If the reviewer’s dots are widely distributed vertically then there is poor agreement caused by some or all of the following:

1. The material presented to them is wanting
2. Progress relative to the measure is insufficient to judge consistently
3. The criteria is not defined clearly or measures multiple features

If many of the reviewer’s dots are to the left, then possibly:

1. They are not sufficiently expert to judge the material
2. The review material is not clear or the criterion poorly defined

For the project I facilitated, the reviewers were experts, the criteria clearly defined and the dots were well grouped vertically. Since they placed their dots on sheets independently, what we captured is a good indicator of the project’s quality. Horizontally, the dots ranged across the scale with some SEMs sufficiently expert to give confidence in the results.

Beyond what was done here, Accord can develop many useful statistics from the raw data. Further, it can fuse qualitative and quantitative evaluations and can collect data from a distributed team via the net.

Step 5 Finalize review: Finally, the material was documented for review by all concerned and use in future, similar projects. In this particular, case the final report was quite detailed due to the size of the investment and the use of a professional facilitator.

Summary
Hopefully, these five steps give you some good ideas for project evaluation. The five steps can make reviews useful exercises rather than a hoop to jump through. The use two-part targets and of Belief Maps can give a much richer window on the progress.

If I left something out or you have experiences that can be used to refine these steps, please share them.

7 Popular Methods for Making Decisions (None of which are advised)

Organizations use many different strategies to make decisions. The following is a list of strategies seen in practice. They all lead to decisions, but they all run the risk of not making good use of the available people and information. Have you ever experienced these?

1. Decision by Running Out of Time: This is the most common form of decision-making. There may be some effort to develop criteria and alternatives, but often time runs out before there is any effort to ensure that a robust decision is made.

2. Decision by chaos: The president of the company says, “I want our new product at the Atlanta trade show in two weeks”. The show is in two weeks. There is no rational way to prepare for the show and make robust decisions. The decisions made in the chaos may need revisiting after the show and work will need to be redone. Some people prefer to work in a chaotic environment, and when in positions of power, they will manufacture chaos as the working environment.

3. Decision by Fiat (or Decision by Authority): This is a very common style in autocratic organizations where a manager or someone else in authority decrees that a certain alternative is his/her favorite. It is often seen when the boss’s idea is chosen in order to preserve the relationship with him/her. This is more justification than decision-making.

4. Decision by Coercion: A champion for one alternative pressures his/her colleagues into submission. Often the loudest voice wins, the others having given up. One colleague referred to this style as “hijacking the process.

5. Decision by Competition: Here concern for who wins is most important, as instanced in most sports. This is often a win-lose situation and the relationship among individual team members is not important.

6. Decision by Voting: Democracy works, but does not often make the best possible choice. This decision making process is a weak form of compromise. Think how most products and businesses would operate if they were designed the same way we elect a president.

7. Decision by Inertia: This style is based on “We did it that way before” which may result in a robust decision, if the previous one was. But, not much progress or innovation is made using this style. Sometimes the tough decision is knowing when to innovate, and when to keep the cruise control on.


I have seen these all. Do you know of other dysfunctional decision making practices, ones that you have experienced?

What can you do to defuse these styles? The first step, like any good program is to realize that there is a problem and that put a name to it. Hopefully, this list can get you through that step. The second is to get general agreement that decision-making is a process. But, both of these steps may be hard as some managers don't want any rationality in the process. There are many ideas in my other blogs and on my web site. Do you have ways of managing these?

4 Surefire Ways to Make a Bad Decision

Bad decisions don’t stick – the issue get revisited again and again, people who agreed with the decision do something else, and resources get wasted on a bad idea. Here are four surefire ways to make a bad decision.

1. Make decisions too soon
Although books like the very popular Blink suggest that generally your first reaction is right, there is much proof to the contrary. Sure, if the house is burning and your first thought is to get out, that is a good thing. But, if your decision has many options, multiple factors to consider, or has the needed knowledge spread amongst multiple people, your knee-jerk decision is probably a poor one.

But, Paul Nutt in his book Why Decisions Fail, a study of over 400 business decisions found a leading cause of decision failure is “premature commitment”. One area where matching decisions to time is in Agile Software Programming which encourages “delaying commitments to the last responsible moment”. A great cartoon by Chris Matts that explains this methodology.

2. Ignore uncertainty
Any decision based on estimates about the future are uncertain and virtually all technical and business decisions are of this type. But, virtually all estimates suck. In an earlier blog post I told of the result of a simple experiment that showed people can’t even make simple estimates, much less real business estimates. There are no “accurate estimates” as an estimate is a distribution with a most likely value and uncertainty about that which is inherent in any real project.

If I ask you to estimate the time it will take to you to get to work, what will you tell me? You could say, “Oh about 15 minutes”, but is this the average time, the longest or the shortest. I wont know unless you give me more information. If I am planning on meeting you at work and I want to be 90% you will be there, then I need to know more about your commute than “15 minutes”.

3. Don’t itemize the important factors
It isn’t really clear how people naturally make decisions. One theory is that you use only the most important factor that can help you differentiate amongst the options. So, for example, if you want to buy a new car, and the most important factor is that it be red, then you choose the red car. If there is more than one red car available, then you use the next most important factor (e.g. cost or body style) and so on. This model is a little simplistic, but it does emphasize that you need to understand the factors to make a decision.

There are methods that can help a team, even a team with little agreement about what is important develop a set of factors that will help make a good decision .

4. Be blind to available information
A team might consider all the options and important factors to develop a robust decision. They write up their recommendation, send it to the ultimate “decider” and he ignores their results and makes a different decision. What a waste of time and money. One manager told me that he didn’t follow the recommendation because the team was not privy to all the information he had. This only compounds the sin.

Conclusion
Practice all four of these and you are just about guaranteed to make weak decisions. How will you know? Your decisions won’t stick - they will be revisited again, people will implement actions not chosen, or the results of the decision will be invisible next week.

If you have examples of these methods please share them.

If you know of other surefire ways to make bad decisions then please share these also, so they can be added to this list.