I just returned from facilitating a one day project review for a government agency. The project was a three year effort by a major aerospace contractor, it cost over $13M and was designed to develop a new technology. I knew nothing about the technology (still don’t), but I was hired by the agency to make the review process rational and useful for future projects. I will share the steps I used with you.

Step 1: Develop project targets. In order to have any usefulness, this must happen before the project begins. I use the term “targets” in the broadest sense, as will be discussed. In this case, the project was quite technical and specific targets for the measurable behavior of the resulting system were established about 3 years ago. I was not part of this process, but if I had been, we could have avoided some later review problems (explained below).

There are three important parts to this step.

What Features? What is important to measure should be addressed up front. Each feature measured should help answer the question “How will we know if the project is successful?” This means you need to understand what success means from the beginning in terms of functional performance, cost and time. Reflect on some recent projects or government actions (e.g. invasion of Iraq) and see if this question can be clearly answered. For the project I facilitated, there were seven clearly defined performance measures for the success.

How measurable? Not all features are measurable. In about 1893 Lord Kelvin said: “When you cannot measure it…your knowledge is of meager and unsatisfactory kind”. In 1956 Frank Knight (professor of four future Nobel Memorial Prize winners in economics) appended Lord Kelvin with: “Oh well, if you cannot measure, measure anyhow”. In the book Making Robust Decisions I spend much time discussing the differences between quantitative (Kelvinesque) and qualitative (Knightlike) criteria. For the project at hand, the targets were all quantitative with numerical targets were set. But, as we will see in a moment, they were not all realistic.

What are both the target and the threshold? It is always best to set two targets – an ideal target and a threshold. The ideal target is the level you would like to achieve if everything goes right, and the threshold is the level you might achieve if nothing goes right. The “target” must be realistic. If it is set beyond what has been previously achieved, then there had better be good evidence for this expected improvement. Setting two “targets” is a good idea whether the criterion is measurable or not. Even if a target is qualitative (Yes or No) then what “Yes” means and what “No” means should be defined ahead of time. For the facilitation job only a single “target” was set for each of seven measures. Some of these targets proved unrealistic, and could not be achieved in spite of the contractor’s best effort. If both a threshold and a target had been set, then at least the level of success could have been evaluated

Step 2: Develop Review Plan: Project reviews can happen at any time in the project. The review I facilitated was a final review, all work was finished. Besides when planning to hold reviews, it is important to identify who is going to do the review. For this case, the reviewers were all Subject Matter Experts (SEMs) flown in from around the country. They represented potential customers for the technology. Thus, the review served not only as an opportunity to measure how well the contractor completed the work, but also to identify future uses for the technology. Finally, the question of where to hold the review is becoming a more important issue. With the increase in capability of virtual meetings and web conferencing, and with increasing travel costs, sometimes “virtual” meeting are best.

Step 3: Measure How Well and How Certain: A quandary facing the government agency that sponsored this project was how to evaluate the project. The problem was that some of the targets were not reached (and no thresholds were set). So, on the face of it, the contractor had failed. But, they had advanced the sate-of-the-art significantly; just some of the targets were not realistic. To accommodate for this, the sponsor decided to treat the evaluations as qualitative and we made use of Belief Maps to help with the assessment.


A Belief Map is a simple grid on which an evaluator puts a single dot for each criterion to reflect how well the effort met the criterion versus how-certain they are of their assessment. In this project, all such assessments were made on paper, independently by each individual SEM, the sheets were collected and the data input into the Accord™ software for analysis. A sample sheet for one of the criteria is shown. It was blank (no dots on the Belief Map) when handed out. Here it is shown with the dots as submitted by seven SEMs.

The use of Belief Maps can help in managing:

• Review material is often incomplete, and may present evolving and uncertain information
• Reviewers have differing levels of understanding about material
• Some reviewers are well prepared, others are not
• Some reviewers have dominant personalities
• Some reviewers leave early or conduct other business

Step 4 Analyze Results to Guide Future Work: In just looking at the completed Belief Maps, much can be learned. If the reviewer’s dots are widely distributed vertically then there is poor agreement caused by some or all of the following:

1. The material presented to them is wanting
2. Progress relative to the measure is insufficient to judge consistently
3. The criteria is not defined clearly or measures multiple features

If many of the reviewer’s dots are to the left, then possibly:

1. They are not sufficiently expert to judge the material
2. The review material is not clear or the criterion poorly defined

For the project I facilitated, the reviewers were experts, the criteria clearly defined and the dots were well grouped vertically. Since they placed their dots on sheets independently, what we captured is a good indicator of the project’s quality. Horizontally, the dots ranged across the scale with some SEMs sufficiently expert to give confidence in the results.

Beyond what was done here, Accord can develop many useful statistics from the raw data. Further, it can fuse qualitative and quantitative evaluations and can collect data from a distributed team via the net.

Step 5 Finalize review: Finally, the material was documented for review by all concerned and use in future, similar projects. In this particular, case the final report was quite detailed due to the size of the investment and the use of a professional facilitator.

Summary
Hopefully, these five steps give you some good ideas for project evaluation. The five steps can make reviews useful exercises rather than a hoop to jump through. The use two-part targets and of Belief Maps can give a much richer window on the progress.

If I left something out or you have experiences that can be used to refine these steps, please share them.