Estimate Confidence – A Demonstrative Example

"Each time we face our fear, we gain strength, courage, and confidence in the doing”

                                                                                                - Theodore Roosevelt

In my previous post, The Schedule – Improving Estimate Confidence, the conclusion likely leaves the impression that for any meaningful project, to get a reasonably high schedule confidence is impractical.  In fact, PERT teaches that you can get high confidence in the schedule with task confidence of only 50%.

It is true, as I said, that the probability of completing the project (a series of critical path tasks) on the expected date is determined by the product of the individual task priorities.  But this isn’t really meaningful.  For example, a series of 10 critical path tasks, each estimated for 10 days at a confidence level of 50% has a probability of less than 1 in 1,000 of completing in 100 days.  But our stakeholders are generally not that rigorous – we are usually allowed to complete within 5-10 days either way of that date for acceptable success, and this dramatically changes the results.

At a task confidence level of 50%, PERT assumes that there is an equal balance between tasks finishing early and tasks finishing late, which at project end will balance out the noise of individual task variance.

An example will better illustrate the situation.  As shown in the Table 1, here is a project with ten critical path tasks.  For simplicity, I’ve given them all the same estimates, but the estimate values aren’t relevant.

In Table 2, the Expected (statistical mean) and Variance values are plugged into a standard deviation z-table so I can show the duration ranges for selected probabilities of success for this hypothetical project.  StdDev is just the square root of Variance and is in the same units of measure as the estimates.  The appropriate multiplier is applied to StdDev and subtracted from (added to) Mean to get Low (High).  Range is the difference between Low and High.  I have taken liberties with rounding throughout this table.  This table would be read that for a 50% project probability of success (for the hypothetical project in Table 1), one would schedule for 109-121 days duration for the project, a range of 12 days.  If a sponsor needed a higher confidence level, they could get 90% confidence with 101-129 days (a range of 28 days).

There are a number of gleanings from this exercise:

• Specifying a fixed completion date sets you up for failure, with a less than 0.1% probability of delivering on that date with just 10 critical path tasks.
• If that is bad, Optimistic scheduling is a disaster.  If you planned to deliver in the optimistic duration above (50 days), your odds drop into the billionths.
• According to PERT, it is possible to deliver a project successfully within a statistically sound range that should be acceptable to most stakeholders.

But all is not champagne and roses – after all we are project managers and the gods do laugh at us.  So what can go wrong in this model?  Let’s review the validity of the assumptions:

• That there is a correlation between PERT and statistical analysis.  I know of no research that supports or confirms this assumption.
• The data points (project estimates) present as a Normal Distribution.  In fact, we know this is not true and that the values skew to the right (meaning that results are more likely to be late rather than early).
• To be statistically meaningful, you need a large number of data points.  Most projects will not have the volume of critical path tasks needed to validate this assumption.
• That results of one task are independent of any other task.  This is talking about statistical dependence rather than project task dependence.  But to be true would have to mean that variance (overage or underage) in one task does not influence the variance of any other task.  Earned Value Management (EVM) results would tend to invalidate this assumption, because EVM experience shows that variance early in a project is a high predictor of comparable variance later in the project.

To round out this discussion on estimate confidence, I will note that PMBoK and the Practice Standard for Project Estimating both consider PERT an analogous estimating technique (though I challenge that if applied at the task level by the task owner).  The only recognized bottom-up estimating technique is the Delphi Method.

Has this series on advanced scheduling techniques provided any insight on weaknesses in the techniques you use?

The Schedule – Improving Estimate Confidence

“My own idea is that these things are as piffle before the wind.

                                                                                                - Daisy Ashford

My previous post on the probability of schedule success, as pessimistic as it was, still left something hanging.  I talked about estimating an individual task to a 50% or 90% confidence level.  Is that just bunkum or is there a way to really get that kind of confidence?

Well, there really is, but a stated confidence level is just a piffle unless the techniques described in this post are used.  My post today is how to really get that level of confidence and how to recognize when it’s real.

The first step is to produce three estimates for each task.  This is not the same as Delphi estimating method, getting estimates from three experts.  Instead we’ll use the PERT three-point estimate, as mentioned in The Schedule – Estimate Task Effort.  For this exercise, the task estimator produces three independent estimates:  an optimistic estimate (O), a pessimistic estimate (P), and a most likely estimate (L).  The expected time is then calculated as E = (O + 4L +P) / 6.  So, if the optimistic estimate is 3 days, the likely estimate is 5 days, and the pessimistic estimate is 13 days, the expected time is (3 + (4 * 5) + 13) / 6, or 36/6, or 6 days.

Next, calculate the standard deviation using the formula SD = (P – O) / 6, or 1.67.  From these two values, we can now produce an estimate to the desired confidence level.

So, the 50% confidence level is determined using the expected time calculation above, or 6 days in this example.

The 84% confidence level is one standard deviation (4.33 – 7.67 days, or, rounding up, 5-8 days).  A confidence level of 98% is achieved by going out two standard deviations, or 3-10 days.  You can even get to a better than 99% confidence level by going out three standard deviations (1 – 11 days).

However, this only gets the individual tasks up to the target probability of success.  As noted in my previous post, the project’s probability of success is determined by multiplying the probabilities of the critical path tasks.  If you have ten critical-path tasks in your project and you estimate each task to a 95% confidence level (two standard deviations), your project still only has about a 62% probability of success.

To get the project to a 90% probability of success, you have to get each task up to a 99.7% probability of success (three standard deviations).  (This will actually get this example project up to a 97% probability of success.)

In my next post, I’ll provide some example numbers to demonstrate just how significant this is.  Which will also show why we as PMs woefully underestimate our projects, why we have such a dismally Chaotic success record, and why attempting to fix it will receive push back from our stakeholders.

CAVEAT:  This math works for standard distributions (you know, those bell curves).  Task estimating is not a standard distribution; the curve is compressed on the left and stretched on the right.  There is a formula/ process for converting a non-standard distribution to a standard distribution, but, even without that, what I’ve described above is still a vast improvement over the typical schedule twaddle.

How often do your projects complete on schedule.