Estimate Confidence – A Demonstrative Example

"Each time we face our fear, we gain strength, courage, and confidence in the doing”

                                                                                                - Theodore Roosevelt

In my previous post, The Schedule – Improving Estimate Confidence, the conclusion likely leaves the impression that for any meaningful project, to get a reasonably high schedule confidence is impractical.  In fact, PERT teaches that you can get high confidence in the schedule with task confidence of only 50%.

It is true, as I said, that the probability of completing the project (a series of critical path tasks) on the expected date is determined by the product of the individual task priorities.  But this isn’t really meaningful.  For example, a series of 10 critical path tasks, each estimated for 10 days at a confidence level of 50% has a probability of less than 1 in 1,000 of completing in 100 days.  But our stakeholders are generally not that rigorous – we are usually allowed to complete within 5-10 days either way of that date for acceptable success, and this dramatically changes the results.

At a task confidence level of 50%, PERT assumes that there is an equal balance between tasks finishing early and tasks finishing late, which at project end will balance out the noise of individual task variance.

An example will better illustrate the situation.  As shown in the Table 1, here is a project with ten critical path tasks.  For simplicity, I’ve given them all the same estimates, but the estimate values aren’t relevant.

In Table 2, the Expected (statistical mean) and Variance values are plugged into a standard deviation z-table so I can show the duration ranges for selected probabilities of success for this hypothetical project.  StdDev is just the square root of Variance and is in the same units of measure as the estimates.  The appropriate multiplier is applied to StdDev and subtracted from (added to) Mean to get Low (High).  Range is the difference between Low and High.  I have taken liberties with rounding throughout this table.  This table would be read that for a 50% project probability of success (for the hypothetical project in Table 1), one would schedule for 109-121 days duration for the project, a range of 12 days.  If a sponsor needed a higher confidence level, they could get 90% confidence with 101-129 days (a range of 28 days).

There are a number of gleanings from this exercise:

• Specifying a fixed completion date sets you up for failure, with a less than 0.1% probability of delivering on that date with just 10 critical path tasks.
• If that is bad, Optimistic scheduling is a disaster.  If you planned to deliver in the optimistic duration above (50 days), your odds drop into the billionths.
• According to PERT, it is possible to deliver a project successfully within a statistically sound range that should be acceptable to most stakeholders.

But all is not champagne and roses – after all we are project managers and the gods do laugh at us.  So what can go wrong in this model?  Let’s review the validity of the assumptions:

• That there is a correlation between PERT and statistical analysis.  I know of no research that supports or confirms this assumption.
• The data points (project estimates) present as a Normal Distribution.  In fact, we know this is not true and that the values skew to the right (meaning that results are more likely to be late rather than early).
• To be statistically meaningful, you need a large number of data points.  Most projects will not have the volume of critical path tasks needed to validate this assumption.
• That results of one task are independent of any other task.  This is talking about statistical dependence rather than project task dependence.  But to be true would have to mean that variance (overage or underage) in one task does not influence the variance of any other task.  Earned Value Management (EVM) results would tend to invalidate this assumption, because EVM experience shows that variance early in a project is a high predictor of comparable variance later in the project.

To round out this discussion on estimate confidence, I will note that PMBoK and the Practice Standard for Project Estimating both consider PERT an analogous estimating technique (though I challenge that if applied at the task level by the task owner).  The only recognized bottom-up estimating technique is the Delphi Method.

Has this series on advanced scheduling techniques provided any insight on weaknesses in the techniques you use?

The Schedule – Improving Estimate Confidence

“My own idea is that these things are as piffle before the wind.

                                                                                                - Daisy Ashford

My previous post on the probability of schedule success, as pessimistic as it was, still left something hanging.  I talked about estimating an individual task to a 50% or 90% confidence level.  Is that just bunkum or is there a way to really get that kind of confidence?

Well, there really is, but a stated confidence level is just a piffle unless the techniques described in this post are used.  My post today is how to really get that level of confidence and how to recognize when it’s real.

The first step is to produce three estimates for each task.  This is not the same as Delphi estimating method, getting estimates from three experts.  Instead we’ll use the PERT three-point estimate, as mentioned in The Schedule – Estimate Task Effort.  For this exercise, the task estimator produces three independent estimates:  an optimistic estimate (O), a pessimistic estimate (P), and a most likely estimate (L).  The expected time is then calculated as E = (O + 4L +P) / 6.  So, if the optimistic estimate is 3 days, the likely estimate is 5 days, and the pessimistic estimate is 13 days, the expected time is (3 + (4 * 5) + 13) / 6, or 36/6, or 6 days.

Next, calculate the standard deviation using the formula SD = (P – O) / 6, or 1.67.  From these two values, we can now produce an estimate to the desired confidence level.

So, the 50% confidence level is determined using the expected time calculation above, or 6 days in this example.

The 84% confidence level is one standard deviation (4.33 – 7.67 days, or, rounding up, 5-8 days).  A confidence level of 98% is achieved by going out two standard deviations, or 3-10 days.  You can even get to a better than 99% confidence level by going out three standard deviations (1 – 11 days).

However, this only gets the individual tasks up to the target probability of success.  As noted in my previous post, the project’s probability of success is determined by multiplying the probabilities of the critical path tasks.  If you have ten critical-path tasks in your project and you estimate each task to a 95% confidence level (two standard deviations), your project still only has about a 62% probability of success.

To get the project to a 90% probability of success, you have to get each task up to a 99.7% probability of success (three standard deviations).  (This will actually get this example project up to a 97% probability of success.)

In my next post, I’ll provide some example numbers to demonstrate just how significant this is.  Which will also show why we as PMs woefully underestimate our projects, why we have such a dismally Chaotic success record, and why attempting to fix it will receive push back from our stakeholders.

CAVEAT:  This math works for standard distributions (you know, those bell curves).  Task estimating is not a standard distribution; the curve is compressed on the left and stretched on the right.  There is a formula/ process for converting a non-standard distribution to a standard distribution, but, even without that, what I’ve described above is still a vast improvement over the typical schedule twaddle.

How often do your projects complete on schedule.

The Schedule – Probability of Success

"Try not to become a man of success, but rather try to become a man of value.”

                                                                                                                - Albert Einstein

I completed the series on the well-formed schedule, but the final post, on adding risk buffers, was pretty light.  I want to take a couple of posts and expand on the topic of risk-based or risk-adjusted scheduling, which also will start some interesting metrics discussion.

How tough is the job of project management?  Planning a new project, the PM prepares a schedule that documents months in advance a completion date.  What is that PM’s probability of success?

It turns out that this is a fairly easy mathematical calculation.  If you have all of the tasks estimated and know the probability of success for each individual task, then you can calculate the project’s probability of success – the probability of completing the project on schedule.  You merely multiply the probabilities of success for each task (in the critical path) together.

For a simple example, if you have a 10-task project (each task in the critical path) and you’ve estimated each task with a 50% likelihood of success (i.e., equally likely to finish early or late), then the likelihood of completing the project on time is 0.50 x 0.50 x 0.50 x … or 0.50¹⁰ or 0.09.8%.  With a very simple project, you have less than 1 in 1,000 likelihood of completing your project on schedule.

Even if you upgrade the individual tasks to a 90% probability estimate, you only have a 35% chance of success (1in 3) in this example.  And with each task you add, the probability of success drops proportionately.  To emphasize the point:  even when you use aggressively conservative scheduling, you will miss the date more often than you hit it.

In other words, we’re screwed before we start.

This sounds very pessimistic.  What are the best practices that can improve this performance?  I explore that in my next post.  Stay tuned…

How do you generate estimates on individual tasks?  What is your expectation for the task success?

The Project Manager’s Cycle – Redux

“What does a project manager do?  What value do I add to your project?  It’s true that I don’t contribute directly to the product or service for which the project exists.  However, if I’m doing my job well, then I add significantly to the productivity and value of the team by improving communications, eliciting priorities, and focusing the team on the key deliverables.  Further, I am your agent, seeing that we perform to plan and communicating back when we diverge from it.”

Note:  I originally wrote this in April/May 2011 and it was intended to be the final post in The Project Manager's Cycle series, but on reviewing my archives today I realized that I never published it.  My apologies... and here it (finally) is.

The Project Manager’s Cycle includes these activities:

• Validate the metrics
• Validate task status
• Incorporate approved change control
• Reschedule/Replan
• Check variances and progress/productivity
• Communicate planned/expected resource utilization to resource managers
• Develop earned value reports
• Review risks and mitigation actions
• Review issues and action items
• Review commitments
• Conduct client/sponsor status meeting
• Conduct project team meeting
• Publish the project status report

A baker’s dozen posts on the core project management elements of monitoring and controlling a project.  This series on The Project Manager’s Cycle been a pleasure to write and has also been beneficial for me.  I developed the outline of the series several years ago, but this journey required that I really flesh out the details, relationships, and components.  I appreciate all of my reader (yes, that would be you) who stayed with me through this.

A picture is worth a few words, it’s said, so I’ve tried to rough out the series in a diagram.  If I ever figure out how to make it available as a download, I’ll enable that.  In the meantime, leave a post or drop me an email and I’ll reply with a full-size PDF of version 1.0 of the diagram.  Next time I’m feeling ambitious, I’ll add the inputs and outputs, though I’m concerned that will make the diagram too busy.

We stepped on the ferris wheel together in January to start this ride.  We’ve gone around the cycle and now it’s time for us to step off.

What of this series has been the most beneficial to you?  Where have I not been clear and need to expand or improve the discussion?

The Schedule – Risk Buffers

“The first step in the risk management process is to acknowledge the reality of risk.  Denial is a common tactic that substitutes deliberate ignorance for thoughtful planning.”

                                                                                                                - Charles Tremper

With this post, we conclude the series on developing the well-formed schedule.  We have taken the WBS, which defines the project scope, added the task dependencies, estimated the effort for each task, assigned resources to each task, and resource-leveled the task assignments.  The final step to complete the well-formed schedule is to add risk buffers so that the schedule has the intended probability of success.

A schedule without risk buffers has virtually no probability of success.  Since you probably want to present a schedule with some probability of success, it obviously follows that you must add some risk buffers.  So now you probably want me to tell you how many and where.

I’ll briefly identify two common practices for identifying risk buffers.  The first is built from the standard risk management process.  The PMI PMBoK risk management process includes the step “Quantify risks.”  You’ve probably wondered what this means.  For risks that should be mitigated, determine the probability of occurrence (a percentage) and the time and cost that must be added to the project if the risk event occurs.  Let’s say for a specific risk you determine there’s a 25% probability that it will occur and that if it occurs it will add 12 weeks (I’m just going to deal with time in this example, but you can extrapolate for costs) to the schedule.  The 25% probability times the 12 weeks is 3 weeks, meaning that you should add a risk buffer of three weeks to your schedule.

Depending on a variety of factors, you can either add all three weeks to the end of the schedule or spread it in appropriate points over the life of the project.

You repeat this for all significant project risks.

A very common alternative to this approach is to use the Critical Chain approach developed and promoted by Elihayu M. Goldratt.  It uses a formulaic approach to determine the appropriate amount and placement of the risk buffers.

This overview is obviously too high a level for practical application.  But with this post I’ve finally presented enough of the PM foundation so that I can introduce some of the more interesting and complex concepts.  I’ll do this starting with a couple of posts on advanced quantification of the schedules, then maybe transition to more fully explaining the development of risk buffers.

It’s taken me two years of posts to get to “the fun stuff.”  What area of project management best practice would you like me to tackle?

The Schedule – Resource Leveling

“Happy people plan actions, they don’t plan results.”

                                                                                                                - Dennis Wholey

We are nearing the end of the series on scheduling basics.  We have taken the WBS, added the dependency chain, estimated the effort and assigned resources.  The next step to completing the well-formed schedule is to level the resource assignments.  This is done to:

• Prevent over allocation of a resource within the project
• Prevent over allocation of a resource across enterprise assignments
• Accommodate non-productive time and absences

First, let’s assume the project is in an organization following best practices.  In that case, all team members record all of their time each cycle to specific project tasks and to operations, administrative, absence, and other non-project activities.  Your project management software integrates with these time records (or time is entered directly into the software, such as Sharepoint wih MS Project EPM, Clarity or Primavera).  In this case, you already have all of the elements in place to just “push the button” on your project management software to resource level the schedule.  You will then want to review and fine tune the results.

On the other hand, if you do not have the luxury of these conditions, you have more work.  For example, you will have to tune and adjust using heuristics to get the right productivity factor so that you don’t over allocate resources within the project;  and it will be much harder to track other assignments and activities to prevent over-allocation across enterprise activities.

In either case, this step will be repeated regularly over the life of the project.  Rescheduling and replanning are regular activities in the Project Manager’s Cycle.

In my next post, we’ll conclude this series of posts on the basics of creating the well-formed schedule by discussing risk buffers.

If you are not in a “best practice” environment, what steps do you follow to produce a resource-leveled schedule?

The Schedule – Resource Assignments

“Time is the scarcest resource and unless it is managed nothing else can be managed.”

                                                                                                                - Peter Drucker

We are now half way through the steps for creating a well-formed schedule.  In my previous post in this series, we added task effort estimates, but we have not yet added resource assignments, which we will now do.

Why add task estimates without considering the resource assignments and then immediately follow that step with the step that adds the resource assignments and then re-estimate each task specifically for the resource?  Let’s consider three scenarios:

For a small, routine project in a fully staffed environment, assigning resources may be a mechanical exercise (you’ve probably done this same type of project so often, the WBS was built practically from a template and you know exactly who is doing each task).  I’ll come back to this first scenario later.

A second scenario is more common:  It is a large project, but some of the resources are known (or the resource skill is known and easily acquired).  However, for some tasks, there may be choices or the resource may not be known until late in the planning cycle.

A third scenario could be a large project, but it is one where the resources are definitely not known and there could still be many negotiations about staffing options.  In fact, the budget, schedule and scope could still be under negotiation.  But a time and cost schedule is needed to get approval to acquire the resources.

For a best practice recommendation, it must be general to all possible cases.  Since it isn’t always possible to know the resources and sometimes it is necessary to have estimates without resources, the general case is to first build the schedule with estimates but not resources.  Then when appropriate, add the resources and re-estimate.

For the first scenario, you certainly can combine these two steps.  Just understand that you are taking a short cut and that in other circumstances or a different environment you will want to follow the best practice.

Getting back to this step, then, you have a partially built schedule with dependencies and estimates, but no resources (or possibly generic resources).  So start assigning resources to tasks (or substituting resources for generic resources), ignoring, for now, resource conflicts and over-allocations.  For each resource assignment, review the task (inputs, outputs, work products, completion conditions, etc.) with the resource and have them re-estimate the task based on their understanding.  Review with the resource any task where there is a significant difference between the original estimate and this new estimate to understand why.  Update the task estimates with these revised values.

Don’t be surprised to find that the new estimate from this step is significantly different from the previous estimate and that you may need to renegotiate budget and scope expectations.  You may even find that as you complete some resource assignments, it forces changes to other previous or planned assignments.  And be prepared for even more variance in the next steps.  That’s why this is a recursive process – it seems like you just keep starting over from the beginning.

But actually, we’re up to Resource Leveling, the next to last step in building the well-formed schedule.  This is the subject of my next post.

Do you have resources estimate their own tasks?