I tend to be an optimistic person, as are most of the people that I’m around. I just generally find optimistic people to be more pleasant and ready for the next adventure, whatever it may be. One of the things my father always said was to why worry about what you can’t control, and in most cases, I agree with his philosophy. However, there is at least one area I’ve experienced in my life and especially in my career where a little pessimism is a good thing.
Making estimates. We all make several estimates a day, sometimes without even being aware of them. Can I stop the car in time at this speed? How much pasta should I prepare? How long will it take my daughters to get ready to leave? (Answers: Stopping – Yes, but not without making my wife nervous; Pasta – Always over-estimated; Daughters – mostly under-estimated.) I find that I am optimistic in my estimates and things usually take longer or involve more supplies than I originally envisioned. I want the task of painting the garage to go quickly, so I succumb to the Fog of Optimism for my estimate.
While I am very seldomly held accountable for these faulty estimates, the estimates that I make at work are a different story. Moreover, on the job, there are additional pressures to make optimistic estimates. Perhaps the boss is concerned that Parkinson’s Law will kick in and the work will expand to fill the available budget and schedule. Or there could be a mindset that optimistically estimating the project and having a shortened schedule will result in a sense of urgency that will prevent people from procrastinating on the important details while they work on the more interesting but less impactful items. The Sales organization may have information that the competition is quoting a lower non-recurring engineering charge for the project, and we will be less competitive on the bid with a less optimistic estimate of the work. Whatever the reasons, there usually is powerful pressure to present an optimistic estimate in the workplace.
In my own experience, I’ve seen two significant and related causes of the Fog of Optimism. The first is ignoring history. If it took over 10,000 hours to do the work for past projects, why would we now think that it will only take 5,000 hours this time? The fog of optimism tends to make us forget how those 10,000 hours were consumed. And it is likely that the estimates for those previous jobs were also in the 5,000 hour range, and we are just not learning from our experiences.
The second is that we don’t want to have our competence questioned. Of course, we aren’t going to make those same budget blowing mistakes again, are we? Having pessimism for our future performance can be a pretty uncomfortable position. Julie Norem, a psychology professor at Wellesley College said of pessimism: “The biggest negative impact potentially comes from other peoples’ reactions. If you’re doing it out loud, other people tend not to like it. They tend to have questions about your competence.” The perceived implication that we may not be the right person or team for the job can certainly bias the estimate towards the optimistic, especially as we are likely to weigh the immediate impact of the estimate over the long-term responsibility of the results.
Steve McConnell, in his book Software Estimation – Demystifying the Black Art, details the various methods for making better estimates. He quotes Fred Brooks in saying “It is very difficult to make a vigorous, plausible, and job-risking defense of an estimate that is derived by no quantitative method, supported by little data, and certified chiefly by the hunches of the managers.” By following a more formal estimation method, the influence of both pessimism and optimism can be reduced. Maybe the point I should be taking from my Dad’s philosophy is to focus more on the control part. But until then, having a little pessimism rather than optimism can be helpful to improve the accuracy of the estimates.
Stay tuned….