The Overconfidence Test

Following on from a previous look at base rates and Bayes’ theorem, I have been considering the topic of overconfidence. All of us make decisions of varying magnitudes on a regular basis, in the face of certainty and uncertainty alike. Due to the tendency of things to err on the side of the latter (especially when it comes to investing!), the use of judgement and estimates becomes an unavoidable aspect of our decision making. Should we leave this process unchecked? Or can improvements be made?

The Problem

Confidence is a powerful force indeed. It can get you through when the going gets tough but at the same time we know what they say about too much of a good thing. There is a difference between justified belief and wishful thinking and, if left unchecked, overconfidence can spell disaster for your portfolio. Having made the mistake myself, I understand the consequences all too well and, worst of all, know how hard it can be to recognise the symptoms in advance.

Hard, but not impossible. Well, the first step to recovery is admitting you have a problem so perhaps that is the place to start. For this purpose, I have borrowed a simple test format first used by Edward Russo and Paul Schoemaker in their 1989 work ‘Decision Traps’. I have seen the test used in a few different places (notably in an excellent paper called The Seven Sins of Fund Management that I would recommend to all), although I have made up my own set of questions in order to first perform a fresh test on myself.

The Test

There are 10 questions, all with numerical answers and with no research necessary. Instead, you are required to estimate the lower and upper bound of your 90% confidence interval for each one:

1. What is the distance between the earth and the moon in kilometres?
2. What is the speed of sound through water in metres per second?
3. In what year was the Louisiana Purchase completed during Thomas Jefferson’s presidency?
4. If spun out to form a single thread, what would be the length of the entire human circulatory system in kilometres?
5. What is the top recorded speed of a pigeon in flight in miles per hour?
6. What has been the annualised return of the S&P 500 since 1970 with dividends reinvested?
7. What is the population of Thailand?
8. At what temperature does iron melt?
9. How many unique chess positions can be reached after 4 moves (by both white and black)?
10. What is the height of Mt. Fuji in feet?

The answers can be found at the bottom of the article.

The Results

How did you do? Remember that the purpose of the test is to state a 90% confidence interval for the questions, not to find the correct answer. Therefore, if you are well calibrated, you will always score 9 out of 10 on such a test, regardless of the questions asked. If you were genuinely clueless as to an answer, then you should have compensated with a wider confidence interval. Sadly, if you did not score 9 out of 10, then it means that your feel for confidence intervals is not properly calibrated.

Russo and Schoemaker performed this test on over 1,000 participants and found that less than 1% actually scored 9 or above.  Most likely, you also fell victim to at least a couple of questions and the implications of this are clear – when we make estimates relating to our investments, we are almost certainly making the exact same mistake.

Given that I have done tests such as this one before, and that I came up with the questions, I rather hoped I would fare quite well. Unfortunately I only actually got 6 right and so clearly I’ve some room for improvement myself…overconfidence really is persistent.

Why Does This Happen?

I suspect this is something we must grapple with ourselves as we come to better understand what motivates us and the decisions we make. For me at least, there seems to be a combination of factors that lead to this errant behaviour. To a degree, it comes from the desire to be knowledgeable and to be perceived as knowledgeable by myself and others. If I got everything right by simply stating a range of 0 to infinity then that would feel like cheating! In other words, it feels better to be precisely right than roughly right, but as Buffett quips, it’s better to be roughly right than precisely wrong.

Another factor might be a misconception about the world which leads us to give intervals that are centred incorrectly. Take the question on the population of Thailand, for example. I got that one wrong and estimated what I thought to be a wide range of 10 million to 60 million. The basis for this decision was that I thought it wasn’t small, but that it definitely wasn’t as large as the UK, and so I went for a range that covered most possibilities below that boundary. It turns out that I have no idea what I’m talking about when it comes to Thailand – it has a surface area over twice that of the UK and the population is a good few million larger too.

A final cause that comes to mind for now is a lack of practice in accounting for uncertainty. I have had no formal training on the subject, and have not put myself through processes such as the one above often enough to develop a knack for it. We all make the mistake of overconfidence, but by the time an investment goes wrong it can be hard to pin down precisely where the error was made and so we don’t learn from it.

In the Seven Sins of Fund Management mentioned above, the brilliant example of comparing the estimates of weather forecasters with those of doctors is used. Weather forecasters make predictions often, have frequent feedback on their accuracy, and as a result demonstrate good confidence interval calibration. Doctors, on the other hand, don’t have the same frequent feedback regarding the accuracy of their estimates and so tend to show poor calibration. As you may have guessed, the evidence suggests that we investors behave more like doctors than weathermen, at least when it comes to making estimates.

The Solution

How can we resolve this issue? In theory, just knowing that this is a problem should be enough to set us on the right track, yet my poor test score suggests that it might not be that easy. It seems continued vigilance is the order of the day, at least on my part.

One way to help eliminate this problem may be to automate the decision making process.  Whether choosing to follow a qualitative checklist or deciding to pursue a systematic quantitative methodology, it is likely that this kind of tool can help you to reduce dependence on wayward estimates. Not only will this improve results, but it has the added bonus of being incredibly straight forward to implement.

Of course, putting such a plan into action is one thing, but sticking to it over the long years ahead is another matter. Perhaps ironically, some confidence is necessary to do that yet we must always strive to remain aware of the risk that complacency brings. Just because a systematic strategy has worked in the past, it does not mean it will work forever and so we should regularly revisit our assumptions and reasoning lest we fall victim to the very trap we are trying to avoid.

If you deem an estimate absolutely necessary, then it is hard to advise a solution other than extreme caution and diligence when performing your work. While estimating the lower and upper bounds for a growth rate or a PE multiple, for example, make sure to select a range that is appropriately wide and honest. It is very hard to do this consistently and I am increasingly sceptical of such processes, but it is vital if you want to produce useful valuations.

If, as you (hopefully) realised from the questions above, you come to an estimate that requires a very large range for you to be certain, then it’s probably best to just let that one go. Charlie Munger has often said that there are only 3 kinds of investments decisions; yes, no and too difficult – and 98% is too difficult! We would do well to remember that…

Concluding Thoughts

Overconfidence is something that I know I have been guilty of in the past and I do not want to make the same mistakes again. It can be difficult to admit ignorance, especially if you have already feigned competence, so learning when to say you’re wrong and when to recognise that you don’t know doesn’t always come naturally. The skills can (and should) be learnt however, and they may prove indispensable to you throughout your investing career. In summary: 

• Overconfidence is something that plagues us all – it is a mistake you are very likely to make when using estimates in your investment process and it could cost you dearly.
• There appear to be a number of causes but they may stem from character, misconceptions or a lack of experience in estimation.
• Luckily this can be remedied in a number of ways from automation to simple prudence.

Test answers:

384,400 km, 1,484 mps, 1803, 100,000 km, 90 mph, 10.19%, 67 million, 1,538 ºC, 72,078, 12,388 ft.

All have been sourced from the top couple of google links so apologies if there are any inaccuracies. Search your own answers if you wish but any differences should have a negligible impact on the test outcome.

Hope you enjoyed the article, if so then please share it with your network and feel free to leave any comments or questions below.