The Overconfidence Test

8th Oct '15 by Tom Firth

4 comments 10359 reads

Following on from a previous look at base rates and Bayes’ theorem, I have been considering the topic of overconfidence. All of us make decisions of varying magnitudes on a regular basis, in the face of certainty and uncertainty alike. Due to the tendency of things to err on the side of the latter (especially when it comes to investing!), the use of judgement and estimates becomes an unavoidable aspect of our decision making. Should we leave this process unchecked? Or can improvements be made?

The Problem

Confidence is a powerful force indeed. It can get you through when the going gets tough but at the same time we know what they say about too much of a good thing. There is a difference between justified belief and wishful thinking and, if left unchecked, overconfidence can spell disaster for your portfolio. Having made the mistake myself, I understand the consequences all too well and, worst of all, know how hard it can be to recognise the symptoms in advance.

Hard, but not impossible. Well, the first step to recovery is admitting you have a problem so perhaps that is the place to start. For this purpose, I have borrowed a simple test format first used by Edward Russo and Paul Schoemaker in their 1989 work ‘Decision Traps’. I have seen the test used in a few different places (notably in an excellent paper called The Seven Sins of Fund Management that I would recommend to all), although I have made up my own set of questions in order to first perform a fresh test on myself.

The Test

There are 10 questions, all with numerical answers and with no research necessary. Instead, you are required to estimate the lower and upper bound of your 90% confidence interval for each one:

What is the distance between the earth and the moon in kilometres?
What is the speed of sound through water in metres per second?
In what year was the Louisiana Purchase completed during Thomas Jefferson’s presidency?
If spun out to form a single thread, what would be the length of the entire human circulatory system in kilometres?
What is the top recorded speed of a pigeon in flight in miles per hour?
What has been the annualised return of the S&P 500 since 1970 with dividends reinvested?
What is the population of Thailand?
At what temperature does iron melt?
How many unique…

Unlock with Email

Unlock with Google

Already have an account?
Login here

PhilipHanson

8th Oct '15

I read about this test in Nassim Nicholas Taleb's book, The Black Swan. I just took the test being careful to expand much more widely my range to ensure I really was 90% confident about each answer.

I got 4/10 right.

This is really powerful stuff.

Tom Firth

Thanks for posting your results - I was hoping some people might. It really is inexplicably hard...I've tried it out on a few people I know as well and no one has passed yet. It's a very small sample size of course but Russo and Schoemaker found that less than 1 in 100 people complete the task succesfully and I can really believe that!

Powerful stuff indeed, and a little disconcerting! If we can't even get this right, then our hopes of correctly estimating reasonable ranges for forecasts are pretty bleak... The hardest thing of all though is to actually take action on what we learn about ourselves through experiments like this one. Most people just note the result as a curiosity and then go back to business as they always have done.

Tom

Damian Cannon

It's a good test Tom. I managed 8/10 by being conservative but then I got over-confident with the last question! Just goes to show how our brains work against us sometimes.

Anyway I very much agree that a checklist is a powerful tool for avoiding our internal bias. I've developed one this year and it's definitely saved me from buying shares which really aren't appropriate for me.

Damian

Not bad! 8 is the best score from my small sample yet. To be fair, I think the test format doesn't do quite what it is intended to because it favours somebody who does happen to know a few of the facts...then again, believing you know the fact wrongly is precisely what can lead you astray in this one.

I tried to make them obscure and varied for this reason but I can't seem think of a superior way of guaging something like this quickly. The obvious one is to get people to make predictions about the future but that takes a little longer to do.

All the best with the checklists, I'm a big fan too!

Tom

The Overconfidence Test

The Problem

The Test

Unlock the rest of this article with a 14 day trial

About the Author

Tom Firth

4 comments

Unlock the metrics for all these stocks and more...

Unlock the metrics for all these stocks and more...