Base Rates And Bayes’ Theorem

21st Sep '15 by Tom Firth

16 comments 17650 reads

The base rate fallacy and its impact on decision making was first popularised by Amos Tversky and Daniel Kahneman in the early 1970’s. In short, it describes the tendency of people to focus on case specific information and to ignore broader base rate information when making decisions involving probabilities. If we look at the investment process through this probabilistic lens, what can consideration of base rates and Bayes’ theorem offer us?

Base Rates

These are most easily described and understood with an example, which I have shamelessly sourced from Wikipedia. Example 1 given on the Wikipedia page is clear and easy to picture. You are told that “John is a man who wears gothic inspired clothing, has long black hair, and listens to death metal.” You are then asked “How likely is it that he is a Christian, and how likely is it that he is a Satanist?”

This is where we find out that our minds are poorly primed to deal intuitively with probabilistic reasoning. The description of John practically has the word Satanist on the tip of our tongues, and when the question comes, we are all too eager to declare that he is much more likely to be a Satanist than a Christian.

This is the base rate fallacy in a nutshell. Despite John’s appearance increasing the probability that he considers himself a Satanist, the fact is that there are around 2 billion Christians in the world and very few Satanists. This means that the odds are still overwhelmingly in favour of John being a Christian.

Bayes’ Theorem

Bayes’ theorem was developed by Rev. Thomas Bayes and was first published in 1763, 2 years after his death. The theorem concerns the incorporation of new information into old, in order to accurately determine the revised probability of an event in light of the new information. As with the base rate fallacy, this process is best outlined with an example, for which I will use example 2 on the same Wikipedia page linked above.

The scenario looks at a driver being stopped and breathalysed and aims to calculate the probability that a driver who fails the test is actually over the limit. We are told that if a person is actually drunk, the test will indicate so 100% of the time but, in addition to this, 5% of people tested will display a false positive – the test says they are drunk when they…

Unlock with Email

Unlock with Google

Already have an account?
Login here

pka

22nd Sep '15

Tom, thanks for an interesting and useful article. I found it a bit confusing when I first read it, because I had wrongly assumed from the title that it is about the Bank of England's base rate, but of course it is nothing to do with that!

Tom Firth

Thanks - my apologies for the confusion! I chose the title because the dash of alliteration made it sound punchy (at least in my mind...). In retrospect perhaps I should have opted for plain old clarity instead.

Have a good evening,

Tom

Mechanical Bull

Very interesting read. I am familiar with Bayes theorem and I am a big fan of StockRanks but I hadn't made the connection. Now you have pointed it out it it seems blindingly obvious!

23rd Sep '15

Tom, I think your article is excellent, but it's use of the mathematical term Bayes Theorem might frighten a lot of people who are not mathematicians. I think you could express the same ideas using the less daunting term 'conditional probability'.

Cuthbert C

A really excellent and thought provoking piece, thank you.

I'm read Kahneman so have already grappled with Bayes Theorem and found it fascinating to see how absolutely counter intuitive the outcomes are when it's applied to apparently simple problems.

There is an old rubric to the effect that it is more important to invest in the right sector than it is to invest in the right stock - and actually that is really a restatement of Bayesian thinking. (The right sector is the one with the most favourable base rate. ) So the learning I take from that is to spend more time choosing sectors than identifying individual stocks. In fact it might be sensible to buy baskets of stocks in the chosen sector rather than just one or two. If house building is the place to be then it's more important to capture the sectoral gains than it is to agonise about which individual stock is best.

Which might also strengthen the case for IT's or OEICs or ETF's which provide broad coverage of target sectors.

Thanks again

iwright7

24th Sep '15

Tom,

Yes great article. I have been listening to an excellent audiobook in the car (also available as a book) called, "The Drunkard's Walk: How Randomness Rules" by Prof L. Mlodinow . In fact it is the opposite of drunken rationale and takes you though a history of the development of randomness theory and the need for the evolutionary human brain to look for cause and effect patterns that are either not there, or that we misinterpret.

Quite a few of his examples relate to gambling, but they could equally as well be attributed to our "investment" decisions. An example is scrutiny (and subsequent demolition) of Fortune 500 companies who hire or fire their CEO's for what turns out to be random short term financial success of failure.

Does make me think that I am not quite so good a stock picker after all and that Stockrank factors which remove my stock picker logic should be given more prominence. Better still when my logic and high Stockrank numbers happen to coincide, or is this just another random event? Ian

P.S. Amazon through www.audible.co.uk have a good selection of investment audiobooks for instant download to a smartphone - Great for listening to in the car on a long journey. I also recommend: Reminisences of a Stockmarket Trader, One up on Wall St and Where are the Customers Yachts, in particular

cig

Where do you stop with this line of thinking though? Some people do say broad asset class choices matter more than sector picks... The problem is the broader the asset the more efficient the market and the harder it is to do selection... or should we all trade currencies?

A good stock picker may be better off shorting their sectors to get the relative perf of their stock picks if they want to avoid base risk.

Hi Ian,

Thanks for the book recommendation, had a quick look on Amazon and it looks like an interesting read. Not a bad shout to get it as an audio book too - I spend a lot of time reading (too much according to some) and have been looking around for material to listen to while I run etc.

Always good to question your own stock picking skills in my view. At the very least, how else could you improve them but through rigorous and regular assessment? And if you do discover that ignorance runs a little deeper than you hoped, well, then there's a hedge for that by the name of diversification. No shame in hedging your bets, it just helps to take the pressure off your own analysis after all. Ultimately, most of us are in this game to protect and grow our capital...not to convince ourselves and others that we're great stock pickers! As far as I'm concerned, whatever works, works.

Good luck with your investing,

Tom

Thanks for the feedback - I quite enjoyed writing this one. I very recently started Kahneman's book myself (after it sitting in the ever growing 'to read' pile for months) and as you say he covers Bayes' Theorem well. I'm only about half way through but his thinking on the subject is great and has added some clarity to my own ideas about this particular tendency affects the investment process - hence the article!

Interesting what you say about picking sectors, it makes sense in the Bayesian context and the house builders you mention are quite a good example. Pretty much any house builder you bought a few years ago would have done extremely well and if you knew the sector was undervalued, you could have saved yourself a lot of effort by just buying a basket of them.

If you will allow me to play Devil's advocate for a minute though, how would you say that picking sectors is different from picking stocks? Is it easier? If so, why? The evidence would suggest that experts and amateurs alike are poor forecasters whether it comes to company earnings or macro events - it seems the future just isn't all that clear, whatever the scale!

If I was to employ such a strategy, my worry would be that I've essentially replaced one forecasting problem (the stock picking problem) with another almost identical forecasting problem (the sector picking problem). Why would I be more likely to get it right just because I'm analysing a different aspect of the future?

Hope that makes sense. I'm not saying I disagree, I'm just curious as to how you (or anyone else?) might address those concerns.

Thanks,

Tom

25th Sep '15

Tom,

In relation to stockpicking I am reminded of the book, "Simple, But Not Easy" - Stockpicking is simple but its not easy to be successful. When I started more serious investing I spent a lot of time reading over 50 books and looking for web based information that would give me an edge over the market. I came across the US Guru screens on AAII whose performance data goes back 10 years or more: http://www.aaii.com/stock-screens?a=menubarHome - Click on the different year tags for % gain rankings.

It is remarkable just how many of these US "Guru" screen selections have beaten the US market, without direct human intervention. I concluded that what was needed was a historically successful set (or sets) of screening criteria and an investment approach that suits your personality so you stick with it. It is turning out to be the same market beating success story in the UK with many of the Stocko Guru and Stockrank screen selections to date.

That's not to say that I don't pick shares too because that is part of the fun of investing, but picking them from a pre-selection of shares that meet your criteria, does give an added confidence factor. In fact at the moment I have a stockpicked quality/momentum type portfolio and a more recently a rules based high Stockrank portfolio to see what happens. So stockpicking for me its understanding that I have all the human bias's and need all the help I can get! Ian

I've just finished reading the book 'How to make a million - slowly' by Lord John Lee, who has been an extremely successful private investor over many years. This is illustrated by the fact that he was one of the first investors in the UK to have an ISA portfolio worth a million pounds.

Although John Lee obviously has great skill as a stock-picker, I think it is very interesting [in the light of this excellent article by Tom Firth on Bayes Theorem and conditional probability] how John Lee has increased the odds of long-term success by the rules he uses to reduce the size of the pool of stocks that he picks from.

The rules that John Lee uses, according to his book, include the following [I assume he won't mind me summarising them here,as this is likely to increase sales of his book]:

- He tends to buy stocks of small, rather than big, companies. [Small companies tend to perform better over the long-run than larger ones, although that is not the case in every year.]

- He tries to buy stocks that are on modest valuations, which he defines as stocks that have an attractive yield and a low price earnings ratio and /or a discount to net asset value / real worth. [It is well known that 'value' stocks and stocks with high dividend yields tend as a group to out-perform over the long-run.]

- He prefers to hold stocks for many years, rather than regularly 'churning' his portfolio, and he lets profitable holdings run. [This greatly reduces his transaction costs, and transaction costs act like a tax on performance, so I think this is likely to improve his long-term results.]

- He looks for established companies with a record of profitability and dividend payments. He avoids start-ups and biotech or exploration stocks. [Of course, some start-ups, biotechs and exploration stocks go onto doing extremely well, but the odds of selecting those in advance are small; by excluding such companies I think he improves his probability of out-performing the stock market as a whole.]

- He looks for moderately optimistic or better chairman's / CEO's most recent comments. [I think this reduces the probability of him selecting a stock that will perform badly in the short-term.]

- He prefers conservative, cash-rich companies or those with low levels of debt. [This must greatly reduce the probability of any companies in his portfolio going bankrupt. Also I think the stocks of such companies would tend to be less volatile than those of highly-indebted ones, and it is known that low-volatility stocks tend to perform better over the long-run.]

- He prefers 'family-run' companies in which the directors have large shareholdings themselves, have 'clean' reputations and have an attitude of being 'stewards' of their shareholders money. [This aligns the interests of the management with those of the shareholders and reduces the chances of fraud by the management. Again I think this must improve the probability of long-term success of the stocks in his portfolio.]

- He prefers companies that have had few changes in their directors and few changes in their auditors. [Again, this reduces the chances of fraud by the management at the expense of shareholders.]

- He uses a 20% stop-loss rule to sell any poorly-performing stocks, but he ignores stop-losses if there is a major overall market fall. He says this is a way of limiting the size of his loss if he has made a bad selection of a particular stock, thereby preserving capital for better use elsewhere. [I think another way to look at this rule is he is using negative momentum to make some selling decisions, and it is well known that stocks with recent negative momentum tend to under-perform the market as a whole over the short-term.]

- He likes to invest in companies in which a number of directors are buying stocks in their own company using their own savings (as opposed to being granted options). [Again I think this must improve the probability of out-performance by those stocks of the market as a whole.]

I think his use of the above rules over the years must have greatly improved the 'conditional probabilities' [which could, in principle, be calculated mathematically using Bayes Theorem] of him constructing portfolios of stocks that significantly out-performed the FTSE All Share index. So even if he had selected his stocks at random from the pool that remained after removing those stocks that did not satisfy his rules, I suspect he would still have done very well over the years (although perhaps not as well as he actually has done after using his skill and judgement in selecting individual stocks from that pool).

Another rule he has is that he likes to attend Annual General Meetings of companies in his portfolio, or of companies in which he is considering investing, and to have discussions with directors if he can, so that he has a better understanding of the businesses of those companies and a feel for whether the management is honest and trustworthy. [This must improve his stock-picking success rate, although it is hard to quantify this, and many private investors don't have the time to do this.]

27th Sep '15

PKA

yes but what on earth does any of that have to do with Bayes Theorem? or the base rate fallacy?

28th Sep '15

Tournesol wrote: "yes but what on earth does any of that have to do with Bayes Theorem? or the base rate fallacy?"

Tom Firth's article above has a section entitled "Applying the Theory". I was using Lord John Lee as an example of someone who been extremely successful at investing over many years, and whose success supports what Tom Firth wrote in that section. This is because I think a large part of John Lee's success was probably due to the rules he used to restrict the pool of stocks from which he constructed his portfolios. I think that greatly improved the conditional probabilities (which could in principle be calculated using Bayes Theorem if one had all the data) of successful outcomes from his portfolios over the long-term. In other words, he greatly improved his 'base rate' probabilities of investing success by following those rules. Of course, John Lee's rules are not the only way to do that.

29th Sep '15

PKA

"..... In other words, he greatly improved his 'base rate' probabilities of investing success by following those rules...."

I really think you are talking about something quite unrelated to the subject under discussion here. By your logic almost all successful investors could be said to be applying Bayes Theory. Such a statement would be so broad and so nebulous as to be of no value.

Seems to me that your thought process leads to the idea of emulating investment heroes - "What would Warren Buffett do?" kind of stuff which is at base rather unedifying.

What the thread originator was getting at with Bayes was the need to separate the general/shared characteristics of a group or class of objects (their base rate) from the specific differences between individuals.

So in the example given we were directed to consider that although satanists often have certain characteristics their numbers are small. Christians might possess the same characteristics only rarely but their numbers are big. Consequently there are more Christians who look like satanists than there are satanists who look like satanists. I cannot find any of that reflected in your discussion of John Lee's approach that will help others to emulate it.

I don't want to snark about this I just do not relate what you are saying to the subject under discussion.

You ask

"If you will allow me to play Devil's advocate for a minute though, how would you say that picking sectors is different from picking stocks? Is it easier? If so, why? The evidence would suggest that experts and amateurs alike are poor forecasters whether it comes to company earnings or macro events - it seems the future just isn't all that clear, whatever the scale!

If I was to employ such a strategy, my worry would be that I've essentially replaced one forecasting problem (the stock picking problem) with another almost identical forecasting problem (the sector picking problem). Why would I be more likely to get it right just because I'm analysing a different aspect of the future? "

I'd look at things from a different angle.

Suppose you came to the realisation that the oil sector was poised to outperform. Spare production capacity was at an all time low. Consumption was growing strongly. Depletion was increasing. Population growth was strong. Economic development was bringing many new consumers into the marketplace. Existing consumers were increasing their consumption. Etc etc etc. In other words the base rate for share price growth in the oil sector would likely be stronger than the base rate for some other sector - say retail. Obviously you would want to invest in companies in that sector. You would be making a sector based decision. You could if you wished simply buy the sector in toto by using a collective or by buying a basket of shares.

Someone else who fancies themselves at stock picking would be sticking individual companies under their microscope and assessing their potential as individuals. But if the individual company was in a sector that was going downwards then even a strong outperformance of its peers might still deliver a dismal performance in absolute terms.

What I'm trying to say is that if builders or banks are in a period of decline then the answer is to avoid those sectors not to invest time and energy trying to pick the best stocks therein. And if oil companies are in the ascendant then you can harvest much of the potential gains without succeeding in picking the very best stock.

I am not saying that it is easy to figure out sectoral vectors (direction and magnitude of movement). Much of the time it is really difficult to get a read on most of the market. But it is frequently possible to get a bearing on just one or two sectors - banks, oil companies, house builders and to act accordingly without having to complement that insight by picking the top performing individual stocks.

My own experience is that it has several times been possible to call the oil sector and to position oneself with advantage. I do not claim any generalised success in other sectors but I'm working on it. To date my second best sector based calls have been in fixed income pref shares, where I arrived late but still in time to join in. In my opinion just a few successful calls which are used as the basis for significant investments and which are held for significant periods can deliver life changing returns.

I'm currently intending to pursue the use of investment trusts to allow me to step back from stock selection and spend more time on sector selection. I think that is the rational response to the Bayesian insights.

Interesting, thanks for getting back to me. That all makes sense and in particular your 3rd paragraph clarifies nicely. Especially once you consider that these trends can persist for extended periods of time I suppose it could indeed be easier to identify a sector that is performing well and is likely to continue to do so.

From a personal perspective, I am still a little wary as I do not have full faith in my ability to reliably identify such trends in a timely manner due to my inexperience, ignorance and so on. Therefore I think it makes sense for me to apply Bayesian thinking to an area that I might consider to be a little more timeless. Value stocks, for example - it seems self evident that buying dollars for 50 cents will always prove to be profitable.

By the way, I thought that what you said here:

"So in the example given we were directed to consider that although satanists often have certain characteristics their numbers are small. Christians might possess the same characteristics only rarely but their numbers are big. Consequently there are more Christians who look like satanists than there are satanists who look like satanists"

really summarised the idea concisely and in very simple language - I may have to borrow your phrasing in the future!

All the best,

Tom

Base Rates and Bayes’ Theorem

Base Rates

Bayes’ Theorem

Unlock the rest of this article with a 14 day trial

About the Author

Tom Firth

16 comments

Unlock the metrics for all these stocks and more...

Unlock the metrics for all these stocks and more...