Looking in the Rear-View Mirror: The Perils of Data Snooping

Saturday, Jun 08 2013 by
Looking in the RearView Mirror The Perils of Data Snooping

Some of the GuruModels that we feature on the site are based on the experiences of practitioners who have taken the time to articulate their approach (e.g. Jim Slater or Ben Graham). But other strategies, usually those developed by academics, have been based on what is known as  “back-testing". This involves looking at historical data and simulating what would have happened if you’d used a particular technique in the past. It is usually done without factoring in trading costs or taxes, given the complexity of factoring in these investor-specific aspects. One example of a Model derived using backtesting, which many readers will know we are huge fans of, is the Piotroski F-Score.

Looking at the past like this is a powerful technique. However, it's very important to be mindful of the pitfalls of investment simulations and "naive backtesting". It is possible for many strategies to look great in backtests but most actually disappoint upon implementation. Simulations are usually always based on a 95% confidence interval, but in reality, investors are disappointed far more than 5% of the time (this has not been the case with Piotroski though!). To understand why, it's important to be mindful of the following critical issues with backtested results. 

How good is the underlying database? 

First up, it's important to consider the calibre of the database(s) on which the back-testing is performed. As John Freeman notes in his excellent article on the subject, “Behind the Smoke and Mirrors: Gauging the Integrity of Investment Simulations”, data errors can manifest themselves in persistent and insidious ways. There are many ways to be fooled. The most important issues include:

i) Survivorship bias

This is the tendency for failed companies to be excluded from performance studies because they have gone bust or otherwise disappeared (e.g. a takeover). For example, a mutual fund company's selection of funds today will include only those that are successful now. Many losing funds are closed and merged into other funds to hide poor performance. In theory, 90% of funds could truthfully claim to have performance in the first quartile of their peers if the peer group includes funds that have closed.

ii) Look ahead bias

This is the bias created by the use of new or revised data not available at the time of the historical trading decision. An example of this would be where a trade is simulated based…

Unlock this article instantly by logging into your account

Don’t have an account? Register for free and we’ll get out your way


As per our Terms of Use, Stockopedia is a financial news & data site, discussion forum and content aggregator. Our site should be used for educational & informational purposes only. We do not provide investment advice, recommendations or views as to whether an investment or strategy is suited to the investment needs of a specific individual. You should make your own decisions and seek independent professional advice before doing so. Remember: Shares can go down as well as up. Past performance is not a guide to future performance & investors may not get back the amount invested. ?>

Do you like this Post?
8 thumbs up
0 thumbs down
Share this post with friends

Please subscribe to submit a comment

About Stockopedia Features

Stockopedia Features

Stockopedia Features covers in-depth stories on strategies, companies and themes that are relevant to online investors. Investing is hard work. We don't try to over-simplify complex concepts - we prefer to try to help you navigate the detail.  more »

Stock Picking Tutorial Centre

Let’s get you setup so you get the most out of our service
Done, Let's add some stocks
Brilliant - You've created a folio! Now let's add some stocks to it.

  • Apple (AAPL)

  • Shell (RDSA)

  • Twitter (TWTR)

  • Volkswagon AG (VOK)

  • McDonalds (MCD)

  • Vodafone (VOD)

  • Barratt Homes (BDEV)

  • Microsoft (MSFT)

  • Tesco (TSCO)
Save and show me my analysis