Lessons in Investing from Science
Confirm Ye Not
Here's what ought to be a really boring idea - we need scientists in general and psychologists and economists in particular to stop hypothesising after results are known (HARKing, geddit?). Instead they need to state what they're looking for before they conduct their experiments because otherwise they cherrypick the results they find to confirm hypotheses they never previously had.
The underlying problem is our old foe, confirmation bias. And the solution for scientists and social scientists alike is known as pre-registration. It would be no bad thing for investors to demand a similar process for fund managers and financial experts. Or, for that matter, to apply some of the ideas to their own investing strategies.
No No Negatives
It's been known for years that a lot of scientific research isn't very reliable. There are numerous problems, chief amongst them being the non-publication of negative results: an issue known as publication bias. There's no kudos in showing that your hypotheses were wrong, so researchers and corporations tend to bury the data, but it's still valuable information that should be shared: scientists see further by standing on the shoulders of others, we shouldn't be encouraging them to shrug them off because they've got bored.
Repeat, Again
Once we move to the social sciences then the problems are even worse. Human beings are terrible things to experiment on, being inclined to change their minds, develop opinions about the experiments and to second-guess what the researchers would like them to do, just to be nice.
All too many experiments in the social sciences turn out to be flawed because of social or situational factors that didn't seem important at the time. Given this you'd think that repeating experiments to make sure the results held would be even more important for psychologists than it is for researchers in the hard sciences. Well, guess again.
According to research by Matthew Makel, Jonathan Plucker and Boyd Hegarty only a little over 1% of psychology studies have ever been replicated. Everything else is simply a matter of faith in the integrity and lack of bias of the original researchers. Which is not science: in the words of John Tukey, quoted at the head of their paper:
"Confirmation comes from repetition. Any attempt to avoid this statement leads to failure and more probably to destruction."
Pre-Register
The best solution to this we've yet found is known as pre-registration: studies have to be registered in advance, and the hypotheses under investigation stated up front before the research is done. This prevents the experimenters from looking at their data after the event and picking out interesting positive correlations which they didn't control for, but which are likely to get published.
"17 out of 30 studies (57%) published prior to 2000 showed a significant benefit of intervention on the primary outcome in comparison to only 2 among the 25 (8%) trials published after 2000".
Some of this may be because the low-hanging fruit on the subject was been picked earlier, but it's a scary result all the same. It seems likely that because the researchers can no longer consciously or unconsciously pick the results they prefer they remove the possibility of confirmation bias - and the fall is so dramatic it places the previous results in question. And, of course, it's not clear how many of those have been replicated.
Pre-registration isn't universally popular: there is much rending of white coats and grinding of molars over the issue. Opponents argue that it risks putting scientists in a creative straight-jacket. Although when respectable peer reviewed journals start publishing papers alleging the existence of extra-sensory perception based on ...
"Anomalous processes of information or energy transfer that are currently unexplained in terms of known physical or biological mechanisms"
... then you have to wonder whether the creative juices maybe need a touch of reduction - oh, and the results of this experiment don't seem to be replicable, bet they never saw that coming.
So, what other group of people do we know who are given to making ad-hoc hypotheses, investing loads of money in them, and then ignoring the results while cherrypicking specific successes in order to publicly claim that they were successful? OK, apart from politicians.
Investing Feedback
Investors have all of these faults, and a few more. If we truly wanted to become better investors then we'd pre-register our hypotheses - including our expected timescales - and then measure our results against the results. Doubtless the outcome would frequently be embarrassing, but the evidence that we do have suggests that getting real feedback about our performance is the only way to improve predictive capability in complex systems like the stockmarket (see: Depressed Investors Don't Need Feedback. Everyone Else Does).
The other thing this would do would be to force us to face up to the reality that we can be successful by luck and can fail through no fault of our own. In complex adaptive systems we simply cannot predict every possible situation, we can only hope to be able to predict a little better than average. But a little better is enough to make a turn, so every percentage point improvement we can make is worth it.
Commit and Document
"Commitment-treatment group participants have a 12.3 (9.6) percent higher probability of increasing their savings by more than 20 percent after six (twelve) months, relative to the control group participants, and an 11 (6.4) percent higher probability of increasing their savings by more than 20 percent, relative to the marketing group participants. The increase in savings over the twelve months suggests that the savings response to the commitment treatment is a lasting change, not merely a short-term response to the new product"
I suspect that even a non-financial reward system based on peer support would facilitate uptake.
HARK, hear ...
Avoiding HARKing is the future of the hard and the soft sciences. And, by analogy, as investors if we don't have hypotheses about what we're investing in then we're simply the modern equivalents of astrologers. And, if we have hypotheses, we should write them down and test whether they're right, not simply crow about the random successes and ignore the equally random failures.
It's worrying, of course, that this isn't already the basic investing process. But to be honest it's even more worrying that it doesn't seem to be the basic scientific process. Genius and creativity has its place in all human activity - Kepler came up with his third law of planetary motion by mapping orbits to harmonic ratios, believing these to be a sign of heavenly perfection. But Kepler was a mad genius who happened to be correct, so here's my hypothesis: relying on mad geniuses for humanity's future and your family's well-being is probably not prudent.
Disclaimer
This is not financial advice. Our content is intended to be used and must be used for information and education purposes only. Please read our disclaimer and terms and conditions to understand our obligations.
6 comments
Excellent post, Timarr. More rigour in our analysis and approach to investing is sorely needed.
In order to avoid any bias I have adopted a system which is to come up with a hypothesis and then to back-test it to destruction against the full LSE equity market. I have been doing this for 12 months now.
The biggest hurdle i have come across with this approach are false positives. I'll give you a simplified example. Say I have come up with a system that traps a share that has recently rocketed. Back testing this system against the full market will give me not only this share but a lot more that turn out to be real dogs. The false positives.
My target now is to achieve at least a 60/40 ratio of "true" positives to false positives. I then apply the rule of letting winners run but cutting losses at 10% maximum, creating a bias in favour of the winners. The best i have achieved so far is 55/45.
Am I chasing fool's gold? Maybe. But my semi-scientific approach keeps my feet firmly on the ground. If back testing a potential system shows no real gains, then I ditch it and look for another hypothesis.
Sounds like a sound approach but one of the issues with backtesting that it always skews in favor of the past. This is a bias of all studies. For example, between 1950 and 2000, being long the S&P 500 above the 2 day MA and in cash below it, you'd have beat the buy n' hold in the vast majority of years (according to bullmarkets.co). But after 2000 it just didn't work anymore. Markets changed. The new choppy markets unwound the effectiveness of short term MA strategies. But a backtester might still look at his backtest and conclude that in the long run the system should work. It didn't.
Further, people in general tend to be arrogantly conclusive in their thinking. A liberal friend of mine once told me that he'd read a study which "showed" that immigration hadn't resulted in lower wages. Now I have no idea whether immigration effected wages or not, but the idea that such a study is conclusive is ridiculous. We can't ever know the counter-factual. The idea that an academic can tell us what would have happened had we gone down a different road is rank speculation in the guise of serious research.
My point is... you might have a lot of historical true positives that turn out to be false positives in the future. Personally, I tried technicals for years, and found backtesting provided a lot of false hope. Doesn't mean you won't do well with it of course. But be extremely suspicious of good backtest results. They are frequently poor indicators of future performance.
Nick, I like where your reasoning and models start in your comment, because I use the primitive form of Brownian motion in my models. My observation suggests that ultimately agent actions must agree with their computations, and their computations must agree with the underlying conditions.
Thus the “information” and “work” put into the markets should always be accessible once the signal is separated from the noise.
I once tried putting some shares through a fourier transform. There were some problems with my methodology, though. I was trying to see if there were any cycles. Nothing leapt out at me particularly, and the frequencies looked pretty random.
What I find bizarre is the pattern of the Footsie over the last 5 years. From 2013-4 there is a very strong channeling going on. It seems a little suspicious, actually. Maybe I just need to replace my tin foil hat, though.
The problem is that the performance of the market is anti-inductive as Taleb calls it. That is, you cannot deduce how it works using induction. Because it contains intelligent agents it is able to observe attempts to understand it and react to them.
In particular, if you did ever find a way to beat the market (average) 100% of the time, the one thing you could not afford to do is publish it!
For me, a useful model to understand stock movements is based on geometric brownian motion. You have to try to identify the 'drift' component in the face of a much larger volatility component. The volatility component will integrate to zero over a long enough time frame (by definition) and the drift component gradually asserts itself. Of course a complicating factor is that the 'drift' for any given stock can change over time.
Although you can often view the drift quite easily by eye using a log-price scale and a suitable time frame (a year or longer), ultimately the drift is linked to fundamental factors. While those remain strong a stock will perform well over "long enough" time periods.
By contrast, over short time periods the market attempts to find a price for each stock which means that the expected return for any stock is the same (so good stocks become more "expensive" than riskier stocks). Trying to outwit this mechanism means exploiting so-called inefficiencies in the market. This is a zero-sum game which one strongly suspects can only be consistently won with information not generally available to all market participants.
Great post Timarr, over the years I've made a number of howlers through not understanding my own bias , more specifically I view current events through the prism of my experience and end up surprised when history fails to be an infallible guide to the future. Basically numpty mistakes..