Having researched and designed your trading system by working through the steps we have discussed so far it is now time to back-test your ideas to see if the system ‘works. Research is conducted via back-testing concepts against available historical data. This is where your programming skills are required, unless you wish to conduct your research manually or visually.
A word of caution regarding visual back-testing: your bias of wanting the rules that you are testing to work will block out instances of price action that result in loss trades, missing trends by exiting too early. These are called ‘scotomas’ or ‘blind spots’. This is also referred to as ‘perceptual blindness’ where price action that contravenes what you believe should happen (or wish to happen) according to your chosen rules is simply not seen on the chart. This occurs in live trading as well as visual back-testing. When a consistent mindset is achieved the researcher becomes far more objective – objectivity is a subset of consistency – and perceptual blindness is mostly overcome when doing research or in live trading.
Perceptual blindness when doing research will result in a skew towards omitting losing trades, including trades that don’t actually meet the rules and overstating winning trades, which will make the research results look far better than they really are. There is a high probability of disaster taking such a set of rules into live trading.
Computer based research will always be far more objective than visual back-testing. Merely learning a programming language is not enough. Programming techniques also need to be learned which will be covered in any half decent programming course.
Choose a concept or combination of concepts from the list discussed in last week’s blog, or elsewhere, including the internet, magazines such as Stocks & Commodities and books that contain ‘base’ trading systems. Then code into your chosen research tool (e.g. Wealth-Lab, TradeStation, Trade Navigator) the criteria to filter price action and/or volume and to determine entry and exit points. Remember that the chosen concept needs to match your stated requirements for the system you are researching. For example, using a momentum indicator on daily charts simply will not produce a profitable, let alone robust, system that has average hold periods of one to five days.
Typically, a filter should be used to improve the win rate of your system and control the opportunity to trade that is required in your stated objectives. A filter could be:
• a moving average
• comparative relative strength against an Index
• inter-market analysis
• fundamental analysis
• Elliot Wave
• Fibonacci retracements
• a combination of the above, or others
Once you have programmed your research concept, and removed all the bugs from the code, the next step is to test your rules against a set of ‘in sample’ historical data. Ensure that you use clean data that closely represents price action in historical live trading. I suggest that you spend some time researching this topic, especially if you are working with futures markets where historical data mostly needs to be normalised for research purposes. Futures contracts expire at regular intervals and if the data is not ‘back adjusted’ to reflect changes in the values of contract months the data will be virtually irrelevant for back testing purposes..
‘In sample’ testing data should be historical data that has suitable information to generate sufficient trades to provide a large enough statistical sample for the timeframe you are trading. For example, if your trading timeframe is intra-day using five minute bars, then one to two months’ data will be sufficient. If your trading timeframe is medium to long term using daily and weekly bars, then five years of historical data should be used and > 10 years for monthly charts.
As a rule of thumb, the ‘in sample’ data set should cover at least two cycles each of markets moving up, down and sideways. This usually requires a subjective ‘eye-ball’ call of the overall market. Again, be careful not to succumb to your personal biases when using your subjectivity. Basic psychology books will inform you of the generic biases that we all have. These biases may assist us in other parts of our lives such as surviving in business or society but hinder us when it comes to trading.
‘Out of sample’ testing data should then be used to test the system rules on a different set of price action data. You are looking for similar results between the two sets of sample data or, as a minimum, one set not being drastically different to the other.
●——————————————●————————-●—–
In sample data Out of sample data
If testing a single market, such as an index or a currency, then using ‘in sample’ and ‘out of sample’ is very important. However, if testing an equities system, then testing price action across, say, 200 to 500 stocks over many years, will provide a wide variety of price action as a test bed. It then becomes more important to test the system rules in different types of all market action, i.e. rising, sideways and falling markets. Using an overall market index like the All-Ords to determine whether the overall market was a bull or a bear should then be brought into the sample test data and tested with trading rules. Such rules might fall under the heading of risk management rules.
Another issue that requires mention is liquidity, which is more applicable to equity trading systems than single market trading systems. Ensure that your research code includes criteria for excluding price action where there is insufficient liquidity to trade. For example, researching price action where an average of $10,000 a day is traded for a particular instrument will be too illiquid for a portfolio that will trade average position sizes of $10,000. Remember, price action that was “illiquid” 10 years ago may be very liquid now. In my experience, trapping illiquid trades that you simply could not have traded historically artificially improves an edge.
In next week’s journal posting I will discuss the welding together of the concepts we have been discussing over the past few weeks to arrive at a winning trading system.