Warning: Cannot modify header information - headers already sent by (output started at /data/web/virtuals/188780/virtual/www/index.php:1) in /data/web/virtuals/188780/virtual/www/wp-includes/pluggable.php on line 1219
24. Testing Criteria of Continuous Walk Forward Analysis - MultiCharts | Algotradingacademy.com

GET YOUR FREE WEBINAR NOW

CLICK HERE
24. Testing Criteria of Continuous Walk Forward Analysis – MultiCharts

24. Testing Criteria of Continuous Walk Forward Analysis – MultiCharts

This chapter´s content follows the previous chapter in which we explained what the Continuous Walk Forward analysis (CWFA) is. Let´s recall that in the ideal case we are looking for a CWFA where all 30 Walk Forward analyses (WFA) pass the defined test criteria. In such case, the overall results for the given WFA include a box saying “PASS” (see Fig. 1). If the test criteria are not met, the box for this WFA says “FAILED”.

Cluster Walk Forward Analysis comprising 30 Walk Forward Analyses all of which passed the demanding test criteria (PASS)_1

Fig. 1: Continuous Walk Forward Analysis comprising 30 Walk Forward Analyses all of which passed the demanding test

In this chapter we will introduce the various test criteria that must be met within a WFA for the different number of runs and the Out-of-Sample (OOS) percentage in order to obtain the “PASS” result (box). There are five CWFA test criteria in MultiCharts:

 

Test criterion no. 1:  Overall Profitability

The default setting of CWFA is that the overall profitability of all OOS runs must be higher than 0. In Fig. 2 you can see that in total the selected ATS is profitable on OOS.

Test criterion - overall profitability

Fig. 2: Test criterion – overall profitability

Test criterion no. 2: Walk Forward Robustness

Walk Forward Efficiency is based on the “Robustness Index” which represents and compares the Out-of-Sample (OOS) and In-Sample (IS) results. In essence, it is an indicator of the trading system´s quality. Robustness Index is calculated as follows:

Robustness Index = (OOS profit * 365 / number of OOS days) / (IS profit * 365) / number of IS days)

The construction of Robustness Index shows that it compares the efficiency of OOS profits against IS profits. Since the time periods of IS and OOS differ (OOSs are shorter in CWFA), it is necessary to standardise the generated profits and therefore we recalculate them to per-day values. The test criteria include the requirement that Robustness Index for each Walk Forward analyse was at least 50%. This is the default setting of CWFA in MultiCharts. Beginners may naturally ask the question: “Only 50%? Isn´t it too little?” In my opinion it is not. After including commissions and slippage into the ATS´s performance calculation I consider 50% for a WFA a realistic estimate. We must keep in mind that we search for the best input parameter setting within the IS period ​​according to the selected Fitness Function (FF). Sometimes, a WFA´s Robustness Index may exceed 100%. In such case we know that the OOS run was more profitable than the IS run in the same IS/OOS-days ratio. In Fig. 3 you can see the result of Continuous Walk Forward Analysis for 30 WFAs. You can see that the “Higher Average Cluster” chose the WFA 9 period in the 3 x 3 format with the highest average Robustness index´s value of 88.4%. This is a very high value which implies a really highly robust ATS. Matrix Average ​​then has the value of 78.2% which represents an average Robustness Index value for all 30 WFAs. Again, 78.2% is a very optimistic value. This CWFA was applied to the Intraday Breakout ATS which shows a high robustness on all markets with equity indices. Advanced traders will maybe find it interesting that the system had stable and symmetrical results both for Long and Short side.

 Results of Walk Forward Efficiency within Cluster Walk Forward Analysis

Fig. 3: Results of Walk Forward Efficiency within Continuous Walk Forward Analysis

 

Test criterion no. 3: Consistency of Profits

The test criterion is that each Walk Forward analysis must show the consistency of profits higher than 50%. In other words, we want more than half of the runs to be profitable. When looking at Fig. 3, all Walk Forward analyses easily complied with this requirement. You can see 100% in the selected blue box which means that all 10 OOS runs were profitable.

Test criterion no. 4: Distribution of Profits

The default setting of this criterion in MultiCharts is that a single run may not contribute by more than 50% to the overall profit of all OOS runs. The logic behind this is clear. Of course, we do not want to use trading systems that do not bring stable profits throughout the testing period. The fact that one run contributes by 50% to the overall profitability is actually negative since such a run may be only a coincidence and may not repeat again. In Fig. 4 you can see that the Out-of-Sample run no. 4 (OOS4) brought the profit of $10,229 while the total profit from all runs was $17,338. In Fig. 5 you can see the Walk Forward Summary, which is one of the tabs that can be displayed in MultiCharts´s CWFA. In this tab you can find, among other important information, the exact values of ​​profits/losses from individual WFA runs.

Cluster Walk Forward Analysis and the Distribution of Profits testing criterion

Fig. 4: Continuous Walk Forward Analysis and the Distribution of Profits testing criterion

Walk Forward Summary (Out-of-Sample)

Fig. 5: Walk Forward Summary (Out-of-Sample)

 

Test criterion no. 5: Maximum Drawdown in %

The last test criterion is the maximum tolerable drawdown. There are actually three possible settings (see Fig. 6).

 Maximum Drawdown test criterion settingFig. 6: Maximum Drawdown test criterion testing

 

  • The first possibility is that the maximum drawdown is exceeded in any one of the runs by 60% of the initial capital. The initial capital value is set in the test criteria settings. Thus, if we had a capital of $20,000, the maximum drawdown for each WFA should not exceed $12,000.
  • The second possibility is that the maximum drawdown exceeds the actual cumulative profit in any of the runs by 60%. For example, the tested system generated a net profit of $40,000 before the Out-of-Sample run no. 8. The maximum drawdown for the selected WFA should not therefore exceed $24,000.
  • The third possibility is that the maximum drawdown exceeds the cumulative profit before the beginning of the drawdown by 60%. For example, the net profit in the middle of the 8th Out-of-Sample run was $80,000. Thus the maximum drawdown would be $48,000.

In order to have a quality CWFA we need to perform at least 5 WFA runs so that we avoid the risk of accidental and statistically invalid results. On the other hand, if the number of runs is absurdly high, we face the risk that we will have only few trades in individual Out-of-Sample and thus we get statistically invalid results. All these aspects must be taken into account when evaluating the CWFA results.

To sum it up, in this chapter you learned what testing criteria must a trading system meet within CWFA so that we can consider it robust for the selected number of runs and Out-of-Sample %.

Petr Tmej

(c) Algotradingacademy.com

Previous: 23. Continuous Walk Forward Analysis – Sophisticated Tool for Robustness Testing

Next: 25. The Biggest Myth around Algorithmic Trading

Author: Renata Tmejová

http://www.algotradingacademy.com

The trader, the co-founder of QuantOn Solutions hedge fund, the lecturer in one person who has been successfully trading US futures via algorithmic trading systems (ATS) for many years. He is the main “brain” of the team Algotradingacademy.com. Petr´s mission is to provide relevant and necessary knowledge and skills in ATS trading to his clients so that they can become successful traders too. In 2009, Peter graduated from the Technical University of Ostrava, Department of Quality Control. The studies were mainly focused on Probability Theory and Statistical Processes. If you would like to get know more about his background and how the trading influenced his life you can find his story here: https://aostrading.cz/en/petr-tmej/. Peter honestly describes all his trading journey there. Furthermore, he reveals also his past and present trading results. His story can be quite inspiring for you since he began trading with very low initial investment. Nowadays, he can enjoy the success he has already achieved in live trading with his hedge fund QuantOn Solutions.

  • What Do You Need to Know About Maximum Drawdown?

    13.05.2018

    This performance indicator has for investors about the same importance as Total […]

    Read more
  • 29. Diversification as a Basic Assumption of Winning Traders?

    29.04.2018

    If there in the world of trading and investing exists something like […]

    Read more
  • 23. Continuous Walk Forward Analysis – Sophisticated Tool for Robustness Testing

    23.04.2018

    In the previous chapter we explained what the Walk Forward Analysis (WFA) […]

    Read more

CONTACT