Is this system testing ?





Do you trust what you see ?

Let me try.

Q1: Is this system testing ?
A1: Yes.

Q2: Do you trust what you see ?
A2: Yes, with some reservations.

My reservations are:

1. I do not know how long a period has this test been done. Judging from the Total nos of trades (13), I think it must be rather short. And since this is Nikkie's data, I take a peek at the chart and I think it must cover mostly the trending part of the market. I would like to see how this system handle the highlighted range bound period.

2. The effect of point 1 above, will be reflected in the equity curve and draw down when the test is extended to include the range bound period which last longer than 1 year. I have to see those and be comfortable with it before I use this system.

3. I am not sure if this trading system is designed specifically for this instrument. Can it be apply to other instrument? How is the performance?

Suppose I'm testing 2 systems.
A 20 Period Moving Average and a 20 Period Channel Breakout.

And here are the results

Nikkei 20MA (with a lot of data)


Nikkei Channel Breakout (with a lot of data)


Nikkei 20MA 07/30/2003 to 6/30/2005


Nikkei Channel Breakout 07/30/2003 to 6/30/2005


Simsci 20MA (with a lot of data)


Simsci Channel Breakout (with a lot of data)


How much data ?
How can we compare the results ?
Are the results valid anyway ?
Now what other questions we can ask?

Thanks

A. Let me put it out front that I am no expert in trading system evaluation. I am doing this on a "L plate" basis, and I welcome all comments.

From the 2 systems testing report presented, I tabulate the profit factor as followed:



I noted the following:

1. 20 Period Channel breakout system (20P BO) performs better then the 20 Period Moving Average (20P MA) system.

2. 20P MA is having a profit factor of 1.1+, not very attractive, no matter trading Nikkie or Simsci. 20P BO is slightly better, higher on Nikkie (2.49). But I must must say that this is still far off the TurtleFarm trading on Capitaland, which has a factor of more then 4.

3. In fact, when using lesser data (07/30/2003 - 6/30/2005), 20P MA has a negative expectancy.

Thus my conclusion is trading 20P BO has a better chance to survive.


B. Next, let look into the more problematic areas.

I noted that the 2 system testing report are generated differently, from 2 different softwares. I cannot tell if these 2 softwares are using the same definition of terms, same algorithm, same setting and same starting capital. So when I do the comparison on the details, I run the risk of comparing apples to oranges.

For example, the 20P BO has slippage, but I am not sure if the 20P MA has the same allowance. Another thing is the drawdown, 20P MA is "Maximum Intraday Drawdown", but 20P BO is Maximum Drawdown, which I think is pegged to end of day price.

If I will to ignore these differences, then I come up with the following table, where MD/NP is (max drawdown / net profit)



I noted the following:

1. Generally the 20P BO still outperform 20P MA; with a maximum drawdown that is less the then half of the net profit generated.

2. It is not possible for me to trade a system that has a max drawdown that is more then the total net profit, especially when long term data are used. 20P MA will knock me off by its max drawdown before I even see profit.

3. The 20P BO short term MD/NP ratio is high, I think 20P BO must have some bad patches lately, making its net profit lower and thus the high ratio value. If this is not the case, then it sound another alarm (see below).

4. Someone said that we should take the maximum drawdown in the system test report and multiply by 2 to get the realistic drawdown, and half the net profit as realistic profit when trading real money. If we will to apply this harsh test to both the system, then it is I would not be trading these system at all. If we will to look at the testing report on Capitaland using TurtleFarm, we have Net Profit 118,081.66 and Max Drawdown 7188.88, a MD/NP ratio of 0.061. Even we double the drawdown, half the profit, we still have a ratio of less than 0.25.


C. Having said all these, I still yet to see the equity curves of both the systems. I prefer a smooth up swinging equity curve, as it make me easier to stay with the system. To wide a swing will frighten me away.


D. Just my thoughts, and welcome all learned friend point out my mistakes.

HC,

you are too humble!

as the thread imply,
let's keep this thread to system testing.
while systems comparisons are part of the test,
i think it would be more productive to be neutral,
not to promote or bash any systems.

Let's us come up with a proper framework for testing trading system.
thanks

Thanks to central limit theorem,



We do not know anything about the systems yet,
but even if the 20 period simple moving average system
or the 20 period channel breakout system show very nice profits,
we have to ask
is it all down to luck?

I am lost.

What mean is that? Average profit per trade ? If so, the number under the 20 MA columns does not tally. And that Std Dev is about the distribution of profit/loss of each trade? And Z ?

Sorry for not being clear.

Consider the example.
A system has generated the trades as shown below.



Now we make an assumption, that the returns are of a normal population.
And we make use of the fact that

T = (Average P/L) / {(Stdev P/L)/SQRT(# of Trades)}
having a t distribution with (# of Trades - 1) degrees of freedom.

Of course, if our assumption is wrong, then T is meaningless.

But if we have a large sample size,
by virtue of central limit theorem,
then

z = (Average P/L) / {(Stdev P/L)/SQRT(# of Trades)}

and for 95% confidence that there is no illusion,
z > 1.645 will be nice.

I am lost, again. (It is my fault, I am the one who must say sorry).

It looks like you are trying to ascertain 95% confidence of having that average return based on the data on the return all the positive and negative trades. But I am weak at statistics, so if you go beyond mean and std dev, I am lost. I am not able to add anything that is of value. Maybe you can teach me instead?

May I add here that I am not sure how useful it is to applying the normal distribution methodology here, as I doubted the the trade returns' distribution is normal in the first place.

For consistency, I would prefer to have a view on the equity curve, over a long period where it cover up/down/sideway market and see how it behave in each period.

(CSK, if you re reading this, I am trying to plot the equity curve with the price chart in TurtleFarm, but still cannot get it done. Can you add this in in your next version of TurtleFarm?)

Hi HC,

(1) You can say that

(2) Don't be too bothered by the theory. Yes P/L is hardly normal distribution.
But thank you for central limit theorem.

clt does not imply that the p/l distribution becomes normal as n gets large, no!
but we can take it that when n is large, by clt, many statistics we use will make sense.

so as i had said in the capitalland thread, we want to have a sufficent # of trades, (>30) for testing purposes.

"(CSK, if you re reading this, I am trying to plot the equity curve
with the price chart in TurtleFarm, but still cannot get it done.
Can you add this in in your next version of TurtleFarm?)"

The chart is plotted with date scale on the x-axis while equity curve
is plotted with trade number on the x-axis and in a serarate display
in the Performance Report. I can add a subgraph to the chart window
but there is a problem. Only a portion of the equity curve corresponding
to the price chart can be displayed at any one time. the entire slop
of the equity curve will be be seen. I see how I can implement this.


On applying statitical analysis to results from a trend following system:

I am not sure about using this. It is normal for a trend following
system to have a low percentage profitable trades. It can be 30% yet
still showing profits overall. And where the system has risk and money
management rules, the $ losses per trade tend to be small while the
$ wins from profitable trades may be big.

Also the position size where calculated based on market conditions
and account equity will not be constant position size in terms of lots
or number of shares. Therefore the individual trade P/L cannot be
directly measured against one another.

For a trend following system the outliers results bring in the most
money. Statistically, outliers trades are very small numbers.

"Only a portion of the equity curve corresponding to the price chart can be displayed at any one time. the entire slop of the equity curve will be be seen. I see how I can implement this."

CSK:

I was thinking along the line to add an additional equity curve option to the price chart, not replacing the existing equity curve in the performance report. I do see the importance of the existing arrangement of having existing curve plotted again the nos of trade as x-axis.

Just to clarify in case I did not make myself clear earlier.

To add one more thought on the issue of the trade return distribution is normal or not:

Let say the probability of price move is normally distributed, meaning that price at T+n is most likely to stay at the same price P as that of time T; the probability of price be P+1 tick and P-1 ticks are the same but each is less that the probability when price remain at P; the probability of price be P+2 tick and P-2 ticks are the same but each is less that the probability when price remain at P or P+1 tick or P-1 tick; and so on.

Next, if I have a strict cut loss rule in my trading system that say cut after price drop 5 ticks from entry price, the normal distribution of probability will the truncated and become zero (slippage & brokerage ignored) once price drop below P-5 ticks. So even with a normally distribution price movement probability, this simple rule will make the trade return probability distribution no longer normal.



To extrapolate this observation further, if the entry and exit rule are asymmetrical in a trading system (which I think is true in most cases), even with a normal price distribution probability, the trade return is unlikely to remain normally distributed.