Comparing backtest results against a benchmark

Algos, strategies, code
1

Hi, I was wondering what is the right approach to compare returns from a strategy to benchmark such as Nifty?

  1. Approach 1:
    If I am backtesting over a 20 Y period and comparing the returns to Nifty, should I take into account addition and removal of stocks from Nifty over this period. (For e.g. between 2003 and 2012 Asian Paints was not a part of Nifty, hence should I exclude Asian Paints from the backtest and only introduce the data into the backtest from 2013 and likewise for other such stocks) and then compare the returns of the strategy to the returns of Nifty?

OR

  1. Approach 2:
    It doesn’t matter whether or not a stock was part of Nifty since we are considering Nifty as the benchmark and trying to check via the backtest if our strategy is outperforming the returns the index has provided irrespective of whether a stock was part of the index or not?

If approach 1 is a sound way, how do we compare returns from a strategy that comprises of stocks that were never a part of Nifty?

2

I don’t know about back testing but just saying, approach 1 looks like an overkill. If the back tested results are 1.5 times or 2 times to that of Nifty, irrespective of the additions and exclusions, that is very good in my opinion. If it is not too tedious, I guess you can go with approach 1, to be absolute.

I cannot recollect exactly, but I think I did read about a similar question being asked the person who back tested answered, but I don’t remember if that person said that he tested with exclusions or he tested with Nifty as a whole.

3

Approach 2

4

Don’t you think additions and removals matter in approach 1 because trades done after being removed from Nifty or before being added to Nifty will show an incorrect picture of the returns?

Also, what do you think about comparing returns of stocks that were never a part of Nifty?

5

Thanks

6

Again I don’t know back testing, so what I think is that, why is this comparison taking place, so that despite the back testing results being acceptable or even exciting by themselves, we have an option to see how the general consensus and the availability of an index fares with our back testing. So to that extent the benchmark does help. I don’t know how much more the back test results can bring to the table more than what they are (assuming they are better than Nifty), a couple more percentage points perhaps, or a few more points.

It is up to you to go into the depths and bring out the absolute numbers, if you feel the need for it.

And of course, testing in and out, in every possible way one can think of, is obviously better :+1:

2 Likes
7

Thanks, maybe I can try both approaches, no harm in doing so. If you ever come across the post you were talking about earlier, do post the link here.

8

I have not really tested stuff in this timeframe, but this is what i think.

  1. If the backtest uses Nifty Index for universe then yes, you will need historical composition else you will have
    some kind of future leak as you will select the stocks that made it.
  2. If this is not the case ( say universe is top 100 volume irrespective of index) then i dont think its needed. You can debate what your reference benchmark should be and choose whatever is closest. Then compare both returns and risk. Returns without risk is meaningless.
  3. An alternative to index would be to use your stock universe itself as baseline. Just take the buy and hold returns of your universe and then compare your system against it.
1 Like
9

Yeah point 3 makes a lot of sense too, thanks for pointing that out.

10

Dear Returns and situations are not same at that condition things were. different now go to running train