Quantile normalization: a simple trick to reduce overfitting

To understand whether a strategy is able to perform in the future, the first question to ask is probably whether our strategy really showed great performance in the historic back-test or all it was doing was just describing past data accurately.

In regards to this, something I recently started doing when searching the strategy parameters space (i.e. “optimizing the strategy”), is to evaluate only the 98% quantile of the returns, that is leaving out the top 2% trades/days.

This approach has 2 main advantages:

1- It reduces the chances that the search algorithm gets stuck to some particular parameters values catching one-of/rare patterns in market history (e.g. 1987 crash, 2010 flash crash, or just  particular squeezes).

2- If the search has positive outcome, what you find is a strategy that is profitable even if the top 2% of his trades never occurred, which means that the strategy is relatively robust.

Of course depending on the trading strategy in use, you may want to change the actual percentage value (if you are using a MR strategy, usually having high win %, you might want to use an higher threshold), but you get the idea.

In some extent, this is similar to using the Omega ratio with a variable threshold level (given by the 98% quantile). However, I find this quantile normalization approach somewhat more powerful, as it allows you to calculate the full range of strategy statistics (including your chosen fitness function) across these “normalised” returns – in contrast with the single number spit out using the Omega ratio.

Andrea

Advertisements

About mathtrading

My name is Andrea La Rosa and I am a quant trader based in the UK. In the past I worked as a quant in the prop desk of an investment bank, before deciding to fully dedicate myself to quantitative trading.
This entry was posted in On backtesting, Performance Metrics and tagged , , . Bookmark the permalink.

3 Responses to Quantile normalization: a simple trick to reduce overfitting

  1. Pingback: Trimmed performance estimators | Math Trading

  2. Andrew Selby says:

    This is a very interesting concept. I like the idea of testing the robustness of a system by eliminating the top 2% of trades. Real really cool stuff…

    • mathtrading says:

      Thanks for the kudos Andrew. Not considering the top (worst) trades doesn’t only allow to test for robustness – in some sense it can actually improve the robustness of a system if used within the optimization process, as it allows to reduce the data snooping caused by optimizing some systems and then discard them if not robust enough.

      Andrea

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s