Order matching algorithms

In today’s markets dominated by High-Frequency algos, room for profits for non-HF (and more importantly, non-HF aware) guys is generally speaking reduced. The proportional performance impact of HF is likely to be bigger the smaller is your average trade and the shorter your holding period.

However, in my experience this doesn’t have to be necessary the case: simply put, as in any business you have to adapt to the competitors  and in this case one way of doing it is to pay more attention and improve the execution side of your trading. This is not always easily doable (see the “Timestamp fraud” reported by Zerohedge), but there are some low-hanging fruits that can be picked as a first step.

If this statement may sound kind of vague to you, I have an example based on my experience that supports it and that I think that could be useful to others (while hopefully not having too much of an impact on my strategies).

While all my models are fully automated, I still like to look at markets and particularly at order books when my orders are being executed.
Something that I noticed quite some time ago when trading 30y US bond futures was that whenever my limit orders were executed, I was immediately at a loss.
What this means is better explained by an example. Say that we had an order book that looked like this:

and that my sell limit order was included in those 750 @ 134.6.

Whenever I was executed, the mid-price would then immediately move against me, and the book would then look something like this:


Basically what was happening was that my order was always one of the last to be executed, so the simple fact that it got filled meant that there were no more offers (bids) at my level, and the best bid and offer would move up (down) one tick.

A quick investigation on the CME website revealed that the cause for this was the type of order matching algo being used by the exchange, a First In, First Out (FIFO) algo.

What is a matching algorithm?
CME explains it as:

A matching algorithm is a technique to allocate matched quantities, used when an aggressor order matches with one or multiple resting orders. Algorithms apply to both outright and implied matching.

In Rajeev Ranjan’s website you can find a more in-depth introduction to Order Matching Algorithms (as well as other resources on HFT/algo trading).

In the example above, my trading model was instructed to send the limit order only when the price was close enough to my desired level, which always made me one of the last to join the queue and hence one of the last to be filled, according to the FIFO paradigm.

In practical terms, what this meant was that I was always executed in the worst possible scenarios, that is when the price would continue in the opposite direction of my order, and at the same time I was never executed in the best scenarios, that is when the price would “touch” my level and then reverse back in my favour.

As you can imagine, a simple workaround for me was to send my limit orders (when operating under FIFO matching algos) as early as possible, but generally speaking, this observation can suggest different things to different people. For day traders that are not trading in an automated fashion, operating under FIFO matching algorithms could often mean increasing one’s Maximum Adverse Execution by one tick (which can be quite a lot, depending on what one is doing ), unless one is able to play around it.

Similarly to this case, there are other situations when the order matching algo in use and trades execution in general can become as important as the strategies/trade ideas themselves.

Another example of making good use of order matching algorithms could be that of a trader operating under a pro-rata matching algorithm, typical of Eurodollar (IR) futures. If you really want a fill of X lots, you could just send an order that is somewhat bigger than X – with the extra amount being dictated by how aggressive you want/need to be – and once filled try to cancel the remaining lots (DISCLAIMER: of course by doing this you are actively risking of being filled in all the lots, so just don’t take my word on this being a good practice and do it at your own risk).

Of course paying attention to the matching algorithm is just scratching the surface of the High-Frequency world, but I would think that in some situations it’s an easy “scratch” to do and one that could directly add some value.

To conclude this post, let me clearly say that for how good our market simulator is, trades execution can’t always be modelled beforehand. This doesn’t mean that we should give up trying to make simulations as realistic (and somewhat conservative) as possible, e.g. in terms of fills and slippage (here’s a nice post on what is slippage by Prof. Tucker Balch). Rather, we should just remember that there is no real substitute for personal first hand observation and interaction with the world.
All in all, it shouldn’t really come as a surprise that simple observation is a powerful tool, being it the first step of the scientific method.



Posted in Trading Strategies Design | Tagged , , , , , , , | Leave a comment

Feature selection in trading algorithms

Lately I have been looking for a more systematic way to get around overfitting and in my quest I found it useful to borrow some techniques from the Machine Learning field.

If you think about it, a trading algorithm is just a form of AI applied to prices series. This statement, although possibly obvious, puts us in the position to apply a number of Machine Learning techniques to our trading strategies design.

Expanding what discussed here (and here), it seems intuitive that the more features in a model, the more generally speaking the model might be subject to overfitting. This problem is known as the bias-variance trade-off and is usually summarised by the graph on the right.

bias-variance tradeoff

As complexity increases, performance in the training set increases while prediction power degrades

What’s possibly less intuitive is that the specific features used in relation with the dynamics to predict play a key role in determining whether we are overfitting past data, so that the error behaviour showed in the graph is just a generalization.

Something particularly interesting is that the use of the very same feature (e.g. in our application an indicator, a take profit or stop loss mechanism, etc) might or might not cause overfitting according to the dynamics we are trying to fit.

The reason behind this is that some phenomena (or some times even variants of the same phenomenon) simply can’t be described by some features.

As an example, imagine you are trying to forecast the future sales of a sportwear store in Australia. A “good” feature to use could be the season of the year, as (say) Aussies are particularly keen in water sports and so springs and summers tend to show the best sales for the year.
Now imagine trying to forecast the future sales of a similar sportwear store located somewhere in the US. It might be the case that US citizens don’t have a preference for any particular season, as in the summer they practice water sports and in the winter they go skiing. In this new scenario, a model using the season of the year as a feature is more likely to result in an overfitted model because of the different underlying dynamics.

Back to financial markets, an example of this could be how a stop loss mechanism tends to be (generally speaking and according to my experience) a good feature for trend-following strategies, but not for mean-reversion strategies (and viceversa for target profit orders). A possible explanation of this could be that trends are well described by the absence of big adverse movements, but their full extension can’t be known beforehand (but this is just me trying to rationalize my empirical findings).

So, how do you understand which features are good candidates?
Luckily for us, there are a whole bunch of techniques developed in the Machine Learning field to operate feature selection. I recommend the following 2003 paper for an overview of the methods: An Introduction to variable and feature selection by Isabelle Guyon. Any text of Machine Learning should also cover some of the techniques, as it does the exceptional Stanford’s Machine Learning class in Coursera 
Any other readers’ recommendation (or comment) is of course very welcome.


Posted in On backtesting, Trading Strategies Design | Tagged , , , , | 14 Comments

Resources for Quantitative Trading

Since a few readers asked me about this, here is a list of resources/tools useful in approaching quantitative trading.

For a similar (and more detailed in some aspects) list, you can have a look at this series of 3 posts from Quantivity:

In regards to books, there are obviously so many that are worth including, here I am selecting only among those that I personally read and hence feel comfortable to recommend.
For reviews on more financial books that you’ll ever be able to read, I suggest to have a look at Reading the Markets.

If there is some particular resource/link that you feel I am missing, please feel free to share it and I will include it in this list.

Where to look for ideas

  • Observing the markets
  • Understanding the inside-out of financial instruments
  • Academic Papers and Journals in general
  • Newspapers
  • Blogs (see blog roll)
  • Forums (Elite Trader, Tradersplace, Trade2win, Big Mike, Wilmott, Quantnet, Nuclear Phynance)

Statistical/Mathematical tools

Really anything coming from a number of fields:

  • Statistics
  • Machine Learning
  • Signal Processing
  • Physics (Complex Systems, Chaos Theory,…)
  • Econometrics
  • Stochastics Calculus

Market Data

Main Programming Languages used

  • C++ (execution mainly)
  • C# (execution mainly)
  • Java (execution mainly)
  • Python (execution and research)
  • Matlab (execution and research)
  • R (research mainly)





Posted in Resources for trading, Trading Strategies Design | Tagged , , , , , | 4 Comments

Trimmed performance estimators

This is a quick follow-up on my previous post on Quantile normalization.

Instead of removing just the top X quantile of returns/trades when optimizing a strategy’s parameters space, my recent approach has been to remove the top and bottom X quantiles, so effectively using a robust trimmed estimator of performance instead of the estimator itself.

The advantages are symmetric to those discussed in the previous post, as long as your backtest allows for realistic modelling of trades execution – e.g. if  you are using stop orders and trade bars (as opposed to tick data), you probably want to add an amount of slippage in some way proportional to the size of bar (specification needed because a conservative modelling of limit orders is easier to achieve).

Trimming out the worst returns is particular useful in case of strategies having single big losses (such are mean-reversion strategies of some kind usually), whereas trimming the best returns is more useful for strategies with big positive days (e.g. trend-following strategies).

Two (of many) possible variants are:

-To preserve autocorrelations of a strategy’s returns, one could decide to remove blocks of trades/days, instead of individual trades/days (in a similar fashion to what one does when bootstrapping blocks of trades/days).

-To preserve the number of samples in our results instead of removing the top (worst) days, one could replace them with the average/median positive (losing) days.

Something else to note is that if your performance measure makes use of std deviation (as it’s the case for Sharpe Ratio), trimming the tails of the returns from its computation is likely to result in an overestimation of the performance.

Finally, here’s the Matlab code:

normalise_excess_pnl = 1;
normalisation_quantile = 0.98;

if normalise_excess_pnl

best_daily_pnl = quantile(pnl_daily,normalisation_quantile);
worst_daily_pnl = quantile(pnl_daily,1-normalisation_quantile);

pnl_daily(pnl_daily>=best_daily_pnl ) = [];
pnl_daily(pnl_daily<=worst_daily_pnl ) = [];



(I usually have the variable normalise_excess_pnl automatically initialised to 1 or 0 from the external environment, according to whether or not I’m running an optimisation).


Posted in On backtesting, Performance Metrics | Tagged , , , | Leave a comment