Verne Global

AI / ML / DL | Finance | HPC |

31 January 2019

Why hedge funds are investing in HPC

Written by Shane Richmond (Guest)

Shane Richmond is a freelance technology writer and former Technology Editor of The Daily Telegraph. You can follow him at @shanerichmond

High performance computing (HPC) is of increasing importance in the world of hedge funds. What are they using it for and what does its future look like?

Hedge funds are one of those financial institutions that most people have heard of without really understanding what they do or how. These funds originated as private investment vehicles that sought to balance, or 'hedge', their risk by pairing long and short positions. They're appealing to investors because they often have less regulation and more flexibility than, say, mutual funds.

Over the last few decades they have grown significantly in popularity and their financial strategies have expanded far beyond the original long/short pairings. What's interesting from a technology point of view is that they increasingly do their work with the aid of HPC. What are the implications of that and how is it likely to change in future?


Before we go any further, I should warn you that you should not only never take financial advice from me, but also you should treat my explanations of how the world of investment works as the roughest of rough guides. I'm going to tell you why this is interesting from an HPC perspective but if you want to understand the finance side, then I recommend reading some experts.

For our purposes, we're interested in quantitative hedge funds, known as "Quants". They use algorithms or other system-based strategies for making their investments, rather than the judgements of investment managers. That means using algorithms to identify undervalued stocks or predicting future price movements with statistical analysis of influencing factors.

These things can be done at 'high frequency', which requires a strong IT infrastructure, or at 'low frequency', where the competitiveness of maths modelling is most important. As with lots of algorithmic work, these aren't things that human investment experts can't do, they are just things that computers can do more quickly, or more cheaply or by considering vastly greater quantities of data.


The rise of algorithmic trading has raised fears that automated systems will have an incentive to cause "flash crashes", driving the market down suddenly and then taking advantage of low prices. Andrei Kirilenko, the former chief economist at the Commodity Futures Trading Commission and author of the regulator's report into the 2010 flash crash, recently told the FT: “We just have to accept that financial markets are nearly fully automated and try to make sure that things don’t get so technologically complex and inter-connected that it’s dangerous to the financial system.”

There are worries that as algorithms become more advanced, it will be impossible to reverse engineer decision making and thus unpick a bad decision. However, under MiFID II regulations algorithms must be thoroughly tested to ensure they won't disrupt the market and trading systems must be synchronised, with every action recorded on Coordinated Universal Time (UTC). All of this compliance adds to the already large amount of compute required.

The whole process is power-hungry and, like many HPC operations, the further you can cut costs, the better. There's also a skills shortage and hedge funds are competing for data center expertise with a broad range of sectors. For all those reasons, hedge funds typically use third-party hosting for hardware and software.


Increasingly, hedge funds are moving beyond algorithmic trading and looking at the possibilities of artificial intelligence (AI) or machine learning (ML). This really does move beyond the capabilities of human investment managers, allowing computers to test multiple situations or consume large amounts of data and then improve their processes so that they are more efficient or more profitable.

In its 2018 poll of hedge fund professionals, BarclayHedge found that 56 per cent had used AI or ML - nearly three times the total from the year before (20%). Just over two-thirds of those questioned said they were using the technology for idea generation. Other uses included portfolio construction (58%), risk management (33%) and trade execution (27%). The survey had a small sample, but the results suggest a rapid extension of AI and ML into hedge fund activity.

Does that spell the end for humans in the hedge fund world? Not at all. Quant experts are in high demand and a good candidate with a PhD can expect a starting salary in excess of $200,000. Hedge funds have been building relationships with universities to increase their chances of snapping up quant talent.

Investing is still about risk and even the smartest algorithms can't eliminate that. They are likely to keep getting closer, though. I’ve already said you shouldn’t take investment advice from me but, I’ll offer you some anyway: don't take the short position on algorithmic trading.


Sign up for the Verne Global newsletter

Opinion, thought leadership and news delivered directly to your inbox once a month.