The Bare Metal Cloud: Software-Defined HPC with Real Performance


At the high end of High-Performance Computing, the disruptive arrival of cloud computing has often been met with skepticism by those accustomed to making decisions upon hard evidence - a rare commodity in the nebulous cloud sector.

Conversely, with data centers at a scale that stretches the imagination, hyper-scale cloud infrastructures often look upon conventional HPC infrastructure and its pyrrhic sacrifices for finely-tuned efficiency at the expense of all else, as though they are humouring a bygone era.

Both prejudices seem myopic and ill-informed. The smart and open-minded technologists in both sectors are seeing how technologies from HPC and methodologies from cloud can combine to form a hybrid infrastructure, already proving to be a thoroughbred combination. For example, the recent acquisition by Nvidia of Mellanox for $6.9B was driven by the potential for cloud infrastructure to be improved by HPC best practice.

What of the potential for HPC to be reshaped by cloud? The opportunity for transformation is massive. An early stampede for cloud-hosted HPC created premature solutions that offered the flexibility of cloud, but also with the performance of cloud. The second wave, spear-headed by Verne Global’s hpcDIRECT service, truly combines the best of both worlds.

Cloud’s Continuum of Convenience

Cloud infrastructure is no single thing. I view it as a range of compromises, each weighed against use case and requirements, to provide the best balance of flexibility versus performance.

At one end of this continuum of convenience is the most flexible forms of virtualisation, in which processor, I/O and network performance are traded for features such as resource over-commitment and live migration.

With increasingly aggressive optimisation, virtualisation’s flexibility reduces, but a dramatic boost in performance is achieved. High-performance virtualisation is an adequate balance of compromises for many.

At the furthest extreme of cloud performance, platforms for true HPC, such as hpcDIRECT, require bare metal performance.

In comparison, conventional HPC infrastructure management comprises a narrow range of offerings. The best products available do not approach the flexibility of bare metal cloud, and none could be said to offer more than primitive capabilities for software-defined infrastructure.

The advantages of bare metal cloud infrastructure for true HPC are just as clear as the economics of the cloud model.

Iceland: A Land of Giants

With it’s deep roots in Iceland, Verne Global’s vision for hpcDIRECT brings together a globally-unique combination: advanced bare metal cloud infrastructure, HPC expertise, low operating costs and renewable energy.

The advanced bare metal cloud provides flexibility through support for a broad range of hardware options, including GPUs. HPC performance is delivered through pioneering integration of Infiniband into the bare metal cloud. Software-defined HPC blueprints integrate components such as BeeGFS and Slurm. Cloud methodologies ensure usability through end-to-end automation and the provision of telemetry, logging and monitoring services - as standard.

OpenStack: Where Clouds are Made of Metal

Verne Global’s hpcDIRECT service is a prominent use case for the OpenStack Foundation’s new Bare Metal SIG. Through its advanced configuration of OpenStack Ironic, hpcDIRECT brings new levels of convenience to technical computing, without sacrificing performance, and reducing the time to results for its users.

Stig will be presenting at the HPC & AI MeetUp on Thursday 27th June in London. Come and join the group!

Written by Stig Telfer (Guest)

See Stig Telfer (Guest)'s blog

Stig is the CTO of StackHPC. He has a background in R&D working for various prominent technology companies, particularly in HPC and software-defined networking. Stig is also co-chair of the OpenStack Scientific Working Group,

Related blogs

HPC Perspectives - Dr Seid Koric

This is the first of what will be several profiles of leaders in High Performance Computing (HPC), all answering the same six questions. My hope is that this will help us all to better understand HPC, including its origins, a variety of career paths, successes, and influences in the field. I hope you enjoy learning as much as I have.

First up, Dr. Seid Koric, technical director at the National Center for Supercomputing Applications (NCSA) at the University of Illinois at Urbana-Champaign (UIUC). Seid is a legend in HPC, multiple winner of HPCwire’s Top Supercomputing Achievement, and someone who I have been fortunate enough to know for about 20 years.

Read more

Cloud options are challenging dedicated HPC infrastructure

Speaking at a conference in November last year, Bernd Mohr, general chair of the Juelich Supercomputing Center, described HPC as “a new gold rush” . He said: “We are the modern pioneers, pushing the bounds of science for the betterment of society.” That’s a pretty grandiose claim but much of the work being done in HPC does have groundbreaking potential. It’s use in the discovery of new medicines, for example, has allowed pharmaceutical firms to massively accelerate their research.

Read more

TrueHPC at Industrial Scale

As we prepare to descend upon Frankfurt this week for ISC18, science will take its appropriate place at center stage. Throughout the week, the gathering will hear how through science we are challenging the world’s toughest problems, dissecting those problems down to their foundations and then building them back up by methodically moving every minute detail that has been learned into the powerful realm of the supercomputer. Science shows us that the best innovations are created by literally starting from scratch and building from there.

Read more

We use cookies to ensure we give you the best experience on our website, to analyse our website traffic, and to understand where our visitors are coming from. By browsing our website, you consent to our use of cookies and other tracking technologies. Read our Privacy Policy for more information.