Containers for HPC

WHGA/001 Auditorium (Paul Scherrer Institut)

WHGA/001 Auditorium

Paul Scherrer Institut

Paul Scherrer Institut (PSI), 5232 Villigen, Switzerland
Derek Heinrich Feichtinger (Paul Scherrer Institut), Michele De Lorenzi (CSCS)

The earliest mentionings of Virtual Machine (VM) technology dates back to the 1960s. The much lighter container technologies began to appear in the early 2000s with FreeBSD Jail and Solaris Zones. The last few years have brought massive acceptance of containers when Docker introduced an easy workflow model and central registries for container creation and deployment. However, the use of these technologies within the HPC community has been limited due to Docker being mostly targeted at typical cloud deployment of standard applications with full isolation, and not for running huge collaborative HPC workloads.

An HPC container solution needs to be able to integrate and interface safely to essential services on an HPC system, e.g. the batch system, the shared parallel filesystems, and high speed networks.  Also, there still is the expectation of lower performance and higher complexity on the administrator side for integrating these technologies into HPC clusters. Recently, the topic has gained a higher interest in the community, since specialized HPC container or container-compatibility variants and tools have been developed.

In this forum we would like to discuss the current state of the art, the effects and the potential of these technologies within the field of HPC computing, and how we can leverage them in our environments already now.

Key questions
  • What use cases are there for containers and general virtualization technologies in HPC?

  • Container Technology: What are the differences between Docker, Shifter and Singularity?

  • What is the performance overhead of using container solutions vs. running HPC workloads on bare metal?

  • What is required to successfully use container or container-like solutions on HPC systems? What are important best practices and strategies to deploy containers efficiently on HPC systems? Are there container orchestration tools for these environments and how do they integrate in HPC scheduling systems?

  • What are the pros and cons for users and system administrators?

  • Development process: From containerized application development on a laptop to the deployment on HPC systems. Experiences.

  • Applications persistency: Are containers + virtual machines the key to application persistency and result reproducibility?

  • How mature is container technology today? How big is the risk of having to adapt the configuration recipes over time in order to conserve the execution environment?

  • Container security: Implications and different approaches to solving current security related problems.
  • Achim Gsell
  • Alain Studer
  • Alex Upton
  • Alexandre Kachler
  • Alvise Dorigo
  • Aral Can Kaymaz
  • Balazs Laurenczy
  • Carlos Fenoy
  • Christian Bolliger
  • Christian Clémençon
  • Christiane Pousa Ribeiro
  • Christopher Gwilliams
  • Colin McMurtrie
  • Daniel Pfenniger
  • Daniel Webster
  • Derek Feichtinger
  • Dmitry Ozerov
  • Douglas Potter
  • Gianfranco Sciacca
  • Hamid Hussain-Khan
  • Hans-Rudolf Hotz
  • Hardik Kothari
  • Heiner Billich
  • Hüsser Peter
  • Ivan Usov
  • Ivano Talamo
  • Jean-Claude De Giorgi
  • Jens-Christian Fischer
  • Jiri Kuncar
  • Joost VandeVondele
  • Jorge Molina
  • Leonardo Sala
  • Maffioletti Sergio
  • Markus Reinhardt
  • Martin Jacquot
  • Michele De Lorenzi
  • Miguel Gila
  • Mikhail Serkov
  • Mohamed Chebbi
  • Nicholas Cardo
  • Nick Holway
  • Nico Färber
  • Nina Loktionova
  • Olivier Byrde
  • Pascal Häussler
  • Pawel Bednarek
  • Pierre Berthier
  • Pierre Dubath
  • Raluca Hodoroaba
  • Ricardo Silva
  • Sadaf Alam
  • Samuel Fux
  • Sankar Martial
  • Sebastien Moretti
  • Silvan Hostettler
  • Sofiane Sarni
  • Steve Litster
  • Steven Armstrong
  • Thomas Kramer
  • Thomas Wuest
  • Thorsten Steenbock
  • Tristan Carel
  • Tyanko Aleksiev
  • Urban Borštnik
  • Valeri Markushin
  • Valerio Zanetti
  • Volker Flegel
  • Wolfgang Zipfel