Containers for HPC

Thursday 26 Oct 2017, 08:00 → 17:00 Europe/Zurich

WHGA/001 Auditorium (Paul Scherrer Institut)

Derek Heinrich Feichtinger (Paul Scherrer Institut), Michele De Lorenzi (CSCS)

Introduction

The earliest mentionings of Virtual Machine (VM) technology dates back to the 1960s. The much lighter container technologies began to appear in the early 2000s with FreeBSD Jail and Solaris Zones. The last few years have brought massive acceptance of containers when Docker introduced an easy workflow model and central registries for container creation and deployment. However, the use of these technologies within the HPC community has been limited due to Docker being mostly targeted at typical cloud deployment of standard applications with full isolation, and not for running huge collaborative HPC workloads.

An HPC container solution needs to be able to integrate and interface safely to essential services on an HPC system, e.g. the batch system, the shared parallel filesystems, and high speed networks.  Also, there still is the expectation of lower performance and higher complexity on the administrator side for integrating these technologies into HPC clusters. Recently, the topic has gained a higher interest in the community, since specialized HPC container or container-compatibility variants and tools have been developed.

In this forum we would like to discuss the current state of the art, the effects and the potential of these technologies within the field of HPC computing, and how we can leverage them in our environments already now.

Key questions

• What use cases are there for containers and general virtualization technologies in HPC?

• Container Technology: What are the differences between Docker, Shifter and Singularity?

• What is the performance overhead of using container solutions vs. running HPC workloads on bare metal?

• What is required to successfully use container or container-like solutions on HPC systems? What are important best practices and strategies to deploy containers efficiently on HPC systems? Are there container orchestration tools for these environments and how do they integrate in HPC scheduling systems?

• What are the pros and cons for users and system administrators?

• Development process: From containerized application development on a laptop to the deployment on HPC systems. Experiences.

• Applications persistency: Are containers + virtual machines the key to application persistency and result reproducibility?

• How mature is container technology today? How big is the risk of having to adapt the configuration recipes over time in order to conserve the execution environment?

• Container security: Implications and different approaches to solving current security related problems.

Coffee and registration

Welcome and introduction

Conveners: Derek Feichtinger (PSI), Michele De Lorenzi (CSCS)

Keynote Presentation: Versatile HPC Platforms with Containers

The talk will present how container technologies can help to deploy more versatile HPC platforms and how they are helping scientists running on Piz Daint. From a technical standpoint, we will discuss why we started using containers, what are our current use cases and how we are contributing back to the Shifter community adding GPU support.

Convener: Miguel Gila (CSCS)

picture IMG_9235.jpg

slides MiguelGila_Versatile-HPC-Platforms-with-Containers_20171026.pdf

Pipeline Interoperability Using Containers in the Context of Personalized Health Research

Research in personalized health, as for example within the upcoming Swiss Personalized Health Network (SPHN) or related research initiatives, has stringent requirements on data confidentiality, reproducibility and interoperability between the different University hospitals and academic compute centers of Switzerland. Container technologies like Docker and Singularity provide efficient, yet flexible, means of abstraction to accommodate data work flows under such requirements within existing IT compute and data infrastructures. In this talk, I will elaborate on the experience we have gained from a collaborative containerization / pipeline interoperability pilot project carried out within the framework of the Swissuniversity / e-science coordination team on a national scale with the participation of all major Swiss universities.

Convener: Balazs Laurenczy (ETH Zurich)

picture IMG_9238.jpg

slides BalazsLaurenczy_PipelineInteroperabilityUsingContainers_20171026.pdf

Shifter and Docker SLURM Integration

The goal of this project is to deploy computing environments on Docker images in SLURM schedulers. When running HPC clusters, user codes often rely on several libraries and dependencies. However, each HPC cluster/system has its own environment, which makes performance portability as well as result reproducibility and repeatability almost impossible.
For SLURM, this integration is done using Shifter; a prototype implementation that NERSC is developing and experimenting with as a scalable way of deploying containers in an HPC environment.

Convener: Ricardo Silva (EPFL)

picture 02_Ricardo_Silva.jpeg

slides RicardoSilva_Shifter-And-Docker-SLURM-Integration_20171026.v2.pdf

A Data Science’s Perspective of the Use of Containers: Use Cases from Bioinformatics to Digital Humanities Projects

We have used containers to address the following issues:
- the integration of various libraries and framework with complicated setup
- the rise of reproducibility issues in scientific studies
- the need for data interoperability and safety
Containers are versatile tools that could offer suitable solutions to address such data science issues. Through a series of use cases ranging from life science to digital humanities projects, we illustrate how we took advantage of the flexibility of containers to deliver innovative products to our users.

Convener: Martial Sankar (SIB)

picture Martial_Sankar.jpeg

slides Sankar_HPCEvent_171024.pdf

12:15 Lunch and networking

Status of Plans to Use Containers in the Worldwide LHC Computing Grid

The WLCG collaboration is coordinating efforts by the LHC experiments to introduce use of containers in their computing models. This presentation reports on the current status and plans for the experiments workflows.

Convener: Gianfranco Sciacca (University of Bern)

slides Sciacca_Gianfranco_Status_of_Plans_to_Use_Containers_in_the_Worldwide_LHC_Computing_Grid_20171026.pdf

Community Development

Convener: Michele De Lorenzi (CSCS)

picture 02_Community_Development.jpeg IMG_9236.jpg

Valeri Markushin Recognition Ceremony

slides IMG_9262.jpg

14:05 Coffee break

Visit of SwissFEL

picture

• IMG_9273.jpg
• IMG_9274.jpg
• IMG_9275.jpg
• IMG_9276.jpg
• SwissFEL_Presentation.png

Farewell and end of the meeting

picture IMG_9281.jpg