Containers for HPC

chaired by Derek Heinrich Feichtinger (Paul Scherrer Institut), Michele De Lorenzi (CSCS)
Thursday, 26 October 2017 from to (Europe/Zurich)
at Paul Scherrer Institut ( WHGA/001 Auditorium )
Paul Scherrer Institut (PSI), 5232 Villigen, Switzerland
Description Introduction

The earliest mentionings of Virtual Machine (VM) technology dates back to the 1960s. The much lighter container technologies began to appear in the early 2000s with FreeBSD Jail and Solaris Zones. The last few years have brought massive acceptance of containers when Docker introduced an easy workflow model and central registries for container creation and deployment. However, the use of these technologies within the HPC community has been limited due to Docker being mostly targeted at typical cloud deployment of standard applications with full isolation, and not for running huge collaborative HPC workloads.

An HPC container solution needs to be able to integrate and interface safely to essential services on an HPC system, e.g. the batch system, the shared parallel filesystems, and high speed networks.  Also, there still is the expectation of lower performance and higher complexity on the administrator side for integrating these technologies into HPC clusters. Recently, the topic has gained a higher interest in the community, since specialized HPC container or container-compatibility variants and tools have been developed.

In this forum we would like to discuss the current state of the art, the effects and the potential of these technologies within the field of HPC computing, and how we can leverage them in our environments already now.

Key questions
  • What use cases are there for containers and general virtualization technologies in HPC?

  • Container Technology: What are the differences between Docker, Shifter and Singularity?

  • What is the performance overhead of using container solutions vs. running HPC workloads on bare metal?

  • What is required to successfully use container or container-like solutions on HPC systems? What are important best practices and strategies to deploy containers efficiently on HPC systems? Are there container orchestration tools for these environments and how do they integrate in HPC scheduling systems?

  • What are the pros and cons for users and system administrators?

  • Development process: From containerized application development on a laptop to the deployment on HPC systems. Experiences.

  • Applications persistency: Are containers + virtual machines the key to application persistency and result reproducibility?

  • How mature is container technology today? How big is the risk of having to adapt the configuration recipes over time in order to conserve the execution environment?

  • Container security: Implications and different approaches to solving current security related problems.
Go to day
  • Thursday, 26 October 2017
    • 09:30 - 10:00 Coffee and registration
    • 10:00 - 10:15 Welcome and introduction
      Convener: Derek Feichtinger (PSI), Michele De Lorenzi (CSCS)
    • 10:15 - 11:00 Keynote Presentation: Versatile HPC Platforms with Containers
      The talk will present how container technologies can help to deploy more versatile HPC platforms and how they are helping scientists running on Piz Daint. From a technical standpoint, we will discuss why we started using containers, what are our current use cases and how we are contributing back to the Shifter community adding GPU support.
      Convener: Miguel Gila (CSCS)
      Material: picture unknown type file slides pdf file
    • 11:00 - 11:30 Pipeline Interoperability Using Containers in the Context of Personalized Health Research
      Research in personalized health, as for example within the upcoming Swiss Personalized Health Network (SPHN) or related research initiatives, has stringent requirements on data confidentiality, reproducibility and interoperability between the different University hospitals and academic compute centers of Switzerland. Container technologies like Docker and Singularity provide efficient, yet flexible, means of abstraction to accommodate data work flows under such requirements within existing IT compute and data infrastructures. In this talk, I will elaborate on the experience we have gained from a collaborative containerization / pipeline interoperability pilot project carried out within the framework of the Swissuniversity / e-science coordination team on a national scale with the participation of all major Swiss universities.
      Convener: Balazs Laurenczy (ETH Zurich)
      Material: picture unknown type file slides pdf file
    • 11:30 - 11:45 Shifter and Docker SLURM Integration
      The goal of this project is to deploy computing environments on Docker images in SLURM schedulers. When running HPC clusters, user codes often rely on several libraries and dependencies. However, each HPC cluster/system has its own environment, which makes performance portability as well as result reproducibility and repeatability almost impossible.
      For SLURM, this integration is done using Shifter; a prototype implementation that NERSC is developing and experimenting with as a scalable way of deploying containers in an HPC environment.
      Convener: Ricardo Silva (EPFL)
      Material: picture unknown type file slides pdf file
    • 11:45 - 12:15 A Data Science’s Perspective of the Use of Containers: Use Cases from Bioinformatics to Digital Humanities Projects
      We have used containers to address the following issues:
      - the integration of various libraries and framework with complicated setup
      - the rise of reproducibility issues in scientific studies
      - the need for data interoperability and safety
      Containers are versatile tools that could offer suitable solutions to address such data science issues. Through a series of use cases ranging from life science to digital humanities projects, we illustrate how we took advantage of the flexibility of containers to deliver innovative products to our users.
      Convener: Martial Sankar (SIB)
      Material: picture unknown type file slides pdf file
    • 12:15 - 13:15 Lunch and networking
    • 13:15 - 13:45 Status of Plans to Use Containers in the Worldwide LHC Computing Grid
      The WLCG collaboration is coordinating efforts by the LHC experiments to introduce use of containers in their computing models. This presentation reports on the current status and plans for the experiments workflows.
      Convener: Gianfranco Sciacca (University of Bern)
      Material: slides pdf file
    • 13:45 - 14:00 Community Development
      Convener: Michele De Lorenzi (CSCS)
      Material: picture unknown type filedown arrow
    • 14:00 - 14:05 Valeri Markushin Recognition Ceremony
      Material: slides unknown type file
    • 14:05 - 14:20 Coffee break
    • 14:20 - 15:45 Visit of SwissFEL
      Material: picture unknown type filedown arrow
    • 15:45 - 15:50 Farewell and end of the meeting
      Material: picture unknown type file