The earliest mentionings of Virtual Machine (VM) technology dates back to the 1960s. The much lighter container technologies began to appear in the early 2000s with FreeBSD Jail and Solaris Zones. The last few years have brought massive acceptance of containers when Docker introduced an easy workflow model and central registries for container creation and deployment. However, the use of these technologies within the HPC community has been limited due to Docker being mostly targeted at typical cloud deployment of standard applications with full isolation, and not for running huge collaborative HPC workloads.
An HPC container solution needs to be able to integrate and interface safely to essential services on an HPC system, e.g. the batch system, the shared parallel filesystems, and high speed networks. Also, there still is the expectation of lower performance and higher complexity on the administrator side for integrating these technologies into HPC clusters. Recently, the topic has gained a higher interest in the community, since specialized HPC container or container-compatibility variants and tools have been developed.
In this forum we would like to discuss the current state of the art, the effects and the potential of these technologies within the field of HPC computing, and how we can leverage them in our environments already now.
Key questions
- What use cases are there for containers and general virtualization technologies in HPC?
- Container Technology: What are the differences between Docker, Shifter and Singularity?
- What is the performance overhead of using container solutions vs. running HPC workloads on bare metal?
- What is required to successfully use container or container-like solutions on HPC systems? What are important best practices and strategies to deploy containers efficiently on HPC systems? Are there container orchestration tools for these environments and how do they integrate in HPC scheduling systems?
- What are the pros and cons for users and system administrators?
- Development process: From containerized application development on a laptop to the deployment on HPC systems. Experiences.
- Applications persistency: Are containers + virtual machines the key to application persistency and result reproducibility?
- How mature is container technology today? How big is the risk of having to adapt the configuration recipes over time in order to conserve the execution environment?
- Container security: Implications and different approaches to solving current security related problems.
