Cloud and Containers

Thursday 16 May 2019, 09:30 → 16:30 Europe/Zurich

Pharmazentrum Hörsaal 2 (University of Basel)

Martin Jacquot (University of Basel), Michele De Lorenzi (CSCS)

Playlist with all video presentations can be viewed here.

Introduction

The HPC landscape changed enormously over the last years; traditionally, it was led by computational physics, chemistry and other disciplines requiring huge processing performance and running on large number of nodes. Nowadays more and more scientific disciplines need computing resources and introduce new requirements for the HPC environments. Many of these new applications necessitate only a fraction of one compute node for a short time while others need more powerful nodes than those regularly available. At the same time, software development and distribution increasingly develop towards portable virtualization technologies required for research reproducibility, and scientists rely more and more on containers to run their experiments. Infrastructure as a service (IaaS) addresses many of the requirements needed by the researchers of the different scientific disciplines. In this context, OpenStack one of the leading open source IaaS platforms, support virtualized infrastructures, bare-metal infrastructures, GPU devices, I/O virtualization, and containers orchestration.

In this forum, we would like to explore how system architects and system managers can build HPC environment on top of IaaS infrastructures like OpenStack.

Key questions

 

• What are the differences between traditional HPC clusters and IaaS cloud based HPC clusters?
• What are the advantages and the limitations of HPC infrastructures built on top of OpenStack?
• How to handle big data in an OpenStack environment in term of scalability, confidentiality and performance.
• How to deploy containers (docker, singularity, …) on OpenStack
• What is the overhead of the virtualization, how does this affect the performance on the CPU, on the I/O or on the network. Which workloads/processes are affected?
• How to deploy and manage the containerized applications. Which automation tool fills the requirements, kubernetes, ansible, mesos. …?
• What are the new skills needed by the system administrators and the users to work on these environments?
• How to best share the knowledge between the different HPC centers?

 

09:30 Coffee and Registration

Welcome and Introduction

Conveners: Martin Jacquot (University of Basel), Michele De Lorenzi (CSCS)

Keynote Presentation. Deployment of the BioMedIT Infrastructure Based on OpenStack

sciCORE is deploying a secure IT infrastructure in Basel for the Swiss Personalized Health Network (SPHN) national initiative, in the context of the project BioMedIT. The purpose of this project is to support the personalized health research and setup the equipment providing the appropriate technology and fulfil the security requirements. In this presentation, we will introduce the constraints of BioMedIT nodes in term of functionality, interoperability and security, explain the choice of OpenStack, the architecture of the infrastructure and describe the technical management of the project.

Convener: Martin Jacquot (University of Basel)

Deployment of the BioMedIT Infrastructure Based on OpenStack.pdf

Martin Jacquot.png

Kubernetes for Biomedical Analysis

SIB Swiss Institute of Bioinformatics BioMedIT project is developing the computing infrastructure which will enable biomedical analyses on sensitive human data across multiple sites as part of the Swiss Personalized Health Network (SPHN). A key component of this work is to enable portable workflows which can take the analysis to the data. The research analyses being done vary widely from exploratory data analysis in interactive notebooks to machine learning. This presentation will focus on our experience assessing Kubernetes to support these biomedical workloads, and the benefits it provides to researchers in the community.

Convener: Kevin Sayers (University of Basel)

Kevin Sayers_01.png

Kevin Sayers_02.JPG

Kevin_Sayers_Kubernetes_For_Biomedical_Analysis.pdf

Sarus: An OCI-Compliant Container Engine for HPC

Sarus is a tool for HPC systems that provides the means to instantiate feature-rich containers from Docker images. It has been designed to address the unique requirements of HPC installations, such as high performance, security oriented to multi-tenant systems and integration with resource managers. Sarus leverages the Open Container Initiative (OCI) specifications to extend the capabilities of runc through dedicated hook programs in order to enable a variety of features. This session will also illustrate several use cases for OCI hooks in HPC and show how their usage can allow Sarus containers to achieve native performance on supercomputing systems.

Convener: Alberto Madonna (CSCS)

Alberto Madonna.JPG

Sarus Container Engine for HPC- hpc-ch Forum Cloud and Containers.pdf

12:00 Lunch and Networking

Site Visit

The participants will be split into two groups. Each group will visit in turn:

• The Imaging Core Facility
  https://www.biozentrum.unibas.ch/facilities/technology-platforms/technology-platforms-a-z/overview/unit/imcf/

• The NMR Spectroscopy Group of Sebastian Hiller
  https://www.biozentrum.unibas.ch/research/researchgroups/overview/unit/hiller/

Both facilities will present their infrastructure and services.

Site-visit_01.JPG

Site-visit_02.JPG

Bringing Cloud Accessibility to HPC Technology, a Use Case Driven Approach

In the recent years, the development of new scientific workflows and scientific community frameworks have brought new access and automation requirements to HPC centre. Whereas IaaS and PaaS is partially answering such requirements on a Cloud technology perspective, a significant work is required to obtain a comparable flexibility for HPC technology. In this talk, I will present a set of use cases perspectives and requirements which have driven new enhancements of HPC technology at CSCS. I will elaborate on the challenges and the developments that have been undergone at CSCS to enable Cloud accessibility to HPC technology.

Convener: Maxime Martinasso (CSCS)

CloudAccessForHPC.pdf

Maxime Martinasso.JPG

Benchmarking MPI Applications in Singularity Containers on Traditional HPC and Cloud Infrastructures

Singularity is a container solution that promises to both integrate MPI applications seamlessly and run containers without privilege escalation. These benefits make Singularity a potentially good candidate for the scientific high-performance computing (HPC) community. However, the performance overhead introduced by Singularity is unclear. In this work we will analyse the overhead and the user experience on both traditional HPC and cloud infrastructures.

Convener: Andrei Plamada (ETH Zurich)

Andrei-Plamada-Benchmarking-MPI-Applications-in-Singularity-Containers-on-Traditional-HPC-and-Cloud-Infrastructures-16-05-2019.pdf

Andrei Plamada.JPG

15:00 Coffee Break

Towards Reproducible Data Analysis Using Cloud and Container Technologies

Reproducing published experiments or extending existing data analysis pipelines is a challenge that is becoming increasingly prevalent in the wider research community. In this talk, an overview of the challenges faced by both research infrastructure providers and Science IT units are presented, along with best practices to improve the reproducibility of data analysis using cloud and container technologies.

Convener: Sergio Maffioletti (University of Zurich)

Sergio Maffioletti.JPG

Towards Reproducible Data Analysis Using Container Technologies.pdf

Using GPUs in OpenStack and OpenShift Environments

OpenStack and OpenShift are platforms for delivering infrastructure in a cloud operational model (self-service, scalable, pay-as-you-go). SWITCH operates SWITCHengines for IaaS and in conjunction with APPUiO an OpenShift container orchestration platform. There has been a growing demand for GPU resources, and SWITCHengines has both NVIDIA P100 and T4 GPUs for use. This talk examines the provisioning of GPU based VMs and Containers, and various use cases in research and education.

Conveners: Adrian Kosmaczewski (VSHN AG), Jens-Christian Fischer (SWITCH)

fischer_GPU on OpenStack_OpenShift_2019-05-16.pdf

1 Farewell and End of the Meeting

Farewell.JPG