Cloud and Containers

Europe/Zurich
Pharmazentrum Hörsaal 2 (University of Basel)

Pharmazentrum Hörsaal 2

University of Basel

Biozentrum Klingelbergstrasse 50/70 4056 Basel
Martin Jacquot (University of Basel), Michele De Lorenzi (CSCS)
Description

Playlist with all video presentations can be viewed here.

Introduction

The HPC landscape changed enormously over the last years; traditionally, it was led by computational physics, chemistry and other disciplines requiring huge processing performance and running on large number of nodes. Nowadays more and more scientific disciplines need computing resources and introduce new requirements for the HPC environments. Many of these new applications necessitate only a fraction of one compute node for a short time while others need more powerful nodes than those regularly available. At the same time, software development and distribution increasingly develop towards portable virtualization technologies required for research reproducibility, and scientists rely more and more on containers to run their experiments. Infrastructure as a service (IaaS) addresses many of the requirements needed by the researchers of the different scientific disciplines. In this context, OpenStack one of the leading open source IaaS platforms, support virtualized infrastructures, bare-metal infrastructures, GPU devices, I/O virtualization, and containers orchestration.

In this forum, we would like to explore how system architects and system managers can build HPC environment on top of IaaS infrastructures like OpenStack.

Key questions

 

  • What are the differences between traditional HPC clusters and IaaS cloud based HPC clusters?
  • What are the advantages and the limitations of HPC infrastructures built on top of OpenStack?
  • How to handle big data in an OpenStack environment in term of scalability, confidentiality and performance.
  • How to deploy containers (docker, singularity, …) on OpenStack
  • What is the overhead of the virtualization, how does this affect the performance on the CPU, on the I/O or on the network. Which workloads/processes are affected?
  • How to deploy and manage the containerized applications. Which automation tool fills the requirements, kubernetes, ansible, mesos. …?
  • What are the new skills needed by the system administrators and the users to work on these environments?
  • How to best share the knowledge between the different HPC centers?

 

Participants
  • Adrian Kosmaczewski
  • Alberto Madonna
  • Alex Upton
  • Alexander Kashev
  • Alfio Lazzaro
  • Andrei Plamada
  • Angelo Mangili
  • Ann Harding
  • Arnaud Fortier
  • Benedikt Thelen
  • Bernd Rinn
  • Carlos Fenoy
  • Christian Bolliger
  • Christoph Witzig
  • Diego Moreno
  • enrico favero
  • Filippo Stenico
  • Fotis Georgatos
  • Gianfranco Sciacca
  • Graeme Baillie
  • Guilherme Peretti-Pezzi
  • Guillermo Losilla
  • Hans-Rudolf Hotz
  • Henry Lütcke
  • Jani Heikkinen
  • Jaroslaw Surkont
  • Jens-Christian Fischer
  • Kachler Alexandre
  • Kevin Sayers
  • Konrad Jaggi
  • Kristin Gnodtke
  • Maffioletti Sergio
  • Marco Borelli
  • Markus Reinhardt
  • Martin Jacquot
  • Mathias Rechenmann
  • Matthias Kraushaar
  • Mattia Belluco
  • Maxime Martinasso
  • Maximilian Koch
  • Michael Rolli
  • Michele De Lorenzi
  • Nick Holway
  • Nico Färber
  • Nina Mujkanovic
  • Olivier Byrde
  • Pablo Fernandez
  • Patrick Zosso
  • Pawel Bednarek
  • Philippe Steiner
  • Pierre Berthier
  • Pierre Dubath
  • Ricardo Silva
  • Riccardo Murri
  • Roberto Fabbretti
  • Sabine Österle
  • Silvia Schaub
  • Szymon Gadomski
  • Sébastien Moretti
  • Teo Brasacchio
  • Theofilos Manitaras
  • Thomas Kramer
  • Thomas Wuest
  • Tim Robinson
  • Tobias Ternent
  • Wolfgang Zipfel
    • 09:30 10:00
      Coffee and Registration 30m
    • 10:00 10:15
      Welcome and Introduction
      Conveners: Martin Jacquot (University of Basel), Michele De Lorenzi (CSCS)
    • 10:15 11:00
      Keynote Presentation. Deployment of the BioMedIT Infrastructure Based on OpenStack

      sciCORE is deploying a secure IT infrastructure in Basel for the Swiss Personalized Health Network (SPHN) national initiative, in the context of the project BioMedIT. The purpose of this project is to support the personalized health research and setup the equipment providing the appropriate technology and fulfil the security requirements. In this presentation, we will introduce the constraints of BioMedIT nodes in term of functionality, interoperability and security, explain the choice of OpenStack, the architecture of the infrastructure and describe the technical management of the project.

      Convener: Martin Jacquot (University of Basel)
    • 11:00 11:30
      Kubernetes for Biomedical Analysis

      SIB Swiss Institute of Bioinformatics BioMedIT project is developing the computing infrastructure which will enable biomedical analyses on sensitive human data across multiple sites as part of the Swiss Personalized Health Network (SPHN). A key component of this work is to enable portable workflows which can take the analysis to the data. The research analyses being done vary widely from exploratory data analysis in interactive notebooks to machine learning. This presentation will focus on our experience assessing Kubernetes to support these biomedical workloads, and the benefits it provides to researchers in the community.

      Convener: Kevin Sayers (University of Basel)
    • 11:30 12:00
      Sarus: An OCI-Compliant Container Engine for HPC

      Sarus is a tool for HPC systems that provides the means to instantiate feature-rich containers from Docker images. It has been designed to address the unique requirements of HPC installations, such as high performance, security oriented to multi-tenant systems and integration with resource managers. Sarus leverages the Open Container Initiative (OCI) specifications to extend the capabilities of runc through dedicated hook programs in order to enable a variety of features. This session will also illustrate several use cases for OCI hooks in HPC and show how their usage can allow Sarus containers to achieve native performance on supercomputing systems.

      Convener: Alberto Madonna (CSCS)
    • 12:00 13:00
      Lunch and Networking 1h
    • 13:00 14:00
      Site Visit

      The participants will be split into two groups. Each group will visit in turn:

      • The Imaging Core Facility
        https://www.biozentrum.unibas.ch/facilities/technology-platforms/technology-platforms-a-z/overview/unit/imcf/

      • The NMR Spectroscopy Group of Sebastian Hiller
        https://www.biozentrum.unibas.ch/research/researchgroups/overview/unit/hiller/

      Both facilities will present their infrastructure and services.

    • 14:00 14:30
      Bringing Cloud Accessibility to HPC Technology, a Use Case Driven Approach

      In the recent years, the development of new scientific workflows and scientific community frameworks have brought new access and automation requirements to HPC centre. Whereas IaaS and PaaS is partially answering such requirements on a Cloud technology perspective, a significant work is required to obtain a comparable flexibility for HPC technology. In this talk, I will present a set of use cases perspectives and requirements which have driven new enhancements of HPC technology at CSCS. I will elaborate on the challenges and the developments that have been undergone at CSCS to enable Cloud accessibility to HPC technology.

      Convener: Maxime Martinasso (CSCS)
    • 14:30 15:00
      Benchmarking MPI Applications in Singularity Containers on Traditional HPC and Cloud Infrastructures

      Singularity is a container solution that promises to both integrate MPI applications seamlessly and run containers without privilege escalation. These benefits make Singularity a potentially good candidate for the scientific high-performance computing (HPC) community. However, the performance overhead introduced by Singularity is unclear. In this work we will analyse the overhead and the user experience on both traditional HPC and cloud infrastructures.

      Convener: Andrei Plamada (ETH Zurich)
    • 15:00 15:15
      Coffee Break 15m
    • 15:15 15:45
      Towards Reproducible Data Analysis Using Cloud and Container Technologies

      Reproducing published experiments or extending existing data analysis pipelines is a challenge that is becoming increasingly prevalent in the wider research community. In this talk, an overview of the challenges faced by both research infrastructure providers and Science IT units are presented, along with best practices to improve the reproducibility of data analysis using cloud and container technologies.

      Convener: Sergio Maffioletti (University of Zurich)
    • 15:45 16:15
      Using GPUs in OpenStack and OpenShift Environments

      OpenStack and OpenShift are platforms for delivering infrastructure in a cloud operational model (self-service, scalable, pay-as-you-go). SWITCH operates SWITCHengines for IaaS and in conjunction with APPUiO an OpenShift container orchestration platform. There has been a growing demand for GPU resources, and SWITCHengines has both NVIDIA P100 and T4 GPUs for use. This talk examines the provisioning of GPU based VMs and Containers, and various use cases in research and education.

      Conveners: Adrian Kosmaczewski (VSHN AG), Jens-Christian Fischer (SWITCH)
    • 16:15 16:30
      Farewell and End of the Meeting 15m