Automated Provisioning of Virtual Workstations and Servers for a Broad Audience of Researchers

• Roberto Fabbretti (University of Lausanne)

Containers and Virtualization for the HPC Cluster

• Nicolas Alejandro Kowenski (ETH Zurich)

A story on how we performed an extreme culture and technology shift. From a mix of technologies to a fully Kubernetes based infrastructure. Including virtualization with a cloud-native / cutting edge approach.

Sarus: Highly Scalable Docker Containers for HPC Systems

• Alberto Madonna (CSCS)

Sarus is a container engine for HPC systems that provides the means to instantiate high-performance containers from Docker images.It has been designed to address the unique requirements of HPC containers, such as integration with hardware accelerators, quick deployments at scale, security and permissions enforcement on multi-tenant hosts, and parallel filesystems. Sarus leverages the Open Container Initiative (OCI) specifications to extend the capabilities of a standard runtime through dedicated hook programs, implementing container customization at deployment time and without the user's intervention. This presentation will highlight how OCI hooks can enable portable, infrastructure-agnostic images to achieve native performance on top of HPC-specific devices such as GPUs and high-speed interconnects. Thanks to their standalone nature and standard interface, OCI hooks can be independently developed to target specific features, and can be configured according to the characteristics of particular host systems. The same container image can thus be used across the whole development workflow, from early tests on a personal workstation to deployments at scale on flagship systems, while benefiting from the advantages of each platform.

AiiDAlab – an Ecosystem for Developing, Executing, and Sharing Scientific Workflows

• Aliaksandr Yakutovich (EPFL)

Cloud platforms allow users to execute tasks directly from their web browser and are a key enabling technology not only for commerce but also for computational science. Research software is often developed by scientists with limited experience in (and time for) user interface design, which can make research software difficult to install and use for novices. When combined with the increasing complexity of scientific workflows (involving many steps and software packages), setting up a computational research environment becomes a major entry barrier. AiiDAlab is a web platform that enables computational scientists to package scientific workflows and computational environments and share them with their collaborators and peers. By leveraging the AiiDA workflow manager and its plugin ecosystem, developers get access to a growing range of simulation codes through a python API, coupled with automatic provenance tracking of simulations for full reproducibility. Computational workflows can be bundled together with user-friendly graphical interfaces and made available through the AiiDAlab app store. Being fully compatible with open-science principles, AiiDAlab provides a complete infrastructure for automated workflows and provenance tracking, where incorporating new capabilities becomes intuitive, requiring only Python knowledge.

Dynamic Provisioning of Storage Resources: A Case Study with Burst Buffers

• Maxime Martinasso (CSCS)

Complex applications and workflows needs are often exclusively expressed in terms of computational resources on HPC systems. In many cases, other resources like storage or network are not allocatable and are shared across the entire HPC system. By looking at the storage resources in particular, any workflow or application should be able to select both its preferred data manager and its required storage capability or capacity. To achieve such a goal, new mechanisms should be introduced. In this work, we present such a tool that dynamically provision a data management system on top of storage devices. We propose a proof-of-concept that is able to deploy, on-demand, a parallel filesystem across intermediate storage nodes on a Cray XC50 system. We show how this mechanism can be easily extended to support more data managers and any type of intermediate storage. Finally, we evaluate the performance of the provisioned storage system with a set of benchmarks.

Cloud Bursting: a First Experience

• Ricardo Silva (EPFL)

We will report on a proof-of-concept carried out by SCITAS using Google Cloud Platform (GCP) resources to analyze the state of the art and evaluate the capabilities available in GCP. We evaluated both the technical feasibility and the performance of the solutions from an HPC perspective (including performance benchmarks). We defined a semi-automatic procedure to deploy, in less than 10 minutes, a fully usable cluster. The following architectures were evaluated: - a cluster in the cloud, offering the same software stack as our bare-metal clusters (external elastic computing) - a hybrid approach with an on-site headnode and dynamic allocation of compute nodes in the cloud (hybrid elastic computing) - a hybrid approach with an on-site compute node using accelerators in the cloud (external compute resources)

Community Development

• Michele De Lorenzi (CSCS)
• Christian Bolliger (ETH Zurich)

Tool: MURAL (a digital workspace for visual collaboration)

Farewell and End of the Meeting