Reproducible High-Throughput HPC Workloads with AiiDA

• Giovanni Pizzi (EPFL)

Room: Akademie I The ever-growing availability of computing power and the sustained development of advanced computational methods present new challenges driven by the sheer amount of calculations and data to manage. In this talk I will present AiiDA (aiida.net), a robust open-source high-throughput infrastructure addressing the challenges arising from the needs of automated workflow management and data provenance recording. I will discuss the developments and capabilities required to reach sustained performance, with AiiDA supporting throughputs of tens of thousands processes/hour, each of them ranging from local processes to massive HPC jobs. I will also discuss how AiiDA automatically preserves and stores the full data provenance in a relational database making it queryable and traversable, thus enabling high-performance data analytics and reproducibility of the research.

AiiDA Lab – an Ecosystem for Developing, Executing and Sharing Scientific Workflows

• Kristjan Eimre (Empa)

Room: Akademie I Computational simulations constitute an integral part of modern science. Running scientific software, however, requires expert knowledge not only in the modelled scientific phenomena but often also in accessing and operating compute resources, programming, installing and compiling software, and other topics related to computation. In the presentation I will introduce AiiDA lab, an open source platform that provides the means to develop, execute and share computational science workflows as an intuitive web service available to anyone who knows how to use a web browser. AiiDA lab is built on top of AiiDA, an infrastructure that provides the automatic workflow engine and provenance tracking, and Jupyter, which powers the graphical user interfaces.

FirecREST: a RESTful API to HPC Systems

• Eirini Koutsaniti (CSCS)

Room: Akademie I As science gateways are becoming an increasingly popular digital interface for scientific communities, it is important for High-Performance Computing centers to provide a modern and externally accessible interface such as Web-enabled APIs. Such interface provides access to HPC center resources to allow scientific web portals to submit jobs or move data in and out the HPC center. This work presents the FirecREST API, a RESTful Web API infrastructure that allows scientific communities to access the various integrated resources and services available on HPC systems.

Real-Time Services for Large Volume Experiment-Data Analysis Utilizing Supercomputing and Cloud Technologies (SELVEDAS)

• Mei-Chih Chang (PSI)

Room: Akademie I The ongoing developments in accelerators, detectors and experiment automation is leading to a rapid growth of data generated during experiments. A viable solution is utilizing suitable infrastructures that allow additional remote high performance capacity for processing and analysis of data from the experimental facilities with larger data volumes and higher processing needs. The SELVEDAS project proposes a hybrid cloud infrastructure, offering scalable and extensible services for data management and analysis to Swiss academic users by leveraging high performance computing (HPC), storage, networking as well as cloud technologies and orchestration. The on-demand services perform as a highly efficient remote data processing system providing fast feedback and analysis with the long time storage and archival of petabytes of data.

Interactive HPC on Piz Daint with JupyterLab

• Maxime Martinasso (CSCS)

Room: Akademie I The last few months have seen a significant rise in the number of users accessing Piz Daint interactively via Jupyter notebooks, and we expect this trend to increase further in the future. We discuss some of the challenges faced in providing interactive computing in a multi-user, batch-oriented computing environment such as Piz Daint. We describe our centrally-maintained JupyterLab software stack and some of the tools we provide to enable customized workflows.