19–22 Sept 2022
Paul Scherer Institute, Villigen, Switzerland
Europe/Zurich timezone

Scientific Data Management at European XFEL.

20 Sept 2022, 18:46
20m
WHGA/Auditorium and online (Paul Scherer Institute, Villigen, Switzerland)

WHGA/Auditorium and online

Paul Scherer Institute, Villigen, Switzerland

Paul Scherrer Institute Forschungsstrasse 111 CH-5232-Villigen-PSI
Poster

Speaker

Janusz Malka (European XFEL GmbH)

Description

Data Management is essential to make research data easily accessible and usable. Important ingredients of data management include data policies and data workflows.
The data workflows are based on the policies which are implemented by defining a set of parameters stored in the metadata catalogue. The role of the metadata catalogue in relation to the data management services and underlying hardware solutions for the data storage systems will be presented. The architecture of the storage system consists of four layers, each addressing a different set of challenges. The first – online - is designed as a fast cache for the data generated directly at the scientific instruments during experiments. The second layer – offline - provides the performance for data processing during and after the beamtimes. The third layer - dCache disk pool - delivers the capacity to the system for long-term storage and the last one - tape archive - provides data safety and long-term archive. The storage system is able to accept 2PB/day of raw data, demonstrating the real capabilities with all sub-services being involved in this process. The storage system is connected to the high-performance computing cluster supporting remote data analysis and alternatively allows external users to export data outside of the European XFEL facility.

Email address of presenting author janusz.malka@xfel.eu

Primary authors

Bharathi Vanganuru (European XFEL GmbH) Carsten Patzke (Deutsches Elektronen-Synchrotron DESY) Christian Voss (Deutsches Elektronen-Synchrotron DESY) Christopher Youngman (European XFEL GmbH) Djelloul Boukhelef (European XFEL GmbH) Fabio Dall'Antonia (European XFEL GmbH) Gianpietro Previtali (European XFEL GmbH) Illia Derevianko (European XFEL GmbH) Janusz Malka (European XFEL GmbH) Janusz Szuba (European XFEL GmbH) Juergen Hannappel (Deutsches Elektronen-Synchrotron DESY) Kars Ohrenberg (Deutsches Elektronen-Synchrotron DESY) Kilian Schwarz (Deutsches Elektronen-Synchrotron DESY) Krzysztof Wrona (European XFEL GmbH) Luis Maia (European XFEL GmbH) Martin Gasthuber (Deutsches Elektronen-Synchrotron DESY) Maurizio Manetti (European XFEL GmbH) Mwai Karimi (Deutsches Elektronen-Synchrotron DESY) Nasser Alqudami (European XFEL GmbH) Philipp Schmidt (European XFEL GmbH) Steve Aplin (European XFEL GmbH) Thomas Kluyver (European XFEL GmbH) Tigran Mkrtchyan (Deutsches Elektronen-Synchrotron DESY) Uwe Ensslin (Deutsches Elektronen-Synchrotron DESY)

Presentation materials