19–22 Sept 2022
Paul Scherer Institute, Villigen, Switzerland
Europe/Zurich timezone

Fast analysis feedback with automated data processing pipelines at PETRA-III

21 Sept 2022, 13:50
20m
WHGA/Auditorium and online (Paul Scherer Institute, Villigen, Switzerland)

WHGA/Auditorium and online

Paul Scherer Institute, Villigen, Switzerland

Paul Scherrer Institute Forschungsstrasse 111 CH-5232-Villigen-PSI
Oral NOBUGS 2022

Speaker

Vijay Kartik (DESY)

Description

The high data rates of next-generation X-ray detectors coming into use at PETRA-III beamlines have triggered intense activity on the topics of `live' data processing and fast feedback to beamline users.

The vision is to provide users analyzed and reduced data in near-real time, which can be used to judge whether their data acquisition is producing meaningful output or not. This can in turn help them to tune their data acquisition or to stop and modify their current experiment/sample setup before continuing the experiment further. Fast analysis feedback is essential to keep the experimenter in the loop, able to make agile decisions based on the science without being overwhelmed by the large amounts of raw data being produced by the detectors.

We present MENTO, a data processing toolkit that is automatically triggered during data acquisition, and which remotely runs external data analysis software on-demand using the DESY high-performance computing (HPC) cluster, `Maxwell'. The processed results are transparently made available at the beamline so that users can immediately evaluate the experiment without having to manually handle any raw data. MENTO is set up to require no input from the users except to point to the desired analysis, and the entire processing pipeline is then managed automatically, including data input, access to the HPC cluster, job submissions to a batch processing scheduler, and result writing. MENTO integrates easily with the experiment control systems currently used at DESY, and is in production at a few PETRA-III beamlines already. At the coherence applications beamline P10 in particular, it is augmented with a graphical user interface for visualizing reduced and pre-processed data, giving the user direct real-time visual feedback about the integrity of the acquired data, potentially flagging beam damage to samples, and thus pre-empting acquiring unusable data, allowing users to make the most of their limited time at the beamline.

Email address of presenting author vijay.kartik@desy.de

Primary authors

Vijay Kartik (DESY) Michael Sprung (DESY) Anton Barty (Coherent Imaging Division, CFEL, DESY)

Presentation materials