19–22 Sept 2022
Paul Scherer Institute, Villigen, Switzerland
Europe/Zurich timezone

High-Rate Data Acquisition, Streaming, and Processing at the EMBL PETRA III Beamlines

21 Sept 2022, 15:10
20m
WHGA/Auditorium and online (Paul Scherer Institute, Villigen, Switzerland)

WHGA/Auditorium and online

Paul Scherer Institute, Villigen, Switzerland

Paul Scherrer Institute Forschungsstrasse 111 CH-5232-Villigen-PSI
Oral NOBUGS 2022

Speaker

Marina Nikolova (EMBL)

Description

The EMBL beamlines, P14, TREXX, P13, P12, support macromolecular crystallography, SAXS and X-ray imaging experiments. A common denominator between these techniques is the usage of high-frame-rate megapixel detectors and cameras. In an optimal scenario the data that they produce would immediately undergo some preliminary analysis and provide feedback to the users, allowing them to make informed decisions about the sample, the beamline environment and the further progress of the experiment. We have designed our software system and computing and hardware infrastructure with these goals in mind placing a particular emphasis on robustness at high loads. In recent developments we introduced a multi-node data stream receiver structure enabling us to transfer, process and store data at the maximum frame rate of the DECTRIS EIGER 2 X 4M detector of 1120 frames per second in 8-bit mode. The consolidated results of the crystallographic analysis using DOZOR [1] are displayed in real-time in the GUI, MXCuBE [2], and are additionally used to generate live highlights around the reflections in the diffraction viewer, ADXV [3].
The backbone of the system is a 40Gbit InfiniBand, a petabyte parallel cluster file system BeeGFS storage and a collection of servers that take care of acquisition and experiment control, stream receiving, data processing, image tracking, etc. A concerted effort has been made to harmonize workflows and re-use computing infrastructure across experiment types. To this end, acquisition, data stream sending and receiving software has been developed for an X-ray imaging camera, PCO – emulating the architecture of our EIGER setup. The streamed frames are flat-field corrected and subsequently displayed to the user in real-time. Improvements to the system to achieve also a live tomographic reconstruction are in progress.

[1] Popov, A. N. & Bourenkov, G. (2016). DOZOR. ESRF, Grenoble, France
[2] Oscarsson, et al. J. Synchrotron Rad. (2019). 26, 393-405
[3] Arvai, A. (2015). The ADXV User Manual, https://www.scripps.edu/tainer/arvai/adxv/AdxvUserManual.pdf

Email address of presenting author nikolova@embl-hamburg.de

Primary authors

Co-authors

Mr Andrea d'Amato (EMBL) Dr Elizabeth Duke (EMBL) Ivars Karpics (European XFEL) Dr Jonas Albers (EMBL) Dr Kirill Kovalev (EMBL) Dr Stefan Fiedler (EMBL) Thomas R. Schneider (EMBL c/o DESY)

Presentation materials