Sep 19 – 22, 2022
Paul Scherer Institute, Villigen, Switzerland
Europe/Zurich timezone

Integrated Real-Time Auto-Processing at Diamond Light Source

Sep 21, 2022, 2:30 PM
WHGA/Auditorium and online (Paul Scherer Institute, Villigen, Switzerland)

WHGA/Auditorium and online

Paul Scherer Institute, Villigen, Switzerland

Paul Scherrer Institute Forschungsstrasse 111 CH-5232-Villigen-PSI
Oral NOBUGS 2022


Jacob Filik (Diamond Light Source)


As data rates and experimental complexity increase it is critical that facilities reduce the burden of reduction and processing of raw experimental data for users. While this statement itself is clear and simple, the reality behind implementing generic real-time auto-processing is not.

Broadly the problem can be split into two categories: Infrastructure and User Experience. Infrastructure requirements include things like data and metadata storage and access, cross process communication between different systems, and access to High Performance Computing resources. User Experience deals with how can these separate systems come together to provide a flexible and usable system, but most critically – how do we give facility users the confidence that the data is processed correctly, and with full provenance, so that they will use it for real-time experimental decision making or as the basis for publication?

Here we present the system deployed for the Physical Sciences at the Diamond Light Source and show how technologies like HDF5 (with SWMR), message brokers (like ActiveMQ) and information management systems (like ISPyB), can be used to build a versatile system for generic real-time data processing.

Email address of presenting author

Primary author

Jacob Filik (Diamond Light Source)


Dr Dean Keeble (Diamond Light Source) Mr James Hall (Diamond Light Source) Mr Karl Levik (Diamond Light Source) Dr Peter Chang (Diamond Light Source) Richard Gildea (Diamond Light Source Ltd) Dr Satheesh Maheswaran (Diamond Light Source) Dr Tim Snow (Diamond Light Source)

Presentation materials