Sep 19 – 22, 2022
Paul Scherer Institute, Villigen, Switzerland
Europe/Zurich timezone

Zocalo: a high-throughput data processing framework

Sep 21, 2022, 2:10 PM
WHGA/Auditorium and online (Paul Scherer Institute, Villigen, Switzerland)

WHGA/Auditorium and online

Paul Scherer Institute, Villigen, Switzerland

Paul Scherrer Institute Forschungsstrasse 111 CH-5232-Villigen-PSI
Oral NOBUGS 2022


Richard Gildea (Diamond Light Source Ltd)


Zocalo is a data processing system developed to support live analysis at Diamond Light Source across multiple science areas, including crystallography and cryo-EM. Zocalo has been designed to be low-latency and fault-tolerant, and is used to orchestrate a wide variety of data processing tasks, including fast X-ray centring data analysis, automated data reduction with xia2, combining multiple data sets with xia2.multiplex, user-instigated reprocessing via SynchWeb and providing live feedback for serial crystallography experiments (SSX).

Zocalo is made up of several main components. Services start in the background and wait for work comprising discrete short-lived tasks, for example finding diffraction spots on a single image, or inserting results into a database. In contrast, wrappers are used for longer-running tasks, such as data reduction of a diffraction experiment. Recipes describe the connections between services and wrappers, enabling highly flexible and dynamic data processing workflows. Messages are passed between components via an underlying message broker.

Recent improvements to Zocalo include support for the RabbitMQ message broker, and support for running services on Kubernetes, both of which help further improve the high-availability and fault-tolerance of the system. System metrics can be exported via Prometheus enabling monitoring of system performance and automatic alerting of potential problems with Grafana and Alertmanager.

Email address of presenting author

Primary author

Richard Gildea (Diamond Light Source Ltd)


Dr Markus Gerstel (Diamond Light Source Ltd) Nicholas Devenish (Diamond Light Source Ltd) Irakli Sikharulidze (Diamond Light Source Ltd) Dan Hatton (Diamond Light Source Ltd) Anna Horstmann (Diamond Light Source Ltd) Graeme Winter (Diamond Light Source Ltd)

Presentation materials