Speaker
Description
Zocalo is a data processing system developed to support live analysis at Diamond Light Source across multiple science areas, including crystallography and cryo-EM. Zocalo has been designed to be low-latency and fault-tolerant, and is used to orchestrate a wide variety of data processing tasks, including fast X-ray centring data analysis, automated data reduction with xia2, combining multiple data sets with xia2.multiplex, user-instigated reprocessing via SynchWeb and providing live feedback for serial crystallography experiments (SSX).
Zocalo is made up of several main components. Services start in the background and wait for work comprising discrete short-lived tasks, for example finding diffraction spots on a single image, or inserting results into a database. In contrast, wrappers are used for longer-running tasks, such as data reduction of a diffraction experiment. Recipes describe the connections between services and wrappers, enabling highly flexible and dynamic data processing workflows. Messages are passed between components via an underlying message broker.
Recent improvements to Zocalo include support for the RabbitMQ message broker, and support for running services on Kubernetes, both of which help further improve the high-availability and fault-tolerance of the system. System metrics can be exported via Prometheus enabling monitoring of system performance and automatic alerting of potential problems with Grafana and Alertmanager.
| Email address of presenting author | richard.gildea@diamond.ac.uk |
|---|
