Efficient use of HDF5 with high data rate x-ray detectors
Current and future x-ray detectors - like the Eiger detector developed and build at PSI - deliver data at sustained rates of several times 10Gbit/s. The HDF5 library and Nexus data format are widely accepted solutions for data storage at synchrotron and x-ray free electron light sources.The two day workshop and course will focus on the efficient storage of x-ray detector data at rates of 10-100Gbit/s and beyond in HDF5/Nexus files. The workshop will give a detailed and accurate presentation of the HDF5 library's current capabilities and limitations to write detector data at high rates. Planned or ongoing HDF5 developments and their applicability to data storage at light sources will be presented, too.
Two members of the HDF group will attend the workshop and will lead the course section which starts each day. In the afternoon the participants are encouraged to work on proof-of-concept implementations and on the workshop report.
Participants need a good understanding of the HDF5 data model and related concepts like datasets, groups, attributes and data types. Familiarity with the tools h5dump and h5view and some high level programming API to HDF5 like python's h5py module.
The course section will focus on advanced HDF5 features and library implementation like compression, chunking, filter, adding own filter routines, links and references, external storage, parallel compression and parallel writing, single writer multiple reader, low level file driver, ... . The course will focus on concepts, capabilities and limitations and will not present the HDF5 library's API in detail.
