hdf5_schema.md

HDF5 Schema

This document describes the current ScintiPix HDF5 output structure.

It covers:

• simulation output written by scintipix
• optical transport output written by src/optics/OpticalTransport.py

The current intensifier module consumes this transport output in memory. It can also optionally write a separate persisted intensifier-stage HDF5 dataset. A downstream Timepix sensor stage can also write a persisted sensor HDF5 dataset.

This is a reference for the current writer schema. It does not describe legacy aliases or older ad hoc outputs.

Output Layout

Simulation output is written under the simulatedPhotons/ stage.

Typical paths:

• <run_root>/simulatedPhotons/photon_optical_interface_hits.h5
• <run_root>/simulatedPhotons/photon_optical_interface_hits_<subrun>.h5

Optical transport output is written under the transportedPhotons/ stage.

Typical paths:

• <run_root>/transportedPhotons/photons_intensifier_hits.h5
• <run_root>/transportedPhotons/photons_intensifier_hits_<subrun>.h5

Optional intensifier output is written under the sensor/ stage.

Typical paths:

• <run_root>/sensor/intensifier_output_events_<subrun>.h5
• <run_root>/sensor/timepix_hits_<subrun>.h5

Simulation Datasets

Simulation HDF5 files contain these datasets:

• /primaries
• /secondaries
• /photons

Selection semantics:

• /primaries contains primaries that created at least one secondary in the scintillator volume.
• /secondaries contains secondaries linked to at least one detected optical-interface photon hit.
• /photons contains one row per detected optical-interface photon hit.

/primaries

Fields:

• gun_call_id
• primary_track_id
• primary_species
• primary_x_mm
• primary_y_mm
• primary_energy_MeV
• primary_interaction_time_ns
• primary_created_secondary_count
• primary_generated_optical_photon_count
• primary_detected_optical_interface_photon_count

Notes:

• gun_call_id is the Geant4 event ID.
• primary_track_id is the event-local Geant4 track ID of the primary.
• primary_interaction_time_ns is the first recorded scintillator interaction time for the primary in the Geant4 global time frame and is written as NaN when no such interaction time was recorded. Source creation and pulse timing values are used internally to set the Geant4 primary vertex time; they are not persisted in /primaries.
• The three *_count fields summarize activity attributed to the primary ancestry inside the scintillator.

/secondaries

Fields:

• gun_call_id
• primary_track_id
• secondary_track_id
• secondary_species
• secondary_origin_x_mm
• secondary_origin_y_mm
• secondary_origin_z_mm
• secondary_origin_energy_MeV
• secondary_end_x_mm
• secondary_end_y_mm
• secondary_end_z_mm

Notes:

• secondary_track_id is the event-local Geant4 track ID of the secondary.
• secondary_end_*_mm is written as NaN when no usable end position was recorded for that secondary.

/photons

Fields:

• gun_call_id
• primary_track_id
• secondary_track_id
• photon_track_id
• photon_creation_time_ns
• photon_origin_x_mm
• photon_origin_y_mm
• photon_origin_z_mm
• photon_scint_exit_x_mm
• photon_scint_exit_y_mm
• photon_scint_exit_z_mm
• optical_interface_hit_x_mm
• optical_interface_hit_y_mm
• optical_interface_hit_time_ns
• optical_interface_hit_dir_x
• optical_interface_hit_dir_y
• optical_interface_hit_dir_z
• optical_interface_hit_pol_x
• optical_interface_hit_pol_y
• optical_interface_hit_pol_z
• optical_interface_hit_energy_eV
• optical_interface_hit_wavelength_nm

Notes:

• /photons stores one row per detected optical-interface crossing.
• photon_creation_time_ns and optical_interface_hit_time_ns share the same Geant4 event-local global-time basis.
• photon_scint_exit_*_mm is written as NaN when no scintillator-exit crossing was recorded for that photon.
• The optical_interface_hit_* fields capture position, time, direction, polarization, energy, and wavelength at the optical-interface crossing.

Optical Transport Dataset

Transport HDF5 files contain:

• copied /primaries
• copied /secondaries
• /transported_photons

/transported_photons

Fields:

• source_photon_index
• gun_call_id
• primary_track_id
• secondary_track_id
• photon_track_id
• intensifier_hit_x_mm
• intensifier_hit_y_mm
• intensifier_hit_z_mm
• reached_intensifier
• in_bounds

Notes:

• source_photon_index is the row index in the source /photons dataset used for transport.
• intensifier_hit_*_mm is written as NaN when the photon misses the lens or sensor in transport.
• reached_intensifier is True when transport produced a finite hit on the intensifier plane.
• in_bounds is True when the hit falls inside the configured intensifier input screen. If no input screen is defined, reached hits are treated as in-bounds by definition.

Transport File Attributes

Transport HDF5 files also write root-level attributes describing provenance and transport assumptions.

Common attributes:

• source_hdf5
• run_id
• lens_name
• lens_zmx_path
• object_plane
• optical_interface_represents
• transport_engine
• transport_chunk_rows
• transport_chunk_target_mib
• generated_utc

Optional attributes:

• lens_smx_path
• intensifier_model
• intensifier_input_screen_defined
• intensifier_input_screen_diameter_mm
• intensifier_input_screen_center_mm
• intensifier_input_screen_magnification
• intensifier_input_screen_coordinate_frame

Notes:

• intensifier_input_screen_defined is False when no input screen geometry is configured.
• When an input screen is configured, the related diameter, center, magnification, and coordinate-frame attributes are also written.

Optional Intensifier Output Dataset

When intensifier.write_output_hdf5 is true, the intensifier pipeline also writes a standalone HDF5 file under sensor/.

Intensifier output HDF5 files contain:

• copied /primaries
• copied /secondaries
• /intensifier_output_events

/intensifier_output_events

Fields:

• source_photon_index
• gun_call_id
• primary_track_id
• secondary_track_id
• photon_track_id
• output_x_mm
• output_y_mm
• output_time_ns
• signal_amplitude_arb
• total_gain
• wavelength_nm

Notes:

• /intensifier_output_events stores one row per final in-memory intensifier output event.
• source_photon_index links the event back to the source /photons row used by optical transport.
• signal_amplitude_arb is the current sensor-agnostic output amplitude from the phosphor stage.
• total_gain is the combined gain after the simplified dual-stage MCP model.

Intensifier Output File Attributes

Common attributes:

• source_hdf5
• transport_hdf5
• run_id
• intensifier_model
• generated_utc

Timepix Sensor Output Dataset

The Timepix sensor pipeline writes a standalone HDF5 file under sensor/.

Timepix output HDF5 files contain:

• copied /primaries
• copied /secondaries
• /timepix_hits

/timepix_hits

Fields:

• gun_call_id
• primary_track_id
• secondary_track_id
• x_pixel
• y_pixel
• time_of_arrival_ns
• time_over_threshold_ns
• contribution_count

Notes:

• /timepix_hits stores one row per final merged Timepix pixel hit.
• gun_call_id, primary_track_id, and secondary_track_id are currently representative IDs from the first contributing intensifier event in that merged pixel hit.
• time_of_arrival_ns is currently written as 0.0 for every row. The current Timepix stage uses intensifier event times internally for merge and dead-time behavior, but does not yet persist a calibrated downstream ToA.
• time_over_threshold_ns is currently a ToT-like proxy derived from phosphor-stage signal_amplitude_arb and clipped by the configured max_tot_ns.
• contribution_count records how many mapped intensifier events were merged into the final pixel-hit row.

Timepix Output File Attributes

Common attributes:

• source_hdf5
• transport_hdf5
• run_id
• intensifier_model
• sensor_model
• generated_utc

Schema Sources

The current writer schema is defined in:

• sim/include/structures.hh
• sim/src/SimIO.cc
• src/optics/OpticalTransport.py
• src/common/hdf5_schema.py