We currently delete runs which are marked 'processed' (https://github.com/XENON1T/pax/blob/master/bin/delete_decider#L140) and have ended (DAQ stopped) more than two hours ago.
However, marking a run as processed happens when the trigger is done (https://github.com/XENON1T/pax/blob/master/pax/plugins/io/MongoDB.py#L455), while event writing to file might not be finished. This is particularly likely if the trigger just blazed through a few runs when it's catching up with the DAQ after crash. If this is for a run that stopped more than two hours ago, the run might be deleted while final event writing is still in progress.
There is currently a one-minute delay between deciding a run should be deleted and actually starting deletion. We can increase this to e.g. five minutes as a temporary workaround, but of course a more robust solution would be either
- Recording the time when the trigger is already done (perhaps we already do this), and base deletion based on this.
- Only mark a run as 'processed' when all eventbuilding has finished.
We currently delete runs which are marked 'processed' (https://github.com/XENON1T/pax/blob/master/bin/delete_decider#L140) and have ended (DAQ stopped) more than two hours ago.
However, marking a run as processed happens when the trigger is done (https://github.com/XENON1T/pax/blob/master/pax/plugins/io/MongoDB.py#L455), while event writing to file might not be finished. This is particularly likely if the trigger just blazed through a few runs when it's catching up with the DAQ after crash. If this is for a run that stopped more than two hours ago, the run might be deleted while final event writing is still in progress.
There is currently a one-minute delay between deciding a run should be deleted and actually starting deletion. We can increase this to e.g. five minutes as a temporary workaround, but of course a more robust solution would be either