Missing iir params in status frame.

[msilvafe]

We found when processing data at level3 that some wafers within some obs (according to @amaurea ~5% of satp3 data he used to map) is missing the iir_params information. Some examples are:

1. obsid: obs_1717850846_satp3_1111111 wafers: mv5, mv23
2. obsid: obs_1717908165_satp3_1111111 wafers: mv23

I looked at obs_1717908165_satp3_1111111 and confirmed that iir_params are present on all other wafers just missing on mv23. Then I went to the level2 data and confirmed it didn't exist there or in the status frame however fscale and enabled parameters related to the iir filter were present. So this seems like a failure where that info from pysmurf doesn't make it into the status frame.

[mhasself]

Can you easily check the TOD to see if the PSD shows the expected roll-off? It would be nice to confirm that, although the parameters are missing, the filter was both configured as usual and enabled.

[msilvafe]

Can you easily check the TOD to see if the PSD shows the expected roll-off? It would be nice to confirm that, although the parameters are missing, the filter was both configured as usual and enabled.

Yes the readout filter is still on based on the PSD:

[jlashner]

Hey @msilvafe, any chance there's a record of obs_ids that have failed preprocessing on satp1 due to this problem? If there is a cache of failed obs-ids that would be helpful for me to try to figure out under what conditions this was happening.

[mmccrackan]

@jlashner. The error did not seem to occur at all on the satp1 site-pipeline run that I just ran, only on the satp3 run. Here are the list of obs_ids from satp3 just in case.
iir_param_obs_id.txt

[ykyohei]

Confirmed that satp3 doesn't have this issue anymore after we stopped doing soft hammer. https://simonsobs.atlassian.net/wiki/x/AoAHL
But satp3 have this issue in about 800 (obs_id, band, wafer_slot). satp3_iir_missing.txt

[msilvafe]

Confirmed that satp3 doesn't have this issue anymore after we stopped doing soft hammer. https://simonsobs.atlassian.net/wiki/x/AoAHL But satp3 have this issue in about 800 (obs_id, band, wafer_slot). satp3_iir_missing.txt

It's on my todo list to work on a fix for this. Its a little bit more complicated because we don't want a blanket if iir missing then fill with default in case there's new failures in the future which we don't want to fill or we want to fill with different defaults so we need to implement something that checks that its in the time range where we've confirmed the source of the failure is known (soft hammer) and have confirmed the defaults to fill with. The proposed short + long term fix were documented by @mhasself in this issue: simonsobs/sotodlib#969 (comment)