Reference pixels are not populated in awesimsoss

[nespinoza]

Hey @hover2pi,

I'm not sure this is expected behavior or not, but the reference pixels are not populated in awesimsoss. This has as a consequence that the RefPixelStep in the JWST pipeline does not do anything, and thus the odd/even pattern (i.e., the horizontal strips in the image below) remain in the data:

It might also be that this pattern remains nonetheless (reference pixel correction or not), but I pressume that step would at least mitigate its impact a bit.

I'm not sure what the solution is really (going to tag @KevinVolkSTScI here). Populating them perhaps would be nice by forcing no signal from the PSF gets into those pixels (in order to simulate their "insensibility to light"), but that I pressume would involve playing around with the lab detector data used to generate these detector effects to begin with.

Cheers
Néstor

[KevinVolkSTScI]

These stripes are not the even/odd effect, at least not if you mean the obvious bands in the image above that are ~10 pixels wide in y. The even/odd effect is really an even to odd pixel effect in the y pixel numbers. (Of course, maybe I just do not see it in the above image if it is really at pixel row level...). I have seen such banding in images due to aliasing in the image display, so you might check that out. If you really see even/odd row effects in the data we have to find where these come from.

If we want to get a bit more fancy we can simulate the detector noise in the ramps or use real darks as a base signal as Mirage does. But do we want to do this? Using real darks gives the bad pixel properties but the number of such darks available is limited. Using simulated darks takes some time and does not fully reproduce the real noise and there are no bad pixels.

[nespinoza]

Hey @KevinVolkSTScI --- I think you are right about my plot, which is not the optimal way of showing this. Here are 10 adjacent vertical cuts along the spatial direction (centered around column 1000) that better show the even/odd effect:

A better close up to the pattern:

I'll take a more detailed look at this next week. I have all the pipeline products, so can directly check where this comes from --- but this pattern also shows up on the raw (*uncal.fits) data. The product I'm showing here is after doing the dark-frame correction in Detector 1 (with all the magic before that step already applied).

N.

[KevinVolkSTScI]

There is some issue with the values being -550 or so on the average. Something is being subtracted wrongly. I do not remember what the code is doing, so I have to look at the input _uncal file to try to see what is going on. It might just be a mis-match in the pedestal that is causing both the offset and the row to row variation.

With the reference pixels all set to zero in the simulation the pipeline even/odd correction will give no result if the pedestal subtraction is OK. The first thing to do is look at the output of the superbias subtraction step and then the output of the reference pixel subtraction step to see where the effect is coming from. If this structure is in the uncal files it has to come from either the pedestal or the flat field, if awesimsoss uses a flat. But if this structure is in the images after the superbias subtraction step it will not be changed by the reference pixel correction step as that should do nothing. But you also have to verify that the reference pixels actually have no signal after the pedestal subtraction.

[nespinoza]

Hi @KevinVolkSTScI --- you are totally right, there is an evident offset. Indeed, the plot I showed above is the super-bias corrected frame. So the mismatch I believe has to be between the superbias the pipeline uses and the one that is being injected into awesimsoss.

In fact, when I plot the superbias being used by the pipeline (jwst_niriss_superbias_0120.fits) and the uncal file simulated by awesimsoss, this is what I see:

And --- you guessed it --- the difference between both is about ~550 counts.

@hover2pi, thoughts on this?

[KevinVolkSTScI]

OK so this means is that we need to revise the code to use the current pipeline reference file for the pedestal. That should be relatively straightforward to do.

[hover2pi]

Yup, this has been a TODO for me for a while (#68 ). Both the pedestal and the photometric scaling are wrong in awesimsoss because I have the reference files (incorrectly) hardcoded. I'll open up a PR now to tie the reference files to the jwst repo. The only issue is that it's been difficult (near impossible?) to get Travis to install the pipeline so the code can be tested.

[nespinoza]

As I wrote on the comment on that PR, let me know when you push the changes --- I can give it a quick validation/testing with the simulations I'm working with. I have an iPython notebook ready to pass any dataset through each step of the pipeline one by one that might be handy for this.