Attempt to correct readout pattern for extended objects with continuum seems to result in a poor readout pattern correction (amp bias correction?)

[kfkaplan]

I am currently working on a target that is extended and has a lot of continuum and fills the IGRINS slit and the PLP's readout pattern removal code seems to overcorrect for the readout pattern (row amplifier bias? not really clear to me exactly what it is correcting). Below are some images of my target in the H band showing the overcorrection where the continuum is bright. I suspect stray light from the bright continuum hitting the detector beyond the order edges is causing this.

I tracked the issue down to the PLP's readout pattern removal code (in the qlook and reimplement_cr_reject branches). Specifically in
igrins/procedures/readout_pattern_helper.py the following function

    if mask is None:
        mask = np.zeros(d.shape, dtype=bool)
    else:
        mask = mask.copy()

    mask[:4] = True
    mask[-4:] = True

    p = [pipes[k] for k in ['p64_1st_order',
                            'col_wise_bias_c64',
                            'amp_wise_bias_r2',
                            'col_wise_bias']]

    return apply_pipe(d, p, mask=mask)

where amp_wise_bias_r2 appears to be the cause.

I tracked the "pipe" for amp_wise_bias_r2 to igrins/procedures/readout_pattern.py where it points to the following function

class PatternAmpP2(PatternBase):
    @classmethod
    def get(kls, d, mask=None):
        """
        returns a tuple of two 32 element array. First is per-amp bias values.
        The second is the [1,-1] amplitude for each amp.
        """
        d = np.ma.array(d, mask=mask).filled(np.nan)

        do = d.reshape(32, 32, 2, -1)
        av = np.nanmedian(do, axis=[1, 3])

        amp_bias_mean = np.mean(av, axis=1)
        amp_bias_amp = av[:, 0] - amp_bias_mean

        return amp_bias_mean, amp_bias_amp

    @classmethod
    def broadcast(kls, d, av_p):
        av, p = av_p
        k = p[:, np.newaxis] * np.array([1, -1])
        v = np.zeros((32, 32, 2, 1)) + k[:, np.newaxis, :, np.newaxis]
        avv = av.reshape(32, 1, 1, 1) + v
        return avv.reshape(2048, 1)

To try to resolve the issue, I tried to simply comment out amp_wise_bias_r2 in igrins/procedures/readout_pattern_helper.py like so:

    if mask is None:
        mask = np.zeros(d.shape, dtype=bool)
    else:
        mask = mask.copy()

    mask[:4] = True
    mask[-4:] = True

    p = [pipes[k] for k in ['p64_1st_order',
                            'col_wise_bias_c64',
                            #'amp_wise_bias_r2',
                            'col_wise_bias']]

    return apply_pipe(d, p, mask=mask)

and the result is a stacked H band frame that looks much better here when I blink it with what I had before:

It is very tempting to just leave amp_wise_bias_r2 turned off as a solution, but my main worry is that I am disabling pattern correction or bias subtraction that should be left on, so I am not sure this is the solution. My attempts to modify class PatternAmpP2(PatternBase) did not result in any improvement.

[ericasaw]

is this from the new pattern noise removal we are trying to implement for Gemini South data? or was there an old pattern removal section in the plp?

[kfkaplan]

This is from the old pattern removal section in the PLP. It is unrelated to the additional correction we are adding. (As a side note, the new additional correction is pretty much done, I was going to commit it when I ran into this issue and spent the day trying to solve it).

[ericasaw]

Do we need to have both running when the new correction is implemented?

[kfkaplan]

Yes I found we need both. My new correction gets rid of the small scale "wavy" pattern while the PLP's existing readout pattern correction gets rid of the larger scale pattern so both working together gives the best result.

[ericasaw]

Is this only a problem with this extended data you’re working with right now? or does it fail for other targets to?

…

On Mon, Jan 29, 2024 at 8:47 PM kfkaplan ***@***.***> wrote: Yes I found we need both. My new correction gets rid of the small scale "wavy" pattern while the PLP's existing readout pattern correction gets rid of the larger scale pattern so both working together gives the best result. — Reply to this email directly, view it on GitHub <#27 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AMC7LDLHKNESKN6R3QZ3X6LYRBNKJAVCNFSM6AAAAABCQPDA62VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSMJVHE3TGOBWGU> . You are receiving this because you commented.Message ID: ***@***.***>

[kfkaplan]

We will need to test more but I've only seen this in the extended data I am working with. Specifically it seems to only affect it if there is a lot of extended continuum. It doesn't affect 99.9% of targets, hence why we never really noticed it before.

[kfkaplan]

Similar issues seem to arise when only A (on) nods are reduced without the B (off) nod subtracted in the K-band. The K-band has stray light background that falls on the inter-order pixels. It seems the pattern removal is getting confused by the flux on the inter-order pixels and that is what might be causing this problem. In the above example, Uranus has bright enough continuum filling the slit, the light might be leaking into the inter-order pixels and since it is not subtracted in the OFF, the H-band where Uranus has a lot of bright continuum shows this issue.