Channels missing from bias step mapping

[eehealy]

I am looking for help debugging why so many channels are not responding to bias group mapping. I have two LF UFMs with NDFs looking out to the room. After running uxm_setup, there is tracking on 372 channels. The noise looks normal (~120pA/rt(Hz)) on all bands with some lower noise channels indicating some TESes are saturated, but most of these channels are not responding to bias group mapping / taking IV curves.

Here is a plot from take_bgmap. Do folks have advice on how to determine why so many channels are not responding?

Thanks!

[jlashner]

Generally the first thing to check is to look at TODs for channels that are not responding, something like:

rc = np.where(bsa.bgmap == -1)[0][10]
plt.plot(bsa.am.signal[rc])

If you could post some of those plots that would be very helpful

[msilvafe]

Sure looks like there's just not getting bias line signal going down to the UFM could be:

1. Open in UFM
2. Open in cold wiring
3. Open in warm wiring/unplugged/poorly plugged warm connector
4. RTM/CC not working right so not getting signal out.

I think can you check these things probably in order 4->1?

[eehealy]

4 - How would you typically test this? The green lights on the cryocard are on. I could try setting voltages on the bias lines then probe that the voltages are right at the connector to the fridge.

3 - Just confirmed that things appeared to be mated properly. We can remate connections after debugging #4

1-2 - We probed the electrical lines before plugging things into smurf, and everything looked continuous/un-shorted. It seems unlikely thought not impossible that we blew out 4 bias lines when plugged things into smurf and that wouldn't explain the low but nonzero response to bg mapping on BL-3. We can try probing again if other debugging avenues dry up.

[jlashner]

This doesn't answer your question.... but can you post the TODs / use the bias_steps.plot_steps function to show the response for the channels assigned to BG 3? Usually when there are only a couple of channels assigned to an otherwise non-functioning bias-line like that it might mean those channels are just crosstalk.

[eehealy]

seems to be the case! this is the only channel on BL3

[msilvafe]

@eehealy ya let's start with 4: best way is to do as you say, unplug db50 cable from the cryocard. Plug in the breakout board at the output of the cryocard. Command biases or run the bgmap command and check on an oscilloscope that you're seeing the commanded voltage or set of square wave pulses as expected on each problematic biasgroup (in this case, that seems like all of them).

[dpdutcher]

My thoughts go straight to a problem within the UFM, which I hope isn't the case! You said everything probed out as continuous before you plugged in SMuRF? If the perimeter bonds broke or if BL+ is shorted to BL-, things would probe out fine, but no TESs would respond. I've not seen either of those things develop over time, usually only the first time an LF UFM is tested.

• Which BL on each UFM is working? This is the second time you've cooled down with UFM-Lp3. Is that one behaving as you expect, or are there non-responding BL now that worked before?
• 372 tracking channels is pretty low, I'd expect something closer to 65*4*2 = 520. Any idea what's going on with the other channels? And how they're distributed across UFMS or mux chips?
• Channels that have lower noise (somewhere in the 30-60 pA/rtHz range) could be saturated TESs OR they could be uncoupled resonators -- each LF UFM has 79 uncoupled resonators, distributed differently across the bands/mux chips. Saturated TESs should respond to a bias tickle, uncoupled resonators won't.
• You said the channels aren't responding to bias group mapping / taking IV curves. Are you using an IV function that depends on a bias group map, such that those two tests are the same thing, or have you tried a separate IV function that doesn't rely on an input bias group map?

[eehealy]

Thanks for weighing in!

I would have expected to see bias group mapping to work for BLs 0, 1, 2, 3, 7, 8, 9, and we are only seeing 0 (Lp4) and 8 (Lp3). We are using one DC chain for both UFMs with a breakout "hydra" cable to the UFMs. Lp4 BLs 0 (5 on UFM MDM), 1, 2, 3 map to 0, 1, 2, 3 of the hydra cable. Lp3 BLs 0, 1, 2 (5 on UFM MDM), 3 map to 6, 7, 8, 9 with BL-0/6 having been open since arriving to UC. Though BL-8 was responding to bias steps in the last cooldown, the detectors weren't responding to IVs, which we didn't attempt to debug.

I wasn't sure whether the tracking issue was abnormal, but it might help explain why we had many fewer detectors compared to Princeton in the last cooldown. From uxm_setup, it looks like 465 resonators responded to a FR, so 75% yield for tracking does seem low.

Good point about the uncoupled resonators! Here are the noise histograms, based on the numbers it looks like the low noise ones are mostly uncoupled resonators.

I am using the sodetlib iv.py function take_iv passing in specific bias lines as an argument and running serially. Here is a snippet from running just with BL-0.

bias_groups = [0] iva = iv_op.take_iv( S, cfg, bias_groups=bias_groups, wait_time=0.01, bias_high=15, overbias_wait=0.5, bias_low=0, bias_step=0.025, overbias_voltage=19, cool_wait=20, high_current_mode=False, run_serially=True, serial_wait_time=30, run_analysis=True, show_plots=True, )

[eehealy]

Ok so we probed the UFMs again and all the bias lines are still reading as continuous. Interestingly, a short on BL-8 has returned, but no other changes since probing before plugging into smurf.

Next we checked the voltage output at smurf, and seem to have found some sort of issue. For this check, we put a 1k resistor across the BL pins, set smurf to put out 4V on that bias line, and probed across the resistor.

1. For BL-0, which has some IV response but is not optically responding, we saw a voltage drop (of 2V)
2. For BL-1, which is one of the bias lines that has no bias step response, we are not seeing any voltage drop across the 1k resistor.

Daniel just shared this document, so we will follow the steps in Section 6.3 (which is like what Max suggested earlier with the scope)

[msilvafe]

Sounds good Erin, let us know how the QC steps go but it does sound like you could have some smurf parts in need of repair.

[eehealy]

Just sent an email to the smurf hardware listserv for help debugging. Updating the results of the QC checks for completeness, though this seemingly isn't a sodetlib issue anymore.

As the table shows, the inline resistance of many of the bias lines is not as expected. In addition, there seem to be issues switching between high and low current mode. The full details of the debugging, including the commands we used, etc. can be found here (under "Readout")

[msilvafe]

We found an issue with that S.set_tes_bias_high_current_mode(bg) function and tried to address it in a new function in sodetlib sdl.set_current_mode() (link to code). Can you try using that instead and see if it solves your mode switching issues? I will talk w/ @dpdutcher about updating that QC g-doc with this function instead.

Also for measuring resistance I'm not sure you can read it from BL+ to BL- especially when there's voltage out of the DACs because this is a differential drive so measuring from one side to the other is through the DACs. I think you have to measure from one side of the CC to the other (SCSI side to DB50 side). Although interesting that you get some reasonable-ish numbers for some of the bias Ines.

[dpdutcher]

Max, you're referring to probing BL+ to BL- on the breakout board? Those are the instructions we gave for measuring both bias line resistance and high-to-low current ratio. But you might be right that if there's voltage out of the DACs, that measurement may be off...I can test that and see what happens.

[eehealy]

Thanks for the replies.

I made two changes -- I switched to the main branch and used the set_current_mode() function. That yielded different and better results for the QC check:

Fortunately, I could continue testing with these issues, since we weren't planning to use BL 6 or 8.

Of course, now I have encountered a new problem. After switching to main, setup_amps() is not working. I used all_off() after debugging before plugging in the DC cable to the fridge. I didn't realize that this didn't turn off the amplifiers, so when I ran get_amplifier_biases(), I was surprised they were reading out the normal voltages, but they seemed to be working OK. Then I ran setup_amps() and got this error:
[ 2024-01-22 17:41:28 ] Gate voltage adjustment would go out of range (-2.0, 0). Unable to change hemt1_drain_current to desired value [ 2024-01-22 17:41:28 ] Failed determining hemt1 gate voltage

Not sure if this has to do with switching branches / config files or what. Let me know if I should start a new discussion for this issue..

[msilvafe]

We were seeing this on the lat & sat when the cables weren't seated all the way in. Can you double check that. Are you using the jackscrews/posts?

[eehealy]

huzzah! the connector to the fridge barely moved but using the jackposts resolved the amp issue.

After moving to the new main branch, running uxm_setup again, things are looking better. Here is a bias group map:

We know that BL-8 has issues, so I didn't include that in the bg mapping. Not sure what is going on with BL-2, but the IVs look ok on 0, 1, 3, 7 and 9 so we are moving forward with testing.

Thanks all! We can close out this discussion.