In late November and early December, 2015, users noticed that bias images and short exposures of the sky were visibly noisier than usual. We contacted Peter Onaka and his team, and they quickly suggested that the problem might be some sort of cross-talk with the ion pump (a device which maintains the vacuum of the dewar, but which also features high voltages).
I checked the behavior of the camera by grabbing all BIAS frames taken between
Nov 01, 2015 = MJD 57331
and
Dec 12, 2015 = MJD 57368
I defined two rectangular areas in each frame:
Within each region, I computed simple statistics: the mean and standard deviation of pixel values, in raw counts. You can find all the measurements in an ASCII text file linked below. The two columns after "overscan" are the mean and stdev in the overscan region, and the two columns after "data" are the mean and stdev in the target region.
The graphs below show the behavior of the pixels in bias frames over the study period Nov 01, 2015 - Dec 12, 2015.
As you can see, there was period during which both the mean values and the stdev values were elevated: MJD 359 - 361 (= Dec 2 - Dec 5, 2015), and MJD 364.9-365.05 (= Dec 7-8, 2015) The stdev values are particularly useful for identifying periods of unusual behavior, as the mean values tend to drift on timescales of weeks.
Note that cycling power to HDI will cause the first image taken to show a characteristic pattern of bright diagonal lines -- it reminds me of Venetian blinds. The stdev of pixel values in such a "re-start" image are very high -- typically 50 to 60 counts.
Peter Onaka provided this guidance.
Symptom: Con was seeing bias frame pixel variations at a much higher level than normal. As much as 300 to 400 adu. Looking at the one amp fits file in ds9, we can see high and low single pixel variations (with some trailing) at ~17 and 25 pixels. Even though this frequency is too high for the ion pump "pulsing", I asked Con to turn it off with the console command: power ionpump off This got rid of the pickup (or reduced it significantly) and we turned it back on with the command: power ionpump on Con said the noise level went up slightly, but acceptable - he's going to proceed and see what happens through the night. Technical Background ------------------------------ Con and I both suspect that something might have changed recently but the experiment did affect the noise. We have seen pickup from a pulsing ion pump on other systems but HDI has not been affected (at least until now). The ion pump is a: Kernco Modion, titanium sputtering, typ. 0.5 liters/sec, starting pressure: <1E-3 Torr, output voltage 3500VDC You'll notice the "3500VDC output voltage" = this is the anode voltage that is "pulsing" when you see the LED display pulsing. We're always worried about noise pickup from this and it's also the motivation to want the pump on steady vs pulsing. Con said the vacuum and temperature graphs looked good for the past few days so we don't suspect a vacuum problem. The ion pump may have changed it EMI signature with the power off/on cycle, I can't be sure. Recomendations -------------------- I suggest leaving the ion pump on until there is another problem, then repeat the power off/on experiment.
We followed Peter's advice, cycling power to the ion pump on Dec 8, 2015 = MJD 365. After doing so, the noise values dropped back to their nominal values, and have remained there for the past four days.