This note collects information concerning the noise problems that plagued HDI for several days in March, 2016. It is possible that this same issue caused the high noise state seen for prolonged periods of time during Dec 2015 - Jan 2016, but that is not yet established.
Under ordinary circumstances, the readout noise in HDI images has a root-mean-square (RMS) about 5 or 6 counts. One can easily measure it by looking at the overscan columns of the chip, even in ordinary target or flatfield images. I have written scripts to measure the values of pixels in the region defined as follows:
start_row = 10
number of rows = 4130
start_col = 4100
number of cols = 40
In March, 2016, the noise had its ordinary properties at the start of the month. The graph below shows time on the horizontal axis: JD - 2457000 = 449 corresponds to UT March 1. The vertical axis shows the standard deviation from the mean in units of counts; look only at the red symbols, which correspond to the overscan region.
The noise jumps up from its nominal value of about 6 to a new, high value of about 30 counts in the middle of the night of JD 457 = Mar 9. The first image showing this high-noise state is
http://hdiserver.kpno.noao.edu/ds/hdi/c7457t0023o/
taken at 2016-03-10T04:23:20.
The noise remained in this high state for the next several nights, as shown in the graph below.
The noise does not only increase the RMS values of pixels -- it has a clear periodic signature that suggests electronic interference. Below are sample cutouts of two images taken on the night that the noise appeared: one before, one after. Each image is 400 x 400 pixels, zoomed in by a factor of two. The contrast between pure black and pure white pixels is set to 300 counts.
This is a 5-second, R-band image of the night sky.
This is a 5-second, V-band image of the night sky.
The first clue was provided by Flynn Haase, during the afternoon of Friday, Mar 11, 2016 = JD 459. Flynn took a series of bias frames with the telescope moved to different positions. He found that in one position, the high noise disappeared:
I started taking positional biases. I took 5 biases at zenith - noise was present; I then went far north (mirror cover park) and took another 5 - and the noise *WENT AWAY*; I went back to zenith and took 5 more - noise was back; then back again to mirror cover park - NO noise!Now, this is *very* preliminary, and based only on the jpg quickviews in the datastore, but this seems to be the start of something here.
That night, the observer, Nathalie Haurberg, and her student, Izak, took many bias images while moving the telescope to different positions. They confirmed that when the telescope was pointing to the North (as in a mirror-cover park), the noise went back to its low value; but at all other positions, the noise was high.
The following night, Mar 12, 2016 = JD 460, Nathalie and Izak continued their tests. They opened the HDI enclosure to inspect the location of cable inside as the telescope moved. In Nathalie's words,
So, after Peter's inquiry about another photo of the control box in the "low-noise" orientation, Izak (my student) and I opened up the box for the photo. I left Izak in the dome to watch what happened in the box as we moved the telescope from zenith to the mirror cover park position (which is from a high-noise to low-noise state over the last few days, generally). He witnessed no motion of the interior cables except for the single braided ground cable which Flynn referred to in his previous email. That cable moved significantly. In the zenith park that cable appeared to be totally free floating (which was in itself alarming as that is not a good place for a ground cable to be). In the mirror cover park position that cable appeared to be lightly touching the side of the aluminum box. When the biases in both states were examined carefully, it was clear that the repeating noise was much lower in the mirror cover park state, but still slightly present. This led us to conclude that this almost certainly was a grounding issue where, in the free floating position there was no ground, and there was a tenuous grounding to the box in the mirror cover park.The braided ground cable that appeared to be the culprit is fed through a wide hole at the top of the box that is then connected to a ground terminal between the filter wheel and camera box. After consulting briefly with John Salzer, we decided to perform a controlled experiment. We pulled the cable out through that hole (as it was connected to nothing and would disturb nothing else in the box to do so). We then connected it to a common ground with the other ground terminals and Voila! no more noise. We then disconnected the cable from the ground altogether and the noise was back to what it has been. It appears as though this cable serves to connect all the ground terminals as a common ground. It apparently either broke at some point or became disconnected from its proper terminal. Since pulling the cable out resulted in us not having to touch with anything INSIDE the control box, we chose to keep it that way and have connected that cable from the exterior ground terminal to the interior edge of the box (as that seems to be the same electric ground as the other terminals).
The result: EVERYTHING IS WORKING! The strange ringing noise is gone and all seems well with the camera, telescope, and everything in between. We currently have the wire just slipped in between the exterior plates and the interior "truss" structure of the box and taped into place. This is likely not the way we would like to leave it permanently, but it is currently working and we are taking data. We plan to stay here until dawn and get as many photons as possible. We will be up in the morning to work with Flynn - but it appears our noisy culprit is identified!
For the next few nights (at least), the noise has remained in the nominal low state.
E-mail discussions initially focused on a large, braided cable which connected the HDI electronics box (at left side of picture below) to the power supply (at right side of picture below). This "braided cable" is marked with yellow circles in the picture below. The braided cable was observed to shift position as the telescope moved.
The image below was taken while the telescope was at mirror-cover park, pointing north: a low-noise position.
Image taken while telescope pointing north: low noise
Over the next two days, however, suspicion fell on a smaller, grounding wire which is near the top of the HDI enclosure. The two black-and-green boxes in the figure above mark the positions at which it comes into the enclosure (top green box) and its free-floating terminal (lower green box).
Note, for example, the change in the appearance of the grounding wire when the telescope was moved to zenith park, a high-noise location:
Image taken while telescope pointing at zenith: high noise
The lower end of the grounding wire disappears in this second photograph -- it was swinging freely as the telescope moved, and in this position, was hidden behind other items inside the enclosure.
The hypothesis was that this lower terminal end of the grounding wire was sometimes touching items, sometimes not touching items --- and that caused the noise to appear or disappear.
So, on Mar 13, the observers pulled this grounding wire up through a hole in the top of the enclosure, so that it no longer entered the enclosure. They taped it to the exterior wall of the enclosure, since in this position, the high noise disappeared.
On Mar 14, Flynn Haase tidied up this grounding wire ...
... and then routed it back into the HDI enclosure. He wedged it into a space between one of the frame pieces of the enclosure and one of the walls, as shown in this photograph:
This is only a temporary solution, as Flynn wrote:
Image #9815 shows how the ground cable was secured a bit more permanently. Due to all other items within the HDI enclosure, this is the best view possible. I labeled the cable coming down thru the top of the enclosure, the wrap you saw in image #9811 is behind the conduit to the ion pump. The blue based terminal is secured in between the frame and side panel. There is a mounting screw to the side panel just upwards of the terminal that is applying pressure to hold it in place. Due to size differential between the mounting screw diameter and the terminal end, I could not directly secure the ground lead with the mounting screw. As I said in a previous email, I will probably go up again this week and try to find a better way to secure this ground lead. This was the best I could think of in the time I had.