Median Dark Frames

Today, we will finally get to the standard method for removing thermal noise from CCD images. To illustrate, we'll switch to a different set of images.

Using images from Sep 20, 2003

Begin by logging in to spiff and then going to your own directory. For example, after I've logged in, I type


          cd richmond

We're going to use a set of images taken at the RIT Observatory on Sep 20, 2003, for today's activities. I've placed these files into a sub-directory of the $dd directory. You can list the files by typing


          ls $dd/sep20_2003

Please make a copy of all these files into your own directory. Don't worry about the "omitting directory" messages you'll see -- that's not a problem.


          cp $dd/sep20_2003/* .

After you've done so, your directory should look something like this:

The image v585.fit is a 30-second image of the field of V585 Lyrae, a variable star in the constellation Lyra. We'll work with this image quite a bit, so get familiar with it. Display the image with parameters like this:


         tv v585.fit z=850 l=500 zoom=2

It should look something like this:

We want to get rid of the thermal contribution to this image. Some of the bright spots are real stars, but others are hot pixels where many thermal electrons have accumulated.

Is there any way to distinguish real stars from hot pixels?
Pick five different hot pixels, and measure their values. How far above the surrounding pixels do they rise? Hint: zoom in, then zoom in again, so that you can very easily place the cursor on a hot pixel to measure its value.

First try: subtracting a constant

What is the mean level of the single dark image dark30_001.fit? For the example below, I'll assume it's 120, but use the actual value you find.

Let's make a COPY of the real image, so that we can go back and try again later.


        cp v585.fit test1.fit

Okay, now we'll subtract the mean level of one dark frame from the copy. Remember to replace the "120" below with the actual value of the dark image you measured.


        sub test1.fit const=120

Display the test1.fit image after you have subtracted the constant value. What values of z= and l= give a picture that looks similar to the original image when displayed like this:
```
            tv v585.fit z=850 l=500 zoom=2
    
```
Is this subtraction of a constant value a big improvement? Why or why not?

Second try: subtracting a single dark image from the raw target image

Okay, that didn't work very well. Let's try subtracting a single dark image from the raw target image. We'll pick the fifth 30-second dark frame. As before, we'll make a COPY of the target image so that we can use the original for future work.


          cp v585.fit test2.fit

          sub test2.fit dark30_005.fit

Again display the subtracted image test2.fit at a zoom level of 2. Find a set of z= and l= parameters which cause it to look similar to the original image when displayed like this:


               tv v585.fit z=850 l=500 zoom=2

How well did the subtraction work this time?
Take a close look at the image. Zoom in to the area around pixel (160, 130) and scroll around. Do you see anything peculiar? (Hint: try displaying at zoom=2 with z=650 and l=500)
Go back to the dark image dark30_005.fit and examine this area. What do you see?

Third try: subtracting an average dark image from the raw target image

Hmmm. Cosmic rays seem to be a problem. Suppose that we average together all 10 of the 30-second dark frames, and subtract THAT from our target image. The way to average together two images is with the add command; you have to include some extra parameters:


      add image1.fit image2.fit norm outfile=average.fit

You can add together more than 2 images. The norm keyword means "normalize the sum at the end by dividing by the number of input images." So you could add together four images like this:


      add image1.fit image2.fit image3.fit image4.fit norm outfile=average.fit

But this could get pretty annoying if you need to add together 10 images, or 20, or 50. In that case, it makes sense to the use the Unix wildcard character "*" to specify "all files which have names containing a particular pattern." Look:

Can you figure out how to add together all the 30-second dark images to make an average dark frame?

Average together all 10 30-second dark images to create an image called dark-avg.fit
Make a copy of the target image called test3.fit
Subtract the average dark image from test3.fit
Display the result. How does it look? Pay special attention to the area around the cosmic ray at (163, 129).

Rats. Even averaging together 10 dark frames still leaves a residue at the location of a cosmic ray in just one of those dark frames; and that residue could screw up our calculations on the target image.

When you reach this point, look around. If someone nearby is still working on earlier stages of this assignment, offer to help. We'll all do the next part together.

A brief trip to the Land of Statistics

Suppose that you take a survey of incomes among a set of people who work in the Seattle area.

Fred: $10,000
Jane: $12,000
Bob: $11,000
Bill: $50,000,000
Peter: $10,000
Sally: $11,000

You want to describe the results of your survey with just a single number. What best represents the typical income of Seattle residents?

Perhaps you could use the mean, also called the average.

What is the mean of the 6 incomes?
Is that a fair way to describe the "typical" income?

A statistic which does a better job of ignoring isolated outliers is the median. It is simply "the number in the middle of the distribution." Here's how to calculate the median of a set by hand:

sort the values
count the values
walk through the sorted array until you have gone past half the entries; then stop
the current value is the median

If there are N elements in a set, then the median is just element number N/2 in the sorted set.

What is the median of the incomes?
Does this seem more reasonble for the "typical" income?

Final try: subtracting a median dark image from the raw target image

To create an image which is the pixel-by-pixel median of a set of images, you can use the median command (surprise!).


          median dark30_0*.fit outfile=dark-med.fit

Create a median image from all the 30-second darks, with the name dark-med.fit
Make a copy of the original target image; call it test4.fit
Subtract the median dark frame from test4.fit
Display the result. How does it look? Pay special attention to the area around the cosmic ray at (163, 129).

Take a look at all four attempts at subtracting the thermal signal. This is a good way to see them all side-by-side:


            box 1 sr=150 sc=120 nr=30 nc=30
            tv test1.fit box=1 zoom=6  z=650 l=500 
            tv test2.fit box=1 zoom=6  z=650 l=500 xo=200
            tv test3.fit box=1 zoom=6  z=650 l=500 xo=400
            tv test4.fit box=1 zoom=6  z=650 l=500 xo=600

Using the median of a set of dark images is the standard operating procedure for astronomers who use CCDs. It's easiest if all your target images have the same exposure length, say, 30 seconds. At the start of the night (and sometimes at the end, if you are really paranoid), you take a set of 10 or 20 or 50 dark images, each 30 seconds long. You can create a "master dark image" by taking the median of all these individual darks, and then subtract it from your target images.

What could you do if you have target frames with a bunch of different exposure times (10 sec, 30 sec, 60 sec, 600 sec, 1200 sec)? In theory, you could take one set of darks with the first exposure time, and another set of darks with the second exposure time, and another .... but that would take a LOT of time. Is there a better way?

The power of the median

Images we take near sea level don't suffer much from cosmic rays -- they only appear every now and then. The median technique is a bit better than simple averaging, but the difference isn't large.

On the other hand, telescopes on mountaintops, high above much of the Earth's atmosphere, are exposed to much greater levels of radiation from space. And telescopes in orbit are even more vulnerable. The Hubble Space Telescope circles the Earth not far from regions of the Earth's magnetic field which are full of high-energy electrons and protons. Cosmic rays can be really annoying in images taken by these instruments. Take a look at this raw image from HST:

Yuck! There are actually several different kinds of nasty stuff in this image:

the random noise along the left and bottom edges is due to regions of the chip which are not illuminated by light from the sky ... so there's nothing BUT random thermal and readout noise
the bright points and short streaks all over the image are due to cosmic rays
the giant, bright line running horizontally across the image is another satellite which passed through the field of view during the exposure

We can't do anything about the regions of the chip which were not illuminated, but look what happens when we take the median of just two raw HST images of this field.

What is the "median" of two numbers? The smaller of the two.

That's much better. Most of the cosmic rays are gone, though some single-pixel events are still present. The satellite trail is reduced in prominence, too.

After combining about 20 images of this field (taken in the V and I filters), all the cosmic rays are gone, and the satellite trail has completely disappeared.

For more information

The HST images in the examples above were part of a project to find Cepheid stars in the galaxy NGC 2841. You can read the results at The Discovery of Cepheids and a New Distance to NGC 2841 Using the Hubble Space Telescope