SRE-01-11 Updated 9/19/2011 VIRS IR Dark Analysis Report 1: Operational Changes Prepared by: Rachel Klima & the MASCS team 1. Introduction and Motivation Based on laboratory and in-flight calibration from the MESSENGER lunar flyby, it was recommended that the IR detectors on VIRS be operated at below 10 degC. Unfortunately, the thermal environment around Mercury has resulted in considerably warmer temperatures. The goal of this study is to evaluate the current dark subtraction methodology and to determine whether the additional dark signal introduced by these higher temperatures has a regular structure that could be better removed with more frequent dark measurements. A secondary goal is to determine whether there are any other operational changes that could improve the scientific return of VIRS IR data. 2. Data Sets We investigated these issues with two primary data sets: (1) commissioning orbits where the temperature was warm (generally >30 degC) and a long sequence of data was obtained over the unilluminated surface of the planet (1000+ spectra collected in the dark). A shorter test CDR at a lower temperature (~16 degC) and a dark interleave of 2 was used to evaluate different dark interleaves for cooler operating temperatures.); (2) extended dark sequences taken during quick-cals (20 consecutive dark spectra at integration times of 1 second followed by 20 darks at 2 second integration times). 3. Testing the Dark Interleave Currently, dark measurements are interleaved every 40 spectra (~every 40-80 seconds depending on integration time). In the calibration pipeline, dark measurements from a given EDR are modeled with a 3rd order polynomial, which is then subtracted from the shutter-open measurements. Shown in Figure 1 an example of the measured raw counts for a single detector from shutter-open but dark surface measurements taken over >2000 seconds (from commissioning orbit). Shown in black is a 3rd order polynomial, fit to every 40th data point. A 3rd order polynomial fits the overall slope of the data well, with an R2 value of 0.99. Increasing the number of points used to calculate the polynomial in this long data series results in almost an identical polynomial fit. Some residual structure is evident; thus, we seek to explore whether this structure is regular enough to be better subtracted from the data. Shown in Figure 2 are cubic splines to various dark interleaves. To calculate the splines, every nth (n=40, 30, 20, etc.) data point was extracted for a 'dark' data set to fit the spline to. The residual is the difference between the data and the curve fit. Since the entire data set is dark, this residual should represent the quadratic sum of the readout noise and the dark current shot noise. For the subset of data shown, a cubic spline with a dark interleave of 40 results in a similar residual (Figure 3) to a 3rd order polynomial fit at a dark interleave of 40. Spline fits to higher cadence dark interleaves begin to fit the shorter frequency variability in the dark, though any regular structure is still similar in magnitude to the random noise in the data (Figure 3). Figure 1. Polynomial fit to the 1164.646 nm pixel. (See PDF version of document for figure.) Figure 2. Spline fits to the same data in Fig. 1 using dark interleaves of (A) 40; (B) 30; (C) 20; (D) 10; (E) 5; (F) 2. (See PDF version of document for figure.) Figure 3. Residual counts after a polynomial fit to every 40th data point and after cublic splines at various dark interleaves (DI). (See PDF version of document for figure.) To numerically evaluate the success of the spline fits, we calculated the standard deviation of the residuals at each dark interleave. Shown in Figure 4 are the standard deviations of residual raw counts after different dark subtractions for eight pixels across the detector. Pixels with both low average noise (ie. 165) and high average noise (ie. 104) are both included. In general, a polynomial fit to every 40th spectrum models the dark current as effectively (or better than) splines up to a dark interleave of 20. Higher frequency dark measurements (interleaves of 10, 5, 2) decrease the residual counts, but even in the most extreme case a dark interleave of two decreases the noise roughly by half. Increasing the dark frequency decreases the total number of surface observations, so as a metric of how many surface measurements are collected relative to the average standard deviation, we have divided the percentage of 'lit' surface observations using a given interleave by the standard deviation of the residual counts (Figure 5). When the loss of surface measurements resulting from taking additional darks is considered, any benefit of more frequent dark measurements drops dramatically. Figure 4. Standard deviation of residual counts using different dark interleaves for various pixels. (See PDF version of document for figure.) Figure 5. Percentage of measurements that are 'lit' for a given interleave divided by standard deviation of residual counts for various pixels. (See PDF version of document for figure.) 4. Assessing Individual Detector Performance Some detector elements have been observed to saturate much earlier than others, and others are consistently more affected by the thermal noise. To examine this behavior, over 19110 dark spectra taken during quick cals at temperatures ranging from 6-53 degC were compiled. We first identified the temperatures at which individual pixels were observed to saturate in dark measurements. The temperature at which 1 and 2 second integration time dark measurements are first observed to saturate for each wavelength is shown in Table 1, and a plot showing the raw count trends for a subset of pixels as a function of temperatures recorded by each of the detector temperature sensors is shown in Figure 6. About 20% of the pixels exhibit a break in slope between 30-40 degrees. This may be a result of the temperature sensors not accurately measuring the detector temperature at the higher temperatures. Pixel 165 (Fig. 6) is the only pixel to exhibit a slight decrease in dark current followed by an increase starting around 25 degrees. Figure 6. Raw dark counts as a function of temperature measured on each of the temperature sensors. A one second integration time saturates pixels 2 and 104, and a two second integration time saturates pixels 2, 81, 104 and 189. (See PDF version of document for figure.) To evaluate the performance of individual pixels as a function of temperature, we calculated the standard deviation of corrected counts in 2 degC temperature bins. The binned standard deviations are included in Table 2, and the standard deviations vs. temperature for several pixels at a 1 and 2 second integration time are shown in Figure 7. Figure 7. Standard deviation of corrected counts for several pixels as a function of temperature (temp_2) for one second integration time (left) and two second integration time (right). (See PDF version of document for figure.) As a whole, there is an increase in pixel noise at temperatures above 14 degC (Table 2). However, there is a range in the average pixel noise over all temperatures. Pixels were separated into groups by their average standard deviation, and shown in Figure 8 is an example of the residual corrected counts for representative pixels from each group. Though only two pixels maintain a standard deviation of <20 counts over all temperatures, 23 pixels remain at or below a standard deviation of 23 counts, and 59 remain at or below a standard deviation of 25 counts. As can be seen in Figure 8, these pixels are distributed across the full wavelength range. The pixels that tend to saturate more quickly are also distributed across the detector. Thus, if data is collected binned by two, saturated or noisy data are being combined with the less noisy pixels. The overall pixel noise and number of saturating pixels increase most significantly after 30 degC. However, the well-behaved pixels stay reasonably low. Shown in Figure 9 are example corrected-count spectra for a dark and a lit spectrum at full resolution, as well as with the SD>30 and SD>25 (191 total) pixels omitted. The distribution of better-performing pixels is sufficient to maintain around a 10 nm spectral resolution, even with the noisiest pixels omitted. Figure 8. (top) Corrected IR counts for dark quick-cal measurements at several wavelengths. Temperature shown is Temp 2. The 1281.35 detector is among the most stable across all temperatures, whereas 1138.98 is among the noisiest. Measurements taken at 1 and 2 second integration times are both shown. The apparent decrease in noise after about 37 degC is a result of the 2 second integration time measurements saturating (the calibration sets corrected counts for saturated pixels to zero). (bottom) Standard deviation of corrected IR counts as a function of wavelengths. Pixels with lower standard deviations are distributed across the array. Over 100 detectors have a standard deviation of 30 counts or less, and ~65 have a standard deviation of 25 counts or less. (See PDF version of document for figure.) Figure 9. Example spectra using the full spectrum as measured (red), a spectrum omitting pixels with a standard deviation (across all temperatures) of >30 counts (green) and a spectrum omitting pixels with a standard deviation of >25 counts (blue). Both panels are spectra from VIRSNC_ORB_11088_035817, the panel on the left is of a spectrum collected in the dark (i.e., spectrum should ideally be a flat line at zero) and the panel on the right is corrected counts of the lit surface. (See PDF version of document for figure.) 5. Results/Recommendations Based on these observations, data are now collected unbinned, to preserve the maximum number of good channels. The dark interleave has been changed to collect a minimum of 4 darks per observation, but otherwise an increase in dark frequency has not been recommended. Though the dark fitting procedure behaves nominally for temperature changes on the order of degrees, the 3rd order polynomial fit breaks down when there is a large change in temperature across the observation. In addition, data that are unbounded by dark measurements (ie. the first 39 spectra and final x spectra in an observation) often are poorly fit by the polynomial. To remedy these issues we are exploring several different options. First, we are exploring the options for fitting the dark currents as a moving function, with a restricted temperature range. We are also exploring the effects of using darks from observations immediately preceding or following a given observation, to alleviate the issues with the spectra taken before the first dark and after the last dark are taken. Finally, we are exploring the effectiveness of modeling the dark current as a function of temperature instead of time. The results of further calibration testing, as well as descriptions of any changes implemented to the pipeline calibration, will be detailed in a second dark analysis report. Table 1. Saturation temperatures and average variation in corrected counts for each wavelength. (See PDF version of document for table.) Table 2. Standard deviation of corrected counts for each wavelength, averaged over temperature bins. (See PDF version of document for table.)