SRE-02-11
12/12/2011
VIRS  IR Dark Analysis Report 2: Calibration Changes
Prepared by: Rachel Klima & the MASCS team

1.  Introduction
This is a follow-up report describing the calibration changes implemented 
following the dark analysis report of 9/2011.  All data is currently being 
updated to use the latest calibration, and these changes will be implemented 
for the data being released in PDS 7.  The major changes implemented in this 
calibration version are to exclude saturated data from dark polynomial fits, 
decrease the polynomial order during short observations with few (<5) 
unsaturated dark measurements, and to utilize the 'Noise Spectrum' data 
field to store a standard deviation of the counts measured at the 
observation temperature for each pixel.

2.  Calibration Changes: Unbinned Data
2.1.  Saturation Handling
*  Code checks for values of 2^15, builds a mask that will ignore these 
saturated values when dark polynomial is fit
*  Mask is created with 1's for good values, NaN (or 1e32 in PDS data) for 
saturated values.  This is multiplied by corrected counts and raw counts so 
that no saturated data is passed through beyond the raw spectrum array
*  These changes only currently affect IR data since Vis only saturates in 
lamp measurements
2.2.  Pixel Noise Spectrum
*  Standard deviation of counts was compiled for all dark measurements taken 
during quick cals.  The average standard deviation at any temperature is 
approximated using a 3rd order polynomial for IR data and an exponential for 
Vis data.  The coefficients for either equation are stored in a save file 
for each pixel.  In the noise spectrum, the temperature is checked and then 
the standard deviation of the noise for that temperature is calculated for 
each pixel using the coefficients in the save file.  This is populated in 
the noise spectrum.
*  IR equation: Noise=A0 + A1*Temp + A2*Temp^2 + A3*Temp^3
*  Vis equation: Noise = A0 + A1*exp(Temp*A2)
*  The saturation mask is also multiplier by the noise spectrum, so anything 
that is saturated becomes NaN for SOC data or 1e32 (in PDS data)
2.3.  Dropping polynomial order in dark fits
*  The number of darks is now checked using the number of unsaturated darks, 
not just total number of darks collected.  If there is only one unsaturated 
dark, that single number is subtracted from the neighboring data.  If there 
are two darks, the average of those two points is used for the dark value.  
If there are three or four, a straight line is fit.  If there are 5, a 
second order polynomial is fit, and for IR data if there are 6 or more, a 
3rd order polynomial is used.

3.  Calibration Changes: Binned Data
3.1.  Saturation Handling
3.1.1.  IR data - temperature prediction, DQI changed to flag
*  A table has been constructed using the temperature that any pixel is first 
observed to saturate in dark quick cal measurements.  This temperature is 
rounded down to the nearest whole degree, and then 1 degree is subtracted.  
For a binned pixel, the saturation temperature is then set to the lower of 
the two individual temperatures.  For binned measurements, the data are 
handled similarly to the description for unbinned, but instead of checking 
for the raw saturated value, the saturation mask is built based on the
temperature at which the first half of the binned pixel saturates.
*  When there are saturated binned pixels, the DQI field now properly sets to 
show that there is saturation (previously this would only trigger if both 
pixels of a binned pair were saturated.
3.2.  Pixel Noise spectrum
*  The standard deviation of the noise of binned pixels is calculated by 
first deriving the individual pixel noise using the same equations as above 
for unbinned data, and then combining the noise for each pixel of a binned 
set using the equation:
*  virsnoise_binned= (sqrt((virsnoise[i,*]^2) + (virsnoise[i+1,*]^2)))/2
*  The saturation mask is again multiplied as in unbinned data.
3.3.  Dropping polynomial order in dark fits
*  Dark subtractions are handled the same as for unbinned, but use the 
saturation values derived from the temperatures.
4.  Plans for Future Calibration Improvements
-  change approximation of dark noise for IR - use exponential
-  develop alternative methods for identifying files where the dark solution 
is not performing well