OMEGA software (IDL version)
release SOFT05
february 2008
********************************************************
changes from SOFT01 to SOFT02
- new version of readomega which includes flat-fielding and
  second order corrections for the VIS channel
- new photometric function (mtf050110_25.dat and mtf050110_50.dat)
  with some adjustments for the VIS channel

Instrumental problems:
- additional information on bad spectels
- additional information on a possible digital saturation
  of the dark in the L channel
- complementary remarks on the co-registration of channels, with
  a warning on the validity range of the L channel
- numerical perturbation for the 128 pixel modes after orbit 511
********************************************************
changes from SOFT02 to SOFT03
- more detailed explanation on omega_path
- improved pipeline for VIS channel deconvolution. The impact of 
  the 2 msec read time of the CCD is now taken into account. 
  the offset correction, bias correction, flat and transfer function
  in the visible are also modified (new photometric function: mtf050701)
  new routines: smear_corr.pro, offset_corr.pro

! This results in a much improved overlap with SWIR-C at 1 m.
  it should however be noted that the visible channel has an effective
  spatial resolution which is close to 1 pixel along the line, 
  but close to 4 pixels (5 mrad) along the swath. This prevents
  meaningful comparison of VIS and IR spectra for spectral features
  of very small patches (a few pixels), as they are much weaker
  in the lower resolution VIS channel.
  
- updated information on instrumental effects
********************************************************
changes from SOFT03 to SOFT04

- a new version of readomega is provided, which handles the evolution
  of IR spectels due to radiation damage. Unreliable spectels are
  given very low values. An evolving photometric function is used
  for spectels which are still stable, but with a reduced efficiency 
  
  (see corresponding section "time evolution of bad and hot IR spectels")
  
  a sligthly revised version of the photometric function (for spectel 69) 
  is now implemented so as to be compatible with the new version of
  readomega
  
- the new readomega also removes an estimate of the thermal background 
  from optics
  
- a routine "readpart.pro" is provided so as to read part of a large
  cube when memory size becomes a problem
  the syntax and result are similar to readomega, but the user is
  requested to provide a number of scans to skip and a number of
  scans to read in the cube. Entering "0" as the number of scans
  to read will result in reading all remaining scans
********************************************************
changes from SOFT04 to SOFT05
- recovery procedure for the lack of dark data for the last scan 
  in summed modes (the dark from the previous scan is considered)
  This resulted in unusable spectra for the last line of summed cubes 
- update of the source files for following spectel evolution 
  ("bound" and "rap") 
  
********************************************************
unzipping SOFT05.ZIP will create a new subdirectory, SOFT05, so that
further releases (SOFT06, ...) will not erase previous versions
(this can be useful for comparing the results of the different updates). 

So as to use the new version, the user should copy the full content of 
SOFT05 to the OMEGA working directory, from where IDL should be started.

IMPORTANT ! The first thing a user should do after unzipping a new
software release and copying the contents to the working directory 
is to modify the file:
         omega_path
in the working directory (from which IDL is run)
replacing the first line with the path to the directory
where the user puts data files (.QUB), and the second line
with the path to the directory where the user puts the geometry files (.NAV)
the two directories can be the same
a "/" must be appended for UNIX, LINUX...
a "\" must be appended for windows

An OMEGA observation corresponds to data obtained with the same observation
parameters (scan length, integration time for the VIS and IR channels, 
downtrack summing, compression). There are in general several OMEGA 
observations for the same orbit, as the scan length is changed depending
on the spacecraft altitude so as to best match the sampling rate and
the drift velocity on the surface.

The OMEGA data files and geometry files are named from the corresponding
OMEGA observation: ORBNNNN_S where NNNN is the orbit number (format I4.4)
and S is the rank of the OMEGA observation on that orbit (starting with 0).
The extensions are .QUB for the data, .NAV for the geometry
example: ORB0018_1.QUB and ORB0018_1.NAV correspond to the second 
observation on orbit 18 (scan length: 32 pixels). 

The main procedure for reading level 1b OMEGA data is readomega.pro

After entering IDL and typing ".run readomega", one has simply to enter 
the name of the requested OMEGA observation in answer to the query

example:

IDL (enter)
.run readomega (enter)
OMEGA observation:
obsname (enter)      /* obsname example: ORB0072_2 */

**** NOTE **** THE NAME MUST BE ENTERED WITHOUT THE EXTENSION

readomega first checks that obsname.QUB exists in the directory
indicated by the path for data files in "omega_path". 

If the cube is not found, it prints:
 "file OMEGA_PATH_DATA/obsname.QUB not found"
and goes back to the query. Otherwise, readomega reads the
pds cube, then it look for the corresponding geometry cube 
obsname.NAV in the directory indicated by the path for geometry files 
in the "omega_path" file. If there is no such file, it prints: 
"no corresponding NAV cube". If the NAV file exists, readomega reads 
the geometry cube. 

the resulting variables are:

idat(npix,352,nscan): integer 16 bits, raw OMEGA data
      npix: 16, 32, 64 or 128 pixels
      nscans: number of scans in the cube
      the 352 wavelengths are in the following order:
      IR C (128 values), then IR L (128 values), then Visible (96 values)

wvl(352) OMEGA wavelengths (file 'lambda_0304.dat'). these wavelengths
      are valid at the center of the IR and VIS field. The wavelength
      range changes only slightly across the field of view, in particular
      in the IR.

jdat(npix, 352, nscan): floating point, OMEGA data physical units 
      (W/m2/steradian/m)
      There is a second order contribution to the last 35 spectels of 
      the VIS channel, which is corrected. Nevertherless, these spectels
      are not as reliable as spectels 256 to 315. The L channel (128:255)
      shows variations in terms of internal calibration signal, which
      do not correlate simply with DN levels from Mars or Phobos

note: the last 4 scans (16 pixels), 2 scans (32 pixels)
      or 1 scan (64, 128 pixels) of idat and jdat
      have only IR data (spectels 0 to 255)

There is calibration data at the beginning of each cube for the visible
channel, at the beginning of the ORBNNNN_0.QUB cubes for the IR channels
The number of calibration scans depend on the pixel length (16 to 128)
and the summation (1, 2 or 4 for 128 pixel modes)

      for every cube, the first scan (128 pixels x 4), 3 scans (128 x 2) 
      7 scans (128 x 1), 14 scans (64 pixels), 
      28 scans (32 pixels), 56 scans (16 pixels) of the visible channel
      (spectels 256 and above) correspond to an internal calibration

      for cubes with names NNNN_0 (first cube in a sequence), the first
      6 scans (128 x 4), 12 scans (128 x 2), 24 scans (128 x 1)
      48 scans (64 pixels), 96 scans (32 pixels) or 192 scans (16 pixels)
      of the IR channel (spectels 0 to 255) correspond to an internal 
      calibration (closed shutter, lamp on at 6 different levels, 
      in order 0,4,3,2,1,0

specmars(352): floating point, solar spectrum at the distance of Mars
      at the time of observations. jdat(i,*,n)/specmars(*) provides 
      the I/F values for the spectral range dominated by reflected 
      sunlight. It is meaningless for the spectral range dominated
      by thermal emission (usually above 3.5 m)

sdat0(352,nscan): 
       0:256 contains the corresponding IR dark
       raw data: sdat0(0:255,n)-idat(i,0:255,n)
       this can be useful when there are problems
       with the IR (digital saturation of the dark, detector saturation) 

       detector saturation in the IR (only observed in the C channel
       with 5 msec integration time) can be checked by plotting:
       sdat0(0:127,n)-idat(i,0:127,n). If there is a plateau at values
       close to 300 DN around spectel 41 (the highest DN level), these
       values are saturated.

       Perturbations of the dark are most prominent with 16 pixels, 
       5 msec integration time at high DN levels from channels 20 to 50.
       A simple way to check for such effects is to plot sdat0(41,*)  
       (dark level for spectel 41, the highest DN level
       in the signal). If it is not nearly constant, and there is a dip 
       when the signal is high, the dark is perturbed. A non perturbed
       dark needs then to be recovered from a region of the cube with
       lower signal values (or another cube from the same orbit). 
       The proper level of the data can then be recovered by:
 
       idat(i,0:127,n)=idat -sdat0 (0:127,n) + sdat0 (0:127, unperturbed)


       With 2.5 msec modes, improper adjustment of the polarization 
       parameter can set part of the L channel dark current (orbits 12xx)
       (spectels 128 to 255) beyond the 4095 quantization limit.
       the bottom of the 3 m band is perturbed
       This can be checked by plotting sdat0(128:255,n)
       so as to verify that it does not reach 4095 except for 
       channel 159 (dead).
       This can be recovered by searching for a non saturated dark
       in other cubes, then scaling it (0 level = 4290) and shifting it
       so as to match the part of the dark below 4095
      

sdat1(npix,7,nscan):
       housekeeping data as defined in the EAICD

exposure (float(3)): integration times for the IR (C channel), 
       the IR (L channel) and the VIS channel. The integration time
       is the same for IR C and IR L in all foreseen OMEGA modes.
       It can be either 2.5 msec or 5 msec. the visible integration
       time is in general set as 50 msec.
    
       there are two photometric functions (352 values per files):
          mtfYYMM_25.dat (2.5 msec, release of year YY, month MM)
          mtfYYMM_50.dat ( 50 msec, release of year YY, month MM) 

       The readomega procedure selects the proper exposure time 
       for the IR and for the visible. In the present release, it uses the 
       mtf050110_25.dat nad mtf050110_50.dat files. It also takes into account
       the summation by two internal to the VIS channel for 128 pixel modes

IMPORTANT REMARK: new releases of the photometric function are likely to
       be necessary as the knowledge of the intrument improves.
       with this software release, relative variations of 5% or more
       between spectra are considered reliable. Smaller variations can
       be related to linearity effects, and should be considered with
       caution, in particular if they correlated with I/F (linearity). 
       The confidence on the absolute radiometric calibration is 
       at the 15% level. Three sets of photometric functions (040319, 
       050110 and 050701) are provided, which indicate the importance
       of updates. The 050701 set is used when computing jdat

bits_per_pixel : number of bits per pixel from data compression

summation : number of downtrack summations (only for modes with 128 pixels)
       this parameter can be 1.0, 2.0 or 4.0, in which case both idat
       and sdat0 have values can be a factor of 2 or 4 higher than
       in the nominal situation (summation 1.0). This factor is taken
       care of when computing jdat
      
The corresponding auxiliary pds cube ORBNNNN_X.NAV is read by readomega
if available (otherwise the user gets a "no corresponding geometry file"
message

products:

geocube: geometry cube with as many samples and lines as idat,
         and 51 bands. The definitions of each band are:
   0:   scan value (in DN) for each pixel
   1:   information on the time (hr:min:sec:msec)
   2:   incidence on the ellipsoid  (unit: 0.0001)
   3:   emergence from the ellipsoid (unit: 0.0001)
   4:   incidence wrt the local gravity field (atmospheric science)
   5:   emergence wrt the local gravity field
   6,7 : longitude and latitude of the center of the IR C pixel (0.0001)
   8,9,10: incidence, emergence, phase for the IR C pixel (unit: 0.0001)
   11 : distance in m
   12 : altitude (MOLA - ellipsoid) in m
        if > 65.535 km: altitude above the surface + 65536 m
        (for limb observations)
   13-16: longitudes of the corners of the IFOV (unit: 0.0001)
   17-20: latitudes of the corners of the IFOV (unit: 0.0001)
   21-35: same as 6-20 for the IR L pixel
   36-50: same as 6-20 for the Visible pixel

More details are given in the EAICD

dmars : sun - mars distance, in AU
specmars (352 values) : solar spectrum at OMEGA wavelengths at
        the heliocentric distance of Mars during observations
longi : longitude in  of all OMEGA pixels (C channel)
lati : latitude in  of all OMEGA pixels (C channel)
alt : MOLA altitude, in km, for all pixels in the cube (C channel)

IMPORTANT NOTE
   The geocube data set is very useful to obtain a first estimate
   of the region imaged by each OMEGA cube. Given the remaining uncertainties
   on position and attitude restitution, the observed variations in
   alignment of the C and L channel (from 1 to 3 pixels), and the 
   registration problems between spectels, the positionning error is 
   estimated as 3 mrad or better for most cubes. A specific effort using
   other data sets (in particular comparing the CO2 band strength with
   MOLA altimetry) is required if one requires a better positioning.

        another very useful check at medium to high incidences is 
        provided by the comparison between the cosine of the local 
        incidence (geocube(*,8,*) and the actual signal in the 
        continuum (e.g. spectel 11). This check of shadowing is 
        also useful so as to search for clouds (in particular in 
        Hellas) which destroy the otherwise good correlation
        between the expected illumination (assuming no rayleigh
        scattering) and actual signal 

        UPDATE ON INSTRUMENTAL PROBLEMS

- time evolution of hot and bad IR spectels
     since ground calibration, IR spectel 158 has been dead "cold" (0 value)
     and spectel 78 has been dead "hot" (always high values).
     since orbit 171, spectel 34 is also dead "hot".
     spectels 78, 171, and 34 from orbit 171 on are replaced by averages 
     of their two neighbours so as not to mess up spectral plots

     Cosmic ray degradation slowly increases the number of spectels
     which are very hot (500 DN or more above the dark), but not dead. 
     - 69, 88, 224 since the beginning
     - 188 since orbit 1147
     - 155 since orbit 1990
     the following spectels are moderately hot (100 to 500 DN above 
     the nominal dark)
     at orbit 2000:
     - 55, 66, 79, 85, 121, 127, 200, 222
     for any given cube, dead and hot pixels can be observed by
     plotting the dark: sdat0(0:255,n). They appear as spikes
     (upward for 158, downward for the others)

     very hot pixels are not reliable

     hot pixels have to be considered with caution, as they can 
     result in spikes in the decalibrated spectra. It is possible
     to recover most of the information for these spectral values
     with appropriate filtering tools or by using spectral ratios
     
     readomega now automatically handles hot and bad spectels as a
     function of orbit number. The first orbit after which a problem
     is observed is provided by the "bound060110.dat" file
     which generates the "bound" array (256 values, 1 per spectel)  
     orbit 0 means that the problem was observed during cruise,
     orbit 9999 means that the spectel has not (yet !) evolved.
     the photometric function (mtf) is multiplied by a factor
     1.e30 for spectels which have become unreliable. Otherwise,
     it is multiplied by a factor which is in general closer to 1
     for the 2.5 msec integration time than for 5 msec.
     An integer array (ic) is generated by readomega. It indicates
     which indices correspond to usable spectels on that date. 
     "plot,wvl(ic),jdat(pix,ic,line)" will therefore get rid of
     the spurious values due to unreliable spectels.

- time evolution of the L channel
     The L channel is not fully stable in the cross scan direction.
     The interval between the C and L channels varies between
     1.2 mrad and 3.1 mrad. There are mainly two states, which are
     correlated with the level of the calibration source in the L channel.
     when it is high, the separation is minimal. When it is low, 
     the separation is large. 

     There are (up to now) three series of orbit with a nominally high
     level of the calibration source:
      - orbits 98 to 511
      - orbits 939 to 1221
      - from orbit 2000 to 2600, the level slowly increases
        to reach a value only slightly lower than nominal

     A careful comparison of signals on bright terrains reveals a 
     likely impact on the L photometric function. 

     WHEN THE L CALIBRATION LEVEL IS LOW, THE VALUE OF JDAT WITH 
     THE PRESENT PHOTOMETRIC FUNCTION IS A LOWER LIMIT FROM 128 TO 255

- numerical perturbation problem for modes with 128 pixels
     Since orbit 511, pixels 80 to 95 are perturbed every 32 spectels
     across the 3 spectral ranges. For even lines, these spectels are:
     [12,13,14,15], [44,45,46,47] .....
     For odd lines, these spectels are:
     [28,29,30,31], [60,61,62,63] ....

     This can be (to some extent) corrected by replacing these 
     four values by the mean of the corresponding spectels on the 
     previous and next line, which have a different parity

         
