Mars Science Laboratory (MSL)
Chemistry and Mineralogy (CheMin)
Reduced Data Archive
Software Interface Specification
Version 1.1
March 14, 2013
Susan Slavney
Planetary Data System Geosciences Node
TABLE OF CONTENTS
TABLES......................................................................................................................................... iv
DOCUMENT CHANGE LOG...................................................................................................... v
TBD ITEMS................................................................................................................................... vi
ACRONYMS AND ABBREVIATIONS.................................................................................... vii
GLOSSARY................................................................................................................................ viii
1 Introduction......................................................................................................................... 1
1.1 Purpose and Scope.............................................................................................................. 1
1.2 Content Overview............................................................................................................... 1
1.3 Applicable Documents and Constraints.............................................................................. 2
1.4 Relationships with Other Interfaces.................................................................................... 2
2 Archive Volume Contents................................................................................................... 2
2.1 Root Directory Contents..................................................................................................... 3
2.2 Data Directory Contents and Naming................................................................................. 3
2.3 Index Directory Contents.................................................................................................... 4
2.4 Document Directory Contents............................................................................................ 5
2.5 Catalog Directory Contents................................................................................................. 5
2.6 Label Directory Contents.................................................................................................... 6
2.7 Software Directory Contents............................................. Error! Bookmark not defined.
2.8 Calib Directory Contents..................................................................................................... 6
2.9 Geometry Directory Contents........................................... Error! Bookmark not defined.
2.10 Browse Directory Contents............................................... Error! Bookmark not defined.
2.11 Extras Directory Contents................................................................................................... 6
3 Archive Volume Format...................................................................................................... 7
3.1 File Formats......................................................................................................................... 7
3.1.1 Document File Format............................................................................................. 7
3.1.2 Tabular File Format.................................................................................................. 7
3.1.3 PDS Label Format.................................................................................................... 7
3.1.4 Software File Format............................................................................................... 8
3.1.5 Catalog File Format.................................................................................................. 8
3.1.6 Science Data File Formats........................................................................................ 8
4 Archive Volume Generation................................................................................................ 9
4.1 Data Transfer and Validation Methods............................................................................... 9
4.2 Data Product Sizes and Delivery Rates.............................................................................. 9
4.3 Interface Media Characteristics........................................................................................... 9
4.4 Backup............................................................................................................................... 10
4.5 Revisions........................................................................................................................... 10
4.6 Labeling and Identification............................................................................................... 10
5 Support Staff and Cognizant Persons............................................................................... 10
Table 1. MSL CheMin Derived Data Products............................................................................... 1
Table 2. Processing Levels for Science Data Sets........................................................................... 1
Table 3. CheMin RDR Product Type Identifiers............................................................................ 3
Table 4. Index Table Contents......................................................................................................... 4
Table 5. CheMin Delivery Sizes and Rates..................................................................................... 9
Change |
Date |
Affected Portions |
V1.1 Updated for Release 1 |
3/14/13 |
TBD table, 3.1.3, page heading |
Section |
Description |
|
|
ASCII |
American Standard Code for Information Interchange |
CD-ROM |
Compact Disk - Read-Only Memory |
CD-WO |
Write-Once Compact Disk |
ISO |
International Standards Organization |
JPL |
Jet Propulsion Laboratory |
NSSDC |
National Space Science Data Center |
PDS |
Planetary Data System |
PSG RCE |
Project Science Group Rover Compute Element |
SDVT |
Science Data Validation Team |
SIS |
Software Interface Specification |
TBD |
To Be Determined |
Archive – An archive consists of one or more data sets along with all the documentation and ancillary information needed to understand and use the data. An archive is a logical construct independent of the medium on which it is stored.
Archive Volume, Archive Volume Set – A volume is a unit of media on which data products are stored; for example, one CD-ROM or DVD-ROM. An archive volume is a volume containing all or part of an archive; that is, data products plus documentation and ancillary files. When an archive spans multiple volumes, they are called an archive volume set. Usually the documentation and some ancillary files are repeated on each volume of the set, so that a single volume can be used alone.
Catalog Information – Descriptive information about a data set (e.g. mission description, spacecraft description, instrument description), expressed in Object Description Language (ODL) which is suitable for loading into a PDS catalog.
Data Product – A labeled grouping of data resulting from a scientific observation, usually stored in one file. A product label identifies, describes, and defines the structure of the data. An example of a data product is a planetary image, a spectrum table, or a time series table.
Data Set – An accumulation of data products. A data set together with supporting documentation and ancillary files is an archive.
Rover Compute Element – The MSL onboard computer. This consists of primary (A) and backup (B) computer elements. CheMin data are downlinked from the RCE.
Standard Data Product – A data product generated in a predefined way using well-understood procedures, processed in "pipeline" fashion. Data products that are generated in a nonstandard way are sometimes called special data products.
This Software Interface Specification is intended to be used by those who wish to understand the format and content of the Mars Science Laboratory (MSL) Chemistry and Mineralogy (CheMin) Reduced Data Record (RDR) Archive. Typically, these individuals would be software engineers, data analysts, or planetary scientists.The specifications in this document apply to all CheMin RDR standard product archive volumes that are generated by the MSL Project.
The MSL CheMin RDR archive consists of the following derived products, which are described in the CheMin RDR Software Interface Specification, applicable document 1. All CheMin derived products are generated by the CheMin Science Team. Level 1B data products will be delivered to PDS with data set identification as MSL-M-CHEMIN-4-RDR-V1.0. Tables with Level 2 mineral identifications and abundances will be delivered separately, with data set identification as MSL-M-CHEMIN-5-RDR-V1.0. The separate delivery of Level 2 mineral identification and abundance data allows analysis and revision of these data independently of the processed 1D source data.
Table 1. MSL CheMin Derived Data Products
NASA Level |
CODMAC Level |
Description |
1B |
4 |
Co Ka XRD data summed along 2-theta radii, adjusted for arc length and presented as 2-theta versus intensity tables. |
1B |
4 |
Co Kb XRD data summed along 2-theta radii, adjusted for arc length and presented as 2-theta versus intensity tables. |
1B |
4 |
Single-pixel EDS histogram of energy versus intensity (counts). |
1B |
4 |
Split-pixel EDS histogram of energy versus intensity (counts). |
1B |
4 |
Total EDS histogram of energy versus intensity (counts). |
1B |
4 |
XRD data from stacked raw frames, summed along 2-theta radii, adjusted for arc length and presented as 2-theta versus intensity tables. |
1B |
4 |
XRD data from modified raw frames, summed along 2-theta radii, adjusted for arc length and presented as 2-theta versus intensity tables. |
2 |
5 |
Mineral identification, abundance, and estimated errors. |
NASA and CODMAC data processing levels are described in Table 2.
Table 2. Processing Levels for Science Data Sets
NASA |
CODMAC |
Description |
||
Packet data |
Raw – Level 1 |
Telemetry data stream as received at the ground station, with science and engineering data embedded. |
||
Level-0 |
Edited – Level 2 |
Instrument science data (e.g., raw voltages, counts) at full resolution, time ordered, with duplicates and transmission errors removed. |
||
Level 1-A |
Calibrated - Level 3 |
Level 0 data that have been located in space and may have been transformed (e.g., calibrated, rearranged) in a reversible manner and packaged with needed ancillary and auxiliary data (e.g., radiances with the calibration equations applied). |
||
Level 1-B |
Resampled - Level 4 |
Irreversibly transformed (e.g., resampled, remapped, calibrated) values of the instrument measurements (e.g., radiances, magnetic field strength). |
||
Level 2 |
Derived - Level 5 |
Geophysical parameters, generally derived from Level 1 data, and located in space and time commensurate with instrument location, pointing, and sampling. |
||
Level 3 |
Derived - Level 5 |
Geophysical parameters mapped onto uniform space-time grids. |
||
This Software Interface Specification (SIS) describes the format, content, and generation of the CheMin RDR Archive. Section 2, Archive Volume Contents, describes the structure of the archive volumes and the contents of each file. Section 3, Archive Volume Format, describes the file formats used on the archive volumes. Section 4, Archive Volume Generation, describes the procedure for assembling the archive and transferring it to the Planetary Data System (PDS). Finally, Section 5, Support Staff and Cognizant Persons, lists the individuals responsible for generating and receiving the archive volumes.
This Archive Volume SIS is intended to be consistent with the following documents:
4. Planetary Data System Archive Preparation Guide, Version 1.4, JPL D-31224, April 1, 2010.
5. Planetary Data System Standards Reference, February 27, 2009, Version 3.8, JPL D-7669, Part 2.
This Archive Volume SIS could be affected by changes to the design of the CheMin RDR standard data products (Applicable Document 3).
This section describes the contents of the MSL CheMin RDR Archive volumes, including the file names, file contents, file types, and organization responsible for providing the files.
Files in the Root Directory include an overview of the archive, a description of the volume for the PDS Catalog, and a list of errata or comments about the archive. The following files are contained in the Root Directory.
File Name |
File Contents |
File Provided By |
AAREADME.TXT |
Volume content and format information |
Geosciences Node |
ERRATA.TXT |
A cumulative listing of comments and updates concerning all archive volumes published to date. More than just errata, this file should be updated with release notes for each data release, including an explanation of any revisions or omissions. |
CheMin Team |
VOLDESC.CAT |
A description of the contents of this volume in a PDS format readable by both humans and computers |
Geosciences Node |
Files in the Data Directory are grouped into subdirectories by sol (Mars day) number, with directory names in the form SOLnnnn (example: SOL0001). Files in the Data Directory consist of data products and the PDS labels that describe them. Both data products and labels are provided by the CheMin Team.
The file naming scheme is fully explained in the CheMin RDR SIS, Applicable Document 3. In brief, the scheme is as follows:
<instr><config><sclk><prod><sol><site><drive><seqid><producer><ver>.<ext>
where
instr = CM for CheMin
config = A_ for primary Rover Compute Element (RCE) , B_ for backup RCE, or __ (two underscores) for a special case where partial downlinks may have been obtained from both the primary and the backup RCE and compiled into a single RDR
sclk = 9-character Spacecraft Clock Start Count
prod = 3-character product type identifier, one of the following for CheMin RDRs:
Table 3. CheMin RDR Product Type Identifiers
CheMin RDR Product Type |
Identifier |
Diffraction Single, K-alpha or K-beta |
RD1 |
Diffraction Split, K-alpha or K-beta |
RDS |
Diffraction All, Raw |
RDA |
Diffraction All, Raw |
RTR |
Energy All |
REA |
Energy Single |
RE1 |
Energy Split |
RES |
Mineral Identification and Abundance |
MIN |
sol = 4-digit sol number
site = 3-character site location number
drive = 4-character drive location (position within site)
seqid = 4-character Category subfield and 5-digit sequence ID
producer = "P" for CheMin Principal Investigator; other characters A through L to be assigned to CheMin Co-Investigators
ver = 1-character version identifier
ext = TAB for ASCII text table data, DAT for binary table data, LBL for ASCII text PDS label
Files in the Index Directory are provided to help the user locate products in the archive. The following files are contained in the Index Directory.
File Name |
File Contents |
File Provided By |
INDXINFO.TXT |
A description of the contents of this directory |
Geosciences Node |
INDEX.TAB |
A table listing all data products on this volume |
CheMin Team |
INDEX.LBL |
A PDS detached label that describes INDEX.TAB |
CheMin Team |
The index table contains one row for each product on an archive volume. Table 4 lists the columns of the index table.
Table 4. Index Table Contents
Column Name |
Description |
Source |
VOLUME_ID |
PDS archive volume on which a data product is stored |
Specified at time of index table creation |
PATH_NAME |
Directory path to data product, relative to volume root |
Determined by location of PDS label file |
FILE_NAME |
Name of label file for data product |
PDS label file name |
PRODUCT_ID |
Unique identifier for data product |
PDS label |
PRODUCT_TYPE |
Type of RDR product, from Table 3. |
PDS label |
PRODUCT_VERSION_ID |
Version number of data product |
PDS label |
PRODUCT_CREATION_TIME |
Date and time of data product creation |
PDS label |
RELEASE_ID |
ID of scheduled data release to PDS in which this product was included |
PDS label |
START_TIME |
UTC date and time of start of observation |
PDS label |
SPACECRAFT_CLOCK_START_COUNT |
Spacecraft clock count at start of observation |
PDS label |
The Document Directory contains documentation to help the user understand and use the archive data. The following files are contained in the Document Directory.
File Name |
File Contents |
File Provided By |
DOCINFO.TXT |
A description of the contents of this directory |
Geosciences Node |
CHEMIN_BASICS.PDF |
A PDF file that explains basic operation and data generation from the CheMin instrument |
CheMin Team |
CHEMIN_BASICS.HTM |
CHEMIN_BASICS as hypertext |
CheMin Team |
CHEMIN_BASICS.LBL |
A PDS label that describes both versions of CHEMIN_BASICS |
CheMin Team |
CHEMIN_RDR_AVSIS.PDF |
The Archive Volume SIS (this document) as a PDF file |
CheMin Team |
CHEMIN_RDR_AVSIS.HTM |
The Archive Volume SIS (this document) as hypertext |
CheMin Team |
CHEMIN_RDR_AVSIS.LBL |
A PDS label that describes both versions of CHEMIN_RDR_AVSIS. |
CheMin Team |
CHEMIN_RDR_SIS.PDF |
The Data Product SIS as a PDF file |
CheMin Team |
CHEMIN_RDR_SIS..HTM |
The Data Product SIS as hypertext |
CheMin Team |
CHEMIN_RDR_SIS.LBL |
A PDS label that describes both versions of CHEMIN_RDR_SIS. |
CheMin Team |
MSL_LDD.FUL, .LBL |
MSL Local Data Dictionary and label |
Geosciences Node |
PDSDD.FUL, .LBL |
PDS Data Dictionary and label |
Geosciences Node |
The files in the Catalog Directory provide a top-level understanding of the mission, spacecraft, instruments, and data sets. The files in this directory are coordinated with the PDS data engineer, who is responsible for loading them into the PDS catalog. The following files are found in the Catalog Directory.
File Name |
File Contents |
File Provided By |
CATINFO.TXT |
A description of the contents of this directory |
Geosciences Node |
CHEMIN_INST.CAT |
Instrument information for the PDS catalog |
CheMin Team |
CHEMIN_L1_RDRDS.CAT, CHEMIN_L2_RDRDS.CAT |
Data set information for the PDS catalog |
CheMin Team |
CHEMIN_REF.CAT |
References mentioned in other CHEMIN*.CAT files |
CheMin Team |
MSL_INSTHOST.CAT |
Instrument host (i.e., spacecraft) information for the PDS catalog |
MSL Project |
MSL_MISSION.CAT |
Mission information for the PDS catalog |
MSL Project |
MSL_REF.CAT |
References mentioned in other MSL*.CAT files |
MSL Project |
PERSON.CAT |
Personnel information for the PDS catalog (Team and PDS personnel responsible for generating the archive) |
CheMin Team |
The Label Directory contains format files, e.g., excerpts from PDS labels that describe data format and organization. These files are referred to in the PDS labels that accompany the data products. They are "include" files that are intended to be parsed as if they were part of the PDS labels that refer to them. This is an optional directory. It is present only if the data product labels refer to format files using the ^STRUCTURE keyword. The following files are contained in the Label Directory.
File Name |
File Contents |
File Provided By |
LABINFO.TXT |
A description of the contents of this directory |
Geosciences Node |
CHEMIN_EDH.FMT |
Format of CheMin RDR energy data products |
CheMin Team |
CHEMIN_MIN.FMT |
Format of CheMin mineral abundance data products |
CheMin Team |
CHEMIN_XRD.FMT |
Format of CheMin X-ray Diffraction data products |
CheMin Team |
The Calib Directory contains calibration files used to process the data products, or calibration data needed to use the data products. This directory is optional. It appears only if there are calibration data, plans, or reports included in the archive. The following files are contained in the Calib Directory.
File Name |
File Contents |
File Provided By |
CALINFO.TXT |
A description of the contents of this directory |
Geosciences Node |
CALREPORT.PDF |
PDF documentation of results obtained from analyses of calibration standards carried on the CheMin flight instrument. |
CheMin Team |
CALREPORT.HTM |
CALREPORT as hypertext |
CheMin Team |
CALREPORT.LBL |
PDS label that describes both versions of CALREPORT |
CheMin Team |
The Extras Directory contains a listing of commercial or public-domain software available for processing Level 2 CheMin X-ray diffraction data. The following files are contained in the Extras Directory.
File Name |
File Contents |
File Provided By |
EXTRINFO.TXT |
A description of the contents of this directory |
Geosciences Node |
XRDINFO.PDF |
A listing of commercial or public-domain software capable of processing CheMin XRD data |
CheMin team |
The purpose of this directory is to provide the reader with a list of software that can process and analyze CheMin XRD data. The software listed will be a partial listing, since there are many commercial and public-domain programs that may be used, with varying degrees of sophistication in XRD pattern processing. The list provided will represent software commonly used by various members of the CheMin team, but the ASCII format in which CheMin XRD RDRs are provided can be handled by most software.
This software listing does not cover processing of energy histograms (EDS data). The EDS data are interpretable from widely available lookup tables of X-ray energy. Quantification of EDS data is desired but currently not a CheMin RDR product. Study is being devoted to EDS quantification and, should this prove practical, an amendment will be made to the CheMin RDR archive.
This section describes the format of CheMin RDR Archive Volumes. Data that comprise the Archive are formatted in accordance with Planetary Data System specifications [Applicable Documents 4 and 5].
This section describes file formats for the kinds of files contained on Archive Volumes.
Document files with the .TXT suffix exist in all directories except Data. They are ASCII files which may have embedded PDS labels. Lines in a .TXT file end with a carriage return character (ASCII 13) and a line feed character (ASCII 10). This allows the files to be readable under various operating systems.
Documents in the Document directory may contain formatting and figures that cannot be rendered as ASCII text. These documents are given as PDF (Portable Document Format) files, which may be viewed with free PDF reader software. To comply with the PDS requirement that a text copy of the documentation be archived, an HTML version of each PDF file is also provided.
Tabular files (.TAB suffix) exist in the Index directory and the Data directory. Tabular files are ASCII files formatted for direct reading into many database management systems on various computers. All fields are separated by spaces and/or commas, and character fields are enclosed in double quotation marks ("). (Character fields are padded with spaces to keep quotation marks in the same columns of successive records.) Character fields are left justified, and numeric fields are right justified. The "start byte" and "bytes" values listed in the labels do not include the commas between fields or the quotation marks surrounding character fields. The records are of fixed length, and the last two bytes of each record contain the ASCII carriage return (ASCII 13) and line feed (ASCII 10) characters. This allows a table to be treated as a fixed length record file on computers that support this file type and as a text file with embedded line delimiters on those that don't.
All tabular files in the archive are described by detached PDS labels. A label has the same name as the data file it describes, but with the extension .LBL. For examples of PDS labels for each type of CheMin data product, see the CheMin RDR Data Product SIS [Applicable Document 3]. Other types of files in the archive may have either detached or attached PDS labels. An attached label is embedded at the beginning of the file it describes.
A PDS label, whether attached or detached from its associated file, consists of lines of ASCII text in the form of keyword = value statements that provide descriptive information about the data file. The label is intended to be readable both by humans and by software. Details of the syntax and semantics of PDS labels can be found in the PDS Standards Reference (Applicable Document 5), and definitions of the keywords used in the label can be found by using the PDS Data Dictionary Lookup web service at http://pds.nasa.gov/tools/ddlookup/data_dictionary_lookup.cfm.
Lines of text in detached labels end with a carriage return character and a line feed character. This allows the files to be read under various operating systems.
No executable software files are posted within the CheMin Reduced Data Archive.
Catalog files (suffix .CAT) exist in the Root and Catalog directories. Like PDS labels, they are made up of keyword = value statements, but most of their content is descriptive text intended to be read by the user. They contain descriptions of the data set, instrument, spacecraft, and mission, as well as personnel contact information and references to published literature. They are called Catalog Files because they are loaded into the PDS online catalog to make the information available to users searching for data.
CheMin RDR data products are ASCII text tables with detached PDS labels, as described in section 3.1.2 above.
For more information about the format and content of the data products, see the CheMin RDR Data Product SIS [Applicable Document 3].
CheMin RDR data products are generated by the CheMin Science Team, either at JPL or at investigator institutions. The RDR data products are prepared from calibrated and transformed CheMin EDR (raw) data products according to the CheMin RDR Data Product SIS [3]. The team uses metadata from telemetry data headers to populate the PDS labels. The CheMin Team delivers data products to the Geosciences Node of the PDS, along with the other components of the archive for which the team is responsible, according to the archive delivery schedule in the MSL Archive Plan [2] and agreement between the CheMin Team and the Geosciences Node [6]. Deliveries occur approximately every 90 sols.
Table 5 summarizes expected sizes and production rates for the CheMin RDR Standard Products.
Table 5. CheMin Delivery Sizes and Rates
CheMin RDR Product ID |
CheMin RDR Product Type |
Product Size |
Production Rate (e.g., number of products per sol or per 90-sol delivery) |
Expected Number of Products for Primary Mission (669 sols) |
Expected Total Data Volume for Primary Mission |
RD1 |
Diffraction Single, K-alpha or K-beta |
816 Kb |
3 |
22 |
350 Kb |
RDS |
Diffraction Split, K-alpha or K-beta |
816 Kb |
3 |
22 |
350 Kb |
RDA |
Diffraction All, K-alpha or K-beta |
816 Kb |
<1 |
8 |
130 Kb |
RTR |
Diffraction All, Raw |
816 Kb |
<1 |
5 |
80 Kb |
RDF |
Diffraction, Film Mode |
816 Kb |
<1 |
5 |
80 Kb |
REA |
Energy All |
100 Kb |
3 |
22 |
2200 Kb |
RE1 |
Energy Single |
100 Kb |
3 |
22 |
2200 Kb |
RES |
Energy Split |
100 Kb |
3 |
22 |
2200 Kb |
MIN |
Mineral Identification and Abundance |
5 Kb |
3 |
22 |
100 Kb |
All volumes in the CheMin RDR Standard Product Archive conform to ISO 9660 standards as specified in the PDS Standards Reference [5].
The PDS Geosciences Node maintains backup copies of MSL CheMin RDR archives according to its general policy for backups, including off-site storage belonging to the Node and long-term storage at the National Space Science Data Center.
It is expected that CheMin RDR data products may be revised during the course of the mission due to improvements in the understanding of the instrument, better calibration information, etc. When the CheMin Team delivers revised versions of CheMin RDR data products to the PDS, the team must identify the revisions at or before the time of delivery. Labels for revised products must have a later value for the PRODUCT_CREATION_TIME keyword and an updated value for the PRODUCT_VERSION_ID keyword. They may also have a different value in the VERSION field of the file name as described in section 2.2, Data Directory Contents and Naming. The reasons for the revisions must be documented in ERRATA.TXT.
Previous versions of revised products are handled according to PDS policy for "draft" data sets, as the CheMin RDR archive is considered a work in progress until the final delivery. If revisions consist of minor changes to labels or corrections of typographical errors, the previous versions are not retained. If the revisions are done to correct serious errors in previously released products such that the older products are unfit for use, then the older products are not retained. In the typical case of revisions done as part of routine updates in calibration, etc., the Geosciences Node retains previous versions, making them accessible to users, for at least six months.
The CheMin RDR Archive has unique PDS data set identifiers that reflect either Level 1B data (labeled MSL-M-CHEMIN-4-RDR-V1.0) or Level 2 data (labeled MSL-M-CHEMIN-5-RDR-V1.0). Use of separate labels allows for expected multiple or iterative analyses of mineral abundances or for advanced pattern analysis methods, based on the methods used in generating Level 2 mineral identification and abundance data, separate from Level 1B data processing.
The volume on which the CheMin RDR Archive is stored has the PDS-assigned unique identifier MSLCMN_1XXX. This is the value of the keyword VOLUME_ID in catalog files and in the index table.
If ever the archive is divided among multiple physical volumes for data transfer or for another reason, the XXX in the volume ID is replaced by sequential volume numbers 001, 002, etc.
The CheMin Principal Investigator is David Blake, NASA Ames Research Center. The CheMin Archive Representatives are CheMin Science Team members David Vaniman, Planetary Science institute, and Michael Wilson, NASA Ames Research Center. The MSL Deputy Project Scientist, who is responsible for coordinating MSL archiving, is Joy Crisp, Jet Propulsion Laboratory.
The PDS Geosciences Node Archivist for the CheMin RDR Archive is Susan Slavney, Washington University, St. Louis; she is also the MSL Mission Lead for PDS. The PDS Geosciences Node Manager is Raymond Arvidson, Washington University. The PDS Data Engineer for MSL is Betty Sword, PDS Engineering Node, Jet Propulsion Laboratory.