DYNAMIC ALBEDO OF NEUTRONS
RDR ARCHIVE VOLUME
SOFTWARE INTERFACE
SPECIFICATION
July 19, 2012
Prepared by:
___________________________________
Karl Harshman
Approved by:
___________________________________
Igor Mitrofanov
Principal Investigator, Lunar Exploration Neutron Detector
Approved by:
___________________________________
Edwin Grayzeck
PDS Program Manager
TABLE OF CONTENTS
1. Introduction................................................................................................................................. 1
1.1. Purpose and Scope............................................................................................................... 1
1.2. Content Overview................................................................................................................ 1
1.3. Applicable Documents and Constraints............................................................................... 1
1.4. Relationships with Other Interfaces.................................................................................... 2
2. Archive Volume Contents........................................................................................................... 2
2.1. Root Directory Contents...................................................................................................... 2
2.2. Data Directory Contents and Naming................................................................................. 3
2.3. Index Directory Contents.................................................................................................... 4
2.4. Document Directory Contents............................................................................................. 4
2.5. Catalog Directory Contents................................................................................................. 4
2.6. Label Directory Contents..................................................................................................... 5
2.7. Calibration Directory Contents............................................................................................ 5
2.8. Software Directory Contents............................................................................................... 6
3. Archive Volume Format.............................................................................................................. 6
3.1. File Formats......................................................................................................................... 6
3.1.1. Document File Format............................................................................................... 6
3.1.2. Tabular File Format................................................................................................... 7
3.1.3. PDS Label Format..................................................................................................... 7
3.1.4. Software File Format................................................................................................. 7
3.1.5. Catalog File Format................................................................................................... 7
3.1.6. Science Data File Formats......................................................................................... 7
4. Archive Volume Generation........................................................................................................ 8
4.1. Data Transfer and Validation Methods............................................................................... 8
4.2. Data Product Sizes and Delivery Rates............................................................................... 8
4.3. Interface Media Characteristics........................................................................................... 8
4.4. Backup and Duplicates........................................................................................................ 8
4.5. Labeling and Identification................................................................................................. 8
5. Support Staff and Cognizant Persons......................................................................................... 9
DOCUMENT CHANGE LOG
Change |
Date |
Affected Portions |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
TBD ITEMS
Section |
Description |
|
|
ACRONYMS AND ABBREVIATIONS
ASCII |
American Standard Code for Information Interchange |
EDR |
Experiment Data Record |
ISO |
International Standards Organization |
NSSDC |
National Space Science Data Center |
PDS |
Planetary Data System |
RDR |
Reduced Data Record |
SIS |
Software Interface Specification |
TBD |
To Be Determined |
GLOSSARY
Archive – An archive consists of one or more data sets along with all the documentation and ancillary information needed to understand and use the data. An archive is a logical construct independent of the medium on which it is stored.
Archive Volume, Archive Volume Set – A volume is a unit of media on which data products are stored; for example, one CD-ROM or DVD-ROM. An archive volume is a volume containing all or part of an archive; that is, data products plus documentation and ancillary files. When an archive spans multiple volumes, they are called an archive volume set. Usually the documentation and some ancillary files are repeated on each volume of the set, so that a single volume can be used alone.
Catalog Information – Descriptive information about a data set (e.g. mission description, spacecraft description, instrument description), expressed in Object Description Language (ODL) which is suitable for loading into a PDS catalog.
Data Product – A labeled grouping of data resulting from a scientific observation, usually stored in one file. A product label identifies, describes, and defines the structure of the data. An example of a data product is a planetary image, a spectrum table, or a time series table.
Data Set – An accumulation of data products. A data set together with supporting documentation and ancillary files is an archive.
Standard Data Product – A data product generated in a predefined way using well-understood procedures, processed in "pipeline" fashion. Data products that are generated in a nonstandard way are sometimes called special data products.
This Software Interface Specification is intended to be used by those who wish to understand the format and content of the Mars Science Laboratory, MSL Dynamic Albedo of Neutron Detector, DAN Reduced Data Record, RDR Archive. Typically, these individuals would be software engineers, data analysts, or planetary scientists.
The specifications in this document apply to all MSL DAN RDR standard product archive volumes that are generated by the MSL DAN Project.
The MSL DAN RDR archive includes the following five types of products.
DAN_DERIVED_ENG Reduced housekeeping data
DAN_DERIVED_PASSIV Reduced passive science data
DAN_DERIVED_ACTIVE Reduced active science data.
DAN_AVERAGED_PASSIV Normalized, averaged passive data
DAN_AVERAGED_ACTIVE Normalized, averaged active data
The RDR data products are generated by the DAN team at the University of Arizona and delivered to the PDS Geosciences Node at Washington University, St. Louis.
This Software Interface Specification (SIS) describes the format, content, and generation of the MSL DAN RDR Archive. Section 2, Archive Volume Generation, describes the procedure for transferring data products to archive media. Section 3, Archive Volume Contents, describes the structure of the archive volumes and the contents of each file. Section 4, Archive Volume Format, describes the file formats used on the archive volumes. Finally, Section 5, Support Staff and Cognizant Persons, lists the individuals responsible for generating the archive volumes.
This Data Product SIS is responsive to the following MSL documents:
1. Mars Exploration Program Data Management Plan, Arvidson et al., Rev. 4.0, June 15, 2011.
2. Mars Science Laboratory Project Archive Generation, Validation and Transfer Plan, Joy Crisp, JPL D-35281, May 28, 2010.
3. DAN FDD, JPL D-34220, MSL 375-1230
4. MSL DAN EDR SIS, JPL D-38113, SIS-SCI014-MSL.
This SIS is also consistent with the following Planetary Data System documents:
Finally, this SIS is meant to be consistent with the contract negotiated between the Mars Science Laboratory and the DAN Principal Investigator (PI) in which reduced data records and documentation are explicitly defined as deliverable products.
This Archive Volume SIS could be affected by changes to the design of the MSL DAN standard data products (Applicable Documents 2, 3 and 4).
This section describes the contents of the MSL DAN Archive volumes, including the file names, file contents, file types, and organization responsible for providing the files.
DAN RDR archive volumes will include the following directories.
Directory Contents
Volume root directory Introduction and errata text files
CALIBRATION Calibration Documents files
CATALOG Descriptions of data set, instruments, spacecraft, and mission as
found in the PDS Catalog
DOCUMENT Documentation files
INDEX Volume and cumulative index tables
LABEL Descriptions of data file formats, referenced by PDS labels
DATA Data files and their detached PDS labels
The contents of these directories are described in detail in Sections 2.1 to 2.7.
Files in the Root Directory include an overview of the archive, a description of the volume for the PDS Catalog, and a list of errata or comments about the archive. The following files are contained in the Root Directory.
File Name |
File Contents |
File Provided By |
AAREADME.TXT |
Volume content and format information |
DAN |
ERRATA.TXT |
A cumulative listing of comments and updates concerning all archive volumes published to date |
DAN |
VOLDESC.CAT |
A description of the contents of this volume in a PDS format readable by both humans and computers |
DAN |
DAN RDR data products are aggregated by sol; i.e., one product contains one sol’s worth of data. Sol is defined to be one Mars day. Data for each sol day will be in a subdirectory named in the format DDD, indicating the sol of the data acquisition period. Within each subdirectory will be up to five PDS labels and associated data files for the five product types listed in section 1.2.
This schematic shows an example of the nesting of directories for data acquired on Sol 0002.
<Volume Root>
-------------------
|
-DATA
|- SOL0002
| |
| | -- <DAN_DERIVED_ENG files acquired on Sol 2>
| |
| | --<DAN_DERIVED_PASSIV files acquired on Sol 2>
| |
| | --<DAN_DERIVED_ACTIVE files acquired on Sol 2>
| |
| | --<DAN_AVERAGED_PASSIV files acquired on Sol 2>
| |
| | --<DAN_AVERAGED_ACTIVE files acquired on Sol 2>
| |
File Name |
File Contents |
File Provided By |
DNA_351797691REN_0550000000_______M1.DAT,.LBL |
Housekeeping data |
DAN |
DNA_351797691RPA_0550000000_______M1.DAT,.LBL |
Passive science data |
DAN |
DNA_351797691RAC_0550000000_______M1.DAT,.LBL |
Active science data |
DAN |
DNA_351797691RAP_0550000000_______M1.DAT,.LBL |
Averaged passive data |
DAN |
DNA_351797691RAA_0550000000_______M1.DAT,.LBL |
Averaged active data |
DAN |
Files in the Index Directory are provided to help the user locate products on this archive volume and on previously released volumes in the archive. The following files are contained in the Index Directory.
File Name |
File Contents |
File Provided By |
INDXINFO.TXT |
A description of the contents of this directory |
DAN |
INDEX.TAB |
A table listing all data products on this volume |
DAN |
INDEX.LBL |
A PDS detached label that describes INDEX.TAB |
DAN |
The Document Directory contains documentation to help the user understand and use the archive data. The following files are contained in the Document Directory.
File Name |
File Contents |
File Provided By |
DOCINFO.TXT |
A description of the contents of this directory |
DAN |
DAN_RDR_SIS.PDF |
The Data Product SIS as a PDF file |
DAN |
DAN_RDR_SIS.HTM |
The Data Product SIS as an HTML file |
DAN |
DAN_RDR_SIS.LBL |
A PDS detached label that describes DAN_RDR_SIS.PDF and HTML |
DAN |
DAN_ARCHIVE_VOL_SIS.PDF |
The Archive Volume SIS (this document) as a PDF file |
DAN |
DAN_ARCHIVE_VOL_SIS.HTM |
The Archive Volume SIS (this document) as an HTML file |
DAN |
DAN_ARCHIVE_VOL_SIS.LBL |
A PDS detached label that describes both DAN_ARCHIVE_VOL_SIS.PDF and HTML. |
DAN |
PDSDD.FUL |
The complete PDS Data Dictionary |
Geosciences |
PDSDD.LBL |
A PDS detached label that describes PDSDD.FUL |
Geosciences |
MSL_LDD.FUL |
The MSL Local Data Dictionary |
Geosciences |
MSL_LDD.LBL |
A PDS detached label that describes MSL_LDD.FUL |
Geosciences |
Additional documents may be added.
The files in the Catalog Directory provide a top-level understanding of the mission, spacecraft, instruments, and data sets. The files in this directory are coordinated with the PDS data engineer, who is responsible for loading them into the PDS catalog. The following files are found in the Catalog Directory.
File Name |
File Contents |
File Provided By |
CATINFO.TXT |
A description of the contents of this directory |
DAN |
DAN_RDR_DS.CAT |
RDR Data set information for the PDS catalog |
DAN |
MSL_INSTHOST.CAT |
Instrument host (i.e., spacecraft) information for the PDS catalog |
MSL |
DAN_INST.CAT |
Instrument information for the PDS catalog |
DAN |
MSL_MISSION.CAT |
Mission information for the PDS catalog |
MSL |
DAN_PERSON.CAT |
Personnel information for the PDS catalog (Team and PDS personnel responsible for generating the archive) |
DAN |
DAN_REF.CAT |
References mentioned in other DAN*.CAT files |
DAN |
MSL_REF.CAT |
References mentioned in other MSL*.CAT files |
MSL |
The Label Directory contains files that describe data format and organization. These files are referred to in the PDS labels that accompany the data products. They are "include" files that are intended to be parsed as if they were part of the PDS labels that refer to them. The following files are contained in the Label Directory.
File Name |
File Contents |
File Provided By |
LABINFO.TXT |
A description of the contents of this directory |
DAN |
DAN_RDR_DERIVED_ENGFMT |
Housekeeping and Engineering data column descriptions |
DAN |
DAN_RDR_DERIVED_PASSIV.FMT |
Derived passive Neutron spectra data column descriptions |
DAN |
DAN_RDR_DERIVED_ACTIV.FMT |
Derived active Neutron spectra data column descriptions |
DAN |
DAN_RDR_AVERAGE_PASSIV.FMT |
Averaged passive Neutron spectra data column descriptions |
DAN |
DAN_RDR_AVERAGE_ACTIVE.FMT |
Averaged active Neutron spectra data column descriptions |
DAN |
|
|
|
The Calibration Directory contains calibration files used to process the data products, or calibration data needed to use the data products. The following files are contained in the Calibration Directory.
File Name |
File Contents |
File Provided By |
CALINFO.TXT |
A description of the contents of this directory |
DAN |
DAN_CALIBRATION_REPORT.PDF |
Description of instrument and data calibration performed by DAN team in a PDF document form |
DAN |
DAN_CALIBRATION_REPORT.HTM DAN_CALIBRATION_REPORT.LBL |
Description of instrument and data calibration performed by DAN team in an HTML document form Description of above document types
|
DAN |
|
|
|
There are currently no plans to provide software.
This section describes the format of MSL DAN RDR Archive Volumes. Data that comprise the Archive will be formatted in accordance with Planetary Data System specifications [Applicable Documents 5 and 6].
This section describes file formats for the kinds of files contained on Archive Volumes.
Document files with the .TXT suffix exist in the Root, Index, Catalog, Document, and Label directories. They are ASCII files which may have embedded PDS labels. Lines in a .TXT file end with a carriage return character (ASCII 13) and a line feed character (ASCII 10). This allows the files to be readable under various operating systems.
Documents in the Document directory may contain formatting and figures that cannot be rendered as ASCII text. Therefore each document is given in two formats, hypertext and PDF. The hypertext file contains ASCII text plus hypertext markup language (HTML) commands that enable it to be viewed in a Web browser such as Netscape Navigator or Microsoft Internet Explorer. The hypertext file may be accompanied by ancillary files such as images and style sheets that are incorporated into the document by the Web browser. The second format, PDF (Portable Document Format) is a proprietary format of Adobe Systems Incorporated that is frequently used for distributing documents. Adobe offers free software, Acrobat Reader, for viewing PDF files.
Tabular files (.TAB suffix) exist in the Index directory. Tabular files are ASCII files formatted for direct reading into many database management systems on various computers. All fields are separated by commas, and character fields are enclosed in double quotation marks ("). (Character fields are padded with spaces to keep quotation marks in the same columns of successive records.) Character fields are left justified, and numeric fields are right justified. The "start byte" and "bytes" values listed in the labels do not include the commas between fields or the quotation marks surrounding character fields. The records are of fixed length, and the last two bytes of each record contain the ASCII carriage return and line feed characters. This allows a table to be treated as a fixed length record file on computers that support this file type and as a text file with embedded line delimiters on those that don't.
All tabular files are described by PDS labels, either embedded at the beginning of the file or detached. If detached, the PDS label file has the same name as the data file it describes, with the extension .LBL; for example, the file INDEX.TAB is accompanied by the detached label file INDEX.LBL in the same directory.
All data files in the archive have PDS labels, either embedded at the beginning of the file or detached in a separate file. For examples of PDS labels for each type of data product, see the Data Product SISs [Applicable Documents 3 and 4].
A PDS label, whether embedded or detached from its associated file, consists of lines of ASCII text in the form of keyword = value statements that provide descriptive information about the data file. The label is intended to be readable both by humans and by software. Details of the syntax and semantics of PDS labels can be found in the PDS Standards Reference (Applicable Document 6), and definitions of the keywords used in the label can be found by using the PDS Data Dictionary Lookup web service at http://pds.jpl.nasa.gov/tools/data_dictionary_lookup.cfm.
Lines of text in detached labels end with a carriage return character (ASCII 13) and a line feed character (ASCII 10). This allows the files to be read under various operating systems.
None provided at this time.
Catalog files (suffix .CAT) exist in the Root and Catalog directories. Like PDS labels, they are text files formatted as keyword = value statements. They contain descriptions of the data set, instrument, spacecraft, and mission, as well as personnel contact information and references to published literature. They are called Catalog Files because they are loaded into the PDS online catalog to make the information available to users searching for data.
DAN RDR data files are in the form of binary time-series tables. The binary tables include columns of most-significant-byte-first (“big-endian”) unsigned integers. One file contains one sol’s worth of data.
For more information about the format and content of the data products, see the Data Product SISs [Applicable Documents 3 and 4].
DAN RDR products are generated by the DAN team at the Lunar and Planetary Laboratory, University of Arizona. The products are delivered to the PDS Geosciences Node at Washington University electronically using ftp according to the schedule in the MSL Data Management and Archive Plan (Applicable Document 1).
The DAN team releases RDR data products according to the schedule outlined in the MSL Archive Generation, Validation and Transfer Plan [Applicable Document 2].
All volumes in the MSL DAN RDR Standard Product Archive conform to ISO 9660 standards [ISO 9660, 1988].
Backup copies of DAN RDR products will be stored at the University of Arizona until the final versions of the products have been archived on physical media with the PDS.
Duplicate copies of DAN archive volumes on physical media will be stored at the PDS Geosciences Node, the PDS Central Node and the NSSDC.
Each DAN RDR archive volume will be identified by a unique volume ID formed according to the scheme MSLDAN_1XXX where:
MSL represents the mission,
DAN represents the instrument, and
1XXX is an RDR volume sequence number. If DAN RDR data need to be stored on multiple physical volumes, the XXX will be replaced with a volume sequence number starting with 001.
If the entire DAN RDR data set is revised, and the data set ID version number is incremented, the keyword VOLUME_VERSION_ID in the file VOLDESC.CAT will be incremented on volumes containing the revised data set.
Igor Mitrofanov, DAN Principal Investigator, Russian Space Institute
William Boynton, DAN Co Investigator, University of Arizona
Karl Harshman, DAN SOC Manager, University of Arizona
Questions and comments regarding the DAN RDR Archive Volume may be directed to the PDS Geosciences Node, Washington University, geosci@wunder.wustl.edu.