THEMIS – ERG Presentation TDAS Usage of the CDF File Format Jim

THEMIS - ERG Presentation TDAS Usage of the CDF File Format Jim Lewis UCB Space Sciences Lab THEMIS – ERG Presentation March 18-19, 2010

THEMIS - ERG Presentation Introduction TDAS has adopted the CDF file format as its primary data storage mechanism. CDF libraries and documentation are available from http://cdf.gsfc.nasa.gov/ Supported languages include IDL, C, Fortran, Java, Perl Supported platforms include Windows, MacOS, Linux, Solaris, and others CDF files contain data variables with fixed-size records. Available data types include signed or unsigned integers with various bit depths; single or double precision floating point, CDF EPOCH (8 byte) or CDF EPOCH16 (16 byte) timestamps, and fixed-length character strings. THEMIS – ERG Presentation March 18-19, 2010

THEMIS - ERG Presentation CDFs contain metadata describing global properties (mission name and description, PI, informative links) and per-variable properties (units, coordinate systems, associations with related “support data” variables specifying sample times, axis labels, etc.) The CDF libraries hide platform-specific details like floating point representations or big-endian versus little-endian conventions. The CDF format imposes very few constraints on how data may be represented, but for interoperability between projects, some agreed-upon data and metadata conventions are desirable. THEMIS – ERG Presentation March 18-19, 2010

THEMIS - ERG Presentation L0, L1, and L2 processing flows for THEMIS L0 CCSDS packets (a raw binary, non-CDF format) are extracted from incoming telemetry files. Packet-level decompression is applied if necessary. Corrupted data is identified when possible, and removed from L0 products. L0 packets require deep knowledge of S/C and instrument subsystems to be usefully interpreted, and are not intended for end users. Exception: TDAS still uses L0 products for working with the ESA instrument, but plans are in place to migrate to use of L1 data. L0 packets are processed to produce L1 CDFs. For THEMIS, the most important operation at this stage is time-tagging individual samples, since each L0 packet contains multiple samples but only a single timestamp. THEMIS – ERG Presentation March 18-19, 2010

THEMIS - ERG Presentation L1 CDFs are uncalibrated, and are expressed in each instrument’s native units and coordinate system. L1 CDFs are combined with calibration, ephemeris, and attitude data to produce L2 CDFs. L2 CDFs are calibrated, and data is expressed in standard engineering units and geophysically relevant coordinate systems. They represent the THEMIS “key parameter data”. THEMIS – ERG Presentation March 18-19, 2010

THEMIS - ERG Presentation For interoperability with SPDF, THEMIS uses the ISTP metadata conventions described at http://spdf.gsfc.nasa.gov/sp use of cdf.html ISTP specifies a required set of global attributes describing the mission, the instrument that produced the data set, and links to mission documentation. ISTP specifies a required set of per-variable attributes describing each variable, its associations with other variables appearing in that CDF, units, coordinate systems, etc. A detailed list of ISTP requirements is beyond the scope of this presentation, but we can provide examples of how they’re implemented on THEMIS. Further documentation is available in the THEMIS L1 File Definition document. THEMIS – ERG Presentation March 18-19, 2010

THEMIS - ERG Presentation THEMIS-specific metadata conventions TDAS uses a variable attribute DEPEND TIME to associate each data variable with a time variable, which should be a doubleprecision floating point quantity representing a Unix timestamp (seconds since 1970-01-01). ISTP requires a variable attribute DEPEND 0, which should refer to a variable of type CDF EPOCH or CDF EPOCH16. It is impractical to store both sets of timestamps due to file size considerations. The CDF EPOCH variable exists in the CDF, but contains no data. Instead, the CDF EPOCH values are to be calculated from the Unix epoch and timestamps. THEMIS and SPDF use the DATA VERSION global attribute and file naming convention somewhat differently. THEMIS – ERG Presentation March 18-19, 2010

THEMIS - ERG Presentation How TDAS load routines work Basic operating principle: given a data source (e.g. tha, thb, FYKN), a data level (L1 or L2), a data version (V00, V01, V02), data type (FGM, SCM, ASI, GMAG, etc), and a time range (usually dates, sometimes dates times), generate a list of filenames or URLs from which data variables need to be loaded. File management: For each supported mission, TDAS maintains user-configurable parameters describing the remote filesystem or URL root, and a local filesystem root. TDAS can be configured to ignore the remote data location and work directly from the local data repository. Since THEMIS data is freely available as soon as it’s processed, TDAS contains no provisions for passwords, cookies, certificates, or other authentication or authorization methods. THEMIS – ERG Presentation March 18-19, 2010

THEMIS - ERG Presentation After the remote data is downloaded to the local repository, the requested variables, time tags, and metadata attributes are read from the local repository using CDF library commands. Data from multiple files is aggregated into a single entity (a TPLOT variable), which contains data values, timestamps, and metadata values. The TDAS load routines contain “hooks” for post-processing steps that can be performed after each data set is loaded. This allows data loading, calibration, and coordinate transformations to be performed with a single call to the load routine. Calibration and coordinate transforms may require combining data from several types of CDFs. For example, the attitude and ephemeris data in the THEMIS L1 STATE CDFs are required (and can be automatically loaded) for many choices of output coordinates. THEMIS – ERG Presentation March 18-19, 2010

THEMIS - ERG Presentation Developing TDAS-compatible CDFs New CDF types should have all required ISTP metadata. Data variables should have the THEMIS DEPEND TIME attribute defined to point to a variable containing double-precision Unix timestamp data. The CDF naming convention should support the mapping of data source, processing level, data type, data version, and time range onto a list of URLs or filenames. The CDF data should be made available either by HTTP download or a locally available filesystem. If HTTP downloads are not feasible due to access control issues, the TDAS remote/local file management functionality can be bypassed, so that TDAS only looks in the local repository, and file distribution to end users will occur outside of TDAS. THEMIS – ERG Presentation March 18-19, 2010

THEMIS - ERG Presentation Tradeoff considerations for design of CDF file structures Download time, versus processing time, versus level of interoperability. Example: THEMIS L2 FGM CDFs contain the same magnetometer data in several coordinate systems. We discovered that it was faster to load, calibrate and cotrans the smaller L1 data sets “on the fly” than it was to download the larger L2 files. So why include redundant data in the L2 files? End users might want to import it into some other toolchain which doesn’t support the required calibration or coordinate transform functions. Multiple CDF types, versus everything in one file. The original THEMIS concept was to have one daily data CDF file, combining all instruments. But this would have introduced a great deal of “churn”, whenever reprocessing was necessary to fix a processing bug or add a CDF variable. Splitting the CDFs by instrument type gave us finer-grained control of reprocessing required to implement changes or bug fixes. THEMIS – ERG Presentation March 18-19, 2010

THEMIS - ERG Presentation Summary A CDF is a platform-independent container for storing data, support data, and metadata. SPDF and TDAS have different, but compatible, conventions for how CDF metadata is expressed. TDAS naming conventions allow client software to generate a list of filenames or URLs to load, given the type and time range of data requested. TDAS load routines take CDFs as input, and produce TPLOT variables as output. TDAS calibration, cotrans, and analysis routines take TPLOT variables as input, and produce new TPLOT variables as output. Command-line plotting routines generate plots from TPLOT variables. The TDAS GUI imports/exports between TPLOT variables and its own internal format. THEMIS – ERG Presentation March 18-19, 2010

THEMIS - ERG Presentation Overview TDAS software extends SSL routines for THEMIS mission. Collaborative software development through SVN. Combines mission specific routines for loading/calibration/coordinate transformation. With general purpose analysis routines. Made possible by acceptance of common conventions for data representation inside IDL. THEMIS – ERG Presentation March 18-19, 2010

THEMIS - ERG Presentation SVN SVN allows provides automatic version management. Merges additions/changes from multiple programmers at different locations. Identifies conflicts if programmers modify same code. Tracks modification history. Provides central repository for code. Server side scripts create releases/nightly builds/processing builds automatically. THEMIS – ERG Presentation March 18-19, 2010

THEMIS - ERG Presentation Command Line Loading If CDF conventions are followed load routines can be developed using two techniques. #1 thm load xxx, highly parameterized routine does all the work. #2 Use file retrieve/file http copy to download file. cdf2tplot loads data into tplot variables. Simple example of #1 themis/spacecraft/thm load bau.pro Simple example of #2 themis/spacecraft/thm load scmode.pro THEMIS – ERG Presentation March 18-19, 2010

THEMIS - ERG Presentation GUI Loading Requires command line load routine Requires interface panel specific to mission to load data. Top level base must fit inside IDL widget tab. Widget ID provided as parameter. (see: themis/thm ui new/thm ui init load window.pro) Load panel must, call CL load routine, add loaded tplot variables to GUI thm ui loaded data object, and cleanup tplot variables after load. Tplot variables and GUI variables are stored in separate namespace to prevent accidental collision/interaction with CL routines. THEMIS – ERG Presentation March 18-19, 2010

THEMIS - ERG Presentation Topic #1 Development/Management of TDAS programs. A: ERG folder/development branch. Discussion Item: How often do you plan to update ERG branch? Multiple times a day? Once a week? How quickly will ERG code be changing? Agree maintaining separate repositories is ideal. But we need a policy to prevent development fork and maintain compatibility between ERG & TDAS Suggest we provide you authentication for our SVN server. Automated process(nightly) or programmer(periodically) checks out ERG branch from ERG servers, checks into TDAS servers. We can provide example scripts that perform automatic svn builds. NOTE: The less frequently that copies are synchronized the greater the probability that a fork occurs which prevents backward compatibility. This has occurred in the past between TDAS and WIND. Would like to avoid in the future, if doesn't require too much effort. We also have SVN email notifications that occur when changes are made to our repositories. We can put your developers and/or scientists on this list, so that you will be aware of changes throughout TDAS. THEMIS – ERG Presentation March 18-19, 2010

THEMIS - ERG Presentation Topic #1 Development/Management of TDAS programs. B: General purpose General Purpose(GP) routines present greater risk of compatibility problems. But we recognize modifications/additions to GP routines are necessary and beneficial. Initially, we recommend that any routines are added to the ERG branch, with 'erg ' prefix. Additions will be checked into ssl general with prefix removed after code review. Discussion Items: Do you have any formal QA processes in mind? (QA Scripts, Test Suites, and/or automated unit tests) What policies do you have in mind for Help & Bug reports? Any plans on synchronization with TDAS repositories? THEMIS – ERG Presentation March 18-19, 2010

THEMIS - ERG Presentation Topic #2 Naming conventions Informal convention is: A all routines for a mission should use the same prefix (ex: tha , erg ) B all tplot variables for a mission should use the same prefix (ex: tha fgs dsl, erg state pos) C For ease of load, it is generally useful to maintain these designations inside the CDF, as well. D Can setup mailing list and/or periodic meetings for technical discussions as needed. THEMIS – ERG Presentation March 18-19, 2010

THEMIS - ERG Presentation Topic 3: Rules of the road Currently we maintain rules of the road in CDFs and on THEMIS website, but do not explicitly post them from within the software. External package IDL GEOPACK, posts rules of the road to command line upon first usage during a session. Most TDAS missions use common initialization routines( ie thm init) Could add to erg init; would present notification, and update data structure to remember preference; same technique used to store configuration parameters could remember user response. Should automatically appear when using the GUI if erg load routine were added to themis gui. Side note: It is important to not assume the presence of a graphical display. Recommend either using only command-line query or checking display availability and falling back to CL. THEMIS – ERG Presentation March 18-19, 2010

THEMIS - ERG Presentation Topic #4 Access control Uncertain if we can support CLUSTER-style access controls at this time. THEMIS doesn't have any explicit access control system. Does support multiple url-configurable repositories. If a repository provides access it can download. Individual repositories can control access based upon their own criterion. For more restricted access, users are required to manually download and place data in appropriate directory. Potential ways to extend access control. Extend file http copy to support https, Create download utility which uses IDLnetURL objects. On command line access control would probably be coupled with the download process, but cdf2tplot and analysis could be completely independent. THEMIS – ERG Presentation March 18-19, 2010

THEMIS - ERG Presentation Topic #5: 2d plotting routines Support some 2d plotting routines tplot:specplot, for time-series data. plotxy & tplotxy: for isotropic 2d line data. plotxyz: for isotropic 3d line data. plotxyvec: for isotropic 2d vector arrows. accgm plot: for generating maps/coordinate grids using aacgm coordinates thm map set & thm map add for creating map-mosiacs Additional routines & improvements are made as specific requests are received and prioritized. THEMIS – ERG Presentation March 18-19, 2010

THEMIS - ERG Presentation Topic #6 IDL VM Testing has not been performed with GUI in VM to verify. (We may have missed places, or there may be VM compatibility issues that are undocumented) GUI using VM should be possible. When developing THEMIS GUI we made sure to avoid all restrictions on IDL VM. Releasing on VM would require creating an IDL save file of the GUI and debugging any platform related issues. Difficulty would depend largely on the extent of undocumented compatibility issues and the time to fix those issues. TDAS-Web: Unaware of any technical limitations. THEMIS – ERG Presentation March 18-19, 2010

THEMIS - ERG Presentation Topic #7 GUI development What is CUI? GUI has many more inter-dependencies than CL routines. Development coordination will be a much more significant task. Currently there is not any standard API for extension of GUI. At the moment extension would require direct modification of GUI internals. GUI interfaces via several objects: thm ui loaded data(read/write data products) , thm ui windows, thm ui window(top level of display settings hierarchy), thm ui draw object(Redraw, query information about displayed panels, manage real-time features), thm ui call sequence(Record actions for document replay), thm ui message bar, thm ui history(logging) There are also a number of standard panel widgets(calender, spinner, data tree) THEMIS – ERG Presentation March 18-19, 2010

THEMIS - ERG Presentation Topic #8 User statistics THEMIS Statistics collected by SSL servers. What statistics are required? Current mechanism does not provide more detailed statistics. THEMIS – ERG Presentation March 18-19, 2010

THEMIS - ERG Presentation Examples: Other missions TDAS CL supports ACE, FAST, GOES, KYOTO(DST), LANL, STEREO, WIND. Several GMAG & ASI networks: UCLAGBO,UCLA-EPO, CARISMA(UAlberta), MACCS(Augsburg), GIMA(UAlaska). See ssl general/missions/ace/ for examples. (ace init.pro, ace load swepam.pro) TDAS GUI supports ACE,WIND,GOES, plus all GMAGS. See themis/thm ui new/panels/thm ui load data file/ for examples. (thm ui init load window.pro,thm ui goes data.pro,thm ui g oes data load.pro) THEMIS – ERG Presentation March 18-19, 2010

THEMIS - ERG Presentation THEMIS Data Analysis Software Graphical User Interface THEMIS – ERG Presentation March 18-19, 2010

THEMIS - ERG Presentation THEMIS – ERG Presentation March 18-19, 2010

THEMIS - ERG Presentation Main GUI Window THEMIS – ERG Presentation March 18-19, 2010

THEMIS - ERG Presentation Load Data Window THEMIS – ERG Presentation March 18-19, 2010

THEMIS - ERG Presentation Plot Layout Window THEMIS – ERG Presentation March 18-19, 2010

THEMIS - ERG Presentation Resulting Display from Plot/Layout Selections THEMIS – ERG Presentation March 18-19, 2010

THEMIS - ERG Presentation Line Options Window THEMIS – ERG Presentation March 18-19, 2010

THEMIS - ERG Presentation Results from Line Option Selections THEMIS – ERG Presentation March 18-19, 2010

THEMIS - ERG Presentation Data Analysis Window THEMIS – ERG Presentation March 18-19, 2010

THEMIS - ERG Presentation Save Data Window THEMIS – ERG Presentation March 18-19, 2010

THEMIS - ERG Presentation Save Image To File Window THEMIS – ERG Presentation March 18-19, 2010

THEMIS - ERG Presentation Special Features Zoom Control Legend Box Marked Area Of Interest Cursor Tracking Status Bar THEMIS – ERG Presentation March 18-19, 2010

THEMIS - ERG Presentation History Window and Help Request Form THEMIS – ERG Presentation March 18-19, 2010

THEMIS - ERG Presentation THEMIS – ERG Presentation March 18-19, 2010