NASA Logo, External Link
Facebook icon, External Link to NASA STI page on Facebook Twitter icon, External Link to NASA STI on Twitter YouTube icon, External Link to NASA STI Channel on YouTube RSS icon, External Link to New NASA STI RSS Feed AddThis share icon
 

Record Details

Record 1 of 1
Re-Organizing Earth Observation Data Storage to Support Temporal Analysis of Big Data
NTRS Full-Text: Click to View  [PDF Size: 936 KB]
Author and Affiliation:
Lynnes, Christopher(NASA Goddard Space Flight Center, Greenbelt, MD, United States)
Abstract: The Earth Observing System Data and Information System archives many datasets that are critical to understanding long-term variations in Earth science properties. Thus, some of these are large, multi-decadal datasets. Yet the challenge in long time series analysis comes less from the sheer volume than the data organization, which is typically one (or a small number of) time steps per file. The overhead of opening and inventorying complex, API-driven data formats such as Hierarchical Data Format introduces a small latency at each time step, which nonetheless adds up for datasets with O(10^6) single-timestep files. Several approaches to reorganizing the data can mitigate this overhead by an order of magnitude: pre-aggregating data along the time axis (time-chunking); storing the data in a highly distributed file system; or storing data in distributed columnar databases. Storing a second copy of the data incurs extra costs, so some selection criteria must be employed, which would be driven by expected or actual usage by the end user community, balanced against the extra cost.
Publication Date: Dec 11, 2017
Document ID:
20170012171
(Acquired Jan 03, 2018)
Subject Category: COMPUTER SYSTEMS
Report/Patent Number: GSFC-E-DAA-TN49796
Document Type: Oral/Visual Presentation
Meeting Information: AGU Fall Meeting; 11-15 Dec. 2017; New Orleans, LA; United States
Meeting Sponsor: American Geophysical Union; Washington, DC, United States
Financial Sponsor: NASA Goddard Space Flight Center; Greenbelt, MD, United States
Organization Source: NASA Goddard Space Flight Center; Greenbelt, MD, United States
Description: 27p; In English
Distribution Limits: Unclassified; Publicly available; Unlimited
Rights: No Copyright; Work of the U.S. Government - Public use permitted
NASA Terms: EARTH OBSERVING SYSTEM (EOS); DATA SYSTEMS; INFORMATION SYSTEMS; EARTH SCIENCES; DATA MANAGEMENT; DATA STORAGE; DOCUMENTS; DATA BASES; COSTS; TIME SERIES ANALYSIS
Other Descriptors: DATA STORAGE; TEMPORAL ANALYSIS; BIG DAT
› Back to Top
Find Similar Records
NASA Logo, External Link
NASA Official: Gerald Steeman
Site Curator: STI Program
Last Modified: January 03, 2018
Contact Us