NASA Logo

NTRS

NTRS - NASA Technical Reports Server

Back to Results
A Spatiotemporal Indexing Approach for Efficient Processing of Big Array-Based Climate Data with MapReduceClimate observations and model simulations are producing vast amounts of array-based spatiotemporal data. Efficient processing of these data is essential for assessing global challenges such as climate change, natural disasters, and diseases. This is challenging not only because of the large data volume, but also because of the intrinsic high-dimensional nature of geoscience data. To tackle this challenge, we propose a spatiotemporal indexing approach to efficiently manage and process big climate data with MapReduce in a highly scalable environment. Using this approach, big climate data are directly stored in a Hadoop Distributed File System in its original, native file format. A spatiotemporal index is built to bridge the logical array-based data model and the physical data layout, which enables fast data retrieval when performing spatiotemporal queries. Based on the index, a data-partitioning algorithm is applied to enable MapReduce to achieve high data locality, as well as balancing the workload. The proposed indexing approach is evaluated using the National Aeronautics and Space Administration (NASA) Modern-Era Retrospective Analysis for Research and Applications (MERRA) climate reanalysis dataset. The experimental results show that the index can significantly accelerate querying and processing (10 speedup compared to the baseline test using the same computing cluster), while keeping the index-to-data ratio small (0.0328). The applicability of the indexing approach is demonstrated by a climate anomaly detection deployed on a NASA Hadoop cluster. This approach is also able to support efficient processing of general array-based spatiotemporal data in various geoscience domains without special configuration on a Hadoop cluster.
Document ID
20170003163
Document Type
Reprint (Version printed in journal)
Authors
Li, Zhenlong (South Carolina Univ. Columbia, SC, United States)
Hu, Fei (George Mason Univ. Fairfax, VA, United States)
Schnase, John L. (NASA Goddard Space Flight Center Greenbelt, MD United States)
Duffy, Daniel Q. (NASA Goddard Space Flight Center Greenbelt, MD United States)
Lee, Tsengdar (NASA Headquarters Washington, DC United States)
Bowen, Michael K. (Patuxent Technology Partners, LLC Greenbelt, MD, United States)
Yang, Chaowei (George Mason Univ. Fairfax, VA, United States)
Date Acquired
April 7, 2017
Publication Date
January 12, 2016
Publication Information
Publication: International Journal of Geographical Information Science
Volume: 31
Issue: 1
ISSN: 1365-8816
Subject Category
Computer Systems
Report/Patent Number
GSFC-E-DAA-TN40490
Funding Number(s)
CONTRACT_GRANT: NNG14HH38I
CONTRACT_GRANT: NSF IIP-1338925
CONTRACT_GRANT: NNG13HQ01C
CONTRACT_GRANT: NNG12PP37I
CONTRACT_GRANT: NSF CNS-1117300
CONTRACT_GRANT: NSF PLR-1349259
CONTRACT_GRANT: NSF ICER-1343759
Distribution Limits
Public
Copyright
Other
Keywords
big data
MapReduce
climate analytics