NASA Logo

NTRS

NTRS - NASA Technical Reports Server

Back to Results
A Low-Cost Clustered Archive Approach for Storing Remote Sensing DataAs part of NASA's Earth Observing System (EOS) Data and Information System (EOSDIS), the Moderate Resolution Imaging Spectrometer (MODIS) Data Processing System (MODAPS) is now processing data from two instruments on the EOS flag ship spacecraft Terra and Aqua. Between the two, MODAPS is generating over 1 Terrabyte of data per day and has surpassed 3 Petabytes of total data. The bulk of the data is stored near-line in StorageTek Powerhorn tape jukeboxes. Accessing data that has been moved to tape involves submitting an order, scheduling the tape, and waiting for the data to become available. I am developing a low cost clustered archive that could enable storing a very large amount of data such as the MODIS data described above in an organized fashion on a cluster of commodity hardware using low cost SATA hard drives such that the files are directly available online. The system takes full advantage of Open Source software, using the GNULinux operating system, PostgreSQL relational database and the Apache HTTP Server. This poster session will depict my approach, the interface to the archive, a brief discussion of the internals of the system and some performance numbers from my prototyping. I will also describe various costs and benefits of this approach versus the traditional large tape jukebox approach currently in use in the EOSDIS.
Document ID
20040081371
Acquisition Source
Goddard Space Flight Center
Document Type
Conference Paper
Authors
Tilmes, Curt
(NASA Goddard Space Flight Center Greenbelt, MD, United States)
Date Acquired
August 21, 2013
Publication Date
January 1, 2004
Subject Category
Computer Operations And Hardware
Meeting Information
Meeting: 3rd USENIX Conference on File and Storage Technologies
Location: San Francisco, CA
Country: United States
Start Date: January 1, 2004
Distribution Limits
Public
Copyright
Work of the US Gov. Public Use Permitted.

Available Downloads

There are no available downloads for this record.
No Preview Available