NASA Logo, External Link
Facebook icon, External Link to NASA STI page on Facebook Twitter icon, External Link to NASA STI on Twitter YouTube icon, External Link to NASA STI Channel on YouTube RSS icon, External Link to New NASA STI RSS Feed AddThis share icon
 

Record Details

Record 1 of 1
Using Docker Containers to Extend Reproducibility Architecture for the NASA Earth Exchange (NEX)
NTRS Full-Text: Click to View  [PDF Size: 94 KB]
Author and Affiliation:
Votava, Petr(California State Univ. at Monterey Bay, Seaside, CA, United States);
Michaelis, Andrew(California State Univ. at Monterey Bay, Seaside, CA, United States);
Spaulding, Ryan(ADNET Systems, Inc., Reston, VA, United States);
Becker, Jeffrey C.(Computer Sciences Corp., Moffett Field, CA, United States)
Abstract: NASA Earth Exchange (NEX) is a data, supercomputing and knowledge collaboratory that houses NASA satellite, climate and ancillary data where a focused community can come together to address large-scale challenges in Earth sciences. As NEX has been growing into a petabyte-size platform for analysis, experiments and data production, it has been increasingly important to enable users to easily retrace their steps, identify what datasets were produced by which process chains, and give them ability to readily reproduce their results. This can be a tedious and difficult task even for a small project, but is almost impossible on large processing pipelines. We have developed an initial reproducibility and knowledge capture solution for the NEX, however, if users want to move the code to another system, whether it is their home institution cluster, laptop or the cloud, they have to find, build and install all the required dependencies that would run their code. This can be a very tedious and tricky process and is a big impediment to moving code to data and reproducibility outside the original system. The NEX team has tried to assist users who wanted to move their code into OpenNEX on Amazon cloud by creating custom virtual machines with all the software and dependencies installed, but this, while solving some of the issues, creates a new bottleneck that requires the NEX team to be involved with any new request, updates to virtual machines and general maintenance support. In this presentation, we will describe a solution that integrates NEX and Docker to bridge the gap in code-to-data migration. The core of the solution is saemi-automatic conversion of science codes, tools and services that are already tracked and described in the NEX provenance system, to Docker - an open-source Linux container software. Docker is available on most computer platforms, easy to install and capable of seamlessly creating and/or executing any application packaged in the appropriate format. We believe this is an important step towards seamless process deployment in heterogeneous environments that will enhance community access to NASA data and tools in a scalable way, promote software reuse, and improve reproducibility of scientific results.
Publication Date: Dec 12, 2016
Document ID:
20170001275
(Acquired Feb 06, 2017)
Subject Category: COMPUTER SYSTEMS; EARTH RESOURCES AND REMOTE SENSING
Report/Patent Number: ARC-E-DAA-TN38171
Document Type: Conference Paper
Meeting Information: 2016 AGU Fall Meeting; 12-16 Dec. 2016; San Francisco, CA; United States
Meeting Sponsor: American Geophysical Union; Washington, DC, United States
Contract/Grant/Task Num: NNA07CA29C; NNX12AD05A
Financial Sponsor: NASA Ames Research Center; Moffett Field, CA United States
Organization Source: NASA Ames Research Center; Moffett Field, CA United States
Description: 1p; In English
Distribution Limits: Unclassified; Publicly available; Unlimited
Rights: Copyright; Distribution under U.S. Government purpose rights; under NASA contract NNA07CA29C and grant NNX12AD05A
NASA Terms: EARTH SCIENCES; DATA STORAGE; CLIMATE; SUPERCOMPUTERS; SOFTWARE REUSE; DEPLOYMENT; MAINTENANCE; MIGRATION
Other Descriptors: EARTH EXCHANGE; SUPERCOMPUTING; DATA
Availability Notes: Abstract Only
› Back to Top
Find Similar Records
NASA Logo, External Link
NASA Official: Gerald Steeman
Site Curator: STI Program
Last Modified: February 06, 2017
Contact Us