Dissemin is shutting down on January 1st, 2025

Published in

Institute of Electrical and Electronics Engineers, IEEE Transactions on Geoscience and Remote Sensing, 11(51), p. 5090-5097, 2013

DOI: 10.1109/tgrs.2013.2266929

Links

Tools

Export citation

Search in Google Scholar

Provenance Capture and Use in a Satellite Data Processing Pipeline

Journal article published in 2013 by Scott Jensen, Beth Plale ORCID, Mehmet S. Aktas, Yuan Luo, Peng Chen, Helen Conover
This paper is available in a repository.
This paper is available in a repository.

Full text: Download

Green circle
Preprint: archiving allowed
Green circle
Postprint: archiving allowed
Red circle
Published version: archiving forbidden
Data provided by SHERPA/RoMEO

Abstract

With the interdependencies that exist between data in a scientific processing pipeline, the ability to track the provenance of the scientific process through multiple stages is necessary to determining the usability of the resulting data product. In this paper we study the capture of provenance from an existing NASA instrument ingest pipeline. Since instrumenting the scientific code for a production system is not feasible, we show how provenance events can be scavenged from log files to generate detailed provenance graphs. Through extensions to the Karma provenance system, which have been implemented on a test instance of the AMSR-E production data pipeline, we determine that when the volume of provenance information is high, provenance graph visualizations provide a good tool for monitoring the ingest pipeline and identifying processing differences in ways not seen before. Two novel uses of provenance that we present in this paper are comparisons between processing runs and forward provenance for viewing downstream dependencies.