Institute of Electrical and Electronics Engineers, IEEE Transactions on Geoscience and Remote Sensing, 11(51), p. 5090-5097, 2013
DOI: 10.1109/tgrs.2013.2266929
Full text: Download
With the interdependencies that exist between data in a scientific processing pipeline, the ability to track the provenance of the scientific process through multiple stages is necessary to determining the usability of the resulting data product. In this paper we study the capture of provenance from an existing NASA instrument ingest pipeline. Since instrumenting the scientific code for a production system is not feasible, we show how provenance events can be scavenged from log files to generate detailed provenance graphs. Through extensions to the Karma provenance system, which have been implemented on a test instance of the AMSR-E production data pipeline, we determine that when the volume of provenance information is high, provenance graph visualizations provide a good tool for monitoring the ingest pipeline and identifying processing differences in ways not seen before. Two novel uses of provenance that we present in this paper are comparisons between processing runs and forward provenance for viewing downstream dependencies.