Systems and methods for using provenance information for data retention in stream-processing

摘要

A system and method for determining data usage based on provenance information, in a stream-processing system, includes progressively setting usage information for output stream data objects (SDOs), determining input SDOs that an output SDO depends on, based on a provenance dependency function; recursively feeding back the usage information for a subset of SDOs that can be discarded; and discarding the subset of SDOs. A system and method for data retention based on usage information, in a stream-processing system, includes managing retention of SDOs by deleting SDOs that are determined to be of null usage; and enhancing retention characteristics of SDOs that are deemed to have usage.

主权项

1. A method for determining data usage based on provenance information, in a stream-processing system, the method comprising:
progressively setting usage information, comprising a usage count corresponding to a number of downstream recipients, for output stream data objects (SDOs) comprising information that is transmitted in the stream-processing system; determining with a processor input SDOs that an output SDO depends on, based on a provenance dependency function; recursively feeding back the usage information for a subset of SDOs that can be discarded; and discarding the subset of SDOs.