History preserving data pipeline system and method

摘要

A history preserving data pipeline computer system and method. In one aspect, the history preserving data pipeline system provides immutable and versioned datasets. Because datasets are immutable and versioned, the system makes it possible to determine the data in a dataset at a point in time in the past, even if that data is no longer in the current version of the dataset.

主权项

1. A method for preserving history of a derived dataset, the method comprising:
at one or more computing devices comprising one or more processors and storage media storing one or more computer programs executed by the one or more processors to perform the method, perform operations of:
storing a first version of a derived dataset;wherein the first version of the derived dataset is derived from at least a first version of another dataset by executing a first version of derivation program associated with the derived dataset;storing a first build catalog entry, the first build catalog entry associated with the derived dataset and comprising an identifier of the first version of the other dataset and comprising an identifier of the first version of the derivation program;wherein the first build catalog entry comprises a name of the derived dataset and an identifier of the first version of the derived dataset;updating the other dataset to produce a second version of the other dataset;storing a second version of the derived dataset;wherein the second version of the derived dataset is derived from at least the second version of the other dataset by executing the first version of the derivation program associated with the derived dataset;storing a second build catalog entry, the second build catalog entry associated with the derived dataset and comprising an identifier of the second version of the other dataset and comprising an identifier of the first version of the derivation program; andwherein the second build catalog entry comprises the name of the derived dataset and an identifier of the second version of the derived dataset.