Fault tolerant batch processing

摘要

Among other aspects disclosed are a method and system for processing a batch of input data in a fault tolerant manner. The method includes reading a batch of input data including a plurality of records from one or more data sources and passing the batch through a dataflow graph. The dataflow graph includes two or more nodes representing components connected by links representing flows of data between the components. At least one but fewer than all of the components includes a checkpoint process for an action performed for each of multiple units of work associated with one or more of the records. The checkpoint process includes opening a checkpoint buffer stored in non-volatile memory at the start of processing for the batch.

主权项

1. A method performed by one or more computer systems that include memory for processing a batch of input data in a fault tolerant manner, the method including:
performing computations on the batch of input data, wherein at least one but fewer than all of the computations includes a checkpoint process for multiple units of work associated with the batch; wherein the checkpoint process includes:
for a unit of work from the batch,
if a result from performing an action for the unit of work was previously saved in a checkpoint buffer stored in memory, using the saved result to complete performing the computations on the unit of work without performing the action again; orif the result from performing the action for the unit of work is not saved in the checkpoint buffer, performing the action to complete performing the computations on the unit of work and saving the result from performing the action in the checkpoint buffer.