Fault tolerant batch processing

摘要

Processing a batch of input data includes reading the batch including multiple records and passing the batch through a dataflow graph At least one but fewer than all of the graph components includes a checkpoint process for an action performed for each of multiple units of work associated with one or more of the records The checkpoint process includes opening a checkpoint buffer at the start of processing If a result from performing the action for a unit of work was previously saved in the checkpoint buffer, the saved result is used to complete processing of the unit of work without performing the action again If a result from performing the action for the unit of work is not saved in the checkpoint buffer, the action is performed to complete processing of the unit of work and the result from performing the action is saved in the checkpoint buffer.