System for partitioning batch processes

摘要

A system for processing a batch job comprises a processor and a memory. The processor is configured to receive a job name for a job submitted to execute, to receive one or more job parameters, and to determine one or more nodes to run the job. The processor is configured to determine one or steps, where for each step: a step is executed on a node using a state of data associated with a start state of the step; and upon completion of executing the step, a result is stored to a durable storage. The durable storage stores the state of data associated with the start state of the step and the completion state of the step and are accessible by other execution processes as associated with either the start state of the step or the completion state of the step. The memory of the system is coupled to the processor and configured to provide processor with instructions.

主权项

1. A system for processing a batch job, comprising:
a processor configured to:
receive a job name for a job submitted to execute;receive one or more job parameters, wherein the job comprises a flow of operations performed on a set of data objects stored in a database;determine a job entry moment for the job, wherein the job entry moment comprises a change identifier that represents a view of the set of data objects stored in the database at a point in time consistent with a start of the job, wherein the job is isolated from ongoing update activity in the database at the job entry moment;determine at least two nodes to run the job;determine one or more steps;for each step:
execute a step of the one or more steps on the at least two nodes in parallel using a state of the set of data objects associated with a start state of the step;upon completion of executing the step, store a result to a durable storage, wherein the durable storage stores a state of data associated with a completion state of the step, and wherein the state of the set of data objects associated with the start state of the step and the completion state of the step are accessible by other execution processes as associated with either the start state of the step or the completion state of the step; a memory coupled with the processor, wherein the memory is configured to provide the processor with instructions.