Method and System for a Scheduled Map Executor

摘要

Method, system, and programs for data processing. In one example, a record is received. The record is separated into multiple partitions. Each partition is submitted to an iterative pipeline, wherein the iterative pipeline comprises two or more processing stages that are run in parallel and one of the multiple partitions passes through one of the two or more processing stages in one iteration. The multiple partitions are passed through two or more processing stages. The partitions that have passed through each of the two or more processing stages are merged.

主权项

1. A method, implemented on at least one machine each of which has at least one processor, storage, and a communication platform connected to a network for processing data, the method comprising the steps of:
generating a queue of one or more records for processing; setting a rate at which the one or more records are requested for processing; retrieving one or more records listed on the queue; submitting the retrieved records to a proxy record reader; removing the records submitted to the proxy reader from the queue; and submitting the retrieved records to an actual record reader.