METHOD AND SYSTEM FOR ITERATIVE PIPELINE

摘要

Method, system, and programs for data processing. In one example, a record is received. The record is separated into multiple partitions. Each partition is submitted to an iterative pipeline, wherein the iterative pipeline comprises two or more processing stages that are run in parallel and one of the multiple partitions passes through one of the two or more processing stages in one iteration. The multiple partitions are passed through two or more processing stages. The partitions that have passed through each of the two or more processing stages are merged.

主权项

1. A method implemented on at least one machine, each of which has at least one processor, storage, and a communication platform connected to a network for processing data, comprising:
receiving a record; separating the record into multiple partitions; submitting the multiple partitions in series to an iterative pipeline, wherein the iterative pipeline comprises two or more processing stages that run concurrently and each of the multiple partitions serially passes through one of the two or more processing stages in one iteration, wherein the two or more of the processing stages concurrently pass two or more of the multiple partitions.