Distributed processing system throttling

摘要

Methods, systems, and apparatus are described for throttling a distributed processing system. In one aspect, a method includes identifying records being processed by a distributed processing system that performs agent processes, each of the records including a corresponding timestamp; determining, based on timestamps of the records that have been processed by a first agent process, a first agent progress; identifying a dependent agent process performed by the distributed processing system, wherein the dependent agent process processes only records that have been processed by the first agent process; determining, based on timestamps of records that have been processed by the dependent agent process, a dependent agent progress; and throttling performance of the first process based on the first agent progress and the dependent agent progress.

主权项

1. A method implemented by data processing apparatus, the method comprising:
identifying records being processed by a distributed processing system that performs a plurality of agent processes, each of the records including a corresponding timestamp specifying a time for the record prior to a time that the record is processed by a first agent process in the distributed processing system; determining, based on one or more of the timestamps of one or more of the records that have been processed by the first agent process, a first agent progress for the first agent process; identifying a second agent process performed by the distributed processing system, wherein the second agent process processes only records that have been processed by the first agent process and is processing records that have been processed by the first agent process concurrently with the first agent process processing records to be provided to the second agent process; determining, based on one or more of the timestamps of one or more records that have been processed by the second agent process, a second agent progress for the second agent process; and throttling performance of the first agent process by the distributed processing system based on the first agent progress and the second agent progress.