Incremental Batch Method for Transforming Event-driven Metrics and Measures within a Map/Reduce Data Center

摘要

A method for a plurality of processors configured to perform steps in a map/reduce network operation adds incremental batch transformation of sequential measures recorded by time periods and uploaded asynchronously from their capture on mobile devices. The method creates and tracks measure states for each measure. The current recurrence of a measure is a transformation of selected past recurrences and measures. Measure state is propagated according to rules. The ID for a current measure is derived from the IDs of its cache measures and the IDs of its trigger measures. A batch incremental enrichment transforms one or more measures from one or more recurrences into at least one output measure that may be transformed again by the same or another batch incremental enrichment. The apparatus determines if the value needs to be overwritten with a newer value by the type of transformation, the recurrence id and the state.

主权项

1. An incremental batch process at a processor which does not completely reprocess the last 30 days of data in each daily batch run comprising:
retaining intermediate measures; fully processing data collected in the period since last incremental batch process; reprocessing intermediate measures when used to build potential new measure; reprocessing intermediate measures when used to build a measure that may be changed; replacing an obsolete measure; providing a unique ID for each generated measure; and determining when a generated measure cannot have changed.