Real-time log joining on a continuous stream of events that are approximately ordered

摘要

Aspects of the disclosure relate generally to a distributed event joining system for joining primary and secondary events based on a common key. In one example, a continuous stream of primary events may be stored at a cache layer. Each primary event may be associated with a primary event key which may be used to generate an index map for the cache layer. After a time T, the stream of primary events may be sent to long term memory. The system may also receive secondary events, each associated with a primary key. The primary key may be used to identify a primary event in the cache layer. If the primary event is not available in the cache layer, the long term memory may be searched for the primary key. Paired events may then be submitted to a distributed joined event storage system.

主权项

1. A method comprising:
receiving a continuous stream of primary events corresponding to search queries submitted by one or more users, each received primary event being associated with a corresponding primary event key identifying a particular search query; storing the received primary events in a short term memory; generating, by a processor, an index map that tracks primary event keys of any received primary events currently stored in the short term memory; after a time T has passed, storing the received primary events in a long term memory; receiving a secondary event corresponding to an advertisement selected by one of the one or more users, the secondary event being associated with a particular primary key; and using the index map to determine whether the particular primary event key is included in the short term memory.