RESTORING NON-TRANSACTIONAL MESSAGES IN QUEUES FOR ADVERTISEMENT DATA FLOW PROCESSING

摘要

The disclosed online system includes a number of processing blocks, including a feeding service that sends a data stream and a consuming service that receives a data stream, without sending back acknowledgements for the data (i.e., “nontransactional”). The system handles failure in the feeding service and/or the consuming service by adding reference points into the data stream and backing up a current sum that is maintained by the consuming service. Upon a failure of the consuming service, the system obtains the last reference point, restores the backup copy of the database, and starts reading the backup copy from that check point. To address a failure of the feeding service, the feeding service creates an identifier for each message in the data stream, and upon failure, the consuming service checks for messages that have the same identifier within a specified period of time, and discards any duplicate messages.

主权项

1. A method comprising:
receiving, by a first processing service, a data stream from an ad server, the data stream comprising a first set of messages and at least one reference point, each message from the first set of messages formatted as an ordered set including an ad identifier and cost information about a served ad; for each message in the first set of messages, identifying a cost limit identifier associated with the ad identifier; generating, by the first processing service, a second set of messages, each message from the second set of messages formatted as an ordered set including the cost limit identifier and the cost information about the served ad; storing, by a second processing service, the second set of messages in a database at specified interval; responsive to detecting the at least one reference point, storing, by the second processing service, a copy of the database in a file system; responsive to a failure of the second processing service, retrieving by the second processing service the at least one reference point; retrieving the copy of the database from the file system; and reading contents of the copy the database from a location corresponding to the at least one reference point, the copy of the data base retrieved from the file system.