System and method for handling out-of-order data supplied by a real-time feed

摘要

A system and associate method handle out-of-order data supplied by a real-time feed, and ingests the real-time feed fast enough to keep up with the feed rate while storing the data in a database in a time-ordered or other sequential manner without discarding any data. The present system adds a second unordered list for out-of-order data received from the feed or from a replay feed. A data element received from the feed, which has a time stamp earlier than the last data element placed in the ordered list, is placed in the unordered list. If replay data is received, the replay data elements are placed in the unordered list without verifying the time stamp. The data is then flushed from these memory lists to a database. Both the ordered list and the unordered list are inputted into the database. The database handles the ordering and merging of these two lists on insertion. The ordered data is still available to the user for real-time query in memory while all data elements are available for analysis in the database.