Big-fast data connector between in-memory database system and data warehouse system

摘要

Embodiments of the present invention include systems and methods for insuring better query consistency between at least two different databases, where one faster database has more up-to-date information than another slower database, and wherein updates are typically applied to the faster database first and then to the slower database. In embodiments, the systems and methods also insure that a query to the slower database is not performed until a set of one or more updates from the faster database have been applied to that slower database.

主权项

1. An in-memory database system that supports fast ingest data transfer between the in-memory database system and a data warehouse system, the in-memory database system comprising:
an in-memory database instance configured to receive data events in real-time or near real-time and update the in-memory database instance with the received data events; a listener configured to receive notifications of data events from the in-memory database instance to select a queue of a plurality of queues in which the data event will be stored, and to store the data event in the selected queue; and a plurality of queues, each queue comprising a proxy table and a corresponding write-behind listener, wherein each write-behind listener is configured to determine whether an update condition has been satisfied for a corresponding proxy table of the queue and, in response to determining that an update condition is satisfied, to store the data event in the data warehouse system, and wherein the in-memory database system is configured to receive a query requesting query results that satisfy the query from both historical data in the data warehouse system and data events stored in the in-memory database instance that have not yet been stored in data warehouse system, to generate a time stamp for the query, to wait for the data warehouse system to be updated with the data event stored in the in-memory database instance that occurred before a time represented by the time stamp, and to generate query results that satisfy the query after the data warehouse system is updated with the data event stored in the in-memory database instance that occurred before the time represented by the time stamp for the query.