Multi-threaded in-memory processing of a transaction log for concurrent access to data during log replay

摘要

A dataset is recovered after a server reboot while clients access the dataset. In response to the reboot, not-yet-completed transactions in a log are parsed to create, for each of the dataset blocks modified by these active transactions, a respective block replay list of the active transactions that modify the block. Once the block replay lists have been created, clients may access specified blocks of the dataset after on-demand recovery of the specified blocks. The on-demand recovery is concurrent with a background recovery task that replays the replay lists. To accelerate log space recovery, the parsing of the log inserts each replay list into a first-in first-out queue serviced by multiple replay threads. The queue can also be used as the cache writeback queue, so that the cache index is used for lookup of the replay list and the recovery state of a given block.

主权项

1. A method of recovery of a dataset in response to reboot of a data processor of a data storage system, the data storage system having data storage storing the dataset and a log of records of transactions upon the dataset, said method comprising the data processor executing computer instructions stored on a non-transitory computer readable storage medium to perform the steps of:
(a) parsing records in the log of transactions not-yet-completed by the time of the reboot in order to produce a data structure identifying dependencies between the not-yet-completed transactions; and then (b) performing a background task of replay of the not-yet-completed transactions in order to recover the dataset, and concurrent with the background task of replay of the not-yet-completed transactions in order to recover the dataset, responding to a request from a client for access to a specified block of data in the dataset by performing on-demand recovery of the specified block and then performing client access to the recovered specified block, and the on-demand recovery of the specified block accessing the data structure identifying dependencies between the not-yet-completed transactions in order to recover the specified block.