DATA MANAGEMENT IN DISTRIBUTED FILE SYSTEMS

摘要

Technology is disclosed for managing data in a distributed processing system (“the technology”). In various embodiments, the technology pushes “cold” data from a primary storage of the distributed processing system to a backup storage thereby maximizing the usage of the space on the primary storage to store “hot” data on which most data processing activities are performed in the distributed processing system. The cold data is retrieved from the backup storage into the primary storage on demand, for example, upon receiving an access request from a client. While the primary storage stores the data in a format specific to the distributed processing system, the backup storage stores the data in a different format, for example, format corresponding to the type of backup storage.

主权项

1. A method, comprising:
storing, by a data node of a distributed data processing system, blocks of data on a first data storage system and a second data storage system, the first data storage system configured to store the blocks of data in a format different from a format of the blocks of data stored in the second storage system; identifying, at the data node and based on a data eviction policy, one or more of the blocks of data stored on the first storage system as cold data; determining, at the data node, whether the cold data is stored on the second storage system; and responsive to a determination that the cold data is stored on the second storage system, deleting the cold data from the first storage system.