Methods and apparatus for archiving system having enhanced processing efficiency

摘要

Methods and apparatus for archiving files in a system with primary storage and secondary storage that reduces thread contention. A single input reader thread generates list of files for processing by multiple map threads that evaluate each file in a list against a number of archive policies, such as move, purge, index and delete. Reduce threads take the lists from the map threads and generates list of files for action under a given policy, e.g., move, purge, index, delete. This arrangement eliminates thread contention for files and lists.

主权项

1. A method comprising,
scanning files in a primary storage system based upon a plurality of file archiving policies to determine if any of the files should be archived to a second storage system by: providing a single input reader thread to scan folders for files and generate a plurality of lists of files; providing a plurality of map threads including a first one of the plurality of map threads to process a first one of the plurality of lists of files and a second one of the plurality of map threads to process a second one of the plurality of lists of files, the first one of the plurality of map threads processing each file in the first one of the plurality of lists of files to determine if the files meet criteria for each of a plurality of policies for archiving files, the second one of the plurality of map threads processing each file in the second one of the plurality of lists of files to determine if the fries meet the criteria for each of a plurality of policies for archiving files; and providing a plurality of reduce threads to process output from the plurality of map threads to group files by a key value and output a first list for a first one of the plurality of policies for archiving files and a second list for a second one of the plurality of policies, wherein there is no contention between map threads and no contention between threads accessing the first and second lists for the plurality of policies for archiving files, and wherein the plurality of policies for archiving files includes a move policy, a purge policy, an index policy, and a delete policy.