Scalable file management for a shared file system

摘要

Managing a shared file system comprising a directory and files stored on a multiple storage devices shared by plural processing nodes, is provided. A plurality of parallel directory traversal records are partitioned into a plurality of ranges to allow for the records in each range to be written independently in parallel by plural processing nodes during parallel directory traversal. Parallel operations are performed comprising parallel directory traversal of all directory paths and files in the shared file system.

主权项

1. A method for managing a shared file system, comprising:
performing traversal by multiple processing nodes in parallel of all directory paths and files in shared directories in the shared file system; partitioning identified directory traversal records into ranges for the records in each range to be written independently in parallel by the multiple processing nodes; scanning file inodes in parallel, each inode comprising file attributes of files stored in said shared directories and is assigned an inode number, and each range is assigned a range number; mapping each inode number to a range number using a mapping function; performing parallel execution threads in one or more processing node utilizing local mutual exclusions to resolve contention for access to local process resources, wherein during the scan of the file inodes, each processing node accesses a row of N sub-ranges that logically comprises a complete range, wherein N is a positive number; and re-balancing a processing workload across the multiple processing nodes based on a monitored workload queue length at each processing node by a master node.