File System Optimization by Log/Metadata Analysis

摘要

A mechanism is provided in a data processing system for optimization of a distributed file system by log data analysis. A name node in the distributed file system initiates a log analysis map/reduce job on one or more connected data nodes via a map/reduce processing framework in the distributed file system and receives result data resulting from the log analysis map/reduce job from the one or more connected data nodes via the map/reduce processing framework in the distributed file system. The name node performs analysis on the received result data and generates an optimization plan for the one or more connected data nodes based on results of the analysis. The name node initiates the optimization plan on the one or more connected data nodes via the map/reduce processing framework in the distributed file system.