Data collecting method, data collecting apparatus and network management device

摘要

The present invention provides a data collection method and apparatus and a network management device. The method includes: a network management device collecting data files to be processed reported by a network element device; assigning the data files to be processed as a plurality of tasks; adding the assigned tasks into a task queue and extracting tasks from the task queue one by one for processing. According to the present invention, the task work load can be automatically adjusted according to the computer configuration and parameter configuration, and the maximum efficiency of data processing can be achieved under different scenarios.

主权项

1. A data collection method, comprising:
a network management device collecting data files to be processed reported by a network element device; assigning the data files to be processed as a plurality of tasks; and adding the plurality of tasks into a task queue and extracting tasks from the task queue one by one for processing; wherein the step of assigning the data files to be processed as a plurality of tasks comprises: evaluating a total quantity or capacity of the data files to be processed; evaluating local calculation ability; obtaining local configuration parameters; calculating task payload of each thread according to the quantity or capacity of the data files to be processed, the local calculation ability and the configuration parameter; and assigning the data files to be processed as the plurality of tasks based on the task payload; wherein the local calculation ability comprises: local CPU processing speed and memory capacity, and the configuration parameter comprises: a number of threads and maximum activation time of the threads; wherein the task payload of each thread is equal to the total data quantity or capacity divided by a number of processing batch and then divided by the number of threads; wherein the number of processing batch=Max (the total data CPU processing time/(the number of threads*the maximum activation time), the total capacity of the data/memory capacity), wherein, Max represents to take a maximum; wherein the total data CPU processing time is equal to the total quantity or total capacity of the data files to be processed divided by the CPU processing speed.