METHOD OF AND SYSTEM FOR COLLECTING NETWORK DATA

摘要

The invention discloses a method of collecting network data. This method is applicable to collection of data of network documents, published on a website, related respectively to M subjects, wherein M is a positive integer, the method including: configuring webpage link addresses, of network data to be collected, into queues of corresponding types according to types corresponding to the webpage link addresses of the network data to be collected, wherein the webpage link addresses of the network data to be collected are link addresses of webpages where the data of the network documents related respectively to the M subjects are located; obtaining webpage source codes corresponding to the webpage link addresses, of the network data to be collected, in the queues of the corresponding types; and extracting the data of the network documents corresponding to URLs corresponding to the webpage source codes according to the URL information and collection depth values of the URLs.

主权项

1. A method of collecting network data, applicable to collection of data of network documents, published on a website, related respectively to M subjects, wherein M is a positive integer, the method comprising:
configuring webpage link addresses, of network data to be collected, into queues of corresponding types according to types corresponding to the webpage link addresses of the network data to be collected, wherein the webpage link addresses of the network data to be collected are link addresses of webpages where the data of the network documents related respectively to the M subjects are located; obtaining webpage source codes corresponding to the webpage link addresses, of the network data to be collected, in the queues of the corresponding types; and extracting the data of the network documents corresponding to Uniform Resource Locators (URLs) corresponding to the webpage source codes according to URL information and collection depth values of the URLs.