GENERIC SCHEDULING

摘要

A system and method for customized scheduling of sources, including breaking down a source of content into at least two categories, including posts and engagements, and gathering content related to a specific source. A scheduler handles scheduling of posts and engagement for a single source and entities that are due to be crawled are sent to a scheduling queue, in which each content type for a source can have its own queue. A process points to the correct scheduler queue in order to request content to be crawled, attaches to the proper queue, processes requests, queries the social network for content, parses the response and sends any new data to be saved to the system.

主权项

1. A method for managing source content, comprising:
identifying content of a source as content of a first category representing top level content or as content of a second category representing content driven from top level content, for storage to a system database; adding a source that is due to be crawled to a scheduling queue, wherein each content category for the source has its own queue within the scheduling queue, wherein the source is added to the scheduling queue based on a record of crawl time values for each of the content categories of the source; and attaching a worker process to a queue and processing a crawling request for a source, querying the source for content and parsing a response from the source, and saving content of the response to the system database.