COMPUTERIZED SYSTEMS AND METHODS FOR DISTRIBUTED FILE COLLECTION AND PROCESSING

摘要

Systems and methods are disclosed for distributed file collection and processing. According to certain embodiments, a producer system may detect a producer configuration file that specifies a file type, identify at least one file of the specified file type, and send the at least one identified file to a collector system. A conductor system may receive a request from a consumer system and allocate the at least one identified file to the consumer system based on the request. The collector system may receive the at least one identified file from the producer system. The consumer system may send the request to the conductor system and pull the at least one identified file from the collector system.

主权项

1. A system for distributed file collection and processing, the system comprising:
a producer system that comprises a first memory that stores a first set of instructions and a first processor in communication with the first memory and configured to execute the first set of instructions to:
detect a producer configuration file that specifies a file type;identify at least one file of the specified file type; andsend the at least one identified file to the collector system; a conductor system that comprises a second memory that stores a second set of instructions and a second processor in communication with the second memory and configured to execute the second set of instructions to:
receive a request from the consumer system; andallocate the at least one identified file to the consumer system based on the request; a collector system that comprises a third memory that stores a third set of instructions and a third processor in communication with the third memory and configured to execute the third set of instructions to receive the at least one identified file from the producer system; and a consumer system that comprises a fourth memory that stores a fourth set of instructions and a fourth processor in communication with the fourth memory and configured to execute the fourth set of instructions to:
send the request to the conductor system; andpull the at least one identified file from the collector system.