| 主权项 |
1. A system comprising:
one or more processing devices; a central storage configured to:
store one or more input files from a client;receive an executable file from a client, the executable file compiled with one or more frameworks that specify a structured, serialized data format; andstore the executable file; one or more storage devices storing instructions that, when executed by the one or more processing devices cause the one or more processing devices to implement:
a work order frontend configured to:
receive, from the client, a message that contains (i) a reference to one or more input files stored in the central storage and (ii) a reference to the executable file stored in the central storage;transmit, to a global-level manager, a work order that includes (i) the reference to the one or more input files and (ii) the reference to the executable file;a global-level manager configured to:
receive, from the work order frontend, the work order;access work order parameters associated with the work order, the work order parameters defining a requirement of the processing of the work order and including at least a deadline for completion of the work order;determine a service level agreement to meet the work order parameters, the service level agreement including a price;receive an indication from the client that the service level agreement is accepted;partition the one or more input files into multiple shards of at least a first and second shard type, wherein the first shard type corresponds to shards of a larger size, and wherein the second shard type corresponds to shards of a smaller size, and wherein each first type shard is associated with at least one or more second type shards, partition the work order into multiple jobs, each job being associated with a specific first type shard, all second type shards associated with that specific first type shard, and the executable file;distribute the jobs among a plurality of clusters to be processed using underutilized computing resources in the clusters;monitor the processing of the jobs to insure that the deadline for completion of the work order will be met;provide a work order output to the client; andthe plurality of clusters, wherein each cluster includes one or more task level managers configured to process a job distributed to the cluster using underutilized computing resources in the clusters by loading the executable file into a native client environment and executing the executable file in the native client environment, and generate a job output based on the processing, wherein the job outputs are combined to form the work order output;wherein the native client environment is configured to limit the executable file's interaction with resources outside of the native client environment to receiving input through an input channel in the structured, serialized data format and sending output through an output channel and wherein the input received through the input channel only comprises one or more of the shards. |
