Distributing processing of array block tasks

摘要

A technique includes distributing a plurality of tasks among a plurality of worker nodes to perform a processing operation on an array. Each task is associated with a set of a least one data block of the array, and an order of the tasks is defined by an array-based programming language. Distribution of the tasks includes, for at least one of the worker nodes, selectively reordering the order defined by the array-based programming language to regulate an amount of data transferred to the worker node.

主权项

1. A method comprising:
distributing a plurality of tasks among a plurality of worker nodes to perform a processing operation on an array, each task of the plurality of tasks being associated with a set of at least one data block of the array and an order of the tasks being defined by an array-based programming language, wherein distributing the plurality of tasks comprises: for at least one of the worker nodes, selectively reordering the order defined by the array-based programming language to regulate an amount of data transferred to the worker node; selecting a given worker node of the plurality of worker nodes to perform a given task of the plurality of tasks; communicating at least one block of data associated with the given task with the given worker node; and using the given worker node to determine a format of the communicated at least one block of data, wherein using the given worker node comprises selecting a given candidate format based at least in part on a time estimated by the worker node to perform a format conversion into the given candidate format and a cost of communicating using the candidate format.