Data streaming infrastructure for remote execution in a constrained environment

摘要

A method, apparatus, and program product stream data between a host element and a target element in a parallel computing system. The method comprises partitioning data used to perform a task into a plurality of data streams based at least in part upon data associated with the task and partitioning a data stream into a plurality of data stream views based at least in part on a memory size associated with a target element configured to process the data stream. The method further comprises sequentially communicating at least a portion of the data stream views to the target element via a data streaming infrastructure implemented on at least a host element and the target element, wherein the portion of the data streaming infrastructure on the host element transfers the portion of the data stream views to the portion of the data streaming infrastructure on the target element.

主权项

1. A method of streaming data between a host element and a target element in a parallel computing system, the method comprising, in the host element:
partitioning data used to perform a task into a plurality of data streams based at least in part upon data associated with the task; partitioning a data stream from the plurality of data streams into a plurality of data stream views based at least in part on a memory size associated with the target element configured to process the data stream, wherein partitioning the data stream based at least in part on the memory size associated with the target element includes dynamically configuring sizes of the plurality of data stream views to fit within an amount of memory available in the target element for processing data stream views; and sequentially communicating at least a portion of the plurality of data stream views to the target element via a data streaming infrastructure implemented on at least the host element and the target element, wherein a first portion of the data streaming infrastructure configured on the host element transfers the portion of the plurality of data stream views to a second portion of the data streaming infrastructure configured on the target element.