SIMULATION ENVIRONMENT FOR DISTRIBUTED PROGRAMS

摘要

A dataflow of a distributed application is visualized in a locally simulated execution environment. A scheduler receives a job graph which includes a graph of computational vertices that are designed to be executed on multiple distributed computer systems. The scheduler queries a graph manager to determine which computational vertices of the job graph are ready for execution in a local execution environment. The scheduler queries a cluster manager to determine the organizational topology of the distributed computer systems to simulate the determined topology in the local execution environment. The scheduler queries a data manager to determine data storage locations for each of the computational vertices indicated as being ready for execution in the local execution environment. The scheduler also indicates an instance of each computational vertex to be spawned and executed in the local execution environment based on the organizational topology and indicated data storage locations.

主权项

1. A computing system for visualizing a dataflow of a distributed application in a locally simulated execution environment, the computing system comprising:
one or more processors; and one or more storage media having stored computer-executable instructions which, when executed by the one or more processors, cause the computing system to perform a method for visualizing a dataflow of a distributed application in a locally simulated execution environment, wherein the method includes:
an act of receiving a subscription to receive dataflow process data from a dataflow visualizer, the dataflow visualizer being configured to display various aspects of a simulated distributed application execution;an act of monitoring dataflow during execution of the distributed application, wherein the distributed application comprises a plurality of computational vertices that correspond to a job graph of the distributed application; andan act of sending the monitored dataflow data to the dataflow visualizer for display, wherein the dataflow data is displayed as the application is being executed.