Formula-based load evaluation in distributed streaming platform for real-time applications

摘要

Software for a distributed streaming platform receives an application that runs on a streaming platform. The application is structured as a directed acyclic graph (DAG) with instances of operators as nodes and streams as edges between nodes. The application is associated with a pre-defined hint that is a key-value pair. The software launches the application by assigning the instances of operators to containers provided by the streaming platform and initiating the streams. Then the software reads a value for the pre-defined hint and transmits the value to the application through an application programming interface (API) exposed by the streaming platform. The software receives a request from the application through the API to make a dynamic adjustment. And the software makes the dynamic adjustment and re-launches the application using a recovery policy.

主权项

1. A method, comprising the operations of:
receiving an application that runs on a streaming platform, wherein the application is structured as a directed acyclic graph (DAG) with instances of operators as nodes and streams as edges between nodes and wherein the application is associated with a pre-defined hint that is a key-value pair; launching the application by assigning the instances of operators to containers provided by streaming platform and initiating the streams; reading a value for the pre-defined hint and transmitting the value to the application through an application programming interface (API) exposed by the streaming platform; receiving a request, based at least in part on the value, from the application through the API to make a dynamic adjustment to one or more of the instances; and making the dynamic adjustment and re-launching the application using a recovery policy, wherein the dynamic adjustment includes a dynamic partition of at least one instance into a plurality of instances, wherein the at least one instance reads from and/or writes to a hash table or hash map associated with the instance, wherein the pre-defined hint is a count of entries in the hash table or hash map and, wherein each of the operations is executed by one or more processors in real time or near real time rather than offline.