Scalable data feed system

摘要

A scalable data feed system is disclosed. The data feed system may be offered as a cloud service that can serve many enterprises or tenants that require data to be pulled from information sources such as FTP, POP3, databases, line of business systems, a topic subscription, or an RSS feed, and pushed the data to information sinks, such as SMTP, email, FTP, mobile phones, and other devices and services. A pull agent pumps data from pull sources and pushes the data out to push agent counterparts. The push agent transforms and sends the data in messages to push sink, such as FTP, SMTP, or a mobile device. Both the pull agent and the push agent services are implemented as replicated services over a structured overlay network for high availability that can serve multiple requests to pump out data from multiple pull sources and push the data to multiple information sinks.

主权项

1. A system for pushing data to a plurality of data sinks, the system comprising:
a processor; and a memory coupled to the processor, the memory storing computer-readable instructions that, upon execution by the processor, cause the system to:
publish a protocol specifying a data format supported by an assigned data sink of the plurality of data sinks, wherein the push service is tasked with pushing incoming data to the assigned data sink;receive a message indicating incoming data is available for delivery to the assigned data sink;schedule delivery of the incoming data from a push job queue to the assigned data sink; andcreate a new runtime instance of the push service that transforms incoming data according to the protocol supported by the assigned data sink.