Connector interface for data pipeline

摘要

Methods and systems for a connector interface in a data pipeline are disclosed. A pipeline comprising two data source nodes and an activity node is configured. Each data source node represents data from a different data source, and the activity node represents a workflow activity that uses the data as input. Two connectors which implement the same connector interface are triggered. In response, data is acquired at each connector from the corresponding data source through the connector interface. The data is sent from the connectors to the activity node through the connector interface. The workflow activity is performed using the acquired data.

主权项

1. A system, comprising:
one or more computing devices configured to implement a plurality of computing resources, wherein the plurality of computing resources comprise a first data source and a second data source; and one or more computing devices configured to implement a configurable workflow service, wherein the configurable workflow service is operable to:
configure a pipeline comprising a first data source node, a second data source node, and an activity node, wherein the first data source node represents first data from the first data source, wherein the second data source node represents second data from the second data source, and wherein the activity node represents a workflow activity that uses the first data and the second data as input;trigger a first connector represented by the first data source node and a second connector represented by the second data source node, wherein the first connector and second connector implement a common connector interface comprising a plurality of methods, wherein at least some of the plurality of methods are implemented differently by the first connector than by the second connector;in response to the triggering the first connector, acquire the first data at the first connector from the first data source through the connector interface;in response to the triggering the second connector, acquire the second data at the second connector from the second data source through the connector interface;send the first data from the first connector to the workflow activity through the connector interface;send the second data from the second connector to the workflow activity through the connector interface; andperform the workflow activity using the first data and the second data.