Managing Data Items Across Multiple Data Services

摘要

Methods and systems for managing data items across multiple data services are disclosed. The method includes establishing a first connector with a source data service and a second connector with a destination data service. The method also includes receiving a job implicating data items that are to be transferred to the destination data service. For each item, the method includes determining whether a set of operations needs to be performed on the data item to comply with requirements of the destination data service. When a set of operations needs to be performed, the method includes instantiating an operation pipeline including one or more operators based on the set of operations. The method further includes receiving the data item via the first connector, transforming the data item to a transformed data item using the pipeline, and transmitting the transformed data item to the destination data service via the second connector.

主权项

1. A method comprising:
establishing, at a processing device, a first connector with a source data service and a second connector with a destination data service; receiving, at the processing device, a job, the job implicating one or more data items associated with the source data service that are to be transferred to the destination data service; for each data item:
determining, at the processing device, whether a set of operations need to be performed on the data item to comply with requirements of the destination data service;when a set of operations need to be performed on the data item:
instantiating, at the processing device, an operation pipeline based on the set of operations, the operation pipeline including one or more operators, each operator corresponding to a respective operation in the set of operations;receiving, at the processing device, the data item via the first connector;transforming, at the processing device, the data item to a transformed data item using the pipeline; andtransmitting, at the processing device, the transformed data item to the destination data service via the second connector.