Rule-based extraction, transformation, and loading of data between disparate data sources

摘要

Methods and apparatus are presented for extracting, transforming, and loading data from one database to another database. For example, an extraction, transformation, and loading (ETL) component may access an operational log of a given database in order to detect an update to the database. Upon detecting the update, the ETL component may extract a subset of data from the operational log, where the extraction of the subset of data is based on one or more rules. Once the subset of data has been extracted, the ETL component may transform the extracted subset of data from the operational log into a format for another, target database, where the data format for the other, target database is different from a data format for the given, source database. The ETL component may then load the subset of data transformed into the data format for the other, target database into the target database.

主权项

1. A method, comprising:
performing, by one or more computing devices:
accessing an operational log to detect an update to a transactional database, wherein the transactional database updates the operational log when transactional data is received, wherein the update indicates a delta from a previous version of the operational log, and wherein the transactional data is generated when Internet marketing is performed and comprising an attribute for a keyword, an ad, a group that includes certain keywords or ads, or a marketing campaign;in response to detecting an update, extracting a subset of data from the operational log without accessing the transactional database;in response to detecting the update, transforming the extracted subset of data into transformed data by aggregating the subset of data having a same attribute value for the attribute and occurring within a same time range; andstoring the transformed data in a corresponding time range specific field of a data object, the data object having one or more time range specific fields.