| 摘要 |
<p>A generic approach for automatically ingesting data into an HDFS (Hadoop File System) based data warehouse includes a datahub server, a generic pipelined data loading framework, and a meta-data model that, together, address data loading efficiency, data source heterogeneities, and data warehouse schema evolvement. The loading efficiency is achieved via the MapReduce scale-out solution. The meta-data model is comprised of configuration files and a catalog. The configuration file is setup per ingestion task. The catalog manages the data warehouse schema. When a scheduled data loading task is executed, the configuration files and the catalog collaboratively drive the datahub server to load the heterogeneous data to their destination schemas automatically.</p> |
