Background format optimization for enhanced sql-like queries in hadoop

摘要

A format conversion engine for Apache Hadoop that converts data from its original format to a database-like format at certain time points for use by a low latency (LL) query engine. The format conversion engine comprises a daemon that is installed on each data node in a Hadoop cluster. The daemon comprises a scheduler and a converter. The scheduler determines when to perform the format conversion and notifies the converter when the time comes. The converter converts data on the data node from its original format to a database-like format for use by the low latency (LL) query engine. Start Receives a query plan Reviews schema information, including file formats N < Converted format Y 604 I' 606 Defines query fragments for Defines query fragments for the original format based on the converted, destination the query plan format based on the query plan Retrieves data of the appropriate file format according to the query fragments Transforms the data into in-memory tuples Performs a query on the in-memory tuples End