SQL QUERY PROCESSING METHOD USING MAPREDUCE

摘要

The present invention relates to a method of processing big data in a data processing system consisting of: a web server (10) for providing data for processing big data; a master node (20) for dividing data of the web server (10) and delivering divided data into various distribution nodes such that a given task can be processed in parallel; a mapper (30) as a subnode for receiving map tasks allocated by the master node (20); and Reducer (40) as a subnode for receiving Reduce tasks. The method of processing big data comprises: an SQL query analyzing step (S10) of receiving the data from the web server (10) and determining, when an SQL query occurs, which data attributes are to be delivered to the mapper (30) and the Reducer (40); a data dividing step (S20) of extracting only the attributes which are acquired in the SQL query analyzing step (S10) from input data and dividing the extracted data for the mapper (30) and the Reducer (40); a map step (S30) of transmitting the divided data to the mapper (30) and outputting only an identification (ID) of a record which satisfies predefined SQL conditions; and a Reduce step (S40) of receiving record information for a record ID from a distributed file system (DFS) on the Reducer (40) by using the record ID outputted from the mapper (30), and then performing a computation process. According to the present invention, it is possible to process a large amount of data generated from a network service in a short period of time.