Method for Preparing Numerous Data for Efficient Manipulation using Interning

摘要

A data-processing technique for increasing data-size capacity and improving query speed on large datasets where fields within records are replaced by integers representing distinct values of those fields, the integers drawn from a densely-populated range so that record selection, aggregation and other actions can be performed efficiently using bit sets and other data structures.

主权项

1. A method of preparing a plurality of multi-field data records for improved query performance, comprising:
for each record of the plurality of multi-field data records: reading the record; interning a value of a data field of the record to obtain an integer representing the value of the data field; and creating a memory structure to represent the record, said memory structure containing the representative integer and excluding the value of the data field.