SYSTEM FOR ORGANIZING AND FAST SEARCHING OF MASSIVE AMOUNTS OF DATA

摘要

A system to collect and store in a special data structure arranged for rapid searching massive amounts of data. Performance metric data is one example. The performance metric data is recorded in time-series measurements, converted into unicode, and arranged into a special data structure having one directory for every day which stores all the metric data collected that day. The performance metric data is collected by one or more probes running on machines about which data is being collected. The performance metric data is compressed prior to transmission to a server over any data path. The data structure at the server where analysis is done has a subdirectory for every resource type. Each subdirectory contains text files of performance metric data values measured for attributes in a group of attributes to which said text file is dedicated. Each attribute has its own section and the performance metric data values are recorded in time series as unicode hex numbers as a comma delimited list. Analysis of the performance metric data is done using regular expressions. For speed, cache memory is used. Performance metric data outside the start time and end time named by the user on a query screen is eliminated before the regular expression is applied.

主权项

1. A computer apparatus programmed to execute one or more searches on one or more levels of nesting, said searches defined by a user in a search query, and wherein each said search contains a filter condition where said filter condition may be stated as a regular expression, said filter condition to be applied to performance metric data or configuration attribute data or event data, and wherein said performance metric data is collected from one or more attributes of one or more instance of one or more resource types being monitored, and wherein said performance metric data comprises one or more time series of numerical values, each time series of performance metric data having been collected from one said attribute of one said instance of a resource type over a plurality of time slots, each of said numerical value of performance metric data having been converted to a Unicode character so as to create a time series of Unicode characters from each said time series of performance metric data, each said time series of Unicode characters stored in a non relational database file system coupled to said computer, and wherein said computer is programmed at least with an operating system and a search program implementing a query definition language having rules of syntax and syntax building blocks with which said user can compose said search query, each said syntax building block having one and only one definition, said search program further structured to control said computer to parse each said search query and control said computer to carry out one or more searches expressed in said search query by accessing from said non relational database file system Unicode characters representing performance metric data, or configuration attribute data or event data from said non relational database file system and applying sequentially to said accessed data each said filter condition expressed in each search expressed in said search query, each said search having its filter condition applied only to data from one or more instances of a resource type identified in a search of said search query and which were qualified by the next previous search if there is more than one search expressed in said search query, said syntax building blocks including a Unicode reverse mapping syntax building block which, when encountered in a search's filter condition, causes said computer to map Unicode characters representing performance metric data numerical values accessed from said non relational database file back to numerical values prior to applying said search's filter condition thereto.