Intelligent data mining and processing of machine generated logs

摘要

According to some embodiments, a method and an apparatus of analyzing log files comprises sampling a log and determining a structure associated with the log file based on the sampling and a pattern within the structure. If the structure and the pattern are stored in a repository, data from the log file will be exported into a database based on the determined pattern.

主权项

1. A method of analyzing log files, the method comprising:
sampling a log file comprising a plurality of structures; determining, via a processor, one of the plurality of structures associated with the log file based on the sampling and a pattern within the one of the plurality of structures; determining a type of delimiter associated with the log file; determining if the one of the plurality of structures and the pattern are stored in a repository; parsing the log fields based on the type of delimiter; discovering log field content types based on the log file's data, patterns, distinct values and regular expressions; assigning log field content types to the parsed log fields; determining that a variety of field names are possible based on content from previously stored log file patterns within the repository; presenting field name options to a user to select a field name based on the determined variety of field names; standardizing the parsed log fields based on a selected field name from the previously stored log file patterns within the repository; and exporting data from the log file into a database.