Systems and methods for filtering low utility value messages from system logs

摘要

Systems and methods disclosed herein provide intelligent filtering of system log messages having low utility value. In providing the filtering, the systems and methods determine the utility value of a system log message and delete the message from the system log if the message is determined to be of low utility value. As such, embodiments herein provide an system log filter, which reduces the amount of data stored in the system log based on the utility value of the message.

主权项

1. A computer-implemented method comprising:
determining, by a log filter, a utility value of each message of a plurality of messages of a system log, the utility value of a respective message based on predictability of information in the respective message, the predictability of information based on a set of one or more messages preceding the respective message in the systemlog; purging each respective message of the plurality of messages which are determined to have a utility values that is lower than a threshold utility value; wherein before purging messages determined to have a low utility value: identifying duplicative messages within one or more respective subsequence of the plurality of messages; deleting, within each respective subsequence of the plurality of messages, all but one of the duplicative messages; after deleting all but one of the duplicative messages within each respective subsequence of the plurality of messages: computing a utility score for each remaining message of the system log, comparing the utility score of each remaining message to the threshold utility value, wherein each message having a utility score lower than the threshold utility value is determined to have a low utility value, identifying subsequences of the remaining messages, wherein the subsequences of the remaining messages comprise a threshold number of messages of low utility value, and determining, for each of said subsequences of the remaining messages, that no duplicative messages exist within the respective subsequences of the remaining messages; and storing, in a memory, each respective message of the plurality of messages which are determined to have a utility value that is equal to or higher than the threshold utility value.