Real-time detection and classification of anomalous events in streaming data

摘要

A system is described for receiving a stream of events and scoring the events based on anomalousness and maliciousness (or other classification). The events can be displayed to a user in user-defined groupings in an animated fashion. The system can include a plurality of anomaly detectors that together implement an algorithm to identify low probability events and detect atypical traffic patterns. The atypical traffic patterns can then be classified as being of interest or not. In one particular example, in a network environment, the classification can be whether the network traffic is malicious or not.

主权项

1. A method of detecting and classifying anomalous events,
comprising: receiving an input log file including a plurality of events, wherein each event comprises a data set; for each event, providing multiple contexts that group the data set into different sub-groups, wherein one or more anomaly detectors are coupled to each context; generating an anomaly score for each context by using each context's one or more anomaly detectors, so that each event is associated with at least two anomaly scores generated for different contexts; for each event, combining at least the at least two anomaly scores generated for different contexts to generate an overall event score so as to classify the event as being normal or abnormal, wherein combining the anomaly score for each context further includes using domain knowledge in the combination, wherein the using domain knowledge includes modifying functions used in the combination based on whether the event is targeting a protected resource, whether the event violates a network rule, and/or whether a role of a network machine associated with the event is unexpected; and outputting a plurality of the overall event scores for the input log file, wherein the outputting includes:
displaying the events as dots that travel across a display in time steps as the events are being received as streaming data, anddisplaying at least one anomaly score and a maliciousness score associated with each event.