Network event capture and retention system

摘要

Methods and apparatus are provided to monitor and analyze activity occurring on a networked computer system. In some embodiments, a method is provided for capturing, in a data structure, at least a portion of a notification describing a network event provided by a node on a computer network, identifying a data element (e.g., an IP address of the node) within the notification, and updating an index and/or summary based on the data element. The data structure may be stored in a file system maintained on a site, and sites may exchange information related to the notification data stored on each. In some embodiments, a query which is issued to a site may be processed using data transferred from other sites, and/or may be split into one or more additional queries which may be transmitted for processing to other sites.

主权项

1. A method, comprising:
collecting and storing a plurality of transmission events as network event data elements in a plurality of data structures, each transmission event being reported by one or more nodes of a network and stored in compressed form in at least one storage site; extracting said plurality of transmission events stored as network event data elements in said plurality of data structures; based on a set of predefined network event characteristics and the extracted plurality of transmission events, creating indices which identify data structures and locations of network event data elements within those data structures, wherein each index identifies a data structure and a respective location of a network event data element within that data structure; receiving a query that requests particular transmission event information; based on the query, accessing the indices to identify a location of at least one network event data element by apportioning said query into multiple partial queries that request the particular transmission event information and sending the multiple partial queries to different storage sites which store said plurality of data structures; and in response to the multiple partial queries, receiving query results from the different storage sites and combining the query results to form an analyzable aggregation of transmission event information; wherein the query results contain particular transmission events in the compressed form to maximize the amount of data conveyed in each disk cycle.