Streaming method and system for processing network metadata

摘要

A method and system for processing network metadata is described. Network metadata may be processed by dynamically instantiated executable software modules which make policy-based decisions about the character of the network metadata and about presentation of the network metadata to consumers of the information carried by the network metadata. The network metadata may be type classified and each subclass within a type may be mapped to a definition by a unique fingerprint value. The fingerprint value may be used for matching the network metadata subclasses against relevant policies and transformation rules. For template-based network metadata such as NetFlow v9, an embodiment of the invention can constantly monitor network traffic for unknown templates, capture template definitions, and informs administrators about templates for which custom policies and conversion rules do not exist. Conversion modules can efficiently convert selected types and/or subclasses of network metadata into alternative metadata formats.

主权项

1. A method of processing network metadata generated on a network transmitting network traffic using one or more network protocols, the network including devices at least some of which receive network traffic through an ingress interface and transmit network traffic through an egress interface, the method comprising the steps of:
receiving network metadata from a plurality of sources in a data processing system, in at least one data format; determining the type or character of said network metadata; processing said network metadata to extract useful information therefrom; converting at least a portion of said network metadata into one or more different data formats that are used in said data processing system for other system metadata, in response, at least in part, to the results of said determining step; wherein said processing step is performed while said network metadata is in transition on said network between a network device that generated said network metadata and a device that is able to store said network metadata; and wherein said processing step is achieved by applying at least one policy governing network metadata processing, wherein said at least one policy is applied for the purpose of deduplicating flow information reported by flow exporters, and wherein said at least one policy includes the steps of:
receiving a flow record containing IP addresses of communicating endpoints and the IP address of a device which provided the flow record;noting the time when said record was received;maintaining in memory information about the IP addresses of communicating endpoints, IP addresses of flow reporting devices and the time of last observed communication; said information being an indicator of the frequency of at least some communication paths utilized by said flow reporting device;selecting based upon said memory information a flow reporting device with the highest frequency of use among other flow reporting devices which report said flow and designating said highest frequency flow reporting device as an authoritative source of information about communications between said network endpoints;in the event that more than one flow reporting device has same highest frequency of use reflected in said memory information, further disambiguating the identity of said authoritative source of information about communications between said network endpoints based upon criteria selected form the group consisting of, the time when flow reporting devices reported said flow, a Time-To-Live counter reported in the flow records submitted by the flow reporting devices, and the next hop IP address reported in the flow records submitted by the flow reporting devices;forwarding for further processing a flow record about communication between said communicating endpoints received from the authoritative flow reporting device; anddiscarding flow records about communication between said communicating endpoints received from other flow reporting devices.