Storing network bidirectional flow data and metadata with efficient processing technique

摘要

A processing technique provides an improved indexing arrangement that enables storage, filtering and querying of metadata used to retrieve packets captured from a network and persistently stored in a data repository. A packet capture engine records the packets in packet capture (PCAP) formats from a network link at a substantially high packet transfer rate to persistent storage of the data repository in a sustained manner. Efficient filtering and querying of the metadata to retrieve the stored packets may be achieved, in part, by organizing the metadata as one or more metadata repositories. The processing technique uses the Berkeley Packet Filter (BPF) language as an interface of a BPF engine to search or index the stored packets in response to queries. The BPF engine processes BPF expressions used as precursors to the indexing arrangement to enable access to the repositories when searching and locating stored packets matching the expressions.

主权项

1. A method comprising:
intercepting a first packet from a network link at a node coupled to the network link, the node including persistent storage devices organized as a plurality of volumes having a hierarchical file system; computing a first hash value based on a network flow of the first packet; recording the first packet in a packet capture (PCAP) format as a first PCAP record including the first hash value and first flow metadata appended to the first PCAP record; copying the first PCAP record to a first metadata repository stored as a first file on a first volume of the hierarchical file system of the node, wherein the first metadata repository stores a plurality of second PCAP records having second hash values and second flow metadata; and concurrently searching and retrieving one or more of the second PCAP records of the first metadata repository while copying the first packet to a data repository stored as a second file of a second volume of the hierarchical file system of the node to realize a substantially high sustained packet transfer rate of the network link.