SYSTEMS AND METHODS FOR IN-LINE REMOVAL OF DUPLICATE NETWORK PACKETS

摘要

Systems and methods are disclosed for in-line removal of duplicate network packets in network packet streams operating at high speeds (e.g., 1-10 Gbps and above). A hash generator applies at least one hash algorithm to incoming packets to form one or more different hash values. The packet deduplication systems and methods then use the one or more hash values for each incoming packet to identify data stored for previously received backs and use the identified data to determine if incoming packets are duplicate packets. Duplicate packets are then removed from the output packet stream thereby reducing duplicate packets for downstream processing. A deduplication window can further be utilized to limit the amount of data stored for previous packets based upon one or more parameters, such as an amount of time that has passed and/or a number of packets for which data has been stored. These parameters can also be selected, configured and/or adjusted to achieve desired operational objectives.