Seamless monitoring of network traffic is critically important for enterprise network and security administrators: it lies at the very foundation of network threat detection and remediation. Regrettably, packet duplication has commonly been an undesirable side-effect of network traffic monitoring. Packet duplication produces redundant information in the monitoring traffic which can, besides overloading monitoring tools, result in packet drops, increased reporting of false positives, and seriously hamper the efficiency of SoC and NoC tools.

Packet recording has become a significant method to enable long-term traffic analysis and the stored information has been proven very valuable in case of security breach and calamity root cause analysis. Duplicate packets not only greatly increase the storage capacity, but the stored duplicate packets also often result in incorrect analysis results.

The common cause of duplicate packets in network monitoring traffic is through the use of the port mirroring feature of network switching devices, which is also known as a Switched Port Analyzer (SPAN). Using SPAN is a very common method of implementing network visibility. This feature, included in most enterprise-grade switches and routers, is a technique for copying packets traversing one or more switch ports sent to one or more network analysis tools.

The use of the SPAN port mirroring feature inevitably creates packet duplication. Specifically, port mirroring can be configured to only specify for packets into or out of a switch port. However, typically network administrators want a copy of both. The problem is when both the ingress and egress ports are mirrored; this results in duplicate packets being seen by network analysis tools. The timestamps may be different but the packet contents are the same. This challenge will be further compounded when SPAN is used for multiple connected devices.

In larger networks, tapping of traffic from multiple network segments, even the use of passive TAPs often results in packet duplication. Traffic traversing multiple network segments may be tapped by different TAPs and forwarded to the monitoring and inspection tools.

To avoid the adverse effects of packet duplication, the monitoring tools themselves may be forced to remove packet duplicates prior to implementing traffic analysis of the traffic. This scenario presents a number of challenges including bandwidth consumption on monitoring tools and consumption of precious processing resources on analysis tools impacting the CPU resources available for critical network analysis functions. The potential drop and processing resources available to the monitoring and analysis tool when it performs its own deduplication can be as high as 50%.