4 minutes reading time (822 words)

Resolving the Challenge of Duplicate Packets

Seamless monitoring of network traffic is critically important for enterprise network and security administrators: it lies at the very foundation of network threat detection and remediation. Regrettably, packet duplication has commonly been an undesirable side-effect of network traffic monitoring. Packet duplication produces redundant information in the monitoring traffic which can, besides overloading monitoring tools, result in packet drops, increased reporting of false positives, and seriously hamper the efficiency of SoC and NoC tools.

Packet recording has become a significant method to enable long-term traffic analysis and the stored information has been proven very valuable in case of security breach and calamity root cause analysis. Duplicate packets not only greatly increase the storage capacity, but the stored duplicate packets also often result in incorrect analysis results.

The common cause of duplicate packets in network monitoring traffic is through the use of the port mirroring feature of network switching devices, which is also known as a Switched Port Analyzer (SPAN). Using SPAN is a very common method of implementing network visibility. This feature, included in most enterprise-grade switches and routers, is a technique for copying packets traversing one or more switch ports sent to one or more network analysis tools.

The use of the SPAN port mirroring feature inevitably creates packet duplication. Specifically, port mirroring can be configured to only specify for packets into or out of a switch port. However, typically network administrators want a copy of both. The problem is when both the ingress and egress ports are mirrored; this results in duplicate packets being seen by network analysis tools. The timestamps may be different but the packet contents are the same. This challenge will be further compounded when SPAN is used for multiple connected devices.

In larger networks, tapping of traffic from multiple network segments, even the use of passive TAPs often results in packet duplication. Traffic traversing multiple network segments may be tapped by different TAPs and forwarded to the monitoring and inspection tools.

To avoid the adverse effects of packet duplication, the monitoring tools themselves may be forced to remove packet duplicates prior to implementing traffic analysis of the traffic. This scenario presents a number of challenges including bandwidth consumption on monitoring tools and consumption of precious processing resources on analysis tools impacting the CPU resources available for critical network analysis functions. The potential drop and processing resources available to the monitoring and analysis tool when it performs its own deduplication can be as high as 50%.

Packet Deduplication Solution

Packet deduplication refers to the capability for removing packet duplicates prior to network data being forwarded or transmitted to network analysis tools for the purpose of monitoring, analyzing, and recording. This typically causes a substantial reduction in the volume of traffic handled by such tools enabling an increase in their operational efficiency, a reduction in false-positive errors generated, and an elimination of security breaches that could exist in implementations without deduplication. Without duplicate packets being identified and removed first, analysis tools may generate erroneous alarms and/or produce compromised data and results.

In some cases, advanced network switches implement a basic level of software-based Layer 2 (L2) and Layer 3 (L3) deduplication as an optional feature on a per-port basis prior to forwarding traffic to an inline security tool.L2 deduplication removes identical Ethernet frames where the Ethernet header and the entire IP packet match, while L3 deduplication removes TCP or UDP packets where only the IP packet match. In such a case, the network switch checks for duplicates and removes only the immediately-previous packet if the duplicate arrives within a fixed type interval (typically in the order of a millisecond) of the original packet.

Deduplication and the Value of Network Packet Brokers

As enterprise network architectures continue to expand, network bandwidth levels continue to dramatically increase, newer tools for security, performance management, and monitoring keep getting deployed, a comprehensive network visibility layer is required.

That is exactly why in recent years Network Packet Brokers (NPBs) have come into play. Advanced network packet brokers allow a high level of deep packet inspection and processing including aggregation, filtering, and load balancing of traffic across the range of security and monitoring tools at data rates of up to 100 Gb/s.

With fine-grained, hardware-based deduplication typically built into advanced NPBs, the challenge of duplicate packets degrading the performance of security and monitoring tools is tackled in the best manner. With the advanced NPBs architecture, packets are sent to an internal packet processor for fine-grained, flexible flow-based deduplication and optimized for delivery to everything from IDS and IPS to forensics, network analyzers, data storage, and more.

Flow-based deduplication permits the elimination of duplicate packets using the range of attributes shared by IP packets in a flow including source IP, destination IP, protocol, and source and destination port. It may also enable the selection of inbound and outbound interfaces, CoS/QoS markings, TCP flags, and others.

Original article by Niagara Networks

Shaw’s quarterly mobile service revenues climb 11%...
Canada launches 3800MHz policy and licence framewo...


No comments made yet. Be the first to submit a comment
Wednesday, 06 July 2022

Captcha Image

By accepting you will be accessing a service provided by a third-party external to https://telnetnetworks.ca/

Contact Us


Telnet Networks Inc.
1324 Andersen Drive
Kingston, ON  K7P 0C6


(800) 561-4019



For More Information about Telnet Networks, our products, or our services, or to request a quote please feel free to contact us directly.