How Duplication and Deduplication Amplifies Visibility

André Vink By: André Vink August 19, 2019

The vast majority of the time, packet duplication is seen as a serious problem for network visibility

When duplication occurs on the network traffic, it may indicate a network issue, however, often packet duplication is associated with the deployment of the network visibility layer itself, whereby network tools and appliances receive for processing the same packet from different locations or often the very same packet itself because of span port utilization.

Beyond providing insights for network troubleshooting, packet duplication will negatively impact your network visibility and IT performance as a whole, and without a plan for packet deduplication, you’ll start to experience four significant performance, monitoring, and security issues.

1. Diminished Capacity of Network Tools

Bandwidth capacity is always a concern for network visibility. It’s one of the most important reasons to upgrade SPAN port mirroring to network tap connectivity. 

But upgrading to taps and monitoring tools that can support the right network speeds isn’t a cure-all for network visibility. When monitoring tools and visibility devices are tasked with capturing duplicate packets, you can’t efficiently manage the bandwidth capacity of your network tool’s ports.

Because networking demands are increasing so rapidly, it’s almost impossible for IT to continuously upgrade security and monitoring devices to keep pace. Instead, you have to maximize efficiency to avoid dropping packets and experiencing performance delays.

Packet deduplication improves network visibility by ensuring that you aren’t wasting port bandwidth capacity on duplicate data.

2. Wasted Storage Space on Security and Monitoring Tools

In recent years, many network visibility concerns have revolved around implementing in-line security devices to defend complex data centers. This kind of real-time analysis is essential to business performance and data protection. But not all packets are analyzed in real time.

For out-of-band use cases like performance analysis, forensic packet analysis, anomaly analysis, and compliance audits, you need access to packet data after the fact, and that means having some level of storage capability on certain networking devices.

Storing networking data can cost you tens of thousands of dollars per terabyte, though. You can’t afford to waste the little space each device might have. But when you don’t have a plan for minimizing packet duplication, that’s exactly what you’re doing.

If networking devices are set to store packets for a certain amount of time, it’s possible that each batch is half filled with duplicate data. This is wasted space that could otherwise be used to store more packets for longer periods of time, helping you spot security issues, performance delays, and other problems that impact your business. 

Adding packet deduplication capabilities to your network visibility layer goes beyond filtering on a device level, opening up storage capacity to maximize the effectiveness of your tools.

3. Dropped Packets for Network Analyzers

Creating a network visibility layer with taps, network packet brokers, and bypass switches can go a long way in eliminating data bottlenecks across your data center. But when you’re asking network analyzers built for 1G or 10G networks to capture data on a 40G network, you need to do more to address traffic bottlenecks.

Packet deduplication is an excellent way to reduce the load placed on network analyzers that are already tasked with monitoring high-speed networks. Instead of overrunning network analyzers with duplicate traffic, you can ensure analysis is reserved for instances that will bring value to your network, business, and operations.

When you don’t have packet deduplication, you risk having packets dropped without anyone in IT realizing it. As a result, you might miss out on key security, performance, or troubleshooting insights.

4. Increased Mean Time to Resolution

Your IT team has no shortage of data at its fingertips. And while that may seem great for troubleshooting scenarios, the reality is that finding root causes in all your data can be a significant challenge. 

Think about duplicate packets as an abundance of false data in your network. Without packet deduplication capabilities, troubleshooting requires you to weed out all that false data just to uncover the true packets that require analysis. Then, you still have to find root causes within the original packets.

Packet deduplication minimizes the volume of data you collect, giving your IT team an easier path to discover root causes and reduce mean time to resolution for all reported issues.

Bringing Packet Deduplication to Your Network Visibility Layer

Packet deduplication is more than a simple extra capability to have in your network. It’s absolutely essential for ensuring network visibility, maintaining strong data protection, and maximizing ROI on security and monitoring tools. 

While some network tools can perform packet de-duplication that typically results in a major drop in the tools' performance by shifting deduplication to the visibility layer, you are increasing the efficiency of the network tool, as it can now 'focus' resources on its intended purpose which essentially is not deduplication...

That’s why you should expect advanced packet deduplication capabilities from your network packet brokers. Hardware-based deduplication together with other Network Intelligence capabilities enhance your visibility layer and enables your tools succeed in the face of increasing demands. 

But if it were so easy to build packet deduplication into your network, no one would have the problems we’ve listed here. That’s where we can help.

Contact us today and talk to a Niagara Network visibility expert who can help you unlock all the benefits and capabilities of advanced packet deduplication.New call-to-action