The IPFIX (IP Flow Information Export) protocol was standardized by the Internet Engineering Task Force (IETF) in 2013 to be a multi-vendor universal metadata protocol for exporting IP flow information from network devices, such as switches, routers, firewalls to network monitoring and analysis applications or “collector” systems. The IETF IPFIX standard defines how flow information is exported, formatted and transferred from the IPFIX “agent” devices to collector systems for further segmentation, analysis and logging.
Based on Netflow Version 9, IPFIX utilizes similar procedures for exporting a “flow” to a collector, which operates in a many-to-many relationship with exporter network devices so that an exporter can transmit flow information to multiple collectors, each of which can collect information from any number of exporter devices.
Flow data utilized by IPFIX consists of all IP data traffic that belongs to the same “connection” or “conversation” between two devices on a particular protocol. Flow information is periodically transmitted to the collector devices without any interaction by the receiver and can be customized to include a range of pre-defined or user-defined information/data types. This flexibility is one of the protocol’s strong suits, as vendors can create custom templates with custom information they wish to collect and analyze.
NetFlow and IPFIX protocols are examples of “metadata-based” techniques which can provide valuable operational insight for data center network performance, security and other applications. In the precise sense, metadata is ‘data about data’ (as opposed to the basic data itself). For data center IP networks, metadata records document packet flows. In a flow record, the “who” and “whom” are IP addresses and port numbers, and the “how long” is byte and packet counts.
In addition to metadata-based flow analysis, direct data capture and analysis of the underlying data packets themselves can also be used for network performance and security troubleshooting. However, this typically involves a level of technical complexity and expense that in most situations does not produce more actionable understanding versus an effective system for the collection and analysis of metadata comprising network flow records.
The metadata supplied by IPFIX and Netflow protocols is similar to how your phone bill shows your calls, displaying the source, destination and volume rather than showing or listening to the actual content of the conversations. With this information, you can gain useful insights about how to manage your traffic at a lower impact on your network management strategy (when compared to full packet capture).
Some of the reasons for using metadata vs. full packet capture:
While, at a top level, IPFIX is comparable to Netflow, there is a definite and strong push for IPFIX use by end-users and support by networking and security vendors keen to adopt a much more pervasive and flexible protocol.
For reasons of backward compatibility, IPFIX supports the similar base set of 79 field types as with Netflow V9. However, IPFIX goes beyond this to support a total of 238 field types, allowing massive scope for monitoring any type of flow data required. Also, unlike NetFlow, IPFIX also allows for variable length fields which means that a field has to conform to no fixed length. Variable length fields can make transmitting information that varies frequently such as URLs (which vary from site to site), messages, and HTTP hosts easier.
IPFIX transport has to fulfil certain reliability and security requirements. Therefore Stream Control Transmission Protocol (SCTP) has been chosen as the preferred transport protocol for IPFIX for all fully compliant implementations while TCP and UDP can be used as optional protocols for backward compatibility. Preference to SCTP is significant because it is congestion-aware and reduces bandwidth use in case of congestion, thus preventing the monitoring application to reduce the performance of the monitored network.
IPFIX allows networking hardware vendors to specify a Vendor ID to create their own proprietary information to be exported. This enables the capturing and gathering of almost any data which typically may have required Syslog or SNMP directly using IPFIX for exporting it from collector devices for further analysis and monitoring.
Lastly and perhaps most importantly, as an open IETF standard, IPFIX benefits from the collective engineering efforts of thousands of individuals in the Internet community as well as support within products offered by dozens of companies in the networking marketplace.
IPFIX Protocol Overview
The IPFIX protocol consists of the following key comprises:
Figure 1. IPFIX Protocol Overview
IPFIX Protocol Extensibility
The IPFIX standard also enables extensibility of network flow export as networks evolve. IPFIX can be very flexible in the fields included in the record definitions including enabling management functions that can be similar (or better, in some situations) to Windows Management Instrumentation (WMI), SNMP and syslog information to IPFIX.
Thus, network managers are able to export whichever fields seem appropriate from an IPFIX-compliant device when troubleshooting network or security issues or planning for future network growth or expansion. Such extensibility is increasingly important as network technologies such as IP multicast and IPv6 grow in popularity, and managers need a better understanding of how they affect networked environments.
To ensure easy implementation of such extensibility, IPFIX-compliant devices export templates itemizing those flow keys configured for export. Flow collection and reporting applications can read those templates to understand which keys are exported, so that network managers need not adjust application configurations themselves.
IPFIX Flow Reporting Using Network Packet Brokers (NPBs)
As enterprise networks expand and add tools for security, performance management, and monitoring, network packet brokers (NPBs) provide a comprehensive network visibility layer enabling aggregation, filtering, and load balancing of traffic across the range of security and monitoring tools.
Network metadata protocols in general and IPFIX in particular has become increasingly important to cybersecurity in recent years. As attacks become more advanced, IPFIX provides as much information as possible about network traffic to security devices such as next-generation firewalls (NGFW), intrusion detection and prevention systems (IDS/IPS) and security web gateways (SWG)
Support of network metadata protocols like IPFIX within discrete networking devices such as network switches and routers produced data that existed in a silo. By integrating metadata engine capabilities into NPBs, you can forward contextual packet data to security and monitoring tools for deeper, precise and coordinated analysis.
Support of metadata protocols like IPFIX in NPBs also enables improved overall network performance. As metadata generation is not the primary task of the network switch or router, its generation may be affected by the device load which is why even under normal conditions, each network device may only be able to sample a subset of the traffic for metadata processing.
An NPB exporting IPFIX metadata into network security and performance monitoring tools can generate metadata on all traffic, thus providing comprehensive coverage. In addition, aggregating multiple inputs from the network into a single NPB translates into more efficient network security and performance monitoring capabilities. Deduplication functionality can decrease the amount of traffic analyzed while producing the same quality of results, for example. The NPB will generate the metadata for all traffic links and can forward traffic metadata as well as selected raw data packets at the same time to security and performance monitoring devices.
IPFIX Best Practices and Use Cases
IPFIX has wide-ranging use cases which together can be classified in 3 main categories: flow analysis, threat detection and performance monitoring. As IPFIX-enabled devices are typically deployed at key locations enabling broad-based monitoring of servers and network infrastructure devices, the resultant flow record data provides a comprehensive set of connection summaries.
Flow analysis is the elementary service provided by IPFIX and includes the following aspects.
At a top-level, IPFIX flow data can be used for a range of threat detection use cases. IPFIX-based detailed flow data based on subnet, IP address, port number, or any number of other network traffic attributes can be used for detecting changes in network behavior (‘anomaly detection’).
The central observation points at which IPFIX flow export devices are usually deployed are particularly useful for the detection of the range of advanced security threats including DDoS attacks, network scans, worm spreading, and botnet communication. The reason is that such attacks impact network traffic metrics that can be derived from flow records, such as the volume of traffic in terms of packets and bytes, the number of active flows during a specific time interval, suspicious port numbers commonly used by worms, and suspicious destination hosts for traffic.
As a result, IPFIX flow reporting can be used to identify attacks on the network such as denial of service (DoS), viruses, and worms. Changes in network behavior are represented clearly with IPFIX data and understanding these deviations from previously typical traffic patterns are helpful in identifying harmful anomalies.
Export of URL information using IPFIX flow records is a key advantage for malware threat detection. For example, it allows security teams to determine which URL a user may have clicked on before triggering malware as well as how many other people may have clicked on the same URL.
Following is a list of common threat detection and network forensics use cases enabled by IPFIX:
Network infrastructure is the business-critical foundation of every modern enterprise. Its bottlenecks, interruptions and other challenges can affect employee productivity and customer perception. IPFIX-based flow reporting can help administrators to avoid such situations, proactively address performance problems and differentiate between problems caused by the underlying network and those caused by upper applications and services.
IPFIX flow records typically provide various performance statistics associated with IP traffic, such as RTT (Round Trip Time), Delay, Jitter and others. Probes at critical points in the network deliver performance metrics as a part of IPFIX message export to IPFIX collectors which are able to store and report on these statistics. Response time metrics for critical application and monitoring of round trip time between specific servers and network systems (e.g. server response time over HTTP/HTTPS, latency for server to storage access) can be gained as indicators for network performance. Beyond validating network performance, export of IPFIX flow records to IPFIX collectors also helps in enhancing visibility into network utilization and network capacity planning.
Full support of metadata protocols including IPFIX by Niagara Networks’ advanced N2 NPB is enabled by the optional Packetron packet processing module based on x86 architecture. The Packetron module offloads IPFIX flow monitoring and removes the requirement for flow monitoring by discrete network router and switch devices and consequent impact on performance of such devices. Packetron offers a wide selection of Network Intelligence applications enabling the range of IPFIX use cases, including in-house applications developed on top of the Niagara Packetron Architecture as well as 3rd party applications.
IPFIX is an industry standard protocol for transmitting IP flow data from network devices, such as servers, switches and routers to “collector” systems for network monitoring and analysis.
IPFIX metadata provides information about IP traffic flows that gives network administrators with extremely beneficial operational insight into data center network performance, security and other applications. In contrast, direct capture of IP packet traffic necessitates a degree of technical intricacy and cost that typically generates more limited insight vis-à-vis IPFIX-based metadata collection and analysis.
While IPFIX is analogous to the alternate Netflow V9 flow metadata protocol, there is a clear preference for IPFIX by end-users and networking vendors eager to implement a multi-vendor, open standard protocol providing essential flexibility. IPFIX leads Netflow in supporting 238 field types which permits enormous room for monitoring the range of flow data required. In addition, IPFIX supports variable length fields for simplifying the transmission of information that varies frequently such as URLs and HTTP hosts. IPFIX also allows a Vendor ID to be specified to enable network device vendor-specific information to be exported.
The IPFIX architecture defines IP traffic flows as IP packets having a collection of shared properties detected at an observation point within the network. A metering process comprising a series of stages for associating packets with specific flows is followed by exporting of flow records to collector systems for analysis and storage.
The open IPFIX standard also permits flexibility for network flow exports in face of network evolution. Network managers can export any fields from IPFIX compliant devices which seem suitable for troubleshooting network or security issues or for future network growth.
IPFIX generation by Network Packet Brokers (NPBs) allows enhanced aggregate network performance while enabling comprehensive traffic analysis for security and performance monitoring applications. As metadata collection is not the main function of the network router or switch, its ability to generate metadata may be impacted by the device overhead. In addition, NPBs can export unified, all-inclusive flow records to security and monitoring tools for comprehensive analysis.
IPFIX has broad use cases which can be categorized into 3 main classes: flow analysis, threat detection and performance monitoring. As IPFIX-enabled devices are mainly deployed at significant positions in the network, it enables broad-based monitoring of hosts and network infrastructure devices with the resulting flow record data presenting an extensive set of connection summaries.
IPFIX flow data can also be optimally utilized for threat detection use cases. IPFIX enables generation of exhaustive flow data based on subnet, IP address, port number, or any number of other network traffic attributes which can be exploited for distinguishing deviations in network behavior (‘anomaly detection’). As a result, IPFIX flow reporting is used to identify and remediate network attacks such as denial of service (DoS), viruses, and worms. Changes in network behavior are characterized unambiguously with IPFIX data and recognizing such aberrations from earlier typical traffic patterns is critical in identifying harmful anomalies.
Network administrators can utilize IPFIX-based flow reporting to deal with network performance problems with foresight as well as distinguish between challenges instigated by the underlying network and those triggered by higher-level applications and services.
Full support of metadata protocols including IPFIX by Niagara Networks’ advanced N2 NPB is enabled by the optional Packetron packet processing module based on x86 architecture. For more information on how metadata protocol support within Packetron-enabled N2 NPBs enable optimally flexible network monitoring and analysis capabilities for your network, contact Niagara Networks to arrange a consultation today.