Packet Capture with ibdump & Wireshark
InfiniBand packets can be captured and analyzed using Wireshark, but the capture process differs from traditional Ethernet networks.
Why Not tcpdump?
Section titled “Why Not tcpdump?”Standard packet capture tools like tcpdump or windump do not work for InfiniBand traffic. This is because InfiniBand bypasses the kernel networking stack entirely — data moves directly between user space and the HCA via RDMA, so kernel-level packet hooks never see the traffic.
ibdump
Section titled “ibdump”ibdump is the InfiniBand-specific packet capture tool. It captures raw InfiniBand packets directly from the HCA and writes them to a file that Wireshark can open.
Basic Usage
Section titled “Basic Usage”ibdump -d <HCA_name> -w <output_file>-d <HCA_name>: Specifies which HCA device to capture from (e.g.,mlx5_0). Useibv_devicesto list available HCAs.-w <output_file>: Writes the captured packets to a file in a format compatible with Wireshark (.pcap).
Example
Section titled “Example”ibdump -d mlx5_0 -w /tmp/ib_capture.pcapThis captures all InfiniBand traffic on mlx5_0 and saves it to /tmp/ib_capture.pcap. You can then open this file in Wireshark for analysis.
Analyzing in Wireshark
Section titled “Analyzing in Wireshark”Wireshark includes built-in dissectors for InfiniBand protocols. Once you open an ibdump capture file, Wireshark will decode the InfiniBand headers — including Local Route Headers (LRH), Global Route Headers (GRH), Base Transport Headers (BTH), and payload data — allowing you to inspect individual packets, filter by QP number, and trace RDMA operations.