Packet Structure

InfiniBand packets are structured with a series of headers that define routing, transport, and data integrity. The structure is designed to be efficient for hardware processing.

Packet Overview

An InfiniBand packet consists of the following components:

1. Local Route Header (LRH) - 8 bytes (Always present)
2. Global Route Header (GRH) - 40 bytes (Optional, for inter-subnet routing)
3. Base Transport Header (BTH) - 12 bytes (Always present)
4. Extended Transport Headers - Variable (Dependent on OpCode)
5. Payload - 0 to 4096 bytes (The actual data)
6. Invariant CRC (ICRC) - 4 bytes
7. Variant CRC (VCRC) - 2 bytes

Routing Headers

Local Route Header (LRH)

• Size: 8 Bytes
• Purpose: Used for local switching within a subnet.
• Key Fields:
  • SL (Service Level): Used for QoS (Quality of Service) to map to Virtual Lanes.
  • DLID (Destination Local ID): The address of the destination port in the local subnet.
  • SLID (Source Local ID): The address of the source port.
  • LNH (Link Next Header): Indicates what header follows (e.g., BTH or GRH).

Note: Unlike Ethernet MAC addresses which are burned into the hardware, LIDs are dynamically assigned by the Subnet Manager (SM) when the link comes up. They are unique only within the local subnet.

For more details, see LRH and GRH InfiniBand Headers.

Global Route Header (GRH)

• Size: 40 Bytes
• Purpose: Used for routing between different subnets (inter-subnet). It is effectively an IPv6 header.
• Key Fields:
  • SGID (Source GID): 128-bit global identifier of the source.
  • DGID (Destination GID): 128-bit global identifier of the destination.
  • TClass (Traffic Class): Similar to DiffServ in IP.
  • FlowLabel: Used to identify a sequence of packets.

For more details, see LRH and GRH InfiniBand Headers.

Transport Headers

Base Transport Header (BTH)

• Size: 12 Bytes
• Purpose: Specifies the transport operation and destination Queue Pair (QP).
• Key Fields:
  • OpCode: Defines the type of packet (e.g., Send, RDMA Write, RDMA Read, Acknowledge).
  • DestQP (Destination Queue Pair): The specific queue pair number at the destination.
  • PSN (Packet Sequence Number): Used to detect missing or out-of-order packets.
  • P_Key (Partition Key): Used for network isolation (similar to VLANs).

Extended Transport Headers

Depending on the OpCode in the BTH, additional headers may follow. Common examples include:

• RDMA Extended Transport Header (RETH): Contains the Virtual Address and R_Key (Remote Key) for RDMA operations.
• Datagram Extended Transport Header (DETH): Used for Unreliable Datagram (UD) transport.
• Atomic Extended Transport Header (AtomicETH): Used for atomic operations like FetchAdd or CmpSwap.

Data & Integrity

Payload

The actual data being transferred. The Maximum Transmission Unit (MTU) typically ranges from 256 bytes to 4096 bytes.

Cyclical Redundancy Checks (CRC)

• ICRC (Invariant CRC): Covers fields that do not change from hop to hop.
• VCRC (Variant CRC): Covers the entire packet, including fields that do change (like the VL field in the LRH) during switching.