Addressing

InfiniBand uses a multi-layered addressing scheme to identify devices and route traffic. This ensures scalable and efficient communication both within a local subnet and across multiple subnets.

Overview

The three primary identifiers in InfiniBand are:

GUID (Globally Unique Identifier): The physical address of the device (burnt-in).
LID (Local Identifier): The data link address used for local switching (assigned by SM).
GID (Global Identifier): The network address used for routing between subnets (similar to IPv6).

graph TD
    Node[InfiniBand Node]
    
    Node --> Physical[Physical Layer]
    Physical --> GUID["GUID<br/>(64-bit Hardware ID)"]
    
    Node --> Link[Link Layer]
    Link --> LID["LID<br/>(16-bit Subnet Address)"]
    
    Node --> Network[Network Layer]
    Network --> GID["GID<br/>(128-bit Global Address)"]
    
    style GUID fill:#dbeafe,stroke:#2563eb
    style LID fill:#d1fae5,stroke:#10b981
    style GID fill:#fee2e2,stroke:#db2777
    style Node fill:#f3f4f6,stroke:#374151,color:#000000
    style Physical fill:#f3f4f6,stroke:#374151,color:#000000
    style Link fill:#f3f4f6,stroke:#374151,color:#000000
    style Network fill:#f3f4f6,stroke:#374151,color:#000000

GUID (Globally Unique Identifier)

The GUID is a 64-bit identifier that is permanently assigned to the hardware by the manufacturer. It is analogous to a MAC address in Ethernet.

Scope: Globally unique.
Assignment: Burned into the device firmware/hardware.
Structure:
- Node GUID: Identifies the device (HCA, Switch, Router) itself.
- Port GUID: Identifies a specific port on the device.
Usage: Used by the Subnet Manager (SM) to identify and authenticate devices during discovery. It is not used for packet switching in the data path.

LID (Local Identifier)

The LID is a 16-bit address assigned to each port by the Subnet Manager (SM) during fabric initialization. It is the primary address used for switching packets within a single subnet.

Scope: Unique only within the local subnet.
Assignment: Dynamic (assigned by the SM). Can change if the subnet is re-initialized.
Structure: 16-bit integer (Range: 1 to 49151). 0 is reserved; values above 0xBFFF are reserved for multicast.
Usage:
- Used in the Local Route Header (LRH).
- Switches use the Linear Forwarding Table (LFT) to look up the destination LID (DLID) and determine the output port.
LID Mask Control (LMC): The SM can assign a range of LIDs to a single port to enable multiple paths (LID Mask Control).

Key Difference: In Ethernet, switches learn MAC addresses. In InfiniBand, switches are explicitly programmed by the SM to route based on LIDs.

GID (Global Identifier)

The GID is a 128-bit address used for routing packets between different subnets (Inter-Subnet Routing). It is effectively a valid IPv6 address.

Scope: Globally unique across the entire network (multiple subnets).
Assignment: Constructed from the subnet prefix and the port GUID.
Structure:
- Upper 64 bits: Subnet Prefix (assigned by the SM, default is 0xFE80::).
- Lower 64 bits: Interface ID (typically the Port GUID).
Usage:
- Used in the Global Route Header (GRH).
- Required when traffic needs to pass through a Router to reach a destination in a different subnet.
- Also used by Upper Layer Protocols (ULPs) like RDMA CM (Communication Manager) to establish connections.

Summary Table

Feature	GUID	LID	GID
Full Name	Globally Unique Identifier	Local Identifier	Global Identifier
Size	64-bit	16-bit	128-bit
Analogy	Ethernet MAC Address	Layer 2 Switch Address	IPv6 Address
Scope	Global (Hardware)	Local Subnet	Global Network
Assigned By	Manufacturer	Subnet Manager	SM Prefix + GUID
Used In	Discovery & Management	LRH (Local Switching)	GRH (Inter-Subnet)