Network Layer

The Network Layer is responsible for routing packets between different InfiniBand subnets. While the Link Layer handles traffic within a single subnet (up to 48k nodes), the Network Layer uses routers to connect multiple subnets, effectively extending the fabric beyond this limitation.

InfiniBand Routing

Routing in InfiniBand allows for scalable and resilient fabric designs.

Benefits

• Scaling: Enables the fabric to scale beyond the 48,000 node limit of a single subnet.
• Isolation: Isolates subnets to provide fault resilience; issues in one subnet do not necessarily propagate to others.
• Subnet Management: Allows for independent subnet management. Each subnet has its own Subnet Manager (SM), ensuring that SM issues are contained.
• Connectivity: Facilitates the integration of existing topologies without requiring a complete redesign of the network.

Global Route Header (GRH) & GIDs

Routing decisions at the Network Layer are made using the Global Identifier (GID) found within the Global Route Header (GRH) of an InfiniBand packet.

Global Identifier (GID)

The GID is a 128-bit field used to identify a single end port or a multicast group. It is globally unique across subnets, similar to an IPv6 address.

• Structure:
  • Subnet Prefix (64 bits): Identifies the specific subnet.
  • Port GUID (64 bits): Uniquely identifies the port within the subnet.

Operations

• Routing: Routers use the GID to determine the path for packets traversing between subnets.
• Traffic Types: Routers support both unicast and multicast operations.
• Default GID: Each HCA port is automatically assigned a default GID. This GID is usable only within the local subnet, functioning similarly to an IPv6 link-local address.

Useful OFED Commands

The following commands are useful for inspecting Network Layer information:

• ibv_devices: Displays all InfiniBand devices installed on the server.
• ibaddr: Displays GID and LID information for the active ports.