Fabric Initialization

The InfiniBand fabric, comprised of one or more subnets, relies on the Subnet Manager (SM) for initialization. This process involves discovering the topology, assigning identifiers, calculating routes, and configuring ports.

Every managed node (switches and HCAs) runs a Subnet Management Agent (SMA) to communicate with the SM.

Initialization Steps

1. Physical Connection

The process begins when all cables are physically connected and devices are powered on. Ports start in a Polling state.

2. Subnet Discovery

When the SM starts, it initiates the discovery process.

• The SM connects to every directly connected node.
• It gathers information about switches and ports.
• Discovered switches act as gateways for the SM to discover further neighbors.
• The SM uses Subnet Management Packets (SMPs) to send and receive topology information.

3. Information Gathering

The SM sends SMPs via Virtual Lane 15 (VL15) to gather detailed information. There are two types of SMP routing used during this phase:

Directed-Route SMPs

Used before the fabric is fully configured and LIDs are assigned.

• Routing: Forwarded based on a vector of port numbers defining a path through the subnet.
• Purpose: Allows communication before forwarding tables are set up. Used to discover physical connections.
• Mechanism: Only the Subnet Manager Interface (SMI) processes these packets.
• Types:
  • Get Node Info: Retrieves Node Type, Number of Ports, GUID.
  • Get Port Info: Retrieves MTU, Width, VLs.

Discovery Example:

1. SM starts with a connected switch (e.g., Leaf1).
2. SM sends a Direct Routed SMP (Get NodeInfo) to Leaf1.
3. Leaf1 responds with its type and connected ports.
4. SM uses Leaf1 to send Get NodeInfo to Leaf1’s neighbors (e.g., Spine1).
5. This process repeats recursively until all Switches and HCAs are found.

LID-Routed SMPs

Used once the fabric is initialized.

• Routing: Routed using standard switch forwarding tables.
• Purpose: Standard management operations after initialization.

Data Gathered

• Topology: Switches, HCAs, Ports/Links, Node GUIDs, Port Numbers.
• Node Info: Node Type, Port Count, GUID, Node Description.
• Port Info: MTU, Supported VLs, Link Width, Link Speed.

4. LID Assignment

After discovery, the SM assigns Local Identifiers (LIDs) to all ports. LIDs are used for routing packets within the subnet.

• HCAs: Receive a single LID per port.
• Switches:
  • Single IC: Receive a single LID.
  • Modular Switches: Receive a LID per Switch IC (switch blade).

5. Path Establishment (Routing)

The SM calculates paths between all nodes in the subnet.

• Algorithm: The simplest and most common is Min-Hop (minimum number of hops).
• Primary Path: If multiple paths exist, the best one is selected.
• Backup Paths: Remaining paths are stored as alternatives for failover.
• LFT Population: The SM populates the Linear Forwarding Table (LFT) on each switch. The LFT maps destination LIDs to output ports.

6. Port Configuration

The SM configures port parameters based on capabilities and policy.

• LID: The port configures its own LID as assigned by the SM.
• Width: Number of active physical lanes (e.g., 4x, 8x).
• MTU: Maximum Transmission Unit (payload size).
• Speed: Operating speed (e.g., HDR, NDR).

QoS Configuration

If Quality of Service (QoS) is configured, the SM pushes settings to all switches:

• VLs (Virtual Lanes): Logically separated flows over a single physical link.
• SL to VL Mapping: Maps Service Levels (SL) to Virtual Lanes (VL).
• VL Arbitration: Mechanism for output ports to select which VL to serve next.

QoS Packet Flow: When a switch receives a packet, it checks the Local Route Header (LRH) for the Destination LID (DLID) and Service Level (SL).

1. Output Port: Determined by the LFT (using DLID).
2. Virtual Lane: Determined by the SL-to-VL Mapping table.

7. Subnet Activation

Ports transition through several states during initialization.

Physical States

1. Polling: Default state after power-on. Link is attempting to detect a connection.
2. Training: Establishing link synchronization (speed/width negotiation).
3. Link Up: Physical link is established and ready.

Logical States

1. Down: Physical link is down.
2. Initialize (Init): Physical link is up. Can only send SMPs and Flow Control packets. Fabric is not yet fully initialized.
3. Armed: Port is configured by the SM but not yet active for data.
4. Active: Port is fully operational and can transmit data.
   • Transition to Active occurs after a successful “Link Integrity” check (validating VCRC on a test packet).

Useful OFED Utilities

Tools to inspect the initialized fabric:

• ibswitches: Lists all switches in the fabric with their LIDs and GUIDs.
• ibroute: Queries the forwarding table of a specific switch.
  • Usage: ibroute <Switch_LID>
  • Displays the LFT, showing which output port maps to which destination LID.