Data Link Layer

The Data Link Layer is responsible for critical services including packet layout, point-to-point link operations, and switching within a local subnet.

It handles several key functions:

1. Packet Management
2. Layer 2 Addressing
3. Quality of Service (QoS)
4. Packet Forwarding
5. Flow Control
6. Data Integrity

Packet Management

A packet is a routable unit that transfers data in a fabric. InfiniBand generally distinguishes between two main types of packets:

• Link Management Packets: Used for link configuration and maintenance, including virtual lane support.
• Data Packets: Contain payload and operations (send, read, write, acks).

Layer 2 Addressing (LIDs)

Local Identifiers (LIDs) are used to route packets within a subnet. They reside in the Local Route Header (LRH).

• Assignment: LIDs are assigned by the Subnet Manager during initialization or topology changes.
• Structure:
  • HCAs: Assigned a LID per port.
  • Switches:
    • Single Unit switches receive a single LID for the entire switch.
    • Modular switches receive a LID per switch module in the chassis.
• Capacity: Each subnet can support up to 48k unicast LIDs and 16k multicast LIDs.

Quality of Service (QoS)

QoS enables different priorities for applications, users, and data flows. It is implemented using Service Levels (SL) and Virtual Lanes (VL), both of which are defined in the Layer 2 header (LRH).

Service Level (SL)

The Service Level is a field in the LRH that defines the class of the packet, allowing differentiation between data streams for various applications.

Virtual Lanes (VL)

Virtual Lanes provide a mechanism for creating multiple virtual links within a single physical link. They are critical for implementing QoS:

• Mapping: Packets are mapped to specific VLs based on their service level and classification.
• Resource Isolation: Each VL gets its own buffers and is configured with different weights and priorities.
• Scheduling: Each VL is configured with an arbiter that controls transmission scheduling.

The InfiniBand specification allows for up to 16 Virtual Lanes:

• VL15: Reserved for Management Traffic only.
• VL0: Default for Data Traffic (Unclassified).
• VL1-14: Used for classified data traffic.

Packet Forwarding

Switches make forwarding decisions by examining the LID in the packet’s Local Route Header (LRH).

1. Lookup: The switch compares the destination LID against its Linear Forwarding Table (LFT).
2. Mapping: The LFT maps the Destination LID to an exit port.
3. QoS Enforcement: The LFT may also include SL-to-VL mappings to ensure the packet is forwarded on the correct Virtual Lane.

Relevant Commands

• ibswitches: Pulls the LID and GUID of all switches in the fabric.
• ibroute <lid>: Displays the forwarding table of a specific switch (requires the switch LID).

Flow Control

InfiniBand is a high-throughput, lossless fabric that leverages link-layer flow control to minimize packet loss.

• Credit-Based: Flow control is credit-based.
  • The receiver sends credits to the sender indicating available receive buffer space.
  • The sender waits for credits before transmitting.
• Per-VL Allocation: Credits are allocated per Virtual Lane, ensuring that congestion and latency in one flow do not block others.
• Drop Timer: Packets are not queued indefinitely; a drop timer prevents persistent congestion by dropping packets if they cannot be delivered within a set time.

Data Integrity (Error Detection)

InfiniBand uses Cyclic Redundancy Checks (CRCs) to ensure packet validity. Checksums are appended to the end of the packet and verified by the destination.

Two distinct CRCs are used:

1. ICRC (Invariant CRC): A 4-byte value used to detect bit errors across the entire fabric. It covers all fields that do not change from source to destination.
2. VCRC (Variant CRC): A 2-byte value that covers the entire packet on a per-hop basis.

Note: HCAs or switches will drop packets if the CRCs do not match.

Useful OFED Utilities

• iblinkinfo: Displays all nodes in the fabric, including LIDs, GUIDs, host names, and link speeds.
• ibnetdiscover: Performs fabric discovery and lists all nodes in the fabric along with connectivity information.