Upper Layer

The Upper Layer is the layer closest to the host application. It is responsible for providing the interface between the application and the InfiniBand fabric.

Its three core responsibilities are:

1. Upper Layer Protocols (ULPs)
2. Management Service Protocols
3. Software Transport Verbs

Upper Layer Protocols (ULPs)

Upper Layer Protocols (ULPs) are protocols that run on top of the InfiniBand transport layer. They can be native InfiniBand RDMA protocols or other standard networking protocols adapted to run over InfiniBand.

Examples of common ULPs include:

• MPI (Message Passing Interface): A library interface for distributed and parallel computing, widely used in High Performance Computing (HPC) applications.
• IPoIB (IP over InfiniBand): Enables standard TCP/IP networking over an InfiniBand fabric by encapsulating IP packets. This allows existing network applications to work over IB without modification, although it may not fully utilize RDMA performance.
• SDP (Sockets Direct Protocol): An interface for standard socket-based applications that reduces latency and CPU usage by utilizing RDMA instead of the traditional OS network stack.
• SRP (SCSI RDMA Protocol): Allows a computer to access SCSI block storage devices directly over RDMA, bypassing the CPU for high-performance storage access.
• iSER (iSCSI Extensions for RDMA): Eliminates traditional iSCSI and TCP bottlenecks by enabling zero-copy RDMA data transfer and offloading CRC calculations. This provides extremely high-performance storage connectivity.
• NFS over RDMA: Enables Network File System (NFS) to leverage the RDMA features of InfiniBand for faster file access.

Management Service Protocols

These protocols handle the management functions of the InfiniBand subnet. They are separated into Subnet Management and Subnet Services.

Subnet Management (QP0)

Subnet Management uses a special class of Management Datagram (MAD) called a Subnet Management Packet (SMP).

• Queue Pair: Always directed to QP0.
• Virtual Lane: Always uses VL15.
• Flow Control: SMPs are not subject to flow control and are sent with the highest priority (no QoS).

How to Identify an SMP:

1. Packet payload includes a MAD Header.
2. Destination QP in the transport layer is QP0.
3. Virtual Lane in the Local Route Header is 15.

General Services (QP1)

Management Services (or General Services) use a special MAD class called a General Management Packet (GMP).

• Queue Pair: Typically mapped to QP1. All GMPs received on QP1 are processed by General Services Agents (GSA).
• Virtual Lane: Uses any VL except 15 (typically VL0 by default, like standard traffic).
• Flow Control: GMPs are subject to flow control.

How to Identify a GMP:

1. Packet payload includes a GMP MAD.
2. Destination QP is QP1.
3. Virtual Lane is typically VL0 (but can use others based on QoS configuration).

Software Transport Verbs

Software Transport Verbs define the application’s software interface to the Host Channel Adapter (HCA) and the InfiniBand fabric.

• The InfiniBand specification defines the concept of the verbs but does not strictly define the API itself.
• The most common implementation is developed by the Open Fabrics Alliance (OFA).
• Verbs describe the methods an application uses to request services from the RDMA transport service.

Common Semantics

These verbs allow an application to request actions from the messaging service:

• rdma_post_send: Used by the sender application. Tells the HCA to send data via RDMA.
• rdma_post_recv: Used by the receiver application. Tells the HCA that a buffer is ready to receive data, and notifies the app when data is available.
• rdma_post_read: Used by a remote host that wants to read data from another host’s memory. Tells the local HCA to request data from a remote HCA (RDMA Read).