ibsim (InfiniBand Simulator)

ibsim: InfiniBand Simulator

ibsim is a software simulator for InfiniBand fabrics. It emulates the behavior of the InfiniBand management layer, allowing users to run standard diagnostic tools and subnet managers against a virtual network topology.

What is ibsim?

At its core, ibsim intercepts Management Datagram (MAD) traffic. Instead of sending these packets to physical hardware via the kernel driver (typically /dev/umadX), it routes them to a simulator process. This is achieved using a preloaded shared library, libumad2sim.so, which wraps the standard libibumad calls.

This allows you to run unmodified InfiniBand utilities like OpenSM, ibnetdiscover, and smpquery on a machine that has no InfiniBand hardware installed.

What it Can Do

Simulate Arbitrary Topologies: You can define complex network structures (Switches, CA/HCAs) using a text-based .net file format.
Run Real Management Tools: Execute opensm, iblinkinfo, ibtracert, and other infiniband-diags tools against the simulated fabric.
Test Subnet Manager Configurations: Verify how OpenSM handles specific topologies, routing engines, or partition configurations.
Inject Faults: Simulate link failures or packet drops to test fault tolerance and recovery mechanisms.
Educational Use: Learn how InfiniBand discovery and management work without needing expensive hardware.

What it Can’t Do (Limitations)

No Data Traffic: ibsim focuses on the Management layer (MADs). It simulates the control plane, not the data plane. You cannot run ib_write_bw or ib_send_lat to test throughput or latency.
No Hardware Emulation: It does not emulate the actual hardware registers or firmware behavior of specific HCAs (e.g., ConnectX-6). It emulates standard InfiniBand specifications.
Performance Testing: It is not a tool for performance benchmarking of the network itself.

How to Run ibsim

The standard way to run ibsim involves compiling the source and using LD_PRELOAD.

1. Build from Source

git clone https://github.com/linux-rdma/ibsim
cd ibsim
make

2. Start the Simulator

# Start the simulator with a topology file
./ibsim/ibsim -s net-examples/net.1

3. Run Clients

To run a client (like ibnetdiscover), you must preload the wrapper library so it talks to the simulator instead of real hardware.

LD_PRELOAD=./umad2sim/libumad2sim.so ibnetdiscover

Interacting with the Simulator: The `ibsim` Command Console

Once ibsim is running, it provides an interactive command prompt (sim>) that lets you inspect and modify the simulated network—perfect for on-the-fly testing, learning, and debugging. Here are some of the most commonly used commands and how you can leverage them:

Useful ibsim Console Commands

1. View Current Topology: `Dump`

Outputs all nodes, links, and port statuses in the simulation. Extremely helpful for verifying the current state.

sim> Dump

You can also inspect a specific node:

sim> Dump "nodeid"

2. Connect or Reconnect Nodes: `Link` / `ReLink`

Manually add or restore connections between ports or nodes.

Link two nodes (by name or GUID) and specify which port on each:

sim> Link "leaf-r01-01"[1] "spine-01"[3]

Restore all previously disconnected links for a node:

sim> ReLink "leaf-r01-01"

Restore a specific port’s link:

sim> ReLink "leaf-r01-01"[3]

3. Disconnect Nodes: `Unlink` and `Clear`

Unlink all ports for a node (network partitioning/simulation of a node failure):

sim> Unlink "spine-01"

Unlink just one port:

sim> Unlink "spine-01"[4]

Reset and unlink (clears port state):

sim> Clear "leaf-r01-01"[2]

4. Route Calculation

Check simulated routing between any two LIDs:

sim> Route <from-lid> <to-lid>

For example:

sim> Route 6 14

5. Modify Node/Port Identifiers: `Guid`

Change the GUID of a node or a port:

sim> Guid "leaf-r01-01" 0x200020002

Or for a specific port:

sim> Guid "leaf-r01-01"[2] 0x300000005

6. Inject Errors: `Error`

Inject error rates into specific links/ports for fault tolerance testing:

sim> Error "leaf-r01-02"[1] 0.05

Set error only on a specific attribute (e.g., PortInfo, NodeInfo):

sim> Error "leaf-r01-01"[3] 0.2 21

7. Setting LIDs: `Baselid`

Assign a new LID (Local Identifier) to a node’s port:

sim> Baselid "leaf-r01-01"[4] 32

8. Setting Performance Counters: `PerformanceSet`

Handy for simulating counters and status for management traffic:

sim> PerformanceSet "leaf-r01-01"[1] PortCounters.PortXmtDiscards=1000

9. Listing Clients

See which diagnostic tools (clients) are connected to your simulation:

sim> Attached

10. Other Utility Commands

!<filename>: Run commands from a script file.
Verbose [level]: Toggle simulator debug output.
Wait <sec>: Pause simulator prompt (useful in scripts).
Quit: Exit the simulator.

Example: A Typical ibsim Command Session

# Dump the network
sim> Dump

# Disconnect a link to simulate a cable failure
sim> Unlink "leaf-r01-02"[1]

# Reconnect it
sim> ReLink "leaf-r01-02"[1]

# Simulate a routing query
sim> Route 6 24

Pro Tip: Use `Dump` Often

The Dump command is invaluable for seeing the real-time state of your topology—especially after you make any modifications. If you are writing scripts or testing OpenSM behavior, get in the habit of running Dump before and after actions.

See the ibsim GitHub README for further examples or the full command set.

Making it Easier: ibsim Control Panel

Working with LD_PRELOAD and manually crafting topology files can be cumbersome. I built the ibsim Control Panel so that I could play with ibsim without wasting a bunch of time on the setup.

Repository: github.com/zbrooks442/ibsim_control_panel

Everything is available to you in a simple interface that dockerizes the ibsim environment and provides:

Visual Topology Editing: Drag-and-drop switches and nodes to design your network without writing .net files manually.
Simulation Management: Start and stop the simulator and OpenSM instances with a single click.
Web Terminal: An in-browser terminal pre-configured with the correct environment variables (LD_PRELOAD) to run diagnostic commands immediately.
Cross-Platform: Runs on macOS and Windows via Docker, bypassing the Linux-only requirement of the raw tools.