We will introduce the configuration of the Mellanox OFED stack.

Mellanox OFED Stack

1. mthca HCA (IB) driver

mthca is a low-level driver for Mellanox Technologies HCA devices InfiniHost, InfiniHost III Ex, InfiniHost III Lx.

2. mlx4 VPI driver

mlx4 is a low-level driver for Mellanox Technologies' ConnectX adapters. This ConnectX can act as an InfiniBand adapter, Ethernet NIC, or Fiber Channel over Ethernet HBA, and this driver in OFED 1.4 supports the InfiniBand and Ethernet NIC variants. In order to fit into the supported morphology, this driver is split into four modules:

mlx4_core

mlx4_core handles low-level functions like device initialization and firmware command processing. It also controls resource allocation so that InfiniBand and Ethernet functions can share devices without interfering with each other.

mlx4_ib

mlx4_ib handles InfiniBand-specific functions and connects to the InfiniBand midlayer.

mlx4_en

mlx4_en handles ethernet specific functions and connects to its netdev midlayer. 10GigE drivers under driver/net/mlx4.

mlx4_fc

mlx4_fc handles FCoE functionality using ConnectX's Fiber Channel hardware offload.

3.Mid-layer Core

Core services include Management Interface (MAD), Connection Interface (CM) and Subnet Administrator Interface (SA). This stack contains elements for both user-mode and kernel applications. This core service runs in the kernel and provides access to user mode for verbs, CM and management.

4. Open FCoE

FCoE functionality is based on the Open-FCoE project. Mellanox OFED includes open-fcoe.org modules for libfc and fcoe.

5.ULPs

IPoIB

The IPoIB (IP over IB) driver implements a network interface over InfiniBand. IPoIB encapsulates IP datagrams over InfiniBand. IPoIB adds headers to IP datagrams and sends the results over the InfiniBand transport service. The initial value of the transport service is Reliable Connection (RC), but it can also be set to Unreliable Datagram (UD). (This interface supports unicast, multicast and broadcast).

RDS

RDS ( Reliable Datagram Sockets ) is a socket API that provides reliable and orderly arrival datagram transfer between sockets over TCP/IP or RC ( Reliable Connection).

SDP

SDP (Socket Direct Protocol) is a byte stream transfer protocol that provides TCP stream behavior (semantics) and takes advantage of InfiniBand's advanced protocol offload capabilities. SDP thus retains the traditional TCP APIs and behaviors (semantics) used in most network applications today, while achieving lower CPU and memory bandwidth utilization compared to TCP.

SRP

SRP (SCSI RDMA Protocol) is designed to take full advantage of the protocol offload and RDMA properties offered by the InfiniBand architecture. SRP makes it easy to use SCSI software over the InfiniBand architecture. The SRP driver (known as the SRP initiator) differs from traditional low-level SCSI drivers in Linux in that the SRP initiator does not take control of the local HBA. Instead, SRP initiators control connections to IO controllers (known as SRP targets), providing access to remote storage on the InfiniBand fabric.

6.MPI

MPI (Message Passing Interface) is a library that allows parallel software development to enable parallel computers, clusters, and heterogeneous networks. Mellanox OFED implements the following MPI over InfiniBand.
・Open MPI – Implementation of open source MPI-2 by the Open MPI Project
・OSU MVAPICH – Implementation of MPI-1 by Ohio State University
Mellanox OFED includes MPI benchmark tests such as OSU BW/LAT, Intel MPI Benchmark and Presta.

7. Infiniband Subnet Manager

All InfiniBand compliant ULPs require proper operation of the subnet manager running in the InfiniBand fabric at all times. A subnet manager can run on any node or InfiniBand switch. OpenSM is an InfiniBand compliant subnet manager and is installed as part of Mellanox OFED.

8.Diagnostic Utilities

Mellanox OFED includes two diagnostic packages for network and data center administrators:
・ibutils - Mellanox Technologies diagnostic utilities
・infiniband-diags – OpenFabrics Alliance Infiniband diagnostic tool

9.Performance Utilities

A suite of tests written on uverbs for performance evaluation as microbenchmarks
It can be used for tuning and functional testing for hardware and software. See PERF_TEST_README.txt in docs/.

*The above content is an excerpt from the Mellanox OFED User Manual 1.40.
For details, please learn User Manual from Hope Page of Mellanox.