Redundant configuration with ib-bonding
Bonding can be used with InfiniBand as well as with Ethernet. This ib-bonding works on ipoib and can currently only be used in active / backup mode ( mode=1) (confirmed in MLNX_OFED-1.5.2-2.1.0). The settings are almost the same as for Ethernet.
Write to network script (using RHEL5.5)
1. Add the following to the master (bond0) interface.
TYPE=Bonding MTU=<Match with slave MTU>
Example: /etc/sysconfig/network-scripts/ifcfg-bond0
DEVICE=bond0
IPADDR=192.168.1.1
NETMASK=255.255.255.0
NETWORK=192.168.1.0
BROADCAST=192.168.1.255
ONBOOT=yes
BOOTPROTO=none
USERCTL=no
TYPE=Bonding
MTU=65520
* MTU=65520 must be set to the same value as all IPoIB slaves must be operating in Connected Mode (CM). When used in Datagram Mode, MTU=2044.
Example: /etc/sysconfig/network-scripts/ifcfg-bond0
DEVICE=bond0
IPADDR=192.168.1.1
NETMASK=255.255.255.0
NETWORK=192.168.1.0
BROADCAST=192.168.1.255
ONBOOT=yes
BOOTPROTO=none
USERCTL=no
TYPE=Bonding
MTU=65520
* MTU=65520 must be set to the same value as all IPoIB slaves must be operating in Connected Mode (CM). When used in Datagram Mode, MTU=2044.
2. Add the following to the slave (ib) interface.
SLAVE=yes
MASTER=
TYPE=InfiniBand
PRIMARY=
Example: /etc/sysconfig/network-scripts/ifcfg-ib0
DEVICE=ib0
USERCTL=no
ONBOOT=yes
MASTER=bond0
SLAVE=yes
BOOTPROTO=none
TYPE=InfiniBand
PRIMARY=yes
3. Add the following description to /etc/modprobe.conf.
alias bond0 bonding
options bond0 miimon=100 mode=1 max_bonds=1
* max_bonds is the number of bonds. You can check the operation as follows.
ib0 interface failed
Automatically send data on backup ib1
See ib-bonding.txt in the MLNX_OFED-1.5.2-2.10 installation directory for details.