For example, I will update the driver for the Mellanox ConnectX-3 Pro EN / HP 544+ QSFP (764284-B21) network adapter that I installed in the HPE DL380 Gen10 server with Ubuntu Server 18.04.
To update the driver, you need to switch to the root user:
sudo -i
First I looked at the version of the driver installed:
modinfo mlx4_core | grep version
version: 4.0-0
modinfo mlx4_core | grep ^version:|sed 's/version: * //g'
4.0-0
ethtool -i ens3
After running these commands on multiple Ubuntu Server 18.04 servers, I noticed that the default driver is version 4.0-0.
Let’s see if the system sees the network adapter, what it is called and its Part number:
lspci -vvv
lspci | grep Mellanox
12:00.0 Network controller: Mellanox Technologies MT27520 Family [ConnectX-3 Pro]
lspci -vv -s 12:00.0 | grep "Part number" -A 3
[PN] Part number: 764284-B21
[EC] Engineering changes: A3
[SN] Serial number: CN7644033S
[V0] Vendor specific: PCIe FDR x8 13W
By the way, I tried to install at the same time network adapters on different chips (Intel XL710-QDA1 and Mellanox ConnectX-3 / HP 544+ QSFP) in the HPE DL380 Gen10 server and noticed that they do not work at the same time and cannot be seen by the lspci command.
Then I looked at the driver on the official website
https://www.mellanox.com/products/infiniband-drivers/linux/mlnx_ofed
Downloaded LTS version MLNX_OFED_LINUX-4.9-2.2.4.0-ubuntu18.04-x86_64.tgz.
Since in the newest version MLNX_OFED_LINUX-5.1-2.5.8.0-ubuntu18.04-x86_64.tgz at the time of this writing, there were drivers starting from ConnectX-4, but for ConnectX-3 there were none.
I unpacked the downloaded archive:
tar -xzf MLNX_OFED_LINUX-4.9-2.2.4.0-ubuntu18.04-x86_64.tgz
cd MLNX_OFED_LINUX-4.9-2.2.4.0-ubuntu18.04-x86_64
To start the update process, you need to run the installation script:
./mlnxofedinstall
This will install the full utilities package and update the driver. The script will detect all Mellanox network adapters and offer to select specific or all, as well as perform a firmware update. Make sure that the correct version of the operating system distribution is specified in the distro file.
But I decided to update the driver and install the minimal set of utilities:
./mlnxofedinstall --upstream-libs --dpdk
During installation, some packages will be installed and removed, in case of an error, you can install or remove them manually, and then run the script again, for example:
apt install tk bison debhelper flex autoconf automake graphviz dpatch gfortran libgfortran4 quilt dkms chrpath tcl m4 swig libnl-route-3-dev ixnfo.com
apt remove ...
After a successful update, restart the driver:
/etc/init.d/openibd restart
Restart the server and make sure the network is working:
reboot
If you only need to update the firmware, then run:
./mlnxofedinstall --fw-update-only
An example of creating a backup copy of the firmware and configuration of the network adapter:
flint -d 12:00.0 ri firmware_backup.bin
flint -d 12:00.0 dc config_backup.ini
Close the ports if necessary:
iptables -A INPUT --protocol tcp --in-interface ens2d1 --dport 8888 -j DROP
iptables -A INPUT --protocol tcp --in-interface ens2d1 --dport 8889 -j DROP
Modules that are automatically loaded when the operating system starts:
ConnectX®-2/ConnectX®-3/ConnectX®-3 Pro:
mlx4_en, mlx4_core, mlx4_ib
ConnectX®-4/ConnectX®-4 Lx/ConnectX®-5:
mlx5_core, mlx5_ib
If you need to unload driver modules, then the mlx*_core module must be unloaded last, for example:
modprobe -r mlx4_ib
modprobe -r mlx4_en
modprobe -r mlx4_core
An example of loading modules:
modprobe mlx4_core
modprobe mlx4_ib
modprobe mlx4_en
I will give a list of important packages that are installed by the driver update script or via apt install:
rdma-core, libibmad, opensm, opensm-libs, ibutils-libs, ibutils, infiniband-diags, perftest, mstflint
See also my articles:
- Firmware Update HP 544+QSFP (764284-B21) Mellanox ConnectX-3 Pro EN
- Installing and updating the ixgbe driver
- Mellanox ConnectX errors “Could not join netdev: No space left on device”