For example, I will update the driver for the Mellanox ConnectX-3 Pro EN / HP 544+ QSFP (764284-B21) network adapter that I installed in the HPE DL380 Gen10 server with Ubuntu Server 18.04.
To update the driver, you need to switch to the root user:
First I looked at the version of the driver installed:
modinfo mlx4_core | grep version version: 4.0-0 modinfo mlx4_core | grep ^version:|sed 's/version: * //g' 4.0-0 ethtool -i ens3
After running these commands on multiple Ubuntu Server 18.04 servers, I noticed that the default driver is version 4.0-0.
Let’s see if the system sees the network adapter, what it is called and its Part number:
lspci -vvv lspci | grep Mellanox 12:00.0 Network controller: Mellanox Technologies MT27520 Family [ConnectX-3 Pro] lspci -vv -s 12:00.0 | grep "Part number" -A 3 [PN] Part number: 764284-B21 [EC] Engineering changes: A3 [SN] Serial number: CN7644033S [V0] Vendor specific: PCIe FDR x8 13W
By the way, I tried to install at the same time network adapters on different chips (Intel XL710-QDA1 and Mellanox ConnectX-3 / HP 544+ QSFP) in the HPE DL380 Gen10 server and noticed that they do not work at the same time and cannot be seen by the lspci command.
Then I looked at the driver on the official website
Downloaded LTS version MLNX_OFED_LINUX-4.9-220.127.116.11-ubuntu18.04-x86_64.tgz.
Since in the newest version MLNX_OFED_LINUX-5.1-18.104.22.168-ubuntu18.04-x86_64.tgz at the time of this writing, there were drivers starting from ConnectX-4, but for ConnectX-3 there were none.
I unpacked the downloaded archive:
tar -xzf MLNX_OFED_LINUX-4.9-22.214.171.124-ubuntu18.04-x86_64.tgz cd MLNX_OFED_LINUX-4.9-126.96.36.199-ubuntu18.04-x86_64
To start the update process, you need to run the installation script:
This will install the full utilities package and update the driver. The script will detect all Mellanox network adapters and offer to select specific or all, as well as perform a firmware update. Make sure that the correct version of the operating system distribution is specified in the distro file.
But I decided to update the driver and install the minimal set of utilities:
./mlnxofedinstall --upstream-libs --dpdk
During installation, some packages will be installed and removed, in case of an error, you can install or remove them manually, and then run the script again, for example:
apt install tk bison debhelper flex autoconf automake graphviz dpatch gfortran libgfortran4 quilt dkms chrpath tcl m4 swig libnl-route-3-dev ixnfo.com apt remove ...
After a successful update, restart the driver:
Restart the server and make sure the network is working:
If you only need to update the firmware, then run:
An example of creating a backup copy of the firmware and configuration of the network adapter:
flint -d 12:00.0 ri firmware_backup.bin flint -d 12:00.0 dc config_backup.ini
Close the ports if necessary:
iptables -A INPUT --protocol tcp --in-interface ens2d1 --dport 8888 -j DROP iptables -A INPUT --protocol tcp --in-interface ens2d1 --dport 8889 -j DROP
Modules that are automatically loaded when the operating system starts:
mlx4_en, mlx4_core, mlx4_ib
If you need to unload driver modules, then the mlx*_core module must be unloaded last, for example:
modprobe -r mlx4_ib modprobe -r mlx4_en modprobe -r mlx4_core
An example of loading modules:
modprobe mlx4_core modprobe mlx4_ib modprobe mlx4_en
I will give a list of important packages that are installed by the driver update script or via apt install:
rdma-core, libibmad, opensm, opensm-libs, ibutils-libs, ibutils, infiniband-diags, perftest, mstflint
See also my articles:
- Firmware Update HP 544+QSFP (764284-B21) Mellanox ConnectX-3 Pro EN
- Installing and updating the ixgbe driver
- Mellanox ConnectX errors “Could not join netdev: No space left on device”