Here are some recommendations for setting up a server on which Accel-ppp will work.
1) In the BIOS, disable Hyper-threading so that the softirq load from the network card is not distributed to virtual cores, by the way, when HT is enabled and since there are more cores with virtual, and if virtual cores are not used, then on Zabbix charts the average CPU utilization is lower, although it is not and is confusing, therefore, I recommend monitoring the loading of each kernel, and not in general, for example, as I wrote in the article: Monitoring CPU usage in Zabbix. You can also draw the idle of each CPU core to see how much % is free and whether some cores are overloaded.
See the general softirq graph with Hyper-threading on and off::
2) If irqbalance doesn’t distribute the network adapter interruptions by processor cores, then disable irqbalance and manually configure RPS/RSS.
Distribution of network card interrupts across processor cores
3) In the BIOS, select the plan “CPU performance”, and also set other possible options to “Performance”.
Also, select “Always Power ON”, so that the server will turn on itself after the power disappears and appears, you can also disable the network adapter with regular ethernet ports.
Also in the operating system, set “performance” as I described in this article:
Changing CPU Scaling Governor on Linux
4) Disable unnecessary offloads on network interfaces, I note that they are needed to improve performance, but sometimes because of them the shaper does not work correctly, for example, I had a very low sending speed with tso, gro, gso enabled, so I disabled it:
ethtool -K ens2f1 tso off gro off gso off
/sbin/ethtool -K ens2f1 rxhash off lro off sg off rxvlan off txvlan off rx off tx off > /dev/null 2>&1
/sbin/ethtool -N ens2f1 rx-flow-hash udp4 sdfn > /dev/null 2>&1
5) Update the driver for the network adapter, I note that on some versions of the drivers, the occupied space of RAM can quickly increase.
If it increases, you can, for example, unload and load the ixgbe module to see if it has a problem (attention, the connection to the server will be lost):
rmmod ixgbe && modprobe ixgbe
6) Enable forwarding of packages, increase the “nf_conntrack_max” and optimize other parameters in /etc/sysctl.conf.
Tuning nf_conntrack
7) If the server uses Zabbix with a standard Linux template, then disable “Network interface discovery” in the template or for the host, otherwise, for example, with a large number of ipoe interfaces and when Zabbix constantly scans them, there will be performance problems. We will also specify the ListenIP parameter to the Zabbix agent – on which IP address it should work (by default, everything is 0.0.0.0). Make sure it’s running on the correct IP address, and make sure there are no other processes running on “all interfaces”:
netstat -tulpn | grep zabbix_agentd
netstat -tulpn | less
netstat -tulpn > netstat.txt
8) Enable rp_filter (I strongly recommend not to disable it, except in extreme cases, or specify 2 instead of 1):
sysctl -a | grep "rp_filter"
nano /etc/sysctl.conf
net.ipv4.conf.default.rp_filter=1
net.ipv4.conf.all.rp_filter=1
sysctl -p
We will protect clients by prohibiting traffic to them on ports that are accidentally opened for them (compiled a small list of iptables rules, I recommend applying at least them):
iptables -A FORWARD -d 172.16.0.0/12 -p tcp -m tcp --dport 25 -m comment --comment SMTP_Blocked_to_local -j DROP
iptables -A FORWARD -d 172.16.0.0/12 -p udp -m udp --dport 53 -m comment --comment DNS_Blocked_for_local -j DROP
iptables -A FORWARD -d 172.16.0.0/12 -p tcp -m tcp --dport 53 -m comment --comment DNS_Blocked_for_local -j DROP
iptables -A FORWARD -d 172.16.0.0/12 -p tcp -m tcp --dport 111 -m comment --comment Blocked_for_local -j DROP
iptables -A FORWARD -d 172.16.0.0/12 -p udp -m udp --dport 111 -m comment --comment Blocked_for_local -j DROP
iptables -A FORWARD -d 172.16.0.0/12 -p udp -m udp --dport 123 -m comment --comment NTP_Blocked_for_local -j DROP
iptables -A FORWARD -d 172.16.0.0/12 -p tcp -m tcp --dport 135 -m comment --comment Blocked_for_local -j DROP
iptables -A FORWARD -d 172.16.0.0/12 -p tcp -m tcp --dport 139 -m comment --comment NetBIOS_Blocked_for_local -j DROP
iptables -A FORWARD -d 172.16.0.0/12 -p udp -m udp --dport 137 -m comment --comment NetBIOS_Blocked_for_local -j DROP
iptables -A FORWARD -d 172.16.0.0/12 -p udp -m udp --dport 138 -m comment --comment NetBIOS_Blocked_for_local -j DROP
iptables -A FORWARD -d 172.16.0.0/12 -p tcp -m tcp --dport 445 -m comment --comment NetBIOS_Blocked_for_local -j DROP
iptables -A FORWARD -d 172.16.0.0/12 -p udp -m udp --dport 161 -m comment --comment SNMP_Blocked_for_local -j DROP
iptables -A FORWARD -d 172.16.0.0/12 -p tcp -m tcp --dport 179 -m comment --comment BGP_Blocked_for_local -j DROP
iptables -A FORWARD -d 172.16.0.0/12 -p udp -m udp --dport 1900 -m comment --comment UPnP_Blocked_for_local -j DROP
iptables -A FORWARD -d 172.16.0.0/12 -p udp -m udp --dport 11211 -m comment --comment memcached_attacks_block_for_local -j DROP
See also my articles:
- How to disable auto-update on Ubuntu 18
- Changing TX and RX network interface buffers
- Configuring the Network in Linux
- Installing Accel-ppp on Ubuntu 18
- Setting up BGP in Quagga
- Test iops using fio