Logging activity using IPTables

Using iptables, you can write network activity to the log file, that is, which data is sent to and from where it comes from.
Example command:

sudo iptables -t filter -A FORWARD -s 192.168.1.0/24 -m tcp -p tcp --dport 80 -j LOG --log-prefix "iptables: "

That the information was written not in rsyslog a file, and separately, we will create a file:

sudo nano /etc/rsyslog.d/10-iptables.conf

And add the following to it:

:msg, contains, "iptables: " -/var/log/iptables.log
& ~

To apply the changes, restart rsyslog:

sudo /etc/init.d/rsyslog reload

Done, the network activity specified in the first rule will be written to the file /var/log/iptables.log.

It is also desirable to configure logrotate to remove old logs, to save disk space.

See also:
Configuring logrotate for log rotation
Configuring IPTables

Monitoring of Apache2 parameters in Zabbix

Install the necessary components:

sudo apt-get install curl

Activate the module information about apache2 (usually it is activated initially):

sudo a2enmod info

Open the configuration file of the module and specify the IP address of the zabbix server to allow it to view information about apache2 (if apache2 is on the local machine, then access is usually allowed by specifying local or 127.0.0.1):

sudo nano /etc/apache2/mods-enabled/status.conf

In the nano editor, CTRL+X is used to exit and y/n to save or discard changes. Example of specifying IP:

<Location /server-status>
    SetHandler server-status
    Require local
    Require ip 192.168.1.5
</Location>

Restart the web server to apply the changes:

sudo service apache2 restart

Now the information about apache2 is available through the browser by the link http://HOST/server-status

Let’s move on to Zabbix. Create a folder for scripts if it does not exist:

mkdir /etc/zabbix/scripts/
chown root:zabbix -R /etc/zabbix/scripts/
chmod 750 /etc/zabbix/scripts/

In the zabbix server configuration file, we specify the path to this folder:

sudo nano /etc/zabbix/zabbix_server.conf
ExternalScripts=PATH

Now I will give the content of the script:

#!/bin/bash
if [[ -z "$1" || -z "$2" || -z "$3" ]]; then
  exit 1
fi
##### PARAMETERS #####
RESERVED="$1"
METRIC="$2"
URL="$3"
STATSURL="${URL}?auto"
#
CACHE_TTL="55"
CACHE_FILE="/tmp/zabbix.apache2.`echo ${URL} | md5sum | cut -d" " -f1`.cache"
EXEC_TIMEOUT="2"
NOW_TIME=`date '+%s'`
##### RUN #####
if [ -s "${CACHE_FILE}" ]; then
  CACHE_TIME=`stat -c"%Y" "${CACHE_FILE}"`
else
  CACHE_TIME=0
fi
DELTA_TIME=$((${NOW_TIME} - ${CACHE_TIME}))
#
if [ ${DELTA_TIME} -lt ${EXEC_TIMEOUT} ]; then
  sleep $((${EXEC_TIMEOUT} - ${DELTA_TIME}))
elif [ ${DELTA_TIME} -gt ${CACHE_TTL} ]; then
  echo "" >> "${CACHE_FILE}" # !!!
  DATACACHE=`curl -sS --insecure --max-time ${EXEC_TIMEOUT} "${STATSURL}" 2>&1`
  echo "${DATACACHE}" > "${CACHE_FILE}" # !!!
  echo "URL=${URL}"  >> "${CACHE_FILE}" # !!!
  chmod 640 "${CACHE_FILE}"
fi
#
if [ "${METRIC}" = "accesses" ]; then
  cat "${CACHE_FILE}" | grep -i "accesses" | cut -d':' -f2 | head -n1
fi
if [ "${METRIC}" = "kbytes" ]; then
  cat "${CACHE_FILE}" | grep -i "kbytes" | cut -d':' -f2 | head -n1
fi
if [ "${METRIC}" = "cpuload" ]; then
  cat "${CACHE_FILE}" | grep -i "cpuload" | cut -d':' -f2 | head -n1
fi
if [ "${METRIC}" = "uptime" ]; then
  cat "${CACHE_FILE}" | grep -i "uptime" | cut -d':' -f2 | head -n1
fi
if [ "${METRIC}" = "avgreq" ]; then
  cat "${CACHE_FILE}" | grep -i "ReqPerSec" | cut -d':' -f2 | head -n1
fi
if [ "${METRIC}" = "avgreqbytes" ]; then
  cat "${CACHE_FILE}" | grep -i "BytesPerReq" | cut -d':' -f2 | head -n1
fi
if [ "${METRIC}" = "avgbytes" ]; then
  cat "${CACHE_FILE}" | grep -i "BytesPerSec" | cut -d':' -f2 | head -n1
fi
if [ "${METRIC}" = "busyworkers" ]; then
  cat "${CACHE_FILE}" | grep -i "BusyWorkers" | cut -d':' -f2 | head -n1
fi
if [ "${METRIC}" = "idleworkers" ]; then
  cat "${CACHE_FILE}" | grep -i "idleworkers" | cut -d':' -f2 | head -n1
fi
if [ "${METRIC}" = "totalslots" ]; then
  cat "${CACHE_FILE}" | grep -i "Scoreboard" | cut -d':' -f2 | sed -e 's/ //g' | wc -c | awk '{print $1-1}'
fi
#
exit 0

Let’s make the script file executable:

chown root:zabbix /etc/zabbix/scripts/apache2-status.sh
chmod 550 /etc/zabbix/scripts/apache2-status.sh

Script validation example:

sudo -u zabbix /etc/zabbix/scripts/apache2-status.sh none accesses http://HOST/server-status

Open the zabbix agent configuration file in the editor:

sudo nano /etc/zabbix/zabbix_agentd.conf

And we will specify the following parameters:

UserParameter=apache2[*],/etc/zabbix/scripts/apache2-status.sh "none" "$1" "$2"

Some commands allow you to see the number of apache2 processes and connections to port 80, for this you do not need a script, for example, you can specify:

UserParameter=apache2.count_processes,ps aux | grep apache | wc -l
UserParameter=connections_on_80_port,netstat -na | grep :80 | wc -l

Restart the zabbix agent to apply the changes:

sudo /etc/init.d/zabbix-agent restart

Let’s check:

zabbix_get -s 127.0.0.1 -k "apache2[accesses,http://HOST/server-status]"

Now we will create a template and add data elements, an example of created data items:

apache2[KEY,http://HOST/server-status]

Exported ready template – apache2-status

We also need to add a macro to the monitored host:

Macro: {$APACHE_STATS_URL}
Value: http://HOST/server-status

Done.

The number of apache2 processes can be obtained from the Zabbix agent by creating on the Zabbix server a data element with a key:

proc.num[apache2]

Executing a SQL query from a script in Linux

It was necessary recently to write a script that executes a sql query into the MySQL database.
Example content:

# Description, here I wrote for others that the script is added to the crontab, so that it is not moved
mysql -u USER -pPASSWORD -h 127.0.0.1 -e "UPDATE nika_system.abon SET otkl=0 WHERE depozit > '10' AND (otkl='-1' OR otkl='-2');";

In order for the script to run automatically, open the crontab file in any text editor (in the nano editor CTRL+X for the output and y/n for saving or canceling the changes):

sudo nano /etc/crontab

Add the following line to it:

0 9 * * * root /home/nika/scripts/reset_credit_nika.sh > /dev/null 2>&1

Now every day at 9 am the script will be executed.

Here is an example of daily automatic output of data from a sql table into a text file:

#!/bin/bash
mysql -u USER -pPASSWORD -h 192.168.1.1 -s -N -e "SELECT id FROM nika_system.abon WHERE tarif=109;";

In /etc/crontab we add:

0 8 * * * root /scripts/freektb.sh > /srv/samba/dir/mirazh/$(date +%Y-%m-%d).txt

For security reasons, it’s better not to specify the password in scripts, see my article – Connecting to MySQL from localhost without entering a password

Firmware Update D-Link DIR-815

For the test, I will update the firmware on the router D-Link DIR-815 revision A1.
The router has firmware 1.00 (Fri 06 Aug 2010), on official FTP found version 1.04, b03 (Wed May 15, 2013).

To update the firmware in the D-Link DIR-815 router, perform the following necessary steps:

1) Let’s see the revision on the label under the router and download the new firmware from the official FTP for it http://ftp.dlink.ru/pub/Router/DIR-815/Firmware/
Updating the firmware of the router for the wrong revision can lead to its failure.

2) Open the router settings by typing in the browser address http://192.168.0.1 (can be 192.168.1.1) and enter the default login — admin without password.

3) In the opened interface at the top, open the tab “Tools“, left in the menu select “Firmware“. On the opened page the current version of the firmware will be displayed, if it is older than downloaded, then click “Browse” and select the previously downloaded new firmware file, and then click “Upgrade” to start the update process.

We will wait for the update to be completed, usually about 5 minutes. After the termination the router itself will reboot.
Categorically, you can not turn off the power when the firmware is updated.

Script to check for an empty directory

Recently noticed on one of his backup scripts that he stopped working and just created an empty directory with the current date, so there was an idea to add to it a check on an empty directory with a notification sent to the email.

Example of a script for checking for an empty directory, taking into account hidden files:

if [ `ls -a /backups/dir/ | grep -v "^\.$" | grep -v "^\..$" | wc -l` = "0" ]; then echo "Empty dir"; else echo "Non empty dir"; fi

Example of checking the directory with the name of the current date and if it is empty – by sending an email:

if [ `ls -a /backups/`date +%Y-%m-%d`/ | grep -v "^\.$" | grep -v "^\..$" | wc -l` = "0" ]; then (echo "Subject:Empty dir"; echo "Empty dir";) | sendmail info@example.com; else echo "Non empty dir"; fi

How to catch broadcast storms on FoxGate switches

First of all, let’s look at the statistics of active traffic on ports:

show interface ethernet counter rate

Then we’ll look at the packet counters, especially pay attention to the BroadCast (pkts) column:

show interface ethernet counter packet

For a specific port, we will execute the command to view the statistics of the network interface several times:

show interface ethernet 1/25

And let’s pay attention to how the incoming and outgoing values of the broadcast packets change, if they do not change, then the broadcast packets do not go through this port, if the digit increases rapidly, then maybe there is a broadcast storm.

If necessary, we will enter the competing mode and set the limit of transmitted broadcast packets in kilobits (minimum value 1) for the required ports:

config
Interface Ethernet1/1
storm-control broadcast 50
Interface Ethernet1/2
storm-control broadcast 50
Interface Ethernet1/3
storm-control broadcast 50
...

Note that with a low broadcast bandwidth limit, DHCP broadcast requests from clients can also be blocked.

See also:
Block DHCP servers on FoxGate switches
Configuring the Foxgate S6224-S2 Switch

Updating the Linux kernel on Hetzner servers

It took one day to update the CentOS kernel on the server in Hetzner, which worked without rebooting for about 2 years.

I looked at various information about the system, the versions of the installed kernels and the running kernel:

lsb_release -a
uname -r
uname -a
cat /proc/version
sudo rpm -q kernel
ls /boot | grep vmlinuz

Updated kernel:

yum -y update

Rebooted the server using the link in cPanel https://example.com:2087/scripts/dialog?dialog=reboot

You can also use the command:

reboot

After rebooting the server did not start, I immediately made a request for technical support via the https://robot.your-server.de.
Technical support responded within two minutes, as they reported, the server did not respond to keystrokes, there was a black screen, after the physical shutdown and power-up, it successfully launched.
Here such there are cases of a reset, it is necessary to be ready to everything.
At the next reboots, hangup no longer occurred.

Dying gasp Alarm on Huawei OLT

Sometimes on Huawei OLT you can see Dying gasp Alarm (0x2e11a00b).

Since they often occur, most likely they can be seen by running the commands:

display alarm active alarmid 0x2e11a00b
display alarm history alarmid 0x2e11a00b

I will give an example:

ALARM 531377 FAULT MINOR 0x2e11a00b SERVICE QUALITY 2017-12-06 08:19:27+02:00
ALARM NAME : The dying-gasp of GPON ONTi (DGi) is generated
SRVEFF : SA
PARAMETERS : FrameID: 0, SlotID: 5, PortID: 6, ONT ID: 32, Equipment ID:
TL-GP110
DESCRIPTION : The ONT is power off
CAUSE : The power supply of the ONT is abnormal
ADVICE : Check the power supply of the ONT and ensure that the power
supply works in the normal state

These alarms occur when power is lost or the power cord is removed from the ONT, at this moment ONT manages to send the Dying gasp message to the OLT.
Dying gasp is also used on many other devices, for example, to correctly terminate the connection.

Opening and analysis of files created with NetFlow

Here is an example of viewing statistics:

cat ft-v05.2015-05-01.000759+0300 | flow-stat -f 10 -S 3 | less
cat ft-v05.2015-05-01.000759+0300 | flow-stat -n -p -w -f 26 -S 2 | less
flow-cat ft-v05.2017-12-07.170236+0200 | flow-print | less

You can see the flow-stat tips by typing:

man flow-stat

I will give an example of exporting data to another format (for example, in csv, which can then be opened in windows by any convenient program):

flow-cat ft-v05.2015-05-01.000759+0300 | flow-export -f 2 | less
flow-cat ft-v05.2015-05-01.000759+0300 | flow-export -f 2 > csv.csv

You can display the list with the specified source address with the following command:

flowdumper -se '"192.168.0.1" eq $srcip' ft-v05.2015-06-17.175701+0300

Export the list only with the specified source address to a file, for example, with the following command:

flowdumper -se '"192.168.0.1" eq $srcip' ft-v05.2015-06-17.175701+0300 > file.txt

Example of exporting from multiple files:

flowdumper -se '"192.168.0.1" eq $srcip' /backup/flows/acct/first/2016/2016-01/ft-v05.2015-06-17* > /backup/result.txt

See also:
Installing and using flow-tools