IT Blog

How to fix the problem with mdadm disks

I received three email messages from one of the servers on Hetzner with information about raids md0, md1, md2:

DegradedArray event on /dev/md/0:example.com
This is an automatically generated mail message from mdadm
running on example.com
A DegradedArray event had been detected on md device /dev/md/0.
Faithfully yours, etc.
P.S. The /proc/mdstat file currently contains the following:
Personalities : [raid6] [raid5] [raid4] [raid1]
md2 : active raid6 sdb3[1] sdd3[3]
208218112 blocks super 1.0 level 6, 512k chunk, algorithm 2 [4/2] [_U_U]
md1 : active raid1 sdb2[1] sdd2[3]
524224 blocks super 1.0 [4/2] [_U_U]
md0 : active raid1 sdb1[1] sdd1[3]
12582784 blocks super 1.0 [4/2] [_U_U]
unused devices:

I looked at the information about RAID and disks:

cat /proc/mdstat
cat /proc/partitions
mdadm --detail /dev/md0
mdadm --detail /dev/md1
mdadm --detail /dev/md2
fdisk -l | grep '/dev/sd'
fdisk -l | less

I was going to send a ticket to the tech support and plan to replace the dropped SSD disks.
SMART recorded information about the dropped discs in the files, there was also their serial number:

smartctl -x /dev/sda > sda.log
smartctl -x /dev/sdc > sdc.log

Remove disks from the raid if you can:

mdadm /dev/md0 -r /dev/sda1
mdadm /dev/md1 -r /dev/sda2
mdadm /dev/md2 -r /dev/sda3

mdadm /dev/md0 -r /dev/sdc1
mdadm /dev/md1 -r /dev/sdc2
mdadm /dev/md2 -r /dev/sdc3

If any partition of the disk is displayed as working, and the disk needs to be extracted, then first mark the partition not working and then delete, for example, if /dev/sda1, /dev/sda2 are dropped, and /dev/sda3 works:

mdadm /dev/md0 -f /dev/sda3
mdadm /dev/md0 -r /dev/sda3

In my case, having looked at the information about the dropped discs, I found that they are whole and working, even better than active ones.

I looked at the disk partitions:

fdisk /dev/sda
p
q
fdisk /dev/sdc
p
q

They were marked the same way as before:

Disk /dev/sda: 120.0 GB, 120034123776 bytes
255 heads, 63 sectors/track, 14593 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00015e3f
Device Boot Start End Blocks Id System
/dev/sda1 1 1567 12582912+ fd Linux raid autodetect
/dev/sda2 1567 1633 524288+ fd Linux raid autodetect
/dev/sda3 1633 14594 104109528+ fd Linux raid autodetect

Therefore, after waiting for the synchronization of each returned these discs back to the raid:

mdadm /dev/md0 -a /dev/sda1
mdadm /dev/md1 -a /dev/sda2
mdadm /dev/md2 -a /dev/sda3

mdadm /dev/md0 -a /dev/sdc1
mdadm /dev/md1 -a /dev/sdc2
mdadm /dev/md2 -a /dev/sdc3

At the end, the command cat /proc/mdstat was already displayed with [UUUU].

If the disks are replaced with new ones, then they need to be broken in the same way as the ones installed.
An example of partitioning the disk /dev/sdb is similar to /dev/sda with MBR:

sfdisk -d /dev/sda | sfdisk --force /dev/sdb

Example of partitioning /dev/sdb with GPT and assigning a random UUID disk:

sgdisk -R /dev/sdb /dev/sda
sgdisk -G /dev/sdb

Also on the newly installed disk you need to install the bootloader:

grub-install --version
grub-install /dev/sdb
update-grub

Either through the menu grub (hd0 is /dev/sda, hd0,1 – /dev/sda2):

cat /boot/grub/device.map
grub
device (hd0) /dev/sda
root (hd0,1)
setup (hd0)
quit

If the grub installation is performed from the rescue disk, you need to look at the partition list and mount it, for example if RAID is not used:

ls /dev/[hsv]d[a-z]*[0-9]*
mount /dev/sda3 /mnt

If you are using software RAID:

ls /dev/md*
mount /dev/md2 /mnt

Either LVM:

ls /dev/mapper/*
mount /dev/mapper/vg0-root /mnt

And execute chroot:

chroot-prepare /mnt
chroot /mnt

After mounting, you can restore GRUB as I wrote above.

How to add a Windows user from the command line

It took one day to add a user to Windows 10 from the command line, because nothing happened when the add button was pressed from the control panel.

The first step is to start the command prompt as administrator, for this, in the start menu, type “cmd” or simply find the shortcut “Command line” and click on it with the right mouse button select “Run as administrator “.

At the command prompt, execute the add user command (where NAME is the user name):

net user NAME /add

Finish, the new user can already be seen in the “Control Panel\User Accounts\User Accounts\Account Management“

How to renew SSL certificate on iRedMail

I will give an example of updating a self-signed certificate on Ubuntu Server with iRedMail installed.

Switch to the root user:

Mount NTFS partitions on Linux

After connecting the disk to the server, let’s see a list of all the disks and find the name of the desired one:

sudo fdisk -l

I’ll give an example of mounting NTFS partition of a disk in Ubuntu (since I had a disk partitioned into two partitions, drive C and D, then they were found in the system as /dev/sdb1 and /dev/sdb2, both mounted to the created directories):

sudo mkdir /newhdd1
sudo mount -t ntfs /dev/sdb1 /newhdd1
sudo mkdir /newhdd2
sudo mount -t ntfs /dev/sdb2 /newhdd2

Since before this disk was used in the Windows system, I had a mount error:

The disk contains an unclean file system (0, 0).
Metadata kept in Windows cache, refused to mount.
Failed to mount ‘/dev/sdb1’: The operation is not allowed
The NTFS partition is in an unsafe state. Please resume and shutdown
Windows fully (no hibernation or fast restarting), or mount the volume
read-only with the ‘ro’ mount option.

In this case, you can mount the partition in read-only mode:

sudo mount -t ntfs -o ro /dev/sdb1 /newhdd1

Either fix the partitions with the command:

sudo ntfsfix /dev/sdb1
sudo ntfsfix /dev/sdb2

And after that, mount with full access:

sudo mount -t ntfs /dev/sdb1 /newhdd1
sudo mount -t ntfs /dev/sdb2 /newhdd2

You can unmount it like this:

sudo umount -t ntfs /dev/sdb1 /newhdd1
sudo umount -t ntfs /dev/sdb2 /newhdd2

Managing disk partitions in Ubuntu using fdisk

Let’s review the existing disks:

Adding a disk to LVM

Suppose we have already configured LVM, for example, as I described in this article – Setting up and using LVM

Switch to the root user:

sudo -i

If there is no hot-swap drive, turn off the server, connect a new disk, turn on the server and look at the name of the new disk (in my case it’s /dev/sdd):

fdisk -l

Let’s see the existing groups and how much space is left:

vgdisplay

Let’s see a list of physical volumes:

pvdisplay

Let’s start marking a new disk:

fdisk /dev/sdd
n
p
1
Enter
Enter
t
8e
w

Now create a physical volume:

pvcreate /dev/sdd1

Let’s see a list of logical volumes:

lvdisplay

We extend it by adding a new partition (where ixnfo is a volume group):

vgextend ixnfo /dev/sdd1

See the list of physical volumes as follows:

pvscan

Let’s look at the path of the logical volume (in my case /dev/ixnfo/temp) and add a new section:

lvextend /dev/ixnfo/temp /dev/sdd1

Let’s see the size of the mounted logical volume:

df -h

So the size did not change, we’ll fix it with the command:

resize2fs /dev/ixnfo/temp

Done.

Setting up and using LVM

LVM (Logical Volume Management) allows you to compile multiple disks and areas from disks into one logical volume and then split again as you like.

PV (Physical Volume) — partition or whole disk
VG (Volume Group) — a single disk assembled from physical volumes
LV (Logical Volume)

Switch to the root user:

sudo -i

Install LVM if it is not already installed (Ubuntu/Debian):

apt-get install lvm2

Let’s look at the information about the disks:

fdisk -l

On the test I have /dev/sda with the system and not marked /dev/sdb.

Let’s make the physical partition all /dev/sdb without partitioning:

pvcreate /dev/sdb

To view the list of physical volumes, use the command:

pvdisplay

Create a volume group named ixnfo:

vgcreate ixnfo /dev/sdb

If necessary, delete as follows:

vgremove ixnfo

Example of viewing existing groups and how much space is left:

vgdisplay

For the test, create a logical volume “temp” of 100 megabytes:

lvcreate -L100 -n temp ixnfo

To view the list of logical volumes, use the command:

lvdisplay

Let’s format it:

mkfs.ext4 -L temp /dev/ixnfo/temp

Create a folder, mount the created volume:

mkdir /mnt/temp
mount /dev/ixnfo/temp /mnt/temp

You can unmount it like this:

umount /mnt/temp/

Installing Zurmo on Linux

On the test I will install Zurmo in Ubuntu Server 16.04 and PHP 7.

Access Control Apache2

Access control Apache2 version 2.4 slightly different from 2.2, for example, to allow access to all, in version 2.4 you need to specify:

Require all granted

Allow access to the specified IP addresses:

Require local
Require ip 192.168.56.1 192.168.22.10

Allow all but the specified IP:

Require all granted
Require not ip 192.168.56.1

Allow the specified host:

Require host example.com

Prohibit all:

Require all denied

And in the version of Apache2 2.2, permit everyone access like this:

Order allow,deny
Allow from all

Prohibit all:

Order deny,allow
Deny from all

Allow access by specified IP:

Order allow,deny
Allow from 192.168.56.1 192.168.22.10

Allow the specified host:

Order Deny,Allow
Deny from all
Allow from example.com

After the changes in the Apache2 configuration, a reboot is required (if the changes were in the .htaccess file, then the reboot is not required):

sudo service apache2 restart

See also:
Using .htaccess

Change connect_timeout in MySQL

connect_timeout – the number of seconds that the mysql server waits for the connection package before terminating the connection.

Connect to MySQL and see the current value:

mysql -u USER -p
show variables like "connect_timeout";
quit;

The value of connect_timeout can be specified in the file /etc/mysql/my.cnf, for example:

[mysqld]
connect_timeout=10

In real time, you can change by executing the SQL query (after restarting MySQL it will be reset to the standard or specified in the configuration file):

SET GLOBAL connect_timeout=10;

The standard value is 10, the minimum value is 2, the maximum is 31536000.