I will give an example of simple monitoring of a mdadm software raid in Zabbix.Continue reading “Monitoring mdadm in Zabbix”
On the test, when installing Ubuntu Server 14.04 LTS, we will create software RAID1.
I note that when creating RAID will be used mdadm automatically.
I will connect two identical disks to the server (similarly, you can try to create RAID on a virtual machine, for example, created in VirtualBox).
Already quite often I wrote in Hetzner to replace disks in a raid and in this article I will describe one of the cases.
And so, one morning, after a disk dropped out of a raid, mdadm sent me a message by email.
Continue reading “How did I make a request to Hetzner to replace the disk in the raid”
Here is an example of migrating a running Ubuntu system to a software RAID1.
In the process, you will need to perform two reboots.
The first step is to switch to the root user if not yet:
Let’s see a list of disks and partitions:
fdisk -l fdisk -l | grep '/dev/sd' lsblk -o NAME,UUID
Suppose that the system uses one disk, for example /dev/sda and has one main partition, /dev/sda1.
For the test, I installed a clean Ubuntu Server 18.04, the disk was parted by default, swap was the file on the same partition.
To create a raid, we connect another disk of the same size, it will be called /dev/sdb.
I recommend reading my article Description of RAID types.
You can install mdadm in Ubuntu using the command:Continue reading “mdadm – utility for managing software RAID arrays”
There was once a case, one disk dropped out of the raid and when the server was loaded in the logs a message was displayed:
md: kicking non-fresh sda1 from array
Since the disk was not in the raid, the data on it was outdated.
First of all, we’ll check the disk for errors, for example, as I wrote in the article below, and try to determine why he was excluded from the raid.
In my case, the disk was completely working, so looking at the information about the raid:
cat /proc/mdstat mdadm --detail /dev/md0
Returned it back to the raid:
mdadm /dev/md0 -a /dev/sda1
After some time, the data was synchronized to disk and the error did not appear any more.
I received three email messages from one of the servers on Hetzner with information about raids md0, md1, md2:
DegradedArray event on /dev/md/0:example.com
This is an automatically generated mail message from mdadm
running on example.com
A DegradedArray event had been detected on md device /dev/md/0.
Faithfully yours, etc.
P.S. The /proc/mdstat file currently contains the following:
Personalities : [raid6] [raid5] [raid4] [raid1]
md2 : active raid6 sdb3 sdd3
208218112 blocks super 1.0 level 6, 512k chunk, algorithm 2 [4/2] [_U_U]
md1 : active raid1 sdb2 sdd2
524224 blocks super 1.0 [4/2] [_U_U]
md0 : active raid1 sdb1 sdd1
12582784 blocks super 1.0 [4/2] [_U_U]
I looked at the information about RAID and disks:
cat /proc/mdstat cat /proc/partitions mdadm --detail /dev/md0 mdadm --detail /dev/md1 mdadm --detail /dev/md2 fdisk -l | grep '/dev/sd' fdisk -l | less
I was going to send a ticket to the tech support and plan to replace the dropped SSD disks.
SMART recorded information about the dropped discs in the files, there was also their serial number:
smartctl -x /dev/sda > sda.log smartctl -x /dev/sdc > sdc.log
Remove disks from the raid if you can:
mdadm /dev/md0 -r /dev/sda1 mdadm /dev/md1 -r /dev/sda2 mdadm /dev/md2 -r /dev/sda3 mdadm /dev/md0 -r /dev/sdc1 mdadm /dev/md1 -r /dev/sdc2 mdadm /dev/md2 -r /dev/sdc3
If any partition of the disk is displayed as working, and the disk needs to be extracted, then first mark the partition not working and then delete, for example, if /dev/sda1, /dev/sda2 are dropped, and /dev/sda3 works:
mdadm /dev/md0 -f /dev/sda3 mdadm /dev/md0 -r /dev/sda3
In my case, having looked at the information about the dropped discs, I found that they are whole and working, even better than active ones.
I looked at the disk partitions:
fdisk /dev/sda p q fdisk /dev/sdc p q
They were marked the same way as before:
Disk /dev/sda: 120.0 GB, 120034123776 bytes
255 heads, 63 sectors/track, 14593 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00015e3f
Device Boot Start End Blocks Id System
/dev/sda1 1 1567 12582912+ fd Linux raid autodetect
/dev/sda2 1567 1633 524288+ fd Linux raid autodetect
/dev/sda3 1633 14594 104109528+ fd Linux raid autodetect
Therefore, after waiting for the synchronization of each returned these discs back to the raid:
mdadm /dev/md0 -a /dev/sda1 mdadm /dev/md1 -a /dev/sda2 mdadm /dev/md2 -a /dev/sda3 mdadm /dev/md0 -a /dev/sdc1 mdadm /dev/md1 -a /dev/sdc2 mdadm /dev/md2 -a /dev/sdc3
At the end, the command cat /proc/mdstat was already displayed with [UUUU].
If the disks are replaced with new ones, then they need to be broken in the same way as the ones installed.
An example of partitioning the disk /dev/sdb is similar to /dev/sda with MBR:
sfdisk -d /dev/sda | sfdisk --force /dev/sdb
Example of partitioning /dev/sdb with GPT and assigning a random UUID disk:
sgdisk -R /dev/sdb /dev/sda sgdisk -G /dev/sdb
Also on the newly installed disk you need to install the bootloader:
grub-install --version grub-install /dev/sdb update-grub
Either through the menu grub (hd0 is /dev/sda, hd0,1 – /dev/sda2):
cat /boot/grub/device.map grub device (hd0) /dev/sda root (hd0,1) setup (hd0) quit
If the grub installation is performed from the rescue disk, you need to look at the partition list and mount it, for example if RAID is not used:
ls /dev/[hsv]d[a-z]*[0-9]* mount /dev/sda3 /mnt
If you are using software RAID:
ls /dev/md* mount /dev/md2 /mnt
ls /dev/mapper/* mount /dev/mapper/vg0-root /mnt
And execute chroot:
chroot-prepare /mnt chroot /mnt
After mounting, you can restore GRUB as I wrote above.
See also my other articles:
How did I make a request to Hetzner to replace the disk in the raid
The solution to the error “md: kicking non-fresh sda1 from array”
The solution to the warning “mismatch_cnt is not 0 on /dev/md*”
mdadm – utility for managing software RAID arrays
Description of RAID types
Diagnostics HDD using smartmontools
Recovering GRUB Linux