How did I make a request to Hetzner to replace the disk in the raid

Already quite often I wrote in Hetzner to replace disks in a raid and in this article I will describe one of the cases.

And so, one morning, after a disk dropped out of a raid, mdadm sent me a message by email.

I looked at the status of the raids:

cat /proc/mdstat

Information was displayed (as you can see, the first disc fell out):

Personalities : [raid1] [raid6] [raid5] [raid4] 
md2 : active raid6 sdc3[5] sdd3[3] sdb3[1]
      208218112 blocks super 1.0 level 6, 512k chunk, algorithm 2 [4/3] [_UUU]
      
md1 : active raid1 sdc2[5] sdd2[3] sdb2[1] sda2[4](F)
      524224 blocks super 1.0 [4/3] [_UUU]
      
md0 : active raid1 sdc1[5] sdd1[3] sdb1[1]
      12582784 blocks super 1.0 [4/3] [_UUU]

I looked at what drives are present in the system:

fdisk -l | grep '/dev/sd'

In my case, there were sdb, sdc, sdd, and sda was missing:

Disk /dev/sdc: 120.0 GB, 120034123776 bytes
/dev/sdc1               1        1567    12582912+  fd  Linux raid autodetect
/dev/sdc2            1567        1633      524288+  fd  Linux raid autodetect
/dev/sdc3            1633       14594   104109528+  fd  Linux raid autodetect
Disk /dev/sdd: 1500.3 GB, 1500301910016 bytes
/dev/sdd1               1        1567    12582912+  fd  Linux raid autodetect
/dev/sdd2            1567        1633      524288+  fd  Linux raid autodetect
/dev/sdd3            1633       14594   104109528+  fd  Linux raid autodetect
Disk /dev/sdb: 1500.3 GB, 1500301910016 bytes
/dev/sdb1               1        1567    12582912+  fd  Linux raid autodetect
/dev/sdb2            1567        1633      524288+  fd  Linux raid autodetect
/dev/sdb3            1633       14594   104109528+  fd  Linux raid autodetect

I was convinced that the system does not see the disk:

smartctl -x /dev/sda
Smartctl open device: /dev/sda failed: No such device

To make a request for a replacement disk, you must provide the serial number of the non-working disk, if the disk is not visible to the system, then you must report the serial number of all working disks.
I created the application through the site robot.your-server.de

Let’s look at the serial numbers of the working disks:

smartctl -x /dev/sdb
smartctl -x /dev/sdc
smartctl -x /dev/sdd

I have the following information:

...
=== START OF INFORMATION SECTION ===
Model Family:     SandForce Driven SSDs
Device Model:     Corsair CSSD-F120GB2
Serial Number:    10446526320009980370
LU WWN Device Id: 5 000000 009980370
Firmware Version: 1.1
User Capacity:    120,034,123,776 bytes [120 GB]
Sector Size:      512 bytes logical/physical
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   8
ATA Standard is:  ATA-8-ACS revision 6
Local Time is:    Tue Nov 20 21:40:16 2018 EET
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
AAM feature is:   Unavailable
APM feature is:   Unavailable
Rd look-ahead is: Enabled
Write cache is:   Enabled
ATA Security is:  Disabled, NOT FROZEN [SEC1]
...

In my cases, Hetzner was replacing drives for about half an hour, free of charge on a used SSD, about five years old.

After the disk is replaced, let’s see if the system sees it and split it in the same way as the one installed:

fdisk -l | grep '/dev/sd'
sfdisk -d /dev/sdc | sfdisk --force /dev/sda

Next, you need to add a disk to your raids and wait for synchronization:

mdadm /dev/md0 -a /dev/sda1
cat /proc/mdstat
mdadm /dev/md1 -a /dev/sda2
cat /proc/mdstat
mdadm /dev/md2 -a /dev/sda3
cat /proc/mdstat

Now it remains to install GRUB2:

grub-install /dev/sda
update-grub

Or GRUB1 (hd0 is /dev/sda, hd0,1 – /dev/sda2):

cat /boot/grub/device.map
grub
device (hd0) /dev/sda
root (hd0,1)
setup (hd0)
quit

I used GRUB1, so I installed it:

grub
Probing devices to guess BIOS drives. This may take a long time.
    GNU GRUB  version 0.97  (640K lower / 3072K upper memory)
 [ Minimal BASH-like line editing is supported.  For the first word, TAB
   lists possible command completions.  Anywhere else TAB lists the possible
   completions of a device/filename.]
grub> device (hd0) /dev/sda
device (hd0) /dev/sda
grub> root (hd0,1)
root (hd0,1)
 Filesystem type is ext2fs, partition type 0xfd
grub> setup (hd0)
setup (hd0)
 Checking if "/boot/grub/stage1" exists... yes
 Checking if "/boot/grub/stage2" exists... yes
 Checking if "/boot/grub/e2fs_stage1_5" exists... yes
 Running "embed /boot/grub/e2fs_stage1_5 (hd0)"...  27 sectors are embedded.
succeeded
 Running "install /boot/grub/stage1 (hd0) (hd0)1+27 p (hd0,1)/boot/grub/stage2 /boot/grub/grub.conf"... succeeded
Done.
grub> quit

Done.

PS. I will give an example of a backup MBR/GPT:

sfdisk --dump /dev/sda > sda_parttable_mbr.bak
sfdisk /dev/sda < sda_parttable_mbr.bak
sgdisk --backup=sda_parttable_gpt.bak /dev/sda
sgdisk --load-backup=sda_parttable_gpt.bak /dev/sda

See also my article:
How to fix the problem with mdadm disks

Join the Conversation

1 Comment

Leave a Reply

  1. Hi, I am on centos 7.8, none of the grub commands exist for me, only grub2…
    For instance I don’t have grub-install command/binary, but I did try grub2-install /dev/sdb in which case I got:

    grub2-install /dev/sdb
    Installing for i386-pc platform.
    grub2-install: warning: Couldn’t find physical volume `(null)’. Some modules may be missing from core image..
    grub2-install: warning: Couldn’t find physical volume `(null)’. Some modules may be missing from core image..
    Installation finished. No error reported.

    Do you happen to know the steps for grub2? Thanks.

Discover more from IT Blog

Subscribe now to keep reading and get access to the full archive.

Continue reading