Software RAID on Linux with mdadm

Software RAID groups several disks into a single volume that survives the failure of one of them. On Linux, the standard tool is mdadm: it creates redundant arrays, monitors them and lets you replace a disk online. This guide shows how to choose a RAID level, create an array with a spare disk, make it persistent and repair it. For intermediate to advanced administrators, on Debian, Ubuntu or RHEL. Commands tested on AlmaLinux 10 (mdadm 4.4).

What you will learn

• Choose the right RAID level (0, 1, 5, 6, 10)
• Create an mdadm array with a spare disk
• Make it persistent at boot (mdadm.conf + initramfs)
• Monitor the array and replace a failed disk

Prerequisites

• A Linux server with sudo and the mdadm package installed
• At least 2 free disks (or partitions) of the same size
• Know how to identify disks and manage partitions

Certification scope

Software RAID is not explicitly listed in the published objectives of the LFCS (storage domain: LVM, file systems, remote storage, swap, automounting, performance) nor in those of the RHCSA EX200. It is an operations skill: knowing how to create an array, make it persistent and replace a failed disk remains the daily work of an administrator of physical servers.

Choose a RAID level

The level defines the trade-off between capacity, performance and fault tolerance. Choose it before creating the array, it is not easy to change afterwards.

Level	Min. disks	Fault tolerance	Usable capacity	For what
RAID 0	2	none	100%	Pure performance, disposable data
RAID 1	2	1 disk	50%	System disk, small critical volumes
RAID 5	3	1 disk	(n-1)/n	Good capacity / safety trade-off
RAID 6	4	2 disks	(n-2)/n	Large arrays (long rebuild)
RAID 10	4	1 per mirror	50%	Performance and redundancy (databases)

Swipe to see more

RAID 0 protects nothing

RAID 0 spreads data over several disks with no redundancy at all: losing a single disk destroys the whole array. Never use it for data that matters. And remember: RAID is not a backup, it protects against the hardware failure of a disk, not against a deletion, ransomware or logical corruption.

Create a RAID 5 array with a spare disk

The spare disk is idle until a failure, then takes over automatically. That is what turns redundancy into resilience with no intervention.

1. Create the array: 3 active disks (--raid-devices=3) plus 1 spare (--spare-devices=1).

   sudo mdadm --create /dev/md0 --level=5 \
     --raid-devices=3 /dev/sdb /dev/sdc /dev/sdd \
     --spare-devices=1 /dev/sde

   The command is interactive, which surprises the first time and blocks any script that ignores it. It asks a question about the write-intent bitmap (the typos are mdadm's own), then asks for confirmation:

   To optimalize recovery speed, it is recommended to enable write-indent
   bitmap, do you want to enable it now? [y/N]?
   mdadm: Defaulting to version 1.2 metadata
   mdadm: array /dev/md0 started.

   With no answer (in a script, or with standard input closed), mdadm gives up: mdadm: create aborted., exit code 1. To automate, answer in advance by passing the matching options:

   sudo mdadm --create /dev/md0 --level=5 \
     --raid-devices=3 /dev/sdb /dev/sdc /dev/sdd \
     --spare-devices=1 /dev/sde \
     --bitmap=internal --run

2. Follow the synchronisation in /proc/mdstat.

   cat /proc/mdstat

   md0 : active raid5 sde[4](S) sdd[3] sdc[1] sdb[0]
         241664 blocks super 1.2 level 5, 512k chunk, algorithm 2 [3/3] [UUU]

   The (S) marks the spare. The [UUU] notation gives the state of each active disk (U = up, _ = missing); here all 3 are healthy.

3. Check the details of the array.

   sudo mdadm --detail /dev/md0

   Raid Level : raid5
   Array Size : 241664 (236.00 MiB 247.46 MB)
        State : clean
   Active Devices : 3
   Working Devices : 4
   Spare Devices : 1

The /dev/md0 array is ready: 3 active disks and 1 spare (Working Devices : 4).

Format and mount

An mdadm array behaves like an ordinary disk. Format it and mount it:

sudo mkfs.xfs /dev/md0
sudo mkdir -p /mnt/raid
sudo mount /dev/md0 /mnt/raid

For a persistent mount, add it to /etc/fstab by UUID, with the nofail option so that a degraded array does not block the boot.

Make the array persistent

An array created in memory is not reassembled automatically after a reboot until it is declared in the configuration and included in the initramfs. It is the number one cause of a RAID that "disappears" after a reboot. The procedure differs by distribution.

Debian / Ubuntu

sudo mdadm --detail --scan | sudo tee -a /etc/mdadm/mdadm.conf
sudo update-initramfs -u

RHEL / AlmaLinux

On RHEL, the mdadm package is not installed on a minimal image (rpm -q mdadm answers package mdadm is not installed), and the /etc/mdadm.conf file does not exist either: you have to create both.

sudo dnf install -y mdadm
sudo mdadm --detail --scan | sudo tee -a /etc/mdadm.conf
sudo dracut --force

What dracut adds to the image is not your /etc/mdadm.conf: an inspection with lsinitrd finds no trace of it. It is the mdraid module, with the mdadm and mdmon binaries and their units, without which the initramfs would not know how to assemble an array. The step is still needed, but for that reason.

The generated line identifies the array by UUID, regardless of the order of the disks:

ARRAY /dev/md0 metadata=1.2 spares=1 UUID=76755a1f:40a789ed:c6b8d707:16526c13

Monitor the RAID

An unmonitored RAID is dangerous: if a disk failure goes unnoticed, a second disk that gives up means total loss. The mdmonitor service sends an alert by mail at the slightest degradation.

Declare the notification address at the top of mdadm.conf, then enable the service (confirmed enabled on AlmaLinux):

mdadm.conf

MAILADDR root@localhost

sudo systemctl enable --now mdmonitor

You can test the state at any time: mdadm --detail --test /dev/md0 returns 0 on a healthy array and 1 as soon as it is degraded, handy in a monitoring script.

Replace a failed disk

When a disk fails, the workflow is always the same: mark, remove, add. If a spare is present, the rebuild starts even before the disk is physically replaced.

1. Mark the disk as failed (simulated here with --fail).

   sudo mdadm /dev/md0 --fail /dev/sdb

   In /proc/mdstat, the disk switches to (F) and the spare takes over at once:

   md0 : active raid5 sde[4] sdd[3] sdc[1] sdb[0](F)
         241664 blocks super 1.2 level 5, 512k chunk, algorithm 2 [3/2] [_UU]
         [=>...................]  recovery =  9.0% (7424/241664) finish=1.9min speed=1856K/sec

   Read carefully: [3/2] and [_UU], not [UUU]. Out of three expected disks, only two are active, and the _ marks the missing one. The spare has been enlisted, but it still has to be rebuilt.

   mdadm --detail says it plainly:

        State : clean, degraded, recovering
   Active Devices : 2
   Working Devices : 3
   Failed Devices : 1
   Rebuild Status : 9% complete

2. Remove the failed disk from the array.

   sudo mdadm /dev/md0 --remove /dev/sdb

3. Add the replacement disk, which becomes the new spare.

   sudo mdadm /dev/md0 --add /dev/sdf

Follow the rebuild with watch -n1 cat /proc/mdstat: the recovery = XX% line gives the progress and the estimated end.

The rebuild window is the real moment of danger

It is the most misunderstood point of RAID, and the most costly. A spare disk does not make the failure painless: it shortens the window during which you are no longer protected, it does not remove it.

As long as /proc/mdstat shows [_UU] and a recovery line, the array is degraded. In RAID 5, that means a second failure destroys all the data. On this guide's demonstration volumes, a few hundred megabytes, the rebuild takes one second and you see nothing pass. On an array of several terabytes, it lasts hours, and it is precisely during those hours that disks from the same batch, of the same age and stressed by the rebuild, are the most likely to fail in turn.

Two practical consequences: RAID 6 is justified as soon as disks are large, because it tolerates the second failure; and replacing the failed disk is urgent, even when a spare has already taken over.

Redundancy is only restored when /proc/mdstat goes back to [3/3] [UUU] and mdadm --detail shows State : clean again without degraded.

RAID and LVM

To combine redundancy and flexibility, the recommended order is RAID below, LVM on top: mdadm creates /dev/md0, which becomes an LVM physical volume (LVM). You keep the resilience of RAID and the flexibility of logical volumes.

Alternative: LVM RAID (lvcreate --type raid5 ...) builds RAID directly into LVM, relying on the same kernel drivers as mdadm. Prefer mdadm for a pure RAID layer, proven (notably for the system disk); LVM RAID for a unified LVM stack.

Traps to avoid

RAID incidents rarely come from mdadm itself: they come from a wrong assumption about what RAID protects, or from a forgotten persistence step. The six traps below come up most often, and each one has already cost data to someone who thought they were protected.

• RAID 0 and critical data: no redundancy, to ban outside disposable caches.
• No monitoring: without mdmonitor + MAILADDR, a failure stays invisible until disaster strikes.
• Forgetting the initramfs: without update-initramfs/dracut after changing mdadm.conf, the array does not come back at boot.
• RAID 5/6 write hole: after a sudden stop, parity can become inconsistent. To close it, create the array with a PPL consistency policy (--consistency-policy=ppl, RAID 5) or a journal on SSD; a write-intent bitmap speeds up resync but does not close the hole. Beware, this bitmap is not set by default: mdadm --create offers it through an interactive question, and an N or a closed standard input leaves it out.
• Believing a spare makes the failure painless: during the whole rebuild, the array is degraded and a second failure is fatal in RAID 5.
• Confusing RAID and backup: see the warning above.

Troubleshooting

Two commands are enough for the diagnosis: cat /proc/mdstat gives the state of each disk at a glance, and mdadm --detail the state of the array and the progress of a rebuild. Always start from them before touching a disk.

Symptom	Likely cause	Fix
Array missing after reboot	mdadm.conf or initramfs not updated	Regenerate the config + update-initramfs/dracut
/proc/mdstat shows [U_U]	A failed disk	mdadm --detail then replace the disk
Slow rebuild	I/O load or kernel limit	cat /proc/mdstat (speed), wait, avoid writes
mdadm: cannot open /dev/sdX: Device or resource busy	Disk member of an array still active	mdadm --stop (or --fail then --remove), then mdadm --zero-superblock /dev/sdX before reuse
No failure alert	mdmonitor inactive	systemctl enable --now mdmonitor + MAILADDR

Swipe to see more

Put it into practice

Persistence is the classic weak point of an array: it works at creation, then disappears at the first reboot if mdadm.conf and the initramfs were forgotten. This lab has you build a RAID 1 with mdadm on a VM fitted with blank disks, and the validation checks that the array is still assembled after a reboot, not only that it was at creation.

Guided lab Practice

Learn how to do it

Build a software RAID 1 with mdadm

$ dsoxlab run l2-raid-mdadm

Environment: VM·Duration: ~30 min

How it is checked: the state of the system is checked after a reboot, not the command you typed.

First time? What DSOXLab is and how to install it

DSOXLab is the command line that prepares a disposable environment (an isolated shell or a VM), runs the lab scenario, then checks the real state of the system and computes a score. Nothing is installed or broken on your workstation.

It needs Python 3.11 or later, uv or pipx, and git. The installation guide details these prerequisites; on Windows, or on a Mac without Python, use the appliance instead.

This lab starts virtual machines: it also needs ansible-playbook, a hypervisor (KVM or Incus), Terraform and hardware virtualization (/dev/kvm), which inside a VirtualBox VM means nested virtualization. The first time, dsoxlab start l2-raid-mdadm checks all of it before creating the machines. Prepare your machine for vm labs.

Install the CLI and add the catalogue, once:

uv tool install dsoxlab   # or: pipx install dsoxlab
dsoxlab catalog add https://github.com/stephrobert/linux-dsoxlab-training

Before you run the lab, check:

• DSOXLab installed
• catalogue available
• compatible environment (isolated shell or VM)

To check, nothing is probed from this page: run dsoxlab doctor. Read the diagnosis

Run this lab, then validate your work:

dsoxlab run l2-raid-mdadm
dsoxlab check l2-raid-mdadm

See every lab: dsoxlab list-labs. Track your progress: dsoxlab progress. The full dsoxlab guide covers scoring and the pitfalls.

Key points

• Choose the level first: RAID 1 (mirror), RAID 5 (trade-off), RAID 6 (double parity), RAID 10 (perf + redundancy).
• A spare (--spare-devices) triggers an automatic rebuild at the first failure, but the array stays degraded ([_UU]) during the whole rebuild: a second failure there is fatal in RAID 5.
• /proc/mdstat and mdadm --detail are the two inspection tools: [UUU] = healthy, [_UU] = degraded, (S) = spare, (F) = failed.
• mdadm --create is interactive: two questions, and create aborted. with rc=1 if nobody answers. In a script, --bitmap=internal --run.
• Making the array persistent = mdadm.conf + initramfs (update-initramfs on Debian, dracut --force on RHEL). The initramfs embeds the mdraid module, not your configuration file.
• Enable mdmonitor: an unmonitored RAID is not resilient.
• RAID is not a backup; RAID 0 protects nothing.

Knowledge check

Check that the essentials of this guide are acquired. The questions only cover what has just been explained here.

Knowledge check

Check what you have learned with this interactive quiz

6 questions

6 min

70% to pass

How it works

• The timer starts when you click Start
• Multiple choice, true/false and short answer questions
• You can move between questions
• Detailed results are shown at the end

Start the quiz

Starts the quiz and the timer

×

See in the glossary →