Reducing the number of devices in a RAID-5

(This blog entry has already been published in French on Nibbles' microblog)

Since 2006, linux RAID allows to grow a RAID-5 volume by adding new devices, and thus allowing to grow the filesystem underneath in order to have more free space. The opposite operation was missing, that is shrinking a RAID-5 volume by removing devices, assuming the underneath filesystem has already been reduced (otherwise data would be lost).
This summer, Neil Brown developed this feature among others, as he announced on his blog.

To be able to use it, you will need:

• linux kernel >= 2.6.31
• mdadm >= 3.1, not yet announced but available in Neil Brown's personal git repository in branch devel-3.1

I suggest you to try this new feature with LVM volumes.

We create 4 logical volumes and use them to create a RAID-5 volume. You can see this logical volumes as real disks.

# lvcreate -n test1 -L 4M vg1

# lvcreate -n test2 -L 4M vg1

# lvcreate -n test3 -L 4M vg1

# lvcreate -n test4 -L 4M vg1

# mdadm -C /dev/md9 --level=5 --raid-devices=4 /dev/vg1/test1 /dev/vg1/test2 /dev/vg1/test3 /dev/vg1/test4

# mdadm -w /dev/md9

# cat /proc/mdstat
[...]
md9 : active raid5 dm-9[0] dm-12[3] dm-11[2] dm-10[1]
12096 blocks level 5, 64k chunk, algorithm 2 [4/4] [UUUU]
[...]

Then, set up a filesystem on that RAID-5 volume.

# mkfs.ext3 /dev/md9

Now imagine the filesystem is used normally, but one day we need to remove one of these logical volumes (think of a disk), and we don't want the RAID-5 to be degraded. If the filesystem has enough free space, we can reduce it and benefit from this new feature.

Before starting, check if linux kernel >= 2.6.31.

$ uname -a
Linux x5452.stalkr.net 2.6.31.2 #1 SMP Tue Oct 6 15:28:59 CEST 2009 i686 GNU/Linux

Reduce the filesystem on the RAID-5.

# resize2fs -M /dev/md9

The mdadm version featuring RAID-5 volume reduction hasn't been announced yet, so we first get the branch devel-3.1 of mdadm from Neil Brown's personal git repository.

$ git clone git://neil.brown.name/mdadm /tmp/mdadm
Initialized empty Git repository in /tmp/mdadm/.git/
remote: Counting objects: 5943, done.
remote: Compressing objects: 100% (3525/3525), done.
remote: Total 5943 (delta 4540), reused 3118 (delta 2412)
Receiving objects: 100% (5943/5943), 1.71 MiB | 95 KiB/s, done.
Resolving deltas: 100% (4540/4540), done.

$ cd /tmp/mdadm

$ git branch --track devel-3.1 origin/devel-3.1
Branch devel-3.1 set up to track remote branch refs/remotes/origin/devel-3.1.

$ git checkout devel-3.1
Switched to branch "devel-3.1"

$ make
[...]

We see the new option --array-size:

# ./mdadm --grow --help
Usage: mdadm --grow device options
[...]
--array-size=  -Z   : Change visible size of array.  This does not change
  any data on the device, and is not stable across restarts.

Size of the array with 4 logical volumes in RAID-5 : (4-1)*(device size) = 3*4M = 12M

# ./mdadm -Q -D /dev/md9
[...]
Array Size : 12096 (11.81 MiB 12.39 MB)
[...]

Reducing size of the array to a size corresponding to 3 logical volumes in RAID-5 : (3-1)*(device size) = 2*4M = 8M
According to the manual, this reduce is only visual (disappears after reboot), the time to trigger the effective reduction.

# ./mdadm /dev/md9 --grow --array-size=8064

# ./mdadm -Q -D /dev/md9
[...]
Array Size : 8064 (7.88 MiB 8.26 MB)
[...]

Now we trigger the effective reduction of the RAID-5 by reducing the number of devices.
Just like in an extend operation, mdadm needs a file to backup the critical section.

# ./mdadm /dev/md9 --grow --raid-devices=3 --backup-file=/tmp/backup
mdadm: Need to backup 384K of critical section..

The RAID-5 volume is now on 3 logical volumes + 1 logical volume for hot-spare.

# cat /proc/mdstat
[...]
md9 : active raid5 dm-9[0] dm-12[3](S) dm-11[2] dm-10[1]
8064 blocks level 5, 64k chunk, algorithm 2 [3/3] [UUU]
[...]

Win! We can now remove the hot-spare if needed elsewhere.

And also, since we reduced the filesystem to its minimum size, we resize the filesystem to take all the available space.

# resize2fs /dev/md9

Big thanks goes to Neil Brown and the other contributors of the Linux RAID project for this very useful feature.