After uppgrading to Ubuntu 12.04 LTS in my server grub started to complain and give errors about my raid array, everything still seems to be working but it is a bit unnerving to have grub giving errors.
Setting up grub-pc (1.99-21ubuntu3.7) ...
error: found two disks with the index 2 for RAID md0.
error: superfluous RAID member (3 found).
I'm getting a lot of these errors when grub is updated.
Facts are: I have three identical discs in a raid5 set up, on two of the discs there is one primary partition which is added to the array but the third disk is added without a primary partition. ie. mdam --manage /dev/md_d0 --add /dev/sdc
I'm guessing that this is the reason for grub complaining.
Since discovering this problem I have disabled the disc that is missing a primary partition, created a primary partition, verified that it looks the same with cfdisk /dev/xxx
and then Print Partition table
. And tried to add the new partition to the raid array and that is when I get the message that the partition is to small to be added to the raid array.
> sudo mdadm /dev/md_d0 --add /dev/sdc1
mdadm: /dev/sdc1 not large enough to join array
The partition tables all look the same,
Partition Table for /dev/sdc
First Last
# Type Sector Sector Offset Length Filesystem Type (ID) Flag
-- ------- ----------- ----------- ------ ----------- -------------------- ----
Pri/Log 0 2047* 0# 2048*Free Space None
1 Primary 2048* 3907029167* 0 3907027120*Linux raid auto (FD) None
Partition Table for /dev/sdd
First Last
# Type Sector Sector Offset Length Filesystem Type (ID) Flag
-- ------- ----------- ----------- ------ ----------- -------------------- ----
Pri/Log 0 2047* 0# 2048*Free Space None
1 Primary 2048* 3907029167* 0 3907027120*Linux raid auto (FD) None
Partition Table for /dev/sde
First Last
# Type Sector Sector Offset Length Filesystem Type (ID) Flag
-- ------- ----------- ----------- ------ ----------- -------------------- ----
Pri/Log 0 2047* 0# 2048*Free Space None
1 Primary 2048* 3907029167* 0 3907027120*Linux raid auto (FD) None
Or actualy if print the partition table as raw data in cfdisk there is some differences but I can not decipher what that means.
-> diff sde.raw sdc.raw
1c1
Disk Drive: /dev/sde
---
Disk Drive: /dev/sdc
30c30
0x1B0: 00 00 00 00 00 00 00 00 B7 E9 70 74 00 00 00 20
---
0x1B0: 00 00 00 00 00 00 00 00 4B 0C 58 1C 00 00 00 20
I realized that I could try to copy the mbr from one of the working discs using dd, but still I get the same error even though the partition table is identical in the raw output from cfdisk.
$ sudo dd if=/dev/sdd of=/tmp/sdd-mbr.bin bs=512 count=1
$ sudo dd if=/tmp/sdd-mbr.bin of=/dev/sdc bs=512 count=1
$ cat /proc/partitions
major minor #blocks name
8 48 1953514584 sdd
8 49 1953513560 sdd1
8 32 1953514584 sdc
8 33 1953513560 sdc1
8 64 1953514584 sde
8 65 1953513560 sde1
Now the raw comparison of the partition table gives identical output and the partitions seems to be of the same size but I still get the same error when trying to add /dev/sdc1
to the array.
I guess my question is if is there is any way to fix this without having to take the whole array apart and recreate it from scratch ?
Output from mdadm -D /dev/md_d0
the array is still rebuilding since I added /dev/sdc again.
/dev/md_d0:
Version : 0.90
Creation Time : Sat Aug 14 21:06:13 2010
Raid Level : raid5
Array Size : 3907028992 (3726.03 GiB 4000.80 GB)
Used Dev Size : 1953514496 (1863.02 GiB 2000.40 GB)
Raid Devices : 3
Total Devices : 3
Preferred Minor : 0
Persistence : Superblock is persistent
Update Time : Fri Jan 11 18:36:06 2013
State : clean, degraded, recovering
Active Devices : 2
Working Devices : 3
Failed Devices : 0
Spare Devices : 1
Layout : left-symmetric
Chunk Size : 64K
Rebuild Status : 53% complete
UUID : 74998045:22316376:01f9e43d:ac30fbff (local to host server)
Events : 0.19988
Number Major Minor RaidDevice State
3 8 32 0 spare rebuilding /dev/sdc
1 8 64 1 active sync /dev/sde
2 8 48 2 active sync /dev/sdd
I realize now that it looks like the whole array is made up of the actual devices rather than the partiotions, then the question is why sdd1 & sde1 partitios are left on the harddrives and over written on /dev/sdc as soon as I add it to the array.