Sunday, April 28, 2024
 Popular · Latest · Hot · Upcoming
10
rated 0 times [  10] [ 0]  / answers: 1 / hits: 23331  / 3 Years ago, sat, october 16, 2021, 3:06:32

I've just read this in another post about improving RAID5/6 write speeds:




After increasing stripe cache & switching to external bitmap, my speeds are 160 Mb/s writes, 260 Mb/s reads. :-D




I've already found out how to increase the stripe cache and this worked pretty well but I'd like to know more about an external bitmap. I have an incredibly fast (540MB/s) RAID0 SSD that would do well if a bitmap does what I think it does but I'm still very unsure. I've only known about them as long as I've known this post.



A few questions:




  • What is a bitmap (in terms of mdadm)?

  • What are the advantages of an internal bitmap (over external)?

  • What are the advantages of an external bitmap (over internal)?

  • How do I switch between the two?



I should add that while this is a I'm-bored-let's-break-something thread, I do value the data stored on the RAID array. If doing this is going to put data at significant risk, please let me know.


More From » performance

 Answers
7

What's a bitmap:



A mdadm bitmap, also called a "write intent bitmap", is a mechanism to speed up RAID rebuilds after an unclean shutdown or after removing and re-adding a disk.



With a bitmap, writing data to the RAID goes like this:




  • Update bitmap: Mark the RAID chunks you are about to write to as dirty.

  • Write the data to the RAID.

  • Update bitmap: Mark the RAID chunks that were just written as clean.



The advantage of a bitmap is that if the system goes down in the middle of a write, the rebuild needs to check only the chunks marked as dirty, rather than the whole multi-TB RAID. This can speed up the rebuild process from taking several hours to completing in just a few seconds.



The drawback is lower write performance under normal use (outside rebuilds), since mdadm does additional disk access to update the bitmap.



External vs internal:




  • external: Stored as a file on a disk outside the RAID. The advantage over an internal bitmap is better write performance during normal use (outside rebuilds).

  • internal: Stored as RAID metadata. The advantage over an external bitmap is that you don't need a non-RAID disk and you save a bit on configuration (the path to the bitmap).



According to the mdadm man page:



Note:  external bitmaps are only known to work on ext2 and ext3.
Storing bitmap files on other filesystems may result in serious
problems.


According to a post by Neil Brown, the mdadm author, external bitmaps should work on ext4 too:



I haven't looked inside ext4 but I am fairly confident that external bitmaps 
will work properly.


HOWTO:



Bitmaps are added and removed using mdadm --grow --bitmap=XXX ..., where the XXX is one of:




  • --bitmap=internal: Create an internal bitmap.

  • --bitmap=/var/my_bitmap.bin: Create an external bitmap at the specified path. The path must reside outside the RAID. A bitmap=... parameter must be added to the ARRAY entry in /etc/mdadm/mdadm.conf, and the --bitmap=... parameter must be passed if you are assembling the RAID from the command line.

  • --bitmap=none: Remove/disable any bitmaps.



References:





Comments:



IMO, bitmaps are perhaps primarily of interest for RAID levels 5 and 6, since these have the slowest rebuilds.



I switched from RAID 5 to RAID 10 myself; the rebuilds are so much faster that I don't feel the need for a bitmap, and RAID 10 seems to require far fewer rebuilds in the first place.



My RAID 5 setup used to drop a disk something like once a month, causing 12-14 hour rebuilds. The RAID 10 has only dropped a disk once in half a year, rebuilt in less than an hour.



I don't know if the frequent disk drops I experienced was caused by something other than the RAID level, but the RAID 10 has been far more stable and rebuild speed isn't much of a concern anymore.


[#44440] Sunday, October 17, 2021, 3 Years  [reply] [flag answer]
Only authorized users can answer the question. Please sign in first, or register a free account.
issfullus

Total Points: 264
Total Questions: 126
Total Answers: 107

Location: Comoros
Member since Mon, Dec 19, 2022
1 Year ago
;