Many of you will say “RAID 5, of course!” and for those of you that do I would like to ask why? Is it because you really understand the differences between the different RAID levels? Granted that you may but have you ever had to do a RAID rebuild of a RAID 5 volume? How long did it take? Was it actually successful?
What RAID level would I use for my VMFS? RAID 10, of course! 😉
Needless to say this post is not about asking questions, but instead give you insight into my years of experience with this very subject. RAID 5 was an acceptable choice when the cost of hard drives were so high as to be prohibitive. However, these days quite the opposite is true. The benefits for using RAID 10 (performance, recovery, etc) far outweigh the costs for the additional drives required.
Art S. Kagel has written an excellent article about the different RAID levels (including RAID 3 and 4) as well as why he also chooses RAID 10 over RAID 5:
http://www.miracleas.com/BAARF/RAID5_versus_RAID10.txt
If you are currently using RAID 5 for you critical uptime production VMFS then you may want to consider creating a new RAID group, carve out a LUN and Storage VMotion your VM’s.
Lets not forget raid DP or raid 6 in most cases, is also gaining some movement in the storage world. Additionally, Raid 50 is used in the Dell Equallogic Array's, as a option as well.
While it might seem feasible to use RAID-10 for VMFS storage, the cost of doubling the number of disks makes it prohibitive for larger implementations. RAID-10 has two primary benefits over most other types of RAID, namely performance (IOPS – Input/Output operations Per Second) and the ability to survive multiple disk failures (assuming no 2 disks from the same mirror group fail).
In a mid-range (and especially with a high-end) storage system the read/write cache makes up for the difference in IOPS performance between RAID-5 and RAID-10 so that benefit is usually negated.
You are correct in that the rebuild time on a large RAID-5 set can be abysmal (especially as drive sizes continue to increase). Most storage vendors counter this in one of a couple different kinds of ways. The traditional method of using small RAID-5 groups and striping LUNs across many groups of them (what EMC would call Symmetrix Metavolumes, CLARiiON MetaLUNs or Celerra Striped Volumes), combined with 1 or more hot spare drives per 15-30 drives this usually results in an acceptable rebuild time (since the RAID groups are usually 4+1), yet maintains the performance increase of a very large number of spindles. Additionally many storage arrays will allocate a queue per LUN which (if supported by the host bus adapter) can result in further performance improvements.
NetApp on the other hand uses a modified RAID-4 for horizontal parity calculations, they have now combined this with diagonal parity to create RAID-DP (a RAID-6 implementation), the diagonal parity is only used in a double disk drive failure and the implementation results in only a small write performance hit (2-3%) compared to other RAID-6 implementations.
Other arrays implement wide striping to stripe across most or all of the disks in the entire array automatically. This can make a big difference in the complexity of management of the storage array by making much easier to implement.
Companies like 3PAR have even another way of doing things – I'd get into it, but I think Marc Farley has it covered beautifully here: http://www.storagerap.com/2008/12/basics-of-wid…
To take Art's quote a tweak it beyond his limited 8/5 drive implementation to give you an idea… “If a drive costs $1000 USD then switching from a 200 disk RAID-10 array to a 125 drive RAID-5 array will save 75 drives or $75,000 USD.” – I'm sorry but most of my clients would consider that a substantial savings (don't forget that's just raw disk costs – you would need to add an additional 5 drive shelves to accommodate those disks too and those can run 10K+ each, so we are likely well above $120,000 USD) and feel that one of the above methods (depending on their SAN vendor of choice) more than make up for the reliability concerns parity based RAID brings to the table.
My 2¢
My vote: RAID 50. EQL style. But I will use RAID 10 in many of my SMB installs to create a performance tier for Exchange and SQL (2 VMs typically). I guess we can do that affordably because we're usually talking less than 3 trays of disk.
Nice comment, Andrew. Very informative. And now I'm into StorageRap. 🙂 But would you ever use RAID 10? What application/scenario warrants its use?
I have nothing against RAID-10, I just don't see it as the best choice for “all” VMFS datastores. I see its value in smaller implementations, although like you mentioned, newer RAID levels like RAID-50 are becoming more popular there.
RAID-10 is still my choice for either RDM or VMFS stored Oracle archive logs, SQL transaction logs, Exchange logs, etc, anything where sequential read/writes are the norm. But I would avoid mixing those LUNs across the same set of disks, instead preferring to allocate spindles directly to the VMs (again this is really only a problem in larger database/mail servers with a high number of transactions where you can't afford seek time).
That raises a great question in our smaller Compellent-VMware solutions. Compellent allows us to create RAID-10 and RAID-5 (5 disk) and RAID-5 (9-disk) LUNs all on the same sets of hard disks. Let's say I have a 2 VMFS LUNS on a single tray of FC: 1 RAID-10 and 1 RAID-5. If VMs on both LUNs generate large amounts of IO, will the VMs on RAID-10 volumes perform any better than the VMs on the RAID 5 volumes? What would the results look like if a tray was dedicated RAID 10 and a tray was dedicated RAID 5? Sounds like homework for Sean for when we add our 2nd Compellent storage center.
I can't really answer that one, Compellent uses a totally different model from many “classic”
storage arrays (like 3PAR – not the same techniques but they are another vendor with a truly innovative model) and its Storage Center software handles automatic tiering at a block level across the entire array (I'm not sure how much their FastTrack technology makes a difference here, but it may). I've not had the chance to get hands on with their product before but I would imagine that given their model, the more disks the better. 🙂
For those of you that are lucky, I suggest using RAID-DP, of course this is only available from NetApp but it does offer you the greatest protection of double parity disks with the lowest cost (compared to RAID10). For those of you that can not afford building out a RAID 10 RAID group there is always RAID 6 (ADG) which offers you performance and protection.
One thing a little off topic (yet having to do with disks) is disk performance. One thing you cannot forget is that with the more VMs you load on your disks, the faster they need to be. Of course if your using SATA disks you can just add more and more spindles. But, the better alternative would be to use high speed SAS or Fibre Channel (FC) disks. SAS/FC can provide as much as three times the performance of a SATA disk.
“RAID 6 (ADG) which offers you performance and protection”
Protection over RAID-5 yes, but performance – no way. On a traditional RAID-6 implementation the additional parity calculation has a write penaly usually 12-25% (yes I've seen some bad ones) worse than that of RAID-5. That's what I love about the NetApp implementation (RAID-DP) it has such a minimal impact (since they are still using RAID-4, single parity disks, just doubling the number of disks – 1 parity disk for horizontal parity, 1 for diagonal).
You've got a great point there about the SATA vs SAS/FC argument. “It's all about the IOPS” 😉
“RAID 6 (ADG) which offers you performance and protection”
Protection over RAID-5 yes, but performance – no way. On a traditional RAID-6 implementation the additional parity calculation has a write penaly usually 12-25% (yes I've seen some bad ones) worse than that of RAID-5. That's what I love about the NetApp implementation (RAID-DP) it has such a minimal impact (since they are still using RAID-4, single parity disks, just doubling the number of disks – 1 parity disk for horizontal parity, 1 for diagonal).
You've got a great point there about the SATA vs SAS/FC argument. “It's all about the IOPS” 😉
“RAID 6 (ADG) of course, which gives you a better performance and good protection…
Thanks for your post,I will come back!
I use RAID no. 10. It’s useful and helpful in many ways. I hope I won’t be changing it unless I find something excessively better than this or it gets updated.
Is it true that writes on RAID6 are still slow? 3Ware controllers now calculate parity operations in parallel so in theory it should be equivalent to only one parity calculation like in RAID5.
Given a lot of spindles, RAID6 speeds should be approaching that of RAID10 and gives you much more efficient use of disks, (n-2 as opposed to n/2 for RAID10).
I'm building a RAID for vm's to sit on at the moment and am wondering whether I should use 6 or 10.
Most people recommending a particular RAID level seem to be basing their decision on gut instinct or on what they think they know about RAID levels in general but it is very difficult to find any actual benchmarks. I may do some.
Sorry for resurrecting this thread but it still seems a valid topic.
Hello Simon,
This question is quite a loaded one. It actually got brought up at the VMworld Ask the experts session:http://vbrownbag.comm/2009/09/vmworld-2009-ask-the-experts-follow-up/
Specifically: # Ask the experts – “Hardware raid config to use” <- It depends. YES IT DOES!
Basically it depends on your IO & redundancy requirements, connection methods, etc.
I would work with your storage vendor or reseller to determine what is best for your solution.
Simon… further asking your question on twitter got similar responses:
From @jasemccarty: you hit the nail on the head. It depends on the workload 100%. Good to tune VMFS for the workload on top of it.
From @scott_lowe: Like others have said, too many variables to make a blanket recommendation: array architecture, drive types, I/O needs, etc.
Thanks for the responses. I'll have a read of the VMWord article.
Cheers all.
Hey I'v been reading your blog from quite some time now and I just wanted to say keep up the good work.
Awesome! Thanks. We've got some big things planned in the future as well.
Hey I'v been reading your blog from quite some time now and I just wanted to say keep up the good work.
Having perfect wedding dresses for your special day is important for every bridals.
巨人倍増枸杞カプセルは最新の超臨界流体抽出技術を採用、睾丸中の有機化合物の働きを活性化し、成分が直接ペニスの海綿体内部に浸透し毛細血管を 拡張します。
血液の流入を促進し、睾丸の有機酵素含有量を増大、ペニスの硬さと太さを増大させます。
巨人倍増枸杞カプセルは細胞の再生力を促進、睡眠時の睾丸の働きを活発化させ、ペニス組織の発育を促進することによってペニスの大きさそのものを更に成長させます。
最先端技術によりペニスの勃起力を大幅に向上！超強力新型精力剤な製品ですね！
巨人倍増:http://www.dazebrands.com/goods3.html