IO Domains – What are they? Why use them?
Today I wanted to introduce the concept of IO domains for your storage planning. IO Domains are just that, areas or domains, separated by either policy, performance, or both.
First lets take a look at some of the potential problems and pitfalls they’ll help you avoid:
- VMFS Corruption
- Redundancy
- Performance
- Flexibility
Corruption!
File systems don’t get corrupt anymore do they? Damn the backups, we’ve got VMFS! It is a robust, journaled, enterprise class file system, it can’t get corrupt, can it? Yes. Yes it can. When it does, it’d likely be best to have an updated copy of your resume ready, as frankly, it’s not pretty. Hopefully it’s not on the Datastore too 🙂
Not to paint a bad picture of VMFS, it is good, fast, solid and reliable. The problem I wanted to point out here, is building a HUGE VMFS volume out of many little 2 TB extents. When you lose access to one of those extents, “Bad Things” happen. What will happen to the running VM’s that lost their disks? Blamo, an RPE (Resume Producing Event)
Redundancy
This could likely be bundled in there with corruption, as having a single VMFS is prone to well… problems in that VMFS. So having more than one is really the way to go anyways. This can be accomplished in any number of ways, and most storage vendors have their own method of replicating data between their storage devices. As DR (Disaster Recovery), or Business Continuity is very implementation specific, I’ll leave that choice up to the reader.
Performance
Think on this for a moment. Each of your VM’s lives in a file or set of files (excluding RDM’s) that is. Each of these in turn lives on VMFS. IO operations in the VM are virtualized, and passed to the underlying disk file. These in turn get a bit of overhead. Now, multiply that, for the number of IO’s each of your VM’s will be doing. Do you smell the bottleneck? Keep an eye out on the IO with esxtop and the number of VMs per Datastore, accounting for the busy VMs.
Flexibility
Think on this one. That little VW Bug at the circus, the one with all the clowns in it. Have that picture in your head? I’ll wait. Now, picture your VM’s in the same situation, no room to grow and stretch, both in size, and in terms of IOPS (IO Operations Per Second).
IO Domains – Some Examples
As stated above IO Domains are to segregate different IO workloads from one another. This can be accomplished in a few ways. We’ll cover an example or two here. Keep in mind, these are generalizations, and will likely have little to do with your own environment. They’re only to serve as examples of different workloads and how you can design for them.
Example – Multi-Tier Application
Any number of web 2.0 sites, e-commerce, CMS, Blogs (like this one) have a few conceptual tiers. Their web front ends or WWW servers, their Application Engines (e-commerce/CMS, etc), and finally their database back ends. Each of these has different requirements, from both a performance perspective, as well as logical and security separation as well.
Looking at the IO workloads – Each of these tiers has it’s own unique requirements, lets look at those:
The web tier – In our example, the web tier will be serving dynamically generated pages, using the web programming language of your choice. All of the code on each web node is static, and there is a large amount of locally stored images. Each of these web nodes is nearly identical (IP address and machine names are different) and can rapidly be re-deployed from template. Being that the required redundancy isn’t high in this case, and most of the content can be cached in the web servers memory for faster serving, we could easily get away with a lower end NAS, or SATA / iSCSI SAN solution for this tier. Say a basic set of 10k rpm spindles (disks) in a raid 5 or 10. This would meet both the performance and redundancy requirements, and would keep costs down, so you can fit more VM’s into this layer should you need to spin up in a hurry.
The App Tier – This tier functions as say a big calculator, or translator, it has a need to run queries against the database, and to serve them up to the web front ends quickly. These boxes can also be deployed from template, but require more customization at deployment time, and thus we should build in more redundancy in the design. Here we can choose to go with 15k RPM disks, and either SAS or Fiber Channel drives in a Raid 10. This will provide better read & write performance (as we’re doing quite a bit of both) as well as added redundancy.
The Database Tier – This is where the wheels come off. Really. Instead of a single LUN or set of spindles, we’re going to get all kinds of fancy here. After all, you do want 300k simultaneous users on your app don’t you? Thought so. Now what we’ve set our goals at the stars, lets figure out how to do this. The database servers in this case all need to be built, and installed from a generic OS install template, and have additional disks added for things like, data, logs, etc.
Starting with the OS and working out from there. Since the OS installs will be fairly generic, and have very little IO load of their own, they can be put onto any raid5 array (NFS, iSCSI, SAN) however, you will want to keep them on the same RPM spindles as the remainder.
The Data disks will have a high level of redundancy required. For this something like a 4+ disk raid 10, on 15k fiber channel disks, and one LUN per database server would be best. This provides for both the data redundancy, IO separation for performance. The transaction logs have similar requirements and should be set up similarly, also utilizing separate spindles where possible.
Conclusion
Breaking out your storage in this fashion will allow all of these VM’s to exist happily, have room to expand (in terms of space and IOPS) , and provide a high level of redundancy and performance.
We’ve covered a lot. If you have any questions, or concerns, drop me a line in the comments.