Understanding Storage Performance
Comparing storage performance is a bit more difficult than meets the eye. This comes up quite a bit as I frequently address the differences between RAM and Flash-based SSDs. This particularly comes up when comparisons are made between TMS’s Flagship Flash system, the RamSan-630 and the Flagship RAM system, the RamSan-440. The Flash system has higher IOPS but the RAM system has lower latency.
There are three independent metrics of storage performance: response time, IOPS, and bandwidth. Understanding the relationships between these metrics is the key to understanding storage performance.
Bandwidth is really just a limitation of the design or standards that are used to connect storage. It is the maximum number of bytes that can be moved in a specific time period; response time overhead or concurrency do not play a factor. IOPS are nothing more than the number of I/O transactions that can be performed in a single second. Determining the maximum theoretical IOPS for a given transfer size is as simple as dividing the maximum bandwidth by the transfer size. For a storage system with a single one Gbps iSCSI connection (~100 MB/s bandwidth) and a workload of 64 KB transfers, then the maximum IOPS will be ~1,500. If the transfer size is a single sector (512 bytes), then the maximum IOPS will be ~200,000 – a notable difference. At this upper limit, bandwidth will more than likely not be the performance limiter.
There is another relationship that ties together response time and concurrency – Little’s law. Little’s law governs the concurrency of a system needed to achieve a desired amount of throughput. For storage, Little’s Law is: (Outstanding I/Os) ÷ (response time) = IOPS. I consider this the most important formula in storage performance. If you boil this down, ultimately the limitation of IOPS performance is the ability of a system to handle Outstanding I/Os concurrently. Once that limit is reached, the I/Os get clogged up and the response time increases rapidly. This is the reason a common tactic to increase storage performance has been to simply add disks – each additional disk increases the concurrent I/O capabilities.
Interestingly, the IOPS performance isn’t limited by the response time. Lower response times merely allow a given level of IOPS to be achieved at lower levels of concurrency. There are practical limits on the level of concurrency that can be achieved by the interfaces to the storage (e.g. the execution throttle setting in an HBA), and many applications have fairly low levels of concurrent I/O, but the response time by itself does not limit the IOPS. This is why even though Flash media has a higher response time than RAM, Flash systems that handle a high level of concurrent I/O can achieve as good as or better IOPS performance.