SAN Shared File Systems with SSDs
On a recent post I discussed how PCIe SSDs fit in with the rise of shared-nothing scale-out clusters and why exceptionally large clusters favor direct attached storage. There are two elements that are required for this type of architecture to be successful: a logical way to partition the data and a need to scan large chunks of data from those partitions. These elements are prevalent in many applications, however, not to all of them. There are applications where “Total Access” to the data is required and the scale-out model is ineffective as most of the data is fetched across the network rather than accessing it locally.
Applications requiring total access are common in the scientific computing, financial modeling, and government areas. They are characterized by being able to be effectively parallelize the processing, but the data that each process needs could be located anywhere in a large dataset. This results in lots of small random access that traditional cluster designs are not well suited to. There have been three primary methods to tackle these problems: add a task specific preprocessing step to allow subsequent accesses to be less random, create a cluster where each compute node’s memory can be accessed over a network, or deploy the applications on a large memory SMP server.
There is a new option that I have seen getting deployed more and more often: using high capacity SSDs and a SAN shared file system. A SAN shared filesystem provides the locking to allow multiple servers to directly access the block storage concurrently. This provides the ease of use of a file system with the performance benefits of block storage access. If you add SSDs into this setup you can build a very powerful solution. The basic setup looks like the following:
Using SSDs as the primary storage allows the shared file system to handle small block random workloads in addition to the standard high bandwidth workloads that are the mainstay of most SAN shared file system deployments. It is relatively easy to construct a system that has a couple hundred cores of processing power and tens to hundreds of TBs of flash. This system can tackle the types of workloads that were previously reserved for the large memory SMP systems of the past.
Workloads that require “Total Access” to a large data set are limited in performance by the network and process steps that connect the compute resources and the storage. The big benefit of a SAN shared file system is first, that the shared components – the SAN resources – can use simple block level communication and multiple high bandwidth interfaces to have a very high performance and low latency. Secondly, solutions can scale-up the performance almost linearly since the coordination processes run on the servers rather than the storage. When customers have extreme application performance needs this is the architecture I look to first.