diff options
| -rw-r--r-- | docs/testing/user/introduction.rst | 196 |
1 files changed, 190 insertions, 6 deletions
diff --git a/docs/testing/user/introduction.rst b/docs/testing/user/introduction.rst index 49e3220..0099c39 100644 --- a/docs/testing/user/introduction.rst +++ b/docs/testing/user/introduction.rst @@ -25,13 +25,13 @@ performance metrics in the shortest reasonable time. How Does StorPerf Work? ======================= -Once launched, StorPerf presents you with a ReST interface, along with a +Once launched, StorPerf presents a ReST interface, along with a `Swagger UI <https://swagger.io/swagger-ui/>`_ that makes it easier to form HTTP ReST requests. Issuing an HTTP POST to the configurations API -causes StorPerf to talk to your OpenStack's heat service to create a new stack -with as many agent VMs and attached Cinder volumes as you specify. +causes StorPerf to talk to OpenStack's heat service to create a new stack +with as many agent VMs and attached Cinder volumes as specified. -After the stack is created, you can issue one or more jobs by issuing a POST +After the stack is created, we can issue one or more jobs by issuing a POST to the jobs ReST API. The job is the smallest unit of work that StorPerf can use to measure the disk's performance. @@ -45,8 +45,187 @@ measured start to "flat line" and stay within that range for the specified amount of time, then the metrics are considered to be indicative of a repeatable level of performance. -What Data Can I Get? -==================== +StorPerf Testing Guidelines +=========================== + +First of all, StorPerf is not able to give pointers on how to tune a +Cinder implementation, as there are far too many backends (Ceph, NFS, LVM, +etc), each with their own methods of tuning. StorPerf is here to assist in +getting a reliable performance measurement by encoding the test +specification from SNIA, and helping present the results in a way that makes +sense. + +Having said that, there are some general guidelines that we can present to +assist with planning a performance test. + +Workload Modelling +------------------ + +This is an important item to address as there are many parameters to how +data is accessed. Databases typically use a fixed block size and tend to +manage their data so that sequential access is more likely. GPS image tiles +can be around 20-60kb and will be accessed by reading the file in full, with +no easy way to predict what tiles will be needed next. Some programs are +able to submit I/O asynchronously where others need to have different threads +and may be synchronous. There is no one size fits all here, so knowing what +type of I/O pattern we need to model is critical to getting realistic +measurements. + +System Under Test +----------------- + +The unfortunate part is that StorPerf does not have any knowledge about the +underlying OpenStack itself – we can only see what is available through +OpenStack APIs, and none of them provide details about the underlying +storage implementation. As the test executor, we need to know +information such as: the number of disks or storage nodes; the amount of RAM +available for caching; the type of connection to the storage and bandwidth +available. + +Measure Storage, not Cache +-------------------------- + +As part of the test data size, we need to ensure that we prevent +caching from interfering in the measurements. The total size of the data +set in the test must exceed the total size of all the disk cache memory +available by a certain amount in order to ensure we are forcing non-cached +I/O. There is no exact science here, but if we balance test duration against +cache hit ratio, it can be argued that 20% cache hit is good enough and +increasing file size would result in diminishing returns. Let’s break this +number down a bit. Given a cache size of 10GB, we could write, then read the +following dataset sizes: + +* 10GB gives 100% cache hit +* 20GB gives 50% cache hit +* 50GB gives 20% cache hit +* 100GB gives 10% cache hit + +This means that for the first test, 100% of the results are unreliable due to +cache. At 50GB, the true performance without cache has only a 20% margin of +error. Given the fact that the 100GB would take twice as long, and that we +are only reducing the margin of error by 10%, we recommend this as the best +tradeoff. + +How much cache do we actually have? This depends on the storage device being +used. For hardware NAS or other arrays, it should be fairly easy to get the +number from the manufacturer, but for software defined storage, it can be +harder to determine. Let’s take Ceph as an example. Ceph runs as software +on the bare metal server and therefore has access to all the RAM available on +the server to use as its cache. Well, not exactly all the memory. We have +to take into account the memory consumed by the operating system, by the Ceph +processes, as well as any other processes running on the same system. In the +case of hyper-converged Ceph, where workload VMs and Ceph run on the systems, +it can become quite difficult to predict. Ultimately, the amount of memory +that is left over is the cache for that single Ceph instance. We now need to +add the memory available from all the other Ceph storage nodes in the +environment. Time for another example: given 3 Ceph storage nodes with +256GB RAM each. Let’s take 20% off to pin to the OS and other processes, +leaving approximately 240GB per node This gives us 3 x 240 or 720GB total RAM +available for cache. The total amount of data we want to write in order to +initialize our Cinder volumes would then be 5 x 720, or 3,600 GB. The +following illustrates some ways to allocate the data: + +* 1 VM with 1 3,600 GB volume +* 10 VMs each with 1 360 GB volume +* 2 VMs each with 5 360 GB volumes + +Back to Modelling +----------------- + +Now that we know there is 3.6 TB of data to be written, we need to go back to +the workload model to determine how we are going to write it. Factors to +consider: + +* Number of Volumes. We might be simulating a single database of 3.6 TB, so + only 1 Cinder volume is needed to represent this. Or, we might be + simulating a web server farm where there are hundreds of processes + accessing many different volumes. In this case, we divide the 3.6 TB by + the number of volumes, making each volume smaller. +* Number of Virtual Machines. We might have one monster VM that will drive + all our I/O in the system, or maybe there are hundreds of VMs, each with + their own individual volume. Using Ceph as an example again, we know that + it allows for a single VM to consume all the Ceph resources, which can be + perceived as a problem in terms of multi-tenancy and scaling. A common + practice to mitigate this is to use Cinder to throttle IOPS at the VM + level. If this technique is being used in the environment under test, we + must adjust the number of VMs used in the test accordingly. +* Block Size. We need to know if the application is managing the volume as a + raw device (ie: /dev/vdb) or as a filesystem mounted over the device. + Different filesystems have their own block sizes: ext4 only allows 1024, + 2048 or 4096 as the block size. Typically the larger the block, the better + the throughput, however as blocks must be written as an atomic unit, larger + block sizes can also reduce effective throughput by having to pad the block + if the content is smaller than the actual block size. +* I/O Depth. This represents the amount of I/O that the application can + issue simultaneously. In a multi-threaded app, or one that uses + asynchronous I/O, it is possible to have multiple read or write requests + outstanding at the same time. For example, with software defined storage + where there is an Ethernet network between the client and the storage, + the storage would have a higher latency for each I/O, but is capable of + accepting many requests in parallel. With an I/O depth of 1, we spend + time waiting for the network latency before a response comes back. With + higher I/O depth, we can get more throughput despite each I/O having higher + latency. Typically, we do not see applications that would go beyond a + queue depth of 8, however this is not a firm rule. +* Data Access Pattern. We need to know if the application typically reads + data sequentially or randomly, as well as what the mixture of read vs. + write is. It is possible to measure read by itself, or write by itself, + but this is not typical behavior for applications. It is useful for + determining the potential maximum throughput of a given type of operation. + +Fastest Path to Results +----------------------- + +Once we have the information gathered, we can now start executing some tests. +Let’s take some of the points discussed above and describe our system: + +* OpenStack deployment with 3 Control nodes, 5 Compute nodes and 3 dedicated + Ceph storage nodes. +* Ceph nodes each have 240 GB RAM available to be used as cache. +* Our application writes directly to the raw device (/dev/vdb) +* There will be 10 instances of the application running, each with its own + volume. +* Our application can use block sizes of 4k or 64k. +* Our application is capable of maintaining up to 6 I/O operations + simultaneously. + +The first thing we know is that we want to keep our cache hit ratio around +20%, so we will be moving 3,600 GB of data. We also know this will take a +significant amount of time, so here is where StorPerf helps. + +First, we use the configurations API to launch our 10 virtual machines each +with a 360 GB volume. Next comes the most time consuming part: we call the +initializations API to fill each one of these volumes with random data. By +preloading the data, we ensure a number of things: + +* The storage device has had to fully allocate all of the space for our + volumes. This is especially important for software defined storage like + Ceph, which is smart enough to know if data is being read from a block that + has never been written. No data on disk means no disk read is needed and + the response is immediate. +* The RAM cache has been overrun multiple times. Only 20% of what was + written can possibly remain in cache. + +This last part is important as we can now use StorPerf’s implementation of +SNIA’s steady state algorithm to ensure our follow up tests execute as +quickly as possible. Given the fact that 80% of the data in any given test +results in a cache miss, we can run multiple tests in a row without having +to re-initialize or invalidate the cache again in between test runs. We can +also mix and match the types of workloads to be run in a single performance +job submission. + +Now we can submit a job to the jobs API to execute a 70%/30% mix of +read/write, with a block size of 4k and an I/O queue depth of 6. This job +will run until either the maximum time has expired, or until StorPerf detects +steady state has been reached, at which point it will immediately complete +and report the results of the measurements. + +StorPerf uses FIO as its workload engine, so whatever workload parameters we +would like to use with FIO can be passed directly through via StorPerf’s jobs +API. + +What Data Can We Get? +===================== StorPerf provides the following metrics: @@ -57,4 +236,9 @@ StorPerf provides the following metrics: These metrics are available for every job, and for the specific workloads, I/O loads and I/O types (read, write) associated with the job. +For each metric, StorPerf also provides the set of samples that were +collected along with the slope, min and max values that can be used for +plotting or comparison. + As of this time, StorPerf only provides textual reports of the metrics. + |