SSD Benchmarking 201, Part 1: Building a Repeatable Benchmark Job

Benchmarking 101, Part 2 met fio only in passing, as the engine under KDiskMark and as the custom job file that Anvil stands in for. Benchmarking 201 starts there. A benchmark result is only as repeatable as the job that produced it. That job sets the transfer size, the share of reads, how random the offsets are, how many I/Os stay in flight, how long the tool waits before it records, and how long it records. Change one of them and the number moves, so nobody can check a result published without its job, not even the person who ran it.

I ran my enterprise SSD tests as a set of Iometer configuration files called from batch files, and the set still reads like a specification. This part goes through it one setting at a time, gives the fio option that does the same job, and checks a real result from a Micron 9100 MAX 2.4TB against the file that produced it. Steady state and latency percentilesA latency percentile gives the time within which that share of commands finished. The 99th percentile is the time that 99% of commands beat. get the next part to themselves.

What a job file pins down

An Iometer job is built from access specifications, and fio writes the same settings as named options. The table pairs them.

The rows come from the Iometer User’s Guide for version 1.1.0 and the fio documentation at revision 3.42.

My 4K random read spec is one line, with 4,096 byte transfers making up 100% of the I/O at 100% reads and 100% random, aligned to 4,096 bytes. Server mixes take more lines. My File Server spec spreads eight sizes from 512B to 64KiB, 60% of them 4KiB, all at 80% reads. Database is a single 8KiB line at 67% reads, and Email Server is a single 8KiB line at 50%.

In fio, rwmixread defaults to 50%, so a randrw job that leaves it out runs half reads without saying so. randrepeat is on by default and seeds the random offsets the same way on every run, which is what lets two runs of one job hit the same pattern.

A job is more than its I/O pattern. Every config in my set still stores the disk targets it was last saved with, four Kingston 400GB drives, so opening a config for a new drive and starting it would test the old ones. My SNIA-style config also names its smallest transfers 512K while the size field says 512 bytes. Only the size field reaches the drive, so that’s the line I check before a run.

Queue depth is workers times outstanding I/Os

Iometer has two settings for load. Its Dynamo program generates the I/O from worker threads, and each worker keeps its own number of outstanding I/Os against each disk. The queue depth (QD)IOPS counts commands finished per second. Queue depth is how many commands are outstanding at once, and a drive needs a deeper queue to reach its top IOPS. the drive sees is the product of the two. Benchmarking 101, Part 1 counts queue depth per thread, the way CrystalDiskMark labels its tests, and in this series QD always means the total the drive sees. In my low queue depth configs the worker count steps through 1, 2, 4, and 8 with one outstanding I/O each, for QD1 to QD8. From QD16 up, 16 workers stay fixed and each one’s outstanding I/Os step through 1, 2, 4, 8, and 16, for QD16 to QD256, so no worker in the sweep ever holds more than 16.

fio makes the same split. numjobs clones a job into separate threads or processes, and iodepth is the number of I/Os each clone keeps in flight, so numjobs=16 with iodepth=16 is QD256. group_reporting adds the clones into one result. The depth only counts with an asynchronous engine, because fio’s documentation says an iodepth above 1 does nothing for a synchronous one. Windows gets native asynchronous I/O through its default engine windowsaio. On Linux the asynchronous engines are libaio and io_uring, and libaio may only queue with direct=1, which asks for unbuffered I/O.

Iometer’s results file can show whether a run held its depth. Little’s law ties the numbers together. The I/Os in flight equal the I/O operations per second (IOPS) times the average response time. At QD256 in my June 2016 run, the 9100 MAX ran 758,048 IOPS of 4KiB random reads at a 0.337ms average. Dividing 256 by 0.337ms gives about 760,000 IOPS, within 0.3% of what Iometer recorded, so the drive really had 256 I/Os waiting. When the product falls well short of the label, fewer I/Os were in flight than the config asked for, and the run’s queue depth label overstates what the drive saw.

Preconditioning goes in the job

A fresh drive and a used one behave like different devices, which Benchmarking 101, Part 1 covers, so my batch files put the preparation in the same run as the test. Each block size gets its own write precondition right before its sweeps. For 4K and 8K random work that is 6h of random writes at QD256 with 8 workers at 32 outstanding I/Os and no ramp, and the write sweep and then the read sweep follow straight on. Each queue depth step records for 300s after a 30s ramp. In its December 2003 user guide for version 1.1.0, Iometer describes the ramp as a wait after all workers start and before statistics are recorded, and it puts typical ramp times at 15 to 60s.

The server mixes and the SNIA-style rounds first get a sequential fill. That config writes 128KiB sequential data for 2h at QD256, with 4 workers at 64 outstanding I/Os each. Its name says 2x capacity, but it stops on the clock. A 2x fill of a 2.4TB drive is 4.8TB, and finishing that in 7,200s takes at least about 667MB/s of sustained writes, so a drive slower than that ends the fill short of its name. My main batch file runs the fill once before the server mixes, and each mix then runs for 6h in its own pattern at QD256, with 8 workers at 32 outstanding I/Os, right before its sweeps.

Batch files make the sequence repeatable. Each one starts Iometer once per config, each with its own results file, and Iometer’s batch mode then runs the series and records it with nobody at the keyboard. Time stamps in the results files show it. On June 18, 2016, the 9100 MAX’s QD1 to QD8 random write run started at 16:29:26 and its QD16 to QD256 run at 16:51:27, 22 minutes later, which is four 330s steps with no gap between them.

In fio, ramp_time runs the workload before any numbers are logged, and runtime with time_based holds a step for its full time even after it has covered the file. fio can also end a precondition on a measurement instead of a clock. Its steadystate option takes a criterion such as IOPS slope and a limit, and it stops the job once the criterion stays under the limit for the set window.

My SNIA-style job runs the same idea at full length. One config holds 56 access specs, eight block sizes from 512B to 1MiB times seven read shares from 0 to 100%, each run for 60s with 16 workers at 16 outstanding I/Os. Its batch file runs one sequential fill and then the whole config 25 times back to back.

Logging a run so it can be repeated

Iometer’s results file is a CSV with a row per run for the whole test and for each worker and disk. Version 1.1.0 records IOPS, MB/s, average and maximum response time, CPU use, and a latency histogram. Its 21 buckets run from 0 to 50µs up to 10s and over. It also writes the drive’s model string and a time stamp. It has no percentile columns, though the buckets can give rough ones, which the next part works through.

fio reports percentiles itself. clat_percentiles gives completion latency percentiles, percentile_list sets a job’s own list of up to 20, and --output-format=json or json+ writes a file a script can read, the second with the full latency bucket dump. I worked from revision 3.42 of the fio documentation, and options change between versions, so the version belongs with the job. Repeating a run takes the job file, the results file it wrote, the tool and its version, and the drive’s model and firmware, all kept together.

Sweeping all the way to QD256 mattered on the 9100 MAX in my June 2016 runs, where each point is one 300s step. Random reads climbed the whole way, from 6,491 IOPS at QD1 with a 0.153ms average response time to 758,048 at QD256. Random writes stopped climbing at QD16, with 300,697 IOPS there and 311,479 at QD256, 3.6% more, while the average response time rose about 15.5 times, from 0.053ms to 0.822ms. Past QD16 the write numbers measure the queue more than the flash. My reading is that the drive was already writing as fast as it could, so each added I/O only waited longer.

My 2022 job for an E18DC-based enterprise NVMe engineering sample went the other way. One 4KiB random sweep stepped 11 read shares from 0 to 100% in 10% steps across 9 queue depths, from QD1 to QD256, for 99 runs in a single results file. That job runs one spec at every pairing of read share and depth, in place of the server mixes.

The question a job file leaves open

A job file says how a number was made. It doesn’t say whether the drive was ready to be measured. My random preconditions ran for a fixed 6h, and the next part of this series checks that choice against the steady-state rules in the SNIA Solid State Storage Performance Test Specification and against fio’s steadystate option.

Figure 1.5 is the raw material for that part. In my June 2016 run, the 9100 MAX’s 0.337ms average at QD256 sits inside the 200 to 500µs bucket, which held 72.0% of its reads, and the slowest reads reached the 2 to 5ms bucket. That part turns buckets like these into the percentiles and quality of service (QoS) figures that enterprise buyers compare.