SSD Benchmarking 201, Part 4: Reading Published Enterprise SSD Reviews

Part 3 laid out my enterprise test runs on three drives. Most buyers never see runs like those. They see a review with a chart and a winner, and the chart rarely carries the job that made it. This part sets five published reviews of the same three drives beside my results. TweakTown, StorageReview and ServeTheHome tested the Intel DC P4510 2TB, PC Perspective tested the Micron 9100 MAX 2.4TB, and KitGuru tested the Kingston DCP1000 1.6TB. Where a review gives its method and its numbers in its text, I matched its settings to a step of my sweep and compared. Where it doesn’t, there was nothing to match.

Every reviewer number below is that outlet’s own measurement, linked at first use. My side comes from the Iometer 1.1.0 job files Part 1 walked through, run on the 9100 MAX in June 2016, the DCP1000 in May 2017 and the P4510 in February 2018. I ran each sweep once, and each step’s result is the one average Iometer recorded. Before each set of sweeps the drive got a 6h precondition at QD256. Random writes at that size came before the 4KiB and 8KiB tests, sequential writes before the 128KiB tests, and a run of the mix itself before each server mix. Each queue depthIOPS counts commands finished per second. Queue depth is how many commands are outstanding at once, and a drive needs a deeper queue to reach its top IOPS. step recorded for 300s after a 30s ramp. Where a review’s numbers differ from mine and its own page doesn’t say why, the reason given here is my inference and is worded as one.

Five reviews, five test setups

Each row lists what that review states about its setup. A cell that says “not stated” means the review’s text doesn’t give it.

TweakTown’s method is the closest to mine. It secure erases each drive, writes its whole capacity at least twice with 128KB sequential data, preconditions it at QD256 with the workload it’s about to measure, runs that workload for 5min at each queue depth, and plots the precondition one second at a time so a reader can see the drive settle. For this review its test system ran a Xeon E5-2699 V3 under Windows Server 2012 R2 with Intel’s own NVMe driver. My batch files share its last two steps, a precondition at QD256 and then a fixed time at each queue depth. They have no erase step, and only the server mixes start from a 2x sequential fill.

StorageReview’s synthetic tests run vdbench on Ubuntu 16.04.3. They fill the drive and then work inside a partition of 25% of its capacity. ServeTheHome ran fio on the same 2TB drive and set it against an Intel DC P3700. PC Perspective built its own suite and ran it on a desktop Core i7 clocked to 4.5GHz, choosing that over server hardware to push the processor harder. KitGuru used the consumer tools Benchmarking 101, Part 2 covers, on a desktop, and ran every test five times.

The tool name tells a reader the least of anything in that table. The drive’s state before the test, the share of the drive the test touches and the number of I/Os waiting are what set the result.

TweakTown and my P4510 runs agree within 4%

Each of TweakTown’s preconditioning runs on the 2TB drive ends in a steady-state average at QD256. I put each one next to the QD256 step of my matching sweep.

Every pair sits within about 4%. They aren’t taken at quite the same moment, since TweakTown’s value is a steady-state average over its precondition run and mine comes from the sweep step that followed my precondition. Its 4K random write precondition ran 16,000s, about 4.44h, and the 2TB drive settled at about 4,000s. Mine ran 6h on the clock, 21,600s or 5.4 times as long as the drive needed to settle in TweakTown’s run, so my sweep steps most likely started from the same settled state its averages describe. Two testers on different machines with different samples got the same write rate, which suggests that for steady-state writes on this drive the result belongs to the drive and the job far more than to the host.

TweakTown also reads its sweeps a particular way. It treats QD1 to QD9 as the operating region that matters, and it found both P4510 models doing close to twice the 9100 MAX’s Database result at QD1 and QD2, with the 9100 MAX ahead on random writes from QD4 up. My runs of the same two drives show the same crossover. At QD1 the P4510 led by 1.9 times in the Database mix (12,470 against 6,638 IOPS) and by 1.8 times in Email Server (13,904 against 7,518). At QD256 the order flipped, and the 9100 MAX led by 2.7 times (249,237 against 93,558) and 2.9 times (206,786 against 70,791). Which drive wins depends on the end of the sweep a reader weighs. TweakTown argues for the low end, and a server that keeps its queues deep would be better served by the other drive.

None of the review texts gives the queue depth where the 2TB P4510’s or the 9100 MAX’s 4KiB random writes stop climbing, and my sweeps show it. The P4510 reached 86,560 IOPS by QD4, 97.5% of its QD256 result, and every deeper step after that added almost nothing. The 9100 MAX kept climbing until QD16, where it ran 300,697 IOPS, 96.5% of its 311,479 at QD256, and its top rate is 3.5 times the P4510’s. At QD1 the order reverses. The P4510 finished each write in an average of 20.5µs and ran 47,586 IOPS, 2.9 times the 9100 MAX’s 16,531 IOPS at 0.060ms. At QD2 the P4510 still led with 70,906 IOPS against 37,597, and by QD4 the 9100 MAX had passed it with 112,028. That puts the crossover between QD2 and QD4, the same boundary TweakTown drew from its own runs. The two shapes suggest the P4510 is tuned to finish one write fast and the 9100 MAX to keep many writes in flight, though a sweep can’t show the firmware behind either.

StorageReview’s write result measures a different drive state

On reads, StorageReview and I land in the same place. Its vdbench run put the 2TB P4510 at a peak of 621,469 IOPS of 4K random reads at 205µs with 128 threads. My QD128 step ran 611,984 IOPS at 0.209ms, so its peak is 1.6% higher at a matching count of 128.

Writes split apart. StorageReview’s 4K random write peak with 64 threads was 233,299 IOPS at 540µs, 2.6 times my 88,313 IOPS at QD64, where my average response time was 0.724ms. Its own page points at the likely cause. Its test fills the whole drive and then works in a partition of 25% of the capacity, about 500GB on the 2TB model, and the review says plainly that this isn’t a steady-state test and that its sustained write figures come out higher for it. My drive had been written across its whole span for 6h first.

Writes confined to a quarter of the space likely leave the drive far more free blocks to clean up in the background, while random reads likely depend far less on that cleanup. Neither number is wrong. They answer different questions, and only StorageReview’s note about the partition tells a reader which question its chart answers.

KitGuru tested the DCP1000 as one RAID 0 volume

The DCP1000 is four 400GB NVMe drives on one card, and the two tests loaded it in different ways. KitGuru built the four into a RAID 0 array after wiping them, then ran an 8KB random test with 8 threads over a 16GB span. Reads peaked at 729,361 IOPS and writes at 421,412 IOPS. IOMeter is among the tools on KitGuru’s setup page, but its 8KB page doesn’t name the tool or a queue depth.

My job treated the card as four separate targets. Iometer ran 16 workers, each on one of the four drives, so at QD256 each drive held QD64. My 8KiB random read at QD256 ran 664,583 IOPS, or 5,444MB/s, and my 8KiB random write after preconditioning ran 101,951 IOPS.

KitGuru’s write result is 4.1 times mine, and its read result is 9.7% above mine. The 16GB span is 1% of the card’s 1.6TB, and the drives started freshly wiped, so I’d expect the writes never reached a point where the drive had to clean up behind itself. That would cover most of the write gap. The read gap could come from the small span, the RAID 0 striping or a deeper queue, and the review doesn’t give enough to tell which. Firmware may differ as well. KitGuru’s sample ran firmware E7FTD4.7, while Iometer recorded E7FT03.C on each of the four drives in mine.

Results that can’t be matched

ServeTheHome’s P4510 review shows its fio results only as charts, and its text gives no queue depth, job file or preconditioning for them. I couldn’t place any of its bars beside a step of my sweep. Its second page does something none of the other four reviews state they did. It runs the fio tests on an AMD EPYC system and an unnamed non-x86 platform to look for Intel-specific platform effects, and then tests an nginx WordPress content delivery workload. Those parts test the host and the application, which a job-level comparison can’t.

PC Perspective’s 9100 MAX review plots IOPS against read share in 10% steps at every queue depth. It dropped the File Server and Web Server mixes, calling them legacy workloads that date from 2000, while my job set still runs both. Its numbers live in its charts, though the text reads its upper-right 4KB point as 3GB/s of random access. My QD256 step on the same drive ran 758,048 IOPS of 4KiB random reads, about 3.1GB/s. That’s as close as its text lets me get, and reading values off its chart images would be guesswork.

How to read a review’s chart

Before comparing two drives on a published chart, look for each of these in the review’s text:

• Which job made each bar: tool, block size, read share, and queue depth, and whether that depth counts per thread, the way Benchmarking 101 counts it, or in total, as Part 1 of this series does.
• How the drive was prepared: freshly wiped, filled once, or written to steady state, and how long the precondition ran. Benchmarking 101, Part 1 covers why a fresh drive flatters, and Part 2 of this series covers how to tell when a drive has settled.
• How much of the drive the test touched: the whole drive, a partition, or a few gigabytes.
• Whether the value is a peak, the average of one step, or a steady-state average.
• Which host ran it, when the result sits near the interface’s limit or at QD1, the two places where I’d expect the processor and driver to show most.
• Which units it uses: IOPS or MB/s, and decimal or binary megabytes.

On most review charts only the units and the kind of value can be read off the chart itself. The drive’s state, the span and the host have to come from the text, when the text gives them at all.

A review that answers those questions in its text can be checked against any other run, and TweakTown’s came within 4% of mine most likely because its job was so close to mine. With the job files from Part 1, a reader can run the matching step on their own drive and set it beside any review that states its job.