SSD Benchmarking 201, Part 3: Three Enterprise SSDs on One Test Method

Part 1 of this series built a repeatable Iometer job, and Part 2 covered when a drive is ready to measure and how to read its latency. Here that job runs on three enterprise SSDs I tested between 2016 and 2018: a Micron 9100 MAX 2.4TB, a Kingston DCP1000 1.6TB, and an Intel DC P4510 2TB. Each one comes out ahead somewhere. Where it leads depends on the queue depth (QD)IOPS counts commands finished per second. Queue depth is how many commands are outstanding at once, and a drive needs a deeper queue to reach its top IOPS. and on how much of the load is writes.

All the results are mine, recomputed from the raw Iometer rows of each run instead of the summary sheets I built at the time. They’re given in I/O operations per second (IOPS), with the average response time beside them wherever it changes the reading. Two wider sets follow the three. One checks workers against depth on 16 drives from the same years, and one takes eight drives from 2019 across the whole read mix. A 2022 job on an engineering sample comes last.

Three drives, one job file

All three drives ran on my test system with the set of Iometer 1.1.0 configs that Part 1 walks through. Before its 4KiB and 8KiB sweeps, each drive got a 6h random write precondition at QD256. Each queue depth step then recorded for 300s after a 30s ramp. QD on this page always means the total number of I/Os the drive has in flight, as in Part 1. From QD1 to QD8 the job adds workers at one outstanding I/O each, and from QD16 to QD256 it holds 16 workers and raises each one’s outstanding I/Os. Each sweep number on this page comes from a single 300s step, and only my SNIA-style results average more than one reading, the mean of rounds 22 to 25 of 25. Each of the 56 access specs in a round ran for 60s at QD256, and the rounds followed a 2h sequential fill.

Iometer wrote each drive’s model and firmware strings into its results files, along with the number of targets it saw.

Source: my Iometer 1.1.0 results files.

Kingston’s datasheet describes the DCP1000 as four M.2 SSDs on one card and rates the card on the four combined. Iometer saw them as four 400GB targets, so every DCP1000 number on this page is the sum across all four, and the last section looks at what that does to a comparison.

The 9100 MAX’s 8KiB random read results on this page start at QD16.

At QD1 the newest drive answers first

QD1 stands for a single thread that waits for each I/O before it sends the next. Benchmarking 101, Part 2 ties random reads at QD1 to boot and app launches on a desktop, and on a server the same depth is one request stream with nothing queued behind it. With one I/O in flight, IOPS is close to the inverse of the average response time, so the drive that answers each request soonest wins. The P4510 returned 4KiB random reads in 0.079ms on average, for 12,653 IOPS. On the DCP1000 a read took 0.106ms for 9,408 IOPS, from one of its four drives, and the 9100 MAX took 0.153ms for 6,491. Writes at QD1 came back sooner than reads on all three. They took 0.020ms on the P4510 for 47,586 IOPS, 0.039ms on the DCP1000 for 25,313, and 0.060ms on the 9100 MAX for 16,531.

At 67% reads, the 8KiB Database mix keeps the same order at low depth. At QD1 the P4510 ran 12,470 IOPS to the 9100 MAX’s 6,638, almost twice as many. Between QD8 and QD16 the 9100 MAX caught up. At QD8 the P4510 still led, 53,033 to 49,424, and at QD16 the 9100 MAX led 79,204 to 72,595.

For reads and the read-heavy Database mix, a host that seldom keeps more than eight I/Os in flight gets more from the P4510. A host that stacks deeper queues gets more from the 9100 MAX, and the next section shows how much more once writes join the load.

Writes split the drives at depth

Each drive ran seven read shares at every block size in the SNIA-style rounds, all at QD256. The chart shows the 4KiB results at four of them.

Moving 5% of the I/O to writes cost the P4510 43% of its all-read rate and the DCP1000 41%. For the 9100 MAX the same change cost 16%. With every I/O a write, the 9100 MAX ran 3.5 times the P4510’s rate. At 8KiB the gap held, 169,261 write IOPS against 48,897, while their all-read results sat close together at 377,987 and 361,567.

My QD sweep shows where each drive’s writes stop climbing. The P4510’s stopped at QD8, as the next section shows, and the other two flattened deeper in the sweep. The 9100 MAX reached 300,697 IOPS at QD16 and 311,479 at QD256, and the DCP1000 reached 177,199 at QD32 and 182,622 at QD256, with one lower step between them that I can’t explain. Past those depths each of the three gained under 4%, so the depth where a drive’s writes stop climbing tells more about it than its QD256 number.

Four server mixes tell the same story. Database is 8KiB at 67% reads, Email Server is 8KiB at 50%, File Server spreads eight sizes from 512B to 64KiB at 80%, and Web Server spreads ten sizes from 512B to 512KiB with no writes at all.

Source: my Iometer QD sweeps, IOPS at 16 workers with 16 outstanding I/Os each.

On Web Server the P4510 came within 2% of the 9100 MAX. In the three mixes that write, the 9100 MAX ran 2.3 to 2.9 times the P4510’s rate. So the choice between those two comes down to the write share. For a read-only service the P4510 gives up almost nothing, and for anything that writes under deep queues the 9100 MAX is the better drive by a wide margin.

Latency is the other half of every IOPS number

Part 1 checked the 9100 MAX’s QD256 read run against Little’s law, which says the I/Os in flight equal IOPS times the average response time. Both other drives pass it. The P4510 ran 657,240 IOPS at 0.389ms, and 256 / 0.389ms gives 658,098, within 0.2%. For the DCP1000, 1,089,673 IOPS at 0.235ms sits within 0.1% of the 1,089,362 the law gives.

Read the other way, the law says what happens past a drive’s peak. Once IOPS stops rising, the average response time is the queue depth divided by a fixed rate, so each doubling of the queue doubles the wait. My P4510 4KiB write runs show it. From QD16 to QD256 their rate rose 1.6%, from 87,403 to 88,758 IOPS, while the average response time went from 0.183ms to 2.884ms, about 16 times as long. Part 1 found the 9100 MAX’s writes doing the same past QD16. A write result at QD256 on either drive mostly measures how long the queue was.

Sequential reads level off the same way. In 128KiB sequential reads the 9100 MAX reached 3,144MB/s at QD32 and 3,196MB/s at QD256, and the P4510 reached 3,306MB/s and 3,310MB/s. By QD256 the average response time had grown to 10.497ms on the 9100 MAX and 10.137ms on the P4510, for less than 2% more data than at QD32.

Iometer logs the host’s CPU load too, and at these rates it isn’t small. My test system ran at 28.1% CPU during the 9100 MAX’s QD256 read step, 26.0% during the P4510’s, and 39.5% during the DCP1000’s 1,089,673 IOPS. None of these runs shows the host holding a drive back, and the CPU column is what would show it, so it belongs beside the IOPS in any result near a million.

What four drives on one card do to the numbers

Iometer’s results show where each worker sent its I/O. On the DCP1000, worker 1 went to the first target, worker 2 to the second, and so on around the four, with worker 5 back on the first. At QD1 one 400GB drive did all the work, for 9,408 IOPS. At QD4 each drive held one I/O, and the card ran 37,695 IOPS, four times the QD1 rate within 0.2%.

Average read response time was 0.106ms at QD1 and still 0.106ms at QD16, where each drive held four I/Os and the card ran 150,532 IOPS. At QD256 each drive held 64 and delivered 268,206 to 276,620 IOPS. Together they ran 1,089,673. A QD label on this card is four times the depth any one of its drives saw. QD16 on the DCP1000 is four drives at QD4, and setting it beside QD16 on a single drive compares two different loads.

Kingston rates the 1.6TB card at 1,100,000 IOPS of 4K random reads and 200,000 of writes at steady state, measured across its four drives. My SNIA-style rounds came in at 1,085,819 and 211,167. Those writes still trailed the 9100 MAX on its own, which wrote 337,722 IOPS at the same settings. With one worker per drive, the card’s 128KiB sequential reads reached 7,152MB/s at QD32 against its rated 6,800MB/s.

My runs addressed each of the four drives directly. Kingston’s datasheet says the card can be configured as a RAID volume through host software, and a stripe between the job and the drives changes which drive each I/O reaches, so a RAID setup needs runs of its own before its numbers can stand beside these.

Workers or depth, two ways to reach QD256

Every drive in my 2016 to 2018 set ran QD256 4KiB random writes twice. The 6h precondition used 8 workers at 32 outstanding I/Os each, and the QD256 step of the sweep that followed used 16 workers at 16. A controller that handled one shape of load better than the other would split those two results. I set the last hour of each precondition, summed second by second from Iometer’s per-worker log, against the sweep step on 16 drives.

On 15 of the 16 the two agreed within 2.3%. The 9100 MAX ran 311,309 IOPS in its precondition’s last hour and 311,479 in the sweep, and a Toshiba HK4R 1,920GB ran 17,298 and 17,297. Only the DCP1000 differed, with the sweep 5.5% higher. Its precondition was still sliding, from 184,919 IOPS in the first hour to 173,145 in the last, and on that card both layouts give each of its four drives QD64 anyway. The two runs in each pair come from different points in a drive’s life, so they only agree once a drive is steady. Where a drive was steady, the flash set the write rate and the host’s split of QD256 made no difference I could measure.

Read results seem to favor added workers, but the sweep can’t tell. It doubles the load twice in a row. QD8 to QD16 doubles the workers, and QD16 to QD32 doubles each worker’s outstanding I/Os. Reads on the 9100 MAX went from 66,444 to 132,818 to 250,164 IOPS. Nine of the 16 drives gained 1.83 to 2.00 times from the added workers and 1.61 to 1.91 times from the added depth. The other seven stopped at QD32, with gains of 1.48 to 1.64 times and then 1.00 to 1.17 times. Those seven are three Micron drives (5100 ECO, 5100 MAX and M510DC) plus the Kingston DC400, Samsung SM863, and Toshiba HK4E and HK4R.

Because the depth step always comes second and starts from twice the load, a smaller gain there says where a drive ran out of parallel work and nothing about workers against depth. Only a test that holds the total fixed and trades one for the other answers that question. My QD256 write pair is that test, and on these 16 drives the answer was no difference.

Optane, low-latency flash and TLC NAND across the read mix

In 2019 I ran a newer job on eight drives, so its results stay apart from the 2016 to 2018 set. After a 4KiB random write precondition at QD128, it stepped the read share from 0 to 100% in 10% steps at every queue depth from QD1 to QD256, 99 steps per drive. Three of the drives use three kinds of memory. Intel’s 905P brief lists 3D XPoint for the Optane SSD 905P, Samsung’s 983 ZET brief lists its own Low Latency V-NAND, and Micron’s 9300 brief lists 64-layer 3D TLC NAND. I tested the 905P and the 983 ZET at 960GB and the 9300 PRO at 7.68TB.

The memory shows up in QD1 reads. A 4KiB read took 0.0168ms on average on the 905P, 0.0241ms on the 983 ZET, and 0.113ms on the 9300 PRO. Writes at QD1 hid the difference, at 0.0184ms, 0.0187ms and 0.0189ms. Across the mix at QD1 the 905P barely moved, from 53,499 IOPS with all writes to 58,854 with all reads. Over the same range at QD1 the 983 ZET slid from 52,632 to 41,111, and the 9300 PRO fell from 52,275 to 8,825, about six times lower, as reads took over.

Deep queues turn the all-read order around. With every request a read at QD256, the 9300 PRO ran 847,866 IOPS and the 983 ZET 789,649, against 579,632 on the 905P. With all writes the 905P ran 583,711 IOPS against 147,277 on the 9300 PRO and 79,503 on the 983 ZET. The 905P’s lowest point at that depth came at 40% reads, 511,459 IOPS or about 12% under its all-write rate, while both flash drives gained with every step toward all reads. At QD16 and 70% reads, the 905P ran 530,219 IOPS against 182,537 on the 983 ZET and 148,968 on the 9300 PRO.

Four other drives in the 2019 set took 0.0925 to 0.1167ms per QD1 read, close to the 9300 PRO. That group is an Intel DC P3520 1.2TB and a Toshiba PX04 3.2TB, along with a Samsung 860 DCT 3.84TB and a Toshiba HK4E 800GB. Optane earns its place on mixed loads and single requests, where it answered reads five to seven times sooner than the 9300 PRO and those four. Once enough all-read requests are queued, both flash drives pass it.

A 2022 sample on a newer job

My 2022 job for an E18DC-based enterprise NVMe engineering sample at 1,920GB ran on a different test system, so its numbers stay apart from the drives above. After a 3h 4KiB random write precondition at QD256, it swept 11 read shares from 0 to 100% at 9 queue depths. At QD256 the sample read 646,007 IOPS at 0.396ms and wrote 48,522 at 5.276ms.

Like my 2019 job, this one steps through every 10% read share. At 90% reads the sample ran 319,837 IOPS, about half its all-read rate, and no other 10% step at that depth cost as much. On this drive at QD256, the first writes added to a read load did more damage than any later ones.

What a comparison has to match

Part 4 sets published reviews of enterprise drives beside my runs. Before I compare a review’s number with one of mine, I match its block size, read share, total queue depth, and preconditioning.

The DCP1000 adds one more item to that list, how the depth is split across workers and targets, because QD4 on that card was four drives at QD1. A review that states all five can sit beside my runs number for number, and that is the test Part 4 puts each review through.