SSD Benchmarking 201, Part 5: One Controller at Three Capacities

Part 4 of this series set published reviews beside my own runs and found that a result only transfers when the job behind it matches. Capacity has to match too. An enterprise SSD line usually offers several capacities on one controller, and a review tests one or two of them. In July 2022 I ran the same Iometer job on an E18DC-based enterprise NVMe engineering sample at 480GB, 960GB and 1,920GB, so capacity is the only thing that changes from one set of numbers to the next.

At queue depth (QD)IOPS counts commands finished per second. Queue depth is how many commands are outstanding at once, and a drive needs a deeper queue to reach its top IOPS. 1, reads ran the same at every capacity. Writes and deep-queue reads grew with it, by close to double from 480GB to 960GB and by about a third again from 960GB to 1,920GB. Every result here is mine, recomputed from the raw Iometer rows of each run. I name the sample by its controller and capacity only. It ran on a different test system from the older drives in Part 3, so its numbers stay apart from theirs.

The same job at three capacities

Each capacity got the same steps. A 3h 4KiB random write precondition at QD256, with 16 workers x 16 outstanding I/Os, left the drive written over before anything was measured. After it came the same 4KiB sweep that Part 1 describes for this sample, 11 read shares at each of 9 queue depths. A 128KiB sequential precondition and sweep ran separately. QD here is the total number of I/Os in flight, as in Part 1.

Phison’s E18DI datasheet, for the industrial version of the E18 family, lists up to 8 NAND channels with 32 chip enables behind a PCIe 4.0 x4 link. A bigger capacity on the same controller puts more flash dies behind those channels. I don’t have this sample’s die count, so this page reads the results without one.

The Parallelism Lab models the same effect with die count and capacity you can set. The 3h preconditions already showed the split, with 4KiB random writes averaging 17,850 IOPS at 480GB, 36,474 at 960GB and 56,247 at 1,920GB.

QD1 reads didn’t change with capacity

A 4KiB random read at QD1 waits on one die to sense its page and send it over one channel. With nothing else in flight, extra dies have no second read to work on, however many a drive has. The sample ran 13,095 IOPS at 480GB, 13,319 at 960GB and 13,078 at 1,920GB, each at an average response time of 0.075 to 0.076ms.

Capacity starts to count once more reads are waiting. At QD4 the three were still within 10% of each other, at 45,957, 49,301 and 50,448 IOPS. By QD32 the 1,920GB drive ran 263,839 IOPS, 1.99 times the 480GB drive’s 132,340. By QD256 the three reached 264,124, 496,209 and 646,007 IOPS.

With more reads waiting, a controller can keep several dies busy at once, which is the interleaving that Agrawal and co-authors show lets reads and writes on independent dies overlap. Flash Foundations, Part 5 covers how channels and dies split that work. An average QD256 read took 0.969ms at 480GB and 0.396ms at 1,920GB, most likely because each read queued behind fewer others on its die.

Random writes followed capacity, not queue depth

After the precondition, 4KiB random writes hardly moved with depth at any capacity. The 480GB drive ran 17,825 IOPS at QD1 and 18,440 at QD256, the 960GB drive 35,337 and 36,046, and the 1,920GB drive 48,271 and 48,522.

On a drive the precondition has written over, new writes most likely share the dies with the controller’s own cleanup of old blocks. More I/Os in flight don’t add dies, so they didn’t add write speed, but more capacity did. From 480GB to 960GB the QD256 rate rose 1.95 times, and from 960GB to 1,920GB it rose 1.35 times.

Extra depth turned into waiting instead, and at QD256 an average 4KiB write took 13.882ms at 480GB, 7.102ms at 960GB and 5.276ms at 1,920GB. A job that piles on depth to find a drive’s write speed mostly measures how long its queue is.

Mixed loads follow the writes, and at 70% reads and QD256 the sample ran 54,110 IOPS at 480GB, 102,365 at 960GB and 136,659 at 1,920GB. Its writes kept each capacity far below its all-read result, as Part 3 showed for the 1,920GB drive across the whole read mix.

Sequential writes doubled with each capacity step

128KiB sequential writes held one rate at every depth from QD1 to QD256. They ran 368.1 to 369.3MB/s at 480GB, 784.7 to 788.1MB/s at 960GB and 1,600.8 to 1,605.8MB/s at 1,920GB. Each step up in capacity doubled the rate, 2.12 times and then 2.04 times at QD256. Once a drive is writing at its sustained rate, a queue gives it nothing more to do.

Sequential reads ran into a different limit. The 480GB drive peaked at 4,169.4MB/s at QD128. The 960GB and 1,920GB drives both peaked above 7GB/s, at 7,149.4MB/s and 7,118.4MB/s, close to the 7.9GB/s a PCIe 4.0 x4 link carries in each direction. At 960GB and 1,920GB the host link was most likely the limit, not the flash. At QD1 all three read 128KiB blocks at 1,232.3 to 1,345.7MB/s, with the 1,920GB drive the slowest.

What one capacity tells you about another

A result belongs to the capacity it was measured on. On this design, a review of the 1,920GB model would overstate the 480GB model’s random writes 2.63 times, its deep-queue random reads 2.45 times and its sequential writes 4.34 times, while getting its QD1 reads right. Sequential reads would transfer down to 960GB and no further.

Read any enterprise review’s write and deep-queue numbers as describing its capacity only, since light, low-depth reads were the one result that carried across all three sizes here. Before trusting a chart of the largest model, check which capacity the review tested, and test or find results for the capacity you plan to buy. That puts capacity on the list of settings a fair comparison has to match, next to the job, the drive’s state and the host.