SSD Benchmarking 101, Part 5: How Many Runs Make a Result

Part 2 of this series asked for at least three runs of each synthetic tool, and it treated runs that vary by more than 5% as a sign that the test environment isn’t under control, while Part 3 ran each PCMark 10 trace more than once. This part checks that rule against the same tests repeated on 84 retail drives from my test bench, and measures how far the passes actually landed from each other.

Most results barely moved. Sequential reads at queue depth 1 and the two trace-based scores moved enough, on enough drives, to cover the gap between two similar products.

The same test, run again

Each drive got 9 CrystalDiskMark 8.0.4 runs in one session, all on the Default profile with a 1GiB test and a count of 2. Most runs swept block size or thread count, and only some held the standard rows, so SEQ1M Q8T1 came up in two runs, SEQ1M Q1T1 in three and RND4K Q1T1 in four. A repeated row sat beside different tests in each run, so its spread may include some effect of those neighbors as well as plain run-to-run noise. I also ran PCMark 10 Storage Full and 3DMark Storage three times each on 83 of the drives.

Inside one run, CrystalDiskMark doesn’t average. Its source code does one extra internal pass first and throws it away, then shows the highest score of the rest, so each figure here is already the better of 2 internal passes. From here on, a pass means one run’s figure, and the spread is between those runs.

For every drive and test I took the highest pass minus the lowest, as a share of the median, and call that the spread here. A small spread means the passes all landed close together. When two drives differ by less than their own spreads, those runs can’t rank them.

Most results barely moved

Five of the six CrystalDiskMark rows had a median spread of 0.34% or less. SEQ1M Q8T1 reads were the steadiest, at 0.09% over two passes. RND4K Q1T1 reads had a median of 0.29% over four, and RND4K Q1T1 writes 0.34% over four. The widest drive on that write row reached 3.68%, and none went past 5%.

A median hides the odd drive, such as one whose RND4K Q1T1 reads spread 87.23% across four passes and another whose SEQ1M Q8T1 reads spread 49.58% across two. One pass on either could have landed anywhere in that range, and nothing in the screenshot would say so. A highest-minus-lowest spread also tends to grow with the number of passes, so a row measured four times can show more spread than one measured twice for that reason alone.

Sequential reads at QD1 moved the most

SEQ1M Q1T1 reads didn’t behave like the other rows. Their median spread was 2.6% over three passes, and the 90th percentile reached 13.89%. Of 84 drives, 45 spread more than 2% and 23 more than 5%.

My data can’t say why this row moves so much, but it’s the CrystalDiskMark result I’d trust least from one pass, and the one where a review’s screenshot and a reader’s own run are most likely to disagree.

Trace scores moved more than most synthetic rows

PCMark 10 Storage Full had a median spread of 1.37% over three passes, and 3DMark Storage 1.73%. At the 90th percentile they reached 3.65% and 5.43%. Of 83 drives, 24 spread more than 2% in PCMark 10 and 36 did in 3DMark. Only 4 and 12 spread more than 5%.

Part 3 treats these scores as the closest thing to daily use, and they still are, but each one needs its passes beside it.

When a gap between two drives is real

Set any gap against the spread before calling a winner. On 24 of my 83 drives, PCMark 10’s own passes spread more than 2%, so a 2% lead over another drive could come down to which pass got reported. On 23 of 84 drives, a 5% gap in SEQ1M Q1T1 reads sits inside the drive’s own spread.

That also tests Part 2’s rule. On most rows, passes within 5% of each other were the normal result. On SEQ1M Q1T1 reads, 23 of 84 drives went past it on the same bench, so on that row a wide spread doesn’t prove the setup is at fault.

These rules follow from that, for my own runs and for reading anyone else’s:

• Run each test more than once and report the median of the passes, which one slow or fast pass can’t move. My data shows the spread at two to four passes; it doesn’t show how many passes are enough.
• Report the spread beside the median, or at least the lowest and highest pass.
• Treat a gap smaller than either drive’s own spread as a tie.
• Rerun a test whose passes look far apart before trusting any one of them.

With those rules, any two results can be set side by side and the gap judged as real or as a tie. The enterprise series asks the same question of server drives, where steady state asks the same question of one long run instead of several short ones.