Flash Foundations companion · Walkthrough
Follow a Read
One 4KiB read, followed from the host to the NAND cell and back on a Samsung 990 Pro 2TB. Step through ten frames, switch the drive design or the page wear, and see where the time goes. For the other direction, follow a write.
The walkthrough
One 4KiB read, ten frames
Each frame lights the parts the read touches and moves it to its next stop. Drag the slider, use the arrow buttons or press play, and switch the drive design or the page wear to change only the frames that differ.
This walkthrough needs JavaScript. The full list of frames is below.
How to read it
NVMe is the command set the computer uses to talk to the drive over PCIe. The host puts each command in a submission queue and rings the doorbell, a register on the drive. The FTL is the controller firmware that maps each logical address to the place in the flash that holds it. ECC is the error-correcting code the controller uses to fix bits that flipped in a page it reads.
The scene is the one in Write of Passage, with the same part names. A dashed outline marks a teaching label, not a block of one drive. The strip of pages stands for one block of the die. The mint outline is the page to read and the filled page is the page being sensed.
The tag on the moving block shows what it is: CMD for the command, 4KiB for the data and CQE for the completion entry. The DRAM switch swaps the DRAM manager for the SRAM. The wear switch lights the retry loop in frame 8.
Click or tab to the walkthrough, then use the left and right arrow keys to step, Home and End to jump, and the space bar to play or pause. With reduced motion turned on, the block jumps from stop to stop instead of gliding.
Flash Foundations, Part 5 covers the controller, Part 6 the flash translation layer and Part 8 error correction.
The bench
Where does the time go?
Pick a page type, a wear point and a drive design. The bar adds up the stages of one read, and the marker is what a 990 Pro measured. A link to a setup can be shared with the Copy link button.
The bench needs JavaScript.
How to read it
Sensing the page is the largest published cost of a fresh read, and wear multiplies it, because every read-retry step repeats the sense, the transfer and the decode. The study behind the wear points tested 160 TLC chips. Pages a year old after 2,000 program/erase cycles averaged 19.9 retry steps, and the read took 21 times as long.
The marker is the 990 Pro’s random 4KiB read at queue depth 1, which averaged 40.73µs. Queue depth 1 means the host waits for each read before it sends the next. A fresh lower-page estimate comes to about 70.4µs, longer than the drive measured. The runs were fresh, on an empty drive, so the benchmark most likely read freshly written data from the SLC cache. That is the tester’s judgment, because the benchmark does not report the mode.
Same test, different answer
On a Phison E21T engineering sample with SK hynix V7 TLC, 1TB, the first run’s random 4KiB read averaged 59.81µs. The next three runs, about four minutes apart, averaged 46.07µs. The benchmark does not say which mode each run read from, so the lab shows the split without naming a cause.
The same walkthrough as text
All ten frames
Host command
The host puts a read command, 64 bytes long, in a queue in its own memory. It names the logical block address to read.
Doorbell
The host then writes one register on the drive, the doorbell, to say a command is waiting.
Fetch
The NVMe engine on the drive fetches the command across PCIe and reads which logical block the host wants.
Map lookup
The flash translation layer, or FTL, keeps a map of where each logical block lives. A core looks up the address in the map held in DRAM and finds the physical page.
Channel and die
The controller sends the read down the channel of the die that holds the page. Dies on a channel share its wires, and chip enable says which one listens.
Page sense
The die senses a whole 16KiB page into its page buffer, which takes tR, 40µs on average. The lower and upper pages use two read levels and the middle page uses three, so their times differ.
ECC decode
The page crosses the channel to the controller, where ECC decodes it. In the research setup this lab borrows, one decode takes 20µs.
Read-retry, not needed
A fresh page decodes on the first try, so there is no retry. The loop stays dark until the page is worn.
Transfer back
DMA copies the 4KiB across PCIe into host memory. The link carries up to 7.88GB/s each way before packet overhead, about 0.52µs for this read.
Completion
The drive posts a completion entry of at least 16 bytes to a queue in host memory, and the host picks it up. The read is done.
On a DRAM-less drive
Choose DRAM-less in the walkthrough to see this design. Frame 4 reads differently, as below, and lights the SRAM instead of the DRAM manager. The other frames are the same.
Map lookup, entry cached
A DRAM-less drive caches only part of the map. This entry is cached, so the lookup needs no extra flash read.
Map lookup, extra read
A DRAM-less drive caches only part of the map. When the entry is missing, it reads that piece of the map from flash first, an extra flash read before the data read.
On a worn page
Choose Worn in the walkthrough to see this. Frame 8 reads differently and lights the retry loop. The other frames are the same.
Read-retry loop
When a decode fails, the controller senses the page again at shifted voltages and decodes again. Every retry repeats the sense, the transfer and the decode, and worn pages need many.
How the model works
The walkthrough is data. Each of the ten frames has a title, a caption and the list of parts it lights, using the part names of the Write of Passage scene. Where each part is drawn is that lab’s. The stop of the read in each frame and its route between frames are this lab’s own.
The bench adds stages. The measured sequential 4KiB read, 14.05µs, stands in for the host, queue and controller time. On top of it go the die’s page read time, 40µs, scaled by page type, the channel transfer, an ECC decode of 20µs and any read-retry steps.
Inside the die, a sequencer runs the read, and latches beside the bit lines hold the sensed bits until the page leaves. Each plane senses its own pages by comparing cells against read levels, so tR depends on page type. A channel carries the page to the controller.
The lower page takes about 34.3µs and the middle page about 51.4µs. The channel moves 4KiB in about 2.0µs. On a DRAM-less drive that misses its map cache, one more page read adds about 62.0µs.
The gap between the estimate and the measurement is shown as it is, even when the estimate is longer. The Atlas shows the parts these stops pass through in 3D: open the Atlas at the cell.
Everything runs in your browser. The page stores nothing itself. Google Analytics counts every page on this site and sees only the page path.
What it leaves out
- The ECC time comes from the simulated drive in a read-retry study, and the retry counts from that study’s tests of other 3D TLC chips, not from this drive.
- The page-type split is a model built on the die’s single printed figure.
- The measured sequential read is a floor for the host, queue and controller time only by inference. The benchmark cannot check it.
- The benchmark does not say whether a read came from the SLC cache or from TLC.
- The map read on a DRAM-less miss is a model. It takes the die’s average read time, the same transfer and the same decode, and a hit in the host memory buffer adds nothing because no source gives its cost.
- Wear uses only the points the read-retry study prints, and two of them are “at least” counts. Nothing is filled in between.
- The scene is a teaching datapath, not one product. It does not draw queue depth, read size or read caching.
- It does not draw a write. The program loop (ISPP), force unit access (FUA) and the volatile write cache (write cache) are in Write of Passage.
- The sources and method name the paper, datasheet or measurement behind every number.