Eyes on Flash/ Driven by Insight
Contact

Search Eyes on Flash

Tip: press / or Ctrl+K on any page to open search.

Type to search.

    Flash Foundations companion · Walkthrough

    Write of Passage

    An app saves 4KB. Fourteen frames later it is a pattern of charge in a NAND cell. Step through the trip, or press play, and see where the write waits, where it is copied and where the drive's map changes.

    Start the walkthrough The timing bench All frames as text How it works

    The walkthrough

    One 4KB write, fourteen frames

    Each frame lights the parts the write touches and moves the block to its next stop. Choose a drive with DRAM or a DRAM-less drive, then drag the slider, use the arrow buttons, or press play. Play waits on each frame about as long as it takes to read the caption, and it stops on the last frame.

    This walkthrough needs JavaScript. The full list of frames is below.

    How to read it

    The scene has three regions. The host holds the application, its command queue and the completion entry. The controller holds the blocks the write passes through, from the PCIe interface to the channels. The NAND region holds one die and the open block, drawn as a strip of eight pages.

    A part with a dashed outline is a teaching label, not a block of a particular drive. In the strip of pages, a dashed copper outline is the old copy of the data, a mint outline is the page the controller has reserved, and a solid mint page holds the new data.

    Click or tab to the walkthrough, then use the left and right arrow keys to step, Home and End to jump, and the space bar to play or pause. Play moves on one frame at once, then waits about as long as the caption takes to read. With reduced motion turned on, play still works but the block jumps from stop to stop instead of gliding. On a small screen the scene is cut to a window around the block.

    Frames 6 and 7 are the flash translation layer. Flash Foundations, Part 6 explains the map, and the FTL & GC Sandbox lets you change how a drive places writes. Frame 9 is error correction, which Part 8 covers.

    The bench

    How long does the write take?

    A drive can say a write is done long before the data is in the cell. Change the drive design, cell type, channel speed, page size and host link, and see both times. A configuration can be shared with the Copy link button.

    The bench needs JavaScript.

    How to read it

    The first segment of the bar is the time the drive takes to say the write is complete. It is one second divided by one drive's rated random write rate at queue depth 1, which is a drive with a write cache doing a burst. The data is not in the cell yet. The rest of the bar is what still has to happen: the page crosses the channel to the die, and the cells are programmed. A write sent with FUA, or followed by a flush, is not complete until the end of the bar.

    Programming is almost all of it. Try a faster channel and the bar barely shrinks. Try SLC mode and it shrinks a lot. On a DRAM-less drive the map cache hit rate adds a flash read for every lookup that misses, which is why a small cache hurts. The gap between the two markers is the window Power Loss Lab lets you cut the power inside.

    The model is addition. It leaves out the time firmware and the controller take between the stages, because no public figure exists, so real drives are slower than the total. The older die and slow channel preset is a 2022 datasheet part, not a current one.

    The same walkthrough as text

    All fourteen frames

    1. Host issues the write

      An app asks to save 4KB. The file system turns that into a write to a logical block address, and the drive never sees the file name.

    2. Command queued

      The host puts an NVMe write command, 64 bytes, in a queue in its own memory.

    3. Drive notices

      It then writes one register on the drive, the doorbell, to say there is work.

    4. Controller fetches

      The NVMe engine fetches the command, then DMA copies the 4KB across PCIe into the controller's Host FIFO.

    5. Staged in DRAM

      On a drive with a DRAM cache, the data usually passes through DRAM next. A DRAM-less drive uses other memory. Either way it is not in NAND yet.

    6. Map lookup

      The FTL keeps a map of where each logical block lives. A core looks up this address and finds the old physical page.

    7. New page chosen

      Flash pages cannot be overwritten in place. The FTL picks a fresh page, points the map at it, and marks the old page invalid.

    8. Gathered into a program unit

      A NAND page is much larger than 4KB (16KiB on one TLC part), so the controller gathers more writes to fill one program unit.

    9. Protected

      The ECC encoder adds LDPC parity so errors can be corrected when the data is read back.

    10. Scheduled

      A CRC guards the unit on its way out, and the channel controller picks one channel and one chip enable.

    11. Over the channel

      The unit crosses the channel to the chosen die. Dies on a channel share its wires, and chip enable says which one listens.

    12. Programmed

      Programming is a loop of small charge pulses, each followed by a verify read. Published TLC figures run from about 700 microseconds (a simulator) to 1.9 milliseconds (an older radiation-tolerant part), against about 10 microseconds to move 16KiB.

    13. Made durable

      The drive saves enough mapping information in NAND to rebuild the map after a power cut. Only a write sent with force unit access, or followed by a Flush, is reported done here. A normal write was reported done earlier, from the volatile cache.

    14. The stale copy

      The old page stays in the block as invalid data. Garbage collection later copies any still-valid pages out of the block and erases the whole block.

    On a DRAM-less drive

    Choose DRAM-less under the title of the walkthrough to see this design. Frames 5 and 6 read differently, as below, and frame 8 lights the SRAM instead of the DRAM manager. The other frames are the same.

    1. Held in a small buffer

      A DRAM-less drive has no DRAM to wait in, so it holds the data in a small buffer on the controller. In the drives tested, that buffer was not in the host's memory. It is not in NAND yet.

    2. Map lookup

      The full map lives in flash, and only part of it is cached on the controller. If the entry is not cached, the drive reads that piece of the map from flash first, an extra flash read before the data read.

    How the model works

    The walkthrough is data. Each of the fourteen frames has a title, a caption and a list of the parts it lights, and the part names follow the Atlas SSD datapath. Where each part is drawn, where the block stops in each frame and the route it takes between frames sit in a separate file, so a newer Atlas model can replace the drawing without touching the captions or their order.

    The block moves along right-angle routes between neighboring frames. Jump to a distant frame and it simply appears there. The eight-page strip in the NAND region follows the write: from frame 7 the old page is marked invalid and a new page is reserved, from frame 12 the new page holds the data, and on frame 14 the old copy is picked out because it is still in the block.

    Everything runs in your browser. The page stores nothing itself. Google Analytics counts every page on this site and sees only the page path.

    What it leaves out

    • It is a teaching datapath, not one product. A real controller draws, orders and names its blocks differently, and a drive can differ in many ways. The DRAM-less design shown is the one the sources describe: a small buffer and a partly cached map on the controller, with the full map in flash.
    • Where a DRAM-less drive holds queued write data is not settled in the sources, and nothing says how large its SRAM is. The walkthrough draws the buffer as SRAM, as the Atlas does, and its claim is filed as unverified. Whether a drive borrows host memory (HMB) varies, and the host can take it back.
    • The bench adds up stage times and nothing else. The 12.5 microseconds for the drive to say it is done comes from one drive's rated random write rate and the datasheet gives no block size. The map cache hit rate is a teaching value. A multi-level page read uses the datasheet's single-plane TLC figure for MLC and TLC alike, and the TLC and MLC program times are from a research simulator.
    • The four channels with two chip enables each, in frame 11, are teaching counts and not the layout of any drive.
    • Host FIFO and the CRC at the NAND FIFO are illustrative labels from the Atlas that follow an older controller data path brief. They are not blocks of a particular drive.
    • The page numbers in the open block and the threshold voltage curves in frame 12 are illustrative, not measurements.
    • The programming times quoted in frame 12 are a range. The low end is a setting in a research simulator rather than a measurement of a chip, and the high end is a typical figure from the datasheet of an older die.
    • It does not draw wear leveling, an SLC cache, encryption, retries or the cost of garbage collection. Frame 14 only points at garbage collection.
    • The sources and method name the paper, datasheet or specification behind every figure in a caption.

    Keep going