Back to the Follow a Read

Sources

Generated from the Flash Library ledger for follow-a-read. Do not edit by hand: file new sources with flash-library add-source and claims with flash-library add-claim, then re-export.

Status: primary (the original paper, spec or measurement), secondary (a review or a vendor summary), book-lead (a textbook pointed to it), model (a modeled value), illustrative (a teaching value), own-data (measured here), unverified (no source yet).

Claim Value Status Source
C-FAR-001 Samsung 990 Pro 2TB, CrystalDiskMark 8.0.4, 1 GiB file on an empty drive: RND 4KiB read at Q1T1 averages 40.73 us (24,507 IOPS) in Default mode; three more Default runs give 40.73, 40.74 and 40.74 us. Full Power Mode gives 39.73 us (CrystalDiskMark_20221011141422.txt). Mean latency as seen by the application, so the Windows storage stack is included. One drive. 40.73 us; 24,507 IOPS (Default); 39.73 us (Full Power Mode) own-data _own-data/z690-samsung-990-pro-2tb-cdm/Default/CrystalDiskMark_20221008163342.txt
C-FAR-003 Random minus sequential 4KiB read at Q1T1 on the Samsung 990 Pro 2TB: 40.73 - 14.05 = 26.7 us, the extra time a read takes when it must go to the NAND for a fresh page. Arithmetic on two own-data runs (CrystalDiskMark_20221008163342.txt and CrystalDiskMark_20221008165630.txt). It is less than the die’s published 40 us TLC tR, so these reads were not full TLC page reads; most likely the freshly written test file sat in the SLC cache (an inference, not measured). 26.7 us model _own-data/z690-samsung-990-pro-2tb-cdm/Default/CrystalDiskMark_20221008165630.txt
C-FAR-004 Samsung 990 Pro 2TB, CrystalDiskMark, Default mode, RND read at Q1T1: 4KiB 40.74 us, 8KiB 44.93 us, 16KiB 51.87 us, 32KiB 66.99 us. Runs CrystalDiskMark_20221008164854.txt and CrystalDiskMark_20221008165242.txt continue the sweep: 64KiB 95.69 us, 128KiB 102.20 us, 256KiB 118.52 us, 512KiB 151.08 us, 1MiB 227.74 us, 2MiB 384.50 us, 4MiB 702.51 us, 8MiB 1,341.45 us. One drive. 44.93 / 51.87 / 66.99 us (8 / 16 / 32KiB) own-data _own-data/z690-samsung-990-pro-2tb-cdm/Default/CrystalDiskMark_20221008164506.txt
C-FAR-005 Samsung 990 Pro 2TB, CrystalDiskMark, Default mode, RND 4KiB read: Q1T2 41.62 us (47,923 IOPS), Q1T3 41.51 us (72,071 IOPS), Q16T16 176.41 us (1,400,152 IOPS). More reads in flight raise IOPS while each read takes about as long or longer. One drive. Q1T2 41.62 us / 47,923 IOPS; Q1T3 41.51 us / 72,071 IOPS; Q16T16 176.41 us / 1,400,152 IOPS own-data _own-data/z690-samsung-990-pro-2tb-cdm/Default/CrystalDiskMark_20221008164118.txt
C-FAR-006 Samsung 990 Pro 2TB, CrystalDiskMark, Default mode: RND 4KiB read at Q32T1 averages 160.07 us (193,496 IOPS), against 40.73 us (24,507 IOPS) at Q1T1: about 7.9 times the IOPS for about 3.9 times the latency. One drive. 160.07 us; 193,496 IOPS own-data _own-data/z690-samsung-990-pro-2tb-cdm/Default/CrystalDiskMark_20221008163342.txt
C-FAR-007 Tom’s Hardware review of the Samsung 990 Pro (2022-10-18): the drive uses Samsung’s Pascal controller (8nm, Arm-based) and 176-layer V7 TLC; on the 2TB model the packages (K9DVGY8JRD-DCK0) carry 512Gb dies; DRAM is LPDDR4. The review does not give the channel count. Pascal controller; 176-layer V7 TLC; 512Gb dies on 2TB; LPDDR4 secondary Samsung 990 Pro SSD Review: The Return of the King (Tom, 2022-10-18) https://www.tomshardware.com/reviews/samsung-990-pro-ssd-review
C-FAR-008 Tom’s Hardware 990 Pro review: Samsung told the reviewer Pascal can take advantage of read caching, without saying how; the reviewer notes it is common to keep hot data in pSLC. Supports, but does not prove, that a freshly written benchmark file is read from the SLC cache. read caching (not described) secondary Samsung 990 Pro SSD Review: The Return of the King (Tom, 2022-10-18) https://www.tomshardware.com/reviews/samsung-990-pro-ssd-review
C-FAR-009 A 2TB drive (16,000 Gb) built from 512Gb dies needs 32 dies (16,384 Gb raw, the extra is spare area). Arithmetic on the review’s die size; the die count is not printed in the review. 32 dies model Samsung 990 Pro SSD Review: The Return of the King (Tom, 2022-10-18) https://www.tomshardware.com/reviews/samsung-990-pro-ssd-review
C-FAR-010 Samsung 7th-generation V-NAND 512Gb TLC (ISSCC 2021, paper 30.3, Fig 30.3.7): page read time tR 40 us, 16KB page, 2.0Gb/s interface (toggle DDR); the text and die photo show four planes. A paper figure for the die, not a measurement in one drive. tR 40 us; 16KB page; 4 planes; 2.0 Gb/s per pin primary A 512Gb 3-bit/Cell 7th-Generation 3D NAND Flash Memory with 184MB/s Write Throughput and 2.0Gb/s Interface (2021)
C-FAR-011 Park et al., ASPLOS 2021, Eq. 2 and 3: a page read takes the sensing time tR, plus the transfer to the controller tDMA, plus the ECC decode tECC, plus read-retry time; each retry step repeats all three. Eq. 1: tR = NSENSE x (tPRE + tEVAL + tDISCH); NSENSE is 1 for SLC and up to 3 for TLC. tREAD = tR + tDMA + tECC + tRETRY; tRETRY = NRR x (tR + tDMA + tECC) primary Reducing Solid-State Drive Read Latency by Optimizing Read-Retry (ASPLOS 2021, arXiv:2104.09611) (arXiv, 2021-04-19) https://arxiv.org/pdf/2104.09611
C-FAR-012 Park et al., ASPLOS 2021, Table 1 and Section 6: timings taken from the 160 real 3D TLC chips they characterized: tPRE 24 us, tEVAL 5 us, tDISCH 10 us, average tR 90 us; the simulated SSD’s ECC corrects 72 bits per 1KiB codeword in tECC 20 us, and a 16KiB page moves in tDMA 16 us at 1Gb/s. Vendor not named; one research setup. tPRE 24 us; tEVAL 5 us; tDISCH 10 us; tR avg 90 us; tECC 20 us; tDMA 16 us per 16KiB at 1Gb/s primary Reducing Solid-State Drive Read Latency by Optimizing Read-Retry (ASPLOS 2021, arXiv:2104.09611) (arXiv, 2021-04-19) https://arxiv.org/pdf/2104.09611
C-FAR-013 Park et al., ASPLOS 2021, Fig 3b gives the TLC Gray code E=111, P1=110, P2=100, P3=000, P4=010, P5=011, P6=001, P7=101. Counting where each bit flips between neighbouring states gives 2, 3 and 2 read levels for the three pages, 7 in all, which matches the text’s ‘up to 3’. Derived from the figure; other dies may use another code. 2 / 3 / 2 sensing steps (LSB / CSB / MSB) model Reducing Solid-State Drive Read Latency by Optimizing Read-Retry (ASPLOS 2021, arXiv:2104.09611) (arXiv, 2021-04-19) https://arxiv.org/pdf/2104.09611
C-FAR-014 Park et al., ASPLOS 2021, Section 3.1 and Fig 5, 160 real 3D TLC chips: a fresh page (no P/E cycling, no retention age) reads with no retry; at 1-year retention and 2K P/E cycles reads average 19.9 retry steps, raising tREAD 21 times; 54.4% of reads need at least seven retry steps at 6-month retention even with no P/E cycling; at 1K P/E cycles and 3-month retention every read needs at least eight. One research setup, vendor not named. 0 retry steps fresh; 19.9 avg at 1-year retention and 2K P/E (tREAD 21x); 54.4% need 7+ at 6 months, 0 P/E primary Reducing Solid-State Drive Read Latency by Optimizing Read-Retry (ASPLOS 2021, arXiv:2104.09611) (arXiv, 2021-04-19) https://arxiv.org/pdf/2104.09611
C-FAR-015 Sean, who ran the Samsung 990 Pro 2TB CrystalDiskMark tests, says they were fresh runs on an empty drive, so the 4KiB reads were most likely of freshly written data read from the SLC cache rather than settled TLC. The tester’s judgment; CrystalDiskMark does not report the mode. Explains why the measured 40.73 us random read is no longer than the die’s 40 us TLC tR alone. fresh run; likely SLC cache reads own-data _own-data/z690-samsung-990-pro-2tb-cdm/README.md
C-FAR-016 Phison E21T engineering sample with SK hynix V7 TLC, 1TB, CrystalDiskMark on an empty drive: the first run’s RND 4KiB Q1T1 read is 68.411 MB/s (59.81 us); the next three runs, about 4 minutes apart, give 88.851, 88.655 and 88.827 MB/s (46.04, 46.13, 46.05 us). The whole first run is slower (SEQ 1MiB Q1T1 2,299 against 3,012 MB/s), and runs 1 and 2 used different CDM profiles. Which mode each run read from is unknown (Sean, 2026-10-10); CDM does not report it. Public naming: controller and flash only, never the archive file or folder names. run 1 68.411 MB/s (59.81 us); runs 2-4 about 88.8 MB/s (46.04-46.13 us) own-data _own-data/z690-e21t-v7-1tb-cdm/README.md
C-FAR-017 Phison E27 engineering samples with Kioxia BiCS6 162L TLC, CrystalDiskMark on empty drives: the first run’s RND 4KiB Q1T1 read is 75.109 MB/s (54.47 us) on the 1TB and 72.583 MB/s (56.36 us) on the 2TB; the next three runs give about 97 to 99 MB/s (41.1 to 42.1 us). The whole first run is slower (2TB SEQ 1MiB Q1T1 3,266 against 4,498 MB/s in run 2). Runs 1 and 2 used different CDM profiles. Mode unknown; CDM does not report it. Public naming: controller and flash only. 1TB: run 1 54.47 us, runs 2-4 41.14-42.07 us; 2TB: run 1 56.36 us, runs 2-4 41.22-42.00 us own-data _own-data/z690-e27-bics6-cdm/README.md
C-FAR-018 Team Group T-Create Expert 2TB, CrystalDiskMark with the drive 58% full (1104 of 1908GiB): the first run’s RND 4KiB Q1T1 read is 51.117 MB/s (80.04 us); the next three runs give about 75.0 MB/s (54.52 to 54.56 us). The whole first run is slower (SEQ 1MiB Q1T1 1,755 against 2,866 MB/s in run 2). Runs 1 and 2 used different CDM profiles. Mode unknown; CDM does not report it. run 1 80.04 us; runs 2-4 54.52-54.56 us; drive 58% full own-data _own-data/z690-team-group-t-create-expert-2tb-cdm/README.md
C-FAR-019 Samsung 970 EVO Plus 1TB and 2TB, CrystalDiskMark on empty drives, RND 4KiB Q1T1 read over runs 1 to 4: 1TB 43.8 / 56.0 / 57.4 / 81.0 MB/s (93.39 / 73.07 / 71.28 / 50.49 us); 2TB 34.2 / 51.8 / 52.8 / 79.8 MB/s (119.70 / 78.93 / 77.43 / 51.23 us). In runs 1 to 3 the test is the last read of the run; in run 4 it is the first, right after the test file is made, and it is fastest. Sequential reads stay steady across runs 1 and 2. Mode unknown; CDM does not report it. 1TB 93.39 / 73.07 / 71.28 / 50.49 us; 2TB 119.70 / 78.93 / 77.43 / 51.23 us (runs 1-4) own-data _own-data/z690-samsung-970-evo-plus-cdm/README.md
C-FAR-020 NVMe 1.4: command submission needs at most one MMIO register write; for NVMe over PCIe a command counts as submitted when the host’s write to the Submission Queue Tail Doorbell completes and moves the tail pointer past the slot holding the command. One doorbell write may submit one or more commands. Limit: this is the fast path; whether the host learns of completion by interrupt or polling is the host’s choice. one MMIO register write (SQ tail doorbell) primary NVM ExpressTM Base Specification (NVMe, 2019)
C-FAR-021 NVMe 1.4 Write command (section 6.15): the host names the data by Starting LBA (SLBA), the 64-bit logical block address of the first logical block to write (Command Dwords 10 and 11), and Number of Logical Blocks (NLB), how many logical blocks to write (Command Dword 12, a 0’s based value). There is no file name in the command; a logical block is the smallest addressable unit for Read and Write, and an LBA range is a starting LBA plus a number of logical blocks (definitions 1.6.22 to 1.6.24). Limit: the logical block size (often 512 B or 4 KiB) is set per namespace format, so NLB counts blocks, not bytes. starting LBA (64-bit) + number of logical blocks (0’s based) primary NVM ExpressTM Base Specification (NVMe, 2019)
C-FAR-022 Samsung 990 PRO carries a DRAM cache of 1 GB LPDDR4 on the 1 TB model and 2 GB LPDDR4 on the 2 TB model, about 1 GB per TB of flash. One drive family; some drives have no DRAM. 1 GB per TB primary https://download.semiconductor.samsung.com/resources/data-sheet/Samsung_NVMe_SSD_990_PRO_Datasheet_Rev.1.0.pdf (download.semiconductor.samsung.com, unknown) https://download.semiconductor.samsung.com/resources/data-sheet/Samsung_NVMe_SSD_990_PRO_Datasheet_Rev.1.0.pdf
C-FAR-023 NVMe 1.4: a submission queue entry (a command) is 64 bytes and a completion queue entry is at least 16 bytes (sections 4.2 and 4.6). 64 B command; 16 B completion primary NVM ExpressTM Base Specification (NVMe, 2019)
C-FAR-024 Rule of thumb: a 4-byte mapping entry per 4 KiB page is a ratio of 1 in 1,024, which matches about 1 GB of DRAM per TB of flash. Arithmetic only; the Samsung datasheet gives cache sizes, not the entry size. Controllers can store the map differently. 4 B per 4 KiB; about 1 GB per TB model no saved source yet
C-FAR-025 Samsung SSD 990 PRO data sheet (Rev 1.0, October 2022): at queue depth 1 with one thread, random read is rated up to 22K IOPS, the same figure for the 1 TB and the 2 TB model. The sheet writes K without defining it; read here as 1,000. Limits: the data sheet does not state the block size of the random test (it never says 4 KB), these are Up to ratings from IOmeter 1.1.0 on an AMD Ryzen 7 5800X with an ASRock X570 Taichi board under Windows 10, and the sheet gives no read latency. For this drive the QD 1 read rating (22K) is far below the QD 1 write rating (80K) because a write can complete into the drive’s cache while a read must wait for the flash. That reason is an inference from the two ratings, not a statement in the data sheet. One drive family, one 2022 revision. 22K IOPS random read at QD 1, Thread 1 (1 TB and 2 TB) primary https://download.semiconductor.samsung.com/resources/data-sheet/Samsung_NVMe_SSD_990_PRO_Datasheet_Rev.1.0_10129514072296.pdf (download.semiconductor.samsung.com, unknown) https://download.semiconductor.samsung.com/resources/data-sheet/Samsung_NVMe_SSD_990_PRO_Datasheet_Rev.1.0_10129514072296.pdf
C-FAR-026 Model: one second divided by the rated QD 1 random read of 22,000 IOPS (C-WOP-045) is 1 / 22,000 s = about 45.5 microseconds per read as the host sees it. Limits: an average at queue depth 1 for one drive family (Samsung 990 PRO, 2022 data sheet), from an Up to rating, including the host software stack. It is not the NAND read time (compare tR in the NAND datasheet claims) and the block size behind the rating is not stated by the data sheet. 45.5 us per read (1 s / 22,000 IOPS), QD 1 average model no saved source yet
C-FAR-027 PCI-SIG FAQ: PCIe 3.0 added 8.0 Gigabits per second per lane per direction of raw bandwidth, PCIe 4.0 added 16.0 and PCIe 5.0 adds 32.0 (earlier generations 2.5 and 5.0). The 3.0 and 4.0 FAQ pages state the same rates as 8.0 GT/s and 16.0 GT/s bit rates. Limits: this is the raw signalling rate before line coding, packet headers and flow control; the FAQ calls it raw bandwidth. The FAQ pages are undated and are PCI-SIG’s own answers, not the base specification text. 8.0 / 16.0 / 32.0 GT/s per lane per direction (PCIe 3.0 / 4.0 / 5.0) primary PCI-SIG FAQ - PCI Express 5.0 (pcisig.com, unknown) https://pcisig.com/faq?field_category_value%5B%5D=pci_express_5.0
C-FAR-028 Model: usable bandwidth of an x4 link per direction before packet overhead = lane rate x 4 lanes x 128/130 / 8 bits per byte. PCIe 3.0: 8e9 x 4 x 128/130 / 8 = 3.938 GB/s (3.94). PCIe 4.0: 16e9 x 4 x 128/130 / 8 = 7.877 GB/s (7.88). PCIe 5.0: 32e9 x 4 x 128/130 / 8 = 15.754 GB/s (15.75). GB here is decimal (10 to the 9th bytes). Limits: this is the ceiling after line coding only; transaction-layer packet headers, link-layer framing and flow control come on top, so a real transfer gets less. It is a rate, not a latency. x4 per direction: 3.94 / 7.88 / 15.75 GB/s (PCIe 3.0 / 4.0 / 5.0), before packet overhead model no saved source yet
C-FAR-029 Model: time to move 4,096 bytes over an x4 link at the usable rate before packet overhead = 4,096 B / (lane rate x 4 x 128/130 / 8). PCIe 3.0: 4,096 / 3.938e9 = 1.040 us. PCIe 4.0: 4,096 / 7.877e9 = 0.520 us. PCIe 5.0: 4,096 / 15.754e9 = 0.260 us. Limits: wire time for the payload bytes only. Packet headers (TLP and DLLP) and flow control add to it, and so do the controller, doorbell and completion steps; this is a transfer time, not the latency of a whole write. 4,096 B over x4: 1.04 / 0.52 / 0.26 us (PCIe 3.0 / 4.0 / 5.0), before packet overhead model no saved source yet
C-FAR-030 Samsung 990 Pro 2TB, CrystalDiskMark 8.0.4, Default mode, 1 GiB file on an empty drive: SEQ 4KiB read at Q1T1 averages 14.05 us (70,794 IOPS). Mean latency as seen by the application. One drive. The lab treats this as the floor of host stack, NVMe and controller time; that use is an inference (C-FAR-003), not part of this measurement. 14.05 us; 70,794 IOPS own-data _own-data/z690-samsung-990-pro-2tb-cdm/Default/CrystalDiskMark_20221008165630.txt
C-FAR-031 Samsung 990 PRO datasheet (Rev 1.0, 2022): the interface is PCIe Gen 4.0 x4 with NVMe 2.0, backward compatible with PCIe 3.0. Rated random read at QD1, thread 1 is up to 22K IOPS on both capacities, measured with IOmeter; the sheet gives no block size. PCIe Gen 4.0 x4, NVMe 2.0 primary https://download.semiconductor.samsung.com/resources/data-sheet/Samsung_NVMe_SSD_990_PRO_Datasheet_Rev.1.0_10129514072296.pdf (download.semiconductor.samsung.com, unknown) https://download.semiconductor.samsung.com/resources/data-sheet/Samsung_NVMe_SSD_990_PRO_Datasheet_Rev.1.0_10129514072296.pdf
C-FAR-032 NVM Command Set Specification 1.0d, Read command: Command Dwords 10 and 11 carry the Starting LBA (SLBA), the 64-bit address of the first logical block to be read; Command Dword 12 bits 15:00 carry the Number of Logical Blocks (NLB) to be read, a 0’s based value. SLBA (64-bit) + NLB (0’s based) primary NVM Express NVM Command Set Specification, Revision 1.0d (NVM Express, Inc., 2023-12-28) https://nvmexpress.org/wp-content/uploads/NVM-Express-NVM-Command-Set-Specification-1.0d-2023.12.28-Ratified.pdf
C-FAR-033 NVMe Base Specification 2.1: a Submission Queue is a circular buffer the host uses to submit commands; the host updates the SQ Tail doorbell register when there are new commands; the controller fetches SQ entries in order from the Submission Queue and may execute them in any order. host writes SQ Tail doorbell; controller fetches SQ entries in order primary _sources/ff-unv/ff1/nvme-base-2.1.md
C-FAR-034 Cai et al.: each flash channel connects the controller to its dies over a bus that is typically 8 or 16 bits wide, carrying both data and flash commands. Control pins mark what is on the bus: address latch enable (ALE) for an address, command latch enable (CLE) for a command, rising edges of write enable (WE) and read enable (RE) for each piece of data written or read. Each die on a channel has its own chip enable (CE) signal. The bus broadcasts address, data and commands to every die on the channel, and only the die whose CE is active acts on it. Limit: a description of a typical SSD channel from a survey, not a specific drive; ONFI and Toggle interfaces differ in signal detail. 8 or 16 bit bus; ALE, CLE, WE, RE; one CE per die primary _sources/cell-lab-depth/cai-survey-proc-ieee-2017.md
C-FAR-035 Cai et al. (Proceedings of the IEEE, September 2017), section 2.1, state: ‘Data in a block is written at the unit of a page, which is typically between 8 and 16 kB in size in NAND flash memory. All read and write operations are performed at the granularity of a page.’ Limits: a 2017 survey, a typical range not a spec for any one part; the one TLC datasheet saved for this project has a 16 KiB page (C-WOP-025), so a 4 KB host write is smaller than one flash page. page typically 8 to 16 kB; reads and writes at page granularity primary _sources/cell-lab-depth/cai-survey-proc-ieee-2017.md
C-FAR-036 Follow a Read follows one 4KiB random read (CrystalDiskMark RND 4KiB Q1T1) on a Samsung 990 Pro 2TB, the drive measured on Sean’s Z690 bench. 4KiB random read; Samsung 990 Pro 2TB own-data _own-data/z690-samsung-990-pro-2tb-cdm/Default/CrystalDiskMark_20221008163342.txt
C-FAR-037 Tom’s Hardware’s Crucial T700 4TB review calls the Samsung 990 Pro ‘the eight-channel, PCIe 4.0 990 Pro’, so its Pascal controller has eight NAND channels. The review tested the 4TB model; the controller is the same Pascal across capacities (C-FAR-007). With 32 dies on the 2TB, an even spread would be 4 dies a channel; that spread is an inference, not stated. 8 channels (990 Pro, Pascal) secondary Crucial T700 4TB SSD Review: Large capacity and max performance for PCIe 5.0 (Tom, 2023-12-23) https://www.tomshardware.com/pc-components/ssds/crucial-t700-4tb-ssd-review/2
C-FAR-038 Kioxia patent US12027203B2 (a generic NAND die): the die holds the address in an address register and the command in a command register, and a sequencer controls the whole die from the command. A read is a 00h command, the address over five cycles (two column, three row) and 30h; the sequencer starts the read and sets the die busy; when it is ready again the controller toggles REn to clock the data out. Generic NAND, not a statement about the Samsung die. address register, command register, sequencer; 00h, 5 address cycles, 30h; busy, then REn primary US12027203B2 - Semiconductor memory device to hold 5-bits of data per memory cell - Google Patents (patents.google.com, 2024-07-02) https://patents.google.com/patent/US12027203B2/en
C-FAR-039 US12027203B2: in a read the sense amplifier precharges the bit line, the row decoder selects the word line, and the driver applies VREAD to the unselected word lines and the read voltage (one read level) to the selected word line; a page with several read levels applies them one after another. Generic NAND. precharge BL; VREAD on unselected WLs; read voltage on the selected WL primary US12027203B2 - Semiconductor memory device to hold 5-bits of data per memory cell - Google Patents (patents.google.com, 2024-07-02) https://patents.google.com/patent/US12027203B2/en
C-FAR-040 US12027203B2: each bit line has a sense amplifier unit with a sense section, a logic section and data latches (five, ADL to EDL, in this 5-bit-per-cell design) plus latch XDL. Data enters and leaves the die only through XDL, which works as the die’s cache. Reading a page with several read levels stores the first level’s result in a data latch and combines each later level into it with the logic section. The number of data latches on a TLC die is not stated here; one per bit is an inference. one sense section and data latches per bit line, plus XDL as the cache; levels combined in a latch primary US12027203B2 - Semiconductor memory device to hold 5-bits of data per memory cell - Google Patents (patents.google.com, 2024-07-02) https://patents.google.com/patent/US12027203B2/en
C-FAR-041 Micron TN-29-01 (2004, a 2Gb SLC part): a page read is two command latch cycles and five address latch cycles, then R/B# goes low for tR. PAGE READ CACHE MODE (31h) moves a sensed page to the cache register while the next array read starts, and the note says it does not provide a benefit for reading a single page. Generic, old part; timings not used for the 990 Pro. 2 command latch and 5 address latch cycles, then busy for tR; cache read gives no benefit for a single page primary TN-29-01: NAND Flash Performance Increase Using PAGE READ CACHE MODE Command (2004)
C-FAR-042 Samsung ISSCC 2021 paper 30.3 (7th-gen V-NAND, the 990 Pro 2TB’s die family): the die has a pad area and four planes; each plane has a 128Gb cell array, peripheral circuits for word lines, bit lines and control logic, a row decoder and a 16KB page buffer, and the circuits sit under the cell array. Read throughput 1.84GB/s. 4 planes; per plane: WL, BL and control-logic circuits, row decoder, 16KB page buffer, all under the array primary _sources/ff-unv/ff4/lib-samsung-isscc2021-30-3-176l-fulltext.txt
C-FAR-043 Shim et al. 2010 (adaptive partitioning for page-level FTLs), section III-D: in a DFTL-style map, a read whose mapping is not in the cache must first read one translation page from flash, costed at one page-read time. A research FTL, not a statement about a particular drive. a map-cache miss costs one translation-page read primary An Adaptive Partitioning Scheme for
C-FAR-044 Phison PS5021-E21T consumer brochure (2020): the E21T is a DRAM-less PCIe 4.0 controller with Host Memory Buffer (HMB) support and four NAND channels. The E21T engineering sample in chapter 10 uses this controller. DRAM-less; HMB support; 4 channels primary Phison PS5021-E21T consumer brochure (Phison, 2020) https://web.archive.org/web/20210301071648/https://www.phison.com/phocadownload/CES_2021_Gaming/Consumer_PS5021-E21T.pdf
C-FAR-045 Kim and Kim (PLOS ONE 2020) describe a DRAM-less SSD as one that has no DRAM in its controller; it is sold for lower cost, lower power use and a smaller form factor. Their Fig 1 draws the controller as an interface controller, an embedded processor with SRAM, and a flash memory controller, with the DRAM controller and DRAM shown dashed as absent. They state that without controller DRAM the drive cannot use it to cache read data, written data or the address mapping table, so I/O performance is degraded. Limit: the paper says no DRAM in the controller (the drive); it gives no SRAM capacity, and Fig 1 is a generic architecture, not a specific product. no DRAM in the controller; embedded processor with SRAM shown in Fig 1 (SRAM size not given) primary HMB in DRAM-less NVMe SSDs: Their usage and effects on performance (PLOS ONE, 2020)
C-FAR-046 US9563382B2 (SanDisk, filed 2015) describes a drive whose control tables (FTL metadata) live in its non-volatile memory and are loaded into a small on-device cache that may be SRAM (called L1); the drive can extend that cache into a region of host DRAM reached over the host bus (L2, the host memory buffer); the flash itself is the last level (L3). To find a needed table entry the drive checks the device cache first, then the host cache, and only if it is in neither reads it from non-volatile memory. Limit: one patented design, not a finding about all drives; the patent describes using the host cache for data the drive can lose or rebuild. L1 device cache (may be SRAM), L2 host memory, L3 NAND primary Methods, systems, and computer readable media for providing flexible host memory buffer (2017)
C-FAR-047 In the Kim and Kim (PLOS ONE 2020) tests the host NVMe driver sets the HMB size within a range each drive specifies: SSD-A (SP A80) and SSD-C (Tammuz M730) accepted 8 MB to 480 MB, and SSD-B (HP EX900) accepted only 64 MB. Limit: these are the sizes those three drives allowed in that study, not what an operating system grants by default, and the paper does not report what size a normal desktop grants. 8 MB to 480 MB (SSD-A and SSD-C); 64 MB only (SSD-B) primary HMB in DRAM-less NVMe SSDs: Their usage and effects on performance (PLOS ONE, 2020)
C-FAR-048 In DFTL (ASPLOS 2009) a lookup that misses the cached mapping table in SRAM must fetch the needed mapping from its translation page on flash before the data page can be read. If the entry that has to be evicted to make room was not updated, it is simply dropped and the miss costs one translation page read; otherwise the worst case is two translation page reads (victim’s page and requested page) plus one translation page write (the victim’s updated page). The authors report that about 97% of evictions in the read-dominant TPC-H trace needed no extra operation. Limit: simulation of a research FTL; the paper gives the extra operations, not a measured latency for a real DRAM-less drive. clean victim: 1 translation page read; worst case: 2 translation page reads and 1 translation page write; about 97% of TPC-H evictions needed no write primary DFTL: A Flash Translation Layer Employing Demand-based Selective Caching of Page-level Address Mappings (2009)
C-FAR-049 Samsung 990 Pro 2TB (firmware 0B2QJXD7), CrystalDiskMark 8.0.4, 1 GiB x2 file on an empty drive, Windows 11 build 22000, Default mode: RND 4KiB write at Q1T1 averages 14.28 us (69,624.3 IOPS, 285.181 MB/s); three more Default runs give 14.31, 14.33 and 14.30 us. Mean latency as seen by the application, so the Windows storage stack is included. One drive, one run per file. Arithmetic check: 1 s / 69,624 = 14.36 us against the printed 14.28 us. 14.28 us; 69,624.3 IOPS; 285.181 MB/s (Default) own-data _own-data/z690-samsung-990-pro-2tb-cdm/Default/CrystalDiskMark_20221008163342.txt
C-FAR-050 Samsung 990 Pro 2TB (firmware 0B2QJXD7), CrystalDiskMark 8.0.4, 1 GiB x2 file on an empty drive, Windows 11 build 22000, Default mode: SEQ 4KiB write at Q1T1 averages 14.29 us (69,605.7 IOPS, 285.105 MB/s), equal to the random write within 0.01 us. Same floor as the SEQ 4KiB read of 14.05 us (C-FAR-030). One drive, one run. 14.29 us; 69,605.7 IOPS; 285.105 MB/s own-data _own-data/z690-samsung-990-pro-2tb-cdm/Default/CrystalDiskMark_20221008165630.txt
C-FAR-051 Samsung 990 Pro 2TB (firmware 0B2QJXD7), CrystalDiskMark 8.0.4, 1 GiB x2 file on an empty drive, Windows 11 build 22000, Full Power Mode (Samsung Magician): RND 4KiB write at Q1T1 averages 13.35 us (74,506.1 IOPS); the next three runs give 13.32, 13.39 and 13.31 us. SEQ 4KiB write averages 13.34 us (74,518.1 IOPS, run 20221011143710). About 1 us below Default, as for reads (C-FAR-001: 39.73 against 40.73 us). One drive. 13.35 us; 74,506.1 IOPS (RND); 13.34 us; 74,518.1 IOPS (SEQ) own-data _own-data/z690-samsung-990-pro-2tb-cdm/Full Power Mode/CrystalDiskMark_20221011141422.txt
C-FAR-052 Samsung 990 Pro 2TB (firmware 0B2QJXD7), CrystalDiskMark 8.0.4, 1 GiB x2 file on an empty drive, Windows 11 build 22000, Default mode, RND 4KiB write by queue and thread count: Q1T2 15.37 us (129,361.8 IOPS); Q1T3 16.48 us (180,646.7 IOPS); Q32T1 184.98 us (167,503.7 IOPS); Q16T16 168.52 us (1,512,688 IOPS). Only these points were run; nothing is interpolated. Writes scale across threads (14.3 to 16.5 us) far better than reads do at Q32. One drive. Q1T2 15.37 us; Q1T3 16.48 us; Q32T1 184.98 us; Q16T16 168.52 us / 1,512,688 IOPS own-data _own-data/z690-samsung-990-pro-2tb-cdm/Default/CrystalDiskMark_20221008164118.txt
C-FAR-053 Samsung 990 Pro 2TB (firmware 0B2QJXD7), CrystalDiskMark 8.0.4, 1 GiB x2 file on an empty drive, Windows 11 build 22000, Default mode, RND write latency at Q1T1 by block size: 4K 14.30, 8K 15.56, 16K 17.17, 32K 20.29, 64K 25.48, 128K 35.78, 256K 56.05, 512K 96.99 us; 1MiB 179.09, 2MiB 343.93, 4MiB 670.37, 8MiB 1,323.64 us. From 4K to 32K the cost is 0.214 us per KiB (about 4.7 GB/s), so size cost looks like transfer time, not program time. The 4K entry is the 164506 run (14.30 us), not the 163342 run (14.28 us). One drive, one run per size. 4K 14.30 us to 8MiB 1,323.64 us; 0.214 us per KiB from 4K to 32K own-data _own-data/z690-samsung-990-pro-2tb-cdm/Default/CrystalDiskMark_20221008164506.txt
C-FAR-054 Samsung 990 Pro 2TB, CrystalDiskInfo (firmware 0B2QJXD7, NVM Express 2.0, PCIe 4.0 x4) lists the drive’s Features as S.M.A.R.T., TRIM and VolatileWriteCache. Shows the drive reports a volatile write cache; says nothing about its size or where it lives (DRAM, SRAM or SLC). Features: S.M.A.R.T., TRIM, VolatileWriteCache own-data _own-data/z690-samsung-990-pro-2tb-cdm/Default/SMART/DiskInfo - 1.txt
C-FAR-055 US9170939B1 (Western Digital Technologies and Skyera, filed 2013): after the last logical segment of a write command has been transferred into the controller’s write buffer, the controller sends the acknowledgement to the host that the write completed; the buffer is de-allocated after the data is moved into the NAND die’s temporary register, before the program completes. One design, not Samsung’s; shows the shape of the idea. ack after transfer into the write buffer; buffer freed before the program completes primary US9170939B1 - Early de-allocation of write buffer in an SSD - Google Patents (patents.google.com, 2013-11-26) https://patents.google.com/patent/US9170939/en
C-FAR-056 Tom’s Hardware’s Samsung 990 Pro review says volatile memory such as SRAM is used by the controller to cache mapping information and buffer data to commit to the non-volatile NAND. A general statement about SSD controllers in the review, not a Samsung disclosure. controller SRAM caches mapping information and buffers data to commit to NAND secondary Samsung 990 Pro SSD Review: The Return of the King (Tom, 2022-10-18) https://www.tomshardware.com/reviews/samsung-990-pro-ssd-review
C-FAR-057 SPA-SSD (2019), Sec II-A: 3D TLC usually programs the three pages of a wordline (a page set) together, one-shot, so the delay of a write includes the wait to gather a full page set; data may wait in volatile DRAM only a limited ‘program delay’ before it must be programmed. Charge-trap 3D TLC in general; the simulator’s 100 ms program delay is a model value in that paper (Table I), not a 990 Pro figure. 3 pages per wordline programmed together; limited program delay in DRAM primary SPA-SSD: Exploit Heterogeneity and Parallelism of 3D SLC-TLC Hybrid SSD to Improve Write Performance (2019 IEEE 37th International Conference on Computer Design (ICCD), 2019)
C-FAR-058 Reader notes on a 3D flash text: one-shot TLC programming was introduced with Samsung V-NAND Gen2 (Im et al., ISSCC 2015, pp 130-131) and is used for 3D TLC and QLC. A book lead: the primary ISSCC paper is not saved here. Never name the book on the site. one-shot TLC since V-NAND Gen2 (ISSCC 2015) secondary _sources/3d-flash-memories/notes-ch1-6.md
C-FAR-059 Model: the 7th-gen die has a 16KB page buffer per plane and four planes (C-FAR-042). If it programs a full TLC page set per plane, one one-shot TLC unit = 3 x 16 KiB = 48 KiB per plane (24.6 us over the channel at 2.0 GB/s) and 192 KiB for four planes (98 us). In SLC mode a unit is 1 page per plane (16 KiB; 64 KiB for four planes). The unit size is arithmetic and assumes a full page set; Samsung does not print it. 48 KiB per plane (24.6 us); 192 KiB four planes (98 us); SLC mode 16 KiB per plane model _sources/ff-unv/ff4/lib-samsung-isscc2021-30-3-176l-fulltext.txt
C-FAR-060 Model: at the measured Q1T1 write pace of 69,624 per second (F-01), collecting 4 KiB writes takes 4 / 69,624 = 57.5 us per 16 KiB page, 16 / 69,624 = 230 us for a 64 KiB four-plane SLC-mode unit, and 48 / 69,624 = 689 us for a 192 KiB four-plane TLC unit. Assumes the controller fills the open page in arrival order (log-structured); real firmware may program partial units after a short wait. No 990 Pro figure. 57.5 us per 16 KiB; 230 us per 64 KiB; 689 us per 192 KiB model _own-data/z690-samsung-990-pro-2tb-cdm/Default/CrystalDiskMark_20221008163342.txt
C-FAR-061 No encode time is published for Pascal, the 990 Pro or the 7th-gen flash. Nearest leads from a book on NAND flash: BCH and Reed-Solomon encoding takes few controller cycles of latency while decoding can take many (Ch 2, sec 2.4, p 43). A second book (3D flash, Ch 7, Fig 7.2a) puts a high-speed encoder on each SSD channel with a shared decoder. BCH-era text, not LDPC; never name the books on the site. The ASPLOS tECC of 20 us (C-FAR-012) is a decode time for a research SSD. Show encode as ‘inline with the transfer, no figure’ or omit. no published encode time; encode is a few controller cycles (BCH-era book text) secondary Inside NAND Flash Memories (2010)
C-FAR-062 Model: 16,384 B / 2.0e9 B/s = 8.19 us to clock one 16 KiB page into the die over an 8-bit channel at 2000 MT/s (the 7th-gen die’s maximum, C-FAR-010); a 4 KiB piece takes 2.05 us, three pages (one-shot TLC, one plane) 24.6 us, four planes 98.3 us, four SLC-mode pages 32.8 us. Burst only: spare bytes, command and address cycles excluded (with Frontgrade’s 2,208 spare bytes the 16 KiB page is 18,592 B = 9.30 us, another part). Pascal’s actual channel clock is not published; ASPLOS used 1 Gb/s (16 KiB in 16 us, C-FAR-012). 8.19 us per 16 KiB; 24.6 us for three pages; 98.3 us for four planes model no saved source yet
C-FAR-063 Earlier Samsung TLC dies: 5th gen program time 500 us, described by Samsung (2018 press line) as the fastest write speed to date and about 30% better than the previous generation. Other generations in the flash-db: 3rd gen (48 WL) 660 us, 4th gen (64 WL) 700 us (see C-PL-100 for the 700 us). Different generations, not the 990 Pro die; the 5th-gen number is one press line about ‘write speed’. 5th gen 500 us; 3rd gen 660 us; 4th gen 700 us secondary _sources/ff-c2/samsung-5thgen-2018.md
C-FAR-064 Model, NOT PUBLISHED: tPROG of the Samsung 7th-gen die (512Gb, 184 MB/s write throughput, 4 planes x 16,384 B). 4 x 16,384 B / 184 MB/s = 356 us. The same formula on the 4th gen (2 x 16,384 / 51 MB/s = 642 us against a printed 700 us) and the 3rd gen (615 us against 660 us) runs 7 to 9% low, so the corrected range is about 380 to 390 us. Show only as a model, about 0.36 to 0.39 ms, not published; the convention behind Samsung’s write throughput is not stated. about 356 to 390 us (model, not published) model _sources/ff-unv/ff4/lib-samsung-isscc2021-30-3-176l-fulltext.txt
C-FAR-065 Park and Kim (MDPI Applied Sciences 12(21), 10697, 2022), Sec 1 and Fig 1: TLC programming time was about 800 us in 2014 and current TLC NAND products are required to operate at under 400 us; four-plane operation is mainstream. A vendor-neutral range, not the Samsung die. about 800 us (2014); under 400 us (current TLC) secondary A Review of Cell Operation Algorithm for 3D NAND Flash Memory (MDPI Applied Sciences 12(21) 10697 (Park, Kim), 2022-10-22) https://www.mdpi.com/2076-3417/12/21/10697
C-FAR-066 Park and Kim (2022, same review): the number of verify pulses is much larger than the number of program pulses during the whole program time, because every cell must be verified individually while one program pulse acts on all cells. Qualitative; no pulse count. verify pulses outnumber program pulses (no count) secondary A Review of Cell Operation Algorithm for 3D NAND Flash Memory (MDPI Applied Sciences 12(21) 10697 (Park, Kim), 2022-10-22) https://www.mdpi.com/2076-3417/12/21/10697
C-FAR-067 Samsung ISSCC 2021 paper 30.3 (7th-gen V-NAND): two-step bitline forcing (TBF) needs 3 steps of cell verify and an extra dynamic latch; two-step BL/WL forcing (TBWF) uses WL voltage VPGM - Vforce + 1/2 ISPP and is applied in the final program loop to improve the programming time (tPROG). Qualitative: the die runs a multi-loop program and verify with tuned final loops. No loop count. TBF: 3 verify steps; TBWF in the final program loop primary _sources/ff-unv/ff4/lib-samsung-isscc2021-30-3-176l-fulltext.txt
C-FAR-068 Cai et al. (2017) sec 3.4 describe binary, foggy and fine program passes with data buffered in SLC for MLC/TLC flows; modern 3D TLC uses one-shot programming (F-27). The survey’s example is planar; do not show foggy-fine for the 990 Pro. binary, foggy, fine passes (planar example) primary _sources/cell-lab-depth/cai-survey-proc-ieee-2017.md
C-FAR-069 Samsung 990 Pro 2TB, Iometer 1.1.0 write saturation (1 MiB sequential, QD32, 100% write, started 2022-10-08 10:09:07, 1 s rows): about 6,200 MB/s for 39 s (the first 39 rows sum to about 242 GB), then a fall at row 40 (10:09:46) to a median of about 1,400 MB/s. Site-derived figures: cache 243.3 GB, burst 6,221 MB/s, post-cache average 1,403 MB/s, total 2,756.1 GB in 1,830 s (1.38 times capacity). Not 4KiB: it tells the cache size and speeds, nothing about 4KiB latency. Drive state before the run unknown (DiskInfo shows 4,161 GB host writes at 10:08). about 6,200 MB/s for 39 s; then about 1,400 MB/s; cache 243.3 GB own-data _own-data/z690-samsung-990-pro-2tb-cdm/Default/WS/instWS-30min.csv
C-FAR-070 Model: 32 dies x 184 MB/s = 5,888 MB/s is the die-level TLC program ceiling; the measured burst of 6,221 MB/s is above it and the data sheet’s 6,900 MB/s sequential write is far above it. The post-cache 1,403 MB/s is 24% of the ceiling. Consistent with the burst running in SLC mode (one bit per cell, faster per byte) and the later rate being direct-to-TLC with fold and housekeeping. Not proven; die count and ceiling are inferences. 5,888 MB/s ceiling; 6,221 MB/s burst; 1,403 MB/s = 24% model no saved source yet
C-FAR-071 Samsung 990 Pro 2TB, Iometer 1 MiB write runs of five minutes after 30 s, 1 min, 5 min and 30 min of idle (all following the full-drive write run): 1,362, 1,360, 1,363 and 1,355 IOPS (1,429, 1,426, 1,429 and 1,421 MB/s decimal); the site’s cache-recovery row shows 0 GB written at speed after every idle step. The main run wrote 1.38 x capacity, so the drive was full: this half-shows the cache-then-fold story. It does NOT show the fold working after idle. 1,362 / 1,360 / 1,363 / 1,355 IOPS after 30 s / 1 / 5 / 30 min idle; 0 GB at speed own-data _own-data/z690-samsung-990-pro-2tb-cdm/Default/WS/WS-idle1-30Sec.csv
C-FAR-072 Samsung TurboWrite on sibling drives: a fixed default region (4GB on 250/500GB, 6GB on 1TB and larger) plus an Intelligent region that varies with free space; the 980 Pro 1TB totals 114GB at 5,000 MB/s, 2,000 MB/s after. NOT the 990 Pro: no 990 Pro TurboWrite size is published in any saved source. 980 Pro 1TB: 114GB at 5,000 MB/s, then 2,000 MB/s secondary Samsung 980 Pro M.2 NVMe SSD Review: Redefining Gen4 Performance (Updated) (Tom, 2021-02-17) https://www.tomshardware.com/reviews/samsung-980-pro-m-2-nvme-ssd-review
C-FAR-073 US10008250B2 (Intel): SLC write buffering for TLC; data written in SLC mode ‘waterfalls’ to the TLC region during idle time between host writes; the SLC region is revolving/dynamic with no added NAND. Intel assignee, not Samsung. SLC buffer drains to TLC during idle time primary US10008250B2 - Single level cell write buffering for multiple level cell non-volatile memory - Google Patents (patents.google.com, 2015-03-27) https://patents.google.com/patent/US10008250/en
C-FAR-074 Tom’s Hardware 990 Pro review: Samsung says Pascal can use read caching; the reviewer notes keeping hot data in pSLC is common; the review lists ‘sustained write performance is lower than expected’ as a con. The only 990 Pro text in the library that touches cache behaviour; no SLC size and no write-cache description. no SLC size; sustained write lower than expected secondary Samsung 990 Pro SSD Review: The Return of the King (Tom, 2022-10-18) https://www.tomshardware.com/reviews/samsung-990-pro-ssd-review
C-FAR-075 Samsung SSD 990 PRO data sheet (Rev 1.0, October 2022): at queue depth 1 with one thread, random write is rated up to 80K IOPS, the same figure for the 1 TB and the 2 TB model. The sheet writes K without defining it; read here as 1,000. Limits: the data sheet does not state the block size of the random test (it never says 4 KB), these are Up to ratings from IOmeter 1.1.0 on an AMD Ryzen 7 5800X with an ASRock X570 Taichi board under Windows 10, and footnote 4 says write results were measured with Intelligent TurboWrite active (the drive’s fast write cache), so this is a best-case burst rating on a fresh test system, not sustained or steady-state write speed. One drive family, one 2022 revision. 80K IOPS random write at QD 1, Thread 1 (1 TB and 2 TB) primary https://download.semiconductor.samsung.com/resources/data-sheet/Samsung_NVMe_SSD_990_PRO_Datasheet_Rev.1.0_10129514072296.pdf (download.semiconductor.samsung.com, unknown) https://download.semiconductor.samsung.com/resources/data-sheet/Samsung_NVMe_SSD_990_PRO_Datasheet_Rev.1.0_10129514072296.pdf
C-FAR-076 Model: one second divided by the rated QD 1 random write of 80,000 IOPS (C-WOP-044) is 1 / 80,000 s = 12.5 microseconds per write as the host sees it. Limits: this is an average at queue depth 1 for one drive family (Samsung 990 PRO, 2022 data sheet), taken from an Up to rating measured with the drive’s write cache active (Intelligent TurboWrite). It is the time until the host is told the write is done, not a time to NAND, and it includes the host software stack and the transfer to the drive. The block size behind the rating is not stated by the data sheet. 12.5 us per write (1 s / 80,000 IOPS), QD 1 average model no saved source yet (original: arithmetic on C-WOP-044 (Samsung 990 PRO rated QD 1 random write))
C-FAR-077 NVMe 1.4: a controller reports in the Identify Controller VWC field whether a volatile write cache is present, and the host enables or disables it with the Volatile Write Cache feature (06h, optional). The Flush command asks the controller to make the contents of that cache non-volatile, and it applies to commands already completed before the Flush was submitted. A Write command with the Force Unit Access (FUA) bit set must put its data on non-volatile media before the controller indicates completion; with FUA cleared the bit has no effect. A cache that is guaranteed to reach non-volatile media on power loss counts as non-volatile and the feature does not apply to it. Limit: the spec makes the cache optional and reported by the drive; the claim that many consumer drives complete writes before data is on stable media comes from the Linux kernel documentation (C-FM-186 and C-PLL-045), not from NVMe. VWC bit; Flush; FUA bit primary NVM ExpressTM Base Specification (NVMe, 2019)
C-FAR-078 Linux kernel documentation: many storage devices, especially consumer ones, have volatile write-back caches and signal completion before data reaches non-volatile storage. A flush makes earlier completed writes durable, and a FUA write completes only after its data is committed. primary Explicit volatile write back cache control — The Linux Kernel documentation (docs.kernel.org, unknown) https://docs.kernel.org/block/writeback_cache_control.html
C-FAR-079 A Flush makes every write that had already completed durable; a FUA write completes only once it is on non-volatile media primary Explicit volatile write back cache control — The Linux Kernel documentation (docs.kernel.org, unknown) https://docs.kernel.org/block/writeback_cache_control.html
C-FAR-080 With the drive’s write cache off, a write completes only after it reaches NAND primary Explicit volatile write back cache control — The Linux Kernel documentation (docs.kernel.org, unknown) https://docs.kernel.org/block/writeback_cache_control.html
C-FAR-081 Many drives with capacitor-backed PLP report no volatile write cache so Flush returns at once; some real PLP drives report a cache and make Flush cheap instead secondary NVMe power-loss protection: knowing whether your drive has PLP at all (netdata.cloud, unknown) https://www.netdata.cloud/guides/nvme/nvme-power-loss-protection-plp/
C-FAR-082 Cai et al. describe the write data path of an SSD controller: host data goes first to a host FIFO buffer, then to the DRAM manager, which buffers the data in DRAM and tracks where it is; when the controller is ready to write to NAND, the DRAM manager reads the data back from DRAM, the controller encodes it into an ECC codeword, generates a CRC for it and puts both in a NAND flash FIFO buffer, and the data is then dispatched through the flash interface to the NAND. The survey takes the path from a 2011 Micron brief (Rollins), whose figures also show a DRAM attached to the buffer manager between the host FIFO and the NAND FIFO. Limit: one described data path, from a Micron SATA-era brief, not every drive. It varies by drive: DRAM-less drives use other buffers (on-chip SRAM, or host memory buffer), and controllers may acknowledge a write at different points in this path. secondary _sources/cell-lab-depth/cai-survey-proc-ieee-2017.md
C-FAR-083 Host FIFO, as drawn in the SSD Atlas teaching datapath (nvme-dma block): the first buffer that host write data enters inside the controller, before the DRAM manager. It is a teaching label from the SSD Atlas, not a vendor block of any particular drive. The label follows the Micron 2011 brief (Rollins), which describes data moving from the host transport into a host FIFO and then to the buffer manager, with parity generated on entry and checked on exit. Limit: that brief covers Micron SATA SSDs, so the name and the placement of the block belong to that design. illustrative A Comparison of Client and Enterprise SSD Data Path Protection
C-FAR-084 CRC at the NAND FIFO, as drawn in the SSD Atlas teaching datapath (ecc-ldpc block): a CRC is generated on the ECC-encoded data just before it enters the NAND FIFO and checked when it leaves, so corruption in that buffer is caught before the data reaches the flash interface. It is a teaching label from the SSD Atlas, not a vendor block of any particular drive. The Micron 2011 brief (Rollins) states that a CRC is generated just before the data enters the NAND FIFO and checked when exiting, and Cai et al. relay the same path. Limit: Micron SATA SSDs of that era; whether a given modern NVMe controller does this is not public. illustrative A Comparison of Client and Enterprise SSD Data Path Protection
C-FAR-085 Cai et al.: when the host writes a page of data that already has an older copy, the FTL does not overwrite it. The old copy’s physical location is marked invalid in the block’s metadata and the new data is written to a page in the flash block currently open for writes, with the logical-to-physical map pointing at the new page. Elsewhere the survey explains why: a flash cell must be erased before it is reprogrammed, because programming can only add charge. Limit: general FTL behaviour from a survey, not one vendor’s firmware; erase works on whole blocks, so invalid pages are reclaimed only when their block is erased. primary _sources/cell-lab-depth/cai-survey-proc-ieee-2017.md
C-FAR-086 Kang et al.: NAND flash does not support in-place update. Once a page is written it must be erased before the same page can be written again, and because reads and writes work on pages while erase works on much larger blocks, an FTL is needed to hide the mismatch. Limit: a 2006 paper about block-level mapping schemes, cited here only for the no in-place update property. primary A Superblock-based Flash Translation Layer for NAND Flash Memory (2006)
C-FAR-087 Cai et al.: over time page invalidations leave blocks mostly invalid. The FTL periodically performs garbage collection: it picks highly fragmented blocks, migrates any remaining valid pages to a new block, and erases the entire block, which then joins the free list. Blocks with the fewest valid pages are often chosen first. The survey also notes that this remapping adds extra flash writes (write amplification). Limit: general behaviour, not one drive; policy (greedy or otherwise), timing and background scheduling vary by firmware. primary _sources/cell-lab-depth/cai-survey-proc-ieee-2017.md
C-FAR-088 Zhou et al. (FAST 2021): with 4 KB pages and 4 byte entries the logical-to-physical table is about 0.1% of SSD capacity, about 1 GB per TB. about 0.1% of capacity primary _sources/ff-unv/ff6/usenix-fast21-zhou.pdf
C-FAR-089 Frontgrade UT81NDQ512G8T organisation: a page of 16,384 bytes plus 2,208 spare bytes and 2,304 pages per block. One TLC part; pages are 8 to 16 kB in general. 16 KiB + 2,208 B; 2,304 pages per block primary _sources/ff-contradictions/frontgrade-ut81ndq512g8t.pdf
C-FAR-090 US20180357268A1 (Samsung, filed 2017) describes an FTL for SSDs with low or no DRAM or SRAM. Most metadata, including the logical-to-physical map, block status, program-erase counts, valid counts and read counts, is kept only on NAND (SLC blocks). A host write’s mapping update is not written to NAND at once; it is logged in a small RAM-resident volatile journal (a 16 KB SRAM journal in its example), copied to an SLC journal page on flash when that fills, and committed into the flash-resident L2P pages when the SLC journal pages are full. Reads must check the journals (Bloom filters are used) for a newer mapping. Limit: one patented method; it says what is written to NAND and when, but not how a drive rebuilds its map after a power loss, or what happens to the RAM journal contents if power fails. For the general scan-and-replay idea on drives with DRAM see C-WOP-013 and C-WOP-014, which are background, not equivalent. map on NAND; updates logged in a small RAM journal (16 KB SRAM in the example), copied to an SLC journal, then committed to flash L2P pages primary Data journaling for large solid state storage devices with low DRAM/SRAM (2018)
C-FAR-091 Zhou et al. (FAST 2021) describe conventional SSD behaviour: the L2P table is persisted on flash and usually cached in DRAM; for every logical page written the FTL also persists the reverse P2L mapping (the logical page number) and a write timestamp as housekeeping metadata on flash, and the latest L2P mappings in DRAM may be lost on sudden power failure. After power-off the FTL scans the persistent metadata in write order and rebuilds the latest L2P table; superblocks carry head and tail metadata for fast recovery, similar to conventional SSDs. Limit: background in a paper about a different design (Remap-SSD); the general mechanism is persisted per-page mapping metadata plus a scan, and the word journal is not what this paper uses for it. secondary _sources/ff-unv/ff6/usenix-fast21-zhou.pdf
C-FAR-092 US9927985B2 (SK hynix, filed 2016) describes one method of keeping the logical-to-physical (L2P) table recoverable: the up-to-date L2P table lives in volatile memory, an often outdated copy lives in flash, and the controller journals segments of the volatile table to the flash copy as data is written. After a sudden power loss the table in volatile memory is rebuilt from the flash copy, using per-page physical-to-logical data kept in metadata pages and replaying the dirty window of writes not yet journaled. The method estimates rebuild time and triggers more journaling to keep it under a limit. Limit: a patent describing one method, not a finding about all SSDs; journal is the patent’s term for it, and the general mechanism is persisted mapping metadata that is replayed or scanned after power loss (see the Zhou claim). primary US9927985B2 - Method of dynamic table journaling - Google Patents (patents.google.com, 2016-05-27) https://patents.google.com/patent/US9927985/en
C-FAR-093 Clocking one 16 KiB (16,384 byte) page over an 8-bit NAND channel at 1,600 MT/s takes 16,384 B divided by 1.6 GB/s, about 10.2 us (about 4.6 us at 3,600 MT/s). Limit: channel burst only, after the array has already read the page into the data register; spare bytes, command overhead and ECC are not included. The 1,600 MT/s figure is one speed grade (ONFI 4.2) seen in a 2021 review, and a given flash part runs one rung of the ONFI ladder. 10.2 us at 1,600 MT/s (16,384 B over an 8-bit channel) model no saved source yet (original: arithmetic on 16 KiB page size (Flash-Cosmos Table 1, Frontgrade datasheet) and ONFI 4.2 1,600 MT/s (C-SA-041))
C-FAR-094 Program time per page without a cache register is page / (tPROG + page / bus); with one the transfer hides behind the program; Park et al: 16 KB page, tPROG 500 microseconds, 166 MB/s gives 27.3 MB/s without a cache and 32.8 with 16 KB; 500 µs; 166 MB/s; 27.3 MB/s; 32.8 MB/s primary _sources/parallelism-depth/park-program-cache-busy-time.md
C-FAR-095 4th-gen V-NAND 512Gb TLC (64 WL) (samsung-4th-gen-v-nand-512gb-tlc-64-wl): tPROG (us) is 700 700µs primary A 512-Gb 3-b/Cell 64-Stacked WL 3-D-NAND Flash Memory (IEEE Journal of Solid-State Circuits, 2018)
C-FAR-096 Cai et al.: NAND cells are programmed by incremental step-pulse programming (ISPP). The controller applies a short high-voltage programming pulse to the wordline, then verifies the cell’s threshold voltage, and repeats pulse then verify until the cell reaches its target. Cells in the wordline are programmed together, and when a cell reaches its target while others have not, ISPP inhibits further pulses to that cell by turning off the SSL transistor of its bitline. Limit: the survey focuses on floating-gate cells (charge-trap cells are analogous, not identical). It gives no pulse count, step size or per-pulse timing, so a caption must not state one. The text names the scheme incremental but does not give the step size, so say rising only as the name implies. primary _sources/cell-lab-depth/cai-survey-proc-ieee-2017.md
C-FAR-097 Incremental step-pulse programming (Suh and colleagues): pulse, sense Vt, step up a little, stop when the cell lands in the window. The saved paper confirms ISPP with 0.5V steps up to 20V, verify after each pulse, and programmed Vt held within 0.6V. 0.5V steps, Vt within 0.6V primary https://doi.org/10.1109/4.475701
C-FAR-098 Frontgrade UT81NDQ512G8T datasheet (version 1.0.0, 1 November 2022), Table 50 SLC Array Characteristics: READ PAGE operation time (tR) 57 us typical, 60 us maximum; SNAP READ (tRSNAP) 27 us typical, 37 us maximum; PROGRAM PAGE operation time (tPROG) 226 us typical, 750 us maximum; block erase 15 ms typical, 30 ms maximum. Limits: one older TLC die operated in SLC mode, 2022 datasheet; the SLC read figures are for this part’s SLC mode only and the tPROG typical (226 us) is a datasheet value to set beside the research-chip value of 200 us. SLC mode: tR 57 us typ / 60 us max; tRSNAP 27 / 37 us; tPROG 226 us typ / 750 us max primary _sources/ff-contradictions/frontgrade-ut81ndq512g8t.pdf
C-FAR-099 Flash-Cosmos (MICRO 2022), section 5.1: the authors characterize 160 real 48-layer 3D TLC NAND chips with a 16 KiB page, and state that under regular SLC-mode programming the chips have a read latency tR of 22.5 us and a program latency tPROG of 200 us. This is the real-chip description and is separate from C-WOP-005, which covers the simulated SSD in Table 1 (tPROG 200 / 500 / 700 us for SLC / MLC / TLC mode; Table 1 repeats tR 22.5 us for SLC mode as a configuration value). Limits: SLC mode only, one research setup on 48-layer chips, no TLC-mode tR for the real chips; the paper’s separate worked example that assumes tR of 60 us is an illustration, not these chips. SLC mode on real 48-layer chips: tR 22.5 us, tPROG 200 us primary https://arxiv.org/pdf/2209.05566 (arxiv.org, unknown) https://arxiv.org/pdf/2209.05566
C-FAR-100 A low-latency 96-layer SLC part reports a 75µs program time and a 4µs read. 75µs; 4µs primary A 128Gb 1-bit/Cell 96-Word-Line-Layer 3D Flash Memory to Improve the Random Read Latency With tProg = 75 μs and tR = 4 μs (IEEE Journal of Solid-State Circuits, 2021)
C-FAR-101 Frontgrade UT81NDQ512G8T datasheet (2022): effective TLC program page time 1,900 us typical. An older 64-layer-era die; current dies differ. 1,900 us effective TLC program page time (typical) primary _sources/ff-contradictions/frontgrade-ut81ndq512g8t.pdf
C-FAR-102 Frontgrade UT81NDQ512G8T datasheet (version 1.0.0, 1 November 2022), Table 49 TLC Array Characteristics: READ PAGE operation time without VPP (tR) 88 us typical, 150 us maximum; SNAP READ (tRSNAP) 51 us typical, 100 us maximum; PROGRAM PAGE effective time per page (tPROG_eff) 1,900 us typical; PROGRAM PAGE per command (tPROG) 9,500 us maximum; block erase 15 ms typical, 30 ms maximum. Notes: read numbers assume the flag check is off, the maximum is the worst case with all shared pages programmed, and read-retry options 8 to 15 can push tR maximum to 480 us. Limits: one older die, TLC mode, a 2022 datasheet; the table gives no separate single-plane row (the features list gives 74 us for that, see the previous claim). TLC Table 49: tR 88 us typ / 150 us max; tRSNAP 51 us typ / 100 us max; tPROG_eff 1,900 us typ; tPROG 9,500 us max primary _sources/ff-contradictions/frontgrade-ut81ndq512g8t.pdf
C-FAR-103 Flash-Cosmos Table 1 lists program latency tPROG of 200 / 500 / 700 us for SLC / MLC / TLC mode in its simulated 2 TB SSD built on 48-layer 3D TLC NAND with 16 KiB pages. The paper’s own measurement on 160 real 48-layer 3D TLC chips states only SLC-mode tR of 22.5 us and tPROG of 200 us; the MLC and TLC figures are the simulator’s configuration values. Limit: 48-layer chips reported in 2022, one research model, not a datasheet. Newer dies differ. The Frontgrade datasheet figure for effective TLC page program time is about 1,900 us typical (see C-SA-045). 200 / 500 / 700 us (SLC / MLC / TLC mode) primary https://arxiv.org/pdf/2209.05566 (arxiv.org, unknown) https://arxiv.org/pdf/2209.05566
C-FAR-104 A page whose program was cut by power loss (even if it read back error-free) showed errors after about 1000 reads and reached 3.1e-3 bit error rate, while a fully programmed page showed errors only after 2.8 million reads (B-MLC32-2, cut-off 1.35 ms). 1000 vs 2.8 million reads; 3.1e-3 primary no saved source yet
C-FAR-105 Burst vs maker rating for seven drives: every rating within 10% of measured burst. Samsung 990 Pro 2TB burst 6,221MB/s vs 6,900 rated; its CrystalDiskMark write median 6,805MB/s (default mode folder). Computed by src/lib/myths-write-stats.ts from src/data/myths/write-runs.json (scripts/derive-myths-data.mjs over D:/D Data/Archive/- Z690 SSD results/- Internals). within 10%; 6,221MB/s; 6,805MB/s own-data no saved source yet (original: slc-cache)
C-FAR-106 Median GB written at speed after idle, by whether the main write run had written the drive’s capacity. Full drives (35; 33 with a 30-minute run): median 0GB after 30 seconds, 1 minute, 5 minutes and 30 minutes. Drives not full (20; 19 with a 30-minute run): median about 30GB, 42GB, 99GB and 125GB at the four steps (29.85GB, 41.9GB, 98.8GB, 125.3GB), and a median of 53% of the main-run cache back at 30 minutes. Computed from src/data/myths/cache-recovery.ts. 0GB; 30GB; 42GB; 99GB; 125GB; 53% own-data no saved source yet (original: slc-cache)
C-FAR-107 Fold work only moves data already staged in the SLC cache and is bounded by it; in the studied drive the cache (about 65GB) is cleared during idle time ~65GB cache example secondary no saved source yet
C-FAR-108 An SLC cache (pSLC) stores one bit per cell in TLC or QLC NAND, across two voltage levels, so writes program fast (arXiv 2409.14360v3) 1 bit per cell; 2 levels secondary _sources/ff-contradictions/arxiv-2409.14360-slc-cache.md
C-FAR-109 SLC cache stores one bit per cell across two voltage levels, so it programs fast. Cited inline to arXiv 2409.14360 (saved), which describes the SLC cache as storing one bit in the original TLC 1 bit per cell, 2 levels secondary no saved source yet
C-FAR-110 Flash-Cosmos background text says block erase latency tBERS is much longer (e.g., 3 to 5 ms) than program latency tPROG (e.g., 200 to 700 us), because erase works on a whole block while programming works on a page. Limit: the paper gives 3 to 5 ms as an example in its background section, not as a measurement from its 48-layer chips or a Table 1 entry. A datasheet for an older 64-layer-era die lists 15 ms typical, 30 ms maximum (C-SA-045). Say a few milliseconds, not a precise figure. about 3 to 5 ms secondary https://arxiv.org/pdf/2209.05566 (arxiv.org, unknown) https://arxiv.org/pdf/2209.05566
C-FAR-111 Cai et al. describe the LDPC read flow as three stages: hard decoding of the page read with the optimal read voltages, then soft decoding in several levels that each add extra reads at new reference voltages, then superpage-level parity recovery if all of that fails. Figure 30b gives average stage latencies of 80 us per hard-decode iteration, 80 us per soft-decode level and 10 ms for parity recovery. Limit: the figure does not separate the decoder’s own time from the NAND read in each iteration, so no LDPC decoder-only time is given by this source. Figure values are adapted by the authors from industry talks, not a measurement of one drive. about 80 us per hard-decode iteration; about 80 us per soft-decode level; about 10 ms for superpage parity recovery primary _sources/cell-lab-depth/cai-survey-proc-ieee-2017.md
C-FAR-112 Kim and Kim (PLOS ONE 2020) probed whether three DRAM-less drives use HMB as a write buffer by flooding the drive with writes, then timing a re-write of growing size before and after a flush. SSD-A shows a write buffer: re-write latency is a clean curve below a request size of 964 KB and spreads and rises sharply above it, and the behaviour is the same with HMB off, so the buffer is not in HMB. SSD-B and SSD-C show no clear curve but also no difference with HMB off, so the authors infer HMB is not their write buffer either. The paper says the drives use other media for this and does not name the medium; it notes that the published specification of SSD-C lists an SLC cache. Limit: one probing method on three 2019 drives; the 964 KB figure is one drive; the paper does not say the buffer is SRAM. write buffer of about 964 KB found on SSD-A and not in HMB; SSD-C spec lists an SLC cache; medium not identified primary HMB in DRAM-less NVMe SSDs: Their usage and effects on performance (PLOS ONE, 2020)
C-FAR-113 Kim, Lee and Kim (IEEE Access 2019) propose, as a research prototype on the FEMU emulator (QEMU 2.9.0, Linux 4.13.10), using host memory lent through HMB as the drive’s write buffer: the host’s block layer writes straight into the HMB buffer, the controller later flushes dirty buffers to NAND, and the buffer table and LRU lists live in the HMB space. On 13 MSR Cambridge traces the emulated average I/O latency improved by 59.0%, 63.2%, 78.5%, 85.4% and 86.2% for HMB buffers of 8, 32, 128, 512 and 2,048 MB, and a 512 MB buffer held almost all writes. Limit: an emulator study and a research proposal, not a shipping design; the paper does not discuss dirty data held in host memory across power loss or withdrawal of the HMB, and the 2020 study found the commercial drives it tested did not use HMB as a write buffer. research proposal on an emulator; average latency improved 59.0% to 86.2% for 8 MB to 2,048 MB HMB buffers primary HMB-SSD: Framework for Efficient Exploiting of the Host Memory Buffer in the NVMe SSD (IEEE Access, 2019)
C-FAR-114 DFTL (ASPLOS 2009) notes that although the Global Translation Directory is kept in SRAM, storing it on non-volatile storage aids recovery from power failure (citing another paper). Limit: a single sentence; the paper gives no recovery procedure and does not say what happens to dirty cached mappings on power loss. GTD on non-volatile storage aids power-failure recovery (one sentence, no procedure) primary DFTL: A Flash Translation Layer Employing Demand-based Selective Caching of Page-level Address Mappings (2009)
C-FAR-115 ONFI specifications page, release histories: ONFI 3.2 (June 2013) scales NV-DDR2 I/O speed to 533 MT/s; ONFI 4.2 (February 2020) extends NV-DDR3 to 1333, 1466 and 1600 MT/s; ONFI 5.0 (May 2021) extends NV-DDR3 to 2400 MT/s and adds NV-LPDDR4 at up to 2400 MT/s; ONFI 5.1 (August 2022) extends NV-DDR3 and NV-LPDDR4 to 3600 MT/s; ONFI 6.0 (published January 2026) extends NV-LPDDR4 to 4800 MT/s. Limits: these are the fastest speeds each release defines, not the speed of any shipping part (a part runs one rung, and the Frontgrade datasheet part tops out at 667 MT/s); ONFI 6.0 drops the older SDR and NV-DDR2 and NV-DDR3 definitions. The saved page is the ONFI summary page (retrieved 2026-10-05; the live page refused an automatic re-fetch on 2026-10-08), not the specification PDFs. 533 / 1,600 / 2,400 / 3,600 / 4,800 MT/s for ONFI 3.2 / 4.2 / 5.0 / 5.1 / 6.0 primary ONFI specifications (ONFI, page current 2026) https://onfi.org/specs.html
C-FAR-116 Frontgrade UT81NDQ512G8T datasheet (4Tb TLC NAND, version 1.0.0, 1 November 2022), features list: SNAP READ operation time without VPP 51 us typical; single-plane READ PAGE operation time 74 us typical without VPP (73 us with VPP); multi-plane READ PAGE operation time 88 us typical without VPP. Limits: typical values for one part, an older die with a 16,384 byte page (the datasheet says B17A industrial die source and gives no layer count), in TLC mode; current dies differ. The detailed array table (Table 49) lists only one READ PAGE row with 88 us typical and 150 us maximum, so the 74 us single-plane figure appears in the features list but not as its own table row (see the next claim). The datasheet also adds a busy-to-ready step tRTABSY of 10 us typical. TLC read: 74 us single-plane, 88 us multi-plane, 51 us snap read (typical, features list) primary _sources/ff-contradictions/frontgrade-ut81ndq512g8t.pdf