Memory and Storage for SI, Part 2: High Bandwidth Flash

Memory and Storage for SI starts with devices. Part 1 asked what an analog memory has to do to stand in for a synapse, and which of those memories have left the lab. This part turns to NAND flash, which the memory map left holding capacity, and to the job High Bandwidth FlashHigh Bandwidth Flash stacks NAND dies the way HBM stacks DRAM and places them on the processor’s package, trading some speed for far more capacity beside the GPU. (HBF) gives it. HBF puts stacked dies on the package of an SI accelerator, beside its HBMHigh Bandwidth Memory stacks DRAM dies on a base die and links them with thousands of vertical wires, giving a very wide, short path to a processor on the same package.. The question is what HBF is, what its draft specification says, where its bandwidth comes from and what it costs in writes and retention. I read the specification and the papers that simulate it, and I mark every vendor statement as one.

The memory beside the GPU is small

The memory map ended with NAND doing the job it does best, which is holding a very large amount of data at a price people will pay. Most of the memories on the memory map try to replace flash. HBF doesn’t. It keeps the NAND and gives it a new place, as a capacity tier on the accelerator’s package.

Capacity is the reason. NVIDIA’s H200 carries 141GB of HBM3e at 4.8TB/s. NVIDIA’s Vera Rubin NVL72 spec table lists 288GB of HBM4 at 19.2TB/s in its Rubin GPU column, and NVIDIA’s developer blog quotes up to 22TB/s for the same part.

Large language models need room for their weights and for the KV cacheA KV cache stores the key and value vectors a language model computes for each token, so later tokens reuse them instead of recomputing them. It grows with context length., the stored attention state of every token in every conversation being served. SPLASH is a September 2026 paper from the National University of Singapore, and its authors give an example from their own arithmetic. Sixty-four concurrent requests of one million tokens each to Qwen3-235B-A22B need 12.9TB of KV cache, enough to fill the HBM of 68 B200 GPUs before any weights are loaded.

They also say why HBM can’t simply grow. A GPU gets its memory bandwidth from wide interfaces along the edge of the chip, and that edge is a fixed length. Working Memory, Part 4 follows that limit through the HBM stack. Capacity can grow without more edge only if the memory gets denser, and flash is denser than DRAMDRAM stores each bit as charge on a tiny capacitor behind one transistor. The charge leaks away, so every cell must be refreshed many times a second, and the data is gone when power is removed.. In their example the same 12.9TB fits in the HBF of five GPUs.

Keep one number from the memory map in mind. In the memory hierarchy, DRAM answers a column access in about 16ns, and an SLC NAND chip takes 25µs to read the first byte of a page. The HBF specification gives no read time for its NAND (more on that below), so the gap between those two numbers is the one to keep in mind.

What HBM is

HBM is DRAM dies stacked on a base dieThe base die sits at the bottom of an HBM stack and connects the DRAM dies above it to the processor. It holds the interface and test logic and can carry logic of its own. and joined by through-silicon vias (TSVs), vertical wires that pass through each die, and set beside the processor on a very wide bus. The cell inside each die is still the one-transistor, one-capacitor (1T1C) DRAM cell. Working Memory, Part 4 explains the stack, why its bus is wide and why the edge of the chip limits it, and the Atlas takes one apart layer by layer.

What the HBF specification says

On August 3, 2026, Sandisk and SK hynix released the HBF specification through the Open Compute Project (OCP). The release counts that as six months after the consortium began work in February, and the announcement of a dedicated OCP workstream is dated February 25, about five months earlier. Its full title is the “High Bandwidth Flash (HBF) High-Level Base Die Specification”, version 0.7.0, and both named authors work at Sandisk or SK hynix. Google and Tenstorrent joined as consortium members during the work, per the release, without a date.

Read the title closely. It marks a 0.7 draft that defines the base die and the host side (the part the GPU talks to). The NAND and the link between the base die and the NAND dies are left undefined. Several physical limits in it are still marked “TBD” for version 1.0.

The stack

An HBF stack is a base die with up to 16 NAND dies above it, joined by TSVs that carry signals from the base die up into the NAND. Much of what an SSD controller does falls to the base die, which runs the host interface, error correction and bad-block management. A read counter tells the host when data needs refreshing. Per the draft, the package is approximately 16mm by 10.975mm and 775µm tall, with the same height for 4, 8, 12 and 16-high stacks. Working Memory, Part 4 has the HBM numbers for comparison. An HBM4 stack is also 775µm nominal, and HBM4 spreads 2,048 bits over 32 channels where an HBF stack has up to 16 host channels of 64 bits.

This has been tried before

Stacking NAND dies with TSVs is not new. In 2015 Toshiba showed a prototype with up to 16 NAND dies joined by TSVs. It had an I/O rate over 1Gbps and, Toshiba said, about 50% less power than Toshiba’s current products. Sixteen dies made a 256GB package. In 2017 Toshiba Memory built a TSV version of its 48-layer BiCSBiCS FLASH is Kioxia’s name for its 3D NAND. It stacks control gate plates between insulator layers, punches holes through them in one step and fills them to form cells. TLC. It had 16 dies for 1TB and an interface of up to 1,066Mbps. Toshiba Memory said the power efficiency of one package, measured as data rate per watt, was about twice that of the same generation of BiCS bonded with wires.

Those parts were built for storage packages. HBF takes the same technique and uses it to feed a GPU.

The host interface

On the GPU side, HBF uses the advanced-package form of the chiplet interconnect standard UCIeUCIe is an open standard for the links between chiplets inside one package, so dies from different makers can be joined. 3.0. A stack has up to 16 independent host channels, each 64 bits wide, carrying AXI transactions over a UCIe streaming link. Per the spec’s assumption, protocol overhead leaves 75% of the raw rate usable. Each host channel reaches its own set of NAND dies, and a request on one channel never touches another channel’s dies. The spec defines three speed grades, from 8GT/s to 32GT/s, in the table below.

A full stack of 16 dies holds 512GiB. SK hynix’s release rounds that to 512GB and the three grades to about 0.4TB/s to 3.0TB/s. The spec itself mixes units, too. Its feature list says 3.072TiB/s where the table says 3,072GB/s, and the table above follows the spec’s table.

How data goes in and out

HBF uses 4KiB NAND pages. A read can be as small as 64B or as large as 4KiB, in 64B steps, inside one page. A write is always a full 4KiB page, and inside a NAND block the host has to write pages in order. There is no garbage collectionGarbage collection copies the valid pages out of mostly stale blocks and erases those blocks, so there is always free space for new writes., and the device never moves live data the way an SSD’s flash translation layerThe flash translation layer is the firmware that maps host addresses to physical flash locations and hides the rules of erase blocks and wear. does. Wear levelingWear leveling spreads erases evenly over all blocks. Dynamic leveling picks young blocks for new writes, while static leveling also moves cold data off them. is done by remapping addresses, either automatically by the base die or by the host swapping heavily worn zones for lightly worn ones.

I didn’t find a read latency for the NAND in the spec. Its timing tables carry units with blank values, and the only latency it describes belongs to an optional SRAMSRAM is fast memory of latching cells with several transistors each. It needs no refresh but takes far more area per bit than DRAM, so it serves as on-chip memory. scratchpad, “in the order of L3 cache”, which says nothing about the NAND.

Sandisk’s slides add one thing the spec leaves out, which is how the NAND is built. Its 2025 Investor Day deck says HBF is built on its BiCS 3D NAND with CBAIn a CMOS bonded array the memory array and the control circuits are built on separate wafers and joined face to face. Xtacking is one such design. wafer bonding, and that its stacking keeps die warpage low enough for 16-high stacks. CBA is a technology Sandisk and Kioxia developed together, and as Kioxia defines it, the CMOS logic and the memory array are made on separate wafers and then bonded together. I didn’t find a BiCS generation named in the deck, so this guide names none, and everything in this paragraph is a vendor statement.

Where the bandwidth comes from

HBF gets its bandwidth from running a great many reads at once, and the NAND itself is no faster. An SSD gets its speed the same way. Every die reads at the same time, and inside each die every plane reads at the same time. Flash Foundations Part 3 covers dies and planes, and the Parallelism Lab lets you watch the effect.

Sandisk agrees. Its CTO Alper Ilkbahar put it this way in a 2025 company blog: “it’s bandwidth that matters.” A footnote on the same blog concedes that HBF has higher latency and a larger page sizeA page is the smallest unit a NAND die programs or reads, commonly 16KiB of data plus a spare area for error-correction codes and metadata. than HBM.

HBF’s specification describes the same idea. To read a model at full speed, the host writes it in 4KiB pages spread across every bank or plane on all 16 channels (the spec counts 16 banks per channel). It then reads all of those pages “in parallel in one HBF read latency time”, so a single read latency covers many pages.

Look at the host side to see how far the interface has been widened. A normal NAND channel on the current open NAND interface standard, ONFI 6.0, tops out at 4,800MT/s, about 4.8GB/s on an 8-bit bus (this guide’s arithmetic). One HBF host channel at grade 3 has a raw link rate of 256GB/s. It isn’t a fair comparison with the NAND inside the stack. The draft leaves the link between the base die and the NAND dies to each vendor, and I didn’t find a speed for it in the draft. The 256GB/s describes the host side only.

Real parts show how much parallelism a NAND dieA NAND die is one piece of silicon holding flash memory. A drive spreads its work across many dies, and each can run an operation on its own. can offer. Kioxia’s XL-FLASH, a low-latency part sold as a 128Gb SLC die or a 256Gb MLC die, lists a 4KB page, 16 planes and a read latency under 5µs. Kioxia’s page doesn’t say which die or generation each figure belongs to. Beyond NAND, Part 2 uses its latency. Part 4 of this series covers the SSDs Kioxia builds on XL-FLASH for GPU-initiated reads. Micron called its 232-layer NAND the first six-plane TLC in production, with each plane able to read on its own. SPLASH’s model of an HBF stack uses 64 planes per die and 16 dies, 1,024 planes in all.

The weak side: writes, wear and retention

HBF’s specification is direct about the first problem: “Unlike HBM (High Bandwidth Memory), HBF is subject to endurance limitations.” It leaves endurance and read-disturb limits “product specific” rather than giving a number. A device is rated for 10 years or until its endurance indicator reaches 100%, whichever comes first, and the host is expected to read the wear counters and act on them.

With power on, data retention is guaranteed for 24 hours at 85°C. Refresh falls to the host, at intervals the spec puts at typically every 24 to 48 hours. It also has to refresh a block after a product-specific number of reads, because each read wears down the health of the data in that block. A refresh means the host rewrites the whole block, so refresh spends write bandwidth and wear too. Per the spec, a refresh and a read sent to the same die at the same time will slow the read. With power off, the spec says data may not be kept past a product-specific time, as with HBM, and it recommends SSDs for permanent storage.

That cuts against Sandisk’s own 2025 fact sheet, which says HBF “retains its data even when a power source is removed” and needs no refresh power. My reading is that they describe different things. The fact sheet covers the NAND cell, and the spec covers what a product will guarantee. Where they disagree, this guide goes with the spec.

Writes are the other weak side. The spec says HBF is “optimized for high-bandwidth read operations”, and Sandisk’s 2026 slides call it read-optimized memory. A UC Berkeley and FuriosaAI study assumes 8GB/s of write bandwidth per stack against 1,000GB/s of read. Its authors say they borrowed typical SSD write figures because few HBF write specifications are published.

So HBF suits data that is written once and read many times. That means model weights and the long-lived part of a KV cache. The spec itself suggests putting weights and KV-cache write traffic in separate groups of channels so that wear can be managed.

So far the lifetime figures are modeled. SPLASH estimates 5.0 years for its 3TB SLC tier at an assumed write amplification of 1.15. Berkeley’s study estimates 4.77 years under one scheduler and 14.82 years when it reserves 10% of capacity as an admission buffer, assuming uniform wear, 100,000 program/eraseA program/erase cycle is one write of a block followed by its erase. Each cycle wears the oxide, and the cell type sets how many cycles it tolerates. cycles and a write amplification of 1.0. SPLASH assumes the same 100,000 cycles, and both lifetimes come from models.

Two newer simulations add their own figures. In HBFSim, HBF lasts 2.0 to 6.9 years at a published peak load of Llama-3.1-70B, longer when more stacks share the KV-cache write traffic, and its authors note that no public HBF hardware exists to check that against. The authors of SETA estimate their HBF cards need no replacement for flash wear over a 10-year deployment. A 2026 letter by Choi and colleagues names read reclaim as a cause of HBF wear in its title, Read-Reclaim-Induced Endurance Degradation in High-Bandwidth Flash for LLM Inference.

What the research shows, and what it doesn’t

SPLASH used no HBF hardware and evaluated its design through simulation, since its authors had no commercial HBF hardware to use. Every result below is simulated or comes from a vendor.

The papers had to guess

Without a part to measure, every paper sets its own parameters, and they don’t agree. “Not stated” means the paper doesn’t give the value. Characterizing HBF is the Berkeley and FuriosaAI study.

Assumed read times run from about 4µs to 20µs across the papers, so any result built on one of these numbers is only as good as that guess.

The academic results

SPLASH pairs sparse attention with HBF, reading only the parts of a long KV cache that matter for the next token. Across 15 model and context-length points, its simulated decode throughput per GPU is 11.4 times that of an HBM-only system at a 100ms target per token (geometric mean).

A study from Berkeley and FuriosaAI models high-throughput serving of multi-turn agent workloads. Its fastest HBF configurations cut total completion time by 36.1% to 87.0% against HBM-only. Time per output token got worse at the same time. On its 5x trace, every hybrid configuration had a higher mean time per output token than HBM-only serving. HBF traded latency for throughput in that study, which fits a memory whose strength is width.

SETA, from Park and colleagues, moves whole agent sessions rather than only their KV caches. Sessions start on a conventional GPU tier and move to a GPU-HBF tier as they mature. Against a GPU-only baseline, it reports 4.0 to 35 times lower time to first token when a session resumes, 1.5 to 2.1 times lower turn latency and 1.9 to 2.2 times higher serving throughput. Its HBF stack holds 256GB and matches HBM’s 0.67TB/s of read bandwidth, with a 5µs read and 10 times less write bandwidth, and it puts an HBF card at 0.39 times the cost of a GPU. Its latency and throughput results come from an in-house simulator, an analytical model fitted to measured H100 kernel timings. The cost ratio is the authors’ own estimate.

HBF-Sim comes from East China Normal University. It is a simulator rather than a result, and its authors are plain about it. Of their default read, program and erase times they write: “These are model parameters rather than hardware measurements.”

HBFSim is a different simulator, from Mohamed bin Zayed University of Artificial Intelligence. It models HBF with host-managed flash translation beside HBM, and its authors note that no public HBF hardware is available to validate it against. In its runs, HBF serves data that does not fit in HBM 11.0 to 19.0 times faster than host DRAM. It also finds HBM does more good holding the running sequences’ KV cache than acting as a cache for HBF.

Sandisk’s results

Sandisk’s 2025 simulation read the 8-bit weights of Llama 3.1 405B and put HBF within 2.2% of a hypothetical HBM with unlimited capacity.

At its August 2026 Investor Day, Sandisk showed a larger simulation, “based on internal testing”. It ran Qwen3-480B-A35B, 960GB in bfloat16, on a multi-turn agentic workload. Each GPU got 4TB of HBF or 192GB of HBM, and both sides were given the same 12.8TB/s of memory bandwidth per GPU. Here HBF takes the place of HBM rather than sitting beside it, and both sides spill to LPDDRLPDDR is the low-power branch of DDR memory for phones and laptops. It is soldered close to the processor and built to idle at very little power. at 128GB/s. Sandisk’s slide says four HBF GPUs matched the token output of eight HBM GPUs. The equal bandwidth was set as an input to the simulation.

Flash as memory on real hardware today

The nearest thing to HBF on real hardware is flash behind a DRAM front tier on CXLCXL runs over the same physical layer as PCIe but adds cache-coherent memory access, so it moves cache lines rather than blocks of storage.. TempoKV comes from Samsung Semiconductor. It used an H100 with a CXL memory device whose DRAM is backed by a Samsung NVMe SSD. Its lesson is timing. Staging the KV cache of a 64K-token Llama-3.1-8B-Instruct prompt from the SSD one second ahead of the request cut the median time to first token from 1.23s to 0.51s. On that setup, with an 8B Llama prefix on an H100, the flash helped only once its latency was hidden. TempoKV puts the SSD behind a DRAM tier across a CXL link, where HBF stacks NAND dies on the accelerator’s package.

LM-CXD, from Sogang University, Samsung and ETRI, had to emulate its CXL-SSD, “as no CXL-SSD is commercially available”. On that emulator, a stock CXL-SSD was about 3x slower than local DRAM, measured as time to first token.

This connects back to Beyond NAND, Part 2. When Micron ended 3D XPoint it shifted toward memory built on CXL, and when Intel wound down Optane it said it was still embracing CXL. Flash Foundations Part 10 covers where CXL stands.

Where it stands

Here is the timeline from the companies’ own statements:

None of the Sandisk and SK hynix statements cited here says that any HBF sample has gone to a customer. On its October 29 earnings call, Sandisk said the first HBF memory die has taped out with fabrication now underway and that it remains on track to deliver samples of its first HBF-based products in 2027. The call didn’t mention the 2025 target for memory samples in the second half of 2026, and Sandisk’s 2026 slides neither repeat nor withdraw it. HBF today amounts to a draft specification and a taped-out die.

What it means for NAND

In the memory map, NAND wins where cost per bit matters most, and most of the memories in Beyond NAND try to win a place by being faster than flash or by lasting longer under writes. HBF does neither. NAND is chosen here for how much it holds per die, and the specification hands the limits that follow to the host, which must handle wear leveling, refresh and the write order. My read is that reads are the cheap part and refresh is the costly one, because every refresh is a full-block rewrite that spends write bandwidth and wear, the two things HBF has least of.

Two things will settle it. The first is real parts. Sandisk plans samples of its first HBF inference product for 2027, while the papers here assume read times from 4µs to 20µs and the specification gives none. The second is whether anyone publishes write and refresh numbers, since every lifetime figure above comes from a model. Part 3 follows the same idea one step closer to the processor, with Samsung’s concept for flash inside the processor’s own package.