Working Memory, Part 4: High Bandwidth Memory

Working Memory has so far followed one cell. Part 2 opened one DRAM cellA DRAM cell is one access transistor and one storage capacitor. Reading it drains the capacitor onto the bit line, so every read is followed by a write back. and followed it out to three buses. They were a DIMMA DIMM is a memory module: a small board of DRAM chips that plugs into a slot on the motherboard., a phone and a graphics card. Part 3 looked at what could replace the cell and the array built from it. The fastest processors use a fourth arrangement of the same cell. Memory and Storage for SI, Part 2 puts flash beside an SI acceleratorAn SI accelerator is a processor built for the vector and matrix math of SI models, such as a GPU, which does far more of that math than a CPU., and the comparison it makes is with what already sits there, which is a stack of DRAM dies called High Bandwidth MemoryHigh Bandwidth Memory stacks DRAM dies on a base die and links them with thousands of vertical wires, giving a very wide, short path to a processor on the same package., or HBM. This part is the account that comparison needs. A reader who finishes it can read an accelerator’s memory line (stacks, capacity, TB/s), say where each number comes from, and say why the answer to “more bandwidth” was a wider bus beside the processor and not a new cell.

Five things are left to other pages. The Atlas takes a stack apart layer by layer. Generations are in the HBM Timeline, and the dies are in the DRAM Timeline. Stacking inside a single die, which is a different idea, is in Part 3. HBF is the flash stack that the SI part sets beside this memory, and its own page covers it. The series page lists the rest.

Why a processor needs bandwidth

A large SI processor is often limited by how fast memory can feed it rather than by how fast it can compute. Bandwidth is the quantity HBM is built to raise, and three lines of evidence show how it became the limit and what raising it costs in power.

Numbers here carry one of six tags. A standard value comes from a JEDEC release or standard. A paper value is a published measurement or model. A vendor value is a company’s own product claim, and the company is named in the sentence. A tutorial value comes from course or conference slides and is used for vocabulary and examples, never as the only support for a figure. Derived means arithmetic on numbers already given, shown in the text. Secondary means trade press or a blog, used only where no primary source was found. Per-pin rates are written in Gb/s, and the bandwidth of a device or a stack in GB/s or TB/s.

Compute grew faster than memory

Gholami and colleagues (IEEE Micro, 2024) measured the trend over 20 years. Peak server FLOPS grew about 3.0 times every two years, while DRAM bandwidth grew about 1.6 times and interconnect bandwidth about 1.4 times (paper). Over the whole 20 years that comes to roughly 60,000 times for compute, 100 times for DRAM bandwidth and 30 times for interconnect. Memory has become the main bottleneck in SI applications, they conclude, particularly in serving, and bandwidth can dominate for decoder Transformer models (paper).

Why a token waits on memory

A language model’s decode phase produces one token at a time. In NVIDIA’s technical blog on inference that phase is memory-bound because how fast weights, keys, values and activations move from memory to the GPU sets the latency, and the arithmetic waits on it (vendor).

SK hynix gives one example in numbers for HBM3E. If Llama 3 70B is driven by a single GPU with four of its HBM3E stacks, the company says, the GPU can read all 70 billion parameters 35 times within a second (SK hynix, vendor). SK hynix presents it as its own illustration, so the 35 reads a second carry no token rate. For scale, the H100 SXM5 is rated at over 3TB/s (NVIDIA, vendor). NVIDIA’s spec table gives the Rubin GPU 19.2TB/s (NVIDIA, vendor).

Capacity runs short as well once many requests share one GPU. From the H100 SXM’s 80GB to the Blackwell Ultra B300’s 288GB, HBM per GPU grew 3.6 times, a step Micron charts in its October 2026 technology webinar (vendor). Micron’s white paper on agentic workloads ran a five-agent workflow on one Blackwell GPU with 192GB of HBM3E, and a single request used at most 36% of HBM bandwidth (vendor). In a second test on the same 8TB/s, 192GB configuration, inference time grew about 8.2 times from 100 to 1,000 concurrent requests and about 105.6 times at 10,000, where the key-value cache filled every byte set aside for it (vendor). These are Micron’s own benchmarks without peer review, so they show a direction more than a rule.

Bandwidth costs power

O’Connor and colleagues (NVIDIA and the University of Texas at Austin, MICRO 2017) worked out what a fixed power budgetA power budget caps how much power a drive may draw at once. Firmware meets it by running fewer NAND operations at the same time. buys. Inside a 60W memory budget, GDDR5 at 14.0pJ/bit reaches about 536GB/s, and HBM2 at 3.9pJ/bit, which the paper’s text gives as 3.97, reaches about 1.9TB/s (paper). Even HBM2 cannot pass 2TB/s inside that budget, they add. Memory interfaces already take 40% or more of an SoCThe controller is a system on a chip with CPU cores, hardware engines, memory and interfaces to both the host and the NAND.’s power budget, according to a Rambus tutorial (tutorial). So the answer had to raise bandwidth per watt as well as bandwidth, and that is the job the stack does.

Figure 4.1 has two panels. Its left panel indexes the three trends to 1 and draws them on a log axis from the rates the paper states, which shows direction and gap rather than measured points. Its right panel shows the 60W budget as two bars. Read together, they show compute pulling away from memory bandwidth. In the same 60W, GDDR5 reaches about 536GB/s, under a third of HBM2’s 1.9TB/s (derived).

What a stack is made of

An HBM stack is a short tower of ordinary DRAM dies standing on one extra die, set beside the processor on a slab of silicon. It has three named parts, which are the DRAM dies, the base die they stand on, and the interposer whose wires reach the processor.

DRAM dies on a base die

HBM is core DRAM dies stacked on a base dieThe base die sits at the bottom of an HBM stack and connects the DRAM dies above it to the processor. It holds the interface and test logic and can carry logic of its own., in SK hynix’s description. The base die carries through-silicon vias (TSVs), vertical wires cut through each die, and at the bottom of the stack it connects to the GPU, which controls the HBM (vendor). NVIDIA’s account of Pascal has HBM2 as one or more vertical stacks of memory dies linked by TSVs and microbumpsA microbump is a tiny solder joint that connects one stacked die to the next, landing on the through-silicon vias of the die below., with over 5,000 TSVA through-silicon via is a vertical copper wire etched through a thinned die, so stacked dies can pass signals and power straight down. holes in a single 8Gb HBM2 die (vendor). For its HBM4 part, SK hynix puts the count at over 20,000 in its packaging talk (vendor, as reported). Every die in the stack is DRAM, a point the fifth section returns to.

Beside the processor, on silicon

The stack does not sit on the processor. A passive silicon interposerAn interposer is a slab of silicon or other material with very fine wiring that sits under a processor and its memory stacks and connects them side by side. carries the wires between the stack and the processor die, and all of it sits in one package (NVIDIA, vendor). A silicon interposer allows fine-pitch connections between processor and DRAM, and that is also why HBM is harder and costlier to build (Rambus tutorial, tutorial). AMD’s Fury was the first high-volume interposer product and the first use of TSVs and microbumps in graphics, with 4GB of HBM on a 4,096-bit interface at 512GB/s (AMD, Hot Chips 27, vendor). SK hynix names TSMC’s CoWoSCoWoS is TSMC’s packaging that places a processor and its HBM stacks side by side on a shared silicon interposer. as the process that joins the processor and the HBM on an interposer (vendor). That packaging step is one of the four supply ceilings The Memory Cycle follows.

Why silicon and not the package’s organic substrate? Because of the wire count. The HBM4 interface has 2,048 data signals, and its SPHBM4 variant needs 4:1 serialization down to 512 signals to relax the bump pitch enough for an organic substrate (JEDEC, standard). Organic routing allows a longer channel and potentially more stacks, by the same release.

Independent channels

In HBM3 the interface is split into independent channels, each with its own 64-bit data bus at double data rateDouble data rate memory moves data on both the rising and the falling edge of the clock. DDR also names the line of PC memory standards, numbered by generation., and the channels need not be synchronous with each other (JEDEC, standard). SK hynix’s HBM at Hot Chips 2014 was already 1,024 bits wide, as eight 128-bit channels (vendor).

Figure 4.2 is a side view, drawn as a schematic and not to scale. At the bottom is the substrate with the interposer on it, and the processor and one stack stand side by side on the interposer. Follow the dense band of wiring between the base die and the processor. It lets the stack carry a bus of 1,024 or 2,048 wires. For a real stack opened layer by layer, see the Atlas, since this figure only names the parts.

The bus is wide, not fast

HBM gets its bandwidth from the number of wires rather than the speed of each one. Two JEDEC standards, the vendors’ rates beyond them and a DIMM at the same pin rate all show it.

HBM3: 1,024 bits

JEDEC’s HBM3 standard runs at up to 6.4Gb/s per pin, which gives 819GB/s per stack. It has 16 channels, and with two pseudo channelsA pseudo channel is one of the two halves an HBM channel is split into. The halves share the channel’s command bus but each decodes and runs its own commands on its own half of the data pins. each that makes 32 virtual channels. Stacks are 4, 8 or 12 dies high, with provision for 16. Device densities span 4GB to 64GB. The release adds symbol-based ECCError-correcting codes add parity bits to each block of data so the controller can find and fix bit errors on read. on-die (JEDEC, standard). The 64GB figure is the maximum at 16-highStack height is how many DRAM dies an HBM stack holds. More dies give more capacity in the same footprint but make the stack harder to build and cool. with 32Gb dies; JEDEC does not say parts are sold at that height. The later revision, JESD238B.01, lists speed bins up to 8.0Gb/s (standard).

HBM3E has no JEDEC document of its own. Vendors use the name for parts built to that revision at its higher bins. JEDEC defines the bins (standard), and the HBM3E name is the vendors’ own. Micron’s HBM3E comes as a 24GB 8-high part that ships with the NVIDIA H200 and a production-capable 36GB 12-high part, and the family delivers over 1.2TB/s (vendor).

HBM4: 2,048 bits

HBM4 doubles the width. Its standard defines a 2,048-bit interface at up to 8Gb/s for up to 2TB/s, with 32 channels and stacks 4 to 16 dies high (JEDEC, standard). Each channel has two pseudo channels, dies are 24Gb or 32Gb, a 16-high stack reaches 64GB, and the interface stays compatible with HBM3 controllers. On NVIDIA’s side, HBM4 doubles the interface width over HBM3e (NVIDIA, vendor).

Vendors run past the standard

Several pin rates appear in HBM4 coverage, each from a different source. JEDEC’s top bin is 8Gb/s. A Rambus tutorial gives 10Gb/s as its upper rate, though the same slide still puts HBM4 at up to 2TB/s per stack, the 8Gb/s figure (tutorial). Samsung rates its HBM4 at a consistent 11.7Gb/s with headroom to 13Gb/s and a maximum of 3.3TB/s per stack (Samsung, vendor). SK hynix claims over 2.8TB/s for its own HBM4 (SK hynix, vendor). The vendor figures are the vendors’ own, above the standard’s 8Gb/s.

Micron’s October 2026 technology webinar works out the same 2.8TB/s for a 2,048-bit interface at 11Gb/s, as slide arithmetic with no rated part named as of the webinar (Micron, vendor). Micron separately announced high-volume manufacturing of a 36GB 12-high HBM4 in March 2026 (vendor).

Arithmetic shows how the numbers fit. Across 2,048 pins, 8Gb/s is about 2TB/s, 10Gb/s about 2.6TB/s, 11Gb/s about 2.8TB/s, 11.7Gb/s about 3.0TB/s, and 13Gb/s about 3.3TB/s (derived). So Samsung’s 3.3TB/s is its 13Gb/s top rate rather than its 11.7Gb/s steady rate.

The same pin rate as a DIMM

The Rambus tutorial’s base DIMM example is a 200MHz core with 6.4Gb/s I/O (tutorial). A DDR5 unbuffered DIMM is 64 bits wide, as two 32-bit channels (JEDEC JESD308B, standard), so that works out to 51.2GB/s (derived: 6.4 times 64 divided by 8). HBM3 runs at the same 6.4Gb/s. Its 1,024 data bits give 819.2GB/s, 16 times the DIMM’s bandwidth from 16 times its width (1,024 against 64) at an identical pin rate (derived: 6.4 times 1,024 divided by 8). Part 2 explains the DIMM’s bus in ranks, channels and DDR5 sub-channels.

Figure 4.3 draws each interface as a rectangle. Its horizontal side is the number of data bits and its vertical side the pin rate, so the area is the bandwidth. A DIMM is a sliver. HBM3’s rectangle is no taller, only 16 times wider. HBM4’s rectangles are twice as wide again, and the dashed one reaches higher only because Samsung runs its pins faster than the standard’s top bin.

Why the chip edge sets the limit

A processor can only put memory wires along its edge, and that edge is a fixed length. More bandwidth therefore meant more wires along that edge, close to the die, and that ended GDDRGDDR is DRAM built for graphics cards. It trades capacity and power for a very high data rate per pin on a wide bus soldered around the GPU. beside the largest GPUs.

Bandwidth lives on the shoreline

The SPLASH authors (National University of Singapore) write that GPU bandwidth comes from wide interfaces along a finite perimeter, which makes shoreline a fixed resource. More bandwidth therefore depends on packaging advances, while capacity can grow without more shoreline if memory gets denser, and HBM inherits DRAM’s density (SPLASH, paper). The Rambus tutorial’s terms are the same. Host shoreline is the millimeters of die given to the host’s memory PHY, and distance is the critical determinant of I/O power (tutorial). Wider interfaces such as HBM optimize bandwidth per shoreline, counted in GB/s per millimeter (tutorial).

A die stops growing at the reticle

A die cannot be larger than the reticle, the exposure field of the chip-making machine. As NVIDIA describes it, Blackwell is two reticle-limit dies joined by a 10TB/s link into one GPU (NVIDIA, vendor), and Rubin as two reticle-limited compute dies on one package, with up to 288GB of HBM4 in 12-high stacks (vendor). The full GH100 has six HBM3 or HBM2e stacks and twelve 512-bit memory controllers. The H100 SXM5 ships with five HBM3 stacks and ten 512-bit memory controllers (NVIDIA, vendor).

So the edge grows by adding dies and a larger interposer. A CoWoS-S silicon interposer can be up to 3.3 times the reticle size, about 2,700mm², holding the processors and the stacks (TSMC, vendor). AMD’s Instinct MI455X carries 432GB of HBM in 12 stacks at 23.3TB/s (AMD, vendor). From the memory side, JEDEC’s SPHBM4 points the same way, because organic routing allows longer channels and potentially more stacks (standard).

Why HBM replaced GDDR beside the big GPU

GDDR is narrow per device. A GDDR6 device has two 16-bit channels, 32 bits in all (Micron datasheet, vendor). The devices are soldered to the board (tutorial). On a traditional GDDR5 board, numerous discrete chips surround the GPU (NVIDIA, vendor). The tutorial sets GDDR at lower cost than HBM and at the expense of energy per bit (tutorial).

NVIDIA’s Tesla P100 was the turn. Instead of “the numerous discrete memory chips surrounding the GPU” in GDDR5 board designs, it carried four 4-die HBM2 stacks for 16GB and 720GB/s of peak bandwidth. Its eight 512-bit memory controllers make 4,096 bits in total, against the 384-bit GDDR5 interfaces of the Tesla parts before it (NVIDIA, vendor). AMD’s Fury X reached 512GB/s against the 320GB/s of the GDDR5 R9 290X. According to AMD, the GPU and its memory take about a third of the board area, 55mm by 55mm against 110mm by 90mm, and that GDDR5 was entering the inefficient region of the power and performance curve (AMD, Hot Chips 27, vendor).

AMD’s efficiency numbers need a caution. Its chart shows 42.6GB/s per watt for the HBM device against 10.66 for GDDR5, and the deck’s endnote for that chart gives 35+GB/s per watt for the HBM device (vendor). The two HBM figures disagree inside one deck, so read the direction and not the number. SK hynix’s Hot Chips talk gives a cleaner comparison for the first HBM: I/O power 42% lower than GDDR5 per Gb/s per pin (vendor), and the energy figures in the first section say why that mattered.

Eventually the board ran out of edge. Around 2010, adding more GDDR devices ran past shoreline and package limits, and GDDR was historically placed about 30mm to 100mm from the GPU, according to Silicon Codesign (a secondary blog source, so its distance is an estimate rather than a measurement).

GDDR moved to the boards where it fits. NVIDIA’s GB202 chip powers the GeForce RTX 5090. The full GB202 has sixteen 32-bit memory controllers, 512 bits in all. NVIDIA’s whitepaper lists the RTX 5090 at 28Gb/s on a 512-bit bus, for 1.792TB/s (NVIDIA, vendor). That is less than the 2TB/s one JEDEC HBM4 stack allows (derived). Part 2 shows the GDDR bus in same cell, three buses.

Figure 4.4 draws the two arrangements in plan view as schematics. On the left, sixteen GDDR packages ring a GPU on a board and set a 512-bit bus. On the right, one GH100 die has six stack sites along two edges, and five are filled. A strip beneath counts stacks per package for three products. A board ring can add devices only by pushing them farther from the GPU, while the interposer puts the wires right at the die edge.

Not a DIMM, and not a new cell

A stack is neither a module of chips on a board nor a new memory cell, and each of those lines is drawn below.

What a DIMM does instead

A DIMM puts DRAM packages side by side on a module in a board slot, on a 64-bit bus that gives the 51.2GB/s derived in the bus section above, at the Rambus tutorial’s 6.4Gb/s I/O (tutorial). It wins on capacity per slot. Micron’s DDR5 range includes 128GB RDIMMs built from 32Gb monolithic dies and MRDIMMsAn MRDIMM is a server memory module whose buffer chips run two ranks at once and merge their data onto the channel, so the module moves more data than a standard registered DIMM. JEDEC standardizes its buffer chips. up to 256GB (vendor). In September 2026 Micron sampled a 512GB DDR5 RDIMMA registered DIMM adds a chip that buffers the command and address signals, so a server can carry more modules per channel at the cost of a little latency. at up to 9,200MT/s (Micron, vendor). A 12-high HBM3E stack holds 36GB (vendor). Adding DIMMs is limited by shoreline, board routing and heat as well (tutorial).

The same cell in every die

The core diesA core die is one of the DRAM dies stacked above the base die in an HBM stack. of a stack are DRAM dies built on the vendor’s DRAM process. Samsung’s HBM4 paper (ISSCC 2026) says its core dies use its sixth-generation DRAM process and its base die a 4nm FinFET logic process (paper). SK hynix’s HBM4 uses 1b DRAM core dies, with the base die built with TSMC (TrendForce, secondary). The paper by O’Connor and colleagues describes a traditional HBM2 die as having 16 DRAM banksA bank is an independent array inside a DRAM die with its own sense amplifiers. Different banks can work on different rows at the same time. that share a single wide I/O interface (paper).

Bits cost more wafer in a stack, even with the same cell. Micron’s fiscal 2026 annual report warns that HBM needs more wafers and more cleanroom space than conventional DRAM on the same node to make the same number of bits, so a shift of capacity back from HBM could sharply raise the supply of ordinary DRAM (vendor).

The Rambus tutorial has HBM and GDDR placing the I/O sense amplifierA sense amplifier compares the current or voltage of a selected cell against a reference to decide which stored level the cell holds. in the middle of a bank to shorten the path, where DDR and LPDDRLPDDR is the low-power branch of DDR memory for phones and laptops. It is soldered close to the processor and built to idle at very little power. put it at the end of the bank to save cost (tutorial). Moving it is a bank-layout choice rather than a new cell. HBM2’s access energy is about 3.97pJ/bit, and most of it goes into moving data from the row buffer across the die and down to the base-layer interfaces, plus opening a 1KB row, with the I/O itself a small remainder (paper).

HBM is a stack of separate dies joined by TSVs (SK hynix, vendor). HBM is stacked-die memory, which makes it unlike a monolithic die such as 3D NAND flash. A monolithic 3D DRAM3D DRAM stacks DRAM cells in layers on one die, the way 3D NAND does, instead of shrinking them further across a flat array. chip is a different thing, and Semiconductor Engineering sees it as something that could later feed into the HBM architecture (Semiconductor Engineering, secondary). Part 2 explains the cell inside each die, and Part 3 covers stacking inside one die.

Logic on the base die

A stack can differ from plain DRAM at its base die. SK hynix’s floorplan figure shows an interface region, a TSV area and a test port (vendor). SK hynix made it on its own process through HBM3E and moves the HBM4 base die to a TSMC logic process so more function fits in the limited space, which also lets it build customized HBM (SK hynix, vendor). A custom base die can also hold the memory controllerA memory controller is the circuit, usually inside the processor, that turns reads and writes into DRAM commands and schedules them around refresh and timing rules., a die-to-die link to the compute die and custom logic, in a slide Samsung shows in the Rambus tutorial (tutorial).

Samsung’s HBM-PIM is a different arrangement. It puts an SI engine inside each memory bank of the DRAM dies, so the logic sits in the memory dies rather than on the base die. Samsung claims more than twice the system performance with more than 70% less energy (Samsung, vendor).

Figure 4.5 sets the two side by side as a schematic, with a different scale in each column. One column holds the board trace and the slot, the other the interposer and the stack. Across the bottom runs the point of the figure: whichever arrangement a die sits in, inside it is the same one-transistor, one-capacitor array in banks, the cell Part 2 opens.

Stacks on the processor

Micron’s roadmap takes the base die further. At its October 2026 technology webinar, Micron set out HBM4 in 2026 and HBM4E as the first generation it plans with both JEDEC and custom base dies, followed by HBM5, HBM5E and HBM6 through 2030 (vendor). Alongside that line it plans vertical HBM, or vHBM, with memory stacked on the compute die for the shortest memory-to-compute path, and a vertical edge inference memory, vEIM, whose first version targets 2TB/s at under 1pJ/bit (vendor). Hybrid copper bonding, 16-high stacks and a heat spreader over the PHY are the packaging steps it lists to get there. All of the timing is Micron’s target for high-volume manufacturing, which it calls directional and subject to change.

The width and height that HBF is measured against

High Bandwidth Flash sets a stack of NAND dies beside the same processor and compares it with HBM.

Width

Width is covered in the section on the bus. Samsung’s HBM4 paper adds 2,048 I/Os, 3.3TB/s and 36GB in each cube (paper).

Height

JEDEC fixes the height of the stack as well as the number of dies. An HBM3 stack is 720µm nominal for 4, 8 and 12 dies high (standard). An HBM4 stack is 775µm nominal for 4 to 16 dies high (standard). Fitting 12 dies where 8 went takes thinner dies. SK hynix made each die 40% thinner to put twelve 3GB dies, 36GB, in the thickness of its 8-layer 24GB part (SK hynix, vendor). Samsung’s 12-layer stack meets the same height as 8-layer parts, with a 7µm gap between dies, at up to 1,280GB/s in its 2024 release (Samsung, vendor). SK hynix began volume production of 36GB HBM3E in 12-high stacks (vendor). Samsung’s 12-high HBM4 runs 24GB to 36GB (Samsung, vendor), and a Hot Chips packaging talk lists SK hynix HBM4 climbing to 48GB, with 16-high under qualification (vendor, as reported). The most both standards allow is 64GB, at 32Gb dies and 16-high (standard).

What Memory and Storage for SI, Part 2 does with these

That part sets a stack of NAND dies beside the same kind of processor, on the same kind of interposer and base die, and measures it against these widths and this height. Its first section names the problem and its stack section draws the flash side, and the HBF numbers stay on that page.

Figure 4.6 draws three stacks on one height axis. Both HBM3 columns have the same 720µm height, with 8 and 12 dies, and the HBM4 column is 775µm with 16. Die layers are drawn evenly, so only the overall height is to scale. Within one standard, more dies means thinner dies at the same height, and HBM4 allows 55µm more height for up to 16.

Two things are worth watching from here. One is the base die, which SK hynix moves to a TSMC logic process and Micron plans to offer in custom form from HBM4E. Next comes vHBM, Micron’s plan to stack memory on the compute die, a roadmap target with no rated part behind it so far (vendor).

Part 5 turns from how these parts work to which of them ship today and which only sample, across DDR5, LPDDR, GDDR and HBM.