Memory and Storage for SI, Part 1: Memory as a Synapse

This is the first part of Memory and Storage for SI. It leans on Beyond NAND, Part 5 for the arithmetic of the crossbar and takes up the cells that hold its weights. The question is what it takes for a memory cell to serve as a weight, and how far the cells that try have gotten. Over in the Crosspoint Lab, a solved circuit runs the multiply, and the memory map has the memory hierarchy these cells sit in. The series page lists the other parts.

What an analog memory has to do

Beyond NAND, Part 5 ended with a crossbar that multiplies, with a conductance in every cell, a voltage on every row and a sum of currents on every column. That picture leaves out where the conductances come from. Here each cell has to be a weight, which means a value that can be set, nudged in small steps, read many times and trusted between updates.

A weight is a conductance

In a memristive synapse the weight is the combined conductance of the device or devices, the input voltage from the neuron is applied to them, and the summed device currents are the synapse output, as Boybat and colleagues describe it. That is the crossbar of Beyond NAND, Part 5 seen from the cell. The cell stores a conductance, and a read gives a current.

A signed weight takes a pair of conductances held in the same position of a pair of arrays, as Beyond NAND, Part 5 explains and Gokmen and Vlasov assume in their design. Hardware shows how many devices that takes. Le Gallo and colleagues report that IBM’s 64-core phase-change chip stores each weight in four devices, two for each polarity. Joshi and colleagues ran a ResNet-32 experiment that programmed each of its 361,722 weights on just two phase-change devices of a prototype chip in a differential pairA differential pair stores one signed weight in two cells and subtracts their currents, one cell for the positive part and one for the negative., with the rest of the network simulated in software. How many devices a weight gets is a design choice, and the rest of this part shows what each choice buys.

Three jobs, not one

A digital memory cell does one thing well. It holds a level, a few bits at most, and gives it back when read. A cell used as a weight has three jobs. It is read many times, once in every pass through the network. It is updated, in small steps, over and over. And it has to hold its value in between.

Updates are the unusual job. Gokmen and Vlasov contrast them with ordinary storage. Storage in a nonvolatile memory cell is digital and typically does not exceed a few bits, and its write overwrites every earlier value, so it does not depend on history. A training device in their simulated design does not need to be resolved with a high signal-to-noise ratio. It needs a large number of small steps to keep the history of updates, with a lower estimate of 600 states, and high endurance only for small increments, with no demand for full swings between levels. A coincidence event is the moment a pulse on a row and a pulse on a column arrive together at one crosspoint. Each pulse is at half the switching voltage, so a single pulse barely moves the device, and only where the two meet does it see the full voltage and take a small step. Because only coincidences change a device, every cell in the array can be updated at once.

Figure 1.1 draws the weight and two of its jobs, the read and the update, as three panels. Panel one is the weight, two conductances and the gap between their tops. Panel two is a read, volts in on the rows and currents adding down a column. Panel three is an update, equal small steps up and equal small steps down, one pulse at a time. Nothing in the figure is a measured value. Numbers for how well a real device does each job come next.

Two ways to use it

A weight can be written once or written all the time. In inference the network was trained elsewhere, digitally and with noise injected on the weights in Joshi and colleagues’ experiment, and the trained weights were transferred to the devices and then read. In training in place, the array changes its own weights as it learns. Inference needs a cell that holds still. Training needs a cell that moves by the same small amount every time it is asked to, in either direction, and still holds. Most of what follows is about how hard that second job is. Back in the fifth section, the cost of the two to the chip around the array comes up again.

Two kinds of network

A survey frames a second split, by the kind of network. In spiking networks the connections are updated by local learning rules such as spike-timing-dependent plasticity. In deep networks the arrays do analog matrix-vector products inside backpropagation, in the words of the abstract by Burr and colleagues. This part follows the deep-network use. Not every neuromorphic chip puts its weights in a new kind of memory. Chips whose sources name a memory as the weight are sorted from the ones whose sources do not in the last section.

How a synapse is scored

A cell that can hold, move and repeat has to be judged on several things at once, and the sources do not all define them the same way. Plain-word names for the metrics come first, then the targets that simulations ask for, then the ones that are hard to meet.

The scorecard

Two reviews list what to measure. Burr and colleagues judge devices proposed as synapses on conductance dynamic rangeDynamic range is the ratio between the highest and lowest conductance a cell can be set to, and it limits how many distinct weights one cell can hold., nonlinearity and asymmetry of the conductance response, retention, endurance, required switching power and device variability, as their abstract puts it. Yu’s review names multilevel states, weight-update nonlinearity and asymmetry, and variation and noise as the desired characteristics, and its abstract says it surveys phase-change, resistive, ferroelectric and floating-gate candidates.

In plain words:

• Dynamic range is how far apart the highest and lowest conductance a cell reaches are, as a ratio.
• Nonlinearity is when the conductance change per pulse depends on the conductance the cell already has. The sources define the metric in different ways, so this part says which one each number belongs to.
• Asymmetry is when an up pulse and a down pulse of the same size change the conductance by different amounts.
• Read noise and write noise are the scatter between two reads of one weight and the scatter in where an update ends up.
• Retention and endurance are how long a value lasts and how many updates a cell survives.
• Switching power and variability are the current an update takes and how much cells differ from each other and from cycle to cycle.

Noise budgets from simulation

Simulations that ask how bad a device can be before training fails give the most useful numbers. They are requirements that come out of models and say nothing about any measured device. For backpropagation on a resistive-memory accelerator, Agarwal and colleagues find that read noise should have a standard deviation below 5% of the weight range. Write noise should stay below 0.4% of the weight range and up to 300% of a characteristic update step, for the datasets they tested. To allow parallel reads and writes, the write current should be below 100nA per device, and all three figures come from the abstract.

Gokmen and Vlasov stress-tested a different design. Their simulations also set a range requirement, a lower estimate of 600 states between the bottom and the top of a weight. Their simulated backpropagation tolerated up to 150% noise in each weight update and up to 10% read noise on columns or rows, each at a 0.3% error penalty. Because the two sets of numbers come from different networks and definitions, they do not merge into one specification. They agree on the shape. A read has to be clean, while a single update can be far noisier than its own size.

What is hard to meet

Of the specifications Gokmen and Vlasov derived, asymmetry between up and down steps was the most restrictive, with a 5% asymmetry threshold. A global asymmetry can be compensated by pulse design, but device-to-device variation in asymmetry can be compensated only if devices are updated one at a time. Agarwal and colleagues reach the same place from the other side. Asymmetric nonlinearity in the change of conductance against pulse number causes weight decay and cuts accuracy sharply, while moderate symmetric nonlinearity had no effect in their simulations.

Tolerances also add up. With every stress-tested parameter at its individual threshold, Gokmen and Vlasov’s simulated MNIST test error reached 5.0%, 3.0 percentage points above the baseline.

Figure 1.2 puts the scorecard in one table. It has gaps, and it says so: no on/off ratioThe on/off ratio is how many times more current a low-resistance cell passes than a high-resistance one at the same voltage. A bigger ratio makes reads easier., retention time or endurance count appears as a requirement, and it borrows no figure from elsewhere. Gokmen and Vlasov ask only for endurance to small steps. Simulations call the accented row the most restrictive.

Candidates measured so far

Few devices have been measured against this list. Jerry and colleagues showed a ferroelectric transistor synapse with 5-bit symmetric potentiation and depression, a 45x tunable range in conductance and a 75ns update pulse. Onen and colleagues made silicon-compatible protonic programmable resistors that shuttled protons in nanoseconds and showed symmetric, linear and reversible modulation with a 20x dynamic range. Both are device demonstrations reported in abstracts. Beyond NAND, Part 4 covers the ferroelectric physics.

Tools that turn a metric into accuracy

Chen and colleagues’ abstract says NeuroSim maps synaptic device properties up through array circuits to network topology, and that they used it to study how nonideal analog memory properties affect online learning accuracy and to compare SRAMSRAM is fast memory of latching cells with several transistors each. It needs no refresh but takes far more area per bit than DRAM, so it serves as on-chip memory., digital and analog memory designs. IBM’s open simulator for analog arrays models device-to-device and cycle-to-cycle variation, response curves, and weight and output noise, and provides statistical programming-noise and drift models calibrated on phase-change hardware, according to Rasch and colleagues.

Updates that miss the target

The crossbar of Beyond NAND, Part 5 multiplies by reading weights. Updates are the other side, and the question is what happens when the array has to change them.

What an ideal update looks like

An ideal weight moves up by one small step for an up pulse and down by the same step for a down pulse, wherever it sits in its range. A device with unequal up and down steps has one conductance where the two steps are equal, called its symmetry point. Device-to-device variation scatters those points, so across an array they do not all sit at zero weight, as Gokmen and Haensch show.

What went wrong in practice

In-place training on analog nonvolatile memory has generally scored below software training, because of insufficient dynamic range and excessive weight-update asymmetry, as the abstract of Ambrogio and colleagues puts it. Beyond NAND, Part 5 has the lopsided steps measured on a resistive crossbar. The simulations above answer what they do to learning. Asymmetric steps decay the weights and cut accuracy. Each of the three fixes below spends something to get that accuracy back, whether hardware, devices or area.

Fix one: add hardware

Ambrogio and colleagues combined long-term storage in phase-change memoryPhase-change memory stores data in a material that switches between a crystalline, low-resistance state and an amorphous, high-resistance state when heated. with near-linear updates on volatile capacitors, and moved weights between them with polarity inversion to cancel device-to-device variation. Their mixed hardware and software scheme reached software-equivalent accuracy on MNIST, CIFAR-10 and CIFAR-100 test sets, according to the abstract. Beyond NAND, Part 5 gives the size of that hardware and says its efficiency figure was calculated rather than measured. It needs two kinds of device and a transfer step with polarity inversion between them.

Fix two: several devices per weight

Boybat and colleagues spread one weight over several devices. Their multi-memristive synapse updates only one of the N devices at a time, selected by a global counter, so the combined conductance has a wider range and finer steps than any single device. A simulation using a measured model of phase-change nonlinearity gave a conventional two-device differential synapse below 15% on MNIST. Ideal double-precision weights score 97.8% on the same network. With several devices per synapse and an arbitration counter, the same simulation exceeded 88.9% in the differential architecture and exceeded 90% in the non-differential one. The authors also ran unsupervised learning of temporal correlations on a spiking network using over a million phase-change devices. Each weight takes several devices instead of one or two, plus the counter.

Figure 1.3 shows the problem and the fix. Its left panel is an illustrative shape with no data behind it. Up pulses raise the conductance quickly and the curve flattens, down pulses lower it more slowly, and the two paths never retrace each other, which leaves a symmetry point between them. Its right panel is sourced and simulated: a two-device synapse falls below 15% with a measured phase-change nonlinearity, and extra devices bring most of the accuracy back. Dashed bars mark bounds, since the source states a limit instead of a point value.

Fix three: change the algorithm

Gokmen and Haensch’s Tiki-Taka algorithm leaves the devices alone. Device asymmetry adds an unintended cost term to stochastic gradient descent. Tiki-Taka minimizes that term together with the network’s objective, and it matched symmetric-device accuracy in simulations of fully connected, convolutional and LSTM networks. This is a simulation result. It moves the fix from the device into the training method, and it pays in area: it needs two sets of weight matrices for each layer, up to a factor of 2 in the worst case, which the authors say stacking the devices in layers above the transistors would eliminate, and they judge the overall cost minimal.

The projection and the gap

An early design study made the case for all this effort. Gokmen and Vlasov’s 2016 study projected 30,000x acceleration over state-of-the-art microprocessors and 84 TOPS/W for training a network of about 1 billion weights on an array of resistive processing units. That is a projection from simulation, not a measurement. Set it beside the phase-change chips in the last section, which hold tens of millions of weights and run inference with the weights programmed in advance.

Noise, drift and the weights you can trust

Once a weight is set it still moves. This section covers how training prepares for noise and drift, and how many bits a weight really holds. The read-side numbers are in Beyond NAND, Part 5.

Time

Beyond NAND, Part 5’s “Time” gives the drift law as a power of time, G(t) = G(t0)(t/t0)^-v, where t is the time since programming, and what drift did to a network with no compensation. For IBM’s devices v averages about 0.06, according to Joshi and colleagues, whose paper has an open copy. The new point here is the common fix and why it falls short. A per-core rescale of the matrix-multiply result cancels the average drift, but the exponent v depends on the conductance state and varies between devices, so the error of the multiply still grows with time, as Le Gallo and colleagues found.

Training for the hardware you will get

Because the hardware is noisy, the network can be trained to expect it. Joshi and colleagues trained ResNet-type networks with noise on the weights. A simulator of phase-change devices gave 93.7% on CIFAR-10 and 71.6% top-1 on ImageNet after mapping to phase-change memory. Those two figures are simulated. Their hardware/software experiment is the ResNet-32 one above, with the rest of the network simulated.

Rasch and colleagues retrained with a realistic analog-tile model and reached iso-accuracy on many networks, including CNNs, RNNs and transformers, in simulation. They found that nonidealities that add noise to inputs or outputs hurt more than noise on the weights. Noise in training does not cover everything. Luquin and colleagues posted a modeling study as a preprint. Gaussian-noise fine-tuning gave BERT-type transformers resilience against converter quantization and phase-change read noise, but was less effective against IR-drop, the voltage lost along the wires. IR-drop is deterministic and nonlinear, and it depends on all the inputs applied at once. The Crosspoint Lab plays out what spoils the sum and working around the damage on one small array.

How precise a weight really is

On the 64-core chip, a direct measurement shows how precise a phase-change weight is. Compared with a digital engine with 8-bit input and output and N-bit weights, mapping each weight onto strictly one device of the right polarity gave a precision close to 3-bit weights on the chip, and allowing up to two devices gave between 3-bit and 4-bit weights. A ResNet-9 ran on CIFAR-10 at 92.81% accuracy when a weight could use up to two devices and 92.23% when it used strictly one, against a software baseline of 93.67%. Bayat and colleagues built a different design, two passive 20x20 metal-oxide memristor arrays trained off the array and loaded afterward (ex situ), and it scored within 3% of its simulated classification fidelity.

Figure 1.4 pairs the two halves of this section. Its left panel is drawn from the drift law instead of from measurements. Even an average device keeps losing conductance as time passes, and two dashed neighbors show that devices differ, so one rescale cannot cancel them all. Its right panel plots accuracy on an axis that starts at 91 and skips zero. Both chip results sit within a point and a half of the software baseline, and the noise-trained simulation sits beside them as a different network and a different kind of result.

Fixing precision outside the array

A mixed-precision scheme moves the last bits out of the array. Analog memory does the bulk of a computation while a digital machine iteratively corrects the result. It solved a system of 5,000 linear equations using 998,752 phase-change devices in the experiment of Le Gallo and colleagues. That matrix was a problem to solve and held no network’s weights, but the trade is the same one. Analog memory does the bulk of the arithmetic, and a digital step supplies the accuracy, so the system keeps the energy and area efficiency of in-memory computing.

The converters, the periphery, and training against inference

Converters, amplifiers and control sit around the array, and an array that is also written needs more of them. What the chip designs say about keeping them small comes first. The share of power and area they take in two simulated designs is in Beyond NAND, Part 5’s converter section.

What the converters do to the budget

Ankit and colleagues simulated a memristor accelerator in which larger crossbars need converters of higher resolution, and converter overhead grows non-linearly with resolution, which offsets the saving from sharing peripherals over more cells. The same simulated accelerator was reported at 8.3x the peak area efficiency and 1.65x the peak power efficiency of a Google TPU. A review of analog accelerators says that exploiting memory arrays has proved hard mainly because of the overhead of the peripheral circuits and the nonideal properties of the memory devices, as the abstract of Xiao and colleagues puts it.

Hardware has made a different choice. According to the authors of the 64-core chip, earlier analog-weight designs needed slow high-resolution converters that often have to be multiplexed across columns. Their chip places a compact current-controlled-oscillator converter on each row instead, so a whole matrix-vector multiply runs in parallel. Each row converter delivers two 12-bit unsigned results, one for the positive current and one for the negative. That pair is the signed weight of the first section showing up again at the edge. Its on-chip calibration does not have to be redone for a new network. In its peak case, 8-bit input and output matrix-vector multiplies alone in the low-precision one-phase read mode with all 64 cores busy, the 14nm chip delivers 63.1 TOPS at 9.76 TOPS/W. In the four-phase, high-precision mode used for the accuracy tests, the chip is about four times lower on both.

Inference and training want different things

An inference array is programmed once and read for as long as it is used. Ambrogio and colleagues, building a larger chip, say why: because nonvolatile devices have finite endurance and slow, power-hungry programming, analog-SI systems must be fully weight stationary, with every weight programmed before inference begins, as their analog-SI chip paper states. A training array does the opposite. Each cell is read and nudged all the time, so the history requirement and the asymmetry limits of the earlier sections apply to every cell, and so does the endurance for small increments that Gokmen and Vlasov describe. Updating also adds circuits of its own, namely stochastic-pulse circuits on every row and column, so that row and column pulses coincide at every cell in parallel.

Figure 1.5 sets one weight’s life under each use on one time axis. Inference spends a short burst of writes and then reads. Training interleaves a read and a small write across the whole axis. As a schematic, the figure claims nothing beyond that contrast. Chips in the next section sit on the top panel.

What has shipped, and what is research

Three kinds of thing get called neuromorphic or analog SI, and each gets its own subsection below. First come research chips that ran. Second are a vendor’s claims about its own chiplets. Third are other neuromorphic chips whose sources name no new memory for their weights.

Research chips with weights in phase-change memory

IBM’s 64-core chip, above, is a 14nm research chip. A related 14nm analog-SI chip combined 35 million phase-change devices across 34 tiles and reached up to 12.4 TOPS/W chip-sustained performance, according to Ambrogio and colleagues. On the MLPerf speech-recognition network, a system of five such chips reached near software-equivalent accuracy with more than 45 million weights mapped onto more than 140 million phase-change devices. The 12.4 and 9.76 TOPS/W figures come from different chips and tests, so they are not a ranking.

Early chips were small. According to the authors of the 64-core chip, earlier single- or few-core in-memory chips, in several memory technologies, ran only small networks of about 100,000 weights or fewer on simple tasks such as MNIST, or emulated larger networks by reprogramming one layer at a time. They add that multi-core chips supporting more than 1M weights came more recently. They also say why a memory and not SRAM: SRAM-based analog designs need off-chip weight buffers because their cells are too large to hold weights past 10MB at reasonable area, and SRAM cells lose the weights when power cycles. Both IBM chips run inference with weights programmed in advance, and neither updates its weights in place.

One vendor, with weights in flash

Mythic says its analog compute-in-memory processor stores SI model weights in flash memory and computes in the analog domain, and claims up to 100x system-level performance per watt per dollar against comparable conventional digital or GPU inference. It also says on its technology page that a single chiplet runs edge SI at 0.7W and that 1,024 or more chiplets can be combined for large-model inference. These are vendor claims from two marketing pages, and no datasheet or independent measurement is cited for them, so this guide does not call the part a product on sale and does not carry the 100x figure into any comparison.

Other neuromorphic chips, for contrast

The better-known neuromorphic chips take a different route, and none of them is described as holding its weights in a phase-change, resistive or other new memory. Merolla and colleagues report IBM’s TrueNorth packing 4096 neurosynaptic cores, 1 million programmable spiking neurons and 256 million configurable synapses into 5.4 billion transistors, and drawing 63mW while running 400-by-240-pixel video at 30 frames per second. Davies and colleagues describe Intel’s first Loihi as a 60mm² chip in a 14nm process with programmable synaptic learning rules.

Loihi and its successor Loihi 2 model spiking neurons with a digital discrete-time approach instead of analog circuit dynamics, according to Orchard and colleagues. That describes the neuron model and says nothing about what holds the weights.

Hala Point packages 1,152 Loihi 2 processors made on Intel 4 into 1.15 billion neurons and 128 billion synapses and draws at most 2,600W, and Intel calls it a research prototype. Intel reports efficiency as high as 15 TOPS/W on a pruned 8-bit multilayer perceptron with 10:1 sparsity and 10% activation, in a test with random-noise input.

Modha and colleagues describe IBM’s NorthPole inference chip as eliminating off-chip memory by putting compute and memory together on the die, and it scored 25x the frames per second per watt of a GPU on a comparable 12nm process on ResNet50, according to the abstract.

Those efficiency figures use different precision and different tests. Set side by side they would mislead, so this guide does not.

Where that leaves memory as a synapse

In-place training on analog memory appears here as simulations, as the mixed phase-change and capacitor design above, as a spiking-network experiment on over a million devices, and as the small resistive arrays trained in place in Beyond NAND, Part 5. The phase-change chips built at scale do inference with weights programmed in advance, and the one commercial flash-based claim here is a vendor claim. TrueNorth, Loihi, Hala Point and NorthPole are not described as using a new memory for the weights, and only Loihi and Loihi 2 are described as digital.

Figure 1.6 starts from a metric and follows each branch to the failure that one source ties to it in this part. Asymmetric steps decay the weights, read noise over its budget loses accuracy, IR-drop slips past noise training, drift error grows despite a rescale, and converter overhead grows non-linearly with resolution. Part 2 turns to NAND flash stacked beside an accelerator, and the series page lists the rest.

The ferroelectric and protonic devices above have shown symmetric steps as single devices, and neither has been shown in an array that is written all the time. Whether one can do that is the open question.