AI Memory: State of the Market

Not investment advice. This video summarizes public sources and MESH's CORTEX notes as of October 2026. Company figures are the companies' own claims.

Transcript

0:00 · Intro

Here's a plain-English look at AI memory, as of October 2026. We'll cover why memory became the bottleneck, which kind of memory each style of AI compute depends on, and what that means for the market.

0:14 · Why now

Why does memory matter so much? Because compute has outrun it. David Patterson and Xiaoyu Ma found that from 2012 to 2022, Nvidia GPU 64-bit compute rose 80 times, while memory bandwidth rose only 17 times. And AI leans on the slow side. Generating each new token, called decode, is memory-bound: the chip waits on memory, not math. One analysis finds that in agentic work with long outputs, decode can take around 90 percent of the time.

0:44 · The ladder

AI systems use a ladder of memory, trading speed for capacity. At the top is SRAM on the chip itself: the fastest, but only megabytes to gigabytes. Next is HBM, stacks of DRAM packaged right beside the processor, with hundreds of gigabytes per GPU. Below that sit cheaper LPDDR, GDDR and DDR memory, then shared pools over CXL, and finally flash, which holds the most but is the slowest. Start with GPUs and custom accelerators.

1:11 · GPUs & SRAM

They run on HBM. Nvidia's Rubin GPU carries 288 gigabytes of HBM4 at 22 terabytes per second. AMD's MI455X carries 432 gigabytes, and Google's Ironwood TPU has 192. That memory is expensive. Epoch AI estimates that HBM and advanced packaging make up about two-thirds of the cost to produce an Nvidia B200. A second camp keeps the model on the chip, in SRAM. Groq's latest LPU holds 500 megabytes of SRAM at 150 terabytes per second, and Cerebras puts 44 gigabytes on a single wafer. That makes decode very fast, but capacity is tiny: Nvidia's 256-chip LPX rack holds just 128 gigabytes in total. Nvidia bought in anyway. In December 2025 it licensed Groq's technology and hired its founder, in a deal CNBC reported at 20 billion dollars. So the newest trend is to split the work by phase. Prefill, reading the prompt, is compute-bound. Decode is memory-bound. So each phase can get different hardware and different memory. Nvidia's Rubin platform adds Groq-based LPX racks for decode, and AWS is pairing Trainium for prefill with Cerebras for decode. Nvidia also announced Rubin CPX, a prefill chip on cheaper GDDR7, though it was later reportedly redesigned around HBM4. Then there's the KV cache, the model's working memory of a conversation, which grows with every token and every user.

2:34 · Tiers & edge

It can't stay in HBM for long, so it spills down the ladder. Nvidia's Vera CPU adds up to 1.5 terabytes of LPDDR5X that works as one pool with the GPU's HBM, and its BlueField-4 storage adds a flash tier below that. SanDisk and SK hynix have also published a spec for high-bandwidth flash, up to 512 gigabytes per stack, with SanDisk's first samples now expected in 2027. On phones and laptops, it's the same story in miniature. On-device AI runs on LPDDR5X, and Apple's researchers note that generating tokens is bounded by memory bandwidth, not compute. Apple's base M5 has 153 gigabytes per second of bandwidth, and Qualcomm's top Snapdragon X2 has 228. Some startups are designing around HBM altogether. d-Matrix computes inside SRAM and is moving to stacked 3D DRAM. Positron is building on LPDDR5X, with two terabytes per accelerator, though its chip hasn't taped out. Others compute inside the memory itself. In Samsung's own demo, its processing-in-memory LPDDR5X tripled an 8-billion-parameter model's token rate, and analog startups EnCharge and Mythic have each raised over 100 million dollars. Put it together, and each style of compute leans on a different rung of the ladder. GPUs and accelerators on HBM, fast decode chips on SRAM, CPUs and edge devices on LPDDR, and the KV cache on DRAM, CXL and flash.

4:09 · Makers

For the memory makers, this is a boom. SK hynix posted a 76 percent operating margin in the second quarter. Micron's latest quarterly revenue was 54 billion dollars, nearly five times a year earlier. And Samsung is back: Counterpoint estimates its share of HBM revenue rose from 21 to 33 percent between the first and second quarters, while SK hynix slipped from 58 to 50. All three say they now ship HBM4, though TrendForce says low yields still limit volume. Micron sizes the HBM market at about 35 billion dollars in 2025, rising to around 100 billion by 2028.

4:38 · Market

Yole's mid-2025 forecast reached a similar figure only in 2030. TrendForce now expects the whole memory market to reach about 889 billion dollars in 2026, and more than 1.28 trillion in 2027. By its estimate, memory is set to make up 47 percent of big cloud providers' capital spending this year. The flip side is a shortage. Conventional DRAM contract prices roughly doubled in the first quarter of 2026, according to TrendForce. It ripples outward: TrendForce says Nvidia has halved the LPDDR memory on its Vera Rubin superchip modules, and may get only about 60 percent of what it needs next year. TrendForce now expects HBM prices to jump 121 percent in 2027.

5:30 · Startups

Investors are betting big on chips that rethink memory. In 2026, Etched raised 700 million dollars at a 21 billion dollar valuation, SambaNova raised a billion at 11 billion, and Positron 875 million at 5 billion. MatX raised more than 500 million, and d-Matrix raised 275 million at a 2 billion dollar valuation in late 2025. And the exits are big. Cerebras went public in May 2026, with a market value near 40 billion dollars at pricing. Nvidia reportedly paid over 900 million dollars for Enfabrica's team and technology, and Marvell agreed to buy CXL switch maker XConn for about 540 million. There are real risks.

6:11 · Outlook

Memory is cyclical. Past supercycles have rolled into downturns within one or two years, and big new fab capacity arrives in 2028. Shortages force compromises: TrendForce says chipmakers are weighing shorter 8-high HBM stacks, and even less HBM per device. And new tiers can slip. High-bandwidth flash samples moved from 2026 to 2027, and Counterpoint notes that most of Nvidia's custom HBM claims can't yet be checked.

6:38 · Recap

To recap. Compute has outrun memory, and decode is memory-bound. Each style of compute maps to a rung: HBM for GPUs, SRAM for fast decode, LPDDR for CPUs and the edge, and DRAM, CXL and flash for the KV cache. Memory makers are booming, and prices have spiked. Investors are funding chips that rethink memory, and the exits have arrived. The open question is whether supply catches up before the cycle turns. That's the state of AI memory in October 2026. This video summarizes public sources and MESH's CORTEX notes. It isn't investment advice, and company figures are the companies' own claims.