Google's TurboQuant Compresses AI Memory 6x, Rattles Chip Industry

Google Research has published what may be the most consequential inference optimization paper of the year. TurboQuant, a new compression algorithm for large language model runtime memory, achieves 6x compression of KV caches with zero accuracy loss and zero retraining required.

On NVIDIA H100 GPUs, the 4-bit variant delivers up to 8x speedup in computing attention logits. The paper will be presented at ICLR 2026, and open-source code is expected around Q2 2026.

The internet has already dubbed it "Pied Piper," after the fictional compression company in HBO's Silicon Valley. The financial markets were less amused — memory chip stocks, including Micron, took a significant hit as investors reassessed demand projections for high-bandwidth memory.

How TurboQuant Works

The algorithm uses a two-step approach to compress the KV cache — the runtime working memory that large language models use to track context during inference.

Step one: PolarQuant. Random rotation of data vectors creates a distribution that is more amenable to high-quality quantization. This step alone enables aggressive compression without the information loss that typically accompanies naive quantization.

Step two: Quantized Johnson-Lindenstrauss (QJL). A single bit of residual compression eliminates the bias introduced by quantization, preserving model accuracy. The combination compresses KV caches down to 3 bits per value — a dramatic reduction from the standard 16-bit floating point representation.

The key breakthrough is that this works as a drop-in replacement. No model retraining. No architecture changes. Any existing large language model can benefit immediately.

Why the Chip Industry Is Nervous

The KV cache is one of the primary bottlenecks in serving large language models. As context windows have grown — from 4K tokens to 128K and beyond — the memory required to store KV caches has ballooned. This has been a major driver of demand for high-bandwidth memory (HBM), the premium memory chips manufactured by companies like Micron, SK Hynix, and Samsung.

A 6x reduction in KV cache size directly threatens that demand. If inference providers can serve the same models with dramatically less memory, the multi-billion-dollar HBM buildout that the industry has been planning may need to be significantly scaled back.

The market reaction was immediate. Micron and other memory chip makers saw billions wiped from their market capitalizations as analysts raced to model the implications.

The Inference Economics Revolution

For AI companies running large-scale inference, TurboQuant could cut costs by 50% or more. The savings come from two sources: less memory hardware needed per GPU, and faster computation enabling higher throughput per server.

At current cloud pricing, inference costs for a GPT-4-class model run roughly $0.01-0.03 per 1,000 tokens. A 50% reduction would accelerate the adoption of AI across cost-sensitive applications — customer service, code generation, document processing — where unit economics are currently marginal.

Open Questions

Several important caveats remain. The paper demonstrates results on specific model architectures and hardware configurations. Real-world deployment across the diverse landscape of production AI systems may reveal edge cases or compatibility issues.

Google has not yet released the code, and independent reproduction is pending. The ICLR presentation will provide the first opportunity for the broader research community to scrutinize the methodology.

There is also the question of whether competing hardware vendors — NVIDIA, AMD, Intel — will optimize their architectures to work with TurboQuant-style compression, potentially recapturing some of the efficiency gains at the hardware level.

The Bigger Picture

TurboQuant represents something the AI industry has been waiting for: a software breakthrough that meaningfully changes the hardware economics of AI. For the past three years, the dominant narrative has been that AI progress requires ever-larger clusters of ever-more-expensive chips. TurboQuant suggests that algorithmic innovation can bend that cost curve.

If the results hold up in production, this is not just an optimization — it is a structural shift in how the AI infrastructure market operates.

Sources: Google Research Blog, TechCrunch, VentureBeat, Financial Content