One Deep Concept #01

PagedAttention: From Operating System Virtual Memory to Large Language Model Memory Systems

English Version

Introduction

Large Language Models have created a new category of computing problems.

At first, researchers focused mainly on model intelligence:

  • larger parameters
  • better training data
  • improved architectures

However, when models entered real-world deployment, another bottleneck appeared:

How do we efficiently run many LLM requests at the same time?

The answer is not only about faster GPUs.

It is about system design.

PagedAttention is a representative example showing that AI infrastructure innovation often comes from connecting ideas across fields.

The core idea:

LLM KV Cache Problem

        ↓

Dynamic Memory Management Problem

        ↓

Operating System Virtual Memory

        ↓

Paging Mechanism

        ↓

PagedAttention

1. Concept

PagedAttention is a memory management technique designed for efficient Large Language Model inference.

The key observation:

Traditional KV Cache management assumes memory must be continuous.

PagedAttention breaks this assumption.

Instead of storing KV Cache as one large continuous block:

Traditional:

[Token1][Token2][Token3][Token4][Token5]

it divides memory into smaller blocks:

Block 0
Block 1
Block 2
Block 3

Logical continuity is maintained while physical memory can be scattered.

This is similar to how operating systems manage virtual memory.

2. Background

Transformer inference repeatedly uses previous token information.

Attention requires:

Attention(Q,K,V)
=
softmax(QK^T / sqrt(d))V

During generation, previous Keys and Values are reused.

Therefore models store them in:

KV Cache

KV Cache trades memory for computation:

More Memory
      ↓
Less Recalculation
      ↓
Faster Generation

But as context length and users increase, KV Cache becomes a major memory bottleneck.

3. Problem

The challenge is not simply memory size.

It is memory management.

Requests have unpredictable lengths:

Request A → 500 tokens

Request B → 8000 tokens

Request C → 100 tokens

Pre-allocating maximum memory causes waste.

This creates:

  • Internal fragmentation
  • External fragmentation

A GPU may have free memory but still cannot allocate a large continuous region.

4. Core Innovation

PagedAttention borrows the abstraction of operating system paging.

Operating systems separate:

Virtual Memory

↓

Physical Memory

through page tables.

PagedAttention applies the same idea:

Logical KV Blocks

↓

Block Table

↓

Physical GPU Blocks

The model sees continuous memory.

The hardware stores flexible blocks.

5. Engineering Trade-offs

Advantages:

  • Higher memory utilization
  • More concurrent requests
  • Better serving throughput
  • Support for prefix sharing

Costs:

  • More complicated kernels
  • Additional address translation
  • Block size optimization problems

The lesson:

Every optimization moves complexity somewhere else.

6. Researcher’s View

PagedAttention teaches a broader innovation method:

Observe bottleneck

↓

Abstract the problem

↓

Find similar solved problems

↓

Transfer mature ideas

↓

Create new systems

Future directions may include:

  • Token-level KV virtualization
  • Semantic-aware memory management
  • Hierarchical AI memory systems
  • Predictive KV allocation

中文翻译与深度解释

引言

大模型的发展让计算机系统面临新的挑战。

早期重点是：

  • 更大的参数量
  • 更好的数据
  • 更强的模型结构

但当大模型真正部署到生产环境后，一个新的问题出现：

如何让大量用户同时高效使用大模型？

这已经不只是模型问题，而是系统工程问题。

PagedAttention 的价值就在这里：

它证明了 AI 系统创新不仅来自新的算法，也来自跨领域思想迁移。

1. 什么是 PagedAttention？

PagedAttention 是一种优化 LLM 推理过程中 KV Cache 管理的方法。

传统方法：

一整个连续内存区域保存 KV Cache

问题：

每个用户生成长度不同，导致大量浪费。

PagedAttention：

KV Cache

↓

拆成 Block

↓

灵活管理显存

它借鉴了操作系统中的分页机制。

2. 为什么需要 KV Cache？

Transformer 每生成一个 Token，都需要关注历史 Token。

如果每次重新计算：

Token1
Token2
Token3
...

计算量巨大。

因此保存：

Key
Value

避免重复计算。

但是：

模型越大
+
上下文越长
+
用户越多

↓

KV Cache 越大

最终显存成为瓶颈。

3. PagedAttention 的核心思想

它真正解决的问题不是：

“如何减少 KV Cache。”

而是：

“如何像操作系统管理内存一样管理 KV Cache。”

传统：

逻辑地址 = 物理地址

PagedAttention：

逻辑地址

↓

映射表

↓

物理地址

因此：

模型认为数据连续。

GPU 可以自由安排位置。

4. 工程启示

PagedAttention 最重要的价值不是一个技术细节。

它展示了一种研究方法：

发现问题

↓

抽象本质

↓

寻找其他领域类似问题

↓

迁移成熟思想

↓

重新设计系统

很多重大创新都来自这种方法。

Research Reflection

真正需要培养的能力不是记住：

“PagedAttention 是 KV Cache 分块。”

而是学习：

看到一个问题时：

不要马上寻找答案。

先问：

这个问题本质属于什么类别？

因为：

具体问题

↓

抽象问题

↓

跨领域连接

↓

创新设计

这才是 AI 时代工程师和研究者的重要能力。