Back to Mind & Machine

DeepSeek-V3: Dancing in Hardware Chains

The Art of Making AI Cheaper & Faster

The Big Idea

Think of AI Like a Hospital
A dense model is like calling ALL doctors for every patient (expensive!)
A MoE model is like a hospital with 256 specialists — only call the 8 most relevant ones!
671B
Total "Brain Size"
37B
Actually Used
256
Specialists
8+1
Called Each Time
E1
E2
...
E256

Only 9 wake up: 8 specialists + 1 always-on helper

Problem 1: The "Superstar" Problem

What Goes Wrong
The Hospital
90% patients!

One "superstar" gets ALL patients! Others become idle & forget their skills.

The Solution: Always-On GP
Balanced Care
GP
Complex cases →
Heart
Brain
Bones
Lungs

GP handles basics! Specialists focus on complex cases.

Problem 2: The "Jack of All Trades"

What Goes Wrong
Dr. A
Both do
"okay"
at everything
Dr. B
2x cost, same quality!
The Solution: Tiny Specialized Experts
BEFORE: 8 Big "Jacks of All Trades"
Heart
+ Brain
Heart
+ Brain
Bone
+ Skin
Expensive, mediocre quality
Transform
AFTER: 256 Tiny Specialists
+250
Cheap, superior quality
Think: 256 specialist nurses vs 2 general practitioners!

Problem 3: The Overworked Doctor

What Goes Wrong
Overloaded!
Too many!
Idle...
Idle...
The Solution: Smart Queue System
-0.5
Low priority
"I'm busy, send elsewhere"
+0.8
High priority
"Send me more!"
+0.6
Higher priority
"I can help!"

Like doctors voluntarily adjusting their workload based on availability!

Problem 4: The Multi-Building Hospital

What Goes Wrong

Expert data lives on different GPUs (like different hospital buildings). Moving data between them is SLOW!

Building A
1
2
3
SLOW!
Building B
50
51
52
The Solution: Local Neighborhood Care

Each patient can only use doctors from 3 nearby buildings! Forces local specialization — no more "commuting"!

GPU 0
1
2
GPU 1
3
4
GPU 2
5
6

Doctors must specialize locally — no freeloading!

Why Exactly 8? Hardware Limits!

H800 GPUs have limited memory bandwidth (due to sanctions):

37GB
Activation Memory/GPU
3.35TB/s
Max Speed
400GB/s
Inter-GPU (slow!)
Think of it like a whiteboard
The 37GB is like a whiteboard — you need space to work, but it clears after each patient! It's temporary activation memory needed per GPU during a forward pass, NOT the stored model weights.
More experts = more communication between GPUs = more waiting time!

Here's how the percentages are calculated:
  • 8 experts: Communication time fits perfectly inside computation time = 100% speed (no waiting!)
  • 10 experts: Communication takes 25% longer than computation = 25% slowdown
  • 12 experts: Communication takes 50% longer than computation = 50% slowdown
The Sweet Spot: 8 Experts
100%
8 Experts
Perfect!
75%
10 Experts
Slower...
50%
12 Experts
Too slow!

More experts = more waiting between GPUs = slower training!

The Summary

1
Always-On GP

One expert handles basics, others specialize

2
256 Tiny Experts

Small but specialized = better quality

3
Smart Queue

Auto-balance workload without punishment

4
Local Only

Use nearby GPUs to avoid waiting

5
Hot Replication

Busy experts clone themselves when needed

6
8+1 = 9

9 experts cover broader knowledge

How 6 Solutions Work Together
GP handles basics
→
256 tiny experts
→
auto-balance
→
local-only
→
replicate
→
8+1=9
The Bottom Line: DeepSeek-V3 proves you can build GPT-4 level AI for ~$5.5M by making smart trade-offs. It's not "the best architecture" — it's the best architecture FOR LIMITED HARDWARE. When hardware improves, the architecture will evolve!