A few days back I was reading this post about AI Infrastructure at periodic, when this line WideEP Tensor parallelism (TP) with latent attention (MLA) is not ideal for high-throughput workloads. caught my eye.
I’m aware of TP but really didn’t understand how it conflates with MLA attention so here’s why
But first lets understand few concepts first :
Background
KV cache
-
When a language model generates text token by token, each new token needs to “look back” at all previous tokens via attention.
To avoid recomputing everything from scratch at every step, the model stores the Key (K) and Value (V) vectors for every past token — this stored memory is the KV cache. -
The problem: this cache grows with sequence length and gets huge. For long contexts or many simultaneous users, the KV cache is often the single biggest consumer of GPU memory during inference — bigger than the model weights themselves. So reducing KV cache size is a big deal for serving efficiency (it lets you fit more users/requests on the same GPUs = higher throughput).
Tensor parallelism (TP)
-
A single large model often doesn’t fit on one GPU. Tensor parallelism shards individual layers ( basically matrix multiplications) across multiple GPUs, so each GPU holds a slice of the work and they communicate to combine results.
-
For classic multi-head attention, this splits naturally: e.g., with 8 attention heads and 4 GPUs, each GPU handles 2 heads, including the K/V vectors for just those heads. So the KV cache is also naturally split across GPUs — no duplication, each GPU stores only its own slice.
-
Attention heads = Splitting the columnar vector/tensor representation of a token across n pieces and running the attention computation across such n pieces with each such head/computation attending to different kinds of patterns(one might track adjacent
words, semantics, etc)
MLA (Multi-head Latent Attention)
-
MLA is a technique (from DeepSeek’s models) that compresses the K and V vectors for all heads into one small shared “latent” vector, rather than keeping separate full-size K/V per head.
-
This is great for memory efficiency in general — the whole point is to shrink the KV cache dramatically by not storing redundant per-head information.
-
Deepseek-V3 and above models implements this attention technique. Read more here.
Mixture of Experts (MoE) and Expert Parallelism (EP)
- A normal feedforward layer (FFN) runs every token through the same weights.
- MOE replaces it with many smaller FFNs called experts. A small router sends each token to only a few of them (e.g. top-2 of 64).
- Expert parallelism places whole experts on different GPUs and routes tokens to wherever their expert lives:
- Router picks experts for each token.
- Tokens are sent to the right GPUs (all-to-all).
- Each GPU runs its local experts.
- Results are sent back (all-to-all).
- So basically sharding by small feedforward neural networks instead of the columnar matrix in TP.
- WideEP means spreading experts very thin across a large number of GPUs.
- Attention stays on a small group (or even a single GPU per replica), with each replica handling its own requests and its own KV cache. This is data-parallel attention. No TP duplication.
- The MoE layers scale out wide with EP, so you still use many GPUs for capacity.
- Net result: scale the model without multiplying the KV cache.
The problem: TP + MLA
-
Here’s the crux: in standard multi-head attention, K/V naturally splits by head across TP ranks (GPUs).
But in MLA, there’s just one shared latent KV representation used to reconstruct all the heads’ worth of attention information. -
Since TP still needs to split the computation (the different attention heads) across GPUs, but there’s only one compressed latent KV cache to draw from, every GPU participating in tensor parallelism needs a full copy of that latent KV cache to do its share of the head computation — you can’t split a single shared latent vector into pieces per GPU the way you could split separate per-head K/V.
-
So instead of the KV cache being divided across N GPUs (1/N each, like in normal MHA), every one of the N GPUs stores the entire KV cache — it gets duplicated N times, which is the exact issue MLA is trying to avoid.
-
Hence TP can’t go hand in hand with MLA since it will end up wasting GPU memory that could otherwise hold more concurrent requests, directly hurting throughput — the opposite of what you want
An Alternative : WideEP + MLA
-
So instead of using TP to shard attention (which causes this KV duplication problem under MLA), the idea is to use EP to scale out the MoE feed-forward layers instead, since experts split naturally without this duplication issue.
-
This avoids paying the MLA-duplication tax while still scaling the model across many GPUs.
I would recommend reading the complete post of AI Infrastructure at Periodic for more.