DeepSeek-V3 New Paper is coming! Unveiling the Secrets of Low-Cost Large Model Training through Hardware-Aware Co-design

A newly released 14-page technical paper from the team behind DeepSeek-V3, with DeepSeek CEO Wenfeng Liang as a co-author, sheds light on the “Scaling Challenges and Reflections on Hardware for AI Architectures.” This follow-up to their initial technical report delves into the intricate relationship between large language model (LLM) development, training, and the underlying hardware infrastructure. The paper moves beyond the architectural specifics of DeepSeek-V3 to explore how hardware-aware model co-design can effectively address the limitations of current hardware, ultimately enabling cost-efficient large-scale training and inference.

The rapid scaling of LLMs has exposed critical bottlenecks in current hardware architectures, particularly concerning memory capacity, computational efficiency, and interconnect bandwidth. DeepSeek-V3, trained on a cluster of 2048 NVIDIA H800 GPUs, serves as a compelling case study demonstrating how a synergistic approach between model design and hardware considerations can overcome these limitations. This research focuses on the interplay between hardware architecture and model design in achieving economical large-scale training and inference, aiming to provide actionable insights for efficiently scaling LLMs without compromising performance or accessibility.

Key areas of focus in the paper include:

DeepSeek-V3 incorporates several key architectural innovations, as illustrated in Figure 1 of the paper, including the DeepSeekMoE architecture and Multi-head Latent Attention (MLA). These designs directly tackle the core challenges of scaling LLMs: memory efficiency, cost-effectiveness, and inference speed.

LLMs exhibit exponential growth in memory demands, outpacing the slower growth of high-speed memory like HBM. While multi-node parallelism offers a solution, optimizing memory usage at the source remains crucial. DeepSeek addresses this bottleneck with Multi-head Latent Attention (MLA), which employs projection matrices to compress the key-value (KV) representations of all attention heads into a smaller latent vector, trained jointly with the model. During inference, only this compressed latent vector needs to be cached, significantly reducing memory consumption compared to storing full KV caches for each head.

Beyond MLA, DeepSeek highlights other valuable techniques for KV cache size reduction, providing inspiration for future advancements in memory-efficient attention mechanisms:

Table 1 in the paper compares the per-token KV cache memory footprint of DeepSeek-V3, Qwen-2.5 72B, and LLaMA-3.1 405B. DeepSeek-V3 achieves a remarkable reduction, requiring only 70 KB per token, significantly lower than LLaMA-3.1 405B’s 516 KB and Qwen-2.5 72B’s 327 KB.

For sparse computation, DeepSeek developed DeepSeekMoE, an advanced Mixture-of-Experts (MoE) architecture (Figure 1, bottom right). MoE models offer two key advantages in terms of cost-effectiveness:

DeepSeek prioritizes both system-level maximum throughput

and single-request latency for inference speed. To maximize throughput, the model employs a dual micro-batch overlapping architecture from the outset, intentionally overlapping communication latency with computation.

Furthermore, DeepSeek decouples the computation of MLA and MoE into distinct stages. While one micro-batch performs part of the MLA or MoE computation, the other concurrently executes the corresponding scheduling communication. Conversely, during the second micro-batch’s computation phase, the first micro-batch undertakes the combine communication step. This pipelined approach enables seamless overlap of all-to-all communication with continuous computation, ensuring full GPU utilization. In production, DeepSeek utilizes a prefill and decode separation architecture, assigning large-batch prefill and latency-sensitive decode requests to different-sized expert-parallel groups, maximizing system throughput under real-world serving conditions.

The paper also touches upon the importance of test-time scaling for reasoning models and highlights the critical role of high token output speed in reinforcement learning workflows and for reducing user-perceived latency in long inference sequences. Optimizing inference speed through hardware-software co-innovation is therefore paramount for the efficiency of reasoning models.

While quantization techniques like GPTQ and AWQ have significantly reduced memory requirements primarily for inference, DeepSeek has pioneered the use of FP8 mixed-precision training for a large-scale MoE model. Despite NVIDIA’s Transformer Engine supporting FP8, DeepSeek-V3 marks a significant step as the first publicly known large model to leverage FP8 for training. This achievement, resulting from close collaboration between infrastructure and algorithm teams, along with extensive experimentation, significantly reduces computational costs while maintaining model quality, making large-scale training more feasible. Figure 1 illustrates the FP8 precision used in the forward and backward passes during training.

DeepSeek also employs low-precision compression for network communication within the DeepSeek-V3 architecture. During EP parallelism, tokens are scheduled using fine-grained FP8 quantization, reducing communication volume by 50% compared to BF16, thereby significantly shortening communication time.

Beyond traditional floating-point formats, DeepSeek experimented with a novel data type called LogFMT-nBit (Logarithmic Floating-Point Formats).

DeepSeek currently utilizes the NVIDIA H800 GPU SXM architecture (Figure 2), which, while based on the Hopper architecture similar to the H100, features reduced FP64 compute performance and NVLink bandwidth (400 GB/s down from 900 GB/s in H100) due to regulatory requirements. This significant reduction in intra-node scaling bandwidth poses challenges for high-performance workloads. To compensate, each node is equipped with eight 400G Infiniband (IB) CX7 network interface cards (

NICs) to enhance inter-node scaling capabilities.

To navigate the limitations of the H800 architecture, the DeepSeek-V3 model incorporates hardware-aware design considerations for parallelization, including: avoiding Tensor Parallelism (TP), enhancing Pipeline Parallelism (PP), and accelerating Expert Parallelism (EP). Specific details of these strategies are available in the original paper.

A key aspect of model co-design is “node-aware routing” for the TopK expert selection strategy in the MoE architecture. Given the approximately 4:1 bandwidth difference between intra-node (NVLink, ~160 GB/s effective) and inter-node (IB, ~40 GB/s effective per NIC) communication, DeepSeek designed the routing to leverage the higher intra-node bandwidth. By grouping the 256 routing experts (4 per GPU in an 8-node, 64-GPU setup) into 8 groups of 32 experts, each residing on a single node, and algorithmically ensuring that each token is routed to at most 4 nodes, DeepSeek mitigates the IB communication bottleneck and improves effective communication bandwidth during training. Tokens destined for experts on the same node can be sent via IB once and then forwarded via NVLink, reducing redundant IB traffic.

While node-aware routing reduces bandwidth demands, the bandwidth disparity between NVLink and IB complicates the implementation of communication-intensive kernels. Currently, GPU Streaming Multiprocessors (SMs) handle both network message processing and data forwarding via NVLink, consuming significant compute resources. DeepSeek advocates for integrating intra-node (scale-up) and inter-node (scale-out) communication into a unified framework.

Integrating dedicated co-processors for network traffic management and seamless forwarding between NVLink and IB domains could reduce software complexity and maximize bandwidth utilization. Hardware support for dynamic traffic deduplication could further optimize strategies like DeepSeek-V3’s node-aware routing. DeepSeek also explores eme