Tổng quan về mô hình Kimi K3: 2,8 nghìn tỷ tham số, lượng tử hóa MXFP4 và ý nghĩa của trọng số mở đối với cộng đồng

Moonshot AI publicly released Kimi K3 on July 16, 2026, with full open-source weights promised by July 27. At 2.8 trillion parameters, it is the first open-source model to reach the 3-trillion-parameter class. This article provides a structured model overview for the research and MLOps community.

Kimi K3 introduces several architectural innovations that collectively yield approximately 2.5x scaling efficiency improvement over its predecessor K2.

A hybrid linear attention mechanism that replaces standard quadratic attention in a subset of layers. KDA maintains full expressiveness for critical layers while reducing the computational cost of attention across the model's 1M-token context window. For a detailed treatment of KDA's design and its relationship to prior linear attention work, see thearchitecture deep dive on Hashnode.

A drop-in replacement for standard residual connections. Rather than uniformly accumulating layer outputs, AttnRes allows each layer to selectively retrieve representations from arbitrary earlier layers. This is particularly impactful in MoE architectures where different experts activate at different depths.

The MoE framework that manages 896 experts with 16 active per token. Key components include latent-space routing, Quantile Balancing for load management, and soft dropping for overflow tokens. For the full technical analysis, see theMoE deep dive on Zenn.dev.

K3 employsquantization-aware training (QAT)starting from the supervised fine-tuning stage, not post-training quantization. This is a critical distinction — the model learns to compensate for quantization error during training, resulting in significantly less quality degradation.

The format choices are deliberate:

The MXFP4 weight format means the full 2.8T model requires approximately1.4 TB of weight storage— substantially less than the ~5.6 TB that FP16 weights would demand. This brings self-hosting within reach of organizations with multi-node GPU clusters (e.g., 8-16 nodes of 8x H100/B200).

Moonshot's own serving uses theMooncake disaggregated inference infrastructure, which separates prefill and decode across different node pools and achieves a reported 90% cache hit rate on coding workloads. This architecture enables the aggressive cached input pricing of $0.30/MTok.

K3 leads all models on SWE Marathon and Program Bench, suggesting particular strength in sustained coding sessions — consistent with the 1M-token context window enabling full-repository understanding.

Expert specialization analysis: With 896 experts, researchers can probe whether experts specialize by language, domain, task type, or some other dimension. This data could inform future MoE designs.

Pruning and distillation: Can the 896-expert pool be reduced to 256 or 128 experts for deployment on smaller clusters? What is the quality-compute tradeoff curve?

Attention Residuals ablation: AttnRes is described as a "drop-in replacement" for residual connections. Testing this in smaller models could validate whether the technique generalizes beyond K3's scale.

Fine-tuning dynamics: How does LoRA or QLoRA interact with an already-quantized MoE model? The MXFP4 base presents novel challenges for adapter-based fine-tuning.

Moonshot acknowledges three areas where K3 falls short:

Kimi K3 represents a significant milestone: frontier-level performance in an open-source package. The combination of Stable LatentMoE, KDA, and quantization-aware training makes the 2.8T parameter count economically viable for both API serving and self-hosting.

The weight release on July 27 will be the real test — whether the community can reproduce Moonshot's benchmark numbers, adapt the model for domain-specific use cases, and advance the underlying architectural innovations.

For a broader analysis of what Kimi K3 means for the open-source AI ecosystem, see thiscomprehensive overview on wan27.org.