DeepSeek Releases Open-R2 Architecture with Native Speculative Decoding
The research lab unveils an open architecture demonstrating 3.2x faster inference latency on commodity hardware without quality degradation.
- Open-R2 integrates draft-model speculative verification into single-tensor runtime pipelines.
- Achieves up to 3.2x latency improvement on consumer RTX 4090 and Apple Silicon chips.
- Weights published under permissive Apache 2.0 license on HuggingFace.
Researchers at the frontier of open-weights artificial intelligence have formally published Open-R2, a next-generation transformer variant tailored for ultra-low latency speculative decoding on consumer and edge hardware.
The model release, which includes checkpoint weights ranging from 8B to 70B active parameters, specifically attacks the memory-bandwidth bottleneck that has historically plagued local model execution.
Figure 1: Telemetry and latent projection mapping across parallel draft-head verification kernels.
Overcoming the Memory Bandwidth Bottleneck
Traditional autoregressive generation requires reading gigabytes of parameter weights from VRAM for every single token produced. In consumer environments—where memory bandwidth is bounded by PCIe lanes and unified memory buses—this creates an immovable ceiling on tokens-per-second.
Open-R2 sidesteps this through a co-trained draft head embedded directly into the backbone network. Unlike conventional speculative decoding setups that require hosting two independent models (a small draft model and a large target model), Open-R2 uses internal latent projections to propose up to 5 tokens in parallel per forward pass.
Input Tokens -> Backbone Transformer -> Shared Latent Layer
├── Propose Draft Tokens (k=5)
└── Single-Pass Verification KernelAccording to benchmark runs validated across independent testbeds:
- Nvidia RTX 4090 (24GB): Reached 84 tokens/sec on fp8 quantization, up from 26 tokens/sec on standard Llama-3 class architectures.
- Apple M3 Max (128GB Unified): Sustained 61 tokens/sec on full 16-bit precision across long-context reasoning prompts.
Community Verification and Availability
The release has already sparked rapid integration efforts within the llama.cpp and vLLM communities. Maintainer Georgi Gerganov confirmed that pull requests supporting the custom speculative verification kernel have entered review, with initial quantization flags targeting GGUF formats expected within 48 hours.
The weights and reference inference kernels are accessible immediately on HuggingFace and GitHub under an unrestricted Apache 2.0 license.