Snowflake/Arctic-LSTM-Speculator-gpt-oss-120b

ArcticSpeculator

Build the fastest OSS vllm-based speculative decoding system for your own model, using ArcticTraining and ArcticInference!

Throughput (tokens/s) of gpt-oss-120b on 8xH100 using vLLM below:

method	ShareGPT	HumanEval
vLLM V1 Baseline	220.2	220.7
ArcticSpeculator	377.3	400.0

For more details about ArcticSpeculator and how to use it:

• ❄️ Using Arctic-Inference and Arctic-Training for improving real-world speculative decoding Performance (blog)
• 🚀 Getting started guide using ArcticTraining

See all of the speculators we have released via our Speculators Collection