furiosa-ai/Llama-3.1-8B-Instruct-FP8

Model Overview

• Model Architecture: Meta-Llama-3
  • Input: Text
  • Output: Text
• Model Optimizations:
• Maximum Context Length: 32k tokens
  • Maximum Prompt Length: 32768 tokens
  • Maximum Generation Length: 32768 tokens
• Intended Use Cases: Intended for commercial and non-commercial use. Same as Meta-Llama-3.1-8B-Instruct, this models is intended for assistant-like chat.
• Release Date: 04/12/2025
• Version: v2025.2
• License(s): Llama3.1
• Supported Inference Engine(s): Furiosa LLM
• Supported Hardware Compatibility: FuriosaAI RNGD
• Preferred Operating System(s): Linux
• Quantization:
  • Tool: Furiosa Model Compressor v0.6.2, included in Furiosa SDK 2025.2
  • Weight: float8, Activation: float8, KV cache: float8
  • Calibration: [mgoin/ultrachat_2k] dataset, 128 samples

Description

This model is the pre-compiled version of the Meta-Llama-3.1-8B-Instruct, which is an auto-regressive language model that uses an optimized transformer architecture.

Usage

To run this model with Furiosa-LLM, follow the example command below after installing Furiosa-LLM and its prerequisites.

furiosa-llm serve furiosa-ai/Llama-3.1-8B-Instruct-FP8