Llama-v3.2-3B-Instruct: Optimized for Mobile Deployment

State-of-the-art large language model useful on a variety of language understanding and generation tasks

Llama 3 is a family of LLMs. The model is quantized to w4a16 (4-bit weights and 16-bit activations) and part of the model is quantized to w8a16 (8-bit weights and 16-bit activations) making it suitable for on-device deployment. For Prompt and output length specified below, the time to first token is Llama-PromptProcessor-Quantized's latency and average time per addition token is Llama-TokenGenerator-Quantized's latency.

This model is an implementation of Llama-v3.2-3B-Instruct found here.

This repository provides scripts to run Llama-v3.2-3B-Instruct on Qualcomm® devices. More details on model performance across various devices, can be found here.

WARNING: The model assets are not readily available for download due to licensing restrictions.

Model Details

Model Type: Model_use_case.text_generation
Model Stats:
- Input sequence length for Prompt Processor: 128
- Context length: 4096
- Precision: w4a16 + w8a16 (few layers)
- Num of key-value heads: 8
- Model-1 (Prompt Processor): Llama-PromptProcessor-Quantized
- Prompt processor input: 128 tokens + position embeddings + attention mask + KV cache inputs
- Prompt processor output: 128 output tokens + KV cache outputs
- Model-2 (Token Generator): Llama-TokenGenerator-Quantized
- Token generator input: 1 input token + position embeddings + attention mask + KV cache inputs
- Token generator output: 1 output token + KV cache outputs
- Use: Initiate conversation with prompt-processor and then token generator for subsequent iterations.
- Minimum QNN SDK version required: 2.27.7
- Supported languages: English.
- TTFT: Time To First Token is the time it takes to generate the first response token. This is expressed as a range because it varies based on the length of the prompt. The lower bound is for a short prompt (up to 128 tokens, i.e., one iteration of the prompt processor) and the upper bound is for a prompt using the full context length (4096 tokens).
- Response Rate: Rate of response generation after the first response token.

| Model | Precision | Device | Chipset | Target Runtime | Response Rate (tokens per second) | Time To First Token (range, seconds) |---|---|---|---|---|---| | Llama-v3.2-3B-Instruct | w4a16 | Snapdragon 8 Elite QRD | Snapdragon® 8 Elite Mobile | GENIE | 23.4718 | 0.088195 - 2.82225 | -- | -- | | Llama-v3.2-3B-Instruct | w4a16 | Snapdragon X Elite CRD | Snapdragon® X Elite | GENIE | 18.4176 | 0.12593600000000002 - 4.029952000000001 | -- | -- | | Llama-v3.2-3B-Instruct | w4a16 | SA8255P ADP | Qualcomm® SA8255P | GENIE | 14.02377 | 0.187414 - 5.997256999999999 | -- | -- | | Llama-v3.2-3B-Instruct | w4 | Snapdragon 8 Elite QRD | Snapdragon® 8 Elite Mobile | GENIE | 13.83 | 0.088195 - 2.82225 | -- | -- |

Deploying Llama 3.2 3B on-device

Please follow the LLM on-device deployment tutorial.

Installation

Install the package via pip:

pip install "qai-hub-models[llama-v3-2-3b-instruct]"

Configure Qualcomm® AI Hub to run this model on a cloud-hosted device

Sign-in to Qualcomm® AI Hub with your Qualcomm® ID. Once signed in navigate to Account -> Settings -> API Token.

With this API token, you can configure your client to run models on the cloud hosted devices.

qai-hub configure --api_token API_TOKEN

Navigate to docs for more information.

Demo off target

The package contains a simple end-to-end demo that downloads pre-trained weights and runs this model on a sample input.

python -m qai_hub_models.models.llama_v3_2_3b_instruct.demo

The above demo runs a reference implementation of pre-processing, model inference, and post processing.

NOTE: If you want running in a Jupyter Notebook or Google Colab like environment, please add the following to your cell (instead of the above).

%run -m qai_hub_models.models.llama_v3_2_3b_instruct.demo

Run model on a cloud-hosted device

In addition to the demo, you can also run the model on a cloud-hosted Qualcomm® device. This script does the following:

Performance check on-device on a cloud-hosted device
Downloads compiled assets that can be deployed on-device for Android.
Accuracy check between PyTorch and on-device outputs.

python -m qai_hub_models.models.llama_v3_2_3b_instruct.export

Deploying compiled model to Android

The models can be deployed using multiple runtimes:

TensorFlow Lite (.tflite export): This tutorial provides a guide to deploy the .tflite model in an Android application.
QNN (.so export ): This sample app provides instructions on how to use the .so shared library in an Android application.

View on Qualcomm® AI Hub

Get more details on Llama-v3.2-3B-Instruct's performance across various devices here. Explore all available models on Qualcomm® AI Hub

License

The license for the original implementation of Llama-v3.2-3B-Instruct can be found here.
The license for the compiled assets for on-device deployment can be found here

References

Community

Join our AI Hub Slack community to collaborate, post questions and learn more about on-device AI.
For questions or feedback please reach out to us.

Downloads last month: -; Downloads are not tracked for this model. How to track