Duplicate from RedHatAI/Llama-4-Scout-17B-16E-Instruct-FP8-dynamic

Co-authored-by: Eldar Kurtic <[email protected]>

.gitattributes

@@ -0,0 +1,36 @@
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tar filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zst filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text
+tokenizer.json filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,450 @@
+---
+library_name: vllm
+language:
+- ar
+- de
+- en
+- es
+- fr
+- hi
+- id
+- it
+- pt
+- th
+- tl
+- vi
+base_model:
+- meta-llama/Llama-4-Scout-17B-16E-Instruct
+pipeline_tag: image-text-to-text
+tags:
+- facebook
+- meta
+- pytorch
+- llama
+- llama4
+- neuralmagic
+- redhat
+- llmcompressor
+- quantized
+- FP8
+license: other
+license_name: llama4
+---
+
+<h1 style="display: flex; align-items: center; gap: 10px; margin: 0;">
+  Llama-4-Scout-17B-16E-Instruct-FP8-dynamic
+  <img src="https://www.redhat.com/rhdc/managed-files/Catalog-Validated_model_0.png" alt="Model Icon" width="40" style="margin: 0; padding: 0;" />
+</h1>
+  
+<a href="https://www.redhat.com/en/products/ai/validated-models" target="_blank" style="margin: 0; padding: 0;">
+<img src="https://www.redhat.com/rhdc/managed-files/Validated_badge-Dark.png" alt="Validated Badge" width="250" style="margin: 0; padding: 0;" />
+</a>
+
+## Model Overview
+- **Model Architecture:** Llama4ForConditionalGeneration
+  - **Input:** Text / Image
+  - **Output:** Text
+- **Model Optimizations:**
+  - **Activation quantization:** FP8
+  - **Weight quantization:** FP8
+- **Release Date:** 04/15/2025
+- **Version:** 1.0
+- **Model Developers:** Red Hat (Neural Magic)
+
+
+### Model Optimizations
+
+This model was obtained by quantizing activations and weights of [Llama-4-Scout-17B-16E-Instruct](https://huggingface.co/meta-llama/Llama-4-Scout-17B-16E-Instruct) to FP8 data type.
+This optimization reduces the number of bits used to represent weights and activations from 16 to 8, reducing GPU memory requirements (by approximately 50%) and increasing matrix-multiply compute throughput (by approximately 2x).
+Weight quantization also reduces disk size requirements by approximately 50%. The [llm-compressor](https://github.com/vllm-project/llm-compressor) library is used for quantization.
+
+## Deployment
+
+This model can be deployed efficiently on vLLM, Red Hat Enterprise Linux AI, and Openshift AI, as shown in the example below.
+
+Deploy on <strong>vLLM</strong>
+
+```python
+from vllm import LLM, SamplingParams
+from transformers import AutoTokenizer
+
+model_id = "RedHatAI/Llama-4-Scout-17B-16E-Instruct-FP8-dynamic"
+number_gpus = 4
+
+sampling_params = SamplingParams(temperature=0.7, top_p=0.8, max_tokens=256)
+
+tokenizer = AutoTokenizer.from_pretrained(model_id)
+
+prompt = "Give me a short introduction to large language model."
+
+llm = LLM(model=model_id, tensor_parallel_size=number_gpus)
+
+outputs = llm.generate(prompt, sampling_params)
+
+generated_text = outputs[0].outputs[0].text
+print(generated_text)
+```
+
+vLLM also supports OpenAI-compatible serving. See the [documentation](https://docs.vllm.ai/en/latest/) for more details.
+
+
+<details>
+  <summary>Deploy on <strong>Red Hat AI Inference Server</strong></summary>
+  
+```bash
+podman run --rm -it --device nvidia.com/gpu=all -p 8000:8000 \
+ --ipc=host \
+--env "HUGGING_FACE_HUB_TOKEN=$HF_TOKEN" \
+--env "HF_HUB_OFFLINE=0" -v ~/.cache/vllm:/home/vllm/.cache \
+--name=vllm \
+registry.access.redhat.com/rhaiis/rh-vllm-cuda \
+vllm serve \
+--tensor-parallel-size 8 \
+--max-model-len 32768  \
+--enforce-eager --model RedHatAI/Llama-4-Scout-17B-16E-Instruct-FP8-dynamic
+```
+</details>
+
+<details>
+  <summary>Deploy on <strong>Red Hat Enterprise Linux AI</strong></summary>
+  
+```bash
+# Download model from Red Hat Registry via docker
+# Note: This downloads the model to ~/.cache/instructlab/models unless --model-dir is specified.
+ilab model download --repository docker://registry.redhat.io/rhelai1/llama-4-scout-17b-16e-instruct-fp8-dynamic:1.5
+```
+
+```bash
+# Serve model via ilab
+ilab model serve --model-path ~/.cache/instructlab/models/llama-4-scout-17b-16e-instruct-fp8-dynamic
+  
+# Chat with model
+ilab model chat --model ~/.cache/instructlab/models/llama-4-scout-17b-16e-instruct-fp8-dynamic
+```
+See [Red Hat Enterprise Linux AI documentation](https://docs.redhat.com/en/documentation/red_hat_enterprise_linux_ai/1.4) for more details.
+</details>
+
+<details>
+  <summary>Deploy on <strong>Red Hat Openshift AI</strong></summary>
+  
+```python
+# Setting up vllm server with ServingRuntime
+# Save as: vllm-servingruntime.yaml
+apiVersion: serving.kserve.io/v1alpha1
+kind: ServingRuntime
+metadata:
+ name: vllm-cuda-runtime # OPTIONAL CHANGE: set a unique name
+ annotations:
+   openshift.io/display-name: vLLM NVIDIA GPU ServingRuntime for KServe
+   opendatahub.io/recommended-accelerators: '["nvidia.com/gpu"]'
+ labels:
+   opendatahub.io/dashboard: 'true'
+spec:
+ annotations:
+   prometheus.io/port: '8080'
+   prometheus.io/path: '/metrics'
+ multiModel: false
+ supportedModelFormats:
+   - autoSelect: true
+     name: vLLM
+ containers:
+   - name: kserve-container
+     image: quay.io/modh/vllm:rhoai-2.20-cuda # CHANGE if needed. If AMD: quay.io/modh/vllm:rhoai-2.20-rocm
+     command:
+       - python
+       - -m
+       - vllm.entrypoints.openai.api_server
+     args:
+       - "--port=8080"
+       - "--model=/mnt/models"
+       - "--served-model-name={{.Name}}"
+     env:
+       - name: HF_HOME
+         value: /tmp/hf_home
+     ports:
+       - containerPort: 8080
+         protocol: TCP
+```
+
+```python
+# Attach model to vllm server. This is an NVIDIA template
+# Save as: inferenceservice.yaml
+apiVersion: serving.kserve.io/v1beta1
+kind: InferenceService
+metadata:
+  annotations:
+    openshift.io/display-name: Llama-4-Scout-17B-16E-Instruct-FP8-dynamic # OPTIONAL CHANGE
+    serving.kserve.io/deploymentMode: RawDeployment
+  name: Llama-4-Scout-17B-16E-Instruct-FP8-dynamic          # specify model name. This value will be used to invoke the model in the payload
+  labels:
+    opendatahub.io/dashboard: 'true'
+spec:
+  predictor:
+    maxReplicas: 1
+    minReplicas: 1
+    model:
+      modelFormat:
+        name: vLLM
+      name: ''
+      resources:
+        limits:
+          cpu: '2'			# this is model specific
+          memory: 8Gi		# this is model specific
+          nvidia.com/gpu: '1'	# this is accelerator specific
+        requests:			# same comment for this block
+          cpu: '1'
+          memory: 4Gi
+          nvidia.com/gpu: '1'
+      runtime: vllm-cuda-runtime	# must match the ServingRuntime name above
+      storageUri: oci://registry.redhat.io/rhelai1/modelcar-llama-4-scout-17b-16e-instruct-fp8-dynamic:1.5
+    tolerations:
+    - effect: NoSchedule
+      key: nvidia.com/gpu
+      operator: Exists
+```
+
+```bash
+# make sure first to be in the project where you want to deploy the model
+# oc project <project-name>
+
+# apply both resources to run model
+
+# Apply the ServingRuntime
+oc apply -f vllm-servingruntime.yaml
+
+# Apply the InferenceService
+oc apply -f qwen-inferenceservice.yaml
+```
+
+```python
+# Replace <inference-service-name> and <cluster-ingress-domain> below:
+# - Run `oc get inferenceservice` to find your URL if unsure.
+
+# Call the server using curl:
+curl https://<inference-service-name>-predictor-default.<domain>/v1/chat/completions
+        -H "Content-Type: application/json" \
+        -d '{
+    "model": "Llama-4-Scout-17B-16E-Instruct-FP8-dynamic",
+    "stream": true,
+    "stream_options": {
+        "include_usage": true
+    },
+    "max_tokens": 1,
+    "messages": [
+        {
+            "role": "user",
+            "content": "How can a bee fly when its wings are so small?"
+        }
+    ]
+}'
+
+```
+
+See [Red Hat Openshift AI documentation](https://docs.redhat.com/en/documentation/red_hat_openshift_ai/2025) for more details.
+</details>
+
+## Creation
+
+<details>
+  <summary>Creation details</summary>
+  This model was created with [llm-compressor](https://github.com/vllm-project/llm-compressor) by running the code snippet below. 
+
+
+```python
+#!/usr/bin/env python3
+"""
+This script loads an LLM model and applies FP8 quantization to
+weights and activations. Activations are dynamically quantized, i.e. during
+actual runtime.
+"""
+
+import argparse
+import torch
+from transformers import AutoTokenizer, AutoModelForCausalLM, Llama4ForConditionalGeneration
+from llmcompressor.modifiers.quantization import QuantizationModifier
+from llmcompressor import oneshot
+from compressed_tensors.quantization import (
+    QuantizationScheme,
+    QuantizationArgs,
+    QuantizationType,
+    QuantizationStrategy,
+)
+
+
+def parse_arguments():
+    """Parse command line arguments."""
+    parser = argparse.ArgumentParser(description="Quantize a causal language model")
+    parser.add_argument(
+        "--model_path",
+        type=str,
+        required=True,
+        help="Path to the pre-trained model",
+    )
+    parser.add_argument(
+        "--quant_path",
+        type=str,
+        required=True,
+        help="Output path for the quantized model",
+    )
+    return parser.parse_args()
+
+
+def main():
+    """Main function to load and quantize the model."""
+    args = parse_arguments()
+
+    print(f"Loading model from {args.model_path}...")
+    model = Llama4ForConditionalGeneration.from_pretrained(
+        args.model_path,
+        device_map="auto",
+        torch_dtype="auto",
+        trust_remote_code=True,
+    )
+
+    quant_scheme = QuantizationScheme(
+        targets=["Linear"],
+        weights=QuantizationArgs(
+            num_bits=8,
+            type=QuantizationType.FLOAT,
+            strategy=QuantizationStrategy.CHANNEL,
+            symmetric=True,
+            observer="mse",
+        ),
+        input_activations=QuantizationArgs(
+            num_bits=8,
+            type=QuantizationType.FLOAT,
+            strategy=QuantizationStrategy.TOKEN,
+            symmetric=True,
+            dynamic=True,
+        ),
+        output_activations=None,
+    )
+
+    recipe = QuantizationModifier(
+        targets="Linear",
+        config_groups={"group_0": quant_scheme},
+        ignore=[
+            're:.*lm_head',
+            're:.*self_attn',
+            're:.*router',
+            're:.*vision_model',
+            're:.*multi_modal_projector',
+        ]
+    )
+
+    print("Applying quantization...")
+    oneshot(
+        model=model,
+        recipe=recipe,
+        trust_remote_code_model=True,
+    )
+
+    model.save_pretrained(args.quant_path, save_compressed=True, skip_compression_stats=True, disable_sparse_compression=True)
+    print(f"Quantized model saved to {args.quant_path}")
+
+
+if __name__ == "__main__":
+    main()
+```
+</details>
+ 
+
+
+## Evaluation
+
+The model was evaluated on the OpenLLM leaderboard tasks (v1 and v2), long context RULER, multimodal MMMU, and multimodal ChartQA.
+All evaluations are obtained through [lm-evaluation-harness](https://github.com/EleutherAI/lm-evaluation-harness).
+
+<details>
+  <summary>Evaluation details</summary>
+
+  **OpenLLM v1**
+  ```
+  lm_eval \
+    --model vllm \
+    --model_args pretrained="RedHatAI/Llama-4-Scout-17B-16E-Instruct-FP8-dynamic",dtype=auto,add_bos_token=True,max_model_len=4096,tensor_parallel_size=8,gpu_memory_utilization=0.7,enable_chunked_prefill=True,trust_remote_code=True \
+    --tasks openllm \
+    --batch_size auto 
+  ```
+
+  **OpenLLM v2**
+  ```
+  lm_eval \
+    --model vllm \
+    --model_args pretrained="RedHatAI/Llama-4-Scout-17B-16E-Instruct-FP8-dynamic",dtype=auto,add_bos_token=False,max_model_len=16384,tensor_parallel_size=8,gpu_memory_utilization=0.5,enable_chunked_prefill=True,trust_remote_code=True \
+    --tasks leaderboard \
+    --apply_chat_template \
+    --fewshot_as_multiturn \
+    --batch_size auto 
+  ```
+
+  **Long Context RULER**
+  ```
+  lm_eval \
+    --model vllm \
+    --model_args pretrained="RedHatAI/Llama-4-Scout-17B-16E-Instruct-FP8-dynamic",dtype=auto,add_bos_token=False,max_model_len=524288,tensor_parallel_size=8,gpu_memory_utilization=0.9,enable_chunked_prefill=True,trust_remote_code=True \
+    --tasks ruler \
+    --metadata='{"max_seq_lengths":[131072]}' \
+    --batch_size auto 
+  ```
+
+  **Multimodal MMMU**
+  ```
+  lm_eval \
+    --model vllm-vlm \
+    --model_args pretrained="RedHatAI/Llama-4-Scout-17B-16E-Instruct-FP8-dynamic",dtype=auto,add_bos_token=False,max_model_len=1000000,tensor_parallel_size=8,gpu_memory_utilization=0.9,enable_chunked_prefill=True,trust_remote_code=True,max_images=10 \
+    --tasks mmmu_val \
+    --apply_chat_template \
+    --batch_size auto 
+  ```
+
+  **Multimodal ChartQA**
+  ```
+  export VLLM_MM_INPUT_CACHE_GIB=8
+  lm_eval \
+    --model vllm-vlm \
+    --model_args pretrained="RedHatAI/Llama-4-Scout-17B-16E-Instruct-FP8-dynamic",dtype=auto,add_bos_token=False,max_model_len=1000000,tensor_parallel_size=8,gpu_memory_utilization=0.9,enable_chunked_prefill=True,trust_remote_code=True,max_images=10 \
+    --tasks chartqa \
+    --apply_chat_template \
+    --batch_size auto 
+  ```
+
+</details>
+
+### Accuracy
+
+|                                                | Recovery (%) | meta-llama/Llama-4-Scout-17B-16E-Instruct | RedHatAI/Llama-4-Scout-17B-16E-Instruct-FP8-dynamic<br>(this model) |
+| ---------------------------------------------- | :-----------: | :---------------------------------------: | :-----------------------------------------------------------------: |
+| ARC-Challenge<br>25-shot                       | 100.36       | 69.37                                     | 69.62                                                               |
+| GSM8k<br>5-shot                                | 99.24        | 90.45                                     | 89.76                                                               |
+| HellaSwag<br>10-shot                           | 99.94        | 85.23                                     | 85.18                                                               |
+| MMLU<br>5-shot                                 | 99.94        | 80.54                                     | 80.49                                                               |
+| TruthfulQA<br>0-shot                           | 99.17        | 61.41                                     | 60.90                                                               |
+| WinoGrande<br>5-shot                           | 98.88        | 77.90                                     | 77.03                                                               |
+| **OpenLLM v1<br>Average Score**                    | **99.59**        | **77.48**                                     | **77.16**                                                               |
+| IFEval<br>0-shot<br>avg of inst and prompt acc | 100.91       | 86.90                                     | 87.69                                                               |
+| Big Bench Hard<br>3-shot                       | 99.82        | 65.13                                     | 65.01                                                               |
+| Math Lvl 5<br>4-shot                           | 98.82        | 57.78                                     | 57.10                                                               |
+| GPQA<br>0-shot                                 | 100.53       | 31.88                                     | 32.05                                                               |
+| MuSR<br>0-shot                                 | 102.18       | 42.20                                     | 43.12                                                               |
+| MMLU-Pro<br>5-shot                             | 99.82        | 55.70                                     | 55.60                                                               |
+| **OpenLLM v2<br>Average Score**                    | **100.28**       | **56.60**                                     | **56.76**                                                               |
+| RULER<br>seqlen = 131072<br>niah_multikey_1    | 101.36       | 88.20                                     | 89.40                                                               |
+| RULER<br>seqlen = 131072<br>niah_multikey_2    | 100.72       | 83.60                                     | 84.20                                                               |
+| RULER<br>seqlen = 131072<br>niah_multikey_3    | 96.19        | 78.80                                     | 75.80                                                               |
+| RULER<br>seqlen = 131072<br>niah_multiquery    | 100.79       | 95.40                                     | 96.15                                                               |
+| RULER<br>seqlen = 131072<br>niah_multivalue    | 97.22        | 73.75                                     | 71.70                                                               |
+| RULER<br>seqlen = 131072<br>niah_single_1      | 100.00       | 100.00                                    | 100.00                                                              |
+| RULER<br>seqlen = 131072<br>niah_single_2      | 100.00       | 99.80                                     | 99.80                                                               |
+| RULER<br>seqlen = 131072<br>niah_single_3      | 100.00       | 99.80                                     | 99.80                                                               |
+| RULER<br>seqlen = 131072<br>ruler_cwe          | 96.19        | 39.42                                     | 37.92                                                               |
+| RULER<br>seqlen = 131072<br>ruler_fwe          | 98.86        | 92.93                                     | 91.87                                                               |
+| RULER<br>seqlen = 131072<br>ruler_qa_hotpot    | 100.00       | 48.20                                     | 48.20                                                               |
+| RULER<br>seqlen = 131072<br>ruler_qa_squad     | 98.81        | 53.57                                     | 52.93                                                               |
+| RULER<br>seqlen = 131072<br>ruler_qa_vt        | 100.35       | 92.28                                     | 92.60                                                               |
+| **RULER<br>seqlen = 131072<br>Average Score**      | **99.49**        | **80.44**                                     | **80.03**                                                               |
+| MMMU<br>0-shot                                 | 97.92        | 53.44                                     | 52.33                                                               |
+| ChartQA<br>0-shot<br>exact_match               | 100.12       | 65.88                                     | 65.96                                                               |
+| ChartQA<br>0-shot<br>relaxed_accuracy          | 99.69        | 88.92                                     | 88.64                                                               |
+| **Multimodal Average Score**                       | **99.38**        | **69.41**                                     | **68.98**                                                               |
+

chat_template.json

@@ -0,0 +1,3 @@
+{
+  "chat_template": "{{- bos_token }}\n{%- if custom_tools is defined %}\n    {%- set tools = custom_tools %}\n{%- endif %}\n{%- if not tools_in_user_message is defined %}\n    {%- set tools_in_user_message = true %}\n{%- endif %}\n{%- if not date_string is defined %}\n    {%- if strftime_now is defined %}\n        {%- set date_string = strftime_now(\"%d %b %Y\") %}\n    {%- else %}\n        {%- set date_string = \"26 Jul 2024\" %}\n    {%- endif %}\n{%- endif %}\n{%- if not tools is defined %}\n    {%- set tools = none %}\n{%- endif %}\n\n{#- This block extracts the system message, so we can slot it into the right place. #}\n{%- if messages[0]['role'] == 'system' %}    \n    {%- if messages[0]['content'] is string %}\n        {%- set system_message = messages[0]['content']|trim %}\n    {%- else %}\n        {#- FIXME: The processor requires an array, always. #}\n        {%- set system_message = messages[0]['content'][0]['text']|trim %}\n    {%- endif %}\n    {%- set messages = messages[1:] %}\n    {%- set user_supplied_system_message = true %}\n{%- else %}\n    {%- set system_message = \"\" %}\n    {%- set user_supplied_system_message = false %}\n{%- endif %}\n\n{#- System message if the user supplied one #}\n{%- if user_supplied_system_message %}\n    {{- \"<|header_start|>system<|header_end|>\\n\\n\" }}\n    {%- if tools is not none %}\n        {{- \"Environment: ipython\\n\" }}\n    {%- endif %}\n    {%- if tools is not none and not tools_in_user_message %}\n        {{- \"You have access to the following functions. To call a function, please respond with JSON for a function call.\" }}\n        {{- 'Respond in the format {\"name\": function name, \"parameters\": dictionary of argument name and its value}.' }}\n        {{- \"Do not use variables.\\n\\n\" }}\n        {%- for t in tools %}\n            {{- t | tojson(indent=4) }}\n            {{- \"\\n\\n\" }}\n        {%- endfor %}\n    {%- endif %}\n    {{- system_message }}\n    {{- \"<|eot|>\" }}\n{%- endif %}\n\n{#- Custom tools are passed in a user message with some extra guidance #}\n{%- if tools_in_user_message and not tools is none %}\n    {#- Extract the first user message so we can plug it in here #}\n    {%- if messages | length != 0 %}\n        {%- set first_user_message = messages[0]['content']|trim %}\n        {%- set messages = messages[1:] %}\n    {%- else %}\n        {{- raise_exception(\"Cannot put tools in the first user message when there's no first user message!\") }}\n{%- endif %}\n    {{- '<|header_start|>user<|header_end|>\\n\\n' -}}\n    {{- \"Given the following functions, please respond with a JSON for a function call \" }}\n    {{- \"with its proper arguments that best answers the given prompt.\\n\\n\" }}\n    {{- 'Respond in the format {\"name\": function name, \"parameters\": dictionary of argument name and its value}.' }}\n    {{- \"Do not use variables.\\n\\n\" }}\n    {%- for t in tools %}\n        {{- t | tojson(indent=4) }}\n        {{- \"\\n\\n\" }}\n    {%- endfor %}\n    {{- first_user_message + \"<|eot|>\"}}\n{%- endif %}\n\n{%- for message in messages %}\n    {%- if not (message.role == 'ipython' or message.role == 'tool' or 'tool_calls' in message) %}\n    {{- '<|header_start|>' + message['role'] + '<|header_end|>\\n\\n' }}\n        {%- if message['content'] is string %}\n            {{- message['content'] }}\n        {%- else %}\n            {%- for content in message['content'] %}\n                {%- if content['type'] == 'image' %}\n                    {{- '<|image|>' }}\n                {%- elif content['type'] == 'text' %}\n                    {{- content['text'] }}\n                {%- endif %}\n            {%- endfor %}\n        {%- endif %}\n        {{- \"<|eot|>\" }}\n    {%- elif 'tool_calls' in message and message.tool_calls|length > 0 %}\n       {{- '<|header_start|>assistant<|header_end|>\\n\\n' -}}\n       {{- '<|python_start|>' }}\n        {%- if message['content'] is string %}\n            {{- message['content'] }}\n        {%- else %}\n            {%- for content in message['content'] %}\n                {%- if content['type'] == 'image' %}\n                    {{- '<|image|>' }}\n                {%- elif content['type'] == 'text' %}\n                    {{- content['text'] }}\n                {%- endif %}\n            {%- endfor %}\n        {%- endif %}\n       {{- '<|python_end|>' }}\n        {%- for tool_call in message.tool_calls %}\n           {{- '{\"name\": \"' + tool_call.function.name + '\", ' }}\n           {{- '\"parameters\": ' }}\n           {{- tool_call.function.arguments | tojson }}\n           {{- \"}\" }}\n        {%- endfor %}\n       {{- \"<|eot|>\" }}\n    {%- elif message.role == \"tool\" or message.role == \"ipython\" %}\n        {{- \"<|header_start|>ipython<|header_end|>\\n\\n\" }}\n        {%- if message.content is mapping or message.content is iterable %}\n            {{- message.content | tojson }}\n        {%- else %}\n            {{- message.content }}\n        {%- endif %}\n        {{- \"<|eot|>\" }}\n    {%- endif %}\n{%- endfor %}\n{%- if add_generation_prompt %}\n    {{- '<|header_start|>assistant<|header_end|>\\n\\n' }}\n{%- endif %}\n"
+}

config.json

@@ -0,0 +1,570 @@
+{
+  "architectures": [
+    "Llama4ForConditionalGeneration"
+  ],
+  "boi_token_index": 200080,
+  "eoi_token_index": 200081,
+  "image_token_index": 200092,
+  "model_type": "llama4",
+  "text_config": {
+    "_attn_implementation_autoset": true,
+    "attention_bias": false,
+    "attention_chunk_size": 8192,
+    "attention_dropout": 0.0,
+    "bos_token_id": 200000,
+    "eos_token_id": [
+      200001,
+      200007,
+      200008
+    ],
+    "for_llm_compressor": true,
+    "head_dim": 128,
+    "hidden_act": "silu",
+    "hidden_size": 5120,
+    "initializer_range": 0.02,
+    "interleave_moe_layer_step": 1,
+    "intermediate_size": 8192,
+    "intermediate_size_mlp": 16384,
+    "max_position_embeddings": 10485760,
+    "model_type": "llama4_text",
+    "no_rope_layers": [],
+    "num_attention_heads": 40,
+    "num_experts_per_tok": 1,
+    "num_hidden_layers": 48,
+    "num_key_value_heads": 8,
+    "num_local_experts": 16,
+    "output_router_logits": false,
+    "pad_token_id": 200018,
+    "rms_norm_eps": 1e-05,
+    "rope_scaling": {
+      "factor": 16.0,
+      "high_freq_factor": 1.0,
+      "low_freq_factor": 1.0,
+      "original_max_position_embeddings": 8192,
+      "rope_type": "llama3"
+    },
+    "rope_theta": 500000.0,
+    "router_aux_loss_coef": 0.001,
+    "router_jitter_noise": 0.0,
+    "torch_dtype": "bfloat16",
+    "use_cache": true,
+    "use_qk_norm": true,
+    "vocab_size": 202048
+  },
+  "torch_dtype": "bfloat16",
+  "transformers_version": "4.51.0.dev0",
+  "vision_config": {
+    "_attn_implementation_autoset": true,
+    "attention_dropout": 0.0,
+    "hidden_act": "gelu",
+    "hidden_size": 1408,
+    "image_size": 336,
+    "initializer_range": 0.02,
+    "intermediate_size": 5632,
+    "model_type": "llama4_vision_model",
+    "multi_modal_projector_bias": false,
+    "norm_eps": 1e-05,
+    "num_attention_heads": 16,
+    "num_channels": 3,
+    "num_hidden_layers": 34,
+    "patch_size": 14,
+    "pixel_shuffle_ratio": 0.5,
+    "projector_dropout": 0.0,
+    "projector_input_dim": 4096,
+    "projector_output_dim": 4096,
+    "rope_theta": 10000,
+    "vision_feature_layer": -1,
+    "vision_feature_select_strategy": "default",
+    "vision_output_dim": 4096
+  },
+  "quantization_config": {
+    "config_groups": {
+      "group_0": {
+        "input_activations": {
+          "actorder": null,
+          "block_structure": null,
+          "dynamic": true,
+          "group_size": null,
+          "num_bits": 8,
+          "observer": null,
+          "observer_kwargs": {},
+          "strategy": "token",
+          "symmetric": true,
+          "type": "float"
+        },
+        "output_activations": null,
+        "targets": [
+          "Linear"
+        ],
+        "weights": {
+          "actorder": null,
+          "block_structure": null,
+          "dynamic": false,
+          "group_size": null,
+          "num_bits": 8,
+          "observer": "mse",
+          "observer_kwargs": {},
+          "strategy": "channel",
+          "symmetric": true,
+          "type": "float"
+        }
+      }
+    },
+    "format": "float-quantized",
+    "global_compression_ratio": null,
+    "ignore": [
+      "vision_model.patch_embedding.linear",
+      "vision_model.model.layers.0.self_attn.q_proj",
+      "vision_model.model.layers.0.self_attn.k_proj",
+      "vision_model.model.layers.0.self_attn.v_proj",
+      "vision_model.model.layers.0.self_attn.o_proj",
+      "vision_model.model.layers.0.mlp.fc1",
+      "vision_model.model.layers.0.mlp.fc2",
+      "vision_model.model.layers.1.self_attn.q_proj",
+      "vision_model.model.layers.1.self_attn.k_proj",
+      "vision_model.model.layers.1.self_attn.v_proj",
+      "vision_model.model.layers.1.self_attn.o_proj",
+      "vision_model.model.layers.1.mlp.fc1",
+      "vision_model.model.layers.1.mlp.fc2",
+      "vision_model.model.layers.2.self_attn.q_proj",
+      "vision_model.model.layers.2.self_attn.k_proj",
+      "vision_model.model.layers.2.self_attn.v_proj",
+      "vision_model.model.layers.2.self_attn.o_proj",
+      "vision_model.model.layers.2.mlp.fc1",
+      "vision_model.model.layers.2.mlp.fc2",
+      "vision_model.model.layers.3.self_attn.q_proj",
+      "vision_model.model.layers.3.self_attn.k_proj",
+      "vision_model.model.layers.3.self_attn.v_proj",
+      "vision_model.model.layers.3.self_attn.o_proj",
+      "vision_model.model.layers.3.mlp.fc1",
+      "vision_model.model.layers.3.mlp.fc2",
+      "vision_model.model.layers.4.self_attn.q_proj",
+      "vision_model.model.layers.4.self_attn.k_proj",
+      "vision_model.model.layers.4.self_attn.v_proj",
+      "vision_model.model.layers.4.self_attn.o_proj",
+      "vision_model.model.layers.4.mlp.fc1",
+      "vision_model.model.layers.4.mlp.fc2",
+      "vision_model.model.layers.5.self_attn.q_proj",
+      "vision_model.model.layers.5.self_attn.k_proj",
+      "vision_model.model.layers.5.self_attn.v_proj",
+      "vision_model.model.layers.5.self_attn.o_proj",
+      "vision_model.model.layers.5.mlp.fc1",
+      "vision_model.model.layers.5.mlp.fc2",
+      "vision_model.model.layers.6.self_attn.q_proj",
+      "vision_model.model.layers.6.self_attn.k_proj",
+      "vision_model.model.layers.6.self_attn.v_proj",
+      "vision_model.model.layers.6.self_attn.o_proj",
+      "vision_model.model.layers.6.mlp.fc1",
+      "vision_model.model.layers.6.mlp.fc2",
+      "vision_model.model.layers.7.self_attn.q_proj",
+      "vision_model.model.layers.7.self_attn.k_proj",
+      "vision_model.model.layers.7.self_attn.v_proj",
+      "vision_model.model.layers.7.self_attn.o_proj",
+      "vision_model.model.layers.7.mlp.fc1",
+      "vision_model.model.layers.7.mlp.fc2",
+      "vision_model.model.layers.8.self_attn.q_proj",
+      "vision_model.model.layers.8.self_attn.k_proj",
+      "vision_model.model.layers.8.self_attn.v_proj",
+      "vision_model.model.layers.8.self_attn.o_proj",
+      "vision_model.model.layers.8.mlp.fc1",
+      "vision_model.model.layers.8.mlp.fc2",
+      "vision_model.model.layers.9.self_attn.q_proj",
+      "vision_model.model.layers.9.self_attn.k_proj",
+      "vision_model.model.layers.9.self_attn.v_proj",
+      "vision_model.model.layers.9.self_attn.o_proj",
+      "vision_model.model.layers.9.mlp.fc1",
+      "vision_model.model.layers.9.mlp.fc2",
+      "vision_model.model.layers.10.self_attn.q_proj",
+      "vision_model.model.layers.10.self_attn.k_proj",
+      "vision_model.model.layers.10.self_attn.v_proj",
+      "vision_model.model.layers.10.self_attn.o_proj",
+      "vision_model.model.layers.10.mlp.fc1",
+      "vision_model.model.layers.10.mlp.fc2",
+      "vision_model.model.layers.11.self_attn.q_proj",
+      "vision_model.model.layers.11.self_attn.k_proj",
+      "vision_model.model.layers.11.self_attn.v_proj",
+      "vision_model.model.layers.11.self_attn.o_proj",
+      "vision_model.model.layers.11.mlp.fc1",
+      "vision_model.model.layers.11.mlp.fc2",
+      "vision_model.model.layers.12.self_attn.q_proj",
+      "vision_model.model.layers.12.self_attn.k_proj",
+      "vision_model.model.layers.12.self_attn.v_proj",
+      "vision_model.model.layers.12.self_attn.o_proj",
+      "vision_model.model.layers.12.mlp.fc1",
+      "vision_model.model.layers.12.mlp.fc2",
+      "vision_model.model.layers.13.self_attn.q_proj",
+      "vision_model.model.layers.13.self_attn.k_proj",
+      "vision_model.model.layers.13.self_attn.v_proj",
+      "vision_model.model.layers.13.self_attn.o_proj",
+      "vision_model.model.layers.13.mlp.fc1",
+      "vision_model.model.layers.13.mlp.fc2",
+      "vision_model.model.layers.14.self_attn.q_proj",
+      "vision_model.model.layers.14.self_attn.k_proj",
+      "vision_model.model.layers.14.self_attn.v_proj",
+      "vision_model.model.layers.14.self_attn.o_proj",
+      "vision_model.model.layers.14.mlp.fc1",
+      "vision_model.model.layers.14.mlp.fc2",
+      "vision_model.model.layers.15.self_attn.q_proj",
+      "vision_model.model.layers.15.self_attn.k_proj",
+      "vision_model.model.layers.15.self_attn.v_proj",
+      "vision_model.model.layers.15.self_attn.o_proj",
+      "vision_model.model.layers.15.mlp.fc1",
+      "vision_model.model.layers.15.mlp.fc2",
+      "vision_model.model.layers.16.self_attn.q_proj",
+      "vision_model.model.layers.16.self_attn.k_proj",
+      "vision_model.model.layers.16.self_attn.v_proj",
+      "vision_model.model.layers.16.self_attn.o_proj",
+      "vision_model.model.layers.16.mlp.fc1",
+      "vision_model.model.layers.16.mlp.fc2",
+      "vision_model.model.layers.17.self_attn.q_proj",
+      "vision_model.model.layers.17.self_attn.k_proj",
+      "vision_model.model.layers.17.self_attn.v_proj",
+      "vision_model.model.layers.17.self_attn.o_proj",
+      "vision_model.model.layers.17.mlp.fc1",
+      "vision_model.model.layers.17.mlp.fc2",
+      "vision_model.model.layers.18.self_attn.q_proj",
+      "vision_model.model.layers.18.self_attn.k_proj",
+      "vision_model.model.layers.18.self_attn.v_proj",
+      "vision_model.model.layers.18.self_attn.o_proj",
+      "vision_model.model.layers.18.mlp.fc1",
+      "vision_model.model.layers.18.mlp.fc2",
+      "vision_model.model.layers.19.self_attn.q_proj",
+      "vision_model.model.layers.19.self_attn.k_proj",
+      "vision_model.model.layers.19.self_attn.v_proj",
+      "vision_model.model.layers.19.self_attn.o_proj",
+      "vision_model.model.layers.19.mlp.fc1",
+      "vision_model.model.layers.19.mlp.fc2",
+      "vision_model.model.layers.20.self_attn.q_proj",
+      "vision_model.model.layers.20.self_attn.k_proj",
+      "vision_model.model.layers.20.self_attn.v_proj",
+      "vision_model.model.layers.20.self_attn.o_proj",
+      "vision_model.model.layers.20.mlp.fc1",
+      "vision_model.model.layers.20.mlp.fc2",
+      "vision_model.model.layers.21.self_attn.q_proj",
+      "vision_model.model.layers.21.self_attn.k_proj",
+      "vision_model.model.layers.21.self_attn.v_proj",
+      "vision_model.model.layers.21.self_attn.o_proj",
+      "vision_model.model.layers.21.mlp.fc1",
+      "vision_model.model.layers.21.mlp.fc2",
+      "vision_model.model.layers.22.self_attn.q_proj",
+      "vision_model.model.layers.22.self_attn.k_proj",
+      "vision_model.model.layers.22.self_attn.v_proj",
+      "vision_model.model.layers.22.self_attn.o_proj",
+      "vision_model.model.layers.22.mlp.fc1",
+      "vision_model.model.layers.22.mlp.fc2",
+      "vision_model.model.layers.23.self_attn.q_proj",
+      "vision_model.model.layers.23.self_attn.k_proj",
+      "vision_model.model.layers.23.self_attn.v_proj",
+      "vision_model.model.layers.23.self_attn.o_proj",
+      "vision_model.model.layers.23.mlp.fc1",
+      "vision_model.model.layers.23.mlp.fc2",
+      "vision_model.model.layers.24.self_attn.q_proj",
+      "vision_model.model.layers.24.self_attn.k_proj",
+      "vision_model.model.layers.24.self_attn.v_proj",
+      "vision_model.model.layers.24.self_attn.o_proj",
+      "vision_model.model.layers.24.mlp.fc1",
+      "vision_model.model.layers.24.mlp.fc2",
+      "vision_model.model.layers.25.self_attn.q_proj",
+      "vision_model.model.layers.25.self_attn.k_proj",
+      "vision_model.model.layers.25.self_attn.v_proj",
+      "vision_model.model.layers.25.self_attn.o_proj",
+      "vision_model.model.layers.25.mlp.fc1",
+      "vision_model.model.layers.25.mlp.fc2",
+      "vision_model.model.layers.26.self_attn.q_proj",
+      "vision_model.model.layers.26.self_attn.k_proj",
+      "vision_model.model.layers.26.self_attn.v_proj",
+      "vision_model.model.layers.26.self_attn.o_proj",
+      "vision_model.model.layers.26.mlp.fc1",
+      "vision_model.model.layers.26.mlp.fc2",
+      "vision_model.model.layers.27.self_attn.q_proj",
+      "vision_model.model.layers.27.self_attn.k_proj",
+      "vision_model.model.layers.27.self_attn.v_proj",
+      "vision_model.model.layers.27.self_attn.o_proj",
+      "vision_model.model.layers.27.mlp.fc1",
+      "vision_model.model.layers.27.mlp.fc2",
+      "vision_model.model.layers.28.self_attn.q_proj",
+      "vision_model.model.layers.28.self_attn.k_proj",
+      "vision_model.model.layers.28.self_attn.v_proj",
+      "vision_model.model.layers.28.self_attn.o_proj",
+      "vision_model.model.layers.28.mlp.fc1",
+      "vision_model.model.layers.28.mlp.fc2",
+      "vision_model.model.layers.29.self_attn.q_proj",
+      "vision_model.model.layers.29.self_attn.k_proj",
+      "vision_model.model.layers.29.self_attn.v_proj",
+      "vision_model.model.layers.29.self_attn.o_proj",
+      "vision_model.model.layers.29.mlp.fc1",
+      "vision_model.model.layers.29.mlp.fc2",
+      "vision_model.model.layers.30.self_attn.q_proj",
+      "vision_model.model.layers.30.self_attn.k_proj",
+      "vision_model.model.layers.30.self_attn.v_proj",
+      "vision_model.model.layers.30.self_attn.o_proj",
+      "vision_model.model.layers.30.mlp.fc1",
+      "vision_model.model.layers.30.mlp.fc2",
+      "vision_model.model.layers.31.self_attn.q_proj",
+      "vision_model.model.layers.31.self_attn.k_proj",
+      "vision_model.model.layers.31.self_attn.v_proj",
+      "vision_model.model.layers.31.self_attn.o_proj",
+      "vision_model.model.layers.31.mlp.fc1",
+      "vision_model.model.layers.31.mlp.fc2",
+      "vision_model.model.layers.32.self_attn.q_proj",
+      "vision_model.model.layers.32.self_attn.k_proj",
+      "vision_model.model.layers.32.self_attn.v_proj",
+      "vision_model.model.layers.32.self_attn.o_proj",
+      "vision_model.model.layers.32.mlp.fc1",
+      "vision_model.model.layers.32.mlp.fc2",
+      "vision_model.model.layers.33.self_attn.q_proj",
+      "vision_model.model.layers.33.self_attn.k_proj",
+      "vision_model.model.layers.33.self_attn.v_proj",
+      "vision_model.model.layers.33.self_attn.o_proj",
+      "vision_model.model.layers.33.mlp.fc1",
+      "vision_model.model.layers.33.mlp.fc2",
+      "vision_model.vision_adapter.mlp.fc1",
+      "vision_model.vision_adapter.mlp.fc2",
+      "multi_modal_projector.linear_1",
+      "language_model.model.layers.0.self_attn.q_proj",
+      "language_model.model.layers.0.self_attn.k_proj",
+      "language_model.model.layers.0.self_attn.v_proj",
+      "language_model.model.layers.0.self_attn.o_proj",
+      "language_model.model.layers.0.feed_forward.router",
+      "language_model.model.layers.1.self_attn.q_proj",
+      "language_model.model.layers.1.self_attn.k_proj",
+      "language_model.model.layers.1.self_attn.v_proj",
+      "language_model.model.layers.1.self_attn.o_proj",
+      "language_model.model.layers.1.feed_forward.router",
+      "language_model.model.layers.2.self_attn.q_proj",
+      "language_model.model.layers.2.self_attn.k_proj",
+      "language_model.model.layers.2.self_attn.v_proj",
+      "language_model.model.layers.2.self_attn.o_proj",
+      "language_model.model.layers.2.feed_forward.router",
+      "language_model.model.layers.3.self_attn.q_proj",
+      "language_model.model.layers.3.self_attn.k_proj",
+      "language_model.model.layers.3.self_attn.v_proj",
+      "language_model.model.layers.3.self_attn.o_proj",
+      "language_model.model.layers.3.feed_forward.router",
+      "language_model.model.layers.4.self_attn.q_proj",
+      "language_model.model.layers.4.self_attn.k_proj",
+      "language_model.model.layers.4.self_attn.v_proj",
+      "language_model.model.layers.4.self_attn.o_proj",
+      "language_model.model.layers.4.feed_forward.router",
+      "language_model.model.layers.5.self_attn.q_proj",
+      "language_model.model.layers.5.self_attn.k_proj",
+      "language_model.model.layers.5.self_attn.v_proj",
+      "language_model.model.layers.5.self_attn.o_proj",
+      "language_model.model.layers.5.feed_forward.router",
+      "language_model.model.layers.6.self_attn.q_proj",
+      "language_model.model.layers.6.self_attn.k_proj",
+      "language_model.model.layers.6.self_attn.v_proj",
+      "language_model.model.layers.6.self_attn.o_proj",
+      "language_model.model.layers.6.feed_forward.router",
+      "language_model.model.layers.7.self_attn.q_proj",
+      "language_model.model.layers.7.self_attn.k_proj",
+      "language_model.model.layers.7.self_attn.v_proj",
+      "language_model.model.layers.7.self_attn.o_proj",
+      "language_model.model.layers.7.feed_forward.router",
+      "language_model.model.layers.8.self_attn.q_proj",
+      "language_model.model.layers.8.self_attn.k_proj",
+      "language_model.model.layers.8.self_attn.v_proj",
+      "language_model.model.layers.8.self_attn.o_proj",
+      "language_model.model.layers.8.feed_forward.router",
+      "language_model.model.layers.9.self_attn.q_proj",
+      "language_model.model.layers.9.self_attn.k_proj",
+      "language_model.model.layers.9.self_attn.v_proj",
+      "language_model.model.layers.9.self_attn.o_proj",
+      "language_model.model.layers.9.feed_forward.router",
+      "language_model.model.layers.10.self_attn.q_proj",
+      "language_model.model.layers.10.self_attn.k_proj",
+      "language_model.model.layers.10.self_attn.v_proj",
+      "language_model.model.layers.10.self_attn.o_proj",
+      "language_model.model.layers.10.feed_forward.router",
+      "language_model.model.layers.11.self_attn.q_proj",
+      "language_model.model.layers.11.self_attn.k_proj",
+      "language_model.model.layers.11.self_attn.v_proj",
+      "language_model.model.layers.11.self_attn.o_proj",
+      "language_model.model.layers.11.feed_forward.router",
+      "language_model.model.layers.12.self_attn.q_proj",
+      "language_model.model.layers.12.self_attn.k_proj",
+      "language_model.model.layers.12.self_attn.v_proj",
+      "language_model.model.layers.12.self_attn.o_proj",
+      "language_model.model.layers.12.feed_forward.router",
+      "language_model.model.layers.13.self_attn.q_proj",
+      "language_model.model.layers.13.self_attn.k_proj",
+      "language_model.model.layers.13.self_attn.v_proj",
+      "language_model.model.layers.13.self_attn.o_proj",
+      "language_model.model.layers.13.feed_forward.router",
+      "language_model.model.layers.14.self_attn.q_proj",
+      "language_model.model.layers.14.self_attn.k_proj",
+      "language_model.model.layers.14.self_attn.v_proj",
+      "language_model.model.layers.14.self_attn.o_proj",
+      "language_model.model.layers.14.feed_forward.router",
+      "language_model.model.layers.15.self_attn.q_proj",
+      "language_model.model.layers.15.self_attn.k_proj",
+      "language_model.model.layers.15.self_attn.v_proj",
+      "language_model.model.layers.15.self_attn.o_proj",
+      "language_model.model.layers.15.feed_forward.router",
+      "language_model.model.layers.16.self_attn.q_proj",
+      "language_model.model.layers.16.self_attn.k_proj",
+      "language_model.model.layers.16.self_attn.v_proj",
+      "language_model.model.layers.16.self_attn.o_proj",
+      "language_model.model.layers.16.feed_forward.router",
+      "language_model.model.layers.17.self_attn.q_proj",
+      "language_model.model.layers.17.self_attn.k_proj",
+      "language_model.model.layers.17.self_attn.v_proj",
+      "language_model.model.layers.17.self_attn.o_proj",
+      "language_model.model.layers.17.feed_forward.router",
+      "language_model.model.layers.18.self_attn.q_proj",
+      "language_model.model.layers.18.self_attn.k_proj",
+      "language_model.model.layers.18.self_attn.v_proj",
+      "language_model.model.layers.18.self_attn.o_proj",
+      "language_model.model.layers.18.feed_forward.router",
+      "language_model.model.layers.19.self_attn.q_proj",
+      "language_model.model.layers.19.self_attn.k_proj",
+      "language_model.model.layers.19.self_attn.v_proj",
+      "language_model.model.layers.19.self_attn.o_proj",
+      "language_model.model.layers.19.feed_forward.router",
+      "language_model.model.layers.20.self_attn.q_proj",
+      "language_model.model.layers.20.self_attn.k_proj",
+      "language_model.model.layers.20.self_attn.v_proj",
+      "language_model.model.layers.20.self_attn.o_proj",
+      "language_model.model.layers.20.feed_forward.router",
+      "language_model.model.layers.21.self_attn.q_proj",
+      "language_model.model.layers.21.self_attn.k_proj",
+      "language_model.model.layers.21.self_attn.v_proj",
+      "language_model.model.layers.21.self_attn.o_proj",
+      "language_model.model.layers.21.feed_forward.router",
+      "language_model.model.layers.22.self_attn.q_proj",
+      "language_model.model.layers.22.self_attn.k_proj",
+      "language_model.model.layers.22.self_attn.v_proj",
+      "language_model.model.layers.22.self_attn.o_proj",
+      "language_model.model.layers.22.feed_forward.router",
+      "language_model.model.layers.23.self_attn.q_proj",
+      "language_model.model.layers.23.self_attn.k_proj",
+      "language_model.model.layers.23.self_attn.v_proj",
+      "language_model.model.layers.23.self_attn.o_proj",
+      "language_model.model.layers.23.feed_forward.router",
+      "language_model.model.layers.24.self_attn.q_proj",
+      "language_model.model.layers.24.self_attn.k_proj",
+      "language_model.model.layers.24.self_attn.v_proj",
+      "language_model.model.layers.24.self_attn.o_proj",
+      "language_model.model.layers.24.feed_forward.router",
+      "language_model.model.layers.25.self_attn.q_proj",
+      "language_model.model.layers.25.self_attn.k_proj",
+      "language_model.model.layers.25.self_attn.v_proj",
+      "language_model.model.layers.25.self_attn.o_proj",
+      "language_model.model.layers.25.feed_forward.router",
+      "language_model.model.layers.26.self_attn.q_proj",
+      "language_model.model.layers.26.self_attn.k_proj",
+      "language_model.model.layers.26.self_attn.v_proj",
+      "language_model.model.layers.26.self_attn.o_proj",
+      "language_model.model.layers.26.feed_forward.router",
+      "language_model.model.layers.27.self_attn.q_proj",
+      "language_model.model.layers.27.self_attn.k_proj",
+      "language_model.model.layers.27.self_attn.v_proj",
+      "language_model.model.layers.27.self_attn.o_proj",
+      "language_model.model.layers.27.feed_forward.router",
+      "language_model.model.layers.28.self_attn.q_proj",
+      "language_model.model.layers.28.self_attn.k_proj",
+      "language_model.model.layers.28.self_attn.v_proj",
+      "language_model.model.layers.28.self_attn.o_proj",
+      "language_model.model.layers.28.feed_forward.router",
+      "language_model.model.layers.29.self_attn.q_proj",
+      "language_model.model.layers.29.self_attn.k_proj",
+      "language_model.model.layers.29.self_attn.v_proj",
+      "language_model.model.layers.29.self_attn.o_proj",
+      "language_model.model.layers.29.feed_forward.router",
+      "language_model.model.layers.30.self_attn.q_proj",
+      "language_model.model.layers.30.self_attn.k_proj",
+      "language_model.model.layers.30.self_attn.v_proj",
+      "language_model.model.layers.30.self_attn.o_proj",
+      "language_model.model.layers.30.feed_forward.router",
+      "language_model.model.layers.31.self_attn.q_proj",
+      "language_model.model.layers.31.self_attn.k_proj",
+      "language_model.model.layers.31.self_attn.v_proj",
+      "language_model.model.layers.31.self_attn.o_proj",
+      "language_model.model.layers.31.feed_forward.router",
+      "language_model.model.layers.32.self_attn.q_proj",
+      "language_model.model.layers.32.self_attn.k_proj",
+      "language_model.model.layers.32.self_attn.v_proj",
+      "language_model.model.layers.32.self_attn.o_proj",
+      "language_model.model.layers.32.feed_forward.router",
+      "language_model.model.layers.33.self_attn.q_proj",
+      "language_model.model.layers.33.self_attn.k_proj",
+      "language_model.model.layers.33.self_attn.v_proj",
+      "language_model.model.layers.33.self_attn.o_proj",
+      "language_model.model.layers.33.feed_forward.router",
+      "language_model.model.layers.34.self_attn.q_proj",
+      "language_model.model.layers.34.self_attn.k_proj",
+      "language_model.model.layers.34.self_attn.v_proj",
+      "language_model.model.layers.34.self_attn.o_proj",
+      "language_model.model.layers.34.feed_forward.router",
+      "language_model.model.layers.35.self_attn.q_proj",
+      "language_model.model.layers.35.self_attn.k_proj",
+      "language_model.model.layers.35.self_attn.v_proj",
+      "language_model.model.layers.35.self_attn.o_proj",
+      "language_model.model.layers.35.feed_forward.router",
+      "language_model.model.layers.36.self_attn.q_proj",
+      "language_model.model.layers.36.self_attn.k_proj",
+      "language_model.model.layers.36.self_attn.v_proj",
+      "language_model.model.layers.36.self_attn.o_proj",
+      "language_model.model.layers.36.feed_forward.router",
+      "language_model.model.layers.37.self_attn.q_proj",
+      "language_model.model.layers.37.self_attn.k_proj",
+      "language_model.model.layers.37.self_attn.v_proj",
+      "language_model.model.layers.37.self_attn.o_proj",
+      "language_model.model.layers.37.feed_forward.router",
+      "language_model.model.layers.38.self_attn.q_proj",
+      "language_model.model.layers.38.self_attn.k_proj",
+      "language_model.model.layers.38.self_attn.v_proj",
+      "language_model.model.layers.38.self_attn.o_proj",
+      "language_model.model.layers.38.feed_forward.router",
+      "language_model.model.layers.39.self_attn.q_proj",
+      "language_model.model.layers.39.self_attn.k_proj",
+      "language_model.model.layers.39.self_attn.v_proj",
+      "language_model.model.layers.39.self_attn.o_proj",
+      "language_model.model.layers.39.feed_forward.router",
+      "language_model.model.layers.40.self_attn.q_proj",
+      "language_model.model.layers.40.self_attn.k_proj",
+      "language_model.model.layers.40.self_attn.v_proj",
+      "language_model.model.layers.40.self_attn.o_proj",
+      "language_model.model.layers.40.feed_forward.router",
+      "language_model.model.layers.41.self_attn.q_proj",
+      "language_model.model.layers.41.self_attn.k_proj",
+      "language_model.model.layers.41.self_attn.v_proj",
+      "language_model.model.layers.41.self_attn.o_proj",
+      "language_model.model.layers.41.feed_forward.router",
+      "language_model.model.layers.42.self_attn.q_proj",
+      "language_model.model.layers.42.self_attn.k_proj",
+      "language_model.model.layers.42.self_attn.v_proj",
+      "language_model.model.layers.42.self_attn.o_proj",
+      "language_model.model.layers.42.feed_forward.router",
+      "language_model.model.layers.43.self_attn.q_proj",
+      "language_model.model.layers.43.self_attn.k_proj",
+      "language_model.model.layers.43.self_attn.v_proj",
+      "language_model.model.layers.43.self_attn.o_proj",
+      "language_model.model.layers.43.feed_forward.router",
+      "language_model.model.layers.44.self_attn.q_proj",
+      "language_model.model.layers.44.self_attn.k_proj",
+      "language_model.model.layers.44.self_attn.v_proj",
+      "language_model.model.layers.44.self_attn.o_proj",
+      "language_model.model.layers.44.feed_forward.router",
+      "language_model.model.layers.45.self_attn.q_proj",
+      "language_model.model.layers.45.self_attn.k_proj",
+      "language_model.model.layers.45.self_attn.v_proj",
+      "language_model.model.layers.45.self_attn.o_proj",
+      "language_model.model.layers.45.feed_forward.router",
+      "language_model.model.layers.46.self_attn.q_proj",
+      "language_model.model.layers.46.self_attn.k_proj",
+      "language_model.model.layers.46.self_attn.v_proj",
+      "language_model.model.layers.46.self_attn.o_proj",
+      "language_model.model.layers.46.feed_forward.router",
+      "language_model.model.layers.47.self_attn.q_proj",
+      "language_model.model.layers.47.self_attn.k_proj",
+      "language_model.model.layers.47.self_attn.v_proj",
+      "language_model.model.layers.47.self_attn.o_proj",
+      "language_model.model.layers.47.feed_forward.router",
+      "language_model.lm_head"
+    ],
+    "kv_cache_scheme": null,
+    "quant_method": "compressed-tensors",
+    "quantization_status": "compressed"
+  }
+}

generation_config.json

@@ -0,0 +1,13 @@
+{
+  "bos_token_id": 200000,
+  "do_sample": true,
+  "eos_token_id": [
+    200001,
+    200007,
+    200008
+  ],
+  "pad_token_id": 200018,
+  "temperature": 0.6,
+  "top_p": 0.9,
+  "transformers_version": "4.51.0.dev0"
+}

model-00001-of-00023.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:36c6c6b4a70837939c51ebbc1a3682c2dfd0695e5ba8f8509d8860088191cf51
+size 4987679352

model-00002-of-00023.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:45564a59bceae860c561c0df3a749fc9cd53f9544f0f3eb7e20c3ea3abaeca50
+size 4993243544

model-00003-of-00023.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5d3709c7ec359d32f8a2e5087abf949fa142aa277e44036af4532b654c412bee
+size 4993249744

model-00004-of-00023.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:cfbd86bd62812b58da4002f26e43d8ee6f6842a30c84cdc6d209dc08a47c80a9
+size 4993385008

model-00005-of-00023.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3a2bdbd7070ebe089dd41dcd30d0974c846923fb28b864a34625828c8bb4720e
+size 4993243448

model-00006-of-00023.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:086ff4c6faf089ed2dcc7e0526d0a88cec285ffbaa7022058f3f3e2028b5e00c
+size 4993249968

model-00007-of-00023.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5c714823d9aa502d897584058d4591782582f98d97fe11533994f2ebce6c39c2
+size 4993243800

model-00008-of-00023.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:39da01bc2ff9b736889b0d1bb4315ca841823e0bbd1a31fb0049e58e51ac9f8b
+size 4993407816

model-00009-of-00023.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:40f590c5cbb908cc5c8d317594859894db13bcac090a7fd46b6d2818197816cc
+size 4993227400

model-00010-of-00023.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:97f05df75e2df713003387ede5cc74881be14267e3b7ac14eec84353a25da127
+size 4993243696

model-00011-of-00023.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d1d7b546425089d7f141df49c480b4ad85c925993e9507f351d470bdf731aadd
+size 4993243736

model-00012-of-00023.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:48e491ca3534c4e178a6fd7c8f2560b01432a0915cae017d735b51aa01d131f6
+size 4993249944

model-00013-of-00023.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:01fb18922b16aa486f4b86dc435125eb4b375c189186f07179fdd6a502cb237f
+size 4993407832

model-00014-of-00023.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a9aa009c000001a690cb7e4106639c19862a5b2deea42f6ae4b7121decbfaa22
+size 4993221256

model-00015-of-00023.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d67b71b3b97463e83aadf441ffa9b6acb53408420c42c3e566dd023404174593
+size 4993249848

model-00016-of-00023.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f4d43bdb18deaaf45a819c87f680e69c6b792449982efb8c300d1599b62cbe04
+size 4993243744

model-00017-of-00023.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:2b1470c606ee6cf40a30b6b9639b5f03cb3f005542d4a120a9edc7bca3bf3d8f
+size 4993243808

model-00018-of-00023.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:7a7c1418d67d020178d1eec2b95a94f1f18e8663186a4e0273a62cc9720fe637
+size 4993413984

model-00019-of-00023.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f2640c96edc392c6d34f7bba8a7c02dba377b31d1c4719abfed3126ae76c458d
+size 4993221256

model-00020-of-00023.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e527ea23f0451c33ace375998d9923b8596f91db1298be0d5b9fbd61286764cb
+size 4993243704

model-00021-of-00023.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c7a9fa2abdef027a6c0b606834d006715aaec059de76eabf78ea382117d312eb
+size 4993249896

model-00022-of-00023.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:6b5275c8540db1f40baf698cf480c844f534fbdc918717ddc654f49c3fe27f45
+size 4993243816

model-00023-of-00023.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:77ac77a23183abb72299176e5782765908f48c37c480b3256686046ce365e64a
+size 4796555224

preprocessor_config.json

@@ -0,0 +1,33 @@
+{
+  "crop_size": null,
+  "data_format": "channels_first",
+  "default_to_square": true,
+  "device": null,
+  "do_center_crop": null,
+  "do_convert_rgb": true,
+  "do_normalize": true,
+  "do_rescale": true,
+  "do_resize": true,
+  "image_mean": [
+    0.5,
+    0.5,
+    0.5
+  ],
+  "image_processor_type": "Llama4ImageProcessorFast",
+  "image_std": [
+    0.5,
+    0.5,
+    0.5
+  ],
+  "input_data_format": null,
+  "max_patches": 16,
+  "processor_class": "Llama4Processor",
+  "resample": 2,
+  "rescale_factor": 0.00392156862745098,
+  "resize_to_max_canvas": false,
+  "return_tensors": null,
+  "size": {
+    "height": 336,
+    "width": 336
+  }
+}

processor_config.json

@@ -0,0 +1,6 @@
+{
+  "fake_image_token": "<|image|>",
+  "image_token": "<|image|>",
+  "patch_size": 14,
+  "processor_class": "Llama4Processor"
+}

recipe.yaml

@@ -0,0 +1,12 @@
+default_stage:
+  default_modifiers:
+    QuantizationModifier:
+      config_groups:
+        group_0:
+          targets: [Linear]
+          weights: {num_bits: 8, type: float, symmetric: true, strategy: channel, observer: mse}
+          input_activations: {num_bits: 8, type: float, symmetric: true, strategy: token,
+            dynamic: true, observer: null}
+          output_activations: null
+      ignore: ['re:.*lm_head', 're:.*self_attn', 're:.*router', 're:.*vision_model', 're:.*multi_modal_projector']
+      targets: [Linear]

special_tokens_map.json

@@ -0,0 +1,5 @@
+{
+  "bos_token": "<|begin_of_text|>",
+  "eos_token": "<|eot|>",
+  "pad_token": "<|finetune_right_pad_id|>"
+}

tokenizer.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:172c9eb4beafc72601690da3ccfcede5c2e6806a8d5ec1fca33e22acea8023a4
+size 27948578

tokenizer.model

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d0bdbaf59b0762c8c807617e2d8ea51420eb1b1de266df2495be755c8e0ed6ed
+size 3622230