Duplicate from starvector/starvector-1b-im2svg

Co-authored-by: Juan A. Rodriguez <[email protected]>

.gitattributes

@@ -0,0 +1,35 @@
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tar filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zst filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,151 @@
+---
+library_name: transformers
+license: apache-2.0
+language:
+- en
+---
+
+# Model Card for StarVector
+
+![image/png](https://cdn-uploads.huggingface.co/production/uploads/65c27c201b5b51dd4814fcd2/ULL7FkrMHA38I8olD7nEh.png)
+
+StarVector is a foundation model for generating Scalable Vector Graphics (SVG) code from images and text. It utilizes a Vision-Language Modeling architecture to understand both visual and textual inputs, enabling high-quality vectorization and text-guided SVG creation.
+
+## Model Details
+
+### Model Description
+
+This is the model card for the StarVector model, a 🤗 transformers model. StarVector is a foundation model for generating Scalable Vector Graphics (SVG) code from images and text. It utilizes a Vision-Language Modeling architecture to understand both visual and textual inputs, enabling high-quality vectorization and text-guided SVG creation.
+
+- **Developed by:** ServiceNow Research, Mila - Quebec AI Institute, ETS, Montreal.
+- **Shared by :** Juan A Rodriguez, Abhay Puri, Shubham Agarwal, Issam H. Laradji, Sai Rajeswar, Pau Rodriguez, David Vazquez, Christopher Pal, Marco Pedersoli.
+- **Model type:** Vision-Language Model for SVG Generation.
+- **Language(s) (NLP):** English.
+- **License:** Apache 2.0
+
+### Model Architecture
+
+The StarVector architecture integrates an image encoder and a Large Language Model (LLM) Adapter to generate SVG code from both image and text inputs. Images are first converted into embeddings using a Vision Transformer (ViT), after which the LLM Adapter maps these embeddings into the LLM's embedding space to create visual tokens. Text prompts are handled through the LLM’s tokenizer and embedder. This unified multimodal approach ensures precise and contextually rich SVG output.
+
+<figure>
+  <img src="https://cdn-uploads.huggingface.co/production/uploads/65c27c201b5b51dd4814fcd2/IVGxASfNr8wfu-agH9Nqj.png" alt="Figure 2: StarVector Architecture">
+  <figcaption>Figure 2: a) StarVector Architecture: StarVector projects images into embeddings via an image encoder, then maps these embeddings to the LLM hidden space using an LLM Adapter, generating Visual Tokens. Text conditioning is achieved with the LLM's tokenizer and embedder. The model learns to map token sequences (visual or textual) to SVG code. The symbol ⊕ denotes mutually exclusive operations (image-to- SVG or text-to-SVG), while ‖ indicates sequence concatenation. Figure 2: b)Vision Model and Adapter: The image encoder employs a Vision Transformer (ViT) to process image patches sequentially. The LLM Adapter non-linearly projects embeddings into visual tokens for LLM integration.</figcaption>
+</figure>
+
+
+
+### Model Sources
+
+- **Repository:** [https://github.com/joanrod/star-vector](https://github.com/joanrod/star-vector)
+- **Paper:** [https://arxiv.org/abs/2312.11556](https://arxiv.org/abs/2312.11556)
+
+## Uses
+
+### Direct Use
+
+Image-to-SVG generation, Text-to-SVG generation.
+
+### Downstream Use
+
+Creation of icons, logotypes, technical diagrams, and other vector graphics.
+
+### Out-of-Scope Use
+
+Generating realistic photographic images or complex 3D graphics.
+
+## Bias, Risks, and Limitations
+
+Potential biases may exist in the model due to the composition of the training data (SVG-Stack). The model's ability to perfectly vectorize all types of images and interpret all textual instructions may have limitations. Users should be aware of these potential issues, especially in critical applications.
+
+### Recommendations
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. Further investigation into the model's behavior across different types of inputs is recommended.
+
+## How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+```Python
+from PIL import Image
+from transformers import AutoModelForCausalLM, AutoTokenizer, AutoProcessor
+from starvector.data.util import process_and_rasterize_svg
+import torch
+ 
+model_name = "starvector/starvector-1b-im2svg"
+ 
+starvector = AutoModelForCausalLM.from_pretrained(model_name, torch_dtype=torch.float16, trust_remote_code=True)
+processor = starvector.model.processor
+tokenizer = starvector.model.svg_transformer.tokenizer
+ 
+starvector.cuda()
+starvector.eval()
+ 
+image_pil = Image.open('assets/examples/sample-18.png')
+ 
+image = processor(image_pil, return_tensors="pt")['pixel_values'].cuda()
+if not image.shape[0] == 1:
+    image = image.squeeze(0)
+batch = {"image": image}
+ 
+raw_svg = starvector.generate_im2svg(batch, max_length=4000)[0]
+svg, raster_image = process_and_rasterize_svg(raw_svg)
+```
+
+## Training Details
+### Training Data
+SVG-Stack: A dataset of over 2 million SVG samples.
+
+### Training Procedure
+The model utilizes a Vision-Language Modeling architecture. Images are projected into embeddings via an image encoder, then mapped to the LLM hidden space using an LLM Adapter, generating Visual Tokens. Text conditioning is achieved with the LLM's tokenizer and embedder. The model learns to map token sequences (visual or textual) to SVG code.
+
+## Evaluation
+### Testing Data & Factors
+#### Testing Data
+SVG-Bench
+
+#### Factors
+SVG-Stack, SVG-Fonts, SVG-Icons, SVG-Emoji, SVG-Diagrams.
+
+## Models
+
+StarVector models achieve state-of-the-art performance on SVG generation tasks
+
+We provide [Hugging Face 🤗 model checkpoints](https://huggingface.co/collections/starvector/starvector-models-6783b22c7bd4b43d13cb5289) for image2SVG vectorization, for 💫 StarVector-8B and 💫 StarVector-1B. These are the results on SVG-Bench, using the DinoScore metric.
+
+| Method             | SVG-Stack | SVG-Fonts | SVG-Icons | SVG-Emoji | SVG-Diagrams |
+|--------------------|-----------|-----------|-----------|-----------|--------------|
+| AutoTrace          | 0.942     | 0.954     | 0.946     | 0.975     | 0.874        |
+| Potrace            | 0.898     | 0.967     | 0.972     | 0.882     | 0.875        |
+| VTracer            | 0.954     | 0.964     | 0.940     | 0.981     | 0.882        |
+| Im2Vec             | 0.692     | 0.733     | 0.754     | 0.732     | -            |
+| LIVE               | 0.934     | 0.956     | 0.959     | 0.969     | 0.870        |
+| DiffVG             | 0.810     | 0.821     | 0.952     | 0.814     | 0.822        |
+| GPT-4-V            | 0.852     | 0.842     | 0.848     | 0.850     | -            |
+| 💫 **StarVector-1B** | 0.926     | 0.978     | 0.975     | 0.929     | 0.943        |
+| 💫 **StarVector-8B** | 0.966     | 0.982     | 0.984     | 0.981     | 0.959        |
+
+**Note:** StarVector models will not work for natural images or illustrations, as they have not been trained on those images. They excel in vectorizing icons, logotypes, technical diagrams, graphs, and charts.
+
+As shown in the table above, StarVector-8B achieves the highest performance across all benchmark datasets, demonstrating its effectiveness in generating high-quality SVG code from images. The model's ability to understand and reproduce complex vector graphics makes it particularly valuable for applications requiring precise vectorization of icons, logos, and technical diagrams.
+
+## Summary
+StarVector represents a significant advancement in the field of vector graphics generation. By combining the power of vision-language models with a comprehensive training dataset, we've created a system that can accurately translate images into high-quality SVG code. The model's performance on SVG-Bench demonstrates its effectiveness across a wide range of vector graphics tasks.
+
+We believe that StarVector will enable new applications in design, illustration, and technical documentation, making vector graphics more accessible and easier to create. We invite the research community to build upon our work and explore new directions in this exciting field.
+
+For more details, please refer to our [paper](https://arxiv.org/abs/2312.11556) and explore our [code](https://github.com/joanrod/star-vector) repository.
+
+## BibTeX entry and citation info
+
+
+```
+@misc{rodriguez2024starvector,
+      title={StarVector: Generating Scalable Vector Graphics Code from Images and Text},
+      author={Juan A. Rodriguez and Abhay Puri and Shubham Agarwal and Issam H. Laradji and Pau Rodriguez and Sai Rajeswar and David Vazquez and Christopher Pal and Marco Pedersoli},
+      year={2024},
+      eprint={2312.11556},
+      archivePrefix={arXiv},
+      primaryClass={cs.CV},
+      url={https://arxiv.org/abs/2312.11556},
+}
+```

added_tokens.json

@@ -0,0 +1,6 @@
+{
+  "<caption-start>": 49155,
+  "<image-start>": 49154,
+  "<svg-start>": 49153,
+  "[PAD]": 49152
+}

config.json

@@ -0,0 +1,34 @@
+{
+  "_name_or_path": "ServiceNow/starvector-1b-im2svg",
+  "adapter_norm": "batch_norm",
+  "adapter_size": "large",
+  "architectures": [
+    "StarVectorForCausalLM"
+  ],
+  "auto_map": {
+    "AutoConfig": "starvector_arch.StarVectorConfig",
+    "AutoModelForCausalLM": "starvector_arch.StarVectorForCausalLM"
+  },
+  "dropout": 0.1,
+  "hidden_size": 2048,
+  "hidden_size_scale": 2,
+  "image_encoder_type": "clip",
+  "image_size": 224,
+  "init_type": "glorot",
+  "max_length_train": 8192,
+  "max_position_embeddings": 8192,
+  "model_type": "starvector",
+  "multi_query": true,
+  "num_attention_heads": 16,
+  "num_hidden_layers": 24,
+  "num_kv_heads": 4,
+  "starcoder_model_name": "bigcode/starcoderbase-1b",
+  "torch_dtype": "float16",
+  "train_LLM": true,
+  "train_image_encoder": false,
+  "transformers_version": "4.40.1",
+  "use_cache": true,
+  "use_flash_attn": true,
+  "vocab_size": 49156,
+  "image_token_index": 49154
+}

model-00001-of-00002.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:bea3a2add4a0107ae405391eaaefd7b95339d33b1ad98ed87862716ebdc02c20
+size 4995740600

model-00002-of-00002.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c71c1a94c42a1408f4a75e519814f10005061a0a81a3c87f0f136d475fa916b6
+size 146964720

model.safetensors.index.json

@@ -0,0 +1,591 @@
+{
+  "metadata": {
+    "total_size": 5142622224
+  },
+  "weight_map": {
+    "model.image_encoder.ln_vision.bias": "model-00002-of-00002.safetensors",
+    "model.image_encoder.ln_vision.weight": "model-00002-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.class_embedding": "model-00001-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.conv1.weight": "model-00001-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.ln_pre.bias": "model-00001-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.ln_pre.weight": "model-00001-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.positional_embedding": "model-00001-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.0.attn.in_proj_bias": "model-00001-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.0.attn.in_proj_weight": "model-00001-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.0.attn.out_proj.bias": "model-00001-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.0.attn.out_proj.weight": "model-00001-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.0.ln_1.bias": "model-00001-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.0.ln_1.weight": "model-00001-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.0.ln_2.bias": "model-00001-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.0.ln_2.weight": "model-00001-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.0.mlp.c_fc.bias": "model-00001-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.0.mlp.c_fc.weight": "model-00001-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.0.mlp.c_proj.bias": "model-00001-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.0.mlp.c_proj.weight": "model-00001-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.1.attn.in_proj_bias": "model-00001-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.1.attn.in_proj_weight": "model-00001-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.1.attn.out_proj.bias": "model-00001-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.1.attn.out_proj.weight": "model-00001-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.1.ln_1.bias": "model-00001-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.1.ln_1.weight": "model-00001-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.1.ln_2.bias": "model-00001-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.1.ln_2.weight": "model-00001-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.1.mlp.c_fc.bias": "model-00001-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.1.mlp.c_fc.weight": "model-00001-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.1.mlp.c_proj.bias": "model-00001-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.1.mlp.c_proj.weight": "model-00001-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.10.attn.in_proj_bias": "model-00001-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.10.attn.in_proj_weight": "model-00001-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.10.attn.out_proj.bias": "model-00001-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.10.attn.out_proj.weight": "model-00001-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.10.ln_1.bias": "model-00001-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.10.ln_1.weight": "model-00001-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.10.ln_2.bias": "model-00001-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.10.ln_2.weight": "model-00001-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.10.mlp.c_fc.bias": "model-00001-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.10.mlp.c_fc.weight": "model-00001-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.10.mlp.c_proj.bias": "model-00001-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.10.mlp.c_proj.weight": "model-00001-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.11.attn.in_proj_bias": "model-00001-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.11.attn.in_proj_weight": "model-00001-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.11.attn.out_proj.bias": "model-00001-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.11.attn.out_proj.weight": "model-00001-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.11.ln_1.bias": "model-00001-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.11.ln_1.weight": "model-00001-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.11.ln_2.bias": "model-00001-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.11.ln_2.weight": "model-00001-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.11.mlp.c_fc.bias": "model-00001-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.11.mlp.c_fc.weight": "model-00001-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.11.mlp.c_proj.bias": "model-00001-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.11.mlp.c_proj.weight": "model-00001-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.12.attn.in_proj_bias": "model-00001-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.12.attn.in_proj_weight": "model-00001-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.12.attn.out_proj.bias": "model-00001-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.12.attn.out_proj.weight": "model-00001-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.12.ln_1.bias": "model-00001-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.12.ln_1.weight": "model-00001-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.12.ln_2.bias": "model-00001-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.12.ln_2.weight": "model-00001-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.12.mlp.c_fc.bias": "model-00001-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.12.mlp.c_fc.weight": "model-00001-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.12.mlp.c_proj.bias": "model-00001-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.12.mlp.c_proj.weight": "model-00001-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.13.attn.in_proj_bias": "model-00001-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.13.attn.in_proj_weight": "model-00001-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.13.attn.out_proj.bias": "model-00001-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.13.attn.out_proj.weight": "model-00001-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.13.ln_1.bias": "model-00001-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.13.ln_1.weight": "model-00001-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.13.ln_2.bias": "model-00001-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.13.ln_2.weight": "model-00001-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.13.mlp.c_fc.bias": "model-00001-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.13.mlp.c_fc.weight": "model-00001-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.13.mlp.c_proj.bias": "model-00001-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.13.mlp.c_proj.weight": "model-00001-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.14.attn.in_proj_bias": "model-00001-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.14.attn.in_proj_weight": "model-00001-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.14.attn.out_proj.bias": "model-00001-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.14.attn.out_proj.weight": "model-00001-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.14.ln_1.bias": "model-00001-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.14.ln_1.weight": "model-00001-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.14.ln_2.bias": "model-00001-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.14.ln_2.weight": "model-00001-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.14.mlp.c_fc.bias": "model-00001-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.14.mlp.c_fc.weight": "model-00001-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.14.mlp.c_proj.bias": "model-00001-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.14.mlp.c_proj.weight": "model-00001-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.15.attn.in_proj_bias": "model-00001-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.15.attn.in_proj_weight": "model-00001-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.15.attn.out_proj.bias": "model-00001-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.15.attn.out_proj.weight": "model-00001-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.15.ln_1.bias": "model-00001-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.15.ln_1.weight": "model-00001-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.15.ln_2.bias": "model-00001-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.15.ln_2.weight": "model-00001-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.15.mlp.c_fc.bias": "model-00001-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.15.mlp.c_fc.weight": "model-00001-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.15.mlp.c_proj.bias": "model-00001-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.15.mlp.c_proj.weight": "model-00001-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.16.attn.in_proj_bias": "model-00001-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.16.attn.in_proj_weight": "model-00001-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.16.attn.out_proj.bias": "model-00001-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.16.attn.out_proj.weight": "model-00001-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.16.ln_1.bias": "model-00001-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.16.ln_1.weight": "model-00001-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.16.ln_2.bias": "model-00001-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.16.ln_2.weight": "model-00001-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.16.mlp.c_fc.bias": "model-00001-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.16.mlp.c_fc.weight": "model-00001-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.16.mlp.c_proj.bias": "model-00001-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.16.mlp.c_proj.weight": "model-00001-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.17.attn.in_proj_bias": "model-00001-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.17.attn.in_proj_weight": "model-00001-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.17.attn.out_proj.bias": "model-00001-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.17.attn.out_proj.weight": "model-00001-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.17.ln_1.bias": "model-00001-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.17.ln_1.weight": "model-00001-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.17.ln_2.bias": "model-00002-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.17.ln_2.weight": "model-00002-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.17.mlp.c_fc.bias": "model-00001-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.17.mlp.c_fc.weight": "model-00001-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.17.mlp.c_proj.bias": "model-00002-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.17.mlp.c_proj.weight": "model-00002-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.18.attn.in_proj_bias": "model-00002-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.18.attn.in_proj_weight": "model-00002-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.18.attn.out_proj.bias": "model-00002-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.18.attn.out_proj.weight": "model-00002-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.18.ln_1.bias": "model-00002-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.18.ln_1.weight": "model-00002-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.18.ln_2.bias": "model-00002-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.18.ln_2.weight": "model-00002-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.18.mlp.c_fc.bias": "model-00002-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.18.mlp.c_fc.weight": "model-00002-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.18.mlp.c_proj.bias": "model-00002-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.18.mlp.c_proj.weight": "model-00002-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.19.attn.in_proj_bias": "model-00002-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.19.attn.in_proj_weight": "model-00002-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.19.attn.out_proj.bias": "model-00002-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.19.attn.out_proj.weight": "model-00002-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.19.ln_1.bias": "model-00002-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.19.ln_1.weight": "model-00002-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.19.ln_2.bias": "model-00002-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.19.ln_2.weight": "model-00002-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.19.mlp.c_fc.bias": "model-00002-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.19.mlp.c_fc.weight": "model-00002-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.19.mlp.c_proj.bias": "model-00002-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.19.mlp.c_proj.weight": "model-00002-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.2.attn.in_proj_bias": "model-00001-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.2.attn.in_proj_weight": "model-00001-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.2.attn.out_proj.bias": "model-00001-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.2.attn.out_proj.weight": "model-00001-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.2.ln_1.bias": "model-00001-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.2.ln_1.weight": "model-00001-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.2.ln_2.bias": "model-00001-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.2.ln_2.weight": "model-00001-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.2.mlp.c_fc.bias": "model-00001-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.2.mlp.c_fc.weight": "model-00001-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.2.mlp.c_proj.bias": "model-00001-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.2.mlp.c_proj.weight": "model-00001-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.20.attn.in_proj_bias": "model-00002-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.20.attn.in_proj_weight": "model-00002-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.20.attn.out_proj.bias": "model-00002-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.20.attn.out_proj.weight": "model-00002-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.20.ln_1.bias": "model-00002-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.20.ln_1.weight": "model-00002-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.20.ln_2.bias": "model-00002-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.20.ln_2.weight": "model-00002-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.20.mlp.c_fc.bias": "model-00002-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.20.mlp.c_fc.weight": "model-00002-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.20.mlp.c_proj.bias": "model-00002-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.20.mlp.c_proj.weight": "model-00002-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.21.attn.in_proj_bias": "model-00002-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.21.attn.in_proj_weight": "model-00002-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.21.attn.out_proj.bias": "model-00002-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.21.attn.out_proj.weight": "model-00002-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.21.ln_1.bias": "model-00002-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.21.ln_1.weight": "model-00002-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.21.ln_2.bias": "model-00002-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.21.ln_2.weight": "model-00002-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.21.mlp.c_fc.bias": "model-00002-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.21.mlp.c_fc.weight": "model-00002-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.21.mlp.c_proj.bias": "model-00002-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.21.mlp.c_proj.weight": "model-00002-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.22.attn.in_proj_bias": "model-00002-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.22.attn.in_proj_weight": "model-00002-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.22.attn.out_proj.bias": "model-00002-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.22.attn.out_proj.weight": "model-00002-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.22.ln_1.bias": "model-00002-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.22.ln_1.weight": "model-00002-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.22.ln_2.bias": "model-00002-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.22.ln_2.weight": "model-00002-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.22.mlp.c_fc.bias": "model-00002-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.22.mlp.c_fc.weight": "model-00002-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.22.mlp.c_proj.bias": "model-00002-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.22.mlp.c_proj.weight": "model-00002-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.3.attn.in_proj_bias": "model-00001-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.3.attn.in_proj_weight": "model-00001-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.3.attn.out_proj.bias": "model-00001-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.3.attn.out_proj.weight": "model-00001-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.3.ln_1.bias": "model-00001-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.3.ln_1.weight": "model-00001-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.3.ln_2.bias": "model-00001-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.3.ln_2.weight": "model-00001-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.3.mlp.c_fc.bias": "model-00001-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.3.mlp.c_fc.weight": "model-00001-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.3.mlp.c_proj.bias": "model-00001-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.3.mlp.c_proj.weight": "model-00001-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.4.attn.in_proj_bias": "model-00001-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.4.attn.in_proj_weight": "model-00001-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.4.attn.out_proj.bias": "model-00001-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.4.attn.out_proj.weight": "model-00001-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.4.ln_1.bias": "model-00001-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.4.ln_1.weight": "model-00001-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.4.ln_2.bias": "model-00001-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.4.ln_2.weight": "model-00001-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.4.mlp.c_fc.bias": "model-00001-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.4.mlp.c_fc.weight": "model-00001-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.4.mlp.c_proj.bias": "model-00001-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.4.mlp.c_proj.weight": "model-00001-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.5.attn.in_proj_bias": "model-00001-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.5.attn.in_proj_weight": "model-00001-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.5.attn.out_proj.bias": "model-00001-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.5.attn.out_proj.weight": "model-00001-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.5.ln_1.bias": "model-00001-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.5.ln_1.weight": "model-00001-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.5.ln_2.bias": "model-00001-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.5.ln_2.weight": "model-00001-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.5.mlp.c_fc.bias": "model-00001-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.5.mlp.c_fc.weight": "model-00001-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.5.mlp.c_proj.bias": "model-00001-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.5.mlp.c_proj.weight": "model-00001-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.6.attn.in_proj_bias": "model-00001-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.6.attn.in_proj_weight": "model-00001-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.6.attn.out_proj.bias": "model-00001-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.6.attn.out_proj.weight": "model-00001-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.6.ln_1.bias": "model-00001-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.6.ln_1.weight": "model-00001-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.6.ln_2.bias": "model-00001-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.6.ln_2.weight": "model-00001-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.6.mlp.c_fc.bias": "model-00001-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.6.mlp.c_fc.weight": "model-00001-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.6.mlp.c_proj.bias": "model-00001-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.6.mlp.c_proj.weight": "model-00001-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.7.attn.in_proj_bias": "model-00001-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.7.attn.in_proj_weight": "model-00001-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.7.attn.out_proj.bias": "model-00001-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.7.attn.out_proj.weight": "model-00001-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.7.ln_1.bias": "model-00001-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.7.ln_1.weight": "model-00001-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.7.ln_2.bias": "model-00001-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.7.ln_2.weight": "model-00001-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.7.mlp.c_fc.bias": "model-00001-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.7.mlp.c_fc.weight": "model-00001-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.7.mlp.c_proj.bias": "model-00001-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.7.mlp.c_proj.weight": "model-00001-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.8.attn.in_proj_bias": "model-00001-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.8.attn.in_proj_weight": "model-00001-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.8.attn.out_proj.bias": "model-00001-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.8.attn.out_proj.weight": "model-00001-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.8.ln_1.bias": "model-00001-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.8.ln_1.weight": "model-00001-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.8.ln_2.bias": "model-00001-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.8.ln_2.weight": "model-00001-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.8.mlp.c_fc.bias": "model-00001-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.8.mlp.c_fc.weight": "model-00001-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.8.mlp.c_proj.bias": "model-00001-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.8.mlp.c_proj.weight": "model-00001-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.9.attn.in_proj_bias": "model-00001-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.9.attn.in_proj_weight": "model-00001-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.9.attn.out_proj.bias": "model-00001-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.9.attn.out_proj.weight": "model-00001-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.9.ln_1.bias": "model-00001-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.9.ln_1.weight": "model-00001-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.9.ln_2.bias": "model-00001-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.9.ln_2.weight": "model-00001-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.9.mlp.c_fc.bias": "model-00001-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.9.mlp.c_fc.weight": "model-00001-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.9.mlp.c_proj.bias": "model-00001-of-00002.safetensors",
+    "model.image_encoder.visual_encoder.transformer.resblocks.9.mlp.c_proj.weight": "model-00001-of-00002.safetensors",
+    "model.image_projection.c_fc.bias": "model-00002-of-00002.safetensors",
+    "model.image_projection.c_fc.weight": "model-00002-of-00002.safetensors",
+    "model.image_projection.c_proj.bias": "model-00002-of-00002.safetensors",
+    "model.image_projection.c_proj.weight": "model-00002-of-00002.safetensors",
+    "model.image_projection.norm.bias": "model-00002-of-00002.safetensors",
+    "model.image_projection.norm.num_batches_tracked": "model-00002-of-00002.safetensors",
+    "model.image_projection.norm.running_mean": "model-00002-of-00002.safetensors",
+    "model.image_projection.norm.running_var": "model-00002-of-00002.safetensors",
+    "model.image_projection.norm.weight": "model-00002-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.0.attn.c_attn.bias": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.0.attn.c_attn.weight": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.0.attn.c_proj.bias": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.0.attn.c_proj.weight": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.0.ln_1.bias": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.0.ln_1.weight": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.0.ln_2.bias": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.0.ln_2.weight": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.0.mlp.c_fc.bias": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.0.mlp.c_fc.weight": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.0.mlp.c_proj.bias": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.0.mlp.c_proj.weight": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.1.attn.c_attn.bias": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.1.attn.c_attn.weight": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.1.attn.c_proj.bias": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.1.attn.c_proj.weight": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.1.ln_1.bias": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.1.ln_1.weight": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.1.ln_2.bias": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.1.ln_2.weight": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.1.mlp.c_fc.bias": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.1.mlp.c_fc.weight": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.1.mlp.c_proj.bias": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.1.mlp.c_proj.weight": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.10.attn.c_attn.bias": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.10.attn.c_attn.weight": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.10.attn.c_proj.bias": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.10.attn.c_proj.weight": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.10.ln_1.bias": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.10.ln_1.weight": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.10.ln_2.bias": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.10.ln_2.weight": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.10.mlp.c_fc.bias": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.10.mlp.c_fc.weight": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.10.mlp.c_proj.bias": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.10.mlp.c_proj.weight": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.11.attn.c_attn.bias": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.11.attn.c_attn.weight": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.11.attn.c_proj.bias": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.11.attn.c_proj.weight": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.11.ln_1.bias": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.11.ln_1.weight": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.11.ln_2.bias": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.11.ln_2.weight": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.11.mlp.c_fc.bias": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.11.mlp.c_fc.weight": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.11.mlp.c_proj.bias": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.11.mlp.c_proj.weight": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.12.attn.c_attn.bias": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.12.attn.c_attn.weight": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.12.attn.c_proj.bias": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.12.attn.c_proj.weight": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.12.ln_1.bias": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.12.ln_1.weight": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.12.ln_2.bias": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.12.ln_2.weight": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.12.mlp.c_fc.bias": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.12.mlp.c_fc.weight": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.12.mlp.c_proj.bias": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.12.mlp.c_proj.weight": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.13.attn.c_attn.bias": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.13.attn.c_attn.weight": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.13.attn.c_proj.bias": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.13.attn.c_proj.weight": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.13.ln_1.bias": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.13.ln_1.weight": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.13.ln_2.bias": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.13.ln_2.weight": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.13.mlp.c_fc.bias": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.13.mlp.c_fc.weight": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.13.mlp.c_proj.bias": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.13.mlp.c_proj.weight": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.14.attn.c_attn.bias": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.14.attn.c_attn.weight": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.14.attn.c_proj.bias": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.14.attn.c_proj.weight": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.14.ln_1.bias": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.14.ln_1.weight": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.14.ln_2.bias": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.14.ln_2.weight": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.14.mlp.c_fc.bias": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.14.mlp.c_fc.weight": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.14.mlp.c_proj.bias": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.14.mlp.c_proj.weight": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.15.attn.c_attn.bias": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.15.attn.c_attn.weight": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.15.attn.c_proj.bias": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.15.attn.c_proj.weight": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.15.ln_1.bias": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.15.ln_1.weight": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.15.ln_2.bias": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.15.ln_2.weight": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.15.mlp.c_fc.bias": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.15.mlp.c_fc.weight": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.15.mlp.c_proj.bias": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.15.mlp.c_proj.weight": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.16.attn.c_attn.bias": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.16.attn.c_attn.weight": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.16.attn.c_proj.bias": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.16.attn.c_proj.weight": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.16.ln_1.bias": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.16.ln_1.weight": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.16.ln_2.bias": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.16.ln_2.weight": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.16.mlp.c_fc.bias": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.16.mlp.c_fc.weight": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.16.mlp.c_proj.bias": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.16.mlp.c_proj.weight": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.17.attn.c_attn.bias": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.17.attn.c_attn.weight": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.17.attn.c_proj.bias": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.17.attn.c_proj.weight": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.17.ln_1.bias": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.17.ln_1.weight": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.17.ln_2.bias": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.17.ln_2.weight": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.17.mlp.c_fc.bias": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.17.mlp.c_fc.weight": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.17.mlp.c_proj.bias": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.17.mlp.c_proj.weight": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.18.attn.c_attn.bias": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.18.attn.c_attn.weight": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.18.attn.c_proj.bias": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.18.attn.c_proj.weight": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.18.ln_1.bias": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.18.ln_1.weight": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.18.ln_2.bias": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.18.ln_2.weight": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.18.mlp.c_fc.bias": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.18.mlp.c_fc.weight": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.18.mlp.c_proj.bias": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.18.mlp.c_proj.weight": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.19.attn.c_attn.bias": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.19.attn.c_attn.weight": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.19.attn.c_proj.bias": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.19.attn.c_proj.weight": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.19.ln_1.bias": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.19.ln_1.weight": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.19.ln_2.bias": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.19.ln_2.weight": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.19.mlp.c_fc.bias": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.19.mlp.c_fc.weight": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.19.mlp.c_proj.bias": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.19.mlp.c_proj.weight": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.2.attn.c_attn.bias": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.2.attn.c_attn.weight": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.2.attn.c_proj.bias": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.2.attn.c_proj.weight": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.2.ln_1.bias": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.2.ln_1.weight": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.2.ln_2.bias": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.2.ln_2.weight": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.2.mlp.c_fc.bias": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.2.mlp.c_fc.weight": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.2.mlp.c_proj.bias": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.2.mlp.c_proj.weight": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.20.attn.c_attn.bias": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.20.attn.c_attn.weight": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.20.attn.c_proj.bias": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.20.attn.c_proj.weight": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.20.ln_1.bias": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.20.ln_1.weight": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.20.ln_2.bias": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.20.ln_2.weight": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.20.mlp.c_fc.bias": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.20.mlp.c_fc.weight": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.20.mlp.c_proj.bias": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.20.mlp.c_proj.weight": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.21.attn.c_attn.bias": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.21.attn.c_attn.weight": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.21.attn.c_proj.bias": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.21.attn.c_proj.weight": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.21.ln_1.bias": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.21.ln_1.weight": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.21.ln_2.bias": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.21.ln_2.weight": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.21.mlp.c_fc.bias": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.21.mlp.c_fc.weight": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.21.mlp.c_proj.bias": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.21.mlp.c_proj.weight": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.22.attn.c_attn.bias": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.22.attn.c_attn.weight": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.22.attn.c_proj.bias": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.22.attn.c_proj.weight": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.22.ln_1.bias": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.22.ln_1.weight": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.22.ln_2.bias": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.22.ln_2.weight": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.22.mlp.c_fc.bias": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.22.mlp.c_fc.weight": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.22.mlp.c_proj.bias": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.22.mlp.c_proj.weight": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.23.attn.c_attn.bias": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.23.attn.c_attn.weight": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.23.attn.c_proj.bias": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.23.attn.c_proj.weight": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.23.ln_1.bias": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.23.ln_1.weight": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.23.ln_2.bias": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.23.ln_2.weight": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.23.mlp.c_fc.bias": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.23.mlp.c_fc.weight": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.23.mlp.c_proj.bias": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.23.mlp.c_proj.weight": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.3.attn.c_attn.bias": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.3.attn.c_attn.weight": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.3.attn.c_proj.bias": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.3.attn.c_proj.weight": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.3.ln_1.bias": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.3.ln_1.weight": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.3.ln_2.bias": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.3.ln_2.weight": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.3.mlp.c_fc.bias": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.3.mlp.c_fc.weight": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.3.mlp.c_proj.bias": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.3.mlp.c_proj.weight": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.4.attn.c_attn.bias": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.4.attn.c_attn.weight": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.4.attn.c_proj.bias": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.4.attn.c_proj.weight": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.4.ln_1.bias": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.4.ln_1.weight": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.4.ln_2.bias": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.4.ln_2.weight": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.4.mlp.c_fc.bias": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.4.mlp.c_fc.weight": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.4.mlp.c_proj.bias": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.4.mlp.c_proj.weight": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.5.attn.c_attn.bias": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.5.attn.c_attn.weight": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.5.attn.c_proj.bias": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.5.attn.c_proj.weight": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.5.ln_1.bias": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.5.ln_1.weight": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.5.ln_2.bias": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.5.ln_2.weight": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.5.mlp.c_fc.bias": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.5.mlp.c_fc.weight": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.5.mlp.c_proj.bias": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.5.mlp.c_proj.weight": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.6.attn.c_attn.bias": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.6.attn.c_attn.weight": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.6.attn.c_proj.bias": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.6.attn.c_proj.weight": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.6.ln_1.bias": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.6.ln_1.weight": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.6.ln_2.bias": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.6.ln_2.weight": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.6.mlp.c_fc.bias": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.6.mlp.c_fc.weight": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.6.mlp.c_proj.bias": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.6.mlp.c_proj.weight": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.7.attn.c_attn.bias": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.7.attn.c_attn.weight": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.7.attn.c_proj.bias": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.7.attn.c_proj.weight": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.7.ln_1.bias": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.7.ln_1.weight": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.7.ln_2.bias": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.7.ln_2.weight": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.7.mlp.c_fc.bias": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.7.mlp.c_fc.weight": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.7.mlp.c_proj.bias": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.7.mlp.c_proj.weight": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.8.attn.c_attn.bias": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.8.attn.c_attn.weight": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.8.attn.c_proj.bias": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.8.attn.c_proj.weight": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.8.ln_1.bias": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.8.ln_1.weight": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.8.ln_2.bias": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.8.ln_2.weight": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.8.mlp.c_fc.bias": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.8.mlp.c_fc.weight": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.8.mlp.c_proj.bias": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.8.mlp.c_proj.weight": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.9.attn.c_attn.bias": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.9.attn.c_attn.weight": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.9.attn.c_proj.bias": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.9.attn.c_proj.weight": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.9.ln_1.bias": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.9.ln_1.weight": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.9.ln_2.bias": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.9.ln_2.weight": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.9.mlp.c_fc.bias": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.9.mlp.c_fc.weight": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.9.mlp.c_proj.bias": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.h.9.mlp.c_proj.weight": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.ln_f.bias": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.ln_f.weight": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.wpe.weight": "model-00001-of-00002.safetensors",
+    "model.svg_transformer.transformer.transformer.wte.weight": "model-00001-of-00002.safetensors"
+  }
+}

preprocessor_config.json

@@ -0,0 +1,14 @@
+{
+  "image_processor_type": "SimpleStarVectorProcessor",
+  "mean": [
+    0.48145466,
+    0.4578275,
+    0.40821073
+  ],
+  "image_std": [
+    0.26862954,
+    0.26130258,
+    0.27577711
+  ],
+  "size": 224
+}

processing_starvector.py

@@ -0,0 +1,77 @@
+from transformers.processing_utils import ProcessorMixin
+from torchvision import transforms
+from torchvision.transforms.functional import InterpolationMode, pad
+from transformers.feature_extraction_sequence_utils import BatchFeature
+
+class SimpleStarVectorProcessor(ProcessorMixin):
+    attributes = ["tokenizer"]  # Only include tokenizer in attributes
+    valid_kwargs = ["size", "mean", "std"]  # Add other parameters as valid kwargs
+    image_processor_class = "AutoImageProcessor"
+    tokenizer_class = "AutoTokenizer"
+
+    def __init__(self, 
+                 tokenizer=None,  # Make tokenizer the first argument
+                 size=224, 
+                 mean=None, 
+                 std=None, 
+                 **kwargs,
+                 ):
+        if mean is None:
+            mean = (0.48145466, 0.4578275, 0.40821073)
+        if std is None:
+            std = (0.26862954, 0.26130258, 0.27577711)
+
+        # Store these as instance variables
+        self.mean = mean
+        self.std = std
+        self.size = size
+        
+        self.normalize = transforms.Normalize(mean=mean, std=std)        
+        
+        self.transform = transforms.Compose([
+            transforms.Lambda(lambda img: img.convert("RGB") if img.mode == "RGBA" else img),
+            transforms.Lambda(lambda img: self._pad_to_square(img)),
+            transforms.Resize(size, interpolation=InterpolationMode.BICUBIC),
+            transforms.ToTensor(),
+            self.normalize
+        ])
+
+        # Initialize parent class with tokenizer
+        super().__init__(tokenizer=tokenizer)
+
+
+    def __call__(self, images=None, text=None, **kwargs) -> BatchFeature:
+        """
+        Process images and/or text inputs.
+        
+        Args:
+            images: Optional image input(s)
+            text: Optional text input(s)
+            **kwargs: Additional arguments
+        """
+        if images is None and text is None:
+            raise ValueError("You have to specify at least one of `images` or `text`.")
+
+        image_inputs = {}
+        if images is not None:
+            if isinstance(images, (list, tuple)):
+                images_ = [self.transform(img) for img in images]
+            else:
+                images_ = self.transform(images)
+            image_inputs = {"pixel_values": images_}
+        
+        text_inputs = {}
+        if text is not None:
+            text_inputs = self.tokenizer(text, **kwargs)
+        return BatchFeature(data={**text_inputs, **image_inputs})
+
+    def _pad_to_square(self, img):
+        # Calculate padding to make the image square
+        width, height = img.size
+        max_dim = max(width, height)
+        padding = [(max_dim - width) // 2, (max_dim - height) // 2]
+        padding += [max_dim - width - padding[0], max_dim - height - padding[1]]
+        return pad(img, padding, fill=255)  # Assuming white padding
+
+
+AutoProcessor.register(SimpleStarVectorProcessor, SimpleStarVectorProcessor)

processor_config.json

@@ -0,0 +1,14 @@
+{
+  "mean": [
+    0.48145466,
+    0.4578275,
+    0.40821073
+  ],
+  "processor_class": "SimpleStarVectorProcessor",
+  "size": 224,
+  "std": [
+    0.26862954,
+    0.26130258,
+    0.27577711
+  ]
+}

special_tokens_map.json

@@ -0,0 +1,51 @@
+{
+  "additional_special_tokens": [
+    "<|endoftext|>",
+    "<fim_prefix>",
+    "<fim_middle>",
+    "<fim_suffix>",
+    "<fim_pad>",
+    "<filename>",
+    "<gh_stars>",
+    "<issue_start>",
+    "<issue_comment>",
+    "<issue_closed>",
+    "<jupyter_start>",
+    "<jupyter_text>",
+    "<jupyter_code>",
+    "<jupyter_output>",
+    "<empty_output>",
+    "<commit_before>",
+    "<commit_msg>",
+    "<commit_after>",
+    "<reponame>"
+  ],
+  "bos_token": {
+    "content": "<|endoftext|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "eos_token": {
+    "content": "<|endoftext|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": {
+    "content": "[PAD]",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "unk_token": {
+    "content": "<|endoftext|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}

starvector_arch.py

@@ -0,0 +1,204 @@
+from transformers import (
+    PretrainedConfig,
+    PreTrainedModel
+)
+from torch.nn import CrossEntropyLoss
+from transformers.models.gpt_bigcode.modeling_gpt_bigcode import CausalLMOutputWithCrossAttentions
+from typing import Optional, Tuple, Union
+import torch
+
+from transformers.processing_utils import ProcessorMixin
+from torchvision import transforms
+from torchvision.transforms.functional import InterpolationMode, pad
+from transformers.feature_extraction_sequence_utils import BatchFeature
+from transformers import AutoProcessor
+
+class SimpleStarVectorProcessor(ProcessorMixin):
+    attributes = ["tokenizer"]  # Only include tokenizer in attributes
+    valid_kwargs = ["size", "mean", "std"]  # Add other parameters as valid kwargs
+    image_processor_class = "AutoImageProcessor"
+    tokenizer_class = "AutoTokenizer"
+
+    def __init__(self, 
+                 tokenizer=None,  # Make tokenizer the first argument
+                 size=224, 
+                 mean=None, 
+                 std=None, 
+                 **kwargs,
+                 ):
+        if mean is None:
+            mean = (0.48145466, 0.4578275, 0.40821073)
+        if std is None:
+            std = (0.26862954, 0.26130258, 0.27577711)
+
+        # Store these as instance variables
+        self.mean = mean
+        self.std = std
+        self.size = size
+        self.normalize = transforms.Normalize(mean=mean, std=std)        
+        
+        self.transform = transforms.Compose([
+            transforms.Lambda(lambda img: img.convert("RGB") if img.mode == "RGBA" else img),
+            transforms.Lambda(lambda img: self._pad_to_square(img)),
+            transforms.Resize(size, interpolation=InterpolationMode.BICUBIC),
+            transforms.ToTensor(),
+            self.normalize
+        ])
+
+        # Initialize parent class with tokenizer
+        super().__init__(tokenizer=tokenizer)
+
+
+    def __call__(self, images=None, text=None, max_length=None, **kwargs) -> BatchFeature:
+        """
+        Process images and/or text inputs.
+        
+        Args:
+            images: Optional image input(s)
+            text: Optional text input(s)
+            **kwargs: Additional arguments
+        """
+        if images is None and text is None:
+            raise ValueError("You have to specify at least one of `images` or `text`.")
+
+        image_inputs = {}
+        if images is not None:
+            if isinstance(images, (list, tuple)):
+                images_ = torch.stack([self.transform(img) for img in images])
+            else:
+                images_ = self.transform(images)
+            image_inputs = {"pixel_values": images_}
+        
+        text_inputs = {}
+        if text is not None:
+            text_inputs = self.tokenizer(
+                text, truncation=True, 
+                add_special_tokens=True, 
+                padding='longest', 
+                max_length=max_length, 
+                return_tensors="pt"
+            )
+
+        return BatchFeature(data={**text_inputs, **image_inputs})
+
+    def _pad_to_square(self, img):
+        # Calculate padding to make the image square
+        width, height = img.size
+        max_dim = max(width, height)
+        padding = [(max_dim - width) // 2, (max_dim - height) // 2]
+        padding += [max_dim - width - padding[0], max_dim - height - padding[1]]
+        return pad(img, padding, fill=255)  # Assuming white padding
+
+
+AutoProcessor.register(SimpleStarVectorProcessor, SimpleStarVectorProcessor)
+
+
+class StarVectorConfig(PretrainedConfig):
+    model_type = "starvector"
+
+    def __init__(
+        self,
+        starcoder_model_name: str = "bigcode/starcoderbase-1b",
+        image_encoder_type: str = "clip",
+        adapter_norm: str = "layer_norm",
+        image_size: int = 224,
+        max_length: int = 8192,
+        max_length_train: int = 8192,
+        use_flash_attn: bool = True,
+        use_cache: bool = True,
+        num_attention_heads: int = 16,
+        num_hidden_layers: int = 24,
+        vocab_size: int = 49152,
+        hidden_size: int = 2048,
+        num_kv_heads: int = 4,
+        torch_dtype: str = "bfloat16",
+        **kwargs,
+    ):
+        kwargs["torch_dtype"] = torch_dtype
+        self.starcoder_model_name = starcoder_model_name
+        self.image_encoder_type = image_encoder_type
+        self.adapter_norm = adapter_norm
+        self.image_size = image_size
+        self.max_length = max_length
+        self.max_length_train = max_length_train
+        self.use_flash_attn = use_flash_attn
+        self.use_cache = use_cache
+        self.num_attention_heads = num_attention_heads
+        self.num_hidden_layers = num_hidden_layers
+        self.vocab_size = vocab_size
+        self.hidden_size = hidden_size
+        self.num_kv_heads = num_kv_heads
+        super().__init__(**kwargs)
+
+class StarVectorForCausalLM(PreTrainedModel):
+    config_class = StarVectorConfig
+    _no_split_modules = []
+    _supports_flash_attn_2 = True
+
+    def __init__(self, config: StarVectorConfig, **kwargs):
+        super().__init__(config)
+        starcoder_model_name = config.starcoder_model_name
+        if 'starcoder2' in starcoder_model_name:
+            from starvector.model.models.starvector_v2 import StarVectorStarCoder2
+            self.model = StarVectorStarCoder2(config=config, **kwargs)
+        else:
+            from starvector.model.models.starvector_v1 import StarVectorStarCoder
+            self.model = StarVectorStarCoder(config=config, **kwargs)
+            
+
+    @property
+    def supports_gradient_checkpointing(self):
+        # If the underlying transformer (e.g., the one in StarCoderModel)
+        # supports gradient checkpointing, delegate to it.
+        if hasattr(self.model, 'svg_transformer'):
+            return getattr(self.model.svg_transformer, 'supports_gradient_checkpointing', False)
+        return False
+
+    def gradient_checkpointing_enable(self):
+        # Optionally, forward this call to the internal transformer.
+        if hasattr(self.model, 'svg_transformer') and hasattr(self.model.svg_transformer, 'gradient_checkpointing_enable'):
+            self.model.svg_transformer.gradient_checkpointing_enable()
+
+    def forward(self,  vision_embeds, input_ids, num_generations, num_logits_to_keep) -> Union[Tuple, CausalLMOutputWithCrossAttentions]:
+        r"""
+        Wrapper for the forward pass of the model.
+        """
+        device = vision_embeds.device
+
+        completion_embeds = self.model._get_embeddings(input_ids)
+        vision_embeds = torch.cat([vision_embeds.repeat(num_generations, 1, 1), completion_embeds], dim=1)
+        attention_mask = torch.ones_like(vision_embeds[:, :, 0]).to(device)
+
+        transformer_outputs = self.model.svg_transformer.transformer.transformer(
+            inputs_embeds=vision_embeds, 
+            attention_mask=attention_mask,
+        )
+        hidden_states = transformer_outputs[0]
+
+        if num_logits_to_keep > 0:
+            lm_logits = self.model.svg_transformer.transformer.lm_head(hidden_states[:, -num_logits_to_keep:, :])
+        else:
+            lm_logits = self.model.svg_transformer.transformer.lm_head(hidden_states)
+        loss = None
+        return CausalLMOutputWithCrossAttentions(
+            loss=loss,
+            logits=lm_logits,
+            past_key_values=transformer_outputs.past_key_values,
+            hidden_states=transformer_outputs.hidden_states,
+            attentions=transformer_outputs.attentions,
+            cross_attentions=transformer_outputs.cross_attentions,
+        )
+
+    def generate_im2svg(self, batch, **kwargs):
+        return self.model.generate_im2svg(batch, **kwargs)
+    
+    def generate_im2text(self, batch, **kwargs):
+        return self.model.generate_im2text(batch, **kwargs)
+
+    def process_images(self, images):
+        return self.model.image_encoder.process_images(images)
+    
+    def gradient_checkpointing_enable(self, gradient_checkpointing_kwargs=None):
+        self.model.svg_transformer.transformer.gradient_checkpointing_enable(gradient_checkpointing_kwargs=gradient_checkpointing_kwargs)
+
+

tokenizer_config.json

@@ -0,0 +1,222 @@
+{
+  "add_bos_token": false,
+  "add_prefix_space": false,
+  "added_tokens_decoder": {
+    "0": {
+      "content": "<|endoftext|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "1": {
+      "content": "<fim_prefix>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "2": {
+      "content": "<fim_middle>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "3": {
+      "content": "<fim_suffix>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "4": {
+      "content": "<fim_pad>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "5": {
+      "content": "<filename>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "6": {
+      "content": "<gh_stars>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "7": {
+      "content": "<issue_start>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "8": {
+      "content": "<issue_comment>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "9": {
+      "content": "<issue_closed>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "10": {
+      "content": "<jupyter_start>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "11": {
+      "content": "<jupyter_text>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "12": {
+      "content": "<jupyter_code>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "13": {
+      "content": "<jupyter_output>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "14": {
+      "content": "<empty_output>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "15": {
+      "content": "<commit_before>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "16": {
+      "content": "<commit_msg>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "17": {
+      "content": "<commit_after>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "18": {
+      "content": "<reponame>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "49152": {
+      "content": "[PAD]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "49153": {
+      "content": "<svg-start>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "49154": {
+      "content": "<image-start>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "49155": {
+      "content": "<caption-start>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    }
+  },
+  "additional_special_tokens": [
+    "<|endoftext|>",
+    "<fim_prefix>",
+    "<fim_middle>",
+    "<fim_suffix>",
+    "<fim_pad>",
+    "<filename>",
+    "<gh_stars>",
+    "<issue_start>",
+    "<issue_comment>",
+    "<issue_closed>",
+    "<jupyter_start>",
+    "<jupyter_text>",
+    "<jupyter_code>",
+    "<jupyter_output>",
+    "<empty_output>",
+    "<commit_before>",
+    "<commit_msg>",
+    "<commit_after>",
+    "<reponame>"
+  ],
+  "bos_token": "<|endoftext|>",
+  "clean_up_tokenization_spaces": true,
+  "eos_token": "<|endoftext|>",
+  "errors": "replace",
+  "extra_special_tokens": {},
+  "model_max_length": 1000000000000000019884624838656,
+  "pad_token": "[PAD]",
+  "processor_class": "SimpleStarVectorProcessor",
+  "tokenizer_class": "GPT2Tokenizer",
+  "unk_token": "<|endoftext|>",
+  "vocab_size": 49152
+}