|
--- |
|
{} |
|
--- |
|
# Instella-VL-1B ✨ |
|
Welcome to the official repository for **Instella-VL-1B**, AMD's first ever Vision-Language Model (VLM). This repository provides a detailed guide for training and inference with **Instella-VL-1B**. Developed from AMD's **Instella-1B** (previously known as [AMD OLMo 1B SFT](https://www.amd.com/en/developer/resources/technical-articles/introducing-the-first-amd-1b-language-model.html) LLM), this model is fully open-source, with both model weights and training code available for AMD GPUs (MI300). Its compact size aims to make it accessible to a broad spectrum of researchers, developers, and enthusiasts, enabling them to build upon, modify, and integrate it into their own projects. |
|
|
|
[[GitHub](https://github.com/AMD-AIG-AIMA/InstellaVL)][[Blog](https://rocm.blogs.amd.com/artificial-intelligence/Instella-BL-1B-VLM/README.html)] |
|
|
|
## Main Results |
|
We compare our model with models which only releases the model weights (with * in the below table) and also models which releases weights, data curvation and all training details. |
|
|
|
<table class="tg"><thead> |
|
<tr> |
|
<td class="tg-0pky"></td> |
|
<td class="tg-c3ow">DeepSeek-VL-1.3B *</td> |
|
<td class="tg-c3ow">InternVL2-1B *</td> |
|
<td class="tg-c3ow">InternVL2.5-1B *</td> |
|
<td class="tg-c3ow">TinyLLaVA-2.4B</td> |
|
<td class="tg-c3ow">TinyLLaVA-1.5B</td> |
|
<td class="tg-c3ow">llava-onevision-1b</td> |
|
<td class="tg-c3ow">MiniCPM-V-2</td> |
|
<td class="tg-c3ow">Instella-VL-1B</td> |
|
</tr></thead> |
|
<tbody> |
|
<tr> |
|
<td class="tg-c3ow">GQA</td> |
|
<td class="tg-c3ow">--</td> |
|
<td class="tg-c3ow">55.06</td> |
|
<td class="tg-c3ow">56.66</td> |
|
<td class="tg-c3ow">61.58</td> |
|
<td class="tg-c3ow">60.28</td> |
|
<td class="tg-c3ow">57.95</td> |
|
<td class="tg-c3ow">--</td> |
|
<td class="tg-c3ow">61.52</td> |
|
</tr> |
|
<tr> |
|
<td class="tg-c3ow">SQA</td> |
|
<td class="tg-c3ow">64.52</td> |
|
<td class="tg-c3ow">89.54</td> |
|
<td class="tg-c3ow">93.90</td> |
|
<td class="tg-c3ow">64.30</td> |
|
<td class="tg-c3ow">59.69</td> |
|
<td class="tg-c3ow">59.25</td> |
|
<td class="tg-c3ow">76.10</td> |
|
<td class="tg-c3ow">83.74</td> |
|
</tr> |
|
<tr> |
|
<td class="tg-c3ow">POPE</td> |
|
<td class="tg-c3ow">85.80</td> |
|
<td class="tg-c3ow">87.40</td> |
|
<td class="tg-c3ow">89.95</td> |
|
<td class="tg-c3ow">85.66</td> |
|
<td class="tg-c3ow">84.77</td> |
|
<td class="tg-c3ow">87.17</td> |
|
<td class="tg-c3ow">86.56</td> |
|
<td class="tg-c3ow">86.73</td> |
|
</tr> |
|
<tr> |
|
<td class="tg-c3ow">MM-Bench</td> |
|
<td class="tg-c3ow">64.34</td> |
|
<td class="tg-c3ow">61.70</td> |
|
<td class="tg-c3ow">68.40</td> |
|
<td class="tg-c3ow">58.16</td> |
|
<td class="tg-c3ow">51.28</td> |
|
<td class="tg-c3ow">44.60</td> |
|
<td class="tg-c3ow">70.44</td> |
|
<td class="tg-c3ow">69.17</td> |
|
</tr> |
|
<tr> |
|
<td class="tg-c3ow">seedbench</td> |
|
<td class="tg-c3ow">65.94</td> |
|
<td class="tg-c3ow">65.90</td> |
|
<td class="tg-c3ow">71.30</td> |
|
<td class="tg-c3ow">63.30</td> |
|
<td class="tg-c3ow">60.04</td> |
|
<td class="tg-c3ow">65.43</td> |
|
<td class="tg-c3ow">66.90</td> |
|
<td class="tg-c3ow">68.47</td> |
|
</tr> |
|
<tr> |
|
<td class="tg-c3ow">MMMU</td> |
|
<td class="tg-c3ow">28.67</td> |
|
<td class="tg-c3ow">32.40</td> |
|
<td class="tg-c3ow">35.60</td> |
|
<td class="tg-c3ow">32.11</td> |
|
<td class="tg-c3ow">29.89</td> |
|
<td class="tg-c3ow">30.90</td> |
|
<td class="tg-c3ow">38.55</td> |
|
<td class="tg-c3ow">29.30</td> |
|
</tr> |
|
<tr> |
|
<td class="tg-c3ow">realworldqa</td> |
|
<td class="tg-c3ow">50.20</td> |
|
<td class="tg-c3ow">51.90</td> |
|
<td class="tg-c3ow">58.30</td> |
|
<td class="tg-c3ow">52.42</td> |
|
<td class="tg-c3ow">46.67</td> |
|
<td class="tg-c3ow">51.63</td> |
|
<td class="tg-c3ow">55.03</td> |
|
<td class="tg-c3ow">58.82</td> |
|
</tr> |
|
<tr> |
|
<td class="tg-c3ow">mmstar</td> |
|
<td class="tg-c3ow">38.30</td> |
|
<td class="tg-c3ow">46.18</td> |
|
<td class="tg-c3ow">47.93</td> |
|
<td class="tg-c3ow">37.17</td> |
|
<td class="tg-c3ow">31.87</td> |
|
<td class="tg-c3ow">37.38</td> |
|
<td class="tg-c3ow">40.93</td> |
|
<td class="tg-c3ow">43.21</td> |
|
</tr> |
|
<tr> |
|
<td class="tg-c3ow"><span style="font-weight:bold">Average</span></td> |
|
<td class="tg-c3ow">-</td> |
|
<td class="tg-c3ow">61.26</td> |
|
<td class="tg-c3ow">65.26</td> |
|
<td class="tg-c3ow">56.84</td> |
|
<td class="tg-c3ow">53.06</td> |
|
<td class="tg-c3ow">54.29</td> |
|
<td class="tg-c3ow">-</td> |
|
<td class="tg-c3ow">62.62</td> |
|
</tr> |
|
<tr> |
|
<td class="tg-c3ow">ocrbench</td> |
|
<td class="tg-c3ow">41.40</td> |
|
<td class="tg-c3ow">74.40</td> |
|
<td class="tg-c3ow">74.20</td> |
|
<td class="tg-c3ow">28.90</td> |
|
<td class="tg-c3ow">34.40</td> |
|
<td class="tg-c3ow">43.00</td> |
|
<td class="tg-c3ow">60.00</td> |
|
<td class="tg-c3ow">67.90</td> |
|
</tr> |
|
<tr> |
|
<td class="tg-c3ow">TextVQA</td> |
|
<td class="tg-c3ow">57.54</td> |
|
<td class="tg-c3ow">69.60</td> |
|
<td class="tg-c3ow">72.96</td> |
|
<td class="tg-c3ow">47.05</td> |
|
<td class="tg-c3ow">49.54</td> |
|
<td class="tg-c3ow">49.54</td> |
|
<td class="tg-c3ow">74.23</td> |
|
<td class="tg-c3ow">71.23</td> |
|
</tr> |
|
<tr> |
|
<td class="tg-c3ow">AI2D</td> |
|
<td class="tg-c3ow">51.13</td> |
|
<td class="tg-c3ow">62.40</td> |
|
<td class="tg-c3ow">67.58</td> |
|
<td class="tg-c3ow">49.58</td> |
|
<td class="tg-c3ow">43.10</td> |
|
<td class="tg-c3ow">57.35</td> |
|
<td class="tg-c3ow">64.40</td> |
|
<td class="tg-c3ow">66.65</td> |
|
</tr> |
|
<tr> |
|
<td class="tg-c3ow">ChartQA</td> |
|
<td class="tg-c3ow">47.40</td> |
|
<td class="tg-c3ow">71.52</td> |
|
<td class="tg-c3ow">75.76</td> |
|
<td class="tg-c3ow">12.96</td> |
|
<td class="tg-c3ow">15.24</td> |
|
<td class="tg-c3ow">61.24</td> |
|
<td class="tg-c3ow">59.80</td> |
|
<td class="tg-c3ow">72.52</td> |
|
</tr> |
|
<tr> |
|
<td class="tg-c3ow">DocVQA</td> |
|
<td class="tg-c3ow">35.70</td> |
|
<td class="tg-c3ow">80.94</td> |
|
<td class="tg-c3ow">82.76</td> |
|
<td class="tg-c3ow">25.82</td> |
|
<td class="tg-c3ow">30.38</td> |
|
<td class="tg-c3ow">71.22</td> |
|
<td class="tg-c3ow">69.54</td> |
|
<td class="tg-c3ow">80.30</td> |
|
</tr> |
|
<tr> |
|
<td class="tg-c3ow">InfoVQA</td> |
|
<td class="tg-c3ow">20.52</td> |
|
<td class="tg-c3ow">46.30</td> |
|
<td class="tg-c3ow">53.62</td> |
|
<td class="tg-c3ow">21.35</td> |
|
<td class="tg-c3ow">24.46</td> |
|
<td class="tg-c3ow">41.18</td> |
|
<td class="tg-c3ow">38.24</td> |
|
<td class="tg-c3ow">46.40</td> |
|
</tr> |
|
<tr> |
|
<td class="tg-c3ow">OCR Average</td> |
|
<td class="tg-c3ow">42.28</td> |
|
<td class="tg-c3ow">67.53</td> |
|
<td class="tg-c3ow">71.15</td> |
|
<td class="tg-c3ow">30.94</td> |
|
<td class="tg-c3ow">32.85</td> |
|
<td class="tg-c3ow">53.92</td> |
|
<td class="tg-c3ow">61.04</td> |
|
<td class="tg-c3ow">67.50</td> |
|
</tr> |
|
</tbody></table> |
|
|
|
### Quick Start |
|
> [!NOTE] |
|
> Follow below packages list for setting up the inference environment. |
|
> ```bash |
|
> pip==25.0 |
|
> wheel==0.45.1 |
|
> setuptools==75.8.0 |
|
> torch==2.6.0 |
|
> torchvision==0.21.0 |
|
> transformers==4.49.0 |
|
> einops==0.8.0 |
|
> ``` |
|
|
|
```python |
|
import torch |
|
from transformers import AutoTokenizer, AutoProcessor, AutoConfig, AutoModelForCausalLM |
|
|
|
from PIL import Image |
|
import requests |
|
from io import BytesIO |
|
|
|
def load_image(image_file): |
|
if image_file.startswith("http") or image_file.startswith("https"): |
|
response = requests.get(image_file) |
|
image = Image.open(BytesIO(response.content)).convert("RGB") |
|
else: |
|
image = Image.open(image_file).convert("RGB") |
|
return image |
|
|
|
|
|
config = AutoConfig.from_pretrained("amd/Instella-VL-1B", trust_remote_code=True) |
|
tokenizer = AutoTokenizer.from_pretrained("amd/Instella-VL-1B", config=config, trust_remote_code=True) |
|
processor = AutoProcessor.from_pretrained("amd/Instella-VL-1B", trust_remote_code=True) |
|
model = AutoModelForCausalLM.from_pretrained("amd/Instella-VL-1B", trust_remote_code=True).to('cuda') # or 'cpu' |
|
model.eval() |
|
|
|
# For single image and text |
|
query="Describe the image." |
|
image=load_image("path/to/your_image") # can be a https:// url |
|
out = processor.encode(query, image, model.get_vision_tower().image_processor, tokenizer, config) |
|
inputs = {k: v.to(model.device) for k, v in out.items() if isinstance(v, torch.Tensor)} |
|
with torch.inference_mode(): |
|
output_ids = model.generate(inputs["input_ids"], images=inputs['image_tensor'], image_sizes=out['image_sizes'], do_sample=True, num_beams=1, temperature=0.2, max_new_tokens=1024, use_cache=True, stopping_criteria=out['stopping_criteria'], eos_token_id=out['eos_token_id']) |
|
outputs = processor.decode(output_ids) |
|
print("InstellaVL: ", outputs) |
|
|
|
# For batch of images and text. |
|
query=["Describe the image.", "What is the color of the dog?"] |
|
image=[load_image("../assets/images/instellavl.png"), load_image("../assets/images/example2_dog.jpg")] |
|
outs = processor.batch_encode(query, image, model.get_vision_tower().image_processor, tokenizer, config) |
|
|
|
for idx, o in enumerate(outs): |
|
ins = {k: v.to(model.device) for k, v in o.items() if isinstance(v, torch.Tensor)} |
|
with torch.inference_mode(): |
|
output_ids = model.generate(ins["input_ids"], |
|
images=ins['image_tensor'], |
|
image_sizes=o['image_sizes'], |
|
do_sample=True, |
|
num_beams=1, |
|
temperature=0.2, |
|
max_new_tokens=1024, |
|
use_cache=True, |
|
stopping_criteria=o['stopping_criteria'], |
|
eos_token_id=o['eos_token_id']) |
|
outputs = processor.decode(output_ids) |
|
print("Query: ", query[idx]) |
|
print("InstellaVL: ", outputs) |
|
``` |
|
|
|
<details> |
|
<summary><b>TL;DR</b>: Loading from locally saved checkpoint</summary> |
|
<p><strong>Note:</strong> Do <code>pip install -e . --no-deps</code> to register/include for InstellaVL repo as <code>instellavl</code> package into Python package list.</p> |
|
|
|
``` python |
|
import torch |
|
|
|
# Import essential modules |
|
from instellavl.constants import DEFAULT_IMAGE_TOKEN, IMAGE_TOKEN_INDEX |
|
from instellavl.conversation import conv_templates, SeparatorStyle |
|
from instellavl.model.builder import load_pretrained_model |
|
from instellavl.utils import disable_torch_init |
|
from instellavl.mm_utils import process_images, tokenizer_image_token, get_model_name_from_path |
|
|
|
from PIL import Image |
|
|
|
import requests |
|
from io import BytesIO |
|
|
|
# Login into HF Hub |
|
from huggingface_hub import login |
|
login(token = "<Your HFtoken id>") # Enter your token |
|
|
|
def load_image(image_file): |
|
if image_file.startswith("http") or image_file.startswith("https"): |
|
response = requests.get(image_file) |
|
image = Image.open(BytesIO(response.content)).convert("RGB") |
|
else: |
|
image = Image.open(image_file).convert("RGB") |
|
return image |
|
|
|
# |
|
# ========= CHANGE IMAGE and Query only HERE ============ |
|
image_file = '/path/to/Instella-VL-repo/assets/images/example2_dog.jpg' # Enter the test image path |
|
query = 'Describe this image.' |
|
# ======================================================= |
|
|
|
disable_torch_init() |
|
conv_mode = 'instella' |
|
|
|
# Model loading |
|
model_path = '<path/to/model-checkpoint-saved-locally>' # Enter your model path, should contain instellavl substring in the name. |
|
model_name = get_model_name_from_path(model_path) |
|
tokenizer, model, image_processor, context_len = load_pretrained_model(model_path, None, model_name, False, False) |
|
model.eval() |
|
model = model.to('cuda') # change to 'cpu' if not 'cuda' |
|
|
|
# Image pre-processing |
|
image = load_image(image_file) |
|
image_tensor = process_images([image], image_processor, model.config) |
|
image_tensor = image_processor.preprocess(image, return_tensors="pt")["pixel_values"].to(model.dtype) |
|
|
|
# Text pre-processing - follow the below logic too when there is no Image: |
|
# if images is not None and len(image_tensor) != 0 and DEFAULT_IMAGE_TOKEN not in text: |
|
# question = DEFAULT_IMAGE_TOKEN + "\n" + text |
|
# else: |
|
# question = text |
|
query = query.replace(DEFAULT_IMAGE_TOKEN, "").strip() |
|
question = DEFAULT_IMAGE_TOKEN + "\n" + query |
|
conv = conv_templates[conv_mode].copy() |
|
conv.append_message(conv.roles[0], question) |
|
conv.append_message(conv.roles[1], None) |
|
prompt_question = conv.get_prompt() |
|
|
|
# Final arrangements required |
|
input_ids = tokenizer_image_token(prompt_question, tokenizer, IMAGE_TOKEN_INDEX, return_tensors="pt").unsqueeze(0) |
|
keywords = [conv.sep] |
|
image_sizes = [image.size] |
|
stopping_criteria = [KeywordsStoppingCriteria(keywords, tokenizer, input_ids)] |
|
terminators = [tokenizer.eos_token_id, tokenizer.convert_tokens_to_ids("|||IP_ADDRESS|||")] |
|
|
|
with torch.inference_mode(): |
|
output_ids = model.generate(input_ids.to(model.device), images=image_tensor.to(model.device), image_sizes=image_sizes, do_sample=True, num_beams=1, temperature=0.2, max_new_tokens=1024, use_cache=True, stopping_criteria=stopping_criteria, eos_token_id=terminators) |
|
|
|
outputs = tokenizer.decode(output_ids[0, input_ids.shape[1] :]).strip() |
|
print("InstellaVL: ", outputs) |
|
``` |
|
</details> |
|
|
|
## Model Architecture |
|
|
|
| Parts | Parameter size | Number of layers | Number of heads | Hidden size | Patch Size | |
|
| ------------- |:-------------:|:-----:|:-----:|:-----:|:-----:| |
|
| Vision Encoder | 300M | 24| 16 | 1024 | 14 | |
|
| MLP | 6.3M | 2 | - | 2048 | - | |
|
| LM | 1.2B | 16 | 16 | 2048 | - | |
|
|
|
We initialize the vision encoder from [CLIP-ViT-L/14@336](https://huggingface.co/openai/clip-vit-large-patch14-336) and initialize LM from [AMD OLMo 1B SFT](https://huggingface.co/amd/AMD-OLMo-1B-SFT) |
|
|
|
## Training Stages |
|
|
|
| Stages | MLP Warmup | Pretraining | Instruction Tuning | |
|
| ------------- |:-------------:|:-----:|:-----:| |
|
| Tunable Parts | Adapter | Entire Model | Entire Model | |
|
|
|
## Hardware |
|
Training was conducted with up to 4 nodes, totaling 32 GPUs. Each node comprises [8 AMD Instinct™ MI300X GPUs](https://www.amd.com/en/products/accelerators/instinct/mi300/mi300x.html) |
|
|
|
**MLP warmup**: 1 node |
|
**Pretraining**: 2 nodes |
|
**Finetune**: 4 nodes |
|
|
|
## Datasets |
|
|
|
### MLP Warmup |
|
[BLIP558K](https://huggingface.co/datasets/liuhaotian/LLaVA-Pretrain) |
|
|
|
<h3 align="center">Pretraining Stage</h3> |
|
|
|
| **Domain** | **Datasets** | **Num of Examples** | **Licenses** | |
|
|---|:---:|---:|:---| |
|
| Image Captions | [BLIP150K](https://huggingface.co/datasets/liuhaotian/LLaVA-Pretrain), [COCO118K](https://huggingface.co/datasets/liuhaotian/LLaVA-Pretrain), [CC3M-Recap](https://huggingface.co/datasets/lmms-lab/LLaVA-ReCap-CC3M), [Pixmo_Cap](https://huggingface.co/datasets/allenai/pixmo-cap) | 3.52M | BSD 3-Clause for BLIP150K, COCO118K; Apache 2 for CC3M-Recap; ODC-BY-1.0 for Pixmo_Cap; see source materials for CC3M-Recap | |
|
| OCR | [SynthDog_EN](https://huggingface.co/datasets/lmms-lab/LLaVA-OneVision-Mid-Data), [SynthDog_ZH](https://huggingface.co/datasets/lmms-lab/LLaVA-OneVision-Mid-Data), [UReader](https://huggingface.co/datasets/lmms-lab/LLaVA-OneVision-Mid-Data), [ART](https://rrc.cvc.uab.es/?ch=14&com=downloads), [COCO-Text](https://bgshih.github.io/cocotext/), [HierText](https://github.com/google-research-datasets/hiertext), [Uber-Text](https://s3-us-west-2.amazonaws.com/uber-common-public/ubertext/index.html), [TextOCR](https://huggingface.co/datasets/lmms-lab/LLaVA-OneVision-Data), [OpenVINO](https://github.com/openvinotoolkit/cvat), [MLT-17](https://rrc.cvc.uab.es/?ch=8&com=downloads) | 913K | Apache 2 for SynthDog_EN, SynthDog_ZH, UReader, TextOCR, OpenVINO; CC By 4.0 for COCO-Text; CC BY-SA 4.0 for HierText, Uber-Text; See source materials for ART, MLT-17 | |
|
| Doc | [DocVQA](https://huggingface.co/datasets/lmms-lab/LLaVA-OneVision-Data), [DocStruct4M](https://huggingface.co/datasets/mPLUG/DocStruct4M) | 410K | Apache 2 | |
|
| Table & Chart & Plot | [Chart2Text](https://github.com/vis-nlp/Chart-to-text/tree/main/pew_dataset/dataset/imgs), [UniChart](https://huggingface.co/datasets/ahmed-masry/unichart-pretrain-data), [PlotQA](https://huggingface.co/datasets/lmms-lab/LLaVA-OneVision-Data), [WidgetCaption](https://huggingface.co/datasets/rootsautomation/RICO-WidgetCaptioning?row=0), [Screen2Words](https://huggingface.co/datasets/rootsautomation/RICO-Screen2Words), [SciGraphQA-295K](https://huggingface.co/datasets/alexshengzhili/SciGraphQA-295K-train), [Paper2Fig100K](https://zenodo.org/records/7299423#.Y2lzonbMKUl), [MMC Instruction](https://huggingface.co/datasets/xywang1/MMC/viewer/MMC-Instruction), [M-Paper](https://huggingface.co/datasets/mPLUG/M-Paper) | 1.97M | GPL-3.0 for Chart2Text; MIT for UniChart, SciGraphQA-295K; Apache 2 for PlotQA, M-Paper; CC By 4.0 for WidgetCaption, Screen2Words, Paper2Fig100K; CC BY-SA 4.0 for MMC Instruction | |
|
| Text Only | [Evol-Instruct-GPT-4](https://huggingface.co/datasets/lmms-lab/LLaVA-OneVision-Mid-Data/tree/main/evol_instruct) | 70K | Apache 2 | |
|
|
|
<h3 align="center">Instruction-tuning Stage</h3> |
|
|
|
| **Domain** | **Datasets** | **Num of Examples** | **Licenses** | |
|
|---|:---:|---:|:---| |
|
| General | [AOKVQA, CLEVR, Hateful Memes, Image Textualization, OKVQA, ScienceQA, ShareGPT-4V, TallyQA, Visual7W, VizWiz, VQAv2, WebSight, ALLaVA Instruct, Cambrian, COCO Caption, IconQA, LLaVA-158K, LLaVAR, RefCOCO, ShareGPT-4O, Vision FLAN, VisText, VQARAD, VSR, InterGPS](https://huggingface.co/datasets/lmms-lab/LLaVA-OneVision-Data), [Image-Paragraph-Captioning, ImageNet, COCO-GOI, COCO-ITM, Visual Dialog, SNLI-VE](https://huggingface.co/datasets/MMInstruction/M3IT), [Web-Landmark, Web-Celebrity, SAM, LAION-GPT-4V-Dataset, OODVQA]( https://huggingface.co/datasets/nyu-visionx/Cambrian-10M/tree/main), [Pixmo_Cap](https://huggingface.co/datasets/allenai/pixmo-cap), [Pixmo_Count](https://huggingface.co/datasets/allenai/pixmo-count), [Pixmo_Points](https://huggingface.co/datasets/allenai/pixmo-points), [Pixmo_Ask_Model_Anything](https://huggingface.co/datasets/allenai/pixmo-ask-model-anything), [SVIT_Core_150K](https://huggingface.co/datasets/BAAI/SVIT), [Localized Narratives](https://huggingface.co/datasets/HuggingFaceM4/the_cauldron) | 2.66M | see source materials for Image-Paragraph-Captioning, ImageNet, COCO-GOI, COCO-ITM, Visual Dialog, SNLI-VE; ODC-BY-1.0 for Pixmo_Cap, Pixmo_Count, Pixmo_Points, Pixmo_Ask_Model_Anything; CC By 4.0 for SVIT_Core_150K, Localized Narratives; Apache 2 for rest of the datasets; | |
|
| Table & Chart & Screen | [AI2D, ChartQA, DocVQA, FigureQA, InfographicVQA, RoBUT-SQA, RoBUT-WTQ, TQA, UReader IE, UReader QA, Chart2Text, , Diagram Image2Text, DVQA, HiTab, LRV Chart, RoBUT WikiSQL, Screen2Words, UReader Caption, UReader KG, VisualMRC](https://huggingface.co/datasets/lmms-lab/LLaVA-OneVision-Data), [TinyChartData](https://huggingface.co/datasets/mPLUG/TinyChartData) | 866K | Apache 2 | |
|
| Doc | [ArxivQA](https://huggingface.co/datasets/MMInstruction/ArxivQA), [DocDownstream-1.0](https://huggingface.co/datasets/mPLUG/DocDownstream-1.0), [DocReason25K](https://huggingface.co/datasets/mPLUG/DocReason25K), [DocStruct4M](https://huggingface.co/datasets/mPLUG/DocStruct4M), [Pixmo_Docs](https://huggingface.co/datasets/allenai/pixmo-docs) | 522K | CC BY-SA 4.0 for ArxivQA; Apache 2 for DocDownstream-1.0, DocReason25K, DocStruct4M; ODC-BY-1.0 for Pixmo_Docs | |
|
| General OCR | [ChromeWriting, IIIT5K, K12 Printing, Rendered Text, TextCaps, HME100K, IAM, TextOCR-GPT-4V](https://huggingface.co/datasets/lmms-lab/LLaVA-OneVision-Data), [SynthDog-EN](https://huggingface.co/datasets/lmms-lab/LLaVA-OneVision-Mid-Data) | 84K | Apache 2 | |
|
| Math & Reasoning | [MAVIS Manual Collection, CLEVR-Math, Geo170K QA, GEOS, GeoMVerse, MapQA, Super-CLEVR, UniGeo, LRV Normal, Visual Genome, MAVIS Data Engine, Geo170K Align, Geometry3K, GeoQA+, TabMWP, GQA, RAVEN, MathVision, KVQA, VCR](https://huggingface.co/datasets/lmms-lab/LLaVA-OneVision-Data), [FinQA](https://huggingface.co/datasets/HuggingFaceM4/the_cauldron), [Design2Code, IDK](https://huggingface.co/datasets/nyu-visionx/Cambrian-10M/) | 460K | CC By 4.0 for FinQA; Apache 2 for rest of the datasets | |
|
| Others | [IQA, MOCHEG, Shapes](https://huggingface.co/datasets/MMInstruction/M3IT), [ALFWorld, Q-Instruct-DB](https://huggingface.co/datasets/nyu-visionx/Cambrian-10M/) | 479K | see source materials for IQA, MOCHEG, Shapes; Apache 2 for ALFWorld, Q-Instruct-DB | |
|
| Text Only | [MathQA, Magpie Pro (L3 MT), Magpie Pro (Qwen2 ST), Magpie Pro (L3 ST)](https://huggingface.co/datasets/lmms-lab/LLaVA-OneVision-Data) | 480K | Apache 2 | |
|
|
|
> [!NOTE] |
|
> Further, to strengthen model’s understanding of science-based and general reasoning questions, as identified through error analysis, we oversampled (almost doubled the volume) specific datasets from the SFT dataset pool as detailed below. |
|
> |
|
> Oversampled (~2x sampling rate): ScienceQA, AI2D, PMC-VQA, Cambrian, and TQA |
|
> |
|
> Further information concerning the training datasets, including applicable licensing terms and use restrictions, may be located at the linked source location. |
|
|
|
|
|
For the details of training hyperparameters, please check [our github repo](https://github.com/AMD-AIG-AIMA/Instella-VL) |
|
|
|
## Contributors |
|
**Core contributors:** [Ximeng Sun](https://sunxm2357.github.io/), [Aditya Kumar Singh](https://rodosingh.github.io), [Gowtham Ramesh](https://www.linkedin.com/in/gowtham1/), [Zicheng Liu](https://zicliu.wixsite.com/mysite) |
|
|
|
**Contributors:** [Pratik Prabhanjan Brahma](https://www.linkedin.com/in/pratik-p-brahma/), [Ze Wang](https://www.linkedin.com/in/ze-wang-1379601a5/), [Jiang Liu](https://joellliu.github.io/), [Jialian Wu](https://jialianwu.com/), [Prakamya Mishra](https://prakamya-mishra.github.io/), [Xiaodong Yu](https://www.xiaodongyu.me/), [Yusheng Su](https://yushengsu-thu.github.io/), [Sudhanshu Ranjan](https://www.linkedin.com/in/sudhanshu-ranjan-33a216124), [Emad Barsoum](https://www.linkedin.com/in/ebarsoum/) |
|
|
|
|
|
## Bias, Risks, and Limitations |
|
This model is made accessible without any safety guarantees. Users should be aware that the model may generate outputs that are sensitive, inaccurate, harmful, biased, or otherwise objectionable based on user prompts. It is crucial for users to conduct comprehensive safety evaluations, implement safety filtering, and verify the model's outputs to mitigate these risks. |
|
|
|
## License |
|
See Files for license and any notices. |
|
|
|
## Citing |
|
|
|
```bibtex |
|
@misc{Instella-VL-1B, |
|
title = {Instella-VL-1B: First AMD Vision Language Model}, |
|
url = {https://huggingface.co/amd/Instella-VL-1B}, |
|
author = {Ximeng Sun, Aditya Singh, Gowtham Ramesh, Jiang Liu, Ze Wang, Sudhanshu Ranjan, Pratik Prabhanjan Brahma, Prakamya Mishra, Jialian Wu, Xiaodong Yu, Yusheng Su, Emad Barsoum, Zicheng Liu}, |
|
month = {March}, |
|
year = {2025} |
|
} |
|
``` |
