Update README.md

README.md

@@ -1,3 +1,100 @@
----
-license: llama3.2
----
+---
+license: llama3.2
+datasets:
+- AdaptLLM/remote-sensing-visual-instructions
+language:
+- en
+base_model:
+- meta-llama/Llama-3.2-11B-Vision-Instruct
+tags:
+- remote-sensing
+---
+# Adapting Multimodal Large Language Models to Domains via Post-Training
+
+This repos contains the **biomedicine MLLM developed from Llama-3.2-11B-Vision-Instruct** in our paper: [On Domain-Specific Post-Training for Multimodal Large Language Models](https://huggingface.co/papers/2411.19930). The correspoding training dataset is in [medicine-visual-instructions](https://huggingface.co/datasets/AdaptLLM/medicine-visual-instructions).
+
+The main project page is: [Adapt-MLLM-to-Domains](https://huggingface.co/AdaptLLM/Adapt-MLLM-to-Domains/edit/main/README.md)
+
+## 1. To Chat with AdaMLLM  
+
+Our model architecture aligns with the base model: Llama-3.2-Vision-Instruct. We provide a usage example below, and you may refer to the official [Llama-3.2-Vision-Instruct Repository](https://huggingface.co/meta-llama/Llama-3.2-11B-Vision-Instruct) for more advanced usage instructions, 
+
+**Note:** For AdaMLLM, always place the image at the beginning of the input instruction in the messages.  
+
+<details>
+<summary> Click to expand </summary>
+
+Starting with transformers >= 4.45.0 onward, you can run inference using conversational messages that may include an image you can query about.
+
+Make sure to update your transformers installation via `pip install --upgrade transformers`.
+
+```bash
+import requests
+import torch
+from PIL import Image
+from transformers import MllamaForConditionalGeneration, AutoProcessor
+
+model_id = "AdaptLLM/remote-sensing-Llama-3.2-11B-Vision-Instruct"
+
+model = MllamaForConditionalGeneration.from_pretrained(
+    model_id,
+    torch_dtype=torch.bfloat16,
+    device_map="auto",
+)
+processor = AutoProcessor.from_pretrained(model_id)
+
+url = "https://huggingface.co/datasets/huggingface/documentation-images/resolve/0052a70beed5bf71b92610a43a52df6d286cd5f3/diffusers/rabbit.jpg"
+image = Image.open(requests.get(url, stream=True).raw)
+
+# NOTE: For AdaMLLM, always place the image at the beginning of the input instruction in the messages.  
+messages = [
+    {"role": "user", "content": [
+        {"type": "image"},
+        {"type": "text", "text": "If I had to write a haiku for this one, it would be: "}
+    ]}
+]
+input_text = processor.apply_chat_template(messages, add_generation_prompt=True)
+inputs = processor(
+    image,
+    input_text,
+    add_special_tokens=False,
+    return_tensors="pt"
+).to(model.device)
+
+output = model.generate(**inputs, max_new_tokens=30)
+print(processor.decode(output[0]))
+```
+</details>  
+
+## 2. To Evaluate Any MLLM on Domain-Specific Benchmarks  
+
+See [remote-sensing-VQA-benchmark](https://huggingface.co/datasets/AdaptLLM/remote-sensing-VQA-benchmark) to reproduce our results and evalaute more MLLMs on the domain-specific benchmarks.
+
+## 3. To Reproduce this Domain-Adapted MLLM  
+
+See [Post-Train Guide](https://github.com/bigai-ai/QA-Synthesizer/blob/main/docs/Post_Train.md) to adapt MLLMs to domains.
+
+## Citation
+If you find our work helpful, please cite us.
+
+[AdaMLLM](https://huggingface.co/papers/2411.19930)
+```bibtex
+@article{adamllm,
+  title={On Domain-Specific Post-Training for Multimodal Large Language Models},
+  author={Cheng, Daixuan and Huang, Shaohan and Zhu, Ziyu and Zhang, Xintong and Zhao, Wayne Xin and Luan, Zhongzhi and Dai, Bo and Zhang, Zhenliang},
+  journal={arXiv preprint arXiv:2411.19930},
+  year={2024}
+}
+```
+
+[Adapt LLM to Domains](https://huggingface.co/papers/2309.09530) (ICLR 2024)
+```bibtex
+@inproceedings{
+cheng2024adapting,
+title={Adapting Large Language Models via Reading Comprehension},
+author={Daixuan Cheng and Shaohan Huang and Furu Wei},
+booktitle={The Twelfth International Conference on Learning Representations},
+year={2024},
+url={https://openreview.net/forum?id=y886UXPEZ0}
+}
+```