PrunaAI/llava-v1.6-vicuna-7b-bnb-4bit

Simply make AI models cheaper, smaller, faster, and greener!

• Give a thumbs up if you like this model!
• Contact us and tell us which model to compress next here.
• Request access to easily compress your own AI models here.
• Read the documentations to know more here
• Join Pruna AI community on Discord here to share feedback/suggestions or get help.

Frequently Asked Questions

• How does the compression work? The model is compressed by using bitsandbytes.
• How does the model quality change? The quality of the model output will slightly degrade.
• What is the model format? We the standard safetensors format.
• How to compress my own models? You can request premium access to more compression methods and tech support for your specific use-cases here.

Usage

from transformers import LlavaNextProcessor, LlavaNextForConditionalGeneration
import torch
from PIL import Image
import requests

processor = LlavaNextProcessor.from_pretrained("PrunaAI/llava-v1.6-vicuna-7b-bnb-4bit")

model = LlavaNextForConditionalGeneration.from_pretrained("PrunaAI/llava-v1.6-vicuna-7b-bnb-4bit") 

# prepare image and text prompt, using the appropriate prompt template
url = "https://github.com/haotian-liu/LLaVA/blob/1a91fc274d7c35a9b50b3cb29c4247ae5837ce39/images/llava_v1_5_radar.jpg?raw=true"
image = Image.open(requests.get(url, stream=True).raw)
prompt = "A chat between a curious human and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the human's questions. USER: <image>\nWhat is shown in this image? ASSISTANT:"


inputs = processor(prompt, image, return_tensors="pt").to("cuda:0")

# autoregressively complete prompt
output = model.generate(**inputs, max_new_tokens=100)

print(processor.decode(output[0], skip_special_tokens=True))

Credits & License

The license of the smashed model follows the license of the original model. Please check the license of the original model liuhaotian/llava-v1.6-vicuna-7b before using this model which provided the base model. The license of the pruna-engine is here on Pypi.

Want to compress other models?

• Contact us and tell us which model to compress next here.
• Request access to easily compress your own AI models here.