davidfred/qwen2-VL-7b-instruct-trl-sft-skin-disease

Qwen/Qwen2-VL-2B-Instruct Fine-Tuned for Skin Disease Classification Model Overview This model is a fine-tuned version of the Qwen/Qwen2-VL-2B-Instruct model, specifically adapted for classifying skin diseases from images. It was trained on a dataset of skin disease images and corresponding labels. Model Details Model Name: Qwen2-VL-7B-Instruct-Skin-Disease Base Model: Qwen/Qwen2-VL-2B-Instruct Fine-Tuning Dataset: Skin Diseases Image Dataset (Kaggle) Training Parameters: Number of epochs: 3 Batch size: 4 (per device) Gradient accumulation steps: 8 Learning rate: 2e-4 Optimizer: AdamW Hardware: NVIDIA GPU (e.g., A100) Training Time: Approximately 20 hours How to Use Installation To use this model, you'll need to install the following dependencies: bash

pip install -U -q git+https://github.com/huggingface/transformers.git git+https://github.com/huggingface/trl.git datasets bitsandbytes peft qwen-vl-utils wandb accelerate pip install -q torch==2.4.1+cu121 torchvision==0.19.1+cu121 torchaudio==2.4.1+cu121 --extra-index-url https://download.pytorch.org/whl/cu121 pip install qwen-vl-utils Loading the Model Load the fine-tuned model and processor: python

from transformers import Qwen2VLForConditionalGeneration, Qwen2VLProcessor import torch

model_id = "your_username/qwen2-7b-instruct-trl-sft-skin-disease" model = Qwen2VLForConditionalGeneration.from_pretrained( model_id, device_map="auto", torch_dtype=torch.bfloat16, ) processor = Qwen2VLProcessor.from_pretrained(model_id) Generating Predictions To generate predictions for a skin disease image, use the following function: python

from qwen_vl_utils import process_vision_info

def generate_text_from_sample(model, processor, sample, max_new_tokens=1024, device="cuda"): text_input = processor.apply_chat_template( sample[1:2], tokenize=False, add_generation_prompt=True ) image_inputs, _ = process_vision_info(sample) model_inputs = processor( text=[text_input], images=image_inputs, return_tensors="pt", ).to(device) generated_ids = model.generate(**model_inputs, max_new_tokens=max_new_tokens) trimmed_generated_ids = [out_ids[len(in_ids):] for in_ids, out_ids in zip(model_inputs.input_ids, generated_ids)] output_text = processor.batch_decode( trimmed_generated_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False ) return output_text[0]

Example usage

sample = { "role": "user", "content": [ { "type": "image", "image": "path/to/your/image.jpg", }, { "type": "text", "text": "What skin disease is shown in this image?", }, ], } output = generate_text_from_sample(model, processor, sample) print(output) System Message The model was trained with the following system message: text

You are a Vision Language Model specialized in identifying skin diseases. Your task is to analyze the provided image and classify the skin disease shown. Focus on delivering accurate, concise labels based on the visual information. Include this system message in your prompts for optimal performance. Performance The model was evaluated on a held-out test set from the skin disease dataset. The performance metrics are as follows: Accuracy: [To be filled after evaluation] Precision: [To be filled after evaluation] Recall: [To be filled after evaluation] F1 Score: [To be filled after evaluation] Limitations The model's performance may vary depending on the quality and resolution of the input images. It may not perform well on skin diseases not represented in the training dataset. The model's predictions should be used as a tool for medical professionals and not as a definitive diagnosis. Ethical Considerations This model should not be used for self-diagnosis or to replace professional medical advice. Ensure that the use of this model complies with privacy regulations regarding medical data. License This model is released under the [License Name] license. Please refer to the license file for more details. This model card provides a comprehensive guide on how to use the fine-tuned Qwen/Qwen2-VL-2B-Instruct model for skin disease classification. Make sure to fill in the performance metrics after evaluating the model on your test set.