README.md

---
license: mit
language:
- zh
metrics:
- bleu
pipeline_tag: question-answering
tags:
- medical
- llama-cpp
- gguf-my-repo
base_model: WangCa/Qwen2.5-7B-Medicine
---

# Triangle104/Qwen2.5-7B-Medicine-Q8_0-GGUF
This model was converted to GGUF format from [`WangCa/Qwen2.5-7B-Medicine`](https://huggingface.co/WangCa/Qwen2.5-7B-Medicine) using llama.cpp via the ggml.ai's [GGUF-my-repo](https://huggingface.co/spaces/ggml-org/gguf-my-repo) space.
Refer to the [original model card](https://huggingface.co/WangCa/Qwen2.5-7B-Medicine) for more details on the model.

---
Model Description
-



Qwen2.5-7B-Instruct-Medical is a medical domain-specific model 
fine-tuned from the Qwen2.5-7B-Instruct model using 340,000 medical 
dialogue samples. This model is optimized to provide accurate and 
contextually relevant responses to medical-related inquiries, making it 
an excellent choice for healthcare applications such as medical 
chatbots, decision support systems, and educational tools.



	
		
	

Model Details
-


Base Model: Qwen2.5-7B-Instruct
Fine-tuning Dataset: 340,000 medical dialogue samples
Training Duration: 51 hours
Hardware Used: 6x NVIDIA RTX 3090 (24GB VRAM)
Optimization Algorithm: AdamW
Training Method: LoRA (Low-Rank Adaptation)
Training Framework: PyTorch



	
		
	

Performance
-


BLEU-4 Score:
Base Model: 23.5 (on a test set of 500 samples)
Fine-tuned Model: 55.7 (on the same test set)




This shows a significant improvement in the model's ability to 
generate more fluent and contextually relevant responses after 
fine-tuning on the medical dialogue dataset.



	
		
	

Intended Use
-


This model is specifically tailored for medical dialogue tasks and can be used for:


Medical question answering
Healthcare chatbots
Clinical decision support systems
Medical education and training



	
		
	

Performance
-


The model exhibits a strong understanding of medical terminology, 
clinical contexts, and patient interactions, making it a powerful tool 
for applications in healthcare and medical research.



	
		
	

Usage
-


To use this model, you can load it using the transformers library in Python:


from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("path_to_model")
tokenizer = AutoTokenizer.from_pretrained("path_to_model")

input_text = "What are the symptoms of diabetes?"
inputs = tokenizer(input_text, return_tensors="pt")
output = model.generate(**inputs)
print(tokenizer.decode(output[0], skip_special_tokens=True))
<




	
		
	

Limitations
-


While this model has been fine-tuned on a medical dialogue dataset, 
it may still make errors or provide inaccurate responses in highly 
specialized medical domains or cases where the input data falls outside 
the training data's coverage. Always ensure human supervision in 
critical healthcare scenarios.



	
		
	

License
-



This model is released under the MIT License.



	
		
	

Acknowledgements
-



Dataset: 340,000 medical dialogues (From Modelscope).
LoRA (Low-Rank Adaptation): This technique was used to efficiently 
fine-tune the model without modifying the full parameter set, allowing 
for faster and more memory-efficient training.

---

## Use with llama.cpp
Install llama.cpp through brew (works on Mac and Linux)

```bash
brew install llama.cpp

```
Invoke the llama.cpp server or the CLI.

### CLI:
```bash
llama-cli --hf-repo Triangle104/Qwen2.5-7B-Medicine-Q8_0-GGUF --hf-file qwen2.5-7b-medicine-q8_0.gguf -p "The meaning to life and the universe is"
```

### Server:
```bash
llama-server --hf-repo Triangle104/Qwen2.5-7B-Medicine-Q8_0-GGUF --hf-file qwen2.5-7b-medicine-q8_0.gguf -c 2048
```

Note: You can also use this checkpoint directly through the [usage steps](https://github.com/ggerganov/llama.cpp?tab=readme-ov-file#usage) listed in the Llama.cpp repo as well.

Step 1: Clone llama.cpp from GitHub.
```
git clone https://github.com/ggerganov/llama.cpp
```

Step 2: Move into the llama.cpp folder and build it with `LLAMA_CURL=1` flag along with other hardware-specific flags (for ex: LLAMA_CUDA=1 for Nvidia GPUs on Linux).
```
cd llama.cpp && LLAMA_CURL=1 make
```

Step 3: Run inference through the main binary.
```
./llama-cli --hf-repo Triangle104/Qwen2.5-7B-Medicine-Q8_0-GGUF --hf-file qwen2.5-7b-medicine-q8_0.gguf -p "The meaning to life and the universe is"
```
or 
```
./llama-server --hf-repo Triangle104/Qwen2.5-7B-Medicine-Q8_0-GGUF --hf-file qwen2.5-7b-medicine-q8_0.gguf -c 2048
```