Files changed (1) hide show
  1. README.md +94 -80
README.md CHANGED
@@ -1,81 +1,95 @@
1
- ---
2
- base_model:
3
- - Qwen/Qwen2.5-32B
4
- library_name: transformers
5
- tags:
6
- - mergekit
7
- - merge
8
- license: apache-2.0
9
- ---
10
- <img src="https://huggingface.co/arcee-train/Virtuoso-Medium-v2/resolve/main/virtuoso-medium.jpg" alt="Virtuoso-Lite Logo" style="display: block; margin: 0 auto;" />
11
-
12
- **Virtuoso-Medium-v2 (32B)** is our next-generation, 32-billion-parameter language model that builds upon the original Virtuoso-Medium architecture. This version is distilled from Deepseek-v3, leveraging an expanded dataset of 5B+ tokens worth of logits. It achieves higher benchmark scores than our previous release (including surpassing Arcee-Nova 2024 in certain tasks).
13
-
14
- ### GGUF
15
- Quantizations available [here](https://huggingface.co/arcee-ai/Virtuoso-Medium-v2-GGUF)
16
-
17
- ### Model Details
18
- - **Architecture Base:** Qwen-2.5-32B
19
- - **Parameter Count:** 32B
20
- - **Tokenizer:**
21
- - Initially integrated with Deepseek-v3 tokenizer for logit extraction.
22
- - Final alignment uses the Qwen tokenizer, using specialized “tokenizer surgery” for cross-architecture compatibility.
23
- - **Distillation Data:**
24
- - ~1.1B tokens/logits from Deepseek-v3’s training data.
25
- - Logit-level distillation using a proprietary “fusion merging” approach afterwards for maximum fidelity.
26
- - **License:** [Apache-2.0](#license)
27
-
28
- ### Background on Deepseek Distillation
29
- Deepseek-v3 serves as the teacher model, from which we capture logits across billions of tokens. Rather than standard supervised fine-tuning, we apply a full logit-level replication. This ensures more precise transference of knowledge, including advanced reasoning in:
30
- - Technical and scientific queries
31
- - Complex code generation
32
- - Mathematical problem-solving
33
-
34
- ### Intended Use Cases
35
- - **Advanced Chatbots & Virtual Assistants**
36
- - **Enterprise Data Analysis & Workflow Automation**
37
- - **Research Simulations & Natural Language Understanding**
38
- - **Educational Tools for STEM Fields**
39
-
40
- ### Evaluations
41
- <img src="https://huggingface.co/arcee-train/Virtuoso-Lite/resolve/main/Benchmarks.png" alt="Virtuoso-Lite Logo" style="display: block; margin: 0 auto;" />
42
-
43
- ### How to Use
44
- Below is a sample code snippet using `transformers`:
45
-
46
- ```python
47
- from transformers import AutoTokenizer, AutoModelForCausalLM
48
-
49
- model_name = "arcee-ai/Virtuoso-Medium-v2"
50
- tokenizer = AutoTokenizer.from_pretrained(model_name)
51
- model = AutoModelForCausalLM.from_pretrained(model_name)
52
-
53
- prompt = "Provide a concise summary of quantum entanglement."
54
- inputs = tokenizer(prompt, return_tensors="pt")
55
- outputs = model.generate(**inputs, max_new_tokens=150)
56
- print(tokenizer.decode(outputs[0], skip_special_tokens=True))
57
- ```
58
-
59
- ### Training & Fine-Tuning
60
- - **Initial Training:** Began with Qwen-32B, calibrated for large-scale text ingestion.
61
- - **Distillation & Merging:**
62
- - Trained on ~1.1B tokens worth of Deepseek-v3 logits.
63
- - Employed “fusion merging” to retain as much teacher expertise as possible.
64
- - Final step included DPO to improve alignment and reduce model hallucinations.
65
- - **Continuous Development:** Additional R1 distillations are in progress to further enhance performance and specialization.
66
-
67
- ### Performance
68
- Thanks to a larger parameter count and a richer training corpus, Virtuoso-Medium-v2 delivers high scores across multiple benchmarks (BBH, MMLU-PRO, MATH, etc.). It frequently surpasses other 30B+ models and even some 70B+ architectures in specific tasks.
69
-
70
- ### Limitations
71
- - **Context Length:** 128k Tokens
72
- - **Knowledge Cut-off:** Training data may not reflect the latest events or developments, leading to gaps in current knowledge beyond June 2024.
73
-
74
- ### Ethical Considerations
75
- - **Content Generation Risks:** Like any language model, Virtuoso-Medium-v2 can potentially generate harmful or biased content if prompted in certain ways.
76
-
77
- ### License
78
- **Virtuoso-Medium-v2 (32B)** is released under the [Apache-2.0 License](https://www.apache.org/licenses/LICENSE-2.0). You are free to use, modify, and distribute this model in both commercial and non-commercial applications, subject to the terms and conditions of the license.
79
-
80
-
 
 
 
 
 
 
 
 
 
 
 
 
 
 
81
  If you have questions or would like to share your experiences using these models, please connect with us on social media. We’re excited to see what you build—and how these models help you innovate!
 
1
+ ---
2
+ base_model:
3
+ - Qwen/Qwen2.5-32B
4
+ library_name: transformers
5
+ tags:
6
+ - mergekit
7
+ - merge
8
+ license: apache-2.0
9
+ language:
10
+ - zho
11
+ - eng
12
+ - fra
13
+ - spa
14
+ - por
15
+ - deu
16
+ - ita
17
+ - rus
18
+ - jpn
19
+ - kor
20
+ - vie
21
+ - tha
22
+ - ara
23
+ ---
24
+ <img src="https://huggingface.co/arcee-train/Virtuoso-Medium-v2/resolve/main/virtuoso-medium.jpg" alt="Virtuoso-Lite Logo" style="display: block; margin: 0 auto;" />
25
+
26
+ **Virtuoso-Medium-v2 (32B)** is our next-generation, 32-billion-parameter language model that builds upon the original Virtuoso-Medium architecture. This version is distilled from Deepseek-v3, leveraging an expanded dataset of 5B+ tokens worth of logits. It achieves higher benchmark scores than our previous release (including surpassing Arcee-Nova 2024 in certain tasks).
27
+
28
+ ### GGUF
29
+ Quantizations available [here](https://huggingface.co/arcee-ai/Virtuoso-Medium-v2-GGUF)
30
+
31
+ ### Model Details
32
+ - **Architecture Base:** Qwen-2.5-32B
33
+ - **Parameter Count:** 32B
34
+ - **Tokenizer:**
35
+ - Initially integrated with Deepseek-v3 tokenizer for logit extraction.
36
+ - Final alignment uses the Qwen tokenizer, using specialized “tokenizer surgery” for cross-architecture compatibility.
37
+ - **Distillation Data:**
38
+ - ~1.1B tokens/logits from Deepseek-v3’s training data.
39
+ - Logit-level distillation using a proprietary “fusion merging” approach afterwards for maximum fidelity.
40
+ - **License:** [Apache-2.0](#license)
41
+
42
+ ### Background on Deepseek Distillation
43
+ Deepseek-v3 serves as the teacher model, from which we capture logits across billions of tokens. Rather than standard supervised fine-tuning, we apply a full logit-level replication. This ensures more precise transference of knowledge, including advanced reasoning in:
44
+ - Technical and scientific queries
45
+ - Complex code generation
46
+ - Mathematical problem-solving
47
+
48
+ ### Intended Use Cases
49
+ - **Advanced Chatbots & Virtual Assistants**
50
+ - **Enterprise Data Analysis & Workflow Automation**
51
+ - **Research Simulations & Natural Language Understanding**
52
+ - **Educational Tools for STEM Fields**
53
+
54
+ ### Evaluations
55
+ <img src="https://huggingface.co/arcee-train/Virtuoso-Lite/resolve/main/Benchmarks.png" alt="Virtuoso-Lite Logo" style="display: block; margin: 0 auto;" />
56
+
57
+ ### How to Use
58
+ Below is a sample code snippet using `transformers`:
59
+
60
+ ```python
61
+ from transformers import AutoTokenizer, AutoModelForCausalLM
62
+
63
+ model_name = "arcee-ai/Virtuoso-Medium-v2"
64
+ tokenizer = AutoTokenizer.from_pretrained(model_name)
65
+ model = AutoModelForCausalLM.from_pretrained(model_name)
66
+
67
+ prompt = "Provide a concise summary of quantum entanglement."
68
+ inputs = tokenizer(prompt, return_tensors="pt")
69
+ outputs = model.generate(**inputs, max_new_tokens=150)
70
+ print(tokenizer.decode(outputs[0], skip_special_tokens=True))
71
+ ```
72
+
73
+ ### Training & Fine-Tuning
74
+ - **Initial Training:** Began with Qwen-32B, calibrated for large-scale text ingestion.
75
+ - **Distillation & Merging:**
76
+ - Trained on ~1.1B tokens worth of Deepseek-v3 logits.
77
+ - Employed “fusion merging” to retain as much teacher expertise as possible.
78
+ - Final step included DPO to improve alignment and reduce model hallucinations.
79
+ - **Continuous Development:** Additional R1 distillations are in progress to further enhance performance and specialization.
80
+
81
+ ### Performance
82
+ Thanks to a larger parameter count and a richer training corpus, Virtuoso-Medium-v2 delivers high scores across multiple benchmarks (BBH, MMLU-PRO, MATH, etc.). It frequently surpasses other 30B+ models and even some 70B+ architectures in specific tasks.
83
+
84
+ ### Limitations
85
+ - **Context Length:** 128k Tokens
86
+ - **Knowledge Cut-off:** Training data may not reflect the latest events or developments, leading to gaps in current knowledge beyond June 2024.
87
+
88
+ ### Ethical Considerations
89
+ - **Content Generation Risks:** Like any language model, Virtuoso-Medium-v2 can potentially generate harmful or biased content if prompted in certain ways.
90
+
91
+ ### License
92
+ **Virtuoso-Medium-v2 (32B)** is released under the [Apache-2.0 License](https://www.apache.org/licenses/LICENSE-2.0). You are free to use, modify, and distribute this model in both commercial and non-commercial applications, subject to the terms and conditions of the license.
93
+
94
+
95
  If you have questions or would like to share your experiences using these models, please connect with us on social media. We’re excited to see what you build—and how these models help you innovate!