Update README.md
Browse files
README.md
CHANGED
@@ -1,9 +1,9 @@
|
|
1 |
This is [trashpanda-org/QwQ-32B-Snowdrop-v0](https://huggingface.co/trashpanda-org/QwQ-32B-Snowdrop-v0) with the `embed_tokens` and `lm_head` tensors replaced with the correctly-sized ones from [Qwen/Qwen2.5-32B-Instruct](https://huggingface.co/Qwen/Qwen2.5-32B-Instruct).
|
2 |
|
3 |
-
At the time of posting there's an ongoing issue where the Qwen2.5 embedding tensors have dimension `152064` (matching the vocab size stated in the config), but the actual tokenizer and vocab included have fewer tokens defined (seemingly Qwen pre-initialized extra embed space for future added tokens). Some LLM software (e.g. Axolotl, Mergekit) have this trigger an automated check and, seeing that the vocab size is less than the embed size, resize the embeddings to match, which breaks compatibility in some places.
|
4 |
-
|
5 |
(Why the instruct model and not QwQ? Because that's the tokenizer trashpanda was aiming for.)
|
6 |
|
|
|
|
|
7 |
```python
|
8 |
from transformers import AutoModelForCausalLM, AutoTokenizer
|
9 |
import torch
|
|
|
1 |
This is [trashpanda-org/QwQ-32B-Snowdrop-v0](https://huggingface.co/trashpanda-org/QwQ-32B-Snowdrop-v0) with the `embed_tokens` and `lm_head` tensors replaced with the correctly-sized ones from [Qwen/Qwen2.5-32B-Instruct](https://huggingface.co/Qwen/Qwen2.5-32B-Instruct).
|
2 |
|
|
|
|
|
3 |
(Why the instruct model and not QwQ? Because that's the tokenizer trashpanda was aiming for.)
|
4 |
|
5 |
+
At the time of posting there's an ongoing issue where the Qwen2.5 embedding tensors have dimension `152064` (matching the vocab size stated in the config), but the actual tokenizer and vocab included have fewer tokens defined (seemingly Qwen pre-initialized extra embed space for future added tokens). Some LLM software (e.g. Axolotl, Mergekit) have this trigger an automated check and, seeing that the vocab size is less than the embed size, resize the embeddings to match, which breaks compatibility in some places.
|
6 |
+
|
7 |
```python
|
8 |
from transformers import AutoModelForCausalLM, AutoTokenizer
|
9 |
import torch
|