Hugging Face's logo

CYFRAGOVPL
/
PLLuM-12B-base-250801

Text Generation
Transformers
Safetensors
Polish
mistral
text-generation-inference
Model card Files Community
Bartłomiej Koptyra commited on
Commit
5633938
·
1 Parent(s): 29a0260

Added model

Browse files
Files changed (13) hide show
  1. .gitattributes +1 -0
  2. README.md +243 -3
  3. config.json +26 -0
  4. generation_config.json +6 -0
  5. model-00001-of-00005.safetensors +3 -0
  6. model-00002-of-00005.safetensors +3 -0
  7. model-00003-of-00005.safetensors +3 -0
  8. model-00004-of-00005.safetensors +3 -0
  9. model-00005-of-00005.safetensors +3 -0
  10. model.safetensors.index.json +371 -0
  11. special_tokens_map.json +23 -0
  12. tokenizer.json +3 -0
  13. tokenizer_config.json +0 -0
.gitattributes CHANGED
@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
33
  *.zip filter=lfs diff=lfs merge=lfs -text
34
  *.zst filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
 
 
33
  *.zip filter=lfs diff=lfs merge=lfs -text
34
  *.zst filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
36
+ tokenizer.json filter=lfs diff=lfs merge=lfs -text
README.md CHANGED
@@ -1,3 +1,243 @@
1
- ---
2
- license: apache-2.0
3
- ---
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: apache-2.0
3
+ language:
4
+ - pl
5
+ pipeline_tag: text-generation
6
+ library_name: transformers
7
+ ---
8
+ <p align="center">
9
+ <img src="https://pllum.org.pl/_nuxt/PLLuM_logo_RGB_color.DXNEc-VR.png">
10
+ </p>
11
+
12
+ # PLLuM: A Family of Polish Large Language Models
13
+
14
+ ## Overview
15
+ PLLuM is a family of large language models (LLMs) specialized in Polish and other Slavic/Baltic languages, with additional English data incorporated for broader generalization. Developed through an extensive collaboration with various data providers, PLLuM models are built on high-quality text corpora and refined through instruction tuning, preference learning, and advanced alignment techniques. These models are intended to generate contextually coherent text, offer assistance in various tasks (e.g., question answering, summarization), and serve as a foundation for specialized applications such as domain-specific intelligent assistants.
16
+ In 2024, PLLuM models were developed by the PLLuM consortium. Since 2025, their development has continued under HIVE AI, a broader alliance of research institutions and organizations delivering digital public services, focused on open language technologies for Polish public administration.
17
+
18
+ ### Key Highlights
19
+ - **Extensive Data Collection**
20
+ We gathered large-scale, high-quality text data in Polish (around 150B tokens after cleaning and deduplication) and additional text in Slavic, Baltic, and English languages. Part of these tokens (28B) can be used in fully open-source models, including for commercial use (in compliance with relevant legal regulations).
21
+
22
+ - **Organic Instruction Dataset**
23
+ We curated the largest Polish collection of manually created “organic instructions” (~40k prompt-response pairs, including ~3.5k multi-turn dialogs). This human-authored instruction set is based on an extensive typology of human-model interactions and it covers a range of subtle aspects of supervised fine-tuning (SFT) that might be overlooked with automated approaches (including large scale distillation of 'strong LLMs'). It was also designed to mitigate negative linguistic transfer from non-Polish textual data used in the pre-training phase.
24
+
25
+ - **Polish Preference Corpus**
26
+ We created the first Polish-language preference corpus, featuring prompts and multiple model responses manually assessed by a demographically diverse team of annotators. This dataset teaches the model not only correctness (factual and linguistic) but also balance and safety—especially for potentially controversial or adversarial topics.
27
+
28
+ - **Evaluation Benchmarks**
29
+ We developed custom benchmarks to evaluate our models on tasks relevant to Polish public administration, where PLLuM achieved top scores among all tested models. In broader Polish-language tasks, PLLuM models also attain state-of-the-art results.
30
+
31
+ ## Model Description
32
+
33
+ Below is a summary of the new PLLuM 12B-nc-250715 models. All model names link to their respective Hugging Face resources, while the base models and licenses link to the relevant source models or license references. The model with the -chat suffix has been aligned to human preferences and is generally safer and more efficient for use in dialogue and general-purpose scenarios.
34
+
35
+ | Model Name | Params | License | Based On |
36
+ |-------------------------------------------------------|----------------------|---------------------------------------------------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------|
37
+ | [PLLuM-12B-nc-base-250715](https://huggingface.co/CYFRAGOVPL/pllum-12b-nc-base-250715) | 12B | [pllum-1.2](https://huggingface.co/CYFRAGOVPL/pllum-12b-nc-base-250715/blob/main/LICENSE) | [Mistral-Nemo-Base-2407](https://huggingface.co/mistralai/Mistral-Nemo-Base-2407) |
38
+ | [PLLuM-12B-nc-instruct-250715](https://huggingface.co/CYFRAGOVPL/pllum-12b-nc-instruct-250715) | 12B | [pllum-1.0](https://huggingface.co/CYFRAGOVPL/pllum-12b-nc-instruct-250715/blob/main/LICENSE) | [Mistral-Nemo-Base-2407](https://huggingface.co/mistralai/Mistral-Nemo-Base-2407) |
39
+ | [PLLuM-12B-nc-chat-250715](https://huggingface.co/CYFRAGOVPL/pllum-12b-nc-chat-250715) | 12B | [pllum-1.1](https://huggingface.co/CYFRAGOVPL/pllum-12b-nc-chat-250715/blob/main/LICENSE) | [Mistral-Nemo-Base-2407](https://huggingface.co/mistralai/Mistral-Nemo-Base-2407) |
40
+
41
+
42
+ ### Model Development
43
+ - **Pretraining**: All models were pretrained or continued-pretrained on large-scale Polish corpora (up to 150B tokens) plus a range of English texts. The training data reflects sources available up to autumn 2024 and does not include more recent content.
44
+ - **Instruction Fine-Tuning**: We fine-tuned the models using approximately 50k manually curated Polish "organic instructions," 30k programmatically derived instructions from high-quality Polish corpora, and 45k synthetic, context-aware instructions generated by high-performing, permissively licensed LLMs such as Mixtral8x22 and Deepseek V3.
45
+ - **Alignment and Preference Learning**: Manually annotated preference data taught the models to produce safer, balanced, and contextually appropriate responses, even in adversarial or sensitive cases.
46
+
47
+ ## Intended Use Cases
48
+ - **General Language Tasks**: Text generation, summarization, question answering, etc.
49
+ - **Domain-Specific Assistants**: Especially effective for Polish public administration and legal or bureaucratic topics where domain-aware retrieval is required.
50
+ - **Research & Development**: Building blocks for downstream AI applications in academic or industrial settings, where a strong command of the Polish language is essential.
51
+
52
+ ## How to Use
53
+ Each PLLuM model can be loaded via the Hugging Face Transformers library (or compatible frameworks). For RAG-based scenarios, pair the model with a relevant vector store or document retrieval system.
54
+
55
+ Below are some recommended steps and code snippets:
56
+
57
+ ### 1. Installation
58
+ Make sure you have the latest versions of `transformers` and `torch` (or another compatible deep learning framework) installed:
59
+ ```bash
60
+ pip install transformers accelerate torch
61
+ ```
62
+
63
+ ### 2. Loading the Model
64
+ Use the following example to load one of the PLLuM models:
65
+ ```python
66
+ from transformers import AutoTokenizer, AutoModelForCausalLM
67
+
68
+ model_name = "CYFRAGOVPL/PLLuM-12B-chat-250715" # Replace with the PLLuM model name of your choice
69
+ tokenizer = AutoTokenizer.from_pretrained(model_name)
70
+ model = AutoModelForCausalLM.from_pretrained(model_name)
71
+ ```
72
+
73
+ ### 3. Using bfloat16 (BF16)
74
+ If your hardware (e.g., newer GPUs) supports bfloat16, you can reduce memory usage and potentially speed up inference:
75
+
76
+ ```python
77
+ import torch
78
+ from transformers import AutoTokenizer, AutoModelForCausalLM
79
+
80
+ model_name = "CYFRAGOVPL/PLLuM-12B-chat-250715"
81
+ tokenizer = AutoTokenizer.from_pretrained(model_name)
82
+
83
+ # Load model in bfloat16 precision
84
+ model = AutoModelForCausalLM.from_pretrained(
85
+ model_name,
86
+ torch_dtype=torch.bfloat16,
87
+ device_map="auto" # automatically places model layers on available devices
88
+ )
89
+ ```
90
+
91
+ ### 4. Generating an Example Text
92
+ ```python
93
+
94
+ prompt = "Napisz krótki wiersz o wiośnie." # EN:"Write a short poem about spring."
95
+ inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
96
+
97
+ outputs = model.generate(
98
+ **inputs,
99
+ max_new_tokens=50,
100
+ do_sample=True,
101
+ top_k=50,
102
+ top_p=0.9,
103
+ temperature=0.3
104
+ )
105
+
106
+ generated_text = tokenizer.decode(outputs[0], skip_special_tokens=True)
107
+ print(generated_text)
108
+ ```
109
+
110
+ ### 5. Expected Output
111
+ Below is a sample (hypothetical) output for the prompt above:
112
+
113
+ ```css
114
+ Przykładowy wiersz o tematyce wiosennej:
115
+
116
+ Wiosna, wiosna, wiosna, ach to ty!
117
+ Kwiecień plecień wciąż przeplata,
118
+ trochę zimy, trochę lata.
119
+ A ja nie mogę się już doczekać,
120
+ kiedy w kalendarzu ujrzę maj.
121
+ Wtedy wszystko wkoło rozkwita,
122
+ a ptaki tak pięknie śpiewają.
123
+ Wiosno, wiosno, czekam z utęsknieniem,
124
+ zrób mi tę przyjemność i przyjdź wreszcie, proszę!
125
+ ```
126
+ Your results may vary depending on model parameters (e.g., temperature, top_k, top_p), hardware, and other settings.
127
+
128
+ ### 6. Retrieval Augmented Generation (RAG)
129
+ PLLuM-12B-nc-instruct-250715 and PLLuM-12B-nc-chat-250715 were additionally trained to perform well in Retrieval Augmented Generation (RAG) setting. The prompt is in .jinja format, where docs is a list of document texts and question is a query that should be answered based on the provided documents. If there is no answer in the provided documents model generates "Nie udało mi się odnaleźć odpowiedzi na pytanie".
130
+
131
+ Prompt:
132
+
133
+ ```
134
+ Numerowana lista dokumentów jest poniżej:
135
+ ---------------------
136
+ <results>{% for doc in docs %}
137
+ Dokument: {{ loop.index0 }}
138
+ {{ doc }}
139
+ {% endfor %}</results>
140
+ ---------------------
141
+ Odpowiedz na pytanie użytkownika wykorzystując tylko informacje znajdujące się w dokumentach, a nie wcześniejszą wiedzę.
142
+ Udziel wysokiej jakości, poprawnej gramatycznie odpowiedzi w języku polskim. Odpowiedź powinna zawierać cytowania do dokumentów, z których pochodzą informacje. Zacytuj dokument za pomocą symbolu [nr_dokumentu] powołując się na fragment np. [0] dla fragmentu z dokumentu 0. Jeżeli w dokumentach nie ma informacji potrzebnych do odpowiedzi na pytanie, zamiast odpowiedzi zwróć tekst: "Nie udało mi się odnaleźć odpowiedzi na pytanie".
143
+
144
+ Pytanie: {{ question }}
145
+ ```
146
+
147
+ ## Training Procedure
148
+ - **Datasets**: ~150B tokens from Polish and multilingual sources, with ~28B tokens available for fully open-source commercial use.
149
+ - **Hyperparameters**: Vary based on model size, typically including Adam or AdamW optimizers, a range of batch sizes, and carefully tuned learning rates.
150
+ - **Hardware & Duration**: Training using [Bem2](https://man.e-science.pl/pl/kdm/bem2) (up to 300xH100 GPUs) and [Helios](https://www.cyfronet.pl/19950,artykul,superkomputer_helios.html) HPCs. The training time for each PLLuM model depends on its parameter size and hardware configuration, ranging from approximately 8 to 25 days on a multi-GPU cluster for models between 8B and 70B.
151
+
152
+ ## Evaluation and Benchmarks
153
+ - **Public Administration**: PLLuM models demonstrated top-tier performance in specialized tasks relevant to government services.
154
+ - **Polish Language Tasks**: Across a variety of internal benchmarks and standard corpora, PLLuM consistently outperforms other models in accuracy, coherence, and safety metrics.
155
+ - **Custom Tests**: A unique preference corpus and alignment tests ensure robust, safe, and contextually accurate responses.
156
+
157
+ ## Limitations and Bias
158
+ - **Potential Hallucinations**: Like other LLMs, PLLuM may occasionally produce factually incorrect or fabricated content.
159
+ - **Sensitivity & Bias**: While extensive preference learning has been done, biases might still emerge, especially in controversial or subjective topics.
160
+ - **Context Length**: Very long context tasks may challenge certain models, depending on memory constraints.
161
+
162
+ ## Ethical Considerations
163
+ PLLuM models are designed for constructive and responsible usage. Users should exercise caution when deploying them in production scenarios, especially for sensitive or regulated domains. Despite efforts to minimize harmful outputs, there is always a risk of generating offensive, biased, or inappropriate text. Human oversight and due diligence are advised.
164
+
165
+ ## Citation
166
+ If you use PLLuM models or any part of this repository in your research or deployment, please cite as follows (BibTeX):
167
+ ```
168
+ @unpublished{pllum2025,
169
+ title={PLLuM: A Family of Polish Large Language Models},
170
+ author={HIVE AI Consortium},
171
+ year={2025}
172
+ }
173
+ ```
174
+
175
+ ## License
176
+ PLLuM 12B-nc-base-250715 model is published under [PLLuM-1.2](https://huggingface.co/CYFRAGOVPL/pllum-12b-nc-base-250715/blob/main/LICENSE) license.
177
+
178
+ ## Creators & Consortium
179
+
180
+ The PLLuM-12B-nc-250715 models are the first to be trained under the banner of HIVE AI – a new initiative uniting scientific institutions and digital service providers working together to develop and implement open language technologies for the Polish public sector. HIVE AI continues the mission of the original PLLuM project, expanding it into a coordinated effort to build, improve, and deploy Polish language models in real-world administrative applications.
181
+
182
+ <table style="border: none; border-collapse: collapse;">
183
+ <tr>
184
+ </td>
185
+ <td align="center" valign="middle" style="border: none;">
186
+ <a href="https://www.nask.pl/">
187
+ <img src="https://pllum.org.pl/_nuxt/nask.Bz8rmSzR.png" alt="nask.Bz8rmSzR.png" width="100">
188
+ </a>
189
+ <br><strong>NASK PIB</strong><br><em>– Project Leader</em>
190
+ <td align="center" valign="middle" style="border: none;">
191
+ <a href="https://pwr.edu.pl/">
192
+ <img src="https://pllum.org.pl/_nuxt/pwr.D1_x0B58.png" alt="pwr.D1_x0B58.png" width="100">
193
+ </a>
194
+ <br><strong>Politechnika Wrocławska</strong>
195
+ </td>
196
+ <td align="center" valign="middle" style="border: none;">
197
+ <a href="https://www.ipipan.waw.pl/">
198
+ <img src="https://clarin.biz/_nuxt/img/ipipan.294d39c.png" alt="ipipan.294d39c.png" width="100">
199
+ </a>
200
+ <br><strong>Instytut Podstaw Informatyki PAN</strong>
201
+ </td>
202
+ <td align="center" valign="middle" style="border: none;">
203
+ <a href="https://ispan.waw.pl/default/">
204
+ <img src="https://pllum.org.pl/_nuxt/is.Dqb94VRb.png" alt="is.Dqb94VRb.png" width="100">
205
+ </a>
206
+ <br><strong>Instytut Slawistyki PAN</strong>
207
+ </td>
208
+ </tr>
209
+ <tr>
210
+ <td align="center" valign="middle" style="border: none;">
211
+ <a href="https://opi.org.pl/">
212
+ <img src="https://pllum.org.pl/_nuxt/opi.CF-COwcC.png" alt="opi.CF-COwcC.png" width="100">
213
+ </a>
214
+ <br><strong>Ośrodek Przetwarzania Informacji PIB</strong>
215
+ <td align="center" valign="middle" style="border: none;">
216
+ <a href="https://www.uni.lodz.pl/">
217
+ <img src="https://pllum.org.pl/_nuxt/ul.aTSgr_W6.png" alt="ul.aTSgr_W6.png" width="100">
218
+ </a>
219
+ <br><strong>Uniwersytet Łódzki</strong>
220
+ </td>
221
+ <td align="center" valign="middle" style="border: none;">
222
+ <a href="https://www.cyfronet.pl/">
223
+ <img src="https://historia.agh.edu.pl/mediawiki/images/0/08/Cyfronet_logo.png" width="100">
224
+ </a>
225
+ <br><strong>Cyfronet AGH</strong>
226
+ </td>
227
+ </td>
228
+ </tr>
229
+ </table>
230
+
231
+
232
+ ## Contact and Support
233
+ For questions or contributions, please reach out via: <[email protected]>
234
+
235
+ We welcome feedback, collaboration, and further exploration of PLLuM models!
236
+
237
+
238
+ ## Acknowledgements
239
+
240
+ Project financed by the Minister of Digital Affairs under the targeted subsidy No. 1/WII/DBI/2025: *“HIVE AI: Development and Pilot Implementation of LLMs in the Polish Public Administration”*
241
+
242
+ **Funding Amount:** approx. 19 mln PLN
243
+ **Contract Signing Date:** 2025-03-25
config.json ADDED
@@ -0,0 +1,26 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "architectures": [
3
+ "MistralForCausalLM"
4
+ ],
5
+ "attention_dropout": 0.0,
6
+ "bos_token_id": 1,
7
+ "eos_token_id": 2,
8
+ "head_dim": 128,
9
+ "hidden_act": "silu",
10
+ "hidden_size": 5120,
11
+ "initializer_range": 0.02,
12
+ "intermediate_size": 14336,
13
+ "max_position_embeddings": 131072,
14
+ "model_type": "mistral",
15
+ "num_attention_heads": 32,
16
+ "num_hidden_layers": 40,
17
+ "num_key_value_heads": 8,
18
+ "rms_norm_eps": 1e-05,
19
+ "rope_theta": 1000000.0,
20
+ "sliding_window": null,
21
+ "tie_word_embeddings": false,
22
+ "torch_dtype": "bfloat16",
23
+ "transformers_version": "4.53.1",
24
+ "use_cache": false,
25
+ "vocab_size": 131072
26
+ }
generation_config.json ADDED
@@ -0,0 +1,6 @@
 
 
 
 
 
 
 
1
+ {
2
+ "_from_model_config": true,
3
+ "bos_token_id": 1,
4
+ "eos_token_id": 2,
5
+ "transformers_version": "4.53.1"
6
+ }
model-00001-of-00005.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:9e8091fa9b2e995b161add226aaccff3d8aefea18eee25b3daafc4097f119d1f
3
+ size 4865522496
model-00002-of-00005.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:dd965e0276c88cbe688920398b18c55ac0cd3af0566c0cbed9660b66670f41c0
3
+ size 4907529424
model-00003-of-00005.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:2863a3018575a7adc295d333bde797e72a9acf692aa933aac5db00c9fd13ef1d
3
+ size 4907529456
model-00004-of-00005.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:e313989ee7b816deab59c2d4693e396135c803c592bcc27abd6745b450ede2e1
3
+ size 4907529456
model-00005-of-00005.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:02089843acbbdfa463f981cac243c48d7b0ced815233ae88aa4d0b531bcf4ace
3
+ size 4907496272
model.safetensors.index.json ADDED
@@ -0,0 +1,371 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "metadata": {
3
+ "total_parameters": 12247782400,
4
+ "total_size": 24495564800
5
+ },
6
+ "weight_map": {
7
+ "lm_head.weight": "model-00005-of-00005.safetensors",
8
+ "model.embed_tokens.weight": "model-00001-of-00005.safetensors",
9
+ "model.layers.0.input_layernorm.weight": "model-00001-of-00005.safetensors",
10
+ "model.layers.0.mlp.down_proj.weight": "model-00001-of-00005.safetensors",
11
+ "model.layers.0.mlp.gate_proj.weight": "model-00001-of-00005.safetensors",
12
+ "model.layers.0.mlp.up_proj.weight": "model-00001-of-00005.safetensors",
13
+ "model.layers.0.post_attention_layernorm.weight": "model-00001-of-00005.safetensors",
14
+ "model.layers.0.self_attn.k_proj.weight": "model-00001-of-00005.safetensors",
15
+ "model.layers.0.self_attn.o_proj.weight": "model-00001-of-00005.safetensors",
16
+ "model.layers.0.self_attn.q_proj.weight": "model-00001-of-00005.safetensors",
17
+ "model.layers.0.self_attn.v_proj.weight": "model-00001-of-00005.safetensors",
18
+ "model.layers.1.input_layernorm.weight": "model-00001-of-00005.safetensors",
19
+ "model.layers.1.mlp.down_proj.weight": "model-00001-of-00005.safetensors",
20
+ "model.layers.1.mlp.gate_proj.weight": "model-00001-of-00005.safetensors",
21
+ "model.layers.1.mlp.up_proj.weight": "model-00001-of-00005.safetensors",
22
+ "model.layers.1.post_attention_layernorm.weight": "model-00001-of-00005.safetensors",
23
+ "model.layers.1.self_attn.k_proj.weight": "model-00001-of-00005.safetensors",
24
+ "model.layers.1.self_attn.o_proj.weight": "model-00001-of-00005.safetensors",
25
+ "model.layers.1.self_attn.q_proj.weight": "model-00001-of-00005.safetensors",
26
+ "model.layers.1.self_attn.v_proj.weight": "model-00001-of-00005.safetensors",
27
+ "model.layers.10.input_layernorm.weight": "model-00002-of-00005.safetensors",
28
+ "model.layers.10.mlp.down_proj.weight": "model-00002-of-00005.safetensors",
29
+ "model.layers.10.mlp.gate_proj.weight": "model-00002-of-00005.safetensors",
30
+ "model.layers.10.mlp.up_proj.weight": "model-00002-of-00005.safetensors",
31
+ "model.layers.10.post_attention_layernorm.weight": "model-00002-of-00005.safetensors",
32
+ "model.layers.10.self_attn.k_proj.weight": "model-00002-of-00005.safetensors",
33
+ "model.layers.10.self_attn.o_proj.weight": "model-00002-of-00005.safetensors",
34
+ "model.layers.10.self_attn.q_proj.weight": "model-00002-of-00005.safetensors",
35
+ "model.layers.10.self_attn.v_proj.weight": "model-00002-of-00005.safetensors",
36
+ "model.layers.11.input_layernorm.weight": "model-00002-of-00005.safetensors",
37
+ "model.layers.11.mlp.down_proj.weight": "model-00002-of-00005.safetensors",
38
+ "model.layers.11.mlp.gate_proj.weight": "model-00002-of-00005.safetensors",
39
+ "model.layers.11.mlp.up_proj.weight": "model-00002-of-00005.safetensors",
40
+ "model.layers.11.post_attention_layernorm.weight": "model-00002-of-00005.safetensors",
41
+ "model.layers.11.self_attn.k_proj.weight": "model-00002-of-00005.safetensors",
42
+ "model.layers.11.self_attn.o_proj.weight": "model-00002-of-00005.safetensors",
43
+ "model.layers.11.self_attn.q_proj.weight": "model-00002-of-00005.safetensors",
44
+ "model.layers.11.self_attn.v_proj.weight": "model-00002-of-00005.safetensors",
45
+ "model.layers.12.input_layernorm.weight": "model-00002-of-00005.safetensors",
46
+ "model.layers.12.mlp.down_proj.weight": "model-00002-of-00005.safetensors",
47
+ "model.layers.12.mlp.gate_proj.weight": "model-00002-of-00005.safetensors",
48
+ "model.layers.12.mlp.up_proj.weight": "model-00002-of-00005.safetensors",
49
+ "model.layers.12.post_attention_layernorm.weight": "model-00002-of-00005.safetensors",
50
+ "model.layers.12.self_attn.k_proj.weight": "model-00002-of-00005.safetensors",
51
+ "model.layers.12.self_attn.o_proj.weight": "model-00002-of-00005.safetensors",
52
+ "model.layers.12.self_attn.q_proj.weight": "model-00002-of-00005.safetensors",
53
+ "model.layers.12.self_attn.v_proj.weight": "model-00002-of-00005.safetensors",
54
+ "model.layers.13.input_layernorm.weight": "model-00002-of-00005.safetensors",
55
+ "model.layers.13.mlp.down_proj.weight": "model-00002-of-00005.safetensors",
56
+ "model.layers.13.mlp.gate_proj.weight": "model-00002-of-00005.safetensors",
57
+ "model.layers.13.mlp.up_proj.weight": "model-00002-of-00005.safetensors",
58
+ "model.layers.13.post_attention_layernorm.weight": "model-00002-of-00005.safetensors",
59
+ "model.layers.13.self_attn.k_proj.weight": "model-00002-of-00005.safetensors",
60
+ "model.layers.13.self_attn.o_proj.weight": "model-00002-of-00005.safetensors",
61
+ "model.layers.13.self_attn.q_proj.weight": "model-00002-of-00005.safetensors",
62
+ "model.layers.13.self_attn.v_proj.weight": "model-00002-of-00005.safetensors",
63
+ "model.layers.14.input_layernorm.weight": "model-00002-of-00005.safetensors",
64
+ "model.layers.14.mlp.down_proj.weight": "model-00002-of-00005.safetensors",
65
+ "model.layers.14.mlp.gate_proj.weight": "model-00002-of-00005.safetensors",
66
+ "model.layers.14.mlp.up_proj.weight": "model-00002-of-00005.safetensors",
67
+ "model.layers.14.post_attention_layernorm.weight": "model-00002-of-00005.safetensors",
68
+ "model.layers.14.self_attn.k_proj.weight": "model-00002-of-00005.safetensors",
69
+ "model.layers.14.self_attn.o_proj.weight": "model-00002-of-00005.safetensors",
70
+ "model.layers.14.self_attn.q_proj.weight": "model-00002-of-00005.safetensors",
71
+ "model.layers.14.self_attn.v_proj.weight": "model-00002-of-00005.safetensors",
72
+ "model.layers.15.input_layernorm.weight": "model-00003-of-00005.safetensors",
73
+ "model.layers.15.mlp.down_proj.weight": "model-00003-of-00005.safetensors",
74
+ "model.layers.15.mlp.gate_proj.weight": "model-00002-of-00005.safetensors",
75
+ "model.layers.15.mlp.up_proj.weight": "model-00003-of-00005.safetensors",
76
+ "model.layers.15.post_attention_layernorm.weight": "model-00003-of-00005.safetensors",
77
+ "model.layers.15.self_attn.k_proj.weight": "model-00002-of-00005.safetensors",
78
+ "model.layers.15.self_attn.o_proj.weight": "model-00002-of-00005.safetensors",
79
+ "model.layers.15.self_attn.q_proj.weight": "model-00002-of-00005.safetensors",
80
+ "model.layers.15.self_attn.v_proj.weight": "model-00002-of-00005.safetensors",
81
+ "model.layers.16.input_layernorm.weight": "model-00003-of-00005.safetensors",
82
+ "model.layers.16.mlp.down_proj.weight": "model-00003-of-00005.safetensors",
83
+ "model.layers.16.mlp.gate_proj.weight": "model-00003-of-00005.safetensors",
84
+ "model.layers.16.mlp.up_proj.weight": "model-00003-of-00005.safetensors",
85
+ "model.layers.16.post_attention_layernorm.weight": "model-00003-of-00005.safetensors",
86
+ "model.layers.16.self_attn.k_proj.weight": "model-00003-of-00005.safetensors",
87
+ "model.layers.16.self_attn.o_proj.weight": "model-00003-of-00005.safetensors",
88
+ "model.layers.16.self_attn.q_proj.weight": "model-00003-of-00005.safetensors",
89
+ "model.layers.16.self_attn.v_proj.weight": "model-00003-of-00005.safetensors",
90
+ "model.layers.17.input_layernorm.weight": "model-00003-of-00005.safetensors",
91
+ "model.layers.17.mlp.down_proj.weight": "model-00003-of-00005.safetensors",
92
+ "model.layers.17.mlp.gate_proj.weight": "model-00003-of-00005.safetensors",
93
+ "model.layers.17.mlp.up_proj.weight": "model-00003-of-00005.safetensors",
94
+ "model.layers.17.post_attention_layernorm.weight": "model-00003-of-00005.safetensors",
95
+ "model.layers.17.self_attn.k_proj.weight": "model-00003-of-00005.safetensors",
96
+ "model.layers.17.self_attn.o_proj.weight": "model-00003-of-00005.safetensors",
97
+ "model.layers.17.self_attn.q_proj.weight": "model-00003-of-00005.safetensors",
98
+ "model.layers.17.self_attn.v_proj.weight": "model-00003-of-00005.safetensors",
99
+ "model.layers.18.input_layernorm.weight": "model-00003-of-00005.safetensors",
100
+ "model.layers.18.mlp.down_proj.weight": "model-00003-of-00005.safetensors",
101
+ "model.layers.18.mlp.gate_proj.weight": "model-00003-of-00005.safetensors",
102
+ "model.layers.18.mlp.up_proj.weight": "model-00003-of-00005.safetensors",
103
+ "model.layers.18.post_attention_layernorm.weight": "model-00003-of-00005.safetensors",
104
+ "model.layers.18.self_attn.k_proj.weight": "model-00003-of-00005.safetensors",
105
+ "model.layers.18.self_attn.o_proj.weight": "model-00003-of-00005.safetensors",
106
+ "model.layers.18.self_attn.q_proj.weight": "model-00003-of-00005.safetensors",
107
+ "model.layers.18.self_attn.v_proj.weight": "model-00003-of-00005.safetensors",
108
+ "model.layers.19.input_layernorm.weight": "model-00003-of-00005.safetensors",
109
+ "model.layers.19.mlp.down_proj.weight": "model-00003-of-00005.safetensors",
110
+ "model.layers.19.mlp.gate_proj.weight": "model-00003-of-00005.safetensors",
111
+ "model.layers.19.mlp.up_proj.weight": "model-00003-of-00005.safetensors",
112
+ "model.layers.19.post_attention_layernorm.weight": "model-00003-of-00005.safetensors",
113
+ "model.layers.19.self_attn.k_proj.weight": "model-00003-of-00005.safetensors",
114
+ "model.layers.19.self_attn.o_proj.weight": "model-00003-of-00005.safetensors",
115
+ "model.layers.19.self_attn.q_proj.weight": "model-00003-of-00005.safetensors",
116
+ "model.layers.19.self_attn.v_proj.weight": "model-00003-of-00005.safetensors",
117
+ "model.layers.2.input_layernorm.weight": "model-00001-of-00005.safetensors",
118
+ "model.layers.2.mlp.down_proj.weight": "model-00001-of-00005.safetensors",
119
+ "model.layers.2.mlp.gate_proj.weight": "model-00001-of-00005.safetensors",
120
+ "model.layers.2.mlp.up_proj.weight": "model-00001-of-00005.safetensors",
121
+ "model.layers.2.post_attention_layernorm.weight": "model-00001-of-00005.safetensors",
122
+ "model.layers.2.self_attn.k_proj.weight": "model-00001-of-00005.safetensors",
123
+ "model.layers.2.self_attn.o_proj.weight": "model-00001-of-00005.safetensors",
124
+ "model.layers.2.self_attn.q_proj.weight": "model-00001-of-00005.safetensors",
125
+ "model.layers.2.self_attn.v_proj.weight": "model-00001-of-00005.safetensors",
126
+ "model.layers.20.input_layernorm.weight": "model-00003-of-00005.safetensors",
127
+ "model.layers.20.mlp.down_proj.weight": "model-00003-of-00005.safetensors",
128
+ "model.layers.20.mlp.gate_proj.weight": "model-00003-of-00005.safetensors",
129
+ "model.layers.20.mlp.up_proj.weight": "model-00003-of-00005.safetensors",
130
+ "model.layers.20.post_attention_layernorm.weight": "model-00003-of-00005.safetensors",
131
+ "model.layers.20.self_attn.k_proj.weight": "model-00003-of-00005.safetensors",
132
+ "model.layers.20.self_attn.o_proj.weight": "model-00003-of-00005.safetensors",
133
+ "model.layers.20.self_attn.q_proj.weight": "model-00003-of-00005.safetensors",
134
+ "model.layers.20.self_attn.v_proj.weight": "model-00003-of-00005.safetensors",
135
+ "model.layers.21.input_layernorm.weight": "model-00003-of-00005.safetensors",
136
+ "model.layers.21.mlp.down_proj.weight": "model-00003-of-00005.safetensors",
137
+ "model.layers.21.mlp.gate_proj.weight": "model-00003-of-00005.safetensors",
138
+ "model.layers.21.mlp.up_proj.weight": "model-00003-of-00005.safetensors",
139
+ "model.layers.21.post_attention_layernorm.weight": "model-00003-of-00005.safetensors",
140
+ "model.layers.21.self_attn.k_proj.weight": "model-00003-of-00005.safetensors",
141
+ "model.layers.21.self_attn.o_proj.weight": "model-00003-of-00005.safetensors",
142
+ "model.layers.21.self_attn.q_proj.weight": "model-00003-of-00005.safetensors",
143
+ "model.layers.21.self_attn.v_proj.weight": "model-00003-of-00005.safetensors",
144
+ "model.layers.22.input_layernorm.weight": "model-00003-of-00005.safetensors",
145
+ "model.layers.22.mlp.down_proj.weight": "model-00003-of-00005.safetensors",
146
+ "model.layers.22.mlp.gate_proj.weight": "model-00003-of-00005.safetensors",
147
+ "model.layers.22.mlp.up_proj.weight": "model-00003-of-00005.safetensors",
148
+ "model.layers.22.post_attention_layernorm.weight": "model-00003-of-00005.safetensors",
149
+ "model.layers.22.self_attn.k_proj.weight": "model-00003-of-00005.safetensors",
150
+ "model.layers.22.self_attn.o_proj.weight": "model-00003-of-00005.safetensors",
151
+ "model.layers.22.self_attn.q_proj.weight": "model-00003-of-00005.safetensors",
152
+ "model.layers.22.self_attn.v_proj.weight": "model-00003-of-00005.safetensors",
153
+ "model.layers.23.input_layernorm.weight": "model-00003-of-00005.safetensors",
154
+ "model.layers.23.mlp.down_proj.weight": "model-00003-of-00005.safetensors",
155
+ "model.layers.23.mlp.gate_proj.weight": "model-00003-of-00005.safetensors",
156
+ "model.layers.23.mlp.up_proj.weight": "model-00003-of-00005.safetensors",
157
+ "model.layers.23.post_attention_layernorm.weight": "model-00003-of-00005.safetensors",
158
+ "model.layers.23.self_attn.k_proj.weight": "model-00003-of-00005.safetensors",
159
+ "model.layers.23.self_attn.o_proj.weight": "model-00003-of-00005.safetensors",
160
+ "model.layers.23.self_attn.q_proj.weight": "model-00003-of-00005.safetensors",
161
+ "model.layers.23.self_attn.v_proj.weight": "model-00003-of-00005.safetensors",
162
+ "model.layers.24.input_layernorm.weight": "model-00004-of-00005.safetensors",
163
+ "model.layers.24.mlp.down_proj.weight": "model-00004-of-00005.safetensors",
164
+ "model.layers.24.mlp.gate_proj.weight": "model-00003-of-00005.safetensors",
165
+ "model.layers.24.mlp.up_proj.weight": "model-00004-of-00005.safetensors",
166
+ "model.layers.24.post_attention_layernorm.weight": "model-00004-of-00005.safetensors",
167
+ "model.layers.24.self_attn.k_proj.weight": "model-00003-of-00005.safetensors",
168
+ "model.layers.24.self_attn.o_proj.weight": "model-00003-of-00005.safetensors",
169
+ "model.layers.24.self_attn.q_proj.weight": "model-00003-of-00005.safetensors",
170
+ "model.layers.24.self_attn.v_proj.weight": "model-00003-of-00005.safetensors",
171
+ "model.layers.25.input_layernorm.weight": "model-00004-of-00005.safetensors",
172
+ "model.layers.25.mlp.down_proj.weight": "model-00004-of-00005.safetensors",
173
+ "model.layers.25.mlp.gate_proj.weight": "model-00004-of-00005.safetensors",
174
+ "model.layers.25.mlp.up_proj.weight": "model-00004-of-00005.safetensors",
175
+ "model.layers.25.post_attention_layernorm.weight": "model-00004-of-00005.safetensors",
176
+ "model.layers.25.self_attn.k_proj.weight": "model-00004-of-00005.safetensors",
177
+ "model.layers.25.self_attn.o_proj.weight": "model-00004-of-00005.safetensors",
178
+ "model.layers.25.self_attn.q_proj.weight": "model-00004-of-00005.safetensors",
179
+ "model.layers.25.self_attn.v_proj.weight": "model-00004-of-00005.safetensors",
180
+ "model.layers.26.input_layernorm.weight": "model-00004-of-00005.safetensors",
181
+ "model.layers.26.mlp.down_proj.weight": "model-00004-of-00005.safetensors",
182
+ "model.layers.26.mlp.gate_proj.weight": "model-00004-of-00005.safetensors",
183
+ "model.layers.26.mlp.up_proj.weight": "model-00004-of-00005.safetensors",
184
+ "model.layers.26.post_attention_layernorm.weight": "model-00004-of-00005.safetensors",
185
+ "model.layers.26.self_attn.k_proj.weight": "model-00004-of-00005.safetensors",
186
+ "model.layers.26.self_attn.o_proj.weight": "model-00004-of-00005.safetensors",
187
+ "model.layers.26.self_attn.q_proj.weight": "model-00004-of-00005.safetensors",
188
+ "model.layers.26.self_attn.v_proj.weight": "model-00004-of-00005.safetensors",
189
+ "model.layers.27.input_layernorm.weight": "model-00004-of-00005.safetensors",
190
+ "model.layers.27.mlp.down_proj.weight": "model-00004-of-00005.safetensors",
191
+ "model.layers.27.mlp.gate_proj.weight": "model-00004-of-00005.safetensors",
192
+ "model.layers.27.mlp.up_proj.weight": "model-00004-of-00005.safetensors",
193
+ "model.layers.27.post_attention_layernorm.weight": "model-00004-of-00005.safetensors",
194
+ "model.layers.27.self_attn.k_proj.weight": "model-00004-of-00005.safetensors",
195
+ "model.layers.27.self_attn.o_proj.weight": "model-00004-of-00005.safetensors",
196
+ "model.layers.27.self_attn.q_proj.weight": "model-00004-of-00005.safetensors",
197
+ "model.layers.27.self_attn.v_proj.weight": "model-00004-of-00005.safetensors",
198
+ "model.layers.28.input_layernorm.weight": "model-00004-of-00005.safetensors",
199
+ "model.layers.28.mlp.down_proj.weight": "model-00004-of-00005.safetensors",
200
+ "model.layers.28.mlp.gate_proj.weight": "model-00004-of-00005.safetensors",
201
+ "model.layers.28.mlp.up_proj.weight": "model-00004-of-00005.safetensors",
202
+ "model.layers.28.post_attention_layernorm.weight": "model-00004-of-00005.safetensors",
203
+ "model.layers.28.self_attn.k_proj.weight": "model-00004-of-00005.safetensors",
204
+ "model.layers.28.self_attn.o_proj.weight": "model-00004-of-00005.safetensors",
205
+ "model.layers.28.self_attn.q_proj.weight": "model-00004-of-00005.safetensors",
206
+ "model.layers.28.self_attn.v_proj.weight": "model-00004-of-00005.safetensors",
207
+ "model.layers.29.input_layernorm.weight": "model-00004-of-00005.safetensors",
208
+ "model.layers.29.mlp.down_proj.weight": "model-00004-of-00005.safetensors",
209
+ "model.layers.29.mlp.gate_proj.weight": "model-00004-of-00005.safetensors",
210
+ "model.layers.29.mlp.up_proj.weight": "model-00004-of-00005.safetensors",
211
+ "model.layers.29.post_attention_layernorm.weight": "model-00004-of-00005.safetensors",
212
+ "model.layers.29.self_attn.k_proj.weight": "model-00004-of-00005.safetensors",
213
+ "model.layers.29.self_attn.o_proj.weight": "model-00004-of-00005.safetensors",
214
+ "model.layers.29.self_attn.q_proj.weight": "model-00004-of-00005.safetensors",
215
+ "model.layers.29.self_attn.v_proj.weight": "model-00004-of-00005.safetensors",
216
+ "model.layers.3.input_layernorm.weight": "model-00001-of-00005.safetensors",
217
+ "model.layers.3.mlp.down_proj.weight": "model-00001-of-00005.safetensors",
218
+ "model.layers.3.mlp.gate_proj.weight": "model-00001-of-00005.safetensors",
219
+ "model.layers.3.mlp.up_proj.weight": "model-00001-of-00005.safetensors",
220
+ "model.layers.3.post_attention_layernorm.weight": "model-00001-of-00005.safetensors",
221
+ "model.layers.3.self_attn.k_proj.weight": "model-00001-of-00005.safetensors",
222
+ "model.layers.3.self_attn.o_proj.weight": "model-00001-of-00005.safetensors",
223
+ "model.layers.3.self_attn.q_proj.weight": "model-00001-of-00005.safetensors",
224
+ "model.layers.3.self_attn.v_proj.weight": "model-00001-of-00005.safetensors",
225
+ "model.layers.30.input_layernorm.weight": "model-00004-of-00005.safetensors",
226
+ "model.layers.30.mlp.down_proj.weight": "model-00004-of-00005.safetensors",
227
+ "model.layers.30.mlp.gate_proj.weight": "model-00004-of-00005.safetensors",
228
+ "model.layers.30.mlp.up_proj.weight": "model-00004-of-00005.safetensors",
229
+ "model.layers.30.post_attention_layernorm.weight": "model-00004-of-00005.safetensors",
230
+ "model.layers.30.self_attn.k_proj.weight": "model-00004-of-00005.safetensors",
231
+ "model.layers.30.self_attn.o_proj.weight": "model-00004-of-00005.safetensors",
232
+ "model.layers.30.self_attn.q_proj.weight": "model-00004-of-00005.safetensors",
233
+ "model.layers.30.self_attn.v_proj.weight": "model-00004-of-00005.safetensors",
234
+ "model.layers.31.input_layernorm.weight": "model-00004-of-00005.safetensors",
235
+ "model.layers.31.mlp.down_proj.weight": "model-00004-of-00005.safetensors",
236
+ "model.layers.31.mlp.gate_proj.weight": "model-00004-of-00005.safetensors",
237
+ "model.layers.31.mlp.up_proj.weight": "model-00004-of-00005.safetensors",
238
+ "model.layers.31.post_attention_layernorm.weight": "model-00004-of-00005.safetensors",
239
+ "model.layers.31.self_attn.k_proj.weight": "model-00004-of-00005.safetensors",
240
+ "model.layers.31.self_attn.o_proj.weight": "model-00004-of-00005.safetensors",
241
+ "model.layers.31.self_attn.q_proj.weight": "model-00004-of-00005.safetensors",
242
+ "model.layers.31.self_attn.v_proj.weight": "model-00004-of-00005.safetensors",
243
+ "model.layers.32.input_layernorm.weight": "model-00004-of-00005.safetensors",
244
+ "model.layers.32.mlp.down_proj.weight": "model-00004-of-00005.safetensors",
245
+ "model.layers.32.mlp.gate_proj.weight": "model-00004-of-00005.safetensors",
246
+ "model.layers.32.mlp.up_proj.weight": "model-00004-of-00005.safetensors",
247
+ "model.layers.32.post_attention_layernorm.weight": "model-00004-of-00005.safetensors",
248
+ "model.layers.32.self_attn.k_proj.weight": "model-00004-of-00005.safetensors",
249
+ "model.layers.32.self_attn.o_proj.weight": "model-00004-of-00005.safetensors",
250
+ "model.layers.32.self_attn.q_proj.weight": "model-00004-of-00005.safetensors",
251
+ "model.layers.32.self_attn.v_proj.weight": "model-00004-of-00005.safetensors",
252
+ "model.layers.33.input_layernorm.weight": "model-00005-of-00005.safetensors",
253
+ "model.layers.33.mlp.down_proj.weight": "model-00005-of-00005.safetensors",
254
+ "model.layers.33.mlp.gate_proj.weight": "model-00004-of-00005.safetensors",
255
+ "model.layers.33.mlp.up_proj.weight": "model-00005-of-00005.safetensors",
256
+ "model.layers.33.post_attention_layernorm.weight": "model-00005-of-00005.safetensors",
257
+ "model.layers.33.self_attn.k_proj.weight": "model-00004-of-00005.safetensors",
258
+ "model.layers.33.self_attn.o_proj.weight": "model-00004-of-00005.safetensors",
259
+ "model.layers.33.self_attn.q_proj.weight": "model-00004-of-00005.safetensors",
260
+ "model.layers.33.self_attn.v_proj.weight": "model-00004-of-00005.safetensors",
261
+ "model.layers.34.input_layernorm.weight": "model-00005-of-00005.safetensors",
262
+ "model.layers.34.mlp.down_proj.weight": "model-00005-of-00005.safetensors",
263
+ "model.layers.34.mlp.gate_proj.weight": "model-00005-of-00005.safetensors",
264
+ "model.layers.34.mlp.up_proj.weight": "model-00005-of-00005.safetensors",
265
+ "model.layers.34.post_attention_layernorm.weight": "model-00005-of-00005.safetensors",
266
+ "model.layers.34.self_attn.k_proj.weight": "model-00005-of-00005.safetensors",
267
+ "model.layers.34.self_attn.o_proj.weight": "model-00005-of-00005.safetensors",
268
+ "model.layers.34.self_attn.q_proj.weight": "model-00005-of-00005.safetensors",
269
+ "model.layers.34.self_attn.v_proj.weight": "model-00005-of-00005.safetensors",
270
+ "model.layers.35.input_layernorm.weight": "model-00005-of-00005.safetensors",
271
+ "model.layers.35.mlp.down_proj.weight": "model-00005-of-00005.safetensors",
272
+ "model.layers.35.mlp.gate_proj.weight": "model-00005-of-00005.safetensors",
273
+ "model.layers.35.mlp.up_proj.weight": "model-00005-of-00005.safetensors",
274
+ "model.layers.35.post_attention_layernorm.weight": "model-00005-of-00005.safetensors",
275
+ "model.layers.35.self_attn.k_proj.weight": "model-00005-of-00005.safetensors",
276
+ "model.layers.35.self_attn.o_proj.weight": "model-00005-of-00005.safetensors",
277
+ "model.layers.35.self_attn.q_proj.weight": "model-00005-of-00005.safetensors",
278
+ "model.layers.35.self_attn.v_proj.weight": "model-00005-of-00005.safetensors",
279
+ "model.layers.36.input_layernorm.weight": "model-00005-of-00005.safetensors",
280
+ "model.layers.36.mlp.down_proj.weight": "model-00005-of-00005.safetensors",
281
+ "model.layers.36.mlp.gate_proj.weight": "model-00005-of-00005.safetensors",
282
+ "model.layers.36.mlp.up_proj.weight": "model-00005-of-00005.safetensors",
283
+ "model.layers.36.post_attention_layernorm.weight": "model-00005-of-00005.safetensors",
284
+ "model.layers.36.self_attn.k_proj.weight": "model-00005-of-00005.safetensors",
285
+ "model.layers.36.self_attn.o_proj.weight": "model-00005-of-00005.safetensors",
286
+ "model.layers.36.self_attn.q_proj.weight": "model-00005-of-00005.safetensors",
287
+ "model.layers.36.self_attn.v_proj.weight": "model-00005-of-00005.safetensors",
288
+ "model.layers.37.input_layernorm.weight": "model-00005-of-00005.safetensors",
289
+ "model.layers.37.mlp.down_proj.weight": "model-00005-of-00005.safetensors",
290
+ "model.layers.37.mlp.gate_proj.weight": "model-00005-of-00005.safetensors",
291
+ "model.layers.37.mlp.up_proj.weight": "model-00005-of-00005.safetensors",
292
+ "model.layers.37.post_attention_layernorm.weight": "model-00005-of-00005.safetensors",
293
+ "model.layers.37.self_attn.k_proj.weight": "model-00005-of-00005.safetensors",
294
+ "model.layers.37.self_attn.o_proj.weight": "model-00005-of-00005.safetensors",
295
+ "model.layers.37.self_attn.q_proj.weight": "model-00005-of-00005.safetensors",
296
+ "model.layers.37.self_attn.v_proj.weight": "model-00005-of-00005.safetensors",
297
+ "model.layers.38.input_layernorm.weight": "model-00005-of-00005.safetensors",
298
+ "model.layers.38.mlp.down_proj.weight": "model-00005-of-00005.safetensors",
299
+ "model.layers.38.mlp.gate_proj.weight": "model-00005-of-00005.safetensors",
300
+ "model.layers.38.mlp.up_proj.weight": "model-00005-of-00005.safetensors",
301
+ "model.layers.38.post_attention_layernorm.weight": "model-00005-of-00005.safetensors",
302
+ "model.layers.38.self_attn.k_proj.weight": "model-00005-of-00005.safetensors",
303
+ "model.layers.38.self_attn.o_proj.weight": "model-00005-of-00005.safetensors",
304
+ "model.layers.38.self_attn.q_proj.weight": "model-00005-of-00005.safetensors",
305
+ "model.layers.38.self_attn.v_proj.weight": "model-00005-of-00005.safetensors",
306
+ "model.layers.39.input_layernorm.weight": "model-00005-of-00005.safetensors",
307
+ "model.layers.39.mlp.down_proj.weight": "model-00005-of-00005.safetensors",
308
+ "model.layers.39.mlp.gate_proj.weight": "model-00005-of-00005.safetensors",
309
+ "model.layers.39.mlp.up_proj.weight": "model-00005-of-00005.safetensors",
310
+ "model.layers.39.post_attention_layernorm.weight": "model-00005-of-00005.safetensors",
311
+ "model.layers.39.self_attn.k_proj.weight": "model-00005-of-00005.safetensors",
312
+ "model.layers.39.self_attn.o_proj.weight": "model-00005-of-00005.safetensors",
313
+ "model.layers.39.self_attn.q_proj.weight": "model-00005-of-00005.safetensors",
314
+ "model.layers.39.self_attn.v_proj.weight": "model-00005-of-00005.safetensors",
315
+ "model.layers.4.input_layernorm.weight": "model-00001-of-00005.safetensors",
316
+ "model.layers.4.mlp.down_proj.weight": "model-00001-of-00005.safetensors",
317
+ "model.layers.4.mlp.gate_proj.weight": "model-00001-of-00005.safetensors",
318
+ "model.layers.4.mlp.up_proj.weight": "model-00001-of-00005.safetensors",
319
+ "model.layers.4.post_attention_layernorm.weight": "model-00001-of-00005.safetensors",
320
+ "model.layers.4.self_attn.k_proj.weight": "model-00001-of-00005.safetensors",
321
+ "model.layers.4.self_attn.o_proj.weight": "model-00001-of-00005.safetensors",
322
+ "model.layers.4.self_attn.q_proj.weight": "model-00001-of-00005.safetensors",
323
+ "model.layers.4.self_attn.v_proj.weight": "model-00001-of-00005.safetensors",
324
+ "model.layers.5.input_layernorm.weight": "model-00001-of-00005.safetensors",
325
+ "model.layers.5.mlp.down_proj.weight": "model-00001-of-00005.safetensors",
326
+ "model.layers.5.mlp.gate_proj.weight": "model-00001-of-00005.safetensors",
327
+ "model.layers.5.mlp.up_proj.weight": "model-00001-of-00005.safetensors",
328
+ "model.layers.5.post_attention_layernorm.weight": "model-00001-of-00005.safetensors",
329
+ "model.layers.5.self_attn.k_proj.weight": "model-00001-of-00005.safetensors",
330
+ "model.layers.5.self_attn.o_proj.weight": "model-00001-of-00005.safetensors",
331
+ "model.layers.5.self_attn.q_proj.weight": "model-00001-of-00005.safetensors",
332
+ "model.layers.5.self_attn.v_proj.weight": "model-00001-of-00005.safetensors",
333
+ "model.layers.6.input_layernorm.weight": "model-00002-of-00005.safetensors",
334
+ "model.layers.6.mlp.down_proj.weight": "model-00002-of-00005.safetensors",
335
+ "model.layers.6.mlp.gate_proj.weight": "model-00001-of-00005.safetensors",
336
+ "model.layers.6.mlp.up_proj.weight": "model-00002-of-00005.safetensors",
337
+ "model.layers.6.post_attention_layernorm.weight": "model-00002-of-00005.safetensors",
338
+ "model.layers.6.self_attn.k_proj.weight": "model-00001-of-00005.safetensors",
339
+ "model.layers.6.self_attn.o_proj.weight": "model-00001-of-00005.safetensors",
340
+ "model.layers.6.self_attn.q_proj.weight": "model-00001-of-00005.safetensors",
341
+ "model.layers.6.self_attn.v_proj.weight": "model-00001-of-00005.safetensors",
342
+ "model.layers.7.input_layernorm.weight": "model-00002-of-00005.safetensors",
343
+ "model.layers.7.mlp.down_proj.weight": "model-00002-of-00005.safetensors",
344
+ "model.layers.7.mlp.gate_proj.weight": "model-00002-of-00005.safetensors",
345
+ "model.layers.7.mlp.up_proj.weight": "model-00002-of-00005.safetensors",
346
+ "model.layers.7.post_attention_layernorm.weight": "model-00002-of-00005.safetensors",
347
+ "model.layers.7.self_attn.k_proj.weight": "model-00002-of-00005.safetensors",
348
+ "model.layers.7.self_attn.o_proj.weight": "model-00002-of-00005.safetensors",
349
+ "model.layers.7.self_attn.q_proj.weight": "model-00002-of-00005.safetensors",
350
+ "model.layers.7.self_attn.v_proj.weight": "model-00002-of-00005.safetensors",
351
+ "model.layers.8.input_layernorm.weight": "model-00002-of-00005.safetensors",
352
+ "model.layers.8.mlp.down_proj.weight": "model-00002-of-00005.safetensors",
353
+ "model.layers.8.mlp.gate_proj.weight": "model-00002-of-00005.safetensors",
354
+ "model.layers.8.mlp.up_proj.weight": "model-00002-of-00005.safetensors",
355
+ "model.layers.8.post_attention_layernorm.weight": "model-00002-of-00005.safetensors",
356
+ "model.layers.8.self_attn.k_proj.weight": "model-00002-of-00005.safetensors",
357
+ "model.layers.8.self_attn.o_proj.weight": "model-00002-of-00005.safetensors",
358
+ "model.layers.8.self_attn.q_proj.weight": "model-00002-of-00005.safetensors",
359
+ "model.layers.8.self_attn.v_proj.weight": "model-00002-of-00005.safetensors",
360
+ "model.layers.9.input_layernorm.weight": "model-00002-of-00005.safetensors",
361
+ "model.layers.9.mlp.down_proj.weight": "model-00002-of-00005.safetensors",
362
+ "model.layers.9.mlp.gate_proj.weight": "model-00002-of-00005.safetensors",
363
+ "model.layers.9.mlp.up_proj.weight": "model-00002-of-00005.safetensors",
364
+ "model.layers.9.post_attention_layernorm.weight": "model-00002-of-00005.safetensors",
365
+ "model.layers.9.self_attn.k_proj.weight": "model-00002-of-00005.safetensors",
366
+ "model.layers.9.self_attn.o_proj.weight": "model-00002-of-00005.safetensors",
367
+ "model.layers.9.self_attn.q_proj.weight": "model-00002-of-00005.safetensors",
368
+ "model.layers.9.self_attn.v_proj.weight": "model-00002-of-00005.safetensors",
369
+ "model.norm.weight": "model-00005-of-00005.safetensors"
370
+ }
371
+ }
special_tokens_map.json ADDED
@@ -0,0 +1,23 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "bos_token": {
3
+ "content": "<s>",
4
+ "lstrip": false,
5
+ "normalized": false,
6
+ "rstrip": false,
7
+ "single_word": false
8
+ },
9
+ "eos_token": {
10
+ "content": "</s>",
11
+ "lstrip": false,
12
+ "normalized": false,
13
+ "rstrip": false,
14
+ "single_word": false
15
+ },
16
+ "unk_token": {
17
+ "content": "<unk>",
18
+ "lstrip": false,
19
+ "normalized": false,
20
+ "rstrip": false,
21
+ "single_word": false
22
+ }
23
+ }
tokenizer.json ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:b0240ce510f08e6c2041724e9043e33be9d251d1e4a4d94eb68cd47b954b61d2
3
+ size 17078292
tokenizer_config.json ADDED
The diff for this file is too large to render. See raw diff