Hugging Face's logo

Nondzu
/
Llama-PLLuM-70B-chat-GGUF

Text Generation
GGUF
Inference Endpoints
conversational
Model card Files Community
Nondzu commited on
Commit
ec9e794
·
verified ·
1 Parent(s): ca25d32

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +77 -3
README.md CHANGED
@@ -1,3 +1,77 @@
1
- ---
2
- license: llama3.1
3
- ---
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+
2
+ ```yaml
3
+ ---
4
+ quantized_by: Nondzu
5
+ pipeline_tag: text-generation
6
+ base_model: CYFRAGOVPL/Llama-PLLuM-70B-chat
7
+ license: llama3.1
8
+ ---
9
+ ```
10
+
11
+ # Llama-PLLuM-70B-chat GGUF Quantizations by Nondzu
12
+
13
+ DISCLAIMER: This is state of the art quantized model. I am not the author of the original model. I am only hosting the quantized models. I do not take any responsibility for the models.
14
+
15
+ This repository contains GGUF quantized versions of the [Llama-PLLuM-70B-chat](https://huggingface.co/CYFRAGOVPL/Llama-PLLuM-70B-chat) model. All quantizations were performed using the [llama.cpp](https://github.com/ggerganov/llama.cpp) (release [b4765](https://github.com/ggml-org/llama.cpp/releases/tag/b4765)). These quantized models can be run in [LM Studio](https://lmstudio.ai/) or any other llama.cpp–based project.
16
+
17
+ ## Prompt Format
18
+
19
+ Use the following prompt structure:
20
+ ```
21
+ ???
22
+ ```
23
+
24
+ ## Available Files
25
+
26
+ Below is a list of available quantized model files along with their quantization type, file size, whether the file is split, and a short description.
27
+
28
+ | Filename | Quant Type | File Size | Split | Description |
29
+ | ------------------------------------------------------------------------------------- | ---------- | --------- | ----- | --------------------------------------------------------------------------------------------- |
30
+ | [Llama-PLLuM-70B-chat-Q2_K.gguf](https://huggingface.co/Nondzu/Llama-PLLuM-70B-chat-GGUF/tree/main) | Q2_K | 25 GB | No | Very low quality but surprisingly usable. |
31
+ | [Llama-PLLuM-70B-chat-Q3_K.gguf](https://huggingface.co/Nondzu/Llama-PLLuM-70B-chat-GGUF/tree/main) | Q3_K | 32 GB | No | Low quality, suitable for setups with very limited RAM. |
32
+ | [Llama-PLLuM-70B-chat-Q3_K_L.gguf](https://huggingface.co/Nondzu/Llama-PLLuM-70B-chat-GGUF/tree/main) | Q3_K_L | 35 GB | No | High quality; recommended for quality-focused usage. |
33
+ | [Llama-PLLuM-70B-chat-Q3_K_M.gguf](https://huggingface.co/Nondzu/Llama-PLLuM-70B-chat-GGUF/tree/main) | Q3_K_M | 32 GB | No | Very high quality, near perfect output – recommended. |
34
+ | [Llama-PLLuM-70B-chat-Q3_K_S.gguf](https://huggingface.co/Nondzu/Llama-PLLuM-70B-chat-GGUF/tree/main) | Q3_K_S | 29 GB | No | Moderate quality with improved space efficiency. |
35
+ | [Llama-PLLuM-70B-chat-Q4_K.gguf](https://huggingface.co/Nondzu/Llama-PLLuM-70B-chat-GGUF/tree/main) | Q4_K | 40 GB | No | Good quality for standard use. |
36
+ | [Llama-PLLuM-70B-chat-Q4_K_M.gguf](https://huggingface.co/Nondzu/Llama-PLLuM-70B-chat-GGUF/tree/main) | Q4_K_M | 40 GB | No | Default quality for most use cases – recommended. |
37
+ | [Llama-PLLuM-70B-chat-Q4_K_S.gguf](https://huggingface.co/Nondzu/Llama-PLLuM-70B-chat-GGUF/tree/main) | Q4_K_S | 38 GB | No | Slightly lower quality with enhanced space savings – recommended when size is a priority. |
38
+ | [Llama-PLLuM-70B-chat-Q5_0.gguf](https://huggingface.co/Nondzu/Llama-PLLuM-70B-chat-GGUF/tree/main) | Q5_0 | 46 GB | No | Extremely high quality – the maximum quant available. |
39
+ | [Llama-PLLuM-70B-chat-Q5_K.gguf](https://huggingface.co/Nondzu/Llama-PLLuM-70B-chat-GGUF/tree/main) | Q5_K | 47 GB | No | Very high quality – recommended for demanding use cases. |
40
+ | [Llama-PLLuM-70B-chat-Q5_K_M.gguf](https://huggingface.co/Nondzu/Llama-PLLuM-70B-chat-GGUF/tree/main) | Q5_K_M | 47 GB | No | High quality – recommended. |
41
+ | [Llama-PLLuM-70B-chat-Q5_K_S.gguf](https://huggingface.co/Nondzu/Llama-PLLuM-70B-chat-GGUF/tree/main) | Q5_K_S | 46 GB | No | High quality, offered as an alternative with minimal quality loss. |
42
+ | [Llama-PLLuM-70B-chat-Q4_0.gguf](https://huggingface.co/Nondzu/Llama-PLLuM-70B-chat-GGUF/tree/main) | Q4_0 | 38 GB | No | Legacy format offering online repacking for ARM/AVX CPU inference. |
43
+ | [Llama-PLLuM-70B-chat-Q6_K.gguf](https://huggingface.co/Nondzu/Llama-PLLuM-70B-chat-GGUF/tree/main) | Q6_K | 54 GB | Yes | Very high quality with quantized embed/output weights. Split into 2 parts due to file size. |
44
+ |    • Part 1: [Q6_K-00001-of-00002.gguf](https://huggingface.co/Nondzu/Llama-PLLuM-70B-chat-GGUF/tree/main) (37 GB)
45
+    • Part 2: [Q6_K-00002-of-00002.gguf](https://huggingface.co/Nondzu/Llama-PLLuM-70B-chat-GGUF/tree/main) (18 GB) | | | | |
46
+ | [Llama-PLLuM-70B-chat-Q8_0.gguf](https://huggingface.co/Nondzu/Llama-PLLuM-70B-chat-GGUF/tree/main) | Q8_0 | 70 GB | Yes | Maximum quality quantization. Available either as a single file or split into 2 parts. |
47
+ |    • Part 1: [Q8_0.gguf-00001-of-00002.gguf](https://huggingface.co/Nondzu/Llama-PLLuM-70B-chat-GGUF/tree/main) (37 GB)
48
+    • Part 2: [Q8_0.gguf-00002-of-00002.gguf](https://huggingface.co/Nondzu/Llama-PLLuM-70B-chat-GGUF/tree/main) (34 GB) | | | | |
49
+
50
+ *Files marked as "split" must be downloaded in full (all parts) to obtain the complete quantized model.
51
+
52
+ ## Downloading Using Hugging Face CLI
53
+
54
+ <details>
55
+ <summary>Click to view download instructions</summary>
56
+
57
+ First, ensure you have the Hugging Face CLI installed:
58
+
59
+ ```bash
60
+ pip install -U "huggingface_hub[cli]"
61
+ ```
62
+
63
+ Then, target a specific file to download:
64
+
65
+ ```bash
66
+ huggingface-cli download Nondzu/Llama-PLLuM-70B-chat-GGUF --include "Llama-PLLuM-70B-chat-Q4_K_M.gguf" --local-dir ./
67
+ ```
68
+
69
+ For files larger than 50 GB that are split into multiple parts, use a wildcard to download all parts at once:
70
+
71
+ ```bash
72
+ huggingface-cli download Nondzu/Llama-PLLuM-70B-chat-GGUF --include "Llama-PLLuM-70B-chat-Q8_0/*" --local-dir ./
73
+ ```
74
+
75
+ You can specify a new local directory (e.g., `Llama-PLLuM-70B-chat-Q8_0`) or download them directly into the current directory (`./`).
76
+
77
+ </details>