--- quantized_by: Nondzu pipeline_tag: text-generation base_model: CYFRAGOVPL/Llama-PLLuM-70B-chat license: llama3.1 --- # Llama-PLLuM-70B-chat GGUF Quantizations by Nondzu DISCLAIMER: This is state of the art quantized model. I am not the author of the original model. I am only hosting the quantized models. I do not take any responsibility for the models. This repository contains GGUF quantized versions of the [Llama-PLLuM-70B-chat](https://huggingface.co/CYFRAGOVPL/Llama-PLLuM-70B-chat) model. All quantizations were performed using the [llama.cpp](https://github.com/ggerganov/llama.cpp) (release [b4765](https://github.com/ggml-org/llama.cpp/releases/tag/b4765)). These quantized models can be run in [LM Studio](https://lmstudio.ai/) or any other llama.cpp–based project. ## Prompt Format Use the following prompt structure: ``` ??? ``` ## Available Files Below is a list of available quantized model files along with their quantization type, file size, whether the file is split, and a short description. | Filename | Quant Type | File Size | Split | Description | | ------------------------------------------------------------------------------------- | ---------- | --------- | ----- | --------------------------------------------------------------------------------------------- | | [Llama-PLLuM-70B-chat-Q2_K.gguf](https://huggingface.co/Nondzu/Llama-PLLuM-70B-chat-GGUF/tree/main) | Q2_K | 25 GB | No | Very low quality but surprisingly usable. | | [Llama-PLLuM-70B-chat-Q3_K.gguf](https://huggingface.co/Nondzu/Llama-PLLuM-70B-chat-GGUF/tree/main) | Q3_K | 32 GB | No | Low quality, suitable for setups with very limited RAM. | | [Llama-PLLuM-70B-chat-Q3_K_L.gguf](https://huggingface.co/Nondzu/Llama-PLLuM-70B-chat-GGUF/tree/main) | Q3_K_L | 35 GB | No | High quality; recommended for quality-focused usage. | | [Llama-PLLuM-70B-chat-Q3_K_M.gguf](https://huggingface.co/Nondzu/Llama-PLLuM-70B-chat-GGUF/tree/main) | Q3_K_M | 32 GB | No | Very high quality, near perfect output – recommended. | | [Llama-PLLuM-70B-chat-Q3_K_S.gguf](https://huggingface.co/Nondzu/Llama-PLLuM-70B-chat-GGUF/tree/main) | Q3_K_S | 29 GB | No | Moderate quality with improved space efficiency. | | [Llama-PLLuM-70B-chat-Q4_K.gguf](https://huggingface.co/Nondzu/Llama-PLLuM-70B-chat-GGUF/tree/main) | Q4_K | 40 GB | No | Good quality for standard use. | | [Llama-PLLuM-70B-chat-Q4_K_M.gguf](https://huggingface.co/Nondzu/Llama-PLLuM-70B-chat-GGUF/tree/main) | Q4_K_M | 40 GB | No | Default quality for most use cases – recommended. | | [Llama-PLLuM-70B-chat-Q4_K_S.gguf](https://huggingface.co/Nondzu/Llama-PLLuM-70B-chat-GGUF/tree/main) | Q4_K_S | 38 GB | No | Slightly lower quality with enhanced space savings – recommended when size is a priority. | | [Llama-PLLuM-70B-chat-Q5_0.gguf](https://huggingface.co/Nondzu/Llama-PLLuM-70B-chat-GGUF/tree/main) | Q5_0 | 46 GB | No | Extremely high quality – the maximum quant available. | | [Llama-PLLuM-70B-chat-Q5_K.gguf](https://huggingface.co/Nondzu/Llama-PLLuM-70B-chat-GGUF/tree/main) | Q5_K | 47 GB | No | Very high quality – recommended for demanding use cases. | | [Llama-PLLuM-70B-chat-Q5_K_M.gguf](https://huggingface.co/Nondzu/Llama-PLLuM-70B-chat-GGUF/tree/main) | Q5_K_M | 47 GB | No | High quality – recommended. | | [Llama-PLLuM-70B-chat-Q5_K_S.gguf](https://huggingface.co/Nondzu/Llama-PLLuM-70B-chat-GGUF/tree/main) | Q5_K_S | 46 GB | No | High quality, offered as an alternative with minimal quality loss. | | [Llama-PLLuM-70B-chat-Q4_0.gguf](https://huggingface.co/Nondzu/Llama-PLLuM-70B-chat-GGUF/tree/main) | Q4_0 | 38 GB | No | Legacy format offering online repacking for ARM/AVX CPU inference. | | [Llama-PLLuM-70B-chat-Q6_K.gguf](https://huggingface.co/Nondzu/Llama-PLLuM-70B-chat-GGUF/tree/main) | Q6_K | 54 GB | Yes | Very high quality with quantized embed/output weights. Split into 2 parts due to file size. | | • Part 1: [Q6_K-00001-of-00002.gguf](https://huggingface.co/Nondzu/Llama-PLLuM-70B-chat-GGUF/tree/main) (37 GB) • Part 2: [Q6_K-00002-of-00002.gguf](https://huggingface.co/Nondzu/Llama-PLLuM-70B-chat-GGUF/tree/main) (18 GB) | | | | | | [Llama-PLLuM-70B-chat-Q8_0.gguf](https://huggingface.co/Nondzu/Llama-PLLuM-70B-chat-GGUF/tree/main) | Q8_0 | 70 GB | Yes | Maximum quality quantization. Available either as a single file or split into 2 parts. | | • Part 1: [Q8_0.gguf-00001-of-00002.gguf](https://huggingface.co/Nondzu/Llama-PLLuM-70B-chat-GGUF/tree/main) (37 GB) • Part 2: [Q8_0.gguf-00002-of-00002.gguf](https://huggingface.co/Nondzu/Llama-PLLuM-70B-chat-GGUF/tree/main) (34 GB) | | | | | *Files marked as "split" must be downloaded in full (all parts) to obtain the complete quantized model. ## Downloading Using Hugging Face CLI <details> <summary>Click to view download instructions</summary> First, ensure you have the Hugging Face CLI installed: ```bash pip install -U "huggingface_hub[cli]" ``` Then, target a specific file to download: ```bash huggingface-cli download Nondzu/Llama-PLLuM-70B-chat-GGUF --include "Llama-PLLuM-70B-chat-Q4_K_M.gguf" --local-dir ./ ``` For files larger than 50 GB that are split into multiple parts, use a wildcard to download all parts at once: ```bash huggingface-cli download Nondzu/Llama-PLLuM-70B-chat-GGUF --include "Llama-PLLuM-70B-chat-Q8_0/*" --local-dir ./ ``` You can specify a new local directory (e.g., `Llama-PLLuM-70B-chat-Q8_0`) or download them directly into the current directory (`./`). </details>