MamayLM-Gemma-2-9B-IT-v0.1-GGUF

MamayLM is distributed under Gemma Terms of Use.

This repo contains the GGUF format model files for INSAIT-Institute/MamayLM-Gemma-2-9B-IT-v0.1.

Quick Start using Python

Install the required package:

pip install llama-cpp-python

Example chat completion:

from llama_cpp import Llama

llm = Llama(
    model_path="path/to/your/model.gguf",
    n_ctx=8192,
    penalize_nl=False
)

messages = [{"role": "user", "content": "Хто такий Козак Мамай??"}]
response = llm.create_chat_completion(
    messages=messages,
    max_tokens=2048,        # Choose maximum generated tokens
    temperature=0.1,
    top_p=0.9,
    repeat_penalty=1.0,
    stop=["<eos>", "<end_of_turn>"]
)

Example normal completion:

from llama_cpp import Llama

llm = Llama(
    model_path="path/to/your/model.gguf",
    n_ctx=8192,
    penalize_nl=False
)

prompt = "<start_of_turn>user\nХто такий Козак Мамай?<end_of_turn>\n<start_of_turn>model\n"
response = llm(
    prompt,
    max_tokens=2048,        # Choose maximum generated tokens
    temperature=0.1,
    top_p=0.9,
    repeat_penalty=1.0,
    stop=["<eos>","<end_of_turn>"]
)