ubergarm

/

GLM-4.5-GGUF

like 14

@Lockout

Thanks for the report! This larger GLM model is quite usable even with older DDR RAM and you're even multi-NUMA node pretty amazing.

Regarding PP speed, you'll llikely get more tok/sec by omitting -rtr and increasing -ub 4096 -b 4096 but may need to offload less to free up the VRAM to support larger batches.

When using -rtr I tend to keep it at default batch sizes and offload as much as possible trying to get a little more TG.

I have a quant cookers guide on ik_llama.cpp discussions with some info https://github.com/ikawrakow/ik_llama.cpp/discussions/434

A quick reference of my usual perplexity procedure (i try to keep it almost exactly the same especially context size as it would otherwise vary a lot and not be useful for comparing):

$ wget https://huggingface.co/datasets/ikawrakow/validation-datasets-for-llama.cpp/resolve/main/wiki.test.raw.gz
$ gunzip wiki.test.raw.gz
$ sha1sum wiki.test.raw
6f1fe2054a940eebfc76b284b09680763b37f5ea  wiki.test.raw

model=/mnt/llms/ubergarm/Qwen3-14B-GGUF/Qwen3-14B-Q8_0.gguf

./build/bin/llama-perplexity \
    -m "$model" \
    --ctx-size 512 \
    -f wiki.test.raw \
    -fa \
    -ngl 99 \
    --seed 1337 \
    --threads 1

The important things to hold consistent are:

• ctx-size
• exact wiki.test.raw file
• fa should be very similar to non-fa but i report using fa path as it is typically what people use
• leave kv-cache unquantized at full f16 as lower sizes very slightly effect perplexity

It is fine to change:

• the seed is not actually used but i leave it as a kind of calling card lol
• adjust threads and offload to match whatever is fastest for you
• it is okay to adjust batch and ubatch (but i avoid going over 4096)

Thanks!

Unfortunately I found out something about 4096. Yes it gives you "faster" prompt processing (double), but it increases your time to reply. In actual chats the PP can be half on smaller prompts regardless of what the bench says. If you're doing big prompts, 4096, if you're chatting RTR and 512 or 1024 ubatch.

I'm testing the IKSS now to see where it falls. Unfortunately gguf scripts don't dump the tensors for me to see file sizes . Likely because I have the mainline gguf package installed. I wish sweep bench printed them in MiB like main, its a nice QOL thing.

Preliminary it looks like this:

Probably have to drop caches since its loading slowly.

@Lockout

Yup, increasing the batch size can increase "latency" for a short prompts psure. And right for lots of long prompts it is beneficial, but every day shorter prompt mult-turn chats rtr and default or smaller batches is fine and might allow one more routed expert layer offloaded on VRAM too.

Unfortunately gguf scripts don't dump the tensors for me to see file sizes . Likely because I have the mainline gguf package installed.

You can view the tensors using the same gguf package from ik_llama.cpp. Unfortunately, hugging face doesn't support this and so most of my quants are not visible in the side-bar which is annoying and inconvenient indeed. Here is the gguf dump script:

$ cd ik_llama.cpp
$ source venv/bin/activate # or however you manage python packages
$ pip install numpy==1.26.4 # a few more packages you'll need too
$ python gguf-py/scripts/gguf_dump.py /mnt/raid/models/ubergarm/GLM-4.5-Air-GGUF/GLM-4.5-Air-main-IQ3_KS.gguf

I wish sweep bench printed them in MiB like main, its a nice QOL thing.

I'm not sure exactly where llama-sweep-bench prints tensors in MiB sizes? sweep-bench isn't even officially on mainline psure. If you have a short clip of the logs and/or command I might understand better. But yeah in general I too like my numbers in GiB and MiB!

llama.cpp prints it when loading.. mis-wrote. I assume loading seq is all the same tho.. it says offloaded tensor xyz to cpu (750mb) etc. In mainline I think you may have to enable verbose now to see it.

For that script, on mainline it calls the gguf_reader which is python. If you have the gguf py package installed it will use it's reader. I also had to patch it to display MB and not bytes. On a large list that's quite distracting.