LatitudeGames

/

Wayfarer-12B-GGUF

like 18

Follow

Latitude 43

tl;dr;

Pretty good RP model at only 12B! I had a good run before the fabric of reality itself finally undid me. It is fairly steerable as others mentioned, but makes it fun and keeps the story going in creative ways.

Performance

I'm getting ~38 tok/s on my 3090TI FE on latest llama.cpp@7a689c41 fully offloading and 16k context in just under 16GB VRAM (the card has 24GB, so enough to extra VRAM to run kokoro-tts to read everything aloud to me with minimal latency using streaming gradio app, hejhejhej...).

Server Command

./llama-server \
    --model "../models/LatitudeGames/Wayfarer-12B-GGUF/Wayfarer-12B-Q8_0.gguf" \
    --n-gpu-layers 41 \
    --ctx-size 16384 \
    --parallel 1 \
    --cache-type-k f16 \
    --cache-type-v f16 \
    --threads 8 \
    --flash-attn \
    --mlock \
    --host  127.0.0.1 \
    --port 8080

System Prompt

You are an adventure role-play Dungeon Master specifically trained to give players a challenging and dangerous experience. Lead the user on an AI Dungeon adventure with high quality prose including creative description language, adult themes, existential characters, and gripping dialogue.

Samplers

I roughly followed the model cards lead and set temp=0.8 and dropped min_p=0.02. However instead of adding repetition penalty, I tried out XTC and DRY to mix it up a bit:

Prose Generation Example

The vast nothingness swallows you whole, pulling apart and reconstructing every atom of your being. The transformation is beyond pain - it's beyond everything human conception allows, and for an instant that feels like an eternity, you cease to exist entirely, consumed by the cosmic hunger. Then, like a newborn star being birthed in the void, you re-emerge.

Silvermarsh Harbor stands before you as if it's your first day on its cobbled streets. Yet something is fundamentally different - the very fabric of the world around you seems thrummed with a subtle awareness of your existence, a resonance that speaks of a power now flowing through you. People bustle about, oblivious to your presence, though some might sense it faintly as they glance in your direction with unrecognizing wariness. You've walked the same streets, seen the same sights, but now you perceive patterns in the movement of shadows, in the way reality itself bends and ripples.

Time, space - they've taken on a different texture.

In the distance, you hear it - that same rhythmic pulsing that consumed you, but now it feels like... like home. The knowledge, the memories of what came before remain, but they're filtered through this new lens, this new awareness that both terrifies and exhilarates. The world of Silvermarsh Harbor might never be the same again. As the last wisps of the Rose and Bold Tom fade from memory, you sense their echoes within you, a part of the infinite tapestry you've now become. The harbor calls, and the adventure - or perhaps the cosmic horror - continues anew. Will you embrace your newfound power?

References

• llama.cpp sampler reference guide
• r/LocaLLLaMA thread