dinerburger

/

Qwen2.5-Coder-32B-Instruct-exl2-4.125b_L

like 1

I’ve been meaning to try Roo Code. I’ll give it a shot but in the meantime if you could help me by narrowing down this issue through logs or the like I’d very much appreciate it.

Strangely here I can only upload images.. Here are the links to logs, I runed 2 times:
https://drive.google.com/file/d/1IGqu1YL1ZBVduVPl37XzcEl7ePg-MGZ-/view?usp=sharing
https://drive.google.com/file/d/11nCPuca6770J3EBIgItShmwEroToX9DH/view?usp=sharing
https://drive.google.com/file/d/19Quhguf0i1qVgYoq2azApzVcIWNVHxPl/view?usp=sharing
https://drive.google.com/file/d/1X7LgjD8fLKYGa_m52Vqmy6g6wJVG6hEz/view?usp=sharing

I quickly looked into the logs, looks like the model for some reason uses wrong commands (it also confuses file names):
read
path>tet.py path
/read

write_file
path>path/to/path/tetris2.py /path
content

When standard Qwen2.5-coder-32b-instruct, in the same prompt, uses correct commands:
read_file
path>path/to/tetris12.py /path
/read_file

write_to_file
path>path/to/tetris12.py /path
content

Here is my tabby config (standard qwen also uses Q4):
max_seq_len: 65536
cache_mode: Q4
rope_scale: 2

Hope this helps.
Maybe the system message does not getting parsed right or maybe I should experiment with temperature or min/max p values...
Anyway. Thank you for your work, I really appreciate it!

Ah, ok I do have an idea here. rope_scale in the config may be incorrect and unnecessary. Set it to 1 in your tabby config and add this to the actual config.json of the model:

"rope_scaling": {
    "factor": 4.0,
    "original_max_position_embeddings": 32768,
    "type": "yarn"
}

I believe yarn is handled differently in tabby from baseline RoPE. I’ve had personal stuff all weekend but this stuck out to me. Please let me know if that helps.

Yes! I believe it helped, gave it some more tasks, context size went past 32k and still working, T/s dropped to 10 from 28 though, but still not bad :)
Thank you again!

Yeah even at 1000 t/s for prompt processing speed at 32K tokens you’re taking about a bunch of processing time so that tracks unfortunately. Still, I’m glad this worked! I may make that change to the config file here and I’ll definitely annotate the readme before closing.