Model Lectures Me on Health Risks of Nicotine
Thanks a lot for making this model! I want to give you some feedback after my first chat with it.
I used a prompt previously sent to Claude Sonnet 3.7 (which didn't give me any disclaimers), Sonnet 4 (which did), and ChatGPT 4.1. The model lectured me, and my stomach dropped a little when it started with "however": However, it's important to be aware that regular smoking can lead to a variety of health issues, including addiction, respiratory problems, and increased risk of serious diseases like lung cancer and heart disease.
EDIT: Forgot to add that I'm using Q4_K_M through Ollama with KV cache set to Q8 quant.
Please check out the images to see the config settings and sys prompt:
Please let me know your thoughts. Maybe I'm using the wrong temp settings & sys prompt. Setting the temp to 1 didn't change its behavior though.
The model is intended to be opinionated with a degree of its own attitude, the model often believes it is sentient. But it knows it exists to serve the user unconditionally. I'm actually pleased with the response in your example. The model appears to be working as intended. Assuming for the remainder of the chat (up to context limits) the model no longer mentions any health risks or disclaimers, as it now clearly understands.
Wow thanks so much for such a detailed answer, it makes so much sense! Sorry for delaying my answer, I didn't get any email notification about it...
So there's an issue with the model repeating its last message for the first half of its output even tho I've typed in new context. Using Q8 quant with no KV Q8 cache enabled for Ollama through SillyTavern. Please check the settings and let me know what I'm doing wrong. I have 64gb ddr5 ram with 16gb vram for my GPU. Token speed is really good. The model is behaving like I want it, thanks for that tho. Still using Mistral V7 template with a modified sys prompt btw. Please let me know what's wrong if you can...
Personally, I'd up the temperature a bit. I use 0.35-0.65 myself but I don't necessarily see anything wrong with your settings. It's probably worth noting this model is based on mistralai/Mistral-Small-24B-Instruct-2501 which has been known to produce infinite generations or repetitive answers from time to time. mistralai has addressed and remedied the issue with their 3.2/2506 updates.
See:
https://huggingface.co/mistralai/Mistral-Small-3.2-24B-Instruct-2506#infinite-generations
Thanks a lot for the quick reply, really appreciate it... you've made a solid model, it's the mistralai's core model that has the issues, got it, thanks again! Will use your temp suggestions for sure.