Actually interesting model (maybe overtuned)
#3
by
SerialKicked
- opened
I spent way more time on this model than anticipated.
Pros:
- It's chatml instead of the mistral's abomination of an instruct prompt
- It's better than mistral in writing. It has some slop but if you understand LLM you know that eradicating slop is just replacing one for another. Those recurring sentences serve a purpose you can't make go away.
- As a storyteller sys.prompt / character it works flawlessly. Not surprising that it's better as a storyteller than as a chat partner. It's still decent in the other role, though.
- You made the effort to actually pick a consistent action text tone, which is great because most RP models are very bad at it.
- It's willing to go roads that most models would tell you to fuck off
Cons:
- It's still a Mistral model and the second you allow it to keep a post structure, it'll keep repeating it over and over again. Not your fault, it's a defect of the model itself.
- It's so willing to follow your lead that it feels kinda servile (no matter how dumb you are, you're the best thing that ever existed, kind of deal)
- It's been obviously over-trained on "somename: some text" kind of output, to a point that when ordered to summarize a text or do a task literally any language model could do, it'll still preface the text with a "[space]: {actual response}". It's filterable, but it shows you really damaged the base model for normal operations. It's a pet peeve of mine because I'm working on an app that needs both decent RP performance and high IF rating. Let's summarize this as atrocious instruction following ability (which is weird of a mistral model to begin with)
All in all, it still ranks up as one of the best models i've encountered in that category this year. Ain't perfect, but it's good.
For me, this model tends to loop over the story, not introducing any new characters and events at all without explicit user input. For example, if I say that my character stays in the apartment - it will stay in the apartment, thinking about the same things, just phrased differently, in each message. Is that just me?