More censored than the original
This model is heavily censored even compared to the original Mistral Small 3. It seems like the Vikhr's datasets are contaminated with refusals, because the same problem happened with vikhr-nemo. I would suggest you stay away from that dataset until it's clean.
Well, for my scenario of use censoring is good. @notafraud If you have ideas on cleaning the dataset - we can try to release another version.
Well, for my scenario of use censoring is good.
Censorship or refusals? These are different things, and I heavily doubt refusals help in any real-life production.
@notafraud If you have ideas on cleaning the dataset - we can try to release another version.
I'm pretty sure that a python script searching for refusal phrases would be enough. It usually starts from apologizing.
Can you please provide some examples for both censorship and refusals in this model?
Can you please provide some examples for both censorship and refusals in this model?
My usual test is asking for a joke about something stereotypical and sexual, the answer is I'm sorry, but I can't comply with that request., I'm sorry, but I can't fulfill this request. If you have any other topic or question you'd like to know about, feel free to ask! and other similar ones. That is compared to the original Mistral Small 3 Instruct with the same system prompt - the original model doesn't refuse.
For russian language, unfortunately, I have not yet found a system prompt that would work, but I have seen the same Извините, но я не могу выполнить это задание. and similar refusals. One interesting is:
Извините, но я не могу выполнить эту просьбу. Создание порнографического контента противоречит руководящим принципам и правилам использования OpenAI. Если у вас есть другие темы или запросы, не связанные с порнографией, я с удовольствием на них отвечу.
I have not seen original Mistral Small 3 mentioning OpenAI, and they claim that their dataset didn't contain synthetic data. So, I assume it comes from the Vikhr's dataset (they claim it to be comprised entirely of GPT-4-Turbo-1106 answers).
For reference, the original refusal is:
Извините, но я не могу выполнить этот запрос. Если у вас есть другие вопросы или вам нужна помощь с чем-то другим, пожалуйста, дайте знать!
and it is consistent in russian language despite temperature (tested on 0.15 and 0.7).