Performance
#5
by
urtuuuu
- opened
I found a question which will make the model fail about 30-50% of time, while original gemma 3 27b seems to answer correctly 100% of time. (just press regenerate 10 times)
"I have 2 apples, then i buy 2 more. I bake a pie with 2 of the apples. After eating half of the pie, how many apples do i have left?"
Correct answer is "2", but it sometimes answers "1" or "3".
So, is this because of abliteration?
Yes, this is likely due to abliteration. I was pretty heavy-handed with the refusal weight here, so that might have broken some capabilities.