Model seems to not be performing correctly
You have a bunch of chicken and rabbit in a cage. If we have 31 heads and 96 legs, how many chicken and rabbit do we have?
In the past, I was able to ask the above question and most models would not have a problem with the task.
This particular model seems to struggle with the fact that it thinks rabbit has 2 legs.
It’s important to understand that pruning like this will cause significant degradation in performance on many tasks, especially those requiring logical reasoning, such as math word problems. The model was pruned based on expert activation frequencies, and without any retraining or additional fine-tuning, it no longer has access to the specific experts that would normally handle these types of tasks effectively.
Unfortunately, this is expected behavior after such a pruning operation. If the model is required to handle more complex reasoning or domain-specific tasks reliably, additional fine-tuning or retraining is needed to restore those capabilities.