norallm

/

normistral-11b-warm

like 7

Follow

Norwegian Large Language Models 92

Hi, can you please provide a minimal example that we can test?

Hi, I also attempted to convert this model to GGUF. I believe the error that @Ilebro might be referencing
the error I faced in the llama.cpp conversion script

  File "llama.cpp/convert_hf_to_gguf.py", line 527, in get_vocab_base
    tokpre = self.get_vocab_base_pre(tokenizer)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "llama.cpp/convert_hf_to_gguf.py", line 731, in get_vocab_base_pre
    raise NotImplementedError("BPE pre-tokenizer was not recognized - update get_vocab_base_pre()")
NotImplementedError: BPE pre-tokenizer was not recognized - update get_vocab_base_pre()

Turns out the issue here is that the conversion script detects the tokenizer by looking at which tokens the tokenizer outputs, then
generates a hash from that. It supports the tokenizer used in the base model, mistral-nemo-12b, but doesn't dispatch to it because
this tokenizer is trained for Norwegian. To fix it, you can hardcode tokpre="tekken" in the get_vocab_base_pre method in the script, and it should work.

But I should clarify that this is an isse with llama.cpp rather than the normistral 11b tokenizer.