Unable to load the custom tokenizer

by tomaarsen - opened May 15

May 15

Hello!

I'm excited to give this a try, but I ran into issues loading the custom tokenizer using the transformers and sentence_transformers snippets from the README. It seems to be caused by this, which does not point to a HF repository (or this repository itself): https://huggingface.co/BAAI/bge-code-v1/blob/main/tokenizer_config.json#L205

Tom Aarsen

hanhainebula

May 15

Hello, @tomaarsen . Thank you for bringing this issue to our attention! The problem has been fixed.

tomaarsen

May 15

Thank you!

tomaarsen changed discussion status to closed May 15

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment