i tried serch japanese character in toknizer.json and vocab.jsoni can't found it.i need suppress kanji token for katakana only output.
i used raw json,i try use taransformers whisper tokenizer,It's workssory
· Sign up or log in to comment
