add model_max_length
#4
by
alistairewj
- opened
model_max_length is present in the BERT base tokenizer config, but not here, and it should be
without this, the model_max_length for the tokenizer defaults to a very large int, and inference crashes if not handled properly