How to use the tokenizer of gemma-3
#36
by
rishav09
- opened
Hi,
I am trying to use the tokenizer of gemma-3 using the below code, but I am unable to do, how to do it-
from transformers import AutoTokenizer
# Load Gemma 3‑27B‑IT’s tokenizer
MODEL_GEMMA = "google/gemma-3-27b-it"
gemma_tokenizer = AutoTokenizer.from_pretrained(MODEL_GEMMA, trust_remote_code=True)
Hi
@rishav09
, Gemma 3 requires transformers version 4.45.0.dev. Please install this specific version using the provided command - !pip install git+https://github.com/huggingface/[email protected]. I have tried replicating the error and able to get the tokenizers successfully. See the attached gist for more details.