google/gemma-3-27b-it · Processor Decode does not remove <end_of

I'm using the example provided on the model's page. When I run:

decoded = processor.decode(generation, skip_special_tokens=True)
print(decoded)

the output doesn't remove the "end_of_turn", which still appears in the decoded text.

This issue occurs specifically when I use the model.generate method:

input_len = inputs["input_ids"].shape[-1]
with torch.inference_mode():
generation = model.generate(**inputs, max_new_tokens=500, do_sample=False)
generation = generation[0][input_len:]

However, when I use the pipeline interface, the special tokens (including "end_of_turn") are properly removed and everything works as expected. Also, using a tokenizer instead of processor to decode the text works just fine.

google
/

gemma-3-27b-it

Processor Decode does not remove <end_of_turn> token