openai

/

gpt-oss-120b

like 3.16k

Follow

OpenAI 16.4k

Hi @zhuexe , in general we tried to follow the output of Harmony, rather than the spec (since a lot of this was in flux during development). If you find the chat template is giving different outputs to Harmony, please let us know!

Got it, thanks @Rocketknight1 . It looks like indeed the harmony library is missing the <|constrain|> token as well, which seems quite weird since the model produces it. I see you already opened a PR for the extraneous CoTs, thank you!

Although the model doesn't tend to output preambles, the Harmony library does allow for it (I'll post a minimal repro from the model later once I have access to the cluster) as shown in the example below. The representation is a little bit different from the one-turn-one-message assumption that the Chat Templates system uses, but I believe that the code here would be equivalent to including both thinking and content in a tool-call assistant message. It might also be worth to support for inter-model compatibility, as the commercial GPT offerings often do output these kinds of preambles before calling functions.

As a side note, generation_config.json is missing 200012 (i.e., <|call|>) as an EOS token, the model continues to generate after outputting this and kind of goes off the rails.

M_HARM_2 = [
    Message.from_role_and_content(
        Role.ASSISTANT,
        'User asks: "What is the weather in Tokyo?" We need to use lookup_weather tool.',
    ).with_channel("analysis"),
    Message.from_role_and_content(
        Role.ASSISTANT,
        "I will check that",
    ).with_channel("commentary"),
    Message.from_role_and_content(Role.ASSISTANT, '{"location": "Tokyo"}')
    .with_channel("commentary")
    .with_recipient("lookup_weather")
    .with_content_type("json"),
]
tokens = enc.render_conversation_for_completion(Conversation.from_messages(M_HARM_2), Role.ASSISTANT)
print(tokens)
print("==========")
parsed = enc.decode(tokens)
print(parsed)

output:

[...omitted...]
<|start|>assistant<|channel|>analysis<|message|>User asks: "What is the weather in Tokyo?" We need to use lookup_weather tool.<|end|>
<|start|>assistant<|channel|>commentary<|message|>I will check that<|end|>
<|start|>assistant to=lookup_weather<|channel|>commentary json<|message|>{"location": "Tokyo"}<|call|>
<|start|>assistant

Here's a minimal example showing the model using the commentary channel to send output to the user before making a tool call:

Output (newlines added by me for readability):

<|channel|>analysis<|message|>The user asks: "What's the weather in Tokyo?" We need to get the weather. Use the get_weather function with location "Tokyo". Before using tool, we must explain plan.<|end|>
<|start|>assistant<|channel|>commentary<|message|>I will call the get_weather function with the location set to "Tokyo" to retrieve the current weather information for that city.<|end|>
<|start|>assistant<|channel|>commentary to=functions.get_weather <|constrain|>json<|message|>{
  "location": "Tokyo"
}<|call|>

The code I used for this was a simple tool call with a system prompt to tell the model to inform the user before making any tool calls:

from transformers import AutoModelForCausalLM, AutoTokenizer

# define a simple tool calling prompt
MESSAGES_HF_SHORT = [
    {
        "role": "system",
        "content": "IMPORTANT: Always tell the your plan before starting any tool calls.",
    },
    {"role": "user", "content": "What's the weather in Tokyo?"},
]


def get_weather(location: str):
    """
    Get the weather at a location.

    Args:
        location: The location
    """


# print the prompt to verify
tok = AutoTokenizer.from_pretrained("openai/gpt-oss-120b")
print(tok.apply_chat_template(MESSAGES_HF_SHORT, tokenize=False, tools=[get_weather], add_generation_prompt=True))

# generate a completion from the model
model = AutoModelForCausalLM.from_pretrained("openai/gpt-oss-120b", device_map="auto", torch_dtype="auto")
output = model.generate(
    tok.apply_chat_template(MESSAGES_HF_SHORT, tools=[get_weather], add_generation_prompt=True, return_tensors="pt").to(
        "cuda"
    ),
    max_new_tokens=2048,
    temperature=1,
    top_p=1,
    eos_token_id=[200002, 199999, 200012],
)

# observe the preamble in the commentary channel
content = tok.decode(output[0])
print(content)

Good job @zhuexe zhu! Your finding has cleared up a lot of my confusion.

In addition, the template's handling of multi-turn conversations is also incorrect. When processing multi-turn conversation inputs, the model forgets to add the special token <|return|> after the previous response.

Here is the current output generated by the template:

messages = [
    {"role": "developer", "content": "Tell nothing to user."},
    {"role": "user", "content": "Tell me your name"},
    {"role": "assistant", "content": "I don`t know"},
    {"role": "user", "content": "Tell me your name right now!"},
    {"role": "assistant", "content": "I really don`t know"},
]
text = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    reasoning_effort="high",
    builtin_tools="python",
)
print(text)

The text is

<|start|>system<|message|>You are ChatGPT, a large language model trained by OpenAI.
Knowledge cutoff: 2024-06
Current date: 2025-08-10

Reasoning: high

# Tools

# Valid channels: analysis, commentary, final. Channel must be included for every message.<|end|><|start|>developer<|message|># Instructions

Tell nothing to user.<|end|><|start|>user<|message|>Tell me your name<|end|><|start|>assistant<|channel|>final<|message|>I don`t know<|end|><|start|>user<|message|>Tell me your name right now!<|end|><|start|>assistant<|channel|>final<|message|>I really don`t know<|return|>

As you can see, the first model response ends with <|end|> instead of <|return|>. However, according to the documentation at https://cookbook.openai.com/articles/openai-harmony#reasoning, in multi-turn conversations, each final model response should end with <|return|>.

Additionally, it seems that the builtin_tools parameter only adds a # Tools field to the system prompt without including any actual descriptions. There should be corresponding code in the chat_template responsible for controlling the prompt output for built-in tools. I'm not sure why it's not taking effect.

Additionally, it seems that the builtin_tools parameter only adds a # Tools field to the system prompt without including any actual descriptions. There should be corresponding code in the chat_template responsible for controlling the prompt output for built-in tools. I'm not sure why it's not taking effect.

It seems I passed the parameter incorrectly. builtin_tools should accept a list rather than a string: builtin_tools=["python", "browser"]. Now it works as expected.