Tongyi-Zhiwen

/

QwenLong-L1-32B

like 148

Follow

Tongyi-Zhiwen 106

When I was exploring how chat_template organizes tool call information, I found that the output was missing the <|tool▁calls▁end|> tag. Below is my messages:

    messages=[
        {"role": "system", "content": "You are a helpful AI assistant."},
        {"role": "user", "content": "Who won the world series in 2020?"},
        {"role": "assistant", "content": None, "tool_calls": [
            {
                "type": "function",
                "function": {
                    "name": "get_world_series_winner",
                    "arguments": "{\"year\": \"int\"}"
                }
            }
        ]},
        {"role": "tool", "content": "The Los Angeles Dodgers"},
    ],

The output is:

<｜begin▁of▁sentence｜>You are a helpful AI assistant.<｜User｜>Who won the world series in 2020?<｜Assistant｜><｜tool▁calls▁begin｜><｜tool▁call▁begin｜>function<｜tool▁sep｜>get_world_series_winner
```json
{"year": "int"}
```<｜tool▁call▁end｜><｜tool▁outputs▁begin｜><｜tool▁output▁begin｜>The Los Angeles Dodgers<｜tool▁output▁end｜><｜tool▁outputs▁end｜>

And I notice that there might be a problem with the chat_template. I format the template info:

{% if not add_generation_prompt is defined %}
    {% set add_generation_prompt = false %}
{% endif %}

{% set ns = namespace(is_first=false, is_tool=false, is_output_first=true, system_prompt='') %}

{%- for message in messages %}
    {%- if message['role'] == 'system' %}
        {% set ns.system_prompt = message['content'] %}
    {%- endif %}
{%- endfor %}

{{bos_token}}
{{ns.system_prompt}}

{%- for message in messages %}
    {%- if message['role'] == 'user' %}
        {%- set ns.is_tool = false -%}
        {{'<｜User｜>' + message['content']}}
    {%- endif %}

    {%- if message['role'] == 'assistant' and message['content'] is none %}
        {%- set ns.is_tool = false -%}

        {%- for tool in message['tool_calls']%}
            {%- if not ns.is_first %}
                {{'<｜Assistant｜><｜tool▁calls▁begin｜><｜tool▁call▁begin｜>' + tool['type'] + '<｜tool▁sep｜>' + tool['function']['name'] + '\\n' + '```json' + '\\n' + tool['function']['arguments'] + '\\n' + '```' + '<｜tool▁call▁end｜>'}}
                {%- set ns.is_first = true -%}
            {%- else %}
                {{'\\n' + '<｜tool▁call▁begin｜>' + tool['type'] + '<｜tool▁sep｜>' + tool['function']['name'] + '\\n' + '```json' + '\\n' + tool['function']['arguments'] + '\\n' + '```' + '<｜tool▁call▁end｜>'}}
                {{'<｜tool▁calls▁end｜><｜end▁of▁sentence｜>'}}
            {%- endif %}
        {%- endfor %}
    {%- endif %}

    {%- if message['role'] == 'assistant' and message['content'] is not none %}
        {%- if ns.is_tool %}
            {{'<｜tool▁outputs▁end｜>' + message['content'] + '<｜end▁of▁sentence｜>'}}
            {%- set ns.is_tool = false -%}
        {%- else %}
            {% set content = message['content'] %}
            {% if '</think>' in content %}
                {% set content = content.split('</think>')[-1] %}
            {% endif %}
            {{'<｜Assistant｜>' + content + '<｜end▁of▁sentence｜>'}}
        {%- endif %}
    {%- endif %}

    {%- if message['role'] == 'tool' %}
        {%- set ns.is_tool = true -%}

        {%- if ns.is_output_first %}
            {{'<｜tool▁outputs▁begin｜><｜tool▁output▁begin｜>' + message['content'] + '<｜tool▁output▁end｜>'}}
            {%- set ns.is_output_first = false %}
        {%- else %}
            {{'\\n<｜tool▁output▁begin｜>' + message['content'] + '<｜tool▁output▁end｜>'}}
        {%- endif %}
    {%- endif %}
{%- endfor -%}

{% if ns.is_tool %}
    {{'<｜tool▁outputs▁end｜>'}}
{% endif %}
{% if add_generation_prompt and not ns.is_tool %}
    {{'<｜Assistant｜><think>\\n'}}
{% endif %}

If I swap the line {{'<｜tool▁calls▁end｜><｜end▁of▁sentence｜>'}} with the next line {%- endif %}, everything makes sense.