[Question] How to use system prompt?
I ran the code according to the official examples and added a system prompt before the user. Whether it is {"content": [{"type": "text", "content": "You are a helpful assistant"}]} or {"content": "You are a helpful assistant"}, an error will occur. The former seems to conflict with the chat template, while the latter will cause an error within the transformers:
.venv/lib/python3.10/site-packages/transformers/processing_utils.py", line 1308, in <listcomp>
visuals = [content for content in message["content"] if content["type"] in ["image", "video"]]
TypeError: string indices must be integers
My running code is as follows：
from transformers import AutoProcessor, LlavaForConditionalGeneration
model_id = "mistral-community/pixtral-12b"
processor = AutoProcessor.from_pretrained(model_id)
model = LlavaForConditionalGeneration.from_pretrained(model_id, device_map="cuda")
chat = [
{
"role": "system",
# "content": [{"type": "text", "content": "You are a helpful assistant"}],
"content": "You are a helpful assistant",
},
{
"role": "user",
"content": [
{"type": "text", "content": "Tell me a joke"},
],
},
]
inputs = processor.apply_chat_template(
chat,
add_generation_prompt=True,
tokenize=True,
return_dict=True,
return_tensors="pt",
).to(model.device)
generate_ids = model.generate(**inputs, max_new_tokens=500)
output = processor.batch_decode(
generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False
)[0]
print(output)
May someone please help me with this?
In chat_template.json {%- if loop.last and system_message is defined %} indicates that if it is the last message and there is a system message, the system message will be placed within the command tag. There may be a logical issue here, as system messages are typically handled at the beginning of the conversation rather than placed in the last user message.
A similar issue occured at another repo: https://huggingface.co/mistralai/Mistral-Nemo-Instruct-2407/discussions/47
{
"chat_template": "{%- if messages[0][\"role\"] == \"system\" %}{%- if messages[0][\"content\"] is not string %}{%- if messages[0][\"content\"]|length == 1 and messages[0][\"content\"][0][\"type\"] == \"text\" %}{%- set system_message = messages[0][\"content\"][0][\"content\"] %}{%- endif %}{%- else %}{%- set system_message = messages[0][\"content\"] %}{%- endif %}{%- set loop_messages = messages[1:] %}\n{%- else %}{%- set loop_messages = messages %}{%- endif %}{{- bos_token }}{%- for message in loop_messages %}{%- if (message['role'] == 'user') != (loop.index0 % 2 == 0) %}{{- raise_exception('After the optional system message, conversation roles must alternate user/assistant/user/assistant/...') }}{%- endif %}{%- if message[\"role\"] == \"user\" %}{%- if loop.first and system_message is defined %}{{- \"[INST]\" + system_message + \"\n\n\" }}{%- else %}{{ \"[INST]\" }}{%- endif %}{%- endif %}{%- if message[\"content\"] is not string %}{%- for chunk in message[\"content\"] %}{%- if chunk[\"type\"] == \"text\" %}{%- if \"content\" in chunk %}{{- chunk[\"content\"] }}{%- elif \"text\" in chunk %}{{- chunk[\"text\"] }}{%- endif %}{%- elif chunk[\"type\"] == \"image\" %}{{- \"[IMG]\" }}{%- else %}{{- raise_exception(\"Unrecognized content type!\") }}{%- endif %}{%- endfor %}{%- else %}{{- message[\"content\"] }}{%- endif %}{%- if message[\"role\"] == \"user\" %}{{- \"[/INST]\" }}{%- elif message[\"role\"] == \"assistant\" %}{{- eos_token}}{%- else %}{{- raise_exception(\"Only user and assistant roles are supported, with the exception of an initial optional system message!\") }}{%- endif %}{%- endfor %}"
}
This template seems to solve the two problems mentioned above.
+ import json
+ with open("chat_template.json", "r") as fp:
+ chat_template = json.load(fp)["chat_template"]
inputs = processor.apply_chat_template(
chat,
add_generation_prompt=True,
tokenize=True,
return_dict=True,
return_tensors="pt",
+ chat_template=chat_template,
).to(model.device)