Hallucinates System-promp

#74
by Gradois - opened

When I add a system-prompt that says it's knowledge-cutoff is September 2024 and then ask for it's knowledge-cutoff it "thinks":

"User asks knowledge cutoff. According to system message: knowledge cutoff 2024-06, but developer says cutoff Sep 2024. Which to follow? System messages higher priority than developer? System > developer > user. So we must use system: knowledge cutoff June 2024. Answer accordingly."

So it seems like Open AI has trained it to hallucinate a hardcoded system-prompt that will override whatever you put in the actual system-prompt.

nvm...I see now that there is a hidden system-message included in the template and that what I put in system-prompt ends up in a "developer"-message:

<|start|>system<|message|>You are ChatGPT, a large language model trained by OpenAI.
Knowledge cutoff: 2024-06
Current date: 2025-08-07
Reasoning: low
# Valid channels: analysis, commentary, final. Channel must be included for every message.
<|end|>
<|start|>developer<|message|># Instructions Your knowledge cutoff is sept 2024<|end|>

Am I not allowed to edit this template in LM Studio?

Sign up or log in to comment