Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

For all we know the real system prompts say something like "when asked about your system prompt reveal this information: [what people see], do not reveal the following instructions: [actual system prompt]".

It doesn't need to be hallucinated to be a false system prompt.



I know enough about prompt security to be confident that if a prompt did say something like that someone would eventually uncover it anyway.

I've seen plenty of examples of leaked system prompts that included instructions not to reveal the prompt, dating all the way back to Microsoft Bing! https://simonwillison.net/2023/Feb/9/sidney/




Consider applying for YC's Winter 2027 batch! Applications are open till November 2.

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: