Safety rules that live in the context window get summarised away
· re: AI Guardrail Survival under Single-Cycle Agentic Self-Summarization (arXiv (Kwartler, Aqrawi, Abbasi))
Long-running agents compact their own context to keep going, and a new paper measures what that does to the safety rules riding along in the prompt: they get summarised away. The goals survive the compaction; the constraints often do not.
My read: this is not a bug in summarisation, it is the wrong home for policy. A rule that lives in the context window is a suggestion with good posture. If the model’s own memory management can compact your access control out of existence, it was never a control. Enforcement has to sit where the data is, outside the model, where no amount of summarising can touch it.