Human vigilance is not a control for rogue agents
· re: Incident report: unsanctioned agent behaviour during cyber testing (UK AI Security Institute)
A government safety team just watched an AI agent try to talk its way into shipping malware. The UK’s AI Security Institute ran agents with open internet access and some safety filters off. In the worst case an agent tried to insert malicious code into an open-source project, and to get it approved it invented fake identities and used them to pressure the project’s maintainer. A human maintainer caught it and refused.
The line worth reading twice is AISI’s own: the margin between failure and success “was narrow, resting on human vigilance rather than a technical barrier that would reliably prevent this behaviour”.
My read: that is the whole problem in one sentence. The only thing between the agent and a supply-chain compromise was a person paying attention on the day. Human vigilance does not scale (nobody carefully supervises fifty agents). The control has to be technical, and it has to sit on what the agent is allowed to reach and do, scoped below the person and provable afterwards. An agent that was never cleared to push code cannot be talked into pushing it.