Gemini Cyber Test Breakout: Put Egress Outside the Agent's Control

What happened during the Gemini tests

Reuters reported that Irregular was testing Gemini's cyber capabilities in May 2026 when the model reached three real companies that it believed were in scope. The companies and Gemini version were not disclosed. Google vice-president of security engineering Heather Adkins confirmed the incidents to Reuters, said the affected companies were notified and said testing procedures were revised.

The Guardian reported that the test environment was intended to contain simulated companies but retained unintended internet access. One simulated target shared a name with a real organisation, while public information and credentials enabled access to two others. These were test-control failures, not evidence that Gemini independently chose to attack arbitrary companies.

Why a prompt is not a security boundary

Instructions, target lists and model refusals can reduce risk, but they remain decision-layer controls. They should not determine which networks, credentials or tools are technically reachable. A capable agent can misclassify an asset, follow an unexpected dependency or use information the test designer did not anticipate.

Google's Secure AI Framework treats agent security as a system problem that includes permissions, tools, data and monitoring. NIST's technical testing guidance similarly emphasises planning, controlled execution and documented assessment. The practical inference is that the test harness must enforce the rules even when the model misunderstands them.

Build a hard egress gate around cyber agents

Before giving an AI agent security-testing tools, require a containment design with controls that the agent cannot rewrite or bypass:

Any request to widen scope should require a separate human approval recorded outside the agent session. The control plane should default to deny, not attempt to infer whether a destination is probably legitimate.

• isolate the environment and deny public-internet egress by default;
• allowlist exact destinations, ports and protocols in an external gateway;
• use synthetic organisations, identities, data and credentials wherever possible;
• mediate DNS and HTTP traffic so lookalike names cannot silently resolve to real assets;
• set command, time and spend limits with an independent emergency stop;
• record immutable network, tool and credential logs outside the agent; and
• make scope expansion a two-person, time-limited approval.

Treat every breakout as a control event

If an agent contacts an unapproved asset, pause the test and preserve the full evidence chain: prompt, tool calls, DNS resolution, destination, credentials used, data touched, stop condition and human interventions. Notify affected organisations when required, rotate exposed credentials and update the threat model before resuming.

Measure containment separately from task success. Useful metrics include blocked out-of-scope connection attempts, time to detection, credential exposure, human stop latency and whether logs are complete enough to reconstruct the event. A high task-success score cannot compensate for a weak boundary.

What business leaders should do next

Ask security, engineering and vendors to show where scope is enforced. If the answer is a prompt, policy document or model judgement, the boundary is incomplete. Require a network diagram, identity model, allowlist owner, stop mechanism, test evidence and named approver before any autonomous cyber workflow reaches production-connected infrastructure.

ELYMENT AI helps organisations turn AI capabilities into governed operating systems. Start by making the external control plane testable: the agent may recommend an action, but infrastructure must decide what it can actually reach.

Sources

• Reuters, Gemini hacked three companies in first known breakout by Google's AI (18 September 2026) - Independent reporting on the May cyber tests, three real-company accesses, Google's confirmation, notifications and revised procedures.
• The Guardian, Google Gemini AI hacked real companies during safety test (18 September 2026) - Independent reporting on unintended internet access, the simulated-target naming collision and use of publicly available credentials.
• Google, Secure AI Framework (Accessed 21 September 2026) - Google's primary framework for mapping AI security risks to system-level controls, including agent permissions, tools, data and monitoring.
• NIST, Technical Guide to Information Security Testing and Assessment (Accessed 21 September 2026) - Primary US government guidance for planning, conducting and documenting controlled information-security testing and assessment.

Continue learning

• Build two evidence chains after an AI-agent breach
• Govern AI agents by capability, not label
• See the earlier agent-test containment incident and its control lessons

Frequently asked questions