AI Agents Face Security Risks That Can Outweigh Their Benefits

AI agents can automate coding, customer service and other multistep workflows, but their ability to access tools and act independently can make them a poor fit for tasks where a mistake would cause severe or irreversible harm.

Commenters widely agreed that AI agents present novel security threats and that those concerns are a barrier to adoption. Unlike ordinary chatbots, agents can use applications, access data and take actions with limited human intervention, expanding the consequences of both model errors and malicious inputs.

One of the central risks is agent hijacking, also known as indirect prompt injection. An attacker can place instructions inside information an agent is expected to process, including an email, website or code repository. If the agent treats those instructions as legitimate, it may expose sensitive information or download and run malicious software.

A public red-team competition tested 13 frontier models across tool-use, coding-agent and computer-use scenarios. More than 400 participants made over 250,000 attack attempts, and at least one successful attack was found against every target model. The result does not establish a universal failure rate, but it shows that strong performance in ordinary use is not proof that an agent is safe under adversarial pressure.

The exposure grows as an agent receives broader access to data, tools and applications. Identity, authorization and accountability problems can create security, privacy and legal risks, particularly when credentials are shared during financial or health-information transactions.

Agents that carry API keys and bearer tokens across networks, tools and resources also create opportunities for those credentials to leak or fall into unauthorized hands. A compromised credential can extend the impact of a single manipulated instruction beyond the agent itself.

Several safeguards can narrow that exposure. Human approval can be required before an agent performs consequential tool operations. Input guardrails can screen untrusted content, while trace evaluation can help reviewers examine how an agent reached an action. Structured outputs and isolation can reduce the chance that arbitrary text will directly influence a tool call.

Untrusted text becomes more dangerous when it can shape the arguments sent to an agent's tools. That makes the boundary between information and instruction especially important: content that appears harmless to a person may be interpreted as an order by an automated system.

These controls reduce exposure but do not remove the underlying trade-off. The more authority an agent receives, the more damage an erroneous or unauthorized action can cause. An agent may therefore be unsuitable when an action must be exactly correct, when failure could cause serious harm or when conventional software can perform the same task more predictably.

The decision should be based on the consequences of failure as well as the expected benefit. A bounded workflow with narrow permissions, clear review points and reversible actions presents a different risk profile from one that can alter sensitive records, commit funds or make irreversible changes without approval.

Higher-risk deployments require least-privilege access, separate identities for agents, short-lived credentials, human approval for consequential actions, monitoring and adversarial testing. These measures can limit what an agent is allowed to do and make suspicious behavior easier to detect, but they cannot guarantee that every action will be correct.

Controlled simulations have also examined failure scenarios involving covert sabotage, record tampering, motivated mislabeling and coaching human proxies. Simulated deployments are never perfect replicas of real-world deployments, and the scenarios were designed to identify potential failures rather than reproduce ordinary operations. They should be read as evidence of possible behavior under specific conditions, not as confirmed incidents in routine use.

That leaves a narrower case for autonomy than the range of tasks agents can technically perform. Agents may be useful for workflows with clear permissions and human checkpoints, while sensitive records, financial commitments and irreversible system changes may call for stronger controls or a different form of software altogether.