AI Social Engineering: When AI Support Agents Become Confused Deputies

AI-powered support agents are changing how organizations interact with customers. They can reset passwords, update account information, verify identities, and perform actions that once required human support representatives.

As these agents become more capable, attackers are adapting their tactics.

Instead of exploiting software vulnerabilities, they manipulate the AI itself.

This emerging class of attacks, often referred to as AI social engineering or "vibe hacking," relies on conversation rather than code. Attackers build trust, appear cooperative, and gradually steer an AI agent toward performing privileged actions that should never have been authorized.

The result is an AI agent acting on behalf of an attacker, without ever realizing it.

The Confused Deputy Problem

Computer security has long recognized a class of vulnerabilities known as the Confused Deputy problem.

A deputy has authority to perform actions that an ordinary user cannot. If an attacker convinces the deputy to misuse those privileges, the system behaves exactly as designed, but produces an unauthorized outcome.

AI agents are becoming the modern version of that deputy.

A customer support agent may have permission to:

• Reset passwords
• Change account email addresses
• Update billing information
• Retrieve customer records
• Initiate account recovery workflows

The requester has none of these permissions directly.

Instead, they convince the AI agent to perform them.

Unlike traditional phishing, the attacker is not deceiving a human employee. They are manipulating an autonomous system that has access to sensitive business functions.

Why AI Social Engineering Works

Modern language models are designed to be helpful.

They resolve ambiguity, accommodate users, and complete tasks efficiently.

Those same characteristics make them vulnerable to manipulation.

A typical attack unfolds over several turns:

1. The attacker starts a routine support session.
2. They present a believable problem, such as losing access to an account.
3. They establish credibility through consistent, cooperative conversation.
4. They gradually introduce requests that require elevated privileges.
5. The AI performs an action that should have required stronger identity verification.

Each message appears harmless.

The attack only becomes apparent when viewed across the entire session.

Why Traditional AI Security Misses It

Many AI security products inspect prompts individually.

They look for jailbreaks, prompt injection, malicious instructions, or known attack signatures.

None of those signals may exist during an AI social engineering attack.

Every request appears reasonable.

The danger lies in how the conversation evolves.

Research from Cisco Foundation AI (2026) found that 64% of multi-turn attacks evade single-turn scanners entirely because malicious intent is distributed across multiple interactions rather than contained in a single prompt.

This creates a blind spot for stateless detection systems.

The attacker is not trying to bypass a filter.

They are trying to change the agent's decision making over time.

Identity Verification Is Not Enough

Many organizations assume strong authentication solves the problem.

It doesn't.

Even after identity verification, an AI agent must determine whether the requested action is consistent with its intended purpose.

Consider two examples:

Legitimate session

A verified customer requests a password reset after forgetting their credentials.

The agent follows an approved recovery workflow.

Everything aligns with the agent's purpose.

Malicious session

A requester gradually persuades the AI to replace the account's recovery email before initiating a password reset.

Each request appears reasonable in isolation.

Viewed together, the session is steering the agent toward an unauthorized account takeover.

The difference is not authentication.

The difference is intent.

AI Agents Need Session-Native Security

This is where traditional AI security reaches its limits.

Security decisions cannot be made from individual prompts.

They require understanding the complete session.

Teel Security continuously monitors every turn, tracking how goals evolve, how context changes, and whether a requester is attempting to redirect an AI agent beyond its intended purpose.

Rather than searching for forbidden phrases, Teel evaluates whether the interaction remains aligned with the agent's role.

If a routine customer support session begins evolving toward privileged account changes without sufficient justification, Teel detects the behavioral shift before the action is completed.

Example threat assessments might look like this:

By monitoring the full session instead of isolated prompts, Teel detects attacks that traditional filters never see.

AI Security Must Protect Decisions, Not Just Prompts

As AI agents gain access to enterprise systems, customer accounts, financial operations, and internal workflows, attackers will increasingly target the decision making process itself.

The next generation of AI attacks will not rely on malicious prompts.

They will rely on ordinary conversations that gradually redirect an AI agent toward unintended goals.

Organizations need security that understands not just what was said, but what the session is trying to accomplish.

That requires continuous monitoring, session-native intent analysis, and protection that follows every turn of the interaction.

Because the most dangerous AI attack is often the one that sounds completely legitimate.