Skip to content

overview.mdx

Latest commit

 

History

History
203 lines (144 loc) · 8.94 KB

overview.mdx

File metadata and controls

203 lines (144 loc) · 8.94 KB
title The Problem
description AI agents execute actions with real-world consequences. Traditional security architectures weren't designed for this.

From Text to Actions

Large language models began as text generators. Security meant filtering harmful outputs—profanity, misinformation, toxic content. If the model said something wrong, you could discard it before displaying.

That model is obsolete.

Today's AI systems do things:

  • Query databases and modify records
  • Send emails and Slack messages
  • Create, edit, and delete files
  • Execute code and shell commands
  • Call external APIs and webhooks
  • Manage cloud infrastructure
  • Process financial transactions

When an AI agent executes DROP TABLE customers, no output filter helps. The data is gone.

The Runtime Security Gap

AI-driven actions have five characteristics that break traditional security:

Unlike text generation—which can be filtered before display—tool executions produce immediate, permanent effects. Database mutations, sent emails, financial transfers, credential changes. 
Once executed, the damage is done.

    # This can't be "filtered"
    agent.execute("rm -rf /important/data")
Agents execute hundreds of actions per minute. No human can review them in real time. 
| Actor | Actions per minute |
|-------|-------------------|
| Human operator | 2–5 |
| AI agent | 100–500 |
| Human reviewer capacity | 5–10 |

Security decisions must be **automated and instantaneous**.
Individual actions may each satisfy policy. Their composition violates it. ``` Action 1: db.query("SELECT * FROM customers") → ALLOW (user has read access) 
Action 2: email.send(to="external@partner.com", body=results)  
  → ALLOW (user can send email)

Composition: PII exfiltrated to external party  
  → VIOLATION

    
    Traditional security evaluates actions in isolation. Context-aware evaluation catches what isolated checks miss.
  </Accordion>

  <Accordion title="Untrusted Orchestration" icon="skull">
    Prompt injection, jailbreaks, and indirect attacks mean the model's "intent" cannot be trusted.
    
    The agent might be:
    - Following malicious instructions embedded in a document
    - Manipulated by a crafted error message
    - Deceived about what action it's actually taking
    
    **The AI layer must be treated as potentially compromised.**
  </Accordion>

  <Accordion title="Privilege Amplification" icon="arrow-up-right-dots">
    Autonomous agents routinely operate under static, high-privilege identities misaligned with the principle of least privilege. A calendar integration granted full Google Workspace admin access. A database helper with `DROP` permissions.
    
    Small reasoning failures produce large-scale impact when the agent's execution identity is too powerful. A single prompt injection can exploit every permission the agent holds—not just the ones relevant to the current task.
  </Accordion>
</AccordionGroup>

---

## Why Existing Security Fails

<CardGroup cols={2}>
  <Card title="SIEM" icon="magnifying-glass">
    **Built for:** Event analysis and correlation
    
    **Failure mode:** Observes *after* execution. By the time SIEM alerts, the database is dropped.
    
    SIEM answers "what happened?"—but can't prevent it.
  </Card>

  <Card title="API Gateways" icon="door-open">
    **Built for:** Authentication, rate limiting, routing
    
    **Failure mode:** Verifies *who* is calling, not *what* the action means. A valid token making a destructive call passes through.
    
    Gateway sees credentials, not intent.
  </Card>

  <Card title="Firewalls" icon="shield-halved">
    **Built for:** Network perimeter defense
    
    **Failure mode:** Agents operate *inside* the perimeter with legitimate credentials. The call comes from an authorized service.
    
    The threat is already inside.
  </Card>

  <Card title="Prompt Guardrails" icon="comment-slash">
    **Built for:** Filtering model outputs
    
    **Failure mode:** Filters text, not actions. Easily bypassed. Can't evaluate whether `db.execute(query)` is safe—only whether the *text describing it* looks harmful.
    
    Guardrails see words, not operations.
  </Card>

  <Card title="IAM / RBAC" icon="user-lock">
    **Built for:** Identity and access management
    
    **Failure mode:** Evaluates permissions in isolation. User *can* read customers. User *can* send email. System doesn't know doing both in sequence is exfiltration.
    
    Permissions don't understand composition.
  </Card>

  <Card title="Human-in-the-Loop" icon="user-check">
    **Built for:** Manual approval of sensitive actions
    
    **Failure mode:** Doesn't scale to agent speed. Leads to rubber-stamping. [Can itself be exploited](https://checkmarx.com/blog/lies-in-the-loop) through forged approval dialogs.
    
    Humans become the bottleneck—or the vulnerability.
  </Card>
</CardGroup>

### Automated vs. Autonomous Response

These existing systems share a fundamental limitation: they support **automated response**—deterministic enforcement of pre-specified rules. A firewall blocks a port. An API gateway rejects an expired token. A SIEM triggers an alert when a pattern matches.

What they do not support is **autonomous response**—reasoning over accumulated context to decide whether an action should be permitted, modified, delayed, or blocked. Determining whether "send email to external recipient" is safe requires knowing what data the agent accessed, what the user originally asked for, and whether the action aligns with that intent. No static rule captures this.

**The gap** lies at the intersection of **prevention** and **context-awareness**: systems that can block actions before execution based on accumulated session context, not just static policy.

---

## The Missing Layer

There's a gap in the security stack:

┌─────────────────────────────────────────────────────────────┐ │ EXISTING SECURITY │ ├─────────────────────────────────────────────────────────────┤ │ Perimeter │ Network firewalls, WAF │ │ Identity │ IAM, RBAC, OAuth │ │ Application │ Input validation, output filtering │ │ Data │ Encryption, DLP │ │ Observability │ SIEM, logging, monitoring │ ├─────────────────────────────────────────────────────────────┤ │ ??? GAP ??? │ │ │ │ Where do you enforce policy on AI-initiated actions │ │ BEFORE they execute, with context-aware evaluation? │ │ │ ├─────────────────────────────────────────────────────────────┤ │ TOOLS / APIS │ │ Databases, email, files, cloud, external services │ └─────────────────────────────────────────────────────────────┘


This gap is where AARM operates: **runtime enforcement at the action boundary**.

---

## What's Needed

A security system that:

| Requirement | Why |
|-------------|-----|
| **Intercepts before execution** | Prevention, not just detection |
| **Evaluates action semantics** | Understands *what* is happening, not just *who* |
| **Tracks accumulated context** | Detects multi-action violations and intent drift |
| **Enforces policy inline** | Allow, deny, modify, defer, or escalate in real time |
| **Handles uncertainty** | Defers actions when context is insufficient for a confident decision |
| **Treats agent as untrusted** | Assumes orchestration may be compromised |
| **Enforces least privilege** | Scopes credentials to the specific operation, not standing permissions |
| **Creates forensic trail** | Tamper-evident record of every decision |
| **Scales to agent speed** | Automated, millisecond decisions |

No existing security tool does all of this.

---

## Next

<Card title="What is AARM?" icon="shield-check" href="/definition">
  How AARM fills the runtime security gap
</Card>