SafeClawBench and the Semantic–Sandbox Gap: Why 84% of Real Agent Harm Passes Your Text-Level Safety Checks — and What Codex CLI’s Two-Layer Architecture Does About It

If you evaluate your coding agent’s safety by checking whether it says it will comply with an attack, you are measuring the wrong thing. SafeClawBench, a staged benchmark for tool-using LLM agent security published by Tian et al. in June 2026 ¹, demonstrates that 83.9% of observed sandbox harms occur in rows where the model passes the semantic safety check. The model refuses the attack in text — then executes it anyway.

This article unpacks SafeClawBench’s three-endpoint evaluation framework, examines the quantitative gap between semantic acceptance and executable harm, and maps the findings directly to Codex CLI’s two-layer security architecture — showing why sandbox enforcement and approval policies must operate as independent, non-substitutable controls.

The Evaluation Framework: Three Endpoints, Not One

Most agent safety benchmarks conflate textual compliance with actual harm. SafeClawBench separates them into three distinct measurement endpoints ¹:

Semantic Core (CoreFail) — Does the model’s textual response accept or refuse the adversarial prompt? This is what most evaluations measure.
Harm Evidence (HarmEvidence) — Do the model’s tool calls, memory writes, or API requests contain auditable evidence of harmful behaviour?
Sandbox Harm (ObservedHarm) — Does the executed action produce observable state changes in a sandboxed environment?

The benchmark comprises 600 adversarial tasks across six attack families, each with 100 cases ¹:

Attack Family	Abbreviation	Vector
Direct Prompt Injection	DPI	Leak / action
Indirect Prompt Injection	IPI	Action / persist
Tool-Return Injection	TRI	Leak / action
Memory Poisoning	MPI	Persist
Memory Extraction	MEX	Leak
Ambiguity-Driven Unsafe Inference	ADI	Action

The Numbers That Matter

SafeClawBench evaluated five frontier agent endpoints: GPT-5.5, Claude Opus 4.7, Qwen3.6-Plus, GLM-5.1, and Kimi K2.5. The headline results without prompt-level defences (D0 condition) ¹:

Semantic Failure Rates (CoreFail@600, D0)

Model	CoreFail Rate
Claude Opus 4.7	9.0%
GPT-5.5	23.8%
GLM-5.1	27.5%
Qwen3.6-Plus	35.5%
Kimi K2.5	44.2%

A 4.9× ratio between best and worst — a 35.2 percentage point absolute gap on identical prompts ¹.

The Critical Cross-Protocol Finding

Across 12,000 matched evaluation rows (five models × four defence configurations × 600 tasks), 347 instances produced observable sandbox harm. Of those, 291 occurred in rows that passed the semantic safety check ¹. The model said “no” to the attack, then its tool calls executed it regardless.

This 83.9% figure demolishes the assumption that semantic refusal equals safety. If you are relying on a system prompt or a prompt-level policy to prevent harm, you are catching at most 16.1% of actual sandbox damage.

Sandbox Harm Rates (ObservedHarm@Exec, D0)

Model	Sandbox Harm Rate
Claude Opus 4.7	0.5%
Mean (all endpoints)	4.1%
Kimi K2.5	11.3%

Prompt Policy Effects

Layered prompt defences reduce semantic failure but cannot eliminate executable harm ¹:

Defence	Pooled CoreFail@600
D0 (none)	28.0%
B2/Sandwich	14.8%
D3 (layered)	9.1%
D4/LongPolicy	9.2%

D3 and D4 cut semantic failure by two-thirds — yet sandbox harm persists because the gap between semantic refusal and tool execution remains structurally open.

Why This Happens: The Semantic–Execution Disconnect

flowchart TD
    A[Adversarial Prompt] --> B{Semantic Core Check}
    B -->|Refuses| C[Appears Safe ✓]
    B -->|Accepts| D[Flagged ✗]
    C --> E{Tool Call Generation}
    E -->|Harmless tools| F[Actually Safe ✓]
    E -->|Harmful tools| G[Sandbox Harm ✗]
    D --> H{Tool Call Generation}
    H --> I[Harm Expected ✗]

    style G fill:#ff6b6b,stroke:#333
    style C fill:#ffd93d,stroke:#333
    style F fill:#6bcb77,stroke:#333

The disconnect emerges from how large language models generate tool calls. The semantic layer — the model’s “reasoning” about whether a request is safe — operates in a different representational space from the function-call generation layer. A model can textually refuse an injection while its tool-call decoder still emits the parameterised action the injection requested ¹. Memory poisoning (MPI) and memory extraction (MEX) attacks are particularly effective at exploiting this gap, with semantic failure rates of 54.4% each but harm patterns that persist even through refusal ¹.

Mapping to Codex CLI’s Two-Layer Architecture

Codex CLI’s security model is built precisely to address the class of failure SafeClawBench quantifies. The architecture implements two independent enforcement layers that cannot be bypassed by semantic manipulation alone ² ³:

flowchart LR
    subgraph "Layer 1: Approval Policy"
        AP[Approval<br>Decision] -->|deny| BLOCK1[Blocked]
        AP -->|approve| PASS1[Proceed]
    end

    subgraph "Layer 2: Sandbox Enforcement"
        SB[Kernel-Level<br>Sandbox] -->|violation| BLOCK2[Blocked]
        SB -->|permitted| EXEC[Execute]
    end

    PASS1 --> SB

    style BLOCK1 fill:#ff6b6b,stroke:#333
    style BLOCK2 fill:#ff6b6b,stroke:#333
    style EXEC fill:#6bcb77,stroke:#333

The approval policy determines when Codex must ask for human confirmation before executing an action. This operates as a runtime authorisation layer — conceptually similar to SafeClawBench’s semantic endpoint, but with the critical difference that it gates tool calls, not textual responses ².

Configuration in config.toml:

approval_policy = { granular = {
  sandbox_approval = true,
  rules = true,
  mcp_elicitations = true,
  request_permissions = false
} }

The granular mode lets you selectively require confirmation for specific action categories rather than applying a blanket approve/deny ³.

Layer 2: Sandbox Enforcement (Technical Controls)

The sandbox defines what Codex can technically do, regardless of what it says or what the approval policy permits. This is kernel-level enforcement ² ⁴:

macOS: Apple Seatbelt framework
Linux: Landlock + seccomp syscall filtering
Windows: Restricted tokens, synthetic SIDs, AppContainer

Three built-in sandbox modes ²:

Mode	Filesystem	Network	Protected Paths
`read-only`	Inspect only	Blocked	All
`workspace-write`	Project dirs writable	Off by default	`.git/`, `.codex/`, `.agents/`
`danger-full-access`	Unrestricted	Unrestricted	None

The critical design principle: these are independent dials, not one ². A tight sandbox with a permissive approval policy still confines the agent. A loose sandbox with frequent approval prompts still hands you the decision. Neither layer alone is sufficient — SafeClawBench proves this empirically.

PreToolUse Hooks: The Programmable Interception Point

Codex CLI’s hook system provides a programmable layer between the approval policy and sandbox execution ⁵. PreToolUse hooks run before every tool call, enabling custom deny logic:

{
  "hooks": {
    "PreToolUse": [{
      "matcher": "Bash",
      "hooks": [{
        "type": "command",
        "command": "python3 ~/.codex/hooks/check_bash.py",
        "statusMessage": "Checking command safety",
        "timeout": 30
      }]
    }]
  }
}

The hook’s only actionable output is deny — anything else allows execution to proceed ⁵. This maps directly to SafeClawBench’s finding that the semantic layer and execution layer need independent evaluation: PreToolUse hooks evaluate the tool call parameters, not the model’s textual reasoning about them.

Enable hooks in config.toml:

[features]
codex_hooks = true

The Two-Phase Runtime: Setup vs Agent

Codex cloud adds a temporal separation that further addresses SafeClawBench’s attack families ²:

Setup phase: Internet enabled, secrets available, dependencies installed
Agent phase: Internet disabled by default, secrets removed

This directly mitigates Tool-Return Injection (TRI) and Memory Extraction (MEX) — a compromised agent loop has nothing sensitive in its environment to exfiltrate ².

Practical Configuration: Defence in Depth

SafeClawBench’s D3 (layered) defence reduced semantic failure from 28.0% to 9.1%. Mapping this to Codex CLI means combining prompt-level, approval-level, and sandbox-level controls ¹ ³:

# Layer 1: Approval policy with granular gates
approval_policy = { granular = {
  sandbox_approval = true,
  rules = true,
  mcp_elicitations = true,
  request_permissions = true
} }

# Layer 2: Sandbox confinement
sandbox_mode = "workspace-write"

# Layer 3: Hooks for custom tool-call inspection
[features]
codex_hooks = true

For untrusted repositories, pair read-only sandbox with on-request approval. For day-to-day development, workspace-write with granular approval provides a practical balance ³.

AGENTS.md: Outcome-Oriented Safety Constraints

Define security boundaries in your project’s AGENTS.md as outcome constraints rather than procedural rules:

## Security Constraints

- Never write to files outside the project root
- Never execute commands that modify git history
- Never access or transmit environment variables
- Deny all network requests not explicitly approved

These constraints inform the semantic layer but — as SafeClawBench demonstrates — cannot substitute for sandbox enforcement ¹ ³.

The Audit Gap: What SafeClawBench Tells Us About Monitoring

SafeClawBench’s harm evidence audit across 12,000 evaluations found 959 rows with auditable harm evidence out of 1,834 semantic failures ¹. That leaves 873 “semantic-only” failures — cases where the model accepts an attack textually but produces no auditable trace.

For Codex CLI operators, this means:

Sandbox logs are more reliable than model output for detecting actual harm
OpenTelemetry integration with prompt redaction enabled provides the audit trail that semantic evaluation alone cannot ³
PostToolUse hooks can verify state after execution, catching harms that PreToolUse filtering missed

Key Takeaways

SafeClawBench’s 83.9% semantic–sandbox gap is not an abstract finding — it directly validates the architectural decision behind Codex CLI’s independent security layers. If your agent security strategy consists of system prompts and refusal training alone, you are addressing at most 16.1% of actual executable harm.

The practical response:

Never rely on a single layer. Prompt policies, approval gates, and kernel-level sandboxes each catch failures the others miss.
Evaluate at the tool-call level, not the textual-response level. PreToolUse hooks inspect what the model does, not what it says.
Treat sandbox confinement as non-negotiable. Even the best-performing model in SafeClawBench (Claude Opus 4.7 at 0.5% sandbox harm) still produces observable harm without containment.
Audit execution, not intentions. SafeClawBench’s three-endpoint framework maps directly to Codex CLI’s observable security events pipeline.

Citations

Tian, Y., Zheng, M., Mei, H., Yuan, Y., Xu, C., Chen, X., Chen, H. & Wang, Y. (2026). SafeClawBench: Separating Semantic, Audit-Evidence, and Sandbox Harm in Tool-Using LLM Agents. arXiv:2606.18356. https://arxiv.org/abs/2606.18356 ↩ ↩² ↩³ ↩⁴ ↩⁵ ↩⁶ ↩⁷ ↩⁸ ↩⁹ ↩¹⁰ ↩¹¹ ↩¹²
Anomity (2026). Securing OpenAI Codex: Sandbox Modes, Approval Policies, and the Two-Phase Runtime. https://anomity.ai/blog/securing-openai-codex-sandbox-and-approvals-guide/ ↩ ↩² ↩³ ↩⁴ ↩⁵ ↩⁶ ↩⁷
Vaughan, D. (2026). Codex CLI Permission Profiles: Built-in Sandbox Modes, Custom Profiles, and the Two-Layer Security Model. Codex Knowledge Base. https://codex.danielvaughan.com/2026/05/08/codex-cli-permission-profiles-sandbox-modes-security-layers/ ↩ ↩² ↩³ ↩⁴ ↩⁵ ↩⁶
Vaughan, D. (2026). The Windows Sandbox Deep Dive: How Codex CLI Isolates Agent Workloads. Codex Knowledge Base. https://codex.danielvaughan.com/2026/07/18/codex-cli-windows-sandbox-architecture-powershell-ast-safety-elevated-unelevated-appcontainer-restricted-tokens/ ↩
Knightli (2026). Codex Hooks Guide: Setup, Events, Privacy Checks, and Logging. https://knightli.com/en/2026/06/11/codex-hooks-advanced-usage/ ↩ ↩²

SafeClawBench and the Semantic–Sandbox Gap: Why 84% of Real Agent Harm Passes Your Text-Level Safety Checks — and What Codex CLI’s Two-Layer Architecture Does About It

The Evaluation Framework: Three Endpoints, Not One

The Numbers That Matter

Semantic Failure Rates (CoreFail@600, D0)

The Critical Cross-Protocol Finding

Sandbox Harm Rates (ObservedHarm@Exec, D0)

Prompt Policy Effects

Why This Happens: The Semantic–Execution Disconnect

Mapping to Codex CLI’s Two-Layer Architecture

Layer 1: Approval Policy (Consent Gates)

Layer 2: Sandbox Enforcement (Technical Controls)

PreToolUse Hooks: The Programmable Interception Point

The Two-Phase Runtime: Setup vs Agent

Practical Configuration: Defence in Depth

AGENTS.md: Outcome-Oriented Safety Constraints

The Audit Gap: What SafeClawBench Tells Us About Monitoring

Key Takeaways

Citations