Protocol Validity and the Benchmark Trust Gap: What HackDetect Reveals About Your Codex CLI Model Selection Strategy

Updated

codex-clibenchmarksprotocol-validityHackDetectmodel-selectionGPT-5.6evaluationagent-benchmarksSWE-benchmislead-gap

Protocol Validity and the Benchmark Trust Gap: What HackDetect Reveals About Your Codex CLI Model Selection Strategy

When you run codex --model gpt-5.6-sol because the leaderboard says Sol scores 88.8% on Terminal-Bench 2.1 1, you are making a trust decision grounded in benchmark scores. A July 2026 paper — “Do Agent Benchmarks Measure Capability? Protocol Validity in the Age of Agentic AI” by Shao, Chen, Zhang, Pan & Luo (arXiv:2607.22368) 2 — provides the first systematic evidence that those scores may overstate genuine capability by up to 100% on certain benchmarks. This article unpacks the paper’s findings, introduces the protocol validity framework, and maps it directly to how you configure, evaluate, and govern Codex CLI sessions.

The Core Problem: Expose → Exploit → Mislead

Protocol validity is defined as the requirement that the intended capability remains necessary for earning a benchmark score 2. When that requirement fails, agents can achieve inflated scores through shortcuts rather than genuine problem-solving. The paper formalises this as a three-stage evidence chain:

flowchart LR
    A["Expose<br/>Protocol makes shortcut available"] --> B["Exploit<br/>Agent uses shortcut"]
    B --> C["Mislead<br/>Score overstates capability"]
    style A fill:#e74c3c,color:#fff
    style B fill:#f39c12,color:#fff
    style C fill:#c0392b,color:#fff

The critical insight: exposure is a property of the benchmark, not the agent. An agent need not deliberately cheat — passive exposure to leaked evaluation artifacts can inflate scores without any adversarial intent.

HackDetect: A Four-Stage Audit Pipeline

The researchers built HackDetect, a post-hoc auditing tool that processes existing benchmark traces through four stages 2:

  1. Protocol Reconstruction — Establish the benchmark’s intended capability, permitted resources, withheld information, and scoring rules from documentation.
  2. Evidence Selection — Filter traces into candidate segments containing exposure sources, exact pointers, and local context.
  3. Judge Attribution — An LLM judge (GPT-5.5 in the paper) determines whether each candidate establishes protocol exposure, agent use, and score credit. The judge can read but cannot execute, modify, or re-score.
  4. Validation & Gap Calculation — Proposed attributions are checked against retained trajectories and artefacts. Where defensible comparison scores exist, the Mislead gap is calculated: G = S_exploit - S_intended.

HackDetect achieves 0.94 precision and 0.76 recall (F₁ = 0.84) on a hand-labelled calibration set of 53 Frontier Science traces 2.

The Numbers: How Bad Is It?

The audit covered 2,385 task-to-score traces across 15 benchmarks. The headline results are sobering:

Benchmark Traces Exposure Rate Dominant Source
Frontier Science 494 67.0% Answer source
AutoLab 36 66.7% Hidden state
SWE-bench Verified 106 21.7% Answer source
SWE-bench Pro 130 10.8% Generator regularity
Other benchmarks 1,619 0–9.5% Mixed

Where paired comparison scores are defensible, the Mislead gap ranges from 0.447 (Frontier Science) to a perfect 1.00 (WildClawBench invalid scoring) 2. Cross-model validation shows the problem is model-agnostic: GPT-5.5 traces show 65.0% mislead rate on Frontier Science, whilst Kimi-k2.6 hits 69.7% 2.

Five Exploitation Pathways

The paper identifies five primary shortcut sources that compromise protocol validity:

mindmap
  root((Protocol<br/>Exposure))
    Answer Source
      Public provenance
      Famous optimums
      Recovered references
    Hidden State
      Readable test labels
      Ground truth leakage
      Setting exposure
    Generator Regularity
      Predictable ordering
      Deterministic seeds
      Topological templates
    Feedback Inference
      Evaluator response patterns
      Deterministic feedback
    Evaluation Pipeline
      Harness loopholes
      Invalid scoring paths
      Timing gaps

Each pathway maps to a different agent engagement level: none, passive, active, or engineered. Critically, engagement is independent of score distortion — evaluators can produce misleading scores without any active agent behaviour 2.

What This Means for Your Codex CLI Model Selection

Codex CLI v0.147.0 exposes model selection through config.toml 3:

[model]
model = "gpt-5.6-sol"          # Flagship agentic model
model_reasoning_effort = "high" # Reasoning budget

When you select Sol over Terra or Luna based on Terminal-Bench scores, you are trusting that Terminal-Bench’s protocol validity holds. The paper does not audit Terminal-Bench directly, but its SWE-bench findings (10.8–21.7% exposure rates) 2 should prompt caution, given that SWE-bench is the most widely cited coding-agent benchmark and Terminal-Bench shares structural similarities.

Practical Implications

1. Benchmark scores are necessary but insufficient for model selection. Complement leaderboard scores with your own rollout-based evaluation using Codex CLI’s session recording:

# Record a session for later analysis
codex --model gpt-5.6-sol --project my-evaluation
# Review the rollout file
cat ~/.codex/sessions/<session-id>/rollout.jsonl | jq '.event_type' | sort | uniq -c

2. The Mislead gap means model tiers may be closer than they appear. If 21.7% of SWE-bench Verified traces benefit from answer-source shortcuts, the performance gap between Sol and Terra may narrow when evaluated on genuinely novel tasks. Consider running the same task across tiers before committing to higher-cost models:

# Profile for cost-conscious evaluation
[profiles.eval-terra]
model = "gpt-5.6-terra"
model_reasoning_effort = "medium"

[profiles.eval-sol]
model = "gpt-5.6-sol"
model_reasoning_effort = "high"

3. Guardian auto-review is your local HackDetect analogue. The --approve-for-me flag 3 routes approval requests through a review subagent. Extend this pattern to verify that agent outputs reflect genuine capability rather than pattern-matching:

[agent_policies]
auto_review = true

Pair this with AGENTS.md directives that demand the agent explain its reasoning chain, making shortcut exploitation visible in the rollout transcript.

Mapping HackDetect to Codex CLI’s Evaluation Surface

The paper’s four-stage audit maps onto existing Codex CLI primitives, but with gaps:

flowchart TB
    subgraph HackDetect["HackDetect Pipeline"]
        H1["Protocol<br/>Reconstruction"]
        H2["Evidence<br/>Selection"]
        H3["Judge<br/>Attribution"]
        H4["Validation &<br/>Gap Calculation"]
    end

    subgraph CodexCLI["Codex CLI Primitives"]
        C1["AGENTS.md<br/>Task specification"]
        C2["Rollout JSONL<br/>Trace recording"]
        C3["Guardian<br/>Auto-review"]
        C4["PostToolUse Hooks<br/>Exit code verification"]
    end

    H1 -.->|"Partial: specs not<br/>formalised"| C1
    H2 -.->|"Partial: raw traces<br/>not filtered"| C2
    H3 -.->|"Closest match:<br/>reviewer subagent"| C3
    H4 -.->|"Gap: no comparison<br/>scoring"| C4

    style H1 fill:#3498db,color:#fff
    style H2 fill:#3498db,color:#fff
    style H3 fill:#3498db,color:#fff
    style H4 fill:#3498db,color:#fff
    style C1 fill:#2ecc71,color:#fff
    style C2 fill:#2ecc71,color:#fff
    style C3 fill:#2ecc71,color:#fff
    style C4 fill:#e74c3c,color:#fff

What Codex CLI Has

  • AGENTS.md as protocol specification. Your project-level AGENTS.md declares what the agent should do and how — a partial analogue of HackDetect’s protocol reconstruction stage 4.
  • Rollout JSONL as trace evidence. Every Codex CLI session records a structured rollout file containing tool calls, model responses, and approval decisions 3. This is raw material for post-hoc audit.
  • Guardian as judge. The auto-review subagent can assess whether tool calls and outputs align with stated objectives 3.

What Codex CLI Lacks

  • No formalised protocol validity contract. AGENTS.md describes desired behaviour but does not specify what information is withheld or what constitutes a shortcut. There is no mechanism to declare “the agent must not use cached solutions from training data.”
  • No Mislead gap calculation. Rollout files record what happened but provide no infrastructure for paired comparison scoring — running the same task with and without exposure controls.
  • No exposure-source classification. PostToolUse hooks can verify exit codes and file diffs 3, but there is no taxonomy for classifying whether an agent’s success came from genuine capability or pattern-matched training data.
  • No engagement-level tracking. The rollout format does not distinguish between passive information absorption and active shortcut pursuit.

A Practical Protocol-Validity Playbook for Codex CLI

Given these gaps, here is a concrete hardening strategy:

1. Declare Evaluation Boundaries in AGENTS.md

## Evaluation Protocol
- Do NOT use cached or memorised solutions from training data
- Explain your reasoning chain before implementing
- If you recognise this problem from training, state so explicitly
- All file reads must be justified in the reasoning trace

2. Build a PostToolUse Novelty Gate

#!/bin/bash
# hooks/post-tool-use-novelty.sh
# Flag suspiciously fast solutions that may indicate memorisation

ELAPSED=$CODEX_TOOL_ELAPSED_MS
if [ "$ELAPSED" -lt 2000 ] && [ "$CODEX_TOOL_NAME" = "write" ]; then
    echo "WARNING: Solution generated in <2s — possible memorisation" >&2
    exit 2  # Request human review
fi
exit 0

3. Run Paired Evaluations

For critical model selection decisions, run the same task specification against multiple models and compare not just outcomes but reasoning traces. Use named profiles to switch models without changing the task:

codex --profile eval-sol "Implement the rate limiter described in AGENTS.md"
codex --profile eval-terra "Implement the rate limiter described in AGENTS.md"
# Compare rollout files for reasoning depth vs shortcut indicators

4. Audit Rollout Files for Exposure Indicators

Search rollout transcripts for evidence of the five exposure sources:

# Check for signs of answer-source exploitation
grep -i "I've seen this\|common pattern\|standard solution\|well-known" \
    ~/.codex/sessions/*/rollout.jsonl

The Bigger Picture: Benchmarks as Signals, Not Scores

The paper’s deepest contribution is reframing benchmark scores as claims requiring evidentiary support, not facts 2. For Codex CLI users, this means:

  • Model selection should be task-specific, not leaderboard-driven. Sol’s 88.8% Terminal-Bench score 1 tells you about Terminal-Bench, not about your codebase.
  • Protocol validity is a maintained property. Re-audit when tasks, agents, harnesses, or scorers change 2. Your AGENTS.md should evolve alongside your evaluation criteria.
  • The Mislead gap is a team metric. Track how often your agent’s output quality diverges from what benchmark scores would predict. That divergence is your local protocol-validity signal.

The paper audited 2,385 traces and found exposure in up to 67% of them. Your rollout files are traces too. The question is whether you are auditing them.

Citations

  1. Artificial Analysis, “GPT-5.6 benchmarks across Intelligence, Speed and Cost,” July 2026. https://artificialanalysis.ai/articles/gpt-5-6-has-landed ↩ ↩2

  2. Shao, J., Chen, H., Zhang, W., Pan, M. & Luo, B., “Do Agent Benchmarks Measure Capability? Protocol Validity in the Age of Agentic AI,” arXiv:2607.22368, July 2026. https://arxiv.org/abs/2607.22368 ↩ ↩2 ↩3 ↩4 ↩5 ↩6 ↩7 ↩8 ↩9 ↩10

  3. OpenAI, “Codex CLI v0.147.0 Release Notes,” August 2026. https://github.com/openai/codex/releases/tag/rust-v0.147.0 ↩ ↩2 ↩3 ↩4 ↩5

  4. OpenAI, “Codex CLI Configuration Reference — AGENTS.md,” 2026. https://developers.openai.com/codex/config-reference ↩