PatchFusion and the Pass@k-to-Pass@1 Problem: Why Your Codex CLI Needs a Patch Selection Strategy

Your coding agent can solve the bug. The question is whether it will solve the bug this time. The reliability gap — the distance between what an agent can do (pass@k) and what it does do on a single shot (pass@1) — is the defining operational problem for anyone running Codex CLI in production pipelines. Three recent research threads converge on a single practical insight: generating multiple candidate patches is cheap; selecting the right one is the hard part.

The Pass@k-to-Pass@1 Gap

Every SWE-bench leaderboard entry reports pass@1: one attempt, one patch, pass or fail. But the same agents evaluated at pass@5 or pass@10 show dramatically higher ceiling performance¹. Jha et al. (2026) formalise this as the reliability gap — the distance between capability (“can succeed once in k tries”) and consistency (“usually succeeds on the first try”)².

The gap is not academic. If your CI pipeline dispatches a Codex CLI cloud task to fix a failing test, and the agent generates a semantically incorrect patch 20% of the time — even when a correct patch exists within its sampling distribution — you have a production problem that no amount of prompt tuning will eliminate³.

graph LR
    A[Issue / Failing Test] --> B[Generate N Candidates]
    B --> C1[Candidate 1]
    B --> C2[Candidate 2]
    B --> C3[Candidate 3]
    C1 --> D[Selection Strategy]
    C2 --> D
    C3 --> D
    D --> E[Final Patch]
    E --> F[PR / Merge]

PatchFusion: Deterministic Selection Without Tests

Yang et al. (July 2026) introduce PatchFusion, a deterministic approach to patch selection that operates without executing test suites at decision time⁴. The core mechanism:

Repair neighbourhood construction — whole-diff agreement across candidates identifies a cluster of patches that modify the same code regions in similar ways.
Representative selection — an auditable representative is chosen from the neighbourhood.
Evidence-constrained fusion (ECF) — shared atomic edits (individual hunks, line changes) are retained; unsupported modifications are pruned.

The key insight is that when multiple independent generation attempts agree on a specific edit atom, that convergence constitutes evidence of correctness — without needing to run the patch against a test suite.

Results on PatchFuseBench

Benchmark	PatchFusion	Best Single Selector	Candidate Ceiling
SWE-bench Verified	426/500	—	96.2% of ceiling⁴
SWE-bench Multilingual	236/300	—	89.7% of ceiling⁴
Defects4J	87/371	—	Outperforms all⁴

PatchFusion recovered 41 bugs on SWE-bench Verified and 27 on Multilingual that no single source solved individually⁴. The ablation shows ECF alone adds +5/+6/+9 solved bugs across the three benchmarks. Execution time: 3.28 ms per bug — two to three orders of magnitude below model-based selectors⁴.

Ensemble Patching in Practice: The Team Atlanta Evidence

Team Atlanta’s 2026 vulnerability patching evaluation provides complementary evidence from a production-adjacent setting³. They ran 10 agent configurations (combining Claude Code, Codex CLI, Gemini CLI, and Copilot with five frontier models) against 63 real crashes from the DARPA AIxCC competition:

Best single configuration: ~71% correct patches, ~80% semantic correctness
Ensemble of N=3: consistently outperformed individual agents
Model choice > framework choice: upgrading from Opus 4.5 to 4.6 had more impact than switching agent framework

Their taxonomy of 145 faulty patches identified five failure modes: altered functionality (55), symptomatic patching (38), weak guards (28), wrong API usage (15), and inappropriate mitigations (15)³. These are precisely the failure modes that ensemble selection can filter — because different agents fail in different ways.

SWE-Replay: Efficient Test-Time Scaling

Where PatchFusion operates on a pool of completed candidates, SWE-Replay (January 2026) optimises the generation phase itself⁵. Rather than sampling N independent trajectories from scratch, it maintains an archive of prior attempts and stochastically branches at critical intermediate steps:

Reduces cost by up to 17.4% while maintaining or improving performance by up to 3.8% on SWE-bench Verified⁵
Bypasses reliance on potentially miscalibrated value models
Generalises across agent scaffolds — including those that synthesise custom bash scripts as tools

The practical implication: you can get more diverse candidates for the same token budget by recycling partial trajectories rather than starting fresh each time.

Mapping to Codex CLI: The Multi-Attempt Stack

Codex CLI already provides the building blocks for multi-candidate workflows. The challenge is assembling them into a coherent selection pipeline.

The `--attempts` Flag (Cloud Tasks)

# Generate 3 independent solutions for the same task
codex cloud exec --env prod-env --attempts 3 "Fix the race condition in worker.go"

# Review each attempt's diff independently
codex cloud diff TASK_ID --attempt 1
codex cloud diff TASK_ID --attempt 2
codex cloud diff TASK_ID --attempt 3

The --attempts flag (1–4) triggers best-of-N execution: Codex spawns multiple independent agents on the same task and surfaces all results for review⁶. This is the simplest entry point, but selection remains manual.

Multi-Model Routing for Diversity

PatchFusion’s effectiveness depends on candidate diversity — patches generated by the same model with the same temperature tend to cluster. Named profiles enable cross-model generation:

# ~/.codex/repair-ensemble.config.toml
[profiles.attempt-opus]
model = "claude-opus-4.8"
model_provider = "anthropic"

[profiles.attempt-gpt]
model = "gpt-5.6-sol"
model_provider = "openai"

[profiles.attempt-gemini]
model = "gemini-3.1-pro"
model_provider = "google"

# Generate diverse candidates via different models
for profile in attempt-opus attempt-gpt attempt-gemini; do
  codex exec -p "$profile" \
    --output-schema ./patch-schema.json \
    "Fix issue #1234. Output only the unified diff." \
    > "candidate-${profile}.patch" &
done
wait

PostToolUse Hooks for Automated Validation

Before selection, each candidate needs basic validation. A PostToolUse hook can gate patches automatically:

# .codex/config.toml
[[hooks]]
event = "PostToolUse"
tool = "apply_patch"
command = "scripts/validate-patch.sh"

#!/bin/bash
# scripts/validate-patch.sh
# Run fast checks: syntax, type checking, affected test subset
set -e
cargo check 2>/dev/null || npm run typecheck 2>/dev/null || true
git diff --stat HEAD | grep -q "." || exit 1  # Ensure non-empty diff

Implementing PatchFusion-Style Selection

The deterministic fusion approach translates directly to a post-processing script:

#!/bin/bash
# scripts/patch-select.sh — Lightweight atomic-edit voting
# Collects hunks from N candidates, selects by consensus

candidates=(candidate-*.patch)
declare -A hunk_votes

# Extract individual hunks and vote
for patch in "${candidates[@]}"; do
  while IFS= read -r hunk; do
    key=$(echo "$hunk" | sha256sum | cut -d' ' -f1)
    hunk_votes["$key"]=$(( ${hunk_votes["$key"]:-0} + 1 ))
  done < <(grep -A999 "^@@" "$patch" | csplit -z -f /tmp/hunk - '/^@@/' '{*}' 2>/dev/null)
done

# Retain hunks with majority agreement (≥ ceil(N/2) votes)
threshold=$(( (${#candidates[@]} + 1) / 2 ))
# ... assemble final patch from majority hunks

⚠️ A production implementation would need proper unified-diff parsing rather than this sketch. The PatchFusion paper’s actual algorithm handles hunk boundaries, context lines, and overlapping edits formally.

AGENTS.md Specification for Ensemble Workflows

## Patch Generation Protocol

When fixing bugs or implementing features:
1. Generate the fix independently — do not reference prior attempts
2. Output a clean unified diff via `git diff`
3. Include a one-line rationale for each hunk
4. If uncertain between two approaches, prefer the minimal change

## Selection Criteria (for ensemble review)
- Prefer patches with fewer modified files
- Prefer patches that modify only the module containing the bug
- Reject patches that suppress errors without addressing root cause

The Cost Arithmetic

The economics of multi-attempt generation versus single-shot repair are counterintuitive:

Strategy	Token Cost	Success Rate	Cost per Successful Fix
Single attempt (pass@1)	1×	~52–71%³	1.4–1.9× base
3 attempts + manual select	3×	~85–90%³	3.3–3.5× base
3 attempts + PatchFusion	3× + 3.28ms	~90–96%⁴	3.1–3.3× base
SWE-Replay (3 budget)	~2.5×⁵	~87–92%	2.7–2.9× base

The break-even calculation: if a failed patch costs you a review cycle (20 minutes of senior developer time), three attempts with automated selection pays for itself whenever your single-shot failure rate exceeds ~10%.

The Semantic Incorrectness Problem

Raw pass rates undercount failures. Team Atlanta found that ~20% of patches passing automated validation contained semantic defects — fixes that mask the symptom without addressing the root cause³. PatchFusion’s atomic-edit voting partially addresses this: symptomatic patches tend to be idiosyncratic (each agent suppresses the error differently), while correct patches converge on the same structural change.

flowchart TD
    A[N Candidate Patches] --> B{Atomic Edit Voting}
    B -->|Consensus edits| C[Fused Patch]
    B -->|Idiosyncratic edits| D[Pruned - likely symptomatic]
    C --> E[PostToolUse Validation]
    E -->|Pass| F[PR with provenance metadata]
    E -->|Fail| G[Escalate to human review]

This maps to Codex CLI’s Guardian auto-review subagent: configure it to flag patches where the fused result differs significantly from any individual candidate, as divergence signals uncertainty.

Practical Recommendations

Start with --attempts 3 for any CI-triggered repair task. The cost is 3× tokens; the reliability improvement is disproportionate.
Diversify models, not temperatures. Cross-model candidates (GPT-5.6 + Claude Opus 4.8 + Gemini 3.1 Pro) produce more structurally diverse patches than the same model at different temperatures³.
Implement atomic-edit voting as a PostToolUse hook or CI step. Even a naive majority-hunk filter catches the most common failure mode (symptomatic patching).
Track your reliability gap. Log pass@1 and pass@3 for your actual task distribution. If the gap exceeds 15 percentage points, ensemble selection will pay for itself.
Use codex exec --output-schema to enforce structured diff output, making automated parsing reliable across attempts.

Citations

Jha, S. et al. “The Reliability Gap: Agent Benchmarks for Enterprise.” simmering.dev, 2026. https://simmering.dev/blog/agent-benchmarks/ ↩
Runloop. “I have Opinions on Pass@K - You should too.” runloop.ai, 2026. https://runloop.ai/blog/i-have-opinions-on-pass-k-you-should-too ↩
Team Atlanta. “Patching Vulnerabilities with Coding Agents in 2026.” team-atlanta.github.io, 2026. https://team-atlanta.github.io/blog/post-patch-2026-ensemble/ ↩ ↩² ↩³ ↩⁴ ↩⁵ ↩⁶ ↩⁷
Yang, B., Hu, X., Ren, L., Chen, Y., Le, B., Bissyandé, T.F., Tian, H. “A Single Patch Is Not Enough: Deterministic Fusion of Repair Candidates.” arXiv:2607.01597, July 2026. https://arxiv.org/abs/2607.01597 ↩ ↩² ↩³ ↩⁴ ↩⁵ ↩⁶ ↩⁷
SWE-Replay authors. “SWE-Replay: Efficient Test-Time Scaling for Software Engineering Agents.” arXiv:2601.22129, January 2026. https://arxiv.org/abs/2601.22129 ↩ ↩² ↩³
OpenAI. “Codex Cloud.” developers.openai.com, 2026. https://developers.openai.com/codex/cloud ↩

PatchFusion and the Pass@k-to-Pass@1 Problem: Why Your Codex CLI Needs a Patch Selection Strategy

The Pass@k-to-Pass@1 Gap

PatchFusion: Deterministic Selection Without Tests

Results on PatchFuseBench

Ensemble Patching in Practice: The Team Atlanta Evidence

SWE-Replay: Efficient Test-Time Scaling

Mapping to Codex CLI: The Multi-Attempt Stack

The --attempts Flag (Cloud Tasks)

Multi-Model Routing for Diversity

PostToolUse Hooks for Automated Validation

Implementing PatchFusion-Style Selection

AGENTS.md Specification for Ensemble Workflows

The Cost Arithmetic

The Semantic Incorrectness Problem

Practical Recommendations

Citations

The `--attempts` Flag (Cloud Tasks)