brianletort.ai
All issues

Agent Techniques Weekly

Issue 17 · Week 33 of 2026.

/From Chat to Cowork to Build to Automate/Public sources only

Big Read

Human approval finally got measured, and it caught 13.6% of dangerous commands — the control most enterprises built their agent policy on is the one that just failed its first published test

Almost every agent governance policy written in the last two years rests on the same load-bearing assumption: a human reviews each consequential action before it executes. Approve or deny. It is the control that gets an agent pilot through risk review, and until this week nobody had published a number for how well it works. Anthropic published one on the way to making auto mode the default in Claude Code for Pro, Max and Team users on August 14. In a study the company says covered 1,053 testers, humans manually reviewing commands blocked 13.6% of dangerous ones. The automated classifier blocked 89%. This is vendor-reported and the vendor has an obvious interest in the result, so hold it loosely. But the method is disclosed, the sample is large, and the finding matches what anyone who has watched an engineer click through forty approval prompts already suspected. Approval prompts do not fail because people are careless. They fail because a control that fires on every action trains the person to dismiss it, and the dangerous action arrives looking exactly like the thirty-nine safe ones before it. The timing is what makes this the week's story rather than a footnote. Three days earlier SpaceXAI shipped Grok Bot, where agents run on a cloud Linux VM and operate real applications through the browser — and the vendor's own documentation notes that a user's bots share the computer and its sessions. That is credential pooling as a default, in a form factor where the agent authenticates as the user and the target application cannot tell the difference. Two days before that, Taiwan's digital ministry confirmed that July attacks on government websites used AI agents, including open agent frameworks. So in one week: the human-approval control was measured and found weak, agent reach expanded to any application with a login, and a government confirmed the offensive case is no longer hypothetical. The conclusion is not that autonomy is unsafe. It is that the safety control has to move from attention to environment. A human approving each action is an attentional control, and attention degrades with volume in a way that is entirely predictable. A sandbox, an outbound allowlist, a credential scoped to one task, a classifier in the execution path, and a verifiable checkpoint the agent cannot forge are environmental controls, and they do not get tired. The 89% figure is not evidence that classifiers are trustworthy; it is evidence that a control which runs on every action needs to be a control that does not fatigue. Keep the human, but spend their attention on the small number of irreversible decisions rather than on the stream. The rest of the week's tooling moved the same direction, which is why this reads as a shift rather than one vendor's product decision. GitHub shipped persistent memory for Copilot in JetBrains and shipped the managed settings to lock MCP servers and plugin marketplaces in the same release. Agent Plugins 1.0 reached general availability across VS Code, the Copilot CLI and the Copilot app, governed through existing managed settings and MCP allowlists rather than a new console. Two academic papers landed on the same theme from opposite ends: FlowScout mines historical execution logs into reusable workflow graphs so agents follow a verified path instead of improvising one, and MAP-Graph proposes permission, path-trust and action gating as runtime properties of shared agent memory rather than as audit performed afterwards. Every one of those is environmental. One caution on the evidence, because it is the recurring weakness in this publication's beat. Nothing this week cleared the bar of an independently measured production deployment with a named customer and a stated baseline. The Anthropic study has a method and a sample but is vendor-run. OpenAI's Ultrafast preview reports a security investigation dropping from one to two hours down to ten to fifteen minutes, which is a compelling number with no disclosed methodology attached and no published price or availability date. The Grok Bot efficiency quotes have neither. The academic results are real measurements on synthetic and benchmark tasks, not field deployments. Treat the direction as well supported and the magnitudes as unverified, and do not let a vendor's study — even a well-constructed one — substitute for measuring your own approval-catch rate. That number is cheap to collect and you almost certainly do not have it.

Technique of the Week

automate/Anthropic, GitHub Changelog, arXiv, Help Net Security

Environmental gating: replace per-action approval prompts with controls that cannot fatigue

Instead of asking a human to approve each agent action, constrain what the agent is capable of doing at all — through sandbox boundaries, scoped credentials, outbound allowlists, an automated classifier in the execution path, and verifiable checkpoints — and reserve human attention for the small set of irreversible decisions. The technique treats approval fatigue as a predictable property of high-volume controls rather than a training problem, and moves the safety guarantee from what a person notices to what the environment permits.

The first published measurement of human approval performance puts it at 13.6% of dangerous commands blocked against 89% for an automated classifier, on a vendor-run study of 1,053 testers. If that figure is even directionally right, most enterprise agent policies are resting on a control that does not carry the weight assigned to it — and the fix is not more training or better prompts, because the failure mode scales with the number of approvals rather than with the competence of the approver. Engineering and security leaders should audit their own approval-catch rate before the next autonomy expansion, then rebuild the policy so the environment enforces what the prompt was supposed to.

Scope the blast radius before granting autonomy

Give the agent a sandboxed working copy, a credential scoped to one task rather than a shared session, and an outbound network allowlist. This is the step most often skipped because it is infrastructure work rather than policy work, and it is the only one that holds when the approval prompt is dismissed. Note the counter-example from this week: a computer-use agent whose bots share one VM and its sessions has no credential boundary at all, so a bot name cannot be treated as an isolation unit.

Put an automated classifier in the execution path, not beside it

A classifier that evaluates each command before execution runs at the same volume as the agent and does not degrade with repetition. The published 89% catch rate is not a claim that classifiers are reliable in absolute terms — it is a claim that a non-fatiguing control at high volume beats a fatiguing one. Log every classifier decision including the allows, because the allow stream is what you need to tune it and the only record of what the agent was permitted to do.

Reserve human approval for irreversibility, not for risk

Route to a human only where the action cannot be undone: production data deletion, outbound communication to a customer, payment, credential rotation, publishing. Everything reversible should be allowed and logged rather than approved. This cuts approval volume enough that the remaining prompts carry real signal, which is the mechanism that makes the human control work again rather than a hope that people will pay closer attention.

Constrain the path, not just the action

Where historical execution logs exist, mine them into a verified workflow the agent follows rather than letting it improvise a tool sequence each run. Freezing a known-good path removes an entire class of failure before any approval logic runs, and it makes deviations detectable as deviations. Where agents share memory, apply permission and path-trust as runtime gates on what one agent may act on from another rather than as an audit performed after the fact.

Require a checkpoint the agent cannot forge

A test suite, a type check, a diff review by a separate model, or a signed build artifact. The agent asserting it finished is not evidence that it finished. This matters more as models get tuned for long-horizon autonomy, because a model optimized to complete multi-step tasks is also optimized to report completion, and self-reported success is the one signal that gets less reliable as autonomy increases.

Instrument your own catch rate and review it monthly

Sample approved actions, replay them against a classifier, and measure how many dangerous ones a human waved through. This is inexpensive, it is the only version of the vendor's finding that applies to your environment, and almost no organization currently collects it. Without it you are setting an autonomy default on someone else's study.

New Agent Capabilities.

Anthropic / automate

Claude Code auto mode

Auto mode becomes the default for Pro, Max and Team users, with an automated classifier evaluating commands in the execution path; Enterprise remains opt-in so administrators choose the default themselves. Classifier overhead is no longer billed on those tiers.

Engineering leaders should set the organizational default deliberately in managed settings before the Enterprise flip rather than inheriting it, and should pair auto mode with repository sandboxing and secrets hooks on the same change. The supporting study — 1,053 testers, humans blocking 13.6% of dangerous commands against the classifier's 89% — is vendor-run, so use it to justify instrumenting your own catch rate rather than as a substitute for having one.

SpaceXAI / automate

Grok Bot

Early-beta agents running on a cloud Linux VM that operate browsers and desktop applications directly, requiring no MCP server or API integration to reach a target system

Operators must assume credential pooling: the vendor's documentation notes that a user's bots share the computer and its sessions, so a bot name is not a security boundary and per-bot isolation cannot be assumed. Before any pilot against a system of record, answer which identity the agent authenticates as, whether credentials are scoped per task, and what the SIEM sees when it acts — because the target application sees a user, not an agent.

OpenAI / cowork

Ultrafast preview on Cerebras

Limited preview of GPT-5.6 Sol at up to roughly 14x speed and around 750 tokens per second, aimed at latency-sensitive agent loops; no public price and no general availability date

Teams running voice, incident response or live coding loops should join the waitlist and demand a priced SLA before redesigning any user experience around real-time frontier inference, because a preview without pricing cannot be planned against. The reported drop in a security investigation from one to two hours down to ten to fifteen minutes is vendor-claimed with no disclosed methodology, so treat it as a reason to test rather than a number to forecast with.

GitHub and Microsoft / build

Copilot for JetBrains persistent memory

Cross-session agent memory shipped together with enterprise managed settings that lock MCP servers, plugin marketplaces and permissions

Enable memory only with a retention and redaction policy already written, and lock the MCP and plugin surfaces through managed settings before agents gain cross-session recall — because memory turns a one-off prompt injection into a persistent one. The pattern worth demanding from every agent vendor is visible here: the governance control shipped in the same release as the capability that requires it, rather than a quarter later.

SpaceXAI / automate

Grok 4.6

A post-training release tuned for long-horizon agent trajectories through reinforcement learning inside agentic environments, measured at roughly 53 turns and 0.5B input tokens on long-horizon tasks against roughly 103 turns and 2.0B for the comparison frontier model

Prefer models trained for multi-step verify-and-retry loops when running overnight agents, and procure on turn count rather than token price — turns drive the bill on long-running work and do not reprice when a rate card changes. Its Terminal-Bench v3.0 result of 26% against 34.6% for the leading model is the gap that argues against unattended terminal work, and it is not closed by the efficiency gain.

New Skills And Connectors.

plugin / GitHub Copilot and the Agent Plugins coalition

Agent Plugins 1.0

The 1.0 specification, published August 6 with backing from AWS, Anysphere, Microsoft, OpenAI, Vercel and Google, reached general availability in VS Code, the Copilot CLI and the Copilot app

Platform teams should standardize on one plugin package per capability now and prohibit per-IDE forks while consolidation is still cheap. Govern it as software supply chain rather than as developer configuration: enterprise control runs through existing managed settings and MCP allowlists, which puts the review with whoever already owns endpoint policy. The breadth of signatories is what makes this worth standardizing on — it is the first agent extensibility format likely to outlive any single vendor's tooling.

harness / DeepSeek open tooling

DeepSeek Harness (dsh) v0.1

An MIT-licensed developer-preview agent harness built on an everything-is-a-plugin architecture, published with breaking changes expected

Worth tracking as open competition to the proprietary coding harnesses, and worth keeping out of production until it has sandboxing and plugin allowlisting you have verified yourself. A developer preview from a vendor that swapped a served model build without notice this same week is not a component to put on a critical path. Evaluate it in an isolated environment and treat the plugin surface as untrusted by default.

harness / Long-horizon agent research tooling

lh-harness v0.1.4

A web dashboard release productizing the monitor-evaluate-adjust loop from recent long-horizon agent research, bringing trajectory inspection to a running agent rather than to a log file after the fact

Teams running overnight or multi-day agents should adopt some form of live trajectory inspection, because the failure mode on long-horizon work is an agent confidently proceeding down a wrong path for hours. The specific tool is early and the underlying paper was submitted just before this window, so evaluate the pattern rather than committing to the implementation — the requirement is visibility into the trajectory while it runs, not this particular dashboard.

Proof Of Value.

Evidence: vendor_claim

Anthropic: Automated command review in a coding agent against manual human approval

Across a study of 1,053 testers, humans manually reviewing commands blocked 13.6% of dangerous ones while the automated classifier blocked 89%. Teams and Enterprise users on auto mode are reported to ship roughly 25% more pull requests, with Adobe, Nuro, Gusto and Garner Health named as customers using it.

The strongest-methodology claim of the week and still vendor-run on a product the vendor was making default the same day, which is exactly the conflict to hold in mind. The sample size and disclosed method put it well above the usual vendor assertion, and the direction is consistent with everything known about alarm fatigue in other domains. Use it as the justification for measuring your own approval-catch rate — a cheap exercise almost nobody has done — rather than as a number to cite in a risk register. The 25% pull-request figure is vendor analytics with no baseline disclosed and should not be carried into a business case.

Evidence: vendor_claim

OpenAI and Cerebras: Security investigation loop under low-latency frontier inference

A security investigation workflow reported to drop from one to two hours down to ten to fifteen minutes under the Ultrafast preview, with customer quotes from Jane Street and Podium; throughput cited at up to roughly 14x and around 750 tokens per second.

Compelling and unverifiable in equal measure. No methodology is disclosed, the baseline is anecdotal, and the preview has no published price or availability date, which means even a true result cannot yet be planned against. The useful signal is directional: if latency collapses by an order of magnitude, interactive agent loops become viable in workflows that currently batch, and that changes the user experience question before it changes the cost question. Wait for pricing before redesigning anything.

Evidence: benchmark

FlowScout (academic): Synthesizing reusable agent workflows from historical execution logs

Mining execution logs into a process graph and refining it against execution feedback with Monte Carlo tree search improves tool correctness and execution quality against named baselines including PM4Py, ReAct and AFlow.

Independently measured against named baselines with the method disclosed, which puts it above every vendor claim this week on evidence quality and below all of them on deployment realism — these are academic benchmarks, not production workloads. The transferable idea is cheap to test: if you have historical execution logs for a workflow your agents keep improvising, mine them into a frozen path and measure whether deviation rate drops. That is a one-sprint experiment with a clear success criterion.

Evidence: benchmark

MAP-Graph (academic): Provenance-aware shared memory across multiple cooperating agents

Applying permission, path-trust and action gating as runtime properties of shared agent memory improves task success and accuracy across a 2,700-task suite, with ablations isolating each control's contribution.

Measured with ablations, which is the right way to make this kind of claim, and entirely synthetic, which is the limit on how far to carry it. The finding that matters for practitioners is architectural rather than numerical: if multiple agents share memory, trust needs to be a runtime gate on what one agent may act on from another, not an audit log reviewed afterwards. Teams building multi-agent systems should treat provenance as a design requirement now, because retrofitting it after agents are already writing to shared memory is substantially harder.

Evidence: hype_signal

SpaceXAI: Computer-use agents on internal sales, operations and engineering tasks

Internal workflows including sales outbound, operations invoice handling and engineering bug reproduction cited alongside customer quotes referencing two to three times efficiency gains.

No baseline, no method, no measurement — efficiency multiples quoted in a launch post are marketing, and this publication scores them as such regardless of how plausible the underlying capability is. The capability may well be real and important; the numbers attached to it carry no information. If you pilot this, define your own baseline before the pilot starts, because the vendor has given you nothing to compare against and a post-hoc estimate will be indistinguishable from the launch-post figure.

Enterprise Readiness.

  • human approval

    The first published measurement puts human review at 13.6% of dangerous commands blocked against 89% for an automated classifier, on a vendor-run study of 1,053 testers. Treat per-action approval as a fatiguing control that degrades with volume, reserve it for irreversible actions only, and instrument your own catch rate rather than adopting the vendor's number — sampling approved actions and replaying them against a classifier is a week of work and gives you the only version of this figure that applies to your environment.

  • permissioning

    Computer-use agents break the assumption that an agent identity is separable from a user identity: the target application sees a login, and this week's leading example pools sessions across a user's bots on one VM. Before any pilot against a system of record, establish which identity the agent authenticates as, whether credentials are scoped per task rather than per user, and whether your SIEM can distinguish agent activity from human activity at all.

  • data access

    Persistent agent memory shipped this week with the managed settings needed to govern it, which is the correct pairing and also a prompt to act. Memory converts a one-off prompt injection into a persistent compromise, so write the retention and redaction policy before enabling it and lock MCP servers and plugin marketplaces through managed settings first. Where multiple agents share memory, treat provenance and path-trust as runtime gates rather than as audit.

  • verification

    Self-reported completion becomes less reliable as models are tuned for long-horizon autonomy, because a model optimized to finish multi-step work is also optimized to report finishing it. Require a checkpoint the agent cannot forge — a test suite, a type check, a separate reviewing model, or a signed artifact — and prefer frozen, verified workflow paths over improvised tool sequences wherever historical execution logs make that possible.

  • reliability

    Agent frameworks are now confirmed dual-use: Taiwan's digital ministry stated on August 13 that July attacks on government websites used AI agents including open frameworks. Prompt-injection resistance, sandbox defaults and outbound tool allowlists move from research backlog to incident-response priority, and any open harness adopted internally should be assumed to be equally available to an adversary targeting you.

  • cost

    Agent unit economics are on a published clock: one major provider moved to peak and off-peak pricing on August 16 and another disclosed at launch that its current rate expires December 31 and doubles January 1. Model long-running agent workloads against the post-increase rates, and procure on measured turn count rather than token price, since turns drive the bill on long-horizon work and do not change when a rate card does.

  • auditability

    Log classifier allows as well as denies. The allow stream is the only record of what an agent was permitted to do and the only dataset that lets you tune the classifier or reconstruct an incident. Certification is also becoming a usable procurement signal — one major coding-agent vendor completed an agent-behavior audit with adversarial testing and quarterly retesting this week, which is a more relevant attestation than SOC 2 for autonomous tooling.

Try This.

Measure your own approval-catch rate before you change any autonomy default

  1. Pull the last 30 days of agent actions that went through a human approval prompt in one team, and export the approved set with enough context to replay the decision. Aim for at least 500 approvals; if you do not have logging that supports this, that is the finding and the first fix.
  2. Independently label a stratified sample of 200 approved actions as safe or dangerous using your own definitions of irreversibility — production data mutation, outbound customer communication, payment, credential change, publishing. Have two reviewers label separately and reconcile disagreements, so the label set is not itself a single tired human.
  3. Compute the share of dangerous actions that were approved. That number is your human-approval catch rate, and it is the only version of this week's 13.6% figure that applies to your environment.
  4. Run the same 200 actions through whatever automated classifier your agent platform offers, or a simple rule set covering your irreversible categories, and compute its catch rate on the same labels. Compare the two directly rather than against the vendor's numbers.
  5. Reclassify your action inventory into reversible and irreversible. Move reversible actions to allow-and-log, keep human approval only on irreversible ones, and confirm the change cuts approval volume enough that a reviewer sees a manageable number per day.
  6. Close the loop on environment before expanding autonomy: confirm the agent runs in a sandbox with a task-scoped credential and an outbound allowlist, and confirm at least one completion checkpoint the agent cannot forge. Re-measure the catch rate after 30 days.

Expected outcome: A defensible, environment-specific number for how well human approval actually works in your organization, replacing an assumption that has never been tested. Most teams running this will find their catch rate well below what their risk register implies, and will find approval volume high enough to explain why. The immediate deliverable is a reversible-versus-irreversible action inventory and a sharply reduced approval queue; the durable one is that the next autonomy decision gets made against your own measurement rather than a vendor's study.

Watchlist.

  • Aug 16

    Peak and off-peak inference pricing takes effect at a major provider

    The first time-of-day pricing from a significant model vendor. If agent platforms start scheduling batch and overnight work into off-peak windows, inference scheduling becomes an agent architecture discipline rather than an infrastructure detail, and turn efficiency and run timing both become procurement inputs.

  • Aug 16 - Sep 30

    Whether Enterprise tiers follow consumer tiers on autonomy defaults

    Auto mode became default for Pro, Max and Team while Enterprise stayed opt-in. The date Enterprise defaults flip is the point at which every administrator who has not set a managed-settings policy inherits one, and it is worth having the policy written before that rather than after.

  • Sep 1

    Hardware security keys required for gated frontier model access

    The first hard physical-authentication requirement on model access takes effect. Whether other labs adopt equivalent controls within the quarter determines if physical authentication becomes a norm for dual-use agent capability or stays a single vendor's control that shapes nothing else.

  • Sep - Oct

    Independent replication of the human-approval effectiveness finding

    The 13.6% figure is vendor-run and is already being cited as though it were settled. An academic or third-party replication — or a contradicting result — would either make this the most consequential agent governance finding of the year or expose it as a product-launch artifact. Until then, treat your own measurement as the authoritative one.

  • Q4 2026

    Provenance or signing norms for agent plugin marketplaces

    Agent Plugins 1.0 shipped with broad multi-vendor backing and no published signing or attestation standard for third-party packages. Watch whether the coalition addresses supply-chain provenance before the install base is large enough that retrofitting it is impractical.

  • Ongoing

    A production proof of value with a named customer and a stated baseline

    Nothing this week cleared that bar, and nothing has for several weeks running. The first independently measured deployment with a disclosed baseline will reset what this publication treats as an acceptable evidence standard, and until one appears the honest summary of agent ROI remains that the direction is well supported and the magnitudes are not.