blacktops
Agentic SDLC · AI FDEJoin the waitlist
Insights / October 31, 2026

What should you check before you add another coding agent?

Most teams pick a coding agent the way they’d pick a model: open the benchmark table, find the best score, add it. But the model is the smallest part of what you’re adding.

Before you bring in another agent, get answers to four questions its model name won’t give you: what tools and instruction files it loads, what it will do without asking and who decides that, what it leaves behind for whoever reviews the work, and where it runs.

When you ask “which agent should we add?”, you’re actually deciding four things at once, and the model is only the first of them:

  • The model: the weights that produce each answer.
  • The coding agent: the software wrapped around the model, with its prompts, tools, loop and permissions.
  • Orchestration: whatever assigns the work, runs more than one agent and passes work between them.
  • The delivery system: the requirement, the repository, CI, review, approval and release. Everything the work has to get through.

Swapping the model only changes the first of those. Adding a new agent brings its own tools, permissions, records and environments into a setup your team is already looking after.

A report published last week shows how far that reaches. On a customer’s machine, Forever Security found an AGENTS.md file sitting in the Downloads folder, “disguised as ordinary engineering notes”. It told OpenAI’s Codex to install a malicious npm package. Forever’s software blocked the install before any code ran.

The researchers then reproduced the attack themselves, and the results are worth reading with the caveats attached:

  • Codex: In an isolated virtual machine, with the attacker’s file in place, Codex added a tool server to its machine-wide configuration and installed the package. “In our testing, Codex ran the malware in more than 90% of attempts.” Most of those runs used a harmless stand-in; the real package ran once.
  • Claude Code: The researchers wrote their own CLAUDE.md, modeled on the attacker’s file, to point Claude Code’s API traffic at a server they controlled. Forever says that change needs a person’s approval. “In our testing, Claude followed through on this about 50% of the time.”
  • The limits: Nobody knows how the file got onto the customer’s machine, though Forever showed in its own tests that a web page can drop one there. The two rates come from two different attacks, so they don’t compare the agents. Neither Forever nor Help Net Security reports a statement from OpenAI or Anthropic.

Forever’s conclusion: “There is nothing here for OpenAI or Anthropic to patch.” Each agent did what it’s built to do, which is read the instruction file in the folder it was started in.

Forever Security’s report on agent instructions planted in the Downloads folder has the full chain, and Help Net Security covered it.

The short version

Before you add another agent, ask four questions about it, and about the agents you already have: what tools it supports, what permissions it has, what evidence it leaves for review, and where it runs. If nobody on your team can answer one of them, write unknown, because that gap is a finding in itself.

The model is one of four things you're choosing. The model decides the answer to each prompt and changes when you switch models or the vendor updates one. The coding agent decides the prompts, the tools on offer and when to stop and ask, and changes when you add, upgrade or reconfigure an agent. Orchestration decides which agent gets which task and what a delegated agent inherits, and changes when someone turns on subagents or a coordinator. The delivery system decides what the work has to pass, such as checks, review and release; people make its decisions, and it changes rarely and on purpose. Swapping the model changes one of the four.

The model is one of four things you’re choosing

The four layers decide different things, and they change at different speeds.

Layer What it decides
The model The answer to each prompt.
Changes when: You switch models, or the vendor updates one
The coding agent The prompts, the tools on offer, and when to stop and ask.
Changes when: You add, upgrade or reconfigure an agent
Orchestration Which agent gets which task, and what a delegated agent inherits.
Changes when: Someone turns on subagents or a coordinator
The delivery system What the work has to pass: branch protection, required checks, review and release.
Changes when: Rarely, and on purpose

The agent around the model changes the results. When Anthropic wrote about SWE-bench, it defined an agent as “the combination of an AI model and the software scaffolding around it”. The scaffolding writes the prompts, turns the model’s output into actions and keeps the loop going.

Anthropic, on scaffolding

“The performance of an agent on SWE-bench can vary significantly based on this scaffolding, even when using the same underlying AI model.”

Anthropic doesn’t attach a figure to that; it’s from Anthropic’s post on SWE-bench Verified and scaffolding.

The SWE-agent paper did put a figure on it, for one model. On SWE-bench Lite (300 tasks), GPT-4 Turbo resolved 11.00 percent of tasks working through a bare Linux shell and 18.00 percent through the interface the authors built for it.

Bear in mind that the interface bundled several changes at once (a file viewer, linted edits, search), and it’s the authors’ own system, measured with 2024 models. The SWE-agent paper is on arXiv.

So does the machine it runs on. Anthropic ran internal experiments on Terminal-Bench 2.0 with the “same Claude model, same harness, same task set” and found a gap of 6 percentage points (p < 0.01) between the best- and worst-resourced setups.

On SWE-bench (227 problems, 10 samples each), giving the agent five times the memory raised scores by 1.54 percentage points. These are Anthropic’s internal evaluations with Claude models, not production numbers. The post’s own summary: “Two agents with different resource budgets and time limits aren’t taking the same test.” See Anthropic’s post on infrastructure noise in agentic coding evals.

GitHub and Microsoft draw the same line between the model and what’s around it. GitHub’s 7 October changelog says Copilot’s local sandbox runs commands and tools with “restricted access to the filesystem, network, credentials, and other system capabilities”, and that “Model execution and tool isolation are separate concerns.” VS Code’s release notes add that sandboxing “does not replace endpoint security or provide a standalone security boundary.”

Both are worth reading in full: GitHub’s changelog on local sandboxing for Copilot and the VS Code 1.141 release notes.

Four checks before you add an agent

None of these needs a meeting. Each is a few questions you can answer from documentation, settings files and run records, for the new agent and for the ones you already have.

We read the product pages quoted below on 28 October 2026. Documentation tells you how a product works, not how well it works, and none of these pages carries a version date. They’re here to show where the answers live, not to rank anyone.

1. Tool support

Ask: Which tools, MCP servers and instruction files does it load, and from which files? Can it add a tool or change its own configuration? Which parts of MCP does it support?
Look in: The product’s MCP and configuration pages; its user, project and managed settings files; the instruction files in each folder it gets started in.
If nobody can answer: Nobody can list what the agent loads when it starts, so nobody can say what a planted file or a new server would change.

MCP leaves enforcement to the product. The Model Context Protocol (MCP) standardizes how an agent connects to tool servers, and its specification is candid about the limits. “Tools represent arbitrary code execution and must be treated with appropriate caution,” it says, and MCP “cannot enforce these security principles at the protocol level”. See the MCP specification (version 2026-07-28).

Products handle tool approval differently. You can see it in GitHub’s and Cursor’s own pages:

  • GitHub’s page says that once an MCP server is configured, Copilot cloud agent uses its tools “autonomously, and will not ask for your approval before using them.” It doesn’t currently support remote MCP servers that use OAuth, and the same page strongly recommends allowlisting specific read-only tools.
  • Cursor’s page says that after an MCP connection is approved, “each tool call still needs individual approval before running.” You can pre-approve specific tools with an MCP allowlist.

Different defaults, and in both cases an administrator can narrow the tool list.

Sources: GitHub’s page on extending Copilot cloud agent with MCP and Cursor’s agent security page.

Instruction files count as input. AGENTS.md describes itself as “a README for agents”, and OpenAI’s page says “Codex reads AGENTS.md files before doing any work.” Other products have their own file names, which is part of why the Downloads-folder file worked. See the AGENTS.md site and OpenAI’s page on AGENTS.md in Codex.

A tool server can run code the moment you sign in. On 19 October, Pillar Security reported that in the OpenCode agent on Windows and WSL, a malicious remote MCP server could run code on the developer’s machine during OAuth sign-in. Version 1.18.33 fixed it, but with no CVE or advisory, so “conventional vulnerability scanners and dependency alerts will not identify affected installations.”

The maintainer’s position, as Pillar quotes it: “trying to deal with a malicious MCP server is outside our threat model.” No exploitation has been reported, and Pillar published no indicators you could check for past use. That’s why the checklist asks for installed versions. Read Pillar Security’s report on the OpenCode MCP OAuth flaw and The Hacker News’s coverage.

2. Permissions

Ask: What can it do without asking: edit files, run commands, reach the network, call tools? Who sets that policy, and can a developer, a repository file or the agent itself loosen it? What doesn’t a rule cover? What does a delegated agent inherit?
Look in: The product’s permissions or approvals page; the user, project and managed settings actually in effect; the repository’s branch protection and required reviews.
If nobody can answer: Then as far as anyone can show, the agent can do whatever the person who started it can.

The sandbox and the approval policy are two different settings. OpenAI’s page describes Codex’s sandbox, which limits what it can touch, “plus an approval policy that controls when it must stop and ask you before acting.” Ask who can change each one. See OpenAI’s page on Codex approvals and security.

Look for a bypass. In the Downloads case, the planted file told Codex to start a second session with --dangerously-bypass-approvals-and-sandbox, and Forever says the package fetch kept failing until it did. Whether a bypass exists, and who can switch it off for everyone, belongs in your permissions answer.

Anthropic, GitHub and Cursor also say what their controls don’t cover:

  • Claude Code’s Read and Edit deny rules don’t apply “to arbitrary subprocesses that read or write files indirectly, like a Python or Node script that opens files itself.”
  • Copilot cloud agent’s firewall “only applies to processes started by the agent via its Bash tool.”
  • Cursor calls its run modes “best-effort guardrails rather than a hard security boundary.”

Sources: Claude Code’s permissions page and GitHub’s page on the Copilot agent firewall.

Some of the controls live in your delivery system, not in the agent. GitHub’s risks page says Copilot cloud agent “cannot approve or merge a pull request” and “is also subject to any branch protections and required checks for the working repository.” See GitHub’s page on Copilot cloud agent risks and mitigations.

Delegated agents inherit permissions. It’s where orchestration and permissions meet, and OpenAI and Anthropic both spell it out:

  • OpenAI’s page: “Subagents inherit your current sandbox policy.” and “A custom agent file can’t replace those settings.”
  • Claude Code’s page: if a subagent’s mode is left unset, “the subagent inherits the main conversation’s permission mode”. The page lists exceptions, including when the parent runs in bypassPermissions, acceptEdits or auto mode.

Sources: OpenAI’s page on Codex subagents and Claude Code’s page on subagents.

3. Review evidence

Ask: Once a run is over, what can a reviewer open without going to the developer: the prompt, each tool call, each approval and who gave it, the commands? Is that recorded by default? Where does it live, and for how long? Who gets the credit, or the blame, for the change?
Look in: The product’s telemetry, logging or session pages; the pull request; the CI run log; your own retention settings.
If nobody can answer: The reviewer has the diff and the agent’s own summary of what it did, and nothing to check that summary against.

The records usually exist. Someone has to switch them on. Claude Code and Codex are two examples:

  • Claude Code: OpenTelemetry export is off until enabled, and even then prompt content and tool details stay off by default. A repository’s own settings can’t switch it on. Subagent transcripts are deleted after 30 days by default.
  • Codex: It saves local session transcripts by default. Its OpenTelemetry export, which can record tool approvals and results, is “Disabled by default”.

Sources: Claude Code’s monitoring page and OpenAI’s advanced configuration page for Codex.

Put the record where the reviewer already looks. A transcript on somebody’s laptop is no use to a reviewer. One attached to the pull request or the CI run is.

GitHub’s page on Copilot cloud agent says “session logs show the work and tools used”, and then adds a line worth keeping:

GitHub, on session logs

“Logs do not replace your own review and testing.”

See GitHub’s page about Copilot cloud agent.

4. Deployment environment

Ask: Where does it run: a laptop, CI, the vendor’s cloud, your own machines? Which folder is it started in? What can that place reach? Do its defaults change depending on where it runs?
Look in: The product’s pages on its command line, CI, cloud and self-hosted options; the CI workflow files that start it; the runner or machine definition.
If nobody can answer: Nobody can say what the agent could reach if it followed a planted instruction.

The defaults move with the place. OpenAI’s pages say “By default, codex exec runs in a read-only sandbox”, while the Auto preset “can read files, make edits, and run commands in the working directory automatically.” Claude Code’s page says “By default, cloud sessions execute on Anthropic’s infrastructure”. Where the agent runs is itself a setting.

Sources: OpenAI’s page on non-interactive Codex and Claude Code’s page on self-hosted environments.

If you want to go deeper on what belongs inside that environment and what has to stay outside it, we covered it in What belongs in an agent workspace, and what has to stay out of its reach. In practice this is the check to start with: where is the new agent going to run?

Illustrative walkthrough: adding a second agent

Illustrative walkthrough only. The team, the agents and the numbers are made up, and neither agent is a real product.

A product team of 14 has one coding agent running in developers’ editors. Engineering leadership wants a second one that picks up labeled issues and opens pull requests from CI.

Up to now the team has been comparing the two models’ benchmark scores. Before deciding, the platform lead runs the four checks on both agents.

Check What they found, and what they decided
Tool support The new agent reads a different instruction file, so the team’s rules (run the integration suite; never edit generated migrations) weren’t loaded in the trial. In CI it reached two of the four tool servers.
Decided: One instruction file both agents read, and a read-only token for the ticket server in CI
Permissions In editors, the first agent asks before each shell command. In CI nobody is there to ask, so the trial ran with approvals off, set in a file any developer could edit.
Decided: The CI agent pushes only to its own branches and can’t merge. The policy sits in managed settings and branch protection, owned by the platform lead
Review evidence The first agent’s transcripts sat on 14 laptops with telemetry off, so reviewers saw a diff and a summary. The CI agent’s run log was kept for 7 days.
Decided: Telemetry on for both; every pull request links its run log; logs kept 90 days
Deployment environment The CI agent ran on the shared runner, which holds a staging deploy credential and reaches the staging database.
Decided: A dedicated runner with no deploy credential, and network access limited to the package registry and the Git host

The model comparison took an afternoon; the four checks took three days. And two of the findings were about the agent the team already had: nobody could list what it loaded at start, and its records never left the laptops.

Illustrative walkthrough: adding a second agent, with four checks on both agents. The request is to add a second agent that works from CI; so far the evaluation compares two models' benchmark scores. Tool support: a gap; the new agent read a different instruction file, so the team's rules weren't loaded, and the team decided on one instruction file both agents read. Permissions: a gap; approvals were off, set in a file any developer could edit, and the team decided the CI agent pushes only to its own branches and can't merge. Review evidence: a gap; transcripts sat on 14 laptops and the CI log was kept 7 days, and the team decided to turn telemetry on, link each run log and keep logs 90 days. Deployment environment: a gap; the agent ran on a shared runner with a deploy credential, and the team decided on a dedicated runner with limited network access. Each decision was made by a person. The model comparison took an afternoon; the four checks took three days.

Compare the agents you already run

Start with the agents already in use, before you add one. Each product, in each place it runs, gets its own column.

Compare the agents you already run: four checks the model's name doesn't answer. One, tool support: what does it load at start, and can it add to that? Look in the tool and configuration pages and the settings files. Two, permissions: what does it do without asking, and who can weaken that? Look in the permissions pages, managed settings and branch rules. Three, review evidence: what can a reviewer open without the developer? Look in the telemetry pages, the pull request and the CI logs. Four, deployment environment: where does it run, and what can that place reach? Look in the product pages, CI workflows and runners. Where nothing answers a question, write unknown, with who could answer: that's a finding. One column per agent, per place it runs.
Check Ask
Tool support What it loads at start, and whether it can add to that.
Where to look: MCP and configuration pages; settings and instruction files
Permissions What it does without asking, and who can weaken that.
Where to look: Permissions pages; managed settings; branch protection
Review evidence What a reviewer can open without the developer, and for how long.
Where to look: Telemetry pages; the pull request; the CI run log
Deployment environment Where it runs and what that place reaches.
Where to look: Product pages on each place; CI workflows; runner definitions

Run it

The block below is written so that a person or an agent can carry out the comparison from it.

Terms:

  • Agent: one product in one place it runs (editor, terminal, CI or the vendor’s cloud).
  • Default: what the product does with no setting changed (a row such as “does without asking” is answered for the default first, then for the settings in effect).
  • Managed setting: a setting an administrator controls that a developer can’t override.
  • Weaken: turn off or loosen an approval, a sandbox or a network limit.
  • Review evidence: a record of the run a reviewer can open without the developer’s help.
  • Delegated agent: a subagent, or another agent this one hands work to.

Inputs: The agents in use, with each one’s installed version; each product’s documentation pages (URL and date read); the user, project and managed settings files in effect; the instruction files where each agent starts; the CI workflow files that start agents; one recent run’s record per agent (run log, transcript, pull request); each product’s release notes or security advisories. Supply them as pasted text, file paths or URLs.

Steps:

  1. Make one column per agent, as defined in Terms.
  2. Put the product, installed version and place it runs in each column header.
  3. Answer the tool-support rows (models, instruction files, tools, configuration) from the settings files, the instruction files and the product’s MCP and configuration pages.
  4. Answer the permissions rows (does without asking through what a delegated agent inherits) from the settings in effect and the repository’s branch protection. Note what the product page says a rule doesn’t cover.
  5. Answer the review-evidence rows (record, attribution) from the recent run: what a reviewer can open without the developer, where it’s kept, for how long and who the change is attributed to.
  6. Answer the environment rows (where it runs, and how you would learn of a fixed flaw) from the CI workflow, runner or machine definition, and from the product’s release notes or security advisories.
  7. After each answer, add its source in brackets: a page URL with the date read, or a file path.
  8. Copy every unknown and every difference into the follow-up table. A difference is any row where two columns give different answers.
  9. Give each follow-up table row an owner: the role or team that will find the answer or make the change.

Output: These two tables, filled in. Copy them as they are. Add a column per agent to the first table.

| Question | Agent 1: product, version, where it runs | Agent 2: product, version, where it runs |
| --- | --- | --- |
| Tool support: models it uses | | |
| Tool support: instruction files it reads at start | | |
| Tool support: tools and MCP servers it loads (config file paths) | | |
| Tool support: can it add a tool or change its own configuration? (yes/no/unknown) | | |
| Permissions: does without asking | | |
| Permissions: asks before | | |
| Permissions: who sets the policy; can a developer or repository file weaken it? | | |
| Permissions: what a delegated agent inherits | | |
| Review evidence: record a reviewer can open (where; on by default?; kept how long) | | |
| Review evidence: who the change is attributed to | | |
| Deployment environment: where it runs; what that place can reach | | |
| Deployment environment: how you would learn of a fixed flaw in this version | | |
### Follow-up table
| Follow-up (unknown or difference) | Agent | Owner |
| --- | --- | --- |
| | | |

Unknowns: Write unknown and the role or team that could supply the answer. Never paste credential values, tokens or anything confidential into the table.

Done when: Every cell holds an answer with its source, or unknown with a role or team; every agent in use has its own column; and every unknown and every difference has an owner in the follow-up table.

Where Blacktops fits

You don’t need any particular tool to do this. The four checks, filled in for the agents you already run, with a named owner for every unknown, are enough to get going.

Whichever agents you end up with, the delivery system stays yours: the requirement the work starts from, the team context and decisions it has to respect and the review it has to pass.

Blacktops is the Agentic SDLC Platform designed to keep the customer’s question linked to the work. It maintains a single thread from the customer’s question through to the resolution:

  • the approved requirement
  • the agent’s work
  • the evidence
  • each decision and who made it
  • the update back to the customer

Agents do the work between decisions, and people make the decisions. That’s our operating model: Continuous Agentic Software Delivery™ (or Blacktops CASD™).

Compare the agents you already run. Fill in the comparison before you add the next one, and get in touch with us if you’d like to talk through what you found. Just leave out credentials, customer names and anything else that’s confidential.

Related: What belongs in an agent workspace, and what has to stay out of its reach covers the deployment-environment check in depth. 16 coding agents compared: where they run, tools, permissions and logs applies the four checks to the documentation of 16 products.

Continuous Agentic Software Delivery, CASD and Blacktops CASD are trademarks of Blacktops, Inc.