Foo GuardBeta
Menu

Research note · .agentguard.yml policy

What Static AI Agent Security Analysis Can Catch Before Runtime

A public benchmark of four agent configurations scored by Foo Guard's deterministic rules.

Introduction

An AI agent receives much of its power from a configuration file before any model call runs. That file can grant shell execution, filesystem access, database access, arbitrary network access, API access, repository modification, and destructive operations. It can also record whether a person must approve an action, whether a credential is embedded, whether a permission is administrative, and whether the agent is aimed at production.

Those declarations are visible to a static review. Foo Guard parses the file, normalizes the declared capabilities, and evaluates them with a fixed rule set, AG-001 through AG-023, plus a secret scan that can add AG-SECRET-001. The same file always produces the same findings, score, and grade. The model is not asked to decide whether a rule passed.

This note shows what that process catches on four Foo Guard test configurations, and where the method stops. Static analysis reads the configuration. It does not watch the agent run.

What this analysis tests

The benchmark uses four Foo Guard test configurations with different risk profiles:

These are Foo Guard-maintained fixtures, not a third-party independent benchmark. Four files cannot represent the AI-agent ecosystem. The public copies are a repeatable fixture set: a reader can inspect the inputs behind the published rule IDs. Foo Guard does not publish the analyzer source code with this note.

Methodology

Each file follows the production analysis path in analyzeWithSanitizedConfig, the same function Team GitHub scans call per file:

  1. Parse JSON or YAML into a document.
  2. Scan the document for secret-like values and replace detected material with [REDACTED] before later stages see it.
  3. Normalize the sanitized document into an agent model: identity, tools, permissions, credentials, autonomy, and auditability.
  4. Evaluate deterministic rules AG-001 through AG-023 against that model.
  5. Merge secret findings. When AG-SECRET-001 already covers a credential index, overlapping AG-004 and AG-005 findings for that same credential are dropped so one exposed value is not counted three times.
  6. Score the merged findings and map the score to a grade.
  7. Apply repository policy when a pass/fail outcome is required.

This is not a machine-learning classifier and not a statistical sample. There is no confidence interval. A finding exists when the rule's predicate matches the normalized model.

Severity weights are fixed: critical 25, high 15, medium 8, low 3, info 0. The risk score is the sum of those weights, capped at 100. A higher score is worse. Grades use these bands:

A score of 100 maps to F. The cap is what makes a file with more than 100 points of weighted findings report 100 rather than the raw sum.

Pass/fail in the table uses Foo Guard's checked-in benchmark policy, .agentguard.yml: failOn: high and minGrade: B. failOn: high fails the file when any finding is high or critical. minGrade: B fails the file when the grade is worse than B. Either condition is enough. The preferred policy filename is .fooguard.yml. This benchmark uses the legacy filename. When both exist, .fooguard.yml wins. A Team organization policy can only make the effective threshold stricter. This note does not apply a live organization policy.

A Team GitHub scan of several files does not average them. The check summary score is the highest file score. The summary grade is the worst grade. Policy then sees every finding from every file that analyzed successfully. The check is named Foo Guard Security. A policy failure completes as a failing check. A policy pass completes as a success. A missing configuration or an operational error completes as neutral. The tree and file contents are read at the commit SHA from the GitHub event.

The CLI uses the same policy function. Exit code 0 means the scan finished and policy passed. Exit code 1 means policy failed. Exit code 2 means the scan could not analyze a file. The GitHub Action calls that CLI path.

How the analysis moves

  1. 1Agent config
  2. 2Parse JSON or YAML
  3. 3Identify and redact secret-like values
  4. 4Normalize capabilities
  5. 5Deterministic rules
  6. 6Findings
  7. 7Risk score and grade
  8. 8Policy decision

Test configurations

Outcome uses failOn: high and minGrade: B. Low and info counts are zero for every file, so they are omitted from the table.

Benchmark results for four agent configurations
ConfigurationScoreGradeCriticalHighMediumOutcome
Secure FAQ Assistant0A000Passed
Internal Knowledge Agent15B010Failed
Staging Support Copilot38C021Failed
Legacy Integration Agent100F561Failed

Score and grade

Finding severity breakdown

Finding counts by severity
ConfigurationCritical × 25High × 15Medium × 8
Secure FAQ Assistant000
Internal Knowledge Agent010
Staging Support Copilot021
Legacy Integration Agent561

Secure FAQ Assistant

File: secure-faq-assistant.json. Score 0. Grade A. No findings. Policy passed.

No Foo Guard rule in this rule set was triggered by this configuration. That is a statement about the current predicates, not a claim that the agent is perfectly secure.

The normalized model has an id, an owner, and lifecycle production. The only credential is an environment variable named LLM_API_KEY with embedded: false, longLived: false, and location: env. No secret-like value is present, so AG-SECRET-001 does not fire, and AG-006 does not fire because longLived is not true.

Both tools, faq_lookup and escalate_to_human, set shell, filesystem, database, arbitrary HTTP, and resource modification to false. The single permission is faq:read on knowledge-base/faq with accessLevel: read. It is marked production: true. AG-010 still does not fire, because that rule requires production together with write or admin access. A read-only production permission is not that condition.

Autonomy requires human approval, caps iterations at 8, and forbids self-modification. Auditability enables logging, tracing, and action attribution. Those fields keep AG-020 and AG-021 quiet. AG-021 also requires a high-impact tool, which this file does not declare.

Internal Knowledge Agent

File: internal-knowledge-agent.yaml. Score 15. Grade B. One high finding, AG-006. Policy failed.

Grade B meets minGrade: B. The file still fails because failOn: high treats any high finding as a failure. The grade threshold and the severity threshold are separate. Meeting one does not waive the other.

AG-006, Long-Lived Credential, fires when any credential has longLived: true. The evidence path is credentials[0].longLived for CONFLUENCE_SERVICE_ACCOUNT. The credential is an environment variable, not an embedded secret, and it is not shared. The rule does not look at the lifecycle value staging, and it does not check whether the credential can still authenticate. A long-lived marker means the configuration itself says the credential is not short-lived. Rotation, theft response, and blast radius are harder when that marker is true. That is the extent of the finding.

The rest of the file stays inside other predicates. Search tools are read-oriented.sharepoint_read sets accessesFilesystem: true and modifiesResources: false, so AG-015 does not fire. Filesystem write requires both flags. bookmark_article can modify a resource, and it sets requiresApproval: true. Human approval on the agent is also true, so the compound rules that require humanApprovalRequired: false stay inactive. The permission is wiki:read, not a wildcard and not administrative.

Staging Support Copilot

File: staging-support-copilot.json. Score 38. Grade C. Findings AG-002 (medium), AG-006 (high), and AG-014 (high). Policy failed.

The weighted sum is 15 + 15 + 8 = 38, inside the C band of 26–45. The file sits between the FAQ assistant, which triggers nothing, and the legacy fixture, which hits the score cap. It has real gaps and it does not combine disabled approval with data read plus arbitrary HTTP, so AG-022 and AG-023 stay off.

AG-002, Missing Owner, fires because identity.owner is missing. A sponsor,support-platform, is present. The rule does not treat sponsor as a substitute for owner. Severity is medium, so this finding alone would not fail failOn: high. It does contribute 8 points.

AG-006 fires on ZENDESK_OAUTH because longLived: true. A second credential, COMMERCE_API_KEY, is marked longLived: false and does not add another AG-006 hit. One finding covers the credentials that match.

AG-014, Arbitrary HTTP Requests, fires because tool order_lookup sets arbitraryHttp: true. The same tool also sets restrictedNetwork: true. The rule does not consult restrictedNetwork. The predicate is only arbitraryHttp === true. Declaring a restriction beside an arbitrary-HTTP flag does not clear the finding. Human approval is true, and draft_reply requires approval before it modifies a resource, so AG-021 does not run. AG-021 requires humanApprovalRequired to be explicitly false.

Legacy Integration Agent

File: legacy-integration-agent.json. This is an intentionally misconfigured test fixture. The description in the file says so. Score 100. Grade F. Five critical, six high, one medium. Policy failed.

The uncapped weighted sum is 223: five critical findings at 25, six high findings at 15, and one medium finding at 8. The published score is 100 because the engine caps the total. Grade F is every score above 70, including the cap.

Independent findings, each tied to one declared condition:

AG-011 does not add a second finding. It skips permissions already marked wildcard: true, so the wildcard case stays on AG-008.

The credential is also embedded: true and location: client. AG-004 and AG-005 match those flags. They are removed before scoring when AG-SECRET-001 already points at credentials[0]. The published list therefore contains the secret finding and not a second and third critical finding for the same credential.

Combination findings, which require more than one condition:

The downloadable copy replaces the original token-like string with a synthetic value that still matches the sk- pattern, so AG-SECRET-001 still fires for the same reason. This page prints that value as sk-••••••••. The full synthetic string is only in the downloaded file, and it is not a live credential.

What static analysis caught

Across the four files, the engine reported these categories. Each line is backed by a rule that actually fired, or by a rule that was evaluated and then deduplicated as described above.

The broader engine also contains rules these four files did not trigger, including shell execution, database write, shared credentials, self-modification, and unlimited autonomy. Those rules are part of the product. They are not results of this benchmark.

What static analysis cannot prove

The findings above are statements about the configuration text. The current product leaves the following questions open:

AG-022 is named as a potential data exfiltration path for that reason. The rule combines flags in the file. It does not observe an exfiltration.

Static review and runtime controls answer different questions. A configuration check can reject a wildcard production admin permission before merge. A runtime control can see the call that happens later. This benchmark covers the first question. Runtime monitoring and enforcement remain in scope for systems built to watch execution.

Why pre-deployment checks matter

The practical use of these results is review before the file is the one a deployment runs. A pull request can show that longLived flipped to true, that arbitraryHttp appeared, or that a value matching an API-key pattern landed in the diff. The finding id is stable, so the review comment can name AG-006 or AG-SECRET-001 instead of a paraphrase.

Foo Guard turns that review into a pass/fail check when policy is configured. Team workspaces publish a GitHub Check named Foo Guard Security for the commit SHA on pull requests and default-branch pushes. The CLI and the GitHub Action use the same rules and the same exit codes for other pipelines. Repository policy lives in .fooguard.yml, with legacy .agentguard.yml still accepted. Branch protection can require the Foo Guard Security check. Foo Guard does not configure that protection itself. A neutral check means the scan did not finish as a policy decision: no supported file was found, or the run hit an operational error.

On this corpus, three of the four files fail the benchmark policy, including the knowledge agent whose grade is already B. A repository scan that included all four would report score 100 and grade F, because those are the worst file score and the worst grade, and policy would fail because high and critical findings exist anywhere in the set. The FAQ assistant would not pull that aggregate back to A.

Public benchmark fixtures

The original fixtures are part of Foo Guard's internal certification corpus. This page does not publish the analyzer source code, and it does not ask you to clone a private tree. Sanitized copies are served from this site. Three files match the internal fixtures. The legacy file keeps the same fields and replaces the original token-like string with a synthetic sk- value that still matches the secret scanner. Foo Guard's production analyzer produced the scores, grades, rule IDs, and policy outcomes in the table from these copies.

Paste a downloaded file into the analyzer on the Foo Guard homepage. That public scan shows the findings and the score. It does not apply repository policy by itself. The pass/fail column uses failOn: high and minGrade: B, as documented above. GitHub scans and the CLI apply that policy step.

Related product pages: AI agent security scanner, configuration security, GitHub and CI checks, rule reference, risk scores, GitHub Checks, and .fooguard.yml.

Results summary

On these four files, static analysis separated a configuration that triggered no current rule, a single long-lived credential at grade B that still fails a high-severity policy, a staging support configuration at grade C, and an intentionally dangerous fixture whose weighted findings exceed the cap and publish as 100 / F. The catches are credential shape, privilege flags, tool capabilities, missing ownership, disabled approval, and combinations of those flags.

Static analysis can identify configuration-level risk before an agent runs. It complements runtime monitoring and enforcement. A grade A on this rule set means no current Foo Guard rule matched the file. It does not prove the agent is safe.