PERSONAL RESEARCH / Agentic coding · Test management

Testing
agent-built software.

Agents can generate and run tests. I am interested in how we choose the right checks, examine the tests themselves and understand what a passing result actually establishes.

Article date

Updated
Base notes dated
Latest source check
DENIZ WETZ / Personal research synthesisALL RESEARCH

THE SHORT VERSION

Key points.

  • Define success before generating tests. Expected behaviour, unacceptable outcomes and acceptance criteria determine which checks matter.
  • Examine the tests as well as the software. Passing tests can preserve a mistaken assumption or leave important paths untested.
  • Keep each result tied to its scope. Functional tests, security analysis and agent evaluations answer different questions.
Back to contents

01RESEARCH NOTE

Testing starts with a decision about success

A test run answers a defined question. Test management also decides which questions matter: expected behaviour, unacceptable outcomes, relevant environments, coverage, stopping conditions and who can accept what remains uncertain. I find this distinction more useful than asking whether an agent “does the testing”.

For a small personal experiment, an acceptance criterion might say that one person can edit their own record but cannot read another person’s record, including through a direct API request. The expected result is the test oracle: the basis for deciding whether the behaviour is correct. An agent can help formulate it, but deriving the expectation only from the code risks preserving the same mistaken assumption.

NIST SSDF 1.1 makes this separation explicit: PW.8.1 concerns selecting executable-code testing; PW.8.2 concerns its scope, design, execution, recorded results and treatment of findings. Generating test files is one part of that work. [01]

Make the evidence inspectable

Two scoped examples, drawn as a reading aid. Software testing and skill evaluation examine different things; neither diagram represents a complete assurance process.

NIST SSDF · Executable software

  1. Select suitable testing
  2. Design and execute checks
  3. Record and address findings

PW.8 connects choosing a method with designing, running and interpreting tests. An available test tool and a completed test run are different evidence. [01]

OpenAI · Agent-skill evaluations

  1. Define success
  2. Capture runs and outputs
  3. Check, score and compare

Kundel and Chua’s skill-evaluation example compares observed behaviour with stated goals. Concrete checks and rubric-based assessment serve different purposes. [03]

Back to contents

02RESEARCH NOTE

What the engineering sources emphasise

These publications describe complementary practices. They are not a benchmark of which coding agent tests best, and the presence of a feature does not prove that a particular run used it.

  1. OpenAI — Define success before writing the skill

    Dominik Kundel and Gabriel Chua / Testing Agent Skills Systematically with Evals / 22 January 2026

    Kundel and Chua recommend a small set of must-pass criteria before writing an agent skill. They distinguish outcome, process, style and efficiency: did the task succeed, did the intended steps occur, were conventions followed, and was the work reasonably economical? Their evaluation connects a prompt to a recorded run, traces and artefacts, then checks and a comparable score. Deterministic checks and rubric-based grading can answer different questions. [03]

    This is guidance for evaluating skills, not a mandate for every software test suite. I take from it the practice of stating success in advance and collecting evidence suited to each criterion. An agent skill’s evaluation and the tests of the application it changes remain distinct.

  2. OpenAI: delegate test writing, retain the specification

    Engineering guidance / Building an AI-Native Engineering Team

    OpenAI recommends delegating an initial set of test cases and tests while engineers check for shortcuts, stubbed tests and coverage that misses the intended feature or user experience. It suggests generating tests separately from implementation and checking that new tests fail before the feature is added. Runnable tests, appropriate permissions and knowledge of the available suites are part of the setup. [02]

    The Codex best-practice guide similarly asks for relevant checks and a clear definition of success in the prompt or project instructions. I read this as support for explicit expectations, not as a promise that every model, surface and task automatically runs a complete project-specific suite. [04]

  3. Anthropic: make the result observable

    Claude Code documentation / Verification and browser preview

    Anthropic recommends checks Claude can execute and inspect: tests, build results, scripts or visual comparisons. The guidance distinguishes a request to check the result, a goal evaluated across turns, a configured script gate and a separate verification agent. It asks for evidence such as commands, output and screenshots rather than an assertion of success. [05]

    The Desktop documentation also describes automatic preview verification, enabled by default, using screenshots and error checks. That is a useful capability with a specific scope. My interpretation is that a preview check alone does not establish a retained browser regression suite, database coverage or complete accessibility and security testing. [06]

  4. Google: review the tests as well as the implementation

    Engineering practices / Jules product publications

    Google’s engineering-practices guide asks reviewers to examine whether tests are correct, useful and capable of failing when the implementation is broken. It treats unit, integration and end-to-end tests as choices appropriate to the change, and tests as code that must remain maintainable. [07]

    The Jules publications illustrate a separate distinction. The June 2025 update describes improved likelihood of writing and running tests. The August 2025 account by Kathy Korevec, Mehadi Hassen and Yiguang (Amanda) Zhang describes an integrated critic that evaluates proposed changes. A model’s critique and an executed test produce different evidence; neither historical publication establishes a universal current test suite. [08] [09]

Back to contents

03RESEARCH NOTE

The tests need scrutiny too

A test can pass because it asserts too little, replaces the important component with a convenient mock, or reproduces the implementation’s assumptions. My practical question is: what plausible failure would this test detect? A regression test should first reproduce the defect it is intended to prevent. Changing an assertion to match a broken result is not a repair.

Test-driven development makes expectations executable before implementation: a meaningful failing test, an implementation that satisfies it, then refactoring while preserving the behaviour. The joint BSI/ANSSI guide identifies AI-assisted test-case generation, including edge cases, as an opportunity to reduce the additional effort of TDD. It separately discusses functional tests, security review and vulnerability scanning. That does not make TDD a complete security method. [10]

For my experiments, I would use test-first work where the expected behaviour can be stated clearly, and keep a separate check of the test’s adequacy. More generated cases or higher line coverage are useful only insofar as they expose relevant failures. They do not by themselves validate the requirements or the assertions.

Back to contents

04RESEARCH NOTE

Separate purpose, scope and method

“Functional”, “security” and “performance” describe purposes. “Positive” and “negative” describe kinds of case. Unit, integration and end-to-end describe the system boundary being exercised. Static and dynamic analysis describe methods. A security negative test can therefore be an integration test. The following examples are my working explanation of these distinctions.

  1. Rules and component contracts

    Unit and contract tests

    A unit test checks a small piece of behaviour. A contract test checks agreed inputs, outputs and failure responses between components. Mocking a remote service can make a test fast and repeatable; it leaves the real provider’s behaviour untested. Passing a contract test means the modelled contract held for the exercised cases.

  2. Real interactions

    Integration tests / Isolated test database

    An integration test can exercise the API, permission checks and a real disposable database together. This matters for transactions, constraints, concurrent writes and row-level access rules. A mocked repository cannot establish that the deployed database enforces the same boundary. The test environment and its differences from deployment remain part of the evidence.

  3. User journeys

    End-to-end and browser tests

    A browser test can follow sign-in, an edit and the resulting saved state. Its coverage depends on the journeys, roles, browsers and substituted services. A screenshot helps inspect layout; it does not establish every keyboard interaction, access rule or backend side effect. Assertions should check the outcome where it matters.

  4. The deployed environment

    Smoke tests, canaries and recovery

    A smoke test checks a small set of essential behaviours after deployment. A canary exposes a change gradually while watching for problems. These complement pre-release testing; they are not full regression coverage. CISA, the FBI and ASD’s ACSC place testing alongside staged rollout, monitoring and recovery in their joint deployment guidance. [20]

Back to contents

05RESEARCH NOTE

What security checks actually check

I include security in the test plan rather than treating “AI reviewed it” as a separate assurance category. The method should match the failure being investigated. This is a concise map of purposes, not a universal package of scanners.

  1. Code, components and credentials

    SAST / SCA / Secret scanning

    SAST analyses code without running the application. SCA examines actual dependencies and versions for known component risks. Secret scanning looks for exposed credentials in its configured scope. These cover different failure classes; a clean dependency report says little about a faulty access-control rule in your own code. [11]

    An SBOM is an inventory of components, not a successful vulnerability test. FOSS describes free and open-source software, not a testing method. Component and licence checks need the actual software inventory and relevant current evidence. [12]

  2. The running application

    DAST and application-specific tests

    DAST sends test requests to a running application and examines its behaviour. Source inspection alone is not a DAST run. Generic scanners do not automatically understand which person may access which record or which approval step a workflow must enforce. OWASP ASVS explicitly distinguishes automated tools from application-specific verification. [11] [26]

  3. Forbidden actions and unusual inputs

    Authorisation, abuse cases and robustness

    An authorisation test might try another person’s record identifier. An abuse case might bypass an expected workflow step. Fuzzing explores many inputs; property-based tests check specified invariants across generated cases. These need suitable expectations and observation points. A passed happy-path test does not answer whether forbidden actions were rejected. [27] [11]

  4. AI-assisted security analysis

    Provider-specific boundaries

    OpenAI describes Codex Security Cloud as complementing SAST. Anthropic places its security guidance alongside existing static-analysis and dependency scanners and distinguishes source review from testing a running service. Google Cloud describes continuous baseline scanning and deeper AI analysis prioritised by risk. These are scoped product recommendations, not evidence that one model review replaces every security method. [13] [14] [16]

    Anthropic’s separate security-review guide distinguishes an on-demand /security-review command from a GitHub Action that reviews pull requests after installation and configuration. It positions both alongside existing security practices and manual review. Automatic execution therefore depends on the actual integration; it is not implied by using Claude Code or by opening any pull request. [15]

Back to contents

06RESEARCH NOTE

Institutional guidance, with its scope intact

The publications below differ in purpose, specificity and authority. I keep those differences visible instead of merging every recommendation into one supposedly universal minimum.

  1. NIST

    SSDF 1.1 / Developer verification guidance

    SSDF 1.1 connects test selection, execution and finding management to the software lifecycle. IR 8397, published in 2021 in response to Executive Order 14028, gives a more concrete set of developer-verification techniques. Its scope and date matter: recommendations such as fuzzing and structural coverage are part of that document, not a newly established worldwide consensus. [01] [11]

  2. BSI and ANSSI

    Joint AI Coding Assistants guidance / 2024

    The joint guide treats generated tests as an opportunity while addressing the quality and security of generated output separately. Its TDD discussion sits alongside developer understanding and additional testing and review. I read this as encouragement to improve verification, not permission to let the same generation process certify its own adequacy. [10]

  3. ENISA

    Secure by Design and Default Playbook v1.0 / July 2026

    Section 4.9 is concrete about SAST and dependency scanning in pull requests, targeted negative tests, appropriate fuzzing or property testing, and review of sensitive changes. It applies the same checks to AI-generated and human-written code. The playbook also ties the baseline to product context, threats and applicable requirements; its practical checklist should retain that context. [17]

  4. CIS and SAFECode

    Software-security assessment guidance / Evidence of practice

    CIS’s account of its updated assessment guide with SAFECode emphasises observable testing and analysis evidence. It says AI-produced code should undergo the same review, testing and validation processes as human-written software. This is useful for evaluating what a process actually does. It does not make a passing status badge a security conclusion. [18]

  5. NSA and CISA; CISA, FBI and ASD’s ACSC

    Two distinct joint publications / Pipeline and release assurance

    NSA and CISA’s CI/CD guidance treats the pipeline itself as an attack surface, including committed-code analysis, credentials and component information. The separate safe-deployment guide by CISA, the FBI and ASD’s ACSC adds staged rollout, monitoring and recovery. The implication for my reading is that testing the application and protecting its delivery process answer different questions. [19] [20]

  6. ISACA — Assessing AI-coding risks

    Mary Carmichael (2025) and Anjali Gopinadhan Nair (2026) / ISACA Now practitioner articles

    Carmichael’s 2025 article relates testing and oversight to a use case’s likely failures, impact and accountable owner. Nair’s 2026 contribution considers the coding assistants themselves through discovery, risk classification, layered controls and continued monitoring. These are authored practice articles, not formal COBIT requirements or independent demonstrations that a particular test package is sufficient. [21] [22]

  7. ISO/IEC 27001 and VDA ISA

    Management and assessment context

    These references have a different purpose from a software testing guide. ISO/IEC 27001 addresses an information security management system; VDA ISA is an assessment questionnaire used in TISAX. My use of them here is to keep scope, applicable requirements and evidence traceable, rather than turn their names into a universal scanner checklist. The applicable edition must be identified for the assessment in question. [23] [24]

Back to contents

07RESEARCH NOTE

OWASP: three publications, three roles

Even one organisation can publish guidance at different levels. ASVS, WSTG and AISVS should remain separately identifiable.

  1. ASVS 5.0.0

    Application security requirements and verification

    ASVS provides application-security requirements and verification levels. Its assessment guidance asks for explicit scope, checked requirements, methods and justified exclusions. It explains why ordinary scanners cannot entirely verify complex access control or business logic, while application-specific automated tests can contribute. A green scan alone is not verification of every applicable ASVS requirement. [25] [26]

  2. WSTG 4.2

    A flexible security-testing process

    WSTG describes methods across requirements, design, development, deployment and operation. It is explicitly adaptable to an organisation’s process. Its role here is to help connect a security question with a suitable procedure; it does not define an ASVS-style level or an identical scan schedule for every application. [28]

  3. AISVS 1.0, Appendix C

    AI-assisted code generation / Requirements within stated levels

    Appendix C addresses AI-assisted code generation. AC.4.2 at Level 2 specifies automated SAST, IAST, DAST, secret, infrastructure-as-code and composition scanning on pull requests containing AI-generated code. AC.4.3 addresses critical findings and merge blocking. These explicit, level-specific requirements should not be quietly weakened into general advice or transferred to every application as a universal minimum. AISVS levels are not ASVS levels. [29]

Back to contents

08RESEARCH NOTE

Agent-built software is not always an AI system

Software written with an AI assistant may contain no model at runtime. Its authorisation, calculations and data handling still need suitable software tests. If the application itself calls a model, there is an additional question: how reliably does that model-assisted behaviour meet the intended task across realistic cases?

NIST’s AI Risk Management Framework treats reliability, validity, safety, security and other trustworthiness characteristics in the context of use. Metrics and thresholds need judgment. My application of that principle is to define representative tasks, expected evidence and unacceptable outcomes before changing prompts, models or retrieval, then evaluate the affected behaviour again. A passing unit suite does not establish the quality of generated answers. [30]

Back to contents

09RESEARCH NOTE

What I take into my experiments

I want a small, explicit chain of evidence: an intended outcome, a relevant failure to guard against, a suitable check, an observed result and a decision about remaining uncertainty. This is my working synthesis of the sources, not a new validated testing framework.

For a changed data boundary I would prioritise negative access tests and the real integration. For a visual change I would inspect the relevant layouts and interaction. For a dependency change I would check the actual version, known vulnerabilities and affected behaviour. Additional depth follows risk and change impact, rather than repeating every possible review without a reason.

The record should state what ran, against which version and environment, with which substitutions and results. Skipped cases, failures and tests that pass only after retries remain visible. Reproducible project commands and required checks can enforce selected gates; an agent’s statement that it is finished cannot substitute for their results. A deterministic trigger also does not make an LLM-based judgment deterministic.

Back to contents

The useful question is not how many tests the agent produced. It is which important behaviours the evidence supports, which failures the checks could detect and what remains outside their scope.

REFERENCES / 30

Sources.

This article compares specific publications, not universal positions of their publishers. Access dates distinguish sources refreshed on 14 October 2026 from earlier documented readings. Where a full document could not be retrieved again, the source note preserves that limitation. The examples and working synthesis are my own interpretation.

  1. NIST — Secure Software Development Framework 1.1 (opens in a new tab)

    GuidancePublished Accessed

    PW.8.1 and PW.8.2, printed p.15 / PDF p.24. This article cites version 1.1 explicitly.

  2. OpenAI — Building an AI-Native Engineering Team (opens in a new tab)

    Engineering articleAccessed

    Testing: What engineers do instead; Delegate / Review / Own; Getting started checklist.

  3. Dominik Kundel and Gabriel Chua — Testing Agent Skills Systematically with Evals (opens in a new tab)

    Engineering articlePublished Accessed

    OpenAI developer article. Sections 1 and 4: define success; deterministic checks and rubric-based grading. Applies to agent-skill evaluations, not a universal application-testing mandate.

  4. OpenAI — Best practices (opens in a new tab)

    DocumentationAccessed

    Improve reliability with testing and review. Maintained documentation.

  5. Anthropic — Best practices for Claude Code (opens in a new tab)

    DocumentationAccessed

    Give Claude a way to verify its work. Maintained documentation; configuration and execution surfaces matter.

  6. Anthropic — Desktop application (opens in a new tab)

    DocumentationAccessed

    Auto-verify changes. Desktop-specific capability; not a complete regression-test guarantee.

  7. Google — What to look for in a code review (opens in a new tab)

    GuidanceAccessed

    Tests. General engineering guidance, not an evaluation of a particular AI product.

  8. Google — Jules Agent Update: Faster, Smarter, More Reliable (opens in a new tab)

    DocumentationPublished Accessed

    Better test habits. Historical product announcement; probabilistic wording.

  9. Kathy Korevec, Mehadi Hassen and Yiguang (Amanda) Zhang — Meet Jules’ sharpest critic and most valuable ally (opens in a new tab)

    Engineering articlePublished Accessed

    Critic augmented generation; What the critic does; Here’s how it works. Historical feature description.

  10. BSI and ANSSI — AI Coding Assistants (opens in a new tab)

    GuidancePublished Accessed

    Sections 2.3 and 3.3, PDF pp.6 and 9. Detailed reading retained from 1 August 2026; publisher page rechecked on 14 October, but a full PDF retrieval was unavailable in this update.

  11. NIST — IR 8397: Guidelines on Minimum Standards for Developer Verification of Software (opens in a new tab)

    GuidancePublished Accessed

    Sections 1.1–1.3 and 2.1–2.11. Dated guidance responding to Executive Order 14028; techniques are scoped, not a universal contemporary mandate.

  12. OWASP — Developer Guide: Dependencies (opens in a new tab)

    GuidanceAccessed

    Dependency management, component inventory, software bills of materials and software composition analysis.

  13. OpenAI — Codex Security Cloud FAQ (opens in a new tab)

    DocumentationAccessed

    Does it replace SAST? Specific to the documented security offering.

  14. Anthropic — Catch security issues as Claude writes code (opens in a new tab)

    DocumentationAccessed

    How this fits with other security tools; source review versus a running site.

  15. Anthropic — Automated Security Reviews in Claude Code (opens in a new tab)

    DocumentationPage date Accessed

    On-demand reviews and the configured GitHub Action. The page displays 16 March 2026 without identifying it as initial publication or last update.

  16. Google Cloud — Vulnerability management with AI (opens in a new tab)

    DocumentationAccessed

    Phase 2: Scan and prioritize. Baseline scans, deep AI scans and risk-based prioritisation.

  17. ENISA — Secure by Design and Default Playbook v1.0 (opens in a new tab)

    GuidancePublished Accessed

    Sections 4.9 and 4.23; published version 1.0, distinct from the consultation draft.

  18. CIS — Turning Secure Software Development into a Measurable Practice (opens in a new tab)

    GuidanceAccessed

    Expanded Guidance on Artificial Intelligence; A Continued Focus on Evidence. Describes the CIS/SAFECode assessment guide update.

  19. NSA and CISA — Defending CI/CD Environments (opens in a new tab)

    GuidancePublished Accessed

    Code analysis, credentials and SBOM/SCA, pp.11–15. Earlier full-document reading retained; indexed excerpts were available on 14 October 2026, but full PDF retrieval was unavailable.

  20. CISA, FBI and ASD’s ACSC — Safe Software Deployment (opens in a new tab)

    GuidancePublished Accessed

    Introduction; Objectives; Key phases. Full text hosted by a coauthor; Australian page dated 25 October 2024.

  21. Mary Carmichael — Is Vibe Coding Ready for Prime Time? (opens in a new tab)

    Practitioner accountPublished Accessed

    Adopting a Risk-Based Approach. Authored practitioner article published by ISACA, not a formal standard.

  22. Anjali Gopinadhan Nair — Securing the AI Frontier (opens in a new tab)

    Practitioner accountPublished Accessed

    A Four-Phase Framework. Authored practitioner article; its reported performance figures are not adopted here.

  23. ISO — ISO/IEC 27001:2022 (opens in a new tab)

    GuidancePublished Accessed

    Public description of the management-system standard only; no claim of complete normative assessment.

  24. ENX — VDA ISA and TISAX Download Center (opens in a new tab)

    GuidanceAccessed

    Version and assessment-scope information. Applicable editions are identified separately; this article is not an ISA control mapping.

  25. OWASP ASVS 5.0.0 — What is the ASVS? (opens in a new tab)

    GuidancePublished Accessed

    Scope of the ASVS; Application Security Verification Levels. Version-pinned source.

  26. OWASP ASVS 5.0.0 — Assessment and Certification (opens in a new tab)

    GuidancePublished Accessed

    Verification reporting; Scope of Verification; Verification Mechanisms; The Role of Automated Security Testing Tools.

  27. OWASP WSTG 4.2 — Introduction (opens in a new tab)

    GuidancePublished Accessed

    Risk-Driven Security Requirements; The Need for a Balanced Approach; Test Automation. Version-pinned source.

  28. OWASP WSTG 4.2 — The Web Security Testing Framework (opens in a new tab)

    GuidancePublished Accessed

    Overview and lifecycle phases. Flexible framework; distinct from a verification level.

  29. OWASP AISVS 1.0 — Appendix C: AI for Code Generation (opens in a new tab)

    GuidanceAccessed

    AC.4.2 and AC.4.3, Level 2. Mutable main branch within the 1.0 directory; this article refers to the retrieved wording.

  30. NIST AI RMF 1.0 — AI Risks and Trustworthiness (opens in a new tab)

    GuidancePublished Accessed

    Section 3 and 3.1: context, metrics, thresholds, validity and reliability. Applies to AI-system behaviour, not merely AI-written code.

Back to contents