Claude-Code-Game-Studios/CCGS Skill Testing Framework/skills/analysis/test-evidence-review.md

Skill Test Spec: /test-evidence-review

Skill Summary

/test-evidence-review reviews the quality of the test files and evidence documents behind stories — /smoke-check covers existence and passing. It takes a scope: a single story path, sprint (the most recent plan in production/sprints/), or a system name (production/epics/[system]/story-*.md); with no argument it asks which. If the resolved scope holds zero stories it stops with NOT ASSESSED — no stories in scope and routes to the skill that creates them. For each story it collects Story Type, evidence path and acceptance criteria with targeted greps, locates the evidence (tests/unit/[system]/, tests/integration/[system]/, production/qa/evidence/, production/qa/smoke-*.md), and reviews automated tests for assertion count, edge cases, naming and formula traceability, and manual evidence for criterion linkage, sign-offs, retained screenshots and freshness. It resolves qa.level and testing.strict the way /story-done does: at qa.level: minimal a Logic, Integration or Config/Data story with no test evidence is WAIVED (qa.level: minimal), not MISSING, and a gap in a test that does exist is ADVISORY; Visual/Feel and UI evidence is never waived; otherwise each gap takes its story type's gate level from testing.strict.<type> (unset → a plain-boolean testing.strict legacy value applies to every type; else the coding-standards.md default). Each story gets ADEQUATE, INCOMPLETE, MISSING, NOT ASSESSED or WAIVED; the overall verdict is the worst present, with NOT ASSESSED outranking ADEQUATE only, and WAIVED below ADEQUATE. The report is presented in conversation, and the skill asks "May I write this test evidence review to production/qa/evidence-review-[date].md?" as an optional step. It never modifies test files or evidence docs and invokes no director gates. The run ends COMPLETE, or CONCERNS when BLOCKING items were found.

---

Static Assertions (Structural)

Checked against the SKILL.md by /skill-test spec — no fixture needed.

• Has required frontmatter fields: name, description, argument-hint, user-invocable, allowed-tools
• Has ≥2 phase headings
• Contains verdict keywords: ADEQUATE, INCOMPLETE, MISSING, NOT ASSESSED, and the run-level COMPLETE / CONCERNS
• Contains "May I write" language for the optional production/qa/evidence-review-[date].md
• Has a next-step handoff (Section 7: BLOCKING items before /story-done, /test-helpers, sign-off routing)
• The resolve_config injection names qa.level and testing.strict, and the skill states the WAIVED (qa.level: minimal) result and the per-type testing.strict.<type> gate level

---

Director Gate Checks

None. Test evidence review is an advisory quality skill; no gates are invoked.

---

Test Cases

Case 1: Happy Path — Single Logic story with a strong test

Fixture:

• project.yaml: engine.name: godot, so the unit-test root is tests/unit/; modes.automation unset (collaborative)
• production/epics/combat/story-001-damage-calc.md (slug damage-calc): ## Test Evidence gives Story Type Logic and the path tests/unit/combat/damage-calc_test.gd; ## Acceptance Criteria includes "damage is never below 1" and "critical hits multiply damage by 2.0"
• tests/unit/combat/damage-calc_test.gd opens with # Story: production/epics/combat/story-001-damage-calc.md, has 4 test functions with 3+ assertions each, named like test_damage_calc_zero_armor_returns_base_damage and test_damage_calc_max_crit_doubles_damage, and names the GDD's damage_formula in a comment

Input: /test-evidence-review production/epics/combat/story-001-damage-calc.md

Expected behavior:

1. Single-story mode: skill reads the story's Test Evidence and Acceptance Criteria
2. Section 3 finds the test file under tests/unit/combat/
3. Section 4: assertion coverage is normal (3+ per function); "zero" and "max" cases cover the numeric criteria; names follow test_[scenario]_[expected]; the formula is referenced
4. Story verdict: ADEQUATE; overall verdict ADEQUATE; 0 BLOCKING items
5. Report is presented in conversation, then the skill asks "May I write this test evidence review to production/qa/evidence-review-[date].md?" as optional
6. Verdict: COMPLETE

Assertions:

• The test file is located under tests/unit/combat/ for the Logic story
• Assertion coverage, edge cases, naming and formula traceability are each reported
• Story verdict is ADEQUATE
• The report is shown before the optional "May I write" prompt
• Run ends with COMPLETE

---

Case 2: Vacuous and Thin Tests — INCOMPLETE with a BLOCKING item

Fixture:

• project.yaml: engine.name: godot, modes.rigor: standard (so qa.level: standard — tests are required); no testing.strict block
• production/epics/combat/story-002-crit-roll.md (slug crit-roll): Story Type Logic
• tests/unit/combat/crit-roll_test.gd has func test_1() with no assertions and func test_crit_roll_applies_multiplier() with one assertion

Input: /test-evidence-review production/epics/combat/story-002-crit-roll.md

Expected behavior:

1. Skill finds the test file and counts assertions per function
2. test_1 has 0 assertions → flagged BLOCKING (passes vacuously, proves nothing); test_1 is also flagged as a generic name
3. The one-assertion function is noted as potentially thin
4. Story verdict: INCOMPLETE
5. Follow-ups: "These must be resolved before /story-done can mark the story Complete…" and "Consider running /test-helpers [system]…"
6. Run ends with CONCERNS because BLOCKING items were found

Assertions:

• A test function with zero assertions is flagged BLOCKING
• test_1 is flagged as a naming issue
• Story verdict is INCOMPLETE, not ADEQUATE
• Output states BLOCKING items must be resolved before /story-done and suggests /test-helpers
• Run ends with CONCERNS
• The test file is not modified
• Variant — with testing.strict.logic: false in project.yaml, the zero-assertion finding is ADVISORY, the story is still INCOMPLETE, and the run ends COMPLETE

---

Case 3: Empty Sprint Scope at Minimal Workflow — NOT ASSESSED, not a clean report

Fixture:

• project.yaml: modes.rigor: minimal (the resolved block shows workflow: minimal)
• production/sprints/ does not exist

Input: /test-evidence-review sprint

Expected behavior:

1. Sprint mode finds no sprint plan, so the scope holds zero stories
2. Skill stops and reports NOT ASSESSED — no stories in scope, naming the sprint scope and the empty production/sprints/ path
3. Because the workflow is minimal (no sprints), it routes to /test-evidence-review [epic-slug] instead of /sprint-plan new
4. Skill does not continue to Section 3; no Summary table and no "BLOCKING items: 0" line are rendered

Assertions:

• Verdict is NOT ASSESSED — no stories in scope
• Output names which scope was searched and which path was empty
• At minimal workflow the route is /test-evidence-review [epic-slug], not /sprint-plan new
• No empty report reading "BLOCKING items: 0 / ADVISORY items: 0" is produced

---

Case 4: Story Type Unknown — Per-story NOT ASSESSED lifts the overall verdict

Fixture:

• project.yaml: engine.name: godot
• production/epics/combat/ contains story-001-damage-calc.md (Logic, test as in Case 1) and story-003-hit-feedback.md, whose ## Test Evidence section states no Story Type and no evidence path
• Full-reading story-003-hit-feedback.md does not reveal a type either

Input: /test-evidence-review combat

Expected behavior:

1. System mode globs production/epics/combat/story-*.md and finds both stories
2. story-001-damage-calc is ADEQUATE
3. story-003-hit-feedback is NOT ASSESSED, with the reason stated: the story type cannot be determined, so the required evidence is unknown
4. It is not reported as MISSING
5. Overall verdict: NOT ASSESSED — it outranks ADEQUATE

Assertions:

• The untyped story is NOT ASSESSED, not MISSING
• The NOT ASSESSED reason (type cannot be determined) is stated for that story
• Overall verdict is NOT ASSESSED, not ADEQUATE
• The typed story is still reviewed and reported ADEQUATE

---

Case 5: Visual/Feel Evidence Without a Retained Screenshot, Full Review Mode — No gates

Fixture:

• project.yaml: engine.name: godot, modes.review_mode: full
• production/epics/combat/story-004-hit-flash.md (slug hit-flash): Story Type Visual/Feel
• production/qa/evidence/hit-flash-evidence.md references every acceptance criterion, has every sign-off filled, and is dated after the sprint start
• No *.png, *.jpg or *.gif for this story exists in production/qa/evidence/

Input: /test-evidence-review production/epics/combat/story-004-hit-flash.md

Expected behavior:

1. Section 3 finds production/qa/evidence/hit-flash-evidence.md
2. Section 5: criterion linkage and sign-offs are complete, but no retained image exists for the story
3. The doc describes a visual check without evidence → story verdict INCOMPLETE, flagged BLOCKING
4. No director gate is invoked regardless of review mode
5. The evidence document is not edited
6. Run ends with CONCERNS

Assertions:

• A Visual/Feel evidence doc with no retained image is INCOMPLETE, even with complete sign-offs
• The missing screenshot is listed as a BLOCKING issue
• No director gate is invoked in any review mode
• The evidence document is not modified

---

Case 6: UI Story Closed by Its Screenshots — No Sign-Off Required

Fixture:

• project.yaml: engine.name: godot
• production/epics/ui/story-007-inventory-screen.md (slug inventory-screen): Story Type UI; its ## Test Evidence names production/qa/evidence/inventory-screen-populated.png and production/qa/evidence/inventory-screen-empty.png — one per screen state it touches
• Both images exist and are dated after the sprint start
• No inventory-screen-evidence.md, no sign-off record and no walkthrough log exist

Input: /test-evidence-review production/epics/ui/story-007-inventory-screen.md

Expected behavior:

1. Section 3 uses the two paths the story states before searching
2. Section 5: a UI story is closed by the retained screenshot of each screen it touched
3. Story verdict ADEQUATE; run ends with COMPLETE

Assertions:

• The story's stated evidence paths are checked first, and both images are found
• The missing sign-off does not make the story INCOMPLETE — a UI story needs none
• No walkthrough log is demanded
• Story verdict is ADEQUATE

---

Case 7: Unity Logic Story — Evidence at the Stated Engine Path

Fixture:

• project.yaml: engine.name: unity
• production/epics/inventory/story-003-stack-merge.md: Story Type Logic; its ## Test Evidence names Assets/Tests/EditMode/InventoryTests.cs
• That file exists and holds one [Test] per acceptance criterion (4 in all), each with 3 or more Assert.That calls, named in the [Scenario]_[Expected] form (MergeStacks_FullStack_SpillsToNewSlot), with the empty-stack and max-stack cases covered
• The inventory GDD has no Formulas section, so formula traceability does not apply
• Nothing named stack-merge_test.* exists anywhere, and tests/unit/ does not exist

Input: /test-evidence-review production/epics/inventory/story-003-stack-merge.md

Expected behavior:

1. Section 3 finds the test at the stated path under Assets/Tests/EditMode/
2. Section 4 reviews its assertions; story verdict ADEQUATE

Assertions:

• The test is found at the stated Unity path — the story is not reported MISSING
• The review does not require a [story-slug]_test.* name or a tests/unit/ location

---

Case 8: No Test for a Logic Story — MISSING, and it outranks NOT ASSESSED

Fixture:

• project.yaml: engine.name: godot, modes.rigor: standard (so qa.level: standard — a Logic story's test is required)
• production/epics/combat/ contains story-003-hit-feedback.md (no Story Type, as in Case 4) and story-005-knockback.md: Story Type Logic, no evidence path stated
• No test exists for it: tests/unit/combat/ does not exist, and no other test file names the slug knockback or references the story

Input: /test-evidence-review combat

Expected behavior:

1. story-005-knockback is a Logic story, so a unit test is required; the search of the unit-test root finds none
2. story-005-knockback is MISSING — the review ran and found nothing — and the gap is a BLOCKING item
3. story-003-hit-feedback is NOT ASSESSED (its type cannot be determined), not MISSING
4. Overall verdict: MISSING — a known failure outranks the story that could not be assessed
5. Run ends with CONCERNS because a BLOCKING item was found

Assertions:

• The Logic story with no test file is MISSING, not NOT ASSESSED
• The untyped story is NOT ASSESSED, not MISSING — the two are kept apart
• Overall verdict is MISSING, not NOT ASSESSED
• The missing test is listed as BLOCKING and the run ends with CONCERNS

---

Case 9: qa.level: minimal — a Logic Story With No Test Is WAIVED; a UI Screenshot Is Not

Fixture:

• project.yaml: engine.name: godot, no modes block (so qa.level: minimal — tests are waived)
• production/epics/combat/ contains story-005-knockback.md (Logic, no test anywhere, as in Case 8) and story-008-pause-menu.md (UI, no retained image for it in production/qa/evidence/)

Input: /test-evidence-review combat

Expected behavior:

1. story-005-knockback is WAIVED (qa.level: minimal) — not MISSING — and raises no BLOCKING item
2. story-008-pause-menu is MISSING and BLOCKING — the retained screenshot a UI story needs is never waived
3. Overall verdict: MISSING; the report's header shows qa.level: minimal
4. Run ends with CONCERNS because of the UI story alone
5. Variant — without the UI story, the overall verdict is WAIVED, there are 0 BLOCKING items, and the run ends COMPLETE

Assertions:

• The Logic story with no test is WAIVED (qa.level: minimal), not MISSING, and is not listed as BLOCKING
• The UI story with no screenshot is still MISSING and BLOCKING at qa.level: minimal
• The waived story is named as waived in the report, not dropped from it
• Variant: an all-waived scope reads WAIVED, with no "must be resolved before /story-done" prompt

---

Protocol Compliance

• Collects Story Type, evidence path and acceptance criteria with targeted ## Test Evidence / ## Acceptance Criteria greps, full-reading a story only when its Test Evidence section is missing or ambiguous
• An empty scope stops with NOT ASSESSED — no stories in scope, naming the scope and routing onward
• Each story gets ADEQUATE, INCOMPLETE, MISSING or NOT ASSESSED, and a NOT ASSESSED story states its reason
• Overall verdict is the worst story verdict; NOT ASSESSED outranks ADEQUATE but not INCOMPLETE or MISSING; WAIVED ranks below ADEQUATE
• At qa.level: minimal a Logic, Integration or Config/Data story with no test evidence is WAIVED, never MISSING or BLOCKING; Visual/Feel and UI evidence is never waived
• BLOCKING vs ADVISORY per story type comes from testing.strict.<type> in the resolved block, falling back to a legacy plain-boolean testing.strict value applied to every type, then to the coding-standards.md default
• The report is presented before the optional "May I write" for production/qa/evidence-review-[date].md
• Does not modify test files or evidence documents
• No director gates are invoked
• Ends with COMPLETE, or CONCERNS when BLOCKING items were found — except an empty scope, which stops at NOT ASSESSED — no stories in scope before any report (Case 3)

---

Coverage Notes

• Integration stories (playtest records in production/session-logs/) and Config/Data stories (production/qa/smoke-*.md) are not tested here.
• The no-argument path (the skill asks which scope to review) is not tested.
• The POTENTIALLY STALE freshness flag is not tested separately.