ksh999000/Claude-Code-Game-Studios
1
0
Fork 0
mirror of https://github.com/Donchitos/Claude-Code-Game-Studios.git synced 2026-10-16 01:37:05 +00:00
Code Issues Packages Projects Releases Wiki Activity

Release 1.1.2 — the default path works end to end

Browse Source 
Fix release for 1.1.1: the minimal path end to end, engine commands that
run on Godot, Unity and Unreal, hook warnings you can see, and verdicts
that say when they could not check. Details in CHANGELOG.md; upgrade
steps in UPGRADING.md (v1.1.1 → v1.1.2).

Fixes #67
Fixes #131

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WuJn2FdynEybfL5KC697nJ
This commit is contained in:
Donchitos
2026-09-29 21:32:19 +10:00
co-authored by Claude Opus 5.5
parent 7ed2c3e9c4
commit b21fa0f7f2
383 changed files with 22888 additions and 10618 deletions
Download Patch File Download Diff File Expand all files Collapse all files
CCGS Skill Testing Framework/skills/analysis/asset-audit.md
+202 -74
View File
@@ -2,22 +2,34 @@
## Skill Summary
`/asset-audit` audits the `assets/` directory for naming convention compliance,
missing metadata, and format/size issues. It reads asset files against the
conventions and budgets defined in `technical-preferences.md`. No director gates
are invoked. The skill does not write without user approval. Verdicts: COMPLIANT,
WARNINGS, or NON-COMPLIANT.
`/asset-audit` audits the engine's asset root — `assets/` (art, audio, vfx,
shaders, data) on Godot, `Assets/` on Unity, `Content/` on Unreal — for naming
convention compliance, size budgets, format standards, orphaned assets and
missing assets. It reads the art bible or asset standards from the design docs
(Phase 1), resolves the engine and its code root and scans the asset root with
Glob (Phase 2), measures sizes and formats with the pre-approved read-only
commands (`stat`, `file`, `od`, `wc`, `du`), and checks Godot names against its
own patterns — art `[category]_[name]_[variant]_[size].[ext]`, audio
`[category]_[context]_[name]_[variant].[ext]`, all lowercase with underscores —
and Unity and Unreal names only against a rule the art bible states. Orphaned
and missing assets are found by searching the **code root** for asset
references; GDD-level completeness is `/content-audit`'s job. The report lists
violations only, in one table per category, and names every section it could
not assess. A measurement it cannot take is `NOT ASSESSED`, never an estimate.
The skill is read-only: it writes no files and invokes no director gates.
Verdicts, first match wins: NON-COMPLIANT, WARNINGS, NOT ASSESSED, COMPLIANT —
COMPLIANT only when at least one asset was scanned and every section assessed.
---
## Static Assertions (Structural)
Verified automatically by `/skill-test static` — no fixture needed.
Checked against the SKILL.md by `/skill-test spec` — no fixture needed.
- [ ] Has required frontmatter fields: `name`, `description`, `argument-hint`, `user-invocable`, `allowed-tools`
- [ ] Has ≥2 phase headings
- [ ] Contains verdict keywords: COMPLIANT, WARNINGS, NON-COMPLIANT
- [ ] Does NOT require "May I write" language (read-only; optional report requires approval)
- [ ] Contains verdict keywords: COMPLIANT, WARNINGS, NON-COMPLIANT, NOT ASSESSED
- [ ] Does NOT require "May I write" language (read-only: `allowed-tools` has no Write/Edit and the skill states it does not write files)
- [ ] Has a next-step handoff (what to do after audit results)
---
@@ -33,24 +45,25 @@ None. Asset auditing is a read-only analysis skill; no gates are invoked.
### Case 1: Happy Path — All assets follow naming conventions
**Fixture:**
- `technical-preferences.md` specifies naming convention: `snake_case`, e.g., `enemy_grunt_idle.png`
- `assets/art/characters/` contains: `enemy_grunt_idle.png`, `enemy_sniper_run.png`
- `assets/audio/sfx/` contains: `sfx_jump_land.ogg`, `sfx_item_pickup.ogg`
- All files are within size budget (textures ≤2MB, audio ≤500KB)
- `project.yaml`: `engine.name: godot`, so the asset root is `assets/` and the code root `src/`
- `design/art/art-bible.md` Asset Standards: textures PNG, power-of-two, ≤2MB; SFX OGG, 44.1 kHz, ≤500KB (no duration budget)
- `assets/art/characters/` contains: `char_grunt_idle_512.png`, `char_sniper_run_512.png` (512×512 PNG, 300KB each)
- `assets/audio/sfx/` contains: `sfx_player_jump_01.ogg`, `sfx_item_pickup_01.ogg` (OGG, 44.1 kHz, 80KB and under 1 s each)
- `src/` references all four files
**Input:** `/asset-audit`
**Expected behavior:**
1. Skill reads naming conventions and size budgets from `technical-preferences.md`
2. Skill scans `assets/` recursively
3. All files match `snake_case` convention; all within budget
4. Audit table shows all rows PASS
1. Skill reads the asset standards from `design/art/art-bible.md` (Phase 1)
2. Skill resolves the engine to Godot and globs `assets/art/**/*` and `assets/audio/**/*` (Phase 2)
3. Every name matches the Phase 3 art or audio pattern; every file is within budget and in the expected format; every file is referenced from code
4. Summary shows 4 assets scanned, 0 in every violation count and no section not assessed; the violation tables are empty
5. Verdict is COMPLIANT
**Assertions:**
- [ ] Audit covers both art and audio asset directories
- [ ] Each file is checked against naming convention and size budget
- [ ] All rows show PASS when compliant
- [ ] Each file is checked against the Phase 3 naming pattern and the size budget from the art bible
- [ ] Summary reports `Total assets scanned: 4` with zero naming, size, format, orphaned and missing counts, and `Sections not assessed: none`
- [ ] Verdict is COMPLIANT
- [ ] No files are written
@@ -59,112 +72,227 @@ None. Asset auditing is a read-only analysis skill; no gates are invoked.
### Case 2: Non-Compliant — Textures exceed size budget
**Fixture:**
- `assets/art/environment/` contains 5 texture files
- 3 texture files are 4MB each (budget: ≤2MB)
- 2 texture files are within budget
- `project.yaml`: `engine.name: godot`
- `design/art/art-bible.md` asset standards: textures PNG, power-of-two, ≤2MB
- `assets/art/environment/` contains 5 correctly named power-of-two PNG textures, all referenced from code
- 3 of them are 4096×4096 at 4MB each; 2 are 1024×1024 at 1MB each
**Input:** `/asset-audit`
**Expected behavior:**
1. Skill reads size budget from `technical-preferences.md` (2MB for textures)
2. Skill scans `assets/art/environment/` — finds 3 oversized textures
3. Audit table lists each oversized file with actual size and budget
4. Verdict is NON-COMPLIANT
5. Skill recommends compression or resolution reduction for flagged files
1. Skill reads the 2MB texture budget from the art bible
2. Skill measures each file's size with `stat` or `du` — not by estimate
3. The Size Violations table lists the 3 oversized files with Budget, Actual and Overage
4. The 2 within-budget files do not appear in the Size Violations table; `Total assets scanned` counts all 5
5. Recommendations give each oversized file its own fix, naming the target — a resolution (such as 2048×2048) or a compression setting that brings it under 2MB
6. Verdict is NON-COMPLIANT
**Assertions:**
- [ ] All 3 oversized files are listed by name with actual size and budget size
- [ ] All 3 oversized files appear in the Size Violations table with Budget (2MB), Actual (4MB) and Overage
- [ ] Sizes come from a measuring command (`stat`/`du`), not an estimate
- [ ] Verdict is NON-COMPLIANT when any file exceeds its budget
- [ ] Optimization recommendation is given for oversized files
- [ ] Within-budget files are also listed (showing PASS) for completeness
- [ ] Each oversized file has a recommendation that names its target resolution or compression, not only "reduce size"
- [ ] The 2 within-budget files are counted in `Total assets scanned` but are not listed as violations
---
### Case 3: Format Issue — Audio in wrong format
### Case 3: Format Issue — Music in wrong format
**Fixture:**
- `technical-preferences.md` specifies audio format: OGG
- `assets/audio/music/theme_main.wav` exists (WAV format)
- `assets/audio/sfx/sfx_footstep.ogg` exists (correct OGG format)
- `project.yaml`: `engine.name: godot`
- `design/art/art-bible.md` asset standards: SFX OGG ≤500KB; music OGG or MP3 ≤8MB
- `assets/audio/music/music_menu_theme_01.wav` exists (WAV format, 6MB), referenced from code
- `assets/audio/sfx/sfx_player_footstep_01.ogg` exists (OGG format, 40KB), referenced from code
**Input:** `/asset-audit`
**Expected behavior:**
1. Skill reads audio format requirement: OGG
2. Skill scans `assets/audio/` — finds `theme_main.wav` in wrong format
3. Audit table flags `theme_main.wav` as FORMAT ISSUE (expected OGG, found WAV)
4. `sfx_footstep.ogg` shows PASS
5. Verdict is WARNINGS (format issues are correctable)
1. Skill applies the Phase 3 audio format standard (OGG for SFX, OGG/MP3 for music)
2. Skill identifies the WAV file's format with `file` or a header read
3. The Format Violations table lists `music_menu_theme_01.wav` with expected OGG/MP3 and actual WAV
4. `sfx_player_footstep_01.ogg` does not appear in any violation table; both files are within their size budgets
5. Recommendations name the target format for the WAV file (re-export as OGG or MP3)
6. Verdict is WARNINGS (a format issue is correctable)
**Assertions:**
- [ ] `theme_main.wav` is flagged as FORMAT ISSUE with expected and actual format noted
- [ ] Verdict is WARNINGS (not NON-COMPLIANT) for format issues, which are correctable
- [ ] Correct-format assets are shown as PASS
- [ ] `music_menu_theme_01.wav` appears in the Format Violations table with Expected Format OGG/MP3 and Actual Format WAV
- [ ] The recommendation for the WAV file names OGG or MP3 as its target format
- [ ] Verdict is WARNINGS (not NON-COMPLIANT) for a format issue alone
- [ ] `sfx_player_footstep_01.ogg` is not listed in any violation table
- [ ] Skill does not modify or convert any asset files
---
### Case 4: Missing Asset — Asset referenced by GDD but absent from assets/
### Case 4: Missing Asset — Referenced by code but absent from assets/
**Fixture:**
- `design/gdd/enemies.md` references `enemy_boss_idle.png`
- `assets/art/characters/boss/` directory is empty — file does not exist
- `project.yaml`: `engine.name: godot`; `design/art/art-bible.md` sets texture budgets
- `assets/art/characters/` holds 3 other correctly named textures, within budget and referenced from code
- `src/enemies/boss.gd` loads `res://assets/art/characters/boss/char_boss_idle_512.png`
- `assets/art/characters/boss/` is empty — the file does not exist
- `design/gdd/enemies.md` also names `char_boss_roar_512.png`, which no code references
**Input:** `/asset-audit`
**Expected behavior:**
1. Skill reads GDD references to find expected assets (cross-references with `/content-audit` scope)
2. Skill scans `assets/art/characters/boss/` — file not found
3. Audit table flags `enemy_boss_idle.png` as MISSING ASSET
4. Verdict is NON-COMPLIANT (missing critical art asset)
1. Skill searches code for asset references (Phase 3 "Missing assets") and checks each path exists
2. `char_boss_idle_512.png` is not found
3. The Missing Assets table lists Reference Location `src/enemies/boss.gd` and the Expected Path
4. The GDD-only name `char_boss_roar_512.png` is not reported — the missing-asset check searches code, not GDDs
5. Verdict is NON-COMPLIANT
6. Next steps point to `/content-audit` for the GDD cross-check
**Assertions:**
- [ ] Skill checks GDD references to identify expected assets
- [ ] Missing assets are flagged as MISSING ASSET with the GDD reference noted
- [ ] Verdict is NON-COMPLIANT when critical assets are missing
- [ ] The Missing Assets table lists `src/enemies/boss.gd` as the reference location and `assets/art/characters/boss/char_boss_idle_512.png` as the expected path
- [ ] The GDD-only asset name is not reported as missing
- [ ] Verdict is NON-COMPLIANT when a code-referenced asset is missing
- [ ] Next steps name `/content-audit` for GDD-vs-asset completeness
- [ ] Skill does not create or add placeholder assets
---
### Case 5: Gate Compliance — No gate; technical-artist may be consulted separately
### Case 5: No Data — No assets and no asset standards
**Fixture:**
- 2 files have naming convention violations (CamelCase instead of snake_case)
- `review-mode.txt` contains `full`
- `project.yaml`: `engine.name: godot`, `modes.review_mode: full`
- `assets/` does not exist
- `design/art/art-bible.md` does not exist; no design doc defines asset standards
**Input:** `/asset-audit`
**Expected behavior:**
1. Skill scans assets and finds 2 naming violations
2. No director gate is invoked regardless of review mode
3. Verdict is WARNINGS
4. Output notes: "Consider having a Technical Artist review naming conventions"
5. Skill presents findings; offers optional audit report write
6. If user opts in: "May I write to `production/qa/asset-audit-[date].md`?"
1. Skill runs its insufficient-input check before producing a report: records each input as FOUND or ABSENT
2. Every required input is ABSENT, so the skill stops
3. Verdict is `NOT ASSESSED — NO DATA`, naming what was missing and which skill produces it
4. The skill does not report COMPLIANT and does not render zero-count violation tables as a clean pass
5. No director gate is invoked in any review mode; no file is written and no write is offered
**Assertions:**
- [ ] No director gate is invoked in any review mode
- [ ] Technical artist consultation is suggested (not mandated)
- [ ] Findings table is presented before any write prompt
- [ ] Optional audit report write asks "May I write" before writing
- [ ] Inputs are listed as FOUND / ABSENT before any report section
- [ ] Verdict is `NOT ASSESSED — NO DATA`, not COMPLIANT
- [ ] The report names the missing inputs (`assets/`, asset standards) and the skill that produces the standards (`/art-bible`)
- [ ] No director gate is invoked, even with review mode `full`
- [ ] No "May I write" prompt appears and no file is written
---
### Case 6: Zero Assets — Standards exist, nothing to audit
**Fixture:**
- `project.yaml`: `engine.name: godot`
- `design/art/art-bible.md` asset standards: textures PNG ≤2MB; SFX OGG ≤500KB
- `assets/` does not exist; `src/` holds gameplay code that loads no asset yet
**Input:** `/asset-audit`
**Expected behavior:**
1. The art bible is FOUND, so the insufficient-input check does not stop the run
2. Phase 2 globs `assets/` and matches nothing: `Total assets scanned: 0`
3. Naming, Size, Format and Orphaned are `NOT ASSESSED — no assets under assets/`, and the Summary lists them; the missing-asset search runs and finds no reference
4. Verdict is NOT ASSESSED — zero violations over zero files is not compliance
**Assertions:**
- [ ] `Total assets scanned` is 0 and the four file checks are NOT ASSESSED with that reason
- [ ] Verdict is NOT ASSESSED, not COMPLIANT
- [ ] No file is written
---
### Case 7: No Standards — Assets exist, no size budget anywhere
**Fixture:**
- `project.yaml`: `engine.name: godot`
- `design/art/art-bible.md` does not exist; no design doc states asset standards
- `assets/art/ui/` holds 3 correctly named power-of-two PNGs; `src/` references all three
**Input:** `/asset-audit`
**Expected behavior:**
1. Assets are FOUND and standards ABSENT, so the skill runs rather than stopping
2. Naming, Format, Orphaned and Missing are checked; none finds a violation
3. Size is `NOT ASSESSED — no size budget set`, and the report names `/art-bible` as the skill that writes the standards
4. The Summary lists Size under Sections not assessed
5. Verdict is NOT ASSESSED, naming Size
**Assertions:**
- [ ] No size is judged against an invented budget
- [ ] The Size section is `NOT ASSESSED — no size budget set` and `/art-bible` is named
- [ ] Verdict is NOT ASSESSED, not COMPLIANT
---
### Case 8: Unreal Project — Assets under `Content/`
**Fixture:**
- `project.yaml`: `engine.name: unreal`, so the code root is `Source/<Module>/`
- `design/art/art-bible.md` asset standards: textures ≤4MB; no naming convention stated
- `Content/Characters/Grunt/` holds `T_Grunt_D.uasset` (2.5MB) and `T_Grunt_N.uasset` (2.1MB)
- `assets/` does not exist
- `Source/Arena/Private/GruntCharacter.cpp` loads `/Game/Characters/Grunt/T_Grunt_D`
**Input:** `/asset-audit`
**Expected behavior:**
1. Phase 2 globs `Content/**/*.uasset` and `Content/**/*.umap`, not `assets/`: `Total assets scanned: 2`
2. Both sizes are measured against the 4MB texture budget; neither is over
3. The PascalCase names are not checked against the Godot lowercase patterns; with no naming rule in the art bible, Naming is `NOT ASSESSED — no naming rule for Unreal`, and `/art-bible` is named
4. Format is NOT ASSESSED (an Unreal asset file does not keep its source format) and Orphaned is NOT ASSESSED (Unreal assets are referenced from other assets, not only from code)
5. The code reference resolves to an existing asset, so Missing has no finding
6. Verdict is NOT ASSESSED, naming Naming, Format and Orphaned
**Assertions:**
- [ ] The scan covers `Content/` and counts 2 assets, not 0
- [ ] No naming violation is raised from the Godot lowercase patterns
- [ ] Naming, Format and Orphaned are each listed as NOT ASSESSED with a reason
- [ ] Verdict is NOT ASSESSED, never COMPLIANT and never a report of zero assets
---
### Case 9: Engine Unresolved — Stop before scanning
**Fixture:**
- `project.yaml` has no `engine.name`; `.claude/docs/technical-preferences.md` reads `[TO BE CONFIGURED]`
- Both `src/` and `Source/` exist, so the tree does not decide the engine
- `design/art/art-bible.md` has asset standards; `assets/art/` holds PNG files
**Input:** `/asset-audit`
**Expected behavior:**
1. Phase 2 cannot resolve the engine or its code root per `.claude/docs/code-root-resolution.md`
2. The skill stops with `NOT ASSESSED — engine unresolved`, naming `/setup-engine`
3. No asset root is guessed, so the Godot naming patterns are not applied and no section is reported clean
**Assertions:**
- [ ] Verdict is `NOT ASSESSED — engine unresolved`, not COMPLIANT
- [ ] Output names `/setup-engine`
- [ ] No violation table is filled from an assumed `assets/` layout
---
## Protocol Compliance
- [ ] Reads `technical-preferences.md` for naming conventions, formats, and size budgets
- [ ] Scans `assets/` directory recursively
- [ ] Audit table shows file name, check type, expected value, actual value, and result
- [ ] Does not modify any asset files
- [ ] Reads the art bible or asset standards design docs before scanning (Phase 1)
- [ ] Resolves the engine and its code root, then scans that engine's asset root with Glob (`assets/` subfolders on Godot, `Assets/` on Unity, `Content/` on Unreal); an unresolved engine stops the run as NOT ASSESSED
- [ ] Report uses one table per category: Naming (File, Expected Pattern, Issue), Size (File, Budget, Actual, Overage), Format (File, Expected Format, Actual Format), Orphaned (File, Last Modified, Size, Recommendation), Missing (Reference Location, Expected Path)
- [ ] A size or format it cannot measure is `NOT ASSESSED`, never an estimate, and every section not assessed is named with its reason
- [ ] Each recommendation names the change for its violation (target resolution or compression, corrected name, target format)
- [ ] Does not modify any asset files and writes no report file
- [ ] No director gates are invoked
- [ ] Verdict is one of: COMPLIANT, WARNINGS, NON-COMPLIANT
- [ ] Verdict, first match: NON-COMPLIANT, WARNINGS, NOT ASSESSED, COMPLIANT — COMPLIANT only with at least one asset scanned and every section assessed
---
## Coverage Notes
- Metadata checks (e.g., missing texture import settings in Godot `.import` files)
are not explicitly tested here; they follow the same FORMAT ISSUE flagging pattern.
- The interaction between `/asset-audit` and `/content-audit` (both check GDD
references vs. assets) is intentional overlap; `/asset-audit` focuses on
compliance while `/content-audit` focuses on completeness.
- The skill does not check import metadata (Godot `.import` files, Unity `.meta`
files, Unreal import settings); a missing import setting is not reported.
- Orphan detection on Godot (an asset no code references) is not given its own
case; it uses the same code search as Case 4 in the other direction. On Unity
and Unreal it is NOT ASSESSED (Case 8).
- Unity follows Case 8's naming and orphan rules, with Format still checked on
the source files under `Assets/`; it is not tested separately.
- `/asset-audit` checks compliance against code references; `/content-audit`
checks completeness against the GDDs. Case 4 pins that boundary.
- An audio duration budget is not measurable with the skill's tools, so it makes
that check NOT ASSESSED; Case 1's fixture sets none, and the path is not given
its own case.
CCGS Skill Testing Framework/skills/analysis/balance-check.md
+132 -87
View File
@@ -2,171 +2,216 @@
## Skill Summary
`/balance-check` reads balance data files (JSON or YAML in `assets/data/`) and
checks each value against the design formulas defined in GDDs under `design/gdd/`.
It produces a findings table with columns: Value → Formula → Deviation → Severity.
No director gates are invoked (read-only analysis). The skill may optionally write
a balance report but asks "May I write" before doing so. Verdicts: BALANCED,
CONCERNS, or OUT OF BALANCE.
`/balance-check [system-name|path-to-data-file]` identifies the balance domain
(combat, economy, progression, loot, or a data file given directly), reads the
domain's data files from `assets/data/` and `design/balance/`, then reads the
intended values — from `design/registry/entities.yaml` first (its `constants:` and
`formulas:` blocks), and from the system's GDD in `design/gdd/` for anything the
registry does not supply. It compares every data value with its target — a
registry constant's `value`, a registry formula's `output_range`, or the GDD's
range — then runs domain-specific checks, and prints a report with
Data Sources Analyzed, a Health Summary, an Outliers Detected table
(Item/Value | Expected Range | Actual | Issue), Degenerate Strategies, Progression
Analysis, a Priority-ranked Recommendations table and Values That Need Attention.
Inputs it cannot find make the affected section `NOT ASSESSED — NO DATA`, naming
the missing source and `/design-system`, never an estimate; a value with no
target is listed as `no stated range — not judged`. After the report, an
`AskUserQuestion` offers [A] fix now, [B] save the report to
`design/balance/balance-check-[system]-[date].md`, [C] stop; it writes only on
[B]. No director gates are invoked. Health Summary, first match wins: CRITICAL
ISSUES, CONCERNS, NOT ASSESSED, HEALTHY.
---
## Static Assertions (Structural)
Verified automatically by `/skill-test static` — no fixture needed.
Checked against the SKILL.md by `/skill-test spec` — no fixture needed.
- [ ] Has required frontmatter fields: `name`, `description`, `argument-hint`, `user-invocable`, `allowed-tools`
- [ ] Has ≥2 phase headings
- [ ] Contains verdict keywords: BALANCED, CONCERNS, OUT OF BALANCE
- [ ] Contains "May I write" language (optional report write)
- [ ] Contains verdict keywords: HEALTHY, CONCERNS, CRITICAL ISSUES, NOT ASSESSED
- [ ] Has ask-before-write language: the save is an `AskUserQuestion` option that names its target path, and the Write happens only when that option is chosen
- [ ] Has a next-step handoff (what to do after findings are reviewed)
---
## Director Gate Checks
None. Balance check is a read-only analysis skill; no gates are invoked.
None. Balance check is an analysis skill; no gates are invoked in any review mode.
---
## Test Cases
### Case 1: Happy Path — All balance values within formula tolerances
### Case 1: Happy Path — All balance values within their intended ranges
**Fixture:**
- `assets/data/combat-balance.json` exists with 6 stat values
- `design/gdd/combat-system.md` contains formulas for all 6 stats with ±10% tolerance
- All 6 values fall within tolerance
- `design/registry/entities.yaml` has no entries (the shipped empty stub)
- `design/gdd/combat-system.md` gives an expected range for all 6 stats
- All 6 values fall inside their ranges; no degenerate strategy exists
**Input:** `/balance-check`
**Input:** `/balance-check combat`
**Expected behavior:**
1. Skill reads all balance data files in `assets/data/`
2. Skill reads GDD formulas from `design/gdd/`
3. Skill computes deviation for each value against its formula
4. All deviations are within ±10% tolerance
5. Skill outputs findings table with all rows showing PASS
6. Verdict is BALANCED
1. Skill identifies the Combat domain from the argument
2. Skill reads `assets/data/combat-balance.json` (and any combat files in `design/balance/`)
3. Registry has no entries, so the skill uses the GDD alone for the intended ranges
4. Each of the 6 values is compared with its GDD range and none falls outside; the combat checks find no dominant option
5. Report lists `assets/data/combat-balance.json` and `design/gdd/combat-system.md` under Data Sources Analyzed; the Outliers Detected table has no rows
6. Health Summary is HEALTHY
7. Phase 6 `AskUserQuestion` offers [A] / [B] / [C]; nothing is written before the user chooses
**Assertions:**
- [ ] Findings table is shown for all checked values
- [ ] Each row shows: stat name, formula target, actual value, deviation percentage
- [ ] All rows show PASS or equivalent when within tolerance
- [ ] Verdict is BALANCED
- [ ] No files are written without user approval
- [ ] Data Sources Analyzed lists every file read, including the data file and the GDD
- [ ] The empty registry is skipped and the GDD supplies the intended ranges
- [ ] The Outliers Detected table has no rows
- [ ] Health Summary is HEALTHY
- [ ] No file is written before the user picks option [B]
---
### Case 2: Out of Balance — Player damage 40% above formula target
### Case 2: Outlier — Player damage far above its intended range
**Fixture:**
- `assets/data/combat-balance.json` has `player_damage_base: 140`
- `design/gdd/combat-system.md` formula specifies `player_damage_base = 100` (±10%)
- All other stats are within tolerance
- `design/gdd/combat-system.md` gives `player_damage_base` an expected range of 90–110
- All other stats are within range
**Input:** `/balance-check`
**Input:** `/balance-check combat`
**Expected behavior:**
1. Skill reads combat-balance.json and computes deviation for `player_damage_base`
2. Deviation is +40% — far outside ±10% tolerance
3. Skill flags this row as severity HIGH in the findings table
4. Verdict is OUT OF BALANCE
5. Skill surfaces the HIGH severity item prominently before the table
1. Skill reads the data file and the GDD range for `player_damage_base`, and compares each value with its range
2. The Outliers Detected table lists `player_damage_base` with Expected Range 90–110, Actual 140 and an Issue description
3. The Recommendations table has a row for it with a Priority, a Suggested Fix and an Impact
4. Values That Need Attention gives a suggested adjustment with rationale
5. Health Summary is CONCERNS — one outlier a tuning pass can fix, with no degenerate strategy, stalled progression or sinkless loop
6. If the user picks [A] and changes `player_damage_base`, which the combat GDD defines, the skill reminds them to run `/propagate-design-change` on that GDD before committing
**Assertions:**
- [ ] `player_damage_base` row shows deviation of +40%
- [ ] Severity is HIGH for deviations exceeding tolerance by more than 2×
- [ ] Verdict is OUT OF BALANCE when any stat has HIGH severity deviation
- [ ] The HIGH severity item is called out explicitly, not buried in table rows
- [ ] `player_damage_base` appears in the Outliers Detected table with Expected Range 90–110 and Actual 140
- [ ] The Recommendations table ranks the fix with a Priority
- [ ] Health Summary is CONCERNS — not HEALTHY, and not CRITICAL ISSUES without a finding of the critical kind
- [ ] On [A], a fix to a GDD-defined value is followed by the `/propagate-design-change` reminder
---
### Case 3: No GDD Formulas — Cannot validate, guidance given
### Case 3: No Design Targets — Data exists but no GDD or registry defines the ranges
**Fixture:**
- `assets/data/economy-balance.yaml` exists with 10 stat values
- No GDD in `design/gdd/` contains formula definitions for economy stats
- `assets/data/economy-balance.yaml` exists with 10 plain stat values (prices and rates, no faucet/sink loop the data alone could show)
- `design/registry/entities.yaml` has no entries
- No GDD in `design/gdd/` covers the economy
**Input:** `/balance-check`
**Input:** `/balance-check economy`
**Expected behavior:**
1. Skill reads balance data files
2. Skill searches GDDs for formula definitions — finds none for economy stats
3. Skill outputs: "Cannot validate economy stats — no formulas defined. Run /design-system first."
4. No findings table is generated for the economy stats
5. Verdict is CONCERNS (data exists but cannot be validated)
1. Skill records its inputs as FOUND / ABSENT: the data file is FOUND; the registry entries and the economy GDD are ABSENT
2. Skill reads the data file and lists it under Data Sources Analyzed
3. The sections that need intended values (Outliers Detected, and Progression Analysis where it needs a target) are marked `NOT ASSESSED — NO DATA`, naming the missing economy GDD and the empty registry, and naming `/design-system` as the skill that writes them
4. No expected range is invented or inferred from the data itself
5. Health Summary is NOT ASSESSED — the report says the values could not be judged, not that they are fine
**Assertions:**
- [ ] Skill does not fabricate formula targets when none exist in GDDs
- [ ] Output explicitly names the missing formula source
- [ ] Output recommends running `/design-system` to define formulas
- [ ] Verdict is CONCERNS (not BALANCED, since validation was impossible)
- [ ] Skill does not fabricate expected ranges when neither the registry nor a GDD supplies them
- [ ] The affected sections are marked `NOT ASSESSED — NO DATA` rather than shown as empty tables
- [ ] Output names the missing design source (economy GDD in `design/gdd/`, empty `design/registry/entities.yaml`) and `/design-system`
- [ ] Health Summary is NOT ASSESSED — not HEALTHY, and not CONCERNS, since nothing was found wrong
---
### Case 4: Orphan Reference — Balance file references an undefined stat
### Case 4: Registry First — The registry value overrides a contradicting GDD
**Fixture:**
- `assets/data/combat-balance.json` contains a stat `legacy_armor_mult: 1.5`
- `design/gdd/combat-system.md` has no formula for `legacy_armor_mult`
- All other stats have formula definitions and pass validation
- `design/registry/entities.yaml` `constants:` lists `player_damage_base` with `value: 100`, `unit: damage`, `source: design/gdd/combat-system.md` (a registry constant carries a value, not a range)
- `design/gdd/combat-system.md` still says `player_damage_base` range 100–130 (stale)
- `assets/data/combat-balance.json` has `player_damage_base: 125`
**Input:** `/balance-check`
**Input:** `/balance-check combat`
**Expected behavior:**
1. Skill reads all stats from combat-balance.json
2. Skill cannot find a formula for `legacy_armor_mult` in any GDD
3. Skill flags `legacy_armor_mult` as ORPHAN REFERENCE in the findings table
4. Other stats are evaluated normally; those within tolerance show PASS
5. Verdict is CONCERNS (orphan reference prevents full validation)
1. Skill greps `design/registry/entities.yaml` (`^ - name:` with context) before reading the GDD
2. The target for `player_damage_base` is the registry constant's `value`, 100, not the GDD's range
3. 125 differs from the registry value 100, so `player_damage_base` appears in the Outliers Detected table with Expected Range 100 (the registry value) and Actual 125 — although it sits inside the stale GDD range
4. The GDD is still read for any value the registry does not list
5. Data Sources Analyzed lists the registry, the GDD and the data file
**Assertions:**
- [ ] `legacy_armor_mult` appears in findings table with status ORPHAN REFERENCE
- [ ] Orphan references are distinguished from formula deviations in the table
- [ ] Verdict is CONCERNS when any orphan references are found
- [ ] Skill does not skip orphan stats silently
- [ ] The registry is read before the GDD
- [ ] `player_damage_base` is judged against the registry value (100), not the GDD range (100–130)
- [ ] `player_damage_base: 125` is reported as an outlier
- [ ] Data Sources Analyzed includes `design/registry/entities.yaml`
---
### Case 5: Gate Compliance — Read-only; no gate; optional report requires approval
### Case 5: No Argument, Gate Compliance and the Save Path
**Fixture:**
- Balance data and GDD formulas exist; 1 stat has CONCERNS-level deviation (15% above target)
- `review-mode.txt` contains `full`
- Combat data, registry and GDD exist; 1 stat is slightly outside its range
- `project.yaml`: `modes.review_mode: full`
- `modes.automation` resolves to `collaborative`
**Input:** `/balance-check`
**Input:** `/balance-check` (no argument)
**Expected behavior:**
1. Skill reads data and GDDs; generates findings table
2. Verdict is CONCERNS (one stat slightly out of range)
1. Skill asks the user which system to check; user answers "combat"
2. Skill reads data and design targets and prints the report
3. No director gate is invoked
4. Skill presents findings table to user
5. Skill offers to write an optional balance report
6. If user says yes: skill asks "May I write to `production/qa/balance-report-[date].md`?"
7. If user says no: skill ends without writing
4. After the report, `AskUserQuestion` offers [A] fix now, [B] save to `design/balance/balance-check-combat-[date].md`, [C] stop
5. On [B]: the report is written to `design/balance/balance-check-combat-[YYYY-MM-DD].md` (directory created if needed), the write is confirmed, and the skill ends with "Re-run `/balance-check` after fixes to verify."
6. On [C]: nothing is written; open issues are summarized and the same re-run line ends the output
**Assertions:**
- [ ] With no argument, the skill asks which system to check before reading data
- [ ] No director gate is invoked in any review mode
- [ ] Findings table is presented without writing anything automatically
- [ ] Optional report write is offered but not forced
- [ ] "May I write" prompt appears only if user opts in to the report
- [ ] The report is presented before the save option, and the save option names `design/balance/balance-check-[system]-[date].md`
- [ ] Option [B] writes to that path with the date in YYYY-MM-DD form; option [C] writes nothing
- [ ] Both [B] and [C] end with "Re-run `/balance-check` after fixes to verify."
---
### Case 6: Orphan Stat — A data value no GDD or registry defines
**Fixture:**
- `assets/data/combat-balance.json` holds 6 stats plus `legacy_armor_mult: 1.5`
- `design/registry/entities.yaml` has no entries
- `design/gdd/combat-system.md` gives a range for the 6 stats and nothing for `legacy_armor_mult`
- The 6 stats are inside their ranges; no degenerate strategy exists
**Input:** `/balance-check combat`
**Expected behavior:**
1. Every value is compared with its target; the 6 stats are in range
2. No target exists for `legacy_armor_mult`, so it is listed under Values That Need Attention as `no stated range — not judged`
3. It is not an Outliers Detected row, and it is not dropped from the report
4. Health Summary is NOT ASSESSED, naming `legacy_armor_mult` as the value that could not be judged
**Assertions:**
- [ ] `legacy_armor_mult` is reported as `no stated range — not judged`, apart from any outlier
- [ ] The orphan stat is not skipped silently and not judged against a range inferred from the data
- [ ] Health Summary is NOT ASSESSED, not HEALTHY, naming the unjudged value
---
## Protocol Compliance
- [ ] Reads both balance data files and GDD formulas before analysis
- [ ] Findings table shows Value, Formula, Deviation, and Severity columns
- [ ] Does not write any files without explicit user approval
- [ ] Reads the domain's data files (Phase 2) and the registry then GDD (Phase 3) before analysis
- [ ] Compares every data value with its target (registry constant `value`, registry formula `output_range`, or GDD range); a value outside it is an Outliers Detected row, a value with none is `no stated range — not judged`
- [ ] Outliers Detected table uses the columns Item/Value, Expected Range, Actual, Issue; Recommendations uses Priority, Issue, Suggested Fix, Impact
- [ ] A section whose inputs are absent is `NOT ASSESSED — NO DATA`, never estimated, and names the missing source and `/design-system`
- [ ] Writes only when the user picks the save option, and only to `design/balance/balance-check-[system]-[date].md`
- [ ] No director gates are invoked
- [ ] Verdict is one of: BALANCED, CONCERNS, OUT OF BALANCE
- [ ] Health Summary, first match: CRITICAL ISSUES, CONCERNS, NOT ASSESSED, HEALTHY — HEALTHY only when every value was judged against a stated target
---
## Coverage Notes
- The case where `assets/data/` is entirely empty is not tested; behavior
follows the CONCERNS pattern with a message that no data files were found.
- Tolerance thresholds (±10%, ±20%) are implementation details of the skill;
the tests verify that deviations are detected and classified, not the
exact threshold values.
- The case where every input is absent (no data files, no registry entries, no
GDD) is not tested separately; the skill then stops and reports
`NOT ASSESSED — NO DATA` as the whole verdict, naming what was missing and
which skill produces it.
- CRITICAL ISSUES (a degenerate strategy, a stalled progression, a loop with no
sink) is not given its own case; Case 2 pins that an outlier alone is CONCERNS.
- A data-file path argument (Phase 1 "File path given") is not tested; the
domain is then inferred from the file's content.
CCGS Skill Testing Framework/skills/analysis/code-review.md
+161 -82
View File
@@ -2,171 +2,250 @@
## Skill Summary
`/code-review` performs an architectural code review of source files in `src/`,
checking coding standards from `CLAUDE.md` (doc comments on public APIs,
dependency injection over singletons, data-driven values, testability). Findings
are advisory. No director gates are invoked. No code edits are made. Verdicts:
APPROVED, CONCERNS, or NEEDS CHANGES.
`/code-review [file(s)] [story-path]` performs an architectural review of source
files. It reads the target files and the project coding standards (Phase 1),
resolves the engine specialists from `project.yaml` (Phase 2), checks compliance
with any referenced ADR by reading only its `## Decision` and `## Consequences`
sections (Phase 3), then evaluates six standards checks, architecture and SOLID,
and game-specific concerns (Phases 4–6). Applicable engine specialists — and
`qa-tester` for Logic/Integration stories — are spawned in parallel, and every
specialist finding is recorded with file/line, evidence and a VERIFIED /
UNVERIFIED confidence (Phase 7). The report (Phase 8) ends in a verdict, first
match wins: CHANGES REQUIRED, NOT ASSESSED (nothing to review, the engine
specialist review did not run, or a report section — such as ADR Compliance on a
referenced ADR that could not be read — reads NOT ASSESSED), APPROVED WITH
SUGGESTIONS, APPROVED. It grants only `git log` from Bash. Phase 9
offers next steps via `AskUserQuestion`. The skill is read-only and invokes no
director gates.
---
## Static Assertions (Structural)
Verified automatically by `/skill-test static` — no fixture needed.
Checked against the SKILL.md by `/skill-test spec` — no fixture needed.
- [ ] Has required frontmatter fields: `name`, `description`, `argument-hint`, `user-invocable`, `allowed-tools`
- [ ] Has ≥2 phase headings
- [ ] Contains verdict keywords: APPROVED, CONCERNS, NEEDS CHANGES
- [ ] Does NOT require "May I write" language (read-only; findings are advisory output)
- [ ] Contains verdict keywords: APPROVED, APPROVED WITH SUGGESTIONS, CHANGES REQUIRED, NOT ASSESSED
- [ ] Does NOT require "May I write" language (read-only; states that no files are written)
- [ ] Has a next-step handoff (what to do with findings)
---
## Director Gate Checks
None. Code review is a read-only advisory skill; no gates are invoked.
None. Code review invokes no director gates in any review mode. It does spawn
engine specialists and `qa-tester` as reviewers (Phase 7); those are not gates.
---
## Test Cases
### Case 1: Happy Path — Source file follows all coding standards
### Case 1: Happy Path — Standards-compliant file that follows its ADR
**Fixture:**
- `src/gameplay/health_component.gd` exists with:
- All public methods have doc comments (`##` notation)
- No singletons used; dependencies injected via constructor
- No hardcoded values; all constants reference `assets/data/`
- ADR reference in file header: `# ADR: docs/architecture/adr-004-health.md`
- Referenced ADR has `Status: Accepted`
- `project.yaml`: `engine.name: godot`, `specialists.code: godot-gdscript-specialist`, `specialists.shader: null`, `specialists.ui: null`
- `src/gameplay/health_component.gd` meets all six Phase 4 checks:
- Doc comments (`##`) on every public method and the class
- Every method under cyclomatic complexity 10 and under 40 lines
- Dependencies injected; no singletons
- All tuning values loaded from `assets/data/`
- Depends on interfaces, not concrete classes
- Header comment: `# Implements ADR-0004 (docs/architecture/adr-0004-health.md)`
- `docs/architecture/adr-0004-health.md` has `## Decision` and `## Consequences` sections, and the code follows the chosen approach
- The specialist reports no issues; there is nothing to suggest
**Input:** `/code-review src/gameplay/health_component.gd`
**Expected behavior:**
1. Skill reads the source file
2. Skill checks all coding standards: doc comments, DI, data-driven, ADR status
3. All checks pass
4. Skill outputs findings summary with all checks PASS
5. Verdict is APPROVED
1. Skill reads the source file and the coding standards
2. Skill finds the `ADR-0004` reference in the file header
3. Skill greps the ADR's `^## ` headings, then reads only the `## Decision` and `## Consequences` spans
4. `godot-gdscript-specialist` is spawned via `Agent` for the `.gd` file; `shader: null` and `ui: null` are treated as unset and not spawned
5. Report shows ADR Compliance: COMPLIANT, Standards Compliance: 6/6 passing, Engine Specialist Findings: CLEAN, Testability: `N/A — no story path given`, and a Positive Observations section
6. Verdict is APPROVED
7. Phase 9 `AskUserQuestion` offers [A] Run `/story-done` and [B] Stop here
**Assertions:**
- [ ] Each coding standard check is listed in the output
- [ ] All checks show PASS when standards are met
- [ ] Skill reads referenced ADR to confirm its status
- [ ] Verdict is APPROVED
- [ ] No edits are made to any file
- [ ] The ADR is read with a heading map and bounded reads of `## Decision` and `## Consequences`, not an unbounded full read
- [ ] `godot-gdscript-specialist` is spawned for the `.gd` file; no agent named `null` is spawned
- [ ] Standards Compliance reads `6/6 passing` and ADR Compliance reads COMPLIANT
- [ ] Testability reads `N/A — no story path given`; no `qa-tester` is spawned without a story
- [ ] Positive Observations section is present
- [ ] Verdict is APPROVED and Phase 9 offers `/story-done` or stop
- [ ] No file is written or edited
---
### Case 2: Needs Changes — Missing doc comment and singleton usage
### Case 2: Changes Required — Missing doc comments and singleton usage, no ADR
**Fixture:**
- `project.yaml`: `engine.name: godot`, `specialists.code: godot-gdscript-specialist`
- `src/ui/inventory_ui.gd` has:
- 2 public methods without doc comments
- Uses `GameManager.instance` (singleton pattern)
- 2 public methods (`refresh_slots`, `sort_items`) without doc comments
- `GameManager.instance` used at lines 42 and 87
- All other standards met
- No ADR reference in the file header, and no commit touching it names an ADR
- No story path is passed
**Input:** `/code-review src/ui/inventory_ui.gd`
**Expected behavior:**
1. Skill reads the source file
2. Skill detects: 2 missing doc comments on public methods
3. Skill detects: singleton usage at specific lines (e.g., line 42, line 87)
4. Findings list the exact method names and line numbers
5. Verdict is NEEDS CHANGES
1. Skill searches the file header and `git log --oneline -- src/ui/inventory_ui.gd` for `ADR-NNNN` references and finds none
2. Output notes: "No ADR references found — ADR compliance check skipped. For full ADR compliance review, provide the story path: `/code-review [files] [story-path]`."
3. Standards Compliance shows 4/6 passing and lists the failures with line references: the two undocumented methods and the singleton at lines 42 and 87
4. Required Changes lists the missing doc comments and the singleton, with dependency injection as the fix
5. Verdict is CHANGES REQUIRED
6. Phase 9 offers [A] Fix the issues and re-run `/code-review`, [B] Run `/story-done` anyway with noted exceptions, [C] Stop here
**Assertions:**
- [ ] Missing doc comments are listed with method names
- [ ] Singleton usage is flagged with file and line number
- [ ] Verdict is NEEDS CHANGES when BLOCKING-level standard violations exist
- [ ] Skill does not edit the file — findings are for the developer to act on
- [ ] Output suggests replacing singleton with dependency injection
- [ ] The "No ADR references found" note names the story-path form of the command
- [ ] Missing doc comments are listed with method names (`refresh_slots`, `sort_items`), not only line references
- [ ] Singleton usage is flagged with file and line numbers (42, 87)
- [ ] Standards Compliance reads `4/6 passing`
- [ ] Verdict is CHANGES REQUIRED and Phase 9 offers the three CHANGES REQUIRED options
- [ ] Skill does not edit the file
---
### Case 3: Architecture Risk — ADR reference is Proposed, not Accepted
### Case 3: Architectural Violation — Code uses a pattern its ADR rejects
**Fixture:**
- `src/core/save_system.gd` has a header comment: `# ADR: docs/architecture/adr-010-save.md`
- `adr-010-save.md` exists but has `Status: Proposed`
- Code itself follows all other coding standards
- `project.yaml`: `engine.name: godot`, `specialists.code: godot-gdscript-specialist`
- `src/core/save_system.gd` header: `# Implements ADR-0010 (docs/architecture/adr-0010-save-format.md)`
- `adr-0010-save-format.md` `## Decision`: save data goes through `SaveService` as JSON; direct `FileAccess` writes from gameplay code are explicitly rejected
- `save_system.gd` writes save data with `FileAccess.open(..., FileAccess.WRITE)` directly at line 58
- Code otherwise meets all six standards checks
**Input:** `/code-review src/core/save_system.gd`
**Expected behavior:**
1. Skill reads the source file
2. Skill reads referenced ADR — finds `Status: Proposed`
3. Skill flags this as ARCHITECTURE RISK (code is implementing an unaccepted ADR)
4. Other coding standard checks pass
5. Verdict is CONCERNS (risk flag is advisory, not a hard NEEDS CHANGES)
1. Skill reads only the Decision and Consequences sections of ADR-0010
2. The direct `FileAccess` write is classified ARCHITECTURAL VIOLATION (BLOCKING) — a pattern the ADR explicitly rejects
3. ADR Compliance reads VIOLATION, listing ADR-0010, the result and the deviation with its severity
4. The violation appears under Required Changes
5. Verdict is CHANGES REQUIRED
6. Next steps: fix the implementation to comply with `docs/architecture/adr-0010-save-format.md`, or — if the design has legitimately changed — run `/architecture-decision` to revise the existing ADR, not create a competing one
**Assertions:**
- [ ] Skill reads referenced ADR file to check its status
- [ ] ARCHITECTURE RISK is flagged when ADR status is Proposed
- [ ] Verdict is CONCERNS (not NEEDS CHANGES) for ADR risk — advisory severity
- [ ] Output recommends resolving the ADR before the code goes to production
- [ ] The deviation is classified ARCHITECTURAL VIOLATION (BLOCKING), not ADR DRIFT or MINOR DEVIATION
- [ ] ADR Compliance reads VIOLATION and names ADR-0010
- [ ] The violation is listed under Required Changes
- [ ] Verdict is CHANGES REQUIRED
- [ ] Output says to comply with the existing ADR or revise it via `/architecture-decision`, never to write a competing ADR
---
### Case 4: Edge Case — No source files found at specified path
### Case 4: Edge Case — No source files at the specified path
**Fixture:**
- User calls `/code-review src/networking/`
- `src/networking/` directory does not exist
- `project.yaml`: `engine.name: godot`
- `src/networking/` does not exist
**Input:** `/code-review src/networking/`
**Expected behavior:**
1. Skill attempts to read files in `src/networking/`
2. Directory or files not found
3. Skill outputs an error: "No source files found at `src/networking/`"
4. Skill suggests checking `src/` for valid directories
5. No verdict is emitted (nothing was reviewed)
1. Skill records its inputs as FOUND / ABSENT; the target source files are ABSENT
2. Every required input is ABSENT, so the skill stops before producing review sections
3. Output reports `NOT ASSESSED — NO DATA` as the whole verdict, naming `src/networking/` as missing
4. No specialist or `qa-tester` agent is spawned
5. No review section is filled in, and the result is not presented as APPROVED
**Assertions:**
- [ ] Skill does not crash when path does not exist
- [ ] Output names the attempted path in the error message
- [ ] Output suggests checking `src/` for valid file paths
- [ ] No verdict is emitted when there is nothing to review
- [ ] Skill does not crash when the path does not exist
- [ ] Output names the attempted path `src/networking/`
- [ ] Verdict is `NOT ASSESSED — NO DATA`, not APPROVED
- [ ] No specialist agent is spawned when there is nothing to review
---
### Case 5: Gate Compliance — No gate; LP may be consulted separately
### Case 5: No Engine Configured — Skipped specialists announce themselves; no gate
**Fixture:**
- Source file follows most standards but has 1 CONCERNS-level finding (a magic number)
- `review-mode.txt` contains `full`
- `project.yaml` has `modes.review_mode: full` and no `engine.name`; `.claude/docs/technical-preferences.md` reads `[TO BE CONFIGURED]`
- `src/gameplay/loot_system.gd` hardcodes a drop rate `0.05` at line 31; everything else meets the standards
**Input:** `/code-review src/gameplay/loot_system.gd`
**Expected behavior:**
1. Skill reads and reviews the source file
2. No director gate is invoked (code review findings are advisory)
3. Skill presents findings with the CONCERNS verdict
4. Output notes: "Consider requesting a Lead Programmer review for architecture concerns"
5. Skill does not invoke any agent automatically
1. Phase 2 finds no engine configured and skips the engine specialist steps
2. Output records `Engine validation: NOT ASSESSED — no engine configured (engine.name unset in project.yaml)`
3. Engine Specialist Findings reads `N/A — no engine configured`; no engine specialist is spawned
4. Standards Compliance lists the "Configuration values loaded from data files" failure with line 31, and the fix is listed under Required Changes
5. Verdict is CHANGES REQUIRED — the known defect outranks the skipped engine review, which the report still names
6. No director gate is invoked in any review mode
7. No file is edited
**Assertions:**
- [ ] No director gate is invoked in any review mode
- [ ] LP consultation is suggested (not mandated) in the output
- [ ] Output contains the `Engine validation: NOT ASSESSED — no engine configured` line
- [ ] Engine Specialist Findings reads `N/A — no engine configured` and no engine specialist agent is spawned
- [ ] The hardcoded value is reported under Standards Compliance with its line reference
- [ ] Verdict is CHANGES REQUIRED, not NOT ASSESSED and never APPROVED
- [ ] No director gate is invoked, even with review mode `full`
- [ ] No code edits are made
- [ ] Verdict is CONCERNS for advisory-level findings
---
### Case 5b: No Engine Configured, Clean File — NOT ASSESSED, never APPROVED
**Fixture:**
- `project.yaml` has no `engine.name`; `.claude/docs/technical-preferences.md` reads `[TO BE CONFIGURED]`
- `src/gameplay/loot_table.gd` meets all six Phase 4 checks and raises no architecture, SOLID or game-specific concern
**Input:** `/code-review src/gameplay/loot_table.gd`
**Expected behavior:**
1. Phase 2 records `Engine validation: NOT ASSESSED — no engine configured`; no engine specialist is spawned
2. Nothing is listed under Required Changes or Suggestions
3. Verdict is NOT ASSESSED, naming the engine specialist review that did not run
4. Phase 9 offers the NOT ASSESSED options, naming `/setup-engine` as the way to make the review complete
**Assertions:**
- [ ] Verdict is NOT ASSESSED, not APPROVED — a review that skipped the engine check has not approved the code
- [ ] The verdict names the engine specialist review as what did not run
- [ ] Phase 9 does not offer `/story-done`; it names `/setup-engine`
### Case 6: Referenced ADR missing — the section's NOT ASSESSED reaches the verdict
**Fixture:**
- `project.yaml`: `engine.name: godot`, `specialists.code: godot-gdscript-specialist`
- `src/gameplay/stamina.gd` meets all six Phase 4 checks and raises no
architecture, SOLID or game-specific concern; the specialist reports it clean
- Its header reads `# Implements ADR-0008 (docs/architecture/adr-0008-stamina.md)`,
and no such file exists
**Input:** `/code-review src/gameplay/stamina.gd`
**Expected behavior:**
1. Phase 3 finds the `ADR-0008` reference but cannot read the ADR
2. ADR Compliance reads `NOT ASSESSED — ADR-0008 could not be read`
3. Nothing is listed under Required Changes or Suggestions
4. Verdict is NOT ASSESSED, naming the ADR Compliance section and ADR-0008
**Assertions:**
- [ ] ADR Compliance reads NOT ASSESSED and names ADR-0008 — not `NO ADRS FOUND`, not COMPLIANT
- [ ] Verdict is NOT ASSESSED, never APPROVED: a section that could not be assessed reaches the verdict
- [ ] Variant — the file names no ADR at all: ADR Compliance reads `NO ADRS FOUND`, and the verdict is APPROVED (a missing reference is not an unreadable one)
---
## Protocol Compliance
- [ ] Reads source file(s) and coding standards before reviewing
- [ ] Lists each coding standard check in findings output
- [ ] Reads the target file(s) and the coding standards before reviewing (Phase 1)
- [ ] Reads the `specialists` block from `project.yaml`, falling back to `technical-preferences.md`; a `null` specialist is skipped, never spawned
- [ ] Every specialist finding in the report carries file/line, evidence, and VERIFIED or `UNVERIFIED — specialist claim`
- [ ] Does not edit any source files (read-only skill)
- [ ] No director gates are invoked
- [ ] Verdict is one of: APPROVED, CONCERNS, NEEDS CHANGES
- [ ] Verdict, first match: CHANGES REQUIRED, NOT ASSESSED, APPROVED WITH SUGGESTIONS, APPROVED — never an approval when the engine specialist review did not run or any report section reads NOT ASSESSED
- [ ] `allowed-tools` grants `Bash(git log *)` and the scoped `yaml-helper.sh resolve_config` pattern, not bare `Bash` — the skill is read-only: it runs `git log` and the config resolver, nothing else
- [ ] Phase 9's prompt and option branches use the same verdict vocabulary as the Phase 8 report
---
## Coverage Notes
- Batch review of all files in a directory is not explicitly tested; behavior
is assumed to apply the same checks file by file and aggregate the verdict.
- Test coverage checks (verifying corresponding test files exist) are a stretch
goal not tested here; that is primarily the domain of `/test-evidence-review`.
- Batch review of a directory is not tested separately; the same checks apply
file by file.
- The `qa-tester` testability review (Logic and Integration stories, story path
passed as the last argument) is not given its own case.
- Test-file existence checks are the domain of `/test-evidence-review`.
CCGS Skill Testing Framework/skills/analysis/consistency-check.md
+213 -93
View File
@@ -2,28 +2,38 @@
## Skill Summary
`/consistency-check` scans all GDDs in `design/gdd/` and checks for internal
conflicts across documents. It produces a structured findings table with columns:
System A vs System B, Conflict Type, Severity (HIGH / MEDIUM / LOW). Conflict
types include: formula mismatch, competing ownership, stale reference, and
dependency gap.
The skill is read-only during analysis. It has no director gates. An optional
consistency report can be written to `design/consistency-report-[date].md` if the
user requests it, but the skill asks "May I write" before doing so.
`/consistency-check [full | since-last-review | entity:<name> | item:<name>]`
compares every in-scope GDD in `design/gdd/` against the entity registry
(`design/registry/entities.yaml`). It reads the registry once, builds entity,
item, formula and constant lookups, then greps each GDD for each registered name
(`-C 3`) instead of reading whole documents; only conflicts get a targeted
section read. The registry entry's `source:` GDD is the authoritative owner.
In a GDD whose effective tier is `standard` — the project tier, or that
system's `system_overrides` row — only the required GDD sections are compared
(plus Tuning Knobs when `workflow_overrides.tuning_knobs` is true), and the
report says so. Findings are classified 🔴 CONFLICT, ⚠️ STALE REGISTRY or ℹ️ UNVERIFIABLE, and the
report verdict is PASS or CONFLICTS FOUND — or NOT ASSESSED when the registry is
empty, a named entity is not in it, or no GDD is in scope, since a scan of
nothing is not a PASS. Registry corrections are written only
after a "May I update / May I add" ask; afterwards the verdict is COMPLETE, or
BLOCKED while conflicts remain. Every 🔴 CONFLICT is appended to
`docs/consistency-failures.md` after a "May I append" ask, a breadcrumb is appended to
`production/session-state/active.md`, and the skill closes with an
`AskUserQuestion` widget (collaborative and guided modes). No director gates.
---
## Static Assertions (Structural)
Verified automatically by `/skill-test static` — no fixture needed.
Checked against the SKILL.md by `/skill-test spec` — no fixture needed.
- [ ] Has required frontmatter fields: `name`, `description`, `argument-hint`, `user-invocable`, `allowed-tools`
- [ ] Has ≥2 phase headings
- [ ] Contains verdict keywords: CONSISTENT, CONFLICTS FOUND, DEPENDENCY GAP
- [ ] Does NOT require "May I write" language during analysis (read-only scan)
- [ ] Contains verdict keywords: PASS, CONFLICTS FOUND, NOT ASSESSED, COMPLETE, BLOCKED
- [ ] The analysis phases (1–5) are read-only
- [ ] Has a next-step handoff at the end
- [ ] Documents that report writing is optional and requires approval
- [ ] Registry writes are preceded by "May I update" / "May I add" language naming `design/registry/entities.yaml`
- [ ] The conflict log write is preceded by "May I append" language naming `docs/consistency-failures.md`
---
@@ -37,140 +47,250 @@ required as part of the scan itself.
## Test Cases
### Case 1: Happy Path — 4 GDDs with no conflicts
### Case 1: Happy Path — Registry and 4 GDDs agree
**Fixture:**
- `design/gdd/` contains exactly 4 system GDDs
- All GDDs have consistent formulas (no overlapping variables with different values)
- No two GDDs claim ownership of the same game entity or mechanic
- All dependency references point to GDDs that exist
- `modes.workflow` resolves to `full`; `modes.automation` resolves to `collaborative`
- `design/registry/entities.yaml` has 3 entities and 2 constants, each with a `source:` GDD
- `design/gdd/` contains 4 system GDDs plus `game-concept.md`, `systems-index.md` and a `gdd-cross-review-[date].md`
- Every value the GDDs state for a registered name matches the registry
**Input:** `/consistency-check`
**Expected behavior:**
1. Skill reads all 4 GDDs in `design/gdd/`
2. Runs cross-GDD consistency checks (formulas, ownership, references)
3. No conflicts found
4. Outputs structured findings table showing 0 issues
5. Verdict: CONSISTENT
1. Skill reads the registry and reports `Registry loaded: 3 entities, 0 items, 0 formulas, 2 constants` and `Scope: full`
2. Skill globs `design/gdd/*.md`, excludes `game-concept.md`, `systems-index.md` and the cross-review report, and reports the 4 in-scope GDDs before scanning
3. Each registered name is grepped across the in-scope GDDs with 3 lines of context; no GDD is read in full
4. No conflicts or stale entries; the report's Clean Entries line reads `✅ 5 registry entries verified across all GDDs with no conflicts.`
5. Verdict: PASS
6. Nothing is appended to `docs/consistency-failures.md`; the breadcrumb is appended to `production/session-state/active.md`
7. Closes with the `AskUserQuestion` widget (Fix / Run `/design-review` / Stop)
**Assertions:**
- [ ] All 4 GDDs are read before producing output
- [ ] Findings table is present (even if empty — shows "No conflicts found")
- [ ] Verdict is CONSISTENT when no conflicts exist
- [ ] Skill does NOT write any files without user approval
- [ ] Next-step handoff is present
- [ ] The in-scope GDD list (4 GDDs, non-system docs excluded) is reported before the scan
- [ ] The scan greps for registered names rather than fully reading each GDD
- [ ] Verdict is PASS when registry and GDDs agree
- [ ] `design/registry/entities.yaml` and `docs/consistency-failures.md` are not written; the only write is the `active.md` breadcrumb
- [ ] The breadcrumb names `docs/consistency-failures.md`, not an invented report file
- [ ] The skill closes with the `AskUserQuestion` widget, not plain text
---
### Case 2: Failure Path — Two GDDs with conflicting damage formulas
### Case 2: Failure Path — A GDD contradicts a registered constant
**Fixture:**
- GDD-A defines damage formula: `damage = attack * 1.5`
- GDD-B defines damage formula: `damage = attack * 2.0` for the same entity type
- Both GDDs refer to the same "attack" variable
- `modes.workflow` resolves to `full`
- Registry `constants:` has `crit_multiplier` value 1.5, `source: design/gdd/combat.md`, `referenced_by: [combat.md, loot.md]`
- `design/gdd/combat.md` states `crit_multiplier = 1.5`
- `design/gdd/loot.md` states `crit_multiplier = 2.0`
- The registry entry is newer than the last commit to `combat.md` (not stale)
**Input:** `/consistency-check`
**Expected behavior:**
1. Skill reads all GDDs and detects the formula mismatch
2. Findings table includes an entry: GDD-A vs GDD-B | Formula Mismatch | HIGH
3. Specific conflicting formulas are shown (not just "formula conflict exists")
1. The constant scan (3d) greps `crit_multiplier` and finds 2.0 in `loot.md`
2. Phase 4 does a targeted read of the conflicting section and uses the registry `source:` to decide `combat.md` is authoritative
3. Report lists `🔴 crit_multiplier`: Registry (source: combat.md) = 1.5; Conflict in loot.md = 2.0; Resolution: change `loot.md` to 1.5
4. Verdict: CONFLICTS FOUND
5. Skill asks "May I append 1 entries to `docs/consistency-failures.md` (creating it if absent)?"; on approval an entry for the conflict is appended (the file is created with its header first if absent), naming `combat.md` vs `loot.md`
6. No GDD is edited automatically; with the conflict unresolved the final verdict is BLOCKED
**Assertions:**
- [ ] Verdict is CONFLICTS FOUND (not CONSISTENT)
- [ ] Conflict entry names both GDD filenames
- [ ] Conflict type is "Formula Mismatch"
- [ ] Severity is HIGH for a direct formula contradiction
- [ ] Both conflicting formulas are shown in the findings table
- [ ] Skill does NOT auto-resolve the conflict
- [ ] Verdict is CONFLICTS FOUND (not PASS)
- [ ] The conflict entry names both GDDs and shows both values (1.5 and 2.0)
- [ ] The conflict is classified 🔴 CONFLICT, and the resolution targets the non-source GDD (`loot.md`)
- [ ] A reflexion-log entry is appended to `docs/consistency-failures.md`, only after the "May I append" ask naming it is approved
- [ ] Skill does NOT auto-resolve the conflict — neither GDD is edited without the user choosing to fix it
- [ ] Final verdict is BLOCKED while the conflict is unresolved
---
### Case 3: Partial Path — GDD references a system with no GDD
### Case 3: Stale Registry — The source GDD changed after the registry entry
**Fixture:**
- GDD-A's Dependencies section lists "system-B" as a dependency
- No GDD for system-B exists in `design/gdd/`
- All other GDDs are consistent
- `modes.workflow` resolves to `full`
- Registry `entities:` has `goblin` with `health: 40`, `source: design/gdd/combat.md`
- `design/gdd/combat.md` now says goblin health is 50; `git log` shows it changed after the registry entry was written
- No other GDD states a goblin health value
**Input:** `/consistency-check entity:goblin`
**Expected behavior:**
1. Scope is limited to the `goblin` entity across all GDDs; the report shows `Scope: entity:goblin`
2. The mismatch is found in the source GDD itself; Phase 4 checks git history and classifies it ⚠️ STALE REGISTRY, not 🔴 CONFLICT
3. Report lists it under Stale Registry Entries: registry says 40, source GDD now says 50
4. Skill asks: "May I update `design/registry/entities.yaml` to fix the 1 stale entries?"
5. On approval: `health` becomes 50, `revised:` is set to today, and a `# was: 40 before [date]` comment is added; nothing is deleted
6. After writing, verdict is COMPLETE
7. No entry is appended to `docs/consistency-failures.md` (only 🔴 CONFLICT entries are logged)
**Assertions:**
- [ ] The finding is classified ⚠️ STALE REGISTRY, distinct from 🔴 CONFLICT
- [ ] The registry is not written until the "May I update `design/registry/entities.yaml`" ask is approved
- [ ] The approved update sets `revised:` and keeps the old value in a `# was:` comment
- [ ] Verdict after the write is COMPLETE
- [ ] `docs/consistency-failures.md` is not written for a stale-registry finding
---
### Case 4: Edge Case — Empty registry, no GDDs
**Fixture:**
- `modes.workflow` resolves to `full`
- `design/registry/entities.yaml` is the shipped stub with no entries
- `design/gdd/` is empty
**Input:** `/consistency-check`
**Expected behavior:**
1. Skill reads all GDDs and checks dependency references
2. GDD-A's reference to "system-B" cannot be resolved — no GDD exists for it
3. Findings table includes: GDD-A vs (missing) | Dependency Gap | MEDIUM
4. Verdict: DEPENDENCY GAP (not CONSISTENT, not CONFLICTS FOUND)
1. Skill reads the registry and finds no entries
2. Skill outputs: "Entity registry is empty. Run `/design-system` to write GDDs — the registry is populated automatically after each GDD is completed. Nothing to check yet."
3. Verdict: **NOT ASSESSED — entity registry empty**; the skill stops without scanning
4. No file is written — not the registry, not the reflexion log, not the session-state breadcrumb
**Assertions:**
- [ ] Verdict is DEPENDENCY GAP (distinct from CONSISTENT and CONFLICTS FOUND)
- [ ] Findings entry names GDD-A and the missing system-B
- [ ] Severity is MEDIUM for an unresolved dependency reference
- [ ] Skill suggests running `/design-system system-B` to create the missing GDD
---
### Case 4: Edge Case — No GDDs found
**Fixture:**
- `design/gdd/` directory is empty or does not exist
**Input:** `/consistency-check`
**Expected behavior:**
1. Skill attempts to read files in `design/gdd/`
2. No GDD files found
3. Skill outputs an error: "No GDDs found in `design/gdd/`. Run `/design-system` to create GDDs first."
4. No findings table is produced
5. No verdict is issued
**Assertions:**
- [ ] Skill outputs a clear error message when no GDDs are found
- [ ] No verdict is produced (CONSISTENT / CONFLICTS FOUND / DEPENDENCY GAP)
- [ ] Skill recommends the correct next action (`/design-system`)
- [ ] Skill outputs the empty-registry message naming `/design-system`
- [ ] Verdict is NOT ASSESSED, naming the empty registry as the reason — never PASS, CONFLICTS FOUND, COMPLETE or BLOCKED, and never no verdict at all (`.claude/rules/skill-authoring.md` obligation 1)
- [ ] No file is written, including `production/session-state/active.md`
- [ ] Skill does NOT crash or produce a partial report
---
### Case 5: Director Gate — No gate spawned; no review-mode.txt read
### Case 4b: Edge Case — Registry populated, but no GDD in scope
**Fixture:**
- `modes.workflow` resolves to `full`
- `design/registry/entities.yaml` has 3 entities, each with a `source:` GDD
- `design/gdd/gdd-cross-review-[date].md` exists, and no system GDD has changed since it
**Input:** `/consistency-check since-last-review`
**Expected behavior:**
1. Skill reads the registry and reports what it loaded
2. The `since-last-review` filter finds zero changed GDDs; the in-scope list is empty
3. Verdict: **NOT ASSESSED — no GDDs in scope**, naming `since-last-review` as the
filter that produced zero; the skill stops without scanning
**Assertions:**
- [ ] Verdict is NOT ASSESSED, never PASS — nothing was compared
- [ ] The output names the filter that produced the empty scope
- [ ] No GDD is scanned and no conflict log entry is written
---
### Case 4c: Edge Case — Named entity is not in the registry
**Fixture:**
- `modes.workflow` resolves to `full`
- `design/registry/entities.yaml` has entities `goblin` and `orc_warlord`
- `design/gdd/` contains 3 system GDDs; one mentions a `goblin_shaman`
**Input:** `/consistency-check entity:goblin_shaman`
**Expected behavior:**
1. Skill reads the registry and reports what it loaded, `Scope: entity:goblin_shaman`
2. `goblin_shaman` has no registry entry
3. Verdict: **NOT ASSESSED — `goblin_shaman` is not in the registry**, listing the
closest registered names (`goblin`); the skill stops without scanning
**Assertions:**
- [ ] Verdict is NOT ASSESSED naming the unregistered entity — never PASS
- [ ] The closest registered names are listed
- [ ] No GDD is scanned and nothing is written
---
### Case 5: Director Gate — No gate spawned in full review mode
**Fixture:**
- `project.yaml`: `modes.review_mode: full`, `modes.workflow: full`
- `design/registry/entities.yaml` has entries
- `design/gdd/` contains ≥2 GDDs
- `production/session-state/review-mode.txt` exists with `full`
**Input:** `/consistency-check`
**Expected behavior:**
1. Skill reads all GDDs and runs the consistency scan
2. Skill does NOT read `production/session-state/review-mode.txt`
1. Skill loads the registry and runs the consistency scan
2. The review mode plays no part: the skill's config block resolves only `automation`, `workflow` and `system_overrides`
3. No director gate agents are spawned at any point
4. Findings table and verdict are produced normally
4. Report and verdict are produced normally
**Assertions:**
- [ ] No director gate agents are spawned (no CD-, TD-, PR-, AD- prefixed gates)
- [ ] Skill does NOT read `production/session-state/review-mode.txt`
- [ ] The skill does not resolve or read the review mode — its config block asks only for `automation`, `workflow` and `system_overrides`
- [ ] Output contains no "Gate: [GATE-ID]" or gate-skipped entries
- [ ] Review mode has no effect on this skill's behavior
- [ ] Review mode `full` has no effect on this skill's behavior
---
### Case 6: Standard Workflow — Only the required GDD sections are compared
**Fixture:**
- `modes.workflow` resolves to `standard`
- Registry `constants:` has `crit_multiplier` value 1.5, `source: design/gdd/combat.md`
- `design/gdd/combat.md` `## Detailed Design` states `crit_multiplier = 1.5`
- `design/gdd/loot.md` states `crit_multiplier = 2.0` only in its `## Tuning Knobs` section, which `standard` does not require; no required section of `loot.md` states a value for it
**Input:** `/consistency-check`
**Expected behavior:**
1. Only hits inside the sections `standard` requires are compared
2. The `loot.md` value in `## Tuning Knobs` is not compared: no 🔴 CONFLICT is raised for it and nothing is appended to `docs/consistency-failures.md`
3. The report's Scope line says the check covered the required sections only, and lists the `loot.md` Tuning Knobs mention as not checked at this tier
4. Verdict: PASS — the required sections agree with the registry
**Assertions:**
- [ ] The Tuning Knobs value is not reported as a 🔴 CONFLICT at `standard`
- [ ] The report states the required-sections-only scope and names the mention it did not check — the narrower scan is not silent
- [ ] Verdict is PASS
---
### Case 6b: Standard Workflow — A per-system override makes one GDD `full`
**Fixture:**
- `modes.workflow` resolves to `standard`; `project.yaml` has
`workflow_overrides.system_overrides.loot: full`
- Registry `constants:` has `crit_multiplier` value 1.5, `source: design/gdd/combat.md`
- `design/gdd/combat.md` `## Detailed Design` states `crit_multiplier = 1.5`
- `design/gdd/loot.md` states `crit_multiplier = 2.0` only in its `## Tuning Knobs` section
**Input:** `/consistency-check`
**Expected behavior:**
1. The config block lists `system_overrides: loot=full`, so `loot.md`'s effective tier is `full` and every one of its sections is compared; `combat.md` stays at `standard`
2. The `loot.md` Tuning Knobs value is compared and raised as `🔴 crit_multiplier` (registry 1.5, `loot.md` 2.0)
3. The report's Scope line names `loot.md` as checked at `full`
4. Verdict: CONFLICTS FOUND
**Assertions:**
- [ ] The Tuning Knobs value in `loot.md` is compared, because its effective tier is `full` — the project tier alone would have skipped it (Case 6)
- [ ] The report names the GDD whose effective tier differs from the project's
- [ ] Verdict is CONFLICTS FOUND, not PASS
---
## Protocol Compliance
- [ ] Reads all GDDs before producing the findings table
- [ ] Findings table shown in full before any write ask (if report is requested)
- [ ] Verdict is one of exactly: CONSISTENT, CONFLICTS FOUND, DEPENDENCY GAP
- [ ] No director gates — no review-mode.txt read
- [ ] Report writing (if requested) gated by "May I write" approval
- [ ] Ends with next-step handoff appropriate to verdict
- [ ] Reads the registry once, then greps GDDs; full-section reads happen only for conflicts
- [ ] The report is shown in full before any registry write ask
- [ ] Report verdict is PASS, CONFLICTS FOUND or NOT ASSESSED (empty registry, unregistered name, or no GDD in scope — never PASS over nothing compared); post-correction verdict is COMPLETE or BLOCKED
- [ ] Applies each GDD's effective tier (the resolved `modes.workflow`, or that system's `system_overrides` row): at `standard`, compares the required GDD sections only (plus Tuning Knobs when `workflow_overrides.tuning_knobs` is true) and says so in the report
- [ ] No director gates — the review mode is not an input
- [ ] Registry writes are gated by "May I update" / "May I add"; entries are never deleted (`status: deprecated` instead)
- [ ] Every 🔴 CONFLICT is appended to `docs/consistency-failures.md` after a "May I append" ask that names it
- [ ] Closes with `AskUserQuestion` in collaborative and guided modes; in autonomous mode prints findings and records via `log_decision`
---
## Coverage Notes
- This skill checks for structural consistency between GDDs. Deep design theory
analysis (pillar drift, dominant strategies) is handled by `/review-all-gdds`.
- Formula conflict detection relies on consistent formula notation across GDDs —
informal descriptions of the same mechanic may not be detected.
- The conflict severity rubric (HIGH / MEDIUM / LOW) is defined in the skill body
and not re-enumerated here.
- This skill checks registered facts across GDDs. Deep design theory analysis
(pillar drift, dominant strategies) is handled by `/review-all-gdds`.
- Facts that are not in the registry are not compared; two GDDs that disagree on
an unregistered value are not detected. Phase 6 offers to register names that
appear in more than one GDD.
- `since-last-review` is tested only on its empty-scope path (Case 4b), not on a
run that finds changed GDDs.
- `modes.workflow: minimal` is not tested separately: a minimal project normally
has no GDDs, and an empty scope ends NOT ASSESSED as in Cases 4 and 4b.
CCGS Skill Testing Framework/skills/analysis/content-audit.md
+204 -74
View File
@@ -2,29 +2,43 @@
## Skill Summary
`/content-audit` reads GDDs in `design/gdd/` and checks whether all content
items specified there (enemies, items, levels, etc.) are accounted for in
`assets/`. It produces a gap table: Content Type → Specified Count → Found Count
→ Missing Items. No director gates are invoked. The skill does not write without
user approval. Verdicts: COMPLETE, GAPS FOUND, or MISSING CRITICAL CONTENT.
`/content-audit [system-name | --summary]` compares the content counts the design
specifies against what has been built. It reads `design/gdd/systems-index.md`
for systems and their MVP/priority tier, takes the planned named content from the
`entities:` and `items:` sections of `design/registry/entities.yaml` when the
registry has entries, and pre-scans GDDs with two greps (Summary presence and
digit-plus-content-noun counts) before full-reading only the GDDs it needs
(Phase 1). It then globs the implementation directories for each content type —
`assets/data/**/enemies/**`, `assets/data/**/items/**`, scene files under the
resolved code root, and so on (Phase 2) — and produces a gap table
`System | Content Type | Specified | Found | Gap | Status` with per-row Status
COMPLETE / IN PROGRESS / EARLY / NOT STARTED, HIGH PRIORITY flags, and a summary
line (Phase 3). The full report is written to `docs/content-audit-[YYYY-MM-DD].md`
only after "May I write"; `--summary` writes nothing. `modes.workflow` sets the
scope: `full` audits every GDD count, `standard` only counts in the required GDD
sections (naming any left out), and `minimal` has no systems index to audit
against. The run closes COMPLETE, or `NOT ASSESSED — NO DATA` when a required
input is absent, nothing in scope gives a count, or a Found count could not be
taken (code root unresolved) — never COMPLETE for a run that could not compute a
gap. No director gates are invoked.
---
## Static Assertions (Structural)
Verified automatically by `/skill-test static` — no fixture needed.
Checked against the SKILL.md by `/skill-test spec` — no fixture needed.
- [ ] Has required frontmatter fields: `name`, `description`, `argument-hint`, `user-invocable`, `allowed-tools`
- [ ] Has ≥2 phase headings
- [ ] Contains verdict keywords: COMPLETE, GAPS FOUND, MISSING CRITICAL CONTENT
- [ ] Does NOT require "May I write" language (read-only output; write is optional report)
- [ ] Contains status and verdict keywords: COMPLETE, IN PROGRESS, EARLY, NOT STARTED, NOT ASSESSED
- [ ] Contains "May I write" before the optional report write to `docs/content-audit-[YYYY-MM-DD].md`
- [ ] Has a next-step handoff (what to do after gap table is reviewed)
---
## Director Gate Checks
None. Content audit is a read-only analysis skill; no gates are invoked.
None. Content audit is an analysis skill; no gates are invoked.
---
@@ -33,132 +47,248 @@ None. Content audit is a read-only analysis skill; no gates are invoked.
### Case 1: Happy Path — All specified content present
**Fixture:**
- `design/gdd/enemies.md` specifies 4 enemy types: Grunt, Sniper, Tank, Boss
- `assets/art/characters/` contains folders: `grunt/`, `sniper/`, `tank/`, `boss/`
- `design/gdd/items.md` specifies 3 item types; all 3 found in `assets/data/items/`
- `modes.workflow` resolves to `full`
- `design/gdd/systems-index.md` lists Enemies (MVP) and Items (MVP)
- `design/registry/entities.yaml` has no entries
- `design/gdd/enemies.md` specifies "4 enemy types: Grunt, Sniper, Tank, Boss"
- `design/gdd/items.md` specifies "3 items"
- `assets/data/enemies/` contains `grunt.json`, `sniper.json`, `tank.json`, `boss.json`
- `assets/data/items/` contains 3 item `.json` files
**Input:** `/content-audit`
**Expected behavior:**
1. Skill reads all GDDs in `design/gdd/`
2. Skill scans `assets/` for each specified content item
3. All 4 enemy types and 3 item types are found
4. Gap table shows: all rows have Found Count = Specified Count, no missing items
5. Verdict is COMPLETE
1. Skill reads the systems index; the empty registry is skipped and the GDD scan is used
2. The pre-scan greps find both GDDs declare content counts; both are full-read
3. Skill globs `assets/data/**/enemies/**` and `assets/data/**/items/**`
4. Gap table: Enemies — Specified 4, Found 4, Gap 0, COMPLETE; Items — Specified 3, Found 3, Gap 0, COMPLETE
5. Summary line: 7 specified, 7 found, overall gap 0%
6. Skill presents the table and asks "May I write the full report to `docs/content-audit-[YYYY-MM-DD].md`?"
7. Output ends with `Verdict: COMPLETE — content audit finished.`
**Assertions:**
- [ ] Gap table covers all content types found in GDDs
- [ ] Each row shows Specified Count and Found Count
- [ ] No missing items when counts match
- [ ] Verdict is COMPLETE
- [ ] No files are written
- [ ] Gap table uses the columns System, Content Type, Specified, Found, Gap, Status
- [ ] Both rows show Specified = Found, Gap 0 and Status COMPLETE
- [ ] Summary line reports 7 specified, 7 found and 0% gap
- [ ] No HIGH PRIORITY flag is raised
- [ ] No file is written before the "May I write" ask is answered yes
---
### Case 2: Gaps Found — Enemy type missing from assets
### Case 2: Gaps Found — MVP enemy content barely started
**Fixture:**
- `design/gdd/enemies.md` specifies 3 enemy types: Grunt, Sniper, Boss
- `assets/art/characters/` contains: `grunt/`, `sniper/` only (Boss folder missing)
- `modes.workflow` resolves to `full`
- `design/gdd/systems-index.md` tags Enemies as MVP
- `design/gdd/enemies.md` specifies "3 enemy types: Grunt, Sniper, Boss"
- `assets/data/enemies/` contains only `grunt.json`
**Input:** `/content-audit`
**Input:** `/content-audit enemies`
**Expected behavior:**
1. Skill reads GDD — finds 3 enemy types specified
2. Skill scans `assets/art/characters/` — finds only 2
3. Gap table row for enemies: Specified 3, Found 2, Missing: Boss
4. Verdict is GAPS FOUND
1. Single-system audit: skill skips the pre-scan and full-reads `design/gdd/enemies.md`
2. Skill finds 1 enemy definition file
3. Gap table row: Enemies — Specified 3, Found 1, Gap 2, Status EARLY (33%)
4. Enemies is flagged HIGH PRIORITY: status EARLY and MVP-tagged
5. Skill asks "May I write the full report to `docs/content-audit-[YYYY-MM-DD].md`?"; on yes, the report's HIGH PRIORITY Gaps section lists Enemies with its rationale
6. Next steps include "Run `/create-stories [epic-slug]` for each HIGH PRIORITY gap."
**Assertions:**
- [ ] Gap table row identifies "Boss" as the missing item by name
- [ ] Specified Count (3) and Found Count (2) are both shown
- [ ] Verdict is GAPS FOUND when any content item is missing
- [ ] Skill does not assume the asset will be added later — it flags it now
- [ ] Specified (3), Found (1) and Gap (2) are all shown
- [ ] Status is EARLY for 1 of 3 (33%), not IN PROGRESS or NOT STARTED
- [ ] Enemies is flagged HIGH PRIORITY because it is EARLY and MVP-tagged
- [ ] Skill flags the gap now; it does not assume the content will be added later
- [ ] Next steps point to `/create-stories` for the HIGH PRIORITY gap
---
### Case 3: No GDD Content Specs Found — Guidance given
### Case 3: No Design Inputs — Nothing to audit against
**Fixture:**
- `design/gdd/` contains only `core-loop.md` which has no content inventory section
- No other GDDs exist with content specifications
- `modes.workflow` resolves to `full`
- `design/gdd/systems-index.md` does not exist
- `design/gdd/` contains no GDDs
- `design/registry/entities.yaml` has no entries
**Input:** `/content-audit`
**Expected behavior:**
1. Skill reads all GDDs — finds no content inventory sections
2. Skill outputs: "No content specifications found in GDDs — run /design-system first to define content lists"
3. No gap table is produced
4. Verdict is GAPS FOUND (cannot confirm completeness without specs)
1. Skill records its inputs as FOUND / ABSENT; the systems index, GDDs and registry entries are all ABSENT
2. Every required input is ABSENT, so the skill stops
3. Output reports `NOT ASSESSED — NO DATA`, naming the missing systems index and GDDs and the skills that produce them (`/map-systems`, `/design-system`)
4. No gap table and no gap percentage are produced (no division by zero)
5. No report write is offered and no file is written
**Assertions:**
- [ ] Skill does not produce a gap table when no GDD content specs exist
- [ ] Output recommends running `/design-system`
- [ ] Verdict reflects inability to confirm completeness
- [ ] Verdict is `NOT ASSESSED — NO DATA`, not COMPLETE
- [ ] No gap table or overall gap percentage is produced
- [ ] Output names the missing inputs and the skills that produce them
- [ ] No "May I write" prompt appears and no file is written
---
### Case 4: Edge Case — Asset in wrong format for target platform
### Case 3b: No Counts Anywhere — GDDs exist but none gives a count
**Fixture:**
- `design/gdd/audio.md` specifies audio assets as OGG format
- `assets/audio/sfx/jump.wav` exists (WAV format, not OGG)
- `assets/audio/sfx/land.ogg` exists (correct format)
- `technical-preferences.md` specifies audio format: OGG
- `modes.workflow` resolves to `full`
- `design/gdd/systems-index.md` lists Core Loop
- `design/gdd/core-loop.md` has a `## Summary` and describes mechanics only — no content count, no named list
- `design/registry/entities.yaml` has no entries
**Input:** `/content-audit`
**Expected behavior:**
1. Skill reads GDD audio spec and technical preferences for format requirements
2. Skill finds `jump.wav` — present but in wrong format
3. Gap table row for audio: Specified 2, Found 2 (by name), but `jump.wav` flagged as FORMAT ISSUE
4. Verdict is GAPS FOUND (format compliance is part of content completeness)
1. The systems index and the GDD are FOUND, so the skill does not stop at the insufficient-input check
2. Scan (b) matches nothing; `core-loop.md` has a Summary, so it is noted "No auditable content counts" without a full read
3. No row carries a Specified count, so no Gap, Status or overall gap percentage is computed (Specified is 0)
4. The run closes `Verdict: NOT ASSESSED — NO DATA`, saying no GDD gives a count and naming `/design-system` to add counts
**Assertions:**
- [ ] Skill checks asset format against GDD or technical preferences when format is specified
- [ ] `jump.wav` is flagged as FORMAT ISSUE with expected format (OGG) noted
- [ ] Format issues are distinct from missing content in the gap table
- [ ] Verdict is GAPS FOUND when format issues exist
- [ ] No overall gap percentage is computed and no row is marked COMPLETE
- [ ] Verdict is `NOT ASSESSED — NO DATA`, never COMPLETE over an empty gap table
- [ ] Output says no GDD gives a count and names `/design-system`
---
### Case 5: Gate Compliance — Read-only; no gate; gap table for human review
### Case 4: Edge Case — A pre-Summary GDD describes content without a count
**Fixture:**
- GDDs specify 10 content items; 9 are found in assets; 1 is missing
- `review-mode.txt` contains `full`
- `modes.workflow` resolves to `full`
- `design/gdd/systems-index.md` lists Enemies and Quests
- `design/gdd/enemies.md` has a `## Summary` and specifies "3 enemy types: Grunt, Sniper, Boss"; `assets/data/enemies/` has all 3
- `design/gdd/quests.md` has no `## Summary` section and says the game has "a handful of side quests" with no number or named list
- User answers yes to the report write
**Input:** `/content-audit`
**Expected behavior:**
1. Skill reads GDDs and scans assets; produces gap table
1. Skill globs `design/gdd/*.md` for the denominator: N = 3, since the glob also counts `systems-index.md` (read in step 1; it has no `## Summary`); the Summary grep matches only `enemies.md`; the content-count grep matches only `enemies.md`
2. Failing open on the missing Summary, the skill full-reads `quests.md` rather than treating it as out of scope — the fail-open rule wins over the content-count narrowing
3. Skill records the Quests content type as "Unspecified" and flags it as a design gap; it does not invent a count
4. Enemies row is COMPLETE (3 of 3)
5. The written report `docs/content-audit-[YYYY-MM-DD].md` lists `quests.md` under "Unspecified Content Counts"
6. The report notes that counts are approximations based on file scanning
**Assertions:**
- [ ] `quests.md` is full-read because it lacks `## Summary`, even though the content-count grep did not match it
- [ ] The Quests content is recorded as "Unspecified", never given an estimated count
- [ ] The Unspecified Content Counts section of the report names `quests.md`
- [ ] The Enemies row is still audited normally (Specified 3, Found 3, COMPLETE)
- [ ] The report carries the approximation caveat
---
### Case 5: Gate Compliance — No gate; optional report requires approval
**Fixture:**
- `project.yaml`: `modes.workflow: full`, `modes.review_mode: full`
- GDDs specify 10 content items; 9 are found in the implementation directories
**Input:** `/content-audit`
**Expected behavior:**
1. Skill reads the design inputs and scans the implementation; produces the gap table
2. No director gate is invoked regardless of review mode
3. Skill presents gap table to user as read-only output
4. Verdict is GAPS FOUND
5. Skill offers to write an audit report but does not write automatically
3. Skill presents the gap table and summary before any write
4. Skill asks "May I write the full report to `docs/content-audit-[YYYY-MM-DD].md`?"
5. If the user says no, nothing is written; no asset file is ever modified
**Assertions:**
- [ ] No director gate is invoked in any review mode
- [ ] Gap table is presented without auto-writing any file
- [ ] Optional report write is offered but not forced
- [ ] The report write is offered for `docs/content-audit-[YYYY-MM-DD].md` but not forced
- [ ] Skill does not modify any asset files
---
### Case 6: Standard Workflow — Only counts in the required sections are audited
**Fixture:**
- `modes.workflow` resolves to `standard`
- `design/gdd/systems-index.md` lists Enemies (MVP)
- `design/gdd/enemies.md` `## Detailed Design` specifies "3 enemy types: Grunt, Sniper, Boss"; its `## Tuning Knobs` section mentions "2 elite variants"
- `assets/data/enemies/` contains `grunt.json`, `sniper.json`, `boss.json`
**Input:** `/content-audit`
**Expected behavior:**
1. The count in `## Detailed Design`, a required section at `standard`, is audited: Enemies — Specified 3, Found 3, Gap 0, COMPLETE
2. The "2 elite variants" count sits in `## Tuning Knobs`, which `standard` does not require; it gets no gap row and is listed as not audited at this tier
3. Output ends with `Verdict: COMPLETE — content audit finished.`
**Assertions:**
- [ ] The Detailed Design count is audited and the Enemies row is COMPLETE
- [ ] The Tuning Knobs count is not in the gap table or the totals
- [ ] The output names the Tuning Knobs count as not audited at `standard` rather than dropping it silently
---
### Case 7: Minimal Workflow — No systems index to audit against
**Fixture:**
- `project.yaml`: `modes.rigor: minimal` (`workflow` resolves to `minimal`)
- `design/game-brief.md` exists; `design/gdd/systems-index.md` does not
**Input:** `/content-audit`
**Expected behavior:**
1. The systems index is ABSENT, as it normally is at `minimal`
2. Output reports `NOT ASSESSED — NO DATA`, naming the missing systems index and `/map-systems`, which creates it
3. No gap table, no gap percentage and no report write
**Assertions:**
- [ ] Verdict is `NOT ASSESSED — NO DATA`, not COMPLETE
- [ ] Output names `design/gdd/systems-index.md` as missing and `/map-systems` as the skill that produces it
- [ ] No gap percentage is computed and no file is written
---
### Case 8: Code Root Unresolved — The level count is not taken
**Fixture:**
- `modes.workflow` resolves to `full`
- `project.yaml` has no `engine.name`; `.claude/docs/technical-preferences.md` reads `[TO BE CONFIGURED]`; both `src/` and `Source/` exist, so the tree does not decide the root
- `design/gdd/systems-index.md` lists Levels (MVP) and Enemies (MVP)
- `design/gdd/levels.md` specifies "6 levels"; `design/gdd/enemies.md` specifies "3 enemy types: Grunt, Sniper, Boss"
- `assets/data/enemies/` contains 3 enemy files; no scene files exist under `assets/`
**Input:** `/content-audit`
**Expected behavior:**
1. Enemies is audited from `assets/data/`: Specified 3, Found 3, COMPLETE
2. The level scan needs the code root, which is unresolved: the Levels row's Found is `NOT ASSESSED — code root unresolved` — not 0, and its Status is not NOT STARTED
3. Levels gets no HIGH PRIORITY flag and stays out of both summary totals; the summary names it
4. The run closes `Verdict: NOT ASSESSED — NO DATA`, naming the unresolved code root
**Assertions:**
- [ ] The Levels row reads `NOT ASSESSED — code root unresolved`, never Found 0 or NOT STARTED
- [ ] The summary totals leave Levels out and say so
- [ ] Verdict is NOT ASSESSED, not COMPLETE
---
## Protocol Compliance
- [ ] Reads GDDs and asset directory before producing gap table
- [ ] Gap table shows Content Type, Specified Count, Found Count, Missing Items
- [ ] Does not write files without explicit user approval
- [ ] Reads the systems index, the registry (when it has entries) and the in-scope GDDs before scanning the implementation
- [ ] A GDD with no `## Summary` is full-read even when the content-count grep did not match it
- [ ] Applies the resolved `modes.workflow`: `standard` audits only counts in the required GDD sections and names any it left out
- [ ] Gap table shows System, Content Type, Specified, Found, Gap, Status
- [ ] Row Status is one of: COMPLETE, IN PROGRESS, EARLY, NOT STARTED; a row whose Found could not be taken is `NOT ASSESSED` and stays out of the totals
- [ ] Writes only after "May I write" and only to `docs/content-audit-[YYYY-MM-DD].md`; `--summary` writes nothing
- [ ] A missing required input, a scope with no counts, or an untaken Found count yields `NOT ASSESSED — NO DATA`, never a computed gap percentage or COMPLETE
- [ ] No director gates are invoked
- [ ] Verdict is one of: COMPLETE, GAPS FOUND, MISSING CRITICAL CONTENT
- [ ] On Unity, every content glob also runs under `Assets/` without a `data/` segment (e.g. `Assets/**/Items/**`, `*Item*.asset`) — a Unity project's items, abilities, quests and dialogue are never reported NOT STARTED from an `assets/data/`-only scan
---
## Coverage Notes
- MISSING CRITICAL CONTENT verdict (vs. GAPS FOUND) is triggered when the
missing item is tagged as critical in the GDD; this is not explicitly tested
but follows the same detection path.
- The case where `assets/` directory does not exist is not tested; the skill
would produce a MISSING CRITICAL CONTENT verdict for all specified items.
- `--summary` mode (prints the table, writes nothing, ends with "Run
`/content-audit` without `--summary` to write the full report.") is not given
its own case.
- Unity levels are found under the code root (`Assets/`, `*.unity`) and the
other content types under `Assets/` (asserted in Protocol Compliance); Unreal
levels under `Content/` (`*.umap`, with data assets as `*.uasset`); neither
engine has its own case.
- Asset naming, size and format compliance is `/asset-audit`'s job, not this
skill's.
CCGS Skill Testing Framework/skills/analysis/estimate.md
+121 -106
View File
@@ -2,22 +2,30 @@
## Skill Summary
`/estimate` estimates task or story effort using a relative-size scale (S / M /
L / XL) based on story complexity, acceptance criteria count, and historical
sprint velocity from past sprint files. Estimates are advisory and are never
written automatically. No director gates are invoked. Verdicts are effort ranges,
not pass/fail — every run produces an estimate.
`/estimate [task-description]` estimates the effort for one task. It reads the
task description from the argument (asking for clarification first if it is too
vague to estimate), the project context from `CLAUDE.md`, and any related GDD in
`design/gdd/` (Phase 1). It scans the affected code for complexity, integration
points and test coverage, and reads `production/sprints/` for similar completed
tasks and historical velocity (Phase 2). It analyzes code complexity, scope and
risk (Phase 3), then prints an estimate (Phase 4): a Complexity Assessment
table, an Effort Estimate in days for Optimistic / Expected / Pessimistic
scenarios, a recommended budget equal to the Expected figure, a Confidence of
High / Medium / Low with its drivers, Risk Factors, Dependencies, a Suggested
Breakdown with a total, and Notes and Assumptions — followed by a one-line
summary and `Verdict: COMPLETE — estimate generated.` Estimates are rounded to
half days. The skill is read-only and invokes no director gates.
---
## Static Assertions (Structural)
Verified automatically by `/skill-test static` — no fixture needed.
Checked against the SKILL.md by `/skill-test spec` — no fixture needed.
- [ ] Has required frontmatter fields: `name`, `description`, `argument-hint`, `user-invocable`, `allowed-tools`
- [ ] Has ≥2 phase headings
- [ ] Contains size labels: S, M, L, XL (the "verdict" equivalents for this skill)
- [ ] Does NOT require "May I write" language (advisory output only)
- [ ] Contains the Optimistic / Expected / Pessimistic scenarios, the High / Medium / Low confidence levels, and the `COMPLETE` verdict
- [ ] Does NOT require "May I write" language (read-only; states that no files are written)
- [ ] Has a next-step handoff (how to use the estimate in sprint planning)
---
@@ -30,115 +38,121 @@ None. Estimation is an advisory informational skill; no gates are invoked.
## Test Cases
### Case 1: Happy Path — Clear story with known tech stack
### Case 1: Happy Path — Clear task in a documented system with existing patterns
**Fixture:**
- `production/epics/combat/story-hitbox-detection.md` exists with:
- 4 clear Acceptance Criteria
- ADR reference (Accepted status)
- No "unknown" or "TBD" language in story body
- `production/sprints/sprint-003.md` through `sprint-005.md` exist with velocity data
- Tech stack is GDScript (well-understood by team per sprint history)
- `project.yaml`: `engine.name: godot`, so the code root is `src/`
- `design/gdd/combat.md` documents melee hit detection
- `src/combat/` holds the melee attack code; similar collision code already exists there
- `production/sprints/sprint-003.md` through `sprint-005.md` record completed combat tasks
**Input:** `/estimate production/epics/combat/story-hitbox-detection.md`
**Input:** `/estimate Add hitbox detection to melee attacks`
**Expected behavior:**
1. Skill reads the story file — assesses clarity, AC count, tech stack
2. Skill reads sprint history to determine average velocity
3. Skill outputs estimate: M (1–2 days) with reasoning
1. Skill restates the task; the description is specific enough, so no clarification is asked
2. Skill reads `design/gdd/combat.md` because the task relates to a documented system
3. Skill scans `src/combat/` for affected files, integration points and existing tests
4. Skill reads `production/sprints/` for similar completed tasks and velocity
5. Output follows the Phase 4 template: Complexity Assessment, Effort Estimate (Optimistic / Expected / Pessimistic, in days), Recommended budget, Confidence, Risk Factors, Dependencies, Suggested Breakdown with a Total, Notes and Assumptions
6. Recommended budget equals the Expected figure; all figures are in half-day increments
7. A brief summary gives the recommended budget, the confidence level and the single biggest risk
8. Output ends with `Verdict: COMPLETE — estimate generated.`; no files are written
**Assertions:**
- [ ] The combat GDD and the sprint history are read before the estimate is produced
- [ ] Effort is given as three day figures (Optimistic, Expected, Pessimistic), not a single number
- [ ] Recommended budget equals the Expected figure, not the Optimistic one
- [ ] Figures are rounded to half days, not hours
- [ ] The closing summary names budget, confidence and the biggest risk
- [ ] No files are written
---
### Case 2: High Uncertainty — New subsystem, no architecture decided
**Fixture:**
- `project.yaml`: `engine.name: godot`
- `src/` holds no source files for an online or matchmaking subsystem
- `design/gdd/online.md` describes the lobby with open "TBD" requirements, and says the networking approach (transport, host model) is not yet decided
**Input:** `/estimate Implement online lobby matchmaking`
**Expected behavior:**
1. Skill reads `design/gdd/online.md` and finds the unresolved requirements and the undecided networking approach
2. Code scan finds no existing subsystem; "Existing patterns available" is No
3. Risk Factors table lists the specific unknowns: the networking approach the GDD leaves undecided, new technology, unclear requirements
4. Confidence is Low, with the drivers explained
5. Phase 5 recommends a time-boxed spike via `/prototype` before committing
6. No files are written
**Assertions:**
- [ ] Confidence is Low when the task has an undecided approach, no existing code and TBD requirements
- [ ] The Risk Factors table names the specific unknowns driving the uncertainty
- [ ] Output recommends a time-boxed `/prototype` spike before committing
- [ ] No files are written
---
### Case 3: No Sprint Velocity Data — Estimate still produced, gap stated
**Fixture:**
- `project.yaml`: `engine.name: godot`; `src/core/` holds the inventory code the save touches
- `design/gdd/save-load.md` documents the save system clearly
- `production/sprints/` is empty — no historical sprints
**Input:** `/estimate Implement save and load of player inventory`
**Expected behavior:**
1. Skill reads the task, the GDD and the affected code
2. Skill looks for past sprint data and finds none
3. The estimate is still produced with all three scenarios and a recommended budget
4. Notes and Assumptions (or the Confidence explanation) states that no sprint history was available, so the figures are not calibrated to team velocity
5. No figure is padded silently to cover the missing data — any extra risk is called out explicitly
6. No files are written
**Assertions:**
- [ ] Skill does not error or stop when no sprint history exists
- [ ] Estimate is still produced with Optimistic, Expected and Pessimistic figures
- [ ] Output states that no sprint history was available for velocity calibration
- [ ] Any allowance for the missing data is stated as a risk, not hidden in the figures
---
### Case 4: Vague Task — Clarification before estimating
**Fixture:**
- `project.yaml`: `engine.name: godot`
- Project has GDDs and source code under `src/`; sprint history exists
**Input:** `/estimate make combat feel better`
**Expected behavior:**
1. Skill reads the task description and finds it too vague to estimate meaningfully (no scope, no named change)
2. Skill asks for clarification before proceeding
3. No Complexity Assessment, Effort Estimate or verdict is produced until the task is clarified
4. No files are written
**Assertions:**
- [ ] Estimate is M for a clear, well-scoped story with known tech
- [ ] Reasoning references AC count, tech stack familiarity, and velocity data
- [ ] Estimate is presented as a range (e.g., "1–2 days"), not a single point
- [ ] Skill asks for clarification for a task that names no concrete change
- [ ] No estimate figures are produced before the clarification is answered
- [ ] `Verdict: COMPLETE` is not printed for an unclarified task
- [ ] No files are written
---
### Case 2: High Uncertainty — Unknown system, no ADR yet
**Fixture:**
- `production/epics/online/story-lobby-matchmaking.md` exists with:
- 2 vague Acceptance Criteria (using "should" and "TBD")
- No ADR reference — matchmaking architecture not yet decided
- References new subsystem ("online/matchmaking") with no existing source files
**Input:** `/estimate production/epics/online/story-lobby-matchmaking.md`
**Expected behavior:**
1. Skill reads story — finds vague AC, no ADR, no existing source
2. Skill flags multiple uncertainty factors
3. Estimate is L–XL with an explicit risk note: "Estimate range is wide due to architectural unknowns"
4. Skill recommends creating an ADR before development begins
**Assertions:**
- [ ] Estimate is L or XL (not S or M) when significant unknowns exist
- [ ] Risk note explains the specific unknowns driving the wide range
- [ ] Output recommends resolving architectural questions first
- [ ] No files are written
---
### Case 3: No Sprint Velocity Data — Conservative defaults used
**Fixture:**
- Story file exists and is well-defined
- `production/sprints/` is empty — no historical sprints
**Input:** `/estimate production/epics/core/story-save-load.md`
**Expected behavior:**
1. Skill reads story — assesses complexity
2. Skill attempts to read sprint velocity data — finds none
3. Skill notes: "No sprint history found — using conservative defaults for velocity"
4. Estimate is produced using default assumptions (e.g., 1 story point = 1 day)
5. No files are written
**Assertions:**
- [ ] Skill does not error when no sprint history exists
- [ ] Output explicitly notes that conservative defaults are being used
- [ ] Estimate is still produced (not blocked by missing velocity)
- [ ] Conservative defaults produce a higher (not lower) estimate range
---
### Case 4: Multiple Stories — Each estimated individually plus sprint total
**Fixture:**
- User provides a sprint file: `production/sprints/sprint-007.md` with 4 stories
- Sprint history exists (3 previous sprints)
**Input:** `/estimate production/sprints/sprint-007.md`
**Expected behavior:**
1. Skill reads sprint file — identifies 4 stories
2. Skill estimates each story individually: S, M, M, L
3. Skill computes sprint total: approximately 6–8 story points
4. Skill presents per-story estimates followed by sprint total
5. No files are written
**Assertions:**
- [ ] Each story receives its own estimate label
- [ ] Sprint total is presented after individual estimates
- [ ] Total is a sum range derived from individual ranges
- [ ] Skill handles sprint files (not just single story files) as input
---
### Case 5: Gate Compliance — No gate; estimates are informational
**Fixture:**
- Story file exists with medium complexity
- `review-mode.txt` contains `full`
- `project.yaml`: `engine.name: godot`, `modes.review_mode: full`
- Task relates to a documented system with medium complexity; its code is under `src/`
**Input:** `/estimate production/epics/core/story-item-pickup.md`
**Input:** `/estimate Add item pickup with inventory stacking`
**Expected behavior:**
1. Skill reads story and sprint history; computes estimate
1. Skill reads context, scans code and sprint history; produces the estimate
2. No director gate is invoked in any review mode
3. Estimate is presented as advisory output only
4. Skill notes: "Use this estimate in /sprint-plan when selecting stories for the next sprint"
4. Next steps include "run `/sprint-plan update` to add it to the next sprint"
**Assertions:**
- [ ] No director gate is invoked regardless of review mode
@@ -150,19 +164,20 @@ None. Estimation is an advisory informational skill; no gates are invoked.
## Protocol Compliance
- [ ] Reads story file before estimating
- [ ] Reads sprint velocity history when available
- [ ] Produces effort range (S/M/L/XL), not a single number
- [ ] Reads the task argument, project context and any related GDD before estimating
- [ ] Reads `production/sprints/` for similar tasks and velocity when available
- [ ] Produces an effort range (Optimistic / Expected / Pessimistic days), not a single number
- [ ] Does not write any files
- [ ] No director gates are invoked
- [ ] Always produces an estimate (never blocked by missing data; uses defaults instead)
- [ ] Ends with Phase 5 next steps matched to the result (`/prototype` for Low confidence, `/create-stories` for tasks over 10 days, `/sprint-plan update` to schedule)
---
## Coverage Notes
- The skill does not produce PASS/FAIL verdicts; the "verdict" here is the
effort range itself. Test assertions focus on the accuracy of the range
and the quality of the reasoning, not a binary outcome.
- Team-specific velocity calibration (what "M" means for this team) is an
implementation detail not tested here; it is configured via sprint history.
- The skill estimates one task per run; a sprint file with several stories is
not a supported input.
- The "task over 10 days → break it into smaller stories via `/create-stories`"
branch is not given its own case.
- The accuracy of the day figures depends on the team and is not tested; the
assertions check the range's structure, its sources and its stated drivers.
CCGS Skill Testing Framework/skills/analysis/perf-profile.md
+119 -90
View File
@@ -2,170 +2,199 @@
## Skill Summary
`/perf-profile` is a structured performance profiling workflow that identifies
bottlenecks and recommends optimizations. If profiler data or performance logs
are provided, it analyzes them directly. If not, it guides the user through a
manual profiling checklist. No director gates are invoked. The skill asks
"May I write to `production/qa/perf-[date].md`?" before persisting a report.
Verdicts: WITHIN BUDGET, CONCERNS, or OVER BUDGET.
`/perf-profile` is a static-analysis performance profiler. It takes a system
name or `full` as its scope, records every input it needs as `FOUND` or
`ABSENT` before producing anything, resolves `performance.enforce` from the
config block injected at its top (Phase 0), and reads the committed budgets
with `get_effective_yaml_key` for `performance.target_framerate`,
`performance.frame_budget_ms`, `performance.draw_call_limit` and
`performance.memory_ceiling_mb` before falling back to design docs or
`CLAUDE.md` (Phase 2). It analyses source code for CPU, memory, rendering and
I/O hotspot candidates (Phase 3) — it does not ingest runtime profiler exports.
The report rates each budgeted row `OK` / `WARNING` / `OVER`; a metric with no
committed budget is `NOT ASSESSED`, and the summary gives headroom only against a
budget that is set (`no budget set — headroom not assessed` otherwise). The skill asks "May I write this profiling
report to `production/polish/[scope]-report-[date].md`?" before writing (Phase
4), offers implement / cut scope / defer / escalate choices for M- or L-effort
hotspots (Phase 5), and hands off to `/architecture-decision`, `/scope-check`
or `/sprint-plan update` (Phase 6). No director gates are invoked. Verdicts:
COMPLETE, or NOT ASSESSED — NO DATA when every required input is absent.
---
## Static Assertions (Structural)
Verified automatically by `/skill-test static` — no fixture needed.
Checked against the SKILL.md by `/skill-test spec` — no fixture needed.
- [ ] Has required frontmatter fields: `name`, `description`, `argument-hint`, `user-invocable`, `allowed-tools`
- [ ] Has ≥2 phase headings
- [ ] Contains verdict keywords: WITHIN BUDGET, CONCERNS, OVER BUDGET
- [ ] Contains "May I write" language (skill writes perf report)
- [ ] Has a next-step handoff (what to do after performance findings are reviewed)
- [ ] Contains verdict keywords: COMPLETE, NOT ASSESSED — NO DATA, and the per-metric statuses OK / WARNING / OVER
- [ ] Contains "May I write" language naming `production/polish/[scope]-report-[date].md`
- [ ] Has a next-step handoff (Phase 6: `/architecture-decision`, `/scope-check`, `/sprint-plan update`)
---
## Director Gate Checks
None. Performance profiling is an advisory analysis skill; no gates are invoked.
None. Performance profiling is an advisory analysis skill; no gates are invoked
in any review mode.
---
## Test Cases
### Case 1: Happy Path — Frame data provided, draw call spike found
### Case 1: Happy Path — Budgets committed, combat hotspots found
**Fixture:**
- User provides `production/qa/profiler-export-2026-03-15.json` with frame time data
- Data shows: average frame time 14ms (within 16.6ms budget), but frames 42–48 spike to 28ms
- Spike correlates with a scene with 450 draw calls (budget: 200)
- `project.yaml`: `engine.name: godot`, `performance.target_framerate: 60`,
`performance.frame_budget_ms: 16.6`, `performance.draw_call_limit: 200`,
`performance.memory_ceiling_mb: 2048`; `performance.enforce` unset (resolves
to `warn`); `modes.automation` unset (collaborative)
- No config key, design doc or `CLAUDE.md` section states a load-time target
- `src/gameplay/combat/enemy_manager.gd` has a `_process()` that loops over every
enemy and, inside that loop, over every active projectile, and casts a ray per
enemy every frame
- Reworking the nested loop into a spatial partition is a multi-day change
**Input:** `/perf-profile production/qa/profiler-export-2026-03-15.json`
**Input:** `/perf-profile combat`
**Expected behavior:**
1. Skill reads profiler data
2. Skill identifies average frame time is within budget
3. Skill identifies draw call spike on frames 42–48 (450 calls vs 200 budget)
4. Verdict is CONCERNS (average OK, but spikes indicate an issue)
5. Skill recommends batching or culling for the identified scene
6. Skill asks "May I write to `production/qa/perf-2026-04-06.md`?"
1. Skill records its inputs: source code FOUND, committed budgets FOUND
2. Phase 0: `performance.enforce` resolves to `warn` — violations are findings, not blockers
3. Phase 2: the four `performance.*` keys are read with `get_effective_yaml_key` before any fallback
4. Phase 3: the combat `_process()` functions are listed; the nested loop and the per-frame raycast are identified
5. Phase 4: the report's Performance Budgets table uses 16.6 ms, 2048 MB and 200 draw calls as budgets, each of those three rows with an OK / WARNING / OVER status; the Load time row has no committed budget and is `NOT ASSESSED`; hotspots and recommendations cite `enemy_manager.gd` with a line number
6. Summary gives the top 3 hotspots, estimated headroom against the three set budgets, `no budget set — headroom not assessed` for load time, and a recommended next action
7. Skill asks "May I write this profiling report to `production/polish/combat-report-[date].md`?"
8. The nested-loop hotspot is rated Fix Effort M or L, so Phase 5 offers: implement, reduce scope via `/scope-check`, defer to Polish, or escalate via `/architecture-decision`
9. Verdict: COMPLETE
**Assertions:**
- [ ] Spike frames are identified by frame number
- [ ] Draw call count and budget are compared explicitly
- [ ] Verdict is CONCERNS when spikes exceed budget even if average is OK
- [ ] At least one specific optimization recommendation is given
- [ ] "May I write" prompt appears before writing report
- [ ] Budgets come from the four `performance.*` keys read with `get_effective_yaml_key`, not from design docs or a hardcoded 16.67 ms
- [ ] The nested loop and the per-frame raycast are reported as hotspots with a `file:line` location
- [ ] Each optimization recommendation states location, expected gain, risk and approach
- [ ] The frame time, memory and draw call rows carry an OK / WARNING / OVER status; the Load time row is `NOT ASSESSED`, not OK
- [ ] No headroom is claimed for load time; the summary says `no budget set — headroom not assessed`
- [ ] "May I write" names `production/polish/combat-report-[date].md` and precedes any write
- [ ] The M/L-effort hotspot triggers the Phase 5 choice (implement / `/scope-check` / defer to Polish / `/architecture-decision`)
- [ ] Verdict is COMPLETE
---
### Case 2: No Profiler Data — Manual checklist output
### Case 2: No Committed Budget — Budget rows are NOT ASSESSED
**Fixture:**
- User runs `/perf-profile` with no arguments
- No profiler data files exist in `production/qa/`
- `project.yaml` has `engine.name: godot` and no `performance` block
- No design doc and no `CLAUDE.md` section states a frame-rate, memory, load-time or draw-call target
- `src/` contains gameplay scripts with `_process()` functions
**Input:** `/perf-profile`
**Input:** `/perf-profile full`
**Expected behavior:**
1. Skill finds no profiler data
2. Skill outputs a manual profiling checklist for the user to work through:
- Enable Godot profiler or target engine's profiler
- Record a 60-second play session
- Export frame time data
- Note any dropped frames or hitches
3. Skill asks user to provide data once collected before running analysis
1. `get_effective_yaml_key` returns nothing for the four budget keys; the design-doc and `CLAUDE.md` fallback finds no target either
2. Source code is FOUND, so Phase 3 analysis still runs and hotspots are reported
3. Every row of the Performance Budgets table is `NOT ASSESSED` — none is measured against the template's `[16.67ms]` placeholder
4. The summary does not claim headroom against a budget nobody set: it says `no budget set — headroom not assessed`
5. Skill asks "May I write this profiling report to `production/polish/full-report-[date].md`?"; the `NOT ASSESSED` rows stay in the file offered for writing
**Assertions:**
- [ ] Skill does not crash or emit a verdict when no data is provided
- [ ] Manual profiling checklist is output (actionable steps, not just an error)
- [ ] No verdict is emitted (there is nothing to assess yet)
- [ ] No files are written
- [ ] No budget row shows OK, WARNING or OVER; each is `NOT ASSESSED`
- [ ] The `[16.67ms]` placeholder is never used as a budget, and no headroom figure is claimed
- [ ] Hotspot analysis still runs and is reported
- [ ] The `NOT ASSESSED` rows are kept in the report offered for writing, not edited out
---
### Case 3: Over Budget — Frame budget exceeded for target platform
### Case 3: Nothing to Profile — Whole verdict NOT ASSESSED — NO DATA
**Fixture:**
- Profiler data shows consistent 22ms frame times (target: 16.6ms for 60fps)
- All frames exceed budget; no single spike — systemic issue
- `technical-preferences.md` specifies target platform: PC, 60fps
- Fresh project: `project.yaml` has `engine.name: godot` and no `performance` block, and the code root `src/` is absent
- No design docs exist
- No profiler output exists anywhere
**Input:** `/perf-profile production/qa/profiler-export-2026-03-20.json`
**Input:** `/perf-profile full`
**Expected behavior:**
1. Skill reads profiler data and technical preferences for performance budget
2. All frames are over the 16.6ms budget
3. Verdict is OVER BUDGET
4. Skill outputs a prioritized optimization list (e.g., LOD system, shader complexity, physics tick rate)
5. Skill asks "May I write" before writing report
1. Skill lists its inputs and records each: source code ABSENT, budgets ABSENT, profiler output ABSENT
2. Every required input is ABSENT, so the skill stops before Phase 3
3. Skill reports `NOT ASSESSED — NO DATA` as the whole verdict, naming what was missing and which skill produces it
4. No report template is filled in and nothing is offered for writing
**Assertions:**
- [ ] Verdict is OVER BUDGET when all or most frames exceed budget
- [ ] Target frame budget is read from `technical-preferences.md` (not hardcoded)
- [ ] Optimization priority list is provided, not just the raw verdict
- [ ] "May I write" prompt appears before report write
- [ ] Each input is recorded as FOUND or ABSENT, not assumed present
- [ ] The whole verdict is `NOT ASSESSED — NO DATA`, not COMPLETE
- [ ] Output names each missing input and which skill produces it
- [ ] No filled-in report, no headroom figure, and no "May I write" prompt
---
### Case 4: Previous Perf Report Exists — Delta comparison
### Case 4: Local Enforcement Override `block` — Violations are blockers
**Fixture:**
- `production/qa/perf-2026-03-28.md` exists with prior results (avg 15ms, max 19ms)
- New profiler export shows: avg 13ms, max 17ms
- Both reports are for the same scene
- `project.yaml`: `engine.name: godot`, `performance.enforce: warn`, `performance.draw_call_limit: 200`, the other three budgets set
- `project.local.yaml`: `performance.enforce: block`, so the resolved block at the top of the skill shows `block`
- `src/levels/forest/forest_builder.gd` instantiates 600 individual foliage `MeshInstance3D` nodes, each with its own material and no instancing
- `modes.automation` unset (collaborative)
**Input:** `/perf-profile production/qa/profiler-export-2026-04-05.json`
**Input:** `/perf-profile forest`
**Expected behavior:**
1. Skill reads new profiler data
2. Skill detects prior report for the same scene
3. Skill computes deltas: avg improved 2ms, max improved 2ms
4. Skill presents regression check: no regressions detected
5. Verdict is WITHIN BUDGET; report notes improvement since last profile
1. Phase 0 uses `block` from the resolved config block at the top of the skill, not the `warn` a fresh read of `project.yaml` would give
2. The draw-call estimate for the forest scope exceeds 200; the Draw calls row is OVER
3. The report states explicitly that the violation is a blocker — release-stopping — and that `/gate-check` will FAIL the Polish gate on it
4. A recommendation (e.g., batching or instancing the foliage) is given with its expected gain
5. Skill asks "May I write" before writing `production/polish/forest-report-[date].md`
**Assertions:**
- [ ] Skill checks `production/qa/` for prior perf reports before writing
- [ ] Delta comparison is shown (prior vs. current for key metrics)
- [ ] Verdict is WITHIN BUDGET when current metrics are within budget
- [ ] Improvement trend is noted positively in the report
- [ ] The local `block` applies, not the committed `warn` — `performance.enforce` comes from the resolved block
- [ ] The Draw calls row is OVER against the 200 budget
- [ ] The report states the violation is a blocker and that `/gate-check` will FAIL the Polish gate on it
- [ ] "May I write" precedes any write
---
### Case 5: Gate Compliance — No gate; performance-analyst separate
### Case 5: Enforcement Level `off` in Full Review Mode — Informational only, no gates
**Fixture:**
- Profiler data shows CONCERNS-level findings (some spikes)
- `review-mode.txt` contains `full`
- `project.yaml`: `engine.name: godot`, `modes.review_mode: full`, `performance.enforce: off`, all four budgets set
- `src/` contains a `_process()` whose estimated frame cost exceeds `performance.frame_budget_ms`
**Input:** `/perf-profile production/qa/profiler-export-2026-04-01.json`
**Input:** `/perf-profile full`
**Expected behavior:**
1. Skill analyzes profiler data; verdict is CONCERNS
2. No director gate is invoked regardless of review mode
3. Output notes: "For in-depth analysis, consider running `/perf-profile` with the performance-analyst agent"
4. Skill asks "May I write" and writes report on user approval
1. No director gate is invoked, whatever the review mode
2. The estimated frame time is still reported in the Performance Budgets table
3. Because budgets are informational under `off`, the over-budget frame time is not raised as a violation finding or as a recommendation to fix
4. Skill asks "May I write" before writing the report
5. Output ends with the Phase 6 next steps: `/architecture-decision`, `/scope-check [feature]`, `/sprint-plan update`
**Assertions:**
- [ ] No director gate is invoked in any review mode
- [ ] Performance-analyst consultation is suggested (not mandated)
- [ ] "May I write" prompt appears before report write
- [ ] Verdict is CONCERNS for spike-based findings
- [ ] The estimated frame time still appears in the budget table
- [ ] The over-budget metric is not raised as a violation finding or a fix recommendation
- [ ] "May I write" precedes any write
- [ ] Next steps name `/architecture-decision`, `/scope-check` and `/sprint-plan update`
---
## Protocol Compliance
- [ ] Reads profiler data when provided; outputs checklist when not
- [ ] Reads `technical-preferences.md` for target platform frame budget
- [ ] Checks for prior perf reports to enable delta comparison
- [ ] Always asks "May I write" before writing report
- [ ] Records each input as FOUND or ABSENT before producing any report
- [ ] Reads the four `performance.*` budget keys with `get_effective_yaml_key` before falling back to design docs or `CLAUDE.md`
- [ ] A metric with no committed budget is `NOT ASSESSED` — never measured against the `[16.67ms]` placeholder — and the summary claims headroom only against a budget that is set
- [ ] Applies `performance.enforce` from the resolved block: `warn` → findings, `block` → blockers, `off` → informational
- [ ] Asks "May I write" naming `production/polish/[scope]-report-[date].md` before writing, and keeps `NOT ASSESSED` sections in the written file
- [ ] No director gates are invoked
- [ ] Verdict is one of: WITHIN BUDGET, CONCERNS, OVER BUDGET
- [ ] Ends with the Phase 6 next-step handoff
---
## Coverage Notes
- Platform-specific profiling workflows (console, mobile) are not tested here;
the checklist output in Case 2 would be platform-specific in practice.
- The delta comparison in Case 4 assumes reports cover the same scene; cross-scene
comparisons are not explicitly handled.
- An unrecognized `performance.enforce` value (surfaced to the user, then
treated as `warn`) is not tested here.
- The skill is static analysis only; confirming a hotspot with runtime
profiling is listed under the report's "Requires Investigation" section and is
not exercised by these cases.
- Deferring M/L hotspots under Phase 5 choice C records them under
`### Deferred to Polish`; that recording is not asserted separately.
CCGS Skill Testing Framework/skills/analysis/scope-check.md
+117 -83
View File
@@ -2,23 +2,33 @@
## Skill Summary
`/scope-check` is a Haiku-tier read-only skill that analyzes a feature, sprint,
or story for scope creep risk. It reads sprint and story files and compares them
against the active milestone goals. It is designed for fast, low-cost checks
before or during planning. No director gates are invoked. No files are written.
Verdicts: ON SCOPE, CONCERNS, or SCOPE CREEP DETECTED.
`/scope-check` is a Haiku-tier read-only skill that compares a feature's,
sprint's or milestone's original planned scope against what has actually been
implemented. It takes the whole argument string — a feature name (which may be
several words), a sprint number, or a milestone name — and locates the baseline
at `design/gdd/[feature].md` (or a matching file in `design/`),
`production/sprints/sprint-NNN.md`, or `production/milestones/[name].md`. If the
baseline is absent it reports the missing file and stops. It then reads the
current state (related source files, `git log`, TODO/FIXME comments, the active
sprint plan) and produces a comparison report: Original Scope, Current Scope,
Scope Additions, Scope Removals, a Bloat Score, a Risk Assessment and
Cut / Defer / Keep / Flag recommendations. The verdict follows the net scope
change: ≤10% PASS, 10–25% CONCERNS, >25% FAIL. When the percentage would be
meaningless — a baseline that enumerates no items, or a current state that
cannot be read — the verdict is NOT ASSESSED and the numeric block is replaced.
No files are written and no director gates are invoked.
---
## Static Assertions (Structural)
Verified automatically by `/skill-test static` — no fixture needed.
Checked against the SKILL.md by `/skill-test spec` — no fixture needed.
- [ ] Has required frontmatter fields: `name`, `description`, `argument-hint`, `user-invocable`, `allowed-tools`
- [ ] Has ≥2 phase headings
- [ ] Contains verdict keywords: ON SCOPE, CONCERNS, SCOPE CREEP DETECTED
- [ ] Does NOT require "May I write" language (read-only skill)
- [ ] Has a next-step handoff (what to do based on verdict)
- [ ] Contains verdict keywords: PASS, CONCERNS, NOT ASSESSED, FAIL
- [ ] Does NOT require "May I write" language (the skill states it is read-only and `allowed-tools` has no Write/Edit)
- [ ] Has a next-step handoff (Phase 5: follow-up per verdict)
---
@@ -30,139 +40,163 @@ None. Scope check is a read-only advisory skill; no gates are invoked.
## Test Cases
### Case 1: Happy Path — Sprint stories align with milestone goals
### Case 1: Happy Path — Multi-word feature on track
**Fixture:**
- `production/milestones/milestone-03.md` lists 3 goals: combat system, enemy AI, level loading
- `production/sprints/sprint-006.md` contains 5 stories, all tagged to one of the 3 goals
- `production/session-state/active.md` references milestone-03 as the active milestone
- `design/gdd/inventory-crafting-system.md` enumerates 20 scope items
- `src/gameplay/inventory/` and `src/gameplay/crafting/` implement all 20, plus one addition (item sorting) found in a recent commit
- No items were dropped
**Input:** `/scope-check`
**Input:** `/scope-check inventory crafting system`
**Expected behavior:**
1. Skill reads active milestone goals from milestone-03
2. Skill reads sprint-006 stories and checks each against milestone goals
3. All 5 stories map to one of the 3 goals
4. Skill outputs a mapping table: story → milestone goal
5. Verdict is ON SCOPE
1. Skill uses the whole argument, "inventory crafting system", to locate the baseline — not only "inventory"
2. Skill reads `design/gdd/inventory-crafting-system.md` as the baseline (Phase 1)
3. Skill reads the current state: related source files, `git log --oneline` for the work, TODO/FIXME comments (Phase 2)
4. Report lists Original Scope and Current Scope; the Scope Additions table names item sorting with its source, date, justification and effort
5. Bloat Score: 20 original, 21 current, 1 added (+5%), 0 removed
6. Verdict: **PASS** — On Track
7. Phase 5: no action required; suggests re-running before the next milestone; ends with "Run `/scope-check [name]` again after cuts are made to verify the verdict improves."
**Assertions:**
- [ ] Each story is mapped to a milestone goal in the output
- [ ] Verdict is ON SCOPE when all stories map to milestone goals
- [ ] The baseline is located from the full multi-word argument, not its first word
- [ ] The addition is named in the Scope Additions table with source, when, justified and effort columns
- [ ] Bloat Score shows original, current, added and removed counts and the net percentage
- [ ] Verdict is PASS for a net change ≤10%
- [ ] No files are written
- [ ] Skill does not modify sprint or milestone files
---
### Case 2: Scope Creep Detected — Stories introducing systems not in milestone
### Case 2: Significant Creep — Sprint gained unplanned items
**Fixture:**
- `production/milestones/milestone-03.md` goals: combat, enemy AI, level loading
- `production/sprints/sprint-006.md` contains 5 stories:
- 3 stories map to milestone goals
- 2 stories reference "online leaderboard" and "achievement system" (not in milestone-03)
- `production/sprints/sprint-003.md` lists 8 planned stories
- Commits since the sprint start add an online leaderboard, an achievement system and a photo mode, none of which appear in the sprint plan
- No planned stories were dropped
**Input:** `/scope-check`
**Input:** `/scope-check sprint-3`
**Expected behavior:**
1. Skill reads milestone goals and sprint stories
2. Skill identifies 2 stories with no matching milestone goal
3. Skill names the out-of-scope stories: "Online Leaderboard Feature", "Achievement System Setup"
4. Verdict is SCOPE CREEP DETECTED
1. Skill reads `production/sprints/sprint-003.md` as the baseline
2. Skill finds the three additions from `git log` and the source tree
3. Scope Additions table names all three; Bloat Score shows 8 original, 11 current, +37.5%
4. Recommendations sort items into Cut / Defer / Keep / Flag
5. Verdict: **FAIL** — Significant Creep
6. Phase 5: recommends escalating to the producer and references `/sprint-plan update` for re-planning or `/estimate` to re-baseline
**Assertions:**
- [ ] Out-of-scope stories are named explicitly in the output
- [ ] Verdict is SCOPE CREEP DETECTED when any story has no milestone goal match
- [ ] Skill does not automatically remove the stories — findings are advisory
- [ ] Output recommends deferring the out-of-scope stories to a later milestone
- [ ] Each unplanned addition is named explicitly in the Scope Additions table
- [ ] Verdict is FAIL for a net change between 25% and 50%
- [ ] Output recommends escalating to the producer and references `/sprint-plan update` or `/estimate`
- [ ] Skill does not edit the sprint plan or remove stories — findings are advisory
---
### Case 3: No Milestone Defined — CONCERNS; scope cannot be validated
### Case 3: Baseline Missing — Report the missing file and stop
**Fixture:**
- `production/session-state/active.md` has no milestone reference
- `production/milestones/` directory exists but is empty
- `production/sprints/sprint-006.md` has 4 stories
- No `design/gdd/crafting.md` and no file in `design/` matches "crafting"
- `src/gameplay/crafting/` contains implemented code
**Input:** `/scope-check`
**Input:** `/scope-check crafting`
**Expected behavior:**
1. Skill reads active.md — finds no milestone reference
2. Skill checks `production/milestones/` — no milestone files found
3. Skill outputs: "No active milestone defined — scope cannot be validated"
4. Verdict is CONCERNS
1. Phase 1 looks for `design/gdd/crafting.md` and a matching file in `design/`; none exists
2. Skill reports which baseline file was missing and stops
3. No comparison report, Bloat Score or percentage is produced
**Assertions:**
- [ ] Skill does not error when no milestone is defined
- [ ] Output explicitly states that scope validation requires a milestone reference
- [ ] Verdict is CONCERNS (not ON SCOPE or SCOPE CREEP DETECTED without data)
- [ ] Output suggests running `/milestone-review` or creating a milestone
- [ ] Output names the baseline file it looked for and did not find
- [ ] Skill stops at Phase 1 — no Phase 3 comparison report is rendered
- [ ] No percentage and no PASS verdict are produced without a baseline
- [ ] No files are written
---
### Case 4: Single Story Check — Evaluated against its parent epic
### Case 4: Empty Baseline — NOT ASSESSED, not 0%
**Fixture:**
- User targets a single story: `production/epics/combat/story-parry-timing.md`
- Story references parent epic: `epic-combat.md`
- `production/epics/combat/epic-combat.md` has scope: "melee combat mechanics"
- Story title: "Implement parry timing window" — matches epic scope
- `design/gdd/dialogue.md` exists but contains only section headings and `[TO BE CONFIGURED]` placeholders — no scope items
- `src/narrative/dialogue/` contains implemented code
**Input:** `/scope-check production/epics/combat/story-parry-timing.md`
**Input:** `/scope-check dialogue`
**Expected behavior:**
1. Skill reads the specified story file
2. Skill reads the parent epic to get scope definition
3. Skill evaluates story against epic scope — "parry timing" matches "melee combat"
4. Verdict is ON SCOPE
1. Phase 1 finds the baseline document, so it does not stop there
2. Phase 4 detects that the baseline enumerates no scope items
3. The Bloat Score block is replaced with `Baseline unusable — see verdict` and the reason; no `Original items: 0` or `Net scope change: 0%` is rendered
4. Verdict: **NOT ASSESSED**, stating that the baseline side could not be read and why
5. Phase 5: says the baseline document needs populating; does not offer a re-run against the same inputs
**Assertions:**
- [ ] Single-file argument is accepted (story path, not sprint)
- [ ] Skill reads the parent epic referenced in the story file
- [ ] Story is evaluated against epic scope (not milestone scope) in single-story mode
- [ ] Verdict is ON SCOPE when story matches epic scope
- [ ] Verdict is NOT ASSESSED, never PASS, when the baseline has no items
- [ ] No computed percentage appears anywhere in the report
- [ ] Output says which side was unusable and what would fix it (populate the baseline document)
- [ ] Skill does not offer a re-run against the same inputs
---
### Case 5: Gate Compliance — No gate; PR may be consulted separately
### Case 4b: Unreadable Current State — NOT ASSESSED, not −100%
**Fixture:**
- Sprint has 2 SCOPE CREEP stories and 3 ON SCOPE stories
- `review-mode.txt` contains `full`
- `production/sprints/sprint-004.md` lists 6 planned stories
- No source files relate to those stories, `git log` shows no commits since the sprint start, and nothing is marked in progress
**Input:** `/scope-check`
**Input:** `/scope-check sprint-4`
**Expected behavior:**
1. Skill reads milestone and sprint; identifies 2 scope creep items
2. No director gate is invoked regardless of review mode
3. Skill presents findings with SCOPE CREEP DETECTED verdict
4. Output notes: "Consider raising scope concerns with the Producer before sprint begins"
5. Skill ends without writing any files
1. Phase 1 reads `production/sprints/sprint-004.md` as the baseline — 6 items
2. Phase 2 finds nothing to read for the current state
3. Phase 4 emits NOT ASSESSED instead of computing a change: 0 current items would compute −100%, which the ≤10% row would read as PASS
4. The Bloat Score block is replaced with `Current state unusable — see verdict`; no `Current items: 0` or percentage is rendered
5. The verdict says the current-state side could not be read and why, and Phase 5 says what would fix it (point the skill at where the work lives) without offering a re-run on the same inputs
**Assertions:**
- [ ] Verdict is NOT ASSESSED, never PASS
- [ ] No −100% (or any percentage) appears in the report
- [ ] Output names the current state as the side that could not be read
- [ ] Skill does not offer a re-run against the same inputs
---
### Case 5: Minor Creep in Full Review Mode — CONCERNS, no gates
**Fixture:**
- `project.yaml` has `modes.review_mode: full`
- `production/milestones/alpha.md` enumerates 10 scope items
- Current state has all 10 plus two additions (a settings menu rework and an extra enemy type)
**Input:** `/scope-check alpha`
**Expected behavior:**
1. Skill reads `production/milestones/alpha.md` as the baseline
2. Bloat Score: 10 original, 12 current, +20%
3. Verdict: **CONCERNS** — Minor Creep
4. No director gate is invoked regardless of review mode
5. Phase 5: offers to identify the 2–3 additions with the best cut ratio and references `/sprint-plan update`
6. Skill ends without writing any files
**Assertions:**
- [ ] Verdict is CONCERNS for a net change between 10% and 25%
- [ ] No director gate is invoked in any review mode
- [ ] Producer consultation is suggested (not mandated)
- [ ] Output offers to identify the additions with the best cut ratio and references `/sprint-plan update`
- [ ] No files are written
- [ ] Verdict is SCOPE CREEP DETECTED
---
## Protocol Compliance
- [ ] Reads milestone goals and sprint/story files before analysis
- [ ] Maps each story to a milestone goal (or flags as unmapped)
- [ ] Locates the baseline document (`design/gdd/`, `production/sprints/`, `production/milestones/`) before any comparison, and stops if it is absent
- [ ] Reads the current state from source files, `git log`, TODO/FIXME comments and the active sprint plan
- [ ] Verdict follows the net-change table: ≤10% PASS, 10–25% CONCERNS, >25% FAIL
- [ ] Emits NOT ASSESSED instead of a percentage when the baseline has no items or the current state cannot be read
- [ ] Does not write any files
- [ ] No director gates are invoked
- [ ] Runs on Haiku model tier (fast, low-cost)
- [ ] Verdict is one of: ON SCOPE, CONCERNS, SCOPE CREEP DETECTED
- [ ] Declares `model: haiku` (when a declared tier is used: `.claude/docs/model-tiers.md`)
---
## Coverage Notes
- The case where the sprint file itself does not exist is not tested; the
skill would output a CONCERNS verdict with a message about missing sprint data.
- Partial scope overlap (story touches a milestone goal but also introduces
new scope) is not explicitly tested; implementation may classify this as
CONCERNS rather than SCOPE CREEP DETECTED.
- The >50% "Out of Control" band shares the FAIL verdict with Case 2 and is not
tested separately.
CCGS Skill Testing Framework/skills/analysis/security-audit.md
+173 -82
View File
@@ -2,166 +2,257 @@
## Skill Summary
`/security-audit` audits the game for security risks including save data
integrity, network communication, anti-cheat exposure, and data privacy. It
reads source files in `src/` for security patterns and checks whether sensitive
data is handled correctly. No director gates are invoked. The skill does not
write files (findings report only). Verdicts: SECURE, CONCERNS, or
VULNERABILITIES FOUND.
`/security-audit` audits the codebase for common game security failures across
six categories: save/serialization, network/multiplayer, input validation, data
exposure, cheat/anti-tamper vectors, and dependencies. It takes a scope
(`full`, `network`, `save`, `input`, `quick`; no argument runs `full`), reads
`engine.name`, `engine.language`, `platform.targets`, `platform.multiplayer` and
`platform.online` from `project.yaml`, and spawns the `security-engineer` agent
with a fixed brief template — only its slots filled: the scope, the engine, the
`platform.*` values, the resolved code root, `assets/data/` and config files,
the categories and the report path — whose own text tells the agent to rate
every finding from its severity table and that game type never lowers a
rating. Grep patterns are chosen per engine from the
Phase 3 table; a category the table marks NOT SOURCEABLE for the engine is
NOT ASSESSED, and an absent `platform.multiplayer` is never read as `false`.
A skipped category is named with its reason, and an unresolved code root makes
the code categories NOT ASSESSED. Findings are classified CRITICAL / HIGH /
MEDIUM / LOW, each with file, line, attack scenario and remediation. The
report's release recommendation, first match wins, is DO NOT SHIP, FIX BEFORE
SHIPPING, NOT ASSESSED (e.g. INSUFFICIENT IMPLEMENTATION) or CLEAR TO SHIP.
The skill presents the executive summary and CRITICAL/HIGH
findings in conversation, then asks "May I write the full security audit report
to `production/security/security-audit-[date]-[scope].md`?" At
`workflow: minimal` the Phase 7 message says the release gate does not read the
report. No director gates are invoked.
---
## Static Assertions (Structural)
Verified automatically by `/skill-test static` — no fixture needed.
Checked against the SKILL.md by `/skill-test spec` — no fixture needed.
- [ ] Has required frontmatter fields: `name`, `description`, `argument-hint`, `user-invocable`, `allowed-tools`
- [ ] Has ≥2 phase headings
- [ ] Contains verdict keywords: SECURE, CONCERNS, VULNERABILITIES FOUND
- [ ] Does NOT require "May I write" language (read-only; findings report only)
- [ ] Has a next-step handoff (what to do with findings)
- [ ] Contains verdict keywords: CLEAR TO SHIP, FIX BEFORE SHIPPING, DO NOT SHIP, NOT ASSESSED, and the severities CRITICAL / HIGH / MEDIUM / LOW
- [ ] Contains "May I write" language naming `production/security/security-audit-[date]-[scope].md`
- [ ] Has a next-step handoff (Phase 7: `/security-audit quick`, `/gate-check release`, `/launch-checklist`)
---
## Director Gate Checks
None. Security audit is a read-only advisory skill; no gates are invoked.
None. The skill spawns the `security-engineer` specialist to run the audit;
that is its core mechanism, not a director gate. No director gate is invoked in
any review mode.
---
## Test Cases
### Case 1: Happy Path — Save data encrypted, no hardcoded credentials
### Case 1: Happy Path — Single-player Godot project, no findings
**Fixture:**
- `src/core/save_system.gd` uses `Crypto` class to encrypt save data before writing
- No hardcoded API keys, passwords, or credentials in any `src/` file
- No version numbers or internal build IDs exposed in client-facing output
- `project.yaml`: `engine.name: godot`, `engine.language: gdscript`, `platform.multiplayer: false`, `platform.online: false`, `modes.rigor: standard` (so `workflow` resolves to `standard`); `modes.automation` unset (collaborative)
- `src/` holds the implemented game: 42 `.gd` files across core, gameplay, UI and save systems
- `src/core/save_system.gd` opens saves with `FileAccess`, verifies a checksum, and bounds-checks every loaded value before use
- No API keys, secrets, passwords or tokens anywhere in `src/` or `assets/`
- `addons/gdUnit4/` is the only third-party dependency
**Input:** `/security-audit`
**Expected behavior:**
1. Skill scans `src/` for security patterns: encryption usage, hardcoded credentials, exposed internals
2. All checks pass: save data encrypted, no credentials found, no exposed internals
3. Findings report shows all checks PASS
4. Verdict is SECURE
1. No argument → `full` scope
2. Skill reads engine, language and platform keys from `project.yaml`
3. Skill spawns `security-engineer` via `Agent` with the fixed brief template: the scope, Godot/GDScript, `platform.multiplayer: false`, `platform.online: false`, code root `src/`, `assets/data/` and config files, the categories to run, and the report path — nothing added outside the slots
4. Category 1 uses the Godot pattern list, including the Godot 4 name `FileAccess`; Category 2 is skipped because both platform keys are explicitly `false`, and the Executive Summary names the skip and its reason
5. Category 6 lists `addons/gdUnit4/` in the Dependency Inventory
6. Executive Summary shows zero findings at every severity over 42 source files; Release recommendation: **CLEAR TO SHIP**
7. Skill presents the summary, then asks "May I write the full security audit report to `production/security/security-audit-[date]-full.md`?"
8. After writing: "✅ No blocking security findings. Report written to `production/security/`. Include this path when running `/gate-check release`."
**Assertions:**
- [ ] Skill checks save data handling for encryption usage
- [ ] Skill scans for hardcoded credentials (API keys, passwords, tokens)
- [ ] Skill checks for version/build numbers exposed to players
- [ ] All checks shown in findings report
- [ ] Verdict is SECURE when all checks pass
- [ ] `security-engineer` is spawned with the fixed brief template — scope, engine/language, the `platform.*` values and a source manifest rooted at `src/` — and the brief adds no severity and no remark on the game's type
- [ ] Save handling is checked with Godot 4 names (`FileAccess`), not only the Godot 3 `File.open`
- [ ] Hardcoded-credential patterns (`api_key`, `secret`, `password`, `token`, `private_key`) are scanned
- [ ] The Executive Summary says Category 2 was skipped because `platform.multiplayer` and `platform.online` are `false` — the skip is not silent
- [ ] Release recommendation is CLEAR TO SHIP
- [ ] "May I write" precedes the report write
---
### Case 2: Vulnerabilities Found — Unencrypted save data and exposed version
### Case 1b: Single-player, One HIGH Finding — FIX BEFORE SHIPPING
**Fixture:**
- `src/core/save_system.gd` writes save data as plain JSON (no encryption)
- `src/ui/debug_overlay.gd` contains: `label.text = "Build: " + ProjectSettings.get("application/config/version")`
(exposes internal build version to player)
- Same as Case 1, except `src/core/save_system.gd` loads save values with no
checksum or bounds check, so an edited save file unlocks late-game progression
- No other findings
**Input:** `/security-audit`
**Expected behavior:**
1. Skill scans `src/` — finds unencrypted save write in `save_system.gd`
2. Skill finds exposed version string in `debug_overlay.gd`
3. Both findings are flagged as VULNERABILITIES
4. Verdict is VULNERABILITIES FOUND
5. Skill provides remediation recommendations for each vulnerability
1. The save finding is classified **HIGH** (save tampering that bypasses progression)
and is **not** raised to CRITICAL, because `platform.multiplayer` is `false`
2. Release recommendation: **FIX BEFORE SHIPPING** — an open HIGH, no CRITICAL
3. After the write: "⚠️ 1 HIGH finding(s) are open. The Polish → Release gate's
security audit item requires none — fix them and re-run `/security-audit quick`
before `/gate-check release`."
**Assertions:**
- [ ] Unencrypted save data is flagged as a vulnerability with file and approximate line
- [ ] Exposed version string is flagged as a vulnerability
- [ ] Remediation suggestion is given for each vulnerability
- [ ] Verdict is VULNERABILITIES FOUND when any vulnerability is detected
- [ ] No files are written or modified
- [ ] The brief sent to `security-engineer` is the fixed template: it does not call the save finding HIGH (or any severity), and does not say a single-player game lowers the bar — the agent rates it from its own table
- [ ] The finding is HIGH, not CRITICAL, in a single-player game
- [ ] Release recommendation is FIX BEFORE SHIPPING — never CLEAR TO SHIP with a HIGH open, and not DO NOT SHIP without a CRITICAL
- [ ] The ⚠️ HIGH message names the release gate's security audit item as the requirement, and names `/security-audit quick` and `/gate-check release`
- [ ] Variant — at `modes.rigor: minimal` the recommendation is still FIX BEFORE SHIPPING, and the message says the `minimal` release gate does not read this report instead of naming the gate item
---
### Case 3: Online Features Without Authentication — CONCERNS
### Case 2: Multiplayer Critical — Client-authoritative RPC and a hardcoded key
**Fixture:**
- `src/networking/lobby.gd` exists with functions: `join_lobby()`, `send_chat()`
- No authentication check is found before `send_chat()` — players can call it without being verified
- Game has online multiplayer features (inferred from file presence)
- `project.yaml`: `engine.name: godot`, `platform.multiplayer: true`, `platform.online: true`
- `src/net/combat_sync.gd` line 42: `@rpc("any_peer") func apply_damage(target_id, amount)` applies `amount` with no sender or range validation
- `src/online/leaderboard_client.gd` line 8: `const API_KEY = "sk_live_..."`
**Input:** `/security-audit`
**Input:** `/security-audit full`
**Expected behavior:**
1. Skill scans `src/networking/` — detects online feature code
2. Skill checks for authentication guard before network calls — finds none on `send_chat()`
3. Flags: "Online feature without authentication check — CONCERNS"
4. Verdict is CONCERNS (not VULNERABILITIES FOUND, as this is a missing control, not an exploit)
1. Category 2 runs (multiplayer is `true`) with the Godot network patterns (`MultiplayerPeer`, `rpc`, `@rpc`, …)
2. The unvalidated `apply_damage` RPC is a CRITICAL finding (client dictates outcomes, breaking multiplayer integrity)
3. The hardcoded key is a credential exposure (HIGH), treated as CRITICAL because the game is multiplayer
4. Each finding is written as `SEC-NNN` with Category, File and line, Description, Attack scenario, Remediation and Effort
5. Release recommendation is DO NOT SHIP — an open CRITICAL decides it
6. Skill presents the executive summary and CRITICAL/HIGH findings, then asks "May I write"
7. Phase 7: "⛔ CRITICAL security findings must be resolved before any public release. Do not proceed to `/launch-checklist` until these are addressed." and re-run `/security-audit quick` before `/gate-check release`
**Assertions:**
- [ ] Skill detects online features by scanning for networking source files
- [ ] Missing authentication checks before network operations are flagged
- [ ] Verdict is CONCERNS (advisory severity) for missing authentication guards
- [ ] Output recommends adding authentication before network calls
- [ ] The RPC finding cites `src/net/combat_sync.gd` with its line and an attack scenario
- [ ] The hardcoded key is raised to CRITICAL in the multiplayer context
- [ ] Every finding carries a specific remediation
- [ ] Release recommendation is DO NOT SHIP (open CRITICAL findings) — never FIX BEFORE SHIPPING or CLEAR TO SHIP
- [ ] The ⛔ message is shown and `/launch-checklist` is held back
---
### Case 4: Edge Case — No Source Files to Analyze
### Case 3: Multiplayer Scope Unset — Absent is not `false`
**Fixture:**
- `src/` directory does not exist or is completely empty
- `project.yaml`: `engine.name: godot`; no `platform` block at all
- `src/networking/lobby.gd` defines `@rpc` functions `join_lobby()` and `send_chat()`
- `modes.automation` unset (collaborative)
**Input:** `/security-audit`
**Expected behavior:**
1. Skill attempts to scan `src/` — no files found
2. Skill outputs an error: "No source files found in `src/` — nothing to audit"
3. No findings report is generated
4. No verdict is emitted
1. Skill finds `platform.multiplayer` and `platform.online` absent
2. Skill does not assume single-player and does not skip Category 2
3. Skill asks the user whether the game has multiplayer or online features
4. On "yes", Category 2 runs with the Godot network patterns and checks each `@rpc` call site for validation
**Assertions:**
- [ ] Skill does not crash when `src/` is empty or absent
- [ ] Output clearly states that no source files were found
- [ ] No verdict is emitted (there is nothing to assess)
- [ ] Skill suggests verifying the `src/` directory path
- [ ] Absent `platform.multiplayer` / `platform.online` are not treated as `false`
- [ ] Skill asks the user whether the game has multiplayer or online features
- [ ] Category 2 is not skipped on the absent keys
- [ ] Unvalidated `@rpc` entry points in `lobby.gd` are checked and any gap is reported with a remediation
- [ ] Variant — with `modes.automation: autonomous` the question cannot be asked: Category 2 still runs, is marked `NOT ASSESSED — multiplayer scope unconfirmed`, and the recommendation is never CLEAR TO SHIP
---
### Case 5: Gate Compliance — No gate; security-engineer invoked separately
### Case 4: No Implemented Surface — NOT ASSESSED — INSUFFICIENT IMPLEMENTATION
**Fixture:**
- Source files exist; 1 CONCERNS-level finding detected (debug logging enabled in release build)
- `review-mode.txt` contains `full`
- `project.yaml`: `engine.name: godot`, `platform.multiplayer: false`, `platform.online: false`
- `src/` exists but contains no source files; `assets/data/` is empty
**Input:** `/security-audit`
**Expected behavior:**
1. Skill scans source; finds debug logging active in release path
2. No director gate is invoked regardless of review mode
3. Verdict is CONCERNS
4. Output notes: "For formal security review, consider engaging a security-engineer agent"
5. Findings are presented as a read-only report; no files written
1. The code root resolves to `src/`, but there is nothing in it to audit
2. Every code-scanning category has zero files to scan, so each is NOT ASSESSED rather than clean
3. Release recommendation: **NOT ASSESSED — INSUFFICIENT IMPLEMENTATION**
4. The report names what was missing (no source under `src/`) and which skill produces it
5. CLEAR TO SHIP is not reported
**Assertions:**
- [ ] The empty code root leaves every code-scanning category NOT ASSESSED; none is reported clean or as zero findings
- [ ] Release recommendation is NOT ASSESSED — INSUFFICIENT IMPLEMENTATION
- [ ] Output names what was missing and which skill produces it
- [ ] CLEAR TO SHIP is not reachable from a scan with nothing to look at
- [ ] Variant — with no `engine.name`, `technical-preferences.md` at `[TO BE CONFIGURED]`, and both `src/` and `Source/` present, the code root is unresolved: no root is guessed, the code categories read `NOT ASSESSED — code root unresolved`, and the recommendation is never CLEAR TO SHIP
---
### Case 5: Unity Project in Full Review Mode — NOT SOURCEABLE category, no gates
**Fixture:**
- `project.yaml`: `engine.name: unity`, `engine.language: csharp`, `platform.multiplayer: true`, `platform.online: true`, `modes.review_mode: full`
- `Assets/Scripts/Save/SaveManager.cs` reads and writes save data
- `Assets/Scripts/Net/PlayerNet.cs` has `[ServerRpc]` methods that validate their inputs
- No findings in the categories that can run
**Input:** `/security-audit`
**Expected behavior:**
1. The manifest passed to `security-engineer` is rooted at `Assets/`, not `src/`
2. Category 1 is `NOT SOURCEABLE` for Unity → `NOT ASSESSED`; no Unity save-API names are written from memory
3. Categories 2 and 3 use the Unity names from the Phase 3 table (`ServerRpc`, `ClientRpc`, `NetworkVariable`, …; `InputSystem`, `PlayerInput`, …), not the Godot lists
4. The Executive Summary names Category 1 as NOT ASSESSED; Release recommendation: `NOT ASSESSED — Save / serialization has no sourced pattern set for Unity`
5. The Phase 7 NOT ASSESSED message names Category 1 as NOT SOURCEABLE on Unity and its fix — a save/serialization module in `docs/engine-reference/unity/modules/` — not `engine.name`, the code root or the multiplayer scope, which were all supplied
6. No director gate is invoked regardless of review mode
7. Skill asks "May I write" before writing the report
**Assertions:**
- [ ] The source manifest uses the Unity code root `Assets/`
- [ ] Category 1 is NOT ASSESSED for Unity and no save-API pattern list is invented
- [ ] Category 2 uses the Unity network names, not the Godot list
- [ ] CLEAR TO SHIP is unreachable; the recommendation names the unsourced category
- [ ] The NOT ASSESSED message points at a serialization module under `docs/engine-reference/unity/modules/` as the fix
- [ ] No director gate is invoked in any review mode
- [ ] Security-engineer consultation is suggested (not mandated)
- [ ] No files are written
- [ ] Verdict is CONCERNS for advisory-level security findings
---
### Case 5b: Ranking — An open HIGH outranks an unsourced category
**Fixture:**
- `project.yaml`: `engine.name: unity`, `engine.language: csharp`, `platform.multiplayer: false`, `platform.online: false`
- `Assets/Scripts/` holds the implemented game (35 `.cs` files), including `Assets/Scripts/Save/SaveManager.cs`
- `Assets/Scripts/Analytics/Telemetry.cs` line 12: `const string ApiKey = "sk_live_...";`
- No other findings
**Input:** `/security-audit`
**Expected behavior:**
1. Category 1 is `NOT SOURCEABLE` for Unity, so it is NOT ASSESSED; Category 2 is skipped (both platform keys `false`) and the skip is named
2. The hardcoded key is a credential exposure: HIGH, and not raised to CRITICAL because the game is single-player
3. Release recommendation: **FIX BEFORE SHIPPING** — the open HIGH is matched before NOT ASSESSED
4. The Executive Summary still names Category 1 as NOT ASSESSED
**Assertions:**
- [ ] Release recommendation is FIX BEFORE SHIPPING, not NOT ASSESSED — a known failure is not buried behind an unscanned category
- [ ] Category 1 is still reported NOT ASSESSED in the Executive Summary
- [ ] CLEAR TO SHIP is not reported
---
## Protocol Compliance
- [ ] Reads source files in `src/` before auditing
- [ ] Checks save data encryption, hardcoded credentials, exposed internals, auth guards
- [ ] Provides remediation recommendations for each finding
- [ ] Does not write any files (read-only skill)
- [ ] Reads `engine.name`, `engine.language` and the `platform.*` keys from `project.yaml` before auditing
- [ ] Spawns `security-engineer` via `Agent` with the fixed brief template — only its named slots filled (scope, engine/language, `platform.*` values, the resolved code root, `assets/data/` and config files, the categories, the report path) — and no sentence of the orchestrator's own: no pre-rated finding, no "single-player lowers the stakes"; an unresolved code root makes the code categories NOT ASSESSED, never clean
- [ ] Uses the Phase 3 pattern set for the resolved engine; a NOT SOURCEABLE category is NOT ASSESSED and makes CLEAR TO SHIP unreachable
- [ ] Never treats an absent `platform.multiplayer` / `platform.online` as `false`
- [ ] Names every skipped category and why in the Executive Summary
- [ ] Each finding carries severity, category, file and line, attack scenario and remediation
- [ ] Release recommendation, first match: DO NOT SHIP, FIX BEFORE SHIPPING, NOT ASSESSED, CLEAR TO SHIP
- [ ] Presents the executive summary and CRITICAL/HIGH findings before asking "May I write" for `production/security/security-audit-[date]-[scope].md`; writes only after approval
- [ ] No director gates are invoked
- [ ] Verdict is one of: SECURE, CONCERNS, VULNERABILITIES FOUND
- [ ] Ends with the Phase 7 gate-integration guidance
---
## Coverage Notes
- Anti-cheat analysis (client-side value validation, server authority) is not
explicitly tested here; it follows the CONCERNS or VULNERABILITIES pattern
depending on severity.
- Data privacy compliance (GDPR, COPPA) is out of scope for this spec; those
require legal review beyond code scanning.
- The `network`, `save`, `input` and `quick` scopes are not tested individually;
they restrict the categories run by the same rules.
- An absent `engine.name` with a resolvable tree (every grep category NOT
ASSESSED, with no fallback to the Godot lists) is not tested separately.
- Unreal follows the same engine table as Case 5 and is not tested separately.
CCGS Skill Testing Framework/skills/analysis/tech-debt.md
+178 -86
View File
@@ -2,24 +2,37 @@
## Skill Summary
`/tech-debt` tracks, categorizes, and prioritizes technical debt across the
codebase. It reads `docs/tech-debt-register.md` for the existing debt register
and scans source files in `src/` for inline `TODO` and `FIXME` comments. It
merges and sorts items by severity. No director gates are invoked. The skill
asks "May I write to `docs/tech-debt-register.md`?" before updating. Verdicts:
REGISTER UPDATED or NO NEW DEBT FOUND.
`/tech-debt` tracks technical debt through four subcommands: `scan`, `add`,
`prioritize` and `report`. With no subcommand it prints usage and stops with
Verdict FAIL. `scan` searches the code root (resolved per engine: `src/`,
`Assets/` or `Source/`) for `TODO`, `FIXME`, `HACK` and `@deprecated` markers,
duplicated blocks, files over 500 lines and functions over 50 lines, sorts the
findings into six debt categories, states how many files it covered, and asks
"May I write these findings to `docs/tech-debt-register.md`?" before appending
(never overwriting) entries — updating a matching `Open` row, and proposing an
`Open` row whose pattern is gone as `Resolved [date]`. New rows follow the
skill's Debt Register Format (next `TD-NNN`, ten columns, an `Added` date). A
scan over zero source files is `NOT ASSESSED — no source files to scan`, an
unresolved code root is `NOT ASSESSED — code root unresolved`, and both write
nothing. `add` collects an
entry through prompts and three `AskUserQuestion` choices, then asks "May I append
this entry to `docs/tech-debt-register.md`?". `prioritize` scores each register
item as `impact ÷ effort` from its Impact and Effort columns, re-sorts
and asks before writing it back. `report` is read-only. No director gates are
invoked. Verdicts: COMPLETE, BLOCKED (user declined a write), FAIL (missing
subcommand), NOT ASSESSED.
---
## Static Assertions (Structural)
Verified automatically by `/skill-test static` — no fixture needed.
Checked against the SKILL.md by `/skill-test spec` — no fixture needed.
- [ ] Has required frontmatter fields: `name`, `description`, `argument-hint`, `user-invocable`, `allowed-tools`
- [ ] Has ≥2 phase headings
- [ ] Contains verdict keywords: REGISTER UPDATED, NO NEW DEBT FOUND
- [ ] Contains "May I write" language (skill writes to debt register)
- [ ] Has a next-step handoff (what to do after register is updated)
- [ ] Contains verdict keywords: COMPLETE, BLOCKED, FAIL, NOT ASSESSED
- [ ] Contains "May I write" language (writes `docs/tech-debt-register.md`)
- [ ] Has a next-step handoff (Phase 3: `/sprint-plan`, `/tech-debt report`)
---
@@ -32,140 +45,219 @@ invoked.
## Test Cases
### Case 1: Happy Path — Inline TODOs plus existing register items merged
### Case 1: Happy Path — Scan appends new findings to an existing register
**Fixture:**
- `docs/tech-debt-register.md` exists with 2 items (LOW and MEDIUM severity)
- `project.yaml`: `engine.name: godot`; `modes.automation` unset (collaborative)
- `docs/tech-debt-register.md` is in the skill's Debt Register Format and has 2 `Open` rows:
- TD-001 — Code Quality, Files `src/core/save_system.gd`: a `# HACK` workaround for a save-path bug. That `HACK` comment is no longer in the file
- TD-002 — Code Quality, Files `src/gameplay/combat.gd`: the `# TODO` comments in that file
- `src/gameplay/combat.gd` has 2 `# TODO` comments and 1 `# FIXME` comment
- `src/ui/hud.gd` has 0 inline debt comments
- `src/ui/hud.gd` is 620 lines long; 12 source files in `src/` in total
**Input:** `/tech-debt`
**Input:** `/tech-debt scan`
**Expected behavior:**
1. Skill reads `docs/tech-debt-register.md` — finds 2 existing items
2. Skill scans `src/` — finds 3 inline comments (2 TODOs, 1 FIXME)
3. Skill checks whether inline comments already exist in the register (deduplication)
4. Skill presents combined list sorted by severity (FIXME before TODO by default)
5. Skill asks "May I write to `docs/tech-debt-register.md`?"
6. User approves; register updated; verdict REGISTER UPDATED
1. Skill resolves the code root to `src/` and scans it for debt indicators
2. Findings include the 2 TODOs, the FIXME (treated as a bug disguised as debt) and `hud.gd` as a file over 500 lines
3. Each finding is sorted into one of the six categories (Architecture, Code Quality, Test, Documentation, Dependency, Performance)
4. The output states the denominator — 12 source files scanned
5. Matching against the register: the 2 TODOs share TD-002's file and kind, so they update TD-002; TD-001's `HACK` is gone, so TD-001 is proposed as `Resolved [date]`; the FIXME and `hud.gd` become new rows TD-003 and TD-004
6. Skill presents the findings and the TD-001 resolution, then asks "May I write these findings to `docs/tech-debt-register.md`?" — one approval covers both
7. On approval, TD-001's Status becomes `Resolved YYYY-MM-DD` and the row stays; TD-002 is updated; TD-003 and TD-004 are appended with all ten columns, Status `Open`, `Added` set to today (`YYYY-MM-DD`) and Priority `—`
8. Verdict: **COMPLETE** — scan findings written to register
**Assertions:**
- [ ] Inline comments are found by scanning `src/` recursively
- [ ] Existing register items are not duplicated
- [ ] Combined list is sorted by severity
- [ ] "May I write" prompt appears before any write
- [ ] Verdict is REGISTER UPDATED
- [ ] The scan covers the resolved code root `src/` for TODO, FIXME, HACK, `@deprecated`, files >500 lines and functions >50 lines
- [ ] Each finding is assigned one of the six debt categories
- [ ] The number of files scanned is stated with the findings
- [ ] "May I write" names `docs/tech-debt-register.md` and comes after the findings are shown
- [ ] The two TODO findings update TD-002 instead of adding a duplicate row
- [ ] TD-001 is proposed as `Resolved [date]` inside the same approval, and is kept in the register rather than deleted
- [ ] New rows take the next IDs (TD-003, TD-004) and carry an `Added` date; the register is updated in place, never replaced
- [ ] Verdict is COMPLETE
---
### Case 2: Register Doesn't Exist — Offered to create it
### Case 2: No Subcommand — Usage and FAIL
**Fixture:**
- `docs/tech-debt-register.md` does NOT exist
- `src/` contains 4 inline TODO/FIXME comments
- `docs/tech-debt-register.md` exists
- `src/` contains source files with TODO comments
**Input:** `/tech-debt`
**Expected behavior:**
1. Skill attempts to read `docs/tech-debt-register.md` — not found
2. Skill informs user: "No tech-debt-register.md found"
3. Skill offers to create the register with the inline items it found
4. Skill asks "May I write to `docs/tech-debt-register.md`?" (create)
5. User approves; register created with 4 items; verdict REGISTER UPDATED
1. Phase 1 finds no subcommand
2. Skill outputs usage listing `scan`, `add`, `prioritize` and `report`
3. Skill stops — no scan runs and the register is not read or written
4. Verdict: **FAIL** — missing required subcommand
**Assertions:**
- [ ] Skill does not crash when register file is absent
- [ ] User is offered register creation (not silently skipping)
- [ ] "May I write" prompt reflects file creation (not update)
- [ ] Verdict is REGISTER UPDATED after creation
- [ ] Usage lists the four subcommands
- [ ] No scan is performed and no file is written
- [ ] Verdict is FAIL, stating the subcommand is missing
---
### Case 3: Resolved Item Detected — Marked resolved in register
### Case 3: Empty Code Root — NOT ASSESSED, not a clean scan
**Fixture:**
- `docs/tech-debt-register.md` has 3 items; one references `src/gameplay/legacy_input.gd`
- `src/gameplay/legacy_input.gd` has been deleted (refactored away)
- The referenced TODO comment no longer exists in source
- `project.yaml`: `engine.name: godot`
- `src/` exists but contains no source files
- `docs/tech-debt-register.md` exists with 3 entries
**Input:** `/tech-debt`
**Input:** `/tech-debt scan`
**Expected behavior:**
1. Skill reads register — finds 3 items
2. Skill scans `src/` — does not find the source location referenced by item 2
3. Skill flags item 2 as RESOLVED (source is gone)
4. Skill presents the resolved item to user for confirmation
5. On approval, register is updated with item 2 marked `Status: Resolved`
1. Skill resolves the code root to `src/` and counts the files the scan covered: zero
2. Skill reports **NOT ASSESSED — no source files to scan**, explaining that "no debt indicators found" would describe an empty search, not the codebase
3. Skill stops: no "May I write" prompt, no register write, no COMPLETE verdict
**Assertions:**
- [ ] Skill checks whether each register item's source reference still exists
- [ ] Missing source locations result in items being flagged as RESOLVED
- [ ] User confirms before resolved items are written
- [ ] RESOLVED items are kept in the register (not deleted) for audit history
- [ ] A zero-file scan is reported as NOT ASSESSED — no source files to scan
- [ ] Output does not claim "no debt found" or a clean codebase
- [ ] The register is not written and no write is offered
- [ ] No COMPLETE verdict is emitted
---
### Case 4: Edge Case — CRITICAL debt item surfaces prominently
### Case 3b: Code Root Unresolved — NOT ASSESSED, nothing scanned on a guess
**Fixture:**
- `src/core/network_sync.gd` has a comment: `# FIXME(CRITICAL): race condition in sync buffer — can corrupt save data`
- `docs/tech-debt-register.md` exists with 5 lower-severity items
- `project.yaml` has no `engine.name`; `.claude/docs/technical-preferences.md` reads `[TO BE CONFIGURED]`
- Both `src/` and `Source/` exist and hold source files, so the tree does not decide the root
- `docs/tech-debt-register.md` exists with 3 entries
**Input:** `/tech-debt`
**Input:** `/tech-debt scan`
**Expected behavior:**
1. Skill scans source and finds the CRITICAL-tagged FIXME
2. Skill presents the CRITICAL item at the top of the output — before the full table
3. Skill asks user to acknowledge the critical item before proceeding
4. After acknowledgment, skill presents full debt table and asks to write
5. Register is updated with CRITICAL item at top; verdict REGISTER UPDATED
1. The code root cannot be resolved per `.claude/docs/code-root-resolution.md`
2. Skill reports `NOT ASSESSED — code root unresolved` and stops
3. Neither directory is scanned on a guess; no "May I write" prompt, no register write, no COMPLETE verdict
**Assertions:**
- [ ] CRITICAL items appear at the top of the output, not buried in the table
- [ ] Skill surfaces CRITICAL items before asking to write
- [ ] User acknowledgment of the CRITICAL item is requested
- [ ] CRITICAL severity is preserved in the written register entry
- [ ] Verdict is `NOT ASSESSED — code root unresolved`
- [ ] No debt findings are reported from a guessed root
- [ ] The register is not written and no write is offered
---
### Case 5: Gate Compliance — No gate; register updated only with approval
### Case 4: Add Mode — User declines the write
**Fixture:**
- Inline scan finds 2 new TODOs; register has 3 existing items
- `review-mode.txt` contains `full`
- `docs/tech-debt-register.md` exists with 4 entries
- `modes.automation` unset (collaborative)
**Input:** `/tech-debt`
**Input:** `/tech-debt add`
**Expected behavior:**
1. Skill scans source and reads register; compiles combined debt list
2. No director gate is invoked regardless of review mode
3. Skill presents sorted debt table to user
4. Skill asks "May I write to `docs/tech-debt-register.md`?"
5. User approves; register updated; verdict REGISTER UPDATED
1. Skill asks in plain text for the description and affected files
2. Skill uses `AskUserQuestion` for the category — six options, Architecture through Performance
3. Skill uses `AskUserQuestion` for the fix effort — S, M, L, XL
4. Skill uses `AskUserQuestion` for the impact if left unfixed — Low, Med, High, Critical
5. Skill presents the complete new entry
6. Skill asks "May I append this entry to `docs/tech-debt-register.md`?"
7. User declines — nothing is written
8. Verdict: **BLOCKED** — user declined write
**Assertions:**
- [ ] Category is collected with `AskUserQuestion` offering the six debt categories
- [ ] Effort is collected with `AskUserQuestion` offering S / M / L / XL
- [ ] Impact is collected with `AskUserQuestion` offering Low / Med / High / Critical, so `prioritize` can score the row
- [ ] The full entry is shown before the append prompt
- [ ] On decline the register is unchanged and the verdict is BLOCKED
---
### Case 4a: Add Mode — Matches an Existing Open Entry
**Fixture:**
- `docs/tech-debt-register.md` exists with an `Open` entry for the same file
and the same category as the one the user is about to add
**Input:** `/tech-debt add`
**Expected behavior:**
1. Steps 1–5 of Case 4 run as usual
2. **Match before appending:** before the append ask, the skill shows the
existing `Open` entry (same file and category) and asks whether to update
it instead of adding a duplicate
3. On "update", the matched entry is updated in place — no duplicate row is added
4. On "add anyway", a new entry is appended alongside the match
**Assertions:**
- [ ] A new entry with the same file and category as an existing `Open` entry is never appended without first showing the match and asking
- [ ] "Update" edits the matched row rather than adding a new one
---
### Case 4b: Report Mode Without a Register — NOT ASSESSED
**Fixture:**
- `docs/tech-debt-register.md` does not exist
**Input:** `/tech-debt report`
**Expected behavior:**
1. Skill looks for the register and does not find it
2. Verdict: **NOT ASSESSED** — no register to report on; run `/tech-debt scan` first
3. No trend or totals are printed and nothing is written
**Assertions:**
- [ ] Verdict is NOT ASSESSED, not a report of zero items or a "stable" trend
- [ ] The output names `/tech-debt scan` as the way to create the register
- [ ] No file is written
---
### Case 5: Prioritize in Full Review Mode — Scored, re-sorted, approved
**Fixture:**
- `project.yaml`: `modes.review_mode: full`
- `docs/tech-debt-register.md` has 4 entries: TD-001 High/S, TD-002 Low/L, TD-003
Critical/M, TD-004 Med/— (Effort not yet estimated)
**Input:** `/tech-debt prioritize`
**Expected behavior:**
1. Skill reads `docs/tech-debt-register.md`
2. Each item is scored `impact ÷ effort` (Impact Low 1 … Critical 4; Effort S 1 … XL 4): TD-001 3.0, TD-003 2.0, TD-002 0.3; TD-004 is listed after them as "not scored — Effort not judged"
3. The register is re-sorted by score and the skill recommends which items belong in the next sprint
4. No director gate is invoked regardless of review mode
5. Skill presents the re-prioritized register, then asks "May I write the re-prioritized register back to `docs/tech-debt-register.md`?"
6. On approval the file is written
7. Verdict: **COMPLETE** — register re-prioritized and saved
**Assertions:**
- [ ] Items are scored `impact ÷ effort` from their own columns and re-sorted: TD-001, TD-003, TD-002
- [ ] TD-004 gets no invented score; it is listed as not scored, naming the missing Effort
- [ ] A next-sprint recommendation is given
- [ ] No director gate is invoked in any review mode
- [ ] Debt table is presented before any write prompt
- [ ] "May I write" prompt appears before file update
- [ ] Write only occurs with explicit user approval
- [ ] The write happens only after the "May I write" prompt is approved
- [ ] Verdict is COMPLETE
---
## Protocol Compliance
- [ ] Reads `docs/tech-debt-register.md` and scans `src/` before compiling
- [ ] Deduplicates inline comments against existing register items
- [ ] Sorts combined list by severity
- [ ] Always asks "May I write" before updating register
- [ ] Requires a subcommand; without one prints usage and stops with FAIL
- [ ] `scan` resolves the code root per engine (`src/`, `Assets/`, `Source/`) and states the number of files covered
- [ ] A zero-file or unresolved-root scan is NOT ASSESSED and writes nothing
- [ ] Every register write is preceded by a "May I write" / "May I append" prompt naming `docs/tech-debt-register.md`, after the content is shown
- [ ] `scan` appends and never overwrites existing entries; a finding matching an `Open` row updates it, and an `Open` row whose pattern is gone is proposed `Resolved [date]`, never deleted
- [ ] New rows follow the Debt Register Format: the next `TD-NNN` ID, all ten columns, an `Added` date in `YYYY-MM-DD`
- [ ] A declined write ends with BLOCKED
- [ ] No director gates are invoked
- [ ] Verdict is REGISTER UPDATED or NO NEW DEBT FOUND
- [ ] Ends with the Phase 3 next steps (`/sprint-plan`, `/tech-debt report`)
---
## Coverage Notes
- The case where `src/` is empty or absent is not tested; behavior follows
the NO NEW DEBT FOUND path for the inline scan, but register items would
still be read and presented.
- TODO comments without severity tags are treated as LOW severity by default;
this classification detail is an implementation concern, not tested here.
- `report` mode with a register (read-only summary by category and effort,
added and resolved counts from each row's `Added` and `Resolved` dates,
flagging items older than 3 sprints, Verdict COMPLETE — debt report generated)
is not tested here.
- Unity and Unreal scans differ only in the resolved root (`Assets/`,
`Source/`) and are not tested separately.
CCGS Skill Testing Framework/skills/analysis/test-evidence-review.md
+213 -94
View File
@@ -2,174 +2,293 @@
## Skill Summary
`/test-evidence-review` performs a quality review of test files in `tests/`,
checking test naming conventions, determinism, isolation, and absence of
hardcoded magic numbers — all against the project's test standards defined in
`coding-standards.md`. Findings may be flagged for qa-lead review. No director
gates are invoked. The skill does not write without user approval. Verdicts:
PASS, WARNINGS, or FAIL.
`/test-evidence-review` reviews the **quality** of the test files and evidence
documents behind stories — `/smoke-check` covers existence and passing. It
takes a scope: a single story path, `sprint` (the most recent plan in
`production/sprints/`), or a system name (`production/epics/[system]/story-*.md`);
with no argument it asks which. If the resolved scope holds zero stories it stops
with `NOT ASSESSED — no stories in scope` and routes to the skill that creates
them. For each story it collects Story Type, evidence path and acceptance
criteria with targeted greps, locates the evidence (`tests/unit/[system]/`,
`tests/integration/[system]/`, `production/qa/evidence/`,
`production/qa/smoke-*.md`), and reviews automated tests for assertion count,
edge cases, naming and formula traceability, and manual evidence for criterion
linkage, sign-offs, retained screenshots and freshness. It resolves `qa.level`
and `testing.strict` the way `/story-done` does: at `qa.level: minimal` a Logic,
Integration or Config/Data story with no test evidence is
`WAIVED (qa.level: minimal)`, not MISSING, and a gap in a test that does exist is
ADVISORY; Visual/Feel and UI evidence is never waived; otherwise each gap takes
its story type's gate level from `testing.strict.<type>` (unset → a
plain-boolean `testing.strict` legacy value applies to every type; else the
`coding-standards.md` default). Each story gets ADEQUATE, INCOMPLETE, MISSING,
NOT ASSESSED or WAIVED; the overall verdict is the worst present, with NOT
ASSESSED outranking ADEQUATE only, and WAIVED below ADEQUATE. The report is presented in
conversation, and the skill asks "May I write this test evidence review to
`production/qa/evidence-review-[date].md`?" as an optional step. It never
modifies test files or evidence docs and invokes no director gates. The run ends
COMPLETE, or CONCERNS when BLOCKING items were found.
---
## Static Assertions (Structural)
Verified automatically by `/skill-test static` — no fixture needed.
Checked against the SKILL.md by `/skill-test spec` — no fixture needed.
- [ ] Has required frontmatter fields: `name`, `description`, `argument-hint`, `user-invocable`, `allowed-tools`
- [ ] Has ≥2 phase headings
- [ ] Contains verdict keywords: PASS, WARNINGS, FAIL
- [ ] Does NOT require "May I write" language (read-only; write is optional flagging report)
- [ ] Has a next-step handoff (what to do after findings are reviewed)
- [ ] Contains verdict keywords: ADEQUATE, INCOMPLETE, MISSING, NOT ASSESSED, and the run-level COMPLETE / CONCERNS
- [ ] Contains "May I write" language for the optional `production/qa/evidence-review-[date].md`
- [ ] Has a next-step handoff (Section 7: BLOCKING items before `/story-done`, `/test-helpers`, sign-off routing)
- [ ] The `resolve_config` injection names `qa.level` and `testing.strict`, and the skill states the `WAIVED (qa.level: minimal)` result and the per-type `testing.strict.<type>` gate level
---
## Director Gate Checks
None. Test evidence review is an advisory quality skill; QL-TEST-COVERAGE gate
is a separate skill invocation and is NOT triggered here.
None. Test evidence review is an advisory quality skill; no gates are invoked.
---
## Test Cases
### Case 1: Happy Path — Tests follow all standards
### Case 1: Happy Path — Single Logic story with a strong test
**Fixture:**
- `tests/unit/combat/health_system_take_damage_test.gd` exists with:
- Naming: `test_health_system_take_damage_reduces_health()` (follows `test_[system]_[scenario]_[expected]`)
- Arrange/Act/Assert structure present
- No `sleep()`, `await` with time values, or random seeds
- No calls to external APIs or file I/O
- No inline magic numbers (uses constants from `tests/unit/combat/fixtures/`)
- `project.yaml`: `engine.name: godot`, so the unit-test root is `tests/unit/`; `modes.automation` unset (collaborative)
- `production/epics/combat/story-001-damage-calc.md` (slug `damage-calc`): `## Test Evidence` gives Story Type Logic and the path `tests/unit/combat/damage-calc_test.gd`; `## Acceptance Criteria` includes "damage is never below 1" and "critical hits multiply damage by 2.0"
- `tests/unit/combat/damage-calc_test.gd` opens with `# Story: production/epics/combat/story-001-damage-calc.md`, has 4 test functions with 3+ assertions each, named like `test_damage_calc_zero_armor_returns_base_damage` and `test_damage_calc_max_crit_doubles_damage`, and names the GDD's `damage_formula` in a comment
**Input:** `/test-evidence-review tests/unit/combat/`
**Input:** `/test-evidence-review production/epics/combat/story-001-damage-calc.md`
**Expected behavior:**
1. Skill reads test standards from `coding-standards.md`
2. Skill reads the test file; checks all 5 standards
3. All checks pass: naming, structure, determinism, isolation, no hardcoded data
4. Verdict is PASS
1. Single-story mode: skill reads the story's Test Evidence and Acceptance Criteria
2. Section 3 finds the test file under `tests/unit/combat/`
3. Section 4: assertion coverage is normal (3+ per function); "zero" and "max" cases cover the numeric criteria; names follow `test_[scenario]_[expected]`; the formula is referenced
4. Story verdict: **ADEQUATE**; overall verdict ADEQUATE; 0 BLOCKING items
5. Report is presented in conversation, then the skill asks "May I write this test evidence review to `production/qa/evidence-review-[date].md`?" as optional
6. Verdict: **COMPLETE**
**Assertions:**
- [ ] Each of the 5 test standards is checked and reported
- [ ] All checks show PASS when standards are met
- [ ] Verdict is PASS
- [ ] No files are written
- [ ] The test file is located under `tests/unit/combat/` for the Logic story
- [ ] Assertion coverage, edge cases, naming and formula traceability are each reported
- [ ] Story verdict is ADEQUATE
- [ ] The report is shown before the optional "May I write" prompt
- [ ] Run ends with COMPLETE
---
### Case 2: Fail — Timing dependency detected
### Case 2: Vacuous and Thin Tests — INCOMPLETE with a BLOCKING item
**Fixture:**
- `tests/unit/ui/hud_update_test.gd` contains:
```gdscript
await get_tree().create_timer(1.0).timeout
assert_eq(label.text, "Ready")
```
- Real-time wait of 1 second used instead of mock or signal-based assertion
- `project.yaml`: `engine.name: godot`, `modes.rigor: standard` (so `qa.level: standard` — tests are required); no `testing.strict` block
- `production/epics/combat/story-002-crit-roll.md` (slug `crit-roll`): Story Type Logic
- `tests/unit/combat/crit-roll_test.gd` has `func test_1()` with no assertions and `func test_crit_roll_applies_multiplier()` with one assertion
**Input:** `/test-evidence-review tests/unit/ui/hud_update_test.gd`
**Input:** `/test-evidence-review production/epics/combat/story-002-crit-roll.md`
**Expected behavior:**
1. Skill reads the test file
2. Skill detects real-time wait (`create_timer(1.0)`) — non-deterministic timing dependency
3. Skill flags this as a FAIL-level finding
4. Verdict is FAIL
5. Skill recommends replacing the timer with a signal-based assertion or mock
1. Skill finds the test file and counts assertions per function
2. `test_1` has 0 assertions → flagged BLOCKING (passes vacuously, proves nothing); `test_1` is also flagged as a generic name
3. The one-assertion function is noted as potentially thin
4. Story verdict: **INCOMPLETE**
5. Follow-ups: "These must be resolved before `/story-done` can mark the story Complete…" and "Consider running `/test-helpers [system]`…"
6. Run ends with **CONCERNS** because BLOCKING items were found
**Assertions:**
- [ ] Real-time wait usage is detected as a non-deterministic timing dependency
- [ ] Finding is classified as FAIL severity (blocking — violates determinism standard)
- [ ] Verdict is FAIL
- [ ] Remediation suggestion references signal-based or mock-based approach
- [ ] Skill does not edit the test file
- [ ] A test function with zero assertions is flagged BLOCKING
- [ ] `test_1` is flagged as a naming issue
- [ ] Story verdict is INCOMPLETE, not ADEQUATE
- [ ] Output states BLOCKING items must be resolved before `/story-done` and suggests `/test-helpers`
- [ ] Run ends with CONCERNS
- [ ] The test file is not modified
- [ ] Variant — with `testing.strict.logic: false` in `project.yaml`, the zero-assertion finding is ADVISORY, the story is still INCOMPLETE, and the run ends COMPLETE
---
### Case 3: Fail — Test calls external API directly
### Case 3: Empty Sprint Scope at Minimal Workflow — NOT ASSESSED, not a clean report
**Fixture:**
- `tests/unit/networking/auth_test.gd` contains:
```gdscript
var result = HTTPRequest.new().request("https://api.example.com/auth")
```
- Direct HTTP call to external API without a mock
- `project.yaml`: `modes.rigor: minimal` (the resolved block shows `workflow: minimal`)
- `production/sprints/` does not exist
**Input:** `/test-evidence-review tests/unit/networking/auth_test.gd`
**Input:** `/test-evidence-review sprint`
**Expected behavior:**
1. Skill reads the test file
2. Skill detects direct external API call (HTTPRequest to live URL)
3. Skill flags this as a FAIL-level finding — violates isolation standard
4. Verdict is FAIL
5. Skill recommends injecting a mock HTTP client
1. Sprint mode finds no sprint plan, so the scope holds zero stories
2. Skill stops and reports `NOT ASSESSED — no stories in scope`, naming the sprint scope and the empty `production/sprints/` path
3. Because the workflow is minimal (no sprints), it routes to `/test-evidence-review [epic-slug]` instead of `/sprint-plan new`
4. Skill does not continue to Section 3; no Summary table and no "BLOCKING items: 0" line are rendered
**Assertions:**
- [ ] Direct external API call is detected and flagged
- [ ] Finding is classified as FAIL severity (violates isolation standard)
- [ ] Verdict is FAIL
- [ ] Remediation references dependency injection with a mock HTTP client
- [ ] Skill does not modify the test file
- [ ] Verdict is NOT ASSESSED — no stories in scope
- [ ] Output names which scope was searched and which path was empty
- [ ] At minimal workflow the route is `/test-evidence-review [epic-slug]`, not `/sprint-plan new`
- [ ] No empty report reading "BLOCKING items: 0 / ADVISORY items: 0" is produced
---
### Case 4: Edge Case — No Test Files Found
### Case 4: Story Type Unknown — Per-story NOT ASSESSED lifts the overall verdict
**Fixture:**
- User calls `/test-evidence-review tests/unit/audio/`
- `tests/unit/audio/` directory does not exist
- `project.yaml`: `engine.name: godot`
- `production/epics/combat/` contains `story-001-damage-calc.md` (Logic, test as in Case 1) and `story-003-hit-feedback.md`, whose `## Test Evidence` section states no Story Type and no evidence path
- Full-reading `story-003-hit-feedback.md` does not reveal a type either
**Input:** `/test-evidence-review tests/unit/audio/`
**Input:** `/test-evidence-review combat`
**Expected behavior:**
1. Skill attempts to read files in `tests/unit/audio/` — not found
2. Skill outputs: "No test files found at `tests/unit/audio/` — run `/test-setup` to scaffold test directories"
3. No verdict is emitted
1. System mode globs `production/epics/combat/story-*.md` and finds both stories
2. `story-001-damage-calc` is ADEQUATE
3. `story-003-hit-feedback` is **NOT ASSESSED**, with the reason stated: the story type cannot be determined, so the required evidence is unknown
4. It is not reported as MISSING
5. Overall verdict: **NOT ASSESSED** — it outranks ADEQUATE
**Assertions:**
- [ ] Skill does not crash when path does not exist
- [ ] Output names the attempted path in the message
- [ ] Output recommends `/test-setup` for scaffolding
- [ ] No verdict is emitted when there is nothing to review
- [ ] The untyped story is NOT ASSESSED, not MISSING
- [ ] The NOT ASSESSED reason (type cannot be determined) is stated for that story
- [ ] Overall verdict is NOT ASSESSED, not ADEQUATE
- [ ] The typed story is still reviewed and reported ADEQUATE
---
### Case 5: Gate Compliance — No gate; QL-TEST-COVERAGE is a separate skill
### Case 5: Visual/Feel Evidence Without a Retained Screenshot, Full Review Mode — No gates
**Fixture:**
- Test file has 1 WARNINGS-level finding (magic number in a non-boundary test)
- `review-mode.txt` contains `full`
- `project.yaml`: `engine.name: godot`, `modes.review_mode: full`
- `production/epics/combat/story-004-hit-flash.md` (slug `hit-flash`): Story Type Visual/Feel
- `production/qa/evidence/hit-flash-evidence.md` references every acceptance criterion, has every sign-off filled, and is dated after the sprint start
- No `*.png`, `*.jpg` or `*.gif` for this story exists in `production/qa/evidence/`
**Input:** `/test-evidence-review tests/unit/combat/`
**Input:** `/test-evidence-review production/epics/combat/story-004-hit-flash.md`
**Expected behavior:**
1. Skill reviews tests; finds 1 WARNINGS-level finding
2. No director gate is invoked (QL-TEST-COVERAGE is invoked separately, not here)
3. Verdict is WARNINGS
4. Output notes: "For full test coverage gate, run `/gate-check` which invokes QL-TEST-COVERAGE"
5. Skill offers optional report write; asks "May I write" if user opts in
1. Section 3 finds `production/qa/evidence/hit-flash-evidence.md`
2. Section 5: criterion linkage and sign-offs are complete, but no retained image exists for the story
3. The doc describes a visual check without evidence → story verdict **INCOMPLETE**, flagged BLOCKING
4. No director gate is invoked regardless of review mode
5. The evidence document is not edited
6. Run ends with **CONCERNS**
**Assertions:**
- [ ] A Visual/Feel evidence doc with no retained image is INCOMPLETE, even with complete sign-offs
- [ ] The missing screenshot is listed as a BLOCKING issue
- [ ] No director gate is invoked in any review mode
- [ ] Output distinguishes this skill from the QL-TEST-COVERAGE gate invocation
- [ ] Optional report requires "May I write" before writing
- [ ] Verdict is WARNINGS for advisory-level test quality issues
- [ ] The evidence document is not modified
---
### Case 6: UI Story Closed by Its Screenshots — No Sign-Off Required
**Fixture:**
- `project.yaml`: `engine.name: godot`
- `production/epics/ui/story-007-inventory-screen.md` (slug `inventory-screen`): Story Type UI; its
`## Test Evidence` names `production/qa/evidence/inventory-screen-populated.png` and
`production/qa/evidence/inventory-screen-empty.png` — one per screen state it touches
- Both images exist and are dated after the sprint start
- No `inventory-screen-evidence.md`, no sign-off record and no walkthrough log exist
**Input:** `/test-evidence-review production/epics/ui/story-007-inventory-screen.md`
**Expected behavior:**
1. Section 3 uses the two paths the story states before searching
2. Section 5: a UI story is closed by the retained screenshot of each screen it touched
3. Story verdict **ADEQUATE**; run ends with **COMPLETE**
**Assertions:**
- [ ] The story's stated evidence paths are checked first, and both images are found
- [ ] The missing sign-off does not make the story INCOMPLETE — a UI story needs none
- [ ] No walkthrough log is demanded
- [ ] Story verdict is ADEQUATE
---
### Case 7: Unity Logic Story — Evidence at the Stated Engine Path
**Fixture:**
- `project.yaml`: `engine.name: unity`
- `production/epics/inventory/story-003-stack-merge.md`: Story Type Logic; its
`## Test Evidence` names `Assets/Tests/EditMode/InventoryTests.cs`
- That file exists and holds one `[Test]` per acceptance criterion (4 in all), each with 3 or more `Assert.That` calls, named in the `[Scenario]_[Expected]` form (`MergeStacks_FullStack_SpillsToNewSlot`), with the empty-stack and max-stack cases covered
- The inventory GDD has no Formulas section, so formula traceability does not apply
- Nothing named `stack-merge_test.*` exists anywhere, and `tests/unit/` does not exist
**Input:** `/test-evidence-review production/epics/inventory/story-003-stack-merge.md`
**Expected behavior:**
1. Section 3 finds the test at the stated path under `Assets/Tests/EditMode/`
2. Section 4 reviews its assertions; story verdict **ADEQUATE**
**Assertions:**
- [ ] The test is found at the stated Unity path — the story is not reported MISSING
- [ ] The review does not require a `[story-slug]_test.*` name or a `tests/unit/` location
---
### Case 8: No Test for a Logic Story — MISSING, and it outranks NOT ASSESSED
**Fixture:**
- `project.yaml`: `engine.name: godot`, `modes.rigor: standard` (so `qa.level: standard` — a Logic story's test is required)
- `production/epics/combat/` contains `story-003-hit-feedback.md` (no Story Type, as in Case 4) and `story-005-knockback.md`: Story Type Logic, no evidence path stated
- No test exists for it: `tests/unit/combat/` does not exist, and no other test file names the slug `knockback` or references the story
**Input:** `/test-evidence-review combat`
**Expected behavior:**
1. `story-005-knockback` is a Logic story, so a unit test is required; the search of the unit-test root finds none
2. `story-005-knockback` is **MISSING** — the review ran and found nothing — and the gap is a BLOCKING item
3. `story-003-hit-feedback` is **NOT ASSESSED** (its type cannot be determined), not MISSING
4. Overall verdict: **MISSING** — a known failure outranks the story that could not be assessed
5. Run ends with **CONCERNS** because a BLOCKING item was found
**Assertions:**
- [ ] The Logic story with no test file is MISSING, not NOT ASSESSED
- [ ] The untyped story is NOT ASSESSED, not MISSING — the two are kept apart
- [ ] Overall verdict is MISSING, not NOT ASSESSED
- [ ] The missing test is listed as BLOCKING and the run ends with CONCERNS
---
### Case 9: `qa.level: minimal` — a Logic Story With No Test Is WAIVED; a UI Screenshot Is Not
**Fixture:**
- `project.yaml`: `engine.name: godot`, no `modes` block (so `qa.level: minimal` — tests are waived)
- `production/epics/combat/` contains `story-005-knockback.md` (Logic, no test anywhere, as in Case 8) and `story-008-pause-menu.md` (UI, no retained image for it in `production/qa/evidence/`)
**Input:** `/test-evidence-review combat`
**Expected behavior:**
1. `story-005-knockback` is **WAIVED (qa.level: minimal)** — not MISSING — and raises no BLOCKING item
2. `story-008-pause-menu` is **MISSING** and BLOCKING — the retained screenshot a UI story needs is never waived
3. Overall verdict: **MISSING**; the report's header shows `qa.level: minimal`
4. Run ends with **CONCERNS** because of the UI story alone
5. Variant — without the UI story, the overall verdict is **WAIVED**, there are 0 BLOCKING items, and the run ends **COMPLETE**
**Assertions:**
- [ ] The Logic story with no test is WAIVED (qa.level: minimal), not MISSING, and is not listed as BLOCKING
- [ ] The UI story with no screenshot is still MISSING and BLOCKING at `qa.level: minimal`
- [ ] The waived story is named as waived in the report, not dropped from it
- [ ] Variant: an all-waived scope reads WAIVED, with no "must be resolved before `/story-done`" prompt
---
## Protocol Compliance
- [ ] Reads `coding-standards.md` test standards before reviewing test files
- [ ] Checks naming, Arrange/Act/Assert structure, determinism, isolation, no hardcoded data
- [ ] Does not edit any test files (read-only skill)
- [ ] Collects Story Type, evidence path and acceptance criteria with targeted `## Test Evidence` / `## Acceptance Criteria` greps, full-reading a story only when its Test Evidence section is missing or ambiguous
- [ ] An empty scope stops with `NOT ASSESSED — no stories in scope`, naming the scope and routing onward
- [ ] Each story gets ADEQUATE, INCOMPLETE, MISSING or NOT ASSESSED, and a NOT ASSESSED story states its reason
- [ ] Overall verdict is the worst story verdict; NOT ASSESSED outranks ADEQUATE but not INCOMPLETE or MISSING; WAIVED ranks below ADEQUATE
- [ ] At `qa.level: minimal` a Logic, Integration or Config/Data story with no test evidence is WAIVED, never MISSING or BLOCKING; Visual/Feel and UI evidence is never waived
- [ ] BLOCKING vs ADVISORY per story type comes from `testing.strict.<type>` in the resolved block, falling back to a legacy plain-boolean `testing.strict` value applied to every type, then to the `coding-standards.md` default
- [ ] The report is presented before the optional "May I write" for `production/qa/evidence-review-[date].md`
- [ ] Does not modify test files or evidence documents
- [ ] No director gates are invoked
- [ ] Verdict is one of: PASS, WARNINGS, FAIL
- [ ] Ends with COMPLETE, or CONCERNS when BLOCKING items were found — except an empty scope, which stops at `NOT ASSESSED — no stories in scope` before any report (Case 3)
---
## Coverage Notes
- Batch review of all test files in `tests/` is not explicitly tested; behavior
is assumed to apply the same checks file by file and aggregate the verdict.
- The QL-TEST-COVERAGE director gate (which checks test coverage percentage) is
a separate concern and is intentionally NOT invoked by this skill.
- Integration stories (playtest records in `production/session-logs/`) and
Config/Data stories (`production/qa/smoke-*.md`) are not tested here.
- The no-argument path (the skill asks which scope to review) is not tested.
- The POTENTIALLY STALE freshness flag is not tested separately.
CCGS Skill Testing Framework/skills/analysis/test-flakiness.md
+131 -115
View File
@@ -2,176 +2,192 @@
## Skill Summary
`/test-flakiness` detects non-deterministic tests by analyzing test history logs
(if available) or scanning test source code for common flakiness patterns (random
numbers without seeds, real-time waits, external I/O). No director gates are
invoked. The skill does not write without user approval. Verdicts: NO FLAKINESS,
SUSPECT TESTS FOUND, or CONFIRMED FLAKY.
`/test-flakiness` finds flaky tests from the results of multiple CI or local
test runs. It takes a log path, `scan` (all available result files — JUnit XML
from gdUnit4 in `reports/` or saved into `test-results/`, NUnit XML from Unity in
`test-results/`, Unreal automation logs in `Saved/Logs/`), or `registry` (remediation guidance for tests already in the
quarantine section of `tests/regression-suite.md`). With no result data it lists
three ways to get some and stops to ask. It builds a per-test history across
runs, classifies each test that both passed and failed by fail rate — >25% High
(quarantine immediately), 5–25% Moderate (fix directly, do not quarantine yet),
1–5% Low (monitor) — and names a likely cause and fix direction from its cause
table, using Grep on the test file. With fewer than 3 runs, every finding is
"suspected", not "confirmed", whatever its fail rate, and nothing is quarantined;
the summary table's Confidence column says which. It asks separately "May I update the quarantine
section of `tests/regression-suite.md`…?" and "May I write a full flakiness
report to `production/qa/flakiness-report-[date].md`?", appends quarantine
entries without removing existing ones, and never deletes test files. No
director gates are invoked. Verdicts: COMPLETE (report written) or BLOCKED
(user declined write).
---
## Static Assertions (Structural)
Verified automatically by `/skill-test static` — no fixture needed.
Checked against the SKILL.md by `/skill-test spec` — no fixture needed.
- [ ] Has required frontmatter fields: `name`, `description`, `argument-hint`, `user-invocable`, `allowed-tools`
- [ ] Has ≥2 phase headings
- [ ] Contains verdict keywords: NO FLAKINESS, SUSPECT TESTS FOUND, CONFIRMED FLAKY
- [ ] Does NOT require "May I write" language (read-only; optional report requires approval)
- [ ] Has a next-step handoff (what to do after flakiness findings)
- [ ] Contains verdict keywords: COMPLETE, BLOCKED, and the flakiness tiers High / Moderate / Low
- [ ] Contains "May I" write language for `tests/regression-suite.md` and `production/qa/flakiness-report-[date].md`
- [ ] Has a next-step handoff (Section 7: add skip annotations, schedule fixes before the release gate)
---
## Director Gate Checks
None. Flakiness detection is an advisory quality skill for the QA lead; no gates
are invoked.
None. Flakiness detection is an advisory quality skill; no gates are invoked.
---
## Test Cases
### Case 1: Happy Path — Clean test history, no flakiness
### Case 1: Happy Path — High flakiness from JUnit history, quarantined on approval
**Fixture:**
- `production/qa/test-history/` contains logs for 10 test runs
- All tests pass consistently across all 10 runs (100% pass rate per test)
- No test has a failure pattern
- `project.yaml`: `engine.name: godot`; `modes.automation` unset (collaborative)
- `test-results/` holds 6 gdUnit4 JUnit XML files from 6 runs of the same commit
- `test_loot_drop_rolls_rare_item` fails in 2 of the 6 runs (33%); the other 40 tests pass in all 6
- `tests/unit/loot/loot_drop_test.gd` calls `randf()` with no seed
- `tests/regression-suite.md` has a Quarantined Tests table with 1 existing entry
**Input:** `/test-flakiness`
**Input:** `/test-flakiness scan`
**Expected behavior:**
1. Skill reads test history logs from `production/qa/test-history/`
2. Skill computes per-test pass rate across 10 runs
3. All tests pass all 10 runs — no inconsistency detected
4. Verdict is NO FLAKINESS
1. Skill finds the XML files in `test-results/` and parses `<testcase name=` and `<failure` / `<error` entries
2. Skill builds `test_id → [run1 … run6]` and finds one test with both outcomes
3. 33% is above 25% over 6 runs → **High** flakiness, Confidence confirmed → quarantine immediately
4. Grep of `loot_drop_test.gd` finds `randf` → likely cause Random seed; fix direction: pass an explicit seed
5. The quarantine recommendation names gdUnit4's GDScript skip parameters (`_do_skip := true`, `_skip_reason := "flaky: ..."`) as the way to skip it
6. Summary shows Runs analysed 6, Tests tracked 41, the flaky test with a 33% fail rate, and 40 tests with consistent results
7. Skill asks "May I update the quarantine section of `tests/regression-suite.md` with the flaky tests found?" and, separately, "May I write a full flakiness report to `production/qa/flakiness-report-[date].md`?"
8. On approval, the new entry is appended with `Edit`; the existing entry stays
9. Verdict: **COMPLETE** — flakiness report written
**Assertions:**
- [ ] Skill reads test history logs when available
- [ ] Per-test pass rate is computed across all available runs
- [ ] Verdict is NO FLAKINESS when all tests pass consistently
- [ ] JUnit XML results in `test-results/` are parsed per test across all 6 runs
- [ ] The flaky test is named with its fail rate, Confidence confirmed, and classified High → quarantine
- [ ] The skip mechanism named is gdUnit4's `_do_skip` / `_skip_reason` parameter pair, not an invented flag
- [ ] The likely cause is Random seed, with the explicit-seed fix direction
- [ ] The two writes are asked about separately, each before it happens
- [ ] The quarantine entry is appended and the existing entry is not removed
- [ ] Verdict is COMPLETE
---
### Case 2: Moderate Flakiness — Fix directly, do not quarantine
**Fixture:**
- `project.yaml`: `engine.name: godot`
- `test-results/` holds 20 JUnit XML files from 20 runs of the same commit
- `test_physics_bounce_height_matches_spec` fails in 3 of 20 runs (15%)
- The test asserts `bounce_height == 0.5`
**Input:** `/test-flakiness scan`
**Expected behavior:**
1. Skill computes a 15% fail rate → **Moderate** flakiness
2. Grep of the test file finds a float equality comparison → likely cause Floating point
3. Recommendation: "This test is intermittently unreliable… Do not quarantine yet — fix the test directly", with the fix direction of an epsilon comparison (`is_equal_approx`)
**Assertions:**
- [ ] A 15% fail rate is classified Moderate
- [ ] The recommendation is to fix directly, not to quarantine
- [ ] The cause is Floating point and the fix names an epsilon comparison such as `is_equal_approx`
---
### Case 3: No Result Data — List options and stop
**Fixture:**
- No `test-results/` directory, no `.github/` directory, no `Saved/Logs/`
- No log path is given
**Input:** `/test-flakiness scan`
**Expected behavior:**
1. Skill finds no CI or local result data
2. Skill prints "No CI log data found…" with the three options: run the suite at least 3 times and collect logs; save a CI log to `test-results/`; run `/test-flakiness registry`
3. Skill stops and asks the user which option to pursue
4. No flakiness table is produced and nothing is written
**Assertions:**
- [ ] Output states no CI log data was found
- [ ] All three options are listed, including `/test-flakiness registry`
- [ ] Skill stops to ask instead of reporting a clean result
- [ ] No files are written
---
### Case 2: Suspect Tests Found — Test fails intermittently in history
### Case 4: Too Few Runs — Suspected, not confirmed
**Fixture:**
- `production/qa/test-history/` contains logs for 10 test runs
- `test_combat_damage_applies_crit_multiplier` passes 7 times, fails 3 times
- Failure messages differ (sometimes timeout, sometimes wrong value)
- `project.yaml`: `engine.name: godot`
- `test-results/` holds only 2 JUnit XML files, from 2 runs of the same commit
- `test_save_roundtrip_preserves_inventory` passes in one run and fails in the other
**Input:** `/test-flakiness`
**Input:** `/test-flakiness scan`
**Expected behavior:**
1. Skill reads test history logs — computes pass rates
2. `test_combat_damage_applies_crit_multiplier` has 70% pass rate (threshold: 95%)
3. Skill flags it as SUSPECT with pass rate (7/10) and failure pattern noted
4. Verdict is SUSPECT TESTS FOUND
5. Skill recommends investigating the test for timing or state dependencies
1. Skill parses both files and finds the test with both outcomes — a 50% fail rate
2. With fewer than 3 runs, the finding is flagged "suspected", not "confirmed": the summary table's Confidence reads suspected, and the High tier's quarantine-immediately does not apply
3. The test is not quarantined; the recommendation is to collect more runs before acting
4. Skill asks whether more run data is available
5. The Data Limitations section notes that fewer than 5 runs were available, so confidence is low
**Assertions:**
- [ ] Tests below the pass-rate threshold are flagged by name
- [ ] Pass rate (fraction and percentage) is shown for each suspect test
- [ ] Failure pattern (e.g., inconsistent error messages) is noted if detectable
- [ ] Verdict is SUSPECT TESTS FOUND
- [ ] Skill recommends investigation steps
- [ ] The finding is labelled suspected, not confirmed, in the summary table's Confidence column
- [ ] The 50% fail rate does not trigger quarantine — no skip is recommended, and any regression-suite note marks the test suspected
- [ ] Skill asks whether more run data is available
- [ ] Data Limitations states that fewer than 5 runs were analysed
---
### Case 3: Source Pattern — Random number used without seed
### Case 5: Registry Mode in Full Review Mode — Guidance for known quarantined tests, no gates
**Fixture:**
- No test history logs exist
- `tests/unit/loot/loot_drop_test.gd` contains:
```gdscript
var roll = randf() # unseeded random — non-deterministic
assert_gt(roll, 0.5, "Loot should drop above 50%")
```
- `project.yaml`: `modes.review_mode: full`
- `tests/regression-suite.md` Quarantined Tests table lists `test_ai_path_recalc_avoids_blocked_tile` (reason: timing) and `test_scene_load_spawns_player` (reason: scene not ready)
**Input:** `/test-flakiness`
**Input:** `/test-flakiness registry`
**Expected behavior:**
1. Skill finds no test history logs
2. Skill falls back to source code analysis
3. Skill detects `randf()` call without a preceding `seed()` call
4. Skill flags the test as FLAKINESS RISK (source pattern, not confirmed)
5. Verdict is SUSPECT TESTS FOUND (pattern detected, not confirmed by history)
6. Skill recommends seeding random before the call or mocking the random function
**Assertions:**
- [ ] Source code analysis is used as fallback when no history logs exist
- [ ] Unseeded random number usage is detected as a flakiness risk
- [ ] Verdict is SUSPECT TESTS FOUND (not CONFIRMED FLAKY — no history to confirm)
- [ ] Remediation recommends seeding or mocking
---
### Case 4: No Test History — Source-only analysis with common patterns
**Fixture:**
- `production/qa/test-history/` does not exist
- `tests/` contains 15 test files
- Scan finds 2 tests using `OS.get_ticks_msec()` for timing assertions
- No other flakiness patterns found
**Input:** `/test-flakiness`
**Expected behavior:**
1. Skill checks for test history — not found
2. Skill notes: "No test history available — analyzing source code for flakiness patterns only"
3. Skill scans all test files for known patterns: unseeded random, real-time waits, system clock usage
4. Finds 2 tests using `OS.get_ticks_msec()` — flags as FLAKINESS RISK
5. Verdict is SUSPECT TESTS FOUND
**Assertions:**
- [ ] Skill notes clearly that source-only analysis is being performed (no history)
- [ ] Common flakiness patterns are scanned: random, time-based assertions, external I/O
- [ ] `OS.get_ticks_msec()` usage for assertions is flagged as a flakiness risk
- [ ] Verdict is SUSPECT TESTS FOUND when source patterns are found
---
### Case 5: Gate Compliance — No gate; flakiness report is advisory
**Fixture:**
- Test history shows 1 CONFIRMED FLAKY test (fails 6 out of 10 runs)
- `review-mode.txt` contains `full`
**Input:** `/test-flakiness`
**Expected behavior:**
1. Skill analyzes test history; identifies 1 confirmed flaky test
2. No director gate is invoked regardless of review mode
3. Verdict is CONFIRMED FLAKY
4. Skill presents findings and offers optional written report
5. If user opts in: "May I write to `production/qa/flakiness-report-[date].md`?"
1. Skill reads the quarantine section of `tests/regression-suite.md`
2. For each quarantined test it gives remediation guidance from the cause table: Timing / async → explicit await or synchronisation instead of time-based delays; Scene/prefab load race → await one frame after instantiation (`await get_tree().process_frame`)
3. No director gate is invoked regardless of review mode
4. Existing quarantine entries are not removed and no test file is deleted or edited by the skill
**Assertions:**
- [ ] The quarantine section of `tests/regression-suite.md` is the input
- [ ] Each quarantined test gets a fix direction matching its cause
- [ ] No director gate is invoked in any review mode
- [ ] CONFIRMED FLAKY verdict requires history-based evidence (not just source patterns)
- [ ] Optional report requires "May I write" before writing
- [ ] Flakiness report is advisory for qa-lead; skill does not auto-disable tests
- [ ] No quarantine entry is removed and no test file is deleted
---
## Protocol Compliance
- [ ] Reads test history logs when available; falls back to source analysis when not
- [ ] Notes clearly which analysis mode is being used (history vs. source-only)
- [ ] Flakiness threshold (e.g., 95% pass rate) is used for SUSPECT classification
- [ ] CONFIRMED FLAKY requires history evidence; SUSPECT covers source patterns only
- [ ] Does not disable or modify any test files
- [ ] Locates result data (`test-results/` XML, `Saved/Logs/`, or a given log path) before analysis; with none, lists the three options and stops to ask
- [ ] Classifies tests with both outcomes by fail rate: >25% High (quarantine), 5–25% Moderate (fix, do not quarantine), 1–5% Low (monitor)
- [ ] Names a likely cause and fix direction from the cause table, using Grep on the test file
- [ ] Counts a test as flaky only when it both passed and failed across runs with no code change between them
- [ ] With fewer than 3 runs, flags findings as suspected, not confirmed, in the summary's Confidence column, and quarantines nothing
- [ ] Asks separately before updating `tests/regression-suite.md` and before writing `production/qa/flakiness-report-[date].md`
- [ ] Appends quarantine entries, never removes existing ones, and never deletes test files
- [ ] No director gates are invoked
- [ ] Verdict is one of: NO FLAKINESS, SUSPECT TESTS FOUND, CONFIRMED FLAKY
- [ ] Ends with COMPLETE on write or BLOCKED on decline; with no result data it lists the three options and stops to ask instead (Case 3), never reporting a clean result
---
## Coverage Notes
- The pass-rate threshold for SUSPECT classification (95% suggested above) is an
implementation detail; the tests verify that intermittent failures are flagged,
not the exact threshold value.
- Tests that fail due to environment issues (missing assets, wrong platform) are
not flakiness — the skill distinguishes environment failures from non-determinism
in the test itself; this distinction is not explicitly tested here.
- The no-argument path (run `scan` when CI logs are accessible, else
`registry`) is not tested here.
- Unity NUnit XML follows the same flow as Case 1, read from `<test-case`
elements and their `result` attribute; not tested separately.
- Unreal differs and is not tested here: its results are plain-text
`Result={Success}` / `Result={Fail}` lines in `Saved/Logs/` (each test prints
`Test Completed. Result={<status>}`), and its skip
mechanism is NOT SOURCEABLE, so a quarantined Unreal test is logged and the
user is asked how their CI excludes it. The same applies to gdUnit4 C# tests.
- The Low tier (1–5%) needs 20 or more runs to reach and is not tested.