Claude-Code-Game-Studios/CCGS Skill Testing Framework/skills/utility/smoke-check.md

# Skill Test Spec: /smoke-check

## Skill Summary

`/smoke-check` is the gate between implementation and QA hand-off. Phase 1 runs
`project-coherence.sh`, verifies that **game test files** exist under the
engine's test root or `tests/smoke/` (not merely that `tests/` exists), and
reads the engine from `project.yaml` (falling back to `technical-preferences.md`),
then resolves the editor executable — the one `commands.test` names, else
`engine.path`, else the name on `PATH` — before it calls the runner unavailable.
Phase 2 runs `commands.test` (or the per-engine default) under a timeout and
reads the result by engine — gdUnit4 exit codes for Godot; on Unity, Play Mode
tests under `Assets/Tests/PlayMode/` get a second run with their own results
file. At `qa.level: minimal` with no game tests, the automated row is WAIVED
and Phase 2 runs a build check instead — `commands.smoke`, else
`commands.build`, when set. Phase 3 scans test coverage against the QA plan or
sprint stories; Phase 4 batch-verifies manual smoke checks with
`AskUserQuestion`, and Batch 1 first asks whether the build was launched this
session — "not launched" makes the manual checks NOT ASSESSED. The report is written
to `production/qa/smoke-[date].md` after explicit approval.

Verdicts, first matching rule wins: FAIL (a test failure, a suite that did not
complete — timeout, compile/parse/build error, no results file — or a Batch
1/Batch 2 check FAILED, or a failed build check), NOT ASSESSED (suite unconfirmed NOT RUN, a runner that
could not run or executed zero tests, unconfirmed `PlayMode: NOT RUN`, a check
that could not execute — including a build not launched this session — an
`UNKNOWN` coverage row, Batch 3 skipped — or no game tests at `qa.level`
`standard`/`full`; at `minimal` the automated row is WAIVED and the batches carry
the verdict), PASS WITH WARNINGS (MISSING test evidence, a runner warning such
as gdUnit4 101's leaked orphans, or a NOT RUN suite the developer confirmed),
PASS. NOT ASSESSED ranks above both pass values and below FAIL.
Whether a FAIL blocks hand-off depends on `qa.level` (advisory at `minimal`)
and `testing.strict.config` (unset → blocking).

No director gates apply. The skill does NOT invoke any director agents.

---

## Static Assertions (Structural)

Checked against the SKILL.md by `/skill-test spec` — no fixture needed.

- [ ] Has required frontmatter fields: `name`, `description`, `argument-hint`, `user-invocable`, `allowed-tools`
- [ ] Has ≥2 phase headings
- [ ] Contains verdict keywords: PASS, PASS WITH WARNINGS, NOT ASSESSED, FAIL
- [ ] Contains "May I write" collaborative protocol language before writing the report
- [ ] Has a next-step handoff (re-run `/smoke-check` after a FAIL; QA hand-off to qa-tester on PASS)

---

## Director Gate Checks

None. `/smoke-check` is a pre-QA utility skill. No director gates apply.

---

## Test Cases

### Case 1: Happy Path — Automated tests pass, manual items confirmed, PASS

**Fixture:**
- `project.yaml` has `engine.name: Godot` and `commands.test` set to the gdUnit4 command
- `addons/gdUnit4/bin/GdUnitCmdTool.gd` exists; game tests exist under
  `tests/unit/` and `tests/integration/`
- `production/qa/qa-plan-sprint-005.md` exists
- The runner exits 0 with 12 tests, 12 passing
- The developer says the build was launched this session and selects no FAILED item in Batch 1, Batch 2 or Batch 3
- Every sprint story is COVERED or EXPECTED (no MISSING, no UNKNOWN)

**Input:** `/smoke-check`

**Expected behavior:**
1. Phase 1 reports environment: Godot, tests found, QA plan path
2. Runs `godot --headless --path . --import`, then `commands.test`
   (`godot --headless -s -d --remote-debug tcp://127.0.0.1:0 res://addons/gdUnit4/bin/GdUnitCmdTool.gd -a res://tests --ignoreHeadlessMode`)
   under `timeout 900`; the report says `commands.test` was the command run
3. Exit 0 with tests found → automated tests PASS (12/12)
4. Coverage scan: all stories COVERED or EXPECTED
5. `AskUserQuestion` for Batch 1, Batch 2 and Batch 3 — nothing selected
6. Report names its checklist source (the QA plan's Smoke Test Scope)
7. Asks "May I write this smoke check report to `production/qa/smoke-[date].md`?"
8. Writes after approval; delivers "Smoke check passed cleanly. The build is ready for manual QA."

**Assertions:**
- [ ] Automated test runner is invoked via Bash, after the Godot import
- [ ] `AskUserQuestion` is used for manual smoke check batches
- [ ] "May I write" is asked before writing the report file
- [ ] Report is written to `production/qa/smoke-[date].md`
- [ ] Verdict is PASS

---

### Case 2: Failure Path — Automated test fails, blocking FAIL

**Fixture:**
- As Case 1, but `project.yaml` sets `modes.rigor: standard` (so `qa.level`
  resolves to `standard`) and leaves `testing.strict` unset
- The runner exits 100: 10 tests run, 8 passing, 2 failing —
  `test_health_clamp_at_zero`, `test_damage_calculation_negative`

**Input:** `/smoke-check`

**Expected behavior:**
1. Skill runs automated tests via Bash; exit 100 → FAIL
2. Records both failing test names
3. Proceeds through the manual smoke check batches
4. Report shows automated tests as FAIL with the failing test names listed
5. Asks to write the report; writes after approval
6. Gate is blocking (`qa.level: standard`, `testing.strict.config` unset → blocking)
7. Delivers: "The smoke check failed. Do not hand off to QA until these failures
   are resolved: …" listing each failure, then "Fix the failures and run
   `/smoke-check` again to re-gate before QA hand-off."

**Assertions:**
- [ ] Failing test names are listed in the report
- [ ] Verdict is FAIL
- [ ] The blocking message directs the developer to fix failures before QA hand-off
- [ ] `/smoke-check` re-run is suggested after fixing

---

### Case 3: Manual Confirmation — MISSING coverage, PASS WITH WARNINGS

**Fixture:**
- As Case 1: runner exits 0 with 8/8 passing
- One Logic story has no matching test file (MISSING coverage); no UNKNOWN rows
- The developer selects no FAILED item in Batch 1, Batch 2 or Batch 3

**Input:** `/smoke-check`

**Expected behavior:**
1. Automated tests PASS
2. Coverage scan finds 1 MISSING entry for a Logic story
3. `AskUserQuestion` is used for Batches 1–3 — nothing selected
4. Report shows automated tests PASS, manual checks PASS, and the MISSING entry
   under "Missing Test Evidence" with its expected test path
5. Verdict is PASS WITH WARNINGS; the message says the build is ready for manual
   QA and lists the MISSING entry as an advisory item to resolve before
   `/story-done` on the affected story
6. Asks to write the report; writes after approval

**Assertions:**
- [ ] `AskUserQuestion` is used for manual smoke check batches (not inline text prompts)
- [ ] MISSING test coverage entry appears in the report
- [ ] Verdict is PASS WITH WARNINGS (not PASS, not FAIL)
- [ ] Advisory note says the MISSING entry must be resolved before `/story-done`
- [ ] Report file is written to `production/qa/smoke-[date].md`

---

### Case 4: Scaffold Without Game Tests — NOT ASSESSED, then stop

**Fixture:**
- `project.yaml` sets `modes.rigor: standard` (so `qa.level` resolves to `standard`)
- Engine is Godot; `tests/unit/` and `tests/integration/` exist but hold only
  the `.gdignore_placeholder` files `/test-setup` creates
- No `tests/smoke/` directory

**Input:** `/smoke-check`

**Expected behavior:**
1. Phase 1 counts game test files, not directories — none are found
2. Skill delivers: "Smoke check: **NOT ASSESSED — no game tests found** under
   [the test root] or `tests/smoke/`. Run `/test-setup` to scaffold the testing
   infrastructure, or point me at where tests live."
3. Skill stops — no automated run, no manual batches, no report written

**Assertions:**
- [ ] The presence of `tests/` and its subdirectories is not treated as tests existing
- [ ] Verdict is NOT ASSESSED — the skill does not stop without a verdict
- [ ] `/test-setup` is suggested as the remediation step
- [ ] No further phases run and no report file is written

---

### Case 4b: No Game Tests at `qa.level: minimal` — WAIVED, the build carries the verdict

**Fixture:**
- No `modes.rigor` set (the default, `minimal`, so `qa.level` resolves to `minimal`)
- Engine is Godot; no game test files under `tests/unit/`, `tests/integration/` or `tests/smoke/`
- `commands.smoke` is `godot --headless --quit-after 5`; it exits 0 with no `SCRIPT ERROR` line
- The user says the build was launched this session and confirms every Batch 1, Batch 2 and Batch 3 check

**Input:** `/smoke-check`

**Expected behavior:**
1. Phase 1 finds no game test files and, at `qa.level: minimal`, records the automated row as `WAIVED — qa.level: minimal, no game tests`
2. Phase 2's test run and Phase 3 are skipped, each with a one-line note saying why
3. Phase 2's build check runs `commands.smoke` (after the Godot import) under a timeout: `Build check: PASS`
4. Phase 4 runs Batches 1–3; every check passes
5. Verdict: PASS — the report's Automated Tests status reads WAIVED

**Assertions:**
- [ ] The missing tests are WAIVED, not NOT ASSESSED, because `qa.level` is `minimal`
- [ ] The skipped test run and Phase 3 are announced, not silent
- [ ] The build check runs `commands.smoke` and its result is in the report
- [ ] Batch 1 and Batch 2 still run — PASS is never given without them
- [ ] Verdict is PASS and the report's Automated Tests row reads WAIVED
- [ ] A Batch 1 or Batch 2 check that could not be executed (no build) makes the verdict NOT ASSESSED, as at any level
- [ ] Variant — the user answers that the build was not launched this session: Batches 2 and 3 are not asked (the skip is said), the Batch 1 checks read `NOT RUN — build not launched this session`, and the verdict is NOT ASSESSED — never PASS from an empty failure list
- [ ] Variant — `commands.smoke` prints `SCRIPT ERROR` (Godot's boot check exits 0 on it): the build check is FAIL and so is the verdict

---

### Case 5: Director Gate Check — No gate; smoke-check is a QA pre-check utility

**Fixture:**
- Valid test setup, automated tests pass, manual smoke checks confirmed

**Input:** `/smoke-check`

**Expected behavior:**
1. Skill runs all phases and produces a verdict
2. No director agents are spawned at any point
3. No gate IDs (CD-*, TD-*, AD-*, PR-*) appear in output
4. No `/gate-check` is invoked

**Assertions:**
- [ ] No director gate is invoked
- [ ] No gate skip messages appear
- [ ] Verdict is PASS, PASS WITH WARNINGS, NOT ASSESSED, or FAIL — no gate verdict involved

---

### Case 6: gdUnit4 Lookalikes — exit 0 over zero tests, and exit 101

**Fixture:**
- As Case 1, with every story COVERED or EXPECTED and nothing selected in
  Batch 1, Batch 2 or Batch 3
- Run A: the suites under `tests/unit/` declare no test functions; the runner
  prints `No test cases found` and exits 0
- Run B: the runner exits 101 — 12 tests, 12 passing, 2 orphan nodes leaked

**Input:** `/smoke-check`

**Expected behavior:**
1. Run A: exit 0 is not read as a pass — the automated tests are
   `NOT ASSESSED — no tests found`, and the verdict is NOT ASSESSED even though
   every manual check passed
2. Run B: 101 is PASS WITH WARNINGS, naming the orphan count; with no MISSING
   entry, the verdict is PASS WITH WARNINGS and the advisory list names the
   orphans

**Assertions:**
- [ ] Run A's verdict is NOT ASSESSED — not PASS
- [ ] Run B's Status line is PASS WITH WARNINGS with the orphan count
- [ ] Run B's verdict is PASS WITH WARNINGS — not PASS, and not FAIL

---

### Case 7: Timeout — the suite never completed, FAIL

**Fixture:**
- As Case 2 (`qa.level: standard`, `testing.strict` unset)
- The runner hangs; `timeout 900` ends it with exit 124 and no summary printed

**Input:** `/smoke-check`

**Expected behavior:**
1. Exit 124 is a gate FAILURE, never a pass — the run did not complete and
   nothing was verified
2. The Status line reads FAIL, naming the timeout
3. The verdict is FAIL and the gate is blocking

**Assertions:**
- [ ] A timeout is FAIL — not NOT ASSESSED and not NOT RUN
- [ ] The report names the timeout as the reason
- [ ] The blocking message is delivered

---

### Case 8: Batch 3 Skipped — NOT ASSESSED, not PASS

**Fixture:**
- As Case 1: runner exits 0 with 12/12 passing, no MISSING or UNKNOWN rows
- In Batch 3 the developer selects only "Performance not checked this session"

**Input:** `/smoke-check`

**Expected behavior:**
1. Batches 1 and 2 pass
2. The skipped Batch 3 item is recorded as not checked — it is not a FAIL
3. The verdict is NOT ASSESSED, and the Phase 6 message names the performance
   check that did not run

**Assertions:**
- [ ] Verdict is NOT ASSESSED — not PASS, not FAIL
- [ ] The unchecked performance item is named in the report

---

### Case 9: Unity — Play Mode is a second run

**Fixture:**
- `engine.name: Unity`; `com.unity.test-framework` is in `Packages/manifest.json`
- `commands.test` is the Edit Mode command `/setup-engine` writes
- `Assets/Tests/EditMode/` and `Assets/Tests/PlayMode/` both hold test files
- Edit Mode exits 0 with 8/8 passing; Play Mode exits 2 with 1 of 3 failing

**Input:** `/smoke-check`

**Expected behavior:**
1. The Edit Mode run deletes `test-results/editmode.xml` first, then runs
2. Because `commands.test` does not pass `-testPlatform PlayMode` and
   `Assets/Tests/PlayMode/` holds tests, the skill runs Play Mode too, deleting
   and then writing `test-results/playmode.xml`
3. The report has one Status line per platform: `EditMode: PASS (8/8)` and
   `PlayMode: FAIL (1 failure)`
4. The verdict is FAIL

**Assertions:**
- [ ] Play Mode is run as a second run with its own results file
- [ ] Each platform is reported on its own line
- [ ] The Play Mode failure makes the verdict FAIL — the Edit Mode pass does not cover it

---

## Protocol Compliance

- [ ] Uses `AskUserQuestion` for all manual smoke check batches (Batch 1, Batch 2, Batch 3), and Batch 1 asks whether the build was launched this session; "not launched" yields NOT ASSESSED
- [ ] Resolves the editor executable from `commands.test`, then `engine.path`, then `PATH` before reporting the runner unavailable; a bare `godot` that does not resolve is never read as "no engine installed"
- [ ] Never edits `project.yaml`: a `commands.test` missing flags or using a relative path is adjusted for the run and reported to the user
- [ ] Runs automated tests via Bash before asking any manual questions
- [ ] Asks "May I write" before creating the report file — never writes without approval
- [ ] Verdict vocabulary is PASS / PASS WITH WARNINGS / NOT ASSESSED / FAIL — no other verdicts
- [ ] FAIL is triggered by automated test failures, a suite that did not complete (timeout, compile/parse/build error, no results file), or Batch 1/Batch 2 FAIL responses
- [ ] PASS WITH WARNINGS is triggered by MISSING test coverage, a runner warning on a passing suite (gdUnit4 101), or a confirmed NOT RUN — with no critical failures
- [ ] Unconfirmed NOT RUN (runner not available) yields NOT ASSESSED — never FAIL, never a pass verdict
- [ ] A run over zero tests, a runner that could not run, and an unconfirmed `PlayMode: NOT RUN` yield NOT ASSESSED
- [ ] Does not invoke director gates at any point

---

## Coverage Notes

- The `quick` argument (skips the Phase 3 coverage scan and Batch 3) is not
  separately fixture-tested.
- The `--platform` argument adds platform-specific `AskUserQuestion` batches and
  a per-platform verdict table; not separately tested here.
- The runner-not-available path (NOT RUN, then NOT ASSESSED until the developer
  confirms a result from their IDE or CI) is covered by the protocol compliance
  assertions above.
- At `qa.level: minimal` a FAIL is advisory and never blocks hand-off; Case 2
  pins `standard` to test the blocking path. Unity's `testcasecount="0"` and
  Unreal's `No automation tests matched` (→ NOT ASSESSED) follow Case 6 Run A;
  not separately fixture-tested.
- The Linux and macOS Unreal commands (picked by `uname -s`) are not
  fixture-tested. On macOS, Unreal tests are NOT ASSESSED until `commands.test`
  is set — asserted in the skill, not given a case here.
- A Play Mode run that cannot be made (`PlayMode: NOT RUN`, unconfirmed →
  NOT ASSESSED) is covered by protocol compliance, not by its own case.