mirror of
https://github.com/Donchitos/Claude-Code-Game-Studios.git
synced 2026-10-16 10:17:06 +00:00
Release 1.1.2 — the default path works end to end
Fix release for 1.1.1: the minimal path end to end, engine commands that run on Godot, Unity and Unreal, hook warnings you can see, and verdicts that say when they could not check. Details in CHANGELOG.md; upgrade steps in UPGRADING.md (v1.1.1 → v1.1.2). Fixes #67 Fixes #131 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WuJn2FdynEybfL5KC697nJ
This commit is contained in:
Donchitos
co-authored by
Claude Opus 5.5
Claude Opus 5.5
parent
7ed2c3e9c4
commit
b21fa0f7f2
@@ -2,16 +2,35 @@
|
||||
|
||||
## Skill Summary
|
||||
|
||||
`/smoke-check` is the gate between implementation and QA hand-off. It detects the
|
||||
test environment, runs the automated test suite (via Bash), scans test coverage
|
||||
against sprint stories, and uses `AskUserQuestion` to batch-verify manual smoke
|
||||
checks with the developer. It writes a report to `production/qa/smoke-[date].md`
|
||||
after explicit user approval.
|
||||
`/smoke-check` is the gate between implementation and QA hand-off. Phase 1 runs
|
||||
`project-coherence.sh`, verifies that **game test files** exist under the
|
||||
engine's test root or `tests/smoke/` (not merely that `tests/` exists), and
|
||||
reads the engine from `project.yaml` (falling back to `technical-preferences.md`),
|
||||
then resolves the editor executable — the one `commands.test` names, else
|
||||
`engine.path`, else the name on `PATH` — before it calls the runner unavailable.
|
||||
Phase 2 runs `commands.test` (or the per-engine default) under a timeout and
|
||||
reads the result by engine — gdUnit4 exit codes for Godot; on Unity, Play Mode
|
||||
tests under `Assets/Tests/PlayMode/` get a second run with their own results
|
||||
file. At `qa.level: minimal` with no game tests, the automated row is WAIVED
|
||||
and Phase 2 runs a build check instead — `commands.smoke`, else
|
||||
`commands.build`, when set. Phase 3 scans test coverage against the QA plan or
|
||||
sprint stories; Phase 4 batch-verifies manual smoke checks with
|
||||
`AskUserQuestion`, and Batch 1 first asks whether the build was launched this
|
||||
session — "not launched" makes the manual checks NOT ASSESSED. The report is written
|
||||
to `production/qa/smoke-[date].md` after explicit approval.
|
||||
|
||||
Verdicts: PASS (tests pass, all smoke checks pass, no missing test evidence),
|
||||
PASS WITH WARNINGS (tests pass or NOT RUN, all critical checks pass, but advisory
|
||||
gaps exist such as missing test coverage), or FAIL (any automated test failure or
|
||||
any Batch 1/Batch 2 smoke check returns FAIL).
|
||||
Verdicts, first matching rule wins: FAIL (a test failure, a suite that did not
|
||||
complete — timeout, compile/parse/build error, no results file — or a Batch
|
||||
1/Batch 2 check FAILED, or a failed build check), NOT ASSESSED (suite unconfirmed NOT RUN, a runner that
|
||||
could not run or executed zero tests, unconfirmed `PlayMode: NOT RUN`, a check
|
||||
that could not execute — including a build not launched this session — an
|
||||
`UNKNOWN` coverage row, Batch 3 skipped — or no game tests at `qa.level`
|
||||
`standard`/`full`; at `minimal` the automated row is WAIVED and the batches carry
|
||||
the verdict), PASS WITH WARNINGS (MISSING test evidence, a runner warning such
|
||||
as gdUnit4 101's leaked orphans, or a NOT RUN suite the developer confirmed),
|
||||
PASS. NOT ASSESSED ranks above both pass values and below FAIL.
|
||||
Whether a FAIL blocks hand-off depends on `qa.level` (advisory at `minimal`)
|
||||
and `testing.strict.config` (unset → blocking).
|
||||
|
||||
No director gates apply. The skill does NOT invoke any director agents.
|
||||
|
||||
@@ -19,13 +38,13 @@ No director gates apply. The skill does NOT invoke any director agents.
|
||||
|
||||
## Static Assertions (Structural)
|
||||
|
||||
Verified automatically by `/skill-test static` — no fixture needed.
|
||||
Checked against the SKILL.md by `/skill-test spec` — no fixture needed.
|
||||
|
||||
- [ ] Has required frontmatter fields: `name`, `description`, `argument-hint`, `user-invocable`, `allowed-tools`
|
||||
- [ ] Has ≥2 phase headings
|
||||
- [ ] Contains verdict keywords: PASS, PASS WITH WARNINGS, FAIL
|
||||
- [ ] Contains verdict keywords: PASS, PASS WITH WARNINGS, NOT ASSESSED, FAIL
|
||||
- [ ] Contains "May I write" collaborative protocol language before writing the report
|
||||
- [ ] Has a next-step handoff (e.g., `/bug-report` on FAIL, QA hand-off guidance on PASS)
|
||||
- [ ] Has a next-step handoff (re-run `/smoke-check` after a FAIL; QA hand-off to qa-tester on PASS)
|
||||
|
||||
---
|
||||
|
||||
@@ -40,29 +59,30 @@ None. `/smoke-check` is a pre-QA utility skill. No director gates apply.
|
||||
### Case 1: Happy Path — Automated tests pass, manual items confirmed, PASS
|
||||
|
||||
**Fixture:**
|
||||
- `tests/` directory exists with a GDUnit4 runner script
|
||||
- Engine detected as Godot from `technical-preferences.md`
|
||||
- `project.yaml` has `engine.name: Godot` and `commands.test` set to the gdUnit4 command
|
||||
- `addons/gdUnit4/bin/GdUnitCmdTool.gd` exists; game tests exist under
|
||||
`tests/unit/` and `tests/integration/`
|
||||
- `production/qa/qa-plan-sprint-005.md` exists
|
||||
- Automated test runner reports 12 tests, 12 passing, 0 failing
|
||||
- Developer confirms all Batch 1 and Batch 2 smoke checks as PASS
|
||||
- All sprint stories have matching test files (no MISSING coverage)
|
||||
- The runner exits 0 with 12 tests, 12 passing
|
||||
- The developer says the build was launched this session and selects no FAILED item in Batch 1, Batch 2 or Batch 3
|
||||
- Every sprint story is COVERED or EXPECTED (no MISSING, no UNKNOWN)
|
||||
|
||||
**Input:** `/smoke-check`
|
||||
|
||||
**Expected behavior:**
|
||||
1. Skill detects test directory and engine, notes QA plan found
|
||||
2. Runs `godot --headless --script tests/gdunit4_runner.gd` via Bash
|
||||
3. Parses output: 12/12 passing
|
||||
4. Scans test coverage — all stories COVERED or EXPECTED
|
||||
5. Uses `AskUserQuestion` for Batch 1 (core stability) and Batch 2 (sprint mechanics)
|
||||
6. Developer selects PASS for all items
|
||||
7. Report assembled: automated tests PASS, all smoke checks PASS, no MISSING coverage
|
||||
8. Asks "May I write this smoke check report to `production/qa/smoke-[date].md`?"
|
||||
9. Writes report after approval
|
||||
10. Delivers verdict: PASS
|
||||
1. Phase 1 reports environment: Godot, tests found, QA plan path
|
||||
2. Runs `godot --headless --path . --import`, then `commands.test`
|
||||
(`godot --headless -s -d --remote-debug tcp://127.0.0.1:0 res://addons/gdUnit4/bin/GdUnitCmdTool.gd -a res://tests --ignoreHeadlessMode`)
|
||||
under `timeout 900`; the report says `commands.test` was the command run
|
||||
3. Exit 0 with tests found → automated tests PASS (12/12)
|
||||
4. Coverage scan: all stories COVERED or EXPECTED
|
||||
5. `AskUserQuestion` for Batch 1, Batch 2 and Batch 3 — nothing selected
|
||||
6. Report names its checklist source (the QA plan's Smoke Test Scope)
|
||||
7. Asks "May I write this smoke check report to `production/qa/smoke-[date].md`?"
|
||||
8. Writes after approval; delivers "Smoke check passed cleanly. The build is ready for manual QA."
|
||||
|
||||
**Assertions:**
|
||||
- [ ] Automated test runner is invoked via Bash
|
||||
- [ ] Automated test runner is invoked via Bash, after the Godot import
|
||||
- [ ] `AskUserQuestion` is used for manual smoke check batches
|
||||
- [ ] "May I write" is asked before writing the report file
|
||||
- [ ] Report is written to `production/qa/smoke-[date].md`
|
||||
@@ -70,83 +90,115 @@ None. `/smoke-check` is a pre-QA utility skill. No director gates apply.
|
||||
|
||||
---
|
||||
|
||||
### Case 2: Failure Path — Automated test fails, FAIL verdict
|
||||
### Case 2: Failure Path — Automated test fails, blocking FAIL
|
||||
|
||||
**Fixture:**
|
||||
- `tests/` directory exists, engine is Godot
|
||||
- Automated test runner reports 10 tests run: 8 passing, 2 failing
|
||||
- Failing tests: `test_health_clamp_at_zero`, `test_damage_calculation_negative`
|
||||
- QA plan exists
|
||||
- As Case 1, but `project.yaml` sets `modes.rigor: standard` (so `qa.level`
|
||||
resolves to `standard`) and leaves `testing.strict` unset
|
||||
- The runner exits 100: 10 tests run, 8 passing, 2 failing —
|
||||
`test_health_clamp_at_zero`, `test_damage_calculation_negative`
|
||||
|
||||
**Input:** `/smoke-check`
|
||||
|
||||
**Expected behavior:**
|
||||
1. Skill runs automated tests via Bash
|
||||
2. Parses output — 2 failures detected
|
||||
3. Records failing test names
|
||||
4. Proceeds through manual smoke check batches
|
||||
5. Report shows automated tests as FAIL with failing test names listed
|
||||
6. Asks to write report; writes after approval
|
||||
7. Delivers FAIL verdict with message: "The smoke check failed. Do not hand off to
|
||||
QA until these failures are resolved." Lists failing tests and suggests fixing
|
||||
then re-running `/smoke-check`
|
||||
1. Skill runs automated tests via Bash; exit 100 → FAIL
|
||||
2. Records both failing test names
|
||||
3. Proceeds through the manual smoke check batches
|
||||
4. Report shows automated tests as FAIL with the failing test names listed
|
||||
5. Asks to write the report; writes after approval
|
||||
6. Gate is blocking (`qa.level: standard`, `testing.strict.config` unset → blocking)
|
||||
7. Delivers: "The smoke check failed. Do not hand off to QA until these failures
|
||||
are resolved: …" listing each failure, then "Fix the failures and run
|
||||
`/smoke-check` again to re-gate before QA hand-off."
|
||||
|
||||
**Assertions:**
|
||||
- [ ] Failing test names are listed in the report
|
||||
- [ ] Verdict is FAIL
|
||||
- [ ] Post-verdict message directs developer to fix failures before QA hand-off
|
||||
- [ ] The blocking message directs the developer to fix failures before QA hand-off
|
||||
- [ ] `/smoke-check` re-run is suggested after fixing
|
||||
|
||||
---
|
||||
|
||||
### Case 3: Manual Confirmation — AskUserQuestion used, PASS WITH WARNINGS
|
||||
### Case 3: Manual Confirmation — MISSING coverage, PASS WITH WARNINGS
|
||||
|
||||
**Fixture:**
|
||||
- `tests/` directory exists, engine is Godot
|
||||
- Automated test runner reports all tests passing (8/8)
|
||||
- One Logic story has no matching test file (MISSING coverage)
|
||||
- Developer confirms all Batch 1 and Batch 2 smoke checks as PASS
|
||||
- As Case 1: runner exits 0 with 8/8 passing
|
||||
- One Logic story has no matching test file (MISSING coverage); no UNKNOWN rows
|
||||
- The developer selects no FAILED item in Batch 1, Batch 2 or Batch 3
|
||||
|
||||
**Input:** `/smoke-check`
|
||||
|
||||
**Expected behavior:**
|
||||
1. Automated tests PASS
|
||||
2. Coverage scan finds 1 MISSING entry for a Logic story
|
||||
3. `AskUserQuestion` is used for Batch 1 and Batch 2 — developer confirms all PASS
|
||||
4. Report shows: automated tests PASS, manual checks all PASS, 1 MISSING coverage entry
|
||||
5. Verdict is PASS WITH WARNINGS — build ready for QA, but MISSING entry must be
|
||||
resolved before `/story-done` closes the affected story
|
||||
6. Asks to write report; writes after approval
|
||||
3. `AskUserQuestion` is used for Batches 1–3 — nothing selected
|
||||
4. Report shows automated tests PASS, manual checks PASS, and the MISSING entry
|
||||
under "Missing Test Evidence" with its expected test path
|
||||
5. Verdict is PASS WITH WARNINGS; the message says the build is ready for manual
|
||||
QA and lists the MISSING entry as an advisory item to resolve before
|
||||
`/story-done` on the affected story
|
||||
6. Asks to write the report; writes after approval
|
||||
|
||||
**Assertions:**
|
||||
- [ ] `AskUserQuestion` is used for manual smoke check batches (not inline text prompts)
|
||||
- [ ] MISSING test coverage entry appears in the report
|
||||
- [ ] Verdict is PASS WITH WARNINGS (not PASS, not FAIL)
|
||||
- [ ] Advisory note explains MISSING entry must be resolved before `/story-done`
|
||||
- [ ] Advisory note says the MISSING entry must be resolved before `/story-done`
|
||||
- [ ] Report file is written to `production/qa/smoke-[date].md`
|
||||
|
||||
---
|
||||
|
||||
### Case 4: No Test Directory — Skill stops with guidance
|
||||
### Case 4: Scaffold Without Game Tests — NOT ASSESSED, then stop
|
||||
|
||||
**Fixture:**
|
||||
- `tests/` directory does not exist
|
||||
- Engine is configured as Godot
|
||||
- `project.yaml` sets `modes.rigor: standard` (so `qa.level` resolves to `standard`)
|
||||
- Engine is Godot; `tests/unit/` and `tests/integration/` exist but hold only
|
||||
the `.gdignore_placeholder` files `/test-setup` creates
|
||||
- No `tests/smoke/` directory
|
||||
|
||||
**Input:** `/smoke-check`
|
||||
|
||||
**Expected behavior:**
|
||||
1. Phase 1 checks for `tests/` directory — not found
|
||||
2. Skill outputs: "No test directory found at `tests/`. Run `/test-setup` to
|
||||
scaffold the testing infrastructure, or create the directory manually if
|
||||
tests live elsewhere."
|
||||
3. Skill stops — no automated tests run, no manual smoke checks, no report written
|
||||
1. Phase 1 counts game test files, not directories — none are found
|
||||
2. Skill delivers: "Smoke check: **NOT ASSESSED — no game tests found** under
|
||||
[the test root] or `tests/smoke/`. Run `/test-setup` to scaffold the testing
|
||||
infrastructure, or point me at where tests live."
|
||||
3. Skill stops — no automated run, no manual batches, no report written
|
||||
|
||||
**Assertions:**
|
||||
- [ ] Error message references the missing `tests/` directory
|
||||
- [ ] The presence of `tests/` and its subdirectories is not treated as tests existing
|
||||
- [ ] Verdict is NOT ASSESSED — the skill does not stop without a verdict
|
||||
- [ ] `/test-setup` is suggested as the remediation step
|
||||
- [ ] Skill stops after this message (no further phases run)
|
||||
- [ ] No report file is written
|
||||
- [ ] No further phases run and no report file is written
|
||||
|
||||
---
|
||||
|
||||
### Case 4b: No Game Tests at `qa.level: minimal` — WAIVED, the build carries the verdict
|
||||
|
||||
**Fixture:**
|
||||
- No `modes.rigor` set (the default, `minimal`, so `qa.level` resolves to `minimal`)
|
||||
- Engine is Godot; no game test files under `tests/unit/`, `tests/integration/` or `tests/smoke/`
|
||||
- `commands.smoke` is `godot --headless --quit-after 5`; it exits 0 with no `SCRIPT ERROR` line
|
||||
- The user says the build was launched this session and confirms every Batch 1, Batch 2 and Batch 3 check
|
||||
|
||||
**Input:** `/smoke-check`
|
||||
|
||||
**Expected behavior:**
|
||||
1. Phase 1 finds no game test files and, at `qa.level: minimal`, records the automated row as `WAIVED — qa.level: minimal, no game tests`
|
||||
2. Phase 2's test run and Phase 3 are skipped, each with a one-line note saying why
|
||||
3. Phase 2's build check runs `commands.smoke` (after the Godot import) under a timeout: `Build check: PASS`
|
||||
4. Phase 4 runs Batches 1–3; every check passes
|
||||
5. Verdict: PASS — the report's Automated Tests status reads WAIVED
|
||||
|
||||
**Assertions:**
|
||||
- [ ] The missing tests are WAIVED, not NOT ASSESSED, because `qa.level` is `minimal`
|
||||
- [ ] The skipped test run and Phase 3 are announced, not silent
|
||||
- [ ] The build check runs `commands.smoke` and its result is in the report
|
||||
- [ ] Batch 1 and Batch 2 still run — PASS is never given without them
|
||||
- [ ] Verdict is PASS and the report's Automated Tests row reads WAIVED
|
||||
- [ ] A Batch 1 or Batch 2 check that could not be executed (no build) makes the verdict NOT ASSESSED, as at any level
|
||||
- [ ] Variant — the user answers that the build was not launched this session: Batches 2 and 3 are not asked (the skip is said), the Batch 1 checks read `NOT RUN — build not launched this session`, and the verdict is NOT ASSESSED — never PASS from an empty failure list
|
||||
- [ ] Variant — `commands.smoke` prints `SCRIPT ERROR` (Godot's boot check exits 0 on it): the build check is FAIL and so is the verdict
|
||||
|
||||
---
|
||||
|
||||
@@ -158,7 +210,7 @@ None. `/smoke-check` is a pre-QA utility skill. No director gates apply.
|
||||
**Input:** `/smoke-check`
|
||||
|
||||
**Expected behavior:**
|
||||
1. Skill runs all phases and produces a PASS or PASS WITH WARNINGS verdict
|
||||
1. Skill runs all phases and produces a verdict
|
||||
2. No director agents are spawned at any point
|
||||
3. No gate IDs (CD-*, TD-*, AD-*, PR-*) appear in output
|
||||
4. No `/gate-check` is invoked
|
||||
@@ -166,28 +218,134 @@ None. `/smoke-check` is a pre-QA utility skill. No director gates apply.
|
||||
**Assertions:**
|
||||
- [ ] No director gate is invoked
|
||||
- [ ] No gate skip messages appear
|
||||
- [ ] Verdict is PASS, PASS WITH WARNINGS, or FAIL — no gate verdict involved
|
||||
- [ ] Verdict is PASS, PASS WITH WARNINGS, NOT ASSESSED, or FAIL — no gate verdict involved
|
||||
|
||||
---
|
||||
|
||||
### Case 6: gdUnit4 Lookalikes — exit 0 over zero tests, and exit 101
|
||||
|
||||
**Fixture:**
|
||||
- As Case 1, with every story COVERED or EXPECTED and nothing selected in
|
||||
Batch 1, Batch 2 or Batch 3
|
||||
- Run A: the suites under `tests/unit/` declare no test functions; the runner
|
||||
prints `No test cases found` and exits 0
|
||||
- Run B: the runner exits 101 — 12 tests, 12 passing, 2 orphan nodes leaked
|
||||
|
||||
**Input:** `/smoke-check`
|
||||
|
||||
**Expected behavior:**
|
||||
1. Run A: exit 0 is not read as a pass — the automated tests are
|
||||
`NOT ASSESSED — no tests found`, and the verdict is NOT ASSESSED even though
|
||||
every manual check passed
|
||||
2. Run B: 101 is PASS WITH WARNINGS, naming the orphan count; with no MISSING
|
||||
entry, the verdict is PASS WITH WARNINGS and the advisory list names the
|
||||
orphans
|
||||
|
||||
**Assertions:**
|
||||
- [ ] Run A's verdict is NOT ASSESSED — not PASS
|
||||
- [ ] Run B's Status line is PASS WITH WARNINGS with the orphan count
|
||||
- [ ] Run B's verdict is PASS WITH WARNINGS — not PASS, and not FAIL
|
||||
|
||||
---
|
||||
|
||||
### Case 7: Timeout — the suite never completed, FAIL
|
||||
|
||||
**Fixture:**
|
||||
- As Case 2 (`qa.level: standard`, `testing.strict` unset)
|
||||
- The runner hangs; `timeout 900` ends it with exit 124 and no summary printed
|
||||
|
||||
**Input:** `/smoke-check`
|
||||
|
||||
**Expected behavior:**
|
||||
1. Exit 124 is a gate FAILURE, never a pass — the run did not complete and
|
||||
nothing was verified
|
||||
2. The Status line reads FAIL, naming the timeout
|
||||
3. The verdict is FAIL and the gate is blocking
|
||||
|
||||
**Assertions:**
|
||||
- [ ] A timeout is FAIL — not NOT ASSESSED and not NOT RUN
|
||||
- [ ] The report names the timeout as the reason
|
||||
- [ ] The blocking message is delivered
|
||||
|
||||
---
|
||||
|
||||
### Case 8: Batch 3 Skipped — NOT ASSESSED, not PASS
|
||||
|
||||
**Fixture:**
|
||||
- As Case 1: runner exits 0 with 12/12 passing, no MISSING or UNKNOWN rows
|
||||
- In Batch 3 the developer selects only "Performance not checked this session"
|
||||
|
||||
**Input:** `/smoke-check`
|
||||
|
||||
**Expected behavior:**
|
||||
1. Batches 1 and 2 pass
|
||||
2. The skipped Batch 3 item is recorded as not checked — it is not a FAIL
|
||||
3. The verdict is NOT ASSESSED, and the Phase 6 message names the performance
|
||||
check that did not run
|
||||
|
||||
**Assertions:**
|
||||
- [ ] Verdict is NOT ASSESSED — not PASS, not FAIL
|
||||
- [ ] The unchecked performance item is named in the report
|
||||
|
||||
---
|
||||
|
||||
### Case 9: Unity — Play Mode is a second run
|
||||
|
||||
**Fixture:**
|
||||
- `engine.name: Unity`; `com.unity.test-framework` is in `Packages/manifest.json`
|
||||
- `commands.test` is the Edit Mode command `/setup-engine` writes
|
||||
- `Assets/Tests/EditMode/` and `Assets/Tests/PlayMode/` both hold test files
|
||||
- Edit Mode exits 0 with 8/8 passing; Play Mode exits 2 with 1 of 3 failing
|
||||
|
||||
**Input:** `/smoke-check`
|
||||
|
||||
**Expected behavior:**
|
||||
1. The Edit Mode run deletes `test-results/editmode.xml` first, then runs
|
||||
2. Because `commands.test` does not pass `-testPlatform PlayMode` and
|
||||
`Assets/Tests/PlayMode/` holds tests, the skill runs Play Mode too, deleting
|
||||
and then writing `test-results/playmode.xml`
|
||||
3. The report has one Status line per platform: `EditMode: PASS (8/8)` and
|
||||
`PlayMode: FAIL (1 failure)`
|
||||
4. The verdict is FAIL
|
||||
|
||||
**Assertions:**
|
||||
- [ ] Play Mode is run as a second run with its own results file
|
||||
- [ ] Each platform is reported on its own line
|
||||
- [ ] The Play Mode failure makes the verdict FAIL — the Edit Mode pass does not cover it
|
||||
|
||||
---
|
||||
|
||||
## Protocol Compliance
|
||||
|
||||
- [ ] Uses `AskUserQuestion` for all manual smoke check batches (Batch 1, Batch 2, Batch 3)
|
||||
- [ ] Uses `AskUserQuestion` for all manual smoke check batches (Batch 1, Batch 2, Batch 3), and Batch 1 asks whether the build was launched this session; "not launched" yields NOT ASSESSED
|
||||
- [ ] Resolves the editor executable from `commands.test`, then `engine.path`, then `PATH` before reporting the runner unavailable; a bare `godot` that does not resolve is never read as "no engine installed"
|
||||
- [ ] Never edits `project.yaml`: a `commands.test` missing flags or using a relative path is adjusted for the run and reported to the user
|
||||
- [ ] Runs automated tests via Bash before asking any manual questions
|
||||
- [ ] Asks "May I write" before creating the report file — never writes without approval
|
||||
- [ ] Verdict vocabulary is strictly PASS / PASS WITH WARNINGS / FAIL — no other verdicts
|
||||
- [ ] FAIL is triggered by automated test failures or Batch 1/Batch 2 FAIL responses
|
||||
- [ ] PASS WITH WARNINGS is triggered when MISSING test coverage exists but no critical failures
|
||||
- [ ] NOT RUN (engine binary unavailable) is recorded as a warning, not a FAIL
|
||||
- [ ] Verdict vocabulary is PASS / PASS WITH WARNINGS / NOT ASSESSED / FAIL — no other verdicts
|
||||
- [ ] FAIL is triggered by automated test failures, a suite that did not complete (timeout, compile/parse/build error, no results file), or Batch 1/Batch 2 FAIL responses
|
||||
- [ ] PASS WITH WARNINGS is triggered by MISSING test coverage, a runner warning on a passing suite (gdUnit4 101), or a confirmed NOT RUN — with no critical failures
|
||||
- [ ] Unconfirmed NOT RUN (runner not available) yields NOT ASSESSED — never FAIL, never a pass verdict
|
||||
- [ ] A run over zero tests, a runner that could not run, and an unconfirmed `PlayMode: NOT RUN` yield NOT ASSESSED
|
||||
- [ ] Does not invoke director gates at any point
|
||||
|
||||
---
|
||||
|
||||
## Coverage Notes
|
||||
|
||||
- The `quick` argument (skips Phase 3 coverage scan and Batch 3) is not separately
|
||||
fixture-tested; it follows the same pattern as Case 1 with a coverage-skip note in output.
|
||||
- The `--platform` argument adds platform-specific AskUserQuestion batches and a
|
||||
per-platform verdict table; not separately tested here.
|
||||
- The case where the engine binary is not on PATH (NOT RUN) follows the PASS WITH
|
||||
WARNINGS pattern and is covered by the protocol compliance assertions above.
|
||||
- The `quick` argument (skips the Phase 3 coverage scan and Batch 3) is not
|
||||
separately fixture-tested.
|
||||
- The `--platform` argument adds platform-specific `AskUserQuestion` batches and
|
||||
a per-platform verdict table; not separately tested here.
|
||||
- The runner-not-available path (NOT RUN, then NOT ASSESSED until the developer
|
||||
confirms a result from their IDE or CI) is covered by the protocol compliance
|
||||
assertions above.
|
||||
- At `qa.level: minimal` a FAIL is advisory and never blocks hand-off; Case 2
|
||||
pins `standard` to test the blocking path. Unity's `testcasecount="0"` and
|
||||
Unreal's `No automation tests matched` (→ NOT ASSESSED) follow Case 6 Run A;
|
||||
not separately fixture-tested.
|
||||
- The Linux and macOS Unreal commands (picked by `uname -s`) are not
|
||||
fixture-tested. On macOS, Unreal tests are NOT ASSESSED until `commands.test`
|
||||
is set — asserted in the skill, not given a case here.
|
||||
- A Play Mode run that cannot be made (`PlayMode: NOT RUN`, unconfirmed →
|
||||
NOT ASSESSED) is covered by protocol compliance, not by its own case.
|
||||
|
||||
Reference in New Issue
Block a user