Release 1.1.2 — the default path works end to end

Fix release for 1.1.1: the minimal path end to end, engine commands that
run on Godot, Unity and Unreal, hook warnings you can see, and verdicts
that say when they could not check. Details in CHANGELOG.md; upgrade
steps in UPGRADING.md (v1.1.1 → v1.1.2).

Fixes #67
Fixes #131

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WuJn2FdynEybfL5KC697nJ
This commit is contained in:
Donchitos
2026-09-29 21:32:19 +10:00
co-authored by Claude Opus 5.5
parent 7ed2c3e9c4
commit b21fa0f7f2
383 changed files with 22888 additions and 10618 deletions
+230 -48
View File
@@ -7,7 +7,7 @@ allowed-tools: Read, Glob, Grep, Bash, Write, AskUserQuestion, Bash(bash "*/.cla
model: sonnet
---
!`bash "${CLAUDE_SKILL_DIR}/../../hooks/yaml-helper.sh" resolve_config --keys automation,qa.level,testing.strict`
!`bash "${CLAUDE_SKILL_DIR}/../../hooks/yaml-helper.sh" resolve_config --keys automation,workflow,qa.level,testing.strict`
@@ -30,7 +30,8 @@ Every `AskUserQuestion` call follows `.claude/docs/automation-modes.md`
`automation_always_ask` categories always prompt).
**`qa.level`**: at `minimal`, smoke-check is **optional** — if
run, a FAIL is advisory and never blocks hand-off; at `standard`, it is required
run, a FAIL is advisory and never blocks hand-off, and a project with no game
tests is checked by its build, launch and critical paths alone (Phase 1 step 1); at `standard`, it is required
before a phase transition; at `full`, before every commit. This sits in front of
the Phase 6 `testing.strict.config` resolution (which only matters once a smoke run
gates). Distinct axis from `workflow`.
@@ -74,11 +75,25 @@ Before running anything, understand the environment:
whose declared engine is not the installed one is worth saying out loud.
1. **Test framework check**: verify that **game** test files exist — not merely
that `tests/` does. Check `tests/unit/`, `tests/integration/` and
`tests/smoke/` for actual test files.
If none are found, **deliver a NOT ASSESSED verdict — do not merely stop.**
that `tests/` does. Check the engine's **test root** (from step 3; the table
is in `.claude/docs/directory-structure.md` — Godot `tests/unit/` and
`tests/integration/`, Unity `Assets/Tests/`, Unreal
`Source/*/Private/Tests/`) and `tests/smoke/` for actual test files. A
**test file** is one the engine's test runner executes — a `*_test.gd` suite,
a C# test class, an Unreal automation test source. Markdown checklists do not
count: `/test-setup` always writes `tests/smoke/critical-paths.md`, and
counting it would hide exactly the zero-test state this step exists to catch.
If none are found **at `qa.level: minimal`**, tests are waived at this level:
record the automated row as `WAIVED — qa.level: minimal, no game tests`, skip
the test run in Phase 2 and all of Phase 3 (say so in one line each), run
Phase 2's **build check**, and go on to Phase 4 — the build check and the
launch and critical-path checks are the smoke check at this level, so
a project with no tests can still reach PASS, but never without a run: a
Batch 1 answer that the build was not launched this session leaves the
verdict NOT ASSESSED.
If none are found **at `standard` or `full`**, **deliver a NOT ASSESSED verdict — do not merely stop.**
"Smoke check: **NOT ASSESSED — no game tests found** under
`tests/unit|integration|smoke`. Run `/test-setup` to scaffold the testing
[the test root] or `tests/smoke/`. Run `/test-setup` to scaffold the testing
infrastructure, or point me at where tests live." Then stop.
> A bare halt is the wrong shape here.
@@ -92,9 +107,8 @@ Before running anything, understand the environment:
> **Do not gate on `tests/` existing.** `/test-setup` creates `tests/unit/`
> and `tests/integration/` with placeholder files, so the directory tree is
> present on any project that ran setup — whether or not a single game test
> was ever written. Count actual test files instead. That is not
> hypothetical: a fixture with no build and zero game tests passed this step
> and went on to score **PASS WITH WARNINGS**.
> was ever written. Count actual test files instead; gated on the directory,
> a project with no build and zero game tests passes this step.
2. **CI check**: check whether `.github/workflows/` contains a workflow file
referencing tests. Note in the report whether CI is configured.
@@ -105,6 +119,17 @@ Before running anything, understand the environment:
(a `[TO BE CONFIGURED]` value means not configured). Store this for test
command selection in Phase 2.
Then find the **editor executable** — none of the three engines installs
onto `PATH` by default on Windows or macOS, so a bare `godot` that does not
resolve says nothing about whether the engine is installed. Take the first
that resolves (the file exists, or `command -v` finds it): the executable
`commands.test` names (its quoted path or first word), else the editor
`engine.path` records, else the engine's name on `PATH` (`godot`). Use it
wherever Phase 2 writes `godot` or `"<Unity editor>"`, quoting a path with a
space; on Unreal, `<UE root>` is `engine.path`. If none resolves, the test
runner is not available (Phase 2) — on Godot,
`NOT ASSESSED — Godot executable not found (commands.test, engine.path, PATH)`.
4. **Smoke test list**: check whether `production/qa/smoke-tests.md` or
`tests/smoke/` exists. If a smoke test list is found, load it for use in
Phase 4. If neither exists, smoke tests will be drawn from the current QA
@@ -113,28 +138,57 @@ Before running anything, understand the environment:
5. **QA plan check**: glob `production/qa/qa-plan-*.md` and take the most
recently modified file. If found, note the path — it will be used in
Phase 3 and Phase 4. If not found, note: "No QA plan found. Run
`/qa-plan sprint` before smoke-checking for best results."
`/qa-plan sprint` before smoke-checking for best results." (At `workflow: minimal`,
which has no sprints, a QA plan is optional — say so rather than recommend one.)
Report findings before proceeding: "Environment: [engine]. Test directory:
Report findings before proceeding: "Environment: [engine]. Editor: [the
resolved executable / not found]. Test directory:
[found / not found]. CI configured: [yes / no]. QA plan: [path / not found]."
---
## Phase 2: Run Automated Tests
Attempt to run the test suite via Bash. Select the command based on the engine
detected in Phase 1:
**Run `commands.test` from `project.yaml` when it is set** — it is the
project's own test command, the one `/setup-engine` wrote and CI runs. Use the
per-engine default below only when `commands.test` is absent, and say in the
report which of the two you ran. Wrap it in a timeout (`timeout 900 …`): **a
timeout (exit 124) is a gate FAILURE, never a pass** — the run never completed
and nothing was verified. macOS has no `timeout`: use `gtimeout 900 …` from
Homebrew's `coreutils` (same exit 124), or, with nothing installed,
`perl -e 'alarm shift; exec @ARGV' 900 …`, which exits 142 when time runs out —
a FAILURE the same way. Then read the result for the engine. On every engine
one outcome looks like success and is not; each is named below.
**Godot 4:**
**Godot 4** — first check that `addons/gdUnit4/bin/GdUnitCmdTool.gd` exists —
the folder has a **capital U**. If it does not, report
`NOT ASSESSED — gdUnit4 not installed at addons/gdUnit4/` instead of running
anything. Then import, whether you run `commands.test` or the
default. A fresh clone has no `.godot/` class cache, and without it gdUnit4's
runner does not load; a cache that predates a new `class_name` fails valid
code with 105. The import takes seconds when nothing changed (`godot` below is
the executable Phase 1 step 3 resolved):
```bash
godot --headless --script tests/gdunit4_runner.gd 2>&1
godot --headless --path . --import
godot --headless -s -d --remote-debug tcp://127.0.0.1:0 res://addons/gdUnit4/bin/GdUnitCmdTool.gd -a res://tests --ignoreHeadlessMode 2>&1
```
If the GDUnit4 runner script does not exist at that path, try:
```bash
godot --headless -s addons/gdunit4/GdUnitRunner.gd 2>&1
```
If neither path exists, note: "GDUnit4 runner not found — confirm the runner
path for your test framework."
gdUnit4's exit code gives the verdict (6.1.3):
- **0** — every test passed: PASS. **But exit 0 over zero tests is not a pass:**
gdUnit4 aborts and still exits 0, so if the output says `No test cases found`,
the verdict is `NOT ASSESSED — no tests found`.
- **101** — every test passed but a test leaked nodes (orphans): PASS WITH
WARNINGS, naming the orphan count from the output. Not a failure.
- **100** — a test failed: FAIL. **105** — a test script does not parse: FAIL.
- **103** — gdUnit4 refused to run headless (`--ignoreHeadlessMode` missing);
**104** — Godot older than 4.3: `NOT ASSESSED`, with that reason.
- **1** — Godot's own error: `GdUnitCmdTool.gd` did not load (`Failed to load
script`), so no test ran. Usually the import above was skipped; after it,
exit 1 is a FAIL.
Every run opens with two `ERROR:` lines about the remote port
(`127.0.0.1:0`) — that is the `--remote-debug` flag working, not a failure.
Never drop the flag: without it a script error opens Godot's interactive
debugger and the run hangs at a `debug>` prompt until the timeout.
**Unity:**
Unity **can** run tests headlessly via shell. Do not skip to reading artifacts.
@@ -150,33 +204,106 @@ give the one-line fix (add `com.unity.test-framework` to `Packages/manifest.json
**Do not fall through to reading stale artifacts** — an unknown-age XML
reported as a pass is worse than no gate.
If present, run the suite with an explicit timeout:
If present, the default is (the **editor** executable — `Unity` on `PATH` may be
Unity's separate CLI). Delete the old results file first, whether you run
`commands.test` or this default:
```bash
timeout 900 "<Unity.exe>" -batchmode -runTests -projectPath . -testPlatform EditMode -testResults test-results/results.xml
rm -f test-results/editmode.xml
timeout 900 "<Unity editor>" -batchmode -runTests -projectPath . -testPlatform EditMode -testResults test-results/editmode.xml
```
Parse `test-results/results.xml` for the `passed` and `failed` counts on the
`<test-run>` element. **A timeout (exit 124) is a gate FAILURE, never a pass** —
the run never completed and nothing was verified.
Exit 0 when every test passes, 2 on a failure. **Exit 1 or 3 is a FAIL**: 1
means the project did not compile (the editor log has the `error CS…` lines),
3 that the run itself failed; 4 (an unknown `-testPlatform`) or any other
non-zero exit is a FAIL too. A compile error writes no results file — which
is why the old one is deleted first: left in place, the last run's pass is read
as this run's. After any exit, no results file is a FAIL. Parse the `<test-run>`
element of the results file for `testcasecount`, `passed` and `failed`. **`testcasecount="0"`
is NOT ASSESSED, never a pass**: tests outside `Assets/` are never compiled, and
Unity then reports `result="Passed"` with exit 0 over zero tests.
**Unreal Engine:**
**Play Mode is a second run.** `-testPlatform EditMode` — this default, and
`commands.test` as `/setup-engine` writes it — never runs the integration tests
`/test-setup` scaffolds under `Assets/Tests/PlayMode/`. When that folder holds a
test file and the command you ran did not already pass `-testPlatform PlayMode`,
run Play Mode the same way, into its own results file, and read it by the same
rules:
```bash
# List most recent Unreal automation logs (bash) — on Windows PowerShell use the fallback below
ls -t Saved/Logs/ 2>/dev/null | grep -i "test\|automation" | head -5 \
|| powershell -Command "Get-ChildItem Saved/Logs/ -ErrorAction SilentlyContinue | Where-Object { $_.Name -match 'test|automation' } | Sort-Object LastWriteTime -Descending | Select-Object -First 5 -ExpandProperty Name"
rm -f test-results/playmode.xml
timeout 900 "<Unity editor>" -batchmode -runTests -projectPath . -testPlatform PlayMode -testResults test-results/playmode.xml
```
If no matching log found: "UE automation tests must be run via the Session
Frontend or CI pipeline. Please confirm test status manually."
Report each platform on its own line — `EditMode: PASS (12/12)`,
`PlayMode: FAIL (1 failure)`. A Play Mode run that could not be made is
reported by name, `PlayMode: NOT RUN — [reason]`: the Edit Mode result does not
cover those tests, so it cannot make the automated-test row a PASS.
**Unreal Engine** — build the editor target first, whether you then run
`commands.test` or the default. `Binaries/` is not committed, and without a
built game module the editor stops with "The game module could not be found",
exit 1, before running any test. The build is incremental — seconds when
nothing changed; a failed build is a FAIL.
**Pick the commands for this machine** with `uname -s`: `Linux` → Linux,
`Darwin` → macOS, anything else (`MINGW*`, `MSYS*`, `CYGWIN*`) → Windows. The
Windows forms below were run on UE 5.7; the Linux and macOS forms come from
Epic's documentation — `docs/engine-reference/unreal/current-best-practices.md`,
"Command Line", has them with their sources. On Windows use
`UnrealBuildTool.exe`, not `Build.bat`: run from bash, `Build.bat` fails on an
engine path with spaces.
```bash
# Windows
timeout 1800 "<UE root>/Engine/Binaries/DotNET/UnrealBuildTool/UnrealBuildTool.exe" <Project>Editor Win64 Development -Project="$(pwd -W 2>/dev/null || pwd)/<Project>.uproject" 2>&1
# Linux
timeout 1800 "<UE root>/Engine/Build/BatchFiles/Linux/Build.sh" <Project>Editor Linux Development -Project="$(pwd -W 2>/dev/null || pwd)/<Project>.uproject" 2>&1
# macOS (timeout: see the macOS note above)
gtimeout 1800 "<UE root>/Engine/Build/BatchFiles/Mac/Build.sh" <Project>Editor Mac Development -Project="$(pwd -W 2>/dev/null || pwd)/<Project>.uproject" 2>&1
```
Then the default. Keep `-stdout -FullStdOutLogOutput`; when `commands.test`
lacks them, add them to the command for this run only: without them the editor
prints nothing, and the lines below are only in `Saved/Logs/<Project>.log`.
Give the project as an **absolute** path — when `commands.test` passes a
relative one, run it with the absolute path instead: UE 5.7
does not find a relative `<Project>.uproject` and exits 1 (`Project file not
found`). `$(pwd -W 2>/dev/null || pwd)` gives the `C:/…` form in Git Bash and
the plain path elsewhere; `$PWD` alone fails when `MSYS_NO_PATHCONV` is set.
Either way, say in the report what you changed and tell the user to fix
`commands.test` in `project.yaml` — never edit `project.yaml` without asking.
`<Project>.` in the filter is the root the tests are named under: the project
name, or the distinct root `/setup-engine` chose when the project's name is
also an engine area — read one test's name string under
`Source/*/Private/Tests/` if unsure.
```bash
# Windows
timeout 1800 "<UE root>/Engine/Binaries/Win64/UnrealEditor-Cmd.exe" "$(pwd -W 2>/dev/null || pwd)/<Project>.uproject" -ExecCmds="Automation RunTests <Project>.; Quit" -unattended -nullrhi -stdout -FullStdOutLogOutput 2>&1
# Linux
timeout 1800 "<UE root>/Engine/Binaries/Linux/UnrealEditor" "$(pwd -W 2>/dev/null || pwd)/<Project>.uproject" -ExecCmds="Automation RunTests <Project>.; Quit" -unattended -nullrhi -stdout -FullStdOutLogOutput 2>&1
```
**macOS has no sourced test command.** Epic documents only the
`UnrealEditor.app` bundle, not a command-line editor inside it. Run a
`commands.test` the user has set and confirmed; with none, report
`NOT ASSESSED — Unreal test command on macOS not sourced; set commands.test to the editor command you use`
instead of guessing a path.
Read the output, not the exit code: a failing test and a run that matched nothing both exit 255.
- `Result={Fail}` on any line → FAIL.
- `No automation tests matched` → `NOT ASSESSED — no tests found` (tests
outside `Source/<Module>/` are never built), not FAIL.
- Exit 0 with `Result={Success}` lines → PASS.
- Any other non-zero exit (`Project file not found`, `The game module … could
not be found`) → FAIL: no test ran. The exception: the editor never started (exit 126 or 127, `command not found`) — that is the runner not being
available, below, and `NOT ASSESSED`.
**Unknown engine / not configured:**
"Engine not configured in `project.yaml` or
`.claude/docs/technical-preferences.md`. Run `/setup-engine` to specify the
engine, then re-run `/smoke-check`."
**If the test runner is not available in this environment** (engine binary not
on PATH, runner script not found, etc.), report clearly:
**If the test runner is not available in this environment** (no editor
executable resolved in Phase 1 step 3, runner script not found, etc.), report
clearly:
"Automated tests could not be executed — engine binary not found on PATH.
Status will be recorded as NOT RUN. Confirm test results from your local IDE
"Automated tests could not be executed — engine executable not found (checked
`commands.test`, `engine.path` and `PATH`; set `engine.path`, or put the
editor's full path in `commands.test`). Status will be recorded as NOT RUN.
Confirm test results from your local IDE
or CI pipeline. Until you do, the verdict is NOT ASSESSED — not FAIL, and not
a pass either: nothing has been observed about this build yet."
@@ -193,6 +320,21 @@ Parse runner output and extract:
- Names of any failing tests (up to 10; if more, note the count)
- Any crash or error output from the runner itself
**Build check — the WAIVED path only** (`qa.level: minimal`, no game tests;
Phase 1 step 1). With no suite to run, run the project's own boot check:
`commands.smoke` when set, else `commands.build` when set, with the executable
Phase 1 step 3 resolved, under the same timeout rules (`timeout 900`, or
`timeout 1800` for `commands.build`; a timeout is a FAIL). On Godot, run the
import line above first. Read the result as `/setup-engine` documents it:
Godot `smoke` exits 0 even on a parse error, so a `SCRIPT ERROR` line in its
output is a FAIL, and so is `Can't run project: no main scene defined` — the
game cannot launch; Unity `smoke` exits 1 on a compile error; any other
non-zero exit is a FAIL. Record `Build check: PASS — [command] exit 0`,
`FAIL — [reason]`, or `NOT RUN — [commands.smoke and commands.build unset |
executable not found]`. A build check that did not run is named in the report
and leaves the launch in Batch 1 to carry the verdict; it never passes on its
own.
---
## Phase 3: Check Test Coverage
@@ -233,7 +375,7 @@ fully close those stories.
## Phase 4: Run Manual Smoke Checks
Draw the smoke test checklist from, in priority order:
Draw the smoke test checklist from, in priority order (and name the one you used in the report):
1. The QA plan's "Smoke Test Scope" section (if QA plan was found in Phase 1)
2. `production/qa/smoke-tests.md` (if it exists)
3. `tests/smoke/` directory contents (if it exists)
@@ -254,8 +396,15 @@ sprint's stories.
Use `AskUserQuestion` to batch-verify. Keep to at most 3 calls.
**Batch 1 — Core stability (always run):**
**Batch 1 — Core stability (always run):** one call, two questions — an empty
failure list means nothing unless somebody launched the build.
```
question: "Did you launch the current build this session?"
multiSelect: false
options:
- "Yes — I launched it and checked the items below"
- "No — I have not launched the current build this session"
question: "Core stability — select any items that FAILED (leave all unselected if everything passed):"
multiSelect: true
options:
@@ -267,6 +416,12 @@ options:
For any selected item, ask the user to briefly describe what failed before generating the report.
If the build was **not launched**, the Batch 1 checks could not be executed:
record them as `NOT RUN — build not launched this session`, skip Batches 2 and
3 and any platform batch (say so — they need a running build), and ignore the
unselected failure list. The verdict is then NOT ASSESSED unless something else
FAILED.
**Batch 2 — Sprint changes and regression (always run):**
```
question: "Sprint changes and regression — select any items that FAILED (leave all unselected if everything passed):"
@@ -354,8 +509,16 @@ Assemble the full smoke check report:
### Automated Tests
**Status**: [PASS ([N] tests, [N] passing) | FAIL ([N] failures) |
NOT RUN ([reason])]
**Status**: [PASS ([N] tests, [N] passing) | PASS WITH WARNINGS ([N] tests
passing; [the runner's warning, e.g. [N] orphan nodes leaked]) | FAIL ([N]
failures, or did not complete: [timeout | compile, parse or build error | no
results file]) | NOT ASSESSED ([reason, e.g. zero tests executed, runner could
not run]) | NOT RUN ([reason]) | WAIVED (qa.level: minimal — no game tests)]
[Unity: one Status line per test platform — `EditMode:` and `PlayMode:`.]
[WAIVED only:] **Build check**: [PASS — `[command]` exit 0 | FAIL — [reason] |
NOT RUN — [commands.smoke and commands.build unset | executable not found]]
[If FAIL, list failing tests:]
- `[test name]` — [brief failure description from runner output]
@@ -381,6 +544,7 @@ will determine whether the automated test row contributes to a FAIL verdict."
### Manual Smoke Checks
- [x] Build launched this session — yes [or: `NOT RUN — build not launched this session`; the batches below were not asked]
- [x] Game launches without crash — PASS
- [x] New game starts — PASS
- [x] [Core mechanic] — PASS
@@ -422,13 +586,23 @@ Any platform with one or more FAIL checks contributes to the overall FAIL verdic
**FAIL** if ANY of:
- Automated test suite ran and reported one or more test failures
- The automated suite did not complete: a timeout, a compile, parse or build
error, or no results file after the run (Phase 2 names each engine's form)
- The build check (WAIVED path) failed: a non-zero exit, a timeout, or a Godot
`SCRIPT ERROR` or no-main-scene line
- Any Batch 1 (core stability) check returned FAIL
- Any Batch 2 (primary sprint mechanic or regression check) returned FAIL
**NOT ASSESSED** if ANY of:
- The automated suite is **unconfirmed NOT RUN** — nobody has reported a result
- The runner could not run (gdUnit4 103 or 104, gdUnit4 or the Unity Test
Framework not installed), or it ran and executed zero tests
(`No test cases found`, `testcasecount="0"`, `No automation tests matched`)
- Unity Play Mode tests exist and are `PlayMode: NOT RUN`, unconfirmed — the
Edit Mode result does not cover them
- A Batch 1 or Batch 2 check could not be executed (no build, engine not
configured, platform unavailable) as opposed to executing and failing
configured, platform unavailable) as opposed to executing and failing —
including a Batch 1 answer that the build was not launched this session
- **Any story's coverage row is `UNKNOWN`** (Phase 3: story file missing or
unreadable). A story nobody could read is not a story with no gaps — without
this line, a run where *every* row is UNKNOWN and the suite passes matches
@@ -438,13 +612,17 @@ Any platform with one or more FAIL checks contributes to the overall FAIL verdic
this line it matches no rule at all and renders as `[-]` beside a PASS
**PASS WITH WARNINGS** if ALL of:
- Automated tests PASS, or NOT RUN **and the developer has confirmed the result
from their local IDE or CI**
- Automated tests PASS or PASS WITH WARNINGS, WAIVED at `qa.level: minimal`, or
NOT RUN **and the developer has confirmed the result from their local IDE or CI**
- All Batch 1 and Batch 2 smoke checks PASS
- One or more Logic/Integration stories have MISSING test evidence
- At least one warning: a Logic/Integration story with MISSING test evidence,
a runner warning on a passing suite (gdUnit4 101 — orphan nodes leaked), or
a NOT RUN suite the developer confirmed
**PASS** if ALL of:
- Automated tests PASS
- Automated tests PASS, or WAIVED (`qa.level: minimal` and no game tests exist —
the build check, when `commands.smoke` or `commands.build` is set, and the
launch and critical-path batches below then carry the verdict)
- All smoke checks in all batches PASS or N/A
- No MISSING test evidence entries
````
@@ -527,10 +705,13 @@ are fixed."
run. Do not hand off to QA on this result:
[Name each check that could not execute, and why: suite unconfirmed NOT RUN,
engine binary absent, no build, platform unavailable]
`PlayMode: NOT RUN`, zero tests executed, runner not installed, engine
executable not found, no build, build not launched this session, platform
unavailable]
[For each, the one thing that would make it runnable — e.g. 'confirm the suite
result from your IDE or CI', 'run `/setup-engine`', 'produce a build'.]
result from your IDE or CI', 'run `/setup-engine`', 'set `engine.path`',
'produce a build', 'launch the build and play the checks'.]
Re-run `/smoke-check` once any of those is resolved."
@@ -546,7 +727,8 @@ their behalf.
"Smoke check passed with warnings. The build is ready for manual QA.
Advisory items to resolve before running `/story-done` on affected stories:
[list MISSING test evidence entries]
[list MISSING test evidence entries, and any runner warning — e.g. the orphan
count]
QA hand-off: share `production/qa/qa-plan-[sprint].md` with the qa-tester
agent to begin manual verification."