Claude-Code-Game-Studios/.claude/skills/smoke-check/references/2-run-tests.md

/smoke-check — Phase 2: Run Automated Tests

Part of /smoke-check. SKILL.md names when to read this file; the settings it resolved and the rules under its later headings still apply here.

Phase 2: Run Automated Tests

In this file:

• commands.test first
• Godot 4
• Unity: Edit Mode, then Play Mode as a second run
• Unreal Engine: build first, commands per OS, macOS
• Unknown engine / not configured
• Test runner not available here
• Build check (the WAIVED path only)

Run commands.test from project.yaml when it is set — it is the project's own test command, the one /setup-engine wrote and CI runs. Use the per-engine default below only when commands.test is absent, and say in the report which of the two you ran. Wrap it in a timeout (timeout 900 …): a timeout (exit 124) is a gate FAILURE, never a pass — the run never completed and nothing was verified. macOS has no timeout: use gtimeout 900 … from Homebrew's coreutils (same exit 124), or, with nothing installed, perl -e 'alarm shift; exec @ARGV' 900 …, which exits 142 when time runs out — a FAILURE the same way. Then read the result for the engine. On every engine one outcome looks like success and is not; each is named below.

Godot 4 — first check that addons/gdUnit4/bin/GdUnitCmdTool.gd exists — the folder has a capital U. If it does not, report NOT ASSESSED — gdUnit4 not installed at addons/gdUnit4/ instead of running anything. Then import, whether you run commands.test or the default. A fresh clone has no .godot/ class cache, and without it gdUnit4's runner does not load; a cache that predates a new class_name fails valid code with 105. The import takes seconds when nothing changed (godot below is the executable Phase 1 step 3 resolved):

godot --headless --path . --import
godot --headless -s -d --remote-debug tcp://127.0.0.1:0 res://addons/gdUnit4/bin/GdUnitCmdTool.gd -a res://tests --ignoreHeadlessMode 2>&1

gdUnit4's exit code gives the verdict (6.1.3):

• 0 — every test passed: PASS. But exit 0 over zero tests is not a pass: gdUnit4 aborts and still exits 0, so if the output says No test cases found, the verdict is NOT ASSESSED — no tests found.
• 101 — every test passed but a test leaked nodes (orphans): PASS WITH WARNINGS, naming the orphan count from the output. Not a failure.
• 100 — a test failed: FAIL. 105 — a test script does not parse: FAIL.
• 103 — gdUnit4 refused to run headless (--ignoreHeadlessMode missing); 104 — Godot older than 4.3: NOT ASSESSED, with that reason.
• 1 — Godot's own error: GdUnitCmdTool.gd did not load (Failed to load script), so no test ran. Usually the import above was skipped; after it, exit 1 is a FAIL.

Every run opens with two ERROR: lines about the remote port (127.0.0.1:0) — that is the --remote-debug flag working, not a failure. Never drop the flag: without it a script error opens Godot's interactive debugger and the run hangs at a debug> prompt until the timeout.

Unity: Unity can run tests headlessly via shell. Do not skip to reading artifacts.

First confirm the Unity Test Framework is installed. A project without it does not fail cleanly — it hangs until something kills it, which is the observation the old "Unity cannot test headlessly" advice was generalised from:

grep -q 'com.unity.test-framework' Packages/manifest.json && echo present || echo ABSENT

If ABSENT, report NOT ASSESSED — Unity Test Framework not installed and give the one-line fix (add com.unity.test-framework to Packages/manifest.json). Do not fall through to reading stale artifacts — an unknown-age XML reported as a pass is worse than no gate.

If present, the default is (the editor executable — Unity on PATH may be Unity's separate CLI). Delete the old results file first, whether you run commands.test or this default:

rm -f test-results/editmode.xml
timeout 900 "<Unity editor>" -batchmode -runTests -projectPath . -testPlatform EditMode -testResults test-results/editmode.xml

Exit 0 when every test passes, 2 on a failure. Exit 1 or 3 is a FAIL: 1 means the project did not compile (the editor log has the error CS… lines), 3 that the run itself failed; 4 (an unknown -testPlatform) or any other non-zero exit is a FAIL too. A compile error writes no results file — which is why the old one is deleted first: left in place, the last run's pass is read as this run's. After any exit, no results file is a FAIL. Parse the <test-run> element of the results file for testcasecount, passed and failed. testcasecount="0" is NOT ASSESSED, never a pass: tests outside Assets/ are never compiled, and Unity then reports result="Passed" with exit 0 over zero tests.

Play Mode is a second run. -testPlatform EditMode — this default, and commands.test as /setup-engine writes it — never runs the integration tests /test-setup scaffolds under Assets/Tests/PlayMode/. When that folder holds a test file and the command you ran did not already pass -testPlatform PlayMode, run Play Mode the same way, into its own results file, and read it by the same rules:

rm -f test-results/playmode.xml
timeout 900 "<Unity editor>" -batchmode -runTests -projectPath . -testPlatform PlayMode -testResults test-results/playmode.xml

Report each platform on its own line — EditMode: PASS (12/12), PlayMode: FAIL (1 failure). A Play Mode run that could not be made is reported by name, PlayMode: NOT RUN — [reason]: the Edit Mode result does not cover those tests, so it cannot make the automated-test row a PASS.

Unreal Engine — build the editor target first, whether you then run commands.test or the default. Binaries/ is not committed, and without a built game module the editor stops with "The game module could not be found", exit 1, before running any test. The build is incremental — seconds when nothing changed; a failed build is a FAIL.

Pick the commands for this machine with uname -s: Linux → Linux, Darwin → macOS, anything else (MINGW*, MSYS*, CYGWIN*) → Windows. The Windows forms below were run on UE 5.7; the Linux and macOS forms come from Epic's documentation — docs/engine-reference/unreal/current-best-practices.md, "Command Line", has them with their sources. On Windows use UnrealBuildTool.exe, not Build.bat: run from bash, Build.bat fails on an engine path with spaces.

# Windows
timeout 1800 "<UE root>/Engine/Binaries/DotNET/UnrealBuildTool/UnrealBuildTool.exe" <Project>Editor Win64 Development -Project="$(pwd -W 2>/dev/null || pwd)/<Project>.uproject" 2>&1
# Linux
timeout 1800 "<UE root>/Engine/Build/BatchFiles/Linux/Build.sh" <Project>Editor Linux Development -Project="$(pwd -W 2>/dev/null || pwd)/<Project>.uproject" 2>&1
# macOS (timeout: see the macOS note above)
gtimeout 1800 "<UE root>/Engine/Build/BatchFiles/Mac/Build.sh" <Project>Editor Mac Development -Project="$(pwd -W 2>/dev/null || pwd)/<Project>.uproject" 2>&1

Then the default. Keep -stdout -FullStdOutLogOutput; when commands.test lacks them, add them to the command for this run only: without them the editor prints nothing, and the lines below are only in Saved/Logs/<Project>.log. Give the project as an absolute path — when commands.test passes a relative one, run it with the absolute path instead: UE 5.7 does not find a relative <Project>.uproject and exits 1 (Project file not found). $(pwd -W 2>/dev/null || pwd) gives the C:/… form in Git Bash and the plain path elsewhere; $PWD alone fails when MSYS_NO_PATHCONV is set. Either way, say in the report what you changed and tell the user to fix commands.test in project.yaml — never edit project.yaml without asking. <Project>. in the filter is the root the tests are named under: the project name, or the distinct root /setup-engine chose when the project's name is also an engine area — read one test's name string under Source/*/Private/Tests/ if unsure.

# Windows
timeout 1800 "<UE root>/Engine/Binaries/Win64/UnrealEditor-Cmd.exe" "$(pwd -W 2>/dev/null || pwd)/<Project>.uproject" -ExecCmds="Automation RunTests <Project>.; Quit" -unattended -nullrhi -stdout -FullStdOutLogOutput 2>&1
# Linux
timeout 1800 "<UE root>/Engine/Binaries/Linux/UnrealEditor" "$(pwd -W 2>/dev/null || pwd)/<Project>.uproject" -ExecCmds="Automation RunTests <Project>.; Quit" -unattended -nullrhi -stdout -FullStdOutLogOutput 2>&1

macOS has no sourced test command. Epic documents only the UnrealEditor.app bundle, not a command-line editor inside it. Run a commands.test the user has set and confirmed; with none, report NOT ASSESSED — Unreal test command on macOS not sourced; set commands.test to the editor command you use instead of guessing a path. Read the output, not the exit code: a failing test and a run that matched nothing both exit 255.

• Result={Fail} on any line → FAIL.
• No automation tests matched → NOT ASSESSED — no tests found (tests outside Source/<Module>/ are never built), not FAIL.
• Exit 0 with Result={Success} lines → PASS.
• Any other non-zero exit (Project file not found, The game module … could not be found) → FAIL: no test ran. The exception: the editor never started (exit 126 or 127, command not found) — that is the runner not being available, below, and NOT ASSESSED.

Unknown engine / not configured: "Engine not configured in project.yaml or .claude/docs/technical-preferences.md. Run /setup-engine to specify the engine, then re-run /smoke-check."

If the test runner is not available in this environment (no editor executable resolved in Phase 1 step 3, runner script not found, etc.), report clearly:

"Automated tests could not be executed — engine executable not found (checked commands.test, engine.path and PATH; set engine.path, or put the editor's full path in commands.test). Status will be recorded as NOT RUN. Confirm test results from your local IDE or CI pipeline. Until you do, the verdict is NOT ASSESSED — not FAIL, and not a pass either: nothing has been observed about this build yet."

Do not treat NOT RUN as an automatic FAIL. Record it, and let the developer's manual confirmation in Phase 4 resolve it. Until that confirmation arrives the verdict is NOT ASSESSED (see the verdict rules in Phase 5), which ranks above both pass values and below FAIL. An unrun suite is not a healthy build; it is an unknown one, and the two need different follow-ups.

Parse runner output and extract:

• Total tests run
• Passing count
• Failing count
• Names of any failing tests (up to 10; if more, note the count)
• Any crash or error output from the runner itself

Build check — the WAIVED path only (qa.level: minimal, no game tests; Phase 1 step 1). With no suite to run, run the project's own boot check: commands.smoke when set, else commands.build when set, with the executable Phase 1 step 3 resolved, under the same timeout rules (timeout 900, or timeout 1800 for commands.build; a timeout is a FAIL). On Godot, run the import line above first. Read the result as /setup-engine documents it: Godot smoke exits 0 even on a parse error, so a SCRIPT ERROR line in its output is a FAIL, and so is Can't run project: no main scene defined — the game cannot launch; Unity smoke exits 1 on a compile error; any other non-zero exit is a FAIL. Record Build check: PASS — [command] exit 0, FAIL — [reason], or NOT RUN — [commands.smoke and commands.build unset | executable not found]. A build check that did not run is named in the report and leaves the launch in Batch 1 to carry the verdict; it never passes on its own.

---