Claude-Code-Game-Studios/CCGS Skill Testing Framework/skills/analysis/test-flakiness.md

# Skill Test Spec: /test-flakiness

## Skill Summary

`/test-flakiness` finds flaky tests from the results of multiple CI or local
test runs. It takes a log path, `scan` (all available result files — JUnit XML
from gdUnit4 in `reports/` or saved into `test-results/`, NUnit XML from Unity in
`test-results/`, Unreal automation logs in `Saved/Logs/`), or `registry` (remediation guidance for tests already in the
quarantine section of `tests/regression-suite.md`). With no result data it lists
three ways to get some and stops to ask. It builds a per-test history across
runs, classifies each test that both passed and failed by fail rate — >25% High
(quarantine immediately), 5–25% Moderate (fix directly, do not quarantine yet),
1–5% Low (monitor) — and names a likely cause and fix direction from its cause
table, using Grep on the test file. With fewer than 3 runs, every finding is
"suspected", not "confirmed", whatever its fail rate, and nothing is quarantined;
the summary table's Confidence column says which. It asks separately "May I update the quarantine
section of `tests/regression-suite.md`…?" and "May I write a full flakiness
report to `production/qa/flakiness-report-[date].md`?", appends quarantine
entries without removing existing ones, and never deletes test files. No
director gates are invoked. Verdicts: COMPLETE (report written) or BLOCKED
(user declined write).

---

## Static Assertions (Structural)

Checked against the SKILL.md by `/skill-test spec` — no fixture needed.

- [ ] Has required frontmatter fields: `name`, `description`, `argument-hint`, `user-invocable`, `allowed-tools`
- [ ] Has ≥2 phase headings
- [ ] Contains verdict keywords: COMPLETE, BLOCKED, and the flakiness tiers High / Moderate / Low
- [ ] Contains "May I" write language for `tests/regression-suite.md` and `production/qa/flakiness-report-[date].md`
- [ ] Has a next-step handoff (Section 7: add skip annotations, schedule fixes before the release gate)

---

## Director Gate Checks

None. Flakiness detection is an advisory quality skill; no gates are invoked.

---

## Test Cases

### Case 1: Happy Path — High flakiness from JUnit history, quarantined on approval

**Fixture:**
- `project.yaml`: `engine.name: godot`; `modes.automation` unset (collaborative)
- `test-results/` holds 6 gdUnit4 JUnit XML files from 6 runs of the same commit
- `test_loot_drop_rolls_rare_item` fails in 2 of the 6 runs (33%); the other 40 tests pass in all 6
- `tests/unit/loot/loot_drop_test.gd` calls `randf()` with no seed
- `tests/regression-suite.md` has a Quarantined Tests table with 1 existing entry

**Input:** `/test-flakiness scan`

**Expected behavior:**
1. Skill finds the XML files in `test-results/` and parses `<testcase name=` and `<failure` / `<error` entries
2. Skill builds `test_id → [run1 … run6]` and finds one test with both outcomes
3. 33% is above 25% over 6 runs → **High** flakiness, Confidence confirmed → quarantine immediately
4. Grep of `loot_drop_test.gd` finds `randf` → likely cause Random seed; fix direction: pass an explicit seed
5. The quarantine recommendation names gdUnit4's GDScript skip parameters (`_do_skip := true`, `_skip_reason := "flaky: ..."`) as the way to skip it
6. Summary shows Runs analysed 6, Tests tracked 41, the flaky test with a 33% fail rate, and 40 tests with consistent results
7. Skill asks "May I update the quarantine section of `tests/regression-suite.md` with the flaky tests found?" and, separately, "May I write a full flakiness report to `production/qa/flakiness-report-[date].md`?"
8. On approval, the new entry is appended with `Edit`; the existing entry stays
9. Verdict: **COMPLETE** — flakiness report written

**Assertions:**
- [ ] JUnit XML results in `test-results/` are parsed per test across all 6 runs
- [ ] The flaky test is named with its fail rate, Confidence confirmed, and classified High → quarantine
- [ ] The skip mechanism named is gdUnit4's `_do_skip` / `_skip_reason` parameter pair, not an invented flag
- [ ] The likely cause is Random seed, with the explicit-seed fix direction
- [ ] The two writes are asked about separately, each before it happens
- [ ] The quarantine entry is appended and the existing entry is not removed
- [ ] Verdict is COMPLETE

---

### Case 2: Moderate Flakiness — Fix directly, do not quarantine

**Fixture:**
- `project.yaml`: `engine.name: godot`
- `test-results/` holds 20 JUnit XML files from 20 runs of the same commit
- `test_physics_bounce_height_matches_spec` fails in 3 of 20 runs (15%)
- The test asserts `bounce_height == 0.5`

**Input:** `/test-flakiness scan`

**Expected behavior:**
1. Skill computes a 15% fail rate → **Moderate** flakiness
2. Grep of the test file finds a float equality comparison → likely cause Floating point
3. Recommendation: "This test is intermittently unreliable… Do not quarantine yet — fix the test directly", with the fix direction of an epsilon comparison (`is_equal_approx`)

**Assertions:**
- [ ] A 15% fail rate is classified Moderate
- [ ] The recommendation is to fix directly, not to quarantine
- [ ] The cause is Floating point and the fix names an epsilon comparison such as `is_equal_approx`

---

### Case 3: No Result Data — List options and stop

**Fixture:**
- No `test-results/` directory, no `.github/` directory, no `Saved/Logs/`
- No log path is given

**Input:** `/test-flakiness scan`

**Expected behavior:**
1. Skill finds no CI or local result data
2. Skill prints "No CI log data found…" with the three options: run the suite at least 3 times and collect logs; save a CI log to `test-results/`; run `/test-flakiness registry`
3. Skill stops and asks the user which option to pursue
4. No flakiness table is produced and nothing is written

**Assertions:**
- [ ] Output states no CI log data was found
- [ ] All three options are listed, including `/test-flakiness registry`
- [ ] Skill stops to ask instead of reporting a clean result
- [ ] No files are written

---

### Case 4: Too Few Runs — Suspected, not confirmed

**Fixture:**
- `project.yaml`: `engine.name: godot`
- `test-results/` holds only 2 JUnit XML files, from 2 runs of the same commit
- `test_save_roundtrip_preserves_inventory` passes in one run and fails in the other

**Input:** `/test-flakiness scan`

**Expected behavior:**
1. Skill parses both files and finds the test with both outcomes — a 50% fail rate
2. With fewer than 3 runs, the finding is flagged "suspected", not "confirmed": the summary table's Confidence reads suspected, and the High tier's quarantine-immediately does not apply
3. The test is not quarantined; the recommendation is to collect more runs before acting
4. Skill asks whether more run data is available
5. The Data Limitations section notes that fewer than 5 runs were available, so confidence is low

**Assertions:**
- [ ] The finding is labelled suspected, not confirmed, in the summary table's Confidence column
- [ ] The 50% fail rate does not trigger quarantine — no skip is recommended, and any regression-suite note marks the test suspected
- [ ] Skill asks whether more run data is available
- [ ] Data Limitations states that fewer than 5 runs were analysed

---

### Case 5: Registry Mode in Full Review Mode — Guidance for known quarantined tests, no gates

**Fixture:**
- `project.yaml`: `modes.review_mode: full`
- `tests/regression-suite.md` Quarantined Tests table lists `test_ai_path_recalc_avoids_blocked_tile` (reason: timing) and `test_scene_load_spawns_player` (reason: scene not ready)

**Input:** `/test-flakiness registry`

**Expected behavior:**
1. Skill reads the quarantine section of `tests/regression-suite.md`
2. For each quarantined test it gives remediation guidance from the cause table: Timing / async → explicit await or synchronisation instead of time-based delays; Scene/prefab load race → await one frame after instantiation (`await get_tree().process_frame`)
3. No director gate is invoked regardless of review mode
4. Existing quarantine entries are not removed and no test file is deleted or edited by the skill

**Assertions:**
- [ ] The quarantine section of `tests/regression-suite.md` is the input
- [ ] Each quarantined test gets a fix direction matching its cause
- [ ] No director gate is invoked in any review mode
- [ ] No quarantine entry is removed and no test file is deleted

---

## Protocol Compliance

- [ ] Locates result data (`test-results/` XML, `Saved/Logs/`, or a given log path) before analysis; with none, lists the three options and stops to ask
- [ ] Classifies tests with both outcomes by fail rate: >25% High (quarantine), 5–25% Moderate (fix, do not quarantine), 1–5% Low (monitor)
- [ ] Names a likely cause and fix direction from the cause table, using Grep on the test file
- [ ] Counts a test as flaky only when it both passed and failed across runs with no code change between them
- [ ] With fewer than 3 runs, flags findings as suspected, not confirmed, in the summary's Confidence column, and quarantines nothing
- [ ] Asks separately before updating `tests/regression-suite.md` and before writing `production/qa/flakiness-report-[date].md`
- [ ] Appends quarantine entries, never removes existing ones, and never deletes test files
- [ ] No director gates are invoked
- [ ] Ends with COMPLETE on write or BLOCKED on decline; with no result data it lists the three options and stops to ask instead (Case 3), never reporting a clean result

---

## Coverage Notes

- The no-argument path (run `scan` when CI logs are accessible, else
  `registry`) is not tested here.
- Unity NUnit XML follows the same flow as Case 1, read from `<test-case`
  elements and their `result` attribute; not tested separately.
- Unreal differs and is not tested here: its results are plain-text
  `Result={Success}` / `Result={Fail}` lines in `Saved/Logs/` (each test prints
  `Test Completed. Result={<status>}`), and its skip
  mechanism is NOT SOURCEABLE, so a quarantined Unreal test is logged and the
  user is asked how their CI excludes it. The same applies to gdUnit4 C# tests.
- The Low tier (1–5%) needs 20 or more runs to reach and is not tested.