mirror of
https://github.com/jmcorgan/fips.git
synced 2026-10-05 19:18:25 +00:00
Johnathan Corgan
da9a12a3dd
The ci nextest profile retries a failing test twice, so a test that fails and then passes leaves the job green, with the retry visible only as "(1 flaky)" in the summary line and a FLAKY line in the log. Recent macOS runs have carried such flakes unnoticed. The Linux job's two junit reporters ignore nextest's <flakyFailure> elements, and the macOS and Windows jobs have no reporter, so no job surfaced them. testing/check-nextest-flaky.sh reads the ci profile's JUnit report and emits a warning annotation and a step-summary line for each test that passed only on retry. It exits 0 so a flake does not red the run, and 2 when there is no report to read. Each of the three unit-test jobs runs it after nextest, on success or failure but not on a cancelled run. Fixture tests over recorded nextest reports (clean, flaky, and a test that failed every attempt) run in local CI and in the GitHub CI parity job. The unit-test jobs cache the whole target directory, so a restored target/nextest/ci/junit.xml from an earlier run could sit where the check reads and hide a run that failed before writing one. Delete the report before nextest runs.
10 lines
812 B
XML
10 lines
812 B
XML
<?xml version="1.0" encoding="UTF-8"?>
|
|
<testsuites name="nextest-run" tests="4" skipped="0" failures="0" errors="0" uuid="96d6f863-2fd5-4341-9a26-0fcb517b97d7" timestamp="2026-10-01T03:24:55.168+00:00" time="0.025">
|
|
<testsuite name="flakydemo" tests="4" skipped="0" errors="0" failures="0">
|
|
<testcase name="tests::also_flips" classname="flakydemo" timestamp="2026-10-01T03:24:55.172+00:00" time="0.016"/>
|
|
<testcase name="tests::always_fails" classname="flakydemo" timestamp="2026-10-01T03:24:55.171+00:00" time="0.020"/>
|
|
<testcase name="tests::steady" classname="flakydemo" timestamp="2026-10-01T03:24:55.168+00:00" time="0.024"/>
|
|
<testcase name="tests::flips_once" classname="flakydemo" timestamp="2026-10-01T03:24:55.168+00:00" time="0.025"/>
|
|
</testsuite>
|
|
</testsuites>
|