mirror of
https://github.com/jmcorgan/fips.git
synced 2026-10-05 11:08:25 +00:00
Surface unit tests that passed only on retry
The ci nextest profile retries a failing test twice, so a test that fails and then passes leaves the job green, with the retry visible only as "(1 flaky)" in the summary line and a FLAKY line in the log. Recent macOS runs have carried such flakes unnoticed. The Linux job's two junit reporters ignore nextest's <flakyFailure> elements, and the macOS and Windows jobs have no reporter, so no job surfaced them. testing/check-nextest-flaky.sh reads the ci profile's JUnit report and emits a warning annotation and a step-summary line for each test that passed only on retry. It exits 0 so a flake does not red the run, and 2 when there is no report to read. Each of the three unit-test jobs runs it after nextest, on success or failure but not on a cancelled run. Fixture tests over recorded nextest reports (clean, flaky, and a test that failed every attempt) run in local CI and in the GitHub CI parity job. The unit-test jobs cache the whole target directory, so a restored target/nextest/ci/junit.xml from an earlier run could sit where the check reads and hide a run that failed before writing one. Delete the report before nextest runs.
This commit is contained in:
Johnathan Corgan
@@ -89,6 +89,10 @@ jobs:
|
||||
# step with ci-local.sh's run_deb_version by hand.
|
||||
- name: Check the Debian version derived for tags and branches
|
||||
run: bash testing/check-deb-version.sh
|
||||
# The unit-test jobs' flaky-test reporter, against recorded nextest
|
||||
# reports. Kept in step with ci-local.sh's run_nextest_flaky by hand.
|
||||
- name: Check the flaky-test reporter against its fixtures
|
||||
run: bash testing/nextest-flaky/test.sh
|
||||
|
||||
fmt:
|
||||
name: Format check
|
||||
@@ -314,9 +318,26 @@ jobs:
|
||||
- name: Install cargo-nextest
|
||||
uses: taiki-e/install-action@nextest
|
||||
|
||||
# The cache restores target/, including the previous run's report. Remove
|
||||
# it so the flaky-test check below reads only this run's, and reports a
|
||||
# missing one when nextest wrote none.
|
||||
- name: Remove a cached nextest report
|
||||
shell: bash
|
||||
run: rm -f target/nextest/ci/junit.xml
|
||||
|
||||
- name: Run unit tests
|
||||
run: cargo nextest run --all --profile ci
|
||||
|
||||
# The ci profile retries a failing test, so a test that passed only on
|
||||
# retry leaves the job green. Annotate each one, and list it in the run
|
||||
# summary, so a flake is seen without failing an unrelated run. The
|
||||
# Linux job's two junit reporters ignore nextest's <flakyFailure>
|
||||
# elements, and the macOS and Windows jobs have no reporter at all.
|
||||
- name: Report tests that passed only on retry
|
||||
if: success() || failure()
|
||||
shell: bash
|
||||
run: bash testing/check-nextest-flaky.sh target/nextest/ci/junit.xml
|
||||
|
||||
- name: Publish test report (Checks tab)
|
||||
uses: dorny/test-reporter@4a2e97665d5fa767581ef38eca97b9694bd4eef4 # v2
|
||||
if: always()
|
||||
@@ -381,9 +402,26 @@ jobs:
|
||||
- name: Install cargo-nextest
|
||||
uses: taiki-e/install-action@nextest
|
||||
|
||||
# The cache restores target/, including the previous run's report. Remove
|
||||
# it so the flaky-test check below reads only this run's, and reports a
|
||||
# missing one when nextest wrote none.
|
||||
- name: Remove a cached nextest report
|
||||
shell: bash
|
||||
run: rm -f target/nextest/ci/junit.xml
|
||||
|
||||
- name: Run unit tests
|
||||
run: cargo nextest run --all --profile ci
|
||||
|
||||
# The ci profile retries a failing test, so a test that passed only on
|
||||
# retry leaves the job green. Annotate each one, and list it in the run
|
||||
# summary, so a flake is seen without failing an unrelated run. The
|
||||
# Linux job's two junit reporters ignore nextest's <flakyFailure>
|
||||
# elements, and the macOS and Windows jobs have no reporter at all.
|
||||
- name: Report tests that passed only on retry
|
||||
if: success() || failure()
|
||||
shell: bash
|
||||
run: bash testing/check-nextest-flaky.sh target/nextest/ci/junit.xml
|
||||
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
# Job 2c – Unit tests (Windows)
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
@@ -413,9 +451,26 @@ jobs:
|
||||
- name: Install cargo-nextest
|
||||
uses: taiki-e/install-action@nextest
|
||||
|
||||
# The cache restores target/, including the previous run's report. Remove
|
||||
# it so the flaky-test check below reads only this run's, and reports a
|
||||
# missing one when nextest wrote none.
|
||||
- name: Remove a cached nextest report
|
||||
shell: bash
|
||||
run: rm -f target/nextest/ci/junit.xml
|
||||
|
||||
- name: Run unit tests
|
||||
run: cargo nextest run --all --profile ci
|
||||
|
||||
# The ci profile retries a failing test, so a test that passed only on
|
||||
# retry leaves the job green. Annotate each one, and list it in the run
|
||||
# summary, so a flake is seen without failing an unrelated run. The
|
||||
# Linux job's two junit reporters ignore nextest's <flakyFailure>
|
||||
# elements, and the macOS and Windows jobs have no reporter at all.
|
||||
- name: Report tests that passed only on retry
|
||||
if: success() || failure()
|
||||
shell: bash
|
||||
run: bash testing/check-nextest-flaky.sh target/nextest/ci/junit.xml
|
||||
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
# Job 2d – PowerShell lint (Windows packaging scripts)
|
||||
#
|
||||
|
||||
@@ -0,0 +1,95 @@
|
||||
#!/bin/bash
|
||||
# ── Surface tests that passed only on retry ─────────────────────────────────
|
||||
# The ci nextest profile (.config/nextest.toml) retries a failing test twice,
|
||||
# so a test that fails and then passes reports green. nextest says so only in
|
||||
# its summary count ("N passed (1 flaky)") and a FLAKY line in the job log,
|
||||
# where nobody looks on a green run; that is how a real race in a test went
|
||||
# unnoticed until someone happened to read the output. The profile's own
|
||||
# comment names the remedy: surface retried-but-passed tests, and keep the
|
||||
# retries so a flake does not red an unrelated run.
|
||||
#
|
||||
# This reads the profile's JUnit report and, for every test case carrying a
|
||||
# <flakyFailure> (an attempt that failed before the final one passed), emits a
|
||||
# GitHub warning annotation and a line in the step summary. A test that failed
|
||||
# every attempt carries <failure> instead and is the nextest step's red, not
|
||||
# this script's concern.
|
||||
#
|
||||
# The report is parsed with awk rather than an XML library so the script runs
|
||||
# unchanged on the Linux, macOS and Windows (Git Bash) runners. It relies on
|
||||
# quick-junit's layout, one element per line, and on the two element names
|
||||
# appearing only as elements: in text and attributes `<` is always escaped.
|
||||
#
|
||||
# Usage: check-nextest-flaky.sh [junit.xml]
|
||||
# Default path: target/nextest/ci/junit.xml, the ci profile's report.
|
||||
# Exit 0 = the report was read (flaky tests, if any, were annotated; they do
|
||||
# not fail the step). Exit 2 = no report, or one with no test cases: nothing
|
||||
# was checked, and that is never reported as a pass.
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
set -uo pipefail
|
||||
|
||||
REPORT="${1:-target/nextest/ci/junit.xml}"
|
||||
|
||||
if [[ ! -s "$REPORT" ]]; then
|
||||
echo "::error title=Flaky-test check::no JUnit report at $REPORT; flaky tests were not checked"
|
||||
exit 2
|
||||
fi
|
||||
|
||||
# One line per flaky test: "<failed attempts><TAB><classname><TAB><name>",
|
||||
# then a final "cases<TAB><count>" line so an empty or truncated report is
|
||||
# told apart from a clean one.
|
||||
if ! parsed="$(awk '
|
||||
function attr(line, key, m, v) {
|
||||
if (match(line, " " key "=\"[^\"]*\"")) {
|
||||
v = substr(line, RSTART + length(key) + 3, RLENGTH - length(key) - 4)
|
||||
gsub(/</, "<", v); gsub(/>/, ">", v); gsub(/"/, "\"", v)
|
||||
gsub(/'/, "\047", v); gsub(/&/, "\\&", v)
|
||||
return v
|
||||
}
|
||||
return ""
|
||||
}
|
||||
function flush() {
|
||||
if (fails > 0) printf "%d\t%s\t%s\n", fails, cls, name
|
||||
fails = 0
|
||||
}
|
||||
/<testcase[ >]/ { flush(); cases++; name = attr($0, "name"); cls = attr($0, "classname") }
|
||||
/<flakyFailure[ >]/ { fails++ }
|
||||
END { flush(); printf "cases\t%d\n", cases }
|
||||
' "$REPORT")"; then
|
||||
echo "::error title=Flaky-test check::could not parse $REPORT; flaky tests were not checked"
|
||||
exit 2
|
||||
fi
|
||||
|
||||
cases="$(printf '%s\n' "$parsed" | awk -F'\t' '$1 == "cases" { print $2 }')"
|
||||
if [[ -z "$cases" || "$cases" -eq 0 ]]; then
|
||||
echo "::error title=Flaky-test check::$REPORT lists no test cases; flaky tests were not checked"
|
||||
exit 2
|
||||
fi
|
||||
|
||||
flaky=0
|
||||
summary=""
|
||||
while IFS=$'\t' read -r fails cls name; do
|
||||
[[ "$fails" == "cases" ]] && continue
|
||||
flaky=$((flaky + 1))
|
||||
# Workflow-command values escape %, CR and LF; names carry none of the
|
||||
# latter, but a % in a test name would otherwise be read as an escape.
|
||||
msg="$cls $name failed $fails attempt(s) before passing on retry"
|
||||
echo "::warning title=Flaky test::${msg//%/%25}"
|
||||
summary+="- \`$cls $name\`: failed $fails attempt(s), then passed"$'\n'
|
||||
done <<< "$parsed"
|
||||
|
||||
if [[ "$flaky" -eq 0 ]]; then
|
||||
echo "check-nextest-flaky: $cases test case(s), none passed only on retry"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
echo "check-nextest-flaky: $flaky of $cases test case(s) passed only on retry"
|
||||
if [[ -n "${GITHUB_STEP_SUMMARY:-}" ]]; then
|
||||
{
|
||||
echo "### Flaky tests ($flaky)"
|
||||
echo ""
|
||||
echo "These failed at least once and passed on a retry, so the run is green."
|
||||
echo ""
|
||||
printf '%s' "$summary"
|
||||
} >> "$GITHUB_STEP_SUMMARY"
|
||||
fi
|
||||
exit 0
|
||||
@@ -1604,6 +1604,18 @@ run_deb_version() {
|
||||
record "deb-version" $rc
|
||||
}
|
||||
|
||||
# The GitHub unit-test jobs run check-nextest-flaky.sh after nextest to
|
||||
# surface tests that passed only on retry. Nothing local runs nextest under the
|
||||
# retrying ci profile, so the checker itself never runs here; its fixture tests
|
||||
# do, so a checker that stopped seeing flaky tests fails here rather than going
|
||||
# quiet on GitHub. Static, about a second, and needs nothing built.
|
||||
run_nextest_flaky() {
|
||||
local rc=0
|
||||
info "[nextest-flaky] Checking the flaky-test reporter against its fixtures"
|
||||
bash "$SCRIPT_DIR/nextest-flaky/test.sh" || rc=$?
|
||||
record "nextest-flaky" $rc
|
||||
}
|
||||
|
||||
# ── Main ───────────────────────────────────────────────────────────────────
|
||||
|
||||
main() {
|
||||
@@ -1626,6 +1638,7 @@ main() {
|
||||
run_comment_refs
|
||||
run_wait_converge
|
||||
run_deb_version
|
||||
run_nextest_flaky
|
||||
|
||||
if [[ "$TEST_ONLY" == true ]]; then
|
||||
run_tests
|
||||
|
||||
@@ -0,0 +1,9 @@
|
||||
<?xml version="1.0" encoding="UTF-8"?>
|
||||
<testsuites name="nextest-run" tests="4" skipped="0" failures="0" errors="0" uuid="96d6f863-2fd5-4341-9a26-0fcb517b97d7" timestamp="2026-10-01T03:24:55.168+00:00" time="0.025">
|
||||
<testsuite name="flakydemo" tests="4" skipped="0" errors="0" failures="0">
|
||||
<testcase name="tests::also_flips" classname="flakydemo" timestamp="2026-10-01T03:24:55.172+00:00" time="0.016"/>
|
||||
<testcase name="tests::always_fails" classname="flakydemo" timestamp="2026-10-01T03:24:55.171+00:00" time="0.020"/>
|
||||
<testcase name="tests::steady" classname="flakydemo" timestamp="2026-10-01T03:24:55.168+00:00" time="0.024"/>
|
||||
<testcase name="tests::flips_once" classname="flakydemo" timestamp="2026-10-01T03:24:55.168+00:00" time="0.025"/>
|
||||
</testsuite>
|
||||
</testsuites>
|
||||
@@ -0,0 +1,72 @@
|
||||
<?xml version="1.0" encoding="UTF-8"?>
|
||||
<testsuites name="nextest-run" tests="4" skipped="0" failures="1" errors="0" uuid="f7324cab-d0eb-42ec-aa2e-bb588c9f47b5" timestamp="2026-10-01T03:24:55.659+00:00" time="0.052">
|
||||
<testsuite name="flakydemo" tests="4" skipped="0" errors="0" failures="1">
|
||||
<testcase name="tests::steady" classname="flakydemo" timestamp="2026-10-01T03:24:55.659+00:00" time="0.009"/>
|
||||
<testcase name="tests::also_flips" classname="flakydemo" timestamp="2026-10-01T03:24:55.660+00:00" time="0.009"/>
|
||||
<testcase name="tests::flips_once" classname="flakydemo" timestamp="2026-10-01T03:24:55.660+00:00" time="0.015"/>
|
||||
<testcase name="tests::always_fails" classname="flakydemo" timestamp="2026-10-01T03:24:55.659+00:00" time="0.015">
|
||||
<failure message="thread 'tests::always_fails' (191023) panicked at src/lib.rs:22:67" type="test failure with exit code 101">thread 'tests::always_fails' (191023) panicked at src/lib.rs:22:67:
|
||||
nope
|
||||
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace</failure>
|
||||
<rerunFailure timestamp="2026-10-01T03:24:55.679+00:00" time="0.016" message="thread 'tests::always_fails' (191028) panicked at src/lib.rs:22:67" type="test failure with exit code 101">thread 'tests::always_fails' (191028) panicked at src/lib.rs:22:67:
|
||||
nope
|
||||
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
|
||||
<system-out>
|
||||
running 1 test
|
||||
test tests::always_fails ... FAILED
|
||||
|
||||
failures:
|
||||
|
||||
failures:
|
||||
tests::always_fails
|
||||
|
||||
test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 3 filtered out; finished in 0.00s
|
||||
|
||||
</system-out>
|
||||
<system-err>
|
||||
thread 'tests::always_fails' (191028) panicked at src/lib.rs:22:67:
|
||||
nope
|
||||
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
|
||||
</system-err>
|
||||
</rerunFailure>
|
||||
<rerunFailure timestamp="2026-10-01T03:24:55.695+00:00" time="0.016" message="thread 'tests::always_fails' (191032) panicked at src/lib.rs:22:67" type="test failure with exit code 101">thread 'tests::always_fails' (191032) panicked at src/lib.rs:22:67:
|
||||
nope
|
||||
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
|
||||
<system-out>
|
||||
running 1 test
|
||||
test tests::always_fails ... FAILED
|
||||
|
||||
failures:
|
||||
|
||||
failures:
|
||||
tests::always_fails
|
||||
|
||||
test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 3 filtered out; finished in 0.01s
|
||||
|
||||
</system-out>
|
||||
<system-err>
|
||||
thread 'tests::always_fails' (191032) panicked at src/lib.rs:22:67:
|
||||
nope
|
||||
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
|
||||
</system-err>
|
||||
</rerunFailure>
|
||||
<system-out>
|
||||
running 1 test
|
||||
test tests::always_fails ... FAILED
|
||||
|
||||
failures:
|
||||
|
||||
failures:
|
||||
tests::always_fails
|
||||
|
||||
test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 3 filtered out; finished in 0.00s
|
||||
|
||||
</system-out>
|
||||
<system-err>
|
||||
thread 'tests::always_fails' (191023) panicked at src/lib.rs:22:67:
|
||||
nope
|
||||
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
|
||||
</system-err>
|
||||
</testcase>
|
||||
</testsuite>
|
||||
</testsuites>
|
||||
@@ -0,0 +1,53 @@
|
||||
<?xml version="1.0" encoding="UTF-8"?>
|
||||
<testsuites name="nextest-run" tests="4" skipped="0" failures="0" errors="0" uuid="3a0fa717-c76b-4698-8d37-2a9d18fce812" timestamp="2026-10-01T03:24:08.148+00:00" time="0.029">
|
||||
<testsuite name="flakydemo" tests="4" skipped="0" errors="0" failures="0">
|
||||
<testcase name="tests::always_fails" classname="flakydemo" timestamp="2026-10-01T03:24:08.148+00:00" time="0.012"/>
|
||||
<testcase name="tests::steady" classname="flakydemo" timestamp="2026-10-01T03:24:08.148+00:00" time="0.013"/>
|
||||
<testcase name="tests::flips_once" classname="flakydemo" timestamp="2026-10-01T03:24:08.162+00:00" time="0.014">
|
||||
<flakyFailure timestamp="2026-10-01T03:24:08.148+00:00" time="0.013" message="thread 'tests::flips_once' (174785) panicked at src/lib.rs:10:13" type="test failure with exit code 101">thread 'tests::flips_once' (174785) panicked at src/lib.rs:10:13:
|
||||
first attempt fails
|
||||
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
|
||||
<system-out>
|
||||
running 1 test
|
||||
test tests::flips_once ... FAILED
|
||||
|
||||
failures:
|
||||
|
||||
failures:
|
||||
tests::flips_once
|
||||
|
||||
test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 3 filtered out; finished in 0.00s
|
||||
|
||||
</system-out>
|
||||
<system-err>
|
||||
thread 'tests::flips_once' (174785) panicked at src/lib.rs:10:13:
|
||||
first attempt fails
|
||||
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
|
||||
</system-err>
|
||||
</flakyFailure>
|
||||
</testcase>
|
||||
<testcase name="tests::also_flips" classname="flakydemo" timestamp="2026-10-01T03:24:08.168+00:00" time="0.009">
|
||||
<flakyFailure timestamp="2026-10-01T03:24:08.148+00:00" time="0.019" message="thread 'tests::also_flips' (174790) panicked at src/lib.rs:18:13" type="test failure with exit code 101">thread 'tests::also_flips' (174790) panicked at src/lib.rs:18:13:
|
||||
first attempt fails <&> "quoted"
|
||||
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
|
||||
<system-out>
|
||||
running 1 test
|
||||
test tests::also_flips ... FAILED
|
||||
|
||||
failures:
|
||||
|
||||
failures:
|
||||
tests::also_flips
|
||||
|
||||
test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 3 filtered out; finished in 0.00s
|
||||
|
||||
</system-out>
|
||||
<system-err>
|
||||
thread 'tests::also_flips' (174790) panicked at src/lib.rs:18:13:
|
||||
first attempt fails <&> "quoted"
|
||||
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
|
||||
</system-err>
|
||||
</flakyFailure>
|
||||
</testcase>
|
||||
</testsuite>
|
||||
</testsuites>
|
||||
@@ -0,0 +1,98 @@
|
||||
#!/bin/bash
|
||||
# ── Fixture tests for check-nextest-flaky.sh ────────────────────────────────
|
||||
# The checker only runs on GitHub, after the ci-profile nextest steps, and a
|
||||
# flaky test there is rare: a checker that stopped seeing them would stay quiet
|
||||
# for months and look exactly like a healthy one. These fixtures pin its
|
||||
# behaviour instead. Each is a real JUnit report from cargo-nextest 0.9.146
|
||||
# under this repository's ci profile (retries = 2), run over a four-test crate:
|
||||
#
|
||||
# clean.xml every test passed on its first attempt
|
||||
# flaky.xml two tests failed once and passed on retry (FLAKY 2/3)
|
||||
# failed.xml one test failed all three attempts (<failure>, <rerunFailure>)
|
||||
#
|
||||
# The flaky fixture must produce one warning per flaky test and a step-summary
|
||||
# entry for each; the clean and failed fixtures must produce none, since a test
|
||||
# that never passed is the nextest step's red, not a flake. A missing report
|
||||
# must not read as clean.
|
||||
#
|
||||
# Exit 0 = every case behaved. Exit 1 = a case did not.
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
set -uo pipefail
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
|
||||
CHECK="$SCRIPT_DIR/../check-nextest-flaky.sh"
|
||||
WORK="$(mktemp -d)"
|
||||
trap 'rm -rf "$WORK"' EXIT
|
||||
|
||||
FAILED=0
|
||||
ok() { echo " ok $*"; }
|
||||
bad() { echo " FAIL $*"; FAILED=$((FAILED + 1)); }
|
||||
|
||||
# expect <description of a pass> <description of a failure> <command...>:
|
||||
# records ok when the command succeeds and a failure otherwise.
|
||||
expect() {
|
||||
local good="$1" poor="$2"
|
||||
shift 2
|
||||
if "$@"; then ok "$good"; else bad "$poor"; fi
|
||||
return 0
|
||||
}
|
||||
|
||||
# run_case <fixture path>: runs the checker with a fresh step summary, leaving
|
||||
# its output in $WORK/out, the summary in $WORK/summary and its status in RC.
|
||||
run_case() {
|
||||
: > "$WORK/summary"
|
||||
GITHUB_STEP_SUMMARY="$WORK/summary" bash "$CHECK" "$1" > "$WORK/out" 2>&1
|
||||
RC=$?
|
||||
WARNINGS="$(grep -c '^::warning title=Flaky test::' "$WORK/out")"
|
||||
return 0
|
||||
}
|
||||
|
||||
echo "check-nextest-flaky fixtures"
|
||||
|
||||
run_case "$SCRIPT_DIR/flaky.xml"
|
||||
expect "flaky: exit 0, so a flake does not red the run" "flaky: exit $RC, expected 0" \
|
||||
test "$RC" -eq 0
|
||||
expect "flaky: one warning per flaky test" "flaky: $WARNINGS warning(s), expected 2" \
|
||||
test "$WARNINGS" -eq 2
|
||||
for t in tests::flips_once tests::also_flips; do
|
||||
if grep -q "^::warning title=Flaky test::flakydemo $t failed 1 attempt(s)" "$WORK/out" \
|
||||
&& grep -q "flakydemo $t\`: failed 1 attempt(s)" "$WORK/summary"; then
|
||||
ok "flaky: $t named in a warning and in the step summary"
|
||||
else
|
||||
bad "flaky: $t missing from the warnings or the step summary"
|
||||
fi
|
||||
done
|
||||
if grep -q 'tests::steady\|tests::always_fails' "$WORK/out" "$WORK/summary"; then
|
||||
bad "flaky: a test that passed first time was reported"
|
||||
else
|
||||
ok "flaky: tests that passed first time are not reported"
|
||||
fi
|
||||
|
||||
run_case "$SCRIPT_DIR/clean.xml"
|
||||
expect "clean: exit 0, no warning, no summary" \
|
||||
"clean: exit $RC, $WARNINGS warning(s), summary $(wc -c < "$WORK/summary") bytes" \
|
||||
test "$RC" -eq 0 -a "$WARNINGS" -eq 0 -a ! -s "$WORK/summary"
|
||||
expect "clean: all four cases were read" "clean: did not report reading four cases" \
|
||||
grep -q '4 test case(s), none passed only on retry' "$WORK/out"
|
||||
|
||||
run_case "$SCRIPT_DIR/failed.xml"
|
||||
expect "failed: a test that never passed is not reported as flaky" \
|
||||
"failed: exit $RC, $WARNINGS warning(s)" \
|
||||
test "$RC" -eq 0 -a "$WARNINGS" -eq 0
|
||||
|
||||
run_case "$WORK/no-such-report.xml"
|
||||
expect "missing report: exit 2, not a clean pass" "missing report: exit $RC, expected 2" \
|
||||
test "$RC" -eq 2 -a "$WARNINGS" -eq 0
|
||||
|
||||
echo '<?xml version="1.0" encoding="UTF-8"?><testsuites name="nextest-run" tests="0"/>' > "$WORK/empty.xml"
|
||||
run_case "$WORK/empty.xml"
|
||||
expect "report with no test cases: exit 2, not a clean pass" \
|
||||
"report with no test cases: exit $RC, expected 2" \
|
||||
test "$RC" -eq 2
|
||||
|
||||
if [[ "$FAILED" -ne 0 ]]; then
|
||||
echo "check-nextest-flaky fixtures: $FAILED case(s) failed"
|
||||
exit 1
|
||||
fi
|
||||
echo "check-nextest-flaky fixtures: all cases passed"
|
||||
exit 0
|
||||
Reference in New Issue
Block a user