mirror of
https://github.com/jmcorgan/fips.git
synced 2026-10-05 11:08:25 +00:00
fr34aky
eaca285ad2
The suite pinned three fixed /24s — 172.31.60, .61 and .62. Compose
project names are unique per run, so two concurrent runs got distinct
container and network *names*, but the address pools are constants and
both runs asked for the same ones. Whichever created a network first
won; the other died at topology start with
failed to create network ..._mc-far: invalid pool request:
Pool overlaps with other one on this address space
having tested nothing. Pushing maint, master and next within a second of
each other is enough to hit it, and the branch that loses looks broken
when it is fine.
The three `MC_*_PREFIX` overrides existed from the start but nothing
ever set them, so the defaults were the only values ever used.
This does what the nat suite already does. A free /24 per network is
claimed under 10.42.0.0/16 before the lab starts, with docker's own
`network create` as the atomic arbiter of who owns what — the run-id
derived offset is deliberately not used here for the same reason it was
rejected there: it makes a collision unlikely rather than impossible,
and a collision is the failure being removed. Both CI labels are stamped
so ci-cleanup.sh's label sweep recovers the networks when a run is
SIGKILLed, which no inline removal can cover.
Since the claim creates the networks, compose has to attach rather than
create, so `docker-compose.external-net.yml` declares the three
external, applied through a new `MC_EXTRA_COMPOSE` hook. Nothing else
sets it: the GitHub matrix runs one job per runner and invokes
`test.sh` directly, and a bare `docker compose up` is a single lab, so
both keep the fixed defaults and the addresses in the README stay
literal. The hook appends to the whole COMPOSE array rather than to the
`up` alone, so teardown addresses the same project — a `down` without
the overlay would not know the networks are external.
The claim lives in ci-local.sh rather than the suite script for the
reason the nat comment gives: the workflow invokes the script directly
and tears down with the base file only, and does not want a claim.
Release does a `compose down` before removing the networks. That order
is load-bearing rather than tidy: `docker network rm` silently no-ops on
a network that still has endpoints attached and reports success, so
without the `down` the removal fails exactly on the path it exists for.
A network left behind would not merely leak — the next invocation in the
run would hit `network with name ... already exists`, which is not a
pool overlap, so the allocator correctly refuses to advance and fails.
Also gives the suite its own compose project. It set none, so it
inherited whichever COMPOSE_PROJECT_NAME the previous suite exported —
which is why the failure named a medium-change network under the nat
project, `fipsci_<runid>_nat_mc-far`. That is not what caused the
overlap, but filing one suite's resources under another's project is a
teardown hazard: `down --remove-orphans` on either would consider the
other's containers orphans.
The /24 claim loop is now shared rather than copied a third time.
`ci_claim_nat_net` keeps its name, its `[nat]` log tag and its exported
prefix, and becomes a two-line caller.
Verified rather than assumed. The lab runs on the claimed prefixes, not
just alongside them: `node-b re-pinned to 10.42.1.10:2121`. With the
first three candidates occupied by squatter networks — the collision
path itself — the allocator advances to 10.42.3/4/5 and the suite passes
on those. Run standalone with no overlay it still renders 172.31.6x and
passes 8/8, so the workflow path is untouched. `nat-cone` passes and
still claims 10.41.0/1, so the shared loop did not disturb it. Networks
are gone after each run. `ci-local.sh --only medium-change` is 17/17.
Not exercised: the partial-claim rollback, which needs a /16 exhausted
part-way to reach. It mirrors ci_claim_nat_networks' shape.
96 lines
4.2 KiB
Markdown
96 lines
4.2 KiB
Markdown
# Transport-Medium Change Lab
|
|
|
|
A node whose network attachment moves under it — WLAN to LAN, Wi-Fi to
|
|
cellular — while its peers stay where they are.
|
|
|
|
```
|
|
node-a ──┬── mc-primary ───┐
|
|
│ ├── router ── mc-far ── node-b
|
|
└── mc-secondary ─┘
|
|
```
|
|
|
|
`node-a` is multi-homed with two equally usable paths to the router. `node-b`
|
|
sits beyond the router and is reachable **only** through it. That last part
|
|
carries the whole design: because `node-b` is off-link, the route to it
|
|
follows `node-a`'s *default* route, which is what the suite moves. Put
|
|
`node-b` on a bridge shared with `node-a` and the directly-connected route
|
|
wins, the source address never changes, and there is nothing left to test.
|
|
|
|
Both of `node-a`'s interfaces stay **up** throughout. Nothing is unplugged.
|
|
The only thing that changes is which of them the default route points at,
|
|
which is what makes this a medium change rather than a link failure — and
|
|
which is why link-state watching alone does not see it.
|
|
|
|
## Running
|
|
|
|
```bash
|
|
./testing/medium-change/scripts/test.sh
|
|
# or
|
|
./testing/ci-local.sh --only medium-change
|
|
```
|
|
|
|
## What it asserts
|
|
|
|
Traffic returning after the move is a weak signal: a peering that was torn
|
|
down by the liveness timeout and rebuilt by a re-dial also ends with traffic
|
|
flowing. The suite therefore checks *continuity*, from `fipsctl show peers` on
|
|
both nodes:
|
|
|
|
| Observation | Meaning |
|
|
| ----------- | ------- |
|
|
| `link_id` unchanged on node-a | the link was never rebuilt |
|
|
| `authenticated_at_ms` unchanged on node-a | no second handshake ran |
|
|
| `transport_addr` changed on node-b | the far side re-pinned to the new source |
|
|
| longest ping gap within budget | the data plane genuinely carried through |
|
|
|
|
The third is what stops the first two from passing vacuously on a topology
|
|
where nothing actually moved.
|
|
|
|
Phases 1 and 2 move the route in each direction, since the two are not
|
|
symmetric — one direction leaves the old interface holding an address the
|
|
routing table has abandoned, the other returns to it.
|
|
|
|
## The negative control
|
|
|
|
Phase 3 repeats the move with `node.netmon.enabled: false` and **requires**
|
|
the outage. If traffic survives with detection off, this topology is not
|
|
exercising the code path and every assertion above is vacuous — so the suite
|
|
fails rather than passing quietly.
|
|
|
|
This is deliberate. A regression test that has never been seen to fail is a
|
|
claim, not a test, and the claim is cheap to make and expensive to trust.
|
|
|
|
## Knobs
|
|
|
|
| Variable | Default | Meaning |
|
|
| -------- | ------- | ------- |
|
|
| `MC_MAX_GAP_SECS` | `5` | longest tolerated break in traffic across a move |
|
|
| `MC_CONTROL_DARK_SECS` | `12` | how long the control must stay dark |
|
|
| `MC_PRIMARY_PREFIX` | `172.31.60` | first access path `/24` |
|
|
| `MC_SECONDARY_PREFIX` | `172.31.61` | second access path `/24` |
|
|
| `MC_FAR_PREFIX` | `172.31.62` | far segment `/24` |
|
|
| `MC_EXTRA_COMPOSE` | unset | extra compose overlays, colon-separated |
|
|
|
|
The gap budget sits far below the 30 s liveness timeout on purpose: a pass
|
|
must mean the move was absorbed, not that the reaper was quick.
|
|
|
|
## Running two of these at once
|
|
|
|
The three prefixes above are fixed defaults, and two runs that both take them
|
|
do not both get them: docker refuses the second with `Pool overlaps with other
|
|
one on this address space`, and the suite fails at topology start having tested
|
|
nothing. The compose project name is unique per run, so the containers and the
|
|
networks get distinct *names* — it is only the address pools that are shared.
|
|
|
|
`testing/ci-local.sh` avoids that by claiming a free `/24` per network under
|
|
`10.42.0.0/16` before it starts the lab, letting docker's own `network create`
|
|
be the arbiter of who owns what, and pointing compose at the result with
|
|
`docker-compose.external-net.yml` via `MC_EXTRA_COMPOSE`. It exports the
|
|
claimed prefixes as the three `MC_*_PREFIX` variables, so everything the suite
|
|
renders from them follows.
|
|
|
|
Nothing else applies that overlay. The GitHub matrix runs one job per runner
|
|
and the README invocation above is a single lab, so both keep the fixed
|
|
defaults and the addresses in this document stay literal. If you want to run
|
|
two by hand on one host, set the three prefixes yourself.
|