mirror of
https://github.com/jmcorgan/fips.git
synced 2026-10-05 19:18:25 +00:00
fr34akyandJohnathan Corgan
b922568dca
An established UDP peer gets its own `connect()`-ed socket for the send fast path. `open_connected_fd` binds the wildcard and then calls `connect(2)`, which makes the kernel resolve the route once and auto-bind the local source address to whichever interface was carrying it at that moment. It never re-evaluates. So after the host changed transport medium — a laptop between WLAN and LAN, a phone between Wi-Fi and cellular — every established peer went on transmitting from an address the routing table had abandoned. The peer, which re-pins to whatever address it last heard from, answered somewhere the node was no longer sending from. The peering stayed marked connected and carried nothing until `link_dead_timeout_secs` tore it down: 60-90s of black-holed traffic per switch on a live node, then a full re-handshake and tree re-convergence. The mirror-image case, the peer rotating its address, was already handled where the rotation is observed. This is the local half, and it had no signal to hang off, because a local move is invisible in the data plane. Medium-change detection supplies that signal. `node.netmon.*` controls it and it is on by default. The node samples a coarse fingerprint of its network attachment — the source addresses the routing table would pick for an off-link destination, plus the set of up, non-loopback interface addresses — and reports a change once the picture settles. A handover is not atomic, so a short debounce coalesces the burst into one event, and a fingerprint that settles back where it started reports nothing. Linux and Android subscribe to `NETLINK_ROUTE` multicast and macOS and FreeBSD to a `PF_ROUTE` socket, both reacting in milliseconds; every other platform samples on a timer, which also runs underneath the kernel sources as a backstop. A backend decides only when to look, so the remaining ones land behind the same seam. The reaction is two steps. Drop the stale connected sockets, which is self-healing rather than disruptive: the wildcard listen socket resolves a route per packet, so sends keep working immediately, and a correctly-bound socket is reinstalled on a later tick. Then heartbeat every peer whose send path cannot block, so the far side re-pins at once rather than waiting out its own interval. That filter is the whole point rather than an optimisation. A connectionless send completes without awaiting the wire. A connection-oriented one awaits an unbounded `write_all` on a stream that the medium change has very likely just stranded, and this reaction runs on the rx loop, so it would hold every other arm of the select for as long as that socket took to fail. A peer on such a transport keeps the periodic heartbeat it had before, with `link_dead_timeout_secs` as the backstop. Covered by unit tests, by a regression test that pins the fan-out filter, and by a new `medium-change` integration suite: a multi-homed node whose default route moves between two live access paths while mesh traffic is in flight, with the far peer off-link behind a router. The changelog entries land under Unreleased rather than in the released `0.5.1` section, since none of this is in that release.
75 lines
3.0 KiB
Markdown
75 lines
3.0 KiB
Markdown
# Transport-Medium Change Lab
|
|
|
|
A node whose network attachment moves under it — WLAN to LAN, Wi-Fi to
|
|
cellular — while its peers stay where they are.
|
|
|
|
```
|
|
node-a ──┬── mc-primary ───┐
|
|
│ ├── router ── mc-far ── node-b
|
|
└── mc-secondary ─┘
|
|
```
|
|
|
|
`node-a` is multi-homed with two equally usable paths to the router. `node-b`
|
|
sits beyond the router and is reachable **only** through it. That last part
|
|
carries the whole design: because `node-b` is off-link, the route to it
|
|
follows `node-a`'s *default* route, which is what the suite moves. Put
|
|
`node-b` on a bridge shared with `node-a` and the directly-connected route
|
|
wins, the source address never changes, and there is nothing left to test.
|
|
|
|
Both of `node-a`'s interfaces stay **up** throughout. Nothing is unplugged.
|
|
The only thing that changes is which of them the default route points at,
|
|
which is what makes this a medium change rather than a link failure — and
|
|
which is why link-state watching alone does not see it.
|
|
|
|
## Running
|
|
|
|
```bash
|
|
./testing/medium-change/scripts/test.sh
|
|
# or
|
|
./testing/ci-local.sh --only medium-change
|
|
```
|
|
|
|
## What it asserts
|
|
|
|
Traffic returning after the move is a weak signal: a peering that was torn
|
|
down by the liveness timeout and rebuilt by a re-dial also ends with traffic
|
|
flowing. The suite therefore checks *continuity*, from `fipsctl show peers` on
|
|
both nodes:
|
|
|
|
| Observation | Meaning |
|
|
| ----------- | ------- |
|
|
| `link_id` unchanged on node-a | the link was never rebuilt |
|
|
| `authenticated_at_ms` unchanged on node-a | no second handshake ran |
|
|
| `transport_addr` changed on node-b | the far side re-pinned to the new source |
|
|
| longest ping gap within budget | the data plane genuinely carried through |
|
|
|
|
The third is what stops the first two from passing vacuously on a topology
|
|
where nothing actually moved.
|
|
|
|
Phases 1 and 2 move the route in each direction, since the two are not
|
|
symmetric — one direction leaves the old interface holding an address the
|
|
routing table has abandoned, the other returns to it.
|
|
|
|
## The negative control
|
|
|
|
Phase 3 repeats the move with `node.netmon.enabled: false` and **requires**
|
|
the outage. If traffic survives with detection off, this topology is not
|
|
exercising the code path and every assertion above is vacuous — so the suite
|
|
fails rather than passing quietly.
|
|
|
|
This is deliberate. A regression test that has never been seen to fail is a
|
|
claim, not a test, and the claim is cheap to make and expensive to trust.
|
|
|
|
## Knobs
|
|
|
|
| Variable | Default | Meaning |
|
|
| -------- | ------- | ------- |
|
|
| `MC_MAX_GAP_SECS` | `5` | longest tolerated break in traffic across a move |
|
|
| `MC_CONTROL_DARK_SECS` | `12` | how long the control must stay dark |
|
|
| `MC_PRIMARY_PREFIX` | `172.31.60` | first access path `/24` |
|
|
| `MC_SECONDARY_PREFIX` | `172.31.61` | second access path `/24` |
|
|
| `MC_FAR_PREFIX` | `172.31.62` | far segment `/24` |
|
|
|
|
The gap budget sits far below the 30 s liveness timeout on purpose: a pass
|
|
must mean the move was absorbed, not that the reaper was quick.
|