mirror of
https://github.com/jmcorgan/fips.git
synced 2026-10-05 11:08:25 +00:00
fr34akyandJohnathan Corgan
b922568dca
An established UDP peer gets its own `connect()`-ed socket for the send fast path. `open_connected_fd` binds the wildcard and then calls `connect(2)`, which makes the kernel resolve the route once and auto-bind the local source address to whichever interface was carrying it at that moment. It never re-evaluates. So after the host changed transport medium — a laptop between WLAN and LAN, a phone between Wi-Fi and cellular — every established peer went on transmitting from an address the routing table had abandoned. The peer, which re-pins to whatever address it last heard from, answered somewhere the node was no longer sending from. The peering stayed marked connected and carried nothing until `link_dead_timeout_secs` tore it down: 60-90s of black-holed traffic per switch on a live node, then a full re-handshake and tree re-convergence. The mirror-image case, the peer rotating its address, was already handled where the rotation is observed. This is the local half, and it had no signal to hang off, because a local move is invisible in the data plane. Medium-change detection supplies that signal. `node.netmon.*` controls it and it is on by default. The node samples a coarse fingerprint of its network attachment — the source addresses the routing table would pick for an off-link destination, plus the set of up, non-loopback interface addresses — and reports a change once the picture settles. A handover is not atomic, so a short debounce coalesces the burst into one event, and a fingerprint that settles back where it started reports nothing. Linux and Android subscribe to `NETLINK_ROUTE` multicast and macOS and FreeBSD to a `PF_ROUTE` socket, both reacting in milliseconds; every other platform samples on a timer, which also runs underneath the kernel sources as a backstop. A backend decides only when to look, so the remaining ones land behind the same seam. The reaction is two steps. Drop the stale connected sockets, which is self-healing rather than disruptive: the wildcard listen socket resolves a route per packet, so sends keep working immediately, and a correctly-bound socket is reinstalled on a later tick. Then heartbeat every peer whose send path cannot block, so the far side re-pins at once rather than waiting out its own interval. That filter is the whole point rather than an optimisation. A connectionless send completes without awaiting the wire. A connection-oriented one awaits an unbounded `write_all` on a stream that the medium change has very likely just stranded, and this reaction runs on the rx loop, so it would hold every other arm of the select for as long as that socket took to fail. A peer on such a transport keeps the periodic heartbeat it had before, with `link_dead_timeout_secs` as the backstop. Covered by unit tests, by a regression test that pins the fan-out filter, and by a new `medium-change` integration suite: a multi-homed node whose default route moves between two live access paths while mesh traffic is in flight, with the far peer off-link behind a router. The changelog entries land under Unreleased rather than in the released `0.5.1` section, since none of this is in that release.
75 lines
2.7 KiB
Bash
Executable File
75 lines
2.7 KiB
Bash
Executable File
#!/bin/bash
|
|
# Pin this node's routing before the daemon starts.
|
|
#
|
|
# Interfaces are resolved by the subnet they carry, never by name. Docker
|
|
# assigns eth0/eth1 in an order that is not the order the networks appear in
|
|
# compose, so a name-based rule silently binds the wrong path on some hosts
|
|
# and the suite then measures nothing — the default route would already be on
|
|
# the interface the test is about to "switch" to.
|
|
set -euo pipefail
|
|
|
|
WAIT_TIMEOUT_SECS="${WAIT_TIMEOUT_SECS:-30}"
|
|
# "<prefix>=<router-host-octet>" for every path this node sits on.
|
|
PRIMARY_PREFIX="${PRIMARY_PREFIX:-}"
|
|
SECONDARY_PREFIX="${SECONDARY_PREFIX:-}"
|
|
FAR_PREFIX="${FAR_PREFIX:-}"
|
|
ROUTER_OCTET="${ROUTER_OCTET:-254}"
|
|
# Which path the default route starts on: primary, secondary, or far.
|
|
DEFAULT_VIA="${DEFAULT_VIA:-primary}"
|
|
|
|
iface_for_subnet() {
|
|
local prefix="$1"
|
|
ip -4 -oneline addr show \
|
|
| awk -v p="${prefix}." '$4 ~ "^"p {print $2; exit}'
|
|
}
|
|
|
|
wait_for_subnet() {
|
|
local prefix="$1" name="$2" deadline=$((SECONDS + WAIT_TIMEOUT_SECS))
|
|
while [ "$SECONDS" -lt "$deadline" ]; do
|
|
if [ -n "$(iface_for_subnet "$prefix")" ]; then
|
|
return 0
|
|
fi
|
|
sleep 0.5
|
|
done
|
|
echo "Timed out waiting for an address on ${prefix}.0/24 (${name})" >&2
|
|
ip -4 -brief addr show >&2 || true
|
|
return 1
|
|
}
|
|
|
|
ip link set lo up
|
|
|
|
for spec in "primary:$PRIMARY_PREFIX" "secondary:$SECONDARY_PREFIX" "far:$FAR_PREFIX"; do
|
|
name="${spec%%:*}"
|
|
prefix="${spec#*:}"
|
|
[ -n "$prefix" ] || continue
|
|
wait_for_subnet "$prefix" "$name"
|
|
ip link set "$(iface_for_subnet "$prefix")" up
|
|
done
|
|
|
|
# The default route. Docker installs one of its own per attached bridge; on a
|
|
# multi-homed container which one wins is not something the suite can depend
|
|
# on, so it is replaced outright rather than adjusted.
|
|
case "$DEFAULT_VIA" in
|
|
primary) via_prefix="$PRIMARY_PREFIX" ;;
|
|
secondary) via_prefix="$SECONDARY_PREFIX" ;;
|
|
far) via_prefix="$FAR_PREFIX" ;;
|
|
*) echo "Unknown DEFAULT_VIA: $DEFAULT_VIA" >&2; exit 1 ;;
|
|
esac
|
|
via_if="$(iface_for_subnet "$via_prefix")"
|
|
# The default route, and deliberately nothing more specific.
|
|
#
|
|
# Every off-link segment in this lab is reachable through the router, so the
|
|
# default covers them all. Adding a per-subnet route as well would be worse
|
|
# than redundant: a /24 to the far segment outranks the default, so moving the
|
|
# default would leave the path to the far node exactly where it was. The suite
|
|
# would then detect a medium change, drop the sockets, and assert against a
|
|
# peer that never actually moved.
|
|
ip route replace default via "${via_prefix}.${ROUTER_OCTET}" dev "$via_if"
|
|
|
|
echo "node: addresses"
|
|
ip -4 -brief addr show | sed 's/^/ /'
|
|
echo "node: routes"
|
|
ip -4 route | sed 's/^/ /'
|
|
|
|
exec /usr/local/bin/entrypoint.sh
|