mirror of
https://github.com/jmcorgan/fips.git
synced 2026-07-30 19:46:15 +00:00
data-plane perf overhaul: off-task encrypt + decrypt, GSO, connected UDP
Moves both AEAD layers (ChaCha20-Poly1305, one round per layer per packet) plus the sendmsg syscall off the rx_loop task onto a per-shard worker pool, adds per-peer connect(2)-ed UDP with SO_REUSEPORT, and uses Linux UDP GSO (sendmsg+UDP_SEGMENT — kernel splits one super-skb into N on-the-wire datagrams in a single TX-stack walk) when packets in a batch are uniform-size. Same kernel primitive WireGuard's in-kernel module and BoringTun use to hit 2.5–3.2 Gbps single-stream. Single TCP stream on a 5-node docker-bridge mesh, 5 x 15 s x P=1: A→D: 1379 → 2708 Mbps (1.96x, RTT +0.12 ms) A→E: 1394 → 2663 Mbps (1.91x, RTT +0.11 ms) E→A: 1406 → 2624 Mbps (1.87x, RTT +0.19 ms) Static-peer pairs only — every CoV under 3%, 0 outliers, 0% ICMP loss. The ~+100 µs RTT is the worker queue handoff cost; AEAD + sendmmsg now run on a separate core in exchange. What lands: - src/node/encrypt_worker.rs: std::thread + crossbeam_channel workers; hash-by-destination dispatch pins a TCP flow to one worker so wire ordering is preserved; per-worker sendmmsg(2) batching up to 32; Linux uses sendmsg(2)+UDP_SEGMENT when packets in a group are uniform-size. - src/node/decrypt_worker.rs: receive-side mirror. Each shard owns its session's recv cipher + replay window in a thread-local HashMap (no shared RwLock/Mutex). Sessions are handed off at promote_connection and re-registered on K-bit flip / rekey cutover. - src/node/handlers/session.rs try_send_session_data_pipelined: FSP+FMP both seal in-place in the worker on one wire-buffer alloc; no intermediate inner_plaintext / fsp_payload Vecs. - src/transport/udp/connected_peer.rs + peer_drain.rs: per-peer connect(2)-ed UDP socket with SO_REUSEPORT (set on the listen socket too — without that, EADDRINUSE on activation and every packet falls back to the wildcard path); the worker sends with msg_name=NULL and the kernel uses its cached 5-tuple. Tick- driven activation in handlers/connected_udp.rs, idempotent. - src/transport/udp/mod.rs: mem::replace the recvmmsg backing buffer instead of buf.to_vec() per packet — single pointer swap, no MTU-sized memcpy. - src/protocol/link.rs SessionDatagramRef: zero-copy borrowed view used by handle_session_datagram for the bulk local-delivery path; handle_session_payload takes the borrowed payload directly (no payload[35..].to_vec()). - src/transport/mod.rs TransportAddr::from_socket_addr: collapses the two-alloc from_string(addr.to_string()) pattern to one. - src/node/handlers/rx_loop.rs: decrypt-fallback drain promoted ahead of packet_rx in the select! (TCP ACK starvation fix); interleaved fallback drain every 32 packets inside the rx burst loop. - noise::Session: send_cipher_clone / recv_cipher_clone / recv_replay_snapshot_owned / take_send_counter / accept_replay so off-task workers can hold a cloned cipher + reserved counter while the dispatcher keeps replay/counter sequencing serial. CipherState::cipher_clone returns a refcount-bumped LessSafeKey. AsyncUdpSocket: AsRawFd so workers issue raw sendmmsg / sendmsg without going through the tokio reactor. - Worker pool sizing: both default to num_cpus, overridable via FIPS_ENCRYPT_WORKERS=N / FIPS_DECRYPT_WORKERS=N. Per-peer connected UDP can be disabled via FIPS_CONNECTED_UDP=0. - src/perf_profile.rs: optional per-stage timing reporter under FIPS_PERF=1 (or FIPS_PIPELINE_TRACE=1). Off by default; zero overhead when disabled. - All cfg(unix)-gated. Windows continues on the existing tokio- based send/recv. Decrypt worker session lifecycle: - Node::unregister_decrypt_worker_session mirrors the existing register helper. Wired at the two natural sites that already iterate peers_by_index: the rekey drain-completion block in handlers/rekey.rs (drops the worker entry for the old our_index once the drain window has expired and the cache_key is unreachable to any in-flight OLD-K packet), and remove_active_peer in handlers/dispatch.rs (drops the worker entry for each of the four index slots: current, rekey, pending, previous). Only our_index is normally registered; unregister_session is fire- and-forget for missing entries, so calling unconditionally on all four slots is correct and bounds the cleanup without per- slot accounting. Without these callers the per-worker sessions HashMap and the Node's decrypt_registered_sessions set would grow monotonically per rekey on long-lived peers. Testing: - testing/static/scripts/bench-multirun.sh: multi-run iperf3 + ping bench. N reruns (default 5), median / min / max / CoV % / per-run outlier flag, avg ping RTT, ICMP loss %, TCP retransmit total. Plain client→dest labels + topology header. Pre-bench peer-convergence check (FIPS_BENCH_CONVERGE_SECS, default 15); per-path route verification via stats.bytes_sent deltas — fails fast if traffic exits via a non-static-peer link. - testing/static/docker-compose.yml: passes FIPS_ENCRYPT_WORKERS / FIPS_DECRYPT_WORKERS / FIPS_PERF through to containers for A/B benchmarking without rebuilds. - testing/static/scripts/iperf-test.sh: same plain client→dest labels + topology header (was multihop/direct/N hop, which conflated topology distance with on-wire path). - .config/nextest.toml: synthetic UDP node tests serialized through a max-threads=1 test group. Localhost handshakes drop on shared CI runners under parallel load; one-at-a-time keeps assertions reliable. - src/node/tests/spanning_tree.rs: repair_missing_edge_handshakes — retries up to 5 times for synthetic edges whose msg1 was dropped, with a drain after each edge retry instead of after each attempt's full burst. - src/node/decrypt_worker.rs::tests: two unit tests asserting WorkerMsg::UnregisterSession removes the worker-thread session HashMap entry (handle_msg_unregister_session_removes_entry) and is a no-op for never-seen cache_keys (handle_msg_unregister_session_idempotent_on_unknown_key), which is the safety invariant the unconditional unregister calls at the four index slots in remove_active_peer rely on. - src/node/encrypt_worker.rs::unix_tests pipelined_send_wire_layout_roundtrips_canonical_decoders: mirrors the encoder geometry of try_send_session_data_pipelined (no coords, the common established-session path), runs the worker's real seal + send via flush_direct_batch_sync, and decodes the resulting wire packet using only canonical receive-side decoders (EncryptedHeader::parse, SessionDatagramRef::decode, FSP header parse, noise::open). Any divergence between the hand-rolled encoder offsets (fsp_aad_offset, fsp_plaintext_offset) and the decoders fails at one of the parse / open / decode steps before the inner-plaintext assertion fires. Complements the existing fsp_preseal_runs_before_outer_fmp_seal test which covers the seal-ordering invariant with synthetic headers but does not exercise the wire-layout invariant. CHANGELOG.md [Unreleased] # Changed entry added describing the worker-pool threading model, hash-by-destination dispatch, sendmmsg/UDP_GSO, per-peer connected UDP, the operator-facing env vars, and the bench numbers above. Cherry-picks from mmalmi/master (paths translated from crates/fips-core/src/ to src/): 9b7c723, 0deb5cb, 13f7339, e036c0e, 3740a68, 3792f83, 8510193, 4910b07, e53f545, e4e2896, 5fe4af5, 1d01ada, 8c37008, e12469e, 6eb2860. Co-authored-by: Johnathan Corgan <johnathan@corganlabs.com>
This commit is contained in:
Martti Malmi
committed by
Johnathan Corgan
Johnathan Corgan
co-authored by
Johnathan Corgan
Johnathan Corgan
parent
6e7e44c8ff
commit
0a5c367edc
@@ -25,6 +25,12 @@ x-fips-common: &fips-common
|
||||
- ./generated-configs/npubs.env
|
||||
environment:
|
||||
- RUST_LOG=info
|
||||
# Passthrough for A/B benchmarking — set on the host shell before
|
||||
# `docker compose up` to override the worker-pool defaults.
|
||||
- FIPS_ENCRYPT_WORKERS=${FIPS_ENCRYPT_WORKERS:-}
|
||||
- FIPS_DECRYPT_WORKERS=${FIPS_DECRYPT_WORKERS:-}
|
||||
- FIPS_PERF=${FIPS_PERF:-0}
|
||||
- FIPS_PERF_INTERVAL_SECS=${FIPS_PERF_INTERVAL_SECS:-5}
|
||||
|
||||
services:
|
||||
# ── Mesh topology ──────────────────────────────────────────────
|
||||
|
||||
Executable
+427
@@ -0,0 +1,427 @@
|
||||
#!/bin/bash
|
||||
# Multi-run iperf3 + ping bench for FIPS mesh. Repeats each test N times
|
||||
# (default 3), reports median + min/max + flags outliers >20% from
|
||||
# median. Adds round-trip latency (ping) and a TCP-retransmit count
|
||||
# (proxy for packet loss across the FIPS overlay) per path.
|
||||
#
|
||||
# Usage:
|
||||
# ./bench-multirun.sh [mesh|chain]
|
||||
#
|
||||
# Environment:
|
||||
# FIPS_BENCH_RUNS=3 Number of iperf3 repetitions per path
|
||||
# FIPS_BENCH_DURATION=15 Seconds per iperf3 run
|
||||
# FIPS_BENCH_PARALLEL=1 Parallel TCP streams (1 = single-stream)
|
||||
# FIPS_BENCH_PING_COUNT=30 ICMP probes per path
|
||||
# FIPS_BENCH_OUTLIER_PCT=20 Flag run as outlier if Δ from median > N%
|
||||
# FIPS_BENCH_OUTPUT=json|text (default text; json emits NDJSON)
|
||||
|
||||
set -e
|
||||
trap 'echo ""; echo "Bench interrupted"; exit 130' INT
|
||||
|
||||
PROFILE="${1:-mesh}"
|
||||
RUNS="${FIPS_BENCH_RUNS:-5}"
|
||||
DURATION="${FIPS_BENCH_DURATION:-15}"
|
||||
PARALLEL="${FIPS_BENCH_PARALLEL:-1}"
|
||||
PING_COUNT="${FIPS_BENCH_PING_COUNT:-30}"
|
||||
OUTLIER_PCT="${FIPS_BENCH_OUTLIER_PCT:-20}"
|
||||
OUTPUT="${FIPS_BENCH_OUTPUT:-text}"
|
||||
|
||||
if ! [[ "$RUNS" =~ ^[1-9][0-9]*$ ]] || [ "$RUNS" -lt 1 ]; then
|
||||
echo "FIPS_BENCH_RUNS must be ≥ 1" >&2; exit 1
|
||||
fi
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
ENV_FILE="$SCRIPT_DIR/../generated-configs/npubs.env"
|
||||
if [ ! -f "$ENV_FILE" ]; then
|
||||
echo "Error: $ENV_FILE not found. Run generate-configs.sh first." >&2
|
||||
exit 1
|
||||
fi
|
||||
# shellcheck source=../generated-configs/npubs.env
|
||||
source "$ENV_FILE"
|
||||
|
||||
# ── Helpers ────────────────────────────────────────────────────────────────
|
||||
|
||||
# Run iperf3 once, print "<mbps> <retransmits>" on success or "FAIL" on
|
||||
# failure. Uses JSON output so we don't have to scrape human-readable
|
||||
# SI prefixes.
|
||||
iperf_once() {
|
||||
local client="$1" dest_npub="$2"
|
||||
local out
|
||||
if ! out=$(docker exec "fips-$client" iperf3 -c "${dest_npub}.fips" \
|
||||
-t "$DURATION" -P "$PARALLEL" -J 2>&1); then
|
||||
echo FAIL
|
||||
return
|
||||
fi
|
||||
python3 - "$out" <<'PYEOF'
|
||||
import json, sys
|
||||
try:
|
||||
d = json.loads(sys.argv[1])
|
||||
bps = d['end']['sum_received']['bits_per_second']
|
||||
# iperf3 reports TCP retransmits in 'sum_sent.retransmits' on
|
||||
# the client side; missing on UDP or for the server-summary view.
|
||||
retr = d.get('end', {}).get('sum_sent', {}).get('retransmits', 0)
|
||||
print(f"{bps/1e6:.2f} {retr}")
|
||||
except Exception as e:
|
||||
print("FAIL")
|
||||
PYEOF
|
||||
}
|
||||
|
||||
# Median + min + max + CoV% + outlier indices over a whitespace-
|
||||
# separated list of floats. CoV% = (stddev / mean) * 100, i.e. the
|
||||
# coefficient of variation in percent — directly comparable across
|
||||
# paths regardless of absolute throughput. Single-value runs print
|
||||
# CoV=0.
|
||||
# Prints: "<median> <min> <max> <cov_pct> <outliers_csv>"
|
||||
stats() {
|
||||
python3 - "$OUTLIER_PCT" "$@" <<'PYEOF'
|
||||
import math
|
||||
import sys
|
||||
pct = float(sys.argv[1])
|
||||
vals = [float(x) for x in sys.argv[2:]]
|
||||
n = len(vals)
|
||||
s = sorted(vals)
|
||||
median = s[n // 2] if n % 2 else (s[n // 2 - 1] + s[n // 2]) / 2
|
||||
lo, hi = s[0], s[-1]
|
||||
mean = sum(vals) / n if n else 0.0
|
||||
if n > 1 and mean > 0:
|
||||
variance = sum((v - mean) ** 2 for v in vals) / (n - 1) # sample stddev
|
||||
cov_pct = math.sqrt(variance) / mean * 100
|
||||
else:
|
||||
cov_pct = 0.0
|
||||
outliers = []
|
||||
if median > 0:
|
||||
for i, v in enumerate(vals):
|
||||
if abs(v - median) / median * 100 > pct:
|
||||
outliers.append(str(i + 1))
|
||||
print(
|
||||
f"{median:.2f} {lo:.2f} {hi:.2f} {cov_pct:.1f}% "
|
||||
f"{','.join(outliers) if outliers else '-'}"
|
||||
)
|
||||
PYEOF
|
||||
}
|
||||
|
||||
# Ping once, print "<min_ms> <avg_ms> <max_ms> <mdev_ms> <loss_pct>"
|
||||
# or "FAIL".
|
||||
ping_path() {
|
||||
local client="$1" dest_npub="$2"
|
||||
local out
|
||||
if ! out=$(docker exec "fips-$client" \
|
||||
ping -c "$PING_COUNT" -i 0.2 -w "$((PING_COUNT * 2))" \
|
||||
-q "${dest_npub}.fips" 2>&1); then
|
||||
echo FAIL
|
||||
return
|
||||
fi
|
||||
# "min/avg/max/mdev = 0.123/0.456/0.789/0.012 ms"
|
||||
local rtt loss
|
||||
rtt=$(echo "$out" | awk -F' = ' '/min\/avg\/max\/mdev/ {print $2}' | awk '{print $1}')
|
||||
loss=$(echo "$out" | awk -F', ' '/packet loss/ {for (i=1;i<=NF;i++) if ($i ~ /packet loss/) print $i}' | awk '{print $1}')
|
||||
if [ -z "$rtt" ]; then
|
||||
echo FAIL
|
||||
return
|
||||
fi
|
||||
echo "${rtt//\// } ${loss:-N/A}"
|
||||
}
|
||||
|
||||
# ── Path definitions ───────────────────────────────────────────────────────
|
||||
#
|
||||
# Each entry is `<client-container> <dest-npub> <label>`. Labels are
|
||||
# plain `client→dest` — the bench measures iperf3 throughput between
|
||||
# the two named nodes. The static topology is printed at start so the
|
||||
# operator can see the configured routing context.
|
||||
#
|
||||
# IMPORTANT — labels DO NOT encode hop count. With peer discovery
|
||||
# enabled and all nodes on the same docker-bridge subnet, every node
|
||||
# pair establishes a direct UDP path within a few ticks regardless of
|
||||
# the static `peers:` list. The bench measures the post-convergence
|
||||
# steady state, which is what real-world FIPS deployments see. To
|
||||
# force on-wire multihop, isolate nodes on distinct docker networks.
|
||||
case "$PROFILE" in
|
||||
mesh|mesh-public)
|
||||
# Static-peer paths only — without mDNS / a Nostr relay in the
|
||||
# test container set, non-adjacent pairs (A↔B, A↔C) can't
|
||||
# establish direct UDP and the bench would either fail
|
||||
# convergence or measure multihop forwarding (apples-to-oranges
|
||||
# vs builds that do have mDNS). Stick to pairs that are in
|
||||
# each other's static `peers:` list so the measurement is
|
||||
# deterministic and the same on every build.
|
||||
PATHS=(
|
||||
"node-a $NPUB_D A→D"
|
||||
"node-a $NPUB_E A→E"
|
||||
"node-e $NPUB_A E→A"
|
||||
)
|
||||
TOPOLOGY_LINES=(
|
||||
"A peers with: D, E"
|
||||
"D peers with: A, C, E"
|
||||
"E peers with: A, C, D"
|
||||
"(Static-peer pairs A↔D, A↔E only — non-adjacent pairs"
|
||||
" are skipped because plain mesh.yaml has no discovery"
|
||||
" transport that converges them onto direct UDP.)"
|
||||
)
|
||||
;;
|
||||
chain)
|
||||
# Chain topology forces intentional multihop forwarding for
|
||||
# non-adjacent pairs: A peers only with B, B peers with A+C,
|
||||
# etc. We bench just A→B (1 hop, direct static peer) so the
|
||||
# comparison is again apples-to-apples without discovery.
|
||||
PATHS=(
|
||||
"node-a $NPUB_B A→B"
|
||||
)
|
||||
TOPOLOGY_LINES=(
|
||||
"Static chain: A — B — C — D — E"
|
||||
"(A↔B is the only static-peer pair we benchmark; multi-hop"
|
||||
" forwarding to C/D/E needs reply-learned routing or mDNS"
|
||||
" to settle, which is not deterministic on a fresh mesh.)"
|
||||
)
|
||||
;;
|
||||
*)
|
||||
echo "Unknown profile: $PROFILE" >&2; exit 1 ;;
|
||||
esac
|
||||
|
||||
# ── Run ────────────────────────────────────────────────────────────────────
|
||||
|
||||
echo "=== FIPS multi-run bench ($PROFILE, ${RUNS}×${DURATION}s, P=${PARALLEL}) ==="
|
||||
echo "Outlier flag: Δ from median > ${OUTLIER_PCT}%"
|
||||
echo ""
|
||||
echo "Topology (static \`peers:\` from $PROFILE.yaml):"
|
||||
for line in "${TOPOLOGY_LINES[@]}"; do
|
||||
echo " $line"
|
||||
done
|
||||
echo ""
|
||||
|
||||
# Wait for peer discovery to converge before measuring — otherwise
|
||||
# the bench randomly measures either direct UDP (post-convergence
|
||||
# steady state) or multihop forwarding through static peers (only
|
||||
# until discovery converges). On the same docker-bridge subnet
|
||||
# convergence should take seconds, not tens of seconds; this is the
|
||||
# tight default. Bump FIPS_BENCH_CONVERGE_SECS for slower test setups
|
||||
# (no Nostr relay, no mDNS, etc.).
|
||||
#
|
||||
# If any path hasn't converged by the deadline, the bench FAILS with
|
||||
# a clear list of unconverged pairs — that's almost always a topology
|
||||
# / discovery misconfiguration that would make the numbers noise.
|
||||
CONVERGE_SECS="${FIPS_BENCH_CONVERGE_SECS:-15}"
|
||||
|
||||
peer_is_direct() {
|
||||
local client="$1" dest_npub="$2"
|
||||
docker exec "fips-$client" fipsctl show peers 2>/dev/null \
|
||||
| python3 -c '
|
||||
import json, sys
|
||||
target = sys.argv[1]
|
||||
try:
|
||||
d = json.load(sys.stdin)
|
||||
for p in d.get("peers", []):
|
||||
if p.get("npub") == target:
|
||||
print("yes" if p.get("connectivity") == "connected" else "no")
|
||||
sys.exit(0)
|
||||
print("no")
|
||||
except Exception:
|
||||
print("no")
|
||||
' "$dest_npub" 2>/dev/null
|
||||
}
|
||||
|
||||
# Per-peer `stats.bytes_sent` snapshot for the given client. Used to
|
||||
# verify that the iperf3 traffic actually went out the intended next-
|
||||
# hop (i.e. the routing protocol picked the static-peer link, not a
|
||||
# multihop alternative).
|
||||
peer_bytes_sent_snapshot() {
|
||||
local client="$1"
|
||||
docker exec "fips-$client" fipsctl show peers 2>/dev/null \
|
||||
| python3 -c '
|
||||
import json, sys
|
||||
try:
|
||||
d = json.load(sys.stdin)
|
||||
for p in d.get("peers", []):
|
||||
npub = p.get("npub") or "?"
|
||||
sent = (p.get("stats") or {}).get("bytes_sent", 0)
|
||||
print(f"{npub} {sent}")
|
||||
except Exception:
|
||||
pass
|
||||
'
|
||||
}
|
||||
|
||||
# Compute deltas between two snapshots and identify which peer
|
||||
# received the most bytes_sent growth, plus the absolute bytes for
|
||||
# the requested target. Prints "<top_npub> <top_delta> <target_delta>".
|
||||
peer_bytes_delta_winner() {
|
||||
local target="$1"; shift
|
||||
local before="$1"; shift
|
||||
local after="$1"; shift
|
||||
python3 - "$target" "$before" "$after" <<'PYEOF'
|
||||
import sys
|
||||
target, before, after = sys.argv[1], sys.argv[2], sys.argv[3]
|
||||
b = {}
|
||||
for line in before.splitlines():
|
||||
if not line.strip(): continue
|
||||
k, v = line.split()
|
||||
b[k] = int(v)
|
||||
a = {}
|
||||
for line in after.splitlines():
|
||||
if not line.strip(): continue
|
||||
k, v = line.split()
|
||||
a[k] = int(v)
|
||||
deltas = {k: a.get(k, 0) - b.get(k, 0) for k in a}
|
||||
target_delta = deltas.get(target, 0)
|
||||
if deltas:
|
||||
top_npub = max(deltas, key=deltas.get)
|
||||
top_delta = deltas[top_npub]
|
||||
else:
|
||||
top_npub, top_delta = "-", 0
|
||||
print(f"{top_npub} {top_delta} {target_delta}")
|
||||
PYEOF
|
||||
}
|
||||
|
||||
declare -A PEER_STATE
|
||||
echo "Waiting for peer convergence (up to ${CONVERGE_SECS}s)…"
|
||||
WAIT_START=$(date +%s)
|
||||
while :; do
|
||||
all_done=1
|
||||
for path in "${PATHS[@]}"; do
|
||||
read -r client npub label <<<"$path"
|
||||
if [ "${PEER_STATE[$label]:-}" = "direct" ]; then
|
||||
continue
|
||||
fi
|
||||
if [ "$(peer_is_direct "$client" "$npub")" = "yes" ]; then
|
||||
PEER_STATE[$label]=direct
|
||||
else
|
||||
all_done=0
|
||||
fi
|
||||
done
|
||||
if [ "$all_done" = 1 ]; then
|
||||
break
|
||||
fi
|
||||
elapsed=$(( $(date +%s) - WAIT_START ))
|
||||
if [ "$elapsed" -ge "$CONVERGE_SECS" ]; then
|
||||
break
|
||||
fi
|
||||
sleep 1
|
||||
done
|
||||
|
||||
unconverged=()
|
||||
for path in "${PATHS[@]}"; do
|
||||
read -r client npub label <<<"$path"
|
||||
state="${PEER_STATE[$label]:-via-forward}"
|
||||
PEER_STATE[$label]="$state"
|
||||
printf ' %-10s %s\n' "$label" "$state"
|
||||
if [ "$state" != "direct" ]; then
|
||||
unconverged+=("$label")
|
||||
fi
|
||||
done
|
||||
echo ""
|
||||
|
||||
if [ "${#unconverged[@]}" -gt 0 ]; then
|
||||
echo "ERROR: peer discovery did not converge for ${#unconverged[@]} of ${#PATHS[@]} paths within ${CONVERGE_SECS}s:" >&2
|
||||
for p in "${unconverged[@]}"; do
|
||||
echo " - $p" >&2
|
||||
done
|
||||
echo "" >&2
|
||||
echo "On a single docker-bridge subnet convergence should take seconds." >&2
|
||||
echo "Check that: (a) all node IPs are reachable peer-to-peer, (b) the" >&2
|
||||
echo "test build includes a working discovery transport (Nostr relay /" >&2
|
||||
echo "mDNS / etc.), (c) FIPS_BENCH_CONVERGE_SECS is high enough for" >&2
|
||||
echo "this setup. Bench aborted to avoid reporting noisy mixed-state" >&2
|
||||
echo "measurements." >&2
|
||||
exit 2
|
||||
fi
|
||||
|
||||
# Header for text output. All bandwidth columns are Mbits/sec
|
||||
# (Mbps). CoV% = sample coefficient of variation (stddev/mean) over
|
||||
# the N runs, in percent — directly comparable across paths.
|
||||
if [ "$OUTPUT" = "text" ]; then
|
||||
printf '%-10s %-12s %12s %12s %12s %8s %10s | %10s %8s %10s\n' \
|
||||
path session 'median Mbps' 'min Mbps' 'max Mbps' 'CoV %' outliers \
|
||||
'avg RTT ms' 'loss %' 'TCP retr'
|
||||
printf '%-10s %-12s %12s %12s %12s %8s %10s | %10s %8s %10s\n' \
|
||||
'------' '------' '------' '------' '------' '------' '------' \
|
||||
'------' '------' '------'
|
||||
fi
|
||||
|
||||
FAIL_COUNT=0
|
||||
|
||||
ROUTE_ERRORS=()
|
||||
|
||||
for path in "${PATHS[@]}"; do
|
||||
read -r client npub label <<<"$path"
|
||||
|
||||
# Snapshot per-peer bytes_sent before iperf3 — used after the
|
||||
# runs to verify the routing protocol actually pushed the test
|
||||
# traffic out the intended static-peer link (and not, say,
|
||||
# via an alternative multihop route the cost-based router could
|
||||
# in principle pick). Same defensive check the iperf-test.sh
|
||||
# path measurement assumes implicitly.
|
||||
bytes_before=$(peer_bytes_sent_snapshot "$client")
|
||||
|
||||
# iperf3 runs
|
||||
bw_runs=()
|
||||
retr_runs=()
|
||||
for i in $(seq 1 "$RUNS"); do
|
||||
result=$(iperf_once "$client" "$npub")
|
||||
if [ "$result" = "FAIL" ]; then
|
||||
FAIL_COUNT=$((FAIL_COUNT + 1))
|
||||
bw_runs+=("0")
|
||||
retr_runs+=("0")
|
||||
else
|
||||
bw=$(echo "$result" | awk '{print $1}')
|
||||
retr=$(echo "$result" | awk '{print $2}')
|
||||
bw_runs+=("$bw")
|
||||
retr_runs+=("$retr")
|
||||
fi
|
||||
done
|
||||
|
||||
bytes_after=$(peer_bytes_sent_snapshot "$client")
|
||||
# `delta_winner` is "<npub_with_largest_bytes_sent_delta> <its_bytes> <target_dest_bytes>"
|
||||
delta_winner=$(peer_bytes_delta_winner "$npub" "$bytes_before" "$bytes_after")
|
||||
read -r winning_npub winning_delta target_delta <<<"$delta_winner"
|
||||
if [ "$winning_npub" != "$npub" ]; then
|
||||
ROUTE_ERRORS+=("$label: traffic exited via $winning_npub ($winning_delta B) instead of $npub ($target_delta B)")
|
||||
fi
|
||||
|
||||
# Stats on bandwidth (median / min / max / coefficient-of-variation / outlier indices)
|
||||
bw_stats=$(stats "${bw_runs[@]}")
|
||||
read -r bw_median bw_min bw_max bw_cov bw_outliers <<<"$bw_stats"
|
||||
# Sum retransmits across runs (a proxy for packet loss volume).
|
||||
retr_sum=$(python3 -c "print(sum(int(x) for x in '${retr_runs[*]}'.split()))")
|
||||
|
||||
# Ping (single run; ping gives its own internal stats already).
|
||||
ping_result=$(ping_path "$client" "$npub")
|
||||
if [ "$ping_result" = "FAIL" ]; then
|
||||
rtt_avg="N/A"
|
||||
loss="N/A"
|
||||
else
|
||||
# min avg max mdev loss
|
||||
read -r _rtt_min rtt_avg _rtt_max _rtt_mdev loss <<<"$ping_result"
|
||||
fi
|
||||
|
||||
session_state="${PEER_STATE[$label]:-via-forward}"
|
||||
if [ "$OUTPUT" = "json" ]; then
|
||||
printf '{"path":"%s","client":"%s","dest_npub":"%s","session":"%s","mbps_runs":[%s],"mbps_median":%s,"mbps_min":%s,"mbps_max":%s,"mbps_cov_pct":"%s","outlier_runs":"%s","rtt_avg_ms":"%s","loss":"%s","tcp_retr_total":%s}\n' \
|
||||
"$label" "$client" "$npub" "$session_state" \
|
||||
"$(IFS=, ; echo "${bw_runs[*]}")" \
|
||||
"$bw_median" "$bw_min" "$bw_max" "$bw_cov" "$bw_outliers" \
|
||||
"$rtt_avg" "$loss" "$retr_sum"
|
||||
else
|
||||
printf '%-10s %-12s %12s %12s %12s %8s %10s | %10s %8s %10s\n' \
|
||||
"$label" "$session_state" "$bw_median" "$bw_min" "$bw_max" \
|
||||
"$bw_cov" "$bw_outliers" \
|
||||
"$rtt_avg" "$loss" "$retr_sum"
|
||||
if [ "$bw_outliers" != "-" ]; then
|
||||
printf '%-10s runs (Mbps): %s\n' " ↑outliers" "${bw_runs[*]}"
|
||||
fi
|
||||
fi
|
||||
done
|
||||
|
||||
echo ""
|
||||
if [ "${#ROUTE_ERRORS[@]}" -gt 0 ]; then
|
||||
echo "ERROR: traffic for ${#ROUTE_ERRORS[@]} of ${#PATHS[@]} paths did not exit via the intended static-peer link:" >&2
|
||||
for e in "${ROUTE_ERRORS[@]}"; do
|
||||
echo " - $e" >&2
|
||||
done
|
||||
echo "" >&2
|
||||
echo "The numbers above measured a different route than the path label" >&2
|
||||
echo "claims. Inspect peer cost / tree topology before trusting them." >&2
|
||||
exit 3
|
||||
fi
|
||||
if [ "$FAIL_COUNT" -gt 0 ]; then
|
||||
echo "WARN: $FAIL_COUNT iperf3 runs failed"
|
||||
exit 1
|
||||
fi
|
||||
echo "OK"
|
||||
@@ -84,40 +84,48 @@ iperf_test() {
|
||||
echo "=== FIPS iperf3 Bandwidth Test ($PROFILE topology) ==="
|
||||
echo ""
|
||||
|
||||
# Print topology so the operator can see the configured routing
|
||||
# context. The bench measures iperf3 throughput between the named
|
||||
# nodes — labels DO NOT encode hop count, since peer discovery
|
||||
# converges every same-subnet pair onto a direct UDP path within a
|
||||
# few ticks regardless of the static `peers:` list.
|
||||
if [ "$PROFILE" = "mesh" ] || [ "$PROFILE" = "mesh-public" ]; then
|
||||
echo "Topology (static \`peers:\` from mesh.yaml):"
|
||||
echo " A peers with: D, E"
|
||||
echo " B peers with: C"
|
||||
echo " C peers with: B, D, E"
|
||||
echo " D peers with: A, C, E"
|
||||
echo " E peers with: A, C, D"
|
||||
echo " (Same docker-bridge subnet — discovery converges every"
|
||||
echo " pair onto a direct UDP path; static \`peers:\` only seeds"
|
||||
echo " the initial mesh, not the steady-state routing.)"
|
||||
elif [ "$PROFILE" = "chain" ]; then
|
||||
echo "Topology (static chain): A — B — C — D — E"
|
||||
echo " (Same docker-bridge subnet — see mesh note above.)"
|
||||
fi
|
||||
echo ""
|
||||
|
||||
# Wait for nodes to converge
|
||||
echo "Waiting ${SETTLE_SECONDS}s for mesh convergence..."
|
||||
sleep "$SETTLE_SECONDS"
|
||||
|
||||
if [ "$PROFILE" = "mesh" ] || [ "$PROFILE" = "mesh-public" ]; then
|
||||
# Test key paths in mesh topology
|
||||
echo ""
|
||||
echo "Testing mesh topology paths:"
|
||||
|
||||
# Direct peer links (client on A, server on D/E)
|
||||
iperf_test node-d node-a "$NPUB_D" "A → D (direct peer)"
|
||||
iperf_test node-e node-a "$NPUB_E" "A → E (direct peer)"
|
||||
|
||||
# Multi-hop paths (client on A, server on B/C)
|
||||
iperf_test node-b node-a "$NPUB_B" "A → B (multi-hop)"
|
||||
iperf_test node-c node-a "$NPUB_C" "A → C (multi-hop)"
|
||||
|
||||
# Reverse test (client on E, server on A)
|
||||
iperf_test node-a node-e "$NPUB_A" "E → A (direct peer)"
|
||||
iperf_test node-d node-a "$NPUB_D" "A→D"
|
||||
iperf_test node-e node-a "$NPUB_E" "A→E"
|
||||
iperf_test node-b node-a "$NPUB_B" "A→B"
|
||||
iperf_test node-c node-a "$NPUB_C" "A→C"
|
||||
iperf_test node-a node-e "$NPUB_A" "E→A"
|
||||
|
||||
elif [ "$PROFILE" = "chain" ]; then
|
||||
echo ""
|
||||
echo "Testing chain topology paths:"
|
||||
|
||||
# Adjacent hop (client on A, server on B)
|
||||
iperf_test node-b node-a "$NPUB_B" "A → B (1 hop)"
|
||||
|
||||
# Multi-hop tests (client on A, server on C/D/E)
|
||||
iperf_test node-c node-a "$NPUB_C" "A → C (2 hops)"
|
||||
iperf_test node-d node-a "$NPUB_D" "A → D (3 hops)"
|
||||
iperf_test node-e node-a "$NPUB_E" "A → E (4 hops)"
|
||||
|
||||
# Reverse multi-hop (client on E, server on A)
|
||||
iperf_test node-a node-e "$NPUB_A" "E → A (4 hops)"
|
||||
iperf_test node-b node-a "$NPUB_B" "A→B"
|
||||
iperf_test node-c node-a "$NPUB_C" "A→C"
|
||||
iperf_test node-d node-a "$NPUB_D" "A→D"
|
||||
iperf_test node-e node-a "$NPUB_E" "A→E"
|
||||
iperf_test node-a node-e "$NPUB_A" "E→A"
|
||||
fi
|
||||
|
||||
echo ""
|
||||
|
||||
Reference in New Issue
Block a user