mirror of
https://github.com/jmcorgan/fips.git
synced 2026-07-30 19:46:15 +00:00
Add ECN congestion signaling and transport congestion detection
Implement hop-by-hop ECN congestion signaling through the FMP layer, transport-level congestion detection via kernel drop counters, and chaos harness integration for end-to-end validation. FMP/session ECN plumbing: - Thread ce_flag parsed at link layer through dispatch_link_message, handle_session_datagram, handle_session_payload, and handle_encrypted_session_msg to session delivery - Replace hardcoded false in session-layer record_recv() with actual ce_flag, activating ecn_ce_count tracking in session MMP ECN congestion detection and CE relay: - Add EcnConfig (node.ecn.*) with configurable loss_threshold (5%) and etx_threshold (3.0) for transit congestion detection - Add send_encrypted_link_message_with_ce() that ORs FLAG_CE into FMP header flags; original method delegates with ce_flag=false - Compute outgoing_ce = incoming_ce || local congestion on next-hop link, enabling hop-by-hop CE relay through transit nodes IPv6 ECN-CE marking: - Mark ECN-CE (0b11) in IPv6 Traffic Class on received DataPackets before TUN delivery when FMP CE flag is set - Only marks ECN-capable packets (ECT(0)/ECT(1)); Not-ECT packets unchanged per RFC 3168 Transport congestion abstraction and UDP kernel drop detection: - Add TransportCongestion struct to transport layer for transport- agnostic local congestion indicators - Replace tokio::UdpSocket with AsyncFd<socket2::Socket> using libc::recvmsg() with ancillary data parsing - Enable SO_RXQ_OVFL for kernel receive buffer drop counter on every packet, wiring up previously-stubbed UdpStats.kernel_drops - Add TransportDropState for per-transport delta tracking with 1s tick sampling via sample_transport_congestion() - Extend detect_congestion() with transport kernel drop check alongside MMP loss metrics Congestion monitoring and control: - Add CongestionStats (ce_forwarded, ce_received, congestion_detected, kernel_drop_events) to NodeStats with snapshot serialization - Wire counters into forwarding path, session handler, and transport drop sampling with rate-limited warn logging (5s interval) - Expose congestion data in show_routing control query and ecn_ce_count in show_mmp peer entries - Add congestion counters to fipstop routing tab in two-column layout Chaos harness integration: - Add query_routing(), query_transports(), snapshot_all_congestion() to chaos control module - Add congestion/kernel-drop log analysis in logs module - Add congestion-stress scenario: 10-node tree, 1 Mbps bandwidth, 5-10% netem loss, heavy iperf3 traffic - Add IngressConfig for tc ingress policing with per-peer policer filters simulating upstream bandwidth bottlenecks - Add iperf3 JSON result capture to traffic manager for throughput measurement across scenarios - Add ECN A/B test scenarios (ecn-ab-on/off.yaml) with ingress policing and comparison script - Enable TCP ECN negotiation (tcp_ecn=1 sysctl) in container entrypoint for end-to-end CE propagation Tests: - 10 ECN unit/integration tests: mark_ipv6_ecn_ce variants, CE relay chain (3-node propagation), EcnConfig serde roundtrip - 3 transport drop congestion detection unit tests Documentation: - Update fips-mesh-layer.md: replace outdated CE Echo stub with full ECN Congestion Signaling section covering detection logic, CE relay, IPv6 marking, session tracking, and monitoring counters - Update fips-configuration.md: add node.ecn.* parameter table and ecn block in complete reference YAML - Update fips-transport-layer.md: add Congestion Reporting section with TransportCongestion struct, congestion() trait method, and per-transport status; document AsyncFd/recvmsg/SO_RXQ_OVFL in UDP - Update chaos README: add congestion/ECN scenario docs, ingress traffic control, and iperf3 JSON capture sections - Update README.md: add ECN to features list and "What works today"; update transport and tooling entries
This commit is contained in:
Johnathan Corgan
@@ -105,6 +105,51 @@ Explicit topologies exercising non-UDP transports.
|
||||
static peer config. Netem mutation (30% fraction, every 20-40s) and link
|
||||
flaps (1 link max, 10-20s down).
|
||||
|
||||
### Congestion and ECN
|
||||
|
||||
Scenarios testing ECN congestion signaling and transport-level congestion
|
||||
detection.
|
||||
|
||||
| Scenario | Nodes | Topology | Duration | What it tests |
|
||||
| ------------------ | ----- | -------- | -------- | ---------------------------------------------------------- |
|
||||
| congestion-stress | 10 | Tree | 120s | CE marking under kernel drops and MMP loss detection |
|
||||
| ecn-ab-on / ecn-ab-off | 6 | Tree | 120s | A/B throughput comparison: ECN enabled vs disabled |
|
||||
|
||||
- **congestion-stress**: 10-node tree with 1 Mbps egress bandwidth caps,
|
||||
5-10% netem loss, and heavy iperf3 traffic. Ingress policing (1000 kbps)
|
||||
and small `recv_buf_size` (4 KB) trigger both MMP loss detection and
|
||||
`SO_RXQ_OVFL` kernel socket drops. Validates end-to-end CE propagation:
|
||||
transit nodes detect congestion, set CE flag, destinations receive
|
||||
CE-marked packets, `ecn_ce_count` reported in MMP.
|
||||
- **ecn-ab-on / ecn-ab-off**: Paired scenarios with identical conditions
|
||||
(6-node tree, 10 Mbps egress, 1000 kbps ingress policing, 10ms link
|
||||
delay, 8 KB recv buffer) differing only in `ecn.enabled`.
|
||||
`ecn-ab-test.sh` runs both and compares throughput and congestion
|
||||
counters. Initial results: +10.2% recv throughput with ECN enabled.
|
||||
|
||||
### Ingress Traffic Control
|
||||
|
||||
Scenarios can include `ingress` configuration to simulate upstream bandwidth
|
||||
bottlenecks using tc ingress policing:
|
||||
|
||||
```yaml
|
||||
ingress:
|
||||
enabled: true
|
||||
tiers_kbps: [1000] # per-peer rate limit in kbps
|
||||
burst_bytes: 10000 # policer burst allowance
|
||||
```
|
||||
|
||||
Per-peer u32 filters on the ingress qdisc (`parent ffff:`) rate-limit
|
||||
inbound packets. Combined with small `recv_buf_size`, this reliably triggers
|
||||
`SO_RXQ_OVFL` kernel socket drops for congestion detection testing.
|
||||
|
||||
### iperf3 JSON Capture
|
||||
|
||||
Traffic sessions capture iperf3 results using `--json` output. Results are
|
||||
collected per-session from containers and saved as `iperf3-results.json` in
|
||||
the scenario output directory, enabling automated throughput analysis across
|
||||
scenario runs.
|
||||
|
||||
## CLI Options
|
||||
|
||||
| Option | Description |
|
||||
|
||||
Executable
+127
@@ -0,0 +1,127 @@
|
||||
#!/usr/bin/env bash
|
||||
# ECN A/B Throughput Test
|
||||
#
|
||||
# Runs two identical chaos scenarios — one with ECN enabled, one disabled —
|
||||
# and compares iperf3 throughput and congestion counter results.
|
||||
#
|
||||
# Usage: ./ecn-ab-test.sh [--seed N] [--duration N]
|
||||
|
||||
set -euo pipefail
|
||||
cd "$(dirname "$0")"
|
||||
|
||||
EXTRA_ARGS=()
|
||||
while [[ $# -gt 0 ]]; do
|
||||
case "$1" in
|
||||
--seed|--duration)
|
||||
EXTRA_ARGS+=("$1" "$2"); shift 2 ;;
|
||||
*)
|
||||
echo "Unknown arg: $1"; exit 1 ;;
|
||||
esac
|
||||
done
|
||||
|
||||
echo "=== ECN A/B Throughput Test ==="
|
||||
echo ""
|
||||
|
||||
# --- Run A: ECN ON ---
|
||||
echo "--- Phase A: ECN ENABLED ---"
|
||||
sudo python3 -m sim scenarios/ecn-ab-on.yaml "${EXTRA_ARGS[@]}" || true
|
||||
echo ""
|
||||
|
||||
# --- Run B: ECN OFF ---
|
||||
echo "--- Phase B: ECN DISABLED ---"
|
||||
sudo python3 -m sim scenarios/ecn-ab-off.yaml "${EXTRA_ARGS[@]}" || true
|
||||
echo ""
|
||||
|
||||
# --- Compare results ---
|
||||
echo "=== Results ==="
|
||||
echo ""
|
||||
|
||||
python3 - <<'PYEOF'
|
||||
import json
|
||||
import os
|
||||
import sys
|
||||
|
||||
def load_results(path):
|
||||
if not os.path.exists(path):
|
||||
return []
|
||||
with open(path) as f:
|
||||
return json.load(f)
|
||||
|
||||
def extract_throughput(results):
|
||||
"""Extract per-session throughput in Mbps from iperf3 JSON."""
|
||||
sessions = []
|
||||
for r in results:
|
||||
meta = r.get("_meta", {})
|
||||
end = r.get("end", {})
|
||||
# sum_sent / sum_received contain aggregate stats
|
||||
sent = end.get("sum_sent", {})
|
||||
recv = end.get("sum_received", {})
|
||||
sent_mbps = sent.get("bits_per_second", 0) / 1e6
|
||||
recv_mbps = recv.get("bits_per_second", 0) / 1e6
|
||||
sessions.append({
|
||||
"client": meta.get("client", "?"),
|
||||
"server": meta.get("server", "?"),
|
||||
"sent_mbps": sent_mbps,
|
||||
"recv_mbps": recv_mbps,
|
||||
})
|
||||
return sessions
|
||||
|
||||
def load_congestion(path):
|
||||
if not os.path.exists(path):
|
||||
return {}
|
||||
with open(path) as f:
|
||||
return json.load(f)
|
||||
|
||||
def print_sessions(label, sessions):
|
||||
# Filter out incomplete sessions (killed at teardown, no valid data)
|
||||
valid = [s for s in sessions if s["recv_mbps"] > 0]
|
||||
incomplete = len(sessions) - len(valid)
|
||||
if not valid:
|
||||
print(f" {label}: no completed iperf3 sessions ({incomplete} incomplete)")
|
||||
return 0
|
||||
print(f" {label}:")
|
||||
total_sent = 0
|
||||
total_recv = 0
|
||||
for s in valid:
|
||||
print(f" {s['client']:>4} -> {s['server']:<4} "
|
||||
f"sent={s['sent_mbps']:7.2f} Mbps recv={s['recv_mbps']:7.2f} Mbps")
|
||||
total_sent += s["sent_mbps"]
|
||||
total_recv += s["recv_mbps"]
|
||||
n = len(valid)
|
||||
print(f" {'':>14} avg sent={total_sent/n:7.2f} Mbps avg recv={total_recv/n:7.2f} Mbps")
|
||||
print(f" {'':>14} completed={n} incomplete={incomplete}")
|
||||
return total_recv / n
|
||||
|
||||
on_results = load_results("sim-results/ecn-ab-on/iperf3-results.json")
|
||||
off_results = load_results("sim-results/ecn-ab-off/iperf3-results.json")
|
||||
|
||||
on_sessions = extract_throughput(on_results)
|
||||
off_sessions = extract_throughput(off_results)
|
||||
|
||||
print("Throughput:")
|
||||
avg_on = print_sessions("ECN ON", on_sessions) or 0
|
||||
print()
|
||||
avg_off = print_sessions("ECN OFF", off_sessions) or 0
|
||||
print()
|
||||
|
||||
if avg_on and avg_off:
|
||||
delta_pct = ((avg_on - avg_off) / avg_off) * 100
|
||||
print(f" Delta: ECN ON vs OFF = {delta_pct:+.1f}% avg recv throughput")
|
||||
print()
|
||||
|
||||
# Congestion counters
|
||||
print("Congestion Counters (final snapshot):")
|
||||
for label, path in [("ECN ON", "sim-results/ecn-ab-on/congestion-snapshot-final.json"),
|
||||
("ECN OFF", "sim-results/ecn-ab-off/congestion-snapshot-final.json")]:
|
||||
snap = load_congestion(path)
|
||||
if not snap:
|
||||
print(f" {label}: no snapshot")
|
||||
continue
|
||||
totals = {"ce_forwarded": 0, "ce_received": 0, "congestion_detected": 0, "kernel_drop_events": 0}
|
||||
for node_id, data in sorted(snap.items()):
|
||||
cong = data.get("congestion", {})
|
||||
for k in totals:
|
||||
totals[k] += cong.get(k, 0)
|
||||
print(f" {label}: ce_fwd={totals['ce_forwarded']} ce_recv={totals['ce_received']} "
|
||||
f"cong_detect={totals['congestion_detected']} kern_drops={totals['kernel_drop_events']}")
|
||||
PYEOF
|
||||
@@ -8,6 +8,12 @@
|
||||
|
||||
set -e
|
||||
|
||||
# Enable TCP ECN negotiation for both IPv4 and IPv6 connections.
|
||||
# Despite the "ipv4" name, this sysctl controls ECN for all TCP.
|
||||
# Without this, IPv6 packets traversing the FIPS mesh carry ECN=0b00
|
||||
# (Not-ECT), and mark_ipv6_ecn_ce() is a no-op per RFC 3168.
|
||||
sysctl -w net.ipv4.tcp_ecn=1 >/dev/null 2>&1 || true
|
||||
|
||||
# Start background services
|
||||
dnsmasq
|
||||
/usr/sbin/sshd
|
||||
|
||||
@@ -0,0 +1,104 @@
|
||||
# Congestion Stress: exercise both MMP-based and kernel-drop congestion detection
|
||||
#
|
||||
# Topology: 10-node tree with all links bandwidth-limited to 1 Mbps
|
||||
# (egress HTB + ingress policing). Heavy iperf3 traffic (8 concurrent
|
||||
# sessions, 8 parallel streams) combined with 5-10% netem loss.
|
||||
#
|
||||
# Congestion detection signals exercised:
|
||||
# 1. MMP loss detection: netem loss exceeds the 5% loss_threshold,
|
||||
# triggering detect_congestion() via MMP metrics on transit nodes.
|
||||
# 2. Kernel socket drops: small recv_buf_size (8KB) combined with
|
||||
# traffic saturation causes SO_RXQ_OVFL on the UDP socket,
|
||||
# triggering the transport drop detection path.
|
||||
# 3. Ingress policing: tc policer on the receive side drops excess
|
||||
# inbound packets, creating bursty arrival patterns.
|
||||
#
|
||||
# ECN is explicitly enabled via fips_overrides.
|
||||
#
|
||||
# Success criteria (verified via post-run congestion snapshot):
|
||||
# - congestion_detected > 0 on at least one forwarding node
|
||||
# - kernel_drop_events > 0 on at least one node
|
||||
# - ce_forwarded > 0 on transit nodes
|
||||
# - ce_received > 0 on destination nodes
|
||||
#
|
||||
# Topology:
|
||||
#
|
||||
# n01 (root)
|
||||
# / \
|
||||
# n02 n03
|
||||
# / \ / \
|
||||
# n04 n05 n06 n07
|
||||
# | |
|
||||
# n08 n09
|
||||
# |
|
||||
# n10
|
||||
|
||||
scenario:
|
||||
name: "congestion-stress"
|
||||
seed: 7
|
||||
duration_secs: 120
|
||||
|
||||
topology:
|
||||
algorithm: explicit
|
||||
num_nodes: 10
|
||||
params:
|
||||
adjacency:
|
||||
- [n01, n02]
|
||||
- [n01, n03]
|
||||
- [n02, n04]
|
||||
- [n02, n05]
|
||||
- [n03, n06]
|
||||
- [n03, n07]
|
||||
- [n04, n08]
|
||||
- [n05, n09]
|
||||
- [n08, n10]
|
||||
subnet: "172.20.0.0/24"
|
||||
ip_start: 10
|
||||
|
||||
bandwidth:
|
||||
enabled: true
|
||||
tiers_mbps: [1]
|
||||
|
||||
ingress:
|
||||
enabled: true
|
||||
tiers_kbps: [1000]
|
||||
burst_bytes: 16000
|
||||
|
||||
netem:
|
||||
enabled: true
|
||||
default_policy:
|
||||
delay_ms: [2, 5]
|
||||
jitter_ms: [0, 1]
|
||||
loss_pct: [5, 10]
|
||||
mutation:
|
||||
interval_secs: {min: 9999, max: 9999}
|
||||
fraction: 0.0
|
||||
policies:
|
||||
baseline:
|
||||
delay_ms: [2, 5]
|
||||
loss_pct: [5, 10]
|
||||
|
||||
link_flaps:
|
||||
enabled: false
|
||||
|
||||
traffic:
|
||||
enabled: true
|
||||
max_concurrent: 8
|
||||
interval_secs: {min: 2, max: 5}
|
||||
duration_secs: {min: 30, max: 60}
|
||||
parallel_streams: 8
|
||||
|
||||
node_churn:
|
||||
enabled: false
|
||||
|
||||
logging:
|
||||
rust_log: "info"
|
||||
output_dir: "./sim-results"
|
||||
|
||||
fips_overrides:
|
||||
node:
|
||||
ecn:
|
||||
enabled: true
|
||||
transports:
|
||||
udp:
|
||||
recv_buf_size: 4096
|
||||
@@ -0,0 +1,80 @@
|
||||
# ECN A/B Test: ECN DISABLED
|
||||
#
|
||||
# Identical to ecn-ab-on.yaml except ecn.enabled = false.
|
||||
# Run both scenarios and compare iperf3 throughput results.
|
||||
#
|
||||
# Same conditions: 10 Mbps egress, 1000 kbps ingress policing, 10ms
|
||||
# link delay, 8KB recv buffer. Without ECN, the only congestion signal
|
||||
# TCP gets is packet loss (timeouts/triple dup-ack).
|
||||
#
|
||||
# n01 (root)
|
||||
# / \
|
||||
# n02 n03
|
||||
# / \ \
|
||||
# n04 n05 n06
|
||||
|
||||
scenario:
|
||||
name: "ecn-ab-off"
|
||||
seed: 42
|
||||
duration_secs: 120
|
||||
|
||||
topology:
|
||||
algorithm: explicit
|
||||
num_nodes: 6
|
||||
params:
|
||||
adjacency:
|
||||
- [n01, n02]
|
||||
- [n01, n03]
|
||||
- [n02, n04]
|
||||
- [n02, n05]
|
||||
- [n03, n06]
|
||||
subnet: "172.20.0.0/24"
|
||||
ip_start: 10
|
||||
|
||||
bandwidth:
|
||||
enabled: true
|
||||
tiers_mbps: [10]
|
||||
|
||||
ingress:
|
||||
enabled: true
|
||||
tiers_kbps: [1000]
|
||||
burst_bytes: 16000
|
||||
|
||||
netem:
|
||||
enabled: true
|
||||
default_policy:
|
||||
delay_ms: [10, 10]
|
||||
jitter_ms: [0, 0]
|
||||
loss_pct: [0, 0]
|
||||
mutation:
|
||||
interval_secs: {min: 9999, max: 9999}
|
||||
fraction: 0.0
|
||||
policies:
|
||||
baseline:
|
||||
delay_ms: [10, 10]
|
||||
loss_pct: [0, 0]
|
||||
|
||||
link_flaps:
|
||||
enabled: false
|
||||
|
||||
traffic:
|
||||
enabled: true
|
||||
max_concurrent: 4
|
||||
interval_secs: {min: 3, max: 5}
|
||||
duration_secs: {min: 20, max: 30}
|
||||
parallel_streams: 4
|
||||
|
||||
node_churn:
|
||||
enabled: false
|
||||
|
||||
logging:
|
||||
rust_log: "info"
|
||||
output_dir: "./sim-results/ecn-ab-off"
|
||||
|
||||
fips_overrides:
|
||||
node:
|
||||
ecn:
|
||||
enabled: false
|
||||
transports:
|
||||
udp:
|
||||
recv_buf_size: 8192
|
||||
@@ -0,0 +1,100 @@
|
||||
# ECN A/B Test: ECN ENABLED
|
||||
#
|
||||
# Identical to ecn-ab-off.yaml except ecn.enabled = true.
|
||||
# Run both scenarios and compare iperf3 throughput results.
|
||||
#
|
||||
# Design rationale:
|
||||
# - Ingress policing at 1000 kbps: unlike HTB (which paces smoothly),
|
||||
# the ingress policer *drops* excess inbound packets at the kernel
|
||||
# level. This creates real packet loss visible to the FIPS UDP
|
||||
# socket, triggering detect_congestion() via transport_drops.
|
||||
# - 10 Mbps egress bandwidth: lets traffic flow freely outbound so
|
||||
# that the ingress policer is the bottleneck. Multiple flows
|
||||
# competing through bottleneck nodes generate excess traffic that
|
||||
# the policer drops.
|
||||
# - 10ms link delay: creates realistic RTT and causes TCP to batch
|
||||
# packets into bursts, making ingress drops more likely.
|
||||
# - 8KB recv_buf_size: small buffer amplifies the effect of bursty
|
||||
# arrivals — dropped packets at ingress + buffer overflow trigger
|
||||
# detect_congestion() via both MMP loss and transport_drops paths.
|
||||
# - tcp_ecn=1 in entrypoint.sh: TCP negotiates ECN so CE marks
|
||||
# reach TCP congestion control, enabling proactive backoff.
|
||||
# - 4 concurrent sessions with 4 parallel streams: moderate load
|
||||
# that creates real contention without overwhelming everything.
|
||||
#
|
||||
# With ECN enabled, transit nodes detect congestion early and set CE
|
||||
# flags, causing TCP to reduce its window proactively. Without ECN,
|
||||
# TCP only learns about congestion from packet drops (timeouts/triple
|
||||
# dup-ack), which is slower and causes more throughput loss.
|
||||
#
|
||||
# n01 (root)
|
||||
# / \
|
||||
# n02 n03
|
||||
# / \ \
|
||||
# n04 n05 n06
|
||||
|
||||
scenario:
|
||||
name: "ecn-ab-on"
|
||||
seed: 42
|
||||
duration_secs: 120
|
||||
|
||||
topology:
|
||||
algorithm: explicit
|
||||
num_nodes: 6
|
||||
params:
|
||||
adjacency:
|
||||
- [n01, n02]
|
||||
- [n01, n03]
|
||||
- [n02, n04]
|
||||
- [n02, n05]
|
||||
- [n03, n06]
|
||||
subnet: "172.20.0.0/24"
|
||||
ip_start: 10
|
||||
|
||||
bandwidth:
|
||||
enabled: true
|
||||
tiers_mbps: [10]
|
||||
|
||||
ingress:
|
||||
enabled: true
|
||||
tiers_kbps: [1000]
|
||||
burst_bytes: 16000
|
||||
|
||||
netem:
|
||||
enabled: true
|
||||
default_policy:
|
||||
delay_ms: [10, 10]
|
||||
jitter_ms: [0, 0]
|
||||
loss_pct: [0, 0]
|
||||
mutation:
|
||||
interval_secs: {min: 9999, max: 9999}
|
||||
fraction: 0.0
|
||||
policies:
|
||||
baseline:
|
||||
delay_ms: [10, 10]
|
||||
loss_pct: [0, 0]
|
||||
|
||||
link_flaps:
|
||||
enabled: false
|
||||
|
||||
traffic:
|
||||
enabled: true
|
||||
max_concurrent: 4
|
||||
interval_secs: {min: 3, max: 5}
|
||||
duration_secs: {min: 20, max: 30}
|
||||
parallel_streams: 4
|
||||
|
||||
node_churn:
|
||||
enabled: false
|
||||
|
||||
logging:
|
||||
rust_log: "info"
|
||||
output_dir: "./sim-results/ecn-ab-on"
|
||||
|
||||
fips_overrides:
|
||||
node:
|
||||
ecn:
|
||||
enabled: true
|
||||
transports:
|
||||
udp:
|
||||
recv_buf_size: 8192
|
||||
@@ -101,3 +101,42 @@ def snapshot_all_mmp(topology: SimTopology) -> dict[str, dict]:
|
||||
else:
|
||||
log.warning("No MMP data from %s", node_id)
|
||||
return result
|
||||
|
||||
|
||||
def query_routing(container: str) -> dict | None:
|
||||
"""Query a node's routing stats (includes congestion counters)."""
|
||||
return query_node(container, "show_routing")
|
||||
|
||||
|
||||
def query_transports(container: str) -> dict | None:
|
||||
"""Query a node's transport state (includes kernel drop counters)."""
|
||||
return query_node(container, "show_transports")
|
||||
|
||||
|
||||
def snapshot_all_congestion(topology: SimTopology) -> dict[str, dict]:
|
||||
"""Query show_routing on all nodes to capture congestion counters.
|
||||
|
||||
Returns {node_id: {"congestion": {...}, "kernel_drops": [...]}}.
|
||||
Nodes that fail to respond are omitted from the result.
|
||||
"""
|
||||
result = {}
|
||||
for node_id in sorted(topology.nodes):
|
||||
container = topology.container_name(node_id)
|
||||
routing = query_routing(container)
|
||||
transports = query_transports(container)
|
||||
if routing is not None:
|
||||
entry = {"congestion": routing.get("congestion", {})}
|
||||
if transports is not None:
|
||||
drops = []
|
||||
for t in transports.get("transports", []):
|
||||
stats = t.get("stats", {})
|
||||
drops.append({
|
||||
"transport_id": t.get("transport_id"),
|
||||
"name": t.get("name"),
|
||||
"kernel_drops": stats.get("kernel_drops"),
|
||||
})
|
||||
entry["kernel_drops"] = drops
|
||||
result[node_id] = entry
|
||||
else:
|
||||
log.warning("No routing data from %s", node_id)
|
||||
return result
|
||||
|
||||
@@ -26,6 +26,8 @@ class AnalysisResult:
|
||||
mmp_session_metrics: list[tuple[str, str]] = field(default_factory=list)
|
||||
handshake_timeouts: list[tuple[str, str]] = field(default_factory=list)
|
||||
panics: list[tuple[str, str]] = field(default_factory=list)
|
||||
congestion_detected: list[tuple[str, str]] = field(default_factory=list)
|
||||
kernel_drop_events: list[tuple[str, str]] = field(default_factory=list)
|
||||
|
||||
def summary(self) -> str:
|
||||
lines = [
|
||||
@@ -41,6 +43,8 @@ class AnalysisResult:
|
||||
f"Handshake timeouts: {len(self.handshake_timeouts)}",
|
||||
f"MMP link samples: {len(self.mmp_link_metrics)}",
|
||||
f"MMP session samples: {len(self.mmp_session_metrics)}",
|
||||
f"Congestion events: {len(self.congestion_detected)}",
|
||||
f"Kernel drop events: {len(self.kernel_drop_events)}",
|
||||
]
|
||||
|
||||
if self.panics:
|
||||
@@ -134,6 +138,11 @@ def analyze_logs(logs: dict[str, str]) -> AnalysisResult:
|
||||
result.mmp_link_metrics.append((container, line))
|
||||
if "MMP session metrics" in line:
|
||||
result.mmp_session_metrics.append((container, line))
|
||||
# Congestion events
|
||||
if "Congestion detected" in line:
|
||||
result.congestion_detected.append((container, line))
|
||||
if "Kernel recv drops first observed" in line:
|
||||
result.kernel_drop_events.append((container, line))
|
||||
|
||||
return result
|
||||
|
||||
|
||||
@@ -8,6 +8,13 @@ u32 filters match destination IP to direct traffic to the right class.
|
||||
├── class 1:1 → netem 11: (peer 1) ← u32 filter: dst=<peer1_ip>
|
||||
├── class 1:2 → netem 12: (peer 2) ← u32 filter: dst=<peer2_ip>
|
||||
└── class 1:99 → pfifo (default, no impairment)
|
||||
|
||||
Optional ingress policing adds rate-based packet dropping on the
|
||||
receive side via tc ingress qdisc + policer filters:
|
||||
|
||||
eth0 ingress (ffff:)
|
||||
├── u32 filter: src=<peer1_ip> → police rate <R>kbit burst <B> drop
|
||||
└── u32 filter: src=<peer2_ip> → police rate <R>kbit burst <B> drop
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
@@ -17,7 +24,7 @@ import random
|
||||
from dataclasses import dataclass, field
|
||||
|
||||
from .docker_exec import docker_exec_quiet, is_container_running
|
||||
from .scenario import BandwidthConfig, LinkPolicyOverride, NetemConfig, NetemPolicy
|
||||
from .scenario import BandwidthConfig, IngressConfig, LinkPolicyOverride, NetemConfig, NetemPolicy
|
||||
from .topology import SimTopology, veth_interface_name
|
||||
|
||||
log = logging.getLogger(__name__)
|
||||
@@ -65,6 +72,9 @@ class LinkNetemState:
|
||||
netem_handle: str # e.g., "11:"
|
||||
params: NetemParams = field(default_factory=NetemParams)
|
||||
rate_mbit: int = 0 # 0 = unlimited (1gbit default)
|
||||
ingress_rate_kbps: int = 0 # 0 = no ingress policing
|
||||
ingress_burst_bytes: int = 32000
|
||||
ingress_filter_prio: int = 0 # u32 filter priority for this peer
|
||||
|
||||
|
||||
@dataclass
|
||||
@@ -85,6 +95,7 @@ class NetemManager:
|
||||
config: NetemConfig,
|
||||
rng: random.Random,
|
||||
bandwidth: BandwidthConfig | None = None,
|
||||
ingress: IngressConfig | None = None,
|
||||
):
|
||||
self.topology = topology
|
||||
self.config = config
|
||||
@@ -102,6 +113,14 @@ class NetemManager:
|
||||
rate = rng.choice(bandwidth.tiers_mbps)
|
||||
self._edge_rates[(a, b)] = rate
|
||||
self._edge_rates[(b, a)] = rate
|
||||
# Per-edge ingress policing: (node_a, node_b) -> rate in kbps
|
||||
self._ingress_config = ingress
|
||||
self._ingress_rates: dict[tuple[str, str], int] = {}
|
||||
if ingress and ingress.enabled:
|
||||
for a, b in topology.edges:
|
||||
rate = rng.choice(ingress.tiers_kbps)
|
||||
self._ingress_rates[(a, b)] = rate
|
||||
self._ingress_rates[(b, a)] = rate
|
||||
# Per-edge policy overrides: canonical "nXX-nYY" -> NetemPolicy
|
||||
# Build a set of canonical edge strings for validation
|
||||
topo_edge_strs = {"-".join(sorted([a, b])) for a, b in topology.edges}
|
||||
@@ -194,6 +213,9 @@ class NetemManager:
|
||||
f"prio {idx} u32 match ip dst {dest_ip}/32 flowid {class_id}"
|
||||
)
|
||||
|
||||
ingress_rate = self._ingress_rates.get((node_id, peer_id), 0)
|
||||
ingress_burst = self._ingress_config.burst_bytes if self._ingress_config else 32000
|
||||
|
||||
state = LinkNetemState(
|
||||
container=container,
|
||||
dest_ip=dest_ip,
|
||||
@@ -201,16 +223,33 @@ class NetemManager:
|
||||
netem_handle=netem_handle,
|
||||
params=params,
|
||||
rate_mbit=rate_mbit,
|
||||
ingress_rate_kbps=ingress_rate,
|
||||
ingress_burst_bytes=ingress_burst,
|
||||
ingress_filter_prio=idx,
|
||||
)
|
||||
container_states[dest_ip] = state
|
||||
|
||||
# Ingress policing: add ingress qdisc + per-peer policer filters
|
||||
if self._ingress_rates:
|
||||
cmds.append(f"tc qdisc add dev {IFACE} ingress 2>/dev/null || true")
|
||||
for idx, (peer_id, dest_ip) in enumerate(ip_peers.items(), start=1):
|
||||
ingress_rate = self._ingress_rates.get((node_id, peer_id), 0)
|
||||
if ingress_rate > 0:
|
||||
ingress_burst = self._ingress_config.burst_bytes if self._ingress_config else 32000
|
||||
cmds.append(
|
||||
f"tc filter add dev {IFACE} parent ffff: protocol ip "
|
||||
f"prio {idx} u32 match ip src {dest_ip}/32 "
|
||||
f"police rate {ingress_rate}kbit burst {ingress_burst} drop"
|
||||
)
|
||||
|
||||
full_cmd = " && ".join(cmds)
|
||||
result = docker_exec_quiet(container, full_cmd, timeout=30)
|
||||
if result is not None:
|
||||
log.info(
|
||||
"Configured per-link netem on %s (%d IP peers)",
|
||||
"Configured per-link netem on %s (%d IP peers%s)",
|
||||
container,
|
||||
len(ip_peers),
|
||||
", ingress policing" if self._ingress_rates else "",
|
||||
)
|
||||
else:
|
||||
log.warning("Failed to configure netem on %s", container)
|
||||
@@ -282,13 +321,27 @@ class NetemManager:
|
||||
f"prio {prio} u32 match ip dst {dest_ip}/32 flowid {state.class_id}"
|
||||
)
|
||||
|
||||
# Re-apply ingress policing
|
||||
has_ingress = any(s.ingress_rate_kbps > 0 for s in container_states.values())
|
||||
if has_ingress:
|
||||
cmds.append(f"tc qdisc add dev {IFACE} ingress 2>/dev/null || true")
|
||||
for dest_ip, state in container_states.items():
|
||||
if state.ingress_rate_kbps > 0:
|
||||
cmds.append(
|
||||
f"tc filter add dev {IFACE} parent ffff: protocol ip "
|
||||
f"prio {state.ingress_filter_prio} u32 match ip src {dest_ip}/32 "
|
||||
f"police rate {state.ingress_rate_kbps}kbit "
|
||||
f"burst {state.ingress_burst_bytes} drop"
|
||||
)
|
||||
|
||||
full_cmd = " && ".join(cmds)
|
||||
result = docker_exec_quiet(container, full_cmd, timeout=30)
|
||||
if result is not None:
|
||||
log.info(
|
||||
"Re-applied IP netem on %s (%d peers)",
|
||||
"Re-applied IP netem on %s (%d peers%s)",
|
||||
container,
|
||||
len(container_states),
|
||||
", ingress policing" if has_ingress else "",
|
||||
)
|
||||
else:
|
||||
log.warning("Failed to re-apply IP netem on %s", container)
|
||||
|
||||
@@ -12,7 +12,7 @@ import time
|
||||
|
||||
from .compose import generate_compose
|
||||
from .config_gen import write_configs
|
||||
from .control import snapshot_all_mmp, snapshot_all_trees
|
||||
from .control import snapshot_all_congestion, snapshot_all_mmp, snapshot_all_trees
|
||||
from .docker_exec import docker_compose
|
||||
from .links import LinkManager
|
||||
from .logs import AnalysisResult, analyze_logs, collect_logs, write_sim_metadata
|
||||
@@ -140,7 +140,8 @@ class SimRunner:
|
||||
# 7. Initialize managers
|
||||
if s.netem.enabled:
|
||||
bw = s.bandwidth if s.bandwidth.enabled else None
|
||||
self.netem_mgr = NetemManager(self.topology, s.netem, self.rng, bandwidth=bw)
|
||||
ig = s.ingress if s.ingress.enabled else None
|
||||
self.netem_mgr = NetemManager(self.topology, s.netem, self.rng, bandwidth=bw, ingress=ig)
|
||||
self.netem_mgr.down_nodes = self._down_nodes
|
||||
log.info("Applying initial per-link netem...")
|
||||
self.netem_mgr.setup_initial()
|
||||
@@ -254,6 +255,15 @@ class SimRunner:
|
||||
log.info("Restoring stopped nodes...")
|
||||
self.node_mgr.restore_all()
|
||||
|
||||
# Collect iperf3 throughput results before containers stop
|
||||
if self.traffic_mgr:
|
||||
iperf_results = self.traffic_mgr.collect_results()
|
||||
if iperf_results:
|
||||
iperf_path = os.path.join(self.output_dir, "iperf3-results.json")
|
||||
with open(iperf_path, "w") as f:
|
||||
json.dump(iperf_results, f, indent=2)
|
||||
log.info("Saved %d iperf3 results to %s", len(iperf_results), iperf_path)
|
||||
|
||||
# Take final tree snapshot while nodes are still running
|
||||
self._take_snapshot("final")
|
||||
|
||||
@@ -298,27 +308,33 @@ class SimRunner:
|
||||
return result
|
||||
|
||||
def _take_snapshot(self, label: str):
|
||||
"""Query all nodes via control socket and save tree/MMP snapshots."""
|
||||
"""Query all nodes via control socket and save tree/MMP/congestion snapshots."""
|
||||
if not self.topology:
|
||||
return
|
||||
log.info("Taking %s snapshot...", label)
|
||||
tree_snap = snapshot_all_trees(self.topology)
|
||||
mmp_snap = snapshot_all_mmp(self.topology)
|
||||
congestion_snap = snapshot_all_congestion(self.topology)
|
||||
|
||||
tree_path = os.path.join(self.output_dir, f"tree-snapshot-{label}.json")
|
||||
mmp_path = os.path.join(self.output_dir, f"mmp-snapshot-{label}.json")
|
||||
congestion_path = os.path.join(self.output_dir, f"congestion-snapshot-{label}.json")
|
||||
os.makedirs(self.output_dir, exist_ok=True)
|
||||
with open(tree_path, "w") as f:
|
||||
json.dump(tree_snap, f, indent=2)
|
||||
with open(mmp_path, "w") as f:
|
||||
json.dump(mmp_snap, f, indent=2)
|
||||
with open(congestion_path, "w") as f:
|
||||
json.dump(congestion_snap, f, indent=2)
|
||||
log.info(
|
||||
"Snapshot %s: %d/%d tree, %d/%d mmp responses",
|
||||
"Snapshot %s: %d/%d tree, %d/%d mmp, %d/%d congestion responses",
|
||||
label,
|
||||
len(tree_snap),
|
||||
len(self.topology.nodes),
|
||||
len(mmp_snap),
|
||||
len(self.topology.nodes),
|
||||
len(congestion_snap),
|
||||
len(self.topology.nodes),
|
||||
)
|
||||
|
||||
def _schedule_next(self, now: float, interval) -> float:
|
||||
|
||||
@@ -120,6 +120,22 @@ class BandwidthConfig:
|
||||
tiers_mbps: list[int] = field(default_factory=lambda: [1, 10, 100, 1000])
|
||||
|
||||
|
||||
@dataclass
|
||||
class IngressConfig:
|
||||
"""Per-link ingress policing via tc ingress qdisc + policer.
|
||||
|
||||
When enabled, each link gets an ingress policer that drops inbound
|
||||
packets exceeding the configured rate. Unlike egress HTB (which
|
||||
paces packets smoothly), the policer drops excess packets at the
|
||||
kernel level, creating bursty arrival patterns that can overflow
|
||||
small socket buffers and trigger SO_RXQ_OVFL kernel drops.
|
||||
"""
|
||||
|
||||
enabled: bool = False
|
||||
tiers_kbps: list[int] = field(default_factory=lambda: [1000])
|
||||
burst_bytes: int = 32000
|
||||
|
||||
|
||||
@dataclass
|
||||
class LoggingConfig:
|
||||
rust_log: str = "info"
|
||||
@@ -137,6 +153,7 @@ class Scenario:
|
||||
traffic: TrafficConfig = field(default_factory=TrafficConfig)
|
||||
node_churn: NodeChurnConfig = field(default_factory=NodeChurnConfig)
|
||||
bandwidth: BandwidthConfig = field(default_factory=BandwidthConfig)
|
||||
ingress: IngressConfig = field(default_factory=IngressConfig)
|
||||
logging: LoggingConfig = field(default_factory=LoggingConfig)
|
||||
# Raw YAML dict appended to each generated FIPS node config.
|
||||
# Allows scenarios to override any FIPS config parameter
|
||||
@@ -271,6 +288,16 @@ def load_scenario(path: str) -> Scenario:
|
||||
raise ValueError("bandwidth.tiers_mbps must be a non-empty list")
|
||||
s.bandwidth.tiers_mbps = [int(t) for t in tiers]
|
||||
|
||||
# Ingress section
|
||||
ig = raw.get("ingress", {})
|
||||
s.ingress.enabled = ig.get("enabled", False)
|
||||
if "tiers_kbps" in ig:
|
||||
tiers = ig["tiers_kbps"]
|
||||
if not isinstance(tiers, list) or not tiers:
|
||||
raise ValueError("ingress.tiers_kbps must be a non-empty list")
|
||||
s.ingress.tiers_kbps = [int(t) for t in tiers]
|
||||
s.ingress.burst_bytes = int(ig.get("burst_bytes", 32000))
|
||||
|
||||
# Logging section
|
||||
lg = raw.get("logging", {})
|
||||
s.logging.rust_log = lg.get("rust_log", "info")
|
||||
@@ -376,3 +403,9 @@ def _validate(s: Scenario):
|
||||
for tier in s.bandwidth.tiers_mbps:
|
||||
if tier <= 0:
|
||||
raise ValueError(f"bandwidth.tiers_mbps: all values must be > 0, got {tier}")
|
||||
if s.ingress.enabled:
|
||||
for tier in s.ingress.tiers_kbps:
|
||||
if tier <= 0:
|
||||
raise ValueError(f"ingress.tiers_kbps: all values must be > 0, got {tier}")
|
||||
if s.ingress.burst_bytes <= 0:
|
||||
raise ValueError(f"ingress.burst_bytes must be > 0, got {s.ingress.burst_bytes}")
|
||||
|
||||
@@ -7,6 +7,7 @@ entrypoint).
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import logging
|
||||
import random
|
||||
import time
|
||||
@@ -26,6 +27,7 @@ class TrafficSession:
|
||||
started_at: float
|
||||
duration_secs: int
|
||||
container: str
|
||||
result_file: str = ""
|
||||
|
||||
|
||||
class TrafficManager:
|
||||
@@ -43,6 +45,7 @@ class TrafficManager:
|
||||
self.rng = rng
|
||||
self.down_nodes = down_nodes or set()
|
||||
self.active_sessions: list[TrafficSession] = []
|
||||
self.completed_results: list[dict] = []
|
||||
|
||||
@property
|
||||
def active_count(self) -> int:
|
||||
@@ -74,10 +77,14 @@ class TrafficManager:
|
||||
)
|
||||
streams = self.config.parallel_streams
|
||||
|
||||
# Start iperf3 in background (nohup, stdout to /dev/null)
|
||||
# Result file inside the container for JSON capture
|
||||
ts = int(time.time())
|
||||
result_file = f"/tmp/iperf3-{client}-{server}-{ts}.json"
|
||||
|
||||
# Start iperf3 in background with JSON output
|
||||
cmd = (
|
||||
f"nohup iperf3 -c {server_npub}.fips -t {duration} "
|
||||
f"-P {streams} > /dev/null 2>&1 &"
|
||||
f"-P {streams} --json > {result_file} 2>&1 &"
|
||||
)
|
||||
result = docker_exec_quiet(container, cmd)
|
||||
if result is not None:
|
||||
@@ -87,6 +94,7 @@ class TrafficManager:
|
||||
started_at=time.time(),
|
||||
duration_secs=duration,
|
||||
container=container,
|
||||
result_file=result_file,
|
||||
)
|
||||
self.active_sessions.append(session)
|
||||
log.info(
|
||||
@@ -103,16 +111,49 @@ class TrafficManager:
|
||||
"""Remove sessions that have completed (based on time)."""
|
||||
now = time.time()
|
||||
grace = 5 # seconds after expected completion
|
||||
before = len(self.active_sessions)
|
||||
self.active_sessions = [
|
||||
s
|
||||
for s in self.active_sessions
|
||||
if now - s.started_at < s.duration_secs + grace
|
||||
]
|
||||
removed = before - len(self.active_sessions)
|
||||
still_active = []
|
||||
for s in self.active_sessions:
|
||||
if now - s.started_at >= s.duration_secs + grace:
|
||||
self._collect_result(s)
|
||||
else:
|
||||
still_active.append(s)
|
||||
removed = len(self.active_sessions) - len(still_active)
|
||||
self.active_sessions = still_active
|
||||
if removed > 0:
|
||||
log.debug("Cleaned up %d expired traffic sessions", removed)
|
||||
|
||||
def _collect_result(self, session: TrafficSession):
|
||||
"""Retrieve iperf3 JSON result from container."""
|
||||
if not session.result_file:
|
||||
return
|
||||
if session.client_node in self.down_nodes:
|
||||
return
|
||||
stdout = docker_exec_quiet(
|
||||
session.container,
|
||||
f"cat {session.result_file} 2>/dev/null; rm -f {session.result_file}",
|
||||
)
|
||||
if stdout is None:
|
||||
return
|
||||
try:
|
||||
data = json.loads(stdout.strip())
|
||||
except (json.JSONDecodeError, ValueError):
|
||||
log.debug("Could not parse iperf3 result for %s -> %s",
|
||||
session.client_node, session.server_node)
|
||||
return
|
||||
data["_meta"] = {
|
||||
"client": session.client_node,
|
||||
"server": session.server_node,
|
||||
"duration_secs": session.duration_secs,
|
||||
}
|
||||
self.completed_results.append(data)
|
||||
|
||||
def collect_results(self) -> list[dict]:
|
||||
"""Return all completed iperf3 JSON results."""
|
||||
# Collect any remaining active sessions that may have finished
|
||||
for s in self.active_sessions:
|
||||
self._collect_result(s)
|
||||
return self.completed_results
|
||||
|
||||
def stop_all(self):
|
||||
"""Kill all iperf3 client processes in running containers."""
|
||||
seen = set()
|
||||
@@ -124,4 +165,7 @@ class TrafficManager:
|
||||
"killall iperf3 2>/dev/null; true",
|
||||
)
|
||||
seen.add(session.container)
|
||||
# Collect results before clearing (iperf3 writes partial JSON on kill)
|
||||
for s in self.active_sessions:
|
||||
self._collect_result(s)
|
||||
self.active_sessions.clear()
|
||||
|
||||
Reference in New Issue
Block a user