mirror of
https://github.com/jmcorgan/fips.git
synced 2026-07-30 19:46:15 +00:00
Add ECN congestion signaling and transport congestion detection
Implement hop-by-hop ECN congestion signaling through the FMP layer, transport-level congestion detection via kernel drop counters, and chaos harness integration for end-to-end validation. FMP/session ECN plumbing: - Thread ce_flag parsed at link layer through dispatch_link_message, handle_session_datagram, handle_session_payload, and handle_encrypted_session_msg to session delivery - Replace hardcoded false in session-layer record_recv() with actual ce_flag, activating ecn_ce_count tracking in session MMP ECN congestion detection and CE relay: - Add EcnConfig (node.ecn.*) with configurable loss_threshold (5%) and etx_threshold (3.0) for transit congestion detection - Add send_encrypted_link_message_with_ce() that ORs FLAG_CE into FMP header flags; original method delegates with ce_flag=false - Compute outgoing_ce = incoming_ce || local congestion on next-hop link, enabling hop-by-hop CE relay through transit nodes IPv6 ECN-CE marking: - Mark ECN-CE (0b11) in IPv6 Traffic Class on received DataPackets before TUN delivery when FMP CE flag is set - Only marks ECN-capable packets (ECT(0)/ECT(1)); Not-ECT packets unchanged per RFC 3168 Transport congestion abstraction and UDP kernel drop detection: - Add TransportCongestion struct to transport layer for transport- agnostic local congestion indicators - Replace tokio::UdpSocket with AsyncFd<socket2::Socket> using libc::recvmsg() with ancillary data parsing - Enable SO_RXQ_OVFL for kernel receive buffer drop counter on every packet, wiring up previously-stubbed UdpStats.kernel_drops - Add TransportDropState for per-transport delta tracking with 1s tick sampling via sample_transport_congestion() - Extend detect_congestion() with transport kernel drop check alongside MMP loss metrics Congestion monitoring and control: - Add CongestionStats (ce_forwarded, ce_received, congestion_detected, kernel_drop_events) to NodeStats with snapshot serialization - Wire counters into forwarding path, session handler, and transport drop sampling with rate-limited warn logging (5s interval) - Expose congestion data in show_routing control query and ecn_ce_count in show_mmp peer entries - Add congestion counters to fipstop routing tab in two-column layout Chaos harness integration: - Add query_routing(), query_transports(), snapshot_all_congestion() to chaos control module - Add congestion/kernel-drop log analysis in logs module - Add congestion-stress scenario: 10-node tree, 1 Mbps bandwidth, 5-10% netem loss, heavy iperf3 traffic - Add IngressConfig for tc ingress policing with per-peer policer filters simulating upstream bandwidth bottlenecks - Add iperf3 JSON result capture to traffic manager for throughput measurement across scenarios - Add ECN A/B test scenarios (ecn-ab-on/off.yaml) with ingress policing and comparison script - Enable TCP ECN negotiation (tcp_ecn=1 sysctl) in container entrypoint for end-to-end CE propagation Tests: - 10 ECN unit/integration tests: mark_ipv6_ecn_ce variants, CE relay chain (3-node propagation), EcnConfig serde roundtrip - 3 transport drop congestion detection unit tests Documentation: - Update fips-mesh-layer.md: replace outdated CE Echo stub with full ECN Congestion Signaling section covering detection logic, CE relay, IPv6 marking, session tracking, and monitoring counters - Update fips-configuration.md: add node.ecn.* parameter table and ecn block in complete reference YAML - Update fips-transport-layer.md: add Congestion Reporting section with TransportCongestion struct, congestion() trait method, and per-transport status; document AsyncFd/recvmsg/SO_RXQ_OVFL in UDP - Update chaos README: add congestion/ECN scenario docs, ingress traffic control, and iperf3 JSON capture sections - Update README.md: add ECN to features list and "What works today"; update transport and tooling entries
This commit is contained in:
Johnathan Corgan
@@ -163,6 +163,25 @@ Controls tree construction and parent selection.
|
||||
Bloom filter size (1 KB), hash count (5), and size classes are protocol
|
||||
constants and not configurable.
|
||||
|
||||
### ECN Signaling (`node.ecn.*`)
|
||||
|
||||
Controls hop-by-hop ECN (Explicit Congestion Notification) signaling. When
|
||||
enabled, transit nodes detect congestion on outgoing links (via MMP loss/ETX
|
||||
metrics or kernel buffer drops) and set the CE flag on forwarded FMP frames.
|
||||
Destination nodes mark ECN-capable IPv6 packets with CE before TUN delivery
|
||||
per RFC 3168, enabling end-host TCP congestion control to react.
|
||||
|
||||
| Parameter | Type | Default | Description |
|
||||
|-----------|------|---------|-------------|
|
||||
| `node.ecn.enabled` | bool | `true` | Enable ECN congestion signaling (CE flag relay and local congestion detection) |
|
||||
| `node.ecn.loss_threshold` | f64 | `0.05` | MMP loss rate threshold for CE marking (0.0–1.0). When the outgoing link's loss rate meets or exceeds this value, forwarded packets are CE-marked. |
|
||||
| `node.ecn.etx_threshold` | f64 | `3.0` | MMP ETX threshold for CE marking (≥1.0). When the outgoing link's ETX meets or exceeds this value, forwarded packets are CE-marked. |
|
||||
|
||||
Congestion detection triggers on any of: outgoing link loss ≥ `loss_threshold`,
|
||||
outgoing link ETX ≥ `etx_threshold`, or kernel receive buffer drops detected on
|
||||
any local transport. CE is relayed hop-by-hop: once set on any hop, the flag
|
||||
stays set for all subsequent hops to the destination.
|
||||
|
||||
### Session / Data Plane (`node.session.*`)
|
||||
|
||||
Controls end-to-end session behavior and packet queuing.
|
||||
@@ -478,6 +497,10 @@ node:
|
||||
mode: full # full | lightweight | minimal
|
||||
log_interval_secs: 30
|
||||
owd_window_size: 32
|
||||
ecn:
|
||||
enabled: true # ECN congestion signaling (CE flag relay)
|
||||
loss_threshold: 0.05 # MMP loss rate threshold for CE marking (5%)
|
||||
etx_threshold: 3.0 # MMP ETX threshold for CE marking
|
||||
control:
|
||||
enabled: true
|
||||
socket_path: null # null = auto ($XDG_RUNTIME_DIR → /run/fips → /tmp fallback)
|
||||
|
||||
@@ -465,12 +465,44 @@ MMP reports on different timers), inter-frame processing delays inflate spin
|
||||
bit RTT measurements unpredictably. Timestamp-echo from ReceiverReports
|
||||
(with dwell-time compensation) is the sole SRTT source.
|
||||
|
||||
### CE Echo
|
||||
### ECN Congestion Signaling
|
||||
|
||||
The CE (Congestion Experienced) echo flag (bit 1 in the FMP flags byte) is
|
||||
reserved for ECN signaling. The transport trait does not currently expose
|
||||
ECN marking, so the CE echo flag is never set. One-way delay trend serves
|
||||
as the sole pre-loss congestion indicator.
|
||||
The CE (Congestion Experienced) flag (bit 1 in the FMP flags byte) provides
|
||||
hop-by-hop congestion signaling through the mesh. Transit nodes detect
|
||||
congestion on outgoing links and set CE on forwarded packets; once set, the
|
||||
flag stays set for all subsequent hops to the destination.
|
||||
|
||||
**Congestion detection** (`detect_congestion()`) triggers on any of:
|
||||
|
||||
- Outgoing link MMP loss rate ≥ `node.ecn.loss_threshold` (default 5%)
|
||||
- Outgoing link MMP ETX ≥ `node.ecn.etx_threshold` (default 3.0)
|
||||
- Kernel receive buffer drops detected on any local transport (via
|
||||
`SO_RXQ_OVFL` on UDP)
|
||||
|
||||
**CE relay**: The forwarding path computes `outgoing_ce = incoming_ce ||
|
||||
local_congestion`. The `send_encrypted_link_message_with_ce()` method ORs
|
||||
`FLAG_CE` into the FMP header flags when ce is true. The original
|
||||
`send_encrypted_link_message()` delegates with `ce_flag=false`, leaving the
|
||||
20+ existing call sites unchanged.
|
||||
|
||||
**IPv6 ECN-CE marking**: When a CE-flagged DataPacket arrives at its final
|
||||
destination, the IPv6 Traffic Class ECN bits are marked CE (0b11) before
|
||||
TUN delivery — but only for ECN-capable packets (ECT(0) or ECT(1)). Not-ECT
|
||||
packets are never marked per RFC 3168. The host TCP stack then echoes ECE in
|
||||
ACKs, triggering sender cwnd reduction through standard congestion control.
|
||||
|
||||
**Session-layer tracking**: The `ecn_ce_count` field in MMP ReceiverReports
|
||||
tracks CE-flagged packets received per link, providing end-to-end visibility
|
||||
into congestion propagation.
|
||||
|
||||
**Monitoring**: `CongestionStats` tracks four counters — `ce_forwarded`,
|
||||
`ce_received`, `congestion_detected`, and `kernel_drop_events` — exposed via
|
||||
`fipsctl show routing` (congestion block) and `fipstop` (routing tab).
|
||||
Rate-limited warn logging (5s interval) alerts on congestion detection events.
|
||||
|
||||
See `node.ecn.*` in
|
||||
[fips-configuration.md](fips-configuration.md#ecn-signaling-nodeecn) for
|
||||
tuning parameters.
|
||||
|
||||
### Operator Logging
|
||||
|
||||
@@ -540,6 +572,7 @@ an attacker sends invalid packets to elicit responses.
|
||||
| Inner header timestamps | **Implemented** |
|
||||
| Path MTU tracking (SessionDatagram) | **Implemented** |
|
||||
| Metrics Measurement Protocol (MMP) | **Implemented** |
|
||||
| ECN congestion signaling (CE relay, IPv6 marking) | **Implemented** |
|
||||
| Rekey with index rotation | Planned |
|
||||
| Allowlist/blocklist | Planned |
|
||||
|
||||
|
||||
@@ -215,8 +215,15 @@ buffer fills in ~2.5 ms; any stall in the async receive loop (decryption,
|
||||
routing, forwarding overhead) causes the kernel to silently drop incoming
|
||||
datagrams.
|
||||
|
||||
FIPS uses the `socket2` crate to configure socket buffers at bind time,
|
||||
before the receive loop starts:
|
||||
FIPS uses `socket2::Socket` wrapped in `tokio::io::unix::AsyncFd` for the
|
||||
UDP receive path. This replaces `tokio::UdpSocket` and enables direct
|
||||
`libc::recvmsg()` calls with ancillary data parsing — specifically the
|
||||
`SO_RXQ_OVFL` socket option, which delivers a cumulative kernel receive
|
||||
buffer drop counter on every received packet. The drop counter feeds into
|
||||
the ECN congestion detection system (see
|
||||
[fips-mesh-layer.md](fips-mesh-layer.md#ecn-congestion-signaling)).
|
||||
|
||||
Socket buffers are configured at bind time via `socket2`:
|
||||
|
||||
| Parameter | Default | Description |
|
||||
| ---------------- | ------- | ------------------------------------ |
|
||||
@@ -486,6 +493,7 @@ start() → lifecycle Bring transport up (bind socket, o
|
||||
stop() → lifecycle Bring transport down
|
||||
send(addr, data) → delivery Send datagram to transport address
|
||||
close_connection(addr)→ () Close a specific connection (no-op for connectionless)
|
||||
congestion() → TransportCongestion Local congestion indicators (optional)
|
||||
discover() → Vec<DiscoveredPeer> Report discovered FIPS endpoints (optional)
|
||||
auto_connect() → bool Auto-connect discovered peers (default: false)
|
||||
accept_connections() → bool Accept inbound handshakes (default: true)
|
||||
@@ -520,6 +528,30 @@ TransportType {
|
||||
|
||||
Predefined types exist for UDP, TCP, Ethernet, WiFi, Tor, and Serial.
|
||||
|
||||
### Congestion Reporting
|
||||
|
||||
Transports optionally report local congestion indicators via a
|
||||
`TransportCongestion` struct, providing a transport-agnostic interface for
|
||||
the node layer's ECN congestion detection:
|
||||
|
||||
```text
|
||||
TransportCongestion {
|
||||
recv_drops: Option<u64> Cumulative kernel-dropped packets (monotonic)
|
||||
}
|
||||
```
|
||||
|
||||
The node samples each transport's congestion state on a 1-second tick via
|
||||
`sample_transport_congestion()`. `TransportDropState` tracks per-transport
|
||||
drop deltas: when new drops appear (rising edge), the `dropping` flag is
|
||||
set, and `detect_congestion()` in the forwarding path triggers CE marking
|
||||
on all forwarded datagrams.
|
||||
|
||||
| Transport | Congestion Source | Mechanism |
|
||||
| --------- | ----------------- | --------- |
|
||||
| UDP | `SO_RXQ_OVFL` kernel drop counter | `recvmsg()` ancillary data on every packet |
|
||||
| TCP | Not yet implemented | Returns `None` (TCP handles congestion internally) |
|
||||
| Ethernet | Not yet implemented | Returns `None` |
|
||||
|
||||
### Transport Addresses
|
||||
|
||||
Transport addresses (`TransportAddr`) are opaque byte vectors. The transport
|
||||
@@ -542,7 +574,7 @@ transitions through `Starting` to `Up` (operational). `stop()` moves to
|
||||
|
||||
| Transport | Status | Notes |
|
||||
| --------- | ------ | ----- |
|
||||
| UDP/IP | **Implemented** | Primary transport, async send/receive, configurable MTU |
|
||||
| UDP/IP | **Implemented** | Primary transport, AsyncFd/recvmsg, SO_RXQ_OVFL kernel drop detection |
|
||||
| TCP/IP | **Implemented** | FMP header-based framing, connect-on-send, per-connection MSS MTU |
|
||||
| Ethernet | **Implemented** | AF_PACKET SOCK_DGRAM, EtherType 0x2121, beacon discovery, Linux only |
|
||||
| WiFi | Future direction | Infrastructure mode = Ethernet driver |
|
||||
|
||||
Reference in New Issue
Block a user