Drop the `tui`, `ble`, and `gateway` cargo features and replace
them with platform cfg gates. Plain `cargo build` now produces
every subsystem appropriate for the target platform with no
feature flags required.
Motivation:
- `default = ["tui", "ble"]` broke `cargo build` on macOS and
Windows because `ble` pulled in `bluer` (BlueZ, Linux-only).
Every non-Linux packager needed `--no-default-features`.
- The feature flags on `ble` and `gateway` were redundant with
their platform-gated deps (`bluer`, `rustables`). The parallel
gating was inconsistent and error-prone.
- `tui` feature protected against a ratatui binary-size concern
that no longer applies in 2026.
Cargo.toml:
- Remove `tui`, `ble`, `gateway` features; `default = []`.
- Promote `ratatui` to a non-optional top-level dependency.
- Move `rustables` from top-level optional into the Linux
target block, non-optional.
- Split `bluer` into its own target block with
`cfg(all(target_os = "linux", not(target_env = "musl")))`
— BlueZ isn't available on musl router targets and
`libdbus-sys` doesn't cross-compile to musl without pkg-config
sysroot setup.
- Drop `required-features` from the `fipstop` and `fips-gateway`
`[[bin]]` entries.
build.rs:
- Emit a `bluer_available` custom cfg when `target_os == "linux"`
and `target_env != "musl"`, for use in place of the verbose
full predicate in source cfg gates.
Source:
- Replace every `#[cfg(feature = "gateway")]` with
`#[cfg(target_os = "linux")]`. Gateway code works on both
glibc and musl Linux (rustables is fine on musl).
- Replace every `#[cfg(feature = "ble")]` with
`#[cfg(bluer_available)]`. BLE-specific code (BluerIo module,
bluer type conversions, BLE transport instance creation,
resolve_ble_addr) is excluded on musl and non-Linux. Generic
`BleAddr`, `BleIo` trait, `MockBleIo`, and `BleTransport<I>`
still compile on all targets.
- `src/bin/fips-gateway.rs`: always compiled, but `main()` is
gated to Linux. Non-Linux stub exits 1 with a diagnostic.
Existing non-Linux packaging scripts don't ship it, so the
stub binary sits unused.
Packaging and CI:
- Drop `--features` and `--no-default-features` flags from every
packaging script and workflow. Defaults now match each
platform's capabilities.
- AUR `fips-git` automatically aligns with stable `PKGBUILD`
(both build with defaults).
Verified: `cargo build --release` with no flags produces all
four binaries on glibc Linux; all unit and integration tests
pass across Linux/macOS/Windows/OpenWrt (musl) in CI.
A malformed FilterAnnounce whose fill ratio produces an implausibly
high false-positive rate is mostly useless for routing and, once
merged into our outgoing filter via bitwise OR, propagates the
saturated state to tree peers one hop per announce tick. A saturated
filter also made estimated_count() return f64::INFINITY, which
compute_mesh_size summed into its cached estimate.
handle_filter_announce now rejects inbound FilterAnnounce whose
derived FPR exceeds `node.bloom.max_inbound_fpr` (new config field,
default 0.05 ≈ fill 0.549 at k=5). Rejection is silent on the wire,
logs at WARN, and increments a new `bloom.fill_exceeded` counter. The
peer's prior stored filter and filter_sequence are left unchanged so
a single rejected announce does not wipe the peer's existing
contribution to aggregation.
After a successful outgoing FilterAnnounce send, a rate-limited WARN
fires if our own filter's FPR exceeds the same cap, surfacing
aggregation drift. Limited to once per 60 seconds via a new
Node.last_self_warn field.
BloomFilter::estimated_count() now takes max_fpr and returns
Option<f64>. Returns None for saturated filters (regardless of cap)
or when the filter's FPR exceeds max_fpr. Callers updated: debug
logs render None as "—", the Debug impl uses f64::INFINITY as "no
cap" and prints "saturated" instead of inf, control-socket JSON
emits null, and compute_mesh_size propagates None into the already-
Option<u64> estimated_mesh_size field.
Implement TCP Wrappers-style peer access control using
/etc/fips/peers.allow and /etc/fips/peers.deny files. Evaluation
order: allow overrides deny, default permit when no files exist.
Three enforcement points: outbound connect (before dialing), inbound
handshake (msg1 receipt, after restart/rekey classification), and
outbound handshake completion (msg2, before peer promotion). Files
support npub, hex pubkey, host alias, and ALL wildcard entries with
automatic mtime-based reload.
Adds fipsctl acl show query, 954-line acl module with unit tests,
and a 6-node Docker integration harness (testing/acl-allowlist/)
exercising insider, outsider, and allowed-remote scenarios. CI
matrix entry included.
Closes#50
Co-authored-by: Johnathan Corgan <johnathan@corganlabs.com>
In-memory time-series history on the daemon: fast ring (1s × 3600)
plus slow ring (1m × 1440) per metric, covering node-level gauges
(mesh size, tree depth, peer count, active sessions), counters
(parent switches, aggregate bytes/packets in/out), loss rate, and
seven per-peer metrics keyed by NodeAddr (srtt_ms, loss_rate,
bytes_in/out, packets_in/out, ecn_ce). The slow ring is produced by
downsampling the fast ring on minute boundaries with Last / Sum /
Mean aggregation chosen per metric type.
Missing data is first-class. New peers back-fill NaN so every ring
shares a time axis with the node rings; peers absent from a tick
sample NaN (keeps alignment, shows as a visible gap); counter metrics
emit NaN on decrease (new link_stats baseline after reconnect) so
deltas aren't polluted. Peers are evicted 24h after last contact.
Downsampling is NaN-aware: mean skips NaN, all-NaN slow windows stay
NaN. Each history window always returns its full span at the chosen
density (1m / 10m / 1h / 24h), front-padded with NaN when the ring
hasn't yet accumulated enough samples, so switching between windows
feels like zooming in or out rather than clipping to whatever has
arrived. NaN serializes to JSON null via a custom serializer.
Control socket queries:
- show_stats_list enumerates registered metrics plus scope field and
peer_retention_seconds.
- show_stats_history returns one metric's series for a given window
and granularity; accepts optional peer (npub) for per-peer metrics.
- show_stats_all_history returns every metric in a single round trip;
accepts optional peer to fetch all seven per-peer metrics.
- show_stats_peers enumerates tracked peers with lifecycle metadata.
- show_stats_history_all_peers returns one metric across all peers
for grid rendering.
- show_status carries short sparkline windows for the dashboard so
the client can render without extra fetches.
fipsctl gains `stats list`, `stats peers`, and `stats history
<metric>` with `--peer` (hostname or npub) and `--plot` for a Unicode
block sparkline. Plot header reports sample count, granularity,
window, and gap count; NaN renders as a blank cell.
fipstop dashboard grows inline sparklines (peer count, mesh size,
aggregate bytes in/out). A new Graphs tab stacks every metric as an
independent mini plot with its own autoscaled range; each plot uses
btop's braille 2×4 filled-area algorithm (25-entry lookup table
packing two samples per character, per-row gradient coloring for the
characteristic btop vertical-band look, rounded borders with embedded
titles). Three modes are cycled with `m`: Node (node-level stack),
MetricByPeer (small-multiples grid, 1 / 2 / 3 columns by terminal
width), PeerByMetric (existing stack scoped to one peer). `n` / `N`
cycles the mode-specific selector (metric or peer), a selector row
shows the current choice, and Graphs-tab refreshes re-fetch
show_stats_peers so selectors track peer churn. Up / Down scrolls
the stack, Left / Right cycles the window, `g` jumps to the tab.
Implements IDEA-0084 (TASK-2026-0062).
Extend the fipsctl control query interface with visibility into internal
state critical for protocol security auditing, mesh troubleshooting, and
operational monitoring.
New command:
fipsctl show identity-cache
Lists every node identity cached by the daemon (learned from DNS
resolution, peer handshakes, sessions, and static config). Shows
npub, IPv6 address, display name, and LRU age alongside the
configured cache capacity.
Extended queries:
show peers — Noise session counters (send_counter, highest received
counter) for rekey urgency assessment. Per-peer replay suppression
and consecutive decrypt failure counts for active attack detection.
Session index visibility for hijack analysis. Rekey lifecycle
state (in_progress, draining, K-bit epoch).
show sessions — Handshake resend count during establishment for
connectivity debugging. Rekey and session health fields
(session_start, K-bit, coords warmup, drain state) when
established.
show cache — Individual coordinate cache entries with tree
coordinates, depth, path MTU, and age. Enables route-level
debugging by showing exactly which destinations have cached
routes and via what tree path. Renames the top-level count
field from "entries" to "count" for clarity.
show routing — Pending discovery lookups expanded from count to
per-target detail (attempt number, age, last sent). Pending
TUN packet queue depth for backpressure visibility. Connection
retry state per peer (retry count, next attempt, auto-reconnect
flag).
Updates fipstop to match the revised show_cache and show_routing
response schemas. Updates README monitoring section with the complete
fipsctl command list.
Co-authored-by: Johnathan Corgan <johnathan@corganlabs.com>
Gate platform-specific code behind cfg attributes and add full Windows
support: TUN device via wintun, TCP control socket on localhost:21210,
Windows Service lifecycle (--install-service/--uninstall-service/--service),
CI build and test matrix, and packaging with ZIP builder and PowerShell
service management scripts.
Key changes:
- Cargo.toml: move tun/libc/rtnetlink behind cfg(unix); add wintun and
windows-service dependencies for Windows
- upper/tun.rs: wintun-based TUN implementation with netsh configuration
for IPv6 address, MTU, and fd00::/8 routing
- control/mod.rs: split into unix_impl/windows_impl; Windows uses TCP on
localhost:21210 with shared connection handler
- bin/fips.rs: refactor main() into run_daemon() accepting a shutdown
signal; add Windows Service support via windows-service crate
- transport/udp/socket.rs: platform-gated modules; Windows uses
tokio::net::UdpSocket (kernel drop count unavailable, returns 0)
- transport/ethernet: gate to cfg(unix); add Windows stub types
- config: platform-conditional default paths (socket, hosts) for Windows
- CI: add windows-latest to build matrix and test-windows job with
cargo-nextest
- packaging/windows: build-zip.ps1, install-service.ps1,
uninstall-service.ps1, and package-windows.yml workflow
- README/docs: Windows build instructions, service management, and
control socket platform differences
Linux and macOS behavior is unchanged.
macOS platform:
- Platform-native TUN interface management with shutdown pipe
- Raw Ethernet transport with macOS socket backend (socket_macos.rs)
- EthernetTransport and TransportHandle::Ethernet ungated from Linux-only
- macOS .pkg packaging (build-pkg.sh, launchd plist, uninstall script)
- CI: macOS build and unit test jobs; x86_64 cross-compiled from
macos-latest via rustup target add x86_64-apple-darwin
Gateway feature flag:
- New opt-in `gateway` Cargo feature activates optional `rustables` dep
- `pub mod gateway` and `Config.gateway` gated behind the feature so
macOS builds never pull in Linux-only nftables bindings
- `fips-gateway` bin has `required-features = ["gateway"]`
- All Linux/OpenWrt/AUR packaging passes `--features gateway`
CI / packaging:
- package-linux, package-macos, package-openwrt now trigger on push to
master/maint/next and on pull requests; release uploads remain tag-gated
- Bloom filter routing fix: fall through to tree routing when no candidate
is strictly closer
- MMP intervals: raise MIN to 1s / MAX to 5s with 5-sample cold-start phase
When bloom filter candidates existed but none were strictly closer to
the destination in tree-coordinate distance, find_next_hop returned
None without trying greedy tree routing. This caused NoRoute failures
in topologies where the tree parent was closer but not a bloom
candidate. Fall through to tree routing when no bloom candidate makes
progress.
BLE transport implementation using L2CAP Connection-Oriented Channels
(SeqPacket mode) via the bluer crate, behind cfg(feature = "ble").
Core transport:
- BleTransport<I> generic over BleIo trait (BluerIo prod, MockBleIo test)
- Connection pool with priority eviction (static > discovered, max 7)
- Connect-on-send via connect_inline() matching TCP behavior
- Per-connection receive loops with pool cleanup on disconnect
Discovery and probing:
- Combined scan_probe_loop using select! over scanner events and a
BinaryHeap delay queue with per-entry random jitter (0-5s) to prevent
herd effects when multiple nodes see the same beacon simultaneously
- Pre-handshake pubkey exchange ([0x00][pubkey:32]) for IK identity
- Cross-probe tie-breaker: smaller NodeAddr's outbound wins (same
convention as FMP/FSP rekey dual-initiation)
- Probed peers reported to DiscoveryBuffer; pool fills through normal
node-layer auto-connect -> send_async -> connect_inline path
Beacon management:
- Periodic advertising: 1s burst every 30s (configurable via
beacon_interval_secs / beacon_duration_secs)
- FIPS service UUID for scan filtering
Configuration (all fields optional with defaults):
- adapter, psm, mtu, max_connections, connect_timeout_ms
- advertise, scan, auto_connect, accept_connections
- beacon_interval_secs (30), beacon_duration_secs (1)
Hardware validated with two BLE nodes:
- 2048-byte MTU, ~60-160ms RTT, zero-config auto-connect
- BLE spike tool at testing/ble/ for standalone adapter validation
42 unit tests + 4 node-level integration tests, all CI-compatible
via MockBleIo (no hardware required). tokio test-util added for
time-dependent scan/probe tests.
Replace discovery flooding with bloom-filter-guided tree routing:
lookups sent only to tree peers (parent + children) whose bloom
filter contains the target. If no tree peer matches, fall back to
non-tree peers with bloom matches before dropping the request. This
produces single-path forwarding through the spanning tree (90%
traffic reduction) while recovering from dead ends caused by stale
bloom filters, tree restructuring, or transit node failures.
Remove visited bloom filter from LookupRequest wire format (-257
bytes per request). Tree routing is inherently loop-free; request_id
dedup handles edge cases during tree restructuring. Add response-
forwarded flag to prevent response routing loops from convergent
request paths.
Add originator-side exponential backoff (30s base, 300s cap) after
lookup timeouts and bloom misses. Backoff resets on topology changes
(parent switch, new peer, first RTT, reconnection).
Add transit-side per-target rate limiting (2s minimum interval) for
forwarded lookups as defense-in-depth.
Add discovery retry within the timeout window (default: send at T=0,
retry at T=5s, fail at T=10s) to compensate for single-path fragility.
Lookups with zero eligible tree peers fail immediately.
Improve discovery logging: promote key events to info (initiation,
success, timeout with failure count). Add debug logging for dedup,
pending packet retry, backoff suppression, forward rate limiting,
and backoff reset.
New config: discovery.backoff_base_secs, backoff_max_secs,
forward_min_interval_secs, retry_interval_secs, max_attempts.
New stats: req_backoff_suppressed, req_forward_rate_limited,
req_bloom_miss, req_no_tree_peer, req_fallback_forwarded.
Removed: req_already_visited, visited bloom filter.
Add TorTransport in src/transport/tor/ supporting three operating modes:
Outbound (socks5 mode):
- Non-blocking SOCKS5 connect via tokio-socks with per-destination
circuit isolation (IsolateSOCKSAuth)
- TorAddr enum for .onion and clearnet address types
- Connection pool with per-connection receive tasks, reuses TCP
stream FMP framing
- connect_async()/connection_state_sync()/promote_connection() follow
the same non-blocking polling pattern as TCP transport
Inbound (directory mode — recommended for production):
- Tor manages the onion service via HiddenServiceDir in torrc
- FIPS reads .onion address from hostname file at startup
- No control port needed — enables Tor Sandbox 1 (seccomp-bpf)
- Accept loop mirrors TCP pattern with DirectoryServiceConfig
Monitoring (control_port mode and optional in directory mode):
- Async control port client supporting TCP and Unix socket connections
via Box<dyn AsyncRead/Write> trait objects
- AUTHENTICATE with cookie or password auth
- 8 GETINFO queries: bootstrap, circuits, traffic, liveness, version,
dormant state, SOCKS listeners
- Background monitoring task polls every 10s, caches TorMonitoringInfo
in Arc<RwLock> for synchronous query access
- Bootstrap milestone logging (25/50/75/100%), stall warning (>60s),
network liveness transitions, dormant mode entry
- Directory mode optionally connects to control port when control_addr
is configured (non-fatal on failure)
Operator visibility:
- show_transports query exposes tor_mode, onion_address, tor_monitoring
(bootstrap, circuit_established, traffic, liveness, version, dormant)
- fipstop transport detail view: Tor mode, onion address, SOCKS5/control
errors, connection stats, Tor daemon status section
- fipstop table view: tor(mode) label with truncated onion address hint
Security hardening:
- Per-destination circuit isolation via IsolateSOCKSAuth
- Unix socket default for control port (/run/tor/control)
- Reference torrc with HiddenServiceDir, VanguardsLiteEnabled,
ConnectionPadding, DoS protections (PoW + intro rate limiting)
Config:
- TorConfig with socks5, control_port, and directory modes
- DirectoryServiceConfig: hostname_file, bind_addr
- control_addr, control_auth, cookie_path, connect_timeout,
max_inbound_connections
Testing:
- 69 unit + integration tests with mock SOCKS5 and control servers
- Docker tests: socks5-outbound (clearnet via Tor) and directory-mode
(HiddenServiceDir onion service)
Documentation:
- Transport layer design doc: Tor architecture, directory mode
- Configuration doc: Tor config tables and examples
TCP (and future Tor) transports previously established connections
synchronously inside send(), blocking the node's RX event loop during
TCP handshake. This is particularly problematic for Tor where SOCKS5
circuit establishment can take 30-120 seconds.
Add a non-blocking connect path:
- ConnectionState enum in transport layer (None/Connecting/Connected/Failed)
- connect_async() on TcpTransport spawns background TCP connect task
- connection_state_sync() polls task completion, promotes to pool
- TransportHandle gains connect() and connection_state() dispatch methods
- Node tracks PendingConnect entries for connection-oriented transports
- initiate_connection() defers handshake for connection-oriented transports
- start_handshake() extracted as separate method for deferred invocation
- poll_pending_connects() in tick handler polls and completes handshakes
- Failed connects trigger retry via schedule_retry()
Connectionless transports (UDP, Ethernet) are unchanged — connect()
is a no-op and connection_state() always returns Connected.
The existing connect-on-send path in send_async() is preserved as
fallback for reconnection after connection drops.
811 tests pass (6 new), clippy clean.
Compute network size estimate by summing bloom filter estimated entry
counts from the spanning tree parent (upward) and children (downward),
leveraging the tree's non-overlapping partition property. Uses the
standard formula n = -(m/k) * ln(1 - X/m) already in BloomFilter.
Surfaces the estimate in three places:
- show_status JSON: "estimated_mesh_size" field
- Periodic info log at MMP log interval (default 30s)
- fipstop Node dashboard State section as "mesh: ~N"
Add a HostMap that resolves human-readable hostnames to npubs,
enabling `gateway.fips` instead of the full `npub1...xyz.fips`.
Two sources populate the map: peer `alias` fields from the YAML
config and an operator-maintained hosts file at /etc/fips/hosts.
The DNS responder auto-reloads the hosts file on each request by
checking the file modification time, so operators can update
mappings without restarting the daemon.
- New src/upper/hosts.rs: HostMap, HostMapReloader, hostname
validation, hosts file parser with auto-reload on mtime change
- DNS resolver checks host map before falling back to direct npub
- Node uses host map for peer display names
- Default hosts file added to both .deb and tarball packaging
- 26 new tests (789 total)
Implement periodic full rekey at both protocol layers using fresh DH
key exchanges. Uses the existing K-bit flag (FLAG_KEY_EPOCH /
FSP_FLAG_K) to coordinate cutover between peers.
FMP layer (IK pattern):
- ActivePeer gains rekey state: pending/previous sessions, K-bit epoch
tracking, drain window, dampening timer
- Handshake state stored on ActivePeer with msg1 sent on existing link
- Encrypted frame handler detects K-bit flips, promotes pending
sessions, falls back to previous session during drain
- Handshake handlers distinguish rekey from new connections using
addr_to_link lookup with identity-based fallback
- Free all session indices (current, rekey, pending, previous) on
peer removal
FSP layer (XK pattern):
- SessionEntry gains parallel rekey fields with XK-specific state
for the 3-message handshake
- Route availability check before FSP rekey initiation
- Encrypted session handler adds K-bit flip detection and dual-session
decrypt fallback
- SessionSetup/Ack/Msg3 handlers extended for rekey paths
Defense-in-depth:
- Consecutive decryption failure detector (threshold=20) triggers
forced peer removal instead of waiting for link-dead timeout
- Identity-based rekey detection as fallback when addr_to_link
doesn't match (e.g., TCP ephemeral ports)
Configuration: RekeyConfig with enabled flag, after_secs (default 120),
and after_messages (default 65536) thresholds.
Logging: info for successful K-bit cutover completions, warn for
failures, debug for intermediate handshake steps, trace for routine
operations (resends, drain cleanup).
Rekey lifecycle:
1. Timer/counter fires -> initiator starts new handshake
2. Old session continues handling traffic during handshake
3. Handshake completes -> initiator cuts over, flips K-bit
4. Responder sees flipped K-bit -> promotes new session
5. Both keep old session for 10s drain window
6. After drain, old session discarded
Integration test: Docker-based multi-phase test exercising both FMP
and FSP rekey with aggressive timers (35s). Verifies connectivity
across all 20 directed pairs survives two consecutive rekey cycles.
Includes rekey topology, docker-compose profile, and CI matrix entry.
Increase ping test convergence wait from 3s to 5s for CI reliability.
Implement hop-by-hop ECN congestion signaling through the FMP layer,
transport-level congestion detection via kernel drop counters, and
chaos harness integration for end-to-end validation.
FMP/session ECN plumbing:
- Thread ce_flag parsed at link layer through dispatch_link_message,
handle_session_datagram, handle_session_payload, and
handle_encrypted_session_msg to session delivery
- Replace hardcoded false in session-layer record_recv() with actual
ce_flag, activating ecn_ce_count tracking in session MMP
ECN congestion detection and CE relay:
- Add EcnConfig (node.ecn.*) with configurable loss_threshold (5%)
and etx_threshold (3.0) for transit congestion detection
- Add send_encrypted_link_message_with_ce() that ORs FLAG_CE into FMP
header flags; original method delegates with ce_flag=false
- Compute outgoing_ce = incoming_ce || local congestion on next-hop
link, enabling hop-by-hop CE relay through transit nodes
IPv6 ECN-CE marking:
- Mark ECN-CE (0b11) in IPv6 Traffic Class on received DataPackets
before TUN delivery when FMP CE flag is set
- Only marks ECN-capable packets (ECT(0)/ECT(1)); Not-ECT packets
unchanged per RFC 3168
Transport congestion abstraction and UDP kernel drop detection:
- Add TransportCongestion struct to transport layer for transport-
agnostic local congestion indicators
- Replace tokio::UdpSocket with AsyncFd<socket2::Socket> using
libc::recvmsg() with ancillary data parsing
- Enable SO_RXQ_OVFL for kernel receive buffer drop counter on every
packet, wiring up previously-stubbed UdpStats.kernel_drops
- Add TransportDropState for per-transport delta tracking with 1s
tick sampling via sample_transport_congestion()
- Extend detect_congestion() with transport kernel drop check
alongside MMP loss metrics
Congestion monitoring and control:
- Add CongestionStats (ce_forwarded, ce_received, congestion_detected,
kernel_drop_events) to NodeStats with snapshot serialization
- Wire counters into forwarding path, session handler, and transport
drop sampling with rate-limited warn logging (5s interval)
- Expose congestion data in show_routing control query and
ecn_ce_count in show_mmp peer entries
- Add congestion counters to fipstop routing tab in two-column layout
Chaos harness integration:
- Add query_routing(), query_transports(), snapshot_all_congestion()
to chaos control module
- Add congestion/kernel-drop log analysis in logs module
- Add congestion-stress scenario: 10-node tree, 1 Mbps bandwidth,
5-10% netem loss, heavy iperf3 traffic
- Add IngressConfig for tc ingress policing with per-peer policer
filters simulating upstream bandwidth bottlenecks
- Add iperf3 JSON result capture to traffic manager for throughput
measurement across scenarios
- Add ECN A/B test scenarios (ecn-ab-on/off.yaml) with ingress
policing and comparison script
- Enable TCP ECN negotiation (tcp_ecn=1 sysctl) in container
entrypoint for end-to-end CE propagation
Tests:
- 10 ECN unit/integration tests: mark_ipv6_ecn_ce variants, CE relay
chain (3-node propagation), EcnConfig serde roundtrip
- 3 transport drop congestion detection unit tests
Documentation:
- Update fips-mesh-layer.md: replace outdated CE Echo stub with full
ECN Congestion Signaling section covering detection logic, CE relay,
IPv6 marking, session tracking, and monitoring counters
- Update fips-configuration.md: add node.ecn.* parameter table and
ecn block in complete reference YAML
- Update fips-transport-layer.md: add Congestion Reporting section
with TransportCongestion struct, congestion() trait method, and
per-transport status; document AsyncFd/recvmsg/SO_RXQ_OVFL in UDP
- Update chaos README: add congestion/ECN scenario docs, ingress
traffic control, and iperf3 JSON capture sections
- Update README.md: add ECN to features list and "What works today";
update transport and tooling entries
Two bugs prevented Path MTU Discovery from working across heterogeneous
links (e.g., ethernet→UDP boundary):
1. ICMPv6 Packet Too Big wrote the MTU as u16 into a 32-bit field
(RFC 4443 §3.2), causing the kernel to read an inflated value.
Fixed by changing build_packet_too_big() to use u32 and writing
all 4 bytes.
2. handle_tun_outbound() only checked the local transport MTU, not
the per-destination PathMtuState updated by MtuExceeded signals.
Added a PathMtuState check after session lookup so subsequent
oversized packets generate ICMPv6 PTB on TUN instead of being
forwarded and dropped at the bottleneck hop.
Added integration test exercising the full PMTUD loop across a 3-node
chain with heterogeneous MTUs: oversized packet → forwarding failure →
MtuExceeded signal → PathMtuState update → ICMPv6 PTB on TUN.
Implement TCP transport for FIPS enabling firewall traversal and serving
as the foundation for future Tor transport. This is the first
connection-oriented transport in the system.
Key design decisions:
- FMP header-based framing: reuses existing 4-byte FMP common prefix for
packet boundary recovery with zero framing overhead
- Session survives TCP reconnection: Noise/MMP/FSP state bound to npub,
not TCP connection; MMP liveness is sole authority for peer death
- Connect-on-send: fresh connection on first send, transparent reconnect
- close_connection() trait method for cross-connection deduplication cleanup
New transport files:
- src/transport/tcp/mod.rs: TcpTransport, connection pool, accept loop
- src/transport/tcp/stream.rs: FMP-aware stream reader (shared with Tor)
Modified: transport trait (close_connection), TcpConfig, TransportHandle
match arms, create_transports(), initiate_connection() for connection-
oriented links, cross-connection tie-breaker cleanup, design docs.
Tree announce loop and TCP stability fixes:
- Preserve tree announce rate-limit state across reconnection: carry
forward last_tree_announce_sent_ms when a peer reconnects so the
rate-limit window isn't reset to zero
- Drop oversize TCP packets at sender: pre-send MTU check returns
MtuExceeded instead of writing to the stream, preventing receiver-side
connection teardown and reset-reconnect cycles
Chaos harness:
- TCP transport support: tcp_edges/has_tcp/tcp_peers in SimTopology,
transport-aware config_gen with per-edge transport type, TCP port 443,
pure-TCP node support
- Include all non-Ethernet edges in directed_outbound()
- Fix netem/links log messages to say "IP-based" instead of "UDP"
- Add tcp-chain, tcp-only, and tcp-mesh scenario files
Static harness:
- Transport-aware config generation (get_default_transport, transport_port)
- TCP transport injection via Python post-processing
- Add tcp-chain topology and docker-compose profile
Implement raw Ethernet transport using AF_PACKET SOCK_DGRAM on Linux
with EtherType 0x88B5 (IEEE experimental range) and 1-byte frame type
prefix (0x00=data, 0x01=beacon).
Transport implementation:
- EthernetConfig with interface, ethertype, MTU, buffer sizes, and
four independent discovery knobs (discovery, announce, auto_connect,
accept_connections)
- PacketSocket/AsyncPacketSocket wrappers with ioctl helpers for
interface index, MAC address, and MTU queries
- EthernetTransport with Transport trait impl, async start/stop/send,
receive loop dispatching data frames and discovery beacons
- Discovery beacons (34 bytes: type + version + x-only pubkey) with
DiscoveryBuffer for peer accumulation and dedup
- Atomic statistics counters (frames, bytes, errors, beacons)
- Platform-gated with #[cfg(target_os = "linux")]
Transport-layer discovery integration:
- Promote auto_connect() and accept_connections() to Transport trait
with default implementations and TransportHandle dispatch
- Extract initiate_connection() so both static peer config and
discovery auto-connect share the same handshake initiation path
- Add poll_transport_discovery() to the tick handler to drain
discovery buffers and auto-connect to discovered peers
- Enforce accept_connections() in handle_msg1() — transports with
accept_connections=false silently drop inbound handshakes
Node integration:
- create_transports() handles Ethernet named instances
- resolve_ethernet_addr() parses "interface/mac" address format
- transport_mtu() generalized for multi-transport operation
Test harness:
- VethPair RAII struct for veth pair lifecycle management
- Three #[ignore] integration tests requiring root/CAP_NET_RAW:
two-node handshake, data exchange, mixed transport coexistence
- Chaos harness: transport-aware topology model, VethManager for
veth pairs between Docker containers, Ethernet-aware config gen,
netem split (HTB+u32 for UDP, root netem for veth), transport-aware
link flaps and node churn with veth re-setup
- Container entrypoint waits for configured Ethernet interfaces
before starting FIPS (handles veth creation timing)
- New scenarios: ethernet-only (4-node ring), ethernet-mesh (6-node
mixed UDP+Ethernet with netem and link flaps)
Documentation:
- fips-transport-layer.md: Ethernet section, beacon discovery, WiFi
compatibility, updated discovery state, trait surface additions,
implementation status table
- fips-configuration.md: Ethernet parameter table, named instances,
peer address format, mixed UDP+Ethernet example, complete reference
- fips-wire-formats.md: Ethernet frame type prefix note
Bump rand (0.8→0.10), rtnetlink (0.14→0.20), tun (0.7→0.8),
simple-dns (0.9→0.11), socket2 (0.5→0.6), and criterion (0.5→0.8).
Migrate all rand call sites: thread_rng()→rng(), gen()→random(),
gen_range()→random_range(), RngCore→Rng trait. Work around secp256k1
0.30 requiring rand 0.8 by generating random bytes directly and
constructing SecretKey from slice.
Migrate rtnetlink to builder-based API: LinkSetRequest replaced with
LinkUnspec builder + change(), RouteAddRequest replaced with
RouteMessageBuilder.
Remove bloom benchmark (criterion 0.8 incompatible with old harness
config).
Track parent switch frequency in a sliding window. When switches exceed
a configurable threshold (default 4 in 60s), impose an extended hold-down
period (default 120s) that prevents non-mandatory parent changes.
Mandatory switches (parent loss, root change, shouldn't-be-root) bypass
dampening. The flap counter resets when the window expires naturally.
Implements TASK-2026-0030 / IDEA-0013.
Guard discovery triggers in handle_path_broken() and
handle_coords_required() with has_cached_identity() check. When the
XK responder receives an error signal before msg3 completes, the
initiator's identity is unknown, making LookupResponse proof
verification impossible. Skip discovery in this case — the handshake
retry mechanism handles recovery.
Downgrade the identity_cache miss log from ERROR to WARN since it's a
known race condition, not a bug.
Cost-based parent selection:
- Replace depth-only parent selection with effective_depth = depth + link_cost
- link_cost computed from locally measured MMP metrics: etx * (1.0 + srtt_ms / 100.0)
- Prevents bottleneck subtrees in heterogeneous networks where a LoRa link
at depth 1 would otherwise always beat fiber at depth 2
- Configurable hysteresis (default 0.2) prevents marginal parent switches
- Configurable hold-down timer (default 30s) suppresses re-evaluation
after parent switch
- Mandatory switches (parent lost, root change) bypass both safeguards
- Link costs passed as HashMap parameter to keep TreeState pure
Periodic re-evaluation:
- evaluate_parent() was only called on TreeAnnounce receipt or parent loss;
after tree stabilization, link degradation went undetected
- Added timer-based re-evaluation (reeval_interval_secs, default 60s) that
calls evaluate_parent() from the tick handler with current MMP link costs
- Respects existing hold-down and hysteresis safeguards
- Short-circuits when disabled or <2 peers
Design documentation:
- Update 7 design docs to reflect cost-based parent selection
- Replace depth-only algorithm descriptions with effective_depth model
- Replace rejected cumulative path cost spec with local-only design rationale
- Rewrite Example 2 (heterogeneous links) for local-only cost model
- Update config docs: parent_switch_threshold replaced by parent_hysteresis,
hold_down_secs, reeval_interval_secs
Chaos simulation enhancements:
- fips_overrides with deep merge for per-scenario FIPS config customization
- Explicit topology algorithm for deterministic test graphs
- Control socket querying via fipsctl for tree/MMP snapshot collection
- Edge existence validation in netem manager
- Per-link netem policy overrides
- 9 new chaos scenarios covering cost avoidance, depth-vs-cost tradeoffs,
stability, mixed topologies, periodic re-evaluation, and bottleneck parent
12 new unit tests, 667 total passing, clippy clean.
Gate peer_inbound_filters() to only collect from tree peers (parent
and children), so outgoing filter computation merges only tree-sourced
information. All peers still receive FilterAnnounce messages and store
filters locally for routing queries — the restriction is only on what
gets merged into outgoing filters.
This prevents bloom filter saturation where mesh shortcuts cause every
node's filter to converge toward the full network. With tree-only
merge, filters contain subtree (from children) + complement (from
parent) + single-hop mesh views.
Implementation:
- Add is_tree_peer() helper to determine tree parent/child relationship
- Gate peer_inbound_filters() to tree peers only (single control point)
- Trigger bloom filter exchange on tree relationship changes
- Add est_entries, set_bits, fill ratio, and tree_peer fields to
FilterAnnounce send/receive debug logs
- Add test_bloom_filter_split_horizon test verifying directional
asymmetry: upward filters contain only the child's subtree, downward
filters contain only the complement
- Add print_filter_cardinality diagnostic helper for test inspection
Design docs:
- fips-bloom-filters.md: Add directional asymmetry and mesh peer filter
subsections, update per-peer filter model, saturation mitigation,
implementation status table
- fips-mesh-operation.md: Update filter propagation description, add
directional asymmetry, tree relationship change trigger
- fips-intro.md: Rewrite bloom propagation paragraph for tree-only merge
Add a new session-layer error signal that transit routers send back to
the source when a forwarded packet exceeds the next-hop transport MTU.
This complements the existing proactive path MTU discovery (min'd at
each hop) by providing immediate feedback when oversized packets are
dropped, closing the transient window before the proactive mechanism
converges.
Wire format: 36-byte payload (msg_type + flags + dest_addr + reporter +
mtu) with FSP phase=0x0 and U flag set, matching the existing
CoordsRequired/PathBroken pattern.
Changes:
- Add SessionMessageType::MtuExceeded (0x22) and MtuExceeded struct with
encode/decode methods to protocol/session.rs
- Add NodeError::MtuExceeded variant to propagate structured MTU info
from TransportError through send_encrypted_link_message()
- Catch MtuExceeded in the forwarding path and send error signal back to
the datagram source via send_mtu_exceeded_error(), rate-limited by the
existing routing_error_rate_limiter
- Handle incoming MtuExceeded at the source by calling
PathMtuState::apply_notification() for immediate MTU decrease
- Add unit tests for encode/decode roundtrip, boundary MTU values, and
too-short payload rejection
Add a Unix domain socket interface for querying node state at runtime.
A spawned tokio task accepts connections and communicates with the main
event loop via mpsc/oneshot channels, keeping all Node access
single-threaded.
Includes:
- src/control/ module with socket lifecycle, JSON protocol, and 11
query handlers (status, peers, links, tree, sessions, bloom, mmp,
cache, connections, transports, routing)
- Separate fipsctl binary for CLI queries (fipsctl show <command>)
- ControlConfig in node configuration (enabled, socket_path)
- Integration into the main select! event loop
Each node generates a random 8-byte startup epoch, encrypted inside
both Noise IK handshake messages (msg1 and msg2). When a peer's msg1
arrives with a different epoch than the stored value, the node tears
down the stale session and processes the msg1 as a new connection,
enabling near-instant restart detection instead of the 30-second
dead timeout.
Wire format impact:
- msg1: 82 -> 106 bytes (added 24-byte encrypted epoch after ss DH)
- msg2: 33 -> 57 bytes (added 24-byte encrypted epoch after se DH)
- Wire msg1: 90 -> 114 bytes, wire msg2: 45 -> 69 bytes
Implement hybrid coordinate cache warming strategy: piggyback coords
via CP flag when they fit within transport MTU, send standalone
CoordsWarmup (0x14) message when they don't. On CoordsRequired or
PathBroken receipt, send CoordsWarmup immediately with source-side
rate limiting (default 2s per destination, configurable).
- Add CoordsWarmup = 0x14 session message type (empty body, CP flag)
- Add send_coords_warmup() following send_session_msg() pattern
- Restructure send_session_data() to send standalone warmup before
data packet when piggybacked coords exceed MTU
- Add immediate CoordsWarmup response in handle_coords_required()
and handle_path_broken() with per-destination rate limiting
- Add coords_response_interval_ms config (node.session)
- Add RoutingErrorRateLimiter::with_interval() constructor
- Zero transit-path changes: existing try_warm_coord_cache() handles
CoordsWarmup identically to CP-flagged data packets
- Update design docs (session layer, wire formats, mesh operation,
configuration)
Replace raw NodeAddr hex strings in log output with human-readable
identifiers using a four-tier lookup: configured alias, active peer
short npub, session endpoint short npub, or truncated hex fallback.
- Add PeerIdentity::short_npub() for compact npub display (npub1xxxx...yyyy)
- Add NodeAddr::short_hex() for compact hex fallback (first 4 bytes + ...)
- Add peer_aliases map populated at startup from peer config
- Add Node::peer_display_name() with four-tier resolution
- Add SessionEntry::remote_pubkey() accessor for session-layer lookups
- Update all 120 log field occurrences across 11 handler/node files
- Pass pre-computed display names to MMP static metric/teardown methods
to work around borrow checker constraints in iterator loops
- PathBroken handler: convert to async, trigger re-discovery via
maybe_initiate_lookup(), reset COORDS_PRESENT warmup counter
(was a stub that only invalidated coord_cache)
- CoordsRequired recovery timing: reset warmup counter in
handle_lookup_response() when discovery completes for an
established session, so COORDS_PRESENT packets fire after
fresh coords are available (not just on CoordsRequired receipt)
- Routing error rate limiting: add RoutingErrorRateLimiter
(100ms per-destination, matching ICMP PTB pattern) to gate
send_routing_error() at transit nodes
- Remove root refresh dead code: the 1800s periodic root
re-announcement in check_tree_state() only propagated to
depth 1 (sequence-only changes don't cascade). Root loss
detection relies on link failure propagation which works
correctly.
Identity cache: remove TTL-based expiry (60s TTL broke active sessions
after expiry since handle_tun_outbound checks identity_cache before
session table). Replace with LRU-only eviction bounded by configurable
identity_size (default 10K). Lookup now touches timestamp for LRU
freshness.
Cache merge: unify coord_cache and route_cache into single coordinate
cache. Both stored NodeAddr→TreeCoordinate; the layer distinction was
conceptual, not functional. Discovery-sourced entries now get the same
TTL+refresh treatment as session-sourced entries. Simplifies
find_next_hop() to single cache lookup.
Parent-change flush: clear coord_cache after recompute_coords() in both
parent-switch paths of handle_tree_announce(). Stale coordinates after
tree reconvergence cause dead-end routing that's more expensive than
re-discovery.
Tested: 493 unit tests passed, clippy clean, Docker mesh 20/20,
Docker chain 6/6.
Add dual-approach MTU handling to prevent TCP connections from hanging
when packets exceed the transport MTU after FIPS encapsulation.
ICMPv6 Packet Too Big:
- Generate RFC 4443 PTB messages for oversized packets at TUN outbound
- Inject back via TUN for local delivery to the application
- Per-source rate limiting (100ms interval, 10s entry expiry)
- MTU check in handle_tun_outbound before session encapsulation
TCP MSS Clamping:
- Intercept SYN packets in run_tun_reader() (outbound)
- Intercept SYN-ACK packets in TunWriter (inbound)
- Clamp MSS option to fit within effective MTU (transport - 127 overhead)
- Recalculate TCP checksum after modification
Code organization:
- ICMP, TCP MSS, and rate limiter modules in upper/ alongside existing
protocol-specific packet handling (dns.rs, tun.rs)
- Shared FIPS_OVERHEAD constant (127 bytes) and effective_ipv6_mtu()
function in upper/icmp.rs
- Node::effective_ipv6_mtu() delegates to the shared function
- run_tun_reader() accepts actual transport MTU from config
Example config corrections:
- UDP transport MTU set to 1472 across all configs (correct max UDP
payload for standard Ethernet: 1500 - 20 IPv4 - 8 UDP)
- Startup logging of effective MTU and max MSS values
Sessions in the Established state that have no activity for 90 seconds
are now automatically removed. This ensures idle sessions are torn down
before transit node coord_cache entries expire (300s TTL), so that when
traffic resumes a fresh SessionSetup re-warms transit node caches with
current coordinates.
The identity cache now stores registration timestamps and expires
entries after 60 seconds via lazy expiry on lookup. This prevents
unbounded growth while allowing natural repopulation through DNS
resolution on next use.
Timer ordering: identity (60s) < session (90s) < coord_cache (300s).
Both timeouts are configurable: node.session.idle_timeout_secs and
node.cache.identity_ttl_secs. Setting idle_timeout_secs to 0 disables
session idle purging.
Changes:
- Add idle_timeout_secs (default 90) to SessionConfig
- Add identity_ttl_secs (default 60) to CacheConfig
- Add timestamp to identity_cache entries, lazy expiry on lookup
- Add purge_idle_sessions() called from tick loop
- Remove #[cfg(test)] from SessionEntry::last_activity()
- 7 new tests covering timeout behavior and edge cases
Transit nodes cache destination coordinates when they forward
SessionSetup messages (via try_warm_coord_cache). These coord_cache
entries have a 5-minute TTL, after which they expire. Once expired,
the transit node can no longer forward data packets for that
destination — find_next_hop returns None and the node sends
CoordsRequired errors back to the source. This creates a permanent
routing failure for any multi-hop path after 5 minutes of the initial
session establishment, even if traffic is actively flowing.
The root cause was that find_next_hop used coord_cache.get(), a
read-only lookup that checks expiry but never extends it. Active
forwarding did not keep the cache warm. Meanwhile, get_and_touch()
existed but only updated last_used without extending expires_at.
Fix:
- find_next_hop now calls coord_cache.get_and_touch() instead of get()
- get_and_touch now calls entry.refresh() instead of entry.touch(),
which extends expires_at by the default TTL on each access
- find_next_hop signature changed from &self to &mut self to allow
the mutable cache access
This ensures that as long as traffic flows through a transit node,
the coord_cache entries stay warm and routing continues to work.
Entries still expire after 5 minutes of inactivity as designed.
Wire the discovery protocol into the data plane path so that
find_next_hop() consults route_cache as a fallback when coord_cache
has no entry. When session initiation fails due to missing routes,
trigger discovery (initiate_lookup) instead of immediately sending
ICMPv6 Destination Unreachable. On discovery completion, retry
session initiation for any pending TUN packets.
Changes:
- find_next_hop() falls back to route_cache when coord_cache misses
- handle_tun_outbound() triggers discovery on session failure
- handle_lookup_response() retries session after discovery completes
- handle_coords_required() triggers discovery for missing coordinates
- Add pending_lookups deduplication with 10-second timeout
- Periodic cleanup of stale lookups in tick handler
- New test: route_cache fallback verification in find_next_hop
Add DNS responder that resolves <npub>.fips queries to FipsAddress IPv6
addresses. Resolution is pure computation (npub → NodeAddr → IPv6) with
identity cache priming as a side effect, enabling subsequent TUN packet
routing to non-peer destinations.
- New dns.rs module: resolve_fips_query(), handle_dns_packet() using
simple-dns crate, run_dns_responder() async UDP server loop
- DnsConfig in config.rs: enabled, bind_addr (127.0.0.1), port (5354)
- Fourth select! arm in RX event loop for DNS identity channel
- DNS task spawn/abort in node lifecycle
- 10 new tests (420 total)
Add examples/two-node-udp/ with standalone walkthrough for testing two
FIPS nodes in Linux network namespaces over a veth pair. Includes
network diagram, config files, DNS routing setup, and troubleshooting.
Johnathan CorganandGitHub
Alex XieandJohnathan Corgan
Tim O'SheaandJohnathan Corgan
OceanSlimandGitHub
Origami74