Files
fips/docs/design/fips-configuration.md
T
Johnathan Corgan 56d39f223b Add ECN congestion signaling and transport congestion detection
Implement hop-by-hop ECN congestion signaling through the FMP layer,
transport-level congestion detection via kernel drop counters, and
chaos harness integration for end-to-end validation.

FMP/session ECN plumbing:

- Thread ce_flag parsed at link layer through dispatch_link_message,
  handle_session_datagram, handle_session_payload, and
  handle_encrypted_session_msg to session delivery
- Replace hardcoded false in session-layer record_recv() with actual
  ce_flag, activating ecn_ce_count tracking in session MMP

ECN congestion detection and CE relay:

- Add EcnConfig (node.ecn.*) with configurable loss_threshold (5%)
  and etx_threshold (3.0) for transit congestion detection
- Add send_encrypted_link_message_with_ce() that ORs FLAG_CE into FMP
  header flags; original method delegates with ce_flag=false
- Compute outgoing_ce = incoming_ce || local congestion on next-hop
  link, enabling hop-by-hop CE relay through transit nodes

IPv6 ECN-CE marking:

- Mark ECN-CE (0b11) in IPv6 Traffic Class on received DataPackets
  before TUN delivery when FMP CE flag is set
- Only marks ECN-capable packets (ECT(0)/ECT(1)); Not-ECT packets
  unchanged per RFC 3168

Transport congestion abstraction and UDP kernel drop detection:

- Add TransportCongestion struct to transport layer for transport-
  agnostic local congestion indicators
- Replace tokio::UdpSocket with AsyncFd<socket2::Socket> using
  libc::recvmsg() with ancillary data parsing
- Enable SO_RXQ_OVFL for kernel receive buffer drop counter on every
  packet, wiring up previously-stubbed UdpStats.kernel_drops
- Add TransportDropState for per-transport delta tracking with 1s
  tick sampling via sample_transport_congestion()
- Extend detect_congestion() with transport kernel drop check
  alongside MMP loss metrics

Congestion monitoring and control:

- Add CongestionStats (ce_forwarded, ce_received, congestion_detected,
  kernel_drop_events) to NodeStats with snapshot serialization
- Wire counters into forwarding path, session handler, and transport
  drop sampling with rate-limited warn logging (5s interval)
- Expose congestion data in show_routing control query and
  ecn_ce_count in show_mmp peer entries
- Add congestion counters to fipstop routing tab in two-column layout

Chaos harness integration:

- Add query_routing(), query_transports(), snapshot_all_congestion()
  to chaos control module
- Add congestion/kernel-drop log analysis in logs module
- Add congestion-stress scenario: 10-node tree, 1 Mbps bandwidth,
  5-10% netem loss, heavy iperf3 traffic
- Add IngressConfig for tc ingress policing with per-peer policer
  filters simulating upstream bandwidth bottlenecks
- Add iperf3 JSON result capture to traffic manager for throughput
  measurement across scenarios
- Add ECN A/B test scenarios (ecn-ab-on/off.yaml) with ingress
  policing and comparison script
- Enable TCP ECN negotiation (tcp_ecn=1 sysctl) in container
  entrypoint for end-to-end CE propagation

Tests:

- 10 ECN unit/integration tests: mark_ipv6_ecn_ce variants, CE relay
  chain (3-node propagation), EcnConfig serde roundtrip
- 3 transport drop congestion detection unit tests

Documentation:

- Update fips-mesh-layer.md: replace outdated CE Echo stub with full
  ECN Congestion Signaling section covering detection logic, CE relay,
  IPv6 marking, session tracking, and monitoring counters
- Update fips-configuration.md: add node.ecn.* parameter table and
  ecn block in complete reference YAML
- Update fips-transport-layer.md: add Congestion Reporting section
  with TransportCongestion struct, congestion() trait method, and
  per-transport status; document AsyncFd/recvmsg/SO_RXQ_OVFL in UDP
- Update chaos README: add congestion/ECN scenario docs, ingress
  traffic control, and iperf3 JSON capture sections
- Update README.md: add ECN to features list and "What works today";
  update transport and tooling entries
2026-03-05 17:05:57 +00:00

23 KiB
Raw Blame History

FIPS Configuration

FIPS uses YAML-based configuration with a cascading multi-file priority system. All parameters have sensible defaults; a node can run with no configuration file at all (it will generate an ephemeral identity and listen on default addresses).

Configuration Loading

Search Paths

When started without the -c flag, FIPS searches for fips.yaml in these locations, lowest to highest priority:

Priority Path Purpose
1 (lowest) /etc/fips/fips.yaml System-wide defaults
2 ~/.config/fips/fips.yaml User preferences
3 ~/.fips.yaml Legacy user config
4 (highest) ./fips.yaml Deployment-specific overrides

All found files are loaded and merged in priority order. Values from higher priority files override those from lower priority files. This allows a system administrator to set site-wide defaults in /etc/fips/fips.yaml while individual deployments override specific values in ./fips.yaml.

CLI Option

fips -c /path/to/config.yaml

When -c is specified, only that file is loaded (search paths are skipped).

Partial Configuration

Every field has a built-in default. A configuration file only needs to specify values that differ from defaults. For example, a minimal config might contain only the identity and peer list, inheriting all other defaults.

YAML Structure

The configuration is organized into five top-level sections:

node:        # Node behavior, protocol parameters, and tuning
tun:         # TUN virtual interface
dns:         # DNS responder for .fips domain
transports:  # Network transports (UDP, Ethernet, Bluetooth, Tor, ...)
peers:       # Static peer list

Control Socket (node.control.*)

Parameter Type Default Description
node.control.enabled bool true Enable the Unix domain control socket
node.control.socket_path string (auto) Socket file path. Default: $XDG_RUNTIME_DIR/fips/control.sock, then /run/fips/control.sock (if root), then /tmp/fips-control.sock

The control socket provides read-only access to node state via the fipsctl command-line tool. See the project README for the command list.

All tunable protocol parameters live under node.*, organized as sysctl-style dotted paths. The top-level sections (tun, dns, transports, peers) handle infrastructure concerns only.

Node Parameters (node.*)

Identity (node.identity.*)

Parameter Type Default Description
node.identity.nsec string (generate random) Hex-encoded secret key. If omitted, an ephemeral identity is generated on each start.

General

Parameter Type Default Description
node.leaf_only bool false Leaf-only mode: node does not forward traffic or participate in routing
node.tick_interval_secs u64 1 Periodic maintenance tick interval (retry checks, timeout cleanup, tree refresh)
node.base_rtt_ms u64 100 Initial RTT estimate for new links before measurements converge
node.heartbeat_interval_secs u64 10 Heartbeat send interval per peer for liveness detection
node.link_dead_timeout_secs u64 30 No-traffic timeout before a peer is declared dead and removed

Resource Limits (node.limits.*)

Controls capacity for connections, peers, and links.

Parameter Type Default Description
node.limits.max_connections usize 256 Max handshake-phase connections
node.limits.max_peers usize 128 Max authenticated peers
node.limits.max_links usize 256 Max active links
node.limits.max_pending_inbound usize 1000 Max pending inbound handshakes

Rate Limiting (node.rate_limit.*)

Handshake rate limiting protects against DoS on the Noise IK handshake path.

Parameter Type Default Description
node.rate_limit.handshake_burst u32 100 Token bucket burst capacity
node.rate_limit.handshake_rate f64 10.0 Tokens per second refill rate
node.rate_limit.handshake_timeout_secs u64 30 Stale handshake cleanup timeout
node.rate_limit.handshake_resend_interval_ms u64 1000 Initial handshake message resend interval
node.rate_limit.handshake_resend_backoff f64 2.0 Resend backoff multiplier (1s, 2s, 4s, 8s, 16s with defaults)
node.rate_limit.handshake_max_resends u32 5 Max resends per handshake attempt

Retry / Backoff (node.retry.*)

Connection retry with exponential backoff.

Parameter Type Default Description
node.retry.max_retries u32 5 Max connection retry attempts
node.retry.base_interval_secs u64 5 Base backoff interval
node.retry.max_backoff_secs u64 300 Cap on exponential backoff (5 minutes)

Auto-reconnect (triggered by MMP link-dead removal) uses the same backoff parameters but bypasses max_retries, retrying indefinitely. See peers[].auto_reconnect below.

Cache Parameters (node.cache.*)

Controls caching of tree coordinates and identity mappings.

Parameter Type Default Description
node.cache.coord_size usize 50000 Max entries in coordinate cache
node.cache.coord_ttl_secs u64 300 Coordinate cache entry TTL (5 minutes)
node.cache.identity_size usize 10000 Max entries in identity cache (LRU, no TTL)

Discovery Protocol (node.discovery.*)

Controls flood-based node discovery (LookupRequest/LookupResponse).

Parameter Type Default Description
node.discovery.ttl u8 64 Hop limit for LookupRequest flood
node.discovery.timeout_secs u64 10 Lookup completion timeout
node.discovery.recent_expiry_secs u64 10 Dedup cache expiry for recent request IDs

Spanning Tree (node.tree.*)

Controls tree construction and parent selection.

Parameter Type Default Description
node.tree.announce_min_interval_ms u64 500 Per-peer TreeAnnounce rate limit
node.tree.parent_hysteresis f64 0.2 Cost improvement fraction required for same-root parent switch (0.01.0)
node.tree.hold_down_secs u64 30 Suppress non-mandatory re-evaluation after parent switch
node.tree.reeval_interval_secs u64 60 Periodic cost-based parent re-evaluation interval (0 = disabled)
node.tree.flap_threshold u32 4 Parent switches in window before dampening engages
node.tree.flap_window_secs u64 60 Sliding window for counting parent switches
node.tree.flap_dampening_secs u64 120 Extended hold-down duration when flap threshold exceeded

Bloom Filter (node.bloom.*)

Parameter Type Default Description
node.bloom.update_debounce_ms u64 500 Debounce interval for filter update propagation

Bloom filter size (1 KB), hash count (5), and size classes are protocol constants and not configurable.

ECN Signaling (node.ecn.*)

Controls hop-by-hop ECN (Explicit Congestion Notification) signaling. When enabled, transit nodes detect congestion on outgoing links (via MMP loss/ETX metrics or kernel buffer drops) and set the CE flag on forwarded FMP frames. Destination nodes mark ECN-capable IPv6 packets with CE before TUN delivery per RFC 3168, enabling end-host TCP congestion control to react.

Parameter Type Default Description
node.ecn.enabled bool true Enable ECN congestion signaling (CE flag relay and local congestion detection)
node.ecn.loss_threshold f64 0.05 MMP loss rate threshold for CE marking (0.01.0). When the outgoing link's loss rate meets or exceeds this value, forwarded packets are CE-marked.
node.ecn.etx_threshold f64 3.0 MMP ETX threshold for CE marking (≥1.0). When the outgoing link's ETX meets or exceeds this value, forwarded packets are CE-marked.

Congestion detection triggers on any of: outgoing link loss ≥ loss_threshold, outgoing link ETX ≥ etx_threshold, or kernel receive buffer drops detected on any local transport. CE is relayed hop-by-hop: once set on any hop, the flag stays set for all subsequent hops to the destination.

Session / Data Plane (node.session.*)

Controls end-to-end session behavior and packet queuing.

Parameter Type Default Description
node.session.default_ttl u8 64 Default SessionDatagram TTL
node.session.pending_packets_per_dest usize 16 Queue depth per destination during session establishment
node.session.pending_max_destinations usize 256 Max destinations with pending packets
node.session.idle_timeout_secs u64 90 Idle session timeout; established sessions with no application data for this duration are removed. MMP reports (SenderReport, ReceiverReport, PathMtuNotification) do not count as activity
node.session.coords_warmup_packets u8 5 Number of initial data packets per session that include the CP flag for transit cache warmup; also the reset count on CoordsRequired/PathBroken receipt
node.session.coords_response_interval_ms u64 2000 Minimum interval (ms) between standalone CoordsWarmup responses to CoordsRequired/PathBroken signals per destination

The anti-replay window size (2048 packets) is a compile-time constant and not configurable.

Metrics Measurement Protocol for per-peer link measurement. See fips-mesh-layer.md for behavioral details.

Parameter Type Default Description
node.mmp.mode string "full" Operating mode: full (sender + receiver reports), lightweight (receiver reports only), or minimal (spin bit + CE echo only, no reports)
node.mmp.log_interval_secs u64 30 Periodic operator log interval for link metrics
node.mmp.owd_window_size usize 32 One-way delay trend ring buffer size

Session-Layer MMP (node.session_mmp.*)

Metrics Measurement Protocol for end-to-end session measurement. Configured independently from link-layer MMP because session reports are routed through every transit link, consuming bandwidth proportional to path length.

Parameter Type Default Description
node.session_mmp.mode string "full" Operating mode: full, lightweight, or minimal
node.session_mmp.log_interval_secs u64 30 Periodic operator log interval for session metrics
node.session_mmp.owd_window_size usize 32 One-way delay trend ring buffer size

Internal Buffers (node.buffers.*)

Channel sizes affecting throughput and memory. Primarily useful for performance tuning under high load or on memory-constrained devices.

Parameter Type Default Description
node.buffers.packet_channel usize 1024 Transport to Node packet channel capacity
node.buffers.tun_channel usize 1024 TUN to Node outbound channel capacity
node.buffers.dns_channel usize 64 DNS to Node identity channel capacity

TUN Interface (tun.*)

Parameter Type Default Description
tun.enabled bool false Enable TUN virtual interface
tun.name string "fips0" Interface name
tun.mtu u16 1280 Interface MTU (IPv6 minimum)

DNS Responder (dns.*)

Resolves <npub>.fips queries to FIPS IPv6 addresses. Resolution is pure computation (npub to public key to address); resolved identities are registered with the node for routing.

Parameter Type Default Description
dns.enabled bool true Enable DNS responder
dns.bind_addr string "127.0.0.1" Bind address
dns.port u16 5354 Listen port
dns.ttl u32 300 AAAA record TTL in seconds

The dns.ttl value should not exceed node.cache.coord_ttl_secs to avoid stale address mappings.

Transports (transports.*)

UDP (transports.udp.*)

Parameter Type Default Description
transports.udp.bind_addr string "0.0.0.0:2121" UDP bind address and port
transports.udp.mtu u16 1280 Transport MTU
transports.udp.recv_buf_size usize 2097152 UDP socket receive buffer size in bytes (2 MB). Linux kernel doubles the requested value internally. Host net.core.rmem_max must be >= this value.
transports.udp.send_buf_size usize 2097152 UDP socket send buffer size in bytes (2 MB). Host net.core.wmem_max must be >= this value.

Ethernet (transports.ethernet.*)

Ethernet transport sends raw frames via AF_PACKET SOCK_DGRAM sockets. Requires CAP_NET_RAW or running as root. Linux only.

Parameter Type Default Description
interface string (required) Network interface name (e.g., "eth0", "enp3s0")
ethertype u16 0x2121 EtherType
mtu u16 (auto) Override MTU. Default: interface MTU minus 1 (for frame type prefix)
recv_buf_size usize 2097152 Socket receive buffer size in bytes (2 MB)
send_buf_size usize 2097152 Socket send buffer size in bytes (2 MB)
discovery bool true Listen for discovery beacons from other nodes
announce bool false Broadcast announcement beacons on the LAN
auto_connect bool false Auto-connect to discovered peers
accept_connections bool false Accept incoming connection attempts from discovered peers
beacon_interval_secs u64 30 Announcement beacon interval in seconds (minimum 10)

Named instances. Multiple Ethernet interfaces can be configured by using named sub-keys instead of flat parameters:

transports:
  ethernet:
    lan:
      interface: "eth0"
      discovery: true
      announce: true
    backbone:
      interface: "eth1"
      announce: false

Each named instance operates independently with its own socket and discovery state. The instance name is used in log messages and the name() method on the Transport trait.

TCP (transports.tcp.*)

TCP transport enables firewall traversal on networks that block UDP but allow TCP (e.g., port 443). Uses FMP header-based framing with zero overhead.

Parameter Type Default Description
transports.tcp.bind_addr string (none) Listen address (e.g., "0.0.0.0:443"). If omitted, outbound-only mode.
transports.tcp.mtu u16 1400 Default MTU. Per-connection MTU derived from TCP_MAXSEG when available.
transports.tcp.connect_timeout_ms u64 5000 Outbound connect timeout in milliseconds
transports.tcp.nodelay bool true TCP_NODELAY (disable Nagle for low latency)
transports.tcp.keepalive_secs u64 30 TCP keepalive interval in seconds (0 = disabled)
transports.tcp.recv_buf_size usize 2097152 Socket receive buffer size in bytes (2 MB)
transports.tcp.send_buf_size usize 2097152 Socket send buffer size in bytes (2 MB)
transports.tcp.max_inbound_connections usize 256 Maximum simultaneous inbound connections
transports.tcp.socks5_proxy string (none) SOCKS5 proxy for outbound connections (implementation deferred)

Named instances. Like other transports, multiple TCP instances can be configured with named sub-keys:

transports:
  tcp:
    public:
      bind_addr: "0.0.0.0:443"
    tor:
      socks5_proxy: "127.0.0.1:9050"
      connect_timeout_ms: 30000

Peers (peers[])

Static peer list. Each entry defines a peer to connect to.

Parameter Type Default Description
peers[].npub string (required) Peer's Nostr public key (npub-encoded)
peers[].alias string (none) Human-readable name for logging
peers[].addresses[].transport string (required) Transport type: udp, tcp, or ethernet
peers[].addresses[].addr string (required) Transport address. UDP/TCP: "ip:port". Ethernet: "interface/mac" (e.g., "eth0/aa:bb:cc:dd:ee:ff")
peers[].addresses[].priority u8 100 Address priority (lower = preferred)
peers[].connect_policy string "auto_connect" Connection policy: auto_connect, on_demand, or manual
peers[].auto_reconnect bool true Automatically reconnect after MMP link-dead removal (exponential backoff, unlimited retries)

Minimal Example

A typical node configuration enabling TUN, DNS, and a single peer:

node:
  identity:
    nsec: "0102030405060708090a0b0c0d0e0f101112131415161718191a1b1c1d1e1f20"

tun:
  enabled: true
  name: fips0
  mtu: 1280

dns:
  enabled: true
  bind_addr: "127.0.0.1"
  port: 53

transports:
  udp:
    bind_addr: "0.0.0.0:2121"
    mtu: 1472

peers:
  - npub: "npub1tdwa4vjrjl33pcjdpf2t4p027nl86xrx24g4d3avg4vwvayr3g8qhd84le"
    alias: "node-b"
    addresses:
      - transport: udp
        addr: "172.20.0.11:2121"
    connect_policy: auto_connect

Mixed UDP + Ethernet Example

A node bridging internet peers (UDP) and a local Ethernet segment with beacon discovery:

node:
  identity:
    nsec: "0102030405060708090a0b0c0d0e0f101112131415161718191a1b1c1d1e1f20"

tun:
  enabled: true

transports:
  udp:
    bind_addr: "0.0.0.0:2121"
    mtu: 1472
  ethernet:
    interface: "eth0"
    discovery: true
    announce: true
    auto_connect: true
    accept_connections: true

peers:
  - npub: "npub1tdwa4vjrjl33pcjdpf2t4p027nl86xrx24g4d3avg4vwvayr3g8qhd84le"
    alias: "internet-peer"
    addresses:
      - transport: udp
        addr: "203.0.113.5:2121"
    connect_policy: auto_connect

Ethernet peers on the local segment are discovered automatically via beacons — no static peer entries needed. Internet peers still require explicit configuration.

All node.* parameters use their defaults. To override specific values, add only the relevant sections:

node:
  identity:
    nsec: "..."
  limits:
    max_peers: 64
  retry:
    max_retries: 10
    max_backoff_secs: 600
  cache:
    coord_size: 100000

Complete Reference

The full YAML structure with all defaults:

node:
  identity:
    nsec: null                       # hex secret key (null = generate ephemeral)
  leaf_only: false
  tick_interval_secs: 1
  base_rtt_ms: 100
  heartbeat_interval_secs: 10
  link_dead_timeout_secs: 30
  limits:
    max_connections: 256
    max_peers: 128
    max_links: 256
    max_pending_inbound: 1000
  rate_limit:
    handshake_burst: 100
    handshake_rate: 10.0
    handshake_timeout_secs: 30
    handshake_resend_interval_ms: 1000
    handshake_resend_backoff: 2.0
    handshake_max_resends: 5
  retry:
    max_retries: 5
    base_interval_secs: 5
    max_backoff_secs: 300
  cache:
    coord_size: 50000
    coord_ttl_secs: 300
    identity_size: 10000
  discovery:
    ttl: 64
    timeout_secs: 10
    recent_expiry_secs: 10
  tree:
    announce_min_interval_ms: 500
    parent_hysteresis: 0.2              # cost improvement fraction for parent switch
    hold_down_secs: 30                  # suppress re-evaluation after switch
    reeval_interval_secs: 60            # periodic cost-based re-evaluation (0 = disabled)
    flap_threshold: 4                    # parent switches before dampening
    flap_window_secs: 60                 # sliding window for flap detection
    flap_dampening_secs: 120             # extended hold-down on flap
  bloom:
    update_debounce_ms: 500
  session:
    default_ttl: 64
    pending_packets_per_dest: 16
    pending_max_destinations: 256
    idle_timeout_secs: 90
    coords_warmup_packets: 5
    coords_response_interval_ms: 2000
  mmp:
    mode: full                       # full | lightweight | minimal
    log_interval_secs: 30
    owd_window_size: 32
  session_mmp:
    mode: full                       # full | lightweight | minimal
    log_interval_secs: 30
    owd_window_size: 32
  ecn:
    enabled: true                    # ECN congestion signaling (CE flag relay)
    loss_threshold: 0.05             # MMP loss rate threshold for CE marking (5%)
    etx_threshold: 3.0               # MMP ETX threshold for CE marking
  control:
    enabled: true
    socket_path: null                # null = auto ($XDG_RUNTIME_DIR → /run/fips → /tmp fallback)
  buffers:
    packet_channel: 1024
    tun_channel: 1024
    dns_channel: 64

tun:
  enabled: false
  name: "fips0"
  mtu: 1280

dns:
  enabled: true
  bind_addr: "127.0.0.1"
  port: 5354
  ttl: 300

transports:
  udp:
    bind_addr: "0.0.0.0:2121"
    mtu: 1280
    recv_buf_size: 2097152           # 2 MB (kernel doubles to 4 MB actual)
    send_buf_size: 2097152           # 2 MB
  # ethernet:                        # uncomment to enable (requires CAP_NET_RAW)
  #   interface: "eth0"              # required: network interface name
  #   ethertype: 0x2121              # default EtherType
  #   mtu: null                      # null = interface MTU - 1 (typically 1499)
  #   recv_buf_size: 2097152         # 2 MB
  #   send_buf_size: 2097152         # 2 MB
  #   discovery: true                # listen for beacons
  #   announce: false                # broadcast beacons
  #   auto_connect: false            # connect to discovered peers
  #   accept_connections: false      # accept inbound handshakes
  #   beacon_interval_secs: 30       # beacon interval (min 10)
  # tcp:                             # uncomment to enable TCP transport
  #   bind_addr: "0.0.0.0:443"      # listen address (omit for outbound-only)
  #   mtu: 1400                      # default MTU
  #   connect_timeout_ms: 5000       # outbound connect timeout
  #   nodelay: true                  # TCP_NODELAY
  #   keepalive_secs: 30             # keepalive interval (0 = disabled)
  #   recv_buf_size: 2097152         # 2 MB
  #   send_buf_size: 2097152         # 2 MB
  #   max_inbound_connections: 256   # resource protection limit
  #   socks5_proxy: null             # SOCKS5 for outbound (deferred)

peers:                               # static peer list
  # - npub: "npub1..."
  #   alias: "node-b"
  #   addresses:
  #     - transport: udp
  #       addr: "10.0.0.2:2121"
  #       priority: 100
  #   connect_policy: auto_connect
  #   auto_reconnect: true           # reconnect after link-dead removal