fips/docs/design/fips-mesh-layer.md

FIPS Mesh Protocol (FMP)

The FIPS Mesh Protocol is the middle layer of the FIPS protocol stack. It sits between the transport layer below and the FIPS Session Protocol (FSP) above. FMP is where anonymous transport addresses become authenticated peers, where the mesh self-organizes, and where forwarding decisions are made.

Role

FMP manages direct peer connections over transports. When a transport delivers a datagram from an unknown address, FMP authenticates the sender through a Noise IK handshake, establishing a cryptographic link. Once authenticated, the link carries all inter-peer communication: spanning tree gossip, bloom filter updates, coordinate discovery, and forwarded session datagrams — all encrypted per-hop.

FMP is the boundary between opaque transport addresses and identified peers. Below FMP, everything is transport-specific addresses (host:port, MAC, .onion). Above FMP, everything is peers identified by public keys and routable by node_addr. The transport layer never sees FIPS-level structure; FSP never sees transport addresses or routing details.

Services Provided to FSP

From the session layer's perspective, FMP is a black box providing three services. FSP knows nothing about transports, links, peers, spanning trees, coordinates, bloom filters, hop counts, or network topology.

Datagram Forwarding

FMP accepts a datagram addressed by source and destination node_addr and delivers it best-effort toward the destination. The datagram travels hop by hop — at each node, FMP decrypts the link layer, reads the destination node_addr, makes a local forwarding decision, and re-encrypts onto the next-hop link.

FSP provides: source node_addr, destination node_addr, hop limit, and an opaque payload (the session-layer encrypted message).

FMP provides: best-effort delivery. No acknowledgment, no retransmission, no ordering guarantee. Datagrams may be dropped, duplicated, or delivered out of order.

Error Signaling

When forwarding fails, FMP signals the source endpoint asynchronously:

• CoordsRequired: A transit node lacks the destination's tree coordinates and cannot make a forwarding decision. The source should re-initiate discovery and reset its coordinate warmup strategy.
• PathBroken: Greedy routing reached a dead end — no peer is closer to the destination than the current node. The source should re-discover the destination's current coordinates.

Both signals travel inside the SessionDatagram envelope (using the existing src/dest/hop_limit addressing) but are generated by transit nodes and are not end-to-end encrypted. They are rate-limited at 100ms per destination to prevent storms during topology changes.

Local Delivery

When a datagram arrives with a destination node_addr matching the local node, FMP delivers it up to FSP for session-layer processing.

Services Required from Transport Layer

FMP requires the following from each transport:

Datagram Delivery

Send and receive raw datagrams to/from transport addresses. The transport handles all medium-specific details. FMP sees only "send bytes to address" and "bytes arrived from address."

MTU Reporting

The maximum datagram size for a given link. FMP needs this to determine how much payload fits in a single packet after link encryption overhead (37 bytes for the encrypted frame wrapper: 16-byte outer header + 5-byte inner header + 16-byte AEAD tag).

Connection Lifecycle

For connection-oriented transports, the transport must establish the underlying connection before FMP can begin the Noise IK handshake. For connectionless transports, datagrams can flow immediately.

Endpoint Discovery (Optional)

When a transport discovers a FIPS-capable endpoint (via beacon, query, or other transport-specific mechanism), it notifies FMP so that link setup can be initiated. Transports without discovery support provide peer addresses through configuration.

Peer Authentication

Noise IK Handshake

Every peer connection begins with a Noise IK handshake that mutually authenticates both parties and establishes symmetric keys for link encryption.

The IK pattern is chosen because:

• The initiator knows the responder's static public key from configuration or discovery, and sends their own static key encrypted in the first message
• The responder learns the initiator's identity from the first message, then responds with their own ephemeral key

After the two-message handshake completes, both parties share symmetric session keys derived from four DH operations (es, ss, ee, se). The handshake provides mutual authentication, forward secrecy, and identity hiding for the initiator.

Epoch Exchange and Peer Restart Detection

Both IK handshake messages carry an encrypted epoch payload — an 8-byte random value generated once at node startup:

• msg1: Ephemeral key (33 bytes) + encrypted static key (49 bytes) + encrypted epoch (24 bytes) = 106 bytes total
• msg2: Ephemeral key (33 bytes) + encrypted epoch (24 bytes) = 57 bytes total

The encrypted epoch (EPOCH_ENCRYPTED_SIZE = 24 bytes) consists of the 8-byte epoch value plus a 16-byte AEAD tag.

On reconnection, each peer compares the received epoch with the previously stored epoch for that peer. An epoch mismatch indicates the peer has restarted (generated a new epoch), triggering full link re-establishment rather than treating the handshake as a simple reconnection. This prevents stale session state from persisting across restarts.

Identity Binding

The Noise handshake binds the link to the peer's cryptographic identity. After handshake completion:

• The peer's public key (FIPS identity) is confirmed
• The node_addr is computed from the public key (SHA-256, truncated to 128 bits)
• All subsequent traffic on the link is authenticated by the Noise session — successful decryption proves the sender is the authenticated peer
• The link is registered in the dispatch table for O(1) packet routing

Reconnection

When a Noise IK msg1 arrives from a peer that already has an authenticated link, FMP accepts the new handshake alongside the existing session. If the new handshake completes successfully, it replaces the old session. This handles legitimate reconnection (network change, process restart, NAT rebinding) without disrupting ongoing traffic until the new session is confirmed.

Auto-Reconnect

When MMP's liveness detection removes a peer (dead timeout exceeded), FMP automatically re-initiates the connection if the peer is configured for it. The auto-reconnect path:

1. check_link_heartbeats() detects the dead peer and calls remove_active_peer(), tearing down the link and triggering tree/bloom reconvergence
2. schedule_reconnect() checks whether the peer is in the auto-connect list with auto_reconnect: true (the default)
3. If eligible, the peer is fed into the retry system with unlimited retries and exponential backoff (same base interval and max backoff as startup retries, configured via node.retry.*)
4. On each retry tick, a fresh Noise IK handshake is initiated toward the peer's configured transport addresses

Auto-reconnect only applies to peers in the static peer list with connect_policy: auto_connect. Inbound-only peers (those not in the local config) are not reconnected — the owning node (the one with the outbound config) is responsible for re-establishing the link.

Handshake Message Retry

Both link-layer (Noise IK msg1/msg2) and session-layer (SessionSetup/ SessionAck) handshakes use message-level retry with exponential backoff within the handshake timeout window. This handles packet loss on the underlying transport without waiting for the full handshake timeout to expire.

Configuration (under node.rate_limit.*):

• handshake_resend_interval_ms (default 1000): initial resend interval
• handshake_resend_backoff (default 2.0): backoff multiplier per resend
• handshake_max_resends (default 5): max resends per handshake attempt

With defaults, resends occur at 1s, 2s, 4s, 8s, 16s — all within the 30s handshake timeout. Under 19% per-attempt loss (10% bidirectional), the probability of all 6 attempts (initial + 5 resends) failing is ~0.005%.

Session-layer resends wrap the stored payload in a fresh SessionDatagram so routing adapts to topology changes between resends. Responder idempotency: duplicate msg1/SessionSetup triggers resend of the stored msg2/SessionAck.

Link Encryption

All traffic between authenticated peers is encrypted. Every packet on a link — gossip messages, routing queries, forwarded session datagrams, disconnect notifications — passes through Noise's ChaCha20-Poly1305 AEAD.

Encrypted Frame Structure

Post-handshake packets are wrapped in an encrypted frame consisting of:

• A 4-byte common prefix (version, phase, flags, payload length)
• A receiver index for O(1) session lookup
• An explicit counter used as the AEAD nonce
• The ciphertext with a Poly1305 authentication tag

The 16-byte outer header (common prefix + receiver index + counter) is used as AAD for the AEAD, binding the header to the ciphertext without encrypting it.

The plaintext inside the encrypted frame begins with a 5-byte inner header (4-byte session-relative timestamp followed by a message type byte), then the message-specific payload.

See fips-wire-formats.md for the complete wire format specification.

What Encryption Provides

• Confidentiality: An observer on the underlying transport sees only encrypted packets, packet timing, and packet sizes
• Integrity: Any modification to a packet is detected by the AEAD tag
• Authentication: Only the authenticated peer can produce valid ciphertext for this link's session keys

What Encryption Does Not Provide

• End-to-end confidentiality: Link encryption protects traffic on a single hop. Each transit node decrypts, reads the routing envelope, and re-encrypts for the next hop. Session-layer encryption (FSP) provides end-to-end confidentiality.
• Traffic analysis protection: Packet timing, sizes, and volume are visible to transport-layer observers.

Index-Based Session Dispatch

Incoming packets are dispatched to the correct Noise session using a receiver index — a random 32-bit value chosen by the receiver during the handshake. This enables O(1) lookup without relying on source addresses.

Each party in a link has two indices:

• our_index: Chosen by us, included by the peer in packets sent to us
• their_index: Chosen by them, included by us in packets sent to them

The tuple (transport_id, receiver_idx) uniquely identifies a session.

Index Properties

• Random: Cryptographically random to prevent guessing
• Unique per transport: No two active sessions on the same transport share an index
• Scoped to transport: The same index value may appear on different transports
• Rotated on rekey: New indices allocated on rekey to prevent cross-session correlation by passive observers

Dispatch Flow

1. Read the 4-byte common prefix to determine the phase (established, handshake msg1, msg2)
2. For established frames (phase 0x0): look up (transport_id, receiver_idx) in the session table — O(1) hash lookup. Unknown indices are rejected before any cryptographic operation.
3. For handshake msg2 (phase 0x2): look up by our sender index to match to a pending outbound handshake
4. For handshake msg1 (phase 0x1): rate-limited processing, creates new state

This approach follows WireGuard's design: source address is informational, not authoritative. Only successful cryptographic verification establishes authenticity.

Roaming

When an encrypted packet successfully decrypts, the sender is the authenticated peer regardless of what transport address the packet arrived from. FMP updates the peer's current address to the packet's source address, and subsequent outbound packets use the updated address.

This allows peers to change transport addresses (e.g., host:port for UDP) without session interruption. The mechanism is:

1. Packet arrives from a different address than expected
2. Receiver index lookup finds the session
3. AEAD decryption succeeds — the sender is cryptographically authenticated
4. Peer's address is updated to the new source address

Roaming is most useful for UDP, where source addresses can change due to NAT rebinding or network changes. For connection-oriented transports, "roaming" manifests as reconnection rather than mid-session address change.

Roaming addresses mid-session NAT rebinding. Establishing the initial UDP path through NAT is a separate concern, addressed by the optional Nostr-mediated overlay discovery and STUN-assisted hole punching feature (see fips-transport-layer.md and fips-configuration.md).

Replay Protection

Each link session maintains per-direction counters:

• Send counter: Monotonically increasing, used as the AEAD nonce for each outbound packet
• Receive window: A sliding bitmap (2048 entries) tracking which counters have been seen

The receive window handles the realities of unreliable transports: packets may arrive out of order, be duplicated, or be lost. The window accepts any counter not yet seen and within 2048 of the highest counter received. Counters older than the window are rejected.

The replay check is performed before decryption to prevent CPU exhaustion from replayed packets that would pass the index lookup but fail decryption.

During link transitions, stale packets from a previous Noise session arrive encrypted with old counters and are correctly rejected. To avoid excessive log volume from these benign bursts, replay detection logging is suppressed after the first 3 occurrences per peer. A summary is emitted when the peer's session is re-established or the peer is removed.

Rate Limiting

Handshake initiation (msg1) is the primary attack surface for unauthenticated traffic. Each msg1 requires Noise DH operations (~200µs on modern CPUs), state allocation, and response generation.

FMP uses a global token bucket rate limiter:

• Burst capacity: Handles legitimate connection storms (e.g., node restart with many configured peers)
• Sustained rate: Limits steady-state new connections per second
• Global scope: Rate limiting is global, not per-source, because UDP source addresses are trivially spoofable

Additional protections:

• Connection limit: Maximum number of pending inbound handshakes, capping memory usage
• Handshake timeout: Stale pending handshakes are cleaned up after a configurable timeout
• Allowlist/blocklist: Optional peer filtering before handshake processing

Disconnect

FMP supports orderly link teardown via a Disconnect message carrying a reason code (shutdown, restart, protocol error, transport failure, resource exhaustion, security violation, configuration change, timeout).

On receiving Disconnect, FMP immediately cleans up state: removes the peer from the peer table, frees the session index, removes the link, and cleans up address mappings. If the departed peer was the tree parent, FMP triggers parent reselection.

Disconnect is best-effort — if the transport is broken, the message won't arrive. Timeout-based detection remains the fallback for detecting failed links.

On node shutdown, Disconnect is sent to all active peers before transports are stopped.

Liveness Detection

FMP detects link liveness through a combination of explicit heartbeats and traffic observation.

Heartbeat

A Heartbeat message (0x51) is sent to each active peer every node.heartbeat_interval_secs (default 10s). The heartbeat is a minimal encrypted frame with no payload beyond the standard inner header (timestamp + message type). Any successfully decrypted frame — data, gossip, MMP report, or heartbeat — resets the peer's last-receive timestamp tracked by the MMP receiver.

Dead Timeout

When no traffic (of any kind) is received from a peer for node.link_dead_timeout_secs (default 30s), the peer is declared dead and removed via remove_active_peer(). This triggers the full teardown cascade: spanning tree parent reselection (if the dead peer was the parent), TreeAnnounce propagation, coordinate cache flush, and bloom filter recompute.

If the dead peer is eligible for auto-reconnect (see [Auto-Reconnect] (#auto-reconnect)), reconnection is scheduled immediately after removal.

The heartbeat is independent of MMP — it is needed because idle links in Lightweight MMP mode have no guaranteed periodic traffic (gossip is event-driven, and MMP reports require at least one side running Full mode).

Link Message Types

FMP defines eight message types carried inside encrypted frames:

Type	Name	Purpose
0x10	TreeAnnounce	Spanning tree state announcements between peers
0x20	FilterAnnounce	Bloom filter reachability updates
0x30	LookupRequest	Coordinate discovery — flood toward destination
0x31	LookupResponse	Coordinate discovery — response with coordinates
0x00	SessionDatagram	Encapsulated session-layer payload for forwarding
0x01	SenderReport	MMP sender-side metrics report
0x02	ReceiverReport	MMP receiver-side metrics report
0x50	Disconnect	Orderly link teardown with reason code
0x51	Heartbeat	Link liveness probe

Additionally, handshake messages (phase 0x1 msg1, phase 0x2 msg2) are sent unencrypted before the link session is established.

TreeAnnounce and FilterAnnounce are exchanged between direct peers only — they are not forwarded. LookupRequest and LookupResponse are forwarded through the mesh (flooded with deduplication). SessionDatagram is forwarded hop-by-hop toward the destination. Disconnect is peer-to-peer.

See fips-mesh-operation.md for how these messages work together to build and maintain the mesh, and fips-wire-formats.md for byte-level message layouts.

Metrics Measurement Protocol (MMP)

Each active peer link runs an instance of the Metrics Measurement Protocol, providing per-link quality metrics to the operator and to the spanning tree layer for cost-based parent selection.

Metrics Tracked

MMP computes the following metrics from the per-frame counter and timestamp fields in the FMP wire format:

• SRTT — Smoothed round-trip time (Jacobson/RFC 6298, α=1/8). Derived from timestamp-echo in ReceiverReports with dwell-time compensation.
• Loss rate — Bidirectional loss inferred from counter gaps. Tracked as both instantaneous (per-interval) and long-term EWMA.
• Jitter — Interarrival jitter (RFC 3550 algorithm) in microseconds.
• Goodput — Bytes per second of payload data (excludes MMP reports).
• OWD trend — One-way delay trend (µs/s, signed). Indicates congestion buildup before loss occurs.
• ETX — Expected Transmission Count, computed from bidirectional delivery ratios. Used in cost-based parent selection via link_cost = etx * (1.0 + srtt_ms / 100.0); not yet used in find_next_hop() candidate ranking.
• Dual EWMA trends — Short-term (α=1/4) and long-term (α=1/32) trend indicators for both RTT and loss, enabling change detection.

Operating Modes

MMP supports three modes, configured via node.mmp.mode:

Mode	Reports Exchanged	Metrics Available
Full (default)	SenderReport + ReceiverReport	All metrics including RTT, loss, jitter, goodput, OWD trend
Lightweight	ReceiverReport only	Loss (from counter gaps), jitter, OWD trend. No RTT.
Minimal	None	Spin bit and CE echo flags only. No computed metrics.

Report Scheduling

Reports are sent at RTT-adaptive intervals, clamped to [100ms, 2s]. A cold-start interval of 500ms is used before SRTT converges. The interval formula is clamp(2 × SRTT, 100ms, 2000ms).

Spin Bit and RTT

The SP (spin bit) flag in the FMP inner header follows the QUIC spin bit pattern: reflected on receive, toggled on send when the reflected value matches the last sent value. The spin bit state machine runs for TX reflection, but RTT samples from the spin bit are discarded. In a mesh protocol where frames are sent irregularly (tree announces, bloom filters, MMP reports on different timers), inter-frame processing delays inflate spin bit RTT measurements unpredictably. Timestamp-echo from ReceiverReports (with dwell-time compensation) is the sole SRTT source.

ECN Congestion Signaling

The CE (Congestion Experienced) flag (bit 1 in the FMP flags byte) provides hop-by-hop congestion signaling through the mesh. Transit nodes detect congestion on outgoing links and set CE on forwarded packets; once set, the flag stays set for all subsequent hops to the destination.

Congestion detection (detect_congestion()) triggers on any of:

• Outgoing link MMP loss rate ≥ node.ecn.loss_threshold (default 5%)
• Outgoing link MMP ETX ≥ node.ecn.etx_threshold (default 3.0)
• Kernel receive buffer drops detected on any local transport (via SO_RXQ_OVFL on UDP)

CE relay: The forwarding path computes outgoing_ce = incoming_ce || local_congestion. The send_encrypted_link_message_with_ce() method ORs FLAG_CE into the FMP header flags when ce is true. The original send_encrypted_link_message() delegates with ce_flag=false, leaving the 20+ existing call sites unchanged.

IPv6 ECN-CE marking: When a CE-flagged DataPacket arrives at its final destination, the IPv6 Traffic Class ECN bits are marked CE (0b11) before TUN delivery — but only for ECN-capable packets (ECT(0) or ECT(1)). Not-ECT packets are never marked per RFC 3168. The host TCP stack then echoes ECE in ACKs, triggering sender cwnd reduction through standard congestion control.

Session-layer tracking: The ecn_ce_count field in MMP ReceiverReports tracks CE-flagged packets received per link, providing end-to-end visibility into congestion propagation.

Monitoring: CongestionStats tracks four counters — ce_forwarded, ce_received, congestion_detected, and kernel_drop_events — exposed via fipsctl show routing (congestion block) and fipstop (routing tab). Rate-limited warn logging (5s interval) alerts on congestion detection events.

See node.ecn.* in fips-configuration.md for tuning parameters.

Operator Logging

MMP emits periodic link metrics at info level (configurable via node.mmp.log_interval_secs, default 30s):

MMP link metrics peer=node-b rtt=2.3ms loss=0.2% jitter=0.1ms goodput=76.0MB/s tx_pkts=1234 rx_pkts=5678

Teardown logs include final SRTT, loss rate, jitter, ETX, goodput, and cumulative tx/rx packet and byte counts.

Security Properties

Threat Resistance

Threat	Mitigation
Connection exhaustion	Token bucket rate limit + connection count limit
CPU exhaustion (msg1 flood)	Rate limit before crypto operations
Replay attacks	Counter-based nonces with sliding window
State confusion	Strict handshake state machine validation
Spoofed encrypted packets	Index lookup + AEAD verification
Spoofed msg2	Index lookup + Noise ephemeral key binding
Address spoofing	Cryptographic authority, not address-based
Session correlation	Index rotation on rekey

Unauthenticated Attack Surface

Only handshake msg1 can be sent by unauthenticated parties. Encrypted frames require a known session index, and msg2 requires a response to a specific ephemeral key. The msg1 attack surface is protected by rate limiting, connection limits, and handshake timeouts.

Authenticated Peer Misbehavior

Authentication establishes identity but does not grant trust. An authenticated peer can send malformed packets (which fail AEAD and are dropped) or high-frequency traffic (rate-limited by higher layers). False tree coordinate claims are constrained by signature verification on TreeAnnounce messages.

Silent Drop Policy

Invalid packets are silently dropped without error responses. This prevents information leakage about internal state and avoids amplification attacks where an attacker sends invalid packets to elicit responses.

Implementation Status

Feature	Status
Noise IK handshake (with epoch)	Implemented
Peer restart detection (epoch mismatch)	Implemented
Link encryption (ChaCha20-Poly1305)	Implemented
Index-based session dispatch	Implemented
Replay protection (sliding window)	Implemented
Roaming (address-follows-crypto)	Implemented
Rate limiting (token bucket)	Implemented
Disconnect with reason codes	Implemented
Heartbeat liveness detection	Implemented
Reconnection handling	Implemented
Auto-reconnect after link-dead removal	Implemented
Handshake message retry (link + session layer)	Implemented
Common prefix framing	Implemented
AAD binding on encrypted frames	Implemented
Inner header timestamps	Implemented
Path MTU tracking (SessionDatagram)	Implemented
Metrics Measurement Protocol (MMP)	Implemented
ECN congestion signaling (CE relay, IPv6 marking)	Implemented
Rekey with index rotation	Implemented
Allowlist/blocklist	Planned

References

• fips-intro.md — Protocol overview and architecture
• fips-transport-layer.md — Transport layer (below FMP)
• fips-session-layer.md — FSP (above FMP)
• fips-mesh-operation.md — How FMP's routing and self-organization work in practice
• fips-wire-formats.md — Byte-level wire format reference