fips/docs/design/fips-mesh-layer.md

# FIPS Mesh Protocol (FMP)

The FIPS Mesh Protocol is the middle layer of the FIPS protocol stack. It sits
between the transport layer below and the FIPS Session Protocol (FSP) above.
FMP is where anonymous transport addresses become authenticated peers, where
the mesh self-organizes, and where forwarding decisions are made.

## Role

FMP manages direct peer connections over transports. When a transport delivers
a datagram from an unknown address, FMP authenticates the sender through a
Noise IK handshake, establishing a cryptographic link. Once authenticated, the
link carries all inter-peer communication: spanning tree gossip, bloom filter
updates, coordinate discovery, and forwarded session datagrams — all encrypted
per-hop.

FMP is the boundary between opaque transport addresses and identified peers.
Below FMP, everything is transport-specific addresses (host:port, MAC, .onion).
Above FMP, everything is peers identified by public keys and routable by
node_addr. The transport layer never sees FIPS-level structure; FSP never sees
transport addresses or routing details.

## Services Provided to FSP

From the session layer's perspective, FMP is a black box providing three
services. FSP knows nothing about transports, links, peers, spanning trees,
coordinates, bloom filters, hop counts, or network topology.

### Datagram Forwarding

FMP accepts a datagram addressed by source and destination node_addr and
delivers it best-effort toward the destination. The datagram travels hop by
hop — at each node, FMP decrypts the link layer, reads the destination
node_addr, makes a local forwarding decision, and re-encrypts onto the
next-hop link.

FSP provides: source node_addr, destination node_addr, hop limit, and an
opaque payload (the session-layer encrypted message).

FMP provides: best-effort delivery. No acknowledgment, no retransmission, no
ordering guarantee. Datagrams may be dropped, duplicated, or delivered out of
order.

### Error Signaling

When forwarding fails, FMP signals the source endpoint asynchronously:

- **CoordsRequired**: A transit node lacks the destination's tree coordinates
  and cannot make a forwarding decision. The source should re-initiate
  discovery and reset its coordinate warmup strategy.
- **PathBroken**: Greedy routing reached a dead end — no peer is closer to the
  destination than the current node. The source should re-discover the
  destination's current coordinates.

Both signals travel inside the SessionDatagram envelope (using the existing
src/dest/hop_limit addressing) but are generated by transit nodes and are not
end-to-end encrypted. They are rate-limited at 100ms per destination to prevent
storms during topology changes.

### Local Delivery

When a datagram arrives with a destination node_addr matching the local node,
FMP delivers it up to FSP for session-layer processing.

## Services Required from Transport Layer

FMP requires the following from each transport:

### Datagram Delivery

Send and receive raw datagrams to/from transport addresses. The transport
handles all medium-specific details. FMP sees only "send bytes to address" and
"bytes arrived from address."

### MTU Reporting

The maximum datagram size for a given link. FMP needs this to determine how
much payload fits in a single packet after link encryption overhead (37 bytes
for the encrypted frame wrapper: 16-byte outer header + 5-byte inner header +
16-byte AEAD tag).

### Connection Lifecycle

For connection-oriented transports, the transport must establish the underlying
connection before FMP can begin the Noise IK handshake. For connectionless
transports, datagrams can flow immediately.

### Endpoint Discovery (Optional)

When a transport discovers a FIPS-capable endpoint (via beacon, query, or
other transport-specific mechanism), it notifies FMP so that link setup can
be initiated. Transports without discovery support provide peer addresses
through configuration.

## Peer Authentication

### Noise IK Handshake

Every peer connection begins with a Noise IK handshake that mutually
authenticates both parties and establishes symmetric keys for link encryption.

The IK pattern is chosen because:

- The **initiator** knows the responder's static public key from configuration
  or discovery, and sends their own static key encrypted in the first message
- The **responder** learns the initiator's identity from the first message,
  then responds with their own ephemeral key

After the two-message handshake completes, both parties share symmetric
session keys derived from four DH operations (es, ss, ee, se). The handshake
provides mutual authentication, forward secrecy, and identity hiding for the
initiator.

### Epoch Exchange and Peer Restart Detection

Both IK handshake messages carry an encrypted epoch payload — an 8-byte
random value generated once at node startup:

- **msg1**: Ephemeral key (33 bytes) + encrypted static key (49 bytes) +
  encrypted epoch (24 bytes) = 106 bytes total
- **msg2**: Ephemeral key (33 bytes) + encrypted epoch (24 bytes) = 57 bytes
  total

The encrypted epoch (EPOCH_ENCRYPTED_SIZE = 24 bytes) consists of the
8-byte epoch value plus a 16-byte AEAD tag.

On reconnection, each peer compares the received epoch with the previously
stored epoch for that peer. An epoch mismatch indicates the peer has
restarted (generated a new epoch), triggering full link re-establishment
rather than treating the handshake as a simple reconnection. This prevents
stale session state from persisting across restarts.

### Identity Binding

The Noise handshake binds the link to the peer's cryptographic identity. After
handshake completion:

- The peer's public key (FIPS identity) is confirmed
- The node_addr is computed from the public key (SHA-256, truncated to 128 bits)
- All subsequent traffic on the link is authenticated by the Noise session —
  successful decryption proves the sender is the authenticated peer
- The link is registered in the dispatch table for O(1) packet routing

### Reconnection

When a Noise IK msg1 arrives from a peer that already has an authenticated
link, FMP accepts the new handshake alongside the existing session. If the new
handshake completes successfully, it replaces the old session. This handles
legitimate reconnection (network change, process restart, NAT rebinding)
without disrupting ongoing traffic until the new session is confirmed.

### Auto-Reconnect

When MMP's liveness detection removes a peer (dead timeout exceeded), FMP
automatically re-initiates the connection if the peer is configured for it.
The auto-reconnect path:

1. `check_link_heartbeats()` detects the dead peer and calls
   `remove_active_peer()`, tearing down the link and triggering tree/bloom
   reconvergence
2. `schedule_reconnect()` checks whether the peer is in the auto-connect list
   with `auto_reconnect: true` (the default)
3. If eligible, the peer is fed into the retry system with unlimited retries
   and exponential backoff (same base interval and max backoff as startup
   retries, configured via `node.retry.*`)
4. On each retry tick, a fresh Noise IK handshake is initiated toward the
   peer's configured transport addresses

Auto-reconnect only applies to peers in the static peer list with
`connect_policy: auto_connect`. Inbound-only peers (those not in the local
config) are not reconnected — the owning node (the one with the outbound
config) is responsible for re-establishing the link.

### Handshake Message Retry

Both link-layer (Noise IK msg1/msg2) and session-layer (SessionSetup/
SessionAck) handshakes use message-level retry with exponential backoff
within the handshake timeout window. This handles packet loss on the
underlying transport without waiting for the full handshake timeout to
expire.

Configuration (under `node.rate_limit.*`):

- `handshake_resend_interval_ms` (default 1000): initial resend interval
- `handshake_resend_backoff` (default 2.0): backoff multiplier per resend
- `handshake_max_resends` (default 5): max resends per handshake attempt

With defaults, resends occur at 1s, 2s, 4s, 8s, 16s — all within the 30s
handshake timeout. Under 19% per-attempt loss (10% bidirectional), the
probability of all 6 attempts (initial + 5 resends) failing is ~0.005%.

Session-layer resends wrap the stored payload in a fresh SessionDatagram so
routing adapts to topology changes between resends. Responder idempotency:
duplicate msg1/SessionSetup triggers resend of the stored msg2/SessionAck.

## Link Encryption

All traffic between authenticated peers is encrypted. Every packet on a link
— gossip messages, routing queries, forwarded session datagrams, disconnect
notifications — passes through Noise's ChaCha20-Poly1305 AEAD.

### Encrypted Frame Structure

Post-handshake packets are wrapped in an encrypted frame consisting of:

- A 4-byte common prefix (version, phase, flags, payload length)
- A receiver index for O(1) session lookup
- An explicit counter used as the AEAD nonce
- The ciphertext with a Poly1305 authentication tag

The 16-byte outer header (common prefix + receiver index + counter) is used as
AAD for the AEAD, binding the header to the ciphertext without encrypting it.

The plaintext inside the encrypted frame begins with a 5-byte inner header
(4-byte session-relative timestamp followed by a message type byte), then the
message-specific payload.

See [../reference/wire-formats.md](../reference/wire-formats.md) for
the complete wire format specification.

### What Encryption Provides

- **Confidentiality**: An observer on the underlying transport sees only
  encrypted packets, packet timing, and packet sizes
- **Integrity**: Any modification to a packet is detected by the AEAD tag
- **Authentication**: Only the authenticated peer can produce valid ciphertext
  for this link's session keys

### What Encryption Does Not Provide

- **End-to-end confidentiality**: Link encryption protects traffic on a single
  hop. Each transit node decrypts, reads the routing envelope, and re-encrypts
  for the next hop. Session-layer encryption (FSP) provides end-to-end
  confidentiality.
- **Traffic analysis protection**: Packet timing, sizes, and volume are visible
  to transport-layer observers.

## Index-Based Session Dispatch

Incoming packets are dispatched to the correct Noise session using a
receiver index — a random 32-bit value chosen by the receiver during the
handshake. This enables O(1) lookup without relying on source addresses.

Each party in a link has two indices:

- **our_index**: Chosen by us, included by the peer in packets sent to us
- **their_index**: Chosen by them, included by us in packets sent to them

The tuple `(transport_id, receiver_idx)` uniquely identifies a session.

### Index Properties

- **Random**: Cryptographically random to prevent guessing
- **Unique per transport**: No two active sessions on the same transport share
  an index
- **Scoped to transport**: The same index value may appear on different
  transports
- **Rotated on rekey**: New indices allocated on rekey to prevent cross-session
  correlation by passive observers

### Dispatch Flow

1. Read the 4-byte common prefix to determine the phase (established,
   handshake msg1, msg2)
2. For established frames (phase 0x0): look up `(transport_id, receiver_idx)`
   in the session table — O(1) hash lookup. Unknown indices are rejected
   before any cryptographic operation.
3. For handshake msg2 (phase 0x2): look up by our sender index to match to a
   pending outbound handshake
4. For handshake msg1 (phase 0x1): rate-limited processing, creates new state

This approach follows WireGuard's design: source address is informational,
not authoritative. Only successful cryptographic verification establishes
authenticity.

## Roaming

When an encrypted packet successfully decrypts, the sender is the
authenticated peer regardless of what transport address the packet arrived
from. FMP updates the peer's current address to the packet's source address,
and subsequent outbound packets use the updated address.

This allows peers to change transport addresses (e.g., host:port for UDP)
without session interruption. The mechanism is:

1. Packet arrives from a different address than expected
2. Receiver index lookup finds the session
3. AEAD decryption succeeds — the sender is cryptographically authenticated
4. Peer's address is updated to the new source address

Roaming is most useful for UDP, where source addresses can change due to NAT
rebinding or network changes. For connection-oriented transports, "roaming"
manifests as reconnection rather than mid-session address change.

Roaming addresses *mid-session* NAT rebinding. Establishing the initial UDP
path through NAT is a separate concern, addressed by the optional
Nostr-mediated overlay discovery and STUN-assisted hole punching feature
(see [fips-transport-layer.md](fips-transport-layer.md) and
[../reference/configuration.md](../reference/configuration.md)).

## Replay Protection

Each link session maintains per-direction counters:

- **Send counter**: Monotonically increasing, used as the AEAD nonce for each
  outbound packet
- **Receive window**: A sliding bitmap (2048 entries) tracking which counters
  have been seen

The receive window handles the realities of unreliable transports: packets may
arrive out of order, be duplicated, or be lost. The window accepts any counter
not yet seen and within 2048 of the highest counter received. Counters older
than the window are rejected.

The replay check is performed before decryption to prevent CPU exhaustion from
replayed packets that would pass the index lookup but fail decryption.

During link transitions, stale packets from a previous Noise session arrive
encrypted with old counters and are correctly rejected. To avoid excessive log
volume from these benign bursts, replay detection logging is suppressed after
the first 3 occurrences per peer. A summary is emitted when the peer's session
is re-established or the peer is removed.

## Rate Limiting

Handshake initiation (msg1) is the primary attack surface for unauthenticated
traffic. Each msg1 requires Noise DH operations (~200µs on modern CPUs),
state allocation, and response generation.

FMP uses a global token bucket rate limiter:

- **Burst capacity**: Handles legitimate connection storms (e.g., node restart
  with many configured peers)
- **Sustained rate**: Limits steady-state new connections per second
- **Global scope**: Rate limiting is global, not per-source, because UDP
  source addresses are trivially spoofable

Additional protections:

- **Connection limit**: Maximum number of pending inbound handshakes, capping
  memory usage
- **Handshake timeout**: Stale pending handshakes are cleaned up after a
  configurable timeout
- **Peer ACL**: Optional allowlist / denylist filtering of peer npubs
  before handshake processing (loaded from `/etc/fips/peers.allow` and
  `/etc/fips/peers.deny`, mtime-watched and reloaded automatically)

## Disconnect

FMP supports orderly link teardown via a Disconnect message carrying a reason
code (shutdown, restart, protocol error, transport failure, resource
exhaustion, security violation, configuration change, timeout).

On receiving Disconnect, FMP immediately cleans up state: removes the peer
from the peer table, frees the session index, removes the link, and cleans up
address mappings. If the departed peer was the tree parent, FMP triggers parent
reselection.

Disconnect is best-effort — if the transport is broken, the message won't
arrive. Timeout-based detection remains the fallback for detecting failed
links.

On node shutdown, Disconnect is sent to all active peers before transports are
stopped.

## Rekey

FMP periodically negotiates a fresh Noise session over each established link
to bound forward-secrecy exposure: limiting AEAD nonce reuse risk, bounding
the volume of ciphertext recoverable from a stolen long-term static key, and
rotating the session indices that addressed packets carry on the wire.

A rekey is initiated when either threshold is reached on the link's current
session: `node.rekey.after_secs` (default 120) elapsed since the link came
up or last rekeyed, or `node.rekey.after_messages` (default 65536) frames
sent. Either side can be the initiator independently. Rekey is on by
default and can be disabled via `node.rekey.enabled: false` (the
configuration tree is documented in
[../reference/configuration.md](../reference/configuration.md)).

### Mechanism

A rekey reuses the Noise IK pattern of the initial handshake, but the two
messages travel over the existing link as ordinary encrypted FMP frames
rather than as plaintext bootstrap packets. The initiator builds a fresh
`HandshakeState`, generates msg1, and sends it through the current session;
the responder consumes msg1, builds msg2, and replies. After both sides
have exchanged messages and finalised the new keys, traffic transitions
from the old session to the new one.

Cutover is signalled in-band by the **K-bit** in the FMP flags byte. The
initiator switches first, on its own schedule once it has read msg2, and
marks its frames under the new session with K; the responder follows on
the first K-marked frame that authenticates against its new session. A
responder drops a pending session the initiator never adopts after a hold
(the drain ceiling, 120 s at stock settings), so the next rekey can
proceed. A new pair of session indices is allocated as part of the new
session, replacing the old indices on subsequent frames (see
[Index Properties](#index-properties)).

### Drain Window

To absorb in-flight reordering across the cutover, the old session is not
discarded immediately. Each peer retains it in a `previous_session` slot
on the active-peer state for `DRAIN_WINDOW_SECS = 10` seconds (a
compile-time constant in `src/node/handlers/rekey.rs`). During the
window, decrypt attempts fall back to `previous_session` when the new
session rejects a frame, so a packet sent under the old keys that
arrives a few hundred milliseconds late still decrypts. After the
window expires, the old session is dropped.

### Dual-Initiation Race

On high-latency links, both sides' rekey timers can fire close enough
together that each peer's msg1 crosses the other in flight. Without
arbitration, each side would act as both initiator and responder, end
up with two different Noise sessions, and lose connectivity at cutover.
FMP arbitrates with a deterministic tie-breaker: the peer with the
**numerically smaller `NodeAddr`** wins the role of initiator and
discards any inbound msg1 it sees during the race; the larger-`NodeAddr`
peer abandons its own initiation and processes the inbound msg1 as
responder. The same tie-breaker is applied to cross-connection races
during initial handshake.

### Operator Visibility

Successful cutover is reported at INFO level on the K-bit observation;
intermediate steps (handshake start, msg1/msg2 exchange, drain-window
fallback decrypts) log at DEBUG/TRACE. Failures (handshake error,
drain-window expiry without cutover) log at WARN.

The end-to-end rekey at the session layer follows a parallel design;
see [fips-session-layer.md](fips-session-layer.md).

## Liveness Detection

FMP detects link liveness through a combination of explicit heartbeats and
traffic observation.

### Heartbeat

A Heartbeat message (0x51) is sent to each active peer at a configurable
interval (`node.heartbeat_interval_secs`). The heartbeat is a minimal
encrypted frame with no payload beyond the standard inner header (timestamp +
message type). Any successfully decrypted frame — data, gossip, MMP report,
or heartbeat — resets the peer's last-receive timestamp tracked by the MMP
receiver.

### Path probes

A peer reachable over more than one transport holds a path per transport
under the one session; each path is heartbeated on its own with a
PathProbe (0x52) answered by a PathAck (0x53) on the same path, and a
path that is going is announced with a PathClose (0x54) on a surviving
one. Layouts in `wire-formats.md`; the mechanism in
`docs/design/fips-multi-path-switchover.md`. A peer with one path is not
path-probed: the bare Heartbeat above keeps its liveness.

### Dead Timeout

When no traffic (of any kind) is received from a peer for the
configured `node.link_dead_timeout_secs` window, the peer is declared dead and
removed via `remove_active_peer()`. This triggers the full teardown cascade:
spanning tree parent reselection (if the dead peer was the parent),
TreeAnnounce propagation, coordinate cache flush, and bloom filter recompute.

If the dead peer is eligible for auto-reconnect (see [Auto-Reconnect]
(#auto-reconnect)), reconnection is scheduled immediately after removal.

The heartbeat is independent of MMP. Gossip is event-driven, Lightweight
produces receiver reports only when traffic arrives, and Minimal emits no
reports at all — so on a fully idle link no MMP-mode combination
guarantees periodic activity. The heartbeat is the always-on liveness
signal.

## Link Message Types

FMP defines several encrypted message types carried inside the
established-frame envelope. They group naturally by purpose:

- **Routing gossip**: TreeAnnounce carries spanning-tree announcements
  between direct peers; FilterAnnounce carries bloom-filter
  reachability updates between direct peers. Both are peer-to-peer
  (not forwarded).
- **Discovery**: LookupRequest is forwarded through tree peers under
  bloom-filter guidance to find a destination's coordinates;
  LookupResponse routes back to the requester via reverse-path lookup
  in `recent_requests`.
- **Forwarded payload**: SessionDatagram carries a session-layer
  payload hop-by-hop toward the destination.
- **Metrics**: SenderReport and ReceiverReport carry the link-layer
  MMP report stream peer-to-peer.
- **Liveness and lifecycle**: Heartbeat is a minimal frame sent
  peer-to-peer to keep the link alive; Disconnect carries an orderly
  teardown reason code peer-to-peer.
- **Paths**: PathProbe, PathAck and PathClose (0x52–0x54) prove, measure
  and withdraw the individual transports a multi-path peer is reachable
  over, peer-to-peer under the one session.

Handshake messages (phase 0x1 msg1, phase 0x2 msg2) travel before
encryption is established and are identified by the FMP common-prefix
`phase` field rather than a `msg_type` byte.

See [../reference/wire-formats.md](../reference/wire-formats.md) for
byte-level message layouts and the canonical FMP message type
catalog, and [fips-mesh-operation.md](fips-mesh-operation.md) for how
these messages work together to build and maintain the mesh.

## Metrics Measurement Protocol (MMP)

MMP runs on every active link to provide per-link quality metrics
(SRTT, loss, jitter, goodput, OWD trend, ETX) to the operator and to
the spanning tree layer for cost-based parent selection. Reports are
exchanged peer-to-peer between direct neighbors at RTT-adaptive
intervals clamped to `[1s, 5s]`, with a 200 ms cold-start floor for
the first five SRTT samples.

The CE (Congestion Experienced) bit in the FMP flags byte carries
hop-by-hop ECN signaling: transit nodes detect congestion on outgoing
links (via MMP loss/ETX or `SO_RXQ_OVFL` kernel drops) and set CE on
forwarded packets, which the destination then mirrors to the IPv6
Traffic Class for ECN-capable flows.

For the full MMP design — operating modes, report scheduling, spin
bit interaction, ECN, and the algorithmic details shared with
session-layer MMP — see [fips-mmp.md](fips-mmp.md). For the
SenderReport and ReceiverReport byte layouts, see
[../reference/wire-formats.md](../reference/wire-formats.md).
Configuration knobs live under `node.mmp.*` and `node.ecn.*` in
[../reference/configuration.md](../reference/configuration.md).

## Security Properties

### Threat Resistance

The link-layer threat-resistance matrix (connection exhaustion, CPU
exhaustion, replay, state confusion, spoofing variants, address
spoofing, session correlation) is consolidated in
[../reference/security.md](../reference/security.md) along with the
session-layer matrix and operator-facing controls.

### Unauthenticated Attack Surface

Only handshake msg1 can be sent by unauthenticated parties. Encrypted frames
require a known session index, and msg2 requires a response to a specific
ephemeral key. The msg1 attack surface is protected by rate limiting,
connection limits, and handshake timeouts.

### Authenticated Peer Misbehavior

Authentication establishes identity but does not grant trust. An authenticated
peer can send malformed packets (which fail AEAD and are dropped) or
high-frequency traffic (rate-limited by higher layers). False tree coordinate
claims are constrained by signature verification on TreeAnnounce messages.

### Silent Drop Policy

Invalid packets are silently dropped without error responses. This prevents
information leakage about internal state and avoids amplification attacks where
an attacker sends invalid packets to elicit responses.

## Implementation Status

| Feature | Status |
| ------- | ------ |
| Noise IK handshake (with epoch) | **Implemented** |
| Peer restart detection (epoch mismatch) | **Implemented** |
| Link encryption (ChaCha20-Poly1305) | **Implemented** |
| Index-based session dispatch | **Implemented** |
| Replay protection (sliding window) | **Implemented** |
| Roaming (address-follows-crypto) | **Implemented** |
| Rate limiting (token bucket) | **Implemented** |
| Disconnect with reason codes | **Implemented** |
| Heartbeat liveness detection | **Implemented** |
| Multi-path (per-path probes, switchover without a handshake) | **Implemented** (`feat/multi-path-switchover`) |
| Reconnection handling | **Implemented** |
| Auto-reconnect after link-dead removal | **Implemented** |
| Handshake message retry (link + session layer) | **Implemented** |
| Common prefix framing | **Implemented** |
| AAD binding on encrypted frames | **Implemented** |
| Inner header timestamps | **Implemented** |
| Path MTU tracking (SessionDatagram) | **Implemented** |
| Metrics Measurement Protocol (MMP) | **Implemented** |
| ECN congestion signaling (CE relay, IPv6 marking) | **Implemented** |
| Rekey with index rotation | **Implemented** |
| Peer ACL (allowlist / denylist) | **Implemented** |

## References

- [fips-concepts.md](fips-concepts.md) — Protocol overview
- [fips-architecture.md](fips-architecture.md) — Layer architecture and
  identity model
- [fips-transport-layer.md](fips-transport-layer.md) — Transport layer (below FMP)
- [fips-session-layer.md](fips-session-layer.md) — FSP (above FMP)
- [fips-mmp.md](fips-mmp.md) — Metrics Measurement Protocol (link + session)
- [fips-mesh-operation.md](fips-mesh-operation.md) — How FMP's routing and
  self-organization work in practice
- [../reference/wire-formats.md](../reference/wire-formats.md) — Byte-level
  wire format reference