mirror of
https://github.com/jmcorgan/fips.git
synced 2026-08-10 16:43:12 +00:00
Johnathan Corgan
5abf9a9325
Restructures /docs/ by reader purpose (tutorials, how-to, reference, design), adds the new-user-progression and operator-recipe content the prior layout lacked, runs an accuracy pass against current source across the pre-existing design docs, and rewrites the gateway feature-set documentation end-to-end around its actual operational profile (a niche feature designed for systems already serving DHCP/DNS to a LAN, with two independent halves — outbound LAN→mesh, inbound mesh→LAN — sharing one nftables table, one binary, and one control socket). Top-level README and getting-started rewritten around two equally-weighted deployment modes (overlay on existing IP networks; ground-up over non-IP transports). ## Additions - 11 new tutorials in docs/tutorials/: an 8-step new-user progression from single-daemon test-mesh peering through to a ground-up two-device mesh, an IPv6-adapter side-trip walkthrough, an Advanced Tutorials index, and a hand-held OpenWrt walk-through for fips-gateway deployment that exercises both halves of the feature. - 12 new how-tos in docs/how-to/: firewall activation, Nostr discovery (resolve / advertise / open across five scenarios), Tor onion (directory + control_port modes), UDP buffer tuning, unprivileged-user setup, persistent identity, host aliases, Bluetooth LE peering, MTU diagnostics, manual Linux-host gateway deployment (covers both halves), gateway troubleshooting (organised by half), and a section index. - 9 new reference docs in docs/reference/: configuration, wire formats, control-socket protocol, four CLI references (fips, fipsctl, fipstop, fips-gateway), security posture matrix, and Nostr events catalog. Configuration and wire-formats are renamed-and-extended from prior design/ versions; the other seven are net-new. - 6 new design docs: fips-concepts, fips-architecture, and fips-prior-work split out of the deleted fips-intro.md; consolidated fips-mmp and fips-mtu aggregations; and a new generic port-advertisement-and-nat-traversal doc (Nostr-signaled port advertisement plus UDP NAT-traversal protocol, FIPS as an example implementation, suitable for eventual NIP submission). - Top-level docs/getting-started.md walking through the binary-installer-only Install story. - packaging/common/hosts pre-populated with the eight public test-mesh nodes so shortnames resolve out of the box on every fresh install. ## Changes - 23 wire-format diagrams relocated to reference/diagrams/ alongside the wire-formats move. - 4 design diagrams corrected against source code (fips-protocol-stack, fips-identity-derivation, fips-coordinate-discovery, fips-routing-decision). - 10 pre-existing design docs reconciled with current source. Numeric corrections: stale link-MMP report bounds (now [1s, 5s] with 200 ms cold-start floor); UDP default MTU (now 1280, IPv6 minimum); node_addr formula (SHA-256(pubkey)[..16]); Noise patterns (IK at link, XK at session); peer-ACL semantics (strict allowlist requires ALL in peers.deny); daemon DNS upstream ([::1]:5354); on-the-wire bloom-filter size (1,071 bytes); obsolete Cargo-feature references (PR #79 dropped them) removed. - Transport framing tightened across the docs: TCP is for UDP-filtered networks (not NAT traversal); Tor is a deployment mode (not failover); WebSocket dropped (not a shipped FIPS transport); WiFi promoted to Implemented via Ethernet in infrastructure mode; classic-Bluetooth row removed (BLE is the only Bluetooth-mode transport). - docs/design/fips-gateway.md rewritten end-to-end to lead with the niche-feature framing and the two-halves structure. Title moved from "FIPS Outbound LAN Gateway" to "FIPS Gateway"; architecture section describes the common machinery (the fips-gateway service, the nftables table, the control socket) before splitting into separate "Outbound Half" and "Inbound Half" sections of equal weight; security considerations split per-half; no Future Work section (speculative directions live in the project tracker, not in protocol design docs). Inbound port forwarding is a first-class half rather than a buried "Implemented Extensions" subsection. - Gateway terminology unified across all gateway docs as a separate Linux service running alongside the fips daemon (its own systemd unit / OpenWrt init script). Container- pattern terms (sidecar) are reserved for the Docker/Kubernetes sidecar deployment examples — the testing/sidecar/ tree, examples/k8s-sidecar/, examples/sidecar-nostr-relay/, examples/wireguard-sidecar-macos/, and the related CHANGELOG / top-level README entries — where the term carries its standard container meaning. - Net-new design body content: rekey section in fips-mesh-layer (Noise IK msg1/msg2 over the established link, K-bit cutover, drain window, smaller-NodeAddr-wins tie-breaker on dual-init); Mesh Size Estimation and Antipoison FPR Cap sections in fips-bloom-filters; Mesh-Interface Query Filter subsection in fips-ipv6-adapter; failure-suppression knobs and clock- skew tolerance in fips-nostr-discovery; loop-rejection and mid-chain ancestor swap added to spanning-tree propagation / stability rules; Priority Chain in fips-mesh-operation renumbered to match the routing-decision diagram. - Top-level README: dropped the stale nostr-discovery cargo-feature parenthetical. docs/README.md and the four section READMEs (tutorials, how-to, reference, design) refreshed for the new structure; index rows reflect both halves of the gateway feature and the new fips-gateway CLI reference. - Cargo.toml [package.metadata.deb] assets path updated for the fips-security.md move; .gitignore /reference/ rule anchored to repo root so docs/reference/ is trackable. - packaging/openwrt-ipk/files/etc/fips/fips.yaml configuration-doc URL updated to the new docs/reference/configuration.md location. ## Deletions - docs/design/fips-intro.md (split into the three new intro design docs). - docs/design/document-relationships.svg (orphan, no longer referenced). - docs/proposals/ tree removed; the only proposal it contained (the Nostr UDP hole-punch protocol) was rewritten as the new generic design/port-advertisement-and-nat-traversal.md.
219 lines
9.2 KiB
Markdown
219 lines
9.2 KiB
Markdown
# Metrics Measurement Protocol (MMP)
|
||
|
||
The Metrics Measurement Protocol provides per-link and per-session
|
||
quality metrics — SRTT, loss, jitter, goodput, ETX, and one-way delay
|
||
trend — using only counter and timestamp fields already present in
|
||
the FMP and FSP wire formats. No additional probing traffic is
|
||
required. The same algorithms and report message format are used at
|
||
both layers; only the routing scope and configuration namespace
|
||
differ.
|
||
|
||
This document is the canonical home for the MMP design. For the
|
||
link-layer instance's role inside FMP, see
|
||
[fips-mesh-layer.md](fips-mesh-layer.md). For the session-layer
|
||
instance's role inside FSP, see
|
||
[fips-session-layer.md](fips-session-layer.md). For the byte-level
|
||
SenderReport and ReceiverReport layouts, see
|
||
[../reference/wire-formats.md](../reference/wire-formats.md).
|
||
|
||
## Two Layers, One Protocol
|
||
|
||
MMP runs at two layers:
|
||
|
||
- **Link-layer MMP**: One instance per active FMP peer link. Reports
|
||
are exchanged peer-to-peer between direct neighbors and measure the
|
||
quality of that single hop.
|
||
- **Session-layer MMP**: One instance per established FSP session.
|
||
Reports are encrypted end-to-end and forwarded through every transit
|
||
link, measuring end-to-end quality independent of hop count.
|
||
|
||
The algorithms (SRTT estimation, jitter computation, loss inference,
|
||
ETX) are identical at both layers. The differences are configuration
|
||
namespace, report intervals, and routing scope. See
|
||
[Layer Differences](#layer-differences) below.
|
||
|
||
## Metrics Tracked
|
||
|
||
MMP computes the following metrics from the per-frame counter and
|
||
timestamp fields:
|
||
|
||
- **SRTT** — Smoothed round-trip time (Jacobson/RFC 6298, α=1/8).
|
||
Derived from timestamp-echo in ReceiverReports with dwell-time
|
||
compensation.
|
||
- **Loss rate** — Bidirectional loss inferred from counter gaps.
|
||
Tracked as both instantaneous (per-interval) and long-term EWMA.
|
||
- **Jitter** — Interarrival jitter (RFC 3550 algorithm) in
|
||
microseconds.
|
||
- **Goodput** — Bytes per second of payload data (excludes MMP
|
||
reports).
|
||
- **OWD trend** — One-way delay trend (µs/s, signed). Indicates
|
||
congestion buildup before loss occurs.
|
||
- **ETX** — Expected Transmission Count, computed from bidirectional
|
||
delivery ratios. Used in cost-based parent selection via
|
||
`link_cost = etx * (1.0 + srtt_ms / 100.0)`, and in bloom-filter
|
||
candidate ranking inside `find_next_hop()` (the same `link_cost`
|
||
is the primary key when choosing among bloom-filter peers, with
|
||
tree distance as the tie-breaker).
|
||
- **Dual EWMA trends** — Short-term (α=1/4) and long-term (α=1/32)
|
||
trend indicators for both RTT and loss, enabling change detection.
|
||
|
||
Session-layer MMP additionally tracks the observed forward-path MTU;
|
||
see [fips-mtu.md](fips-mtu.md) for the end-to-end path-MTU mechanism.
|
||
|
||
## Operating Modes
|
||
|
||
MMP supports three modes:
|
||
|
||
| Mode | Reports Exchanged | Metrics Available |
|
||
| ---- | ----------------- | ----------------- |
|
||
| **Full** (default) | SenderReport + ReceiverReport | All metrics including RTT, loss, jitter, goodput, OWD trend |
|
||
| **Lightweight** | ReceiverReport only | Loss (from counter gaps), jitter, OWD trend. No RTT. |
|
||
| **Minimal** | None | Spin bit and CE echo flags only. No computed metrics. |
|
||
|
||
The mode is configured per layer (`node.mmp.mode` and
|
||
`node.session_mmp.mode`).
|
||
|
||
## Report Scheduling
|
||
|
||
Reports are sent at RTT-adaptive intervals computed as
|
||
`clamp(2 × SRTT, low, high)`. A cold-start interval is used until SRTT
|
||
has converged.
|
||
|
||
| Layer | Adaptive bounds | Cold-start |
|
||
| ----- | --------------- | ---------- |
|
||
| Link | `[1s, 5s]` | 200 ms (first 5 samples) |
|
||
| Session | `[500ms, 10s]` | 1 s |
|
||
|
||
The session-layer bounds are higher because session reports are
|
||
encrypted and forwarded through every transit link, so bandwidth cost
|
||
is proportional to path length.
|
||
|
||
## Spin Bit and RTT
|
||
|
||
The SP (spin bit) flag in the FMP inner header follows the QUIC spin
|
||
bit pattern: reflected on receive, toggled on send when the reflected
|
||
value matches the last sent value. The spin bit state machine runs
|
||
for TX reflection, but **RTT samples from the spin bit are
|
||
discarded**. In a mesh protocol where frames are sent irregularly
|
||
(tree announces, bloom filters, MMP reports on different timers),
|
||
inter-frame processing delays inflate spin bit RTT measurements
|
||
unpredictably. Timestamp-echo from ReceiverReports (with dwell-time
|
||
compensation) is the sole SRTT source.
|
||
|
||
The spin bit lives in the link-layer FMP inner header, so this
|
||
mechanism applies to link-layer MMP only. Session-layer MMP carries
|
||
its spin bit in the FSP encrypted inner header but uses it the same
|
||
way: reflected for diagnostic visibility, not used for SRTT.
|
||
|
||
## ECN Congestion Signaling
|
||
|
||
The CE (Congestion Experienced) flag (bit 1 in the FMP flags byte)
|
||
provides hop-by-hop congestion signaling through the mesh. Transit
|
||
nodes detect congestion on outgoing links and set CE on forwarded
|
||
packets; once set, the flag stays set for all subsequent hops to the
|
||
destination.
|
||
|
||
**Congestion detection** triggers on any of:
|
||
|
||
- Outgoing link MMP loss rate ≥ `node.ecn.loss_threshold` (default 5%)
|
||
- Outgoing link MMP ETX ≥ `node.ecn.etx_threshold` (default 3.0)
|
||
- Kernel receive buffer drops detected on any local transport (via
|
||
`SO_RXQ_OVFL` on UDP)
|
||
|
||
**CE relay**: The forwarding path computes
|
||
`outgoing_ce = incoming_ce || local_congestion`. Once CE is set on a
|
||
packet, it remains set for the rest of the forward path.
|
||
|
||
**IPv6 ECN-CE marking**: When a CE-flagged DataPacket arrives at its
|
||
final destination, the IPv6 Traffic Class ECN bits are marked CE
|
||
(0b11) before TUN delivery — but only for ECN-capable packets (ECT(0)
|
||
or ECT(1)). Not-ECT packets are never marked per RFC 3168. The host
|
||
TCP stack then echoes ECE in ACKs, triggering sender cwnd reduction
|
||
through standard congestion control.
|
||
|
||
**Session-layer tracking**: The `ecn_ce_count` field in MMP
|
||
ReceiverReports tracks CE-flagged packets received per link, providing
|
||
end-to-end visibility into congestion propagation.
|
||
|
||
ECN signaling is a link-layer mechanism. Session-layer MMP only
|
||
observes the CE counter as part of the report stream; CE marking is
|
||
not generated end-to-end. Tuning parameters live under `node.ecn.*`
|
||
in [../reference/configuration.md](../reference/configuration.md).
|
||
|
||
## Send Failure Backoff (Session Layer Only)
|
||
|
||
When a session MMP report cannot be delivered (destination unreachable,
|
||
no route), the sender applies exponential backoff to the probe
|
||
interval — a standard distributed-systems pattern for transient
|
||
failure handling:
|
||
|
||
- Each consecutive failure doubles the interval: 2x, 4x, 8x, 16x, 32x
|
||
- Backoff caps at 32x the base interval (5 consecutive failures)
|
||
- A successful send resets to the normal SRTT-based interval
|
||
- Debug logging is suppressed after 3 consecutive failures; a summary
|
||
is logged when the destination becomes reachable again
|
||
|
||
This prevents wasted CPU and log noise when a session's remote
|
||
endpoint has departed the network but the local session has not yet
|
||
timed out. Link-layer MMP has no equivalent — link-layer reports are
|
||
peer-to-peer over an authenticated link, so delivery failure is
|
||
indistinguishable from link death and the link-liveness mechanism
|
||
takes over.
|
||
|
||
## Layer Differences
|
||
|
||
| Aspect | Link layer | Session layer |
|
||
| ------ | ---------- | ------------- |
|
||
| Routing scope | Peer-to-peer (one hop) | End-to-end (forwarded through every hop) |
|
||
| Configuration namespace | `node.mmp.*` | `node.session_mmp.*` |
|
||
| Report bounds | `[1s, 5s]` | `[500ms, 10s]` |
|
||
| Cold-start interval | 200 ms (first 5 samples) | 1 s |
|
||
| Bandwidth cost | One link | Proportional to path length |
|
||
| Send-failure backoff | Not applicable | Yes |
|
||
| Path-MTU echo | Not applicable | PathMtuNotification (see [fips-mtu.md](fips-mtu.md)) |
|
||
| Idle-timeout interaction | None | Reports do **not** reset session idle timer |
|
||
|
||
## Idle Timeout Interaction (Session Layer Only)
|
||
|
||
MMP reports (SenderReport, ReceiverReport) and PathMtuNotification do
|
||
**not** reset the session idle timer. Only application data
|
||
(DataPacket, type 0x10) resets `last_activity`. This ensures sessions
|
||
with no application traffic tear down after
|
||
`node.session.idle_timeout_secs` (default 90s), while MMP continues
|
||
providing measurement data up to the teardown moment.
|
||
|
||
## Operator Logging
|
||
|
||
Both layers emit periodic metrics at info level. The interval is
|
||
`node.mmp.log_interval_secs` for link-layer (default 30s) and
|
||
`node.session_mmp.log_interval_secs` for session-layer (default 30s).
|
||
|
||
Link-layer:
|
||
|
||
```text
|
||
MMP link metrics peer=node-b rtt=2.3ms loss=0.2% jitter=0.1ms goodput=76.0MB/s tx_pkts=1234 rx_pkts=5678
|
||
```
|
||
|
||
Session-layer:
|
||
|
||
```text
|
||
MMP session metrics session=npub1tdwa...84le rtt=4.3ms loss=0.6% jitter=0.2ms goodput=71.3MB/s mtu=1472 tx_pkts=1234 rx_pkts=5678
|
||
```
|
||
|
||
Teardown logs include final SRTT, loss rate, jitter, ETX, goodput,
|
||
and cumulative tx/rx packet and byte counts.
|
||
|
||
## See also
|
||
|
||
- [fips-mesh-layer.md](fips-mesh-layer.md) — link-layer MMP integration
|
||
inside FMP
|
||
- [fips-session-layer.md](fips-session-layer.md) — session-layer MMP
|
||
integration inside FSP
|
||
- [fips-mtu.md](fips-mtu.md) — PathMtuNotification, the session-only
|
||
end-to-end path-MTU echo
|
||
- [../reference/wire-formats.md](../reference/wire-formats.md) —
|
||
SenderReport (0x01 / 0x11) and ReceiverReport (0x02 / 0x12) byte
|
||
layouts
|
||
- [../reference/configuration.md](../reference/configuration.md) —
|
||
full `node.mmp.*`, `node.session_mmp.*`, and `node.ecn.*` knob tables
|