Fix FIPS_OVERHEAD constant and add CP flag MTU guard

FIPS_OVERHEAD was 150 but had two bugs: the session AEAD tag (16 bytes)
was listed in the comment but missing from the arithmetic, and the
coordinate budget (~60 bytes) was undersized and didn't belong in a
constant representing fixed data path overhead.

Corrected to 106 bytes (the actual fixed overhead without coordinates).
This increases effective_ipv6_mtu from 1322 to 1366 for standard
Ethernet, well above the IPv6 minimum of 1280.

Added a guard in send_session_data() that computes the total wire size
with coordinates before committing to include them. If adding coords
would exceed the transport MTU, the CP flag is skipped and the warmup
counter is not decremented. This prevents silently producing oversized
packets at tree depth 2+.

Updated design docs (ipv6-adapter, wire-formats, mesh-operation) with
corrected overhead values.
This commit is contained in:
Johnathan Corgan
2026-02-19 14:18:34 +00:00
parent 7df1f21429
commit 999144f59a
7 changed files with 100 additions and 47 deletions
+12 -10
View File
@@ -119,17 +119,19 @@ after all layers of wrapping.
| Layer | Overhead | Purpose | | Layer | Overhead | Purpose |
| ----- | -------- | ------- | | ----- | -------- | ------- |
| Link encryption | 37 bytes | 16-byte outer header + 5-byte inner header + 16-byte AEAD tag | | Link encryption | 37 bytes | 16-byte outer header + 5-byte inner header (timestamp + msg_type) + 16-byte AEAD tag |
| SessionDatagram envelope | 36 bytes | type + ttl + path_mtu + src_addr + dest_addr | | SessionDatagram body | 35 bytes | ttl + path_mtu + src_addr + dest_addr (msg_type counted in inner header) |
| FSP header | 12 bytes | 4-byte prefix + 8-byte counter (used as AEAD AAD) | | FSP header | 12 bytes | 4-byte prefix + 8-byte counter (used as AEAD AAD) |
| FSP inner header | 6 bytes | 4-byte timestamp + 1-byte msg_type + 1-byte inner_flags (inside AEAD) | | FSP inner header | 6 bytes | 4-byte timestamp + 1-byte msg_type + 1-byte inner_flags (inside AEAD) |
| Session AEAD tag | 16 bytes | ChaCha20-Poly1305 tag on session-encrypted payload | | Session AEAD tag | 16 bytes | ChaCha20-Poly1305 tag on session-encrypted payload |
| **Minimal total** | **107 bytes** | | | **Data path total** | **106 bytes** | `FIPS_OVERHEAD` constant |
| Coordinates (if present) | ~43 bytes | Depth-dependent, first few packets only |
| **Worst case total** | **150 bytes** | With CP flag set for depth-3 paths |
The `FIPS_OVERHEAD` constant (150 bytes) is used for conservative MTU Coordinate piggybacking (CP flag) adds variable overhead: `2 + entries × 16`
calculations. per coordinate, with both src and dst coords sent. The send path skips the
CP flag if adding coords would exceed the transport MTU.
The `FIPS_OVERHEAD` constant (106 bytes) represents the fixed data path
overhead and is used for MTU calculations.
### Effective IPv6 MTU ### Effective IPv6 MTU
@@ -143,14 +145,14 @@ For typical deployments:
| Transport MTU | Effective IPv6 MTU | Notes | | Transport MTU | Effective IPv6 MTU | Notes |
| ------------- | ------------------ | ----- | | ------------- | ------------------ | ----- |
| 1472 (UDP/Ethernet) | 1322 | Standard deployment | | 1472 (UDP/Ethernet) | 1366 | Standard deployment |
| 1280 (UDP minimum) | 1130 | Below IPv6 minimum | | 1280 (UDP minimum) | 1174 | Below IPv6 minimum |
IPv6 mandates that every link support at least 1280 bytes. The minimum IPv6 mandates that every link support at least 1280 bytes. The minimum
transport path MTU for the IPv6 adapter is therefore: transport path MTU for the IPv6 adapter is therefore:
```text ```text
1280 + 150 = 1430 bytes 1280 + 106 = 1386 bytes
``` ```
Transports with smaller MTUs (LoRa at ~250 bytes, serial at 256 bytes) cannot Transports with smaller MTUs (LoRa at ~250 bytes, serial at 256 bytes) cannot
+2 -2
View File
@@ -515,8 +515,8 @@ routing decisions but retains its own end-to-end encryption and identity.
| LookupResponse | ~400 bytes | Response to discovery | Yes (greedy routed) | | LookupResponse | ~400 bytes | Response to discovery | Yes (greedy routed) |
| SessionDatagram + SessionSetup | ~232402 bytes | Session establishment | Yes (routed) | | SessionDatagram + SessionSetup | ~232402 bytes | Session establishment | Yes (routed) |
| SessionDatagram + SessionAck | ~122 bytes | Session confirmation | Yes (routed) | | SessionDatagram + SessionAck | ~122 bytes | Session confirmation | Yes (routed) |
| SessionDatagram + Data (minimal) | ~107 bytes + payload | Bulk traffic | Yes (routed) | | SessionDatagram + Data (minimal) | 106 bytes + payload | Bulk traffic | Yes (routed) |
| SessionDatagram + Data (with CP) | ~150 bytes + payload | Warmup/recovery | Yes (routed) | | SessionDatagram + Data (with CP) | 106 + coords + payload | Warmup/recovery | Yes (routed) |
| SessionDatagram + CoordsRequired | 70 bytes | Cache miss error | Yes (routed) | | SessionDatagram + CoordsRequired | 70 bytes | Cache miss error | Yes (routed) |
| SessionDatagram + PathBroken | 70+ bytes | Dead-end error | Yes (routed) | | SessionDatagram + PathBroken | 70+ bytes | Dead-end error | Yes (routed) |
| Disconnect | 2 bytes | Link teardown | No (peer-to-peer) | | Disconnect | 2 bytes | Link teardown | No (peer-to-peer) |
+5 -7
View File
@@ -648,14 +648,12 @@ Layer 0: Transport
| Layer | Overhead | Component | | Layer | Overhead | Component |
| ----- | -------- | --------- | | ----- | -------- | --------- |
| Link encryption | 37 bytes | 16 outer header (AAD) + 5 inner header + 16 AEAD tag | | Link encryption | 37 bytes | 16 outer header (AAD) + 5 inner header (timestamp + msg_type) + 16 AEAD tag |
| SessionDatagram | 36 bytes | 1 type + 1 ttl + 2 path_mtu + 16 src + 16 dest | | SessionDatagram body | 35 bytes | 1 ttl + 2 path_mtu + 16 src + 16 dest (msg_type counted in inner header) |
| FSP header | 12 bytes | 4 prefix + 8 counter | | FSP header | 12 bytes | 4 prefix + 8 counter |
| FSP inner header | 6 bytes | 4 timestamp + 1 msg_type + 1 inner_flags (inside AEAD) | | FSP inner header | 6 bytes | 4 timestamp + 1 msg_type + 1 inner_flags (inside AEAD) |
| Session AEAD tag | 16 bytes | Poly1305 tag on session-encrypted payload | | Session AEAD tag | 16 bytes | Poly1305 tag on session-encrypted payload |
| **Minimal total** | **107 bytes** | | | **Data path total** | **106 bytes** | `FIPS_OVERHEAD` constant |
| Coordinates (if present) | ~43 bytes | Varies with tree depth |
| **Worst case** | **150 bytes** | `FIPS_OVERHEAD` constant |
### At Each Transit Node ### At Each Transit Node
@@ -717,8 +715,8 @@ endpoint session keys).
| Scenario | Wire Size | Notes | | Scenario | Wire Size | Notes |
| -------- | --------- | ----- | | -------- | --------- | ----- |
| Encrypted frame minimum | 37 bytes | Empty body | | Encrypted frame minimum | 37 bytes | Empty body |
| SessionDatagram + Data (minimal) | 37 + 36 + 12 + 6 + payload + 16 | 107 + payload | | SessionDatagram + Data (minimal) | 37 + 35 + 12 + 6 + payload + 16 | 106 + payload |
| SessionDatagram + Data (with coords) | ~150 + payload | Worst case | | SessionDatagram + Data (with coords) | 106 + coords + payload | Coords vary with tree depth |
| SessionDatagram + SessionSetup | ~275 bytes | Depth-3, both dirs | | SessionDatagram + SessionSetup | ~275 bytes | Depth-3, both dirs |
| SessionDatagram + CoordsRequired | 37 + 36 + 38 = 111 bytes | Including link overhead | | SessionDatagram + CoordsRequired | 37 + 36 + 38 = 111 bytes | Including link overhead |
+48 -11
View File
@@ -11,7 +11,8 @@ use crate::node::session_wire::{
parse_encrypted_coords, FspCommonPrefix, FspEncryptedHeader, FSP_COMMON_PREFIX_SIZE, parse_encrypted_coords, FspCommonPrefix, FspEncryptedHeader, FSP_COMMON_PREFIX_SIZE,
FSP_FLAG_CP, FSP_HEADER_SIZE, FSP_PHASE_ESTABLISHED, FSP_PHASE_MSG1, FSP_PHASE_MSG2, FSP_FLAG_CP, FSP_HEADER_SIZE, FSP_PHASE_ESTABLISHED, FSP_PHASE_MSG1, FSP_PHASE_MSG2,
}; };
use crate::protocol::encode_coords; use crate::protocol::{coords_wire_size, encode_coords};
use crate::upper::icmp::FIPS_OVERHEAD;
use crate::node::{Node, NodeError}; use crate::node::{Node, NodeError};
use crate::noise::{HandshakeState, HANDSHAKE_MSG1_SIZE, HANDSHAKE_MSG2_SIZE}; use crate::noise::{HandshakeState, HANDSHAKE_MSG1_SIZE, HANDSHAKE_MSG2_SIZE};
use crate::mmp::report::ReceiverReport; use crate::mmp::report::ReceiverReport;
@@ -726,10 +727,7 @@ impl Node {
})?; })?;
// Check warmup counter and get session timestamp // Check warmup counter and get session timestamp
let include_coords = entry.coords_warmup_remaining() > 0; let wants_coords = entry.coords_warmup_remaining() > 0;
if include_coords {
entry.set_coords_warmup_remaining(entry.coords_warmup_remaining() - 1);
}
let timestamp = entry.session_timestamp(now_ms); let timestamp = entry.session_timestamp(now_ms);
let spin_bit = entry.mmp().is_some_and(|m| m.spin_bit.tx_bit()); let spin_bit = entry.mmp().is_some_and(|m| m.spin_bit.tx_bit());
@@ -751,13 +749,54 @@ impl Node {
let inner_flags = FspInnerFlags { spin_bit }.to_byte(); let inner_flags = FspInnerFlags { spin_bit }.to_byte();
let inner_plaintext = fsp_prepend_inner_header(timestamp, msg_type, inner_flags, plaintext); let inner_plaintext = fsp_prepend_inner_header(timestamp, msg_type, inner_flags, plaintext);
// Build FSP flags // Determine whether coords fit within transport MTU.
// Total wire size: FIPS_OVERHEAD (106) + coords_size + plaintext.len()
// The transport MTU is the UDP payload budget available for the FLP frame.
let (include_coords, my_coords, dest_coords) = if wants_coords {
let src = self.tree_state.my_coords().clone();
let dst = self.get_dest_coords(dest_addr);
let coords_size = coords_wire_size(&src) + coords_wire_size(&dst);
let total_wire = FIPS_OVERHEAD as usize + coords_size + plaintext.len();
if total_wire <= self.transport_mtu() as usize {
(true, Some(src), Some(dst))
} else {
trace!("CP flag skipped: {} bytes with coords exceeds transport MTU {}",
total_wire, self.transport_mtu());
(false, None, None)
}
} else {
(false, None, None)
};
// Decrement warmup counter only if we actually include coords
if include_coords
&& let Some(entry) = self.sessions.get_mut(dest_addr)
{
entry.set_coords_warmup_remaining(entry.coords_warmup_remaining() - 1);
}
// Build FSP flags (CP flag only if coords will be included)
let flags = if include_coords { FSP_FLAG_CP } else { 0 }; let flags = if include_coords { FSP_FLAG_CP } else { 0 };
// Build 12-byte FSP header (used as AAD for AEAD) // Build 12-byte FSP header (used as AAD for AEAD)
let payload_len = inner_plaintext.len() as u16; let payload_len = inner_plaintext.len() as u16;
let header = build_fsp_header(counter, flags, payload_len); let header = build_fsp_header(counter, flags, payload_len);
// Re-borrow session for encryption
let entry = self.sessions.get_mut(dest_addr).ok_or_else(|| NodeError::SendFailed {
node_addr: *dest_addr,
reason: "no session".into(),
})?;
let session = match entry.state_mut() {
EndToEndState::Established(s) => s,
_ => {
return Err(NodeError::SendFailed {
node_addr: *dest_addr,
reason: "session not established".into(),
});
}
};
// Encrypt with AAD binding to the FSP header // Encrypt with AAD binding to the FSP header
let ciphertext = session.encrypt_with_aad(&inner_plaintext, &header).map_err(|e| { let ciphertext = session.encrypt_with_aad(&inner_plaintext, &header).map_err(|e| {
NodeError::SendFailed { NodeError::SendFailed {
@@ -769,11 +808,9 @@ impl Node {
// Assemble: header(12) + [coords] + ciphertext // Assemble: header(12) + [coords] + ciphertext
let mut fsp_payload = Vec::with_capacity(FSP_HEADER_SIZE + ciphertext.len() + 200); let mut fsp_payload = Vec::with_capacity(FSP_HEADER_SIZE + ciphertext.len() + 200);
fsp_payload.extend_from_slice(&header); fsp_payload.extend_from_slice(&header);
if include_coords { if let (Some(src), Some(dst)) = (&my_coords, &dest_coords) {
let my_coords = self.tree_state.my_coords().clone(); encode_coords(src, &mut fsp_payload);
let dest_coords = self.get_dest_coords(dest_addr); encode_coords(dst, &mut fsp_payload);
encode_coords(&my_coords, &mut fsp_payload);
encode_coords(&dest_coords, &mut fsp_payload);
} }
fsp_payload.extend_from_slice(&ciphertext); fsp_payload.extend_from_slice(&ciphertext);
+1 -1
View File
@@ -42,7 +42,7 @@ pub use session::{
COORDS_REQUIRED_SIZE, PATH_MTU_NOTIFICATION_SIZE, SESSION_RECEIVER_REPORT_SIZE, COORDS_REQUIRED_SIZE, PATH_MTU_NOTIFICATION_SIZE, SESSION_RECEIVER_REPORT_SIZE,
SESSION_SENDER_REPORT_SIZE, SESSION_SENDER_REPORT_SIZE,
}; };
pub(crate) use session::{decode_optional_coords, encode_coords}; pub(crate) use session::{coords_wire_size, decode_optional_coords, encode_coords};
/// Protocol version for message compatibility. /// Protocol version for message compatibility.
pub const PROTOCOL_VERSION: u8 = 1; pub const PROTOCOL_VERSION: u8 = 1;
+5
View File
@@ -84,6 +84,11 @@ impl fmt::Display for SessionMessageType {
// Coordinate Wire Format Helpers // Coordinate Wire Format Helpers
// ============================================================================ // ============================================================================
/// Wire size of a TreeCoordinate in address-only format: 2 + entries × 16.
pub(crate) fn coords_wire_size(coords: &TreeCoordinate) -> usize {
2 + coords.entries().len() * 16
}
/// Encode a TreeCoordinate as address-only wire format: count(u16 LE) + addrs(16 × n). /// Encode a TreeCoordinate as address-only wire format: count(u16 LE) + addrs(16 × n).
/// ///
/// Session-layer messages serialize coordinates as NodeAddr arrays (16 bytes each), /// Session-layer messages serialize coordinates as NodeAddr arrays (16 bytes each),
+27 -16
View File
@@ -59,24 +59,35 @@ const ICMPV6_HEADER_LEN: usize = 8;
/// Maximum original packet bytes to include in ICMPv6 error. /// Maximum original packet bytes to include in ICMPv6 error.
const MAX_ORIGINAL_PACKET: usize = MIN_IPV6_MTU - IPV6_HEADER_LEN - ICMPV6_HEADER_LEN; const MAX_ORIGINAL_PACKET: usize = MIN_IPV6_MTU - IPV6_HEADER_LEN - ICMPV6_HEADER_LEN;
/// Total FIPS encapsulation overhead (worst case). /// FIPS encapsulation overhead for data packets (no piggybacked coordinates).
/// ///
/// Breakdown: /// This is the fixed overhead for a SessionDatagram carrying an FSP DataPacket,
/// - Noise encryption tag: 16 bytes (FLP link-layer AEAD) /// used to compute `effective_ipv6_mtu()` and TCP MSS clamping. Coordinate
/// - FLP established frame header: 16 bytes (common prefix + receiver_idx + counter) /// piggybacking (CP flag) adds variable overhead on top of this; the send path
/// - FLP inner header: 5 bytes (4-byte timestamp + 1-byte msg_type) /// guards against exceeding the transport MTU when coords are included.
/// - SessionDatagram fields: 35 bytes (ttl + path_mtu + src_addr + dest_addr)
/// - FSP header: 12 bytes (4-byte prefix + 8-byte counter)
/// - FSP inner header: 6 bytes (4-byte timestamp + 1-byte msg_type + 1-byte inner_flags)
/// - Session AEAD tag: 16 bytes (FSP session-layer AEAD)
/// - Coordinates (worst case): ~60 bytes (2 coords with depth 3 each)
/// ///
/// The old 12-byte session header is replaced by FSP header (12) + FSP inner /// Breakdown (traced through the actual send path):
/// header (6) + session AEAD tag (16) = 34 bytes, an increase of 22 bytes. ///
/// However, the old overhead already accounted for the session AEAD tag in /// ```text
/// the "Noise encryption tag" line, and the old header included the counter. /// FLP outer header (cleartext AAD) 16
/// Net change: +6 bytes for FSP inner header. /// common prefix (4) + receiver_idx (4) + counter (8)
pub const FIPS_OVERHEAD: u16 = 16 + 16 + 5 + 35 + 12 + 6 + 60; // 150 bytes /// FLP AEAD ciphertext:
/// timestamp (4) + msg_type (1) 5 [FLP inner header]
/// ttl (1) + path_mtu (2) + src (16) + dst (16) 35 [SessionDatagram body]
/// FSP header (4 prefix + 8 counter) 12 [cleartext AAD]
/// FSP AEAD ciphertext:
/// timestamp (4) + msg_type (1) + flags (1) 6 [FSP inner header]
/// <application data>
/// Poly1305 tag 16 [FSP AEAD]
/// FLP Poly1305 tag 16 [FLP AEAD]
/// ────
/// 106
/// ```
///
/// Note: the FLP inner header msg_type byte IS the SessionDatagram msg_type
/// byte (shared, not double-counted). The "35 bytes" is the SessionDatagram
/// body after msg_type is consumed by the dispatch layer.
pub const FIPS_OVERHEAD: u16 = 16 + 16 + 5 + 35 + 12 + 6 + 16; // 106 bytes
/// Calculate the effective IPv6 MTU for FIPS-encapsulated traffic. /// Calculate the effective IPv6 MTU for FIPS-encapsulated traffic.
/// ///