mirror of
https://github.com/jmcorgan/fips.git
synced 2026-07-30 19:46:15 +00:00
Martti MalmiandJohnathan Corgan
0a5c367edc
Moves both AEAD layers (ChaCha20-Poly1305, one round per layer per packet) plus the sendmsg syscall off the rx_loop task onto a per-shard worker pool, adds per-peer connect(2)-ed UDP with SO_REUSEPORT, and uses Linux UDP GSO (sendmsg+UDP_SEGMENT — kernel splits one super-skb into N on-the-wire datagrams in a single TX-stack walk) when packets in a batch are uniform-size. Same kernel primitive WireGuard's in-kernel module and BoringTun use to hit 2.5–3.2 Gbps single-stream. Single TCP stream on a 5-node docker-bridge mesh, 5 x 15 s x P=1: A→D: 1379 → 2708 Mbps (1.96x, RTT +0.12 ms) A→E: 1394 → 2663 Mbps (1.91x, RTT +0.11 ms) E→A: 1406 → 2624 Mbps (1.87x, RTT +0.19 ms) Static-peer pairs only — every CoV under 3%, 0 outliers, 0% ICMP loss. The ~+100 µs RTT is the worker queue handoff cost; AEAD + sendmmsg now run on a separate core in exchange. What lands: - src/node/encrypt_worker.rs: std::thread + crossbeam_channel workers; hash-by-destination dispatch pins a TCP flow to one worker so wire ordering is preserved; per-worker sendmmsg(2) batching up to 32; Linux uses sendmsg(2)+UDP_SEGMENT when packets in a group are uniform-size. - src/node/decrypt_worker.rs: receive-side mirror. Each shard owns its session's recv cipher + replay window in a thread-local HashMap (no shared RwLock/Mutex). Sessions are handed off at promote_connection and re-registered on K-bit flip / rekey cutover. - src/node/handlers/session.rs try_send_session_data_pipelined: FSP+FMP both seal in-place in the worker on one wire-buffer alloc; no intermediate inner_plaintext / fsp_payload Vecs. - src/transport/udp/connected_peer.rs + peer_drain.rs: per-peer connect(2)-ed UDP socket with SO_REUSEPORT (set on the listen socket too — without that, EADDRINUSE on activation and every packet falls back to the wildcard path); the worker sends with msg_name=NULL and the kernel uses its cached 5-tuple. Tick- driven activation in handlers/connected_udp.rs, idempotent. - src/transport/udp/mod.rs: mem::replace the recvmmsg backing buffer instead of buf.to_vec() per packet — single pointer swap, no MTU-sized memcpy. - src/protocol/link.rs SessionDatagramRef: zero-copy borrowed view used by handle_session_datagram for the bulk local-delivery path; handle_session_payload takes the borrowed payload directly (no payload[35..].to_vec()). - src/transport/mod.rs TransportAddr::from_socket_addr: collapses the two-alloc from_string(addr.to_string()) pattern to one. - src/node/handlers/rx_loop.rs: decrypt-fallback drain promoted ahead of packet_rx in the select! (TCP ACK starvation fix); interleaved fallback drain every 32 packets inside the rx burst loop. - noise::Session: send_cipher_clone / recv_cipher_clone / recv_replay_snapshot_owned / take_send_counter / accept_replay so off-task workers can hold a cloned cipher + reserved counter while the dispatcher keeps replay/counter sequencing serial. CipherState::cipher_clone returns a refcount-bumped LessSafeKey. AsyncUdpSocket: AsRawFd so workers issue raw sendmmsg / sendmsg without going through the tokio reactor. - Worker pool sizing: both default to num_cpus, overridable via FIPS_ENCRYPT_WORKERS=N / FIPS_DECRYPT_WORKERS=N. Per-peer connected UDP can be disabled via FIPS_CONNECTED_UDP=0. - src/perf_profile.rs: optional per-stage timing reporter under FIPS_PERF=1 (or FIPS_PIPELINE_TRACE=1). Off by default; zero overhead when disabled. - All cfg(unix)-gated. Windows continues on the existing tokio- based send/recv. Decrypt worker session lifecycle: - Node::unregister_decrypt_worker_session mirrors the existing register helper. Wired at the two natural sites that already iterate peers_by_index: the rekey drain-completion block in handlers/rekey.rs (drops the worker entry for the old our_index once the drain window has expired and the cache_key is unreachable to any in-flight OLD-K packet), and remove_active_peer in handlers/dispatch.rs (drops the worker entry for each of the four index slots: current, rekey, pending, previous). Only our_index is normally registered; unregister_session is fire- and-forget for missing entries, so calling unconditionally on all four slots is correct and bounds the cleanup without per- slot accounting. Without these callers the per-worker sessions HashMap and the Node's decrypt_registered_sessions set would grow monotonically per rekey on long-lived peers. Testing: - testing/static/scripts/bench-multirun.sh: multi-run iperf3 + ping bench. N reruns (default 5), median / min / max / CoV % / per-run outlier flag, avg ping RTT, ICMP loss %, TCP retransmit total. Plain client→dest labels + topology header. Pre-bench peer-convergence check (FIPS_BENCH_CONVERGE_SECS, default 15); per-path route verification via stats.bytes_sent deltas — fails fast if traffic exits via a non-static-peer link. - testing/static/docker-compose.yml: passes FIPS_ENCRYPT_WORKERS / FIPS_DECRYPT_WORKERS / FIPS_PERF through to containers for A/B benchmarking without rebuilds. - testing/static/scripts/iperf-test.sh: same plain client→dest labels + topology header (was multihop/direct/N hop, which conflated topology distance with on-wire path). - .config/nextest.toml: synthetic UDP node tests serialized through a max-threads=1 test group. Localhost handshakes drop on shared CI runners under parallel load; one-at-a-time keeps assertions reliable. - src/node/tests/spanning_tree.rs: repair_missing_edge_handshakes — retries up to 5 times for synthetic edges whose msg1 was dropped, with a drain after each edge retry instead of after each attempt's full burst. - src/node/decrypt_worker.rs::tests: two unit tests asserting WorkerMsg::UnregisterSession removes the worker-thread session HashMap entry (handle_msg_unregister_session_removes_entry) and is a no-op for never-seen cache_keys (handle_msg_unregister_session_idempotent_on_unknown_key), which is the safety invariant the unconditional unregister calls at the four index slots in remove_active_peer rely on. - src/node/encrypt_worker.rs::unix_tests pipelined_send_wire_layout_roundtrips_canonical_decoders: mirrors the encoder geometry of try_send_session_data_pipelined (no coords, the common established-session path), runs the worker's real seal + send via flush_direct_batch_sync, and decodes the resulting wire packet using only canonical receive-side decoders (EncryptedHeader::parse, SessionDatagramRef::decode, FSP header parse, noise::open). Any divergence between the hand-rolled encoder offsets (fsp_aad_offset, fsp_plaintext_offset) and the decoders fails at one of the parse / open / decode steps before the inner-plaintext assertion fires. Complements the existing fsp_preseal_runs_before_outer_fmp_seal test which covers the seal-ordering invariant with synthetic headers but does not exercise the wire-layout invariant. CHANGELOG.md [Unreleased] # Changed entry added describing the worker-pool threading model, hash-by-destination dispatch, sendmmsg/UDP_GSO, per-peer connected UDP, the operator-facing env vars, and the bench numbers above. Cherry-picks from mmalmi/master (paths translated from crates/fips-core/src/ to src/): 9b7c723, 0deb5cb, 13f7339, e036c0e, 3740a68, 3792f83, 8510193, 4910b07, e53f545, e4e2896, 5fe4af5, 1d01ada, 8c37008, e12469e, 6eb2860. Co-authored-by: Johnathan Corgan <johnathan@corganlabs.com>
106 lines
3.8 KiB
TOML
106 lines
3.8 KiB
TOML
[package]
|
|
name = "fips"
|
|
version = "0.4.0-dev"
|
|
edition = "2024"
|
|
description = "A distributed, decentralized network routing protocol for mesh nodes connecting over arbitrary transports"
|
|
license = "MIT"
|
|
authors = ["Johnathan Corgan <jcorgan@corganlabs.com>"]
|
|
repository = "https://github.com/jmcorgan/fips"
|
|
readme = "README.md"
|
|
|
|
[dependencies]
|
|
ratatui = "0.30"
|
|
secp256k1 = { version = "0.30", features = ["rand", "global-context"] }
|
|
sha2 = "0.10"
|
|
hkdf = "0.12"
|
|
ring = "0.17"
|
|
rand = "0.10.1"
|
|
crossbeam-channel = "0.5"
|
|
thiserror = "2.0"
|
|
bech32 = "0.11"
|
|
serde = { version = "1.0", features = ["derive"] }
|
|
serde_json = "1.0"
|
|
serde_yaml = "0.9"
|
|
dirs = "6.0"
|
|
hex = "0.4"
|
|
clap = { version = "4.6", features = ["derive"] }
|
|
tracing = "0.1"
|
|
tracing-subscriber = { version = "0.3", features = ["env-filter"] }
|
|
tokio = { version = "1", features = ["rt", "macros", "signal", "sync", "net", "time", "process", "io-util"] }
|
|
futures = "0.3"
|
|
simple-dns = "0.11.2"
|
|
socket2 = { version = "0.6.2", features = ["all"] }
|
|
tokio-socks = "0.5"
|
|
portable-atomic = { version = "1", features = ["std"] }
|
|
|
|
nostr = { version = "0.44", features = ["std", "nip59"] }
|
|
nostr-sdk = "0.44"
|
|
|
|
[target.'cfg(unix)'.dependencies]
|
|
tun = { version = "0.8.7", features = ["async"] }
|
|
libc = "0.2"
|
|
|
|
[target.'cfg(target_os = "linux")'.dependencies]
|
|
rtnetlink = "0.21.0"
|
|
rustables = "0.8.7"
|
|
procfs = { version = "0.18", default-features = false }
|
|
|
|
# bluer/BlueZ needs glibc — see build.rs `bluer_available` cfg gate.
|
|
[target.'cfg(all(target_os = "linux", not(target_env = "musl")))'.dependencies]
|
|
bluer = { version = "0.17", features = ["bluetoothd", "l2cap"] }
|
|
|
|
[target.'cfg(windows)'.dependencies]
|
|
wintun = "0.5"
|
|
windows-service = "0.8.1"
|
|
|
|
[package.metadata.deb]
|
|
maintainer = "Johnathan Corgan <jcorgan@corganlabs.com>"
|
|
copyright = "2026 Johnathan Corgan"
|
|
license-file = ["LICENSE", "0"]
|
|
section = "net"
|
|
priority = "optional"
|
|
depends = "libc6, systemd, libdbus-1-3"
|
|
recommends = "bluez"
|
|
extended-description = """\
|
|
FIPS is a distributed, decentralized network routing protocol for mesh \
|
|
nodes connecting over arbitrary transports including UDP, TCP, Ethernet, \
|
|
Tor, and Bluetooth (BLE). It provides encrypted peer-to-peer connectivity \
|
|
with automatic key management, TUN-based virtual networking, and .fips DNS \
|
|
resolution."""
|
|
maintainer-scripts = "packaging/debian/"
|
|
assets = [
|
|
["target/release/fips", "/usr/bin/", "755"],
|
|
["target/release/fipsctl", "/usr/bin/", "755"],
|
|
["target/release/fipstop", "/usr/bin/", "755"],
|
|
["packaging/common/fips.yaml", "/etc/fips/fips.yaml", "600"],
|
|
["packaging/common/hosts", "/etc/fips/hosts", "644"],
|
|
["packaging/common/fips.nft", "/etc/fips/fips.nft", "644"],
|
|
["packaging/debian/fips.service", "/lib/systemd/system/fips.service", "644"],
|
|
["packaging/debian/fips-dns.service", "/lib/systemd/system/fips-dns.service", "644"],
|
|
["packaging/debian/fips-firewall.service", "/lib/systemd/system/fips-firewall.service", "644"],
|
|
["packaging/common/fips-dns-setup", "/usr/lib/fips/fips-dns-setup", "755"],
|
|
["packaging/common/fips-dns-teardown", "/usr/lib/fips/fips-dns-teardown", "755"],
|
|
["packaging/debian/fips.tmpfiles", "/usr/lib/tmpfiles.d/fips.conf", "644"],
|
|
["target/release/fips-gateway", "/usr/bin/", "755"],
|
|
["packaging/debian/fips-gateway.service", "/lib/systemd/system/fips-gateway.service", "644"],
|
|
["docs/design/fips-security.md", "/usr/share/doc/fips/fips-security.md", "644"],
|
|
]
|
|
conf-files = ["/etc/fips/fips.yaml", "/etc/fips/hosts", "/etc/fips/fips.nft"]
|
|
|
|
[dev-dependencies]
|
|
tempfile = "3.15"
|
|
criterion = { version = "0.8.2", features = ["html_reports"] }
|
|
tokio = { version = "1", features = ["test-util"] }
|
|
|
|
[[bin]]
|
|
name = "fipsctl"
|
|
path = "src/bin/fipsctl.rs"
|
|
|
|
[[bin]]
|
|
name = "fips-gateway"
|
|
path = "src/bin/fips-gateway.rs"
|
|
|
|
[[bin]]
|
|
name = "fipstop"
|
|
path = "src/bin/fipstop/main.rs"
|