laantungir/fips
1
0
Fork 0
mirror of https://github.com/jmcorgan/fips.git synced 2026-10-05 11:08:25 +00:00
Code Issues Packages Projects Releases Wiki Activity
Files
1e111fe044ea582bc56451ecb85e92c7a7754acf
fips/testing/static/docker-compose.yml
T
Johnathan Corgan 1e111fe044 fix(gateway): bound live mappings and the new-mapping rate 
Any host that can reach the LAN resolver could ask for one new .fips
name after another, and each got a virtual-IP mapping until the pool's
65,535 addresses ran out. Every mapping also adds to the cost of each
NAT rebuild, each pool tick and shutdown.

The pool now refuses a new name once it holds 1000 live mappings, and
admits new names from a token bucket of 50 that refills at 10 per
second. Both checks sit after the return for a name that already has a
mapping, so names in use keep resolving when new ones are refused. The
ceiling is checked first, so a refusal there takes no token and is
always reported as the ceiling. A token is taken only once an address
has been taken from the free list, so an exhausted pool costs none.
Each limit has its own PoolError variant, and the "Pool allocation
failed" warning names the one that refused. VirtualIpPool::new keeps
its signature and uses the compiled-in limits; with_limits and
allocate_at, which takes the instant that drives the refill, let the
unit tests set small limits and a clock.

The limits come from a flood of the gateway suite's gateway to 500,
1000 and 2000 live mappings with no limits in place. The measurement
record is kept separately. Figures:

- New mappings per second, over 100 creations: 215 for the first 100,
  63 at 500, 47 at 1000, 32 at 2000.
- NAT rebuild, median/max over the 20 adds up to the count: 17.5/26.5
  ms at 500, 22.2/34.3 ms at 1000, 32.2/38.3 ms at 2000.
- Pool tick: 24 to 32 us at 500, 48 to 53 us at 1000, 105 to 115 us at
  2000, beside a conntrack read of 180 to 300 us.
- Shutdown at 2000 mappings: 2.4 s.

These are a best case for router hardware: they come from a container
on a development host, not a router, and that host has no
/proc/net/nf_conntrack, so the tick figures include no conntrack
parsing. The ceiling is half the largest count measured. At 1000 a
rebuild took about 22 ms and shutdown about 1.2 s. The rate is below
the unthrottled creation rate at every count measured, so the bucket
rather than the rebuild sets how fast a flood can fill the pool. From
empty that now takes about 95 s; the unthrottled run passed 1000 in
about 16 s. Ten rebuilds a second at the ceiling cost about a fifth of
the gateway's single runtime thread on that host. The 2000 figures
exist only in the measurement, since the committed phase stops at the
ceiling.

To measure and to keep measuring, the "Added DNAT/SNAT rules" and
"Removed DNAT/SNAT rules" debug lines now carry the mapping count after
the change and the rebuild's duration in microseconds, and are also
emitted, with the error, when a rebuild fails, so a failure at some
count leaves a record of that count. The pool tick logs a debug line
with the mapping count and the durations of the conntrack read and of
the tick. The suite's gateway logs at info because RUST_LOG=info
overrides the entrypoint's --log-level debug, so the gw-gateway service
now enables debug for the NAT manager and the gateway binary only.

The gateway suite's last phase is now the regression. It restarts the
gateway with mappings that outlive the phase and reads the limits from
pool.rs. It fills the pool to the ceiling with the readiness probe's
mapping plus ceiling - 1 new names, retrying rate refusals, then asks
once each for 20 more. It asserts all 20 get SERVFAIL, the live count
equals the ceiling, at least 20 ceiling refusals are logged, the rate
limit refused during the fill and not after it, the probe's name keeps
its address, nothing is reclaimed, no NAT or proxy NDP failure is
logged, and no 10 s window of the fill holds more than burst + 10 x
rate creations. It also reports rebuild durations at 500 and 1000,
ticks as they fall, and the shutdown time at the ceiling. On a tree
with these assertions but no checks in allocate the phase fails five
of them: 1020 mappings, 0 SERVFAIL, no ceiling or rate refusals, and
713 creations in one 10 s window. With the limits it passes in about
140 s, with rebuilds of 17.8/24.5 ms at 500 and 20.9/33.9 ms at 1000
and a 1.2 s shutdown at the ceiling.

The NAT batch phase now also goes through the rate limit, so its
driver retries refusals; it shares the restart, readiness gate and DNS
driver with the new phase instead of carrying its own copies.
2026-09-18 20:32:25 +00:00

458 lines
16 KiB
YAML
Raw Blame History

networks:
# No subnet is requested: docker assigns one from the daemon's address pool,
# which is what lets two concurrent CI runs bring this topology up at the
# same time. A fixed request is honoured verbatim, so two runs asking for the
# same range collide on "Pool overlaps" — the whole reason mesh nodes now
# peer by docker hostname rather than by address.
fips-net:
driver: bridge
labels:
- "com.corganlabs.fips-ci=1"
# IPv4 is dropped so docker auto-assigns it (nothing reads a LAN IPv4 — the
# whole gateway LAN path is IPv6). The fd02::/64 pin stays for the standalone /
# GitHub path; concurrent local runs replace this network with a per-run
# claimed /64 via docker-compose.gateway-external-net.yml (see run_gateway).
gateway-lan:
driver: bridge
labels:
- "com.corganlabs.fips-ci=1"
enable_ipv6: true
ipam:
config:
- subnet: fd02::/64
x-fips-common: &fips-common
image: ${FIPS_TEST_IMAGE:-fips-test:latest}
cap_add:
- NET_ADMIN
devices:
- /dev/net/tun:/dev/net/tun
sysctls:
- net.ipv6.conf.all.disable_ipv6=0
restart: "no"
env_file:
- ./generated-configs${FIPS_CI_NAME_SUFFIX:-}/npubs.env
environment:
- RUST_LOG=info
# Passthrough for A/B benchmarking — set on the host shell before
# `docker compose up` to override the worker-pool defaults.
- FIPS_ENCRYPT_WORKERS=${FIPS_ENCRYPT_WORKERS:-}
- FIPS_DECRYPT_WORKERS=${FIPS_DECRYPT_WORKERS:-}
- FIPS_PERF=${FIPS_PERF:-0}
- FIPS_PERF_INTERVAL_SECS=${FIPS_PERF_INTERVAL_SECS:-5}
services:
# ── Mesh topology ──────────────────────────────────────────────
node-a:
<<: *fips-common
profiles: ["mesh"]
container_name: fips-node-a${FIPS_CI_NAME_SUFFIX:-}
hostname: node-a
volumes:
- ../docker/resolv.conf:/etc/resolv.conf:ro
- ./generated-configs${FIPS_CI_NAME_SUFFIX:-}/mesh/node-a.yaml:/etc/fips/fips.yaml:ro
networks:
fips-net:
node-b:
<<: *fips-common
profiles: ["mesh"]
container_name: fips-node-b${FIPS_CI_NAME_SUFFIX:-}
hostname: node-b
volumes:
- ../docker/resolv.conf:/etc/resolv.conf:ro
- ./generated-configs${FIPS_CI_NAME_SUFFIX:-}/mesh/node-b.yaml:/etc/fips/fips.yaml:ro
networks:
fips-net:
node-c:
<<: *fips-common
profiles: ["mesh"]
container_name: fips-node-c${FIPS_CI_NAME_SUFFIX:-}
hostname: node-c
volumes:
- ../docker/resolv.conf:/etc/resolv.conf:ro
- ./generated-configs${FIPS_CI_NAME_SUFFIX:-}/mesh/node-c.yaml:/etc/fips/fips.yaml:ro
networks:
fips-net:
node-d:
<<: *fips-common
profiles: ["mesh"]
container_name: fips-node-d${FIPS_CI_NAME_SUFFIX:-}
hostname: node-d
volumes:
- ../docker/resolv.conf:/etc/resolv.conf:ro
- ./generated-configs${FIPS_CI_NAME_SUFFIX:-}/mesh/node-d.yaml:/etc/fips/fips.yaml:ro
networks:
fips-net:
node-e:
<<: *fips-common
profiles: ["mesh"]
container_name: fips-node-e${FIPS_CI_NAME_SUFFIX:-}
hostname: node-e
volumes:
- ../docker/resolv.conf:/etc/resolv.conf:ro
- ./generated-configs${FIPS_CI_NAME_SUFFIX:-}/mesh/node-e.yaml:/etc/fips/fips.yaml:ro
networks:
fips-net:
# ── Chain topology (A-B-C-D-E) ────────────────────────────────
chain-a:
<<: *fips-common
profiles: ["chain"]
container_name: fips-node-a${FIPS_CI_NAME_SUFFIX:-}
hostname: node-a
volumes:
- ../docker/resolv.conf:/etc/resolv.conf:ro
- ./generated-configs${FIPS_CI_NAME_SUFFIX:-}/chain/node-a.yaml:/etc/fips/fips.yaml:ro
networks:
fips-net:
chain-b:
<<: *fips-common
profiles: ["chain"]
container_name: fips-node-b${FIPS_CI_NAME_SUFFIX:-}
hostname: node-b
volumes:
- ../docker/resolv.conf:/etc/resolv.conf:ro
- ./generated-configs${FIPS_CI_NAME_SUFFIX:-}/chain/node-b.yaml:/etc/fips/fips.yaml:ro
networks:
fips-net:
chain-c:
<<: *fips-common
profiles: ["chain"]
container_name: fips-node-c${FIPS_CI_NAME_SUFFIX:-}
hostname: node-c
volumes:
- ../docker/resolv.conf:/etc/resolv.conf:ro
- ./generated-configs${FIPS_CI_NAME_SUFFIX:-}/chain/node-c.yaml:/etc/fips/fips.yaml:ro
networks:
fips-net:
chain-d:
<<: *fips-common
profiles: ["chain"]
container_name: fips-node-d${FIPS_CI_NAME_SUFFIX:-}
hostname: node-d
volumes:
- ../docker/resolv.conf:/etc/resolv.conf:ro
- ./generated-configs${FIPS_CI_NAME_SUFFIX:-}/chain/node-d.yaml:/etc/fips/fips.yaml:ro
networks:
fips-net:
chain-e:
<<: *fips-common
profiles: ["chain"]
container_name: fips-node-e${FIPS_CI_NAME_SUFFIX:-}
hostname: node-e
volumes:
- ../docker/resolv.conf:/etc/resolv.conf:ro
- ./generated-configs${FIPS_CI_NAME_SUFFIX:-}/chain/node-e.yaml:/etc/fips/fips.yaml:ro
networks:
fips-net:
# ── Rekey integration test (mesh + aggressive rekey timers) ──
rekey-a:
<<: *fips-common
profiles: ["rekey"]
container_name: fips-node-a${FIPS_CI_NAME_SUFFIX:-}
hostname: node-a
environment:
- RUST_LOG=info,fips::node::handlers::rekey=debug
volumes:
- ../docker/resolv.conf:/etc/resolv.conf:ro
- ./generated-configs${FIPS_CI_NAME_SUFFIX:-}/rekey/node-a.yaml:/etc/fips/fips.yaml:ro
networks:
fips-net:
rekey-b:
<<: *fips-common
profiles: ["rekey"]
container_name: fips-node-b${FIPS_CI_NAME_SUFFIX:-}
hostname: node-b
environment:
- RUST_LOG=info,fips::node::handlers::rekey=debug
volumes:
- ../docker/resolv.conf:/etc/resolv.conf:ro
- ./generated-configs${FIPS_CI_NAME_SUFFIX:-}/rekey/node-b.yaml:/etc/fips/fips.yaml:ro
networks:
fips-net:
rekey-c:
<<: *fips-common
profiles: ["rekey"]
container_name: fips-node-c${FIPS_CI_NAME_SUFFIX:-}
hostname: node-c
environment:
- RUST_LOG=info,fips::node::handlers::rekey=debug
volumes:
- ../docker/resolv.conf:/etc/resolv.conf:ro
- ./generated-configs${FIPS_CI_NAME_SUFFIX:-}/rekey/node-c.yaml:/etc/fips/fips.yaml:ro
networks:
fips-net:
rekey-d:
<<: *fips-common
profiles: ["rekey"]
container_name: fips-node-d${FIPS_CI_NAME_SUFFIX:-}
hostname: node-d
environment:
- RUST_LOG=info,fips::node::handlers::rekey=debug
volumes:
- ../docker/resolv.conf:/etc/resolv.conf:ro
- ./generated-configs${FIPS_CI_NAME_SUFFIX:-}/rekey/node-d.yaml:/etc/fips/fips.yaml:ro
networks:
fips-net:
rekey-e:
<<: *fips-common
profiles: ["rekey"]
container_name: fips-node-e${FIPS_CI_NAME_SUFFIX:-}
hostname: node-e
environment:
- RUST_LOG=info,fips::node::handlers::rekey=debug
volumes:
- ../docker/resolv.conf:/etc/resolv.conf:ro
- ./generated-configs${FIPS_CI_NAME_SUFFIX:-}/rekey/node-e.yaml:/etc/fips/fips.yaml:ro
networks:
fips-net:
# ── Rekey + accept_connections=false on node b ──────────────────
# Exercises the auto_connect-initiator-with-accept-off regression
# class. Same 5-node mesh, but node b's generated config has
# `transports.udp.accept_connections: false` (injected by
# rekey-test.sh's inject-config when REKEY_ACCEPT_OFF_NODES=b).
rekey-accept-off-a:
<<: *fips-common
profiles: ["rekey-accept-off"]
container_name: fips-node-a${FIPS_CI_NAME_SUFFIX:-}
hostname: node-a
environment:
- RUST_LOG=info,fips::node::handlers::rekey=debug,fips::node::handlers::handshake=debug
volumes:
- ../docker/resolv.conf:/etc/resolv.conf:ro
- ./generated-configs${FIPS_CI_NAME_SUFFIX:-}/rekey-accept-off/node-a.yaml:/etc/fips/fips.yaml:ro
networks:
fips-net:
rekey-accept-off-b:
<<: *fips-common
profiles: ["rekey-accept-off"]
container_name: fips-node-b${FIPS_CI_NAME_SUFFIX:-}
hostname: node-b
environment:
- RUST_LOG=info,fips::node::handlers::rekey=debug,fips::node::handlers::handshake=debug
volumes:
- ../docker/resolv.conf:/etc/resolv.conf:ro
- ./generated-configs${FIPS_CI_NAME_SUFFIX:-}/rekey-accept-off/node-b.yaml:/etc/fips/fips.yaml:ro
networks:
fips-net:
rekey-accept-off-c:
<<: *fips-common
profiles: ["rekey-accept-off"]
container_name: fips-node-c${FIPS_CI_NAME_SUFFIX:-}
hostname: node-c
environment:
- RUST_LOG=info,fips::node::handlers::rekey=debug,fips::node::handlers::handshake=debug
volumes:
- ../docker/resolv.conf:/etc/resolv.conf:ro
- ./generated-configs${FIPS_CI_NAME_SUFFIX:-}/rekey-accept-off/node-c.yaml:/etc/fips/fips.yaml:ro
networks:
fips-net:
rekey-accept-off-d:
<<: *fips-common
profiles: ["rekey-accept-off"]
container_name: fips-node-d${FIPS_CI_NAME_SUFFIX:-}
hostname: node-d
environment:
- RUST_LOG=info,fips::node::handlers::rekey=debug,fips::node::handlers::handshake=debug
volumes:
- ../docker/resolv.conf:/etc/resolv.conf:ro
- ./generated-configs${FIPS_CI_NAME_SUFFIX:-}/rekey-accept-off/node-d.yaml:/etc/fips/fips.yaml:ro
networks:
fips-net:
rekey-accept-off-e:
<<: *fips-common
profiles: ["rekey-accept-off"]
container_name: fips-node-e${FIPS_CI_NAME_SUFFIX:-}
hostname: node-e
environment:
- RUST_LOG=info,fips::node::handlers::rekey=debug,fips::node::handlers::handshake=debug
volumes:
- ../docker/resolv.conf:/etc/resolv.conf:ro
- ./generated-configs${FIPS_CI_NAME_SUFFIX:-}/rekey-accept-off/node-e.yaml:/etc/fips/fips.yaml:ro
networks:
fips-net:
# ── Rekey topology with udp.outbound_only=true on node b ────
# Exercises the udp.outbound_only rekey-msg1 admission regression
# observed in production 2026-04-30. Same 5-node mesh as
# rekey-accept-off, but node-b's generated config has
# `transports.udp.outbound_only: true` (forces ephemeral bind +
# accept_connections false) AND node-b's peer-c address is
# rewritten to the docker hostname (`node-c:2121`) so the
# `addr_to_link` lookup misses on the inbound numeric source addr.
# Injected by rekey-test.sh's inject-config when
# REKEY_OUTBOUND_ONLY_NODES=b.
rekey-outbound-only-a:
<<: *fips-common
profiles: ["rekey-outbound-only"]
container_name: fips-node-a${FIPS_CI_NAME_SUFFIX:-}
hostname: node-a
environment:
- RUST_LOG=info,fips::node::handlers::rekey=debug,fips::node::handlers::handshake=debug
volumes:
- ../docker/resolv.conf:/etc/resolv.conf:ro
- ./generated-configs${FIPS_CI_NAME_SUFFIX:-}/rekey-outbound-only/node-a.yaml:/etc/fips/fips.yaml:ro
networks:
fips-net:
rekey-outbound-only-b:
<<: *fips-common
profiles: ["rekey-outbound-only"]
container_name: fips-node-b${FIPS_CI_NAME_SUFFIX:-}
hostname: node-b
environment:
- RUST_LOG=info,fips::node::handlers::rekey=debug,fips::node::handlers::handshake=debug
volumes:
- ../docker/resolv.conf:/etc/resolv.conf:ro
- ./generated-configs${FIPS_CI_NAME_SUFFIX:-}/rekey-outbound-only/node-b.yaml:/etc/fips/fips.yaml:ro
networks:
fips-net:
rekey-outbound-only-c:
<<: *fips-common
profiles: ["rekey-outbound-only"]
container_name: fips-node-c${FIPS_CI_NAME_SUFFIX:-}
hostname: node-c
environment:
- RUST_LOG=info,fips::node::handlers::rekey=debug,fips::node::handlers::handshake=debug
volumes:
- ../docker/resolv.conf:/etc/resolv.conf:ro
- ./generated-configs${FIPS_CI_NAME_SUFFIX:-}/rekey-outbound-only/node-c.yaml:/etc/fips/fips.yaml:ro
networks:
fips-net:
rekey-outbound-only-d:
<<: *fips-common
profiles: ["rekey-outbound-only"]
container_name: fips-node-d${FIPS_CI_NAME_SUFFIX:-}
hostname: node-d
environment:
- RUST_LOG=info,fips::node::handlers::rekey=debug,fips::node::handlers::handshake=debug
volumes:
- ../docker/resolv.conf:/etc/resolv.conf:ro
- ./generated-configs${FIPS_CI_NAME_SUFFIX:-}/rekey-outbound-only/node-d.yaml:/etc/fips/fips.yaml:ro
networks:
fips-net:
rekey-outbound-only-e:
<<: *fips-common
profiles: ["rekey-outbound-only"]
container_name: fips-node-e${FIPS_CI_NAME_SUFFIX:-}
hostname: node-e
environment:
- RUST_LOG=info,fips::node::handlers::rekey=debug,fips::node::handlers::handshake=debug
volumes:
- ../docker/resolv.conf:/etc/resolv.conf:ro
- ./generated-configs${FIPS_CI_NAME_SUFFIX:-}/rekey-outbound-only/node-e.yaml:/etc/fips/fips.yaml:ro
networks:
fips-net:
# ── Gateway integration test (gateway + server + non-FIPS client) ─
gw-gateway:
<<: *fips-common
profiles: ["gateway"]
container_name: fips-gw-gateway${FIPS_CI_NAME_SUFFIX:-}
hostname: gw-gateway
# Privileged required: gateway must enable IPv6 on eth1 (second network,
# attached after container start) and manage nftables NAT rules.
privileged: true
environment:
# Debug for the NAT rebuild and pool tick timing lines only; the
# gateway's --log-level is ignored while RUST_LOG is set.
- RUST_LOG=info,fips::gateway::nat=debug,fips_gateway=debug
- FIPS_TEST_MODE=gateway
sysctls:
- net.ipv6.conf.all.disable_ipv6=0
- net.ipv6.conf.default.disable_ipv6=0
- net.ipv6.conf.all.forwarding=1
- net.ipv6.conf.all.proxy_ndp=1
volumes:
- ../docker/resolv.conf:/etc/resolv.conf:ro
- ./generated-configs${FIPS_CI_NAME_SUFFIX:-}/gateway/node-a.yaml:/etc/fips/fips.yaml:ro
networks:
fips-net:
gateway-lan:
ipv6_address: ${FIPS_GW_LAN6_PREFIX:-fd02}::10
gw-server:
<<: *fips-common
profiles: ["gateway"]
container_name: fips-gw-server${FIPS_CI_NAME_SUFFIX:-}
hostname: gw-server
volumes:
- ../docker/resolv.conf:/etc/resolv.conf:ro
- ./generated-configs${FIPS_CI_NAME_SUFFIX:-}/gateway/node-b.yaml:/etc/fips/fips.yaml:ro
networks:
fips-net:
# Second mesh destination — gives gw-client-2 a distinct npub to target
# so the gateway allocates a separate virtual-IP mapping per LAN client.
# Mirrors gw-server; not on gateway-lan.
gw-server-2:
<<: *fips-common
profiles: ["gateway"]
container_name: fips-gw-server-2${FIPS_CI_NAME_SUFFIX:-}
hostname: gw-server-2
volumes:
- ../docker/resolv.conf:/etc/resolv.conf:ro
- ./generated-configs${FIPS_CI_NAME_SUFFIX:-}/gateway/node-c.yaml:/etc/fips/fips.yaml:ro
networks:
fips-net:
gw-client:
image: ${FIPS_TEST_APP_IMAGE:-fips-test-app:latest}
profiles: ["gateway"]
container_name: fips-gw-client${FIPS_CI_NAME_SUFFIX:-}
hostname: gw-client
cap_add:
- NET_ADMIN
sysctls:
- net.ipv6.conf.all.disable_ipv6=0
volumes:
- ./generated-configs${FIPS_CI_NAME_SUFFIX:-}/gateway/resolv.conf:/etc/resolv.conf:ro
networks:
gateway-lan:
ipv6_address: ${FIPS_GW_LAN6_PREFIX:-fd02}::20
restart: "no"
env_file:
- ./generated-configs${FIPS_CI_NAME_SUFFIX:-}/npubs.env
# Second LAN client — exercises concurrent multi-client mappings.
# Same image and gateway-lan attachment as
# gw-client; the gateway must allocate a distinct virtual IP for it.
gw-client-2:
image: ${FIPS_TEST_APP_IMAGE:-fips-test-app:latest}
profiles: ["gateway"]
container_name: fips-gw-client-2${FIPS_CI_NAME_SUFFIX:-}
hostname: gw-client-2
cap_add:
- NET_ADMIN
sysctls:
- net.ipv6.conf.all.disable_ipv6=0
volumes:
- ./generated-configs${FIPS_CI_NAME_SUFFIX:-}/gateway/resolv.conf:/etc/resolv.conf:ro
networks:
gateway-lan:
ipv6_address: ${FIPS_GW_LAN6_PREFIX:-fd02}::21
restart: "no"
env_file:
- ./generated-configs${FIPS_CI_NAME_SUFFIX:-}/npubs.env
Reference in New Issue View Git Blame Copy Permalink