fips/docs/how-to/troubleshoot-gateway.md

Troubleshoot fips-gateway

Diagnostic recipes for fips-gateway, grouped by which half of the gateway is failing. For gateway design and deployment, see ../design/fips-gateway.md and deploy-gateway.md. For OpenWrt-specific deployment problems, see the OpenWrt deployment tutorial; most of the recipes below apply on OpenWrt as well, but paths and service names differ.

Inspect gateway state via the control socket

Before digging into nftables or conntrack, ask the gateway directly whether it has the mapping or session you expect. fips-gateway exposes a separate control socket (/run/fips/gateway.sock) with its own command set; there is no fipsctl gateway subcommand — talk to the socket directly with nc -U. Each request is a single line of JSON terminated with a newline; the connection is closed after one response.

Pool summary, listen address, NAT counters, uptime, and the loaded config snapshot:

echo '{"command":"show_gateway"}' | sudo nc -U /run/fips/gateway.sock

Per-mapping virtual-IP state (allocated, active, draining):

echo '{"command":"show_mappings"}' | sudo nc -U /run/fips/gateway.sock

If either command returns gateway not yet initialized, the gateway is still in early startup; wait a moment and retry. If a mapping you expect is not in the list, the DNS path didn't allocate it — fall through to the outbound DNS recipes below. If the mapping exists in state: Active but mesh traffic still fails, the problem is downstream of the allocation (firewall, route, masquerade); see the recipes that follow.

For the full command catalog and JSON shapes, see ../reference/control-socket.md#gateway-command-catalog.

Common (either-half) issues

These break both halves at once because they affect the gateway process itself or the shared NAT machinery.

"No gateway section in configuration"

fips-gateway is normally launched by the systemd unit shipped with the package:

sudo systemctl restart fips-gateway
sudo journalctl -u fips-gateway -e

The unit reads the standard FIPS config search paths (typically /etc/fips/fips.yaml). If the unit logs "no gateway section in configuration" or "Gateway section exists but is not enabled", confirm the section is present and enabled: true:

grep -A1 '^gateway:' /etc/fips/fips.yaml

For one-off debugging outside systemd, run the binary directly and point it at a specific config file:

sudo fips-gateway --config /etc/fips/fips.yaml --log-level debug

This is useful to capture stderr in a terminal, but the systemd unit is the supported entry point in production. See ../reference/cli-fips-gateway.md for the full flag list.

Port conflict on the DNS listen port

Symptom: the gateway exits at startup, before it creates the address pool or the NAT table, and the log carries an error such as this one, wrapped here for reading:

cannot bind the gateway DNS listener on [::]:53: Address already in
use (os error 98); another DNS server holds port 53: dnsmasq,
systemd-resolved's stub listener, unbound or BIND; find the holder
with `ss -ulpn 'sport = :53'` or `netstat -ulnp`, or set
gateway.dns.listen to a free port and point the resolver that
forwards .fips at it

Under systemd the unit restarts every five seconds and fails the same way each time; under procd on OpenWrt the service stops respawning after five failures within an hour. The gateway also exits, after removing its NAT table and routes, when the DNS resolver stops while the gateway is running; that log line reads "Gateway DNS resolver stopped; exiting so the service manager restarts the gateway".

The error names the service most likely to hold the port:

• 53: another DNS server, such as dnsmasq, systemd-resolved's stub listener, unbound or BIND.
• 5353: mDNS. The fips daemon's LAN rendezvous (node.rendezvous.lan), avahi-daemon or systemd-resolved's MulticastDNS.
• 5354: the fips daemon's own DNS responder. gateway.dns.listen must not be the daemon's DNS port.
• 5355: LLMNR, held by systemd-resolved unless LLMNR=no.
• Any other port: another process.

Find the actual holder, replacing the port with your own:

sudo ss -ulpn 'sport = :53'
# OpenWrt ships netstat but not ss:
netstat -ulnp

The default listen address, [::1]:5353, is loopback-only on an unprivileged port. Two options:

• Move the gateway. Set gateway.dns.listen to a free port and point the resolver that forwards .fips at the same port. With the loopback default, configure the existing resolver to forward .fips queries to [::1]:5353 (the canonical OpenWrt deployment works this way out of the box).

• Relocate the conflicting resolver. Move it to a different port (or disable it if not needed) and let the gateway bind the port. Practical for systemd-resolved (set DNSStubListener=no in /etc/systemd/resolved.conf); rarely worth it for production resolvers.

IPv6 forwarding disabled

Symptom: gateway exits at startup with "IPv6 forwarding is disabled. Enable with: sysctl -w net.ipv6.conf.all.forwarding=1".

The gateway is completely non-functional without forwarding — packets cannot traverse the NAT pipeline. Enable it:

sudo sysctl -w net.ipv6.conf.all.forwarding=1

Persist via the drop-in shown in deploy-gateway.md. The same section lists proxy_ndp, which is also required for the outbound half.

nftables table missing or not loaded

Symptom: show_gateway reports an active gateway but nft list table inet fips_gateway errors with "No such file or directory".

The table is created by the gateway at startup and rebuilt atomically on every mapping change and on every set_port_forwards call. If the table is missing while the gateway claims to be running, something else (a host firewall script, a nft flush ruleset from another service) deleted it after creation. Restart the gateway to recreate it:

sudo systemctl restart fips-gateway

If a peer service is repeatedly clobbering the table, switch that service to use add table / flush table <name> for its own table rather than flush ruleset, which destroys every table on the host.

Control socket permission errors

Symptom: nc -U /run/fips/gateway.sock fails with "Permission denied" or "No such file or directory".

The socket is owned by root with mode 0770 (group fips). Either run nc as root (sudo nc -U ...) or add your user to the fips group and re-login. If the file does not exist at all, the gateway either failed to start (check journalctl -u fips-gateway) or failed to bind the socket and continued without it (the warning Failed to bind gateway control socket — continuing without it is in the journal in that case).

Session pinning is off

The gateway keeps a mapping while conntrack shows sessions to its virtual IP. At startup it reads conntrack once, the same way each 10 s tick does, and logs which source answered:

• Conntrack source: proc; session pinning is on: sessions are read from /proc/net/nf_conntrack.
• Conntrack source: netlink; session pinning is on: the proc file is absent, and sessions are read by dumping the conntrack table over netlink, as conntrack -L does. The dump needs CAP_NET_ADMIN.
• No conntrack source is readable; session pinning is off: no source could be read, and the line carries the error from each source it tried. Every mapping then reads zero sessions, so a mapping is reclaimed on its TTL and grace period alone, even while a client that has not re-queried DNS still has traffic flowing through it.

The proc file exists only on a kernel built with CONFIG_NF_CONNTRACK_PROCFS, and only once nf_conntrack is loaded. Without it the gateway uses the netlink dump:

ls /proc/net/nf_conntrack
grep NF_CONNTRACK_PROCFS /boot/config-$(uname -r)

Outbound-half diagnostics

Symptoms in this section all involve a LAN client trying to reach a mesh destination through the gateway and failing.

DNS queries fail

Symptom: LAN clients get SERVFAIL or no response when querying .fips names; or the gateway log shows DNS upstream timeouts.

Step 1. Verify the daemon resolver is running and reachable from the gateway host:

dig @::1 -p 5354 hostname.fips AAAA

If this returns no answer or fails, the FIPS daemon's DNS resolver is not running or not enabled. Check that the daemon config has dns.enabled: true (the default) and the daemon is healthy: fipsctl show status.

Step 2. Verify the gateway is listening on its DNS port:

sudo ss -tulnp | grep -E ':(53|5353)\b'

If nothing is listening on the configured dns.listen address, the gateway either failed to start or is bound to a different address. Check the gateway log: sudo journalctl -u fips-gateway -e.

Step 3. Verify the LAN client can reach the gateway's DNS port:

# from the LAN client
dig @<gateway-lan-addr> hostname.fips AAAA

If this hangs, the LAN-side firewall is blocking DNS, or the LAN route to the gateway is missing.

Ping works but TCP does not

Symptom: ping6 <virtual-ip> succeeds from a LAN client, but TCP connections (SSH, HTTP) hang or time out.

This usually means the fips0-side masquerade rule is missing or misconfigured. Inspect the gateway's nftables table:

sudo nft list table inet fips_gateway

In the postrouting chain, look for a rule matching oifname "fips0" with a masquerade verdict. Without masquerade, the destination mesh node sees a source address (from the virtual pool) it cannot route replies to, and return packets are black-holed.

If the rule is missing, restart the gateway — the table is rebuilt atomically on every mapping change and on startup.

Connection timeout to a virtual IP

Symptom: any traffic to a virtual pool address times out, including ping.

Step 1. Verify IPv6 forwarding is still enabled:

sysctl net.ipv6.conf.all.forwarding
# Expect: net.ipv6.conf.all.forwarding = 1

Step 2. Verify the pool route exists:

ip -6 route show table local | grep <pool-cidr>

If the route is missing, the kernel does not recognize pool addresses as locally-owned and drops the packets before NAT can process them. The gateway adds this route at startup; if it's missing, check the gateway log for startup errors.

Step 3. Verify the destination mesh address actually exists in the FIPS daemon's identity cache:

fipsctl show identity-cache | grep <fd00-mesh-addr>

If the entry is missing, the DNS-side mapping never primed the identity cache, which means the daemon resolver did not actually resolve the .fips name. Re-test the DNS path:

dig @::1 -p 5354 hostname.fips AAAA

Virtual IP unreachable from a LAN client

Symptom: client cannot reach the virtual IP at all (no ping, no ARP/ND response).

Step 1. Verify the client has a route to the pool via the gateway:

# from the LAN client
ip -6 route get <virtual-ip>

The output should show the gateway as the next-hop. If it shows something else (or "unreachable"), fix the LAN-side route — see deploy-gateway.md.

Step 2. On the gateway, verify proxy NDP entries exist for allocated virtual IPs:

ip -6 neigh show proxy

If proxy NDP entries are missing, the gateway cannot answer Neighbor Solicitation requests for virtual IPs on the LAN, so clients cannot resolve the link-layer address and packets never leave the client's NIC.

The gateway adds these entries when a mapping is created (i.e., when a .fips DNS query allocates a virtual IP). If they're absent, trigger a DNS query first:

dig @<gateway-lan-addr> hostname.fips AAAA

Then re-check ip -6 neigh show proxy.

Step 3. Verify proxy_ndp is enabled in the kernel:

sysctl net.ipv6.conf.all.proxy_ndp
# Expect: net.ipv6.conf.all.proxy_ndp = 1

If 0, enable it (see deploy-gateway.md).

Inbound-half diagnostics

Symptoms in this section all involve a mesh peer trying to reach a LAN-side service through the gateway and failing.

Mesh peer can't reach <gateway-npub>.fips:<listen_port>

Walk the path from the mesh-side ingress to the LAN target:

Step 1. Verify the port-forward rule is loaded. On the gateway:

sudo nft list table inet fips_gateway

Look in the prerouting chain for a rule of the form

iif "fips0" meta nfproto ipv6 meta l4proto <tcp|udp> \
    <th> dport <listen_port> dnat ip6 to [<target_addr>]:<target_port>

and, in the postrouting chain, a rule of the form

iif "fips0" oif "<lan_interface>" meta nfproto ipv6 masquerade

The port-forward DNAT and the LAN-side masquerade come from gateway.port_forwards[] and the active lan_interface setting. The masquerade is emitted only when at least one port-forward exists. If either rule is missing, restart the gateway — the table is rebuilt atomically on config load.

Step 2. Verify the mesh firewall is not blocking the listen port. If fips-firewall.service is enabled, the default baseline drops everything inbound on fips0 except established/related and ICMPv6. Add an explicit allow rule under /etc/fips/fips.d/:

# /etc/fips/fips.d/gateway-inbound.nft
tcp dport <listen_port> accept

(See enable-mesh-firewall.md for the full drop-in pattern, including source-address restrictions.) Without an allow rule, mesh peers see TCP RSTs (the firewall drops on the way in) or silent UDP loss.

Step 3. Verify the LAN target is reachable from the gateway itself:

ping6 <target_addr>
curl -v http://[<target_addr>]:<target_port>/   # for TCP HTTP

If the target is unreachable from the gateway, the DNAT rule will fire but the inner connection attempt will fail. Fix LAN-side routing or the target service before going further.

Step 4. Verify conntrack is tracking the inbound flow. Try the connection from a mesh peer once:

curl -v http://<gateway-npub>.fips:<listen_port>/

Then on the gateway:

sudo conntrack -L | grep -E '<listen_port>|<target_port>'

You should see a flow tuple in both directions (orig and reply) with the mesh peer's source on fips0 and the gateway's LAN address as the masqueraded source on the LAN side. No conntrack entry suggests the prerouting DNAT didn't match — recheck step 1.

Step 5. Check the gateway log for nftables or rule install errors:

sudo journalctl -u fips-gateway -e | grep -E 'port_forward|nftables'

A "Failed to install port-forward rules" log line at startup means the rule batch was rejected by netlink — usually a transient condition during a config edit, but persistent failures warrant inspecting the rule with nft -d.

Config rejected: IPv4 target

Symptom: fips-gateway exits at startup with a deserialization error referencing port_forwards[N].target and an invalid IPv6 literal.

The target field is typed as SocketAddrV6 and rejects IPv4 literals at parse time:

# fails at config load
- listen_port: 8080
  proto: tcp
  target: "192.168.1.10:80"

Either re-address the LAN service to be reachable on IPv6, or front it with a small IPv6-aware reverse proxy on the gateway and point the target at that proxy.

Config rejected: zero or duplicate listen_port

Symptom: fips-gateway exits at startup with "Invalid gateway.port_forwards: …".

validate_port_forwards() enforces:

• listen_port must be non-zero.
• The pair (listen_port, proto) must be unique across the list (the same port on TCP and UDP simultaneously is allowed; the same port twice on the same proto is not).

Fix the offending entry and reload.

See also

• ../tutorials/deploy-fips-gateway.md — canonical OpenWrt deployment.
• ../design/fips-gateway.md — gateway design, NAT pipeline, virtual IP pool lifecycle, security considerations.
• deploy-gateway.md — manual Linux-host setup.
• Gateway section of the configuration reference — full gateway.* block.
• ../reference/cli-fips-gateway.md — fips-gateway binary CLI flags.
• Gateway command catalog in the control-socket reference — JSON schema for show_gateway and show_mappings.
• enable-mesh-firewall.md — mesh-firewall baseline and drop-ins.