Johnathan Corgan
43b6512503
Any host able to query the LAN resolver could drain the gateway's 65,535-address virtual-IP pool one `.fips` name at a time, and each allocation rebuilt the whole nftables table in a way that could leave the host with no NAT at all. Four changes, each independently useful, close that off. Do not allocate for query types the gateway never answers with an address. handle_query minted a virtual IP for every query type and only then looked at what the client asked, answering an A or HTTPS query with NODATA after creating a mapping for it. The query type is now decided before the pool is touched, and only AAAA and ANY allocate. The refresh an existing mapping used to get from any query type is kept: it came from the reuse path in allocate, so a new pool method does that refresh alone and never creates anything, and the reuse path calls it. Rebuild the NAT table in one netlink transaction. rebuild() deleted the fips_gateway table in a batch of its own and discarded the result, then sent a second batch recreating the table, the chains, the fips0 masquerade and two rules per mapping. Between those sends the host had no NAT table, and a recreate the kernel refused left the table deleted, turning one failed mapping change into a total loss of forwarding until some later rebuild happened to succeed. The delete and the recreate now share one batch. A leading table add makes the delete legal on the first run, since rustables sends it with NLM_F_CREATE and no NLM_F_EXCL and the crate offers no flush. Deciding what to send is now separate from sending it, which is the seam the new unit tests use: they assert one batch, the add-delete-add prefix, and that every chain and rule follows the recreate, without a netlink socket or privileges. Read conntrack once per tick, off the runtime thread, and match by address. The session count searched each /proc/net/nf_conntrack line for `dst=` followed by the virtual IP's compressed Display form, while the kernel prints every tuple with `%pI6`, the full uncompressed form. That string cannot occur in that field, so the count was zero for every mapping on every kernel that has the file: nothing pinned an in-use mapping and one whose client did not re-query DNS was reclaimed about two minutes after its last DNS reference with traffic still flowing. Each `dst=` is now parsed and compared as an address. The read was also per mapping, under the pool lock, on the runtime thread that serves DNS; the tick now takes one snapshot in a blocking task before taking the lock. An unreadable source was silent, because read_to_string's error became zero through unwrap_or(0). Zero stays, since treating it as in-use would pin every mapping forever on a kernel without CONFIG_NF_CONNTRACK_PROCFS, but it is now reported at warn on the first failure and on each change of outcome, and at debug on a repeat. Ship the OpenWrt gateway disabled, and keep its state across upgrades. The generated postinst enabled and started fips-gateway on every install, against the init script's own header, the package README and the deployment tutorial, which all say the service ships disabled. A fresh install now leaves it alone. Upgrades are the awkward case: opkg runs the outgoing package's prerm first, and every released prerm disabled the gateway on its way out without recording whether it had been enabled. The new prerm stops the services on an upgrade but no longer disables them, and leaves a marker the incoming postinst reads. With the marker, enablement survived and the gateway starts only if it was enabled; without it, the outgoing package was a released one whose prerm destroyed that state, so the gateway is re-enabled rather than letting an upgrade turn off a working deployment. That re-enables a hand-disabled gateway once, which the CHANGELOG says. start_service now reads gateway.enabled from fips.yaml before touching anything, since starting a gateway the config disables used to take dnsmasq's `.fips` forwarding away from the daemon and hand it to a port whose daemon exits immediately. The four maintainer-script bodies move out of heredocs in the two build scripts into packaging/openwrt-ipk/scripts/, so the .ipk and the .apk install the same bodies and a test can run what ships. Coverage recorded rather than closed. The conntrack parser's first test builds its line from the kernel's own format string rather than a capture, because this host is built without CONFIG_NF_CONNTRACK_PROCFS and has no /proc/net/nf_conntrack, so the lab exercises only the unreadable path. Kernel acceptance of delete-then-recreate inside one transaction is not asserted by a unit test; the gateway suite is what proves it, since the manager rebuilds at startup and the daemon exits if that fails. The OpenWrt scenarios run the shipped script bodies under ash in a busybox container against stubbed init scripts, and assert their behaviour given opkg's call order, arguments and PKG_UPGRADE as read from opkg-lede's sources, not under a real opkg upgrade on a router image. Admission limits on the pool are deliberately not included here: they need a measurement run before their constants can be chosen.
FIPS OpenWrt Package (apk)
Builds a FIPS .apk for OpenWrt 25+, where apk-tools is the mandatory
package manager. apk is also available opt-in on 24.10 (where opkg remains
the default). For OpenWrt 24.x and earlier, the .ipk package in
../openwrt-ipk/ still works.
Like the .ipk build, this is SDK-free: it cross-compiles with
cargo-zigbuild and assembles the package directly — no OpenWrt SDK image. The
.ipk format is a plain tar.gz we can hand-roll, but the .apk (apk-tools v3
ADB) container is not, so we drive the official apk mkpkg applet — the same
tool OpenWrt's own include/package-pack.mk
calls. The only extra requirement over the .ipk build is the apk binary.
Layout
| File | Purpose |
|---|---|
build-apk.sh |
Cross-compile + assemble the .apk via apk mkpkg |
apk-version.sh |
Map a release tag / commit height to an apk-tools-valid version |
apk-version.test.sh |
Case-table test for apk-version.sh (sh apk-version.test.sh) |
The installed-filesystem payload (init scripts, fips.yaml, sysctl drop-ins,
hotplug, uci-defaults, …) is shared with the .ipk package — there is one
canonical copy in ../openwrt-ipk/files/. build-apk.sh
stages from there, so the two packages ship the same files apart from one
staged rewrite: build-apk.sh changes ethernet.wan.interface in the staged
fips.yaml from eth0 to wan, the OpenWrt 25 DSA port name. Keep the
staging block in build-apk.sh in sync with ../openwrt-ipk/build-ipk.sh.
Versioning
apk-tools enforces a strict version grammar
(<digit>(.<digit>)*(_<suffix><digit>*)*(-r<N>)). apk-version.sh builds a
valid version from structured inputs rather than rewriting an already-flattened
string:
| Input | apk version |
|---|---|
tag v1.2.3 |
1.2.3-r0 |
tag v1.2.3-rc1 |
1.2.3_rc1-r0 |
dev 1234 (commit height) |
0.0.0_git1234-r0 |
The human-readable version (v1.2.3, master.123.abcdef0) is still used for the
artifact filename; only the metadata embedded in the package is normalized.
Building
Prerequisites
| Requirement | Notes |
|---|---|
cargo install cargo-zigbuild + zig |
Rust musl cross-compilation (as for .ipk) |
apk-tools v3 apk binary |
Provides apk mkpkg; not packaged for most distros — build from source |
fakeroot |
Optional; makes packaged files root-owned on an unprivileged build host |
apk-tools is not in Debian/Ubuntu repos, so build the pinned release from source.
Pin the same commit the targeted OpenWrt release ships (see
package/system/apk/Makefile upstream) so the .apk is readable by the device's
apk. CI builds 3.0.5 (b5a31c0d…):
sudo apt-get install -y build-essential meson ninja-build pkg-config \
zlib1g-dev libssl-dev libzstd-dev liblzma-dev lua5.4-dev scdoc
git clone https://gitlab.alpinelinux.org/alpine/apk-tools.git
cd apk-tools && git checkout b5a31c0d865342ad80be10d68f1bb3d3ad9b0866
meson setup build && ninja -C build src/apk
export APK_BIN="$PWD/build/src/apk"
Build the package
# from the repo root
./packaging/openwrt-apk/build-apk.sh --arch aarch64 # or x86_64, mipsel, mips, arm
Output: dist/fips_<version>_<openwrt-arch>.apk. Override the version with
PKG_VERSION (filename) and APK_VERSION (embedded metadata); otherwise both are
derived from git.
Installing on the router
Packages are unsigned (the same posture as our .ipk), so install with
--allow-untrusted:
scp -O dist/fips_<version>_<arch>.apk root@192.168.1.1:/tmp/
ssh root@192.168.1.1 apk add --allow-untrusted /tmp/fips_<version>_<arch>.apk
On OpenWrt 25.x, installing from a signed repository requires the publisher's
key; a single --allow-untrusted package install does not. If we ever publish an
apk feed, add ECDSA (prime256v1) signing via apk mkpkg --sign and distribute the
public key to /etc/apk/keys/.
/etc/fips/fips.yaml is marked as a config file (via
/lib/apk/packages/fips.conffiles), so apk preserves local edits across upgrades,
and /lib/upgrade/keep.d/fips preserves /etc/fips/ across sysupgrade — the
same guarantees as the .ipk package.