fix(gateway): report the conntrack source at startup

The gateway pins a mapping while conntrack shows sessions to its virtual
IP. On a kernel with no readable conntrack source every mapping reads zero
sessions, so session pinning is silently off, and an operator learned that
only from a warning at the first failed tick.

The gateway now reads conntrack once at startup, through the same reader
the tick uses, and logs one line: which source answered and that session
pinning is on, or that no source is readable and session pinning is off,
with the error. The integration test asks the gateway's namespace whether
the proc file exists and checks that the matching line appears exactly
once.
This commit is contained in:
Johnathan Corgan
2026-09-19 03:54:00 +00:00
parent ca2ad9f117
commit 7e8130ebf3
5 changed files with 203 additions and 1 deletions
+5
View File
@@ -164,6 +164,11 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
indistinguishable from an idle one. A kernel built without indistinguishable from an idle one. A kernel built without
`CONFIG_NF_CONNTRACK_PROCFS` has no `/proc/net/nf_conntrack` at all and fails `CONFIG_NF_CONNTRACK_PROCFS` has no `/proc/net/nf_conntrack` at all and fails
identically every tick, so a repeat is logged at debug rather than warn. identically every tick, so a repeat is logged at debug rather than warn.
- The gateway says at startup whether it can read conntrack sessions. It
reads the table once, as each tick does, and logs either the source it read
or that no source is readable and session pinning is off. An operator on a
kernel with no readable source learned this only from a warning at the first
failed tick.
- The NAT table is rebuilt in one netlink transaction. A rebuild deleted the - The NAT table is rebuilt in one netlink transaction. A rebuild deleted the
`fips_gateway` table in a batch of its own, discarded that batch's result, `fips_gateway` table in a batch of its own, discarded that batch's result,
and only then sent the batch that recreated the table, the chains, the and only then sent the batch that recreated the table, the chains, the
+22
View File
@@ -159,6 +159,28 @@ failed to bind the socket and continued without it (the warning
`Failed to bind gateway control socket — continuing without it` is `Failed to bind gateway control socket — continuing without it` is
in the journal in that case). in the journal in that case).
### Session pinning is off
The gateway keeps a mapping while conntrack shows sessions to its
virtual IP. At startup it reads conntrack once, the same way each
10 s tick does, and logs which source answered:
- `Conntrack source: proc; session pinning is on`: sessions are read
from `/proc/net/nf_conntrack`.
- `No conntrack source is readable; session pinning is off`: no
source could be read, and the line carries the error. Every mapping
then reads zero sessions, so a mapping is reclaimed on its TTL and
grace period alone, even while a client that has not re-queried DNS
still has traffic flowing through it.
The proc file exists only on a kernel built with
`CONFIG_NF_CONNTRACK_PROCFS`, and only once `nf_conntrack` is loaded:
```sh
ls /proc/net/nf_conntrack
grep NF_CONNTRACK_PROCFS /boot/config-$(uname -r)
```
## Outbound-half diagnostics ## Outbound-half diagnostics
Symptoms in this section all involve a LAN client trying to reach a Symptoms in this section all involve a LAN client trying to reach a
+30 -1
View File
@@ -72,7 +72,7 @@ fn elapsed_us(started: Instant) -> u64 {
async fn read_conntrack(log: &mut pool::ConntrackReadLog) -> pool::ConntrackSnapshot { async fn read_conntrack(log: &mut pool::ConntrackReadLog) -> pool::ConntrackSnapshot {
use fips::gateway::pool::ConntrackQuerier; use fips::gateway::pool::ConntrackQuerier;
match tokio::task::spawn_blocking(|| pool::ProcConntrack.snapshot()).await { match tokio::task::spawn_blocking(|| pool::SystemConntrack::default().snapshot()).await {
Ok(Ok(snapshot)) => { Ok(Ok(snapshot)) => {
log.observe(None); log.observe(None);
snapshot snapshot
@@ -107,6 +107,31 @@ fn report_unreadable_conntrack(
} }
} }
/// Check once at startup which conntrack source the tick will read, and say so.
///
/// Without this, an operator on a kernel with no readable source learns that
/// session pinning is off only from a warning at the first failed tick.
#[cfg(target_os = "linux")]
async fn report_conntrack_source() {
let probe =
tokio::task::spawn_blocking(|| pool::probe_conntrack(&pool::SystemConntrack::default()))
.await
.unwrap_or_else(|e| {
pool::ConntrackProbe::Missing(pool::ConntrackUnreadable {
proc: std::io::Error::other(e.to_string()),
})
});
match probe {
pool::ConntrackProbe::Found(pool::ConntrackSource::Proc) => {
info!("Conntrack source: proc; session pinning is on")
}
pool::ConntrackProbe::Missing(e) => warn!(
proc_error = %e.proc,
"No conntrack source is readable; session pinning is off"
),
}
}
#[cfg(target_os = "linux")] #[cfg(target_os = "linux")]
#[tokio::main(flavor = "current_thread")] #[tokio::main(flavor = "current_thread")]
async fn main() { async fn main() {
@@ -371,6 +396,10 @@ async fn main() {
std::process::exit(1); std::process::exit(1);
} }
// The NAT table exists by now, so a kernel that provides the proc file
// has loaded nf_conntrack and the probe sees what the first tick will.
report_conntrack_source().await;
// --- Channels --- // --- Channels ---
// Pool events (new/removed mappings) → NAT + net modules // Pool events (new/removed mappings) → NAT + net modules
+125
View File
@@ -159,6 +159,93 @@ impl ConntrackQuerier for ProcConntrack {
} }
} }
/// Where a conntrack snapshot was read from.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum ConntrackSource {
/// `/proc/net/nf_conntrack`.
Proc,
}
impl ConntrackSource {
/// Short name of the source.
pub fn name(self) -> &'static str {
match self {
Self::Proc => "proc",
}
}
}
/// Why no conntrack source could be read.
#[derive(Debug)]
pub struct ConntrackUnreadable {
/// The error reading `/proc/net/nf_conntrack`.
pub proc: std::io::Error,
}
impl ConntrackUnreadable {
/// The error that stands for the whole failed read.
fn into_error(self) -> std::io::Error {
self.proc
}
}
/// The conntrack reader the gateway uses, which also says which source
/// answered.
///
/// The per-tick read and the startup probe both go through this type, so the
/// probe cannot report a source the tick would not use. The querier is a type
/// parameter so tests can substitute fakes.
pub struct SystemConntrack<P = ProcConntrack> {
proc: P,
}
impl<P: ConntrackQuerier> SystemConntrack<P> {
/// A reader over the given proc querier.
pub fn new(proc: P) -> Self {
Self { proc }
}
/// Read conntrack once and say which source the snapshot came from.
pub fn read(&self) -> Result<(ConntrackSource, ConntrackSnapshot), ConntrackUnreadable> {
match self.proc.snapshot() {
Ok(snapshot) => Ok((ConntrackSource::Proc, snapshot)),
Err(proc) => Err(ConntrackUnreadable { proc }),
}
}
}
impl Default for SystemConntrack {
fn default() -> Self {
Self::new(ProcConntrack)
}
}
impl<P: ConntrackQuerier> ConntrackQuerier for SystemConntrack<P> {
fn snapshot(&self) -> Result<ConntrackSnapshot, std::io::Error> {
self.read()
.map(|(_, snapshot)| snapshot)
.map_err(ConntrackUnreadable::into_error)
}
}
/// Outcome of the startup check for a readable conntrack source.
#[derive(Debug)]
pub enum ConntrackProbe {
/// Sessions can be read, from this source.
Found(ConntrackSource),
/// No source can be read, so every mapping reads zero sessions and session
/// pinning is off.
Missing(ConntrackUnreadable),
}
/// Read conntrack once, as a tick would, and report which source answered.
pub fn probe_conntrack<P: ConntrackQuerier>(reader: &SystemConntrack<P>) -> ConntrackProbe {
match reader.read() {
Ok((source, _)) => ConntrackProbe::Found(source),
Err(e) => ConntrackProbe::Missing(e),
}
}
/// Count conntrack lines by the destination addresses they name. /// Count conntrack lines by the destination addresses they name.
/// ///
/// Every `dst=` value is parsed as an address and compared as an address. The /// Every `dst=` value is parsed as an address and compared as an address. The
@@ -1135,4 +1222,42 @@ mod tests {
"a different failure is a different outcome and is worth a line" "a different failure is a different outcome and is worth a line"
); );
} }
/// A conntrack querier that succeeds with an empty snapshot, or fails with
/// a fixed error kind.
struct FixedRead(Option<std::io::ErrorKind>);
impl ConntrackQuerier for FixedRead {
fn snapshot(&self) -> Result<ConntrackSnapshot, std::io::Error> {
match self.0 {
None => Ok(ConntrackSnapshot::default()),
Some(kind) => Err(kind.into()),
}
}
}
#[test]
fn conntrack_probe_names_the_proc_source_when_the_proc_read_succeeds() {
let reader = SystemConntrack::new(FixedRead(None));
match probe_conntrack(&reader) {
ConntrackProbe::Found(source) => {
assert_eq!(source, ConntrackSource::Proc);
assert_eq!(source.name(), "proc");
}
ConntrackProbe::Missing(e) => panic!("expected the proc source, got {e:?}"),
}
}
#[test]
fn conntrack_probe_reports_missing_with_the_error_when_the_proc_read_fails() {
let reader = SystemConntrack::new(FixedRead(Some(std::io::ErrorKind::PermissionDenied)));
match probe_conntrack(&reader) {
ConntrackProbe::Missing(e) => {
assert_eq!(e.proc.kind(), std::io::ErrorKind::PermissionDenied);
}
ConntrackProbe::Found(source) => panic!("expected no source, got {source:?}"),
}
}
} }
+21
View File
@@ -153,6 +153,27 @@ if [ "$DNS_READY" != true ]; then
echo " WARNING: Gateway DNS did not respond within 30s, continuing anyway" echo " WARNING: Gateway DNS did not respond within 30s, continuing anyway"
fi fi
# The gateway names its conntrack source once at startup, before the DNS
# resolver starts, so by now the line is in the log. Ask the gateway's own
# namespace which source it should have found. A failed `docker logs` reds the
# check rather than counting as zero lines.
if docker exec "$GATEWAY" test -e /proc/net/nf_conntrack; then
EXPECT_SRC=proc
else
EXPECT_SRC=none
fi
if GW_START_LOG=$(docker logs "$GATEWAY" 2>&1); then
SRC_PROC=$(grep -cF 'Conntrack source: proc; session pinning is on' <<< "$GW_START_LOG" || true)
SRC_NONE=$(grep -cF 'No conntrack source is readable; session pinning is off' <<< "$GW_START_LOG" || true)
case "$EXPECT_SRC" in
proc) SRC_OK=$([ "$SRC_PROC" -eq 1 ] && [ "$SRC_NONE" -eq 0 ] && echo 0 || echo 1) ;;
*) SRC_OK=$([ "$SRC_NONE" -eq 1 ] && [ "$SRC_PROC" -eq 0 ] && echo 0 || echo 1) ;;
esac
check "Conntrack source line at startup (expect $EXPECT_SRC; proc lines $SRC_PROC, none lines $SRC_NONE)" "$SRC_OK"
else
check "Conntrack source line at startup (docker logs failed)" 1
fi
# Phase 3: Client network setup — route virtual IP pool via gateway # Phase 3: Client network setup — route virtual IP pool via gateway
echo "" echo ""
echo "Phase 3: Client network setup" echo "Phase 3: Client network setup"