Files
fips/src/native/dgram_probe.rs
T
Johnathan Corgan 1ec62e5119 Build the native datagram API on macOS, over SOCK_DGRAM
macOS does not implement SOCK_SEQPACKET for AF_UNIX, so the listener was
gated to Linux and FreeBSD and a Mac got no API at all. It now uses
SOCK_DGRAM there, which macOS does implement and which keeps the message
boundaries the API's contract with its clients rests on.

Both kernels were measured rather than reasoned about, and the Linux answer
alone refuted the replacement the source had proposed. On Linux 6.8 a
connected SOCK_DGRAM pair reports a closed peer not at all: revents stays
empty and recv returns EAGAIN, which is exactly what an idle socket with a
live peer does. SOCK_SEQPACKET on the same kernel sets POLLHUP and returns a
zero-byte read, which is what the receive path keyed on. Darwin does report
the close, with ECONNRESET, errno 54, and does not set POLLHUP. So the
receive path treats ECONNRESET as end of file alongside the existing POLLHUP
rule. One rule accepting either signal is correct on both kernels, where a
rule split by platform would be silently wrong on whichever one it guessed
at. EAGAIN is deliberately not in that company: it means the socket is empty
and the peer alive, so it stays an error and the caller waits again.

The measurements are asserted rather than only written down, so a kernel that
gains or loses the signal reds a test and reopens the question instead of
leaving a stale comment behind. Each carries its result in the assertion
message, since a passing test prints nothing and a negative result would
otherwise be as uninformative as no result: the close probe reports the poll
return, the whole revents bitmask broken out by flag, the recv result and the
errno, which is enough to write the real rule without another round trip. A
portable test walks datagram sizes upward, because Darwin bounds a
unix-domain datagram with the net.local.dgram.maxdgram sysctl, whose default
is small and which is a system tunable rather than something this process
controls, while the API advertises 1362 bytes to its clients.

Every test recv in the seqpacket suite is bounded in time. This is not
tidying. Simulating the Darwin configuration on a kernel that does not report
the close, the end-of-file test hung for over ten minutes rather than
failing, and a hang is not a red: it would have wedged the macOS runner with
no diagnostic instead of naming the assertion. The same simulation now fails
by name in five seconds.

Three things in the native tree compiled on one platform only, all of them in
code that had never been built for Darwin before. suseconds_t is i64 on Linux
and i32 there, so the timeval microseconds field is a cast, matching the
tv_sec line above it; it cannot truncate, because subsec_micros is below
1_000_000 by construction, and the cast is the only form that compiles on
both, since From does not exist for the narrower width and try_from is a
clippy error on the wider one. MSG_CMSG_CLOEXEC does not exist on Apple, so
the recvmsg flags are chosen per platform and each received descriptor is
marked close-on-exec with fcntl where there is no flag to pass; a failure to
set it is reported rather than ignored, since the descriptor is live either
way and the caller must not be told the receive was clean. socketpair takes
SOCK_CLOEXEC in its type argument on Linux and FreeBSD and rejects it on
macOS, so Darwin sets FD_CLOEXEC with a second fcntl. Both windows between a
call and its fcntl are stated in the code rather than closed, since the
daemon spawns no child on this path, and a test asserts both halves of a pair
are close-on-exec on every platform, because the failure is a silent
descriptor leak into a child and nothing else would report it. Every libc
item the native tree uses was then checked against the crate's own Apple
definitions rather than from memory, and those two constants are the only
ones absent.

The close difference turned out to be unhandled in six further places, and
the whole native API agrees on it now. Darwin reports a closed AF_UNIX
SOCK_DGRAM peer as ECONNRESET, and a later send on the disconnected survivor
as EDESTADDRREQ, where Linux SOCK_SEQPACKET gives EPIPE on a write and a
zero-byte read plus POLLHUP on a read. Each site below promised one of those
spellings and saw another.

- The client's recv and send passed ECONNRESET through, so a closed daemon
  half surfaced as errno 54 against the EPIPE the documentation promises.
  The translation is in one function in seqpacket rather than at each call
  site, since only this one condition has two spellings.
- accept propagated the same errno instead of its documented EPIPE. The
  listener's read reports a closed peer as the empty chunk both of its
  callers already read as the far end going away, which leaves accept's
  contract true on both platforms without either caller knowing which it is
  on.
- why() classified a failed hand-off by BrokenPipe alone, so every ordinary
  macOS listener close was counted under the counter an operator reads to
  find a client that stopped reading. It recognises all three errnos now,
  with a test over each.
- A full client buffer ended a flow's only writer. On Linux that never
  arrives, because the send reports EAGAIN and waits for the client to drain;
  Darwin has no sender-side queue to wait on and reports ENOBUFS on the send
  itself. Returning left the registration, the port and the reader alive
  while every later inbound datagram was counted as a full queue for the rest
  of the flow's life, and a client that resumed reading never recovered. The
  datagram is dropped instead, which is what a datagram API does when the far
  end cannot take it.
- The flow pair was never sized, and the two kernels charge a queued message
  to different ends: Linux to the sender's SO_SNDBUF, BSD to the receiver's
  so_rcv. Sizing only the sender, as the listener pair does, left the flow
  pair bounded on Darwin by a system default small enough that a batch held
  for an arriving client could not fit, and the whole flow was destroyed
  before its client ever saw it. Both halves are sized now.
- peer_hung_up polled with an empty events field, on the rule that POLLHUP is
  reported whether or not it is requested. That holds on Linux, where it was
  measured, and not on Darwin, where a poll requesting nothing registers no
  filter. Nothing observable depended on it, because ECONNRESET arrives first
  and both callers act on it earlier. The cost was elsewhere: three
  assertions written as tripwires for a change in Darwin's behaviour could
  not fail there, which is a guard that executes and proves nothing.
  Requesting POLLIN fixes the function and the guards together.

One difference is not an errno at all, and reading the kernel source rather
than a manual page is what found it. Darwin's unp_disconnect sets
SS_CANTRCVMORE and runs soisdisconnected on both ends for SOCK_STREAM. For
SOCK_DGRAM it removes the reflink, clears SS_ISCONNECTED and stops: no
sorwakeup, no socantrcvmore, no soisdisconnected. The closing peer deposits
ECONNRESET in the survivor's so_error and wakes no knote. The registration is
edge-triggered and was made while the socket was healthy, so nothing
re-evaluates it, and recv awaited readiness before its syscall, which left
the ECONNRESET arm sitting behind an await that never returns. A client
closing its descriptor left the daemon's reader parked for ever, and the
flow's port and registry entry held for the node's lifetime. recv reads
before it waits now, because the latched error is visible to a syscall and
only to a syscall, so the attempt that precedes the wait is what sees a close
that has already happened. A close can also land while the task is parked,
which no first attempt can catch, so on Darwin the wait is bounded and the
syscall retried; the error is latched until a read consumes it, so the bound
sets how long a dead flow holds its port rather than deciding whether the
close is seen at all. On Linux this is one extra recv returning EAGAIN before
the wait and changes nothing else, and everywhere else the readiness is
authoritative and the wait stays unbounded. The three tests this predicted
are the three that had failed: end of file on a closed client half, a
listener's port unbound on close, and one flow's port freed while its
connection stays open.

One test asserted a delivery detail rather than the rule it exists to guard.
a_descriptor_lands_on_the_last_complete_line_of_the_read_that_carried_it
asserted that a plain write and the sendmsg following it arrive in one
recvmsg. Linux coalesces them, so the read returns both lines and the
descriptor together; Darwin stops a stream read at the ancillary boundary, so
the plain line arrives by itself and the descriptor-bearing line comes on the
next read. The rule the module rests on is unaffected, and Darwin satisfies
it more easily than Linux, because the read it arrives on holds nothing
later. The test fills until both lines are queued and asserts the rule
instead of the number of reads it took.

The client compiled in /run/fips/api.sock on every platform, and macOS has no
/run for that path to be in. The daemon never had this problem: it resolves
its socket at startup by looking for a directory, and its macOS branch lands
on /var/run/fips. The constant is conditional the same way now, so a client
that is told nothing looks where a packaged daemon on its own platform
actually is. The reference documentation described that branch as
FreeBSD-only and describes both.

Windows stays excluded and cannot be included: it has no SCM_RIGHTS, so there
is no way to pass a descriptor to another process at all, which is the whole
mechanism rather than a detail of it.

The platform statements in the source and in the shipped documentation all
named Linux and FreeBSD and name macOS now, including the configuration
reference, the security reference, the how-to and the walkthrough. The how-to
also states how far the testing goes, because the person who would meet the
gap first is the one enabling the API on a Mac. The end-to-end suite drives a
client container against a node container over a shared volume, which is a
Linux arrangement, so the socket lifecycle, the descriptor hand-off across a
process boundary and the reclaiming of a port when a client exits are covered
on macOS by unit tests rather than by anything that runs a daemon and a
client as two real processes. That is a gap in testing and not a known
defect, and it is a coverage gap rather than a discharged risk. The same
place names the socket-type difference, since a reader who knows the
descriptor is SOCK_DGRAM there can make sense of a close arriving as a
different errno than the Linux documentation elsewhere describes.

The changelog entry for the API is revised rather than followed by a second
one: it now names the socket type each platform uses and the two
end-of-file signals the receive path accepts. The entry describes what the
release ships rather than the order the commits landed in.
2026-08-21 05:48:39 +00:00

322 lines
13 KiB
Rust

//! What a connected `AF_UNIX` `SOCK_DGRAM` pair does at end of file.
//!
//! This module is tests only. It exists to answer, by measurement on each
//! kernel rather than from the manual pages, the one question a macOS port of
//! this API turns on.
//!
//! [`super::seqpacket`] uses `SOCK_SEQPACKET`, which macOS does not implement
//! for `AF_UNIX`. The candidate replacement there is `SOCK_DGRAM`, which macOS
//! does implement and which also keeps message boundaries. What is not
//! transferable is the rule that tells a close apart from an empty datagram:
//! both produce a zero-byte read, and `seqpacket` resolves them with a latched
//! `POLLHUP` measured on Linux 6.8. Datagram poll semantics differ between
//! kernels, so that rule has to be re-established on Darwin before anything is
//! built on it.
//!
//! **The tests below assert the properties an implementation would need.** A
//! failure here is the measurement coming back negative, not a regression: it
//! says this kernel cannot support the `seqpacket` close rule on `SOCK_DGRAM`
//! and that a macOS port needs a different close signal. The same code runs on
//! every unix so the platforms can be compared without the test itself being a
//! variable.
//!
//! `SOCK_CLOEXEC` is deliberately not passed in the type argument, though
//! `super::seqpacket::pair` does pass it. Linux and FreeBSD accept it there and
//! macOS does not, and that difference belongs to the port rather than to this
//! measurement.
use std::io;
use std::os::fd::{AsRawFd, FromRawFd, OwnedFd, RawFd};
/// Create a connected `AF_UNIX` `SOCK_DGRAM` pair.
fn dgram_pair() -> io::Result<(OwnedFd, OwnedFd)> {
let mut fds = [0 as libc::c_int; 2];
// SAFETY: `fds` is a two-element array of the type socketpair writes, and
// the call either fills both entries or reports failure.
let rc = unsafe { libc::socketpair(libc::AF_UNIX, libc::SOCK_DGRAM, 0, fds.as_mut_ptr()) };
if rc != 0 {
return Err(io::Error::last_os_error());
}
// SAFETY: socketpair reported success, so both entries are open descriptors
// this frame now owns.
Ok(unsafe { (OwnedFd::from_raw_fd(fds[0]), OwnedFd::from_raw_fd(fds[1])) })
}
/// One non-blocking receive, returning the byte count or the errno.
fn recv(fd: RawFd, buf: &mut [u8]) -> io::Result<usize> {
// SAFETY: the descriptor is open and the pointer and length describe `buf`.
let n = unsafe { libc::recv(fd, buf.as_mut_ptr().cast(), buf.len(), libc::MSG_DONTWAIT) };
if n < 0 {
return Err(io::Error::last_os_error());
}
Ok(n as usize)
}
/// Send one datagram, returning the byte count or the errno.
fn send(fd: RawFd, buf: &[u8]) -> io::Result<usize> {
// SAFETY: the descriptor is open and the pointer and length describe `buf`.
let n = unsafe { libc::send(fd, buf.as_ptr().cast(), buf.len(), 0) };
if n < 0 {
return Err(io::Error::last_os_error());
}
Ok(n as usize)
}
/// Whether `POLLHUP` is set, by the same rule `seqpacket::peer_hung_up` uses.
///
/// `POLLIN` is requested rather than nothing. An empty `events` registers no
/// filter on Darwin, so a poll asking for nothing reports nothing there and
/// every assertion built on this helper would pass whatever the kernel did.
/// That is the shape of a guard that executes and cannot fail, so it is worth
/// more than a comment: with `POLLIN` requested, a Darwin that began reporting
/// `POLLHUP` would red the tests below instead of slipping past them.
fn hung_up(fd: RawFd) -> bool {
let mut poll = libc::pollfd {
fd,
events: libc::POLLIN,
revents: 0,
};
// SAFETY: `poll` points at one live pollfd and the call cannot block.
let rc = unsafe { libc::poll(&mut poll, 1, 0) };
rc > 0 && (poll.revents & libc::POLLHUP) != 0
}
#[test]
fn a_connected_dgram_pair_keeps_message_boundaries() {
let (a, b) = dgram_pair().expect("AF_UNIX SOCK_DGRAM socketpair");
assert_eq!(send(a.as_raw_fd(), &[1, 2, 3]).unwrap(), 3);
assert_eq!(send(a.as_raw_fd(), &[4, 5]).unwrap(), 2);
// Two sends must read back as two messages of their own lengths. A stream
// socket would hand back all five bytes in one read, which is the failure
// this discriminates.
let mut buf = [0u8; 64];
assert_eq!(recv(b.as_raw_fd(), &mut buf).unwrap(), 3);
assert_eq!(&buf[..3], &[1, 2, 3]);
assert_eq!(recv(b.as_raw_fd(), &mut buf).unwrap(), 2);
assert_eq!(&buf[..2], &[4, 5]);
}
#[test]
fn an_empty_datagram_reads_as_zero_bytes_and_is_not_a_hangup() {
let (a, b) = dgram_pair().expect("AF_UNIX SOCK_DGRAM socketpair");
assert_eq!(send(a.as_raw_fd(), &[]).unwrap(), 0);
let mut buf = [0u8; 64];
assert_eq!(
recv(b.as_raw_fd(), &mut buf).unwrap(),
0,
"an empty datagram must be delivered as a zero-byte message"
);
assert!(
!hung_up(b.as_raw_fd()),
"an empty datagram must not look like a closed peer: if this fails, a \
client could tear down its own flow by sending nothing"
);
}
/// Linux 6.8, measured 2026-08-20 with this test and cross-checked with a C
/// probe over both socket types: a connected `AF_UNIX` `SOCK_DGRAM` pair gives
/// **no close signal at all**. The peer closing leaves `revents` empty and
/// leaves `recv` returning `EAGAIN`, which is what an idle socket with a live
/// peer also does. `SOCK_SEQPACKET` on the same kernel sets `POLLHUP` and
/// returns a zero-byte read.
///
/// This is asserted rather than merely written down so that a kernel which
/// starts reporting the close reds this test and reopens the question.
#[cfg(target_os = "linux")]
#[test]
fn linux_gives_a_dgram_pair_no_close_signal_at_all() {
let (a, b) = dgram_pair().expect("AF_UNIX SOCK_DGRAM socketpair");
drop(a);
let mut buf = [0u8; 64];
let read = recv(b.as_raw_fd(), &mut buf);
let err = read.as_ref().err().map(|e| e.raw_os_error());
assert_eq!(
err,
Some(Some(libc::EAGAIN)),
"expected the closed peer to be indistinguishable from an idle socket, got {read:?}"
);
assert!(
!hung_up(b.as_raw_fd()),
"POLLHUP is now set on a closed SOCK_DGRAM peer: this kernel has gained \
the close signal Linux 6.8 did not have, and the macOS port's design \
question should be reopened"
);
}
/// The open question, and the only thing a Mac is needed for.
///
/// BSD kernels differ from Linux on datagram close reporting, so Darwin may
/// return `ECONNRESET`, or set `POLLHUP`, where Linux reports nothing. Either
/// would give the receive path something to key on.
///
/// **A failure here is the measurement coming back negative, not a
/// regression.** It says Darwin behaves as Linux does, that a `SOCK_DGRAM`
/// descriptor carries no close signal, and that a macOS port must take the
/// close from the client's line-protocol connection instead of from the flow
/// descriptor.
#[cfg(target_os = "macos")]
#[test]
fn darwin_reports_a_closed_dgram_peer_somehow() {
let (a, b) = dgram_pair().expect("AF_UNIX SOCK_DGRAM socketpair");
drop(a);
let mut buf = [0u8; 64];
let read = recv(b.as_raw_fd(), &mut buf);
let hup = hung_up(b.as_raw_fd());
let signalled = hup
|| matches!(&read, Ok(0))
|| read.as_ref().err().is_some_and(|e| {
e.raw_os_error() != Some(libc::EAGAIN) && e.raw_os_error() != Some(libc::EWOULDBLOCK)
});
assert!(
signalled,
"Darwin reports nothing when a connected SOCK_DGRAM peer closes: \
POLLHUP unset and recv gave {read:?}, which is what an idle socket \
gives. The flow descriptor cannot carry the close on this platform."
);
}
/// Which signal Darwin gives: `ECONNRESET`, and not `POLLHUP`.
///
/// Measured 2026-08-20 on `macos-latest`, run 32353220389, and identical across
/// all three of nextest's attempts, so it is the kernel's behaviour and not a
/// race. The exact reading was `poll` returning 0 with an empty `revents`, and
/// `recv` returning errno 54, `ECONNRESET`.
///
/// This is the opposite of `SOCK_SEQPACKET` on Linux, which sets `POLLHUP` and
/// returns a zero-byte read, and it is why
/// [`super::seqpacket::recv_once`](super::seqpacket) treats `ECONNRESET` as end
/// of file alongside the `POLLHUP` rule rather than choosing between them by
/// platform: one rule that accepts either signal is correct on both kernels.
///
/// Asserted rather than only written down, so that a Darwin release which moves
/// to `POLLHUP`, or stops reporting the close at all, reds this test instead of
/// silently changing what the receive path depends on.
#[cfg(target_os = "macos")]
#[test]
fn darwin_signals_a_closed_dgram_peer_with_econnreset() {
let (a, b) = dgram_pair().expect("AF_UNIX SOCK_DGRAM socketpair");
drop(a);
let mut poll = libc::pollfd {
fd: b.as_raw_fd(),
events: libc::POLLIN,
revents: 0,
};
// SAFETY: `poll` points at one live pollfd and the call cannot block.
let rc = unsafe { libc::poll(&mut poll, 1, 0) };
let mut buf = [0u8; 64];
let read = recv(b.as_raw_fd(), &mut buf);
let errno = read.as_ref().err().and_then(|e| e.raw_os_error());
assert_eq!(
errno,
Some(libc::ECONNRESET),
"Darwin no longer reports a closed SOCK_DGRAM peer as ECONNRESET. \
Measured: poll rc={rc}, revents=0x{:04x}, recv={read:?}. The receive \
path treats ECONNRESET as end of file and would now hang instead.",
poll.revents,
);
assert!(
(poll.revents & libc::POLLHUP) == 0,
"Darwin has gained POLLHUP on a closed SOCK_DGRAM peer, revents=0x{:04x}. \
Nothing breaks, since the receive path accepts either signal, but the \
record here is now wrong and the SOCK_SEQPACKET comparison it rests on \
should be re-read.",
poll.revents,
);
}
/// Whether a `SOCK_DGRAM` pair can carry the largest payload the API offers.
///
/// Darwin bounds a unix-domain datagram with the `net.local.dgram.maxdgram`
/// sysctl, whose default is small, and it is a system tunable rather than
/// something this process can rely on. Linux has no equivalent ceiling on an
/// `AF_UNIX` datagram beyond the socket buffer. The API advertises a payload
/// limit of 1362 bytes to its clients, so a kernel that refuses a datagram that
/// size would break the contract the client was told.
///
/// Written to answer on failure as well as on success: the assertion message
/// carries the largest size that did cross, so a negative result names the
/// actual ceiling rather than only saying the hoped-for one was not reached.
#[test]
fn a_dgram_pair_carries_the_largest_payload_the_api_advertises() {
/// The `max_payload` the API reports to a client, from `super::mod`'s
/// wire-derived limit. Duplicated rather than imported because the constant
/// lives inside a module gated to the platforms that have the listener, and
/// this test runs where that module does not exist.
const ADVERTISED_PAYLOAD: usize = 1362;
let (a, b) = dgram_pair().expect("AF_UNIX SOCK_DGRAM socketpair");
// Walk up rather than testing one size, so a failure reports the ceiling.
let mut largest = 0usize;
let mut buf = vec![0u8; ADVERTISED_PAYLOAD * 4];
for size in [64, 256, 1024, ADVERTISED_PAYLOAD, ADVERTISED_PAYLOAD * 2] {
let payload = vec![0xA5u8; size];
if send(a.as_raw_fd(), &payload).is_err() {
break;
}
match recv(b.as_raw_fd(), &mut buf) {
Ok(n) if n == size => largest = size,
_ => break,
}
}
assert!(
largest >= ADVERTISED_PAYLOAD,
"a SOCK_DGRAM pair carried at most {largest} bytes, below the \
{ADVERTISED_PAYLOAD} the API advertises to clients. On Darwin this is \
the net.local.dgram.maxdgram ceiling and the port has to raise it, or \
lower what it advertises, rather than let a client send what it was \
told it could."
);
}
#[test]
fn a_datagram_queued_before_the_close_is_still_readable() {
let (a, b) = dgram_pair().expect("AF_UNIX SOCK_DGRAM socketpair");
assert_eq!(send(a.as_raw_fd(), &[7, 7, 7]).unwrap(), 3);
drop(a);
// Data sent before the close must survive it. A kernel that discards the
// queue on close would lose a client's last datagram.
let mut buf = [0u8; 64];
assert_eq!(
recv(b.as_raw_fd(), &mut buf).unwrap(),
3,
"a datagram queued before the peer closed must still be delivered"
);
assert_eq!(&buf[..3], &[7, 7, 7]);
}
#[test]
fn an_empty_datagram_queued_before_the_close_is_not_read_as_the_close() {
let (a, b) = dgram_pair().expect("AF_UNIX SOCK_DGRAM socketpair");
assert_eq!(send(a.as_raw_fd(), &[]).unwrap(), 0);
drop(a);
// The ordering case the close rule is weakest against. The peer has closed,
// so POLLHUP is set, and the queued empty datagram also reads as zero
// bytes, so `zero && hung_up` cannot tell them apart. Whatever the kernel
// does here, the implementation has to handle it; this test records which
// kernel loses the datagram.
let mut buf = [0u8; 64];
let read = recv(b.as_raw_fd(), &mut buf);
let hup = hung_up(b.as_raw_fd());
assert!(
matches!(read, Ok(0)),
"expected the queued empty datagram to read as zero bytes, got {read:?}"
);
assert!(
!hup,
"POLLHUP is set while an empty datagram is still queued, so a zero-byte \
read plus POLLHUP cannot mean end of file on this kernel: the queued \
datagram would be swallowed and reported as a close"
);
}