mirror of
https://github.com/jmcorgan/fips.git
synced 2026-07-30 19:46:15 +00:00
Add a feature-gated profiler for the rx-loop maintenance tick
The tick arm runs twenty-six housekeeping steps in sequence on the one runtime thread, and is polled last, so anything slow in it holds up inbound packets, TUN traffic and control commands behind it. Field evidence says that happens for over a second at a time, but the attribution behind that is two months old and predates the connect-on-send gate, the control read isolation, and the peer lifecycle rework. This measures it rather than continuing to reason about it. Per step it records exact count, max and total into fixed static counters; a dedicated writer thread drains them every ten seconds to a TSV under /var/log/fips, one file per capture, capped at 32 MB. Nothing accumulates: the counters are swapped to zero each interval and the thread holds no history. Arming is `fipsctl profile tick on`, served in the control accept task so the toggle cannot queue behind the very behaviour it measures, and it does not survive a restart. The whole thing is behind a Cargo feature that is off by default, because the risk worth eliminating is the twenty-six edited call sites in the hot loop. With the feature off the macro expands to the bare expression, which makes the default build's neutrality something you read off the generated code rather than something a benchmark fails to disprove. The measurement that matters is how late each tick is against the deadline it was scheduled for, since the arm is polled last and that lateness is the delay. Two earlier designs derived it from the interval between entries and both under-reported: the schedule is fixed, so a steady delay leaves every gap exactly one period and any gap-derived figure reads zero under precisely the sustained overload this is meant to find. The interval hands back its own deadline, so the delay is now a subtraction with no model behind it, and a test drives three late ticks at a constant gap to keep it that way. CI gains a default-features clippy and a feature-on build and test on both runners, closing the gap left by clippy already running with all features.
This commit is contained in:
@@ -115,6 +115,58 @@ Tell the daemon to drop a peer link.
|
||||
| -------- | ----------- |
|
||||
| `peer` | npub (bech32) or hostname from `/etc/fips/hosts`. |
|
||||
|
||||
### `profile tick <on|off|status>`
|
||||
|
||||
> **Reading the output.** Step durations are wall clock measured across `await`
|
||||
> points, not CPU time: a step that waits on I/O accrues that wait, and other
|
||||
> tasks may run inside the span. That is the intended measure for head-of-line
|
||||
> delay, and it means a large step is not necessarily an expensive one.
|
||||
> `arm_starvation` is measured directly as the entry time minus the deadline
|
||||
> the interval scheduled that tick for. It is not derived from
|
||||
> `tick_entry_gap`, which carries no starvation signal on its own: under a
|
||||
> steady delay every gap is exactly one tick period.
|
||||
|
||||
Start, stop and inspect a capture of the rx-loop tick body. **Present
|
||||
only when both `fipsctl` and the daemon are built with
|
||||
`--features profiling`**; the feature is off by default, so a stock
|
||||
package does not carry this subcommand and a stock daemon reports
|
||||
`profile_tick_*` as an unknown command.
|
||||
|
||||
| Subcommand | Control-socket command | Description |
|
||||
| ---------- | ---------------------- | ----------- |
|
||||
| `profile tick on` | `profile_tick_on` | Create the capture file and start recording. Fails if a capture is already running (naming the active file) or if the directory cannot be written. |
|
||||
| `profile tick off` | `profile_tick_off` | Stop the capture. The writer is woken immediately, drains once more and is joined, so the command returns promptly. Succeeds, reporting nothing active, when no capture is running. |
|
||||
| `profile tick status` | `profile_tick_status` | Report `idle`, `running`, `stopped_by_cap` or `stopped_by_error`, plus the active path, bytes written, flush interval and byte cap. |
|
||||
|
||||
`profile tick on` options:
|
||||
|
||||
| Flag | Argument | Default | Description |
|
||||
| ---- | -------- | ------- | ----------- |
|
||||
| `--dir` | directory path | `/var/log/fips` | Where to write the capture. Created if absent. Use it to profile a non-root `cargo run`, or on a platform whose log root differs. |
|
||||
|
||||
One file is written per capture, named `profile-<UTC timestamp>.tsv`.
|
||||
It opens with a `#`-prefixed header block (node npub, build version,
|
||||
platform, configured tick period, flush interval, byte cap, start
|
||||
time), then a tab-separated column header, then one row per measured
|
||||
step per flush interval:
|
||||
|
||||
```text
|
||||
ts_unix kind domain name count max total unit
|
||||
```
|
||||
|
||||
`kind` is `step` for a timed span and `gauge` for a sampled scalar, so
|
||||
a gauge value never lands under a duration column; `unit` names the
|
||||
unit of `max` and `total` for that row. Every step present in the build
|
||||
gets a row every interval, including zero-count rows. Gauges cover
|
||||
ticks per interval, peer count, the wall gap between successive
|
||||
tick-arm entries, and the arm-starvation delay, which is measured
|
||||
against the deadline the tick was scheduled for rather than derived
|
||||
from the gap.
|
||||
|
||||
A capture stops itself on reaching 32 MB, appending a `#` line saying
|
||||
so; `profile tick status` then reports `stopped_by_cap` until the next
|
||||
`on` or `off` clears it.
|
||||
|
||||
## Exit Codes
|
||||
|
||||
| Code | Meaning |
|
||||
|
||||
@@ -159,6 +159,21 @@ not reproduced here to avoid duplicating the source.
|
||||
Both commands run on the daemon's main task and may block briefly
|
||||
while the node mutates its state.
|
||||
|
||||
#### Profiler toggle (`--features profiling` builds only)
|
||||
|
||||
| Command | Params | Behaviour |
|
||||
| ------- | ------ | --------- |
|
||||
| `profile_tick_on` | `dir` (optional directory path; default `/var/log/fips`) | Creates the capture file, publishes its path, and starts the writer thread. `data`: `state`, `path`, `interval_secs`, `byte_cap`. Errors if a capture is already running (naming the active file) or the directory is unwritable. |
|
||||
| `profile_tick_off` | — | Stops the capture, drains once more, joins the writer. `data`: `state`, `stopped`, `stopped_by_cap`, `stopped_by_error`, `path`, `bytes`. |
|
||||
| `profile_tick_status` | — | `data`: `state` (`idle` / `running` / `stopped_by_cap` / `stopped_by_error`), `path`, `bytes`, `byte_cap`, `interval_secs`. |
|
||||
|
||||
Unlike `connect` and `disconnect`, these three are served in the
|
||||
control accept task rather than on the daemon's main task. All of their
|
||||
state is process statics and none of them needs `&mut Node`, so
|
||||
routing them through the main loop would only make the toggle queue
|
||||
behind the tick body it exists to measure. They are absent from a
|
||||
default build, where the daemon answers them as unknown commands.
|
||||
|
||||
## Gateway Command Catalog
|
||||
|
||||
`fips-gateway` exposes a separate control socket with its own command
|
||||
|
||||
Reference in New Issue
Block a user