Provider scoping (items 1/2/6):
- Key visibility maps on (model_id.lower(), upstream_provider_id), matching
refresh_model_maps, so a disable/override row on one provider never leaks
onto another provider's model, and matching is case-insensitive.
Data safety (items 3/5):
- Degraded OpenRouter fetches (network error, 429, non-200, bad payload)
return None (unknown) instead of []; a provider whose path set is unknown
keeps its previously persisted rows instead of being wiped.
- Endpoint payload parsing moved fully inside try, with a list guard, so
endpoints:null or non-list shapes are swallowed as documented.
- refresh with an empty live upstream list is a no-op; the unfiltered
DELETE in the prune path is gone (prune now keys off enabled DB rows).
Hot path (items 4/12/14):
- Persist uses chunked bulk INSERTs (one statement per 500 rows) instead of
per-row ORM adds; redundant ix_model_paths_model_id index dropped.
- Read routes filter in SQL instead of materializing the whole table, and
output ordering is deterministic (public id + path), independent of rowid.
- Visibility no longer rebuilds fully priced Model objects per override row;
it reads id/forwarded_model_id/canonical_slug straight off ModelRow.
Path/id contract (items 7/8/9/11):
- discovery_path_for_subprovider/discovery_base_paths hooks on
BaseUpstreamProvider, overridden by OpenRouterUpstreamProvider, mirror
_apply_provider_field so discovery and response stamping cannot drift
(openrouter:OpenRouter now correctly maps to unknown).
- openrouter_author_slug falls back to a slash-containing forwarded_model_id,
so admin-created alias rows are discoverable.
- public_model_id splits on the first slash, same as get_base_model_id, so
discovery ids can be sent to chat completions verbatim.
Lifecycle (items 10/13):
- ENABLE_MODEL_PATHS_REFRESH kill switch; interval and flag re-read every
loop iteration, and the task idles (not exits) while disabled.
- First 429 latches and aborts the remaining fan-out for the cycle; a
per-cycle cache dedupes fetches across providers sharing a base URL.
- refresh_model_maps prunes paths of disabled/deleted providers so admin
mutations take effect immediately; rows carry updated_at and both
endpoints expose it.
Tests (item 15) rewritten through the public refresh entry point with
transport-level httpx.MockTransport fakes, FK enforcement on, and coverage
for the periodic loop. Migration re-chained onto 9c4d8e2f1a6b.
Add DATABASE_POOL_SIZE / DATABASE_MAX_OVERFLOW / DATABASE_POOL_TIMEOUT,
consumed like every other typed env var through the pydantic Settings
(constrained Fields, defaults 5/10/30 matching SQLAlchemy's own baseline
so leaving them unset is behaviour-neutral). An out-of-range or
non-integer value fails validation and refuses to boot — the same
fail-loud behaviour a malformed DATABASE_URL already has — rather than
silently starting up misconfigured. create_db_engine sizes the pool from
these and logs the effective values at startup so they can be confirmed
from the boot output during an incident. In-memory SQLite (StaticPool,
which rejects the pool kwargs) is detected and built without them.
pool_pre_ping is deliberately not exposed: the default backend is a local
SQLite file with no network peer to drop idle connections, so it would add
a SELECT 1 per checkout for no benefit — and it detects dead connections,
not the live-but-wedged ones behind the exhaustion this series addresses.
These knobs are infrastructure the node needs before it can open a DB
session, so they can never be sourced from the DB (chicken-and-egg). A new
ENV_ONLY_FIELDS set keeps them out of the persisted settings blob and
stops a DB value from shadowing env in both SettingsService.initialize and
.update, so env stays authoritative.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
fetch_all_balances shared a single AsyncSession across the tasks it ran
with asyncio.gather. AsyncSession is not safe for concurrent use: the
overlapping queries raised "concurrent operations are not permitted" and
left connections wedged until the QueuePool was exhausted, after which
admin/API endpoints returned 500/502 until a container restart.
Read all outstanding user liabilities up front in one short-lived session
with a single grouped query (balances_by_mint_and_unit), then run the
per-mint balance checks concurrently with no session in scope. A failure
reading liabilities now degrades gracefully — the page still reports the
known wallet custody and blanks only the unknowable user/owner split,
tagging each mint with the error — instead of 500-ing the whole page.
periodic_payout reads each liability fresh, immediately before the payout
decision, rather than from a single pre-loop snapshot: the per-mint round
trip is slow, and a user top-up during the cycle would otherwise let a
later mint/unit act on a stale-low liability and over-send funds owed to
users (related to the payout-safety concern in issue #611).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- periodic_payout: fetch liability per mint/unit right before computing
available balance, so a concurrent top-up can only shrink the payout
- refund sweep: atomically claim each refund before redeeming; release the
claim on failure so retries still happen and concurrent sweeps cannot
misreport a sweep as client-collected
- check_invoice_payment: catch BaseException so task cancellation after a
successful mint still emits the reconciliation alert
- tests: DB-guard race test where both mints succeed (exactly one credit);
pool_size=1 test proving the fee payout releases its connection during
the external send
test_get_balance mocked Wallet.with_db but not the module-level _wallets
cache, so a real wallet cached by an earlier unmocked path (e.g. an
admin-withdraw amount-rejection test) could shadow the mock and fail the
assertion depending on collection order. Reset the cache for the test, as
the sibling wallet tests already do.
Three fixes to how a node provisions its own master key when the operator
sets no ROUTSTR_SECRET_KEY:
- Publish the key atomically. It is written to a same-directory temp file,
fsynced, then os.link-ed into place and the directory fsynced. os.link
publishes the complete file in one step, so a crash mid-write can no longer
strand an empty key at the final path that a later boot would read as corrupt
and then fail to decrypt every secret under. os.link also refuses to clobber,
so a racing worker that generated first keeps ownership and the loser adopts
its key.
- Tighten loose permissions on read. A key file that is group/other-readable is
repaired to 0600 rather than trusted, keeping an upgrading node booting.
- Stop printing the key value. The one-time notice names the file to back up and
shouts the backup imperative, but no longer echoes the key itself, which would
leak it into captured stdout / aggregated container logs; the durable 0600 file
is the recovery path.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A node with a legacy plaintext nsec but no ROUTSTR_SECRET_KEY refused to
boot: bootstrap_secrets raised and vault.encrypt required the env key.
That turned encryption at rest into a hard breaking change on auto-upgrade.
Encryption stays mandatory — the nsec is never persisted in plaintext —
but key custody becomes flexible. When no ROUTSTR_SECRET_KEY is set,
encrypt() generates a Fernet key, writes it owner-only (0600) to a key
file, and prints a one-time back-it-up notice, so an upgrading node keeps
running. The read path stays strict: decrypt()/get_fernet() never mint a
key (a fresh key could not match existing ciphertext) and fail fast with
the generation command when none is configured. A malformed env key still
fails fast rather than silently self-provisioning a different key.
The key file defaults beside the SQLite database (ROUTSTR_SECRET_KEY_FILE
overrides), so it rides whatever volume already persists the data instead
of a working-directory path a container recreate would drop.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Two ways a stale legacy NSEC could override or resurrect an nsec the
vault already owns (issue #553):
- initialize() re-applied env/blob values onto live settings after
bootstrap had decrypted the authoritative nsec, so a stale NSEC left
in .env would clobber it on restart (e.g. after rotating the key in
the admin UI). _apply_to_live_settings now never re-applies secret
fields; bootstrap_secrets is their only writer.
- An empty encrypted_nsec could not distinguish "never migrated" from
"intentionally cleared", so clearing the identity via the admin API
and restarting re-imported the old NSEC from env/blob. Record vault
ownership in a new secrets.nsec_managed column (set on legacy import
and on every set_nsec write); bootstrap skips the legacy import once
the vault owns the nsec, so a cleared identity stays cleared.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
upstream_api_key was added to SECRET_FIELDS, so it was stripped from every
blob write — but unlike nsec, nothing migrates it into encrypted storage. A
node carrying it only in the DB blob would load it into memory once, rewrite
the blob without it, and lose it on the next restart, breaking upstream auth.
It is node-scoped config that really belongs on a provider, not a vault
secret, and it has no encrypted home yet. Remove it from SECRET_FIELDS so it
stays in the blob exactly as before; redaction on read and ignore-on-write in
the admin settings endpoint are unchanged. Encrypting it is follow-up work.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Persist and load admin password and nsec from the encrypted Secret store on
boot: generate a temporary admin password on first run (logged once), encrypt
a provided nsec, and fail fast if a stored nsec cannot be decrypted with the
current key. Stop clobbering live secret settings with empty env values.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Encrypt/decrypt secrets at rest with Fernet keyed by ROUTSTR_SECRET_KEY,
and hash/verify admin passwords with scrypt. Self-contained helpers with
no consumers yet.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Make model_obj/provider_fee required (still nullable) on
adjust_payment_for_tokens, get_x_cashu_cost and the private pricing
helpers so a call site that fails to thread the served candidate is a
type error instead of a silent fallback to alias-map re-derivation.
calculate_cost keeps its defaults as the one documented fallback seam.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The failover loop resolved a single Model for the request and reused it
for every provider: a fallback provider was asked to serve the routing
winner's model id and billed at the winner's pricing and fee. The alias
map now keeps (model, provider) candidate pairs, the proxy rebinds both
per attempt, and forwarding, max-cost echo, and settlement all use the
candidate actually being tried. On a failover serve the response's
model field now names the serving candidate's id.
The unified candidate lookup also applies the version-suffix strip
(-YYYYMMDD) that model resolution already had, so version-suffixed
requests no longer resolve a model yet 400 with "no provider found".
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The USD-cost path (and the litellm pricing fallback) resolved the
provider fee via get_provider_for_model(model_id)[0] — the best-ranked
provider for the alias, not the one that served. Settlement callers in
the upstream handlers now pass their own provider_fee through
adjust_payment_for_tokens / get_x_cashu_cost into calculate_cost; the
string-derived fallback remains for callers without a serving provider.
Configured model pricing is unaffected (the fee is already baked into
cached pricing).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
X-Cashu handlers do not rewrite the upstream's echoed model string, so
get_x_cashu_cost previously priced whatever wire name the upstream
reported — the most collapse-prone alias lookup of all. The routed Model
is now threaded from forward_x_cashu_request through the chat and
Responses handler chains (and the litellm messages path) into
get_x_cashu_cost, so cost and refund are computed from the model that
actually served.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Settlement previously re-derived pricing from the response's model string
through the alias map, which resolves to the best-ranked candidate for
that alias — not necessarily the provider/model that actually served the
request. adjust_payment_for_tokens and calculate_cost now accept the
routed Model and bill its pricing directly; the string lookup remains as
a warning fallback for callers without routed identity (e.g. the generic
streaming finalizer).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
9qeklajc
Jeroen UbbinkandClaude Opus 4.8
