Tinfoil PR #385 added model=<name> to the X-Tinfoil-Usage-Metrics
header/trailer. This commit uses that field for accurate billing.
Changes in routstr/upstream/ehbp.py:
- parse_tinfoil_usage_metrics(): extract the model= field as a string
alongside the existing token counts (previously silently discarded
because int() failed on it).
- _build_cost_info(): accept optional actual_model parameter propagated
through to callers when a real mismatch is detected.
- _compute_ehbp_actual_cost(): compare the served model against
model_obj.forwarded_model_id (the expected upstream ID) rather than
model_obj.id (the client-facing alias). This prevents spurious
mismatches when a node runner maps e.g. tinfoil-glm-5-2 -> glm-5-2
and the header correctly reports glm-5-2. On a genuine mismatch
(failover to a different upstream model), look up the actual model's
pricing via get_model_instance() (forwarded_model_id values are
registered as routable aliases in the global model map).
- forward_ehbp_request() / forward_ehbp_x_cashu_request(): use the
actual served model for payment finalization and logging when a
mismatch is detected.
Tests: 6 new scenarios (alias match, real mismatch with alias, unknown
model fallback, old-format compat, cache token details + model), plus
forwarded_model_id set on all existing mock model objects to keep them
passing. All 49 Tinfoil/EHBP unit tests pass.
The prepare_request_body override in OpenRouterUpstreamProvider injected
provider.require_parameters=true on tool-use requests. This is no longer
needed. Removed the override, the now-unused json import, and the
corresponding test suite.
The X-Cashu refund endpoint raised 404 "Refund not found" when the
"in" transaction existed with a request_id but the "out" (refund)
transaction had not been written yet. This is a timing race: the
endpoint is polled while the upstream request is still in flight,
before send_refund() has minted and stored the refund token.
The 404 was indistinguishable from a genuinely-missing refund, so
clients had no signal that retrying would succeed — leading to
stranded refunds when clients gave up.
Replace the third 404 branch with 425 Too Early + Retry-After: 2.
The two earlier 404 branches (no "in" row, no request_id) remain 404
since those genuinely mean no refund will ever exist.
Also adds debug logging on all three not-found/pending branches so
the race is no longer invisible to operators (middleware does not log
request headers).
Adds unit tests for the new 425 pending path and the no-request_id
404 path.
Refs: refund-race-condition
The X-Cashu refund endpoint raised 404 "Refund not found" when the
"in" transaction existed with a request_id but the "out" (refund)
transaction had not been written yet. This is a timing race: the
endpoint is polled while the upstream request is still in flight,
before send_refund() has minted and stored the refund token.
The 404 was indistinguishable from a genuinely-missing refund, so
clients had no signal that retrying would succeed — leading to
stranded refunds when clients gave up.
Replace the third 404 branch with 425 Too Early + Retry-After: 2.
The two earlier 404 branches (no "in" row, no request_id) remain 404
since those genuinely mean no refund will ever exist.
Also adds debug logging on all three not-found/pending branches so
the race is no longer invisible to operators (middleware does not log
request headers).
Adds unit tests for the new 425 pending path and the no-request_id
404 path.
Refs: refund-race-condition
The bare-tail tie-break ranked candidates by prompt price alone, so two
entries sharing a tail where one is cheaper on prompt but far dearer on
completion could resolve to the entry that undercharges output-heavy
traffic — contradicting the "highest-priced wins for money safety" promise.
Rank by the combined prompt + completion per-token cost instead, so the
choice stays deterministic and money-safe whichever way traffic leans. As
before there are zero bare-tail collisions in the live feed, so this changes
no resolved price today; it only governs the latent case.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The generic provider treated any Venice-style model_spec.pricing as
authoritative after only a None check, so a native both-zero price served
the model free, a negative one credited the caller on every request, and a
non-numeric string threw while parsing — the outer catch then dropped the
provider's entire catalog.
Coerce both native prices through the resolver's _as_float and reject
absent / non-numeric / negative / both-zero values, falling through to the
shared litellm→OpenRouter→fail-closed chain instead. This extends the same
money-safety guard the litellm and OpenRouter rungs already apply to the
native source, and keeps one malformed entry from emptying the catalog.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Route the Authorization: Bearer cashu… redemption failures through the
same failure taxonomy the X-Cashu path already uses (token_already_spent /
invalid_token / mint_error / cashu_error, carried in the error "type"),
with matching statuses, so both redemption paths agree on the
classification for a given failure class instead of introducing a third
parallel code.
- "already spent" -> token_already_spent (400)
- fee/melt failures -> mint_error (422)
- invalid/undecodable token -> invalid_token (400, was 401)
- other expected wallet errors -> cashu_error (400)
- unexpected faults still -> internal_error (500)
Also align the msats<=0 defense-in-depth envelope and note it is now
practically unreachable (credit_balance rejects non-positive amounts).
- Anchor the broad invalid/decode buckets to co-occur with "token" so
internal faults (e.g. "Invalid isoformat string") fall through to 500
instead of masquerading as a 401 token error.
- Map the "estimate fees"/"exceed token amount" wallet errors to the
specific "too small to cover swap fees" message instead of the generic
bucket.
- Stop the X-Cashu redemption path from interpolating raw mint/exception
text into mint_error/cashu_error responses (both duplicated blocks).
- Add tests for the fee-gap mapping and the anchored gate.
The generic provider computed modality as "text->text" for any vision model
and "text" otherwise, discarding the source's own modality and mislabelling
image-capable models. Carry OpenRouter's modality string through verbatim, and
when a source doesn't supply one, derive it from the captured input/output
modalities in the same "in->out" shape (e.g. "text+image->text").
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
When a bare model id matched several OpenRouter entries by tail, the first feed
entry won, making the resolved price depend on feed order and risking an
undercharge if a cheaper reseller happened to sort first. Pick the
highest-priced candidate instead: deterministic and money-safe, since
undercharging is the hazard. An exact id match still wins ahead of any tail
match. Measured against the live feed there are zero bare-tail collisions
today, so this only governs the latent case.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
litellm's max_tokens is the completion cap (it equals max_output_tokens for
~94% of models), not the context window, so falling back to it overstated the
context as the output limit. Take context from max_input_tokens alone; a model
that reports none falls through to the id-based estimate downstream, which is
honest about being a guess rather than mislabelling the output cap.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A litellm cost entry that lists a model but prices both tokens at 0 (free
moderation/rerank tiers) was resolved as a real price, importing the model
enabled and served for free. Reject a both-zero (or negative) litellm hit so
the caller falls through to the next source or fails closed, mirroring the
_has_valid_pricing check the OpenRouter feed already applies.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A generic (OpenAI-compatible) upstream whose /models response omits pricing
was silently defaulting to $0.001/M tokens and a 4096 context window. For a
provider like DeepSeek that reports no price, this undercharged real usage by
~280x — a direct money leak — while presenting a plausible-looking price.
Resolve each model through trust-ordered sources instead: the provider's
native schema (Venice's model_spec) first, then litellm's bundled cost map
(curated list prices), then the OpenRouter feed. Capture the richer metadata
those sources carry (cache rates, modalities, max output tokens, context)
rather than only price and context. When no source knows the model, import it
disabled with a warning rather than invent a number, so an operator can price
it before it serves traffic.
Context has no trustworthy source of last resort, but it is not a billing
input, so a model priced without a reported context window falls back to an
id-based estimate. The whole source-incomplete fallback (price chain +
context estimate) lives in one pricing_resolver module so it can later be
hoisted into the base provider unchanged.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The .well-known/ path was removed from Tinfoil attestation routing
since Tinfoil doesn't use it. The test that asserted GET /.well-known/
bypasses model lookup is no longer valid.
EHBP response bodies are opaque encrypted blobs, so cost cannot be injected
into JSON like the normal proxy flow. Instead, surface cost as response headers:
X-Routstr-Cost-Msats — total msats charged (bearer + x-cashu)
X-Routstr-Cost-Usd — USD equivalent (bearer only)
X-Routstr-Input-Cost-Msats — msats attributed to input tokens
X-Routstr-Output-Cost-Msats— msats attributed to output tokens
- Refactor _compute_ehbp_actual_cost from int→dict to carry full cost breakdown
- Add _build_cost_info() and _inject_cost_response_headers() helpers
- Capture adjust_payment_for_tokens return in bearer path (was discarded)
- Update CORS expose_headers, docs, and unit tests
11000 is nutshell's generic, unregistered TransactionError covering many
unrelated failures, so trusting the code alone made any 11000 retryable.
Gate the generic code on the 'not enough inputs' detail text; keep trusting
the registered 11005 (TransactionUnbalanced) on the code alone. Behavior is
unchanged for every existing case; a bare non-shortfall 11000 now surfaces
immediately instead of burning the retry budget.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add TinfoilUpstreamProvider that uses inference.tinfoil.sh as a direct
EHBP upstream. Routstr acts as a blind relay: it forwards the opaque
encrypted body to the Tinfoil enclave without ever seeing plaintext,
and bills from the X-Tinfoil-Usage-Metrics response header.
- New routstr/upstream/tinfoil.py: fetches models from public
GET /v1/models, parses Tinfoil pricing into standard Model/Pricing
schema, supports_ehbp=True, proxies /attestation to atc.tinfoil.sh
- Updated routstr/upstream/ehbp.py:
- parse_tinfoil_usage_metrics() parses prompt=N,completion=N header
- _resolve_ehbp_target_url() honors X-Tinfoil-Enclave-Url from SDK
- _strip_proxy_headers() removes proxy-only headers before forwarding
- _compute_ehbp_actual_cost() converts usage to msats via calculate_cost
- forward_ehbp_request() finalizes with exact token cost when usage
header is present (non-streaming), falls back to max-cost otherwise
- forward_ehbp_x_cashu_request() computes refund from actual cost
when usage is available
- Updated routstr/proxy.py: forward /attestation and /.well-known/ paths
without model/cost/auth lookups
- Updated routstr/upstream/__init__.py and helpers.py: register and
auto-seed Tinfoil provider from TINFOIL_API_KEY env var
- Updated .env.example with TINFOIL_API_KEY
- 22 new unit tests in tests/unit/test_tinfoil_integration.py
- Updated docs/tinfoil-direct-integration.md and docs/ehbp-proxy-support.md
with implementation status and billing behavior table
When an upstream reports a non-zero USD cost but the response carries no
tokens at all (input, output, cache-read and cache-creation all zero), the
USD path billed the full USD-derived amount for an empty response. Return an
all-zero cost (full refund) for that case only.
The gate is tightened relative to the superseded PR #489: a cache-read- or
cache-creation-only turn legitimately reports zero prompt/completion tokens
with a real cost and must still be billed, so the refund only fires when every
token bucket is zero.
Adds unit tests covering both the refund and the still-billed cache-only path.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
redshift
9qeklajcandGitHub
Jeroen UbbinkandClaude Opus 4.8
ShroominicandClaude Opus 4.8