A live upstream provider re-finds its own database row in two places —
PPQ.AI's insufficient-balance self-disable and the base
refresh_models_cache — with WHERE base_url == self.base_url AND
api_key == self.api_key. That uses a rotatable secret as a self-handle:
if the row's key rotates under a live object, it can no longer find
itself.
Carry the row's primary key on the instance as db_id, stamped centrally
by from_db_row via a _build_from_row construction hook that subclasses
override, and look the row up with session.get(UpstreamProviderRow,
db_id). This also closes a latent gap where providers built outside the
init path (auto-topup) never received db_id.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Browser clients need these EHBP protocol headers visible to JavaScript
so the Tinfoil SDK can detect and decrypt encrypted responses. Without
them, CORS hides the headers, the SDK treats the response as a plaintext
proxy error, and users see 'The provider did not respond to this request.'
Node.js scripts are unaffected (no CORS enforcement).
See ../routstr-chat/TINFOIL_CORS_ISSUE.md for full root-cause analysis.
The ui/README.md was still stock create-next-app boilerplate. Replace it with the
real story: the UI is a Next.js static export served by FastAPI from ui_out/, the
two-process dev loop (backend :8000 + `make ui-dev` :3000 with hot reload, auto-
targeting :8000), the build/integration commands, and the cross-origin CORS note.
Also make the "ui_out not found" startup warning actionable — it now points to
`make ui-build` / `make ui-dev` instead of silently saying it skipped serving.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
WHERE guard now matches on hashed_key, so the UPDATE always affects one
row and result.rowcount is always truthy. Remove the conditional and its
unreachable else branch.
EHBP response bodies are opaque encrypted blobs, so cost cannot be injected
into JSON like the normal proxy flow. Instead, surface cost as response headers:
X-Routstr-Cost-Msats — total msats charged (bearer + x-cashu)
X-Routstr-Cost-Usd — USD equivalent (bearer only)
X-Routstr-Input-Cost-Msats — msats attributed to input tokens
X-Routstr-Output-Cost-Msats— msats attributed to output tokens
- Refactor _compute_ehbp_actual_cost from int→dict to carry full cost breakdown
- Add _build_cost_info() and _inject_cost_response_headers() helpers
- Capture adjust_payment_for_tokens return in bearer path (was discarded)
- Update CORS expose_headers, docs, and unit tests
11000 is nutshell's generic, unregistered TransactionError covering many
unrelated failures, so trusting the code alone made any 11000 retryable.
Gate the generic code on the 'not enough inputs' detail text; keep trusting
the registered 11005 (TransactionUnbalanced) on the code alone. Behavior is
unchanged for every existing case; a bare non-shortfall 11000 now surfaces
immediately instead of burning the retry budget.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The Tinfoil SDK expects chunked transfer encoding for streaming
responses. Returning Response with content=body caused the stream to
terminate abruptly instead of ending gracefully. Use StreamingResponse
with a generator that yields the buffered body so the client gets
proper chunked transfer encoding while still benefiting from trailer
capture and exact billing.
Tinfoil returns X-Tinfoil-Usage-Metrics as an HTTP trailer on streaming
responses, but httpx/httpcore silently discard trailers during chunked
transfer decoding. This caused all streaming EHBP requests to fall back
to max-cost billing instead of charging actual token usage.
- Add routstr/upstream/tinfoil_trailer.py: h11-based HTTP client that
preserves trailers via the EndOfMessage event (httpx discards them)
- Rewrite forward_ehbp_request and forward_ehbp_x_cashu_request to use
forward_with_trailer instead of httpx
- Add _extract_usage_from_response() to check both response headers
(non-streaming) and trailers (streaming) for usage metrics
- Buffer EHBP response bodies (acceptable since they are opaque
encrypted blobs the client decrypts regardless)
- Both bearer and X-Cashu paths now bill based on actual token usage
for both streaming and non-streaming requests
The docstring claimed X-Tinfoil-Enclave-Url is honored for GET requests,
but the implementation delegates to the base class which builds URLs from
self.base_url. X-Tinfoil-Enclave-Url is an EHBP-only header for encrypted
POST requests and is not used for unencrypted GETs.
Add TinfoilUpstreamProvider that uses inference.tinfoil.sh as a direct
EHBP upstream. Routstr acts as a blind relay: it forwards the opaque
encrypted body to the Tinfoil enclave without ever seeing plaintext,
and bills from the X-Tinfoil-Usage-Metrics response header.
- New routstr/upstream/tinfoil.py: fetches models from public
GET /v1/models, parses Tinfoil pricing into standard Model/Pricing
schema, supports_ehbp=True, proxies /attestation to atc.tinfoil.sh
- Updated routstr/upstream/ehbp.py:
- parse_tinfoil_usage_metrics() parses prompt=N,completion=N header
- _resolve_ehbp_target_url() honors X-Tinfoil-Enclave-Url from SDK
- _strip_proxy_headers() removes proxy-only headers before forwarding
- _compute_ehbp_actual_cost() converts usage to msats via calculate_cost
- forward_ehbp_request() finalizes with exact token cost when usage
header is present (non-streaming), falls back to max-cost otherwise
- forward_ehbp_x_cashu_request() computes refund from actual cost
when usage is available
- Updated routstr/proxy.py: forward /attestation and /.well-known/ paths
without model/cost/auth lookups
- Updated routstr/upstream/__init__.py and helpers.py: register and
auto-seed Tinfoil provider from TINFOIL_API_KEY env var
- Updated .env.example with TINFOIL_API_KEY
- 22 new unit tests in tests/unit/test_tinfoil_integration.py
- Updated docs/tinfoil-direct-integration.md and docs/ehbp-proxy-support.md
with implementation status and billing behavior table
- Add supports_ehbp + get_ehbp_forwarding_target hooks to BaseUpstreamProvider,
keeping base.py minimal (no large forwarding methods)
- Create routstr/upstream/ehbp.py with forward_ehbp_request and
forward_ehbp_x_cashu_request helpers, plus max-cost finalization
- PPQAIUpstreamProvider: set supports_ehbp = True, implement target to
/private/v1/... with X-Private-Model header
- proxy.py: filter to EHBP-capable providers, route to ehbp helpers
- Fix bearer EHBP payment: reserve upfront, finalize max cost on success
- Fix X-Cashu EHBP: refund full token on failure, refund excess on success
- Update docs/ehbp-proxy-support.md to reflect new architecture
When an upstream reports a non-zero USD cost but the response carries no
tokens at all (input, output, cache-read and cache-creation all zero), the
USD path billed the full USD-derived amount for an empty response. Return an
all-zero cost (full refund) for that case only.
The gate is tightened relative to the superseded PR #489: a cache-read- or
cache-creation-only turn legitimately reports zero prompt/completion tokens
with a real cost and must still be billed, so the refund only fires when every
token bucket is zero.
Adds unit tests covering both the refund and the still-billed cache-only path.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
9qeklajcandGitHub
Jeroen UbbinkandClaude Opus 4.8
redshift
ShroominicandClaude Opus 4.8