mirror of
https://github.com/Routstr/routstr-core.git
synced 2026-08-06 17:54:37 +00:00
Compare commits
5
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
14785e4cde | ||
|
|
1dceffffa7 | ||
|
|
4b74a9cf81 | ||
|
|
3be2da6728 | ||
|
|
17d690e77e |
+20
-87
@@ -113,109 +113,42 @@ All errors follow a consistent JSON structure:
|
|||||||
**Status:** 402
|
**Status:** 402
|
||||||
**Resolution:** Top up API key balance
|
**Resolution:** Top up API key balance
|
||||||
|
|
||||||
### Cashu Token Redemption Errors
|
#### Invalid Token
|
||||||
|
|
||||||
These errors are returned when a Cashu token you pay with cannot be redeemed.
|
|
||||||
They apply to every endpoint that accepts a token:
|
|
||||||
|
|
||||||
- **Per-request payment** via the `X-Cashu` header (chat completions + Responses API).
|
|
||||||
- **API key top-up** via `POST /v1/wallet/topup`.
|
|
||||||
- **Minting an API key** from a token sent in `Authorization: Bearer <cashu-token>`.
|
|
||||||
|
|
||||||
All three share one classifier, so the same failure yields the same HTTP status
|
|
||||||
and sanitized message everywhere. Structured error envelopes (`X-Cashu` and
|
|
||||||
`Authorization: Bearer <cashu-token>`) also expose the same `type` and `code` —
|
|
||||||
branch on `type` (or `code` for finer granularity). `POST /v1/wallet/topup`
|
|
||||||
keeps its existing plain-string `detail` envelope, so branch on status there.
|
|
||||||
|
|
||||||
| `type` | Status | `code` | Retryable | Meaning |
|
|
||||||
|--------|--------|--------|-----------|---------|
|
|
||||||
| `token_already_spent` | 400 | `cashu_token_already_spent` | No | The token was already redeemed. |
|
|
||||||
| `invalid_token` | 400 | `invalid_cashu_token` | No | The token is malformed or cannot be decoded. |
|
|
||||||
| `mint_error` | 422 | `cashu_token_swap_fees_exceed_amount` | No | Token value is too small to cover the mint's swap/melt fees. |
|
|
||||||
| `mint_error` | 422 | `cashu_foreign_mint_swap_failed` | No | Swapping the token from a foreign mint to the primary mint failed. |
|
|
||||||
| `mint_unreachable` | 503 | `cashu_mint_unreachable` | **Yes** | The mint could not be reached (DNS failure, refused/reset connection, timeout). The token is fine — retry once the mint recovers. |
|
|
||||||
| `cashu_error` | 400 | `cashu_token_redemption_failed` | No | The token could not be redeemed for another expected reason. |
|
|
||||||
| `cashu_error` | 400 | `cashu_token_zero_value` | No | The token redeemed to zero (empty/dust token, or value fully consumed by fees). |
|
|
||||||
| `token_consumed` | 500 | `cashu_token_consumed` | No | The token was **spent** (melted/redeemed) but crediting it then failed. Do not retry — the token is gone; contact support to reconcile. |
|
|
||||||
| `api_error` | 500 | `internal_error` | Maybe | Unexpected server-side fault during redemption. |
|
|
||||||
|
|
||||||
!!! important "Retry only `mint_unreachable`"
|
|
||||||
Only `mint_unreachable` (503) means the same token will work again later —
|
|
||||||
everything else is a permanent property of the token and must not be
|
|
||||||
blindly retried. Use exponential backoff for the 503. In particular, a
|
|
||||||
`token_consumed` 500 means the mint already spent the token, so a retry
|
|
||||||
would fail as `token_already_spent`.
|
|
||||||
|
|
||||||
#### Mint Unreachable (retryable)
|
|
||||||
|
|
||||||
```json
|
```json
|
||||||
{
|
{
|
||||||
"error": {
|
"error": {
|
||||||
"type": "mint_unreachable",
|
"type": "payment_error",
|
||||||
"message": "Cashu mint is unreachable",
|
"message": "Invalid Cashu token",
|
||||||
"code": "cashu_mint_unreachable"
|
"code": "invalid_token",
|
||||||
|
"details": {
|
||||||
|
"reason": "Token already spent"
|
||||||
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
```
|
```
|
||||||
|
|
||||||
**Status:** 503
|
**Status:** 400
|
||||||
|
**Resolution:** Use a valid, unspent token
|
||||||
|
|
||||||
**Resolution:** The token is valid — the mint is temporarily down. Retry with
|
#### Mint Unavailable
|
||||||
backoff, or pay with a token from a different mint.
|
|
||||||
|
|
||||||
#### Token Already Spent
|
|
||||||
|
|
||||||
```json
|
```json
|
||||||
{
|
{
|
||||||
"error": {
|
"error": {
|
||||||
"type": "token_already_spent",
|
"type": "payment_error",
|
||||||
"message": "Cashu token already spent",
|
"message": "Cannot connect to Cashu mint",
|
||||||
"code": "cashu_token_already_spent"
|
"code": "mint_unavailable",
|
||||||
|
"details": {
|
||||||
|
"mint_url": "https://mint.example.com",
|
||||||
|
"retry_after": 60
|
||||||
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
```
|
```
|
||||||
|
|
||||||
**Status:** 400
|
**Status:** 503
|
||||||
|
**Resolution:** Try again later or use different mint
|
||||||
**Resolution:** Use a fresh, unspent token. Do not retry with the same token.
|
|
||||||
|
|
||||||
#### Response envelope differs by endpoint
|
|
||||||
|
|
||||||
The `error` object above is identical everywhere, but the surrounding envelope
|
|
||||||
depends on how you paid:
|
|
||||||
|
|
||||||
- **`X-Cashu` header payments** (chat + Responses API) return the object at the
|
|
||||||
top level, alongside a `request_id`:
|
|
||||||
|
|
||||||
```json
|
|
||||||
{
|
|
||||||
"error": { "type": "mint_unreachable", "message": "Cashu mint is unreachable", "code": "cashu_mint_unreachable" },
|
|
||||||
"request_id": "req-abc123"
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
The original token is echoed back in the `X-Cashu` **response header only when
|
|
||||||
it is still spendable** (e.g. `mint_unreachable`, `invalid_cashu_token`, fee
|
|
||||||
errors) so you can recover/retry it. It is **not** echoed for spent/consumed
|
|
||||||
tokens (`cashu_token_already_spent`, `cashu_token_consumed`,
|
|
||||||
`cashu_token_zero_value`, `internal_error`) — retrying those can never succeed.
|
|
||||||
|
|
||||||
- **`Authorization: Bearer <cashu-token>`** (API key minting) wraps it in
|
|
||||||
FastAPI's `detail` field:
|
|
||||||
|
|
||||||
```json
|
|
||||||
{ "detail": { "error": { "type": "mint_unreachable", "message": "Cashu mint is unreachable", "code": "cashu_mint_unreachable" } } }
|
|
||||||
```
|
|
||||||
|
|
||||||
- **`POST /v1/wallet/topup`** returns a plain string message under `detail` —
|
|
||||||
it carries the shared HTTP **status** and **message** (e.g. `503` for an
|
|
||||||
unreachable mint) but not the structured `type`/`code`, so branch on the
|
|
||||||
status code here:
|
|
||||||
|
|
||||||
```json
|
|
||||||
{ "detail": "Cashu mint is unreachable" }
|
|
||||||
```
|
|
||||||
|
|
||||||
### Validation Errors
|
### Validation Errors
|
||||||
|
|
||||||
@@ -414,7 +347,7 @@ class ErrorHandler:
|
|||||||
'rate_limit',
|
'rate_limit',
|
||||||
'upstream_timeout',
|
'upstream_timeout',
|
||||||
'model_overloaded',
|
'model_overloaded',
|
||||||
'cashu_mint_unreachable'
|
'mint_unavailable'
|
||||||
}
|
}
|
||||||
|
|
||||||
# Errors requiring user action
|
# Errors requiring user action
|
||||||
|
|||||||
@@ -1,88 +0,0 @@
|
|||||||
"""add slug to upstream_providers
|
|
||||||
|
|
||||||
Revision ID: c6d7e8f9a0b1
|
|
||||||
Revises: b5e7c9d1f3a2
|
|
||||||
Create Date: 2026-06-29 00:00:00.000000
|
|
||||||
"""
|
|
||||||
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
import sqlalchemy as sa
|
|
||||||
from alembic import op
|
|
||||||
|
|
||||||
from routstr.core.provider_slugs import provider_slug_base, provider_slug_candidate
|
|
||||||
|
|
||||||
revision = "c6d7e8f9a0b1"
|
|
||||||
down_revision = "b5e7c9d1f3a2"
|
|
||||||
branch_labels = None
|
|
||||||
depends_on = None
|
|
||||||
|
|
||||||
|
|
||||||
def _allocate_backfill_slug(provider_type: str, reserved_slugs: set[str]) -> str:
|
|
||||||
base = provider_slug_base(provider_type)
|
|
||||||
suffix_number = 1
|
|
||||||
while True:
|
|
||||||
candidate = provider_slug_candidate(base, suffix_number)
|
|
||||||
if candidate not in reserved_slugs:
|
|
||||||
reserved_slugs.add(candidate)
|
|
||||||
return candidate
|
|
||||||
suffix_number += 1
|
|
||||||
|
|
||||||
|
|
||||||
def _backfill_provider_slugs(conn: sa.Connection) -> None:
|
|
||||||
existing_rows = conn.execute(
|
|
||||||
sa.text(
|
|
||||||
"SELECT slug FROM upstream_providers "
|
|
||||||
"WHERE slug IS NOT NULL AND slug != ''"
|
|
||||||
)
|
|
||||||
)
|
|
||||||
reserved_slugs = {str(row.slug).lower() for row in existing_rows}
|
|
||||||
|
|
||||||
rows_to_backfill = conn.execute(
|
|
||||||
sa.text(
|
|
||||||
"SELECT id, provider_type FROM upstream_providers "
|
|
||||||
"WHERE slug IS NULL OR slug = '' "
|
|
||||||
"ORDER BY id"
|
|
||||||
)
|
|
||||||
)
|
|
||||||
for row in rows_to_backfill:
|
|
||||||
slug = _allocate_backfill_slug(str(row.provider_type), reserved_slugs)
|
|
||||||
conn.execute(
|
|
||||||
sa.text("UPDATE upstream_providers SET slug = :slug WHERE id = :id"),
|
|
||||||
{"slug": slug, "id": row.id},
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
def upgrade() -> None:
|
|
||||||
conn = op.get_bind()
|
|
||||||
inspector = sa.inspect(conn)
|
|
||||||
columns = {c["name"] for c in inspector.get_columns("upstream_providers")}
|
|
||||||
|
|
||||||
if "slug" not in columns:
|
|
||||||
op.add_column(
|
|
||||||
"upstream_providers",
|
|
||||||
sa.Column("slug", sa.String(), nullable=True),
|
|
||||||
)
|
|
||||||
|
|
||||||
_backfill_provider_slugs(conn)
|
|
||||||
|
|
||||||
existing_indexes = {idx["name"] for idx in inspector.get_indexes("upstream_providers")}
|
|
||||||
if "ix_upstream_providers_slug" not in existing_indexes:
|
|
||||||
op.create_index(
|
|
||||||
"ix_upstream_providers_slug",
|
|
||||||
"upstream_providers",
|
|
||||||
["slug"],
|
|
||||||
unique=True,
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
def downgrade() -> None:
|
|
||||||
conn = op.get_bind()
|
|
||||||
inspector = sa.inspect(conn)
|
|
||||||
existing_indexes = {idx["name"] for idx in inspector.get_indexes("upstream_providers")}
|
|
||||||
if "ix_upstream_providers_slug" in existing_indexes:
|
|
||||||
op.drop_index("ix_upstream_providers_slug", table_name="upstream_providers")
|
|
||||||
|
|
||||||
columns = {c["name"] for c in inspector.get_columns("upstream_providers")}
|
|
||||||
if "slug" in columns:
|
|
||||||
op.drop_column("upstream_providers", "slug")
|
|
||||||
+60
-140
@@ -20,11 +20,7 @@ from .payment.cost_calculation import (
|
|||||||
MaxCostData,
|
MaxCostData,
|
||||||
calculate_cost,
|
calculate_cost,
|
||||||
)
|
)
|
||||||
from .wallet import (
|
from .wallet import credit_balance, deserialize_token_from_string
|
||||||
classify_redemption_error,
|
|
||||||
credit_balance,
|
|
||||||
deserialize_token_from_string,
|
|
||||||
)
|
|
||||||
|
|
||||||
logger = get_logger(__name__)
|
logger = get_logger(__name__)
|
||||||
payments_logger = get_logger("routstr.payments")
|
payments_logger = get_logger("routstr.payments")
|
||||||
@@ -82,37 +78,6 @@ async def check_and_reset_limit(key: ApiKey, session: AsyncSession) -> bool:
|
|||||||
return False
|
return False
|
||||||
|
|
||||||
|
|
||||||
def redemption_error_to_http_exception(error: Exception) -> HTTPException:
|
|
||||||
"""Map a Cashu token redemption failure to a sanitized client-facing error.
|
|
||||||
|
|
||||||
Thin wrapper over the shared :func:`classify_redemption_error` so the bearer
|
|
||||||
path stays identical to the X-Cashu and top-up paths.
|
|
||||||
"""
|
|
||||||
classified = classify_redemption_error(error)
|
|
||||||
if classified is None:
|
|
||||||
return HTTPException(
|
|
||||||
status_code=500,
|
|
||||||
detail={
|
|
||||||
"error": {
|
|
||||||
"message": "Internal error during token redemption",
|
|
||||||
"type": "api_error",
|
|
||||||
"code": "internal_error",
|
|
||||||
}
|
|
||||||
},
|
|
||||||
)
|
|
||||||
error_type, status_code, message, error_code = classified
|
|
||||||
return HTTPException(
|
|
||||||
status_code=status_code,
|
|
||||||
detail={
|
|
||||||
"error": {
|
|
||||||
"message": message,
|
|
||||||
"type": error_type,
|
|
||||||
"code": error_code,
|
|
||||||
}
|
|
||||||
},
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
async def validate_bearer_key(
|
async def validate_bearer_key(
|
||||||
bearer_key: str,
|
bearer_key: str,
|
||||||
session: AsyncSession,
|
session: AsyncSession,
|
||||||
@@ -251,17 +216,7 @@ async def validate_bearer_key(
|
|||||||
|
|
||||||
try:
|
try:
|
||||||
hashed_key = hashlib.sha256(bearer_key.encode()).hexdigest()
|
hashed_key = hashlib.sha256(bearer_key.encode()).hexdigest()
|
||||||
try:
|
token_obj = deserialize_token_from_string(bearer_key)
|
||||||
token_obj = deserialize_token_from_string(bearer_key)
|
|
||||||
except Exception as decode_error:
|
|
||||||
# A malformed token is a bad token (400 invalid_cashu_token via
|
|
||||||
# the shared taxonomy), not an auth failure (401) — otherwise it
|
|
||||||
# would fall through to the generic "Invalid API key" handler.
|
|
||||||
raise redemption_error_to_http_exception(
|
|
||||||
ValueError(
|
|
||||||
f"Invalid Cashu token: could not decode token ({decode_error})"
|
|
||||||
)
|
|
||||||
) from decode_error
|
|
||||||
logger.debug(
|
logger.debug(
|
||||||
"Generated token hash", extra={"hash_preview": hashed_key[:16] + "..."}
|
"Generated token hash", extra={"hash_preview": hashed_key[:16] + "..."}
|
||||||
)
|
)
|
||||||
@@ -379,32 +334,19 @@ async def validate_bearer_key(
|
|||||||
"error_type": type(credit_error).__name__,
|
"error_type": type(credit_error).__name__,
|
||||||
},
|
},
|
||||||
)
|
)
|
||||||
await session.rollback()
|
raise credit_error
|
||||||
raise redemption_error_to_http_exception(credit_error) from credit_error
|
|
||||||
|
|
||||||
if msats <= 0:
|
if msats <= 0:
|
||||||
logger.error(
|
logger.error(
|
||||||
"Token redemption returned zero or negative amount",
|
"Token redemption returned zero or negative amount",
|
||||||
extra={"msats": msats, "key_hash": hashed_key[:8] + "..."},
|
extra={"msats": msats, "key_hash": hashed_key[:8] + "..."},
|
||||||
)
|
)
|
||||||
# Defense-in-depth: credit_balance already raises
|
# Defense-in-depth: credit_balance now refuses to commit on a
|
||||||
# ValueError("Redeemed token amount must be positive…") before
|
# zero/negative redemption, but if a row was nonetheless
|
||||||
# returning (wallet.py), so this branch is only reachable if a
|
# persisted, drop it so we never leave an orphan zero-balance key.
|
||||||
# zero/negative row was somehow persisted; drop it so we never
|
|
||||||
# leave an orphan zero-balance key. Reuse the shared taxonomy
|
|
||||||
# (cashu_error) so the envelope matches the mapper above.
|
|
||||||
await session.delete(new_key)
|
await session.delete(new_key)
|
||||||
await session.commit()
|
await session.commit()
|
||||||
raise HTTPException(
|
raise Exception("Token redemption failed")
|
||||||
status_code=400,
|
|
||||||
detail={
|
|
||||||
"error": {
|
|
||||||
"message": "Failed to redeem Cashu token: token yielded no value",
|
|
||||||
"type": "cashu_error",
|
|
||||||
"code": "cashu_token_zero_value",
|
|
||||||
}
|
|
||||||
},
|
|
||||||
)
|
|
||||||
|
|
||||||
await session.refresh(new_key)
|
await session.refresh(new_key)
|
||||||
await session.commit()
|
await session.commit()
|
||||||
@@ -422,7 +364,6 @@ async def validate_bearer_key(
|
|||||||
except HTTPException:
|
except HTTPException:
|
||||||
raise
|
raise
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
await session.rollback()
|
|
||||||
logger.error(
|
logger.error(
|
||||||
"Cashu token redemption failed",
|
"Cashu token redemption failed",
|
||||||
extra={
|
extra={
|
||||||
@@ -437,7 +378,7 @@ async def validate_bearer_key(
|
|||||||
status_code=401,
|
status_code=401,
|
||||||
detail={
|
detail={
|
||||||
"error": {
|
"error": {
|
||||||
"message": "Invalid or expired Cashu key",
|
"message": f"Invalid or expired Cashu key: {str(e)}",
|
||||||
"type": "invalid_request_error",
|
"type": "invalid_request_error",
|
||||||
"code": "invalid_api_key",
|
"code": "invalid_api_key",
|
||||||
}
|
}
|
||||||
@@ -1079,37 +1020,32 @@ async def adjust_payment_for_tokens(
|
|||||||
# actual cost exceeded discounted reservation (due to tolerance_percentage)
|
# actual cost exceeded discounted reservation (due to tolerance_percentage)
|
||||||
if cost_difference > 0:
|
if cost_difference > 0:
|
||||||
# Always release the reservation and charge min(actual_cost, balance).
|
# Always release the reservation and charge min(actual_cost, balance).
|
||||||
# CASE expressions keep this atomic and safe even when the
|
# Using a CASE expression makes this a single atomic UPDATE — no
|
||||||
# stale-reservation sweeper has already released the reservation.
|
# multi-level fallback needed and balance can never go negative.
|
||||||
chargeable = case(
|
chargeable = case(
|
||||||
(col(ApiKey.balance) >= total_cost_msats, total_cost_msats),
|
(col(ApiKey.balance) >= total_cost_msats, total_cost_msats),
|
||||||
else_=col(ApiKey.balance),
|
else_=col(ApiKey.balance),
|
||||||
)
|
)
|
||||||
overrun_safe_reserved = case(
|
|
||||||
(
|
|
||||||
col(ApiKey.reserved_balance) >= deducted_max_cost,
|
|
||||||
col(ApiKey.reserved_balance) - deducted_max_cost,
|
|
||||||
),
|
|
||||||
else_=0,
|
|
||||||
)
|
|
||||||
|
|
||||||
finalize_stmt = (
|
finalize_stmt = (
|
||||||
update(ApiKey)
|
update(ApiKey)
|
||||||
.where(col(ApiKey.hashed_key) == billing_key.hashed_key)
|
.where(col(ApiKey.hashed_key) == billing_key.hashed_key)
|
||||||
|
.where(col(ApiKey.reserved_balance) >= deducted_max_cost)
|
||||||
.values(
|
.values(
|
||||||
reserved_balance=overrun_safe_reserved,
|
reserved_balance=col(ApiKey.reserved_balance) - deducted_max_cost,
|
||||||
balance=col(ApiKey.balance) - chargeable,
|
balance=col(ApiKey.balance) - chargeable,
|
||||||
total_spent=col(ApiKey.total_spent) + chargeable,
|
total_spent=col(ApiKey.total_spent) + chargeable,
|
||||||
)
|
)
|
||||||
)
|
)
|
||||||
await session.exec(finalize_stmt) # type: ignore[call-overload]
|
result = await session.exec(finalize_stmt) # type: ignore[call-overload]
|
||||||
|
|
||||||
if billing_key.hashed_key != key.hashed_key:
|
if billing_key.hashed_key != key.hashed_key:
|
||||||
child_stmt = (
|
child_stmt = (
|
||||||
update(ApiKey)
|
update(ApiKey)
|
||||||
.where(col(ApiKey.hashed_key) == key.hashed_key)
|
.where(col(ApiKey.hashed_key) == key.hashed_key)
|
||||||
|
.where(col(ApiKey.reserved_balance) >= deducted_max_cost)
|
||||||
.values(
|
.values(
|
||||||
reserved_balance=overrun_safe_reserved,
|
reserved_balance=col(ApiKey.reserved_balance) - deducted_max_cost,
|
||||||
total_spent=col(ApiKey.total_spent) + min(billing_key.balance, total_cost_msats),
|
total_spent=col(ApiKey.total_spent) + min(billing_key.balance, total_cost_msats),
|
||||||
)
|
)
|
||||||
)
|
)
|
||||||
@@ -1117,38 +1053,51 @@ async def adjust_payment_for_tokens(
|
|||||||
|
|
||||||
await session.commit()
|
await session.commit()
|
||||||
|
|
||||||
await session.refresh(billing_key)
|
if result.rowcount:
|
||||||
if billing_key.hashed_key != key.hashed_key:
|
await session.refresh(billing_key)
|
||||||
await session.refresh(key)
|
if billing_key.hashed_key != key.hashed_key:
|
||||||
cost.total_msats = total_cost_msats
|
await session.refresh(key)
|
||||||
logger.info(
|
cost.total_msats = total_cost_msats
|
||||||
"Finalized payment with additional charge",
|
logger.info(
|
||||||
extra={
|
"Finalized payment with additional charge",
|
||||||
"key_hash": key.hashed_key[:8] + "...",
|
extra={
|
||||||
"billing_key_hash": billing_key.hashed_key[:8] + "...",
|
"key_hash": key.hashed_key[:8] + "...",
|
||||||
"charged_amount": total_cost_msats,
|
"billing_key_hash": billing_key.hashed_key[:8] + "...",
|
||||||
"new_balance": billing_key.balance,
|
"charged_amount": total_cost_msats,
|
||||||
"model": model,
|
"new_balance": billing_key.balance,
|
||||||
},
|
"model": model,
|
||||||
)
|
},
|
||||||
await _accumulate_fee(total_cost_msats)
|
)
|
||||||
payments_logger.info(
|
await _accumulate_fee(total_cost_msats)
|
||||||
"FINALIZE",
|
payments_logger.info(
|
||||||
extra={
|
"FINALIZE",
|
||||||
"event": "finalize",
|
extra={
|
||||||
"key_hash": key.hashed_key[:8] + "...",
|
"event": "finalize",
|
||||||
"billing_key_hash": billing_key.hashed_key[:8] + "...",
|
"key_hash": key.hashed_key[:8] + "...",
|
||||||
"model": model,
|
"billing_key_hash": billing_key.hashed_key[:8] + "...",
|
||||||
"cost_reserved": deducted_max_cost,
|
"model": model,
|
||||||
"cost_charged": total_cost_msats,
|
"cost_reserved": deducted_max_cost,
|
||||||
"input_tokens": cost.input_tokens,
|
"cost_charged": total_cost_msats,
|
||||||
"output_tokens": cost.output_tokens,
|
"input_tokens": cost.input_tokens,
|
||||||
"balance": billing_key.balance,
|
"output_tokens": cost.output_tokens,
|
||||||
"reserved_balance": billing_key.reserved_balance,
|
"balance": billing_key.balance,
|
||||||
"total_spent": billing_key.total_spent,
|
"reserved_balance": billing_key.reserved_balance,
|
||||||
"finalize_type": "overrun",
|
"total_spent": billing_key.total_spent,
|
||||||
},
|
"finalize_type": "overrun",
|
||||||
)
|
},
|
||||||
|
)
|
||||||
|
else:
|
||||||
|
# Guard fired: reservation was already released by a concurrent
|
||||||
|
# finalization for this key. Nothing left to do.
|
||||||
|
logger.warning(
|
||||||
|
"Finalization skipped - reservation already released",
|
||||||
|
extra={
|
||||||
|
"key_hash": key.hashed_key[:8] + "...",
|
||||||
|
"billing_key_hash": billing_key.hashed_key[:8] + "...",
|
||||||
|
"attempted_charge": total_cost_msats,
|
||||||
|
"model": model,
|
||||||
|
},
|
||||||
|
)
|
||||||
else:
|
else:
|
||||||
# Refund some of the base cost
|
# Refund some of the base cost
|
||||||
refund = abs(cost_difference)
|
refund = abs(cost_difference)
|
||||||
@@ -1355,35 +1304,6 @@ async def periodic_key_reset() -> None:
|
|||||||
logger.error(f"Error in periodic_key_reset: {e}")
|
logger.error(f"Error in periodic_key_reset: {e}")
|
||||||
|
|
||||||
|
|
||||||
async def periodic_dead_key_prune() -> None:
|
|
||||||
"""Periodically prune dead API keys. Interval <= 0 disables it.
|
|
||||||
|
|
||||||
See ``prune_dead_api_keys`` for eligibility.
|
|
||||||
"""
|
|
||||||
from .core.db import create_session, prune_dead_api_keys
|
|
||||||
|
|
||||||
interval = settings.dead_key_prune_interval_seconds
|
|
||||||
if interval <= 0:
|
|
||||||
logger.info("Dead-key pruning disabled (interval <= 0)")
|
|
||||||
return
|
|
||||||
|
|
||||||
while True:
|
|
||||||
try:
|
|
||||||
await asyncio.sleep(interval)
|
|
||||||
except asyncio.CancelledError:
|
|
||||||
break
|
|
||||||
|
|
||||||
try:
|
|
||||||
async with create_session() as session:
|
|
||||||
await prune_dead_api_keys(
|
|
||||||
session, settings.dead_key_min_age_seconds
|
|
||||||
)
|
|
||||||
except asyncio.CancelledError:
|
|
||||||
break
|
|
||||||
except Exception as e:
|
|
||||||
logger.error(f"Error in periodic_dead_key_prune: {e}")
|
|
||||||
|
|
||||||
|
|
||||||
STALE_RESERVATION_SWEEP_INTERVAL_SECONDS: int = 60
|
STALE_RESERVATION_SWEEP_INTERVAL_SECONDS: int = 60
|
||||||
|
|
||||||
|
|
||||||
|
|||||||
+31
-22
@@ -20,14 +20,7 @@ from .core.db import (
|
|||||||
from .core.logging import get_logger
|
from .core.logging import get_logger
|
||||||
from .core.settings import settings
|
from .core.settings import settings
|
||||||
from .lightning import lightning_router
|
from .lightning import lightning_router
|
||||||
from .wallet import (
|
from .wallet import credit_balance, recieve_token, send_to_lnurl, send_token
|
||||||
classify_redemption_error,
|
|
||||||
credit_balance,
|
|
||||||
is_mint_connection_error,
|
|
||||||
recieve_token,
|
|
||||||
send_to_lnurl,
|
|
||||||
send_token,
|
|
||||||
)
|
|
||||||
|
|
||||||
router = APIRouter()
|
router = APIRouter()
|
||||||
balance_router = APIRouter(prefix="/v1/balance")
|
balance_router = APIRouter(prefix="/v1/balance")
|
||||||
@@ -163,18 +156,30 @@ async def topup_wallet_endpoint(
|
|||||||
raise HTTPException(status_code=400, detail="Invalid token format")
|
raise HTTPException(status_code=400, detail="Invalid token format")
|
||||||
try:
|
try:
|
||||||
amount_msats = await credit_balance(cashu_token, billing_key, session)
|
amount_msats = await credit_balance(cashu_token, billing_key, session)
|
||||||
except Exception as e:
|
except ValueError as e:
|
||||||
# Shared taxonomy so top-up matches the bearer/X-Cashu paths (503 for an
|
error_msg = str(e)
|
||||||
# unreachable mint, 422 for fee/swap failures, 400 for token faults).
|
if "already spent" in error_msg.lower():
|
||||||
classified = classify_redemption_error(e)
|
raise HTTPException(status_code=400, detail="Token already spent")
|
||||||
if classified is None:
|
elif "invalid" in error_msg.lower() or "decode" in error_msg.lower():
|
||||||
logger.error(
|
raise HTTPException(status_code=400, detail="Invalid token format")
|
||||||
"topup_wallet_endpoint: unhandled error",
|
elif "insufficient" in error_msg.lower() or "melt fee" in error_msg.lower():
|
||||||
extra={"error": str(e), "error_type": type(e).__name__},
|
raise HTTPException(
|
||||||
|
status_code=400,
|
||||||
|
detail=f"Token value is too small to cover swap fees. {error_msg}",
|
||||||
)
|
)
|
||||||
raise HTTPException(status_code=500, detail="Internal server error")
|
elif "failed to melt" in error_msg.lower():
|
||||||
_type, status_code, message, _code = classified
|
raise HTTPException(
|
||||||
raise HTTPException(status_code=status_code, detail=message)
|
status_code=400,
|
||||||
|
detail=f"Failed to swap foreign mint token. {error_msg}",
|
||||||
|
)
|
||||||
|
else:
|
||||||
|
raise HTTPException(status_code=400, detail=f"Failed to redeem token: {error_msg}")
|
||||||
|
except Exception as e:
|
||||||
|
logger.error(
|
||||||
|
"topup_wallet_endpoint: unhandled error",
|
||||||
|
extra={"error": str(e), "error_type": type(e).__name__},
|
||||||
|
)
|
||||||
|
raise HTTPException(status_code=500, detail="Internal server error")
|
||||||
return {"msats": amount_msats}
|
return {"msats": amount_msats}
|
||||||
|
|
||||||
|
|
||||||
@@ -436,10 +441,14 @@ async def refund_wallet_endpoint(
|
|||||||
"has_refund_address": bool(key.refund_address),
|
"has_refund_address": bool(key.refund_address),
|
||||||
},
|
},
|
||||||
)
|
)
|
||||||
if is_mint_connection_error(e):
|
if (
|
||||||
raise HTTPException(status_code=503, detail="Mint service unavailable")
|
"mint" in error_msg.lower()
|
||||||
|
or "connection" in error_msg.lower()
|
||||||
|
or "ConnectError" in str(type(e))
|
||||||
|
):
|
||||||
|
raise HTTPException(status_code=503, detail=f"Mint service unavailable: {error_msg}")
|
||||||
else:
|
else:
|
||||||
raise HTTPException(status_code=500, detail="Refund failed")
|
raise HTTPException(status_code=500, detail=f"Refund failed: {error_msg}")
|
||||||
|
|
||||||
await _refund_cache_set(bearer_value, result)
|
await _refund_cache_set(bearer_value, result)
|
||||||
|
|
||||||
|
|||||||
+125
-217
@@ -1,6 +1,5 @@
|
|||||||
import asyncio
|
import asyncio
|
||||||
import json
|
import json
|
||||||
import re
|
|
||||||
import secrets
|
import secrets
|
||||||
from datetime import datetime, timezone
|
from datetime import datetime, timezone
|
||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
@@ -9,7 +8,6 @@ from fastapi import APIRouter, Depends, HTTPException, Query, Request
|
|||||||
from pydantic import BaseModel, RootModel
|
from pydantic import BaseModel, RootModel
|
||||||
from pydantic.v1 import ValidationError as PydanticValidationError
|
from pydantic.v1 import ValidationError as PydanticValidationError
|
||||||
from sqlmodel import select
|
from sqlmodel import select
|
||||||
from sqlmodel.ext.asyncio.session import AsyncSession
|
|
||||||
|
|
||||||
from ..payment.models import _row_to_model, list_models
|
from ..payment.models import _row_to_model, list_models
|
||||||
from ..proxy import refresh_model_maps, reinitialize_upstreams
|
from ..proxy import refresh_model_maps, reinitialize_upstreams
|
||||||
@@ -31,7 +29,6 @@ from .db import (
|
|||||||
)
|
)
|
||||||
from .log_manager import log_manager
|
from .log_manager import log_manager
|
||||||
from .logging import get_logger
|
from .logging import get_logger
|
||||||
from .provider_slugs import allocate_unique_provider_slug
|
|
||||||
from .settings import SettingsService, settings
|
from .settings import SettingsService, settings
|
||||||
|
|
||||||
logger = get_logger(__name__)
|
logger = get_logger(__name__)
|
||||||
@@ -459,18 +456,19 @@ class ModelCreate(BaseModel):
|
|||||||
dependencies=[Depends(require_admin_api)],
|
dependencies=[Depends(require_admin_api)],
|
||||||
)
|
)
|
||||||
async def upsert_provider_model(
|
async def upsert_provider_model(
|
||||||
provider_id: str, payload: ModelCreate
|
provider_id: int, payload: ModelCreate
|
||||||
) -> dict[str, object]:
|
) -> dict[str, object]:
|
||||||
print(payload)
|
print(payload)
|
||||||
logger.info(
|
logger.info(
|
||||||
f"UPSERT_PROVIDER_MODEL called: provider_id={provider_id}, model_id={payload.id}"
|
f"UPSERT_PROVIDER_MODEL called: provider_id={provider_id}, model_id={payload.id}"
|
||||||
)
|
)
|
||||||
async with create_session() as session:
|
async with create_session() as session:
|
||||||
provider = await _get_upstream_provider_by_ref(session, provider_id)
|
provider = await session.get(UpstreamProviderRow, provider_id)
|
||||||
provider_pk = _provider_pk(provider)
|
if not provider:
|
||||||
|
raise HTTPException(status_code=404, detail="Provider not found")
|
||||||
|
|
||||||
# Try to get existing model
|
# Try to get existing model
|
||||||
existing_row = await session.get(ModelRow, (payload.id, provider_pk))
|
existing_row = await session.get(ModelRow, (payload.id, provider_id))
|
||||||
|
|
||||||
if existing_row:
|
if existing_row:
|
||||||
# Update existing model
|
# Update existing model
|
||||||
@@ -526,7 +524,7 @@ async def upsert_provider_model(
|
|||||||
alias_ids=(
|
alias_ids=(
|
||||||
json.dumps(payload.alias_ids) if payload.alias_ids else None
|
json.dumps(payload.alias_ids) if payload.alias_ids else None
|
||||||
),
|
),
|
||||||
upstream_provider_id=provider_pk,
|
upstream_provider_id=provider_id,
|
||||||
enabled=payload.enabled,
|
enabled=payload.enabled,
|
||||||
forwarded_model_id=payload.forwarded_model_id or payload.id,
|
forwarded_model_id=payload.forwarded_model_id or payload.id,
|
||||||
)
|
)
|
||||||
@@ -545,7 +543,7 @@ async def upsert_provider_model(
|
|||||||
dependencies=[Depends(require_admin_api)],
|
dependencies=[Depends(require_admin_api)],
|
||||||
)
|
)
|
||||||
async def update_provider_model_legacy(
|
async def update_provider_model_legacy(
|
||||||
provider_id: str, model_id: str, payload: ModelCreate
|
provider_id: int, model_id: str, payload: ModelCreate
|
||||||
) -> dict[str, object]:
|
) -> dict[str, object]:
|
||||||
"""Legacy PATCH endpoint - redirects to upsert POST endpoint for backward compatibility."""
|
"""Legacy PATCH endpoint - redirects to upsert POST endpoint for backward compatibility."""
|
||||||
logger.info(
|
logger.info(
|
||||||
@@ -558,12 +556,13 @@ async def update_provider_model_legacy(
|
|||||||
"/api/upstream-providers/{provider_id}/models/{model_id:path}",
|
"/api/upstream-providers/{provider_id}/models/{model_id:path}",
|
||||||
dependencies=[Depends(require_admin_api)],
|
dependencies=[Depends(require_admin_api)],
|
||||||
)
|
)
|
||||||
async def get_provider_model(provider_id: str, model_id: str) -> dict[str, object]:
|
async def get_provider_model(provider_id: int, model_id: str) -> dict[str, object]:
|
||||||
async with create_session() as session:
|
async with create_session() as session:
|
||||||
provider = await _get_upstream_provider_by_ref(session, provider_id)
|
provider = await session.get(UpstreamProviderRow, provider_id)
|
||||||
provider_pk = _provider_pk(provider)
|
if not provider:
|
||||||
|
raise HTTPException(status_code=404, detail="Provider not found")
|
||||||
|
|
||||||
row = await session.get(ModelRow, (model_id, provider_pk))
|
row = await session.get(ModelRow, (model_id, provider_id))
|
||||||
if not row:
|
if not row:
|
||||||
raise HTTPException(
|
raise HTTPException(
|
||||||
status_code=404, detail="Model not found for this provider"
|
status_code=404, detail="Model not found for this provider"
|
||||||
@@ -577,11 +576,9 @@ async def get_provider_model(provider_id: str, model_id: str) -> dict[str, objec
|
|||||||
"/api/upstream-providers/{provider_id}/models/{model_id:path}",
|
"/api/upstream-providers/{provider_id}/models/{model_id:path}",
|
||||||
dependencies=[Depends(require_admin_api)],
|
dependencies=[Depends(require_admin_api)],
|
||||||
)
|
)
|
||||||
async def delete_provider_model(provider_id: str, model_id: str) -> dict[str, object]:
|
async def delete_provider_model(provider_id: int, model_id: str) -> dict[str, object]:
|
||||||
async with create_session() as session:
|
async with create_session() as session:
|
||||||
provider = await _get_upstream_provider_by_ref(session, provider_id)
|
row = await session.get(ModelRow, (model_id, provider_id))
|
||||||
provider_pk = _provider_pk(provider)
|
|
||||||
row = await session.get(ModelRow, (model_id, provider_pk))
|
|
||||||
if not row:
|
if not row:
|
||||||
raise HTTPException(
|
raise HTTPException(
|
||||||
status_code=404, detail="Model not found for this provider"
|
status_code=404, detail="Model not found for this provider"
|
||||||
@@ -596,12 +593,10 @@ async def delete_provider_model(provider_id: str, model_id: str) -> dict[str, ob
|
|||||||
"/api/upstream-providers/{provider_id}/models",
|
"/api/upstream-providers/{provider_id}/models",
|
||||||
dependencies=[Depends(require_admin_api)],
|
dependencies=[Depends(require_admin_api)],
|
||||||
)
|
)
|
||||||
async def delete_all_provider_models(provider_id: str) -> dict[str, object]:
|
async def delete_all_provider_models(provider_id: int) -> dict[str, object]:
|
||||||
async with create_session() as session:
|
async with create_session() as session:
|
||||||
provider = await _get_upstream_provider_by_ref(session, provider_id)
|
|
||||||
provider_pk = _provider_pk(provider)
|
|
||||||
result = await session.exec(
|
result = await session.exec(
|
||||||
select(ModelRow).where(ModelRow.upstream_provider_id == provider_pk)
|
select(ModelRow).where(ModelRow.upstream_provider_id == provider_id)
|
||||||
) # type: ignore
|
) # type: ignore
|
||||||
rows = result.all()
|
rows = result.all()
|
||||||
for row in rows:
|
for row in rows:
|
||||||
@@ -620,7 +615,7 @@ class BatchOverrideRequest(BaseModel):
|
|||||||
dependencies=[Depends(require_admin_api)],
|
dependencies=[Depends(require_admin_api)],
|
||||||
)
|
)
|
||||||
async def batch_override_provider_models(
|
async def batch_override_provider_models(
|
||||||
provider_id: str, payload: BatchOverrideRequest
|
provider_id: int, payload: BatchOverrideRequest
|
||||||
) -> dict[str, object]:
|
) -> dict[str, object]:
|
||||||
"""Batch override models for a specific provider."""
|
"""Batch override models for a specific provider."""
|
||||||
logger.info(
|
logger.info(
|
||||||
@@ -628,14 +623,15 @@ async def batch_override_provider_models(
|
|||||||
)
|
)
|
||||||
|
|
||||||
async with create_session() as session:
|
async with create_session() as session:
|
||||||
provider = await _get_upstream_provider_by_ref(session, provider_id)
|
provider = await session.get(UpstreamProviderRow, provider_id)
|
||||||
provider_pk = _provider_pk(provider)
|
if not provider:
|
||||||
|
raise HTTPException(status_code=404, detail="Provider not found")
|
||||||
|
|
||||||
overridden_count = 0
|
overridden_count = 0
|
||||||
|
|
||||||
for model_data in payload.models:
|
for model_data in payload.models:
|
||||||
# Try to get existing model regardless of whether it's enabled or not
|
# Try to get existing model regardless of whether it's enabled or not
|
||||||
existing_row = await session.get(ModelRow, (model_data.id, provider_pk))
|
existing_row = await session.get(ModelRow, (model_data.id, provider_id))
|
||||||
|
|
||||||
if existing_row:
|
if existing_row:
|
||||||
# Update existing
|
# Update existing
|
||||||
@@ -689,7 +685,7 @@ async def batch_override_provider_models(
|
|||||||
if model_data.alias_ids
|
if model_data.alias_ids
|
||||||
else None
|
else None
|
||||||
),
|
),
|
||||||
upstream_provider_id=provider_pk,
|
upstream_provider_id=provider_id,
|
||||||
enabled=model_data.enabled,
|
enabled=model_data.enabled,
|
||||||
)
|
)
|
||||||
session.add(row)
|
session.add(row)
|
||||||
@@ -706,85 +702,6 @@ async def batch_override_provider_models(
|
|||||||
}
|
}
|
||||||
|
|
||||||
|
|
||||||
_SLUG_PATTERN = re.compile(r"^[a-z0-9][a-z0-9-]{1,62}[a-z0-9]$")
|
|
||||||
|
|
||||||
|
|
||||||
def _validate_slug(value: str) -> str:
|
|
||||||
candidate = value.strip().lower()
|
|
||||||
if not _SLUG_PATTERN.fullmatch(candidate):
|
|
||||||
raise HTTPException(
|
|
||||||
status_code=400,
|
|
||||||
detail=(
|
|
||||||
"slug must be 3-64 chars, lowercase letters/digits/hyphens, "
|
|
||||||
"and may not start or end with a hyphen"
|
|
||||||
),
|
|
||||||
)
|
|
||||||
if candidate.isdigit():
|
|
||||||
raise HTTPException(
|
|
||||||
status_code=400,
|
|
||||||
detail="slug must not be all digits",
|
|
||||||
)
|
|
||||||
return candidate
|
|
||||||
|
|
||||||
|
|
||||||
async def _ensure_unique_slug(
|
|
||||||
session: AsyncSession, slug: str, exclude_id: int | None = None
|
|
||||||
) -> None:
|
|
||||||
stmt = select(UpstreamProviderRow).where(UpstreamProviderRow.slug == slug)
|
|
||||||
result = await session.exec(stmt)
|
|
||||||
existing = result.first()
|
|
||||||
if existing and existing.id != exclude_id:
|
|
||||||
raise HTTPException(
|
|
||||||
status_code=409,
|
|
||||||
detail="Provider with this slug already exists",
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
async def _get_upstream_provider_by_ref(
|
|
||||||
session: AsyncSession, provider_ref: str
|
|
||||||
) -> UpstreamProviderRow:
|
|
||||||
if provider_ref.isdigit():
|
|
||||||
provider = await session.get(UpstreamProviderRow, int(provider_ref))
|
|
||||||
else:
|
|
||||||
slug = _validate_slug(provider_ref)
|
|
||||||
result = await session.exec(
|
|
||||||
select(UpstreamProviderRow).where(UpstreamProviderRow.slug == slug)
|
|
||||||
)
|
|
||||||
provider = result.first()
|
|
||||||
|
|
||||||
if not provider:
|
|
||||||
raise HTTPException(status_code=404, detail="Provider not found")
|
|
||||||
return provider
|
|
||||||
|
|
||||||
|
|
||||||
def _provider_pk(provider: UpstreamProviderRow) -> int:
|
|
||||||
if provider.id is None:
|
|
||||||
raise HTTPException(status_code=500, detail="Provider has no database id")
|
|
||||||
return provider.id
|
|
||||||
|
|
||||||
|
|
||||||
def _serialize_provider(
|
|
||||||
provider: UpstreamProviderRow, redact_api_key: bool = True
|
|
||||||
) -> dict[str, object]:
|
|
||||||
return {
|
|
||||||
"id": provider.id,
|
|
||||||
"slug": provider.slug,
|
|
||||||
"provider_type": provider.provider_type,
|
|
||||||
"base_url": provider.base_url,
|
|
||||||
"api_key": "[REDACTED]"
|
|
||||||
if (redact_api_key and provider.api_key)
|
|
||||||
else provider.api_key
|
|
||||||
if not redact_api_key
|
|
||||||
else "",
|
|
||||||
"api_version": provider.api_version,
|
|
||||||
"enabled": provider.enabled,
|
|
||||||
"provider_fee": provider.provider_fee,
|
|
||||||
"provider_settings": json.loads(provider.provider_settings)
|
|
||||||
if provider.provider_settings
|
|
||||||
else None,
|
|
||||||
}
|
|
||||||
|
|
||||||
|
|
||||||
class UpstreamProviderCreate(BaseModel):
|
class UpstreamProviderCreate(BaseModel):
|
||||||
provider_type: str
|
provider_type: str
|
||||||
base_url: str
|
base_url: str
|
||||||
@@ -793,7 +710,6 @@ class UpstreamProviderCreate(BaseModel):
|
|||||||
enabled: bool = True
|
enabled: bool = True
|
||||||
provider_fee: float = 1.01
|
provider_fee: float = 1.01
|
||||||
provider_settings: dict | None = None
|
provider_settings: dict | None = None
|
||||||
slug: str | None = None
|
|
||||||
|
|
||||||
|
|
||||||
class UpstreamProviderUpdate(BaseModel):
|
class UpstreamProviderUpdate(BaseModel):
|
||||||
@@ -804,50 +720,6 @@ class UpstreamProviderUpdate(BaseModel):
|
|||||||
enabled: bool | None = None
|
enabled: bool | None = None
|
||||||
provider_fee: float | None = None
|
provider_fee: float | None = None
|
||||||
provider_settings: dict | None = None
|
provider_settings: dict | None = None
|
||||||
slug: str | None = None
|
|
||||||
|
|
||||||
|
|
||||||
class UpstreamProviderUpdateBySlug(BaseModel):
|
|
||||||
slug: str
|
|
||||||
new_slug: str | None = None
|
|
||||||
provider_type: str | None = None
|
|
||||||
base_url: str | None = None
|
|
||||||
api_key: str | None = None
|
|
||||||
api_version: str | None = None
|
|
||||||
enabled: bool | None = None
|
|
||||||
provider_fee: float | None = None
|
|
||||||
provider_settings: dict | None = None
|
|
||||||
|
|
||||||
|
|
||||||
async def _apply_provider_update(
|
|
||||||
session: AsyncSession,
|
|
||||||
provider: UpstreamProviderRow,
|
|
||||||
payload: UpstreamProviderUpdate,
|
|
||||||
new_slug: str | None = None,
|
|
||||||
) -> None:
|
|
||||||
if new_slug is not None:
|
|
||||||
validated = _validate_slug(new_slug)
|
|
||||||
await _ensure_unique_slug(session, validated, exclude_id=provider.id)
|
|
||||||
provider.slug = validated
|
|
||||||
|
|
||||||
if payload.provider_type is not None:
|
|
||||||
provider.provider_type = payload.provider_type
|
|
||||||
if payload.base_url is not None:
|
|
||||||
provider.base_url = payload.base_url
|
|
||||||
if payload.api_key is not None:
|
|
||||||
provider.api_key = payload.api_key
|
|
||||||
if payload.api_version is not None:
|
|
||||||
provider.api_version = payload.api_version
|
|
||||||
if payload.enabled is not None:
|
|
||||||
provider.enabled = payload.enabled
|
|
||||||
if payload.provider_fee is not None:
|
|
||||||
provider.provider_fee = payload.provider_fee
|
|
||||||
if payload.provider_settings is not None:
|
|
||||||
provider.provider_settings = json.dumps(payload.provider_settings)
|
|
||||||
|
|
||||||
session.add(provider)
|
|
||||||
await session.commit()
|
|
||||||
await session.refresh(provider)
|
|
||||||
|
|
||||||
|
|
||||||
@admin_router.get("/api/upstream-providers", dependencies=[Depends(require_admin_api)])
|
@admin_router.get("/api/upstream-providers", dependencies=[Depends(require_admin_api)])
|
||||||
@@ -855,7 +727,21 @@ async def get_upstream_providers() -> list[dict[str, object]]:
|
|||||||
async with create_session() as session:
|
async with create_session() as session:
|
||||||
result = await session.exec(select(UpstreamProviderRow))
|
result = await session.exec(select(UpstreamProviderRow))
|
||||||
providers = result.all()
|
providers = result.all()
|
||||||
return [_serialize_provider(p) for p in providers]
|
return [
|
||||||
|
{
|
||||||
|
"id": p.id,
|
||||||
|
"provider_type": p.provider_type,
|
||||||
|
"base_url": p.base_url,
|
||||||
|
"api_key": "[REDACTED]" if p.api_key else "",
|
||||||
|
"api_version": p.api_version,
|
||||||
|
"enabled": p.enabled,
|
||||||
|
"provider_fee": p.provider_fee,
|
||||||
|
"provider_settings": json.loads(p.provider_settings)
|
||||||
|
if p.provider_settings
|
||||||
|
else None,
|
||||||
|
}
|
||||||
|
for p in providers
|
||||||
|
]
|
||||||
|
|
||||||
|
|
||||||
@admin_router.post("/api/upstream-providers", dependencies=[Depends(require_admin_api)])
|
@admin_router.post("/api/upstream-providers", dependencies=[Depends(require_admin_api)])
|
||||||
@@ -875,14 +761,7 @@ async def create_upstream_provider(
|
|||||||
detail="Provider with this base URL and API key already exists",
|
detail="Provider with this base URL and API key already exists",
|
||||||
)
|
)
|
||||||
|
|
||||||
if payload.slug:
|
|
||||||
slug = _validate_slug(payload.slug)
|
|
||||||
await _ensure_unique_slug(session, slug)
|
|
||||||
else:
|
|
||||||
slug = await allocate_unique_provider_slug(session, payload.provider_type)
|
|
||||||
|
|
||||||
provider = UpstreamProviderRow(
|
provider = UpstreamProviderRow(
|
||||||
slug=slug,
|
|
||||||
provider_type=payload.provider_type,
|
provider_type=payload.provider_type,
|
||||||
base_url=payload.base_url,
|
base_url=payload.base_url,
|
||||||
api_key=payload.api_key,
|
api_key=payload.api_key,
|
||||||
@@ -899,81 +778,99 @@ async def create_upstream_provider(
|
|||||||
|
|
||||||
await reinitialize_upstreams()
|
await reinitialize_upstreams()
|
||||||
await refresh_model_maps()
|
await refresh_model_maps()
|
||||||
return _serialize_provider(provider)
|
return {
|
||||||
|
"id": provider.id,
|
||||||
|
"provider_type": provider.provider_type,
|
||||||
|
"base_url": provider.base_url,
|
||||||
|
"api_key": "[REDACTED]",
|
||||||
|
"api_version": provider.api_version,
|
||||||
|
"enabled": provider.enabled,
|
||||||
|
"provider_fee": provider.provider_fee,
|
||||||
|
"provider_settings": payload.provider_settings,
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
@admin_router.get(
|
@admin_router.get(
|
||||||
"/api/upstream-providers/{provider_id}", dependencies=[Depends(require_admin_api)]
|
"/api/upstream-providers/{provider_id}", dependencies=[Depends(require_admin_api)]
|
||||||
)
|
)
|
||||||
async def get_upstream_provider(provider_id: str) -> dict[str, object]:
|
async def get_upstream_provider(provider_id: int) -> dict[str, object]:
|
||||||
async with create_session() as session:
|
async with create_session() as session:
|
||||||
provider = await _get_upstream_provider_by_ref(session, provider_id)
|
provider = await session.get(UpstreamProviderRow, provider_id)
|
||||||
return _serialize_provider(provider)
|
if not provider:
|
||||||
|
raise HTTPException(status_code=404, detail="Provider not found")
|
||||||
|
return {
|
||||||
|
"id": provider.id,
|
||||||
|
"provider_type": provider.provider_type,
|
||||||
|
"base_url": provider.base_url,
|
||||||
|
"api_key": "[REDACTED]" if provider.api_key else "",
|
||||||
|
"api_version": provider.api_version,
|
||||||
|
"enabled": provider.enabled,
|
||||||
|
"provider_fee": provider.provider_fee,
|
||||||
|
"provider_settings": json.loads(provider.provider_settings)
|
||||||
|
if provider.provider_settings
|
||||||
|
else None,
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
@admin_router.patch(
|
@admin_router.patch(
|
||||||
"/api/upstream-providers/{provider_id}", dependencies=[Depends(require_admin_api)]
|
"/api/upstream-providers/{provider_id}", dependencies=[Depends(require_admin_api)]
|
||||||
)
|
)
|
||||||
async def update_upstream_provider(
|
async def update_upstream_provider(
|
||||||
provider_id: str, payload: UpstreamProviderUpdate
|
provider_id: int, payload: UpstreamProviderUpdate
|
||||||
) -> dict[str, object]:
|
) -> dict[str, object]:
|
||||||
async with create_session() as session:
|
async with create_session() as session:
|
||||||
provider = await _get_upstream_provider_by_ref(session, provider_id)
|
provider = await session.get(UpstreamProviderRow, provider_id)
|
||||||
|
|
||||||
await _apply_provider_update(session, provider, payload, new_slug=payload.slug)
|
|
||||||
|
|
||||||
await reinitialize_upstreams()
|
|
||||||
await refresh_model_maps()
|
|
||||||
return _serialize_provider(provider)
|
|
||||||
|
|
||||||
|
|
||||||
@admin_router.patch(
|
|
||||||
"/api/upstream-providers", dependencies=[Depends(require_admin_api)]
|
|
||||||
)
|
|
||||||
async def update_upstream_provider_by_slug(
|
|
||||||
payload: UpstreamProviderUpdateBySlug,
|
|
||||||
) -> dict[str, object]:
|
|
||||||
lookup = _validate_slug(payload.slug)
|
|
||||||
async with create_session() as session:
|
|
||||||
result = await session.exec(
|
|
||||||
select(UpstreamProviderRow).where(
|
|
||||||
UpstreamProviderRow.slug == lookup
|
|
||||||
)
|
|
||||||
)
|
|
||||||
provider = result.first()
|
|
||||||
if not provider:
|
if not provider:
|
||||||
raise HTTPException(status_code=404, detail="Provider not found")
|
raise HTTPException(status_code=404, detail="Provider not found")
|
||||||
|
|
||||||
update_payload = UpstreamProviderUpdate(
|
if payload.provider_type is not None:
|
||||||
provider_type=payload.provider_type,
|
provider.provider_type = payload.provider_type
|
||||||
base_url=payload.base_url,
|
if payload.base_url is not None:
|
||||||
api_key=payload.api_key,
|
provider.base_url = payload.base_url
|
||||||
api_version=payload.api_version,
|
if payload.api_key is not None:
|
||||||
enabled=payload.enabled,
|
provider.api_key = payload.api_key
|
||||||
provider_fee=payload.provider_fee,
|
if payload.api_version is not None:
|
||||||
provider_settings=payload.provider_settings,
|
provider.api_version = payload.api_version
|
||||||
)
|
if payload.enabled is not None:
|
||||||
await _apply_provider_update(
|
provider.enabled = payload.enabled
|
||||||
session, provider, update_payload, new_slug=payload.new_slug
|
if payload.provider_fee is not None:
|
||||||
)
|
provider.provider_fee = payload.provider_fee
|
||||||
|
if payload.provider_settings is not None:
|
||||||
|
provider.provider_settings = json.dumps(payload.provider_settings)
|
||||||
|
|
||||||
|
session.add(provider)
|
||||||
|
await session.commit()
|
||||||
|
await session.refresh(provider)
|
||||||
|
|
||||||
await reinitialize_upstreams()
|
await reinitialize_upstreams()
|
||||||
await refresh_model_maps()
|
await refresh_model_maps()
|
||||||
return _serialize_provider(provider)
|
return {
|
||||||
|
"id": provider.id,
|
||||||
|
"provider_type": provider.provider_type,
|
||||||
|
"base_url": provider.base_url,
|
||||||
|
"api_key": "[REDACTED]",
|
||||||
|
"api_version": provider.api_version,
|
||||||
|
"enabled": provider.enabled,
|
||||||
|
"provider_fee": provider.provider_fee,
|
||||||
|
"provider_settings": json.loads(provider.provider_settings)
|
||||||
|
if provider.provider_settings
|
||||||
|
else None,
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
@admin_router.delete(
|
@admin_router.delete(
|
||||||
"/api/upstream-providers/{provider_id}", dependencies=[Depends(require_admin_api)]
|
"/api/upstream-providers/{provider_id}", dependencies=[Depends(require_admin_api)]
|
||||||
)
|
)
|
||||||
async def delete_upstream_provider(provider_id: str) -> dict[str, object]:
|
async def delete_upstream_provider(provider_id: int) -> dict[str, object]:
|
||||||
async with create_session() as session:
|
async with create_session() as session:
|
||||||
provider = await _get_upstream_provider_by_ref(session, provider_id)
|
provider = await session.get(UpstreamProviderRow, provider_id)
|
||||||
deleted_id = _provider_pk(provider)
|
if not provider:
|
||||||
|
raise HTTPException(status_code=404, detail="Provider not found")
|
||||||
await session.delete(provider)
|
await session.delete(provider)
|
||||||
await session.commit()
|
await session.commit()
|
||||||
await reinitialize_upstreams()
|
await reinitialize_upstreams()
|
||||||
await refresh_model_maps()
|
await refresh_model_maps()
|
||||||
return {"ok": True, "deleted_id": deleted_id}
|
return {"ok": True, "deleted_id": provider_id}
|
||||||
|
|
||||||
|
|
||||||
@admin_router.get("/api/provider-types", dependencies=[Depends(require_admin_api)])
|
@admin_router.get("/api/provider-types", dependencies=[Depends(require_admin_api)])
|
||||||
@@ -988,16 +885,17 @@ async def get_provider_types() -> list[dict[str, object]]:
|
|||||||
"/api/upstream-providers/{provider_id}/models",
|
"/api/upstream-providers/{provider_id}/models",
|
||||||
dependencies=[Depends(require_admin_api)],
|
dependencies=[Depends(require_admin_api)],
|
||||||
)
|
)
|
||||||
async def get_provider_models(provider_id: str) -> dict[str, object]:
|
async def get_provider_models(provider_id: int) -> dict[str, object]:
|
||||||
from ..upstream.helpers import _instantiate_provider
|
from ..upstream.helpers import _instantiate_provider
|
||||||
|
|
||||||
async with create_session() as session:
|
async with create_session() as session:
|
||||||
provider = await _get_upstream_provider_by_ref(session, provider_id)
|
provider = await session.get(UpstreamProviderRow, provider_id)
|
||||||
provider_pk = _provider_pk(provider)
|
if not provider:
|
||||||
|
raise HTTPException(status_code=404, detail="Provider not found")
|
||||||
|
|
||||||
db_models = await list_models(
|
db_models = await list_models(
|
||||||
session=session,
|
session=session,
|
||||||
upstream_id=provider_pk,
|
upstream_id=provider_id,
|
||||||
include_disabled=True,
|
include_disabled=True,
|
||||||
apply_fees=False,
|
apply_fees=False,
|
||||||
)
|
)
|
||||||
@@ -1087,11 +985,13 @@ class TopupTokenRequest(BaseModel):
|
|||||||
dependencies=[Depends(require_admin_api)],
|
dependencies=[Depends(require_admin_api)],
|
||||||
)
|
)
|
||||||
async def topup_provider_with_token(
|
async def topup_provider_with_token(
|
||||||
provider_id: str, payload: TopupTokenRequest
|
provider_id: int, payload: TopupTokenRequest
|
||||||
) -> dict:
|
) -> dict:
|
||||||
"""Redeem a Cashu token for an upstream provider."""
|
"""Redeem a Cashu token for an upstream provider."""
|
||||||
async with create_session() as session:
|
async with create_session() as session:
|
||||||
provider = await _get_upstream_provider_by_ref(session, provider_id)
|
provider = await session.get(UpstreamProviderRow, provider_id)
|
||||||
|
if not provider:
|
||||||
|
raise HTTPException(status_code=404, detail="Provider not found")
|
||||||
|
|
||||||
import httpx
|
import httpx
|
||||||
|
|
||||||
@@ -1122,13 +1022,15 @@ async def topup_provider_with_token(
|
|||||||
dependencies=[Depends(require_admin_api)],
|
dependencies=[Depends(require_admin_api)],
|
||||||
)
|
)
|
||||||
async def initiate_provider_topup(
|
async def initiate_provider_topup(
|
||||||
provider_id: str, payload: TopupRequest
|
provider_id: int, payload: TopupRequest
|
||||||
) -> dict[str, object]:
|
) -> dict[str, object]:
|
||||||
"""Initiate a Lightning Network top-up for the upstream provider account."""
|
"""Initiate a Lightning Network top-up for the upstream provider account."""
|
||||||
from ..upstream.helpers import _instantiate_provider
|
from ..upstream.helpers import _instantiate_provider
|
||||||
|
|
||||||
async with create_session() as session:
|
async with create_session() as session:
|
||||||
provider = await _get_upstream_provider_by_ref(session, provider_id)
|
provider = await session.get(UpstreamProviderRow, provider_id)
|
||||||
|
if not provider:
|
||||||
|
raise HTTPException(status_code=404, detail="Provider not found")
|
||||||
|
|
||||||
try:
|
try:
|
||||||
logger.info(
|
logger.info(
|
||||||
@@ -1248,13 +1150,15 @@ async def initiate_provider_topup(
|
|||||||
"/api/upstream-providers/{provider_id}/topup/{invoice_id}/status",
|
"/api/upstream-providers/{provider_id}/topup/{invoice_id}/status",
|
||||||
dependencies=[Depends(require_admin_api)],
|
dependencies=[Depends(require_admin_api)],
|
||||||
)
|
)
|
||||||
async def check_topup_status(provider_id: str, invoice_id: str) -> dict[str, object]:
|
async def check_topup_status(provider_id: int, invoice_id: str) -> dict[str, object]:
|
||||||
"""Check the status of a Lightning Network top-up invoice."""
|
"""Check the status of a Lightning Network top-up invoice."""
|
||||||
from ..upstream.helpers import _instantiate_provider
|
from ..upstream.helpers import _instantiate_provider
|
||||||
from ..upstream.ppqai import PPQAIUpstreamProvider
|
from ..upstream.ppqai import PPQAIUpstreamProvider
|
||||||
|
|
||||||
async with create_session() as session:
|
async with create_session() as session:
|
||||||
provider = await _get_upstream_provider_by_ref(session, provider_id)
|
provider = await session.get(UpstreamProviderRow, provider_id)
|
||||||
|
if not provider:
|
||||||
|
raise HTTPException(status_code=404, detail="Provider not found")
|
||||||
|
|
||||||
# For Routstr providers, proxy the status check
|
# For Routstr providers, proxy the status check
|
||||||
if provider.provider_type == "routstr":
|
if provider.provider_type == "routstr":
|
||||||
@@ -1301,12 +1205,14 @@ async def check_topup_status(provider_id: str, invoice_id: str) -> dict[str, obj
|
|||||||
"/api/upstream-providers/{provider_id}/balance",
|
"/api/upstream-providers/{provider_id}/balance",
|
||||||
dependencies=[Depends(require_admin_api)],
|
dependencies=[Depends(require_admin_api)],
|
||||||
)
|
)
|
||||||
async def get_provider_balance(provider_id: str) -> dict[str, object]:
|
async def get_provider_balance(provider_id: int) -> dict[str, object]:
|
||||||
"""Get the current balance for an upstream provider account."""
|
"""Get the current balance for an upstream provider account."""
|
||||||
from ..upstream.helpers import _instantiate_provider
|
from ..upstream.helpers import _instantiate_provider
|
||||||
|
|
||||||
async with create_session() as session:
|
async with create_session() as session:
|
||||||
provider = await _get_upstream_provider_by_ref(session, provider_id)
|
provider = await session.get(UpstreamProviderRow, provider_id)
|
||||||
|
if not provider:
|
||||||
|
raise HTTPException(status_code=404, detail="Provider not found")
|
||||||
|
|
||||||
# For Routstr providers, proxy the balance check
|
# For Routstr providers, proxy the balance check
|
||||||
if provider.provider_type == "routstr":
|
if provider.provider_type == "routstr":
|
||||||
@@ -1685,13 +1591,15 @@ async def get_lightning_invoices_api(
|
|||||||
"/api/upstream-providers/{provider_id}/routstr/refund",
|
"/api/upstream-providers/{provider_id}/routstr/refund",
|
||||||
dependencies=[Depends(require_admin_api)],
|
dependencies=[Depends(require_admin_api)],
|
||||||
)
|
)
|
||||||
async def refund_routstr_provider_balance(provider_id: str) -> dict[str, object]:
|
async def refund_routstr_provider_balance(provider_id: int) -> dict[str, object]:
|
||||||
"""Refund balance from an upstream Routstr provider back to the local wallet."""
|
"""Refund balance from an upstream Routstr provider back to the local wallet."""
|
||||||
from ..upstream.helpers import _instantiate_provider
|
from ..upstream.helpers import _instantiate_provider
|
||||||
from ..upstream.routstr import RoutstrUpstreamProvider
|
from ..upstream.routstr import RoutstrUpstreamProvider
|
||||||
|
|
||||||
async with create_session() as session:
|
async with create_session() as session:
|
||||||
provider_row = await _get_upstream_provider_by_ref(session, provider_id)
|
provider_row = await session.get(UpstreamProviderRow, provider_id)
|
||||||
|
if not provider_row:
|
||||||
|
raise HTTPException(status_code=404, detail="Provider not found")
|
||||||
|
|
||||||
if provider_row.provider_type != "routstr":
|
if provider_row.provider_type != "routstr":
|
||||||
raise HTTPException(
|
raise HTTPException(
|
||||||
|
|||||||
+1
-65
@@ -9,10 +9,9 @@ from typing import AsyncGenerator
|
|||||||
from alembic import command
|
from alembic import command
|
||||||
from alembic.config import Config
|
from alembic.config import Config
|
||||||
from alembic.util.exc import CommandError
|
from alembic.util.exc import CommandError
|
||||||
from sqlalchemy import UniqueConstraint, delete
|
from sqlalchemy import UniqueConstraint
|
||||||
from sqlalchemy.exc import OperationalError
|
from sqlalchemy.exc import OperationalError
|
||||||
from sqlalchemy.ext.asyncio.engine import create_async_engine
|
from sqlalchemy.ext.asyncio.engine import create_async_engine
|
||||||
from sqlalchemy.orm import aliased
|
|
||||||
from sqlmodel import Field, Relationship, SQLModel, col, func, select, update
|
from sqlmodel import Field, Relationship, SQLModel, col, func, select, update
|
||||||
from sqlmodel.ext.asyncio.session import AsyncSession
|
from sqlmodel.ext.asyncio.session import AsyncSession
|
||||||
|
|
||||||
@@ -126,63 +125,6 @@ async def release_stale_reservations(
|
|||||||
return released
|
return released
|
||||||
|
|
||||||
|
|
||||||
async def prune_dead_api_keys(session: AsyncSession, min_age_seconds: int) -> int:
|
|
||||||
"""Delete dead parentless API keys; return the count removed.
|
|
||||||
|
|
||||||
Dead = 0 balance/reservation/spend/requests, older than the grace period,
|
|
||||||
no parent, no children, no pending invoice. Cashu rows are unlinked (not
|
|
||||||
deleted) first to keep the audit trail.
|
|
||||||
"""
|
|
||||||
cutoff = int(time.time()) - min_age_seconds
|
|
||||||
|
|
||||||
child = aliased(ApiKey)
|
|
||||||
has_children = (
|
|
||||||
select(child.hashed_key).where(
|
|
||||||
col(child.parent_key_hash) == col(ApiKey.hashed_key)
|
|
||||||
)
|
|
||||||
).exists()
|
|
||||||
pending_invoice = (
|
|
||||||
select(LightningInvoice.id)
|
|
||||||
.where(col(LightningInvoice.api_key_hash) == col(ApiKey.hashed_key))
|
|
||||||
.where(col(LightningInvoice.status) == "pending")
|
|
||||||
).exists()
|
|
||||||
|
|
||||||
eligible_hashes = (
|
|
||||||
select(ApiKey.hashed_key)
|
|
||||||
.where(col(ApiKey.balance) == 0)
|
|
||||||
.where(col(ApiKey.reserved_balance) == 0)
|
|
||||||
.where(col(ApiKey.total_spent) == 0)
|
|
||||||
.where(col(ApiKey.total_requests) == 0)
|
|
||||||
.where(col(ApiKey.parent_key_hash).is_(None))
|
|
||||||
.where(
|
|
||||||
(col(ApiKey.created_at).is_(None)) | (col(ApiKey.created_at) < cutoff)
|
|
||||||
)
|
|
||||||
.where(~pending_invoice)
|
|
||||||
.where(~has_children)
|
|
||||||
)
|
|
||||||
|
|
||||||
# Unlink transactions rather than cascade-deleting them, so the financial
|
|
||||||
# audit trail survives. The eligibility predicate is re-evaluated inside both
|
|
||||||
# statements so a key that gained balance mid-run is left untouched.
|
|
||||||
await session.exec( # type: ignore[call-overload]
|
|
||||||
update(CashuTransaction)
|
|
||||||
.where(col(CashuTransaction.api_key_hashed_key).in_(eligible_hashes))
|
|
||||||
.values(api_key_hashed_key=None)
|
|
||||||
)
|
|
||||||
|
|
||||||
result = await session.exec( # type: ignore[call-overload]
|
|
||||||
delete(ApiKey).where(col(ApiKey.hashed_key).in_(eligible_hashes))
|
|
||||||
)
|
|
||||||
await session.commit()
|
|
||||||
|
|
||||||
pruned = int(result.rowcount or 0)
|
|
||||||
logger.info(
|
|
||||||
"Pruned dead API keys",
|
|
||||||
extra={"pruned_keys": pruned, "min_age_seconds": min_age_seconds},
|
|
||||||
)
|
|
||||||
return pruned
|
|
||||||
|
|
||||||
|
|
||||||
class ModelRow(SQLModel, table=True): # type: ignore
|
class ModelRow(SQLModel, table=True): # type: ignore
|
||||||
__tablename__ = "models"
|
__tablename__ = "models"
|
||||||
id: str = Field(primary_key=True)
|
id: str = Field(primary_key=True)
|
||||||
@@ -319,12 +261,6 @@ class UpstreamProviderRow(SQLModel, table=True): # type: ignore
|
|||||||
),
|
),
|
||||||
)
|
)
|
||||||
id: int | None = Field(default=None, primary_key=True)
|
id: int | None = Field(default=None, primary_key=True)
|
||||||
slug: str | None = Field(
|
|
||||||
default=None,
|
|
||||||
unique=True,
|
|
||||||
index=True,
|
|
||||||
description="Stable external slug used for updates via API key.",
|
|
||||||
)
|
|
||||||
provider_type: str = Field(
|
provider_type: str = Field(
|
||||||
description="Provider type: custom, openai, anthropic, azure, openrouter, etc."
|
description="Provider type: custom, openai, anthropic, azure, openrouter, etc."
|
||||||
)
|
)
|
||||||
|
|||||||
@@ -7,26 +7,11 @@ logger = get_logger(__name__)
|
|||||||
|
|
||||||
|
|
||||||
class UpstreamError(Exception):
|
class UpstreamError(Exception):
|
||||||
"""Exception raised when an upstream provider fails.
|
"""Exception raised when an upstream provider fails."""
|
||||||
|
|
||||||
``code`` carries a stable, machine-readable classification (e.g.
|
def __init__(self, message: str, status_code: int = 502):
|
||||||
``UPSTREAM_RATE_LIMIT``) so callers can distinguish failure kinds without
|
|
||||||
string-matching the message. ``details`` holds optional structured,
|
|
||||||
redaction-safe context. Both default to ``None`` for backwards
|
|
||||||
compatibility.
|
|
||||||
"""
|
|
||||||
|
|
||||||
def __init__(
|
|
||||||
self,
|
|
||||||
message: str,
|
|
||||||
status_code: int = 502,
|
|
||||||
code: str | None = None,
|
|
||||||
details: dict[str, object] | None = None,
|
|
||||||
):
|
|
||||||
self.message = message
|
self.message = message
|
||||||
self.status_code = status_code
|
self.status_code = status_code
|
||||||
self.code = code
|
|
||||||
self.details = details
|
|
||||||
super().__init__(message)
|
super().__init__(message)
|
||||||
|
|
||||||
|
|
||||||
|
|||||||
@@ -51,8 +51,6 @@ from pythonjsonlogger import jsonlogger
|
|||||||
from rich.console import Console
|
from rich.console import Console
|
||||||
from rich.logging import RichHandler
|
from rich.logging import RichHandler
|
||||||
|
|
||||||
from .redaction import redact_obj, redact_org_ids
|
|
||||||
|
|
||||||
# Only use RichHandler when stdout is a real TTY. In non-TTY contexts
|
# Only use RichHandler when stdout is a real TTY. In non-TTY contexts
|
||||||
# (docker logs, pipes, CI) Rich pads every line to width and wraps long
|
# (docker logs, pipes, CI) Rich pads every line to width and wraps long
|
||||||
# records, producing visually-empty trailing whitespace and split records.
|
# records, producing visually-empty trailing whitespace and split records.
|
||||||
@@ -182,37 +180,6 @@ class RequestIdFilter(logging.Filter):
|
|||||||
return True
|
return True
|
||||||
|
|
||||||
|
|
||||||
# Standard ``LogRecord`` attributes that are never user-supplied ``extra``
|
|
||||||
# fields; skipped when redacting structured extras (``msg``/``message`` are
|
|
||||||
# handled separately above).
|
|
||||||
_NON_EXTRA_RECORD_ATTRS = frozenset(
|
|
||||||
{
|
|
||||||
"name",
|
|
||||||
"msg",
|
|
||||||
"args",
|
|
||||||
"levelname",
|
|
||||||
"levelno",
|
|
||||||
"pathname",
|
|
||||||
"filename",
|
|
||||||
"module",
|
|
||||||
"exc_info",
|
|
||||||
"exc_text",
|
|
||||||
"stack_info",
|
|
||||||
"lineno",
|
|
||||||
"funcName",
|
|
||||||
"created",
|
|
||||||
"msecs",
|
|
||||||
"relativeCreated",
|
|
||||||
"thread",
|
|
||||||
"threadName",
|
|
||||||
"processName",
|
|
||||||
"process",
|
|
||||||
"taskName",
|
|
||||||
"message",
|
|
||||||
}
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
class SecurityFilter(logging.Filter):
|
class SecurityFilter(logging.Filter):
|
||||||
"""Filter to remove sensitive information from logs."""
|
"""Filter to remove sensitive information from logs."""
|
||||||
|
|
||||||
@@ -236,7 +203,6 @@ class SecurityFilter(logging.Filter):
|
|||||||
"""Filter out sensitive information from log records."""
|
"""Filter out sensitive information from log records."""
|
||||||
try:
|
try:
|
||||||
message = record.getMessage()
|
message = record.getMessage()
|
||||||
message = redact_org_ids(message)
|
|
||||||
standalone_patterns = [
|
standalone_patterns = [
|
||||||
r"Bearer\s+([a-zA-Z0-9_\-\.]{10,})", # Bearer token (must be 10 characters or more to reduce false-positives)
|
r"Bearer\s+([a-zA-Z0-9_\-\.]{10,})", # Bearer token (must be 10 characters or more to reduce false-positives)
|
||||||
r"cashu[A-Z]+([a-zA-Z0-9_\-\.=/+]+)", # Cashu tokens
|
r"cashu[A-Z]+([a-zA-Z0-9_\-\.=/+]+)", # Cashu tokens
|
||||||
@@ -258,16 +224,6 @@ class SecurityFilter(logging.Filter):
|
|||||||
record.msg = message
|
record.msg = message
|
||||||
record.args = ()
|
record.args = ()
|
||||||
|
|
||||||
# Structured `extra={...}` fields are emitted by the JSON formatter
|
|
||||||
# straight from the record dict and never pass through the message
|
|
||||||
# formatting above. Redact organization IDs from any string-valued
|
|
||||||
# extra so they cannot leak via structured logs.
|
|
||||||
for attr, value in list(record.__dict__.items()):
|
|
||||||
if attr in _NON_EXTRA_RECORD_ATTRS:
|
|
||||||
continue
|
|
||||||
if isinstance(value, (str, dict, list, tuple)):
|
|
||||||
record.__dict__[attr] = redact_obj(value)
|
|
||||||
|
|
||||||
except Exception:
|
except Exception:
|
||||||
pass
|
pass
|
||||||
|
|
||||||
|
|||||||
+2
-21
@@ -11,11 +11,7 @@ from starlette.exceptions import HTTPException
|
|||||||
from starlette.responses import Response as StarletteResponse
|
from starlette.responses import Response as StarletteResponse
|
||||||
from starlette.types import Scope
|
from starlette.types import Scope
|
||||||
|
|
||||||
from ..auth import (
|
from ..auth import periodic_key_reset, periodic_stale_reservation_sweep
|
||||||
periodic_dead_key_prune,
|
|
||||||
periodic_key_reset,
|
|
||||||
periodic_stale_reservation_sweep,
|
|
||||||
)
|
|
||||||
from ..balance import balance_router, deprecated_wallet_router
|
from ..balance import balance_router, deprecated_wallet_router
|
||||||
from ..lightning import lightning_router, periodic_invoice_watcher
|
from ..lightning import lightning_router, periodic_invoice_watcher
|
||||||
from ..nostr import (
|
from ..nostr import (
|
||||||
@@ -28,7 +24,6 @@ from ..payment.models import models_router, update_sats_pricing
|
|||||||
from ..payment.price import update_prices_periodically
|
from ..payment.price import update_prices_periodically
|
||||||
from ..proxy import initialize_upstreams, proxy_router, refresh_model_maps_periodically
|
from ..proxy import initialize_upstreams, proxy_router, refresh_model_maps_periodically
|
||||||
from ..upstream.auto_topup import periodic_auto_topup
|
from ..upstream.auto_topup import periodic_auto_topup
|
||||||
from ..upstream.deepseek_v4_pricing_shim import register_deepseek_v4_pricing
|
|
||||||
from ..upstream.litellm_routing import configure_litellm
|
from ..upstream.litellm_routing import configure_litellm
|
||||||
from ..wallet import periodic_payout, periodic_refund_sweep, periodic_routstr_fee_payout
|
from ..wallet import periodic_payout, periodic_refund_sweep, periodic_routstr_fee_payout
|
||||||
from .admin import admin_router
|
from .admin import admin_router
|
||||||
@@ -60,7 +55,6 @@ async def lifespan(_: FastAPI) -> AsyncGenerator[None, None]:
|
|||||||
model_maps_refresh_task = None
|
model_maps_refresh_task = None
|
||||||
key_reset_task = None
|
key_reset_task = None
|
||||||
stale_reservation_task = None
|
stale_reservation_task = None
|
||||||
dead_key_prune_task = None
|
|
||||||
auto_topup_task = None
|
auto_topup_task = None
|
||||||
refund_sweep_task = None
|
refund_sweep_task = None
|
||||||
routstr_fee_task = None
|
routstr_fee_task = None
|
||||||
@@ -71,11 +65,6 @@ async def lifespan(_: FastAPI) -> AsyncGenerator[None, None]:
|
|||||||
# debug logging) before any upstream provider dispatches a request.
|
# debug logging) before any upstream provider dispatches a request.
|
||||||
configure_litellm()
|
configure_litellm()
|
||||||
|
|
||||||
# TEMPORARY: backfill DeepSeek V4 pricing missing from litellm's cost
|
|
||||||
# map (BerriAI/litellm#30430). Remove this call and
|
|
||||||
# deepseek_v4_pricing_shim.py once litellm ships these models.
|
|
||||||
register_deepseek_v4_pricing()
|
|
||||||
|
|
||||||
# Run database migrations on startup
|
# Run database migrations on startup
|
||||||
run_migrations()
|
run_migrations()
|
||||||
|
|
||||||
@@ -137,7 +126,6 @@ async def lifespan(_: FastAPI) -> AsyncGenerator[None, None]:
|
|||||||
stale_reservation_task = asyncio.create_task(
|
stale_reservation_task = asyncio.create_task(
|
||||||
periodic_stale_reservation_sweep()
|
periodic_stale_reservation_sweep()
|
||||||
)
|
)
|
||||||
dead_key_prune_task = asyncio.create_task(periodic_dead_key_prune())
|
|
||||||
auto_topup_task = asyncio.create_task(periodic_auto_topup())
|
auto_topup_task = asyncio.create_task(periodic_auto_topup())
|
||||||
refund_sweep_task = asyncio.create_task(periodic_refund_sweep())
|
refund_sweep_task = asyncio.create_task(periodic_refund_sweep())
|
||||||
routstr_fee_task = asyncio.create_task(periodic_routstr_fee_payout())
|
routstr_fee_task = asyncio.create_task(periodic_routstr_fee_payout())
|
||||||
@@ -177,8 +165,6 @@ async def lifespan(_: FastAPI) -> AsyncGenerator[None, None]:
|
|||||||
key_reset_task.cancel()
|
key_reset_task.cancel()
|
||||||
if stale_reservation_task is not None:
|
if stale_reservation_task is not None:
|
||||||
stale_reservation_task.cancel()
|
stale_reservation_task.cancel()
|
||||||
if dead_key_prune_task is not None:
|
|
||||||
dead_key_prune_task.cancel()
|
|
||||||
if auto_topup_task is not None:
|
if auto_topup_task is not None:
|
||||||
auto_topup_task.cancel()
|
auto_topup_task.cancel()
|
||||||
if refund_sweep_task is not None:
|
if refund_sweep_task is not None:
|
||||||
@@ -210,8 +196,6 @@ async def lifespan(_: FastAPI) -> AsyncGenerator[None, None]:
|
|||||||
tasks_to_wait.append(key_reset_task)
|
tasks_to_wait.append(key_reset_task)
|
||||||
if stale_reservation_task is not None:
|
if stale_reservation_task is not None:
|
||||||
tasks_to_wait.append(stale_reservation_task)
|
tasks_to_wait.append(stale_reservation_task)
|
||||||
if dead_key_prune_task is not None:
|
|
||||||
tasks_to_wait.append(dead_key_prune_task)
|
|
||||||
if auto_topup_task is not None:
|
if auto_topup_task is not None:
|
||||||
tasks_to_wait.append(auto_topup_task)
|
tasks_to_wait.append(auto_topup_task)
|
||||||
if refund_sweep_task is not None:
|
if refund_sweep_task is not None:
|
||||||
@@ -379,10 +363,7 @@ if UI_DIST_PATH.exists() and UI_DIST_PATH.is_dir():
|
|||||||
|
|
||||||
else:
|
else:
|
||||||
logger.warning(
|
logger.warning(
|
||||||
"UI dist directory not found at %s; serving API only. Run `make ui-build` "
|
f"UI dist directory not found at {UI_DIST_PATH}, skipping static file serving"
|
||||||
"to build the static UI served from here, or `make ui-dev` for the Next.js "
|
|
||||||
"dev server with hot reload on :3000 (it targets this backend on :8000).",
|
|
||||||
UI_DIST_PATH,
|
|
||||||
)
|
)
|
||||||
|
|
||||||
@app.get("/", include_in_schema=False)
|
@app.get("/", include_in_schema=False)
|
||||||
|
|||||||
@@ -1,65 +0,0 @@
|
|||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
import re
|
|
||||||
from itertools import count
|
|
||||||
from typing import Collection
|
|
||||||
|
|
||||||
from sqlmodel import select
|
|
||||||
from sqlmodel.ext.asyncio.session import AsyncSession
|
|
||||||
|
|
||||||
from .db import UpstreamProviderRow
|
|
||||||
|
|
||||||
_SLUG_BASE_PATTERN = re.compile(r"[^a-z0-9]+")
|
|
||||||
_MAX_SLUG_LENGTH = 64
|
|
||||||
|
|
||||||
|
|
||||||
def provider_slug_base(provider_type: str) -> str:
|
|
||||||
"""Return a deterministic slug base for a provider type."""
|
|
||||||
base = _SLUG_BASE_PATTERN.sub("-", provider_type.lower()).strip("-")
|
|
||||||
if not base:
|
|
||||||
base = "provider"
|
|
||||||
elif base.isdigit():
|
|
||||||
base = f"provider-{base}"
|
|
||||||
elif len(base) < 3:
|
|
||||||
base = f"{base}-provider"
|
|
||||||
|
|
||||||
if len(base) > _MAX_SLUG_LENGTH:
|
|
||||||
base = base[:_MAX_SLUG_LENGTH].rstrip("-") or "provider"
|
|
||||||
return base
|
|
||||||
|
|
||||||
|
|
||||||
def provider_slug_candidate(base: str, suffix_number: int) -> str:
|
|
||||||
if suffix_number == 1:
|
|
||||||
return base
|
|
||||||
|
|
||||||
suffix = f"-{suffix_number}"
|
|
||||||
max_base_length = _MAX_SLUG_LENGTH - len(suffix)
|
|
||||||
return f"{base[:max_base_length].rstrip('-')}{suffix}"
|
|
||||||
|
|
||||||
|
|
||||||
async def allocate_unique_provider_slug(
|
|
||||||
session: AsyncSession,
|
|
||||||
provider_type: str,
|
|
||||||
reserved_slugs: Collection[str] = (),
|
|
||||||
) -> str:
|
|
||||||
"""Allocate a stable, deterministic provider slug.
|
|
||||||
|
|
||||||
The first provider of a type gets ``openai``; later collisions get
|
|
||||||
``openai-2``, ``openai-3``, etc. ``reserved_slugs`` covers rows staged in
|
|
||||||
memory but not flushed yet, such as settings/env seeding.
|
|
||||||
"""
|
|
||||||
base = provider_slug_base(provider_type)
|
|
||||||
reserved = {slug.lower() for slug in reserved_slugs}
|
|
||||||
|
|
||||||
for suffix_number in count(1):
|
|
||||||
candidate = provider_slug_candidate(base, suffix_number)
|
|
||||||
if candidate in reserved:
|
|
||||||
continue
|
|
||||||
|
|
||||||
result = await session.exec(
|
|
||||||
select(UpstreamProviderRow).where(UpstreamProviderRow.slug == candidate)
|
|
||||||
)
|
|
||||||
if result.first() is None:
|
|
||||||
return candidate
|
|
||||||
|
|
||||||
raise RuntimeError("unreachable")
|
|
||||||
@@ -1,51 +0,0 @@
|
|||||||
"""Redaction helpers for sensitive provider identifiers.
|
|
||||||
|
|
||||||
Single source of truth for stripping account-scoped identifiers (e.g. OpenAI
|
|
||||||
organization IDs) from any text before it is logged, returned to a caller, or
|
|
||||||
written to an audit entry.
|
|
||||||
"""
|
|
||||||
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
import re
|
|
||||||
from typing import Any
|
|
||||||
|
|
||||||
# OpenAI-style organization identifiers look like ``org-<base62>``. Require at
|
|
||||||
# least 6 trailing chars so the already-redacted literal ``org-[REDACTED]`` is
|
|
||||||
# never re-matched (``[`` is not in the character class).
|
|
||||||
_ORG_ID_PATTERN = re.compile(r"\borg-[A-Za-z0-9]{6,}\b")
|
|
||||||
|
|
||||||
ORG_ID_PLACEHOLDER = "org-[REDACTED]"
|
|
||||||
|
|
||||||
|
|
||||||
def redact_org_ids(text: str) -> str:
|
|
||||||
"""Replace OpenAI-style organization IDs with ``org-[REDACTED]``.
|
|
||||||
|
|
||||||
Args:
|
|
||||||
text: Arbitrary text that may embed an ``org-*`` identifier.
|
|
||||||
|
|
||||||
Returns:
|
|
||||||
The text with every organization ID replaced. Non-string input is
|
|
||||||
returned unchanged after coercion to ``str``.
|
|
||||||
"""
|
|
||||||
if not text:
|
|
||||||
return text
|
|
||||||
return _ORG_ID_PATTERN.sub(ORG_ID_PLACEHOLDER, text)
|
|
||||||
|
|
||||||
|
|
||||||
def redact_obj(obj: Any) -> Any:
|
|
||||||
"""Recursively redact organization IDs in arbitrary nested structures.
|
|
||||||
|
|
||||||
Strings are redacted in place; dicts and lists/tuples are walked so that
|
|
||||||
identifiers nested inside structured payloads (e.g. log ``extra`` fields or
|
|
||||||
error ``details``) are also stripped. Other types are returned unchanged.
|
|
||||||
"""
|
|
||||||
if isinstance(obj, str):
|
|
||||||
return redact_org_ids(obj)
|
|
||||||
if isinstance(obj, dict):
|
|
||||||
return {key: redact_obj(value) for key, value in obj.items()}
|
|
||||||
if isinstance(obj, list):
|
|
||||||
return [redact_obj(value) for value in obj]
|
|
||||||
if isinstance(obj, tuple):
|
|
||||||
return tuple(redact_obj(value) for value in obj)
|
|
||||||
return obj
|
|
||||||
@@ -73,14 +73,6 @@ class Settings(BaseSettings):
|
|||||||
stale_reservation_timeout_seconds: int = Field(
|
stale_reservation_timeout_seconds: int = Field(
|
||||||
default=300, env="STALE_RESERVATION_TIMEOUT_SECONDS"
|
default=300, env="STALE_RESERVATION_TIMEOUT_SECONDS"
|
||||||
)
|
)
|
||||||
# Background prune of dead (zero balance, never used) API keys.
|
|
||||||
# Interval 0 disables it; min-age is a grace period (default 1 week).
|
|
||||||
dead_key_prune_interval_seconds: int = Field(
|
|
||||||
default=3600, env="DEAD_KEY_PRUNE_INTERVAL_SECONDS"
|
|
||||||
)
|
|
||||||
dead_key_min_age_seconds: int = Field(
|
|
||||||
default=604_800, env="DEAD_KEY_MIN_AGE_SECONDS"
|
|
||||||
)
|
|
||||||
|
|
||||||
# Network
|
# Network
|
||||||
cors_origins: list[str] = Field(default_factory=lambda: ["*"], env="CORS_ORIGINS")
|
cors_origins: list[str] = Field(default_factory=lambda: ["*"], env="CORS_ORIGINS")
|
||||||
|
|||||||
@@ -418,21 +418,12 @@ def _calculate_from_tokens(
|
|||||||
},
|
},
|
||||||
)
|
)
|
||||||
|
|
||||||
# Fold the cache-read/write cost into the visible ``input_msats`` so a
|
visible_input_msats = int(calc_input_msats + calc_cache_read_msats)
|
||||||
# dashboard that renders I / O / T sees ``input + output == total``
|
|
||||||
# exactly. This mirrors ``_fold_cache_into_input_tokens`` (which rolls the
|
|
||||||
# cache token counts into the visible prompt total). The standalone
|
|
||||||
# ``cache_read_msats`` / ``cache_creation_msats`` fields stay populated for
|
|
||||||
# clients that want the breakdown; nothing sums the components to derive
|
|
||||||
# ``total_msats`` (it is computed independently above), so this is
|
|
||||||
# display-only and does not change what is billed.
|
|
||||||
visible_output_msats = int(calc_output_msats)
|
|
||||||
visible_input_msats = token_based_cost - visible_output_msats
|
|
||||||
|
|
||||||
return CostData(
|
return CostData(
|
||||||
base_msats=0,
|
base_msats=0,
|
||||||
input_msats=visible_input_msats,
|
input_msats=visible_input_msats,
|
||||||
output_msats=visible_output_msats,
|
output_msats=int(calc_output_msats),
|
||||||
total_msats=token_based_cost,
|
total_msats=token_based_cost,
|
||||||
total_usd=total_usd,
|
total_usd=total_usd,
|
||||||
input_tokens=input_tokens,
|
input_tokens=input_tokens,
|
||||||
|
|||||||
@@ -11,8 +11,6 @@ from PIL import Image
|
|||||||
from sqlmodel.ext.asyncio.session import AsyncSession
|
from sqlmodel.ext.asyncio.session import AsyncSession
|
||||||
|
|
||||||
from ..core import get_logger
|
from ..core import get_logger
|
||||||
from ..core.exceptions import UpstreamError
|
|
||||||
from ..core.redaction import redact_org_ids
|
|
||||||
from ..core.settings import settings
|
from ..core.settings import settings
|
||||||
from ..wallet import deserialize_token_from_string
|
from ..wallet import deserialize_token_from_string
|
||||||
|
|
||||||
@@ -420,27 +418,16 @@ def create_error_response(
|
|||||||
status_code: int,
|
status_code: int,
|
||||||
request: Request,
|
request: Request,
|
||||||
token: str | None = None,
|
token: str | None = None,
|
||||||
code: str | int | None = None,
|
|
||||||
details: dict[str, object] | None = None,
|
|
||||||
) -> Response:
|
) -> Response:
|
||||||
"""Create a standardized error response.
|
"""Create a standardized error response."""
|
||||||
|
|
||||||
``code`` is a stable, machine-readable classification (e.g.
|
|
||||||
``UPSTREAM_RATE_LIMIT``); when omitted it defaults to the HTTP status code
|
|
||||||
for backwards compatibility. ``details`` carries optional structured,
|
|
||||||
redaction-safe context.
|
|
||||||
"""
|
|
||||||
error_obj: dict[str, object] = {
|
|
||||||
"message": redact_org_ids(message),
|
|
||||||
"type": error_type,
|
|
||||||
"code": code if code is not None else status_code,
|
|
||||||
}
|
|
||||||
if details is not None:
|
|
||||||
error_obj["details"] = details
|
|
||||||
return Response(
|
return Response(
|
||||||
content=json.dumps(
|
content=json.dumps(
|
||||||
{
|
{
|
||||||
"error": error_obj,
|
"error": {
|
||||||
|
"message": message,
|
||||||
|
"type": error_type,
|
||||||
|
"code": status_code,
|
||||||
|
},
|
||||||
"request_id": getattr(request.state, "request_id", "unknown"),
|
"request_id": getattr(request.state, "request_id", "unknown"),
|
||||||
}
|
}
|
||||||
),
|
),
|
||||||
@@ -448,20 +435,3 @@ def create_error_response(
|
|||||||
media_type="application/json",
|
media_type="application/json",
|
||||||
headers={"X-Cashu": token} if token else {},
|
headers={"X-Cashu": token} if token else {},
|
||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
def create_upstream_error_response(
|
|
||||||
error: UpstreamError,
|
|
||||||
request: Request,
|
|
||||||
fallback_status: int = 502,
|
|
||||||
) -> Response:
|
|
||||||
"""Build an error response from an :class:`UpstreamError`, preserving its
|
|
||||||
structured ``code``, ``details``, and original ``status_code``."""
|
|
||||||
return create_error_response(
|
|
||||||
"upstream_error",
|
|
||||||
str(error),
|
|
||||||
error.status_code or fallback_status,
|
|
||||||
request=request,
|
|
||||||
code=getattr(error, "code", None),
|
|
||||||
details=getattr(error, "details", None),
|
|
||||||
)
|
|
||||||
|
|||||||
@@ -94,10 +94,7 @@ def backfill_cache_pricing(model_id: str, pricing: Pricing) -> Pricing:
|
|||||||
cache reads (DeepSeek hits are 10x cheaper) and undercharges Anthropic
|
cache reads (DeepSeek hits are 10x cheaper) and undercharges Anthropic
|
||||||
cache writes (1.25x). litellm ships per-model USD rates keyed by the exact
|
cache writes (1.25x). litellm ships per-model USD rates keyed by the exact
|
||||||
OpenRouter id (deepseek/deepseek-chat) or by the bare model name
|
OpenRouter id (deepseek/deepseek-chat) or by the bare model name
|
||||||
(gpt-4o, claude-sonnet-4-5), so both spellings are tried. litellm keys are
|
(gpt-4o, claude-sonnet-4-5), so both spellings are tried.
|
||||||
lowercase, but a generic upstream may report a mixed-case id
|
|
||||||
(``deepseek-ai/DeepSeek-V4-Flash``); an exact match is attempted first, then
|
|
||||||
a case-insensitive fallback so such ids still resolve.
|
|
||||||
|
|
||||||
Rates already present (e.g. provided by OpenRouter) are authoritative and
|
Rates already present (e.g. provided by OpenRouter) are authoritative and
|
||||||
never overwritten. Unknown models are returned unchanged.
|
never overwritten. Unknown models are returned unchanged.
|
||||||
@@ -109,26 +106,12 @@ def backfill_cache_pricing(model_id: str, pricing: Pricing) -> Pricing:
|
|||||||
|
|
||||||
import litellm
|
import litellm
|
||||||
|
|
||||||
candidates = (model_id, model_id.split("/", 1)[-1])
|
|
||||||
info: dict | None = None
|
info: dict | None = None
|
||||||
for key in candidates:
|
for key in (model_id, model_id.split("/", 1)[-1]):
|
||||||
candidate = litellm.model_cost.get(key)
|
candidate = litellm.model_cost.get(key)
|
||||||
if isinstance(candidate, dict):
|
if isinstance(candidate, dict):
|
||||||
info = candidate
|
info = candidate
|
||||||
break
|
break
|
||||||
if info is None:
|
|
||||||
# Case-insensitive fallback: a mixed-case upstream id (e.g.
|
|
||||||
# ``deepseek-ai/DeepSeek-V4-Flash``) won't match litellm's lowercase
|
|
||||||
# keys exactly. Build a lowercased index once and retry.
|
|
||||||
lowered = {c.lower() for c in candidates}
|
|
||||||
for key, candidate in litellm.model_cost.items():
|
|
||||||
if (
|
|
||||||
isinstance(key, str)
|
|
||||||
and key.lower() in lowered
|
|
||||||
and isinstance(candidate, dict)
|
|
||||||
):
|
|
||||||
info = candidate
|
|
||||||
break
|
|
||||||
if info is None:
|
if info is None:
|
||||||
return pricing
|
return pricing
|
||||||
|
|
||||||
@@ -232,29 +215,13 @@ def _row_to_model(
|
|||||||
)
|
)
|
||||||
top_provider_dict = json.loads(row.top_provider) if row.top_provider else None
|
top_provider_dict = json.loads(row.top_provider) if row.top_provider else None
|
||||||
|
|
||||||
|
if apply_provider_fee and isinstance(pricing, dict):
|
||||||
|
pricing = {k: float(v) * provider_fee for k, v in pricing.items()}
|
||||||
|
|
||||||
if isinstance(pricing, dict) and float(pricing.get("request", 0.0)) <= 0.0:
|
if isinstance(pricing, dict) and float(pricing.get("request", 0.0)) <= 0.0:
|
||||||
pricing["request"] = max(pricing.get("request", 0.0), 0.0)
|
pricing["request"] = max(pricing.get("request", 0.0), 0.0)
|
||||||
|
|
||||||
parsed_pricing = Pricing.parse_obj(pricing)
|
parsed_pricing = Pricing.parse_obj(pricing)
|
||||||
|
|
||||||
# Fill missing cache-read/write rates from litellm's cost map BEFORE applying
|
|
||||||
# the provider fee, so they carry the same markup as every other component.
|
|
||||||
# DB-stored override pricing (e.g. generic providers) omits cache rates;
|
|
||||||
# without this, ``_row_to_model`` bills cache reads at the full input rate —
|
|
||||||
# the ``_apply_provider_fee_to_model`` path backfills, but the override path
|
|
||||||
# used for admin-configured providers did not.
|
|
||||||
#
|
|
||||||
# Key on ``forwarded_model_id`` (the actual upstream model name litellm
|
|
||||||
# prices) when set: an alias row (id="local-alias",
|
|
||||||
# forwarded_model_id="deepseek-v4-flash") would otherwise look up the alias
|
|
||||||
# and miss the cache rate.
|
|
||||||
pricing_model_id = getattr(row, "forwarded_model_id", None) or row.id
|
|
||||||
parsed_pricing = backfill_cache_pricing(pricing_model_id, parsed_pricing)
|
|
||||||
|
|
||||||
if apply_provider_fee:
|
|
||||||
parsed_pricing = Pricing.parse_obj(
|
|
||||||
{k: float(v) * provider_fee for k, v in parsed_pricing.dict().items()}
|
|
||||||
)
|
|
||||||
model = Model(
|
model = Model(
|
||||||
id=row.id,
|
id=row.id,
|
||||||
name=row.name,
|
name=row.name,
|
||||||
|
|||||||
+13
-7
@@ -24,7 +24,6 @@ from .payment.helpers import (
|
|||||||
calculate_discounted_max_cost,
|
calculate_discounted_max_cost,
|
||||||
check_token_balance,
|
check_token_balance,
|
||||||
create_error_response,
|
create_error_response,
|
||||||
create_upstream_error_response,
|
|
||||||
get_max_cost_for_model,
|
get_max_cost_for_model,
|
||||||
)
|
)
|
||||||
from .payment.models import Model
|
from .payment.models import Model
|
||||||
@@ -212,7 +211,9 @@ async def proxy(
|
|||||||
e,
|
e,
|
||||||
)
|
)
|
||||||
if i == len(all_upstreams) - 1:
|
if i == len(all_upstreams) - 1:
|
||||||
last_error_response = create_upstream_error_response(e, request)
|
last_error_response = create_error_response(
|
||||||
|
"upstream_error", str(e), 502, request=request
|
||||||
|
)
|
||||||
continue
|
continue
|
||||||
return last_error_response or create_error_response(
|
return last_error_response or create_error_response(
|
||||||
"upstream_error", "All upstreams failed", 502, request=request
|
"upstream_error", "All upstreams failed", 502, request=request
|
||||||
@@ -282,10 +283,11 @@ async def proxy(
|
|||||||
last_error = e
|
last_error = e
|
||||||
continue
|
continue
|
||||||
|
|
||||||
if last_error is not None:
|
|
||||||
return create_upstream_error_response(last_error, request)
|
|
||||||
return create_error_response(
|
return create_error_response(
|
||||||
"upstream_error", "All upstreams failed", 502, request=request
|
"upstream_error",
|
||||||
|
str(last_error) if last_error else "All upstreams failed",
|
||||||
|
502,
|
||||||
|
request=request,
|
||||||
)
|
)
|
||||||
|
|
||||||
elif auth := headers.get("authorization", None):
|
elif auth := headers.get("authorization", None):
|
||||||
@@ -341,7 +343,9 @@ async def proxy(
|
|||||||
except UpstreamError as e:
|
except UpstreamError as e:
|
||||||
logger.warning(f"Upstream {upstream.provider_type} failed (GET): {e}")
|
logger.warning(f"Upstream {upstream.provider_type} failed (GET): {e}")
|
||||||
if i == len(upstreams) - 1:
|
if i == len(upstreams) - 1:
|
||||||
last_error_response = create_upstream_error_response(e, request)
|
last_error_response = create_error_response(
|
||||||
|
"upstream_error", str(e), 502, request=request
|
||||||
|
)
|
||||||
continue
|
continue
|
||||||
return last_error_response or create_error_response(
|
return last_error_response or create_error_response(
|
||||||
"upstream_error", "All upstreams failed", 502, request=request
|
"upstream_error", "All upstreams failed", 502, request=request
|
||||||
@@ -521,7 +525,9 @@ async def proxy(
|
|||||||
# If this was the last provider
|
# If this was the last provider
|
||||||
if i == len(upstreams) - 1:
|
if i == len(upstreams) - 1:
|
||||||
await revert_pay_for_request(key, session, max_cost_for_model)
|
await revert_pay_for_request(key, session, max_cost_for_model)
|
||||||
return create_upstream_error_response(e, request)
|
return create_error_response(
|
||||||
|
"upstream_error", str(e), 502, request=request
|
||||||
|
)
|
||||||
|
|
||||||
# Otherwise loop continues to next provider
|
# Otherwise loop continues to next provider
|
||||||
continue
|
continue
|
||||||
|
|||||||
@@ -24,7 +24,7 @@ class AnthropicUpstreamProvider(BaseUpstreamProvider):
|
|||||||
)
|
)
|
||||||
|
|
||||||
@classmethod
|
@classmethod
|
||||||
def _build_from_row(
|
def from_db_row(
|
||||||
cls, provider_row: "UpstreamProviderRow"
|
cls, provider_row: "UpstreamProviderRow"
|
||||||
) -> "AnthropicUpstreamProvider":
|
) -> "AnthropicUpstreamProvider":
|
||||||
return cls(
|
return cls(
|
||||||
|
|||||||
@@ -97,8 +97,6 @@ async def _check_and_topup(row: UpstreamProviderRow) -> None:
|
|||||||
|
|
||||||
# Instantiate provider and check balance
|
# Instantiate provider and check balance
|
||||||
provider = RoutstrUpstreamProvider.from_db_row(row)
|
provider = RoutstrUpstreamProvider.from_db_row(row)
|
||||||
if provider is None:
|
|
||||||
return
|
|
||||||
balance = await provider.get_balance()
|
balance = await provider.get_balance()
|
||||||
|
|
||||||
if balance is None:
|
if balance is None:
|
||||||
|
|||||||
@@ -38,7 +38,7 @@ class AzureUpstreamProvider(BaseUpstreamProvider):
|
|||||||
self.api_version = api_version
|
self.api_version = api_version
|
||||||
|
|
||||||
@classmethod
|
@classmethod
|
||||||
def _build_from_row(
|
def from_db_row(
|
||||||
cls, provider_row: "UpstreamProviderRow"
|
cls, provider_row: "UpstreamProviderRow"
|
||||||
) -> "AzureUpstreamProvider | None":
|
) -> "AzureUpstreamProvider | None":
|
||||||
if not provider_row.api_version:
|
if not provider_row.api_version:
|
||||||
|
|||||||
+125
-246
@@ -2,16 +2,16 @@ from __future__ import annotations
|
|||||||
|
|
||||||
import asyncio
|
import asyncio
|
||||||
import json
|
import json
|
||||||
import math
|
|
||||||
import traceback
|
import traceback
|
||||||
import uuid
|
import uuid
|
||||||
from collections.abc import AsyncGenerator, AsyncIterator, Iterator
|
from collections.abc import AsyncGenerator, AsyncIterator, Iterator
|
||||||
from typing import Any, Mapping, Self, cast
|
from typing import Any, Mapping, cast
|
||||||
|
|
||||||
import httpx
|
import httpx
|
||||||
from fastapi import BackgroundTasks, HTTPException, Request
|
from fastapi import BackgroundTasks, HTTPException, Request
|
||||||
from fastapi.responses import Response, StreamingResponse
|
from fastapi.responses import Response, StreamingResponse
|
||||||
from pydantic.v1 import BaseModel
|
from pydantic.v1 import BaseModel
|
||||||
|
from sqlmodel import select
|
||||||
|
|
||||||
from ..auth import adjust_payment_for_tokens
|
from ..auth import adjust_payment_for_tokens
|
||||||
from ..core import get_logger
|
from ..core import get_logger
|
||||||
@@ -23,7 +23,6 @@ from ..core.db import (
|
|||||||
store_cashu_transaction,
|
store_cashu_transaction,
|
||||||
)
|
)
|
||||||
from ..core.exceptions import UpstreamError
|
from ..core.exceptions import UpstreamError
|
||||||
from ..core.redaction import redact_org_ids
|
|
||||||
from ..payment.cost_calculation import (
|
from ..payment.cost_calculation import (
|
||||||
CostData,
|
CostData,
|
||||||
CostDataError,
|
CostDataError,
|
||||||
@@ -40,12 +39,8 @@ from ..payment.models import (
|
|||||||
list_models,
|
list_models,
|
||||||
)
|
)
|
||||||
from ..payment.price import sats_usd_price
|
from ..payment.price import sats_usd_price
|
||||||
from ..wallet import (
|
from ..payment.usage import normalize_usage
|
||||||
SPENT_TOKEN_CODES,
|
from ..wallet import recieve_token, send_token
|
||||||
classify_redemption_error,
|
|
||||||
recieve_token,
|
|
||||||
send_token,
|
|
||||||
)
|
|
||||||
from . import messages_dispatch
|
from . import messages_dispatch
|
||||||
from .cache_breakpoints import (
|
from .cache_breakpoints import (
|
||||||
inject_anthropic_cache_breakpoints,
|
inject_anthropic_cache_breakpoints,
|
||||||
@@ -53,7 +48,6 @@ from .cache_breakpoints import (
|
|||||||
)
|
)
|
||||||
from .count_tokens import count_tokens_locally
|
from .count_tokens import count_tokens_locally
|
||||||
from .litellm_routing import detect_litellm_prefix
|
from .litellm_routing import detect_litellm_prefix
|
||||||
from .rate_limit import UPSTREAM_RATE_LIMIT, classify_rate_limit
|
|
||||||
|
|
||||||
logger = get_logger(__name__)
|
logger = get_logger(__name__)
|
||||||
|
|
||||||
@@ -96,11 +90,6 @@ class BaseUpstreamProvider:
|
|||||||
base_url: str
|
base_url: str
|
||||||
api_key: str
|
api_key: str
|
||||||
provider_fee: float = 1.05
|
provider_fee: float = 1.05
|
||||||
# Primary key of the ``upstream_providers`` row this instance was built
|
|
||||||
# from. Set by ``from_db_row`` so a live provider can re-find its own row by
|
|
||||||
# stable identity instead of its rotatable ``api_key``. ``None`` for
|
|
||||||
# instances not sourced from a row.
|
|
||||||
db_id: int | None = None
|
|
||||||
_models_cache: list[Model] = []
|
_models_cache: list[Model] = []
|
||||||
_models_by_id: dict[str, Model] = {}
|
_models_by_id: dict[str, Model] = {}
|
||||||
|
|
||||||
@@ -115,7 +104,6 @@ class BaseUpstreamProvider:
|
|||||||
self.base_url = base_url
|
self.base_url = base_url
|
||||||
self.api_key = api_key
|
self.api_key = api_key
|
||||||
self.provider_fee = provider_fee
|
self.provider_fee = provider_fee
|
||||||
self.db_id = None
|
|
||||||
self._models_cache = []
|
self._models_cache = []
|
||||||
self._models_by_id = {}
|
self._models_by_id = {}
|
||||||
|
|
||||||
@@ -133,13 +121,10 @@ class BaseUpstreamProvider:
|
|||||||
return detect_litellm_prefix(self.base_url)
|
return detect_litellm_prefix(self.base_url)
|
||||||
|
|
||||||
@classmethod
|
@classmethod
|
||||||
def from_db_row(cls, provider_row: "UpstreamProviderRow") -> "Self | None":
|
def from_db_row(
|
||||||
"""Instantiate a provider from a database row, carrying its identity.
|
cls, provider_row: "UpstreamProviderRow"
|
||||||
|
) -> "BaseUpstreamProvider | None":
|
||||||
Construction itself is delegated to the ``_build_from_row`` hook (which
|
"""Factory method to instantiate provider from database row.
|
||||||
subclasses override to match their constructor); this wrapper stamps the
|
|
||||||
row's primary key onto the instance as ``db_id`` so the provider can
|
|
||||||
later re-find its own row by identity rather than by its ``api_key``.
|
|
||||||
|
|
||||||
Args:
|
Args:
|
||||||
provider_row: Database row containing provider configuration
|
provider_row: Database row containing provider configuration
|
||||||
@@ -147,19 +132,6 @@ class BaseUpstreamProvider:
|
|||||||
Returns:
|
Returns:
|
||||||
Instantiated provider or None if instantiation fails
|
Instantiated provider or None if instantiation fails
|
||||||
"""
|
"""
|
||||||
provider = cls._build_from_row(provider_row)
|
|
||||||
if provider is not None:
|
|
||||||
provider.db_id = provider_row.id
|
|
||||||
return provider
|
|
||||||
|
|
||||||
@classmethod
|
|
||||||
def _build_from_row(cls, provider_row: "UpstreamProviderRow") -> "Self | None":
|
|
||||||
"""Construct the provider instance from a row (no identity stamping).
|
|
||||||
|
|
||||||
Overridden by subclasses whose constructors differ from the base
|
|
||||||
``(base_url, api_key, provider_fee)`` shape. Callers should use
|
|
||||||
``from_db_row`` instead, which also attaches ``db_id``.
|
|
||||||
"""
|
|
||||||
return cls(
|
return cls(
|
||||||
base_url=provider_row.base_url,
|
base_url=provider_row.base_url,
|
||||||
api_key=provider_row.api_key,
|
api_key=provider_row.api_key,
|
||||||
@@ -186,48 +158,56 @@ class BaseUpstreamProvider:
|
|||||||
|
|
||||||
@staticmethod
|
@staticmethod
|
||||||
def _fold_cache_into_input_tokens(usage: object) -> None:
|
def _fold_cache_into_input_tokens(usage: object) -> None:
|
||||||
"""Fold cache token counts into ``input_tokens`` / ``prompt_tokens``.
|
"""Fold additive cache token counts into Anthropic ``input_tokens``.
|
||||||
|
|
||||||
Cost calculation has already used the per-bucket counts to bill the
|
Cost calculation has already used the per-bucket counts to bill the
|
||||||
request correctly; what the client sees in the visible token total
|
request correctly; what the client sees in Anthropic-shaped visible
|
||||||
should be a single rolled-up prompt count *including* the cache
|
token totals should be a single rolled-up input count *including* the
|
||||||
portion. The standalone ``cache_read_input_tokens`` /
|
cache portion. OpenAI-compatible ``prompt_tokens`` is already inclusive
|
||||||
|
(DeepSeek, OpenAI, OpenRouter, litellm), so adding cache fields there
|
||||||
|
would double-count.
|
||||||
|
|
||||||
|
The standalone ``cache_read_input_tokens`` /
|
||||||
``cache_creation_input_tokens`` fields are left in place for clients
|
``cache_creation_input_tokens`` fields are left in place for clients
|
||||||
that want the breakdown.
|
that want the breakdown.
|
||||||
|
|
||||||
For Anthropic-shaped responses (``input_tokens`` present), the cache
|
|
||||||
fields are forced to ``0`` when the upstream omitted them, so the
|
|
||||||
client always sees a consistent shape.
|
|
||||||
"""
|
"""
|
||||||
if not isinstance(usage, dict):
|
if not isinstance(usage, dict):
|
||||||
return
|
return
|
||||||
|
|
||||||
# Normalise missing cache fields to 0 on Anthropic-shaped usage so
|
# ``prompt_tokens`` is an inclusive OpenAI-compatible grand total.
|
||||||
# downstream consumers can rely on them being present.
|
# Fold only native Anthropic-style usage where ``input_tokens`` excludes
|
||||||
if "input_tokens" in usage:
|
# cache reads/writes.
|
||||||
usage.setdefault("cache_read_input_tokens", 0)
|
if "input_tokens" not in usage or "prompt_tokens" in usage:
|
||||||
usage.setdefault("cache_creation_input_tokens", 0)
|
return
|
||||||
|
|
||||||
|
usage.setdefault("cache_read_input_tokens", 0)
|
||||||
|
usage.setdefault("cache_creation_input_tokens", 0)
|
||||||
|
|
||||||
try:
|
try:
|
||||||
cache_read = int(usage.get("cache_read_input_tokens") or 0)
|
cache_read = int(usage.get("cache_read_input_tokens") or 0)
|
||||||
cache_creation = int(usage.get("cache_creation_input_tokens") or 0)
|
cache_creation = int(usage.get("cache_creation_input_tokens") or 0)
|
||||||
|
input_tokens = int(usage.get("input_tokens") or 0)
|
||||||
except (TypeError, ValueError):
|
except (TypeError, ValueError):
|
||||||
return
|
return
|
||||||
extra = cache_read + cache_creation
|
extra = cache_read + cache_creation
|
||||||
if extra <= 0:
|
if extra > 0:
|
||||||
|
usage["input_tokens"] = input_tokens + extra
|
||||||
|
|
||||||
|
@staticmethod
|
||||||
|
def _add_normalized_usage_fields(response_json: object) -> None:
|
||||||
|
"""Preserve raw usage while adding canonical fields for billing/display."""
|
||||||
|
if not isinstance(response_json, dict):
|
||||||
return
|
return
|
||||||
if "input_tokens" in usage:
|
usage = response_json.get("usage")
|
||||||
try:
|
if not isinstance(usage, dict):
|
||||||
usage["input_tokens"] = int(usage.get("input_tokens") or 0) + extra
|
return
|
||||||
except (TypeError, ValueError):
|
normalized = normalize_usage(usage)
|
||||||
pass
|
if normalized is None:
|
||||||
if "prompt_tokens" in usage:
|
return
|
||||||
try:
|
usage.setdefault("input_tokens", normalized.input_tokens)
|
||||||
usage["prompt_tokens"] = (
|
usage.setdefault("output_tokens", normalized.output_tokens)
|
||||||
int(usage.get("prompt_tokens") or 0) + extra
|
usage.setdefault("cache_read_input_tokens", normalized.cache_read_tokens)
|
||||||
)
|
usage.setdefault("cache_creation_input_tokens", normalized.cache_write_tokens)
|
||||||
except (TypeError, ValueError):
|
|
||||||
pass
|
|
||||||
|
|
||||||
def _apply_provider_field(self, response_json: object) -> None:
|
def _apply_provider_field(self, response_json: object) -> None:
|
||||||
"""Stamp the routstr ``provider`` field onto an upstream response payload.
|
"""Stamp the routstr ``provider`` field onto an upstream response payload.
|
||||||
@@ -611,7 +591,7 @@ class BaseUpstreamProvider:
|
|||||||
preview = body_bytes.decode("utf-8", errors="ignore").strip()
|
preview = body_bytes.decode("utf-8", errors="ignore").strip()
|
||||||
if preview:
|
if preview:
|
||||||
message = preview[:500]
|
message = preview[:500]
|
||||||
return redact_org_ids(message), upstream_code
|
return message, upstream_code
|
||||||
|
|
||||||
async def on_upstream_error_redirect(
|
async def on_upstream_error_redirect(
|
||||||
self, status_code: int, error_message: str
|
self, status_code: int, error_message: str
|
||||||
@@ -654,23 +634,10 @@ class BaseUpstreamProvider:
|
|||||||
body_bytes = b""
|
body_bytes = b""
|
||||||
body_read_error = f"{type(exc).__name__}: {exc}"
|
body_read_error = f"{type(exc).__name__}: {exc}"
|
||||||
|
|
||||||
# ``message`` is already redacted by ``_extract_upstream_error_message``;
|
|
||||||
# the raw body preview is redacted here before it reaches logs or the
|
|
||||||
# forwarded envelope so provider account identifiers never leak.
|
|
||||||
message, upstream_code = self._extract_upstream_error_message(body_bytes)
|
message, upstream_code = self._extract_upstream_error_message(body_bytes)
|
||||||
body_preview = redact_org_ids(
|
body_preview = body_bytes.decode("utf-8", errors="ignore").strip()[:500]
|
||||||
body_bytes.decode("utf-8", errors="ignore").strip()[:500]
|
|
||||||
)
|
|
||||||
is_json_body = _is_json_content_type(content_type)
|
is_json_body = _is_json_content_type(content_type)
|
||||||
|
|
||||||
# Classify upstream rate-limit failures into a stable, structured error.
|
|
||||||
rate_limit = classify_rate_limit(status_code, message, headers)
|
|
||||||
error_code: str | int = upstream_code or status_code
|
|
||||||
error_details: dict[str, object] | None = None
|
|
||||||
if rate_limit is not None:
|
|
||||||
error_code = UPSTREAM_RATE_LIMIT
|
|
||||||
error_details = rate_limit.as_details()
|
|
||||||
|
|
||||||
logger.warning(
|
logger.warning(
|
||||||
"Upstream %s returned %s for model=%s path=%s: %s",
|
"Upstream %s returned %s for model=%s path=%s: %s",
|
||||||
self.provider_type,
|
self.provider_type,
|
||||||
@@ -684,7 +651,6 @@ class BaseUpstreamProvider:
|
|||||||
"model": model_id or "unknown",
|
"model": model_id or "unknown",
|
||||||
"upstream_status": status_code,
|
"upstream_status": status_code,
|
||||||
"upstream_code": upstream_code,
|
"upstream_code": upstream_code,
|
||||||
"error_code": error_code,
|
|
||||||
"upstream_content_type": content_type,
|
"upstream_content_type": content_type,
|
||||||
"upstream_request_id": upstream_request_id,
|
"upstream_request_id": upstream_request_id,
|
||||||
"message_preview": message[:200],
|
"message_preview": message[:200],
|
||||||
@@ -719,41 +685,13 @@ class BaseUpstreamProvider:
|
|||||||
):
|
):
|
||||||
headers.pop(header_name, None)
|
headers.pop(header_name, None)
|
||||||
|
|
||||||
# Propagate a usable retry hint to the caller when the upstream supplied
|
|
||||||
# one but did not echo a ``Retry-After`` header. RFC 7231 delta-seconds
|
|
||||||
# is an integer, so round sub-second hints up to a usable ``1``.
|
|
||||||
if (
|
|
||||||
rate_limit is not None
|
|
||||||
and rate_limit.retry_after_seconds is not None
|
|
||||||
and "retry-after" not in {k.lower() for k in headers}
|
|
||||||
):
|
|
||||||
headers["Retry-After"] = str(max(1, math.ceil(rate_limit.retry_after_seconds)))
|
|
||||||
|
|
||||||
if is_json_body:
|
if is_json_body:
|
||||||
if not content_type:
|
if not content_type:
|
||||||
headers.pop("content-type", None)
|
headers.pop("content-type", None)
|
||||||
headers.pop("Content-Type", None)
|
headers.pop("Content-Type", None)
|
||||||
media_type = content_type or None
|
media_type = content_type or None
|
||||||
# Re-serialise the body with organization IDs stripped. The narrow
|
|
||||||
# ``org-*`` regex preserves the surrounding JSON structure.
|
|
||||||
redacted_text = redact_org_ids(body_bytes.decode("utf-8", errors="ignore"))
|
|
||||||
redacted_body = redacted_text.encode()
|
|
||||||
# Surface the stable rate-limit classification on the forwarded
|
|
||||||
# body so callers can switch on ``error.code`` without parsing the
|
|
||||||
# provider-specific message. Fall back to the redacted bytes if the
|
|
||||||
# body is not a JSON object with an ``error`` mapping.
|
|
||||||
if rate_limit is not None:
|
|
||||||
try:
|
|
||||||
parsed = json.loads(redacted_text)
|
|
||||||
err = parsed.get("error") if isinstance(parsed, dict) else None
|
|
||||||
if isinstance(err, dict):
|
|
||||||
err["code"] = UPSTREAM_RATE_LIMIT
|
|
||||||
err["details"] = error_details
|
|
||||||
redacted_body = json.dumps(parsed).encode()
|
|
||||||
except (ValueError, AttributeError):
|
|
||||||
pass
|
|
||||||
return Response(
|
return Response(
|
||||||
content=redacted_body,
|
content=body_bytes,
|
||||||
status_code=status_code,
|
status_code=status_code,
|
||||||
headers=headers,
|
headers=headers,
|
||||||
media_type=media_type,
|
media_type=media_type,
|
||||||
@@ -764,18 +702,15 @@ class BaseUpstreamProvider:
|
|||||||
for header_name in ("content-type", "Content-Type"):
|
for header_name in ("content-type", "Content-Type"):
|
||||||
headers.pop(header_name, None)
|
headers.pop(header_name, None)
|
||||||
|
|
||||||
error_obj: dict[str, object] = {
|
|
||||||
"message": message or "Upstream returned a non-JSON error response",
|
|
||||||
"type": "upstream_error",
|
|
||||||
"code": error_code,
|
|
||||||
"upstream_status": status_code,
|
|
||||||
"upstream_content_type": content_type or None,
|
|
||||||
"upstream_body_preview": body_preview or None,
|
|
||||||
}
|
|
||||||
if error_details is not None:
|
|
||||||
error_obj["details"] = error_details
|
|
||||||
envelope = {
|
envelope = {
|
||||||
"error": error_obj,
|
"error": {
|
||||||
|
"message": message or "Upstream returned a non-JSON error response",
|
||||||
|
"type": "upstream_error",
|
||||||
|
"code": upstream_code or status_code,
|
||||||
|
"upstream_status": status_code,
|
||||||
|
"upstream_content_type": content_type or None,
|
||||||
|
"upstream_body_preview": body_preview or None,
|
||||||
|
},
|
||||||
"request_id": getattr(request.state, "request_id", None),
|
"request_id": getattr(request.state, "request_id", None),
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -840,9 +775,7 @@ class BaseUpstreamProvider:
|
|||||||
except Exception:
|
except Exception:
|
||||||
pass
|
pass
|
||||||
|
|
||||||
def _process_event(
|
def _process_event(raw_event: bytes) -> Iterator[bytes]:
|
||||||
raw_event: bytes, final: bool = False
|
|
||||||
) -> Iterator[bytes]:
|
|
||||||
"""Process one complete SSE event block (lines up to a blank line).
|
"""Process one complete SSE event block (lines up to a blank line).
|
||||||
|
|
||||||
Handles arbitrary upstream framing across every supported
|
Handles arbitrary upstream framing across every supported
|
||||||
@@ -934,6 +867,7 @@ class BaseUpstreamProvider:
|
|||||||
k: v for k, v in obj.items() if k != "choices"
|
k: v for k, v in obj.items() if k != "choices"
|
||||||
}
|
}
|
||||||
usage_chunk_data["choices"] = []
|
usage_chunk_data["choices"] = []
|
||||||
|
self._add_normalized_usage_fields(usage_chunk_data)
|
||||||
# Forward the content now, without usage, so token
|
# Forward the content now, without usage, so token
|
||||||
# usage is reported exactly once (in the trailer).
|
# usage is reported exactly once (in the trailer).
|
||||||
forward = {k: v for k, v in obj.items() if k != "usage"}
|
forward = {k: v for k, v in obj.items() if k != "usage"}
|
||||||
@@ -945,15 +879,10 @@ class BaseUpstreamProvider:
|
|||||||
)
|
)
|
||||||
return
|
return
|
||||||
usage_chunk_data = obj
|
usage_chunk_data = obj
|
||||||
|
self._add_normalized_usage_fields(usage_chunk_data)
|
||||||
return
|
return
|
||||||
yield prefix + b"data: " + json.dumps(obj).encode() + b"\n\n"
|
yield prefix + b"data: " + json.dumps(obj).encode() + b"\n\n"
|
||||||
else:
|
else:
|
||||||
if final:
|
|
||||||
# Final flush of a truncated tail: the upstream closed
|
|
||||||
# mid-event, so ``data`` is incomplete JSON. Emitting it
|
|
||||||
# as a ``data:`` frame would hand the client invalid
|
|
||||||
# JSON (the "unexpected token" parse error). Drop it.
|
|
||||||
return
|
|
||||||
# Non-JSON data payload (partial fragment already reassembled
|
# Non-JSON data payload (partial fragment already reassembled
|
||||||
# by buffering, or a provider control string). Re-prefix each
|
# by buffering, or a provider control string). Re-prefix each
|
||||||
# line so multi-line ``data`` stays valid SSE framing - a bare
|
# line so multi-line ``data`` stays valid SSE framing - a bare
|
||||||
@@ -972,13 +901,7 @@ class BaseUpstreamProvider:
|
|||||||
# boundary-independent for every provider.
|
# boundary-independent for every provider.
|
||||||
buffer = b""
|
buffer = b""
|
||||||
async for chunk in response.aiter_bytes():
|
async for chunk in response.aiter_bytes():
|
||||||
# Normalize the *joined* buffer, not each chunk in
|
buffer += chunk.replace(b"\r\n", b"\n")
|
||||||
# isolation: a CRLF event delimiter can straddle two
|
|
||||||
# ``aiter_bytes`` chunks (``...\r`` then ``\n...``). A
|
|
||||||
# per-chunk replace would leave a stray ``\r`` and the
|
|
||||||
# ``\n\n`` split would miss the delimiter, merging two
|
|
||||||
# events into one frame and breaking SSE clients.
|
|
||||||
buffer = (buffer + chunk).replace(b"\r\n", b"\n")
|
|
||||||
while b"\n\n" in buffer:
|
while b"\n\n" in buffer:
|
||||||
raw_event, buffer = buffer.split(b"\n\n", 1)
|
raw_event, buffer = buffer.split(b"\n\n", 1)
|
||||||
for out in _process_event(raw_event):
|
for out in _process_event(raw_event):
|
||||||
@@ -986,7 +909,7 @@ class BaseUpstreamProvider:
|
|||||||
|
|
||||||
# Flush any trailing event that lacked a final blank line.
|
# Flush any trailing event that lacked a final blank line.
|
||||||
if buffer.strip():
|
if buffer.strip():
|
||||||
for out in _process_event(buffer, final=True):
|
for out in _process_event(buffer):
|
||||||
yield out
|
yield out
|
||||||
|
|
||||||
async with create_session() as session:
|
async with create_session() as session:
|
||||||
@@ -994,6 +917,8 @@ class BaseUpstreamProvider:
|
|||||||
if fresh_key:
|
if fresh_key:
|
||||||
cost_data: dict
|
cost_data: dict
|
||||||
try:
|
try:
|
||||||
|
if usage_chunk_data is not None:
|
||||||
|
self._add_normalized_usage_fields(usage_chunk_data)
|
||||||
adjustment_input = (
|
adjustment_input = (
|
||||||
usage_chunk_data
|
usage_chunk_data
|
||||||
if usage_chunk_data is not None
|
if usage_chunk_data is not None
|
||||||
@@ -1292,9 +1217,7 @@ class BaseUpstreamProvider:
|
|||||||
except Exception:
|
except Exception:
|
||||||
pass
|
pass
|
||||||
|
|
||||||
def _process_event(
|
def _process_event(raw_event: bytes) -> Iterator[bytes]:
|
||||||
raw_event: bytes, final: bool = False
|
|
||||||
) -> Iterator[bytes]:
|
|
||||||
"""Process one complete SSE event block for the Responses API.
|
"""Process one complete SSE event block for the Responses API.
|
||||||
|
|
||||||
Buffers full events (delimited by a blank line) so parsing is
|
Buffers full events (delimited by a blank line) so parsing is
|
||||||
@@ -1364,11 +1287,6 @@ class BaseUpstreamProvider:
|
|||||||
|
|
||||||
yield prefix + b"data: " + json.dumps(obj).encode() + b"\n\n"
|
yield prefix + b"data: " + json.dumps(obj).encode() + b"\n\n"
|
||||||
else:
|
else:
|
||||||
if final:
|
|
||||||
# Final flush of a truncated tail: upstream closed
|
|
||||||
# mid-event, so ``data`` is incomplete JSON. Dropping it
|
|
||||||
# avoids handing the client an invalid ``data:`` frame.
|
|
||||||
return
|
|
||||||
# Re-prefix each line so multi-line ``data`` stays valid SSE
|
# Re-prefix each line so multi-line ``data`` stays valid SSE
|
||||||
# framing for the client.
|
# framing for the client.
|
||||||
body = b"".join(
|
body = b"".join(
|
||||||
@@ -1381,20 +1299,14 @@ class BaseUpstreamProvider:
|
|||||||
# delimiter so parsing is independent of byte boundaries.
|
# delimiter so parsing is independent of byte boundaries.
|
||||||
buffer = b""
|
buffer = b""
|
||||||
async for chunk in response.aiter_bytes():
|
async for chunk in response.aiter_bytes():
|
||||||
# Normalize the *joined* buffer, not each chunk in
|
buffer += chunk.replace(b"\r\n", b"\n")
|
||||||
# isolation: a CRLF event delimiter can straddle two
|
|
||||||
# ``aiter_bytes`` chunks (``...\r`` then ``\n...``). A
|
|
||||||
# per-chunk replace would leave a stray ``\r`` and the
|
|
||||||
# ``\n\n`` split would miss the delimiter, merging two
|
|
||||||
# events into one frame and breaking SSE clients.
|
|
||||||
buffer = (buffer + chunk).replace(b"\r\n", b"\n")
|
|
||||||
while b"\n\n" in buffer:
|
while b"\n\n" in buffer:
|
||||||
raw_event, buffer = buffer.split(b"\n\n", 1)
|
raw_event, buffer = buffer.split(b"\n\n", 1)
|
||||||
for out in _process_event(raw_event):
|
for out in _process_event(raw_event):
|
||||||
yield out
|
yield out
|
||||||
|
|
||||||
if buffer.strip():
|
if buffer.strip():
|
||||||
for out in _process_event(buffer, final=True):
|
for out in _process_event(buffer):
|
||||||
yield out
|
yield out
|
||||||
|
|
||||||
# Always emit a cost-bearing data chunk
|
# Always emit a cost-bearing data chunk
|
||||||
@@ -2595,16 +2507,9 @@ class BaseUpstreamProvider:
|
|||||||
body_bytes = await response.aread()
|
body_bytes = await response.aread()
|
||||||
except Exception:
|
except Exception:
|
||||||
body_bytes = b""
|
body_bytes = b""
|
||||||
# Redact provider account identifiers before the body text
|
body_preview = body_bytes.decode(
|
||||||
# reaches logs or the raised error.
|
"utf-8", errors="ignore"
|
||||||
body_preview = redact_org_ids(
|
).strip()[:500]
|
||||||
body_bytes.decode("utf-8", errors="ignore").strip()[:500]
|
|
||||||
)
|
|
||||||
rate_limit = classify_rate_limit(
|
|
||||||
response.status_code,
|
|
||||||
body_preview,
|
|
||||||
dict(response.headers),
|
|
||||||
)
|
|
||||||
logger.error(
|
logger.error(
|
||||||
"Upstream %s returned %s for model=%s path=%s: %s",
|
"Upstream %s returned %s for model=%s path=%s: %s",
|
||||||
self.provider_type,
|
self.provider_type,
|
||||||
@@ -2616,7 +2521,6 @@ class BaseUpstreamProvider:
|
|||||||
"provider": self.provider_type,
|
"provider": self.provider_type,
|
||||||
"model": original_model_id or "unknown",
|
"model": original_model_id or "unknown",
|
||||||
"status_code": response.status_code,
|
"status_code": response.status_code,
|
||||||
"error_code": rate_limit.code if rate_limit else None,
|
|
||||||
"reason_phrase": response.reason_phrase,
|
"reason_phrase": response.reason_phrase,
|
||||||
"path": path,
|
"path": path,
|
||||||
"body_preview": body_preview,
|
"body_preview": body_preview,
|
||||||
@@ -2629,8 +2533,6 @@ class BaseUpstreamProvider:
|
|||||||
f"for model {original_model_id or 'unknown'}: "
|
f"for model {original_model_id or 'unknown'}: "
|
||||||
f"{body_preview[:200] or '<empty>'}",
|
f"{body_preview[:200] or '<empty>'}",
|
||||||
status_code=response.status_code,
|
status_code=response.status_code,
|
||||||
code=rate_limit.code if rate_limit else None,
|
|
||||||
details=rate_limit.as_details() if rate_limit else None,
|
|
||||||
)
|
)
|
||||||
|
|
||||||
try:
|
try:
|
||||||
@@ -2927,16 +2829,9 @@ class BaseUpstreamProvider:
|
|||||||
body_bytes = await response.aread()
|
body_bytes = await response.aread()
|
||||||
except Exception:
|
except Exception:
|
||||||
body_bytes = b""
|
body_bytes = b""
|
||||||
# Redact provider account identifiers before the body text
|
body_preview = body_bytes.decode(
|
||||||
# reaches logs or the raised error.
|
"utf-8", errors="ignore"
|
||||||
body_preview = redact_org_ids(
|
).strip()[:500]
|
||||||
body_bytes.decode("utf-8", errors="ignore").strip()[:500]
|
|
||||||
)
|
|
||||||
rate_limit = classify_rate_limit(
|
|
||||||
response.status_code,
|
|
||||||
body_preview,
|
|
||||||
dict(response.headers),
|
|
||||||
)
|
|
||||||
logger.error(
|
logger.error(
|
||||||
"Upstream %s returned %s for model=%s path=%s: %s",
|
"Upstream %s returned %s for model=%s path=%s: %s",
|
||||||
self.provider_type,
|
self.provider_type,
|
||||||
@@ -2948,7 +2843,6 @@ class BaseUpstreamProvider:
|
|||||||
"provider": self.provider_type,
|
"provider": self.provider_type,
|
||||||
"model": original_model_id or "unknown",
|
"model": original_model_id or "unknown",
|
||||||
"status_code": response.status_code,
|
"status_code": response.status_code,
|
||||||
"error_code": rate_limit.code if rate_limit else None,
|
|
||||||
"path": path,
|
"path": path,
|
||||||
"body_preview": body_preview,
|
"body_preview": body_preview,
|
||||||
},
|
},
|
||||||
@@ -2960,8 +2854,6 @@ class BaseUpstreamProvider:
|
|||||||
f"for model {original_model_id or 'unknown'}: "
|
f"for model {original_model_id or 'unknown'}: "
|
||||||
f"{body_preview[:200] or '<empty>'}",
|
f"{body_preview[:200] or '<empty>'}",
|
||||||
status_code=response.status_code,
|
status_code=response.status_code,
|
||||||
code=rate_limit.code if rate_limit else None,
|
|
||||||
details=rate_limit.as_details() if rate_limit else None,
|
|
||||||
)
|
)
|
||||||
|
|
||||||
try:
|
try:
|
||||||
@@ -3986,19 +3878,9 @@ class BaseUpstreamProvider:
|
|||||||
},
|
},
|
||||||
)
|
)
|
||||||
|
|
||||||
redeemed = False
|
|
||||||
try:
|
try:
|
||||||
headers = dict(request.headers)
|
headers = dict(request.headers)
|
||||||
amount, unit, mint = await recieve_token(x_cashu_token)
|
amount, unit, mint = await recieve_token(x_cashu_token)
|
||||||
# Reject a zero/negative redemption (empty/dust token, or a value
|
|
||||||
# fully consumed by fees) before marking the token redeemed, so it
|
|
||||||
# classifies as cashu_token_zero_value like the bearer/top-up paths
|
|
||||||
# rather than being forwarded as a free request.
|
|
||||||
if amount <= 0:
|
|
||||||
raise ValueError(
|
|
||||||
f"Redeemed token amount must be positive, got {amount} {unit}"
|
|
||||||
)
|
|
||||||
redeemed = True
|
|
||||||
headers = self.prepare_headers(dict(request.headers))
|
headers = self.prepare_headers(dict(request.headers))
|
||||||
|
|
||||||
request_id = getattr(request.state, "request_id", None)
|
request_id = getattr(request.state, "request_id", None)
|
||||||
@@ -4042,37 +3924,40 @@ class BaseUpstreamProvider:
|
|||||||
},
|
},
|
||||||
)
|
)
|
||||||
|
|
||||||
# Post-redemption the token is spent; a forwarding failure must not
|
# Use same error handling as regular X-Cashu
|
||||||
# be reported as a retryable redemption error (see handle_x_cashu).
|
if "already spent" in error_message.lower():
|
||||||
if redeemed:
|
|
||||||
return create_error_response(
|
return create_error_response(
|
||||||
"upstream_error",
|
"token_already_spent",
|
||||||
"Payment succeeded but the upstream request failed",
|
"The provided CASHU token has already been spent",
|
||||||
502,
|
400,
|
||||||
request=request,
|
request=request,
|
||||||
code="upstream_request_failed",
|
token=x_cashu_token,
|
||||||
)
|
)
|
||||||
|
|
||||||
classified = classify_redemption_error(e)
|
if "invalid token" in error_message.lower():
|
||||||
if classified is None:
|
|
||||||
return create_error_response(
|
return create_error_response(
|
||||||
"api_error",
|
"invalid_token",
|
||||||
"Internal error during token redemption",
|
"The provided CASHU token is invalid",
|
||||||
500,
|
400,
|
||||||
request=request,
|
request=request,
|
||||||
code="internal_error",
|
token=x_cashu_token,
|
||||||
)
|
)
|
||||||
error_type, status_code, message, error_code = classified
|
|
||||||
# Echo the token back only when it is still spendable, so clients
|
if "mint error" in error_message.lower():
|
||||||
# can recover it; a spent/consumed token is never re-offered.
|
return create_error_response(
|
||||||
echo_token = None if error_code in SPENT_TOKEN_CODES else x_cashu_token
|
"mint_error",
|
||||||
|
f"CASHU mint error: {error_message}",
|
||||||
|
422,
|
||||||
|
request=request,
|
||||||
|
token=x_cashu_token,
|
||||||
|
)
|
||||||
|
|
||||||
return create_error_response(
|
return create_error_response(
|
||||||
error_type,
|
"cashu_error",
|
||||||
message,
|
f"CASHU token processing failed: {error_message}",
|
||||||
status_code,
|
400,
|
||||||
request=request,
|
request=request,
|
||||||
token=echo_token,
|
token=x_cashu_token,
|
||||||
code=error_code,
|
|
||||||
)
|
)
|
||||||
|
|
||||||
async def forward_x_cashu_responses_request(
|
async def forward_x_cashu_responses_request(
|
||||||
@@ -4662,19 +4547,9 @@ class BaseUpstreamProvider:
|
|||||||
},
|
},
|
||||||
)
|
)
|
||||||
|
|
||||||
redeemed = False
|
|
||||||
try:
|
try:
|
||||||
headers = dict(request.headers)
|
headers = dict(request.headers)
|
||||||
amount, unit, mint = await recieve_token(x_cashu_token)
|
amount, unit, mint = await recieve_token(x_cashu_token)
|
||||||
# Reject a zero/negative redemption (empty/dust token, or a value
|
|
||||||
# fully consumed by fees) before marking the token redeemed, so it
|
|
||||||
# classifies as cashu_token_zero_value like the bearer/top-up paths
|
|
||||||
# rather than being forwarded as a free request.
|
|
||||||
if amount <= 0:
|
|
||||||
raise ValueError(
|
|
||||||
f"Redeemed token amount must be positive, got {amount} {unit}"
|
|
||||||
)
|
|
||||||
redeemed = True
|
|
||||||
headers = self.prepare_headers(dict(request.headers))
|
headers = self.prepare_headers(dict(request.headers))
|
||||||
|
|
||||||
request_id = getattr(request.state, "request_id", None)
|
request_id = getattr(request.state, "request_id", None)
|
||||||
@@ -4718,38 +4593,39 @@ class BaseUpstreamProvider:
|
|||||||
},
|
},
|
||||||
)
|
)
|
||||||
|
|
||||||
# Once redeemed the token is spent, so a later forwarding failure
|
if "already spent" in error_message.lower():
|
||||||
# must not surface as a retryable mint_unreachable (spent-token retry
|
|
||||||
# bait). Redemption classification only applies while not redeemed.
|
|
||||||
if redeemed:
|
|
||||||
return create_error_response(
|
return create_error_response(
|
||||||
"upstream_error",
|
"token_already_spent",
|
||||||
"Payment succeeded but the upstream request failed",
|
"The provided CASHU token has already been spent",
|
||||||
502,
|
400,
|
||||||
request=request,
|
request=request,
|
||||||
code="upstream_request_failed",
|
token=x_cashu_token,
|
||||||
)
|
)
|
||||||
|
|
||||||
classified = classify_redemption_error(e)
|
if "invalid token" in error_message.lower():
|
||||||
if classified is None:
|
|
||||||
return create_error_response(
|
return create_error_response(
|
||||||
"api_error",
|
"invalid_token",
|
||||||
"Internal error during token redemption",
|
"The provided CASHU token is invalid",
|
||||||
500,
|
400,
|
||||||
request=request,
|
request=request,
|
||||||
code="internal_error",
|
token=x_cashu_token,
|
||||||
)
|
)
|
||||||
error_type, status_code, message, error_code = classified
|
|
||||||
# Echo the token back only when it is still spendable, so clients
|
if "mint error" in error_message.lower():
|
||||||
# can recover it; a spent/consumed token is never re-offered.
|
return create_error_response(
|
||||||
echo_token = None if error_code in SPENT_TOKEN_CODES else x_cashu_token
|
"mint_error",
|
||||||
|
f"CASHU mint error: {error_message}",
|
||||||
|
422,
|
||||||
|
request=request,
|
||||||
|
token=x_cashu_token,
|
||||||
|
)
|
||||||
|
|
||||||
return create_error_response(
|
return create_error_response(
|
||||||
error_type,
|
"cashu_error",
|
||||||
message,
|
f"CASHU token processing failed: {error_message}",
|
||||||
status_code,
|
400,
|
||||||
request=request,
|
request=request,
|
||||||
token=echo_token,
|
token=x_cashu_token,
|
||||||
code=error_code,
|
|
||||||
)
|
)
|
||||||
|
|
||||||
def _apply_provider_fee_to_model(self, model: Model) -> Model:
|
def _apply_provider_fee_to_model(self, model: Model) -> Model:
|
||||||
@@ -4920,11 +4796,14 @@ class BaseUpstreamProvider:
|
|||||||
"""Refresh the in-memory models cache from upstream API."""
|
"""Refresh the in-memory models cache from upstream API."""
|
||||||
try:
|
try:
|
||||||
async with create_session() as session:
|
async with create_session() as session:
|
||||||
provider = (
|
stmt = select(UpstreamProviderRow).where(
|
||||||
await session.get(UpstreamProviderRow, self.db_id)
|
UpstreamProviderRow.base_url == self.base_url,
|
||||||
if self.db_id is not None
|
UpstreamProviderRow.api_key == self.api_key,
|
||||||
else None
|
|
||||||
)
|
)
|
||||||
|
result = await session.exec(stmt)
|
||||||
|
|
||||||
|
# .first() returns the object or None if not found
|
||||||
|
provider = result.first()
|
||||||
if not provider or not provider.id:
|
if not provider or not provider.id:
|
||||||
raise HTTPException(status_code=404, detail="Provider not found")
|
raise HTTPException(status_code=404, detail="Provider not found")
|
||||||
|
|
||||||
|
|||||||
@@ -1,73 +0,0 @@
|
|||||||
"""TEMPORARY: local DeepSeek V4 pricing shim.
|
|
||||||
|
|
||||||
litellm's bundled cost map does not yet ship ``deepseek-v4-flash`` /
|
|
||||||
``deepseek-v4-pro``. Without an entry, ``backfill_cache_pricing`` cannot find a
|
|
||||||
``cache_read_input_token_cost`` and cache reads fall back to the full input
|
|
||||||
rate — a large overcharge on cache hits (DeepSeek V4 hits are ~0.008-0.02x
|
|
||||||
input, i.e. cached tokens cost 50-120x less than regular input).
|
|
||||||
|
|
||||||
This module injects the missing entries into ``litellm.model_cost`` at startup
|
|
||||||
so the existing backfill path resolves them. Rates mirror the canonical
|
|
||||||
``deepseek`` provider entries now in litellm's ``model_prices`` map
|
|
||||||
(``input_cost_per_token`` is the cache-*miss* rate;
|
|
||||||
``cache_read_input_token_cost`` is the cache-*hit* rate), sourced from
|
|
||||||
https://api-docs.deepseek.com/quick_start/pricing via
|
|
||||||
https://github.com/BerriAI/litellm/pull/26380 (issue
|
|
||||||
https://github.com/BerriAI/litellm/issues/30430).
|
|
||||||
|
|
||||||
=== REMOVAL (once litellm ships these models) ===
|
|
||||||
Delete this file and the single ``register_deepseek_v4_pricing()`` call in
|
|
||||||
``routstr/core/main.py``. Nothing else depends on it. Entries are only added
|
|
||||||
when absent, so a stale shim is harmless after upstream lands — but remove it.
|
|
||||||
"""
|
|
||||||
|
|
||||||
import litellm
|
|
||||||
|
|
||||||
from ..core import get_logger
|
|
||||||
|
|
||||||
logger = get_logger(__name__)
|
|
||||||
|
|
||||||
# USD per token. Mirrors the canonical ``deepseek`` provider entries in
|
|
||||||
# litellm's model_prices map (source: DeepSeek API pricing docs). Keep these in
|
|
||||||
# sync with ``litellm.model_cost["deepseek/deepseek-v4-*"]``.
|
|
||||||
_DEEPSEEK_V4_RATES: dict[str, dict[str, float]] = {
|
|
||||||
"deepseek-v4-flash": {
|
|
||||||
"input_cost_per_token": 1.4e-07,
|
|
||||||
"output_cost_per_token": 2.8e-07,
|
|
||||||
"cache_read_input_token_cost": 2.8e-09,
|
|
||||||
"cache_creation_input_token_cost": 0.0,
|
|
||||||
"input_cost_per_token_cache_hit": 2.8e-09,
|
|
||||||
},
|
|
||||||
"deepseek-v4-pro": {
|
|
||||||
"input_cost_per_token": 4.35e-07,
|
|
||||||
"output_cost_per_token": 8.7e-07,
|
|
||||||
"cache_read_input_token_cost": 3.625e-09,
|
|
||||||
"cache_creation_input_token_cost": 0.0,
|
|
||||||
"input_cost_per_token_cache_hit": 3.625e-09,
|
|
||||||
},
|
|
||||||
}
|
|
||||||
|
|
||||||
|
|
||||||
def register_deepseek_v4_pricing() -> None:
|
|
||||||
"""Inject DeepSeek V4 pricing into ``litellm.model_cost`` if absent.
|
|
||||||
|
|
||||||
Idempotent and non-destructive: a key already present in the cost map
|
|
||||||
(e.g. once litellm ships it) is left untouched. Registers both the bare
|
|
||||||
(``deepseek-v4-flash``) and prefixed (``deepseek/deepseek-v4-flash``)
|
|
||||||
spellings since ``backfill_cache_pricing`` tries both.
|
|
||||||
"""
|
|
||||||
added = []
|
|
||||||
for bare, rates in _DEEPSEEK_V4_RATES.items():
|
|
||||||
for key in (bare, f"deepseek/{bare}"):
|
|
||||||
if key in litellm.model_cost:
|
|
||||||
continue
|
|
||||||
entry: dict[str, object] = dict(rates)
|
|
||||||
entry["litellm_provider"] = "deepseek"
|
|
||||||
entry["mode"] = "chat"
|
|
||||||
litellm.model_cost[key] = entry
|
|
||||||
added.append(key)
|
|
||||||
if added:
|
|
||||||
logger.info(
|
|
||||||
"Registered temporary DeepSeek V4 pricing shim",
|
|
||||||
extra={"models": added},
|
|
||||||
)
|
|
||||||
@@ -20,7 +20,7 @@ class FireworksUpstreamProvider(BaseUpstreamProvider):
|
|||||||
)
|
)
|
||||||
|
|
||||||
@classmethod
|
@classmethod
|
||||||
def _build_from_row(
|
def from_db_row(
|
||||||
cls, provider_row: "UpstreamProviderRow"
|
cls, provider_row: "UpstreamProviderRow"
|
||||||
) -> "FireworksUpstreamProvider":
|
) -> "FireworksUpstreamProvider":
|
||||||
return cls(
|
return cls(
|
||||||
|
|||||||
@@ -51,7 +51,7 @@ class GeminiUpstreamProvider(BaseUpstreamProvider):
|
|||||||
return self._client
|
return self._client
|
||||||
|
|
||||||
@classmethod
|
@classmethod
|
||||||
def _build_from_row(
|
def from_db_row(
|
||||||
cls, provider_row: "UpstreamProviderRow"
|
cls, provider_row: "UpstreamProviderRow"
|
||||||
) -> "GeminiUpstreamProvider":
|
) -> "GeminiUpstreamProvider":
|
||||||
return cls(
|
return cls(
|
||||||
|
|||||||
@@ -45,7 +45,7 @@ class GenericUpstreamProvider(BaseUpstreamProvider):
|
|||||||
)
|
)
|
||||||
|
|
||||||
@classmethod
|
@classmethod
|
||||||
def _build_from_row(
|
def from_db_row(
|
||||||
cls, provider_row: "UpstreamProviderRow"
|
cls, provider_row: "UpstreamProviderRow"
|
||||||
) -> "GenericUpstreamProvider":
|
) -> "GenericUpstreamProvider":
|
||||||
return cls(
|
return cls(
|
||||||
|
|||||||
@@ -20,7 +20,7 @@ class GroqUpstreamProvider(BaseUpstreamProvider):
|
|||||||
)
|
)
|
||||||
|
|
||||||
@classmethod
|
@classmethod
|
||||||
def _build_from_row(cls, provider_row: "UpstreamProviderRow") -> "GroqUpstreamProvider":
|
def from_db_row(cls, provider_row: "UpstreamProviderRow") -> "GroqUpstreamProvider":
|
||||||
return cls(
|
return cls(
|
||||||
api_key=provider_row.api_key,
|
api_key=provider_row.api_key,
|
||||||
provider_fee=provider_row.provider_fee,
|
provider_fee=provider_row.provider_fee,
|
||||||
|
|||||||
@@ -12,7 +12,6 @@ from sqlmodel import select
|
|||||||
|
|
||||||
from ..core import get_logger
|
from ..core import get_logger
|
||||||
from ..core.db import AsyncSession, ModelRow, UpstreamProviderRow, create_session
|
from ..core.db import AsyncSession, ModelRow, UpstreamProviderRow, create_session
|
||||||
from ..core.provider_slugs import allocate_unique_provider_slug
|
|
||||||
from ..payment.models import Model
|
from ..payment.models import Model
|
||||||
from .base import BaseUpstreamProvider
|
from .base import BaseUpstreamProvider
|
||||||
|
|
||||||
@@ -216,6 +215,9 @@ async def init_upstreams() -> list[BaseUpstreamProvider]:
|
|||||||
|
|
||||||
provider = _instantiate_provider(provider_row)
|
provider = _instantiate_provider(provider_row)
|
||||||
if provider:
|
if provider:
|
||||||
|
# Keep provider DB id on runtime instance so model mapping can
|
||||||
|
# bind DB overrides to the correct upstream.
|
||||||
|
setattr(provider, "db_id", provider_row.id)
|
||||||
await provider.refresh_models_cache()
|
await provider.refresh_models_cache()
|
||||||
logger.debug(
|
logger.debug(
|
||||||
f"Initialized {provider_row.provider_type} provider",
|
f"Initialized {provider_row.provider_type} provider",
|
||||||
@@ -248,7 +250,6 @@ async def _seed_providers_from_settings(
|
|||||||
|
|
||||||
providers_to_add: list[UpstreamProviderRow] = []
|
providers_to_add: list[UpstreamProviderRow] = []
|
||||||
seeded_provider_keys: set[tuple[str, str]] = set()
|
seeded_provider_keys: set[tuple[str, str]] = set()
|
||||||
reserved_slugs: set[str] = set()
|
|
||||||
|
|
||||||
provider_classes_by_type = {
|
provider_classes_by_type = {
|
||||||
cls.provider_type: cls
|
cls.provider_type: cls
|
||||||
@@ -278,13 +279,8 @@ async def _seed_providers_from_settings(
|
|||||||
)
|
)
|
||||||
)
|
)
|
||||||
if not result.first():
|
if not result.first():
|
||||||
slug = await allocate_unique_provider_slug(
|
|
||||||
session, provider_type, reserved_slugs
|
|
||||||
)
|
|
||||||
reserved_slugs.add(slug)
|
|
||||||
providers_to_add.append(
|
providers_to_add.append(
|
||||||
UpstreamProviderRow(
|
UpstreamProviderRow(
|
||||||
slug=slug,
|
|
||||||
provider_type=provider_type,
|
provider_type=provider_type,
|
||||||
base_url=base_url,
|
base_url=base_url,
|
||||||
api_key=api_key,
|
api_key=api_key,
|
||||||
@@ -303,13 +299,8 @@ async def _seed_providers_from_settings(
|
|||||||
)
|
)
|
||||||
)
|
)
|
||||||
if not result.first():
|
if not result.first():
|
||||||
slug = await allocate_unique_provider_slug(
|
|
||||||
session, "ollama", reserved_slugs
|
|
||||||
)
|
|
||||||
reserved_slugs.add(slug)
|
|
||||||
providers_to_add.append(
|
providers_to_add.append(
|
||||||
UpstreamProviderRow(
|
UpstreamProviderRow(
|
||||||
slug=slug,
|
|
||||||
provider_type="ollama",
|
provider_type="ollama",
|
||||||
base_url=ollama_base_url,
|
base_url=ollama_base_url,
|
||||||
api_key=ollama_api_key,
|
api_key=ollama_api_key,
|
||||||
@@ -329,13 +320,8 @@ async def _seed_providers_from_settings(
|
|||||||
)
|
)
|
||||||
)
|
)
|
||||||
if not result.first():
|
if not result.first():
|
||||||
slug = await allocate_unique_provider_slug(
|
|
||||||
session, "azure", reserved_slugs
|
|
||||||
)
|
|
||||||
reserved_slugs.add(slug)
|
|
||||||
providers_to_add.append(
|
providers_to_add.append(
|
||||||
UpstreamProviderRow(
|
UpstreamProviderRow(
|
||||||
slug=slug,
|
|
||||||
provider_type="azure",
|
provider_type="azure",
|
||||||
base_url=base_url,
|
base_url=base_url,
|
||||||
api_key=api_key,
|
api_key=api_key,
|
||||||
@@ -356,13 +342,8 @@ async def _seed_providers_from_settings(
|
|||||||
)
|
)
|
||||||
)
|
)
|
||||||
if not result.first():
|
if not result.first():
|
||||||
slug = await allocate_unique_provider_slug(
|
|
||||||
session, "custom", reserved_slugs
|
|
||||||
)
|
|
||||||
reserved_slugs.add(slug)
|
|
||||||
providers_to_add.append(
|
providers_to_add.append(
|
||||||
UpstreamProviderRow(
|
UpstreamProviderRow(
|
||||||
slug=slug,
|
|
||||||
provider_type="custom",
|
provider_type="custom",
|
||||||
base_url=base_url,
|
base_url=base_url,
|
||||||
api_key=api_key,
|
api_key=api_key,
|
||||||
@@ -375,7 +356,7 @@ async def _seed_providers_from_settings(
|
|||||||
session.add(provider)
|
session.add(provider)
|
||||||
logger.info(
|
logger.info(
|
||||||
f"Seeding {provider.provider_type} provider", # type: ignore[str-format]
|
f"Seeding {provider.provider_type} provider", # type: ignore[str-format]
|
||||||
extra={"base_url": provider.base_url, "slug": provider.slug},
|
extra={"base_url": provider.base_url},
|
||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
@@ -410,7 +391,9 @@ def _instantiate_provider(
|
|||||||
return provider
|
return provider
|
||||||
|
|
||||||
if provider_row.provider_type == "custom":
|
if provider_row.provider_type == "custom":
|
||||||
return BaseUpstreamProvider.from_db_row(provider_row)
|
return BaseUpstreamProvider(
|
||||||
|
provider_row.base_url, provider_row.api_key, provider_row.provider_fee
|
||||||
|
)
|
||||||
|
|
||||||
logger.error(
|
logger.error(
|
||||||
f"Unknown provider type: {provider_row.provider_type}",
|
f"Unknown provider type: {provider_row.provider_type}",
|
||||||
|
|||||||
@@ -29,9 +29,7 @@ import litellm
|
|||||||
|
|
||||||
from ..core import get_logger
|
from ..core import get_logger
|
||||||
from ..core.exceptions import UpstreamError
|
from ..core.exceptions import UpstreamError
|
||||||
from ..core.redaction import redact_org_ids
|
|
||||||
from ..payment.models import Model
|
from ..payment.models import Model
|
||||||
from .rate_limit import classify_rate_limit
|
|
||||||
|
|
||||||
logger = get_logger(__name__)
|
logger = get_logger(__name__)
|
||||||
|
|
||||||
@@ -507,33 +505,23 @@ async def dispatch_anthropic_messages(
|
|||||||
try:
|
try:
|
||||||
result = await litellm.anthropic.messages.acreate(**kwargs)
|
result = await litellm.anthropic.messages.acreate(**kwargs)
|
||||||
except Exception as exc:
|
except Exception as exc:
|
||||||
raw_message = getattr(exc, "message", None) or str(exc) or repr(exc)
|
exc_message = getattr(exc, "message", None) or str(exc) or repr(exc)
|
||||||
# Redact provider account identifiers before the message reaches logs
|
|
||||||
# or the surfaced error.
|
|
||||||
exc_message = redact_org_ids(raw_message)
|
|
||||||
exc_status = getattr(exc, "status_code", None)
|
exc_status = getattr(exc, "status_code", None)
|
||||||
exc_response = getattr(exc, "response", None)
|
exc_response = getattr(exc, "response", None)
|
||||||
response_text = None
|
response_text = None
|
||||||
if exc_response is not None:
|
if exc_response is not None:
|
||||||
try:
|
try:
|
||||||
response_text = redact_org_ids(
|
response_text = getattr(exc_response, "text", str(exc_response))
|
||||||
getattr(exc_response, "text", str(exc_response))
|
|
||||||
)
|
|
||||||
except Exception:
|
except Exception:
|
||||||
response_text = "<unreadable>"
|
response_text = "<unreadable>"
|
||||||
status_for_classify = exc_status if isinstance(exc_status, int) else 502
|
|
||||||
rate_limit = classify_rate_limit(
|
|
||||||
status_for_classify, exc_message, getattr(exc, "headers", None)
|
|
||||||
)
|
|
||||||
logger.error(
|
logger.error(
|
||||||
"litellm dispatch failed",
|
"litellm dispatch failed",
|
||||||
extra={
|
extra={
|
||||||
"error": exc_message,
|
"error": exc_message,
|
||||||
"error_type": type(exc).__name__,
|
"error_type": type(exc).__name__,
|
||||||
"status_code": exc_status,
|
"status_code": exc_status,
|
||||||
"error_code": rate_limit.code if rate_limit else None,
|
|
||||||
"llm_provider": getattr(exc, "llm_provider", None),
|
"llm_provider": getattr(exc, "llm_provider", None),
|
||||||
"body": redact_org_ids(str(getattr(exc, "body", "") or "")) or None,
|
"body": getattr(exc, "body", None),
|
||||||
"response_text": response_text,
|
"response_text": response_text,
|
||||||
"model": litellm_model,
|
"model": litellm_model,
|
||||||
"api_base": base_url,
|
"api_base": base_url,
|
||||||
@@ -541,9 +529,7 @@ async def dispatch_anthropic_messages(
|
|||||||
)
|
)
|
||||||
raise UpstreamError(
|
raise UpstreamError(
|
||||||
f"Upstream error via litellm: {exc_message}",
|
f"Upstream error via litellm: {exc_message}",
|
||||||
status_code=status_for_classify,
|
status_code=exc_status if isinstance(exc_status, int) else 502,
|
||||||
code=rate_limit.code if rate_limit else None,
|
|
||||||
details=rate_limit.as_details() if rate_limit else None,
|
|
||||||
) from exc
|
) from exc
|
||||||
|
|
||||||
if not client_stream and hasattr(result, "__aiter__"):
|
if not client_stream and hasattr(result, "__aiter__"):
|
||||||
|
|||||||
@@ -43,7 +43,7 @@ class OllamaUpstreamProvider(BaseUpstreamProvider):
|
|||||||
)
|
)
|
||||||
|
|
||||||
@classmethod
|
@classmethod
|
||||||
def _build_from_row(
|
def from_db_row(
|
||||||
cls, provider_row: "UpstreamProviderRow"
|
cls, provider_row: "UpstreamProviderRow"
|
||||||
) -> "OllamaUpstreamProvider":
|
) -> "OllamaUpstreamProvider":
|
||||||
return cls(
|
return cls(
|
||||||
|
|||||||
@@ -20,7 +20,7 @@ class OpenAIUpstreamProvider(BaseUpstreamProvider):
|
|||||||
)
|
)
|
||||||
|
|
||||||
@classmethod
|
@classmethod
|
||||||
def _build_from_row(
|
def from_db_row(
|
||||||
cls, provider_row: "UpstreamProviderRow"
|
cls, provider_row: "UpstreamProviderRow"
|
||||||
) -> "OpenAIUpstreamProvider":
|
) -> "OpenAIUpstreamProvider":
|
||||||
return cls(
|
return cls(
|
||||||
|
|||||||
@@ -1,4 +1,3 @@
|
|||||||
import json
|
|
||||||
from typing import TYPE_CHECKING
|
from typing import TYPE_CHECKING
|
||||||
|
|
||||||
import httpx
|
import httpx
|
||||||
@@ -19,38 +18,6 @@ class OpenRouterUpstreamProvider(BaseUpstreamProvider):
|
|||||||
supports_anthropic_messages = True
|
supports_anthropic_messages = True
|
||||||
litellm_provider_prefix = "openrouter/"
|
litellm_provider_prefix = "openrouter/"
|
||||||
|
|
||||||
def prepare_request_body(
|
|
||||||
self, body: bytes | None, model_obj: Model
|
|
||||||
) -> bytes | None:
|
|
||||||
"""Set provider.require_parameters on tool-use requests.
|
|
||||||
|
|
||||||
Without it OpenRouter can route a tool call to an endpoint that doesn't
|
|
||||||
support function calling and 404 with "No endpoints found that support
|
|
||||||
tool use". We leave a client-supplied value untouched.
|
|
||||||
"""
|
|
||||||
body = super().prepare_request_body(body, model_obj)
|
|
||||||
if not body:
|
|
||||||
return body
|
|
||||||
|
|
||||||
try:
|
|
||||||
data = json.loads(body)
|
|
||||||
except json.JSONDecodeError:
|
|
||||||
return body
|
|
||||||
|
|
||||||
if not isinstance(data, dict) or not data.get("tools"):
|
|
||||||
return body
|
|
||||||
|
|
||||||
provider = data.get("provider")
|
|
||||||
if not isinstance(provider, dict):
|
|
||||||
provider = {}
|
|
||||||
|
|
||||||
if "require_parameters" in provider:
|
|
||||||
return body
|
|
||||||
|
|
||||||
provider["require_parameters"] = True
|
|
||||||
data["provider"] = provider
|
|
||||||
return json.dumps(data).encode()
|
|
||||||
|
|
||||||
def _apply_provider_field(self, response_json: object) -> None:
|
def _apply_provider_field(self, response_json: object) -> None:
|
||||||
"""Stamp the ``provider`` field for OpenRouter responses.
|
"""Stamp the ``provider`` field for OpenRouter responses.
|
||||||
|
|
||||||
@@ -90,7 +57,7 @@ class OpenRouterUpstreamProvider(BaseUpstreamProvider):
|
|||||||
)
|
)
|
||||||
|
|
||||||
@classmethod
|
@classmethod
|
||||||
def _build_from_row(
|
def from_db_row(
|
||||||
cls, provider_row: "UpstreamProviderRow"
|
cls, provider_row: "UpstreamProviderRow"
|
||||||
) -> "OpenRouterUpstreamProvider":
|
) -> "OpenRouterUpstreamProvider":
|
||||||
return cls(
|
return cls(
|
||||||
|
|||||||
@@ -23,7 +23,7 @@ class PerplexityUpstreamProvider(BaseUpstreamProvider):
|
|||||||
)
|
)
|
||||||
|
|
||||||
@classmethod
|
@classmethod
|
||||||
def _build_from_row(
|
def from_db_row(
|
||||||
cls, provider_row: "UpstreamProviderRow"
|
cls, provider_row: "UpstreamProviderRow"
|
||||||
) -> "PerplexityUpstreamProvider":
|
) -> "PerplexityUpstreamProvider":
|
||||||
return cls(
|
return cls(
|
||||||
|
|||||||
@@ -46,7 +46,7 @@ class PPQAIUpstreamProvider(BaseUpstreamProvider):
|
|||||||
)
|
)
|
||||||
|
|
||||||
@classmethod
|
@classmethod
|
||||||
def _build_from_row(
|
def from_db_row(
|
||||||
cls, provider_row: "UpstreamProviderRow"
|
cls, provider_row: "UpstreamProviderRow"
|
||||||
) -> "PPQAIUpstreamProvider":
|
) -> "PPQAIUpstreamProvider":
|
||||||
return cls(
|
return cls(
|
||||||
@@ -229,14 +229,17 @@ class PPQAIUpstreamProvider(BaseUpstreamProvider):
|
|||||||
f"Disabling PPQ.AI provider ({self.base_url}) due to insufficient balance",
|
f"Disabling PPQ.AI provider ({self.base_url}) due to insufficient balance",
|
||||||
extra={"error": error_message},
|
extra={"error": error_message},
|
||||||
)
|
)
|
||||||
|
from sqlmodel import select
|
||||||
|
|
||||||
from ..core.db import UpstreamProviderRow, create_session
|
from ..core.db import UpstreamProviderRow, create_session
|
||||||
|
|
||||||
async with create_session() as session:
|
async with create_session() as session:
|
||||||
provider = (
|
statement = select(UpstreamProviderRow).where(
|
||||||
await session.get(UpstreamProviderRow, self.db_id)
|
UpstreamProviderRow.base_url == self.base_url,
|
||||||
if self.db_id is not None
|
UpstreamProviderRow.api_key == self.api_key,
|
||||||
else None
|
|
||||||
)
|
)
|
||||||
|
result = await session.exec(statement)
|
||||||
|
provider = result.first()
|
||||||
|
|
||||||
if provider:
|
if provider:
|
||||||
provider.enabled = False
|
provider.enabled = False
|
||||||
|
|||||||
@@ -1,136 +0,0 @@
|
|||||||
"""Detection and parsing of upstream provider rate-limit errors.
|
|
||||||
|
|
||||||
Upstream OpenAI-compatible providers signal rate limits via HTTP 429 and/or a
|
|
||||||
human-readable message such as::
|
|
||||||
|
|
||||||
Rate limit reached for gpt-5.5-2026-04-23 (for limit gpt-5.5) in organization
|
|
||||||
org-XXXX on tokens per min (TPM): Limit 180000000, Used 180000000,
|
|
||||||
Requested 8929. Please try again in 2ms.
|
|
||||||
|
|
||||||
This module classifies those failures into a stable :data:`UPSTREAM_RATE_LIMIT`
|
|
||||||
code and extracts useful debugging fields. All retained text is redacted of
|
|
||||||
organization IDs first.
|
|
||||||
"""
|
|
||||||
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
import re
|
|
||||||
from dataclasses import asdict, dataclass
|
|
||||||
|
|
||||||
from ..core.redaction import redact_org_ids
|
|
||||||
|
|
||||||
# Stable error code callers can switch on to distinguish upstream rate limits
|
|
||||||
# from generic request failures. The literal value matches the identifier named
|
|
||||||
# in issue #555 ("UPSTREAM_RATE_LIMIT") so the public API contract is exact.
|
|
||||||
UPSTREAM_RATE_LIMIT = "UPSTREAM_RATE_LIMIT"
|
|
||||||
|
|
||||||
# Message fragments that indicate a rate-limit even when the status code is not
|
|
||||||
# 429 (some providers wrap it in a 400/500 envelope).
|
|
||||||
_RATE_LIMIT_MARKERS = (
|
|
||||||
"rate limit reached",
|
|
||||||
"rate_limit_exceeded",
|
|
||||||
"rate limit exceeded",
|
|
||||||
"too many requests",
|
|
||||||
)
|
|
||||||
|
|
||||||
_MODEL_RE = re.compile(r"Rate limit reached for ([^\s(]+)", re.IGNORECASE)
|
|
||||||
_LIMIT_NAME_RE = re.compile(r"\(for limit ([^)]+)\)", re.IGNORECASE)
|
|
||||||
_METRIC_RE = re.compile(r"on ([a-z ]+\((?:TPM|RPM|TPD|RPD|IPM)\))", re.IGNORECASE)
|
|
||||||
_LIMIT_RE = re.compile(r"Limit (\d+)", re.IGNORECASE)
|
|
||||||
_USED_RE = re.compile(r"Used (\d+)", re.IGNORECASE)
|
|
||||||
_REQUESTED_RE = re.compile(r"Requested (\d+)", re.IGNORECASE)
|
|
||||||
_RETRY_RE = re.compile(r"try again in ([\d.]+)\s*(ms|s)", re.IGNORECASE)
|
|
||||||
|
|
||||||
|
|
||||||
@dataclass
|
|
||||||
class RateLimitInfo:
|
|
||||||
"""Structured, redaction-safe view of an upstream rate-limit error."""
|
|
||||||
|
|
||||||
code: str
|
|
||||||
message: str
|
|
||||||
model: str | None = None
|
|
||||||
limit_name: str | None = None
|
|
||||||
metric: str | None = None
|
|
||||||
limit: int | None = None
|
|
||||||
used: int | None = None
|
|
||||||
requested: int | None = None
|
|
||||||
retry_after_seconds: float | None = None
|
|
||||||
|
|
||||||
def as_details(self) -> dict[str, object]:
|
|
||||||
"""Return a JSON-serialisable dict for embedding in an error envelope."""
|
|
||||||
return {k: v for k, v in asdict(self).items() if v is not None}
|
|
||||||
|
|
||||||
|
|
||||||
def _looks_like_rate_limit(status_code: int, message: str) -> bool:
|
|
||||||
if status_code == 429:
|
|
||||||
return True
|
|
||||||
lowered = message.lower()
|
|
||||||
return any(marker in lowered for marker in _RATE_LIMIT_MARKERS)
|
|
||||||
|
|
||||||
|
|
||||||
def _parse_retry_after_header(headers: dict[str, str] | None) -> float | None:
|
|
||||||
"""Parse a ``Retry-After`` header (delta-seconds form) into seconds."""
|
|
||||||
if not headers:
|
|
||||||
return None
|
|
||||||
raw = headers.get("retry-after") or headers.get("Retry-After")
|
|
||||||
if raw is None:
|
|
||||||
return None
|
|
||||||
try:
|
|
||||||
return float(str(raw).strip())
|
|
||||||
except (TypeError, ValueError):
|
|
||||||
return None
|
|
||||||
|
|
||||||
|
|
||||||
def _int_or_none(match: re.Match[str] | None) -> int | None:
|
|
||||||
if match is None:
|
|
||||||
return None
|
|
||||||
try:
|
|
||||||
return int(match.group(1))
|
|
||||||
except (TypeError, ValueError):
|
|
||||||
return None
|
|
||||||
|
|
||||||
|
|
||||||
def classify_rate_limit(
|
|
||||||
status_code: int,
|
|
||||||
message: str,
|
|
||||||
headers: dict[str, str] | None = None,
|
|
||||||
) -> RateLimitInfo | None:
|
|
||||||
"""Classify an upstream error as a rate-limit and extract its fields.
|
|
||||||
|
|
||||||
Args:
|
|
||||||
status_code: HTTP status code from the upstream response.
|
|
||||||
message: Upstream error message (may contain sensitive identifiers).
|
|
||||||
headers: Optional upstream response headers, used for ``Retry-After``.
|
|
||||||
|
|
||||||
Returns:
|
|
||||||
A :class:`RateLimitInfo` when the error is a rate-limit, else ``None``.
|
|
||||||
"""
|
|
||||||
message = message or ""
|
|
||||||
if not _looks_like_rate_limit(status_code, message):
|
|
||||||
return None
|
|
||||||
|
|
||||||
redacted = redact_org_ids(message)
|
|
||||||
|
|
||||||
model_match = _MODEL_RE.search(redacted)
|
|
||||||
metric_match = _METRIC_RE.search(redacted)
|
|
||||||
|
|
||||||
retry_after = _parse_retry_after_header(headers)
|
|
||||||
if retry_after is None:
|
|
||||||
retry_match = _RETRY_RE.search(redacted)
|
|
||||||
if retry_match is not None:
|
|
||||||
value = float(retry_match.group(1))
|
|
||||||
retry_after = value / 1000.0 if retry_match.group(2).lower() == "ms" else value
|
|
||||||
|
|
||||||
limit_name_match = _LIMIT_NAME_RE.search(redacted)
|
|
||||||
|
|
||||||
return RateLimitInfo(
|
|
||||||
code=UPSTREAM_RATE_LIMIT,
|
|
||||||
message=redacted,
|
|
||||||
model=model_match.group(1) if model_match else None,
|
|
||||||
limit_name=limit_name_match.group(1).strip() if limit_name_match else None,
|
|
||||||
metric=metric_match.group(1).strip() if metric_match else None,
|
|
||||||
limit=_int_or_none(_LIMIT_RE.search(redacted)),
|
|
||||||
used=_int_or_none(_USED_RE.search(redacted)),
|
|
||||||
requested=_int_or_none(_REQUESTED_RE.search(redacted)),
|
|
||||||
retry_after_seconds=retry_after,
|
|
||||||
)
|
|
||||||
@@ -55,7 +55,7 @@ class RoutstrUpstreamProvider(BaseUpstreamProvider):
|
|||||||
return path.lstrip("/")
|
return path.lstrip("/")
|
||||||
|
|
||||||
@classmethod
|
@classmethod
|
||||||
def _build_from_row(
|
def from_db_row(
|
||||||
cls, provider_row: "UpstreamProviderRow"
|
cls, provider_row: "UpstreamProviderRow"
|
||||||
) -> "RoutstrUpstreamProvider":
|
) -> "RoutstrUpstreamProvider":
|
||||||
import json
|
import json
|
||||||
|
|||||||
@@ -21,7 +21,7 @@ class XAIUpstreamProvider(BaseUpstreamProvider):
|
|||||||
)
|
)
|
||||||
|
|
||||||
@classmethod
|
@classmethod
|
||||||
def _build_from_row(cls, provider_row: "UpstreamProviderRow") -> "XAIUpstreamProvider":
|
def from_db_row(cls, provider_row: "UpstreamProviderRow") -> "XAIUpstreamProvider":
|
||||||
return cls(
|
return cls(
|
||||||
api_key=provider_row.api_key,
|
api_key=provider_row.api_key,
|
||||||
provider_fee=provider_row.provider_fee,
|
provider_fee=provider_row.provider_fee,
|
||||||
|
|||||||
+91
-395
@@ -1,11 +1,8 @@
|
|||||||
import asyncio
|
import asyncio
|
||||||
import re
|
|
||||||
import socket
|
|
||||||
import time
|
import time
|
||||||
import typing
|
import typing
|
||||||
from typing import TypedDict
|
from typing import TypedDict
|
||||||
|
|
||||||
import httpx
|
|
||||||
from cashu.core.base import Proof, Token
|
from cashu.core.base import Proof, Token
|
||||||
from cashu.core.mint_info import MintInfo as _CashuMintInfo
|
from cashu.core.mint_info import MintInfo as _CashuMintInfo
|
||||||
from cashu.wallet.helpers import deserialize_token_from_string
|
from cashu.wallet.helpers import deserialize_token_from_string
|
||||||
@@ -33,172 +30,11 @@ _CashuMintInfo.model_rebuild(force=True)
|
|||||||
logger = get_logger(__name__)
|
logger = get_logger(__name__)
|
||||||
|
|
||||||
|
|
||||||
class MintConnectionError(Exception):
|
|
||||||
"""The mint could not be reached (network transport failure).
|
|
||||||
|
|
||||||
Maps to a 503, not a 4xx: the token is fine, the mint is just unavailable.
|
|
||||||
"""
|
|
||||||
|
|
||||||
|
|
||||||
class TokenConsumedError(Exception):
|
|
||||||
"""A failure that happened AFTER the token's proofs were spent (melt
|
|
||||||
succeeded, or redemption already returned) — e.g. minting on the primary
|
|
||||||
mint or the DB credit then failed.
|
|
||||||
|
|
||||||
Non-retryable: the same token will not work again. Seals the cause chain so
|
|
||||||
a transport error underneath is never re-surfaced as a retryable
|
|
||||||
mint_unreachable.
|
|
||||||
"""
|
|
||||||
|
|
||||||
|
|
||||||
# httpx base classes cover their subclasses. HTTPStatusError is excluded on
|
|
||||||
# purpose — that means the mint answered, just with an error status.
|
|
||||||
_TRANSPORT_EXC_TYPES: tuple[type[BaseException], ...] = (
|
|
||||||
httpx.NetworkError,
|
|
||||||
httpx.TimeoutException,
|
|
||||||
ConnectionError, # refused/reset/aborted
|
|
||||||
socket.gaierror, # DNS failure
|
|
||||||
asyncio.TimeoutError,
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
def is_mint_connection_error(error: BaseException) -> bool:
|
|
||||||
"""True if ``error`` (or anything in its cause/context chain) is a mint
|
|
||||||
transport failure. Walks the chain because some sites re-raise transport
|
|
||||||
errors wrapped in ValueError/MintConnectionError; matches on TYPE, not text.
|
|
||||||
"""
|
|
||||||
seen: set[int] = set()
|
|
||||||
current: BaseException | None = error
|
|
||||||
while current is not None and id(current) not in seen:
|
|
||||||
seen.add(id(current))
|
|
||||||
if isinstance(current, TokenConsumedError):
|
|
||||||
# Sealed: the token was already spent, so whatever transport error
|
|
||||||
# sits underneath must not make this look retryable.
|
|
||||||
return False
|
|
||||||
if isinstance(current, MintConnectionError):
|
|
||||||
return True
|
|
||||||
if isinstance(current, _TRANSPORT_EXC_TYPES):
|
|
||||||
return True
|
|
||||||
current = current.__cause__ or current.__context__
|
|
||||||
return False
|
|
||||||
|
|
||||||
|
|
||||||
# Redemption ``code`` values whose token is spent/consumed/unusable — the
|
|
||||||
# X-Cashu path must NOT echo the original token for these (echoing invites a
|
|
||||||
# retry with a token that can never succeed again).
|
|
||||||
SPENT_TOKEN_CODES: frozenset[str] = frozenset(
|
|
||||||
{
|
|
||||||
"cashu_token_already_spent",
|
|
||||||
"cashu_token_consumed",
|
|
||||||
"cashu_token_zero_value",
|
|
||||||
"internal_error",
|
|
||||||
}
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
def classify_redemption_error(
|
|
||||||
error: Exception,
|
|
||||||
) -> tuple[str, int, str, str] | None:
|
|
||||||
"""Map a token-redemption failure to ``(type, status, message, code)``.
|
|
||||||
|
|
||||||
Single source of truth for every endpoint that redeems a token (bearer,
|
|
||||||
X-Cashu, top-up) so the same failure yields the same taxonomy everywhere.
|
|
||||||
``type`` and ``code`` are stable client contract; ``message`` is sanitized
|
|
||||||
(raw error text stays in logs). Returns None for an unclassified internal
|
|
||||||
fault — the caller emits a generic 500.
|
|
||||||
"""
|
|
||||||
if isinstance(error, TokenConsumedError):
|
|
||||||
return (
|
|
||||||
"token_consumed",
|
|
||||||
500,
|
|
||||||
"Token was redeemed but could not be credited; do not retry",
|
|
||||||
"cashu_token_consumed",
|
|
||||||
)
|
|
||||||
if is_mint_connection_error(error):
|
|
||||||
return (
|
|
||||||
"mint_unreachable",
|
|
||||||
503,
|
|
||||||
"Cashu mint is unreachable",
|
|
||||||
"cashu_mint_unreachable",
|
|
||||||
)
|
|
||||||
lowered = str(error).lower()
|
|
||||||
if "already spent" in lowered:
|
|
||||||
return (
|
|
||||||
"token_already_spent",
|
|
||||||
400,
|
|
||||||
"Cashu token already spent",
|
|
||||||
"cashu_token_already_spent",
|
|
||||||
)
|
|
||||||
if (
|
|
||||||
"insufficient" in lowered
|
|
||||||
or "melt fee" in lowered
|
|
||||||
or "exceed token amount" in lowered
|
|
||||||
or "estimate fees" in lowered
|
|
||||||
):
|
|
||||||
return (
|
|
||||||
"mint_error",
|
|
||||||
422,
|
|
||||||
"Token value is too small to cover swap fees",
|
|
||||||
"cashu_token_swap_fees_exceed_amount",
|
|
||||||
)
|
|
||||||
if "failed to melt" in lowered:
|
|
||||||
return (
|
|
||||||
"mint_error",
|
|
||||||
422,
|
|
||||||
"Failed to swap token from foreign mint",
|
|
||||||
"cashu_foreign_mint_swap_failed",
|
|
||||||
)
|
|
||||||
if ("invalid" in lowered or "decode" in lowered) and "token" in lowered:
|
|
||||||
# Anchored to "token" so internal faults whose text merely contains
|
|
||||||
# "invalid"/"decode" fall through to the 500 branch, not a token error.
|
|
||||||
return (
|
|
||||||
"invalid_token",
|
|
||||||
400,
|
|
||||||
"Invalid Cashu token",
|
|
||||||
"invalid_cashu_token",
|
|
||||||
)
|
|
||||||
if "must be positive" in lowered or "yielded no value" in lowered:
|
|
||||||
# Redeemed to <= 0 (empty/dust token, or value fully consumed by fees).
|
|
||||||
# Consumed, so non-retryable, but its own code — not the generic bucket.
|
|
||||||
return (
|
|
||||||
"cashu_error",
|
|
||||||
400,
|
|
||||||
"Failed to redeem Cashu token: token yielded no value",
|
|
||||||
"cashu_token_zero_value",
|
|
||||||
)
|
|
||||||
if isinstance(error, ValueError):
|
|
||||||
return (
|
|
||||||
"cashu_error",
|
|
||||||
400,
|
|
||||||
"Failed to redeem Cashu token",
|
|
||||||
"cashu_token_redemption_failed",
|
|
||||||
)
|
|
||||||
return None
|
|
||||||
|
|
||||||
|
|
||||||
async def get_balance(unit: str) -> int:
|
async def get_balance(unit: str) -> int:
|
||||||
wallet = await get_wallet(settings.primary_mint, unit)
|
wallet = await get_wallet(settings.primary_mint, unit)
|
||||||
return wallet.available_balance.amount
|
return wallet.available_balance.amount
|
||||||
|
|
||||||
|
|
||||||
async def _redeem_same_mint(
|
|
||||||
wallet: Wallet, token_obj: Token
|
|
||||||
) -> tuple[int, str, str]: # amount, unit, mint_url
|
|
||||||
"""Redeem proofs at their own issuing mint (no cross-mint swap).
|
|
||||||
|
|
||||||
split() re-mints the incoming proofs into fresh ones we own so the sender
|
|
||||||
can't double-spend them. With include_fees=True the mint deducts its NUT-02
|
|
||||||
per-proof input fee, so we end up holding only `amount - input_fees`. Credit
|
|
||||||
that, not the face value, or routstr over-credits the user and its wallet
|
|
||||||
drifts insolvent.
|
|
||||||
"""
|
|
||||||
await wallet.load_mint(keyset_id=token_obj.keysets[0])
|
|
||||||
wallet.verify_proofs_dleq(token_obj.proofs)
|
|
||||||
input_fees = wallet.get_fees_for_proofs(token_obj.proofs)
|
|
||||||
await wallet.split(proofs=token_obj.proofs, amount=0, include_fees=True)
|
|
||||||
return int(token_obj.amount) - input_fees, token_obj.unit, token_obj.mint
|
|
||||||
|
|
||||||
|
|
||||||
async def recieve_token(
|
async def recieve_token(
|
||||||
token: str,
|
token: str,
|
||||||
) -> tuple[int, str, str]: # amount, unit, mint_url
|
) -> tuple[int, str, str]: # amount, unit, mint_url
|
||||||
@@ -212,7 +48,12 @@ async def recieve_token(
|
|||||||
if token_obj.mint not in settings.cashu_mints:
|
if token_obj.mint not in settings.cashu_mints:
|
||||||
return await swap_to_primary_mint(token_obj, wallet)
|
return await swap_to_primary_mint(token_obj, wallet)
|
||||||
|
|
||||||
return await _redeem_same_mint(wallet, token_obj)
|
await wallet.load_mint(keyset_id=token_obj.keysets[0])
|
||||||
|
|
||||||
|
wallet.verify_proofs_dleq(token_obj.proofs)
|
||||||
|
await wallet.split(proofs=token_obj.proofs, amount=0, include_fees=True)
|
||||||
|
|
||||||
|
return token_obj.amount, token_obj.unit, token_obj.mint
|
||||||
|
|
||||||
|
|
||||||
async def send(amount: int, unit: str, mint_url: str | None = None) -> tuple[int, str]:
|
async def send(amount: int, unit: str, mint_url: str | None = None) -> tuple[int, str]:
|
||||||
@@ -275,72 +116,6 @@ async def send_token(amount: int, unit: str, mint_url: str | None = None) -> str
|
|||||||
return token
|
return token
|
||||||
|
|
||||||
|
|
||||||
# A foreign mint's fee_reserve is a non-binding estimate (NUT-05): the mint may
|
|
||||||
# demand more when re-quoting or at melt execution. Instead of padding the
|
|
||||||
# estimate with a safety buffer (which strands the margin at the foreign mint
|
|
||||||
# on every swap), the swap retries with the amount recomputed from the fees the
|
|
||||||
# mint actually demands, up to this many attempts.
|
|
||||||
_MAX_SWAP_ATTEMPTS = 3
|
|
||||||
|
|
||||||
_MINT_ERROR_CODE_RE = re.compile(r"\(Code: (\d+)\)")
|
|
||||||
_MELT_SHORTFALL_RE = re.compile(r"Provided: (\d+), needed: (\d+)")
|
|
||||||
|
|
||||||
# Insufficient-melt-inputs failures differ across mint implementations. 11005 is
|
|
||||||
# the registered "Transaction is not balanced" code (cdk), specific enough to
|
|
||||||
# trust on the code alone. 11000 is nutshell's generic, unregistered
|
|
||||||
# TransactionError covering many unrelated failures, so it only counts as a fee
|
|
||||||
# shortfall alongside the "not enough inputs" detail text. With no code suffix at
|
|
||||||
# all, that same text is the only signal.
|
|
||||||
|
|
||||||
|
|
||||||
def _net_minted_amount(amount_msat: int, token_unit: str, fees: int) -> int:
|
|
||||||
"""
|
|
||||||
Convert the token value minus fees (given in the token unit) into an
|
|
||||||
amount in the primary mint's unit.
|
|
||||||
"""
|
|
||||||
fee_msat = fees * 1000 if token_unit == "sat" else fees
|
|
||||||
remaining_msat = amount_msat - fee_msat
|
|
||||||
if settings.primary_mint_unit == "sat":
|
|
||||||
return int(remaining_msat // 1000)
|
|
||||||
return int(remaining_msat)
|
|
||||||
|
|
||||||
|
|
||||||
def _melt_insufficient_shortfall(error: Exception) -> int | None:
|
|
||||||
"""
|
|
||||||
Classify a melt failure: return the observed shortfall (in the token unit)
|
|
||||||
when the mint rejected the inputs as insufficient, or None when the failure
|
|
||||||
is unrelated to fees and must not be retried (e.g. a Lightning payment
|
|
||||||
failure, where a smaller invoice would not help).
|
|
||||||
|
|
||||||
Cashu errors carry no structured amounts (NUT-00 defines only detail/code,
|
|
||||||
flattened to "Mint Error: <detail> (Code: <code>)" by cashu-py), so the
|
|
||||||
classification uses the code and the shortfall must be inferred: the
|
|
||||||
"Provided: X, needed: Y" amounts are nutshell-specific free text and only
|
|
||||||
refine the shortfall when present; otherwise shrink one unit at a time.
|
|
||||||
"""
|
|
||||||
message = str(error)
|
|
||||||
code_match = _MINT_ERROR_CODE_RE.search(message)
|
|
||||||
code = code_match.group(1) if code_match is not None else None
|
|
||||||
has_shortfall_text = "not enough inputs" in message.lower()
|
|
||||||
|
|
||||||
match code:
|
|
||||||
case "11005": # registered TransactionUnbalanced: trust the code
|
|
||||||
pass
|
|
||||||
case "11000" if has_shortfall_text: # generic nutshell error: needs the text
|
|
||||||
pass
|
|
||||||
case None if has_shortfall_text: # no code suffix: text is the only signal
|
|
||||||
pass
|
|
||||||
case _: # other codes, a bare 11000, or no signal: must not retry
|
|
||||||
return None
|
|
||||||
|
|
||||||
amounts = _MELT_SHORTFALL_RE.search(message)
|
|
||||||
if amounts is not None:
|
|
||||||
provided, needed = int(amounts.group(1)), int(amounts.group(2))
|
|
||||||
if needed > provided:
|
|
||||||
return needed - provided
|
|
||||||
return 1
|
|
||||||
|
|
||||||
|
|
||||||
async def _calculate_swap_amount(
|
async def _calculate_swap_amount(
|
||||||
amount_msat: int,
|
amount_msat: int,
|
||||||
token_unit: str,
|
token_unit: str,
|
||||||
@@ -379,18 +154,26 @@ async def _calculate_swap_amount(
|
|||||||
|
|
||||||
fee_reserve = dummy_melt_quote.fee_reserve
|
fee_reserve = dummy_melt_quote.fee_reserve
|
||||||
input_fees = token_wallet.get_fees_for_proofs(proofs)
|
input_fees = token_wallet.get_fees_for_proofs(proofs)
|
||||||
total_fees = fee_reserve + input_fees
|
if token_unit == "sat":
|
||||||
minted_amount = _net_minted_amount(amount_msat, token_unit, total_fees)
|
fee_msat = (fee_reserve + input_fees) * 1000
|
||||||
|
else:
|
||||||
|
fee_msat = fee_reserve + input_fees
|
||||||
|
|
||||||
|
amount_msat_after_fee = amount_msat - fee_msat
|
||||||
|
|
||||||
|
if settings.primary_mint_unit == "sat":
|
||||||
|
minted_amount = int(amount_msat_after_fee // 1000)
|
||||||
|
else:
|
||||||
|
minted_amount = int(amount_msat_after_fee)
|
||||||
|
|
||||||
if minted_amount <= 0:
|
if minted_amount <= 0:
|
||||||
raise ValueError(f"Fees ({total_fees} {token_unit}) exceed token amount")
|
raise ValueError(f"Fees ({fee_reserve + input_fees} {token_unit}) exceed token amount")
|
||||||
|
|
||||||
logger.info(
|
logger.info(
|
||||||
"swap_to_primary_mint: fee estimation result",
|
"swap_to_primary_mint: fee estimation result",
|
||||||
extra={
|
extra={
|
||||||
"token_amount_sat": amount_msat // 1000,
|
"token_amount_sat": amount_msat // 1000,
|
||||||
"estimated_fee": total_fees,
|
"estimated_fee_sat": fee_msat // 1000,
|
||||||
"estimated_fee_unit": token_unit,
|
|
||||||
"input_fees": input_fees,
|
"input_fees": input_fees,
|
||||||
"minted_amount": minted_amount,
|
"minted_amount": minted_amount,
|
||||||
"minted_unit": settings.primary_mint_unit,
|
"minted_unit": settings.primary_mint_unit,
|
||||||
@@ -403,8 +186,6 @@ async def _calculate_swap_amount(
|
|||||||
"swap_to_primary_mint: fee estimation failed",
|
"swap_to_primary_mint: fee estimation failed",
|
||||||
extra={"error": str(e)},
|
extra={"error": str(e)},
|
||||||
)
|
)
|
||||||
if is_mint_connection_error(e):
|
|
||||||
raise MintConnectionError("Cashu mint is unreachable") from e
|
|
||||||
raise ValueError(f"Failed to estimate fees: {e}") from e
|
raise ValueError(f"Failed to estimate fees: {e}") from e
|
||||||
|
|
||||||
|
|
||||||
@@ -432,9 +213,10 @@ async def swap_to_primary_mint(
|
|||||||
amount_msat = token_amount
|
amount_msat = token_amount
|
||||||
else:
|
else:
|
||||||
raise ValueError("Invalid unit")
|
raise ValueError("Invalid unit")
|
||||||
# If the token is already from the primary mint, we don't need a cross-mint
|
primary_wallet = await get_wallet(settings.primary_mint, settings.primary_mint_unit)
|
||||||
# swap — redeem it same-mint. There's no melt/Lightning fee, but the mint's
|
|
||||||
# NUT-02 input fee still applies; _redeem_same_mint accounts for it.
|
# If the token is already from the primary mint, we don't need to swap
|
||||||
|
# and we definitely don't want to calculate or pay fees.
|
||||||
if token_obj.mint == settings.primary_mint:
|
if token_obj.mint == settings.primary_mint:
|
||||||
logger.info(
|
logger.info(
|
||||||
"swap_to_primary_mint: token already on primary mint, skipping swap",
|
"swap_to_primary_mint: token already on primary mint, skipping swap",
|
||||||
@@ -444,9 +226,8 @@ async def swap_to_primary_mint(
|
|||||||
"unit": token_obj.unit,
|
"unit": token_obj.unit,
|
||||||
},
|
},
|
||||||
)
|
)
|
||||||
return await _redeem_same_mint(token_wallet, token_obj)
|
await token_wallet.split(proofs=token_obj.proofs, amount=0, include_fees=True)
|
||||||
|
return token_amount, token_obj.unit, token_obj.mint
|
||||||
primary_wallet = await get_wallet(settings.primary_mint, settings.primary_mint_unit)
|
|
||||||
|
|
||||||
minted_amount = await _calculate_swap_amount(
|
minted_amount = await _calculate_swap_amount(
|
||||||
amount_msat,
|
amount_msat,
|
||||||
@@ -457,123 +238,67 @@ async def swap_to_primary_mint(
|
|||||||
token_obj.proofs,
|
token_obj.proofs,
|
||||||
)
|
)
|
||||||
|
|
||||||
# The estimate above is non-binding: the mint may demand a higher fee on the
|
mint_quote = await primary_wallet.request_mint(minted_amount)
|
||||||
# real quote or reject the melt outright. Retry the quote/melt cycle with the
|
logger.info(
|
||||||
# amount recomputed from the fees the mint actually demands.
|
"swap_to_primary_mint: mint quote received",
|
||||||
observed_extra_fee = 0
|
extra={"mint_quote_id": mint_quote.quote},
|
||||||
attempt = 0
|
)
|
||||||
while True:
|
|
||||||
attempt += 1
|
|
||||||
mint_quote = await primary_wallet.request_mint(minted_amount)
|
|
||||||
logger.info(
|
|
||||||
"swap_to_primary_mint: mint quote received",
|
|
||||||
extra={"mint_quote_id": mint_quote.quote, "attempt": attempt},
|
|
||||||
)
|
|
||||||
|
|
||||||
melt_quote = await token_wallet.melt_quote(mint_quote.request)
|
melt_quote = await token_wallet.melt_quote(mint_quote.request)
|
||||||
input_fees = token_wallet.get_fees_for_proofs(token_obj.proofs)
|
input_fees = token_wallet.get_fees_for_proofs(token_obj.proofs)
|
||||||
total_needed = melt_quote.amount + melt_quote.fee_reserve + input_fees
|
total_needed = melt_quote.amount + melt_quote.fee_reserve + input_fees
|
||||||
logger.info(
|
logger.info(
|
||||||
"swap_to_primary_mint: melt quote received",
|
"swap_to_primary_mint: melt quote received",
|
||||||
|
extra={
|
||||||
|
"melt_quote_id": melt_quote.quote,
|
||||||
|
"melt_amount": melt_quote.amount,
|
||||||
|
"melt_fee_reserve": melt_quote.fee_reserve,
|
||||||
|
"input_fees": input_fees,
|
||||||
|
"total_needed": total_needed,
|
||||||
|
"token_amount": token_amount,
|
||||||
|
},
|
||||||
|
)
|
||||||
|
|
||||||
|
if total_needed > token_amount:
|
||||||
|
logger.warning(
|
||||||
|
"swap_to_primary_mint: insufficient token amount for melt fees",
|
||||||
extra={
|
extra={
|
||||||
"melt_quote_id": melt_quote.quote,
|
"token_amount": token_amount,
|
||||||
"melt_amount": melt_quote.amount,
|
"melt_amount": melt_quote.amount,
|
||||||
"melt_fee_reserve": melt_quote.fee_reserve,
|
"melt_fee_reserve": melt_quote.fee_reserve,
|
||||||
"input_fees": input_fees,
|
"input_fees": input_fees,
|
||||||
"total_needed": total_needed,
|
"total_needed": total_needed,
|
||||||
"token_amount": token_amount,
|
"shortfall": total_needed - token_amount,
|
||||||
"attempt": attempt,
|
|
||||||
},
|
},
|
||||||
)
|
)
|
||||||
|
raise ValueError(
|
||||||
|
f"Token amount ({token_amount} {token_obj.unit}) is insufficient to cover "
|
||||||
|
f"melt fees. Needed: {total_needed} {token_obj.unit} "
|
||||||
|
f"(amount: {melt_quote.amount} + fee: {melt_quote.fee_reserve} + input_fees: {input_fees})"
|
||||||
|
)
|
||||||
|
|
||||||
if total_needed > token_amount:
|
try:
|
||||||
recomputed = _net_minted_amount(
|
_ = await token_wallet.melt(
|
||||||
amount_msat,
|
proofs=token_obj.proofs,
|
||||||
token_obj.unit,
|
invoice=mint_quote.request,
|
||||||
melt_quote.fee_reserve + input_fees + observed_extra_fee,
|
fee_reserve_sat=melt_quote.fee_reserve,
|
||||||
)
|
quote_id=melt_quote.quote,
|
||||||
if attempt >= _MAX_SWAP_ATTEMPTS or recomputed <= 0:
|
)
|
||||||
logger.warning(
|
except Exception as e:
|
||||||
"swap_to_primary_mint: insufficient token amount for melt fees",
|
logger.error(
|
||||||
extra={
|
"swap_to_primary_mint: melt failed",
|
||||||
"token_amount": token_amount,
|
extra={
|
||||||
"melt_amount": melt_quote.amount,
|
"error": str(e),
|
||||||
"melt_fee_reserve": melt_quote.fee_reserve,
|
"error_type": type(e).__name__,
|
||||||
"input_fees": input_fees,
|
"foreign_mint": token_obj.mint,
|
||||||
"total_needed": total_needed,
|
"token_amount": token_amount,
|
||||||
"shortfall": total_needed - token_amount,
|
"melt_quote_id": melt_quote.quote,
|
||||||
"attempts": attempt,
|
"total_needed": total_needed,
|
||||||
},
|
},
|
||||||
)
|
)
|
||||||
raise ValueError(
|
raise ValueError(
|
||||||
f"Token amount ({token_amount} {token_obj.unit}) is insufficient to cover "
|
f"Failed to melt token from foreign mint {token_obj.mint}: {e}"
|
||||||
f"melt fees. Needed: {total_needed} {token_obj.unit} "
|
) from e
|
||||||
f"(amount: {melt_quote.amount} + fee: {melt_quote.fee_reserve} + input_fees: {input_fees})"
|
|
||||||
)
|
|
||||||
logger.warning(
|
|
||||||
"swap_to_primary_mint: melt quote exceeds token amount, retrying",
|
|
||||||
extra={
|
|
||||||
"total_needed": total_needed,
|
|
||||||
"token_amount": token_amount,
|
|
||||||
"retry_minted_amount": recomputed,
|
|
||||||
"attempt": attempt,
|
|
||||||
},
|
|
||||||
)
|
|
||||||
minted_amount = recomputed
|
|
||||||
continue
|
|
||||||
|
|
||||||
try:
|
|
||||||
_ = await token_wallet.melt(
|
|
||||||
proofs=token_obj.proofs,
|
|
||||||
invoice=mint_quote.request,
|
|
||||||
fee_reserve_sat=melt_quote.fee_reserve,
|
|
||||||
quote_id=melt_quote.quote,
|
|
||||||
)
|
|
||||||
except Exception as e:
|
|
||||||
# A down mint won't fix itself by retrying with a smaller amount.
|
|
||||||
if is_mint_connection_error(e):
|
|
||||||
logger.error(
|
|
||||||
"swap_to_primary_mint: melt failed — mint unreachable",
|
|
||||||
extra={"error": str(e), "foreign_mint": token_obj.mint},
|
|
||||||
)
|
|
||||||
raise MintConnectionError("Cashu mint is unreachable") from e
|
|
||||||
shortfall = _melt_insufficient_shortfall(e)
|
|
||||||
recomputed = 0
|
|
||||||
if shortfall is not None:
|
|
||||||
observed_extra_fee += shortfall
|
|
||||||
recomputed = _net_minted_amount(
|
|
||||||
amount_msat,
|
|
||||||
token_obj.unit,
|
|
||||||
melt_quote.fee_reserve + input_fees + observed_extra_fee,
|
|
||||||
)
|
|
||||||
if shortfall is None or attempt >= _MAX_SWAP_ATTEMPTS or recomputed <= 0:
|
|
||||||
logger.error(
|
|
||||||
"swap_to_primary_mint: melt failed",
|
|
||||||
extra={
|
|
||||||
"error": str(e),
|
|
||||||
"error_type": type(e).__name__,
|
|
||||||
"foreign_mint": token_obj.mint,
|
|
||||||
"token_amount": token_amount,
|
|
||||||
"melt_quote_id": melt_quote.quote,
|
|
||||||
"total_needed": total_needed,
|
|
||||||
"attempts": attempt,
|
|
||||||
},
|
|
||||||
)
|
|
||||||
raise ValueError(
|
|
||||||
f"Failed to melt token from foreign mint {token_obj.mint}: {e}"
|
|
||||||
) from e
|
|
||||||
logger.warning(
|
|
||||||
"swap_to_primary_mint: mint demanded more than quoted at melt, retrying",
|
|
||||||
extra={
|
|
||||||
"shortfall": shortfall,
|
|
||||||
"retry_minted_amount": recomputed,
|
|
||||||
"attempt": attempt,
|
|
||||||
},
|
|
||||||
)
|
|
||||||
minted_amount = recomputed
|
|
||||||
continue
|
|
||||||
|
|
||||||
break
|
|
||||||
|
|
||||||
logger.info(
|
logger.info(
|
||||||
"swap_to_primary_mint: melt succeeded, minting on primary",
|
"swap_to_primary_mint: melt succeeded, minting on primary",
|
||||||
@@ -612,21 +337,21 @@ async def swap_to_primary_mint(
|
|||||||
# Recovery scan ran but did NOT restore the orphaned proofs
|
# Recovery scan ran but did NOT restore the orphaned proofs
|
||||||
# (mint reports them as spent — they're stuck). Refuse to
|
# (mint reports them as spent — they're stuck). Refuse to
|
||||||
# credit the API key balance for proofs we don't actually hold.
|
# credit the API key balance for proofs we don't actually hold.
|
||||||
raise TokenConsumedError(
|
raise ValueError(
|
||||||
f"Swap recovery failed: mint signed outputs but proofs are "
|
f"Swap recovery failed: mint signed outputs but proofs are "
|
||||||
f"unrecoverable (mint reports them spent). "
|
f"unrecoverable (mint reports them spent). "
|
||||||
f"Expected {minted_amount}, recovered {balance_gained}. "
|
f"Expected {minted_amount}, recovered {balance_gained}. "
|
||||||
f"Local wallet DB ('.wallet/') state is corrupted — "
|
f"Local wallet DB ('.wallet/') state is corrupted — "
|
||||||
f"the counter for keyset is stuck at a bad index range."
|
f"the counter for keyset is stuck at a bad index range."
|
||||||
)
|
)
|
||||||
except TokenConsumedError:
|
except ValueError:
|
||||||
raise
|
raise
|
||||||
except Exception as recovery_err:
|
except Exception as recovery_err:
|
||||||
logger.error(
|
logger.error(
|
||||||
"swap_to_primary_mint: recovery failed",
|
"swap_to_primary_mint: recovery failed",
|
||||||
extra={"error": str(recovery_err)},
|
extra={"error": str(recovery_err)},
|
||||||
)
|
)
|
||||||
raise TokenConsumedError(
|
raise ValueError(
|
||||||
f"Mint on primary failed and recovery unsuccessful: {e}"
|
f"Mint on primary failed and recovery unsuccessful: {e}"
|
||||||
) from e
|
) from e
|
||||||
else:
|
else:
|
||||||
@@ -639,10 +364,7 @@ async def swap_to_primary_mint(
|
|||||||
"mint_quote_id": mint_quote.quote,
|
"mint_quote_id": mint_quote.quote,
|
||||||
},
|
},
|
||||||
)
|
)
|
||||||
# Foreign proofs already melted (spent) — non-retryable.
|
raise
|
||||||
raise TokenConsumedError(
|
|
||||||
"Mint on primary failed after successful melt"
|
|
||||||
) from e
|
|
||||||
|
|
||||||
logger.info(
|
logger.info(
|
||||||
"swap_to_primary_mint: completed successfully",
|
"swap_to_primary_mint: completed successfully",
|
||||||
@@ -700,33 +422,15 @@ async def credit_balance(
|
|||||||
extra={"old_balance": key.balance, "credit_amount": amount},
|
extra={"old_balance": key.balance, "credit_amount": amount},
|
||||||
)
|
)
|
||||||
|
|
||||||
# The token is already redeemed (spent) here, so any crediting failure
|
# Use atomic SQL UPDATE to prevent race conditions during concurrent topups
|
||||||
# is post-redemption and non-retryable — surface it as TokenConsumedError
|
stmt = (
|
||||||
# (a key that vanished mid-flight, or an unexpected DB fault), never a
|
update(db.ApiKey)
|
||||||
# retryable/token-error taxonomy.
|
.where(col(db.ApiKey.hashed_key) == key.hashed_key)
|
||||||
try:
|
.values(balance=(db.ApiKey.balance) + amount)
|
||||||
# Atomic UPDATE to prevent race conditions during concurrent topups.
|
)
|
||||||
stmt = (
|
await session.exec(stmt) # type: ignore[call-overload]
|
||||||
update(db.ApiKey)
|
await session.commit()
|
||||||
.where(col(db.ApiKey.hashed_key) == key.hashed_key)
|
await session.refresh(key)
|
||||||
.values(balance=(db.ApiKey.balance) + amount)
|
|
||||||
)
|
|
||||||
result = await session.exec(stmt) # type: ignore[call-overload]
|
|
||||||
# If pruning removed this key after redemption, do not commit a no-op
|
|
||||||
# balance update and pretend the top-up succeeded.
|
|
||||||
if (getattr(result, "rowcount", 0) or 0) == 0:
|
|
||||||
raise TokenConsumedError(
|
|
||||||
"Token redeemed but the API key disappeared before the "
|
|
||||||
"credit could be recorded"
|
|
||||||
)
|
|
||||||
await session.commit()
|
|
||||||
await session.refresh(key)
|
|
||||||
except TokenConsumedError:
|
|
||||||
raise
|
|
||||||
except Exception as db_error:
|
|
||||||
raise TokenConsumedError(
|
|
||||||
"Token redeemed but crediting the balance failed"
|
|
||||||
) from db_error
|
|
||||||
|
|
||||||
logger.info(
|
logger.info(
|
||||||
"credit_balance: Balance updated successfully",
|
"credit_balance: Balance updated successfully",
|
||||||
@@ -863,19 +567,11 @@ async def fetch_all_balances(
|
|||||||
}
|
}
|
||||||
return error_result
|
return error_result
|
||||||
|
|
||||||
# Build the set of mints to inspect. Received tokens are stored against
|
|
||||||
# ``primary_mint`` (which defaults to a real mint even when ``cashu_mints``
|
|
||||||
# is empty), so include it as a fallback — otherwise a node that accepts
|
|
||||||
# payments would still report empty balances when ``cashu_mints`` is unset.
|
|
||||||
mint_urls: list[str] = list(settings.cashu_mints)
|
|
||||||
if settings.primary_mint and settings.primary_mint not in mint_urls:
|
|
||||||
mint_urls.append(settings.primary_mint)
|
|
||||||
|
|
||||||
# Create tasks for all mint/unit combinations
|
# Create tasks for all mint/unit combinations
|
||||||
async with db.create_session() as session:
|
async with db.create_session() as session:
|
||||||
tasks = [
|
tasks = [
|
||||||
fetch_balance(session, mint_url, unit)
|
fetch_balance(session, mint_url, unit)
|
||||||
for mint_url in mint_urls
|
for mint_url in settings.cashu_mints
|
||||||
for unit in units
|
for unit in units
|
||||||
]
|
]
|
||||||
|
|
||||||
|
|||||||
@@ -1,146 +0,0 @@
|
|||||||
"""Regression tests for charging after stale reservation cleanup."""
|
|
||||||
|
|
||||||
import time
|
|
||||||
import uuid
|
|
||||||
from unittest.mock import patch
|
|
||||||
|
|
||||||
import pytest
|
|
||||||
from sqlmodel.ext.asyncio.session import AsyncSession
|
|
||||||
|
|
||||||
from routstr.core.db import ApiKey
|
|
||||||
from routstr.payment.cost_calculation import CostData
|
|
||||||
|
|
||||||
|
|
||||||
def _make_key(balance: int, reserved: int) -> ApiKey:
|
|
||||||
return ApiKey(
|
|
||||||
hashed_key=f"test_{uuid.uuid4().hex}",
|
|
||||||
balance=balance,
|
|
||||||
reserved_balance=reserved,
|
|
||||||
total_spent=0,
|
|
||||||
total_requests=1,
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
def _cost_data(total_msats: int) -> CostData:
|
|
||||||
return CostData(
|
|
||||||
base_msats=0,
|
|
||||||
input_msats=total_msats // 2,
|
|
||||||
output_msats=total_msats - total_msats // 2,
|
|
||||||
total_msats=total_msats,
|
|
||||||
total_usd=0.0,
|
|
||||||
input_tokens=100,
|
|
||||||
output_tokens=100,
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
@pytest.mark.asyncio
|
|
||||||
async def test_overrun_charges_after_reservation_swept(
|
|
||||||
integration_session: AsyncSession,
|
|
||||||
) -> None:
|
|
||||||
"""Overrun finalize must charge even when the reservation was already released."""
|
|
||||||
from routstr.auth import adjust_payment_for_tokens
|
|
||||||
|
|
||||||
deducted_max_cost = 990 # discounted reservation
|
|
||||||
actual_token_cost = 1000 # actual cost overruns the reservation
|
|
||||||
|
|
||||||
# Sweeper has zeroed reserved_balance but left balance untouched.
|
|
||||||
key = _make_key(balance=1000, reserved=0)
|
|
||||||
integration_session.add(key)
|
|
||||||
await integration_session.commit()
|
|
||||||
|
|
||||||
response_data = {
|
|
||||||
"model": "test-model",
|
|
||||||
"usage": {"prompt_tokens": 100, "completion_tokens": 100},
|
|
||||||
}
|
|
||||||
|
|
||||||
with patch(
|
|
||||||
"routstr.auth.calculate_cost",
|
|
||||||
return_value=_cost_data(actual_token_cost),
|
|
||||||
):
|
|
||||||
await adjust_payment_for_tokens(
|
|
||||||
key, response_data, integration_session, deducted_max_cost
|
|
||||||
)
|
|
||||||
|
|
||||||
await integration_session.refresh(key)
|
|
||||||
|
|
||||||
assert key.total_spent == actual_token_cost, (
|
|
||||||
f"Request was not billed (total_spent={key.total_spent}) — free response bug"
|
|
||||||
)
|
|
||||||
assert key.balance == 1000 - actual_token_cost, (
|
|
||||||
f"Balance not charged: {key.balance}"
|
|
||||||
)
|
|
||||||
assert key.balance >= 0
|
|
||||||
assert key.reserved_balance == 0
|
|
||||||
|
|
||||||
|
|
||||||
@pytest.mark.asyncio
|
|
||||||
async def test_free_response_path_closed_end_to_end(
|
|
||||||
integration_session: AsyncSession,
|
|
||||||
patched_db_engine: None,
|
|
||||||
) -> None:
|
|
||||||
"""A reservation released by the real sweeper must not yield a free response."""
|
|
||||||
from routstr.auth import adjust_payment_for_tokens, pay_for_request
|
|
||||||
from routstr.core.db import create_session, release_stale_reservations
|
|
||||||
|
|
||||||
deducted_max_cost = 990
|
|
||||||
actual_token_cost = 1000
|
|
||||||
key_hash = f"test_sweep_{uuid.uuid4().hex}"
|
|
||||||
|
|
||||||
async with create_session() as session:
|
|
||||||
session.add(
|
|
||||||
ApiKey(
|
|
||||||
hashed_key=key_hash,
|
|
||||||
balance=1000,
|
|
||||||
reserved_balance=0,
|
|
||||||
total_spent=0,
|
|
||||||
total_requests=0,
|
|
||||||
)
|
|
||||||
)
|
|
||||||
await session.commit()
|
|
||||||
|
|
||||||
# Reserve the request, then backdate reserved_at so the sweeper treats it as
|
|
||||||
# stale (simulates a stream that outlived stale_reservation_timeout_seconds).
|
|
||||||
async with create_session() as session:
|
|
||||||
key = await session.get(ApiKey, key_hash)
|
|
||||||
assert key is not None
|
|
||||||
await pay_for_request(key, deducted_max_cost, session)
|
|
||||||
await session.refresh(key)
|
|
||||||
assert key.reserved_balance == deducted_max_cost
|
|
||||||
key.reserved_at = int(time.time()) - 10_000
|
|
||||||
session.add(key)
|
|
||||||
await session.commit()
|
|
||||||
|
|
||||||
# Sweeper releases the stale reservation without charging.
|
|
||||||
async with create_session() as session:
|
|
||||||
released = await release_stale_reservations(session, max_age_seconds=300)
|
|
||||||
assert released == 1
|
|
||||||
|
|
||||||
async with create_session() as session:
|
|
||||||
key = await session.get(ApiKey, key_hash)
|
|
||||||
assert key is not None
|
|
||||||
assert key.reserved_balance == 0, "Precondition: sweeper zeroed the reservation"
|
|
||||||
|
|
||||||
response_data = {
|
|
||||||
"model": "test-model",
|
|
||||||
"usage": {"prompt_tokens": 100, "completion_tokens": 100},
|
|
||||||
}
|
|
||||||
with patch(
|
|
||||||
"routstr.auth.calculate_cost",
|
|
||||||
return_value=_cost_data(actual_token_cost),
|
|
||||||
):
|
|
||||||
await adjust_payment_for_tokens(
|
|
||||||
key, response_data, session, deducted_max_cost
|
|
||||||
)
|
|
||||||
|
|
||||||
async with create_session() as session:
|
|
||||||
final = await session.get(ApiKey, key_hash)
|
|
||||||
assert final is not None
|
|
||||||
|
|
||||||
assert final.total_spent == actual_token_cost, (
|
|
||||||
f"Free response: total_spent={final.total_spent}, expected {actual_token_cost}"
|
|
||||||
)
|
|
||||||
assert final.balance == 1000 - actual_token_cost, (
|
|
||||||
f"Balance not charged after sweep: {final.balance}"
|
|
||||||
)
|
|
||||||
assert final.balance >= 0
|
|
||||||
assert final.reserved_balance == 0
|
|
||||||
@@ -1,128 +0,0 @@
|
|||||||
"""A live upstream provider resolves its OWN database row by stable identity
|
|
||||||
(its primary key), not by its mutable/secret ``api_key``.
|
|
||||||
|
|
||||||
Today ``from_db_row`` drops ``provider_row.id`` and the two self-referential
|
|
||||||
paths — PPQ.AI's insufficient-balance self-disable and the base
|
|
||||||
``refresh_models_cache`` — re-find their own row with
|
|
||||||
``WHERE base_url == self.base_url AND api_key == self.api_key``. That uses a
|
|
||||||
rotatable secret as a self-handle: the moment the row's key changes underneath a
|
|
||||||
live object (a rotation racing an in-flight request), the object can no longer
|
|
||||||
find itself. These tests pin the invariant that a provider looks itself up by
|
|
||||||
identity, so the lookup survives a key change (and, later, key encryption).
|
|
||||||
"""
|
|
||||||
|
|
||||||
from unittest.mock import AsyncMock, patch
|
|
||||||
|
|
||||||
import pytest
|
|
||||||
|
|
||||||
from routstr.core.db import UpstreamProviderRow
|
|
||||||
from routstr.upstream.ppqai import PPQAIUpstreamProvider
|
|
||||||
|
|
||||||
|
|
||||||
@pytest.mark.integration
|
|
||||||
@pytest.mark.asyncio
|
|
||||||
async def test_provider_object_carries_its_persistent_identity(
|
|
||||||
integration_session: object,
|
|
||||||
patched_db_engine: None,
|
|
||||||
) -> None:
|
|
||||||
"""``from_db_row`` gives the in-memory object its row's identity (``db_id``)."""
|
|
||||||
row = UpstreamProviderRow(
|
|
||||||
provider_type="ppqai",
|
|
||||||
base_url="https://api.ppq.ai",
|
|
||||||
api_key="sk-original",
|
|
||||||
enabled=True,
|
|
||||||
provider_fee=1.0,
|
|
||||||
)
|
|
||||||
integration_session.add(row) # type: ignore[attr-defined]
|
|
||||||
await integration_session.commit() # type: ignore[attr-defined]
|
|
||||||
await integration_session.refresh(row) # type: ignore[attr-defined]
|
|
||||||
|
|
||||||
provider = PPQAIUpstreamProvider.from_db_row(row)
|
|
||||||
assert provider is not None
|
|
||||||
|
|
||||||
assert provider.db_id == row.id
|
|
||||||
|
|
||||||
|
|
||||||
@pytest.mark.integration
|
|
||||||
@pytest.mark.asyncio
|
|
||||||
async def test_self_disable_targets_own_row_after_key_rotation(
|
|
||||||
integration_session: object,
|
|
||||||
patched_db_engine: None,
|
|
||||||
) -> None:
|
|
||||||
"""PPQ.AI self-disable must disable *its* row even after the key rotated.
|
|
||||||
|
|
||||||
RED (current): the object holds the pre-rotation key, so the
|
|
||||||
``(base_url, api_key)`` lookup misses the row → the provider is never
|
|
||||||
disabled. GREEN: lookup by ``id`` finds it and disables it.
|
|
||||||
"""
|
|
||||||
row = UpstreamProviderRow(
|
|
||||||
provider_type="ppqai",
|
|
||||||
base_url="https://api.ppq.ai",
|
|
||||||
api_key="sk-original",
|
|
||||||
enabled=True,
|
|
||||||
provider_fee=1.0,
|
|
||||||
)
|
|
||||||
integration_session.add(row) # type: ignore[attr-defined]
|
|
||||||
await integration_session.commit() # type: ignore[attr-defined]
|
|
||||||
await integration_session.refresh(row) # type: ignore[attr-defined]
|
|
||||||
|
|
||||||
provider = PPQAIUpstreamProvider.from_db_row(row) # captures sk-original
|
|
||||||
assert provider is not None
|
|
||||||
|
|
||||||
# Key is rotated in the DB while `provider` is still live.
|
|
||||||
row.api_key = "sk-rotated"
|
|
||||||
integration_session.add(row) # type: ignore[attr-defined]
|
|
||||||
await integration_session.commit() # type: ignore[attr-defined]
|
|
||||||
|
|
||||||
with patch("routstr.proxy.reinitialize_upstreams", new=AsyncMock()):
|
|
||||||
await provider.on_upstream_error_redirect(402, "Insufficient balance")
|
|
||||||
|
|
||||||
await integration_session.refresh(row) # type: ignore[attr-defined]
|
|
||||||
assert row.enabled is False
|
|
||||||
|
|
||||||
|
|
||||||
@pytest.mark.integration
|
|
||||||
@pytest.mark.asyncio
|
|
||||||
async def test_refresh_models_cache_finds_own_row_after_key_rotation(
|
|
||||||
integration_session: object,
|
|
||||||
patched_db_engine: None,
|
|
||||||
) -> None:
|
|
||||||
"""``refresh_models_cache`` must resolve its own row after a key rotation.
|
|
||||||
|
|
||||||
``refresh_models_cache`` swallows every exception (it only logs), so the
|
|
||||||
observable proof it found its row is that it reaches ``list_models`` — which
|
|
||||||
is called with the row's ``id`` only *after* the row is resolved. RED
|
|
||||||
(current): the stale-key ``(base_url, api_key)`` lookup returns nothing, the
|
|
||||||
method raises ``404`` internally and returns before ``list_models`` is ever
|
|
||||||
called. GREEN: lookup by ``id`` finds the row and ``list_models`` runs for
|
|
||||||
that ``id``.
|
|
||||||
"""
|
|
||||||
row = UpstreamProviderRow(
|
|
||||||
provider_type="ppqai",
|
|
||||||
base_url="https://api.ppq.ai",
|
|
||||||
api_key="sk-original",
|
|
||||||
enabled=True,
|
|
||||||
provider_fee=1.0,
|
|
||||||
)
|
|
||||||
integration_session.add(row) # type: ignore[attr-defined]
|
|
||||||
await integration_session.commit() # type: ignore[attr-defined]
|
|
||||||
await integration_session.refresh(row) # type: ignore[attr-defined]
|
|
||||||
row_id = row.id
|
|
||||||
|
|
||||||
provider = PPQAIUpstreamProvider.from_db_row(row) # captures sk-original
|
|
||||||
assert provider is not None
|
|
||||||
|
|
||||||
row.api_key = "sk-rotated"
|
|
||||||
integration_session.add(row) # type: ignore[attr-defined]
|
|
||||||
await integration_session.commit() # type: ignore[attr-defined]
|
|
||||||
|
|
||||||
list_models_mock = AsyncMock(return_value=[])
|
|
||||||
with (
|
|
||||||
patch.object(provider, "fetch_models", new=AsyncMock(return_value=[])),
|
|
||||||
patch("routstr.upstream.base.list_models", new=list_models_mock),
|
|
||||||
):
|
|
||||||
await provider.refresh_models_cache()
|
|
||||||
|
|
||||||
list_models_mock.assert_awaited_once()
|
|
||||||
assert list_models_mock.await_args is not None
|
|
||||||
assert list_models_mock.await_args.kwargs["upstream_id"] == row_id
|
|
||||||
@@ -1,283 +0,0 @@
|
|||||||
"""
|
|
||||||
Tests for prune_dead_api_keys — the janitor that removes provably-dead 0/0/0
|
|
||||||
API keys (funded keys fully refunded/expired without ever being used, plus bare
|
|
||||||
orphans), while protecting keys that are still meaningful.
|
|
||||||
"""
|
|
||||||
|
|
||||||
import time
|
|
||||||
import uuid
|
|
||||||
from collections.abc import Awaitable, Callable
|
|
||||||
from typing import Any, cast
|
|
||||||
|
|
||||||
import pytest
|
|
||||||
from sqlalchemy.sql.dml import Update
|
|
||||||
from sqlmodel import col, update
|
|
||||||
|
|
||||||
from routstr.core.db import (
|
|
||||||
ApiKey,
|
|
||||||
CashuTransaction,
|
|
||||||
LightningInvoice,
|
|
||||||
create_session,
|
|
||||||
prune_dead_api_keys,
|
|
||||||
)
|
|
||||||
|
|
||||||
OLD = 100 # min_age_seconds used by the tests
|
|
||||||
NOW = int(time.time())
|
|
||||||
LONG_AGO = NOW - 10_000 # well past the grace period
|
|
||||||
|
|
||||||
|
|
||||||
async def _exists(key_hash: str) -> bool:
|
|
||||||
async with create_session() as session:
|
|
||||||
return (await session.get(ApiKey, key_hash)) is not None
|
|
||||||
|
|
||||||
|
|
||||||
def _dead_key(created_at: int | None) -> ApiKey:
|
|
||||||
return ApiKey(
|
|
||||||
hashed_key=f"dead_{uuid.uuid4().hex}",
|
|
||||||
balance=0,
|
|
||||||
reserved_balance=0,
|
|
||||||
total_spent=0,
|
|
||||||
total_requests=0,
|
|
||||||
created_at=created_at,
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
@pytest.mark.asyncio
|
|
||||||
async def test_prunes_old_refunded_zero_key(patched_db_engine: None) -> None:
|
|
||||||
"""A funded-then-refunded key (0/0/0, NULL parent, old) is pruned."""
|
|
||||||
key = _dead_key(LONG_AGO)
|
|
||||||
async with create_session() as session:
|
|
||||||
session.add(key)
|
|
||||||
await session.commit()
|
|
||||||
|
|
||||||
async with create_session() as session:
|
|
||||||
pruned = await prune_dead_api_keys(session, OLD)
|
|
||||||
|
|
||||||
assert pruned == 1
|
|
||||||
assert not await _exists(key.hashed_key)
|
|
||||||
|
|
||||||
|
|
||||||
@pytest.mark.asyncio
|
|
||||||
async def test_grace_period_protects_fresh_key(patched_db_engine: None) -> None:
|
|
||||||
"""A dead-looking but recently created key is protected by the grace period."""
|
|
||||||
key = _dead_key(int(time.time()))
|
|
||||||
async with create_session() as session:
|
|
||||||
session.add(key)
|
|
||||||
await session.commit()
|
|
||||||
|
|
||||||
async with create_session() as session:
|
|
||||||
pruned = await prune_dead_api_keys(session, OLD)
|
|
||||||
|
|
||||||
assert pruned == 0
|
|
||||||
assert await _exists(key.hashed_key)
|
|
||||||
|
|
||||||
|
|
||||||
@pytest.mark.asyncio
|
|
||||||
async def test_used_key_never_pruned(patched_db_engine: None) -> None:
|
|
||||||
"""Keys with any spend/requests or live balance are never pruned."""
|
|
||||||
spent = _dead_key(LONG_AGO)
|
|
||||||
spent.total_spent = 1
|
|
||||||
requested = _dead_key(LONG_AGO)
|
|
||||||
requested.total_requests = 1
|
|
||||||
funded = _dead_key(LONG_AGO)
|
|
||||||
funded.balance = 1000
|
|
||||||
reserved = _dead_key(LONG_AGO)
|
|
||||||
reserved.reserved_balance = 500
|
|
||||||
|
|
||||||
async with create_session() as session:
|
|
||||||
for k in (spent, requested, funded, reserved):
|
|
||||||
session.add(k)
|
|
||||||
await session.commit()
|
|
||||||
|
|
||||||
async with create_session() as session:
|
|
||||||
pruned = await prune_dead_api_keys(session, OLD)
|
|
||||||
|
|
||||||
assert pruned == 0
|
|
||||||
for k in (spent, requested, funded, reserved):
|
|
||||||
assert await _exists(k.hashed_key)
|
|
||||||
|
|
||||||
|
|
||||||
@pytest.mark.asyncio
|
|
||||||
async def test_parent_and_child_keys_are_not_pruned(
|
|
||||||
patched_db_engine: None,
|
|
||||||
) -> None:
|
|
||||||
"""Pruning must not orphan child keys or delete valid children."""
|
|
||||||
parent = _dead_key(LONG_AGO)
|
|
||||||
child = ApiKey(
|
|
||||||
hashed_key=f"child_{uuid.uuid4().hex}",
|
|
||||||
balance=0,
|
|
||||||
reserved_balance=0,
|
|
||||||
total_spent=0,
|
|
||||||
total_requests=0,
|
|
||||||
created_at=LONG_AGO,
|
|
||||||
parent_key_hash=parent.hashed_key,
|
|
||||||
)
|
|
||||||
async with create_session() as session:
|
|
||||||
session.add(parent)
|
|
||||||
session.add(child)
|
|
||||||
await session.commit()
|
|
||||||
|
|
||||||
async with create_session() as session:
|
|
||||||
pruned = await prune_dead_api_keys(session, OLD)
|
|
||||||
|
|
||||||
assert pruned == 0
|
|
||||||
assert await _exists(parent.hashed_key)
|
|
||||||
assert await _exists(child.hashed_key)
|
|
||||||
|
|
||||||
|
|
||||||
@pytest.mark.asyncio
|
|
||||||
async def test_pending_invoice_protects_key(patched_db_engine: None) -> None:
|
|
||||||
"""A key referenced by a pending topup invoice is never pruned mid-topup."""
|
|
||||||
key = _dead_key(LONG_AGO)
|
|
||||||
invoice = LightningInvoice(
|
|
||||||
id=f"inv_{uuid.uuid4().hex}",
|
|
||||||
bolt11=f"lnbc_{uuid.uuid4().hex}",
|
|
||||||
amount_sats=10,
|
|
||||||
description="topup",
|
|
||||||
payment_hash=uuid.uuid4().hex,
|
|
||||||
status="pending",
|
|
||||||
api_key_hash=key.hashed_key,
|
|
||||||
purpose="topup",
|
|
||||||
expires_at=NOW + 10_000,
|
|
||||||
)
|
|
||||||
async with create_session() as session:
|
|
||||||
session.add(key)
|
|
||||||
session.add(invoice)
|
|
||||||
await session.commit()
|
|
||||||
|
|
||||||
async with create_session() as session:
|
|
||||||
pruned = await prune_dead_api_keys(session, OLD)
|
|
||||||
|
|
||||||
assert pruned == 0
|
|
||||||
assert await _exists(key.hashed_key)
|
|
||||||
|
|
||||||
|
|
||||||
@pytest.mark.asyncio
|
|
||||||
async def test_paid_invoice_does_not_protect_key(patched_db_engine: None) -> None:
|
|
||||||
"""A settled (non-pending) invoice does not keep a dead key alive."""
|
|
||||||
key = _dead_key(LONG_AGO)
|
|
||||||
invoice = LightningInvoice(
|
|
||||||
id=f"inv_{uuid.uuid4().hex}",
|
|
||||||
bolt11=f"lnbc_{uuid.uuid4().hex}",
|
|
||||||
amount_sats=10,
|
|
||||||
description="topup",
|
|
||||||
payment_hash=uuid.uuid4().hex,
|
|
||||||
status="paid",
|
|
||||||
api_key_hash=key.hashed_key,
|
|
||||||
purpose="topup",
|
|
||||||
expires_at=NOW - 1,
|
|
||||||
)
|
|
||||||
async with create_session() as session:
|
|
||||||
session.add(key)
|
|
||||||
session.add(invoice)
|
|
||||||
await session.commit()
|
|
||||||
|
|
||||||
async with create_session() as session:
|
|
||||||
pruned = await prune_dead_api_keys(session, OLD)
|
|
||||||
|
|
||||||
assert pruned == 1
|
|
||||||
assert not await _exists(key.hashed_key)
|
|
||||||
|
|
||||||
|
|
||||||
@pytest.mark.asyncio
|
|
||||||
async def test_key_that_becomes_meaningful_during_prune_survives(
|
|
||||||
patched_db_engine: None, monkeypatch: pytest.MonkeyPatch
|
|
||||||
) -> None:
|
|
||||||
"""Revalidate before unlink/delete so a late top-up cannot be pruned."""
|
|
||||||
key = _dead_key(LONG_AGO)
|
|
||||||
txn = CashuTransaction(
|
|
||||||
id=uuid.uuid4().hex,
|
|
||||||
token="cashuABC",
|
|
||||||
amount=21,
|
|
||||||
unit="sat",
|
|
||||||
type="in",
|
|
||||||
source="apikey",
|
|
||||||
api_key_hashed_key=key.hashed_key,
|
|
||||||
)
|
|
||||||
async with create_session() as session:
|
|
||||||
session.add(key)
|
|
||||||
session.add(txn)
|
|
||||||
await session.commit()
|
|
||||||
|
|
||||||
async with create_session() as session:
|
|
||||||
original_exec = cast(Callable[..., Awaitable[Any]], session.exec)
|
|
||||||
topped_up = False
|
|
||||||
|
|
||||||
async def exec_with_late_topup(
|
|
||||||
statement: Any, *args: Any, **kwargs: Any
|
|
||||||
) -> Any:
|
|
||||||
nonlocal topped_up
|
|
||||||
if (
|
|
||||||
not topped_up
|
|
||||||
and isinstance(statement, Update)
|
|
||||||
and getattr(statement.table, "name", None) == "cashu_transactions"
|
|
||||||
):
|
|
||||||
topped_up = True
|
|
||||||
async with create_session() as topup_session:
|
|
||||||
await topup_session.exec( # type: ignore[call-overload]
|
|
||||||
update(ApiKey)
|
|
||||||
.where(col(ApiKey.hashed_key) == key.hashed_key)
|
|
||||||
.values(balance=42)
|
|
||||||
)
|
|
||||||
await topup_session.commit()
|
|
||||||
return await original_exec(statement, *args, **kwargs)
|
|
||||||
|
|
||||||
monkeypatch.setattr(session, "exec", exec_with_late_topup)
|
|
||||||
pruned = await prune_dead_api_keys(session, OLD)
|
|
||||||
|
|
||||||
assert pruned == 0
|
|
||||||
assert await _exists(key.hashed_key)
|
|
||||||
async with create_session() as session:
|
|
||||||
surviving = await session.get(CashuTransaction, txn.id)
|
|
||||||
assert surviving is not None
|
|
||||||
assert surviving.api_key_hashed_key == key.hashed_key
|
|
||||||
|
|
||||||
|
|
||||||
@pytest.mark.asyncio
|
|
||||||
async def test_transaction_audit_trail_preserved(patched_db_engine: None) -> None:
|
|
||||||
"""Pruning a refunded key keeps its cashu_transactions, unlinked from the key."""
|
|
||||||
key = _dead_key(LONG_AGO)
|
|
||||||
txn = CashuTransaction(
|
|
||||||
id=uuid.uuid4().hex,
|
|
||||||
token="cashuABC",
|
|
||||||
amount=21,
|
|
||||||
unit="sat",
|
|
||||||
type="in",
|
|
||||||
source="apikey",
|
|
||||||
api_key_hashed_key=key.hashed_key,
|
|
||||||
)
|
|
||||||
async with create_session() as session:
|
|
||||||
session.add(key)
|
|
||||||
session.add(txn)
|
|
||||||
await session.commit()
|
|
||||||
|
|
||||||
async with create_session() as session:
|
|
||||||
pruned = await prune_dead_api_keys(session, OLD)
|
|
||||||
|
|
||||||
assert pruned == 1
|
|
||||||
assert not await _exists(key.hashed_key)
|
|
||||||
|
|
||||||
async with create_session() as session:
|
|
||||||
surviving = await session.get(CashuTransaction, txn.id)
|
|
||||||
assert surviving is not None, "Financial audit row must survive key deletion"
|
|
||||||
assert surviving.api_key_hashed_key is None, "Link must be nulled, not dangling"
|
|
||||||
assert surviving.amount == 21
|
|
||||||
|
|
||||||
|
|
||||||
@pytest.mark.asyncio
|
|
||||||
async def test_periodic_prune_disabled_returns_immediately(
|
|
||||||
monkeypatch: pytest.MonkeyPatch,
|
|
||||||
) -> None:
|
|
||||||
"""Non-positive intervals disable the janitor."""
|
|
||||||
from unittest.mock import AsyncMock
|
|
||||||
|
|
||||||
from routstr import auth
|
|
||||||
from routstr.core.settings import settings
|
|
||||||
|
|
||||||
monkeypatch.setattr(settings, "dead_key_prune_interval_seconds", 0)
|
|
||||||
sleep_mock = AsyncMock()
|
|
||||||
monkeypatch.setattr(auth.asyncio, "sleep", sleep_mock)
|
|
||||||
|
|
||||||
await auth.periodic_dead_key_prune()
|
|
||||||
|
|
||||||
sleep_mock.assert_not_called()
|
|
||||||
@@ -1,194 +0,0 @@
|
|||||||
"""
|
|
||||||
Integration tests for reactive swap fee retries via the wallet topup endpoint.
|
|
||||||
|
|
||||||
Foreign-mint tokens are swapped to the primary mint using the foreign mint's
|
|
||||||
melt quote, whose fee_reserve is a non-binding estimate (NUT-05): the mint may
|
|
||||||
demand more when re-quoting or at melt execution. These tests cover the
|
|
||||||
endpoint behaviour in those cases:
|
|
||||||
|
|
||||||
1. The mint demands one sat more at melt time than every quote reported
|
|
||||||
(the mint.cubabitcoin.org incident): the swap retries with a smaller
|
|
||||||
invoice and the topup succeeds, crediting the recomputed amount.
|
|
||||||
2. The real melt quote reports a higher fee_reserve than the estimate: the
|
|
||||||
swap re-quotes from the observed fee and the topup succeeds.
|
|
||||||
3. The mint escalates its fee demands on every attempt: the retry budget is
|
|
||||||
exhausted and the endpoint returns 400 with a clear error (never 500),
|
|
||||||
without ever executing a melt.
|
|
||||||
"""
|
|
||||||
|
|
||||||
from collections.abc import Callable
|
|
||||||
from unittest.mock import AsyncMock, Mock, patch
|
|
||||||
|
|
||||||
import pytest
|
|
||||||
from httpx import AsyncClient, Response
|
|
||||||
|
|
||||||
from routstr.core.settings import settings
|
|
||||||
|
|
||||||
# Captured at collection time, before the integration_app fixture replaces it
|
|
||||||
# with the testmint stub that bypasses swapping (see conftest.py).
|
|
||||||
from routstr.wallet import recieve_token as _real_recieve_token
|
|
||||||
|
|
||||||
PRIMARY_MINT = "http://primary:3338"
|
|
||||||
|
|
||||||
|
|
||||||
def _make_swap_mocks(
|
|
||||||
token_amount: int,
|
|
||||||
fee_reserves: list[int],
|
|
||||||
input_fees: int = 0,
|
|
||||||
mint_url: str = "http://foreign-mint:3338",
|
|
||||||
) -> tuple[Mock, Mock, Mock]:
|
|
||||||
"""Return (token, token_wallet, primary_wallet) mocks that act like a mint.
|
|
||||||
|
|
||||||
Mint quotes pass the requested amount through their ``request`` field and
|
|
||||||
melt quotes echo that amount back, so the mocks stay consistent for
|
|
||||||
whatever amounts the implementation requests. ``fee_reserves`` supplies the
|
|
||||||
fee_reserve of each successive melt quote (the first serves the estimation
|
|
||||||
pass); requesting more quotes than provided fails the test.
|
|
||||||
"""
|
|
||||||
mock_token = Mock()
|
|
||||||
mock_token.mint = mint_url
|
|
||||||
mock_token.unit = "sat"
|
|
||||||
mock_token.amount = token_amount
|
|
||||||
mock_token.keysets = ["keyset1"]
|
|
||||||
mock_token.proofs = [Mock(amount=token_amount)]
|
|
||||||
|
|
||||||
mock_token_wallet = Mock()
|
|
||||||
mock_token_wallet.load_mint = AsyncMock()
|
|
||||||
mock_token_wallet.load_proofs = AsyncMock()
|
|
||||||
mock_token_wallet.get_fees_for_proofs = Mock(return_value=input_fees)
|
|
||||||
|
|
||||||
mock_primary_wallet = Mock()
|
|
||||||
mock_primary_wallet.load_mint = AsyncMock()
|
|
||||||
mock_primary_wallet.load_proofs = AsyncMock()
|
|
||||||
mock_primary_wallet.available_balance = Mock(amount=0)
|
|
||||||
mock_primary_wallet.mint = AsyncMock(return_value=Mock())
|
|
||||||
|
|
||||||
fees = iter(fee_reserves)
|
|
||||||
|
|
||||||
def _next_fee() -> int:
|
|
||||||
try:
|
|
||||||
return next(fees)
|
|
||||||
except StopIteration:
|
|
||||||
raise AssertionError(
|
|
||||||
"more melt quotes requested than fee_reserves provided"
|
|
||||||
) from None
|
|
||||||
|
|
||||||
mock_primary_wallet.request_mint = AsyncMock(
|
|
||||||
side_effect=lambda amount: Mock(quote=f"mint_quote_{amount}", request=amount)
|
|
||||||
)
|
|
||||||
mock_token_wallet.melt_quote = AsyncMock(
|
|
||||||
side_effect=lambda invoice: Mock(
|
|
||||||
quote=f"melt_quote_{invoice}", amount=invoice, fee_reserve=_next_fee()
|
|
||||||
)
|
|
||||||
)
|
|
||||||
mock_token_wallet.melt = AsyncMock(return_value=Mock())
|
|
||||||
|
|
||||||
return mock_token, mock_token_wallet, mock_primary_wallet
|
|
||||||
|
|
||||||
|
|
||||||
def _wallet_router(primary_wallet: Mock, token_wallet: Mock) -> Callable[..., Mock]:
|
|
||||||
"""Route get_wallet calls to the primary or foreign wallet mock by URL."""
|
|
||||||
|
|
||||||
def fake_get_wallet(mint_url: str, unit: str = "sat", load: bool = True) -> Mock:
|
|
||||||
return primary_wallet if mint_url == PRIMARY_MINT else token_wallet
|
|
||||||
|
|
||||||
return fake_get_wallet
|
|
||||||
|
|
||||||
|
|
||||||
async def _post_topup(
|
|
||||||
client: AsyncClient,
|
|
||||||
mock_token: Mock,
|
|
||||||
token_wallet: Mock,
|
|
||||||
primary_wallet: Mock,
|
|
||||||
) -> Response:
|
|
||||||
"""POST /v1/wallet/topup with the swap layer mocked at the mint boundary.
|
|
||||||
|
|
||||||
The conftest's testmint stub for recieve_token is swapped back for the
|
|
||||||
real implementation so the request exercises the actual swap path.
|
|
||||||
"""
|
|
||||||
with patch("routstr.wallet.recieve_token", _real_recieve_token):
|
|
||||||
with patch(
|
|
||||||
"routstr.wallet.deserialize_token_from_string", return_value=mock_token
|
|
||||||
):
|
|
||||||
with patch(
|
|
||||||
"routstr.wallet.get_wallet",
|
|
||||||
side_effect=_wallet_router(primary_wallet, token_wallet),
|
|
||||||
):
|
|
||||||
with patch.object(settings, "primary_mint", PRIMARY_MINT):
|
|
||||||
with patch.object(settings, "primary_mint_unit", "sat"):
|
|
||||||
with patch.object(settings, "cashu_mints", [PRIMARY_MINT]):
|
|
||||||
return await client.post(
|
|
||||||
"/v1/wallet/topup",
|
|
||||||
params={"cashu_token": "cashuAtest_foreign_token"},
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
@pytest.mark.integration
|
|
||||||
@pytest.mark.asyncio
|
|
||||||
async def test_topup_retries_when_melt_demands_more_than_quoted(
|
|
||||||
authenticated_client: AsyncClient,
|
|
||||||
) -> None:
|
|
||||||
"""A 179-sat token where every quote reports fee_reserve=1 but the mint
|
|
||||||
rejects the first melt demanding 180. The retry shrinks the invoice to 177
|
|
||||||
and the topup credits 177 sats (177_000 msats)."""
|
|
||||||
mock_token, token_wallet, primary_wallet = _make_swap_mocks(
|
|
||||||
179, fee_reserves=[1, 1, 1], mint_url="http://mint.cubabitcoin.org"
|
|
||||||
)
|
|
||||||
token_wallet.melt.side_effect = [
|
|
||||||
Exception(
|
|
||||||
"Mint Error: not enough inputs provided for melt. "
|
|
||||||
"Provided: 179, needed: 180 (Code: 11000)"
|
|
||||||
),
|
|
||||||
Mock(),
|
|
||||||
]
|
|
||||||
|
|
||||||
response = await _post_topup(
|
|
||||||
authenticated_client, mock_token, token_wallet, primary_wallet
|
|
||||||
)
|
|
||||||
|
|
||||||
assert response.status_code == 200
|
|
||||||
assert response.json()["msats"] == 177_000
|
|
||||||
assert token_wallet.melt.call_count == 2
|
|
||||||
|
|
||||||
|
|
||||||
@pytest.mark.integration
|
|
||||||
@pytest.mark.asyncio
|
|
||||||
async def test_topup_retries_when_quote_fee_exceeds_estimate(
|
|
||||||
authenticated_client: AsyncClient,
|
|
||||||
) -> None:
|
|
||||||
"""A 1000-sat token estimated at fee 20, but the real quote demands 23.
|
|
||||||
The retry recomputes 1000 - 23 = 977, which fits, and the topup credits
|
|
||||||
977 sats (977_000 msats) with a single melt."""
|
|
||||||
mock_token, token_wallet, primary_wallet = _make_swap_mocks(
|
|
||||||
1000, fee_reserves=[20, 23, 23]
|
|
||||||
)
|
|
||||||
|
|
||||||
response = await _post_topup(
|
|
||||||
authenticated_client, mock_token, token_wallet, primary_wallet
|
|
||||||
)
|
|
||||||
|
|
||||||
assert response.status_code == 200
|
|
||||||
assert response.json()["msats"] == 977_000
|
|
||||||
assert token_wallet.melt.call_count == 1
|
|
||||||
|
|
||||||
|
|
||||||
@pytest.mark.integration
|
|
||||||
@pytest.mark.asyncio
|
|
||||||
async def test_topup_returns_422_when_retries_exhausted(
|
|
||||||
authenticated_client: AsyncClient,
|
|
||||||
) -> None:
|
|
||||||
"""A mint that escalates fee_reserve on every re-quote (1 → 10 → 25 → 50)
|
|
||||||
exhausts the retry budget: clean 422 mint_error/too-small taxonomy, melt
|
|
||||||
never executed."""
|
|
||||||
mock_token, token_wallet, primary_wallet = _make_swap_mocks(
|
|
||||||
1000, fee_reserves=[1, 10, 25, 50]
|
|
||||||
)
|
|
||||||
|
|
||||||
response = await _post_topup(
|
|
||||||
authenticated_client, mock_token, token_wallet, primary_wallet
|
|
||||||
)
|
|
||||||
|
|
||||||
assert response.status_code == 422
|
|
||||||
assert "too small to cover swap fees" in response.json()["detail"]
|
|
||||||
assert token_wallet.melt_quote.call_count == 4 # estimation + 3 attempts
|
|
||||||
token_wallet.melt.assert_not_called()
|
|
||||||
@@ -12,7 +12,10 @@ from sqlmodel import select
|
|||||||
|
|
||||||
from routstr.core.db import ApiKey
|
from routstr.core.db import ApiKey
|
||||||
|
|
||||||
from .utils import ResponseValidator
|
from .utils import (
|
||||||
|
CashuTokenGenerator,
|
||||||
|
ResponseValidator,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
@pytest.mark.integration
|
@pytest.mark.integration
|
||||||
@@ -76,31 +79,29 @@ async def test_api_key_generation_invalid_token(
|
|||||||
# Capture initial state
|
# Capture initial state
|
||||||
await db_snapshot.capture()
|
await db_snapshot.capture()
|
||||||
|
|
||||||
# Non-Cashu bearer values are invalid API keys (401). Malformed values that
|
# Test various invalid tokens
|
||||||
# look like Cashu tokens use the shared Cashu taxonomy (400 invalid_token).
|
invalid_tokens = [
|
||||||
invalid_tokens: list[tuple[str, int, str | None]] = [
|
CashuTokenGenerator.generate_invalid_token(), # Malformed token
|
||||||
("not-a-cashu-token", 401, None),
|
"not-a-cashu-token", # Wrong format
|
||||||
("sk-not-a-real-api-key", 401, None),
|
"cashuA", # Empty token
|
||||||
("cashuA", 400, "invalid_cashu_token"),
|
"cashuA" + "x" * 1000, # Invalid base64
|
||||||
("cashuA" + "x" * 1000, 400, "invalid_cashu_token"),
|
|
||||||
]
|
]
|
||||||
|
|
||||||
for invalid_token, expected_status, expected_code in invalid_tokens:
|
for invalid_token in invalid_tokens:
|
||||||
integration_client.headers["Authorization"] = f"Bearer {invalid_token}"
|
integration_client.headers["Authorization"] = f"Bearer {invalid_token}"
|
||||||
response = await integration_client.get("/v1/wallet/info")
|
response = await integration_client.get("/v1/wallet/info")
|
||||||
|
|
||||||
assert response.status_code == expected_status, (
|
# Should fail with 401
|
||||||
|
assert response.status_code == 401, (
|
||||||
f"Token {invalid_token[:20]}... should be invalid"
|
f"Token {invalid_token[:20]}... should be invalid"
|
||||||
)
|
)
|
||||||
|
|
||||||
# Validate error response
|
# Validate error response
|
||||||
validator = ResponseValidator()
|
validator = ResponseValidator()
|
||||||
error_validation = validator.validate_error_response(
|
error_validation = validator.validate_error_response(
|
||||||
response, expected_status=expected_status, expected_error_key="detail"
|
response, expected_status=401, expected_error_key="detail"
|
||||||
)
|
)
|
||||||
assert error_validation["valid"]
|
assert error_validation["valid"]
|
||||||
if expected_code is not None:
|
|
||||||
assert response.json()["detail"]["error"]["code"] == expected_code
|
|
||||||
|
|
||||||
# Verify no database changes
|
# Verify no database changes
|
||||||
diff = await db_snapshot.diff()
|
diff = await db_snapshot.diff()
|
||||||
|
|||||||
@@ -14,7 +14,6 @@ from httpx import AsyncClient
|
|||||||
from sqlmodel import select
|
from sqlmodel import select
|
||||||
|
|
||||||
from routstr.core.db import ApiKey, CashuTransaction
|
from routstr.core.db import ApiKey, CashuTransaction
|
||||||
from routstr.wallet import MintConnectionError
|
|
||||||
|
|
||||||
|
|
||||||
@pytest.mark.integration
|
@pytest.mark.integration
|
||||||
@@ -504,19 +503,26 @@ async def test_mint_unavailability_handling(
|
|||||||
|
|
||||||
# The global mock in conftest.py is already in place,
|
# The global mock in conftest.py is already in place,
|
||||||
# so we need to temporarily modify it
|
# so we need to temporarily modify it
|
||||||
raw_error = "Mint unavailable: Connection refused"
|
from unittest.mock import patch
|
||||||
|
|
||||||
# Make the send_token method raise a typed mint connection exception.
|
# Make the send_token method raise an exception
|
||||||
with patch(
|
with patch(
|
||||||
"routstr.balance.send_token",
|
"routstr.balance.send_token",
|
||||||
side_effect=MintConnectionError(raw_error),
|
side_effect=Exception("Mint unavailable: Connection refused"),
|
||||||
):
|
):
|
||||||
response = await authenticated_client.post("/v1/wallet/refund")
|
# The exception should propagate as a 503 error (Service Unavailable)
|
||||||
assert response.status_code == 503
|
# But we need to handle it properly
|
||||||
assert response.json()["detail"] == "Mint service unavailable"
|
try:
|
||||||
assert raw_error not in response.text
|
response = await authenticated_client.post("/v1/wallet/refund")
|
||||||
|
# If we get here, check the status code
|
||||||
|
assert response.status_code == 503
|
||||||
|
assert "Mint service unavailable" in response.json()["detail"]
|
||||||
|
except Exception as e:
|
||||||
|
# If the exception propagates, that's also a failure scenario
|
||||||
|
assert "Mint unavailable" in str(e)
|
||||||
|
|
||||||
# Balance should remain unchanged (transaction should roll back)
|
# Balance should remain unchanged (transaction should roll back)
|
||||||
|
# Note: Current implementation might not handle this perfectly
|
||||||
wallet_response = await authenticated_client.get("/v1/wallet/")
|
wallet_response = await authenticated_client.get("/v1/wallet/")
|
||||||
assert wallet_response.status_code == 200
|
assert wallet_response.status_code == 200
|
||||||
assert wallet_response.json()["balance"] == 10_000_000
|
assert wallet_response.json()["balance"] == 10_000_000
|
||||||
|
|||||||
@@ -1,293 +0,0 @@
|
|||||||
import hashlib
|
|
||||||
from types import SimpleNamespace
|
|
||||||
from typing import AsyncGenerator, cast
|
|
||||||
from unittest.mock import AsyncMock, patch
|
|
||||||
|
|
||||||
import httpx
|
|
||||||
import pytest
|
|
||||||
from fastapi import HTTPException
|
|
||||||
from sqlalchemy.ext.asyncio import AsyncEngine, create_async_engine
|
|
||||||
from sqlalchemy.pool import StaticPool
|
|
||||||
from sqlmodel import SQLModel
|
|
||||||
from sqlmodel.ext.asyncio.session import AsyncSession
|
|
||||||
|
|
||||||
from routstr.auth import validate_bearer_key
|
|
||||||
from routstr.core.db import ApiKey
|
|
||||||
from routstr.wallet import MintConnectionError
|
|
||||||
|
|
||||||
|
|
||||||
def _value_error_wrapping_transport() -> ValueError:
|
|
||||||
"""A ValueError re-raised ``from`` a real httpx transport error, mirroring
|
|
||||||
``wallet.py`` wrapping a connection failure. The sanitized classifier must
|
|
||||||
still see the mint-unreachable signal through the ``__cause__`` chain."""
|
|
||||||
try:
|
|
||||||
raise httpx.ConnectError("All connection attempts failed")
|
|
||||||
except httpx.ConnectError as exc:
|
|
||||||
err = ValueError("Failed to estimate fees: connection failed")
|
|
||||||
err.__cause__ = exc
|
|
||||||
return err
|
|
||||||
|
|
||||||
|
|
||||||
def _make_engine() -> AsyncEngine:
|
|
||||||
return create_async_engine(
|
|
||||||
"sqlite+aiosqlite://",
|
|
||||||
poolclass=StaticPool,
|
|
||||||
connect_args={"check_same_thread": False},
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
@pytest.fixture
|
|
||||||
async def session() -> AsyncGenerator[AsyncSession, None]:
|
|
||||||
engine = _make_engine()
|
|
||||||
async with engine.begin() as conn:
|
|
||||||
await conn.run_sync(SQLModel.metadata.create_all)
|
|
||||||
db_session = AsyncSession(engine, expire_on_commit=False)
|
|
||||||
try:
|
|
||||||
yield db_session
|
|
||||||
finally:
|
|
||||||
await db_session.close()
|
|
||||||
await engine.dispose()
|
|
||||||
|
|
||||||
|
|
||||||
@pytest.mark.asyncio
|
|
||||||
async def test_failed_first_cashu_redemption_rolls_back_empty_api_key(
|
|
||||||
session: AsyncSession,
|
|
||||||
) -> None:
|
|
||||||
token = "cashuAfirst_seen_but_redemption_fails"
|
|
||||||
hashed_key = hashlib.sha256(token.encode()).hexdigest()
|
|
||||||
token_obj = SimpleNamespace(mint="http://mint:3338", unit="sat")
|
|
||||||
|
|
||||||
from routstr.core.settings import settings
|
|
||||||
|
|
||||||
with (
|
|
||||||
patch.object(settings, "cashu_mints", ["http://mint:3338"]),
|
|
||||||
patch("routstr.auth.deserialize_token_from_string", return_value=token_obj),
|
|
||||||
patch(
|
|
||||||
"routstr.auth.credit_balance",
|
|
||||||
new=AsyncMock(side_effect=ValueError("token already spent")),
|
|
||||||
),
|
|
||||||
):
|
|
||||||
with pytest.raises(HTTPException):
|
|
||||||
await validate_bearer_key(token, session)
|
|
||||||
|
|
||||||
assert await session.get(ApiKey, hashed_key) is None
|
|
||||||
|
|
||||||
|
|
||||||
@pytest.mark.parametrize(
|
|
||||||
("error", "expected_status", "expected_type", "expected_message", "expected_code"),
|
|
||||||
[
|
|
||||||
(
|
|
||||||
ValueError("Mint Error: Token already spent. (Code: 11001)"),
|
|
||||||
400,
|
|
||||||
"token_already_spent",
|
|
||||||
"Cashu token already spent",
|
|
||||||
"cashu_token_already_spent",
|
|
||||||
),
|
|
||||||
(
|
|
||||||
# Raw httpx transport error propagated unwrapped from cashu.
|
|
||||||
httpx.ConnectError("All connection attempts failed"),
|
|
||||||
503,
|
|
||||||
"mint_unreachable",
|
|
||||||
"Cashu mint is unreachable",
|
|
||||||
"cashu_mint_unreachable",
|
|
||||||
),
|
|
||||||
(
|
|
||||||
# Typed error raised by wallet.py at a wrap site.
|
|
||||||
MintConnectionError("connect to http://mint:3338 refused"),
|
|
||||||
503,
|
|
||||||
"mint_unreachable",
|
|
||||||
"Cashu mint is unreachable",
|
|
||||||
"cashu_mint_unreachable",
|
|
||||||
),
|
|
||||||
(
|
|
||||||
# ValueError wrapping the httpx error in its __cause__ chain.
|
|
||||||
_value_error_wrapping_transport(),
|
|
||||||
503,
|
|
||||||
"mint_unreachable",
|
|
||||||
"Cashu mint is unreachable",
|
|
||||||
"cashu_mint_unreachable",
|
|
||||||
),
|
|
||||||
(
|
|
||||||
# asyncio.TimeoutError is builtin TimeoutError on 3.11+.
|
|
||||||
TimeoutError("Timed out connecting to Cashu mint http://mint:3338"),
|
|
||||||
503,
|
|
||||||
"mint_unreachable",
|
|
||||||
"Cashu mint is unreachable",
|
|
||||||
"cashu_mint_unreachable",
|
|
||||||
),
|
|
||||||
(
|
|
||||||
ValueError(
|
|
||||||
"Token amount (5 sat) is insufficient to cover melt fees. "
|
|
||||||
"Needed: 7 sat (amount: 5 + fee: 1 + input_fees: 1)"
|
|
||||||
),
|
|
||||||
422,
|
|
||||||
"mint_error",
|
|
||||||
"Token value is too small to cover swap fees",
|
|
||||||
"cashu_token_swap_fees_exceed_amount",
|
|
||||||
),
|
|
||||||
(
|
|
||||||
ValueError(
|
|
||||||
"Failed to estimate fees: Fees (7 sat) exceed token amount (5 sat)"
|
|
||||||
),
|
|
||||||
422,
|
|
||||||
"mint_error",
|
|
||||||
"Token value is too small to cover swap fees",
|
|
||||||
"cashu_token_swap_fees_exceed_amount",
|
|
||||||
),
|
|
||||||
(
|
|
||||||
ValueError(
|
|
||||||
"Failed to melt token from foreign mint http://foreign:3338: boom"
|
|
||||||
),
|
|
||||||
422,
|
|
||||||
"mint_error",
|
|
||||||
"Failed to swap token from foreign mint",
|
|
||||||
"cashu_foreign_mint_swap_failed",
|
|
||||||
),
|
|
||||||
(
|
|
||||||
ValueError("could not decode token"),
|
|
||||||
400,
|
|
||||||
"invalid_token",
|
|
||||||
"Invalid Cashu token",
|
|
||||||
"invalid_cashu_token",
|
|
||||||
),
|
|
||||||
(
|
|
||||||
ValueError("some unexpected wallet condition"),
|
|
||||||
400,
|
|
||||||
"cashu_error",
|
|
||||||
"Failed to redeem Cashu token",
|
|
||||||
"cashu_token_redemption_failed",
|
|
||||||
),
|
|
||||||
(
|
|
||||||
ValueError("Redeemed token amount must be positive, got 0 msats"),
|
|
||||||
400,
|
|
||||||
"cashu_error",
|
|
||||||
"Failed to redeem Cashu token: token yielded no value",
|
|
||||||
"cashu_token_zero_value",
|
|
||||||
),
|
|
||||||
],
|
|
||||||
)
|
|
||||||
@pytest.mark.asyncio
|
|
||||||
async def test_redemption_failure_returns_sanitized_error(
|
|
||||||
session: AsyncSession,
|
|
||||||
error: Exception,
|
|
||||||
expected_status: int,
|
|
||||||
expected_type: str,
|
|
||||||
expected_message: str,
|
|
||||||
expected_code: str,
|
|
||||||
) -> None:
|
|
||||||
"""Redemption failures reuse the shared X-Cashu taxonomy (carried in
|
|
||||||
``type``), expose stable sanitized messages and granular machine-readable
|
|
||||||
``code`` values, and leave no orphan ApiKey row."""
|
|
||||||
token = "cashuAredemption_fails_with_specific_error"
|
|
||||||
hashed_key = hashlib.sha256(token.encode()).hexdigest()
|
|
||||||
token_obj = SimpleNamespace(mint="http://mint:3338", unit="sat")
|
|
||||||
|
|
||||||
from routstr.core.settings import settings
|
|
||||||
|
|
||||||
with (
|
|
||||||
patch.object(settings, "cashu_mints", ["http://mint:3338"]),
|
|
||||||
patch("routstr.auth.deserialize_token_from_string", return_value=token_obj),
|
|
||||||
patch(
|
|
||||||
"routstr.auth.credit_balance",
|
|
||||||
new=AsyncMock(side_effect=error),
|
|
||||||
),
|
|
||||||
):
|
|
||||||
with pytest.raises(HTTPException) as exc_info:
|
|
||||||
await validate_bearer_key(token, session)
|
|
||||||
|
|
||||||
assert exc_info.value.status_code == expected_status
|
|
||||||
detail = cast(dict[str, dict[str, object]], exc_info.value.detail)
|
|
||||||
error_detail = detail["error"]
|
|
||||||
assert error_detail["type"] == expected_type
|
|
||||||
assert error_detail["code"] == expected_code
|
|
||||||
assert error_detail["message"] == expected_message
|
|
||||||
assert str(error) not in cast(str, error_detail["message"])
|
|
||||||
assert await session.get(ApiKey, hashed_key) is None
|
|
||||||
|
|
||||||
|
|
||||||
@pytest.mark.asyncio
|
|
||||||
async def test_unexpected_redemption_error_returns_internal_error(
|
|
||||||
session: AsyncSession,
|
|
||||||
) -> None:
|
|
||||||
"""Unexpected (non-wallet) failures surface as generic 500s without
|
|
||||||
leaking internal details, instead of masquerading as token errors."""
|
|
||||||
token = "cashuAredemption_fails_with_internal_error"
|
|
||||||
hashed_key = hashlib.sha256(token.encode()).hexdigest()
|
|
||||||
token_obj = SimpleNamespace(mint="http://mint:3338", unit="sat")
|
|
||||||
|
|
||||||
from routstr.core.settings import settings
|
|
||||||
|
|
||||||
with (
|
|
||||||
patch.object(settings, "cashu_mints", ["http://mint:3338"]),
|
|
||||||
patch("routstr.auth.deserialize_token_from_string", return_value=token_obj),
|
|
||||||
patch(
|
|
||||||
"routstr.auth.credit_balance",
|
|
||||||
new=AsyncMock(side_effect=RuntimeError("db exploded at /var/lib/secret")),
|
|
||||||
),
|
|
||||||
):
|
|
||||||
with pytest.raises(HTTPException) as exc_info:
|
|
||||||
await validate_bearer_key(token, session)
|
|
||||||
|
|
||||||
assert exc_info.value.status_code == 500
|
|
||||||
detail = cast(dict[str, dict[str, str]], exc_info.value.detail)
|
|
||||||
error_detail = detail["error"]
|
|
||||||
assert error_detail["code"] == "internal_error"
|
|
||||||
assert "/var/lib/secret" not in error_detail["message"]
|
|
||||||
assert await session.get(ApiKey, hashed_key) is None
|
|
||||||
|
|
||||||
|
|
||||||
@pytest.mark.asyncio
|
|
||||||
async def test_internal_error_with_invalid_keyword_does_not_masquerade(
|
|
||||||
session: AsyncSession,
|
|
||||||
) -> None:
|
|
||||||
"""A non-wallet fault whose text merely contains "invalid" (but not
|
|
||||||
"token") must fall through to a generic 500, not a 401 token error.
|
|
||||||
|
|
||||||
Guards the anchored `"invalid"/"decode"` + `"token"` gate against stdlib/
|
|
||||||
driver strings like "Invalid isoformat string" leaking as token errors."""
|
|
||||||
token = "cashuAinternal_fault_mentions_invalid"
|
|
||||||
hashed_key = hashlib.sha256(token.encode()).hexdigest()
|
|
||||||
token_obj = SimpleNamespace(mint="http://mint:3338", unit="sat")
|
|
||||||
|
|
||||||
from routstr.core.settings import settings
|
|
||||||
|
|
||||||
with (
|
|
||||||
patch.object(settings, "cashu_mints", ["http://mint:3338"]),
|
|
||||||
patch("routstr.auth.deserialize_token_from_string", return_value=token_obj),
|
|
||||||
patch(
|
|
||||||
"routstr.auth.credit_balance",
|
|
||||||
new=AsyncMock(
|
|
||||||
side_effect=RuntimeError("Invalid isoformat string: '2020-13-99'")
|
|
||||||
),
|
|
||||||
),
|
|
||||||
):
|
|
||||||
with pytest.raises(HTTPException) as exc_info:
|
|
||||||
await validate_bearer_key(token, session)
|
|
||||||
|
|
||||||
assert exc_info.value.status_code == 500
|
|
||||||
detail = cast(dict[str, dict[str, str]], exc_info.value.detail)
|
|
||||||
assert detail["error"]["code"] == "internal_error"
|
|
||||||
assert await session.get(ApiKey, hashed_key) is None
|
|
||||||
|
|
||||||
|
|
||||||
@pytest.mark.asyncio
|
|
||||||
async def test_malformed_cashu_token_returns_400_invalid_token(
|
|
||||||
session: AsyncSession,
|
|
||||||
) -> None:
|
|
||||||
"""A malformed 'cashu...' token that fails to decode maps to 400
|
|
||||||
invalid_cashu_token (shared taxonomy), not the generic 401 invalid_api_key."""
|
|
||||||
token = "cashuAthis_is_not_a_valid_token"
|
|
||||||
|
|
||||||
with patch(
|
|
||||||
"routstr.auth.deserialize_token_from_string",
|
|
||||||
side_effect=ValueError("unable to decode token: bad base64"),
|
|
||||||
):
|
|
||||||
with pytest.raises(HTTPException) as exc_info:
|
|
||||||
await validate_bearer_key(token, session)
|
|
||||||
|
|
||||||
assert exc_info.value.status_code == 400
|
|
||||||
detail = cast(dict[str, dict[str, str]], exc_info.value.detail)
|
|
||||||
assert detail["error"]["type"] == "invalid_token"
|
|
||||||
assert detail["error"]["code"] == "invalid_cashu_token"
|
|
||||||
# Raw decoder text must not leak to the client.
|
|
||||||
assert "base64" not in detail["error"]["message"]
|
|
||||||
+3
-230
@@ -1,13 +1,12 @@
|
|||||||
import json
|
import json
|
||||||
from unittest.mock import AsyncMock, MagicMock, patch
|
from unittest.mock import AsyncMock, MagicMock, patch
|
||||||
|
|
||||||
import httpx
|
|
||||||
import pytest
|
import pytest
|
||||||
from fastapi.responses import JSONResponse
|
from fastapi.responses import JSONResponse
|
||||||
|
|
||||||
from routstr.balance import refund_wallet_endpoint, topup_wallet_endpoint
|
from routstr.balance import refund_wallet_endpoint
|
||||||
from routstr.core.db import ApiKey, CashuTransaction
|
from routstr.core.db import ApiKey, CashuTransaction
|
||||||
from routstr.wallet import MintConnectionError, credit_balance
|
from routstr.wallet import credit_balance
|
||||||
|
|
||||||
|
|
||||||
def _make_cashu_tx(
|
def _make_cashu_tx(
|
||||||
@@ -340,10 +339,7 @@ async def test_apikey_refund_restores_balance_on_mint_failure() -> None:
|
|||||||
|
|
||||||
with (
|
with (
|
||||||
patch("routstr.balance.get_billing_key", AsyncMock(return_value=key)),
|
patch("routstr.balance.get_billing_key", AsyncMock(return_value=key)),
|
||||||
patch(
|
patch("routstr.balance.send_token", AsyncMock(side_effect=Exception("mint down"))),
|
||||||
"routstr.balance.send_token",
|
|
||||||
AsyncMock(side_effect=MintConnectionError("raw mint outage detail")),
|
|
||||||
),
|
|
||||||
patch("routstr.balance.store_cashu_transaction", AsyncMock()),
|
patch("routstr.balance.store_cashu_transaction", AsyncMock()),
|
||||||
patch("routstr.balance._refund_cache_get", AsyncMock(return_value=None)),
|
patch("routstr.balance._refund_cache_get", AsyncMock(return_value=None)),
|
||||||
patch("routstr.balance._refund_cache_set", AsyncMock()),
|
patch("routstr.balance._refund_cache_set", AsyncMock()),
|
||||||
@@ -357,46 +353,10 @@ async def test_apikey_refund_restores_balance_on_mint_failure() -> None:
|
|||||||
)
|
)
|
||||||
|
|
||||||
assert exc_info.value.status_code == 503
|
assert exc_info.value.status_code == 503
|
||||||
assert exc_info.value.detail == "Mint service unavailable"
|
|
||||||
assert "raw mint outage detail" not in exc_info.value.detail
|
|
||||||
# Verify two exec calls: debit + restore
|
# Verify two exec calls: debit + restore
|
||||||
assert session.exec.await_count == 2
|
assert session.exec.await_count == 2
|
||||||
|
|
||||||
|
|
||||||
@pytest.mark.asyncio
|
|
||||||
async def test_apikey_refund_generic_failure_is_sanitized_500() -> None:
|
|
||||||
"""Unexpected send-side failures restore balance without leaking exception text."""
|
|
||||||
from fastapi import HTTPException
|
|
||||||
|
|
||||||
key = _make_api_key(balance=5000, refund_currency="sat")
|
|
||||||
raw_error = "database secret token raw-mint-response"
|
|
||||||
|
|
||||||
session = MagicMock()
|
|
||||||
session.get = AsyncMock(return_value=key)
|
|
||||||
session.exec = AsyncMock(side_effect=[_update_result(1), _update_result(1)])
|
|
||||||
session.commit = AsyncMock()
|
|
||||||
|
|
||||||
with (
|
|
||||||
patch("routstr.balance.get_billing_key", AsyncMock(return_value=key)),
|
|
||||||
patch("routstr.balance.send_token", AsyncMock(side_effect=RuntimeError(raw_error))),
|
|
||||||
patch("routstr.balance.store_cashu_transaction", AsyncMock()),
|
|
||||||
patch("routstr.balance._refund_cache_get", AsyncMock(return_value=None)),
|
|
||||||
patch("routstr.balance._refund_cache_set", AsyncMock()),
|
|
||||||
patch("routstr.balance.logger"),
|
|
||||||
):
|
|
||||||
with pytest.raises(HTTPException) as exc_info:
|
|
||||||
await refund_wallet_endpoint(
|
|
||||||
authorization="Bearer sk-testhash",
|
|
||||||
x_cashu=None,
|
|
||||||
session=session,
|
|
||||||
)
|
|
||||||
|
|
||||||
assert exc_info.value.status_code == 500
|
|
||||||
assert exc_info.value.detail == "Refund failed"
|
|
||||||
assert raw_error not in exc_info.value.detail
|
|
||||||
assert session.exec.await_count == 2
|
|
||||||
|
|
||||||
|
|
||||||
# ---------------------------------------------------------------------------
|
# ---------------------------------------------------------------------------
|
||||||
# no-create guarantee: fresh Cashu/unknown sk- tokens must not create API keys
|
# no-create guarantee: fresh Cashu/unknown sk- tokens must not create API keys
|
||||||
# ---------------------------------------------------------------------------
|
# ---------------------------------------------------------------------------
|
||||||
@@ -441,190 +401,3 @@ async def test_refund_unknown_sk_bearer_returns_401() -> None:
|
|||||||
|
|
||||||
assert exc_info.value.status_code == 401
|
assert exc_info.value.status_code == 401
|
||||||
session.get.assert_awaited_once()
|
session.get.assert_awaited_once()
|
||||||
|
|
||||||
|
|
||||||
# --- Topup redemption error taxonomy (POST /v1/wallet/topup) ------------------
|
|
||||||
|
|
||||||
|
|
||||||
@pytest.mark.asyncio
|
|
||||||
@pytest.mark.parametrize(
|
|
||||||
"error",
|
|
||||||
[
|
|
||||||
httpx.ConnectError("All connection attempts failed"),
|
|
||||||
MintConnectionError("connect to mint refused"),
|
|
||||||
TimeoutError("timed out connecting to mint"),
|
|
||||||
],
|
|
||||||
)
|
|
||||||
async def test_topup_mint_unreachable_returns_503(error: Exception) -> None:
|
|
||||||
"""A down mint must surface 503 (retryable), not 400 or 500 — the token is
|
|
||||||
fine, so the client should retry once the mint recovers."""
|
|
||||||
from fastapi import HTTPException
|
|
||||||
|
|
||||||
key = _make_api_key(balance=1000)
|
|
||||||
session = MagicMock()
|
|
||||||
|
|
||||||
with (
|
|
||||||
patch("routstr.balance.get_billing_key", AsyncMock(return_value=key)),
|
|
||||||
patch("routstr.balance.credit_balance", AsyncMock(side_effect=error)),
|
|
||||||
):
|
|
||||||
with pytest.raises(HTTPException) as exc_info:
|
|
||||||
await topup_wallet_endpoint(
|
|
||||||
cashu_token="cashuAtoken", key=key, session=session
|
|
||||||
)
|
|
||||||
|
|
||||||
assert exc_info.value.status_code == 503
|
|
||||||
assert exc_info.value.detail == "Cashu mint is unreachable"
|
|
||||||
|
|
||||||
|
|
||||||
@pytest.mark.asyncio
|
|
||||||
async def test_topup_already_spent_still_returns_400() -> None:
|
|
||||||
"""Regression: the mint-unreachable short-circuit must not swallow the
|
|
||||||
existing ValueError substring buckets."""
|
|
||||||
from fastapi import HTTPException
|
|
||||||
|
|
||||||
key = _make_api_key(balance=1000)
|
|
||||||
session = MagicMock()
|
|
||||||
|
|
||||||
with (
|
|
||||||
patch("routstr.balance.get_billing_key", AsyncMock(return_value=key)),
|
|
||||||
patch(
|
|
||||||
"routstr.balance.credit_balance",
|
|
||||||
AsyncMock(side_effect=ValueError("Token already spent")),
|
|
||||||
),
|
|
||||||
):
|
|
||||||
with pytest.raises(HTTPException) as exc_info:
|
|
||||||
await topup_wallet_endpoint(
|
|
||||||
cashu_token="cashuAtoken", key=key, session=session
|
|
||||||
)
|
|
||||||
|
|
||||||
assert exc_info.value.status_code == 400
|
|
||||||
assert exc_info.value.detail == "Cashu token already spent"
|
|
||||||
|
|
||||||
|
|
||||||
@pytest.mark.asyncio
|
|
||||||
async def test_topup_zero_value_returns_400_zero_value_message() -> None:
|
|
||||||
"""A dust/zero redemption maps to the documented zero-value message, not the
|
|
||||||
generic redemption-failed one."""
|
|
||||||
from fastapi import HTTPException
|
|
||||||
|
|
||||||
key = _make_api_key(balance=1000)
|
|
||||||
session = MagicMock()
|
|
||||||
|
|
||||||
with (
|
|
||||||
patch("routstr.balance.get_billing_key", AsyncMock(return_value=key)),
|
|
||||||
patch(
|
|
||||||
"routstr.balance.credit_balance",
|
|
||||||
AsyncMock(
|
|
||||||
side_effect=ValueError("Redeemed token amount must be positive, got 0 msats")
|
|
||||||
),
|
|
||||||
),
|
|
||||||
):
|
|
||||||
with pytest.raises(HTTPException) as exc_info:
|
|
||||||
await topup_wallet_endpoint(
|
|
||||||
cashu_token="cashuAtoken", key=key, session=session
|
|
||||||
)
|
|
||||||
|
|
||||||
assert exc_info.value.status_code == 400
|
|
||||||
assert exc_info.value.detail == "Failed to redeem Cashu token: token yielded no value"
|
|
||||||
|
|
||||||
|
|
||||||
@pytest.mark.asyncio
|
|
||||||
async def test_topup_token_consumed_returns_500() -> None:
|
|
||||||
"""A post-redemption crediting failure (token spent) is a non-retryable 500,
|
|
||||||
not a 4xx that invites a retry."""
|
|
||||||
from fastapi import HTTPException
|
|
||||||
|
|
||||||
from routstr.wallet import TokenConsumedError
|
|
||||||
|
|
||||||
key = _make_api_key(balance=1000)
|
|
||||||
session = MagicMock()
|
|
||||||
|
|
||||||
with (
|
|
||||||
patch("routstr.balance.get_billing_key", AsyncMock(return_value=key)),
|
|
||||||
patch(
|
|
||||||
"routstr.balance.credit_balance",
|
|
||||||
AsyncMock(side_effect=TokenConsumedError("credit failed")),
|
|
||||||
),
|
|
||||||
):
|
|
||||||
with pytest.raises(HTTPException) as exc_info:
|
|
||||||
await topup_wallet_endpoint(
|
|
||||||
cashu_token="cashuAtoken", key=key, session=session
|
|
||||||
)
|
|
||||||
|
|
||||||
assert exc_info.value.status_code == 500
|
|
||||||
assert exc_info.value.detail == (
|
|
||||||
"Token was redeemed but could not be credited; do not retry"
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
@pytest.mark.asyncio
|
|
||||||
@pytest.mark.parametrize(
|
|
||||||
("error", "expected_status", "expected_detail"),
|
|
||||||
[
|
|
||||||
(
|
|
||||||
ValueError(
|
|
||||||
"Failed to estimate fees: Fees (7 sat) exceed token amount (5 sat)"
|
|
||||||
),
|
|
||||||
422,
|
|
||||||
"Token value is too small to cover swap fees",
|
|
||||||
),
|
|
||||||
(
|
|
||||||
ValueError(
|
|
||||||
"Token amount (5 sat) is insufficient to cover melt fees."
|
|
||||||
),
|
|
||||||
422,
|
|
||||||
"Token value is too small to cover swap fees",
|
|
||||||
),
|
|
||||||
(
|
|
||||||
ValueError("Failed to melt token from foreign mint http://m: boom"),
|
|
||||||
422,
|
|
||||||
"Failed to swap token from foreign mint",
|
|
||||||
),
|
|
||||||
],
|
|
||||||
)
|
|
||||||
async def test_topup_fee_and_swap_failures_return_422(
|
|
||||||
error: Exception, expected_status: int, expected_detail: str
|
|
||||||
) -> None:
|
|
||||||
"""Fee/swap failures map to 422 (shared taxonomy), matching the bearer and
|
|
||||||
X-Cashu paths — previously top-up flattened these to 400."""
|
|
||||||
from fastapi import HTTPException
|
|
||||||
|
|
||||||
key = _make_api_key(balance=1000)
|
|
||||||
session = MagicMock()
|
|
||||||
|
|
||||||
with (
|
|
||||||
patch("routstr.balance.get_billing_key", AsyncMock(return_value=key)),
|
|
||||||
patch("routstr.balance.credit_balance", AsyncMock(side_effect=error)),
|
|
||||||
):
|
|
||||||
with pytest.raises(HTTPException) as exc_info:
|
|
||||||
await topup_wallet_endpoint(
|
|
||||||
cashu_token="cashuAtoken", key=key, session=session
|
|
||||||
)
|
|
||||||
|
|
||||||
assert exc_info.value.status_code == expected_status
|
|
||||||
assert exc_info.value.detail == expected_detail
|
|
||||||
|
|
||||||
|
|
||||||
@pytest.mark.asyncio
|
|
||||||
async def test_topup_unexpected_non_valueerror_returns_500() -> None:
|
|
||||||
"""A non-ValueError, non-transport fault is an internal error (500), not a
|
|
||||||
sanitized 400 — the merged except must preserve this."""
|
|
||||||
from fastapi import HTTPException
|
|
||||||
|
|
||||||
key = _make_api_key(balance=1000)
|
|
||||||
session = MagicMock()
|
|
||||||
|
|
||||||
with (
|
|
||||||
patch("routstr.balance.get_billing_key", AsyncMock(return_value=key)),
|
|
||||||
patch(
|
|
||||||
"routstr.balance.credit_balance",
|
|
||||||
AsyncMock(side_effect=RuntimeError("db exploded")),
|
|
||||||
),
|
|
||||||
):
|
|
||||||
with pytest.raises(HTTPException) as exc_info:
|
|
||||||
await topup_wallet_endpoint(
|
|
||||||
cashu_token="cashuAtoken", key=key, session=session
|
|
||||||
)
|
|
||||||
|
|
||||||
assert exc_info.value.status_code == 500
|
|
||||||
assert exc_info.value.detail == "Internal server error"
|
|
||||||
|
|||||||
@@ -81,21 +81,6 @@ def test_backfill_strips_vendor_prefix_for_litellm_lookup() -> None:
|
|||||||
assert result.input_cache_read == expected
|
assert result.input_cache_read == expected
|
||||||
|
|
||||||
|
|
||||||
def test_backfill_case_insensitive_lookup() -> None:
|
|
||||||
"""A generic upstream may report a mixed-case id
|
|
||||||
(deepseek-ai/DeepSeek-V4-Flash); litellm keys are lowercase. The
|
|
||||||
case-insensitive fallback still resolves the cache rate."""
|
|
||||||
pricing = Pricing(prompt=1.4e-07, completion=2.8e-07)
|
|
||||||
|
|
||||||
result = backfill_cache_pricing("deepseek-ai/DeepSeek-V4-Flash", pricing)
|
|
||||||
|
|
||||||
expected = litellm.model_cost["deepseek-v4-flash"][
|
|
||||||
"cache_read_input_token_cost"
|
|
||||||
]
|
|
||||||
assert result.input_cache_read == expected
|
|
||||||
assert result.input_cache_read < pricing.prompt # sanity: it's a discount
|
|
||||||
|
|
||||||
|
|
||||||
def test_backfill_fills_cache_write_rate() -> None:
|
def test_backfill_fills_cache_write_rate() -> None:
|
||||||
"""Anthropic cache writes cost more than input (1.25x); billing them at
|
"""Anthropic cache writes cost more than input (1.25x); billing them at
|
||||||
the input rate undercharges. litellm carries the write rate."""
|
the input rate undercharges. litellm carries the write rate."""
|
||||||
@@ -151,93 +136,6 @@ def test_provider_fee_applies_to_backfilled_cache_rates() -> None:
|
|||||||
assert adjusted.pricing.prompt == pytest.approx(2.8e-07 * 2.0)
|
assert adjusted.pricing.prompt == pytest.approx(2.8e-07 * 2.0)
|
||||||
|
|
||||||
|
|
||||||
def test_row_to_model_backfills_cache_rate() -> None:
|
|
||||||
"""The DB-override path (admin-configured providers, e.g. a generic
|
|
||||||
upstream) stores pricing without cache rates. ``_row_to_model`` must
|
|
||||||
backfill them from litellm just like ``_apply_provider_fee_to_model``,
|
|
||||||
otherwise cache reads bill at the full input rate."""
|
|
||||||
import json
|
|
||||||
|
|
||||||
from routstr.core.db import ModelRow
|
|
||||||
from routstr.payment.models import _row_to_model
|
|
||||||
|
|
||||||
row = ModelRow(
|
|
||||||
id="deepseek-v4-flash",
|
|
||||||
name="deepseek-v4-flash",
|
|
||||||
created=0,
|
|
||||||
description="",
|
|
||||||
context_length=1000000,
|
|
||||||
architecture=json.dumps(
|
|
||||||
{
|
|
||||||
"modality": "text",
|
|
||||||
"input_modalities": ["text"],
|
|
||||||
"output_modalities": ["text"],
|
|
||||||
"tokenizer": "unknown",
|
|
||||||
"instruct_type": None,
|
|
||||||
}
|
|
||||||
),
|
|
||||||
# Stored pricing omits input_cache_read (generic provider never sets it).
|
|
||||||
pricing=json.dumps({"prompt": 1.4e-07, "completion": 2.8e-07}),
|
|
||||||
enabled=True,
|
|
||||||
upstream_provider_id=1,
|
|
||||||
)
|
|
||||||
|
|
||||||
with patch(
|
|
||||||
"routstr.payment.models.sats_usd_price", return_value=5.0e-5
|
|
||||||
):
|
|
||||||
model = _row_to_model(row, apply_provider_fee=True, provider_fee=1.0)
|
|
||||||
|
|
||||||
litellm_read = litellm.model_cost["deepseek-v4-flash"][
|
|
||||||
"cache_read_input_token_cost"
|
|
||||||
]
|
|
||||||
assert model.pricing.input_cache_read == pytest.approx(litellm_read)
|
|
||||||
assert model.pricing.input_cache_read < model.pricing.prompt # a discount
|
|
||||||
assert model.sats_pricing is not None
|
|
||||||
assert model.sats_pricing.input_cache_read > 0
|
|
||||||
|
|
||||||
|
|
||||||
def test_row_to_model_backfills_via_forwarded_model_id() -> None:
|
|
||||||
"""An alias row (id != forwarded_model_id) must backfill cache rates from
|
|
||||||
the *forwarded* model name — the real upstream model litellm prices —
|
|
||||||
not the alias id, which litellm doesn't know."""
|
|
||||||
import json
|
|
||||||
|
|
||||||
from routstr.core.db import ModelRow
|
|
||||||
from routstr.payment.models import _row_to_model
|
|
||||||
|
|
||||||
row = ModelRow(
|
|
||||||
id="local-alias", # litellm has no such key
|
|
||||||
name="local-alias",
|
|
||||||
created=0,
|
|
||||||
description="",
|
|
||||||
context_length=1000000,
|
|
||||||
architecture=json.dumps(
|
|
||||||
{
|
|
||||||
"modality": "text",
|
|
||||||
"input_modalities": ["text"],
|
|
||||||
"output_modalities": ["text"],
|
|
||||||
"tokenizer": "unknown",
|
|
||||||
"instruct_type": None,
|
|
||||||
}
|
|
||||||
),
|
|
||||||
pricing=json.dumps({"prompt": 1.4e-07, "completion": 2.8e-07}),
|
|
||||||
enabled=True,
|
|
||||||
upstream_provider_id=1,
|
|
||||||
forwarded_model_id="deepseek-v4-flash",
|
|
||||||
)
|
|
||||||
|
|
||||||
with patch(
|
|
||||||
"routstr.payment.models.sats_usd_price", return_value=5.0e-5
|
|
||||||
):
|
|
||||||
model = _row_to_model(row, apply_provider_fee=True, provider_fee=1.0)
|
|
||||||
|
|
||||||
litellm_read = litellm.model_cost["deepseek-v4-flash"][
|
|
||||||
"cache_read_input_token_cost"
|
|
||||||
]
|
|
||||||
assert model.pricing.input_cache_read == pytest.approx(litellm_read)
|
|
||||||
assert model.pricing.input_cache_read < model.pricing.prompt
|
|
||||||
|
|
||||||
|
|
||||||
# ============================================================================
|
# ============================================================================
|
||||||
# calculate_cost — cached tokens billed at cache rates
|
# calculate_cost — cached tokens billed at cache rates
|
||||||
# ============================================================================
|
# ============================================================================
|
||||||
@@ -284,13 +182,11 @@ async def test_deepseek_cache_hits_billed_at_cache_rate(model_pricing: Mock) ->
|
|||||||
|
|
||||||
assert isinstance(result, CostData)
|
assert isinstance(result, CostData)
|
||||||
# 1000 input @ 1 msat + 9000 cache reads @ 0.1 msat + 500 output @ 2 msat.
|
# 1000 input @ 1 msat + 9000 cache reads @ 0.1 msat + 500 output @ 2 msat.
|
||||||
# input_msats folds the cache-read cost in (1000 + 900) so a dashboard
|
# Client-visible input_msats includes cache-read input cost for display,
|
||||||
# rendering I/O/T sees input + output == total; the cache portion stays
|
# while cache_read_msats keeps the detailed breakdown.
|
||||||
# visible in cache_read_msats.
|
assert result.input_msats == 1900
|
||||||
assert result.cache_read_msats == 900
|
assert result.cache_read_msats == 900
|
||||||
assert result.output_msats == 1000
|
assert result.output_msats == 1000
|
||||||
assert result.input_msats == 1900
|
|
||||||
assert result.input_msats + result.output_msats == result.total_msats
|
|
||||||
assert result.total_msats == 2900
|
assert result.total_msats == 2900
|
||||||
|
|
||||||
|
|
||||||
|
|||||||
@@ -1,73 +0,0 @@
|
|||||||
from contextlib import asynccontextmanager
|
|
||||||
from unittest.mock import AsyncMock, MagicMock, patch
|
|
||||||
|
|
||||||
import pytest
|
|
||||||
|
|
||||||
from routstr.wallet import fetch_all_balances
|
|
||||||
|
|
||||||
|
|
||||||
@asynccontextmanager
|
|
||||||
async def _fake_session(): # type: ignore[no-untyped-def]
|
|
||||||
yield MagicMock()
|
|
||||||
|
|
||||||
|
|
||||||
def _patches(proof_amount: int = 1000): # type: ignore[no-untyped-def]
|
|
||||||
proof = MagicMock(amount=proof_amount)
|
|
||||||
return [
|
|
||||||
patch("routstr.wallet.get_wallet", AsyncMock(return_value=MagicMock())),
|
|
||||||
patch(
|
|
||||||
"routstr.wallet.get_proofs_per_mint_and_unit",
|
|
||||||
MagicMock(return_value=[proof]),
|
|
||||||
),
|
|
||||||
patch(
|
|
||||||
"routstr.wallet.slow_filter_spend_proofs",
|
|
||||||
AsyncMock(side_effect=lambda proofs, wallet: proofs),
|
|
||||||
),
|
|
||||||
patch(
|
|
||||||
"routstr.wallet.db.balances_for_mint_and_unit",
|
|
||||||
AsyncMock(return_value=0),
|
|
||||||
),
|
|
||||||
patch("routstr.wallet.db.create_session", _fake_session),
|
|
||||||
]
|
|
||||||
|
|
||||||
|
|
||||||
@pytest.mark.asyncio
|
|
||||||
async def test_fetch_all_balances_falls_back_to_primary_mint() -> None:
|
|
||||||
"""With empty cashu_mints, balances are still fetched for primary_mint."""
|
|
||||||
from routstr.core.settings import settings
|
|
||||||
|
|
||||||
with patch.object(settings, "cashu_mints", []), patch.object(
|
|
||||||
settings, "primary_mint", "http://primary:3338"
|
|
||||||
):
|
|
||||||
for p in _patches(proof_amount=1000):
|
|
||||||
p.start()
|
|
||||||
try:
|
|
||||||
details, total_wallet, total_user, owner = await fetch_all_balances(
|
|
||||||
units=["sat"]
|
|
||||||
)
|
|
||||||
finally:
|
|
||||||
patch.stopall()
|
|
||||||
|
|
||||||
assert [d["mint_url"] for d in details] == ["http://primary:3338"]
|
|
||||||
assert total_wallet == 1000
|
|
||||||
|
|
||||||
|
|
||||||
@pytest.mark.asyncio
|
|
||||||
async def test_fetch_all_balances_no_duplicate_primary_mint() -> None:
|
|
||||||
"""primary_mint already in cashu_mints is not inspected twice."""
|
|
||||||
from routstr.core.settings import settings
|
|
||||||
|
|
||||||
with patch.object(
|
|
||||||
settings, "cashu_mints", ["http://primary:3338"]
|
|
||||||
), patch.object(settings, "primary_mint", "http://primary:3338"):
|
|
||||||
for p in _patches(proof_amount=1000):
|
|
||||||
p.start()
|
|
||||||
try:
|
|
||||||
details, total_wallet, _total_user, _owner = await fetch_all_balances(
|
|
||||||
units=["sat"]
|
|
||||||
)
|
|
||||||
finally:
|
|
||||||
patch.stopall()
|
|
||||||
|
|
||||||
assert [d["mint_url"] for d in details] == ["http://primary:3338"]
|
|
||||||
assert total_wallet == 1000
|
|
||||||
@@ -11,7 +11,6 @@ import os
|
|||||||
from typing import Any, AsyncIterator
|
from typing import Any, AsyncIterator
|
||||||
from unittest.mock import AsyncMock, MagicMock, patch
|
from unittest.mock import AsyncMock, MagicMock, patch
|
||||||
|
|
||||||
import httpx
|
|
||||||
import pytest
|
import pytest
|
||||||
from fastapi.responses import Response, StreamingResponse
|
from fastapi.responses import Response, StreamingResponse
|
||||||
|
|
||||||
@@ -22,7 +21,6 @@ from routstr.core.db import ApiKey # noqa: E402
|
|||||||
from routstr.payment.cost_calculation import CostData # noqa: E402
|
from routstr.payment.cost_calculation import CostData # noqa: E402
|
||||||
from routstr.payment.models import Architecture, Model, Pricing # noqa: E402
|
from routstr.payment.models import Architecture, Model, Pricing # noqa: E402
|
||||||
from routstr.upstream.base import BaseUpstreamProvider # noqa: E402
|
from routstr.upstream.base import BaseUpstreamProvider # noqa: E402
|
||||||
from routstr.wallet import MintConnectionError, TokenConsumedError # noqa: E402
|
|
||||||
|
|
||||||
# ---------------------------------------------------------------------------
|
# ---------------------------------------------------------------------------
|
||||||
# Fixtures
|
# Fixtures
|
||||||
@@ -1310,320 +1308,3 @@ async def test_dispatch_uses_url_detected_prefix_for_fireworks_custom_row() -> N
|
|||||||
"fireworks_ai/accounts/fireworks/models/glm-5"
|
"fireworks_ai/accounts/fireworks/models/glm-5"
|
||||||
)
|
)
|
||||||
assert captured_kwargs["api_base"] == "https://api.fireworks.ai/inference/v1"
|
assert captured_kwargs["api_base"] == "https://api.fireworks.ai/inference/v1"
|
||||||
|
|
||||||
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
# X-Cashu redemption error taxonomy (unreachable mint + string codes)
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
|
|
||||||
|
|
||||||
@pytest.mark.asyncio
|
|
||||||
@pytest.mark.parametrize(
|
|
||||||
"handler_name",
|
|
||||||
["handle_x_cashu", "handle_x_cashu_responses"],
|
|
||||||
)
|
|
||||||
@pytest.mark.parametrize(
|
|
||||||
"error",
|
|
||||||
[
|
|
||||||
httpx.ConnectError("All connection attempts failed"),
|
|
||||||
MintConnectionError("Cashu mint is unreachable"),
|
|
||||||
TimeoutError("timed out connecting to mint"),
|
|
||||||
],
|
|
||||||
)
|
|
||||||
async def test_x_cashu_mint_unreachable_returns_503(
|
|
||||||
handler_name: str, error: Exception
|
|
||||||
) -> None:
|
|
||||||
"""Both X-Cashu entrypoints classify a down mint as 503 mint_unreachable,
|
|
||||||
not a generic 400 cashu_error."""
|
|
||||||
provider = _make_provider()
|
|
||||||
model = _make_model()
|
|
||||||
request = _make_request()
|
|
||||||
|
|
||||||
with patch(
|
|
||||||
"routstr.upstream.base.recieve_token", new=AsyncMock(side_effect=error)
|
|
||||||
):
|
|
||||||
handler = getattr(provider, handler_name)
|
|
||||||
response = await handler(
|
|
||||||
request=request,
|
|
||||||
x_cashu_token="cashuAtoken",
|
|
||||||
path="v1/chat/completions",
|
|
||||||
max_cost_for_model=10_000,
|
|
||||||
model_obj=model,
|
|
||||||
)
|
|
||||||
|
|
||||||
assert response.status_code == 503
|
|
||||||
body = json.loads(bytes(response.body))
|
|
||||||
assert body["error"]["type"] == "mint_unreachable"
|
|
||||||
assert body["error"]["message"] == "Cashu mint is unreachable"
|
|
||||||
assert body["error"]["code"] == "cashu_mint_unreachable"
|
|
||||||
if str(error) != body["error"]["message"]:
|
|
||||||
assert str(error) not in body["error"]["message"]
|
|
||||||
|
|
||||||
|
|
||||||
@pytest.mark.asyncio
|
|
||||||
@pytest.mark.parametrize(
|
|
||||||
"handler_name",
|
|
||||||
["handle_x_cashu", "handle_x_cashu_responses"],
|
|
||||||
)
|
|
||||||
@pytest.mark.parametrize(
|
|
||||||
(
|
|
||||||
"error",
|
|
||||||
"expected_status",
|
|
||||||
"expected_type",
|
|
||||||
"expected_message",
|
|
||||||
"expected_code",
|
|
||||||
),
|
|
||||||
[
|
|
||||||
(
|
|
||||||
ValueError("Mint Error: Token already spent"),
|
|
||||||
400,
|
|
||||||
"token_already_spent",
|
|
||||||
"Cashu token already spent",
|
|
||||||
"cashu_token_already_spent",
|
|
||||||
),
|
|
||||||
(
|
|
||||||
ValueError("invalid token: could not decode"),
|
|
||||||
400,
|
|
||||||
"invalid_token",
|
|
||||||
"Invalid Cashu token",
|
|
||||||
"invalid_cashu_token",
|
|
||||||
),
|
|
||||||
(
|
|
||||||
# Fee/swap failures now map to a granular 422 on the X-Cashu path,
|
|
||||||
# matching the bearer path (previously flattened to 400).
|
|
||||||
ValueError(
|
|
||||||
"Failed to estimate fees: Fees (7 sat) exceed token amount (5 sat)"
|
|
||||||
),
|
|
||||||
422,
|
|
||||||
"mint_error",
|
|
||||||
"Token value is too small to cover swap fees",
|
|
||||||
"cashu_token_swap_fees_exceed_amount",
|
|
||||||
),
|
|
||||||
(
|
|
||||||
ValueError("Failed to melt token from foreign mint http://m: boom"),
|
|
||||||
422,
|
|
||||||
"mint_error",
|
|
||||||
"Failed to swap token from foreign mint",
|
|
||||||
"cashu_foreign_mint_swap_failed",
|
|
||||||
),
|
|
||||||
(
|
|
||||||
ValueError("some unexpected wallet condition"),
|
|
||||||
400,
|
|
||||||
"cashu_error",
|
|
||||||
"Failed to redeem Cashu token",
|
|
||||||
"cashu_token_redemption_failed",
|
|
||||||
),
|
|
||||||
(
|
|
||||||
# Non-ValueError faults are internal errors (500), not token errors.
|
|
||||||
RuntimeError("db exploded"),
|
|
||||||
500,
|
|
||||||
"api_error",
|
|
||||||
"Internal error during token redemption",
|
|
||||||
"internal_error",
|
|
||||||
),
|
|
||||||
],
|
|
||||||
)
|
|
||||||
async def test_x_cashu_error_code_is_stable_string(
|
|
||||||
handler_name: str,
|
|
||||||
error: Exception,
|
|
||||||
expected_status: int,
|
|
||||||
expected_type: str,
|
|
||||||
expected_message: str,
|
|
||||||
expected_code: str,
|
|
||||||
) -> None:
|
|
||||||
"""X-Cashu emits a stable string ``code`` on every branch, matching the
|
|
||||||
bearer path's taxonomy instead of an int HTTP status."""
|
|
||||||
provider = _make_provider()
|
|
||||||
model = _make_model()
|
|
||||||
request = _make_request()
|
|
||||||
|
|
||||||
with patch(
|
|
||||||
"routstr.upstream.base.recieve_token", new=AsyncMock(side_effect=error)
|
|
||||||
):
|
|
||||||
handler = getattr(provider, handler_name)
|
|
||||||
response = await handler(
|
|
||||||
request=request,
|
|
||||||
x_cashu_token="cashuAtoken",
|
|
||||||
path="v1/chat/completions",
|
|
||||||
max_cost_for_model=10_000,
|
|
||||||
model_obj=model,
|
|
||||||
)
|
|
||||||
|
|
||||||
assert response.status_code == expected_status
|
|
||||||
body = json.loads(bytes(response.body))
|
|
||||||
assert body["error"]["type"] == expected_type
|
|
||||||
assert body["error"]["message"] == expected_message
|
|
||||||
assert body["error"]["code"] == expected_code
|
|
||||||
assert isinstance(body["error"]["code"], str)
|
|
||||||
if str(error) != expected_message:
|
|
||||||
assert str(error) not in body["error"]["message"]
|
|
||||||
|
|
||||||
|
|
||||||
@pytest.mark.asyncio
|
|
||||||
@pytest.mark.parametrize(
|
|
||||||
("handler_name", "forward_attr"),
|
|
||||||
[
|
|
||||||
("handle_x_cashu", "forward_x_cashu_request"),
|
|
||||||
("handle_x_cashu_responses", "forward_x_cashu_responses_request"),
|
|
||||||
],
|
|
||||||
)
|
|
||||||
async def test_x_cashu_transport_error_after_redemption_is_not_retryable(
|
|
||||||
handler_name: str, forward_attr: str
|
|
||||||
) -> None:
|
|
||||||
"""A transport failure while forwarding (after the token is spent) maps to
|
|
||||||
502 upstream_error, never a retryable cashu_mint_unreachable."""
|
|
||||||
provider = _make_provider()
|
|
||||||
model = _make_model()
|
|
||||||
request = _make_request()
|
|
||||||
|
|
||||||
with (
|
|
||||||
patch(
|
|
||||||
"routstr.upstream.base.recieve_token",
|
|
||||||
new=AsyncMock(return_value=(5_000, "sat", "https://mint")),
|
|
||||||
),
|
|
||||||
patch("routstr.upstream.base.store_cashu_transaction", new=AsyncMock()),
|
|
||||||
patch.object(
|
|
||||||
provider,
|
|
||||||
forward_attr,
|
|
||||||
new=AsyncMock(side_effect=httpx.ConnectError("upstream down")),
|
|
||||||
),
|
|
||||||
):
|
|
||||||
handler = getattr(provider, handler_name)
|
|
||||||
response = await handler(
|
|
||||||
request=request,
|
|
||||||
x_cashu_token="cashuAtoken",
|
|
||||||
path="v1/chat/completions",
|
|
||||||
max_cost_for_model=10_000,
|
|
||||||
model_obj=model,
|
|
||||||
)
|
|
||||||
|
|
||||||
assert response.status_code == 502
|
|
||||||
body = json.loads(bytes(response.body))
|
|
||||||
assert body["error"]["type"] == "upstream_error"
|
|
||||||
assert body["error"]["code"] != "cashu_mint_unreachable"
|
|
||||||
|
|
||||||
|
|
||||||
@pytest.mark.asyncio
|
|
||||||
@pytest.mark.parametrize(
|
|
||||||
"handler_name",
|
|
||||||
["handle_x_cashu", "handle_x_cashu_responses"],
|
|
||||||
)
|
|
||||||
async def test_x_cashu_token_consumed_returns_500_and_no_echo(
|
|
||||||
handler_name: str,
|
|
||||||
) -> None:
|
|
||||||
"""A post-redemption failure (token spent, crediting/minting failed) is a
|
|
||||||
non-retryable 500 token_consumed and must NOT echo the spent token back."""
|
|
||||||
provider = _make_provider()
|
|
||||||
model = _make_model()
|
|
||||||
request = _make_request()
|
|
||||||
|
|
||||||
with patch(
|
|
||||||
"routstr.upstream.base.recieve_token",
|
|
||||||
new=AsyncMock(side_effect=TokenConsumedError("credit failed")),
|
|
||||||
):
|
|
||||||
handler = getattr(provider, handler_name)
|
|
||||||
response = await handler(
|
|
||||||
request=request,
|
|
||||||
x_cashu_token="cashuAtoken",
|
|
||||||
path="v1/chat/completions",
|
|
||||||
max_cost_for_model=10_000,
|
|
||||||
model_obj=model,
|
|
||||||
)
|
|
||||||
|
|
||||||
assert response.status_code == 500
|
|
||||||
body = json.loads(bytes(response.body))
|
|
||||||
assert body["error"]["type"] == "token_consumed"
|
|
||||||
assert body["error"]["code"] == "cashu_token_consumed"
|
|
||||||
assert "X-Cashu" not in response.headers
|
|
||||||
|
|
||||||
|
|
||||||
@pytest.mark.asyncio
|
|
||||||
@pytest.mark.parametrize(
|
|
||||||
("handler_name", "error", "echoed"),
|
|
||||||
[
|
|
||||||
# Spent token: must NOT be echoed.
|
|
||||||
("handle_x_cashu", ValueError("Token already spent"), False),
|
|
||||||
("handle_x_cashu_responses", ValueError("Token already spent"), False),
|
|
||||||
# Unspent but unreachable mint: echo so the client can retry the token.
|
|
||||||
("handle_x_cashu", MintConnectionError("mint down"), True),
|
|
||||||
("handle_x_cashu_responses", MintConnectionError("mint down"), True),
|
|
||||||
# Consumed token (post-redemption): must NOT be echoed.
|
|
||||||
("handle_x_cashu", TokenConsumedError("credit failed"), False),
|
|
||||||
("handle_x_cashu_responses", TokenConsumedError("credit failed"), False),
|
|
||||||
],
|
|
||||||
)
|
|
||||||
async def test_x_cashu_echoes_token_only_when_recoverable(
|
|
||||||
handler_name: str, error: Exception, echoed: bool
|
|
||||||
) -> None:
|
|
||||||
provider = _make_provider()
|
|
||||||
model = _make_model()
|
|
||||||
request = _make_request()
|
|
||||||
|
|
||||||
with patch(
|
|
||||||
"routstr.upstream.base.recieve_token", new=AsyncMock(side_effect=error)
|
|
||||||
):
|
|
||||||
handler = getattr(provider, handler_name)
|
|
||||||
response = await handler(
|
|
||||||
request=request,
|
|
||||||
x_cashu_token="cashuAtoken",
|
|
||||||
path="v1/chat/completions",
|
|
||||||
max_cost_for_model=10_000,
|
|
||||||
model_obj=model,
|
|
||||||
)
|
|
||||||
|
|
||||||
if echoed:
|
|
||||||
assert response.headers.get("X-Cashu") == "cashuAtoken"
|
|
||||||
else:
|
|
||||||
assert "X-Cashu" not in response.headers
|
|
||||||
|
|
||||||
|
|
||||||
@pytest.mark.asyncio
|
|
||||||
@pytest.mark.parametrize(
|
|
||||||
("handler_name", "forward_attr"),
|
|
||||||
[
|
|
||||||
("handle_x_cashu", "forward_x_cashu_request"),
|
|
||||||
("handle_x_cashu_responses", "forward_x_cashu_responses_request"),
|
|
||||||
],
|
|
||||||
)
|
|
||||||
@pytest.mark.parametrize("amount", [0, -5])
|
|
||||||
async def test_x_cashu_zero_value_rejected_not_forwarded(
|
|
||||||
handler_name: str, forward_attr: str, amount: int
|
|
||||||
) -> None:
|
|
||||||
"""A token that redeems to <= 0 must be rejected as cashu_token_zero_value
|
|
||||||
(400) and NEVER forwarded as a free request — the X-Cashu path lacked the
|
|
||||||
guard that credit_balance has."""
|
|
||||||
provider = _make_provider()
|
|
||||||
model = _make_model()
|
|
||||||
request = _make_request()
|
|
||||||
|
|
||||||
with (
|
|
||||||
patch(
|
|
||||||
"routstr.upstream.base.recieve_token",
|
|
||||||
new=AsyncMock(return_value=(amount, "sat", "https://mint")),
|
|
||||||
),
|
|
||||||
patch("routstr.upstream.base.store_cashu_transaction", new=AsyncMock()),
|
|
||||||
patch.object(
|
|
||||||
provider,
|
|
||||||
forward_attr,
|
|
||||||
new=AsyncMock(side_effect=AssertionError("must not forward a zero-value token")),
|
|
||||||
),
|
|
||||||
):
|
|
||||||
handler = getattr(provider, handler_name)
|
|
||||||
response = await handler(
|
|
||||||
request=request,
|
|
||||||
x_cashu_token="cashuAtoken",
|
|
||||||
path="v1/chat/completions",
|
|
||||||
max_cost_for_model=10_000,
|
|
||||||
model_obj=model,
|
|
||||||
)
|
|
||||||
|
|
||||||
assert response.status_code == 400
|
|
||||||
body = json.loads(bytes(response.body))
|
|
||||||
assert body["error"]["type"] == "cashu_error"
|
|
||||||
assert (
|
|
||||||
body["error"]["message"]
|
|
||||||
== "Failed to redeem Cashu token: token yielded no value"
|
|
||||||
)
|
|
||||||
assert body["error"]["code"] == "cashu_token_zero_value"
|
|
||||||
# Spent-to-zero token must not be echoed back for retry.
|
|
||||||
assert "X-Cashu" not in response.headers
|
|
||||||
|
|||||||
@@ -1,55 +0,0 @@
|
|||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
import importlib.util
|
|
||||||
from pathlib import Path
|
|
||||||
|
|
||||||
import sqlalchemy as sa
|
|
||||||
|
|
||||||
_MIGRATION_PATH = (
|
|
||||||
Path(__file__).resolve().parents[2]
|
|
||||||
/ "migrations"
|
|
||||||
/ "versions"
|
|
||||||
/ "c6d7e8f9a0b1_add_slug_to_upstream_providers.py"
|
|
||||||
)
|
|
||||||
_spec = importlib.util.spec_from_file_location("provider_slug_migration", _MIGRATION_PATH)
|
|
||||||
assert _spec is not None and _spec.loader is not None
|
|
||||||
migration = importlib.util.module_from_spec(_spec)
|
|
||||||
_spec.loader.exec_module(migration)
|
|
||||||
|
|
||||||
|
|
||||||
def test_slug_migration_backfill_uses_api_safe_deterministic_slugs() -> None:
|
|
||||||
engine = sa.create_engine("sqlite:///:memory:")
|
|
||||||
with engine.begin() as conn:
|
|
||||||
conn.execute(
|
|
||||||
sa.text(
|
|
||||||
"CREATE TABLE upstream_providers ("
|
|
||||||
"id INTEGER PRIMARY KEY, "
|
|
||||||
"provider_type VARCHAR NOT NULL, "
|
|
||||||
"slug VARCHAR NULL"
|
|
||||||
")"
|
|
||||||
)
|
|
||||||
)
|
|
||||||
conn.execute(
|
|
||||||
sa.text(
|
|
||||||
"INSERT INTO upstream_providers (id, provider_type, slug) VALUES "
|
|
||||||
"(1, 'OpenAI Compatible', NULL), "
|
|
||||||
"(2, 'OpenAI Compatible', ''), "
|
|
||||||
"(3, '123', NULL), "
|
|
||||||
"(4, 'x', NULL), "
|
|
||||||
"(5, 'anthropic', 'anthropic')"
|
|
||||||
)
|
|
||||||
)
|
|
||||||
|
|
||||||
migration._backfill_provider_slugs(conn)
|
|
||||||
|
|
||||||
rows = conn.execute(
|
|
||||||
sa.text("SELECT id, slug FROM upstream_providers ORDER BY id")
|
|
||||||
).all()
|
|
||||||
|
|
||||||
assert rows == [
|
|
||||||
(1, "openai-compatible"),
|
|
||||||
(2, "openai-compatible-2"),
|
|
||||||
(3, "provider-123"),
|
|
||||||
(4, "x-provider"),
|
|
||||||
(5, "anthropic"),
|
|
||||||
]
|
|
||||||
@@ -1,144 +0,0 @@
|
|||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
import pytest
|
|
||||||
from sqlalchemy.ext.asyncio import create_async_engine
|
|
||||||
from sqlmodel import SQLModel, select
|
|
||||||
from sqlmodel.ext.asyncio.session import AsyncSession
|
|
||||||
|
|
||||||
from routstr.core.admin import _get_upstream_provider_by_ref
|
|
||||||
from routstr.core.db import UpstreamProviderRow
|
|
||||||
from routstr.core.provider_slugs import (
|
|
||||||
allocate_unique_provider_slug,
|
|
||||||
provider_slug_base,
|
|
||||||
)
|
|
||||||
from routstr.upstream.helpers import _seed_providers_from_settings
|
|
||||||
|
|
||||||
|
|
||||||
@pytest.mark.asyncio
|
|
||||||
async def test_allocate_unique_provider_slug_is_deterministic_with_suffixes() -> None:
|
|
||||||
engine = create_async_engine("sqlite+aiosqlite:///:memory:")
|
|
||||||
async with engine.begin() as conn:
|
|
||||||
await conn.run_sync(SQLModel.metadata.create_all)
|
|
||||||
|
|
||||||
async with AsyncSession(engine) as session:
|
|
||||||
session.add(
|
|
||||||
UpstreamProviderRow(
|
|
||||||
slug="openai",
|
|
||||||
provider_type="openai",
|
|
||||||
base_url="https://api.openai.com/v1",
|
|
||||||
api_key="key-1",
|
|
||||||
)
|
|
||||||
)
|
|
||||||
await session.commit()
|
|
||||||
|
|
||||||
assert await allocate_unique_provider_slug(session, "openai") == "openai-2"
|
|
||||||
assert (
|
|
||||||
await allocate_unique_provider_slug(session, "openai", {"openai-2"})
|
|
||||||
== "openai-3"
|
|
||||||
)
|
|
||||||
|
|
||||||
await engine.dispose()
|
|
||||||
|
|
||||||
|
|
||||||
def test_provider_slug_base_sanitizes_provider_type() -> None:
|
|
||||||
assert provider_slug_base("OpenAI Compatible") == "openai-compatible"
|
|
||||||
assert provider_slug_base("!!!") == "provider"
|
|
||||||
assert provider_slug_base("AI") == "ai-provider"
|
|
||||||
assert provider_slug_base("123") == "provider-123"
|
|
||||||
|
|
||||||
|
|
||||||
@pytest.mark.asyncio
|
|
||||||
async def test_provider_ref_lookup_accepts_existing_numeric_ids_and_slugs() -> None:
|
|
||||||
engine = create_async_engine("sqlite+aiosqlite:///:memory:")
|
|
||||||
async with engine.begin() as conn:
|
|
||||||
await conn.run_sync(SQLModel.metadata.create_all)
|
|
||||||
|
|
||||||
async with AsyncSession(engine) as session:
|
|
||||||
provider = UpstreamProviderRow(
|
|
||||||
slug="openai",
|
|
||||||
provider_type="openai",
|
|
||||||
base_url="https://api.openai.com/v1",
|
|
||||||
api_key="key-1",
|
|
||||||
)
|
|
||||||
session.add(provider)
|
|
||||||
await session.commit()
|
|
||||||
await session.refresh(provider)
|
|
||||||
|
|
||||||
by_id = await _get_upstream_provider_by_ref(session, str(provider.id))
|
|
||||||
by_slug = await _get_upstream_provider_by_ref(session, "openai")
|
|
||||||
|
|
||||||
assert by_id.id == provider.id
|
|
||||||
assert by_slug.id == provider.id
|
|
||||||
|
|
||||||
await engine.dispose()
|
|
||||||
|
|
||||||
|
|
||||||
@pytest.mark.asyncio
|
|
||||||
async def test_seed_providers_from_settings_sets_deterministic_slug(
|
|
||||||
monkeypatch: pytest.MonkeyPatch,
|
|
||||||
) -> None:
|
|
||||||
engine = create_async_engine("sqlite+aiosqlite:///:memory:")
|
|
||||||
async with engine.begin() as conn:
|
|
||||||
await conn.run_sync(SQLModel.metadata.create_all)
|
|
||||||
|
|
||||||
monkeypatch.setenv("OPENAI_API_KEY", "seeded-openai-key")
|
|
||||||
|
|
||||||
class SettingsStub:
|
|
||||||
chat_completions_api_version: str | None = None
|
|
||||||
upstream_base_url: str | None = None
|
|
||||||
upstream_api_key: str = ""
|
|
||||||
|
|
||||||
async with AsyncSession(engine) as session:
|
|
||||||
await _seed_providers_from_settings(session, SettingsStub()) # type: ignore[arg-type]
|
|
||||||
await session.commit()
|
|
||||||
|
|
||||||
result = await session.exec(select(UpstreamProviderRow))
|
|
||||||
providers: list[UpstreamProviderRow] = list(result.all())
|
|
||||||
|
|
||||||
assert [(p.provider_type, p.slug) for p in providers] == [("openai", "openai")]
|
|
||||||
|
|
||||||
await engine.dispose()
|
|
||||||
|
|
||||||
|
|
||||||
@pytest.mark.asyncio
|
|
||||||
async def test_seed_providers_from_settings_keeps_slug_stable_on_reseed(
|
|
||||||
monkeypatch: pytest.MonkeyPatch,
|
|
||||||
) -> None:
|
|
||||||
engine = create_async_engine("sqlite+aiosqlite:///:memory:")
|
|
||||||
async with engine.begin() as conn:
|
|
||||||
await conn.run_sync(SQLModel.metadata.create_all)
|
|
||||||
|
|
||||||
monkeypatch.setenv("OPENAI_API_KEY", "seeded-openai-key")
|
|
||||||
|
|
||||||
class SettingsStub:
|
|
||||||
chat_completions_api_version: str | None = None
|
|
||||||
upstream_base_url: str | None = None
|
|
||||||
upstream_api_key: str = ""
|
|
||||||
|
|
||||||
async with AsyncSession(engine) as session:
|
|
||||||
session.add(
|
|
||||||
UpstreamProviderRow(
|
|
||||||
slug="openai",
|
|
||||||
provider_type="openai",
|
|
||||||
base_url="https://example.invalid/v1",
|
|
||||||
api_key="other-key",
|
|
||||||
)
|
|
||||||
)
|
|
||||||
await session.commit()
|
|
||||||
|
|
||||||
await _seed_providers_from_settings(session, SettingsStub()) # type: ignore[arg-type]
|
|
||||||
await session.commit()
|
|
||||||
await _seed_providers_from_settings(session, SettingsStub()) # type: ignore[arg-type]
|
|
||||||
await session.commit()
|
|
||||||
|
|
||||||
result = await session.exec(
|
|
||||||
select(UpstreamProviderRow).order_by(UpstreamProviderRow.slug)
|
|
||||||
)
|
|
||||||
providers: list[UpstreamProviderRow] = list(result.all())
|
|
||||||
|
|
||||||
assert [(p.provider_type, p.slug) for p in providers] == [
|
|
||||||
("openai", "openai"),
|
|
||||||
("openai", "openai-2"),
|
|
||||||
]
|
|
||||||
|
|
||||||
await engine.dispose()
|
|
||||||
@@ -1,95 +0,0 @@
|
|||||||
import json
|
|
||||||
import os
|
|
||||||
|
|
||||||
os.environ.setdefault("UPSTREAM_BASE_URL", "http://test")
|
|
||||||
os.environ.setdefault("UPSTREAM_API_KEY", "test")
|
|
||||||
os.environ.setdefault("LIGHTNING_ADDRESS", "test@stm.to")
|
|
||||||
|
|
||||||
from routstr.upstream import GenericUpstreamProvider
|
|
||||||
from routstr.upstream.openrouter import OpenRouterUpstreamProvider
|
|
||||||
|
|
||||||
|
|
||||||
def _model(model_id: str = "openai/gpt-4o"): # type: ignore[no-untyped-def]
|
|
||||||
from routstr.payment.models import Architecture, Model, Pricing
|
|
||||||
|
|
||||||
return Model(
|
|
||||||
id=model_id,
|
|
||||||
name=model_id,
|
|
||||||
created=0,
|
|
||||||
description="",
|
|
||||||
context_length=128000,
|
|
||||||
architecture=Architecture(
|
|
||||||
modality="text->text",
|
|
||||||
input_modalities=["text"],
|
|
||||||
output_modalities=["text"],
|
|
||||||
tokenizer="GPT",
|
|
||||||
instruct_type=None,
|
|
||||||
),
|
|
||||||
pricing=Pricing(prompt=0.0, completion=0.0),
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
def _tool_body() -> dict:
|
|
||||||
return {
|
|
||||||
"model": "openai/gpt-4o",
|
|
||||||
"messages": [{"role": "user", "content": "What's the weather?"}],
|
|
||||||
"tools": [
|
|
||||||
{
|
|
||||||
"type": "function",
|
|
||||||
"function": {"name": "get_weather", "parameters": {}},
|
|
||||||
}
|
|
||||||
],
|
|
||||||
}
|
|
||||||
|
|
||||||
|
|
||||||
def _prepare(provider, body: dict) -> dict: # type: ignore[no-untyped-def]
|
|
||||||
out = provider.prepare_request_body(json.dumps(body).encode(), _model())
|
|
||||||
assert out is not None
|
|
||||||
return json.loads(out)
|
|
||||||
|
|
||||||
|
|
||||||
def test_injects_require_parameters_for_tool_request() -> None:
|
|
||||||
data = _prepare(OpenRouterUpstreamProvider(api_key="test"), _tool_body())
|
|
||||||
assert data["provider"]["require_parameters"] is True
|
|
||||||
|
|
||||||
|
|
||||||
def test_generic_provider_on_openrouter_url_is_left_alone() -> None:
|
|
||||||
# Only OpenRouterUpstreamProvider injects; a generic provider pointed at the
|
|
||||||
# same base URL doesn't.
|
|
||||||
provider = GenericUpstreamProvider(base_url="https://openrouter.ai/api/v1")
|
|
||||||
data = _prepare(provider, _tool_body())
|
|
||||||
assert "provider" not in data
|
|
||||||
|
|
||||||
|
|
||||||
def test_no_injection_without_tools() -> None:
|
|
||||||
body = {"model": "openai/gpt-4o", "messages": [{"role": "user", "content": "hi"}]}
|
|
||||||
data = _prepare(OpenRouterUpstreamProvider(api_key="test"), body)
|
|
||||||
assert "provider" not in data
|
|
||||||
|
|
||||||
|
|
||||||
def test_empty_tools_list_does_not_inject() -> None:
|
|
||||||
body = _tool_body()
|
|
||||||
body["tools"] = []
|
|
||||||
data = _prepare(OpenRouterUpstreamProvider(api_key="test"), body)
|
|
||||||
assert "provider" not in data
|
|
||||||
|
|
||||||
|
|
||||||
def test_direct_provider_does_not_inject() -> None:
|
|
||||||
provider = GenericUpstreamProvider(base_url="https://api.openai.com/v1")
|
|
||||||
data = _prepare(provider, _tool_body())
|
|
||||||
assert "provider" not in data
|
|
||||||
|
|
||||||
|
|
||||||
def test_keeps_client_set_require_parameters() -> None:
|
|
||||||
body = _tool_body()
|
|
||||||
body["provider"] = {"require_parameters": False}
|
|
||||||
data = _prepare(OpenRouterUpstreamProvider(api_key="test"), body)
|
|
||||||
assert data["provider"]["require_parameters"] is False
|
|
||||||
|
|
||||||
|
|
||||||
def test_preserves_other_provider_fields() -> None:
|
|
||||||
body = _tool_body()
|
|
||||||
body["provider"] = {"order": ["openai", "azure"]}
|
|
||||||
data = _prepare(OpenRouterUpstreamProvider(api_key="test"), body)
|
|
||||||
assert data["provider"]["order"] == ["openai", "azure"]
|
|
||||||
assert data["provider"]["require_parameters"] is True
|
|
||||||
@@ -20,6 +20,7 @@ comment ever reaches the client. That invariant is exactly what the buggy
|
|||||||
|
|
||||||
import json
|
import json
|
||||||
from collections.abc import AsyncGenerator
|
from collections.abc import AsyncGenerator
|
||||||
|
from typing import cast
|
||||||
from unittest.mock import AsyncMock, MagicMock
|
from unittest.mock import AsyncMock, MagicMock
|
||||||
|
|
||||||
import pytest
|
import pytest
|
||||||
@@ -107,6 +108,76 @@ def _assert_clean(out: list[bytes]) -> list[dict]:
|
|||||||
return objs
|
return objs
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_deepseek_usage_chunk_is_normalized_before_billing() -> None:
|
||||||
|
"""DeepSeek stream trailers keep raw fields and add canonical billing fields."""
|
||||||
|
usage = {
|
||||||
|
"prompt_tokens": 10000,
|
||||||
|
"completion_tokens": 500,
|
||||||
|
"total_tokens": 10500,
|
||||||
|
"prompt_cache_hit_tokens": 9000,
|
||||||
|
"prompt_cache_miss_tokens": 1000,
|
||||||
|
}
|
||||||
|
chunks = [
|
||||||
|
b'data: {"id":"ds","model":"deepseek-chat","choices":[{"delta":{"content":"ok"}}]}\n\n',
|
||||||
|
b"data: "
|
||||||
|
+ json.dumps(
|
||||||
|
{
|
||||||
|
"id": "ds",
|
||||||
|
"model": "deepseek-chat",
|
||||||
|
"choices": [],
|
||||||
|
"usage": usage,
|
||||||
|
}
|
||||||
|
).encode()
|
||||||
|
+ b"\n\n",
|
||||||
|
b"data: [DONE]\n\n",
|
||||||
|
]
|
||||||
|
|
||||||
|
await _drive(chunks)
|
||||||
|
|
||||||
|
adjust_mock = cast(AsyncMock, base.adjust_payment_for_tokens)
|
||||||
|
adjustment_input = adjust_mock.call_args.args[1]
|
||||||
|
billed_usage = adjustment_input["usage"]
|
||||||
|
assert billed_usage["prompt_tokens"] == 10000
|
||||||
|
assert billed_usage["prompt_cache_hit_tokens"] == 9000
|
||||||
|
assert billed_usage["prompt_cache_miss_tokens"] == 1000
|
||||||
|
assert billed_usage["input_tokens"] == 1000
|
||||||
|
assert billed_usage["output_tokens"] == 500
|
||||||
|
assert billed_usage["cache_read_input_tokens"] == 9000
|
||||||
|
assert billed_usage["cache_creation_input_tokens"] == 0
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_fold_cache_tokens_does_not_double_count_inclusive_prompt_tokens() -> None:
|
||||||
|
"""Visible usage mutation must not inflate OpenAI-compatible prompt totals."""
|
||||||
|
usage = {
|
||||||
|
"prompt_tokens": 10000,
|
||||||
|
"completion_tokens": 500,
|
||||||
|
"cache_read_input_tokens": 9000,
|
||||||
|
"prompt_cache_hit_tokens": 9000,
|
||||||
|
}
|
||||||
|
|
||||||
|
BaseUpstreamProvider._fold_cache_into_input_tokens(usage)
|
||||||
|
|
||||||
|
assert usage["prompt_tokens"] == 10000
|
||||||
|
assert usage["cache_read_input_tokens"] == 9000
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_fold_cache_tokens_still_rolls_up_anthropic_input_tokens() -> None:
|
||||||
|
"""Anthropic native input_tokens excludes cache and still needs rollup."""
|
||||||
|
usage = {
|
||||||
|
"input_tokens": 1000,
|
||||||
|
"output_tokens": 500,
|
||||||
|
"cache_read_input_tokens": 9000,
|
||||||
|
"cache_creation_input_tokens": 200,
|
||||||
|
}
|
||||||
|
|
||||||
|
BaseUpstreamProvider._fold_cache_into_input_tokens(usage)
|
||||||
|
|
||||||
|
assert usage["input_tokens"] == 10200
|
||||||
|
|
||||||
|
|
||||||
@pytest.mark.asyncio
|
@pytest.mark.asyncio
|
||||||
async def test_openai_style_plain_stream() -> None:
|
async def test_openai_style_plain_stream() -> None:
|
||||||
"""OpenAI / Groq / Fireworks / xAI / Perplexity: plain data + [DONE]."""
|
"""OpenAI / Groq / Fireworks / xAI / Perplexity: plain data + [DONE]."""
|
||||||
@@ -337,75 +408,3 @@ async def test_multiline_non_json_data_each_line_prefixed() -> None:
|
|||||||
continue
|
continue
|
||||||
assert line.startswith(b"data: "), f"bare line leaked to client: {line!r}"
|
assert line.startswith(b"data: "), f"bare line leaked to client: {line!r}"
|
||||||
assert b"data: line one" in blob and b"data: line two" in blob
|
assert b"data: line one" in blob and b"data: line two" in blob
|
||||||
|
|
||||||
|
|
||||||
@pytest.mark.asyncio
|
|
||||||
async def test_crlf_delimiter_split_across_chunk_boundary() -> None:
|
|
||||||
"""CRLF event delimiter straddling two TCP reads must not merge events.
|
|
||||||
|
|
||||||
Regression: a per-chunk ``replace(b"\\r\\n", b"\\n")`` left a stray ``\\r``
|
|
||||||
when a ``\\r\\n`` of the ``\\r\\n\\r\\n`` delimiter landed at the very end of
|
|
||||||
one ``aiter_bytes`` chunk and the matching ``\\n`` opened the next. The
|
|
||||||
``\\n\\n`` split then missed the boundary, glued two events into one frame
|
|
||||||
with two ``data:`` lines, and the client's ``JSON.parse`` threw on the
|
|
||||||
concatenated payload (the "unexpected token"/"Extra data" crash).
|
|
||||||
"""
|
|
||||||
e1 = b'data: {"id":"x","choices":[{"delta":{"content":"a"}}]}'
|
|
||||||
e2 = b'data: {"id":"x","choices":[{"delta":{"content":"b"}}]}'
|
|
||||||
chunks = [
|
|
||||||
e1 + b"\r\n\r", # delimiter cut mid-CRLF
|
|
||||||
b"\n" + e2 + b"\r\n\r\n",
|
|
||||||
b"data: [DONE]\r\n\r\n",
|
|
||||||
]
|
|
||||||
out = await _drive(chunks)
|
|
||||||
|
|
||||||
# Client-accurate check: a real SSE client concatenates all ``data:`` lines
|
|
||||||
# *within one event* (events are ``\n\n``-delimited) before parsing. A
|
|
||||||
# merged frame would surface here as two objects glued into one payload,
|
|
||||||
# which ``_assert_clean`` (per-line) would miss.
|
|
||||||
blob = b"".join(out)
|
|
||||||
contents: list[str] = []
|
|
||||||
for event in blob.split(b"\n\n"):
|
|
||||||
datas = [
|
|
||||||
ln[len(b"data: ") :]
|
|
||||||
for ln in event.split(b"\n")
|
|
||||||
if ln.startswith(b"data: ")
|
|
||||||
]
|
|
||||||
if not datas:
|
|
||||||
continue
|
|
||||||
payload = b"".join(datas)
|
|
||||||
if payload.strip() == b"[DONE]":
|
|
||||||
continue
|
|
||||||
obj = json.loads(payload) # raises if two events were merged into one
|
|
||||||
for c in obj.get("choices", []):
|
|
||||||
if "delta" in c:
|
|
||||||
contents.append(c["delta"]["content"])
|
|
||||||
assert contents == ["a", "b"]
|
|
||||||
|
|
||||||
|
|
||||||
@pytest.mark.asyncio
|
|
||||||
async def test_truncated_json_tail_on_connection_close() -> None:
|
|
||||||
"""A stream that drops mid-event must not emit the partial JSON downstream.
|
|
||||||
|
|
||||||
Regression: the end-of-stream flush ran ``_process_event`` on the leftover
|
|
||||||
buffer unconditionally. When the upstream connection closed mid-event the
|
|
||||||
leftover was incomplete JSON, which fell through to the raw-forward path and
|
|
||||||
handed the client a ``data: {partial`` frame -> ``Unterminated string`` parse
|
|
||||||
error. The truncated tail must be dropped instead.
|
|
||||||
"""
|
|
||||||
chunks = [
|
|
||||||
b'data: {"id":"x","choices":[{"delta":{"content":"ok"}}]}\n\n',
|
|
||||||
b'data: {"id":"x","choices":[{"delta":{"con', # connection dies here
|
|
||||||
]
|
|
||||||
out = await _drive(chunks)
|
|
||||||
objs = _assert_clean(out) # raises if the partial tail leaked as a data frame
|
|
||||||
contents = [
|
|
||||||
c["delta"]["content"]
|
|
||||||
for o in objs
|
|
||||||
for c in o.get("choices", [])
|
|
||||||
if "delta" in c
|
|
||||||
]
|
|
||||||
# The one complete chunk is delivered; the truncated fragment is dropped
|
|
||||||
# entirely (no second delta), and _assert_clean above guarantees nothing
|
|
||||||
# non-JSON ever reached the client.
|
|
||||||
assert contents == ["ok"]
|
|
||||||
|
|||||||
@@ -1,396 +0,0 @@
|
|||||||
"""Tests for upstream rate-limit detection, classification, and org-ID redaction.
|
|
||||||
|
|
||||||
Covers issue #555: upstream OpenAI-compatible providers return rate-limit
|
|
||||||
errors that embed a sensitive organization ID. The proxy must classify these
|
|
||||||
distinctly (``UPSTREAM_RATE_LIMIT``), preserve useful debugging fields, and
|
|
||||||
never emit a raw ``org-*`` identifier in logs, errors, or returned bodies.
|
|
||||||
"""
|
|
||||||
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
import json
|
|
||||||
from typing import Any
|
|
||||||
from unittest.mock import AsyncMock, MagicMock, Mock, patch
|
|
||||||
|
|
||||||
import httpx
|
|
||||||
import pytest
|
|
||||||
|
|
||||||
from routstr.core.redaction import redact_org_ids
|
|
||||||
from routstr.upstream.base import BaseUpstreamProvider
|
|
||||||
from routstr.upstream.rate_limit import (
|
|
||||||
UPSTREAM_RATE_LIMIT,
|
|
||||||
RateLimitInfo,
|
|
||||||
classify_rate_limit,
|
|
||||||
)
|
|
||||||
|
|
||||||
# The exact scenario from the issue, with a realistic (fake) org identifier.
|
|
||||||
RAW_ORG_ID = "org-abc123XYZ456def"
|
|
||||||
RATE_LIMIT_MESSAGE = (
|
|
||||||
f"Rate limit reached for gpt-5.5-2026-04-23 (for limit gpt-5.5) in "
|
|
||||||
f"organization {RAW_ORG_ID} on tokens per min (TPM): Limit 180000000, "
|
|
||||||
f"Used 180000000, Requested 8929. Please try again in 2ms. Visit "
|
|
||||||
f"https://platform.openai.com/account/rate-limits to learn more."
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
def _make_request(request_id: str = "req-123") -> Mock:
|
|
||||||
request = Mock(spec=["method", "state"])
|
|
||||||
request.method = "POST"
|
|
||||||
request.state = Mock()
|
|
||||||
request.state.request_id = request_id
|
|
||||||
return request
|
|
||||||
|
|
||||||
|
|
||||||
def _make_upstream_response(
|
|
||||||
*,
|
|
||||||
body: bytes,
|
|
||||||
status_code: int = 429,
|
|
||||||
content_type: str | None = "application/json",
|
|
||||||
extra_headers: dict[str, str] | None = None,
|
|
||||||
) -> httpx.Response:
|
|
||||||
headers: dict[str, str] = {}
|
|
||||||
if content_type is not None:
|
|
||||||
headers["content-type"] = content_type
|
|
||||||
if extra_headers:
|
|
||||||
headers.update(extra_headers)
|
|
||||||
return httpx.Response(status_code=status_code, headers=headers, content=body)
|
|
||||||
|
|
||||||
|
|
||||||
@pytest.fixture
|
|
||||||
def provider() -> BaseUpstreamProvider:
|
|
||||||
return BaseUpstreamProvider(
|
|
||||||
base_url="https://privateprovider.xyz", api_key="k", provider_fee=1.0
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
# --------------------------------------------------------------------------- #
|
|
||||||
# Redaction
|
|
||||||
# --------------------------------------------------------------------------- #
|
|
||||||
|
|
||||||
|
|
||||||
def test_redact_org_ids_replaces_identifier() -> None:
|
|
||||||
assert RAW_ORG_ID not in redact_org_ids(RATE_LIMIT_MESSAGE)
|
|
||||||
assert "org-[REDACTED]" in redact_org_ids(RATE_LIMIT_MESSAGE)
|
|
||||||
|
|
||||||
|
|
||||||
def test_redact_org_ids_is_idempotent() -> None:
|
|
||||||
once = redact_org_ids(RATE_LIMIT_MESSAGE)
|
|
||||||
assert redact_org_ids(once) == once
|
|
||||||
|
|
||||||
|
|
||||||
def test_redact_org_ids_leaves_unrelated_text() -> None:
|
|
||||||
assert redact_org_ids("organize the org-chart") == "organize the org-chart"
|
|
||||||
assert redact_org_ids("") == ""
|
|
||||||
|
|
||||||
|
|
||||||
# --------------------------------------------------------------------------- #
|
|
||||||
# Classification
|
|
||||||
# --------------------------------------------------------------------------- #
|
|
||||||
|
|
||||||
|
|
||||||
def test_classify_exact_scenario() -> None:
|
|
||||||
info = classify_rate_limit(429, RATE_LIMIT_MESSAGE)
|
|
||||||
assert isinstance(info, RateLimitInfo)
|
|
||||||
assert info.code == UPSTREAM_RATE_LIMIT
|
|
||||||
assert info.model == "gpt-5.5-2026-04-23"
|
|
||||||
assert info.limit_name == "gpt-5.5"
|
|
||||||
assert info.metric == "tokens per min (TPM)"
|
|
||||||
assert info.limit == 180000000
|
|
||||||
assert info.used == 180000000
|
|
||||||
assert info.requested == 8929
|
|
||||||
assert info.retry_after_seconds == pytest.approx(0.002)
|
|
||||||
# Redaction-safe: no raw org id survives into the structured view.
|
|
||||||
assert RAW_ORG_ID not in info.message
|
|
||||||
assert RAW_ORG_ID not in json.dumps(info.as_details())
|
|
||||||
|
|
||||||
|
|
||||||
def test_classify_by_status_code_without_marker() -> None:
|
|
||||||
info = classify_rate_limit(429, "slow down")
|
|
||||||
assert info is not None
|
|
||||||
assert info.code == UPSTREAM_RATE_LIMIT
|
|
||||||
|
|
||||||
|
|
||||||
def test_classify_by_message_marker_without_429() -> None:
|
|
||||||
info = classify_rate_limit(400, "rate_limit_exceeded for this key")
|
|
||||||
assert info is not None
|
|
||||||
|
|
||||||
|
|
||||||
def test_retry_after_header_takes_precedence() -> None:
|
|
||||||
info = classify_rate_limit(429, RATE_LIMIT_MESSAGE, {"Retry-After": "12"})
|
|
||||||
assert info is not None
|
|
||||||
assert info.retry_after_seconds == pytest.approx(12.0)
|
|
||||||
|
|
||||||
|
|
||||||
def test_non_rate_limit_error_is_not_classified() -> None:
|
|
||||||
assert classify_rate_limit(400, "invalid request: missing field 'model'") is None
|
|
||||||
assert classify_rate_limit(500, "internal server error") is None
|
|
||||||
|
|
||||||
|
|
||||||
# --------------------------------------------------------------------------- #
|
|
||||||
# forward_upstream_error_response integration
|
|
||||||
# --------------------------------------------------------------------------- #
|
|
||||||
|
|
||||||
|
|
||||||
@pytest.mark.asyncio
|
|
||||||
async def test_json_rate_limit_body_is_redacted_and_forwarded(
|
|
||||||
provider: BaseUpstreamProvider,
|
|
||||||
) -> None:
|
|
||||||
body = json.dumps(
|
|
||||||
{"error": {"message": RATE_LIMIT_MESSAGE, "type": "rate_limit_exceeded"}}
|
|
||||||
).encode()
|
|
||||||
upstream = _make_upstream_response(body=body, status_code=429)
|
|
||||||
|
|
||||||
response = await provider.forward_upstream_error_response(
|
|
||||||
_make_request(), "v1/chat/completions", upstream
|
|
||||||
)
|
|
||||||
|
|
||||||
assert response.status_code == 429
|
|
||||||
raw = bytes(response.body).decode()
|
|
||||||
# No raw organization id may survive in the forwarded body.
|
|
||||||
assert RAW_ORG_ID not in raw
|
|
||||||
assert "org-[REDACTED]" in raw
|
|
||||||
# Body remains valid JSON; the original type is preserved while a stable
|
|
||||||
# rate-limit code is injected so callers can switch on it.
|
|
||||||
payload: dict[str, Any] = json.loads(raw)
|
|
||||||
assert payload["error"]["type"] == "rate_limit_exceeded"
|
|
||||||
assert payload["error"]["code"] == UPSTREAM_RATE_LIMIT
|
|
||||||
assert payload["error"]["details"]["model"] == "gpt-5.5-2026-04-23"
|
|
||||||
# A retry hint extracted from the message is surfaced as a header.
|
|
||||||
assert "retry-after" in {k.lower() for k in response.headers}
|
|
||||||
|
|
||||||
|
|
||||||
@pytest.mark.asyncio
|
|
||||||
async def test_non_json_rate_limit_envelope_uses_stable_code(
|
|
||||||
provider: BaseUpstreamProvider,
|
|
||||||
) -> None:
|
|
||||||
upstream = _make_upstream_response(
|
|
||||||
body=RATE_LIMIT_MESSAGE.encode(),
|
|
||||||
status_code=429,
|
|
||||||
content_type="text/plain",
|
|
||||||
)
|
|
||||||
|
|
||||||
response = await provider.forward_upstream_error_response(
|
|
||||||
_make_request(), "v1/chat/completions", upstream
|
|
||||||
)
|
|
||||||
|
|
||||||
payload: dict[str, Any] = json.loads(bytes(response.body))
|
|
||||||
assert payload["error"]["code"] == UPSTREAM_RATE_LIMIT
|
|
||||||
assert payload["error"]["details"]["model"] == "gpt-5.5-2026-04-23"
|
|
||||||
serialized = json.dumps(payload)
|
|
||||||
assert RAW_ORG_ID not in serialized
|
|
||||||
assert "org-[REDACTED]" in serialized
|
|
||||||
|
|
||||||
|
|
||||||
@pytest.mark.asyncio
|
|
||||||
async def test_non_rate_limit_json_error_unchanged(
|
|
||||||
provider: BaseUpstreamProvider,
|
|
||||||
) -> None:
|
|
||||||
body = json.dumps(
|
|
||||||
{"error": {"message": "missing field 'model'", "type": "invalid_request"}}
|
|
||||||
).encode()
|
|
||||||
upstream = _make_upstream_response(body=body, status_code=400)
|
|
||||||
|
|
||||||
response = await provider.forward_upstream_error_response(
|
|
||||||
_make_request(), "v1/chat/completions", upstream
|
|
||||||
)
|
|
||||||
|
|
||||||
assert response.status_code == 400
|
|
||||||
payload: dict[str, Any] = json.loads(bytes(response.body))
|
|
||||||
assert payload["error"]["type"] == "invalid_request"
|
|
||||||
assert "retry-after" not in {k.lower() for k in response.headers}
|
|
||||||
|
|
||||||
|
|
||||||
# --------------------------------------------------------------------------- #
|
|
||||||
# UpstreamError -> proxy response (preserves code/details/status)
|
|
||||||
# --------------------------------------------------------------------------- #
|
|
||||||
|
|
||||||
|
|
||||||
def test_create_upstream_error_response_preserves_structure() -> None:
|
|
||||||
from routstr.core.exceptions import UpstreamError
|
|
||||||
from routstr.payment.helpers import create_upstream_error_response
|
|
||||||
|
|
||||||
info = classify_rate_limit(429, RATE_LIMIT_MESSAGE)
|
|
||||||
assert info is not None
|
|
||||||
err = UpstreamError(
|
|
||||||
f"Upstream error via litellm: {RATE_LIMIT_MESSAGE}",
|
|
||||||
status_code=429,
|
|
||||||
code=info.code,
|
|
||||||
details=info.as_details(),
|
|
||||||
)
|
|
||||||
|
|
||||||
response = create_upstream_error_response(err, _make_request())
|
|
||||||
|
|
||||||
# Original upstream status is preserved (not flattened to 502).
|
|
||||||
assert response.status_code == 429
|
|
||||||
payload: dict[str, Any] = json.loads(bytes(response.body))
|
|
||||||
assert payload["error"]["type"] == "upstream_error"
|
|
||||||
assert payload["error"]["code"] == UPSTREAM_RATE_LIMIT
|
|
||||||
assert payload["error"]["details"]["requested"] == 8929
|
|
||||||
serialized = json.dumps(payload)
|
|
||||||
assert RAW_ORG_ID not in serialized
|
|
||||||
assert "org-[REDACTED]" in serialized
|
|
||||||
|
|
||||||
|
|
||||||
def test_generic_upstream_error_still_defaults_to_502() -> None:
|
|
||||||
from routstr.core.exceptions import UpstreamError
|
|
||||||
from routstr.payment.helpers import create_upstream_error_response
|
|
||||||
|
|
||||||
err = UpstreamError("connection refused") # status_code defaults to 502
|
|
||||||
|
|
||||||
response = create_upstream_error_response(err, _make_request())
|
|
||||||
|
|
||||||
assert response.status_code == 502
|
|
||||||
payload: dict[str, Any] = json.loads(bytes(response.body))
|
|
||||||
assert payload["error"]["type"] == "upstream_error"
|
|
||||||
assert payload["error"]["code"] == 502
|
|
||||||
assert "details" not in payload["error"]
|
|
||||||
|
|
||||||
|
|
||||||
# --------------------------------------------------------------------------- #
|
|
||||||
# Structured log-extra redaction
|
|
||||||
# --------------------------------------------------------------------------- #
|
|
||||||
|
|
||||||
|
|
||||||
def test_security_filter_redacts_org_id_in_extra() -> None:
|
|
||||||
import logging
|
|
||||||
|
|
||||||
from routstr.core.logging import SecurityFilter
|
|
||||||
|
|
||||||
record = logging.LogRecord(
|
|
||||||
name="test",
|
|
||||||
level=logging.ERROR,
|
|
||||||
pathname=__file__,
|
|
||||||
lineno=1,
|
|
||||||
msg="upstream failed",
|
|
||||||
args=(),
|
|
||||||
exc_info=None,
|
|
||||||
)
|
|
||||||
# Simulate an ``extra={"body_preview": ...}`` field carrying an org id.
|
|
||||||
setattr(record, "body_preview", RATE_LIMIT_MESSAGE)
|
|
||||||
|
|
||||||
assert SecurityFilter().filter(record) is True
|
|
||||||
redacted: str = getattr(record, "body_preview")
|
|
||||||
assert RAW_ORG_ID not in redacted
|
|
||||||
assert "org-[REDACTED]" in redacted
|
|
||||||
|
|
||||||
|
|
||||||
def test_security_filter_redacts_org_id_in_nested_extra() -> None:
|
|
||||||
import logging
|
|
||||||
|
|
||||||
from routstr.core.logging import SecurityFilter
|
|
||||||
|
|
||||||
record = logging.LogRecord(
|
|
||||||
name="test",
|
|
||||||
level=logging.ERROR,
|
|
||||||
pathname=__file__,
|
|
||||||
lineno=1,
|
|
||||||
msg="upstream failed",
|
|
||||||
args=(),
|
|
||||||
exc_info=None,
|
|
||||||
)
|
|
||||||
# Nested structures: dict containing a list containing the org id.
|
|
||||||
setattr(record, "body", {"error": {"messages": [RATE_LIMIT_MESSAGE]}})
|
|
||||||
|
|
||||||
assert SecurityFilter().filter(record) is True
|
|
||||||
serialized = json.dumps(getattr(record, "body"))
|
|
||||||
assert RAW_ORG_ID not in serialized
|
|
||||||
assert "org-[REDACTED]" in serialized
|
|
||||||
|
|
||||||
|
|
||||||
# --------------------------------------------------------------------------- #
|
|
||||||
# 5xx-wrapped rate limit through forward_upstream_error_response
|
|
||||||
# --------------------------------------------------------------------------- #
|
|
||||||
|
|
||||||
|
|
||||||
@pytest.mark.asyncio
|
|
||||||
async def test_5xx_wrapped_rate_limit_is_classified(
|
|
||||||
provider: BaseUpstreamProvider,
|
|
||||||
) -> None:
|
|
||||||
# Some providers wrap a rate-limit in a 5xx envelope; classification must
|
|
||||||
# key off the message marker, not only the 429 status.
|
|
||||||
body = json.dumps({"error": {"message": RATE_LIMIT_MESSAGE}}).encode()
|
|
||||||
upstream = _make_upstream_response(body=body, status_code=500)
|
|
||||||
|
|
||||||
response = await provider.forward_upstream_error_response(
|
|
||||||
_make_request(), "v1/chat/completions", upstream
|
|
||||||
)
|
|
||||||
|
|
||||||
assert response.status_code == 500
|
|
||||||
payload: dict[str, Any] = json.loads(bytes(response.body))
|
|
||||||
assert payload["error"]["code"] == UPSTREAM_RATE_LIMIT
|
|
||||||
serialized = json.dumps(payload)
|
|
||||||
assert RAW_ORG_ID not in serialized
|
|
||||||
assert "org-[REDACTED]" in serialized
|
|
||||||
|
|
||||||
|
|
||||||
# --------------------------------------------------------------------------- #
|
|
||||||
# Real proxy loop: structured error surfaced + reservation reverted once
|
|
||||||
# --------------------------------------------------------------------------- #
|
|
||||||
|
|
||||||
|
|
||||||
@pytest.mark.asyncio
|
|
||||||
async def test_proxy_loop_surfaces_rate_limit_and_reverts_once() -> None:
|
|
||||||
from routstr import proxy as proxy_module
|
|
||||||
from routstr.core.db import ApiKey
|
|
||||||
from routstr.core.exceptions import UpstreamError
|
|
||||||
|
|
||||||
info = classify_rate_limit(429, RATE_LIMIT_MESSAGE)
|
|
||||||
assert info is not None
|
|
||||||
|
|
||||||
key = ApiKey(hashed_key="rlkey", balance=10_000)
|
|
||||||
|
|
||||||
request = MagicMock()
|
|
||||||
request.method = "POST"
|
|
||||||
request.headers = {"authorization": "Bearer sk-rlkey"}
|
|
||||||
request.body = AsyncMock(return_value=b'{"model": "test-model"}')
|
|
||||||
request.state = MagicMock()
|
|
||||||
request.state.request_id = "req-rl"
|
|
||||||
|
|
||||||
upstream = MagicMock()
|
|
||||||
upstream.provider_type = "test"
|
|
||||||
upstream.prepare_headers = MagicMock(side_effect=lambda h: h)
|
|
||||||
upstream.forward_request = AsyncMock(
|
|
||||||
side_effect=UpstreamError(
|
|
||||||
f"Upstream error via litellm: {RATE_LIMIT_MESSAGE}",
|
|
||||||
status_code=429,
|
|
||||||
code=info.code,
|
|
||||||
details=info.as_details(),
|
|
||||||
)
|
|
||||||
)
|
|
||||||
|
|
||||||
session = MagicMock()
|
|
||||||
revert_mock = AsyncMock(return_value=True)
|
|
||||||
|
|
||||||
with (
|
|
||||||
patch.object(proxy_module, "get_model_instance", return_value=MagicMock()),
|
|
||||||
patch.object(proxy_module, "get_provider_for_model", return_value=[upstream]),
|
|
||||||
patch.object(
|
|
||||||
proxy_module, "get_max_cost_for_model", AsyncMock(return_value=1_000)
|
|
||||||
),
|
|
||||||
patch.object(
|
|
||||||
proxy_module,
|
|
||||||
"calculate_discounted_max_cost",
|
|
||||||
AsyncMock(return_value=1_000),
|
|
||||||
),
|
|
||||||
patch.object(proxy_module, "check_token_balance", MagicMock()),
|
|
||||||
patch.object(
|
|
||||||
proxy_module, "get_bearer_token_key", AsyncMock(return_value=key)
|
|
||||||
),
|
|
||||||
patch.object(proxy_module, "pay_for_request", AsyncMock(return_value=1_000)),
|
|
||||||
patch.object(proxy_module, "revert_pay_for_request", revert_mock),
|
|
||||||
):
|
|
||||||
response = await proxy_module.proxy(
|
|
||||||
request, "v1/chat/completions", session=session
|
|
||||||
)
|
|
||||||
|
|
||||||
# Original 429 status and the stable code/details survive to the client.
|
|
||||||
assert response.status_code == 429
|
|
||||||
payload: dict[str, Any] = json.loads(bytes(response.body))
|
|
||||||
assert payload["error"]["type"] == "upstream_error"
|
|
||||||
assert payload["error"]["code"] == UPSTREAM_RATE_LIMIT
|
|
||||||
assert payload["error"]["details"]["requested"] == 8929
|
|
||||||
serialized = json.dumps(payload)
|
|
||||||
assert RAW_ORG_ID not in serialized
|
|
||||||
assert "org-[REDACTED]" in serialized
|
|
||||||
# Single upstream failed -> reservation reverted exactly once (no double-charge).
|
|
||||||
revert_mock.assert_awaited_once_with(key, session, 1_000)
|
|
||||||
+91
-1058
File diff suppressed because it is too large
Load Diff
+24
-32
@@ -1,44 +1,36 @@
|
|||||||
# Routstr node admin UI
|
This is a [Next.js](https://nextjs.org) project bootstrapped with [`create-next-app`](https://nextjs.org/docs/app/api-reference/cli/create-next-app).
|
||||||
|
|
||||||
A [Next.js](https://nextjs.org) app (App Router, **static export**) that provides the
|
## Getting Started
|
||||||
admin dashboard for a `routstr-core` node: login, settings, providers, balances,
|
|
||||||
transactions, usage, and logs.
|
|
||||||
|
|
||||||
There is no separate web server in production. `next build` produces a fully static
|
First, run the development server:
|
||||||
export (`next.config.ts` sets `output: 'export'`), and the FastAPI backend serves it
|
|
||||||
directly from `../ui_out/` (see `routstr/core/main.py`). So the UI and the API are
|
|
||||||
served from the **same origin** in production.
|
|
||||||
|
|
||||||
## Developing the UI (hot reload)
|
```bash
|
||||||
|
npm run dev
|
||||||
|
# or
|
||||||
|
yarn dev
|
||||||
|
# or
|
||||||
|
pnpm dev
|
||||||
|
# or
|
||||||
|
bun dev
|
||||||
|
```
|
||||||
|
|
||||||
The everyday loop runs two processes side by side — you do **not** rebuild the static
|
Open [http://localhost:3000](http://localhost:3000) with your browser to see the result.
|
||||||
export while developing:
|
|
||||||
|
|
||||||
1. Start the backend on `:8000` — from the repo root: `make docker-up` (or
|
You can start editing the page by modifying `app/page.tsx`. The page auto-updates as you edit the file.
|
||||||
`uvicorn routstr.core.main:app --reload`).
|
|
||||||
2. Start the Next.js dev server on `:3000` — from the repo root: `make ui-dev`
|
|
||||||
(or `cd ui && pnpm dev`). Edits hot-reload instantly.
|
|
||||||
|
|
||||||
Open http://localhost:3000. With no `NEXT_PUBLIC_API_URL` set, the UI falls back to
|
This project uses [`next/font`](https://nextjs.org/docs/app/building-your-application/optimizing/fonts) to automatically optimize and load [Geist](https://vercel.com/font), a new font family for Vercel.
|
||||||
`http://127.0.0.1:8000` in development (see `lib/api/services/configuration.ts`), so it
|
|
||||||
talks to the local backend out of the box.
|
|
||||||
|
|
||||||
Because dev is cross-origin (`:3000` → `:8000`), it relies on the backend's CORS
|
## Learn More
|
||||||
allowing the UI origin. The default `cors_origins` is `["*"]`; if you tighten CORS,
|
|
||||||
keep `http://localhost:3000` allowed for development.
|
|
||||||
|
|
||||||
## Building the integrated/static UI (what production serves)
|
To learn more about Next.js, take a look at the following resources:
|
||||||
|
|
||||||
To produce the bundle that FastAPI serves from `../ui_out/`:
|
- [Next.js Documentation](https://nextjs.org/docs) - learn about Next.js features and API.
|
||||||
|
- [Learn Next.js](https://nextjs.org/learn) - an interactive Next.js tutorial.
|
||||||
|
|
||||||
- `make ui-build` — builds with local Node/pnpm (`scripts/build-ui.sh`), then moves
|
You can check out [the Next.js GitHub repository](https://github.com/vercel/next.js) - your feedback and contributions are welcome!
|
||||||
`ui/out/*` to `../ui_out/`.
|
|
||||||
- `make ui-build-docker` — same, but inside Docker (no local Node needed).
|
|
||||||
|
|
||||||
`NEXT_PUBLIC_*` variables are read from the repo-root `.env` at build time and baked in.
|
## Deploy on Vercel
|
||||||
For a same-origin deployment leave `NEXT_PUBLIC_API_URL` empty (relative paths); the UI
|
|
||||||
uses `window.location.origin` at runtime. After building, start the backend and open
|
|
||||||
http://localhost:8000 — the dashboard is served at `/` and `/admin`.
|
|
||||||
|
|
||||||
If `../ui_out/` does not exist, the backend logs a warning at startup and serves the API
|
The easiest way to deploy your Next.js app is to use the [Vercel Platform](https://vercel.com/new?utm_medium=default-template&filter=next.js&utm_source=create-next-app&utm_campaign=create-next-app-readme) from the creators of Next.js.
|
||||||
only (hitting a UI route returns a small JSON fallback instead of the dashboard).
|
|
||||||
|
Check out our [Next.js deployment documentation](https://nextjs.org/docs/app/building-your-application/deploying) for more details.
|
||||||
|
|||||||
@@ -297,13 +297,16 @@ export default function ProvidersPage() {
|
|||||||
};
|
};
|
||||||
|
|
||||||
const toggleProviderExpansion = (providerId: number) => {
|
const toggleProviderExpansion = (providerId: number) => {
|
||||||
if (expandedProviders.has(providerId)) {
|
const newExpanded = new Set(expandedProviders);
|
||||||
setExpandedProviders(new Set());
|
if (newExpanded.has(providerId)) {
|
||||||
|
newExpanded.delete(providerId);
|
||||||
|
} else {
|
||||||
|
newExpanded.add(providerId);
|
||||||
|
}
|
||||||
|
setExpandedProviders(newExpanded);
|
||||||
|
if (!newExpanded.has(providerId)) {
|
||||||
setViewingModels(null);
|
setViewingModels(null);
|
||||||
} else {
|
} else {
|
||||||
// Accordion: only one provider open at a time so switching to another
|
|
||||||
// provider's models auto-collapses the previously expanded one.
|
|
||||||
setExpandedProviders(new Set([providerId]));
|
|
||||||
setViewingModels(providerId);
|
setViewingModels(providerId);
|
||||||
}
|
}
|
||||||
};
|
};
|
||||||
|
|||||||
@@ -118,26 +118,6 @@ export function ProviderFormFields({
|
|||||||
/>
|
/>
|
||||||
)}
|
)}
|
||||||
|
|
||||||
<div className='grid gap-2'>
|
|
||||||
<Label htmlFor={`${idPrefix}slug`}>
|
|
||||||
Slug {mode === 'create' ? '(optional, auto-generated)' : ''}
|
|
||||||
</Label>
|
|
||||||
<Input
|
|
||||||
id={`${idPrefix}slug`}
|
|
||||||
value={formData.slug || ''}
|
|
||||||
onChange={(e) =>
|
|
||||||
setFormData((prev) => ({
|
|
||||||
...prev,
|
|
||||||
slug: e.target.value || undefined,
|
|
||||||
}))
|
|
||||||
}
|
|
||||||
placeholder='e.g. openai-prod'
|
|
||||||
/>
|
|
||||||
<p className='text-muted-foreground text-xs'>
|
|
||||||
Stable external key used to update this provider via the admin API.
|
|
||||||
</p>
|
|
||||||
</div>
|
|
||||||
|
|
||||||
<div className='grid gap-2'>
|
<div className='grid gap-2'>
|
||||||
<Label htmlFor={`${idPrefix}base_url`}>Base URL</Label>
|
<Label htmlFor={`${idPrefix}base_url`}>Base URL</Label>
|
||||||
<Input
|
<Input
|
||||||
|
|||||||
@@ -14,7 +14,6 @@ export const ProviderTypeSchema = z.object({
|
|||||||
|
|
||||||
export const UpstreamProviderSchema = z.object({
|
export const UpstreamProviderSchema = z.object({
|
||||||
id: z.number(),
|
id: z.number(),
|
||||||
slug: z.string().nullable().optional(),
|
|
||||||
provider_type: z.string(),
|
provider_type: z.string(),
|
||||||
base_url: z.string(),
|
base_url: z.string(),
|
||||||
api_key: z.string().optional(),
|
api_key: z.string().optional(),
|
||||||
@@ -32,7 +31,6 @@ export const CreateUpstreamProviderSchema = z.object({
|
|||||||
enabled: z.boolean().default(true),
|
enabled: z.boolean().default(true),
|
||||||
provider_fee: z.number().optional(),
|
provider_fee: z.number().optional(),
|
||||||
provider_settings: z.record(z.string(), z.any()).nullable().optional(),
|
provider_settings: z.record(z.string(), z.any()).nullable().optional(),
|
||||||
slug: z.string().optional(),
|
|
||||||
});
|
});
|
||||||
|
|
||||||
export const UpdateUpstreamProviderSchema = z.object({
|
export const UpdateUpstreamProviderSchema = z.object({
|
||||||
@@ -43,7 +41,6 @@ export const UpdateUpstreamProviderSchema = z.object({
|
|||||||
enabled: z.boolean().optional(),
|
enabled: z.boolean().optional(),
|
||||||
provider_fee: z.number().optional(),
|
provider_fee: z.number().optional(),
|
||||||
provider_settings: z.record(z.string(), z.any()).nullable().optional(),
|
provider_settings: z.record(z.string(), z.any()).nullable().optional(),
|
||||||
slug: z.string().optional(),
|
|
||||||
});
|
});
|
||||||
|
|
||||||
export const AdminModelPricingSchema = z.object({
|
export const AdminModelPricingSchema = z.object({
|
||||||
|
|||||||
Reference in New Issue
Block a user