mirror of
https://github.com/Routstr/routstr-core.git
synced 2026-10-05 20:28:23 +00:00
Paperclip Deployment Engineerandthefux
9d0cd41e5a
CORE-UPSTREAM-5XX-NOT-NODE-DOWN
An upstream-attributable failure (provider 5xx, EHBP timeout, transport error
after a Cashu token was redeemed) was forwarded to the caller as the
provider's own 5xx. Callers read that as *this node* being down: they marked
the node unhealthy, dropped it from rotation, or refused to retry an upstream
blip that the node had already retried across candidates.
The node is healthy in those cases — it accepted the request, authenticated
it, reserved payment, tried every candidate, and reverted the reservation.
That is now visible from the response alone:
status 424 (UPSTREAM_ERROR_STATUS)
error.code UPSTREAM_UNAVAILABLE
header X-Routstr-Error-Scope: upstream
error.upstream_status / error.details.upstream_status
the provider's own status, preserved
Applied in routstr/upstream/base.py (forward_upstream_error_response and the
post-redemption x-cashu paths for both chat-completions and responses),
routstr/payment/helpers.py (create_error_response / create_upstream_error_response),
routstr/proxy.py (424 is retryable across candidates on the bearer, EHBP and
unauthenticated-GET loops), routstr/upstream/ehbp.py and
routstr/upstream/tinfoil.py (attestation host). New module
routstr/core/error_scope.py holds the contract constants and the mapping.
Deliberately unchanged:
* rate limits keep 429 + UPSTREAM_RATE_LIMIT (including a rate limit wrapped
in a provider 5xx envelope: status 429, code unchanged) — the retry hint is
worth more than the status class;
* provider-side 4xx passes through unchanged;
* genuine node faults stay 500 and carry no scope header (UpstreamError gained
scope=, set to "node" on internal-exception paths) so "node broken" is still
distinguishable from "upstream broken".
The new status is exported through CORS (x-routstr-error-scope) so browser
clients can read the attribution.
Tests: 15 stale assertions of the old 5xx contract updated (renames keep their
intent: refund still happens, bodies still redacted, pinned requests still do
not fall back), plus new acceptance coverage for 424 + scope header +
upstream_status on the provider, bearer, x-cashu, /v1/messages, EHBP and
unauthenticated-GET paths, node faults staying 500 with no header, and failover
past a 424 to a healthy candidate returning 200.
Docs: docs/api/errors.md (status table, new "Upstream attribution" section,
upstream error examples, retry list now includes 424), docs/api/overview.md
and docs/api/endpoints.md.
Full unit suite: 1649 passed, 1 skipped.
FastAPI Async Unit Tests
This directory contains async unit tests for the Routstr proxy FastAPI application.
Installation
First, ensure you have the development dependencies installed:
uv pip install -e ".[dev]"
Running Tests
To run all tests:
pytest
To run tests with coverage:
pytest --cov=routstr --cov-report=html
To run specific test files:
pytest tests/test_main.py
pytest tests/test_models.py
pytest tests/test_proxy.py
To run only async tests:
pytest -m asyncio
Test Structure
conftest.py- Pytest fixtures and configurationtest_main.py- Tests for main app endpointstest_account.py- Tests for wallet/account management endpointstest_proxy.py- Tests for the proxy functionality with mocked upstreamtest_models.py- Tests for model pricing and data structures
Key Fixtures
async_client- Async HTTP client for testing FastAPI endpointstest_session- In-memory SQLite database session for teststest_api_key- Pre-configured API key with balanceapi_key_with_balance- API key with sufficient balance for proxy tests
Environment Variables
The tests automatically set up required environment variables in conftest.py. No manual configuration needed.
Writing New Tests
- Use
@pytest.mark.asynciofor async tests - Use the provided fixtures for database and client access
- Mock external dependencies (like upstream API calls)
- Test both success and error cases
- Verify database state changes when applicable