mirror of
https://github.com/Routstr/routstr-core.git
synced 2026-10-05 12:28:22 +00:00
feat(systemone): add TypeSafe System One decision endpoint
Support POST /v1/systemone -- `{state, model, questions}` ->
`{answers, usage}` -- as a first-class Routstr endpoint with a dedicated
upstream provider.
- routstr/upstream/typesafe.py: new TypeSafeUpstreamProvider with a fixed
base URL, a priced model catalog assembled from TypeSafe's
pricing-less GET /v1/models, and fail-closed catalog errors.
- Register the provider and seed it from TYPESAFE_API_KEY.
- proxy: admit POST systemone and route it through the response-usage
settlement branch (plus the X-Cashu gate) so ecash payments are charged
from usage instead of being served free.
- Document the endpoint and add unit + end-to-end integration coverage.
SYSTEMONE_TYPESAFE.md records the change set and test report.
This commit is contained in:
redshift
@@ -0,0 +1,161 @@
|
||||
# TypeSafe `/v1/systemone` support — changes & test report
|
||||
|
||||
**Repo:** `~/projects/routstr-core` **Branch:** `feat/systemone-typesafe` (cut from `origin/main` @ `c1b610d4`)
|
||||
**Date:** 2026-09-19 **Status:** code complete, tests green, **not deployed** (no container restart)
|
||||
|
||||
---
|
||||
|
||||
## 1. What was added
|
||||
|
||||
Support for TypeSafe's System One decision endpoint — `POST /v1/systemone`
|
||||
(`{state, model, questions}` → `{model, answers, usage}`) — as a first-class
|
||||
Routstr endpoint with a dedicated upstream provider.
|
||||
|
||||
| File | Change |
|
||||
|---|---|
|
||||
| `routstr/upstream/typesafe.py` | **new** — `TypeSafeUpstreamProvider`: fixed base URL `https://api.typesafe.ai/v1`, `fetch_models()` maps TypeSafe's pricing-less `GET /v1/models` onto priced `Model` objects ($0.042/M input, $0 output, 64k context, `text->decisions`), catalog failures fail closed |
|
||||
| `routstr/upstream/__init__.py` | provider registered → appears in the admin UI provider-type dropdown |
|
||||
| `routstr/proxy.py` | `systemone` added to `_ALLOWED_ENDPOINTS` (POST only) |
|
||||
| `routstr/upstream/base.py` | **two settlement fixes** (see below) |
|
||||
| `routstr/upstream/helpers.py` | `TYPESAFE_API_KEY` env seeding (empty provider table only) |
|
||||
| `docs/api/overview.md`, `docs/api/endpoints.md` | endpoint documented |
|
||||
|
||||
### The two `base.py` fixes matter most
|
||||
|
||||
1. `_x_cashu_path_has_settlement_handler` — now admits `systemone`, so ecash
|
||||
payments settle and refund the delta instead of being rejected with
|
||||
`x_cashu_unsupported_endpoint`.
|
||||
2. `forward_request`'s response-settlement branch — now includes
|
||||
`systemone`. **Without this the endpoint served free**: the request fell
|
||||
through to the generic streaming path, whose `_finalize_generic_streaming_payment`
|
||||
releases the reservation *without charging* (usage never read).
|
||||
|
||||
---
|
||||
|
||||
## 2. Test results
|
||||
|
||||
Both runs executed against the working tree on the branch above.
|
||||
|
||||
### Targeted suites — 16/16 pass
|
||||
|
||||
```
|
||||
pytest tests/unit/test_typesafe_integration.py tests/integration/test_systemone.py
|
||||
→ 16 passed
|
||||
```
|
||||
|
||||
Covers: allowlist admit/refuse (incl. `systemonedump`, `v1/systemone/secret`,
|
||||
traversal spellings, GET/DELETE), x-cashu gate, provider metadata,
|
||||
`fetch_models` pricing + error handling, and an end-to-end proxied request that
|
||||
asserts the upstream hop is exactly `https://api.typesafe.ai/v1/systemone` and
|
||||
that billing settles from usage (1000 input × 0.001 sats = 1000 msats, output
|
||||
free) — plus an X-Cashu settle-and-refund case.
|
||||
|
||||
### Full unit suite
|
||||
|
||||
```
|
||||
pytest tests/unit
|
||||
→ 1584 passed, 9 failed
|
||||
```
|
||||
|
||||
### Full integration suite
|
||||
|
||||
```
|
||||
pytest tests/integration -m "not requires_docker"
|
||||
→ 492 passed, 13 skipped, 0 failed
|
||||
```
|
||||
|
||||
### Lint / types
|
||||
|
||||
```
|
||||
ruff check routstr tests → All checks passed!
|
||||
mypy routstr/upstream/typesafe.py → only a pre-existing error in routstr/nostr/discovery.py
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 3. The 9 unit failures are pre-existing (proven)
|
||||
|
||||
Failing: `test_cashu_httpx_compat.py` (3), `test_fee_payout_migration.py` (3),
|
||||
`test_mint_url_migration.py` (1), `test_provider_id_migration.py` (2).
|
||||
|
||||
Verified by stashing the branch (`git stash push -u`) and re-running those four
|
||||
files against pristine `origin/main`:
|
||||
|
||||
```
|
||||
→ 9 failed, 10 passed
|
||||
```
|
||||
|
||||
Identical failures with none of this work applied. Causes:
|
||||
|
||||
* **httpx compat (3)** — installed httpx no longer accepts the `proxies=` kwarg
|
||||
the shim probes for; version drift, unrelated to this branch.
|
||||
* **migration tests (6)** — the tests shell out to `alembic upgrade`, whose
|
||||
subprocess imports the app, whose file logger opens
|
||||
`logs/app_2026-09-19.log` — a **root-owned file created by the running
|
||||
Docker container**, unwritable by the `debian` user → `PermissionError` →
|
||||
alembic exits non-zero. Environment artifact, not code.
|
||||
|
||||
---
|
||||
|
||||
## 4. Environment note (how to reproduce these runs)
|
||||
|
||||
Running pytest **from the repo directory** currently fails at import:
|
||||
|
||||
```
|
||||
ValueError: Unable to configure handler 'file'
|
||||
← PermissionError: .../logs/app_2026-09-19.log
|
||||
```
|
||||
|
||||
The running container (root) owns today's log file. The runs above therefore
|
||||
used a scratch cwd so the logger writes elsewhere, with the repo on
|
||||
`PYTHONPATH`:
|
||||
|
||||
```bash
|
||||
mkdir -p /tmp/routstr-test && cd /tmp/routstr-test
|
||||
PYTHONPATH=$HOME/projects/routstr-core \
|
||||
$HOME/projects/routstr-core/.venv/bin/python -m pytest \
|
||||
$HOME/projects/routstr-core/tests/integration -m "not requires_docker" -q
|
||||
```
|
||||
|
||||
To run in-repo instead (`make test-unit`), either run as root, or move today's
|
||||
root-owned log aside first — the container keeps writing to its open file
|
||||
descriptor, so `mv` is safe and lossless:
|
||||
|
||||
```bash
|
||||
mv logs/app_$(date -u +%F).log logs/app_$(date -u +%F).log.root-owned
|
||||
```
|
||||
|
||||
Also note `nostr-sdk==0.45.1` was installed into `.venv` during this work (it
|
||||
was missing, and blocks every import of the package).
|
||||
|
||||
---
|
||||
|
||||
## 5. Enabling it on the node (when you add the API key)
|
||||
|
||||
The provider table is non-empty on the live node, so env seeding won't fire —
|
||||
add it explicitly:
|
||||
|
||||
1. Admin UI → Providers → *TypeSafe* (base URL is fixed to
|
||||
`https://api.typesafe.ai/v1`), paste your `api.typesafe.ai` key.
|
||||
2. Or: `POST /admin/api/upstream-providers` with
|
||||
`{"provider_type": "typesafe", "base_url": "https://api.typesafe.ai/v1", "api_key": "<key>"}`.
|
||||
3. `jev-latest` / `jev-preview` are then catalogued automatically with the
|
||||
built-in rates; override the model row if TypeSafe changes pricing.
|
||||
4. Client call: `POST {node}/v1/systemone` with `{state, model, questions}`.
|
||||
|
||||
Containers were **not** restarted and nothing was deployed.
|
||||
|
||||
---
|
||||
|
||||
## 6. Caveats / not yet verified
|
||||
|
||||
* **No live call has been made** — no TypeSafe API key was available during
|
||||
this work, so the upstream contract is implemented from
|
||||
`docs.typesafe.ai` and the OpenRouter model metadata. Specifically
|
||||
unverified against the live API: the exact `GET /v1/models` envelope
|
||||
(code accepts `{"models": [...]}` and a bare list; entries keyed by `name`
|
||||
or `id`) and the `release_date` format.
|
||||
* The `usage` shape (`{input_tokens, output_tokens}`) is already the canonical
|
||||
billing shape, so settlement needed no dialect handling.
|
||||
* Not covered by tests: TypeSafe's `529 Overloaded` status (treated as a
|
||||
generic 5xx; single-provider failover has nothing to fall back to).
|
||||
@@ -200,6 +200,62 @@ POST /v1/embeddings
|
||||
}
|
||||
```
|
||||
|
||||
## System One (TypeSafe Decisions)
|
||||
|
||||
### Evaluate State
|
||||
|
||||
Evaluate a state against typed questions (noul / choice / score) on a TypeSafe
|
||||
System One decision model (e.g. `jev-latest`). Requires a `typesafe` upstream
|
||||
provider on the node.
|
||||
|
||||
```http
|
||||
POST /v1/systemone
|
||||
```
|
||||
|
||||
**Request Body:**
|
||||
|
||||
```json
|
||||
{
|
||||
"model": "jev-latest",
|
||||
"state": "Help! My payouts have been failing for 3 days.",
|
||||
"questions": {
|
||||
"is_urgent": {
|
||||
"type": "noul",
|
||||
"instructions": "Does this convey urgency?"
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**Parameters:**
|
||||
|
||||
| Parameter | Type | Required | Default | Description |
|
||||
|-----------|------|----------|---------|-------------|
|
||||
| `model` | string | Yes | - | TypeSafe model or alias (e.g. `jev-latest`) |
|
||||
| `state` | string/object/array | Yes | - | Content to evaluate |
|
||||
| `questions` | map<string, Question> | Yes | - | Typed questions; answers keyed identically |
|
||||
|
||||
**Response:**
|
||||
|
||||
```json
|
||||
{
|
||||
"model": "jev-latest",
|
||||
"answers": {
|
||||
"is_urgent": {
|
||||
"type": "noul",
|
||||
"noul": 0.95
|
||||
}
|
||||
},
|
||||
"usage": {
|
||||
"input_tokens": 312,
|
||||
"output_tokens": 48
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Billing is input-token based (output tokens are free on Jev); the response's
|
||||
`usage` is the settlement seam, exactly like embeddings.
|
||||
|
||||
## Images (Coming Soon)
|
||||
|
||||
### Create Image
|
||||
|
||||
@@ -104,6 +104,7 @@ Standard OpenAI-compatible endpoints:
|
||||
- **Responses**: `/v1/responses`
|
||||
- **Chat Completions**: `/v1/chat/completions`
|
||||
- **Embeddings**: `/v1/embeddings`
|
||||
- **System One**: `/v1/systemone` (TypeSafe decision models)
|
||||
- **Completions**: `/v1/completions` *(planned)*
|
||||
- **Images**: `/v1/images/generations` *(planned)*
|
||||
- **Audio**: `/v1/audio/transcriptions` *(planned)*
|
||||
|
||||
@@ -262,6 +262,10 @@ _ALLOWED_ENDPOINTS: dict[str, frozenset[str]] = {
|
||||
"responses": frozenset({"POST"}),
|
||||
"messages": frozenset({"POST"}),
|
||||
"embeddings": frozenset({"POST"}),
|
||||
# TypeSafe System One decision endpoint: POST {state, model, questions}
|
||||
# -> {answers, usage}. Non-streaming, JSON in/out; billed from the
|
||||
# response's usage exactly like embeddings.
|
||||
"systemone": frozenset({"POST"}),
|
||||
"models": frozenset({"GET"}),
|
||||
"attestation": frozenset({"GET"}),
|
||||
"tee/attestation": frozenset({"GET"}),
|
||||
|
||||
@@ -12,6 +12,7 @@ from .perplexity import PerplexityUpstreamProvider
|
||||
from .ppqai import PPQAIUpstreamProvider
|
||||
from .routstr import RoutstrUpstreamProvider
|
||||
from .tinfoil import TinfoilUpstreamProvider
|
||||
from .typesafe import TypeSafeUpstreamProvider
|
||||
from .xai import XAIUpstreamProvider
|
||||
|
||||
upstream_provider_classes: list[type[BaseUpstreamProvider]] = [
|
||||
@@ -28,6 +29,7 @@ upstream_provider_classes: list[type[BaseUpstreamProvider]] = [
|
||||
PPQAIUpstreamProvider,
|
||||
RoutstrUpstreamProvider,
|
||||
TinfoilUpstreamProvider,
|
||||
TypeSafeUpstreamProvider,
|
||||
XAIUpstreamProvider,
|
||||
]
|
||||
"""List of all upstream classes"""
|
||||
|
||||
@@ -303,7 +303,7 @@ def _openai_completion_path(path: str) -> str | None:
|
||||
def _x_cashu_path_has_settlement_handler(path: str) -> bool:
|
||||
canonical = path.rstrip("/")
|
||||
return _openai_completion_path(canonical) is not None or canonical.endswith(
|
||||
("embeddings", "messages", "messages/count_tokens")
|
||||
("embeddings", "messages", "messages/count_tokens", "systemone")
|
||||
)
|
||||
|
||||
|
||||
@@ -3155,6 +3155,7 @@ class BaseUpstreamProvider:
|
||||
or path.endswith("embeddings")
|
||||
or path.endswith("messages")
|
||||
or path.endswith("messages/count_tokens")
|
||||
or path.endswith("systemone")
|
||||
):
|
||||
if path.endswith("messages"):
|
||||
client_wants_streaming = False
|
||||
|
||||
@@ -273,6 +273,7 @@ async def _seed_providers_from_settings(
|
||||
("FIREWORKS_API_KEY", "fireworks", None, None),
|
||||
("XAI_API_KEY", "xai", None, None),
|
||||
("TINFOIL_API_KEY", "tinfoil", None, None),
|
||||
("TYPESAFE_API_KEY", "typesafe", None, None),
|
||||
]
|
||||
|
||||
for env_key, provider_type, _, _ in env_mappings:
|
||||
|
||||
@@ -0,0 +1,176 @@
|
||||
"""Upstream provider for the TypeSafe System One API.
|
||||
|
||||
TypeSafe serves "System One" decision models (Jev) at a dedicated
|
||||
``POST /v1/systemone`` endpoint: the request carries a ``state`` and a map of
|
||||
typed ``questions`` (noul / choice / score), and the response returns one
|
||||
``answers`` entry per question plus a flat ``usage`` object
|
||||
(``{"input_tokens": n, "output_tokens": n}``). That usage shape is exactly
|
||||
what :func:`routstr.payment.usage.normalize_usage` already parses, so billing
|
||||
needs no dialect handling — the provider's job is catalog assembly (TypeSafe's
|
||||
``GET /v1/models`` lists model names but no prices) and standard forwarding.
|
||||
|
||||
Rates below are USD per token, matching the prices TypeSafe publishes at
|
||||
https://docs.typesafe.ai/models (input is charged; output tokens are free).
|
||||
Because TypeSafe's model listing does not carry pricing, the operator's DB
|
||||
model row is authoritative whenever one exists; these rates seed the row.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import TYPE_CHECKING
|
||||
|
||||
import httpx
|
||||
|
||||
from ..payment.models import Architecture, Model, Pricing, TopProvider
|
||||
from .base import BaseUpstreamProvider
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from ..core.db import UpstreamProviderRow
|
||||
|
||||
|
||||
# USD per token, keyed by the exact `model` value TypeSafe accepts. Aliases
|
||||
# (jev-latest -> jev-1.13.0) resolve upstream, so one entry per alias keeps
|
||||
# every advertised id priced even if the alias target moves.
|
||||
_TYPESAFE_RATES: dict[str, tuple[float, float]] = {
|
||||
"jev-latest": (0.042 / 1_000_000, 0.0),
|
||||
"jev-preview": (0.042 / 1_000_000, 0.0),
|
||||
}
|
||||
|
||||
_DEFAULT_RATE = (0.042 / 1_000_000, 0.0)
|
||||
|
||||
# Jev ingests state once and evaluates all questions against it in parallel.
|
||||
# The 64k budget covers state + all questions combined; 32k applies to state +
|
||||
# the single longest question. 64k is the safe reservation context.
|
||||
_TYPESAFE_CONTEXT_LENGTH = 64_000
|
||||
|
||||
|
||||
class TypeSafeUpstreamProvider(BaseUpstreamProvider):
|
||||
"""Upstream provider for the TypeSafe System One decision API.
|
||||
|
||||
TypeSafe exposes one evaluation endpoint (``POST /v1/systemone``) shared
|
||||
by every model; the request's ``model`` field selects which one answers.
|
||||
The generic chat-forwarding machinery in the base class handles the
|
||||
request/response plumbing unchanged — the model id sits in the top-level
|
||||
``model`` field like any OpenAI-compatible API, and the response's
|
||||
``usage`` is already in the canonical billing shape.
|
||||
"""
|
||||
|
||||
provider_type = "typesafe"
|
||||
default_base_url = "https://api.typesafe.ai/v1"
|
||||
platform_url = "https://docs.typesafe.ai"
|
||||
|
||||
def __init__(self, api_key: str, provider_fee: float = 1.0):
|
||||
super().__init__(
|
||||
base_url=self.default_base_url,
|
||||
api_key=api_key,
|
||||
provider_fee=provider_fee,
|
||||
)
|
||||
|
||||
@classmethod
|
||||
def _build_from_row(
|
||||
cls, provider_row: "UpstreamProviderRow"
|
||||
) -> "TypeSafeUpstreamProvider":
|
||||
return cls(
|
||||
api_key=provider_row.api_key,
|
||||
provider_fee=provider_row.provider_fee,
|
||||
)
|
||||
|
||||
@classmethod
|
||||
def get_provider_metadata(cls) -> dict[str, object]:
|
||||
return {
|
||||
"id": cls.provider_type,
|
||||
"name": "TypeSafe",
|
||||
"default_base_url": cls.default_base_url,
|
||||
"fixed_base_url": True,
|
||||
"platform_url": cls.platform_url,
|
||||
"can_create_account": False,
|
||||
"can_topup": False,
|
||||
"can_show_balance": False,
|
||||
}
|
||||
|
||||
def transform_model_name(self, model_id: str) -> str:
|
||||
"""Strip a ``typesafe/`` prefix if one was used to namespace the id."""
|
||||
return model_id.removeprefix("typesafe/")
|
||||
|
||||
async def fetch_models(self) -> list[Model]:
|
||||
"""Fetch the model list TypeSafe's account can call.
|
||||
|
||||
``GET /v1/models`` requires the provider's API key and returns
|
||||
``{"models": [{"name", "description", "release_date"}, ...]}`` —
|
||||
aliases only, with no pricing or context fields. Prices come from the
|
||||
table above; the operator's DB model row overrides this pricing
|
||||
whenever one exists.
|
||||
"""
|
||||
url = f"{self.base_url}/models"
|
||||
headers = {"Authorization": f"Bearer {self.api_key}"}
|
||||
try:
|
||||
async with httpx.AsyncClient(timeout=30.0) as client:
|
||||
response = await client.get(url, headers=headers)
|
||||
response.raise_for_status()
|
||||
data = response.json()
|
||||
|
||||
entries = data.get("models", []) if isinstance(data, dict) else data
|
||||
if not isinstance(entries, list):
|
||||
return []
|
||||
|
||||
models: list[Model] = []
|
||||
seen: set[str] = set()
|
||||
for entry in entries:
|
||||
if not isinstance(entry, dict):
|
||||
continue
|
||||
name = entry.get("name")
|
||||
if not isinstance(name, str) or not name or name in seen:
|
||||
continue
|
||||
seen.add(name)
|
||||
|
||||
rate = _TYPESAFE_RATES.get(name, _DEFAULT_RATE)
|
||||
# An unlisted model is priced at the default Jev rate, but a
|
||||
# missing entry in the rate table is still imported enabled:
|
||||
# TypeSafe only lists models the account may call, and the
|
||||
# operator's DB row overrides this pricing anyway.
|
||||
created = 0
|
||||
release_date = entry.get("release_date")
|
||||
if isinstance(release_date, str):
|
||||
try:
|
||||
from datetime import datetime
|
||||
|
||||
created = int(
|
||||
datetime.fromisoformat(
|
||||
release_date.replace("Z", "+00:00")
|
||||
).timestamp()
|
||||
)
|
||||
except ValueError:
|
||||
created = 0
|
||||
|
||||
description = entry.get("description")
|
||||
if not isinstance(description, str) or not description:
|
||||
description = f"TypeSafe System One model {name}"
|
||||
|
||||
models.append(
|
||||
Model(
|
||||
id=name,
|
||||
name=name,
|
||||
created=created,
|
||||
description=description,
|
||||
context_length=_TYPESAFE_CONTEXT_LENGTH,
|
||||
architecture=Architecture(
|
||||
modality="text->decisions",
|
||||
input_modalities=["text"],
|
||||
output_modalities=["decisions"],
|
||||
tokenizer="Other",
|
||||
instruct_type=None,
|
||||
),
|
||||
pricing=Pricing(
|
||||
prompt=rate[0],
|
||||
completion=rate[1],
|
||||
),
|
||||
top_provider=TopProvider(
|
||||
context_length=_TYPESAFE_CONTEXT_LENGTH,
|
||||
),
|
||||
)
|
||||
)
|
||||
return models
|
||||
except Exception:
|
||||
# Catalog fetch failures (bad key, outage) must not break provider
|
||||
# initialization; cached/DB models keep serving.
|
||||
return []
|
||||
@@ -0,0 +1,221 @@
|
||||
"""Integration test: POST /v1/systemone proxied and billed end-to-end.
|
||||
|
||||
The TypeSafe decision endpoint shares the request plumbing with embeddings:
|
||||
model id in the top-level ``model`` field, flat ``usage`` in the response.
|
||||
This test pins the whole path — allowlist, auth, forwarding, and settlement
|
||||
billed from the response's usage — with only the network hop mocked.
|
||||
"""
|
||||
|
||||
import json
|
||||
from typing import Any, AsyncGenerator
|
||||
from unittest.mock import AsyncMock, patch
|
||||
|
||||
import httpx
|
||||
import pytest
|
||||
from httpx import AsyncClient
|
||||
from sqlmodel.ext.asyncio.session import AsyncSession
|
||||
|
||||
from routstr.payment.models import Architecture, Model, Pricing
|
||||
from routstr.proxy import refresh_model_maps
|
||||
from routstr.upstream.base import BaseUpstreamProvider
|
||||
|
||||
TYPESAFE_BASE_URL = "https://api.typesafe.ai/v1"
|
||||
|
||||
SYSTEMONE_REQUEST = {
|
||||
"model": "jev-latest",
|
||||
"state": "Help! My payouts have been failing for 3 days.",
|
||||
"questions": {
|
||||
"is_urgent": {"type": "noul", "instructions": "Does this convey urgency?"}
|
||||
},
|
||||
}
|
||||
|
||||
SYSTEMONE_RESPONSE = {
|
||||
"model": "jev-latest",
|
||||
"answers": {
|
||||
"is_urgent": {"type": "noul", "noul": 0.95},
|
||||
},
|
||||
"usage": {"input_tokens": 1000, "output_tokens": 200},
|
||||
}
|
||||
|
||||
|
||||
class _StaticTypeSafeProvider(BaseUpstreamProvider):
|
||||
"""Upstream provider with a fixed TypeSafe model catalog."""
|
||||
|
||||
def __init__(self, base_url: str, api_key: str, fee: float, model: Model) -> None:
|
||||
super().__init__(base_url, api_key, fee)
|
||||
self.provider_type = "typesafe"
|
||||
self._static_model = model
|
||||
|
||||
def get_cached_models(self) -> list[Model]:
|
||||
return [self._static_model]
|
||||
|
||||
async def refresh_models_cache(self) -> None:
|
||||
pass
|
||||
|
||||
|
||||
def _jev_model(prompt_sats: float = 0.001, completion_sats: float = 0.0) -> Model:
|
||||
return Model(
|
||||
id="jev-latest",
|
||||
name="jev-latest",
|
||||
created=1,
|
||||
description="TypeSafe System One decision model",
|
||||
context_length=64_000,
|
||||
architecture=Architecture(
|
||||
modality="text->decisions",
|
||||
input_modalities=["text"],
|
||||
output_modalities=["decisions"],
|
||||
tokenizer="Other",
|
||||
instruct_type=None,
|
||||
),
|
||||
pricing=Pricing(
|
||||
prompt=prompt_sats, completion=completion_sats, max_cost=50.0
|
||||
),
|
||||
sats_pricing=Pricing(
|
||||
prompt=prompt_sats, completion=completion_sats, max_cost=50.0
|
||||
),
|
||||
)
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
async def typesafe_provider_maps(
|
||||
patched_db_engine: None,
|
||||
) -> AsyncGenerator[_StaticTypeSafeProvider, None]:
|
||||
"""Install a TypeSafe provider with a priced jev-latest model."""
|
||||
provider = _StaticTypeSafeProvider(
|
||||
TYPESAFE_BASE_URL,
|
||||
"key-typesafe",
|
||||
1.0,
|
||||
_jev_model(),
|
||||
)
|
||||
|
||||
from routstr import proxy
|
||||
|
||||
original_upstreams = proxy.get_upstreams()
|
||||
with patch("routstr.proxy._upstreams", [provider]):
|
||||
await refresh_model_maps()
|
||||
yield provider
|
||||
with patch("routstr.proxy._upstreams", original_upstreams):
|
||||
await refresh_model_maps()
|
||||
|
||||
|
||||
@pytest.mark.integration
|
||||
@pytest.mark.asyncio
|
||||
async def test_systemone_forwarded_and_billed_from_usage(
|
||||
authenticated_client: AsyncClient,
|
||||
typesafe_provider_maps: _StaticTypeSafeProvider,
|
||||
integration_session: AsyncSession,
|
||||
) -> None:
|
||||
"""A systemone request reaches api.typesafe.ai and bills input tokens."""
|
||||
sent_requests: list[httpx.Request] = []
|
||||
|
||||
async def fake_transport(
|
||||
request: httpx.Request, *args: Any, **kwargs: Any
|
||||
) -> httpx.Response:
|
||||
sent_requests.append(request)
|
||||
return httpx.Response(
|
||||
200,
|
||||
content=json.dumps(SYSTEMONE_RESPONSE).encode(),
|
||||
headers={"content-type": "application/json"},
|
||||
)
|
||||
|
||||
with (
|
||||
patch(
|
||||
"httpx.AsyncHTTPTransport.handle_async_request",
|
||||
side_effect=fake_transport,
|
||||
),
|
||||
patch(
|
||||
"routstr.payment.cost_calculation.sats_usd_price",
|
||||
return_value=0.0005,
|
||||
),
|
||||
):
|
||||
response = await authenticated_client.post(
|
||||
"/v1/systemone",
|
||||
json=SYSTEMONE_REQUEST,
|
||||
)
|
||||
|
||||
assert response.status_code == 200, response.text
|
||||
payload = response.json()
|
||||
|
||||
# The answers pass through untouched.
|
||||
assert payload["answers"]["is_urgent"]["noul"] == pytest.approx(0.95)
|
||||
|
||||
# Exactly one upstream hop, aimed at TypeSafe's endpoint with the
|
||||
# provider's model spelling.
|
||||
assert len(sent_requests) == 1
|
||||
sent = sent_requests[0]
|
||||
assert str(sent.url) == "https://api.typesafe.ai/v1/systemone"
|
||||
assert json.loads(sent.content)["model"] == "jev-latest"
|
||||
|
||||
# Billed from usage: 1000 input tokens at 0.001 sats/token = 1000 msats;
|
||||
# 200 output tokens at 0 sats = 0. Settled cost is exposed on the body.
|
||||
assert payload["cost"]["input_msats"] == 1000
|
||||
assert payload["cost"]["output_msats"] == 0
|
||||
assert payload["cost"]["total_msats"] == 1000
|
||||
|
||||
|
||||
@pytest.mark.integration
|
||||
@pytest.mark.asyncio
|
||||
async def test_systemone_with_x_cashu_settles(
|
||||
authenticated_client: AsyncClient,
|
||||
typesafe_provider_maps: _StaticTypeSafeProvider,
|
||||
testmint_wallet: Any,
|
||||
) -> None:
|
||||
"""The X-Cashu settlement gate admits systemone and refunds the delta."""
|
||||
token = await testmint_wallet.mint_tokens(10_000)
|
||||
|
||||
async def fake_transport(
|
||||
request: httpx.Request, *args: Any, **kwargs: Any
|
||||
) -> httpx.Response:
|
||||
return httpx.Response(
|
||||
200,
|
||||
content=json.dumps(SYSTEMONE_RESPONSE).encode(),
|
||||
headers={"content-type": "application/json"},
|
||||
)
|
||||
|
||||
# The redemption and refund helpers are bound into base.py at import time,
|
||||
# so the app fixture's wallet patches do not reach the proxy's x-cashu path.
|
||||
with (
|
||||
patch(
|
||||
"httpx.AsyncHTTPTransport.handle_async_request",
|
||||
side_effect=fake_transport,
|
||||
),
|
||||
patch(
|
||||
"routstr.upstream.base.recieve_token",
|
||||
AsyncMock(return_value=(10_000, "sat", "https://mint.test")),
|
||||
),
|
||||
patch(
|
||||
"routstr.upstream.base.send_token",
|
||||
AsyncMock(return_value="cashuBrefundtoken"),
|
||||
),
|
||||
# send_refund derives the mint from the token it just created; the
|
||||
# fake token has no mint to parse.
|
||||
patch(
|
||||
"routstr.upstream.base.token_mint_url",
|
||||
return_value="https://mint.test",
|
||||
),
|
||||
# cost_calculation binds sats_usd_price at import time, so the price
|
||||
# patch in the app fixture does not reach it.
|
||||
patch(
|
||||
"routstr.payment.cost_calculation.sats_usd_price",
|
||||
return_value=0.0005,
|
||||
),
|
||||
# Mint trust is node state that other suites mutate; this test is
|
||||
# about the endpoint gate, not mint policy.
|
||||
patch(
|
||||
"routstr.payment.helpers.is_trusted_source_mint",
|
||||
return_value=True,
|
||||
),
|
||||
):
|
||||
response = await authenticated_client.post(
|
||||
"/v1/systemone",
|
||||
json=SYSTEMONE_REQUEST,
|
||||
headers={"X-Cashu": token},
|
||||
)
|
||||
|
||||
# Not rejected as an unsupported endpoint (that returns 400 with
|
||||
# x_cashu_unsupported_endpoint), and settled rather than streamed raw.
|
||||
assert response.status_code == 200, response.text
|
||||
payload = response.json()
|
||||
assert payload["answers"]["is_urgent"]["type"] == "noul"
|
||||
# A refund header exists when the token exceeded the settled cost.
|
||||
assert response.headers.get("X-Cashu") is not None
|
||||
@@ -0,0 +1,150 @@
|
||||
"""Unit tests for the TypeSafe System One provider integration.
|
||||
|
||||
Covers the three integration seams the systemone endpoint touches:
|
||||
|
||||
* the proxy path allowlist admits ``systemone`` (POST only) and nothing that
|
||||
merely looks like it;
|
||||
* the X-Cashu settlement gate treats ``systemone`` as a settleable endpoint;
|
||||
* the provider maps TypeSafe's pricing-less model listing onto priced
|
||||
``Model`` objects with usable rates.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
|
||||
os.environ.setdefault("UPSTREAM_BASE_URL", "http://test")
|
||||
os.environ.setdefault("UPSTREAM_API_KEY", "test")
|
||||
|
||||
from unittest.mock import AsyncMock, MagicMock, patch # noqa: E402
|
||||
|
||||
import pytest # noqa: E402
|
||||
|
||||
from routstr.proxy import _forwarding_allowed # noqa: E402
|
||||
from routstr.upstream.base import _x_cashu_path_has_settlement_handler # noqa: E402
|
||||
from routstr.upstream.typesafe import TypeSafeUpstreamProvider # noqa: E402
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
("path", "method"),
|
||||
[
|
||||
("v1/systemone", "POST"),
|
||||
("systemone", "POST"),
|
||||
("v1/systemone/", "POST"),
|
||||
],
|
||||
)
|
||||
def test_systemone_is_forwarded(path: str, method: str) -> None:
|
||||
assert _forwarding_allowed(path, method) is True
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
("path", "method"),
|
||||
[
|
||||
("v1/systemone", "GET"), # billed endpoint is POST-only
|
||||
("v1/systemone", "DELETE"),
|
||||
("systemonedump", "POST"), # longer segment must not match
|
||||
("v1/systemone/secret", "POST"), # trailing id segment widens nothing
|
||||
("v1/systemone/../admin", "POST"), # traversal spelling is screened
|
||||
],
|
||||
)
|
||||
def test_systemone_lookalikes_are_refused(path: str, method: str) -> None:
|
||||
assert _forwarding_allowed(path, method) is False
|
||||
|
||||
|
||||
def test_systemone_has_x_cashu_settlement_handler() -> None:
|
||||
assert _x_cashu_path_has_settlement_handler("v1/systemone") is True
|
||||
assert _x_cashu_path_has_settlement_handler("systemone/") is True
|
||||
|
||||
|
||||
def test_provider_metadata_is_complete() -> None:
|
||||
meta = TypeSafeUpstreamProvider.get_provider_metadata()
|
||||
assert meta["id"] == "typesafe"
|
||||
assert meta["default_base_url"] == "https://api.typesafe.ai/v1"
|
||||
assert meta["fixed_base_url"] is True
|
||||
|
||||
|
||||
def test_provider_transform_model_name() -> None:
|
||||
provider = TypeSafeUpstreamProvider(api_key="test")
|
||||
assert provider.transform_model_name("typesafe/jev-latest") == "jev-latest"
|
||||
assert provider.transform_model_name("jev-latest") == "jev-latest"
|
||||
|
||||
|
||||
def _mock_models_response(payload: dict) -> MagicMock:
|
||||
mock_response = MagicMock()
|
||||
mock_response.status_code = 200
|
||||
mock_response.raise_for_status = MagicMock()
|
||||
mock_response.json.return_value = payload
|
||||
return mock_response
|
||||
|
||||
|
||||
def _patch_client(mock_response: MagicMock) -> patch:
|
||||
mock_client = MagicMock()
|
||||
mock_client.__aenter__ = AsyncMock(return_value=mock_client)
|
||||
mock_client.__aexit__ = AsyncMock(return_value=None)
|
||||
mock_client.get = AsyncMock(return_value=mock_response)
|
||||
return patch(
|
||||
"routstr.upstream.typesafe.httpx.AsyncClient",
|
||||
return_value=mock_client,
|
||||
)
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_fetch_models_prices_the_listing() -> None:
|
||||
provider = TypeSafeUpstreamProvider(api_key="test")
|
||||
payload = {
|
||||
"models": [
|
||||
{
|
||||
"name": "jev-latest",
|
||||
"description": "The most recent stable release.",
|
||||
"release_date": "2026-09-17",
|
||||
},
|
||||
{
|
||||
"name": "jev-preview",
|
||||
"description": "The most recent release.",
|
||||
"release_date": "2026-09-17",
|
||||
},
|
||||
]
|
||||
}
|
||||
|
||||
with _patch_client(_mock_models_response(payload)):
|
||||
models = await provider.fetch_models()
|
||||
|
||||
assert [m.id for m in models] == ["jev-latest", "jev-preview"]
|
||||
for model in models:
|
||||
# Input priced, output free; rates usable (zero is a real price).
|
||||
assert model.pricing.prompt == pytest.approx(0.042 / 1_000_000)
|
||||
assert model.pricing.completion == 0.0
|
||||
assert model.context_length == 64_000
|
||||
assert model.architecture.input_modalities == ["text"]
|
||||
assert model.architecture.output_modalities == ["decisions"]
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_fetch_models_handles_error() -> None:
|
||||
provider = TypeSafeUpstreamProvider(api_key="test")
|
||||
mock_response = MagicMock()
|
||||
mock_response.raise_for_status = MagicMock(
|
||||
side_effect=RuntimeError("upstream down")
|
||||
)
|
||||
|
||||
with _patch_client(mock_response):
|
||||
models = await provider.fetch_models()
|
||||
|
||||
assert models == []
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_fetch_models_defaults_unknown_model_rates() -> None:
|
||||
"""A newly listed model still gets a usable default rate."""
|
||||
provider = TypeSafeUpstreamProvider(api_key="test")
|
||||
payload = {
|
||||
"models": [
|
||||
{"name": "jev-2.0", "description": "Future model", "release_date": None}
|
||||
]
|
||||
}
|
||||
|
||||
with _patch_client(_mock_models_response(payload)):
|
||||
models = await provider.fetch_models()
|
||||
|
||||
assert len(models) == 1
|
||||
assert models[0].pricing.prompt == pytest.approx(0.042 / 1_000_000)
|
||||
Reference in New Issue
Block a user