mirror of
https://github.com/Routstr/routstr-core.git
synced 2026-10-05 12:28:22 +00:00
Merge remote-tracking branch 'origin/main' into lightning-refund
This commit is contained in:
9qeklajc
@@ -11,6 +11,9 @@ dist/
|
||||
*.egg
|
||||
.mypy_cache/**
|
||||
|
||||
# MkDocs build output
|
||||
site/
|
||||
|
||||
# Development
|
||||
.notes
|
||||
.*keys.db
|
||||
|
||||
@@ -26,6 +26,20 @@ If you want to run a node, resell API access, or monetize hardware.
|
||||
|
||||
---
|
||||
|
||||
## 👥 For Teams (Remote Nodes)
|
||||
|
||||
If you want one shared Routstr endpoint for a whole team, with per-member identities and per-member usage tracking.
|
||||
|
||||
- **[Overview](teams/index.md)**: What a remote node is, and when to use one.
|
||||
- **[Deploy on Cloudron](teams/deploy-cloudron.md)**: The packaged, supported deployment.
|
||||
- **[Deploy with Docker](teams/deploy-docker.md)**: Run it on any host behind your own TLS.
|
||||
- **[Team Members](teams/team-members.md)**: Bootstrap the first admin and invite people.
|
||||
- **[Connecting Clients](teams/clients.md)**: Wire up Claude Code, Pi, OpenCode, and API keys.
|
||||
- **[Usage and Model Policy](teams/usage-and-policy.md)**: Per-member spend and model allowlists.
|
||||
- **[Security Model](teams/security.md)**: Auth rules and endpoint scoping.
|
||||
|
||||
---
|
||||
|
||||
## 🔌 API Reference
|
||||
|
||||
- **[Overview](api/overview.md)**: Base URL, headers, and standards.
|
||||
|
||||
@@ -0,0 +1,153 @@
|
||||
# Connecting Clients
|
||||
|
||||
A **client** is one agent or application talking to the node. Each client gets its own ID and its own API key (`sk-...`). Clients are how a team node attributes usage: every request is billed against the client that made it.
|
||||
|
||||
Members create and manage **their own** clients. They cannot see or touch anyone else's.
|
||||
|
||||
---
|
||||
|
||||
## Two credentials, two jobs
|
||||
|
||||
The node accepts two entirely different kinds of credential, and confusing them is the most common source of `403`s.
|
||||
|
||||
| Credential | Header | Purpose | Can do |
|
||||
|---|---|---|---|
|
||||
| **API key** | `Authorization: Bearer sk-...` | Inference | Send chat/completion requests. Cannot touch wallets, clients, or npubs. |
|
||||
| **NIP-98** | `Authorization: Nostr <base64-event>` | Management | Manage clients, npubs, wallet, node control. Signed per-request by the member's `nsec`. |
|
||||
|
||||
Your agents use the **API key**. The `routstrd` CLI uses **NIP-98** automatically, which is why it needs your `nsec` in `~/.routstrd/config.json`.
|
||||
|
||||
---
|
||||
|
||||
## Add a client
|
||||
|
||||
From the member's own machine:
|
||||
|
||||
```bash
|
||||
# Name it explicitly
|
||||
routstrd clients add --name "My Laptop"
|
||||
|
||||
# Or use a one-shot integration setup
|
||||
routstrd clients add --claude-code
|
||||
routstrd clients add --pi-agent
|
||||
routstrd clients add --opencode
|
||||
routstrd clients add --openclaw
|
||||
routstrd clients add --hermes
|
||||
```
|
||||
|
||||
The integration flags configure the agent's own config file as well as registering the client, so you do not have to hand-edit anything. Several can be combined in one call.
|
||||
|
||||
On success the CLI prints the credentials and the endpoint to point at:
|
||||
|
||||
```text
|
||||
Client created.
|
||||
|
||||
ID: my-laptop
|
||||
Name: My Laptop
|
||||
API Key: sk-9f2a...
|
||||
|
||||
Access Routstr at: https://team.example.com/v1
|
||||
```
|
||||
|
||||
!!! warning "The API key is a secret"
|
||||
Treat `sk-...` like a password. It bills inference to the team wallet. Do not commit it, and do not paste it into a chat — unlike an npub, it is not safe to share.
|
||||
|
||||
### Adding is idempotent
|
||||
|
||||
Running `clients add` with a name that already exists does not create a duplicate. It looks the client up and prints the existing record — including its API key — so re-running is a safe way to recover a key you lost:
|
||||
|
||||
```text
|
||||
Client 'my-laptop' already exists.
|
||||
|
||||
ID: my-laptop
|
||||
Name: My Laptop
|
||||
API Key: sk-9f2a...
|
||||
```
|
||||
|
||||
### List and delete
|
||||
|
||||
```bash
|
||||
routstrd clients list
|
||||
routstrd clients delete my-laptop
|
||||
```
|
||||
|
||||
`clients list` shows **only your own** clients.
|
||||
|
||||
---
|
||||
|
||||
## Going beyond the CLI
|
||||
|
||||
The agent integrations cover the common tools, but any OpenAI-compatible client works — point it at the node and use the API key:
|
||||
|
||||
```bash
|
||||
curl https://team.example.com/v1/chat/completions \
|
||||
-H "Authorization: Bearer sk-9f2a..." \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"model": "gpt-4o-mini",
|
||||
"messages": [{"role": "user", "content": "hello"}]
|
||||
}'
|
||||
```
|
||||
|
||||
The base URL is always the node host plus `/v1`. Discover available models without any credential at all:
|
||||
|
||||
```bash
|
||||
curl https://team.example.com/v1/models
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## How client ownership works
|
||||
|
||||
This is the part that explains the odd-looking IDs on the node.
|
||||
|
||||
### IDs are derived from the name
|
||||
|
||||
A client's ID is the name lowercased with internal whitespace collapsed to hyphens, then stripped of anything that is not alphanumeric or a hyphen. `"My Laptop!"` becomes `my-laptop`.
|
||||
|
||||
### The node appends an owner suffix
|
||||
|
||||
So that two members can both have a client called `my-laptop` without colliding, the auth proxy appends the **last 7 characters of the owner's npub** to the ID before it reaches the daemon:
|
||||
|
||||
| Where you look | Client ID |
|
||||
|---|---|
|
||||
| Member's `routstrd clients list` | `my-laptop` |
|
||||
| On the node (`cloudron exec`, then `routstrd clients list`) | `my-laptop-4f2x9k7` |
|
||||
|
||||
The suffix is stripped again on the way back, so members always see the clean ID. On the node you deliberately see the suffixed form — **the trailing characters are what tell you which member owns a client.**
|
||||
|
||||
### Ownership is recorded explicitly
|
||||
|
||||
Newly created clients store the owner's npub in an `ownerNpub` field, and the proxy authorises against that field. Clients created before that field existed fall back to matching the ID suffix, so older installs keep working until those clients are recreated.
|
||||
|
||||
### Consequence: admins are not automatically superusers here
|
||||
|
||||
`/clients`, `/clients/add`, and `/clients/delete` are owner-scoped by the calling npub. Even an `admin` cannot list or delete a colleague's clients through these endpoints. Cross-member visibility comes from running the CLI **on the node itself**, where the daemon is unauthenticated on loopback:
|
||||
|
||||
```bash
|
||||
cloudron exec --app routstr.example.com
|
||||
routstrd clients list # all clients, all owners, suffixed IDs
|
||||
```
|
||||
|
||||
That is also the only practical way to clean up a departing member's keys — see [Team Members](team-members.md#what-revocation-does-and-does-not-do).
|
||||
|
||||
---
|
||||
|
||||
## Refreshing models and integrations
|
||||
|
||||
The `clients` command carries options for the daemon's scheduled refresh job, which updates the Routstr 21 model list and re-syncs client integrations:
|
||||
|
||||
```bash
|
||||
routstrd clients --manual-refresh # refresh now, once
|
||||
routstrd clients --disable-automatic-refresh # stop the scheduled job
|
||||
routstrd clients --enable-automatic-refresh # start it again
|
||||
```
|
||||
|
||||
The model list matters because it is also what the [model allowlist](usage-and-policy.md#model-allowlist) is enforced against.
|
||||
|
||||
---
|
||||
|
||||
## Next steps
|
||||
|
||||
- [Usage and Model Policy](usage-and-policy.md) — watch what those clients are spending.
|
||||
- [Security Model](security.md) — the exact rules applied to each credential.
|
||||
@@ -0,0 +1,169 @@
|
||||
# Deploy on Cloudron
|
||||
|
||||
[Cloudron](https://www.cloudron.io/) is the supported deployment target for a team node. The packaged image already contains **both** processes — the `routstrd` daemon and the `routstrd-auth` proxy — supervised inside a single container, with `/app/data` handled as persistent storage and TLS terminated by the platform.
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| **App ID** | `io.routstr.routstrd-auth` |
|
||||
| **Public port** | `8008` (Cloudron proxies it over HTTPS on 443) |
|
||||
| **Health check** | `GET /health` |
|
||||
| **Memory limit** | 512 MB |
|
||||
| **Minimum box version** | Cloudron 9.1.0 |
|
||||
|
||||
---
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- A running Cloudron box with a domain that can get a certificate.
|
||||
- The [`cloudron` CLI](https://docs.cloudron.io/cli/) installed and logged in, **only if** you are building the image yourself:
|
||||
|
||||
```bash
|
||||
npm install -g cloudron
|
||||
cloudron login my.example.com
|
||||
```
|
||||
|
||||
## Install
|
||||
|
||||
### Option A — from the published version list
|
||||
|
||||
The app is published as a custom Cloudron app with a version list (`CloudronVersions.json`), currently at `0.1.26`. Once that app store entry is registered on your Cloudron instance, install it from the dashboard, or:
|
||||
|
||||
```bash
|
||||
cloudron install --appstore-id io.routstr.routstrd-auth --location routstr.example.com
|
||||
```
|
||||
|
||||
### Option B — build the image yourself
|
||||
|
||||
Use this when you want to run a local modification:
|
||||
|
||||
```bash
|
||||
git clone https://github.com/routstr/routstrd-remote
|
||||
cd routstrd-remote
|
||||
|
||||
cloudron build # builds the Dockerfile and pushes it to your registry
|
||||
cloudron install --image <registry>/routstrd-remote:<tag> --location routstr.example.com
|
||||
```
|
||||
|
||||
!!! note "The Dockerfile is the Cloudron image"
|
||||
The repository's `Dockerfile` is built `FROM cloudron/base:5.0.0` and its `CMD` is `cloudron/start.sh`, which prepares `/app/data` and starts `supervisord`. It expects Cloudron's filesystem conventions and should not be confused with a generic Docker image. See [Deploy with Docker](deploy-docker.md) for what that means in practice.
|
||||
|
||||
---
|
||||
|
||||
## Bootstrap the first admin
|
||||
|
||||
The moment the app is healthy, the npub table is **empty**, and nothing except the public endpoints can be reached. Claim it before anything else — while the table is empty, `POST /npubs` is accepted without authentication, so this is the only window in which an unauthenticated registration succeeds.
|
||||
|
||||
On the machine of whoever will be the first admin:
|
||||
|
||||
```bash
|
||||
bun i -g routstrd
|
||||
routstrd remote https://routstr.example.com
|
||||
routstrd npubs register --name "Alice"
|
||||
```
|
||||
|
||||
`routstrd remote` generates a fresh Nostr identity if you do not have one, stores it in `~/.routstrd/config.json`, and prints your npub. `routstrd npubs register` then posts that npub and, because no npubs exist yet, receives `admin`.
|
||||
|
||||
!!! warning "Register immediately after install"
|
||||
Until the first admin registers, anyone who knows the URL can claim the node. Do this as part of the install, not later.
|
||||
|
||||
Verify:
|
||||
|
||||
```bash
|
||||
routstrd npubs list
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Configuration
|
||||
|
||||
Cloudron defaults are set by `cloudron/start.sh` and the two supervisor programs. Everything below can be overridden through the Cloudron **Environment Variables** tab.
|
||||
|
||||
| Variable | Default | Purpose |
|
||||
|---|---|---|
|
||||
| `ROUTSTRD_AUTH_PORT` | `8008` | Public port served by the auth proxy. Must match the manifest's `httpPort`. |
|
||||
| `ROUTSTRD_AUTH_HOST` | `0.0.0.0` | Bind address of the auth proxy. |
|
||||
| `ROUTSTRD_UPSTREAM` | `http://localhost:8009` | Where the daemon listens. Keep this on loopback. |
|
||||
| `ROUTSTRD_PORT` | `8009` | Port the daemon binds. |
|
||||
| `ROUTSTRD_DIR` | `/app/data/routstrd` | Config directory shared by the daemon and the proxy. |
|
||||
| `ROUTSTRD_DB_PATH` | `/app/data/routstrd/routstr.db` | Shared SQLite database. |
|
||||
| `ROUTSTRD_CONFIG_FILE` | `$ROUTSTRD_DIR/config.json` | Daemon config file. |
|
||||
| `ROUTSTRD_AUTH_MODEL_ALLOWLIST` | `false` | Set to `true` to restrict the team to the Routstr 21 model list. See [Usage and Model Policy](usage-and-policy.md). |
|
||||
| `ROUTSTRD_AUTH_ADMIN_NPUBS` | *(unset)* | Optional bootstrap admins. See below. |
|
||||
|
||||
### Bootstrapping admins from the environment
|
||||
|
||||
Instead of the interactive `npubs register` step you can seed admins declaratively. Three variables are accepted and merged: `ROUTSTRD_AUTH_ADMIN_NPUBS`, `ROUTSTRD_AUTH_ADMIN_PUBKEYS`, and `ROUTSTRD_AUTH_BOOTSTRAP_NPUB`. Values are comma- or whitespace-separated and may be either `npub1...` or 64-character hex.
|
||||
|
||||
Rows created this way are tagged `source = 'env'`. At every startup the proxy **reconciles** them: an env-sourced row whose pubkey is no longer present in the environment is **deleted**. This means the environment variables are the source of truth for those rows — removing someone from the variable revokes their access on the next restart.
|
||||
|
||||
!!! tip "Prefer `npubs register` for the first admin"
|
||||
There is deliberately **no** hardcoded default admin npub in the image. An image with a baked-in admin pubkey would hand control of every deployment to the same key.
|
||||
|
||||
### Filesystem layout
|
||||
|
||||
| Path | Lifetime | Contents |
|
||||
|---|---|---|
|
||||
| `/app/code` | replaced on update | Auth proxy source and the `start.sh` / `run-auth.sh` scripts. |
|
||||
| `/app/data` | persistent, backed up | `routstrd/config.json`, `routstrd/routstr.db`, `logs/`, and a `.initialized` marker. |
|
||||
| `/run` | ephemeral | `supervisord` socket and pid file. |
|
||||
|
||||
The startup script writes `authUrl` into `routstrd/config.json` pointing at the local proxy, and generates the container's Nostr identity (`nsec`) on first boot if one is missing. That generated identity is what authorises the daemon's own calls back through the proxy.
|
||||
|
||||
---
|
||||
|
||||
## Backups
|
||||
|
||||
Cloudron's `localstorage` addon makes `/app/data` persistent and includes it in regular backups. The entire data directory is covered, which means the SQLite database at `/app/data/routstrd/routstr.db` travels with it.
|
||||
|
||||
To restore, restore the app from a Cloudron backup — the wallet configuration, npub table, and client records all come back together. Do not hand-copy files between hosts: the daemon's `config.json` holds the container's `nsec`, and losing it breaks the node's ability to authenticate to its own proxy.
|
||||
|
||||
!!! warning "Use `cloudron exec` carefully"
|
||||
The database uses SQLite WAL mode. If you need a manual snapshot, stop the app first (`cloudron stop`) or use `sqlite3 .backup` inside the container — copying the `.db` file while the daemon is writing can produce an inconsistent file.
|
||||
|
||||
---
|
||||
|
||||
## Updates
|
||||
|
||||
```bash
|
||||
cloudron update --app routstr.example.com
|
||||
```
|
||||
|
||||
Updates are the primary lifecycle event on Cloudron and are designed to preserve `/app/data`. After the app comes back, confirm health and that your admin npub is still recognised:
|
||||
|
||||
```bash
|
||||
curl https://routstr.example.com/health
|
||||
routstrd npubs list
|
||||
```
|
||||
|
||||
## Day-to-day operations
|
||||
|
||||
```bash
|
||||
cloudron logs -f --app routstr.example.com # follow both processes' output
|
||||
cloudron exec --app routstr.example.com # shell into the container
|
||||
cloudron stop --app routstr.example.com
|
||||
cloudron start --app routstr.example.com
|
||||
cloudron debug --app routstr.example.com # read-write filesystem, app paused
|
||||
cloudron debug --disable --app routstr.example.com
|
||||
```
|
||||
|
||||
Inside the container you are on the machine that holds the wallet and the shared database, so `routstrd` commands there operate on the node itself rather than as a remote member:
|
||||
|
||||
```bash
|
||||
routstrd npubs list # everyone with access, and their roles
|
||||
routstrd clients list # every client on the node, not just your own
|
||||
routstrd top # interactive usage TUI across all members
|
||||
```
|
||||
|
||||
Both processes log to stdout/stderr and are collected by Cloudron; there are no log files to rotate inside `/app/data`.
|
||||
|
||||
### Failure handling
|
||||
|
||||
`supervisord` runs both programs with `autorestart=true` and a start priority that brings the **daemon up first** (priority 10) and the **proxy second** (priority 20). The proxy's launcher additionally waits — up to 120 seconds — for the database file to exist *and* for the daemon's `/health` to answer before it starts serving. A crash-looping proxy therefore usually means the daemon never became healthy.
|
||||
|
||||
---
|
||||
|
||||
## Next steps
|
||||
|
||||
- [Team Members](team-members.md) — invite the rest of your team.
|
||||
- [Security Model](security.md) — what is exposed and what is not.
|
||||
- [Troubleshooting](troubleshooting.md) — when the proxy will not start.
|
||||
@@ -0,0 +1,170 @@
|
||||
# Deploy with Docker
|
||||
|
||||
The team node is a single container running two supervised processes. You can run that same image on any Docker host and terminate TLS with your own reverse proxy.
|
||||
|
||||
!!! note "Read this first"
|
||||
The image is built `FROM cloudron/base:5.0.0` and its entrypoint is the Cloudron startup script. It works outside Cloudron — it only requires a writable `/app/data` — but its filesystem conventions are Cloudron's, and the repository's `docker-compose.yml` predates this image. The commands below are the ones that match the current image.
|
||||
|
||||
---
|
||||
|
||||
## Build
|
||||
|
||||
```bash
|
||||
git clone https://github.com/routstr/routstrd-remote
|
||||
cd routstrd-remote
|
||||
docker build -t routstr-remote:0.1.26 .
|
||||
```
|
||||
|
||||
The Dockerfile installs Bun (the x64 **baseline** build, chosen because some hosts do not expose AVX/AVX2 and the default binary crashes with `SIGILL`), installs the `routstrd` daemon globally, installs the proxy's dependencies, and copies in the supervisor configuration for both processes.
|
||||
|
||||
## Run
|
||||
|
||||
```bash
|
||||
mkdir -p "$HOME/routstr-remote-data"
|
||||
|
||||
docker run -d \
|
||||
--name routstr-remote \
|
||||
--restart unless-stopped \
|
||||
-p 127.0.0.1:8008:8008 \
|
||||
-v "$HOME/routstr-remote-data:/app/data" \
|
||||
--memory 1g \
|
||||
routstr-remote:0.1.26
|
||||
```
|
||||
|
||||
**Why each flag matters:**
|
||||
|
||||
| Flag | Reason |
|
||||
|---|---|
|
||||
| `-v ...:/app/data` | The only persistent path. It holds `routstrd/config.json` (including the container's `nsec`), `routstrd/routstr.db`, and `logs/`. **Without it the node loses its identity and every npub on restart.** |
|
||||
| `-p 127.0.0.1:8008:8008` | Publish on loopback only and let a TLS reverse proxy in front of it expose the service. Binding `0.0.0.0:8008` on a public host sends API keys over plaintext HTTP. |
|
||||
| `--memory 1g` | Two Bun processes plus the daemon's model and usage state. The Cloudron manifest requests 512 MB; give a bare Docker host at least that, and prefer more. |
|
||||
|
||||
The container listens on exactly two ports: `8008` (public auth proxy) and `8009` (daemon, bound to loopback **inside** the container and deliberately not published).
|
||||
|
||||
Check it came up:
|
||||
|
||||
```bash
|
||||
curl http://127.0.0.1:8008/health
|
||||
docker logs -f routstr-remote
|
||||
```
|
||||
|
||||
### Startup ordering
|
||||
|
||||
The entrypoint starts `supervisord`, which brings up the daemon first and the proxy second. The proxy's launcher then waits for the database file to appear and for `http://localhost:8009/health` to answer, retrying up to 120 times at one-second intervals. If that window expires, the proxy exits with `Timed out waiting for routstrd to become ready.` and restarts.
|
||||
|
||||
## Put TLS in front
|
||||
|
||||
Anything that terminates TLS and forwards to `127.0.0.1:8008` works. Two properties are worth configuring explicitly:
|
||||
|
||||
- **Disable response buffering.** The proxy already sends `X-Accel-Buffering: no` upstream, but your own proxy should also be configured not to buffer, otherwise streamed LLM responses appear to truncate.
|
||||
- **Raise idle timeouts.** Model responses can be silent for a long time while reasoning or waiting on tools. The proxy disables Bun's per-request idle timeout for exactly this reason, but nginx's default 60-second `proxy_read_timeout` will still cut streams. Set it to something generous.
|
||||
|
||||
```nginx
|
||||
location / {
|
||||
proxy_pass http://127.0.0.1:8008;
|
||||
proxy_http_version 1.1;
|
||||
proxy_set_header Host $host;
|
||||
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
|
||||
proxy_set_header X-Forwarded-Proto $scheme;
|
||||
|
||||
proxy_buffering off;
|
||||
proxy_read_timeout 3600s;
|
||||
proxy_send_timeout 3600s;
|
||||
}
|
||||
```
|
||||
|
||||
## Bootstrap the first admin
|
||||
|
||||
Identical to Cloudron — run this from the admin's own machine:
|
||||
|
||||
```bash
|
||||
bun i -g routstrd
|
||||
routstrd remote https://routstr.example.com
|
||||
routstrd npubs register --name "Alice"
|
||||
```
|
||||
|
||||
While the npub table is empty, `POST /npubs` needs no authentication, so the first registration wins. See [Team Members](team-members.md).
|
||||
|
||||
## Configuration
|
||||
|
||||
Override any of the variables from [Deploy on Cloudron](deploy-cloudron.md#configuration) with `-e`, for example:
|
||||
|
||||
```bash
|
||||
docker run -d \
|
||||
--name routstr-remote \
|
||||
--restart unless-stopped \
|
||||
-p 127.0.0.1:8008:8008 \
|
||||
-v "$HOME/routstr-remote-data:/app/data" \
|
||||
-e ROUTSTRD_AUTH_MODEL_ALLOWLIST=true \
|
||||
routstr-remote:0.1.26
|
||||
```
|
||||
|
||||
Inside the container the defaults are `ROUTSTRD_DIR=/app/data/routstrd`, `ROUTSTRD_DB_PATH=/app/data/routstrd/routstr.db`, `ROUTSTRD_UPSTREAM=http://localhost:8009`, `ROUTSTRD_AUTH_HOST=0.0.0.0`, `ROUTSTRD_AUTH_PORT=8008`.
|
||||
|
||||
!!! warning "Use a bind mount, not an anonymous volume"
|
||||
If you recreate the container (`docker rm` then `docker run`), an unnamed volume is orphaned and the node comes back with a **new** Nostr identity and an **empty** npub table — meaning the next person to hit `POST /npubs` becomes admin. Always mount a known host directory.
|
||||
|
||||
## Backups
|
||||
|
||||
The database is SQLite in WAL mode, so stop the container before copying the data directory, or take a proper snapshot from inside:
|
||||
|
||||
```bash
|
||||
docker stop routstr-remote
|
||||
tar czf routstr-remote-$(date +%F).tar.gz -C "$HOME" routstr-remote-data
|
||||
docker start routstr-remote
|
||||
```
|
||||
|
||||
Or, without downtime:
|
||||
|
||||
```bash
|
||||
docker exec routstr-remote sqlite3 /app/data/routstrd/routstr.db ".backup /app/data/routstrd/backup.db"
|
||||
docker cp routstr-remote:/app/data/routstrd/backup.db .
|
||||
```
|
||||
|
||||
## Updates
|
||||
|
||||
```bash
|
||||
docker stop routstr-remote && docker rm routstr-remote
|
||||
git pull
|
||||
docker build -t routstr-remote:new .
|
||||
docker run -d ... routstr-remote:new # same -v and -p flags as before
|
||||
```
|
||||
|
||||
Because `/app/data` is a bind mount, the node's identity, npub table, and client records survive the swap. That is the entire reason the mount is not optional.
|
||||
|
||||
---
|
||||
|
||||
## Running from source (evaluation and development)
|
||||
|
||||
You do not need Docker to try the proxy. Point it at any routstrd daemon's database:
|
||||
|
||||
```bash
|
||||
bun install
|
||||
bun run src/index.ts validate # checks config and opens the DB
|
||||
bun run src/index.ts start # binds 0.0.0.0:8008
|
||||
```
|
||||
|
||||
Useful flags: `--port`, `--host`, `--upstream`, `--db-path`. The `validate` subcommand prints the effective configuration and reports how many npubs are registered, split by role — it is the fastest way to confirm the proxy can see the right database:
|
||||
|
||||
```text
|
||||
Configuration:
|
||||
Port: 8008
|
||||
Host: 0.0.0.0
|
||||
Upstream: http://localhost:8009
|
||||
DB path: /app/data/routstrd/routstr.db
|
||||
Bootstrap admin npubs/pubkeys from env: 0
|
||||
Model allowlist: disabled
|
||||
|
||||
✅ DB accessible. 3 npub(s) registered (1 admin, 2 user).
|
||||
```
|
||||
|
||||
!!! warning "`validate` fails before the daemon has run once"
|
||||
If the database does not exist, validation stops with `Database not found at ... Make sure routstrd has been initialized`. The proxy shares the daemon's database; it never creates the schema itself.
|
||||
|
||||
---
|
||||
|
||||
## Next steps
|
||||
|
||||
- [Team Members](team-members.md) — invite the rest of your team.
|
||||
- [Security Model](security.md) — which endpoints require what.
|
||||
- [Troubleshooting](troubleshooting.md) — startup and streaming failures.
|
||||
@@ -0,0 +1,112 @@
|
||||
# Teams and Remote Nodes
|
||||
|
||||
A normal `routstrd` install is a **single-user daemon** running on your own machine. A **remote node** turns that same daemon into a **shared instance for a team**: one server runs the daemon behind an authentication proxy, and every team member gets their own Nostr identity, their own API keys, and their own usage accounting.
|
||||
|
||||
This is the product sometimes called **Routstrd Remote** — the repository is [`Routstr/routstrd-remote`](https://github.com/routstr/routstrd-remote) and the running service is the package `routstrd-auth`.
|
||||
|
||||
---
|
||||
|
||||
## Who this is for
|
||||
|
||||
- **Small teams and orgs** that want one funded Routstr endpoint instead of one daemon per laptop.
|
||||
- **Anyone who wants per-person spend visibility** without building a billing system.
|
||||
- **Self-hosters** who already run [Cloudron](https://www.cloudron.io/) and want a one-click install.
|
||||
|
||||
## What it gives you
|
||||
|
||||
| Capability | Detail |
|
||||
|---|---|
|
||||
| **One endpoint, many people** | Members point their coding agents at a single HTTPS URL. No per-machine setup beyond one CLI command. |
|
||||
| **Per-member identity** | Each person has their own Nostr keypair (npub). Access is granted and revoked by adding or deleting that npub. |
|
||||
| **Per-member attribution** | Every client registration gets a unique ID, and usage is reported per client, so you can see who is spending what. |
|
||||
| **Two levels of privilege** | `admin` (manage people, move funds, control the node) and `user` (run inference, manage only their own clients). |
|
||||
| **Scoped API keys** | Agent API keys can buy inference but cannot touch the wallet or other members' clients. |
|
||||
| **Model policy** | Optional allowlist restricts the team to approved models. |
|
||||
| **One wallet to fund** | The team tops up a single node wallet rather than N personal wallets. |
|
||||
|
||||
## What it is not
|
||||
|
||||
- It is **not** multi-tenant SaaS. Everyone shares the node's wallet and upstream provider set.
|
||||
- It is **not** a billing or chargeback system. Usage is *attributed* per client; invoicing your teammates is up to you.
|
||||
- It does **not** give each member a separate balance. See [Usage and Model Policy](usage-and-policy.md) for exactly what is tracked.
|
||||
|
||||
---
|
||||
|
||||
## Architecture
|
||||
|
||||
Two processes run together inside one container. Only one of them is reachable from outside.
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
CLI["routstrd CLI<br/>on a member laptop"]
|
||||
Agent["Coding agents<br/>Claude Code, Pi, OpenCode"]
|
||||
App["App holding an sk- API key"]
|
||||
|
||||
TLS["Reverse proxy<br/>TLS termination on 443"]
|
||||
Proxy["routstrd-auth<br/>0.0.0.0:8008 public"]
|
||||
Daemon["routstrd daemon<br/>localhost:8009 no auth"]
|
||||
DB[("routstr.db<br/>shared SQLite")]
|
||||
Providers["Upstream model providers"]
|
||||
|
||||
CLI -->|https| TLS
|
||||
Agent -->|https| TLS
|
||||
App -->|https| TLS
|
||||
TLS -->|http| Proxy
|
||||
Proxy -->|forward| Daemon
|
||||
Proxy -->|npubs and clients| DB
|
||||
Daemon -->|usage and models| DB
|
||||
Daemon -->|inference| Providers
|
||||
```
|
||||
|
||||
**The security property that matters:** the daemon runs with **no authentication at all** because it is bound to `localhost` and never published. The auth proxy is the only public surface. If you expose port `8009`, you have removed the entire security model.
|
||||
|
||||
### Components
|
||||
|
||||
| Component | Role | Bind |
|
||||
|---|---|---|
|
||||
| `routstrd-auth` | Public auth proxy. Validates credentials, enforces roles and model policy, forwards to the daemon. | `0.0.0.0:8008` |
|
||||
| `routstrd` | The inference daemon. Owns the wallet, providers, clients, and usage records. | `localhost:8009` |
|
||||
| `routstr.db` | Shared SQLite database. Holds `routstr_auth_npubs`, `clients`, usage rows, and `sdk_storage` (including the Routstr 21 model list). | on disk |
|
||||
| Reverse proxy | TLS termination and the public hostname. On Cloudron this is managed by the platform. | `443` |
|
||||
|
||||
---
|
||||
|
||||
## Roles
|
||||
|
||||
Registration lives in a single table, `routstr_auth_npubs`, where each row has a `role` of `admin` or `user`.
|
||||
|
||||
| Capability | `admin` | `user` |
|
||||
|---|---|---|
|
||||
| Run inference with own API keys | yes | yes |
|
||||
| Create and delete **own** clients | yes | yes |
|
||||
| Read **own** usage | yes | yes |
|
||||
| List all registered npubs | yes | yes |
|
||||
| Add / update / delete npubs | yes | no |
|
||||
| Send funds from the node wallet | yes | no |
|
||||
| Node control (providers, refunds, stop) | yes | yes |
|
||||
| Read wallet balance / status | yes | yes |
|
||||
|
||||
`user` is the default role when an admin adds someone. Promote with `routstrd npubs update <npub> --role admin`.
|
||||
|
||||
---
|
||||
|
||||
## Bootstrap order
|
||||
|
||||
A fresh node has an empty npub table. That produces exactly one unaudited window, and only one:
|
||||
|
||||
1. **The first person** runs `routstrd npubs register` against the new node. Because the table is empty, `POST /npubs` is accepted **without authentication**, and the caller becomes `admin`.
|
||||
2. From that moment on, **every** npub operation requires NIP-98 auth from an existing admin. A second unauthenticated registration is refused with `409` / "already configured".
|
||||
|
||||
Nobody else can self-register. Team members must be added by an admin. See [Team Members](team-members.md).
|
||||
|
||||
---
|
||||
|
||||
## Next steps
|
||||
|
||||
- **[Deploy on Cloudron](deploy-cloudron.md)** — the supported, packaged deployment with TLS and backups handled for you.
|
||||
- **[Deploy with Docker](deploy-docker.md)** — run the same image anywhere, behind your own reverse proxy.
|
||||
- **[Team Members](team-members.md)** — bootstrap the first admin and invite people.
|
||||
- **[Connecting Clients](clients.md)** — wire up Claude Code, Pi, OpenCode, and raw API keys.
|
||||
- **[Usage and Model Policy](usage-and-policy.md)** — per-member spend tracking and the model allowlist.
|
||||
- **[Security Model](security.md)** — the exact auth rules, public paths, and restricted endpoints.
|
||||
- **[Troubleshooting](troubleshooting.md)** — diagnosing the failures people actually hit.
|
||||
@@ -0,0 +1,172 @@
|
||||
# Security Model
|
||||
|
||||
The whole design rests on one property: **the daemon has no authentication because it is never reachable.** `routstrd` binds loopback-only on port `8009`; the auth proxy on `8008` is the single public surface, and it is the only component that makes authorisation decisions.
|
||||
|
||||
!!! danger "Never publish port 8009"
|
||||
The daemon is unauthenticated by design. Exposing it — or port-forwarding it for debugging, or forgetting to restrict a Docker port mapping to loopback — removes the entire security model at once. Anyone who reaches it can read the wallet, list every member's clients, and spend the team's funds.
|
||||
|
||||
---
|
||||
|
||||
## Request decision flow
|
||||
|
||||
The proxy is **default-deny**. A request is only forwarded if some rule explicitly allows it.
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
A["request arrives on 8008"] --> B{"management path?<br/>npubs, clients, usage"}
|
||||
B -->|yes| C["own handler<br/>NIP-98 required"]
|
||||
B -->|no| D{"GET or HEAD<br/>on a public path?"}
|
||||
D -->|yes| E["forward, no auth"]
|
||||
D -->|no| F{"Authorization header?"}
|
||||
F -->|missing| G["401"]
|
||||
F -->|"Bearer sk-..."| H{"key found in clients?"}
|
||||
H -->|no| I["401"]
|
||||
H -->|yes| J{"restricted path?<br/>wallet, node control"}
|
||||
J -->|yes| K["403"]
|
||||
J -->|no| L["forward with header intact"]
|
||||
F -->|"Nostr event"| M{"valid NIP-98?<br/>url, method, body hash, sig"}
|
||||
M -->|no| N["401"]
|
||||
M -->|yes| O{"pubkey registered?<br/>and role sufficient?"}
|
||||
O -->|no| P["403"]
|
||||
O -->|yes| Q["forward, header stripped"]
|
||||
```
|
||||
|
||||
Two details are easy to miss:
|
||||
|
||||
- **Public means `GET`/`HEAD` only.** The proxy applies the public-path rule only to read methods, because the daemon routes a `POST` to those same paths as a *paid* request. `GET /v1/models` needs no credential; `POST /v1/models` does.
|
||||
- **Management paths are matched before anything else.** `/npubs`, `/clients`, `/clients/add`, `/clients/delete`, `/usage`, and `/usage/summary` are handled by the proxy's own handlers and never forwarded to the daemon wholesale.
|
||||
|
||||
---
|
||||
|
||||
## Public paths
|
||||
|
||||
Reachable with no credential at all (`GET`/`HEAD` only):
|
||||
|
||||
| Path | Purpose |
|
||||
|---|---|
|
||||
| `/health` | Liveness. Used by Cloudron's health check and by the proxy's own upstream probe. |
|
||||
| `/ping` | Lightweight reachability. |
|
||||
| `/models` | Model directory. |
|
||||
| `/v1/models` | OpenAI-compatible model list. |
|
||||
| `/models/*`, `/v1/models/*` | Prefixes covering per-model detail paths. |
|
||||
|
||||
This is intentional — an agent needs to discover models before it has a key, and provider discovery is public information in Routstr.
|
||||
|
||||
---
|
||||
|
||||
## Credential one: API keys (`Bearer sk-...`)
|
||||
|
||||
An API key is looked up in the client records. If no client carries it, the request is rejected with `401 Invalid API key.`
|
||||
|
||||
Keys are **deliberately narrow**. A valid key is refused with `403` on every restricted path:
|
||||
|
||||
| Restricted endpoint | Why |
|
||||
|---|---|
|
||||
| `/wallet/status`, `/wallet/unlock`, `/wallet/balance` | An inference key must not read wallet state. |
|
||||
| `/wallet/receive/cashu`, `/wallet/receive/bolt11` | No minting funds with a key. |
|
||||
| `/wallet/send/cashu`, `/wallet/send/bolt11` | Admin-only in any case. |
|
||||
| `/wallet/mints`, `/wallet/mints/info` | No mint inspection. |
|
||||
| `/stop`, `/refund`, `/refund/xcashu` | No node control. |
|
||||
| `/providers`, `/providers/enable`, `/providers/disable` | `?refresh=true` rewrites the stored provider list. |
|
||||
| `/nwc/*` | No payment-channel access. |
|
||||
| `/npubs`, `/clients/add`, `/clients/delete`, `/usage` | No management surface at all. |
|
||||
|
||||
The rule of thumb: **an API key buys inference, and nothing else.**
|
||||
|
||||
Key handling on the way through: the `Authorization` header is **preserved** so the daemon can validate the key itself, and its own accounting stays authoritative.
|
||||
|
||||
---
|
||||
|
||||
## Credential two: NIP-98 (`Nostr <base64-event>`)
|
||||
|
||||
Management and wallet operations require a signed [NIP-98](https://github.com/nostr-protocol/nips/blob/master/98.md) event. This is not a bearer token — it is a signature over the specific request, so it cannot be replayed against a different endpoint.
|
||||
|
||||
The proxy enforces, in order:
|
||||
|
||||
| Check | Rule |
|
||||
|---|---|
|
||||
| Event kind | must be `27235` |
|
||||
| Timestamp | within **±60 seconds** of now |
|
||||
| `u` tag | must equal the **absolute request URL**, including scheme and host |
|
||||
| `method` tag | must match the HTTP method (case-insensitive) |
|
||||
| `payload` tag | **required when the body is non-empty**; must equal the SHA-256 hex digest of the raw body, compared in constant time |
|
||||
| Signature | verified with `verifyEvent` |
|
||||
|
||||
The proxy then looks the pubkey up in `routstr_auth_npubs`:
|
||||
|
||||
- **Not registered** → `403`. The error message is context-aware: on a node with no npubs at all it tells you to run `routstrd npubs register`; otherwise it says registered auth is required.
|
||||
- **Registered but role insufficient** → `403 Admin access required.`
|
||||
- **Registered and sufficient** → forwarded, with the `Authorization` header **stripped** so it does not reach the daemon or the upstream provider.
|
||||
|
||||
!!! warning "Behind a reverse proxy, forwarded headers are not optional"
|
||||
The `u` tag is checked against the **public** URL the client signed. The proxy reconstructs that URL from `X-Forwarded-Proto` and `X-Forwarded-Host` (falling back to `Host`). If your reverse proxy does not set them, the comparison fails and every NIP-98 request is rejected with `NIP-98 URL tag does not match this request.` See the nginx snippet in [Deploy with Docker](deploy-docker.md#put-tls-in-front).
|
||||
|
||||
!!! note "The ±60 second window means clock skew matters"
|
||||
A client whose clock is more than a minute off will produce events that are rejected as `outside the allowed window`. If one machine alone fails to authenticate, check its clock before suspecting the node.
|
||||
|
||||
---
|
||||
|
||||
## Role requirements by endpoint
|
||||
|
||||
| Endpoint group | Required |
|
||||
|---|---|
|
||||
| `/wallet/send/cashu`, `/wallet/send/bolt11` | `admin` |
|
||||
| `/wallet/status`, `/wallet/unlock`, `/wallet/balance`, `/wallet/receive/*`, `/wallet/mints*`, `/stop`, `/refund*`, `/providers*`, `/nwc/*` | any registered npub (`admin` or `user`) |
|
||||
| `/clients`, `/clients/add`, `/clients/delete` | any registered npub, **scoped to own clients** |
|
||||
| `/usage`, `/usage/summary` | any registered npub, **scoped to own usage** |
|
||||
| `/npubs` read | any registered npub |
|
||||
| `/npubs` create / update / delete | `admin` |
|
||||
| Everything else | valid API key **or** registered npub |
|
||||
|
||||
---
|
||||
|
||||
## Bootstrap window
|
||||
|
||||
While `routstr_auth_npubs` is empty, `POST /npubs` is accepted **without authentication**. This exists solely so a fresh node can be claimed, and it closes permanently after the first registration.
|
||||
|
||||
The practical implication: a node that is deployed and healthy but has not had its first admin register is **unclaimed**. Treat deployment and bootstrap as one operation.
|
||||
|
||||
There is deliberately no hardcoded default admin in the image. The absence of one means an image cannot be shipped with a known admin key — but it also means a half-finished deployment is claimable by whoever finds it first.
|
||||
|
||||
As an alternative to the interactive step, admins can be seeded with `ROUTSTRD_AUTH_ADMIN_NPUBS`, `ROUTSTRD_AUTH_ADMIN_PUBKEYS`, or `ROUTSTRD_AUTH_BOOTSTRAP_NPUB`. Rows created this way are tagged `source = 'env'` and **reconciled at every startup** — remove the value from the environment and the row is deleted, which is a clean way to authorise a node declaratively.
|
||||
|
||||
---
|
||||
|
||||
## CORS
|
||||
|
||||
The proxy answers with:
|
||||
|
||||
```text
|
||||
Access-Control-Allow-Origin: *
|
||||
Access-Control-Allow-Methods: GET, POST, PATCH, DELETE, OPTIONS
|
||||
Access-Control-Allow-Headers: Authorization, Content-Type, X-Cashu, X-Routstr-Model
|
||||
Access-Control-Expose-Headers: X-Cashu, X-Routstr-Request-Id, X-Routstr-Cost-Msats,
|
||||
X-Routstr-Cost-Usd, X-Routstr-Input-Cost-Msats,
|
||||
X-Routstr-Output-Cost-Msats
|
||||
```
|
||||
|
||||
A wildcard origin is safe **here specifically** because the app uses no cookies and no sessions. There is no ambient browser identity for a cross-origin page to borrow — a malicious page cannot make an authenticated request on a visitor's behalf, because every non-public request still needs its own API key or signature.
|
||||
|
||||
!!! warning "If you ever add cookie or session auth, revisit this"
|
||||
The wildcard is only correct while authentication is entirely credential-based. Adding session cookies would turn this into a real vulnerability.
|
||||
|
||||
---
|
||||
|
||||
## Hardening checklist
|
||||
|
||||
- [ ] **Daemon is loopback-only.** Verify `8009` is not published (`docker port routstr-remote`, or check the Cloudron app's port config).
|
||||
- [ ] **TLS everywhere.** No member or agent should ever send an `sk-...` key over plaintext HTTP.
|
||||
- [ ] **First admin registered** immediately after install.
|
||||
- [ ] **Reverse proxy sets `X-Forwarded-Proto` / `X-Forwarded-Host`**, or NIP-98 fails.
|
||||
- [ ] **Streaming hangs are fixed with timeouts, not by buffering.** Disable `proxy_buffering` and raise `proxy_read_timeout`; do not "fix" a truncated stream by publishing the daemon directly.
|
||||
- [ ] **`ROUTSTRD_AUTH_ADMIN_NPUBS` reflects reality** if you use env bootstrapping — those rows are deleted on restart when the variable changes.
|
||||
- [ ] **Departed members have their clients deleted**, not just their npub. Deleting an npub does **not** revoke existing API keys.
|
||||
- [ ] **Backups cover `/app/data`** and are tested, including the `nsec` in `routstrd/config.json`.
|
||||
- [ ] **Model allowlist verified** if you rely on it — it fails open when the model list has not been populated.
|
||||
|
||||
---
|
||||
|
||||
## Next steps
|
||||
|
||||
- [Troubleshooting](troubleshooting.md) — diagnosing `401` and `403` responses.
|
||||
- [Team Members](team-members.md#what-revocation-does-and-does-not-do) — the two-step offboarding that revocation alone does not cover.
|
||||
@@ -0,0 +1,177 @@
|
||||
# Team Members
|
||||
|
||||
Access to a team node is an entry in one table: `routstr_auth_npubs`. Each row holds a Nostr pubkey, an optional display name, and a `role` of `admin` or `user`. Adding someone grants access; deleting their row revokes it.
|
||||
|
||||
There are no passwords, no invite links, and no email addresses. Identity is a Nostr keypair that each person generates on their own machine.
|
||||
|
||||
---
|
||||
|
||||
## The invite loop
|
||||
|
||||
A new member does the first two steps themselves; an existing admin does the third.
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
participant M as New member
|
||||
participant A as Existing admin
|
||||
participant N as Team node
|
||||
|
||||
M->>M: install the routstrd CLI
|
||||
M->>N: set the remote URL
|
||||
N-->>M: generates keypair and prints npub
|
||||
M->>A: send npub out of band
|
||||
A->>N: add npub with role user
|
||||
N-->>A: access confirmed
|
||||
M->>N: add a client integration
|
||||
N-->>M: API key issued
|
||||
```
|
||||
|
||||
### 1. The member installs the CLI and connects
|
||||
|
||||
```bash
|
||||
bun i -g routstrd
|
||||
routstrd remote https://team.example.com
|
||||
```
|
||||
|
||||
`routstrd remote` writes the node URL into `~/.routstrd/config.json` and, **only if no Nostr identity exists yet**, generates one and prints the npub:
|
||||
|
||||
```text
|
||||
Remote daemon URL set to: https://team.example.com
|
||||
|
||||
A new Nostr identity has been generated for remote authentication.
|
||||
Your npub: npub1abc...xyz
|
||||
You can view it in the config file at: /home/bob/.routstrd/config.json
|
||||
```
|
||||
|
||||
If you already had an identity, it is reused and no npub is printed — run `routstrd remote` with no arguments to display the current node and identity.
|
||||
|
||||
!!! tip "Your npub is not a secret"
|
||||
It is a public key and safe to paste into a team chat. The corresponding `nsec` lives in `~/.routstrd/config.json` and **is** a secret: it signs every management request. Never share it, and treat any host that has it as holding that member's credentials.
|
||||
|
||||
### 2. The member sends their npub to an admin
|
||||
|
||||
Out of band — chat, ticket, whatever. There is no self-service join.
|
||||
|
||||
### 3. An admin adds them
|
||||
|
||||
```bash
|
||||
routstrd npubs add npub1abc...xyz --name "Bob"
|
||||
```
|
||||
|
||||
The role defaults to `user`. The new member can now run inference and manage their own clients. To make them an admin, pass `--role admin` (or promote later).
|
||||
|
||||
---
|
||||
|
||||
## Bootstrap: the first admin
|
||||
|
||||
A brand-new node has an empty table, which is a special case: `POST /npubs` is accepted **without authentication** so that someone can claim the node.
|
||||
|
||||
```bash
|
||||
routstrd remote https://team.example.com
|
||||
routstrd npubs register --name "Alice"
|
||||
```
|
||||
|
||||
`npubs register` is deliberately narrow — it refuses to do anything if any npub already exists:
|
||||
|
||||
```text
|
||||
Admin npubs already configured (3). Ask your admin to add your npub.
|
||||
Your npub: npub1...
|
||||
```
|
||||
|
||||
So `register` only ever works once per node. After that, `npubs add` is the command, and it requires admin NIP-98 auth.
|
||||
|
||||
!!! warning "Claim the node during install"
|
||||
Between the app becoming healthy and the first `npubs register`, the node is unclaimed — anybody who reaches the URL can become admin. See [Deploy on Cloudron](deploy-cloudron.md#bootstrap-the-first-admin).
|
||||
|
||||
---
|
||||
|
||||
## Command reference
|
||||
|
||||
All of these talk to the auth proxy over NIP-98, so the caller must be a registered npub, and the mutating ones require the `admin` role.
|
||||
|
||||
| Command | Role needed | Notes |
|
||||
|---|---|---|
|
||||
| `routstrd npubs list` | any registered | Shows role and name for everyone, and marks your own row with `→ you`. |
|
||||
| `routstrd npubs register` | none, **once** | Only succeeds while the table is empty. |
|
||||
| `routstrd npubs add <npub>` | admin | Accepts `npub1...` or a 64-char hex pubkey. `--role admin\|user` (default `user`), `--name`. |
|
||||
| `routstrd npubs update <npub>` | admin | `--role` and/or `--name`. Passing `--name ""` clears the name. |
|
||||
| `routstrd npubs delete <npub>` | admin | Revokes access. |
|
||||
|
||||
Names are trimmed and capped at 64 characters.
|
||||
|
||||
`routstrd npubs list` output looks like this:
|
||||
|
||||
```text
|
||||
Npubs (3):
|
||||
- npub1qqq...4f2 [admin] "Alice" → you
|
||||
- npub1xxx...9k7 [user] "Bob"
|
||||
- npub1zzz...3md [user]
|
||||
```
|
||||
|
||||
If your own npub is missing from the list, the CLI tells you whom to send it to:
|
||||
|
||||
```text
|
||||
Your npub is not in the npub list. Ask an admin to add your npub:
|
||||
npub1yyy...0pl
|
||||
```
|
||||
|
||||
## Underlying HTTP API
|
||||
|
||||
The CLI is a thin wrapper over four endpoints on the auth proxy. Useful for scripting or a custom onboarding form.
|
||||
|
||||
| Method | Path | Auth | Body |
|
||||
|---|---|---|---|
|
||||
| `GET` | `/npubs` | any registered npub | — |
|
||||
| `POST` | `/npubs` | none **if the table is empty**, otherwise admin | `{ "npub": "npub1..." }` or `{ "pubkey": "<hex>" }`, plus optional `role` and `name` |
|
||||
| `PATCH` | `/npubs` | admin | `{ "npub": "npub1..." }` plus `role` and/or `name` (`name: null` clears it) |
|
||||
| `DELETE` | `/npubs/<npub-or-pubkey>` | admin | — (also accepts `/npubs?npub=...`) |
|
||||
|
||||
Responses:
|
||||
|
||||
- `GET /npubs` returns `{ "npubs": [ { "npub": "...", "name": "...", "role": "admin" } ] }`.
|
||||
- Adding a pubkey that is **already registered** returns `409` rather than silently succeeding; use `PATCH` to change an existing entry.
|
||||
- Every mutation is performed by, and recorded against, the requesting admin.
|
||||
|
||||
!!! note "Revocation is immediate"
|
||||
Roles and removals are read from the database on **every request** with no caching layer. Deleting an npub stops that member's management access on their next request. Their **API keys are a separate matter** — see below.
|
||||
|
||||
---
|
||||
|
||||
## What revocation does and does not do
|
||||
|
||||
Both halves of offboarding are separate, and only the first is available through the normal member-facing API.
|
||||
|
||||
**1. Management access — revoked by deleting the npub.**
|
||||
|
||||
```bash
|
||||
routstrd npubs delete npub1abc...xyz
|
||||
```
|
||||
|
||||
Roles and rows are read from the database on **every request** with no caching layer, so the next NIP-98 request from that key is rejected immediately.
|
||||
|
||||
**2. Inference keys — a separate, manually managed thing.**
|
||||
|
||||
The Bearer path validates an `sk-...` key by looking it up in the client records and nothing else. Deleting an npub does **not** touch those rows, so an offboarded member's existing API keys keep working for inference until the client itself is deleted.
|
||||
|
||||
Here the proxy's scoping matters: `/clients`, `/clients/add`, and `/clients/delete` are **strictly owner-scoped** — the proxy filters and authorises by the calling npub. Being an `admin` does not grant access to a *colleague's* client records through those endpoints. So the realistic offboarding paths are:
|
||||
|
||||
- **Have the member delete their own clients** before you remove their npub: `routstrd clients list`, then `routstrd clients delete <id>`.
|
||||
- **Or do it on the node itself.** Inside the container the CLI talks to the daemon directly on loopback, where there is no auth layer and therefore no ownership filter:
|
||||
|
||||
```bash
|
||||
cloudron exec --app routstr.example.com
|
||||
routstrd clients list # every client on the node, all owners
|
||||
routstrd clients delete <id>
|
||||
```
|
||||
|
||||
On the node, client IDs appear **with** their owner suffix (`my-laptop-4f2x9k7`) — see [Connecting Clients](clients.md). That suffix is exactly what tells you which member a client belongs to.
|
||||
|
||||
!!! danger "Removing an npub is not the same as revoking access"
|
||||
If you skip step 2, a departed member's agents continue to consume the team's wallet. Always pair `npubs delete` with deleting their client records.
|
||||
|
||||
---
|
||||
|
||||
## Next steps
|
||||
|
||||
- [Connecting Clients](clients.md) — get each member's agents talking to the node.
|
||||
- [Usage and Model Policy](usage-and-policy.md) — see what each person is spending.
|
||||
@@ -0,0 +1,161 @@
|
||||
# Troubleshooting
|
||||
|
||||
Most team-node problems are one of five things: the daemon never came up, the wrong database, a credential problem, a clock or proxy-header problem, or a networking/timeout issue around streaming.
|
||||
|
||||
---
|
||||
|
||||
## First triage
|
||||
|
||||
Run these in order. Together they answer "is the node up, does it have people, and is the proxy seeing the right database".
|
||||
|
||||
```bash
|
||||
# 1. Is the public surface alive?
|
||||
curl -sS https://team.example.com/health
|
||||
|
||||
# 2. Do both processes run, and what did they log on boot?
|
||||
cloudron logs --app routstr.example.com | tail -50 # or: docker logs routstr-remote
|
||||
|
||||
# 3. Can the proxy see the database, and does it know your people?
|
||||
cloudron exec --app routstr.example.com
|
||||
routstrd-auth validate
|
||||
```
|
||||
|
||||
Step 3 is the most informative. Its output ends with a line like `✅ DB accessible. 3 npub(s) registered (1 admin, 2 user).` — if that count is wrong, or the path is wrong, you have found your problem.
|
||||
|
||||
On startup the proxy also logs a summary, and warns loudly if the node is unclaimed:
|
||||
|
||||
```text
|
||||
routstrd-auth proxy listening on http://0.0.0.0:8008
|
||||
Upstream: http://localhost:8009
|
||||
DB path: /app/data/routstrd/routstr.db
|
||||
Registered npubs: 0
|
||||
Model allowlist: disabled
|
||||
Warning: no registered npub/pubkey. The first admin can be registered without auth using POST /npubs.
|
||||
```
|
||||
|
||||
!!! warning "`Registered npubs: 0` on an established node is an emergency"
|
||||
Either the database was lost (usually a missing volume mount) or the proxy is pointed at the wrong file. While the table is empty the node is claimable by anyone who reaches it. Fix it before anything else — see [Lost identity or empty npub list](#lost-identity-or-empty-npub-list).
|
||||
|
||||
---
|
||||
|
||||
## Error reference
|
||||
|
||||
### Startup
|
||||
|
||||
| Symptom | Cause | Fix |
|
||||
|---|---|---|
|
||||
| `Timed out waiting for routstrd to become ready.` | The proxy's launcher waited 120 seconds for the database file to exist **and** for `http://localhost:8009/health` to answer. The daemon is unhealthy or crashing. | Read the daemon's log lines above this one. It is a daemon problem, not a proxy problem. |
|
||||
| `Database not found at /app/data/routstrd/routstr.db. Make sure routstrd has been initialized (routstrd onboard).` | The proxy shares the daemon's database and never creates the schema. It ran before the daemon had ever initialised. | Start the daemon first (`routstrd start`, or let `supervisord` do it — priority 10 before 20). |
|
||||
| `Invalid admin Nostr pubkey(s): <value>. Use npub or 64-char hex pubkeys.` | A bootstrap admin variable contains something unparseable. | Fix `ROUTSTRD_AUTH_ADMIN_NPUBS` / `_PUBKEYS` / `_BOOTSTRAP_NPUB`. Use `npub1...` or 64-char hex. |
|
||||
| Container exits with `Illegal instruction` / `SIGILL` | Bun's default x64 build needs AVX/AVX2, which some hosts and VMs do not expose. | The shipped image already uses the **baseline** build for this reason. If you build your own, keep `BUN_TARGET=bun-linux-x64-baseline`. |
|
||||
| Proxy crash-loops immediately after a config change | Validation fails, so `start` exits non-zero and `supervisord` restarts it. | Run `routstrd-auth validate` to see the message. |
|
||||
|
||||
### Authentication
|
||||
|
||||
| Response | Meaning | Fix |
|
||||
|---|---|---|
|
||||
| `401 Missing Authorization header. Use 'Authorization: Bearer sk-...' or 'Authorization: Nostr <base64-event>'.` | No credential sent, and the path is not public. Remember public paths are `GET`/`HEAD` only. | Send a credential, or use a read method on a public path. |
|
||||
| `401 Invalid API key.` | The key is not in the client records. | Recover it with `routstrd clients add --name "<existing name>"`, which prints the existing key. |
|
||||
| `403 API keys cannot access this endpoint. Use NIP-98 auth from a registered npub/pubkey.` | You used an `sk-...` key on a wallet, node-control, or management endpoint. | Use the CLI, which signs with NIP-98 automatically. |
|
||||
| `403 Admin access required. Only admin npubs can perform this action.` | The npub is registered but holds the `user` role. | An admin promotes them: `routstrd npubs update <npub> --role admin`. |
|
||||
| `403 This endpoint requires a registered npub/pubkey, but none is configured. Register the first admin with 'routstrd npubs register'.` | The npub table is empty. | Bootstrap the first admin. |
|
||||
| `403 This endpoint requires NIP-98 auth from a registered npub/pubkey.` | The signature was valid but the pubkey is not in the table. | An admin adds it: `routstrd npubs add <npub>`. |
|
||||
| `401 Invalid Authorization format. Expected 'Bearer sk-...' or 'Nostr <base64-event>'.` | The header used a different scheme or a typo'd prefix. | Fix the prefix. |
|
||||
|
||||
A useful diagnostic: the token **type** determines the error. A `403` naming a specific capability means you authenticated successfully and were then refused by policy. A `401` means you did not authenticate at all.
|
||||
|
||||
### NIP-98 signature rejections
|
||||
|
||||
These all return `401` with a precise reason.
|
||||
|
||||
| Message | Cause | Fix |
|
||||
|---|---|---|
|
||||
| `NIP-98 URL tag does not match this request.` | The most common one. The `u` tag holds the URL the client signed, and the proxy reconstructs the public URL from `X-Forwarded-Proto` / `X-Forwarded-Host` (falling back to `Host`). Behind TLS termination, a missing forwarded header makes the reconstructed URL `http://...` while the client signed `https://...`. | Configure the reverse proxy to set both headers. See [Deploy with Docker](deploy-docker.md#put-tls-in-front). |
|
||||
| `NIP-98 event timestamp is outside the allowed window.` | Events must be within **±60 seconds**. Clock skew. | Sync the client's clock (`timedatectl`, NTP). If only one machine fails, it is that machine. |
|
||||
| `NIP-98 payload tag is required for requests with a body.` \| `NIP-98 payload tag does not match the request body hash.` | The signed SHA-256 does not match the body that arrived — something rewrote the body in transit. | Check for a middleware, WAF, or forward proxy that re-encodes request bodies. |
|
||||
| `NIP-98 method tag does not match this request.` | The event was signed for a different method (often a `GET` signature reused on a `POST`). | Sign per request; do not reuse events. |
|
||||
| `Invalid NIP-98 event signature.` \| `Invalid NIP-98 event kind.` | Corrupted token, or not a kind `27235` event. | Regenerate the request with the CLI. |
|
||||
| `Invalid NIP-98 token encoding.` \| `Invalid NIP-98 event JSON.` | The base64 payload is truncated — common when a long `Authorization` header is split or truncated by a client. | Check for a header-size limit in the client or proxy. |
|
||||
|
||||
### Client-side CLI messages
|
||||
|
||||
| Message | Meaning | Fix |
|
||||
|---|---|---|
|
||||
| `The daemon at <url> rejected this account.` then `Register/authorize this npub on the remote daemon first: <npub>` | The node does not recognise your npub. | Send the printed npub to an admin. |
|
||||
| `No remote node is set up.` | No `daemonUrl` in `~/.routstrd/config.json`. | `routstrd remote https://team.example.com`. |
|
||||
| `Your npub is not in the npub list. Ask an admin to add your npub:` | `npubs list` worked (so you *are* registered) but shows you as absent — typically a stale local identity after a config reset. | Re-run `routstrd remote` to display your current npub, and confirm with an admin which one is registered. |
|
||||
| `Daemon is not running` | The local daemon is unreachable. | Only relevant when running from source; `routstrd start`. |
|
||||
|
||||
---
|
||||
|
||||
## Common scenarios
|
||||
|
||||
### Streaming responses get cut off mid-answer
|
||||
|
||||
The symptom is a reply that stops abruptly — often after exactly the same number of seconds — with no error from the model.
|
||||
|
||||
This is **not** a node problem. It is response buffering or an idle timeout in a proxy in front of the node. Model turns can be silent for a long time while reasoning or waiting on tools, and intermediaries treat that silence as a dead connection.
|
||||
|
||||
Fix, in order of what actually bites:
|
||||
|
||||
1. **Disable response buffering** where the proxy talks to `8008`. The node already sends `X-Accel-Buffering: no`, but the fronting proxy must also be told (`proxy_buffering off` in nginx).
|
||||
2. **Raise read timeouts** well above the default (`proxy_read_timeout 3600s`). nginx's 60-second default is the usual culprit.
|
||||
3. **Do not work around it by exposing the daemon.** Publishing `8009` removes authentication entirely.
|
||||
|
||||
The proxy itself disables Bun's per-request idle timeout (`server.timeout(req, 0)`) precisely because valid streams can be quiet for a long time; anything still timing out is outside the node.
|
||||
|
||||
### Lost identity or empty npub list
|
||||
|
||||
If `routstrd npubs list` is empty after a restart, or `routstrd-auth validate` reports `0 npub(s)`, the node lost its data directory. In Docker this is almost always an **unmounted or changed volume**: `/app/data` is the only persistent path.
|
||||
|
||||
The damage is two-fold:
|
||||
|
||||
- The `nsec` in `routstrd/config.json` is gone, so the node has a **new** identity.
|
||||
- The npub table is empty, so `POST /npubs` is **unauthenticated again** — anyone who reaches the URL can claim the node.
|
||||
|
||||
Recover in this order:
|
||||
|
||||
1. **Stop the app** so nobody claims it.
|
||||
2. Restore `/app/data` from a backup, or re-mount the correct volume and restart.
|
||||
3. If data is unrecoverable, accept the new identity and re-register the first admin — then have everyone whose npub was lost send theirs again, and recreate their clients (keys do not survive either).
|
||||
|
||||
### Model requests return `403` unexpectedly
|
||||
|
||||
Either the sender is genuinely outside the allowlist, or the allowlist is enforcing a **stale** list.
|
||||
|
||||
```bash
|
||||
cloudron exec --app routstr.example.com
|
||||
sqlite3 /app/data/routstrd/routstr.db \
|
||||
"SELECT substr(value,1,120) FROM sdk_storage WHERE key = 'routstr21Models';"
|
||||
```
|
||||
|
||||
- **No output:** the daemon never bootstrapped the list, so the proxy is failing **open** and allowing everything. The `403` is coming from somewhere else.
|
||||
- **Output present but missing the model you want:** the list is stale. Refresh it with `routstrd clients --manual-refresh`, and check the scheduled job has not been disabled with `--disable-automatic-refresh`.
|
||||
|
||||
Remember the check applies to `POST`/`PUT`/`PATCH` bodies containing a `model` field only, and comparisons are case-sensitive.
|
||||
|
||||
### A member cannot authenticate at all
|
||||
|
||||
Work down this list:
|
||||
|
||||
1. **Are they registered?** `routstrd npubs list` as an admin.
|
||||
2. **Is their npub the one you registered?** They run `routstrd remote` with no arguments to print the identity actually in their config. A machine with an old or regenerated `nsec` presents a different npub.
|
||||
3. **Is their clock correct?** Off by more than 60 seconds means every NIP-98 event is rejected.
|
||||
4. **Are forwarded headers set?** If *everyone* is failing with a URL tag mismatch, it is the reverse proxy, not the people.
|
||||
5. **Do they have admin-requiring needs?** A `403 Admin access required` is a role problem, not an authentication problem.
|
||||
|
||||
### The node works but nobody can see usage
|
||||
|
||||
Usage is attributed per client, and `/usage` is scoped to the caller's npub. A member with no clients has no usage to show. Aggregate figures require running the CLI **on the node**, where the daemon is unauthenticated and unfiltered:
|
||||
|
||||
```bash
|
||||
cloudron exec --app routstr.example.com
|
||||
routstrd top
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Next steps
|
||||
|
||||
- [Security Model](security.md) — the rules behind the `401`s and `403`s.
|
||||
- [Usage and Model Policy](usage-and-policy.md) — allowlist behaviour and its fail-open case.
|
||||
@@ -0,0 +1,144 @@
|
||||
# Usage and Model Policy
|
||||
|
||||
Two things a team admin cares about: **who is spending what**, and **which models the team may use**. The first is always tracked; the second is an opt-in policy.
|
||||
|
||||
---
|
||||
|
||||
## Usage tracking
|
||||
|
||||
Every request is attributed to the **client** that made it, and every client is owned by exactly one member. That chain is what produces per-person reporting.
|
||||
|
||||
### As a member
|
||||
|
||||
```bash
|
||||
routstrd usage # your own usage summary
|
||||
routstrd top # interactive TUI, alias for 'monitor'
|
||||
routstrd balance # wallet balance and status
|
||||
```
|
||||
|
||||
### As the operator, on the node
|
||||
|
||||
The most complete view is the TUI, run inside the container:
|
||||
|
||||
```bash
|
||||
cloudron exec --app routstr.example.com
|
||||
routstrd top
|
||||
```
|
||||
|
||||
The **clients** tab shows individual usage, and because each client ID carries the **last 7 characters of its owner's npub**, you can tell teammates apart at a glance even when several of them named their client `my-laptop`:
|
||||
|
||||
```text
|
||||
my-laptop-4f2x9k7 1,204 req $3.18
|
||||
my-laptop-9k7qp2 881 req $2.07
|
||||
ci-runner-3md1zz 412 req $0.94
|
||||
```
|
||||
|
||||
Run from the node, this view is **not** filtered to any one member — that is the difference between the CLI on the box and the same CLI on a laptop.
|
||||
|
||||
### Over HTTP
|
||||
|
||||
| Method | Path | Auth | Scope |
|
||||
|---|---|---|---|
|
||||
| `GET` | `/usage` | NIP-98 from a registered npub | The **caller's** usage only |
|
||||
| `GET` | `/usage/summary` | NIP-98 from a registered npub | The **caller's** usage only |
|
||||
|
||||
The proxy forces the scope: it takes the authenticated npub and sets the `npub` query parameter before forwarding to the daemon, so the daemon returns that member's records. There is no query parameter you can pass to widen it — a member cannot read a colleague's usage through the API. Aggregate visibility requires access to the node itself.
|
||||
|
||||
---
|
||||
|
||||
## The wallet
|
||||
|
||||
The team shares **one node wallet**. Funding it is a single action rather than N personal top-ups, and that is one of the main reasons to run a team node.
|
||||
|
||||
| Endpoint | Required role |
|
||||
|---|---|
|
||||
| `/wallet/status`, `/wallet/balance`, `/wallet/mints`, `/wallet/mints/info` | any registered npub |
|
||||
| `/wallet/receive/cashu`, `/wallet/receive/bolt11` | any registered npub |
|
||||
| `/wallet/send/cashu`, `/wallet/send/bolt11` | **admin only** |
|
||||
|
||||
Reading the balance and receiving funds are open to every registered member; **moving funds out is admin-only.** Note the consequence: members can see the team's total balance but cannot withdraw from it. API keys can do none of this.
|
||||
|
||||
Funding, mint management, and payment semantics are the daemon's domain and are documented in the provider and client guides rather than repeated here.
|
||||
|
||||
!!! warning "Usage is attributed, not isolated"
|
||||
There is no per-member balance and no spending cap on this layer. A member's clients spend from the same wallet everyone else does. If you need hard limits, they are not enforced here — track the usage view and manage access accordingly.
|
||||
|
||||
---
|
||||
|
||||
## Model allowlist
|
||||
|
||||
The proxy can restrict the team to the **Routstr 21 model list**. This is **disabled by default**; enable it with:
|
||||
|
||||
```bash
|
||||
ROUTSTRD_AUTH_MODEL_ALLOWLIST=true
|
||||
```
|
||||
|
||||
When enabled, a request naming a model outside the list is rejected with `403` **before it reaches the daemon**, so no tokens are spent. When disabled, every model passes through untouched.
|
||||
|
||||
### How it works
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
A["routstrd CLI or agent"] --> B["routstrd-auth<br/>checks auth, then model"]
|
||||
B -->|"allowed"| C["routstrd daemon"]
|
||||
B -->|"403 not allowed"| A
|
||||
C --> D["upstream provider"]
|
||||
E["Nostr kind 38423<br/>Routstr 21 list"] -->|fetched by daemon| F[("sdk_storage<br/>routstr21Models")]
|
||||
F -->|read on every request| B
|
||||
```
|
||||
|
||||
1. The **daemon's** SDK fetches the Routstr 21 list from Nostr (kind `38423` events) and stores it in the shared SQLite database under the `sdk_storage` table, key `routstr21Models`.
|
||||
2. The **proxy** reads that key from the same database. It has **zero Nostr dependency** — no relay connections, no keys of its own for this purpose.
|
||||
3. The value is read fresh on every request, with no caching, so a list update takes effect immediately.
|
||||
|
||||
Because the proxy shares the daemon's database, this costs a single indexed key lookup — typically under a millisecond.
|
||||
|
||||
### What is and is not checked
|
||||
|
||||
| Request | Checked |
|
||||
|---|---|
|
||||
| `POST` / `PUT` / `PATCH` with a JSON body containing `model` | yes |
|
||||
| `GET` requests, including `/models` and `/v1/models` | no — public paths, forwarded immediately |
|
||||
| Management endpoints (`/npubs`, `/clients`, `/usage`) | no — routed to their own handlers before this check |
|
||||
| Non-JSON bodies | no — no `model` field can be extracted |
|
||||
| Requests with no `model` field | no — forwarded, and the upstream produces the error |
|
||||
|
||||
Model IDs are compared **case-sensitively**, matching how they are stored in the list.
|
||||
|
||||
### Fail-open behaviour
|
||||
|
||||
If `routstr21Models` is absent — typically because the daemon has not bootstrapped it yet — the proxy **fails open and allows every model**. This is deliberate: the alternative is that a fresh node blocks all traffic until Nostr bootstrapping completes. The trade-off is that an allowlist can be silently ineffective early in a node's life, so verify it after enabling:
|
||||
|
||||
```bash
|
||||
# confirm the daemon has populated the list
|
||||
cloudron exec --app routstr.example.com
|
||||
sqlite3 /app/data/routstrd/routstr.db \
|
||||
"SELECT substr(value,1,120) FROM sdk_storage WHERE key = 'routstr21Models';"
|
||||
```
|
||||
|
||||
If that returns nothing, the allowlist is not yet meaningful.
|
||||
|
||||
### Performance note
|
||||
|
||||
When enforcement is enabled, the proxy buffers `POST`/`PUT`/`PATCH` request bodies so it can inspect the `model` field. That is a small latency cost on request upload. **Response streaming is unaffected** — SSE and LLM token streams pass through unbuffered, and the proxy explicitly tells intermediaries not to buffer. When the allowlist is disabled, the body is not buffered at all.
|
||||
|
||||
---
|
||||
|
||||
## Keeping the model list current
|
||||
|
||||
The list is updated by the daemon, not the proxy:
|
||||
|
||||
```bash
|
||||
routstrd clients --manual-refresh # refresh now
|
||||
routstrd clients --disable-automatic-refresh # stop the scheduled job
|
||||
routstrd clients --enable-automatic-refresh # resume it
|
||||
```
|
||||
|
||||
If the scheduled refresh job is disabled and nobody runs a manual refresh, the allowlist enforces a **stale** list — and would eventually stop matching newly approved models.
|
||||
|
||||
---
|
||||
|
||||
## Next steps
|
||||
|
||||
- [Security Model](security.md) — the full endpoint-by-endpoint auth matrix.
|
||||
- [Troubleshooting](troubleshooting.md) — including "why am I getting a 403".
|
||||
@@ -0,0 +1,27 @@
|
||||
"""add model_metadata to model_paths
|
||||
|
||||
Revision ID: a3f1b6c204de
|
||||
Revises: e5a6b7c8d9f0
|
||||
Create Date: 2026-09-15 21:50:00.000000
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import sqlalchemy as sa
|
||||
from alembic import op
|
||||
|
||||
revision = "a3f1b6c204de"
|
||||
down_revision = "e5a6b7c8d9f0"
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
op.add_column(
|
||||
"model_paths",
|
||||
sa.Column("model_metadata", sa.Text(), nullable=False, server_default="{}"),
|
||||
)
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
op.drop_column("model_paths", "model_metadata")
|
||||
@@ -91,6 +91,15 @@ nav:
|
||||
- Advanced Pricing: provider/advanced-pricing.md
|
||||
- Discovery: provider/discovery.md
|
||||
- Tor Support: provider/tor.md
|
||||
- Teams (Remote):
|
||||
- Overview: teams/index.md
|
||||
- Deploy on Cloudron: teams/deploy-cloudron.md
|
||||
- Deploy with Docker: teams/deploy-docker.md
|
||||
- Team Members: teams/team-members.md
|
||||
- Connecting Clients: teams/clients.md
|
||||
- Usage and Model Policy: teams/usage-and-policy.md
|
||||
- Security Model: teams/security.md
|
||||
- Troubleshooting: teams/troubleshooting.md
|
||||
- API Reference:
|
||||
- Overview: api/overview.md
|
||||
- Authentication: api/authentication.md
|
||||
|
||||
+8
-3
@@ -437,9 +437,10 @@ class ModelRow(SQLModel, table=True): # type: ignore
|
||||
class ModelPathRow(SQLModel, table=True): # type: ignore
|
||||
"""Upstream provider path a model is reachable through.
|
||||
|
||||
Discovery/visibility data only. ``model_id`` is intentionally NOT globally
|
||||
unique: it is the client-visible ``/v1/models`` id (``forwarded_model_id or
|
||||
id``) grouped across every provider that exposes the model. A single model
|
||||
Discovery data plus provider-specific model metadata. ``model_id`` is
|
||||
intentionally NOT globally unique: it is the client-visible ``/v1/models``
|
||||
id (``forwarded_model_id or id``) grouped across every provider that exposes
|
||||
the model. A single model
|
||||
can therefore have several rows — one per direct provider path plus one per
|
||||
OpenRouter sub-provider endpoint.
|
||||
"""
|
||||
@@ -476,6 +477,10 @@ class ModelPathRow(SQLModel, table=True): # type: ignore
|
||||
endpoint_name: str | None = Field(
|
||||
default=None, description="Human-readable endpoint display name"
|
||||
)
|
||||
model_metadata: str = Field(
|
||||
default="{}",
|
||||
description="JSON model metadata specific to this provider path",
|
||||
)
|
||||
upstream_provider_id: int = Field(
|
||||
index=True,
|
||||
foreign_key="upstream_providers.id",
|
||||
|
||||
+146
-62
@@ -24,6 +24,12 @@ from ..wallet import (
|
||||
deserialize_token_from_string,
|
||||
is_trusted_source_mint,
|
||||
)
|
||||
from .responses_input import (
|
||||
FILE_ID_URL_PREFIX,
|
||||
count_input_images,
|
||||
input_image_part_to_image_url,
|
||||
responses_input_to_messages,
|
||||
)
|
||||
|
||||
logger = get_logger(__name__)
|
||||
|
||||
@@ -179,7 +185,12 @@ async def calculate_discounted_max_cost(
|
||||
body: dict,
|
||||
model_obj: Any | None = None,
|
||||
) -> int:
|
||||
"""Calculate the discounted max cost for a request using model pricing when available."""
|
||||
"""Calculate the discounted max cost for a request using model pricing when available.
|
||||
|
||||
Completion discounts are trimmed from the largest declared cap among
|
||||
``max_tokens`` and ``max_completion_tokens`` (chat/completions) or
|
||||
``max_output_tokens`` (responses).
|
||||
"""
|
||||
if settings.fixed_pricing:
|
||||
return max_cost_for_model
|
||||
|
||||
@@ -225,17 +236,28 @@ async def calculate_discounted_max_cost(
|
||||
# for work the reservation never covered.
|
||||
prompt_tokens = estimate_prompt_tokens(body)
|
||||
|
||||
# Images are billed as tokens by the upstream but carry no text for
|
||||
# ``estimate_prompt_tokens`` to count, so they are estimated separately and
|
||||
# added on both the chat (``messages``) and Responses (``input``) paths.
|
||||
image_tokens = 0
|
||||
if isinstance(messages, list):
|
||||
image_tokens = await estimate_image_tokens_in_messages(messages)
|
||||
if image_tokens > 0:
|
||||
logger.debug(
|
||||
"Found images in request",
|
||||
extra={
|
||||
"model": model,
|
||||
"image_tokens": image_tokens,
|
||||
},
|
||||
)
|
||||
prompt_tokens += image_tokens
|
||||
image_tokens += await estimate_image_tokens_in_messages(messages)
|
||||
input_data = body.get("input")
|
||||
if isinstance(input_data, list):
|
||||
converted = responses_input_to_messages(input_data)
|
||||
if converted is None:
|
||||
image_tokens += count_input_images(input_data) * _MAX_ORIGINAL_IMAGE_TOKENS
|
||||
else:
|
||||
image_tokens += await estimate_image_tokens_in_messages(converted)
|
||||
if image_tokens > 0:
|
||||
logger.debug(
|
||||
"Found images in request",
|
||||
extra={
|
||||
"model": model,
|
||||
"image_tokens": image_tokens,
|
||||
},
|
||||
)
|
||||
prompt_tokens += image_tokens
|
||||
|
||||
if prompt_tokens > 0:
|
||||
estimated_prompt_delta_sats = (
|
||||
@@ -244,21 +266,39 @@ async def calculate_discounted_max_cost(
|
||||
if estimated_prompt_delta_sats > 0:
|
||||
adjusted = adjusted - math.floor(estimated_prompt_delta_sats * 1000)
|
||||
|
||||
max_tokens_raw = body.get("max_tokens", None)
|
||||
if max_tokens_raw is not None:
|
||||
# Completion caps arrive under several names: ``max_tokens`` (legacy
|
||||
# chat), ``max_completion_tokens`` (modern chat) and ``max_output_tokens``
|
||||
# (Responses API). When a request declares more than one, reserve against
|
||||
# the largest: upstream precedence between the fields varies by provider,
|
||||
# so the smaller cap may not be honored and the reservation must never
|
||||
# under-cover what the upstream could bill.
|
||||
max_tokens_int: int | None = None
|
||||
for cap_field in ("max_tokens", "max_completion_tokens", "max_output_tokens"):
|
||||
cap_raw = body.get(cap_field)
|
||||
if cap_raw is None:
|
||||
continue
|
||||
try:
|
||||
max_tokens_int = int(max_tokens_raw)
|
||||
cap_int = int(cap_raw)
|
||||
except (TypeError, ValueError):
|
||||
logger.warning(
|
||||
"Invalid max_tokens; ignoring in cost adjustment",
|
||||
extra={"max_tokens": str(max_tokens_raw)[:64], "model": model},
|
||||
"Invalid completion token cap; ignoring in cost adjustment",
|
||||
extra={
|
||||
"field": cap_field,
|
||||
"value": str(cap_raw)[:64],
|
||||
"model": model,
|
||||
},
|
||||
)
|
||||
else:
|
||||
estimated_completion_delta_sats = (
|
||||
max_completion_allowed_sats - max_tokens_int * model_pricing.completion
|
||||
)
|
||||
if estimated_completion_delta_sats > 0:
|
||||
adjusted = adjusted - math.floor(estimated_completion_delta_sats * 1000)
|
||||
continue
|
||||
max_tokens_int = (
|
||||
cap_int if max_tokens_int is None else max(max_tokens_int, cap_int)
|
||||
)
|
||||
|
||||
if max_tokens_int is not None:
|
||||
estimated_completion_delta_sats = (
|
||||
max_completion_allowed_sats - max_tokens_int * model_pricing.completion
|
||||
)
|
||||
if estimated_completion_delta_sats > 0:
|
||||
adjusted = adjusted - math.floor(estimated_completion_delta_sats * 1000)
|
||||
|
||||
logger.debug(
|
||||
"Discounted max cost computed",
|
||||
@@ -431,15 +471,46 @@ async def _fetch_image_from_url(url: str) -> bytes | None:
|
||||
return None
|
||||
|
||||
|
||||
# Patch-based image pricing (OpenAI ``detail: "original"``): the image is
|
||||
# covered with 32x32px patches and billed as ceil(patches * multiplier)
|
||||
# tokens, with no 512px-tile downscaling. The API rejects images above
|
||||
# 30,000 patches, so at the 1.2x multiplier documented for the
|
||||
# original-capable model families (gpt-5.4/5.5/5.6) the worst case a
|
||||
# single image can bill is 36,000 tokens.
|
||||
_IMAGE_PATCH_PX = 32
|
||||
_MAX_IMAGE_PATCHES = 30_000
|
||||
_MAX_ORIGINAL_IMAGE_TOKENS = (_MAX_IMAGE_PATCHES * 6 + 4) // 5 # 36,000
|
||||
|
||||
|
||||
def _calculate_original_image_tokens(width: int, height: int) -> int:
|
||||
"""Estimate tokens for an image billed at ``detail: "original"``.
|
||||
|
||||
Patch-based models cover the image with 32x32px patches and bill
|
||||
``ceil(patches * 1.2)`` tokens. The estimate is bounded by the
|
||||
30,000-patch rejection limit, which is more conservative than the
|
||||
per-model resizing patch budgets (e.g. 10,000 patches on gpt-5.4/5.5)
|
||||
so it never under-reserves.
|
||||
"""
|
||||
patches = ((width + _IMAGE_PATCH_PX - 1) // _IMAGE_PATCH_PX) * (
|
||||
(height + _IMAGE_PATCH_PX - 1) // _IMAGE_PATCH_PX
|
||||
)
|
||||
bounded = min(patches, _MAX_IMAGE_PATCHES)
|
||||
return (bounded * 6 + 4) // 5 # ceil(bounded * 1.2) in exact integer math
|
||||
|
||||
|
||||
def _calculate_image_tokens(width: int, height: int, detail: str = "auto") -> int:
|
||||
"""Calculate image tokens based on OpenAI's vision pricing.
|
||||
|
||||
For low detail: 85 tokens
|
||||
For high detail/auto: 85 base tokens + 170 tokens per 512px tile
|
||||
For original detail: patch-based pricing at the original resolution
|
||||
"""
|
||||
if detail == "low":
|
||||
return 85
|
||||
|
||||
if detail == "original":
|
||||
return _calculate_original_image_tokens(width, height)
|
||||
|
||||
if width > 2048 or height > 2048:
|
||||
aspect_ratio = width / height
|
||||
if width > height:
|
||||
@@ -492,7 +563,9 @@ async def estimate_image_tokens_in_messages(messages: list) -> int:
|
||||
continue
|
||||
|
||||
content_type = content_item.get("type")
|
||||
if content_type not in ("image_url", "input_image"):
|
||||
if content_type == "input_image":
|
||||
content_item = input_image_part_to_image_url(content_item)
|
||||
elif content_type != "image_url":
|
||||
continue
|
||||
|
||||
image_url_data = content_item.get("image_url")
|
||||
@@ -504,7 +577,7 @@ async def estimate_image_tokens_in_messages(messages: list) -> int:
|
||||
detail = "auto"
|
||||
elif isinstance(image_url_data, dict):
|
||||
url = image_url_data.get("url", "")
|
||||
detail = image_url_data.get("detail", "auto")
|
||||
detail = image_url_data.get("detail") or "auto"
|
||||
else:
|
||||
continue
|
||||
|
||||
@@ -512,56 +585,67 @@ async def estimate_image_tokens_in_messages(messages: list) -> int:
|
||||
continue
|
||||
|
||||
if url.startswith("data:image/"):
|
||||
try:
|
||||
header, base64_data = url.split(",", 1)
|
||||
image_bytes = base64.b64decode(base64_data)
|
||||
width, height = _get_image_dimensions(image_bytes)
|
||||
tokens = _calculate_image_tokens(width, height, detail)
|
||||
total_image_tokens += tokens
|
||||
logger.debug(
|
||||
"Calculated tokens for base64 image",
|
||||
extra={
|
||||
"width": width,
|
||||
"height": height,
|
||||
"detail": detail,
|
||||
"tokens": tokens,
|
||||
},
|
||||
)
|
||||
except Exception as e:
|
||||
logger.warning(
|
||||
"Failed to process base64 image",
|
||||
extra={"error": str(e)},
|
||||
)
|
||||
total_image_tokens += 85
|
||||
total_image_tokens += _data_url_image_tokens(url, detail)
|
||||
elif url.startswith(FILE_ID_URL_PREFIX):
|
||||
total_image_tokens += _worst_case_image_tokens(detail)
|
||||
elif fetches >= IMAGE_FETCH_MAX_PER_REQUEST:
|
||||
logger.warning(
|
||||
"Skipping image URL fetch above per-request limit",
|
||||
extra={"url": url[:100], "limit": IMAGE_FETCH_MAX_PER_REQUEST},
|
||||
)
|
||||
total_image_tokens += 85
|
||||
total_image_tokens += _worst_case_image_tokens(detail)
|
||||
else:
|
||||
fetches += 1
|
||||
image_bytes_or_none = await _fetch_image_from_url(url)
|
||||
if image_bytes_or_none:
|
||||
width, height = _get_image_dimensions(image_bytes_or_none)
|
||||
tokens = _calculate_image_tokens(width, height, detail)
|
||||
total_image_tokens += tokens
|
||||
logger.debug(
|
||||
"Calculated tokens for URL image",
|
||||
extra={
|
||||
"url": url[:100],
|
||||
"width": width,
|
||||
"height": height,
|
||||
"detail": detail,
|
||||
"tokens": tokens,
|
||||
},
|
||||
)
|
||||
else:
|
||||
total_image_tokens += 85
|
||||
total_image_tokens += _image_bytes_tokens(
|
||||
image_bytes_or_none, detail, source=url[:100]
|
||||
)
|
||||
|
||||
return total_image_tokens
|
||||
|
||||
|
||||
def _worst_case_image_tokens(detail: str) -> int:
|
||||
"""Dimensions unknown: reserve the most ``detail`` can bill."""
|
||||
if detail == "original":
|
||||
return _MAX_ORIGINAL_IMAGE_TOKENS
|
||||
return _calculate_image_tokens(2048, 2048, detail)
|
||||
|
||||
|
||||
def _data_url_image_tokens(url: str, detail: str) -> int:
|
||||
try:
|
||||
_, base64_data = url.split(",", 1)
|
||||
image_bytes = base64.b64decode(base64_data, validate=True)
|
||||
except Exception as e:
|
||||
logger.warning("Failed to decode base64 image", extra={"error": str(e)})
|
||||
return _worst_case_image_tokens(detail)
|
||||
return _image_bytes_tokens(image_bytes, detail, source="data-url")
|
||||
|
||||
|
||||
def _image_bytes_tokens(image_bytes: bytes | None, detail: str, source: str) -> int:
|
||||
if not image_bytes:
|
||||
return _worst_case_image_tokens(detail)
|
||||
try:
|
||||
width, height = Image.open(BytesIO(image_bytes)).size
|
||||
except Exception as e:
|
||||
logger.warning(
|
||||
"Failed to read image dimensions",
|
||||
extra={"error": str(e), "source": source},
|
||||
)
|
||||
return _worst_case_image_tokens(detail)
|
||||
tokens = _calculate_image_tokens(width, height, detail)
|
||||
logger.debug(
|
||||
"Calculated image tokens",
|
||||
extra={
|
||||
"source": source,
|
||||
"width": width,
|
||||
"height": height,
|
||||
"detail": detail,
|
||||
"tokens": tokens,
|
||||
},
|
||||
)
|
||||
return tokens
|
||||
|
||||
|
||||
def create_error_response(
|
||||
error_type: str,
|
||||
message: str,
|
||||
|
||||
@@ -0,0 +1,79 @@
|
||||
"""Convert a Responses API ``input`` into chat ``messages`` via litellm.
|
||||
|
||||
litellm drops ``file_id`` (emits ``url: ""``) and nests a dict-form ``image_url``
|
||||
as-is, so ``input_image`` parts are flattened to ``{image_url: str, detail}`` first.
|
||||
``file_id`` becomes a sentinel URL the image walker treats as unfetchable.
|
||||
"""
|
||||
|
||||
from typing import Any
|
||||
|
||||
from litellm.responses.litellm_completion_transformation.transformation import (
|
||||
LiteLLMCompletionResponsesConfig,
|
||||
)
|
||||
|
||||
from ..core import get_logger
|
||||
|
||||
logger = get_logger(__name__)
|
||||
|
||||
FILE_ID_URL_PREFIX = "file-id:"
|
||||
|
||||
|
||||
def _flatten_input_image(part: dict[str, Any]) -> tuple[str, str]:
|
||||
raw = part.get("image_url")
|
||||
url = raw.get("url", "") if isinstance(raw, dict) else raw
|
||||
detail = part.get("detail") or (
|
||||
raw.get("detail") if isinstance(raw, dict) else None
|
||||
)
|
||||
if not url and part.get("file_id"):
|
||||
url = f"{FILE_ID_URL_PREFIX}{part['file_id']}"
|
||||
return (url if isinstance(url, str) else ""), (detail or "auto")
|
||||
|
||||
|
||||
def _normalize_item(item: Any) -> Any:
|
||||
if not isinstance(item, dict):
|
||||
return item
|
||||
if item.get("type") == "input_image":
|
||||
url, detail = _flatten_input_image(item)
|
||||
return {**item, "image_url": url, "detail": detail}
|
||||
content = item.get("content")
|
||||
if isinstance(content, list):
|
||||
return {**item, "content": [_normalize_item(part) for part in content]}
|
||||
return item
|
||||
|
||||
|
||||
def input_image_part_to_image_url(part: dict[str, Any]) -> dict[str, Any]:
|
||||
"""Reshape an ``input_image`` part found inside chat ``messages``."""
|
||||
url, detail = _flatten_input_image(part)
|
||||
return {"type": "image_url", "image_url": {"url": url, "detail": detail}}
|
||||
|
||||
|
||||
def count_input_images(input_data: Any) -> int:
|
||||
if isinstance(input_data, dict):
|
||||
own = 1 if input_data.get("type") == "input_image" else 0
|
||||
return own + count_input_images(input_data.get("content"))
|
||||
if isinstance(input_data, list):
|
||||
return sum(count_input_images(item) for item in input_data)
|
||||
return 0
|
||||
|
||||
|
||||
def responses_input_to_messages(input_data: Any) -> list[dict[str, Any]] | None:
|
||||
"""Returns ``None`` when the transform fails so the caller can worst-case."""
|
||||
if isinstance(input_data, str):
|
||||
return [{"role": "user", "content": input_data}]
|
||||
if not isinstance(input_data, list):
|
||||
return []
|
||||
try:
|
||||
normalized = [_normalize_item(item) for item in input_data]
|
||||
converted = (
|
||||
LiteLLMCompletionResponsesConfig.transform_responses_api_input_to_messages(
|
||||
input=normalized, # type: ignore[arg-type]
|
||||
responses_api_request={},
|
||||
)
|
||||
)
|
||||
return [dict(message) for message in converted]
|
||||
except Exception as e:
|
||||
logger.warning(
|
||||
"Responses input transform failed; using conservative image fallback",
|
||||
extra={"error": str(e)},
|
||||
)
|
||||
return None
|
||||
+135
-7
@@ -38,9 +38,18 @@ from .payment.models import Model
|
||||
from .upstream import BaseUpstreamProvider
|
||||
from .upstream.ehbp import forward_ehbp_request, forward_ehbp_x_cashu_request
|
||||
from .upstream.helpers import init_upstreams
|
||||
from .upstream.model_paths import (
|
||||
ModelPathSelector,
|
||||
decode_model_path,
|
||||
is_openrouter_base_url,
|
||||
public_model_id,
|
||||
public_provider_url,
|
||||
)
|
||||
from .upstream.request_correction import correct_request, extract_error_message
|
||||
|
||||
logger = get_logger(__name__)
|
||||
|
||||
MODEL_PATH_HEADER = "x-routstr-model-path"
|
||||
proxy_router = APIRouter()
|
||||
|
||||
_upstreams: list[BaseUpstreamProvider] = []
|
||||
@@ -113,6 +122,25 @@ def get_candidates(
|
||||
return None
|
||||
|
||||
|
||||
def _model_ids_match(requested: str, selected: str) -> bool:
|
||||
if requested.lower() == selected.lower():
|
||||
return True
|
||||
return public_model_id(requested).lower() == public_model_id(selected).lower()
|
||||
|
||||
|
||||
def _candidate_for_selector(
|
||||
selector: ModelPathSelector,
|
||||
candidates: list[tuple[Model, BaseUpstreamProvider]],
|
||||
) -> tuple[Model, BaseUpstreamProvider] | None:
|
||||
for model_obj, upstream in candidates:
|
||||
if (
|
||||
upstream.db_id == selector.provider_id
|
||||
and public_provider_url(upstream.base_url) == selector.base_url
|
||||
):
|
||||
return model_obj, upstream
|
||||
return None
|
||||
|
||||
|
||||
def get_model_instance(model_id: str) -> Model | None:
|
||||
"""Get the best-ranked Model instance for a model ID."""
|
||||
candidates = get_candidates(model_id)
|
||||
@@ -416,6 +444,13 @@ async def _proxy(
|
||||
# without model/cost/auth lookups. Do not prefix-match here: paths such as
|
||||
# /attestationjunk must continue through normal authentication.
|
||||
if request.method == "GET" and _is_tinfoil_attestation_path(path):
|
||||
if MODEL_PATH_HEADER in headers:
|
||||
return create_error_response(
|
||||
"unsupported_request",
|
||||
"Model paths do not apply to attestation",
|
||||
400,
|
||||
request=request,
|
||||
)
|
||||
selected_upstreams = _select_unauthenticated_get_upstreams(path, _upstreams)
|
||||
if not selected_upstreams:
|
||||
return create_error_response(
|
||||
@@ -456,6 +491,41 @@ async def _proxy(
|
||||
"upstream_error", "All upstreams failed", 502, request=request
|
||||
)
|
||||
|
||||
selector: ModelPathSelector | None = None
|
||||
if MODEL_PATH_HEADER in headers:
|
||||
selector = decode_model_path(headers[MODEL_PATH_HEADER])
|
||||
if (
|
||||
selector is None
|
||||
or sum(
|
||||
name.lower() == MODEL_PATH_HEADER for name, _ in request.headers.items()
|
||||
)
|
||||
!= 1
|
||||
):
|
||||
return create_error_response(
|
||||
"invalid_request",
|
||||
f"Malformed {MODEL_PATH_HEADER} header",
|
||||
400,
|
||||
request=request,
|
||||
)
|
||||
if not isinstance(model_id, str) or not _model_ids_match(
|
||||
model_id, selector.model_id
|
||||
):
|
||||
return create_error_response(
|
||||
"invalid_request",
|
||||
f"{MODEL_PATH_HEADER} selects model '{selector.model_id}' but the "
|
||||
f"request asks for '{model_id}'",
|
||||
400,
|
||||
request=request,
|
||||
)
|
||||
if "models" in request_body_dict:
|
||||
return create_error_response(
|
||||
"invalid_request",
|
||||
"Model paths cannot be combined with model fallbacks",
|
||||
400,
|
||||
request=request,
|
||||
)
|
||||
model_id = selector.model_id
|
||||
|
||||
candidates = get_candidates(model_id)
|
||||
|
||||
if not candidates:
|
||||
@@ -463,6 +533,51 @@ async def _proxy(
|
||||
"invalid_model", f"Model '{model_id}' not found", 400, request=request
|
||||
)
|
||||
|
||||
if selector is not None:
|
||||
pinned = _candidate_for_selector(selector, candidates)
|
||||
if pinned is None:
|
||||
return create_error_response(
|
||||
"invalid_model_path",
|
||||
f"Model '{selector.model_id}' is not routable through provider "
|
||||
f"{selector.provider_id}",
|
||||
404,
|
||||
request=request,
|
||||
)
|
||||
# Explicit routes must never enter cross-provider failover.
|
||||
candidates = [pinned]
|
||||
|
||||
if selector.endpoint_tag:
|
||||
if (
|
||||
is_ehbp
|
||||
or not request_body_dict
|
||||
or not is_openrouter_base_url(pinned[1].base_url)
|
||||
or _canonical_api_path(path)
|
||||
not in {"chat/completions", "completions", "responses"}
|
||||
):
|
||||
return create_error_response(
|
||||
"unsupported_request",
|
||||
"Endpoint pinning requires an OpenRouter completion or Responses JSON request",
|
||||
400,
|
||||
request=request,
|
||||
)
|
||||
provider_options = request_body_dict.get("provider", {})
|
||||
if not isinstance(provider_options, dict):
|
||||
return create_error_response(
|
||||
"invalid_request",
|
||||
"provider must be an object",
|
||||
400,
|
||||
request=request,
|
||||
)
|
||||
request_body_dict = {
|
||||
**request_body_dict,
|
||||
"provider": {
|
||||
**provider_options,
|
||||
"order": [selector.endpoint_tag],
|
||||
"allow_fallbacks": False,
|
||||
},
|
||||
}
|
||||
request_body = json.dumps(request_body_dict).encode()
|
||||
|
||||
if is_ehbp:
|
||||
candidates = [
|
||||
(model, upstream)
|
||||
@@ -513,11 +628,21 @@ async def _proxy(
|
||||
)
|
||||
elif is_responses_api:
|
||||
return await upstream.handle_x_cashu_responses(
|
||||
request, x_cashu, path, max_cost_for_model, model_obj
|
||||
request,
|
||||
x_cashu,
|
||||
path,
|
||||
max_cost_for_model,
|
||||
model_obj,
|
||||
request_body=request_body,
|
||||
)
|
||||
else:
|
||||
return await upstream.handle_x_cashu(
|
||||
request, x_cashu, path, max_cost_for_model, model_obj
|
||||
request,
|
||||
x_cashu,
|
||||
path,
|
||||
max_cost_for_model,
|
||||
model_obj,
|
||||
request_body=request_body,
|
||||
)
|
||||
except UpstreamError as e:
|
||||
logger.warning(
|
||||
@@ -715,17 +840,20 @@ async def _proxy(
|
||||
)
|
||||
raise
|
||||
|
||||
# Reactive recovery: some models reject one specific request
|
||||
# param (e.g. newer Anthropic models deprecating `temperature`).
|
||||
# When the upstream 400s naming such a param, strip it from the
|
||||
# body and retry the SAME upstream. ``already_stripped`` bounds
|
||||
# this to one retry per distinct param so it always terminates.
|
||||
# Same-provider recovery must not relax an explicit route.
|
||||
if response.status_code == 400 and not is_ehbp:
|
||||
correction = correct_request(
|
||||
request_body,
|
||||
extract_error_message(response),
|
||||
already_stripped,
|
||||
)
|
||||
if correction is not None and selector is not None:
|
||||
corrected_body = json.loads(correction.body)
|
||||
if any(
|
||||
corrected_body.get(field) != request_body_dict.get(field)
|
||||
for field in ("model", "provider")
|
||||
):
|
||||
correction = None
|
||||
if correction is not None:
|
||||
request_body, bad_param = correction.body, correction.label
|
||||
already_stripped.add(bad_param)
|
||||
|
||||
@@ -591,6 +591,7 @@ class BaseUpstreamProvider:
|
||||
"refund-lnurl",
|
||||
"key-expiry-time",
|
||||
"x-cashu",
|
||||
"x-routstr-model-path",
|
||||
]:
|
||||
if headers.pop(header, None) is not None:
|
||||
removed_headers.append(header)
|
||||
@@ -4336,6 +4337,8 @@ class BaseUpstreamProvider:
|
||||
max_cost_for_model: int,
|
||||
model_obj: Model,
|
||||
mint: str | None = None,
|
||||
*,
|
||||
request_body: bytes | None = None,
|
||||
) -> Response | StreamingResponse:
|
||||
"""Forward request paid with X-Cashu token to upstream service.
|
||||
|
||||
@@ -4355,7 +4358,8 @@ class BaseUpstreamProvider:
|
||||
if path.startswith("v1/"):
|
||||
path = path.replace("v1/", "")
|
||||
|
||||
request_body = await request.body()
|
||||
if request_body is None:
|
||||
request_body = await request.body()
|
||||
|
||||
if (
|
||||
path.endswith("messages/count_tokens")
|
||||
@@ -4552,6 +4556,8 @@ class BaseUpstreamProvider:
|
||||
path: str,
|
||||
max_cost_for_model: int,
|
||||
model_obj: Model,
|
||||
*,
|
||||
request_body: bytes | None = None,
|
||||
) -> Response | StreamingResponse:
|
||||
"""Handle X-Cashu payment for Responses API requests.
|
||||
|
||||
@@ -4616,6 +4622,7 @@ class BaseUpstreamProvider:
|
||||
max_cost_for_model,
|
||||
model_obj,
|
||||
mint,
|
||||
request_body=request_body,
|
||||
)
|
||||
except Exception as e:
|
||||
error_message = str(e)
|
||||
@@ -4672,6 +4679,8 @@ class BaseUpstreamProvider:
|
||||
max_cost_for_model: int,
|
||||
model_obj: Model,
|
||||
mint: str | None = None,
|
||||
*,
|
||||
request_body: bytes | None = None,
|
||||
) -> Response | StreamingResponse:
|
||||
"""Forward Responses API request paid with X-Cashu token to upstream service.
|
||||
|
||||
@@ -4693,7 +4702,8 @@ class BaseUpstreamProvider:
|
||||
|
||||
url = f"{self.base_url}/{path}"
|
||||
|
||||
request_body = await request.body()
|
||||
if request_body is None:
|
||||
request_body = await request.body()
|
||||
transformed_body = self.prepare_responses_request_body(request_body, model_obj)
|
||||
|
||||
logger.debug(
|
||||
@@ -5281,6 +5291,8 @@ class BaseUpstreamProvider:
|
||||
path: str,
|
||||
max_cost_for_model: int,
|
||||
model_obj: Model,
|
||||
*,
|
||||
request_body: bytes | None = None,
|
||||
) -> Response | StreamingResponse:
|
||||
"""Handle request with X-Cashu token payment, redeeming token and forwarding request.
|
||||
|
||||
@@ -5360,6 +5372,7 @@ class BaseUpstreamProvider:
|
||||
max_cost_for_model,
|
||||
model_obj,
|
||||
mint,
|
||||
request_body=request_body,
|
||||
)
|
||||
except Exception as e:
|
||||
error_message = str(e)
|
||||
|
||||
+191
-24
@@ -1,9 +1,6 @@
|
||||
"""Model-path discovery service.
|
||||
|
||||
Exposes every selectable upstream route a Routstr model is reachable through.
|
||||
This PR remains discovery-only: request-side routing will consume the opaque
|
||||
selectors in a follow-up.
|
||||
|
||||
A path is a standard percent-encoded query string containing the configured
|
||||
upstream URL, provider ID, client-visible model ID and, for an exact OpenRouter
|
||||
endpoint, its machine-readable tag::
|
||||
@@ -16,11 +13,12 @@ from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import ipaddress
|
||||
import json
|
||||
import random
|
||||
import time
|
||||
from dataclasses import dataclass
|
||||
from typing import TYPE_CHECKING, Any, Callable
|
||||
from urllib.parse import urlencode, urlsplit
|
||||
from urllib.parse import parse_qsl, urlencode, urlsplit
|
||||
|
||||
import httpx
|
||||
from sqlalchemy.dialects.sqlite import insert
|
||||
@@ -60,10 +58,11 @@ ModelKey = tuple[str, int]
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class EndpointIdentity:
|
||||
"""Exact OpenRouter endpoint identity returned by ``/endpoints``."""
|
||||
"""Exact OpenRouter endpoint and its provider-specific model metadata."""
|
||||
|
||||
tag: str
|
||||
provider_name: str | None
|
||||
model_metadata: dict[str, Any]
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
@@ -83,6 +82,7 @@ class DiscoveredPath:
|
||||
model_id: str
|
||||
path: str
|
||||
provider: ConfiguredProviderIdentity
|
||||
model_metadata: dict[str, Any]
|
||||
endpoint_tag: str | None = None
|
||||
endpoint_name: str | None = None
|
||||
|
||||
@@ -132,21 +132,68 @@ def encode_model_path(
|
||||
return urlencode(components)
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class ModelPathSelector:
|
||||
"""Decoded client-supplied route selector."""
|
||||
|
||||
base_url: str
|
||||
provider_id: int
|
||||
model_id: str
|
||||
endpoint_tag: str | None = None
|
||||
|
||||
|
||||
def decode_model_path(path: str) -> ModelPathSelector | None:
|
||||
"""Inverse of ``encode_model_path``; ``None`` when the selector is malformed."""
|
||||
try:
|
||||
pairs = parse_qsl(
|
||||
path,
|
||||
keep_blank_values=True,
|
||||
strict_parsing=True,
|
||||
max_num_fields=4,
|
||||
errors="strict",
|
||||
)
|
||||
except ValueError:
|
||||
return None
|
||||
params = dict(pairs)
|
||||
if len(params) != len(pairs) or params.keys() - {
|
||||
"url",
|
||||
"provider-id",
|
||||
"model-id",
|
||||
"endpoint",
|
||||
}:
|
||||
return None
|
||||
if any(not value.strip() for value in params.values()):
|
||||
return None
|
||||
base_url = params.get("url", "")
|
||||
model_id = params.get("model-id", "")
|
||||
raw_provider_id = params.get("provider-id", "")
|
||||
if not base_url or not model_id or not raw_provider_id:
|
||||
return None
|
||||
try:
|
||||
provider_id = int(raw_provider_id)
|
||||
except ValueError:
|
||||
return None
|
||||
if provider_id <= 0:
|
||||
return None
|
||||
return ModelPathSelector(
|
||||
base_url=base_url,
|
||||
provider_id=provider_id,
|
||||
model_id=model_id,
|
||||
endpoint_tag=params.get("endpoint") or None,
|
||||
)
|
||||
|
||||
|
||||
def _make_http_client() -> httpx.AsyncClient:
|
||||
"""Client factory, separated so tests can substitute a mock transport."""
|
||||
return httpx.AsyncClient()
|
||||
|
||||
|
||||
def is_openrouter_base_url(base_url: str | None) -> bool:
|
||||
"""True when ``base_url`` points at OpenRouter.
|
||||
|
||||
Deliberately separate from ``BaseUpstreamProvider._upstream_accepts_cache_control``:
|
||||
that predicate also returns True for native Anthropic (correct for
|
||||
cache-control, wrong for OpenRouter endpoint discovery). This one keys only
|
||||
on the URL so a ``GenericUpstreamProvider`` aimed at OpenRouter is matched
|
||||
while native Anthropic is not.
|
||||
"""
|
||||
return "openrouter.ai" in (base_url or "")
|
||||
"""Match OpenRouter itself, not compatible providers or lookalike hosts."""
|
||||
try:
|
||||
return urlsplit(base_url or "").hostname == "openrouter.ai"
|
||||
except ValueError:
|
||||
return False
|
||||
|
||||
|
||||
def exposed_model_id(model: object) -> str:
|
||||
@@ -263,9 +310,14 @@ async def _fetch_openrouter_endpoint_subproviders(
|
||||
try:
|
||||
payload = resp.json()
|
||||
data = payload.get("data") if isinstance(payload, dict) else None
|
||||
endpoints = data.get("endpoints") if isinstance(data, dict) else None
|
||||
if not isinstance(data, dict):
|
||||
raise ValueError("data must be an object")
|
||||
endpoints = data.get("endpoints")
|
||||
if not isinstance(endpoints, list):
|
||||
raise ValueError("endpoints must be a list")
|
||||
common_metadata = {
|
||||
key: value for key, value in data.items() if key != "endpoints"
|
||||
}
|
||||
identities: dict[str, EndpointIdentity] = {}
|
||||
for endpoint in endpoints:
|
||||
if not isinstance(endpoint, dict):
|
||||
@@ -281,6 +333,7 @@ async def _fetch_openrouter_endpoint_subproviders(
|
||||
provider_name=provider_name
|
||||
if isinstance(provider_name, str) and provider_name
|
||||
else None,
|
||||
model_metadata={**common_metadata, **endpoint},
|
||||
),
|
||||
)
|
||||
if endpoints and not identities:
|
||||
@@ -341,6 +394,34 @@ async def _load_model_visibility() -> tuple[
|
||||
return overrides_by_key, disabled_model_keys, provider_identities
|
||||
|
||||
|
||||
def _serialize_model_metadata(model: object, model_id: str) -> dict[str, Any]:
|
||||
"""Serialize provider-specific model details into the public API shape."""
|
||||
model_dict = getattr(model, "dict", None)
|
||||
if callable(model_dict):
|
||||
metadata = dict(model_dict())
|
||||
else:
|
||||
metadata = {
|
||||
key: value for key, value in vars(model).items() if not key.startswith("_")
|
||||
}
|
||||
|
||||
for field in (
|
||||
"architecture",
|
||||
"pricing",
|
||||
"sats_pricing",
|
||||
"per_request_limits",
|
||||
"top_provider",
|
||||
"alias_ids",
|
||||
):
|
||||
value = metadata.get(field)
|
||||
if isinstance(value, str):
|
||||
try:
|
||||
metadata[field] = json.loads(value)
|
||||
except (TypeError, ValueError):
|
||||
pass
|
||||
metadata["id"] = model_id
|
||||
return metadata
|
||||
|
||||
|
||||
def _apply_model_visibility(
|
||||
upstream: BaseUpstreamProvider,
|
||||
overrides_by_key: dict[ModelKey, ModelRow] | None,
|
||||
@@ -348,11 +429,10 @@ def _apply_model_visibility(
|
||||
) -> list[object]:
|
||||
"""Return provider models after DB disabled/override state is applied.
|
||||
|
||||
Only the identity fields (``id``, ``forwarded_model_id``,
|
||||
``canonical_slug``) matter for path discovery, so DB override rows are used
|
||||
directly rather than rebuilt into fully priced ``Model`` objects — the
|
||||
pricing pipeline costs ~0.7ms of event-loop CPU per row for data this
|
||||
module immediately discards.
|
||||
DB override rows are used directly rather than rebuilt into priced
|
||||
``Model`` objects. Their JSON metadata fields are decoded when each path is
|
||||
collected, preserving the provider-specific stored values without running
|
||||
the routing price-selection pipeline.
|
||||
"""
|
||||
overrides_by_key = overrides_by_key or {}
|
||||
disabled_model_keys = disabled_model_keys or set()
|
||||
@@ -414,6 +494,7 @@ async def _collect_provider_paths(
|
||||
provider_identity.base_url, provider_identity.id, model_id
|
||||
),
|
||||
provider=provider_identity,
|
||||
model_metadata=_serialize_model_metadata(model, model_id),
|
||||
)
|
||||
|
||||
if not is_openrouter_base_url(upstream.base_url):
|
||||
@@ -453,6 +534,7 @@ async def _collect_provider_paths(
|
||||
endpoint.tag,
|
||||
),
|
||||
provider=provider_identity,
|
||||
model_metadata={**endpoint.model_metadata, "id": model_id},
|
||||
endpoint_tag=endpoint.tag,
|
||||
endpoint_name=endpoint.provider_name,
|
||||
)
|
||||
@@ -512,6 +594,7 @@ async def _persist_provider_paths(
|
||||
"provider_type": discovered.provider.provider_type,
|
||||
"endpoint_tag": discovered.endpoint_tag,
|
||||
"endpoint_name": discovered.endpoint_name,
|
||||
"model_metadata": json.dumps(discovered.model_metadata),
|
||||
"upstream_provider_id": upstream_provider_id,
|
||||
"updated_at": now,
|
||||
}
|
||||
@@ -526,6 +609,7 @@ async def _persist_provider_paths(
|
||||
"provider_type": insert_stmt.excluded.provider_type,
|
||||
"endpoint_tag": insert_stmt.excluded.endpoint_tag,
|
||||
"endpoint_name": insert_stmt.excluded.endpoint_name,
|
||||
"model_metadata": insert_stmt.excluded.model_metadata,
|
||||
"updated_at": insert_stmt.excluded.updated_at,
|
||||
},
|
||||
)
|
||||
@@ -736,10 +820,83 @@ async def refresh_model_paths_periodically(
|
||||
break
|
||||
|
||||
|
||||
def _serialize_path(row: ModelPathRow) -> dict[str, Any]:
|
||||
def _price_in_sats(model: dict[str, Any], provider_fee: float) -> None:
|
||||
"""Run a path's USD rates through the ``/v1/models`` pricing pipeline.
|
||||
|
||||
Metadata copied from the provider model cache is already priced. OpenRouter
|
||||
endpoint metadata is not: it carries that endpoint's own USD rates, which
|
||||
still need the cache backfill, the provider fee and the sats conversion.
|
||||
"""
|
||||
pricing = model.get("pricing")
|
||||
if model.get("sats_pricing") or not isinstance(pricing, dict):
|
||||
return
|
||||
|
||||
from ..payment.models import (
|
||||
Architecture,
|
||||
Model,
|
||||
Pricing,
|
||||
TopProvider,
|
||||
_calculate_usd_max_costs,
|
||||
_update_model_sats_pricing,
|
||||
backfill_cache_pricing,
|
||||
)
|
||||
from ..payment.price import sats_usd_price
|
||||
|
||||
try:
|
||||
model_id = model.get("forwarded_model_id") or model["id"]
|
||||
usd = backfill_cache_pricing(model_id, Pricing.parse_obj(pricing))
|
||||
usd = Pricing.parse_obj({k: v * provider_fee for k, v in usd.dict().items()})
|
||||
priced = Model(
|
||||
id=model_id,
|
||||
name=model.get("name") or model_id,
|
||||
created=0,
|
||||
description="",
|
||||
context_length=model.get("context_length") or 0,
|
||||
architecture=Architecture(
|
||||
modality="text",
|
||||
input_modalities=[],
|
||||
output_modalities=[],
|
||||
tokenizer="",
|
||||
instruct_type=None,
|
||||
),
|
||||
pricing=usd,
|
||||
top_provider=TopProvider(
|
||||
context_length=model.get("context_length"),
|
||||
max_completion_tokens=model.get("max_completion_tokens"),
|
||||
),
|
||||
)
|
||||
(
|
||||
usd.max_prompt_cost,
|
||||
usd.max_completion_cost,
|
||||
usd.max_cost,
|
||||
) = _calculate_usd_max_costs(priced)
|
||||
priced = _update_model_sats_pricing(priced, sats_usd_price())
|
||||
except Exception as exc:
|
||||
# An endpoint with rates we cannot price is still a usable route, so it
|
||||
# is served with its raw upstream pricing rather than dropped.
|
||||
logger.warning(
|
||||
"Could not calculate sats pricing for model path",
|
||||
extra={"model_id": model.get("id"), "error": str(exc)},
|
||||
)
|
||||
return
|
||||
|
||||
if priced.sats_pricing:
|
||||
model["pricing"] = usd.dict()
|
||||
model["sats_pricing"] = priced.sats_pricing.dict()
|
||||
|
||||
|
||||
def _serialize_path(row: ModelPathRow, provider_fee: float) -> dict[str, Any]:
|
||||
endpoint = None
|
||||
if row.endpoint_tag or row.endpoint_name:
|
||||
endpoint = {"tag": row.endpoint_tag, "name": row.endpoint_name}
|
||||
try:
|
||||
model = json.loads(row.model_metadata)
|
||||
except (TypeError, ValueError):
|
||||
model = {}
|
||||
if not isinstance(model, dict):
|
||||
model = {}
|
||||
model.setdefault("id", row.model_id)
|
||||
_price_in_sats(model, provider_fee)
|
||||
return {
|
||||
"path": row.path,
|
||||
"provider": {
|
||||
@@ -748,11 +905,17 @@ def _serialize_path(row: ModelPathRow) -> dict[str, Any]:
|
||||
"type": row.provider_type,
|
||||
},
|
||||
"endpoint": endpoint,
|
||||
"model": model,
|
||||
}
|
||||
|
||||
|
||||
async def _provider_fees(session: "AsyncSession") -> dict[int, float]:
|
||||
rows = (await session.exec(select(UpstreamProviderRow))).all()
|
||||
return {row.id: row.provider_fee for row in rows if row.id is not None}
|
||||
|
||||
|
||||
async def get_all_model_paths() -> dict:
|
||||
"""All models with their exact selectable routes."""
|
||||
"""All models with exact routes and provider-specific model metadata."""
|
||||
async with create_session() as session:
|
||||
rows = (
|
||||
await session.exec(
|
||||
@@ -763,6 +926,7 @@ async def get_all_model_paths() -> dict:
|
||||
)
|
||||
)
|
||||
).all()
|
||||
fees = await _provider_fees(session)
|
||||
|
||||
grouped: dict[str, list[dict[str, Any]]] = {}
|
||||
seen_paths: dict[str, set[str]] = {}
|
||||
@@ -772,7 +936,9 @@ async def get_all_model_paths() -> dict:
|
||||
if row.path in seen_paths.setdefault(row.model_id, set()):
|
||||
continue
|
||||
seen_paths[row.model_id].add(row.path)
|
||||
grouped.setdefault(row.model_id, []).append(_serialize_path(row))
|
||||
grouped.setdefault(row.model_id, []).append(
|
||||
_serialize_path(row, fees.get(row.upstream_provider_id, 1.01))
|
||||
)
|
||||
data = [
|
||||
{
|
||||
"id": grouped_model_id,
|
||||
@@ -806,6 +972,7 @@ async def get_paths_for_model(model_id: str) -> dict:
|
||||
unprefixed_id = public_model_id(model_id)
|
||||
if unprefixed_id != model_id:
|
||||
rows = await load_rows(session, unprefixed_id)
|
||||
fees = await _provider_fees(session)
|
||||
|
||||
seen: set[str] = set()
|
||||
paths: list[dict] = []
|
||||
@@ -815,5 +982,5 @@ async def get_paths_for_model(model_id: str) -> dict:
|
||||
if row.path in seen:
|
||||
continue
|
||||
seen.add(row.path)
|
||||
paths.append(_serialize_path(row))
|
||||
paths.append(_serialize_path(row, fees.get(row.upstream_provider_id, 1.01)))
|
||||
return {"data": paths, "updated_at": updated_at or None}
|
||||
|
||||
@@ -0,0 +1,625 @@
|
||||
"""Model-path routing and fail-closed behavior."""
|
||||
|
||||
import json
|
||||
from typing import Any
|
||||
from unittest.mock import AsyncMock, MagicMock, patch
|
||||
|
||||
import pytest
|
||||
|
||||
from routstr import proxy as proxy_module
|
||||
from routstr.auth import ReservationSnapshot
|
||||
from routstr.core.db import ApiKey
|
||||
from routstr.upstream.model_paths import decode_model_path, encode_model_path
|
||||
|
||||
MODEL_ID = "test-model"
|
||||
|
||||
|
||||
def _make_upstream(db_id: int, status_code: int = 200) -> MagicMock:
|
||||
upstream = MagicMock()
|
||||
upstream.db_id = db_id
|
||||
upstream.base_url = "http://localhost"
|
||||
upstream.provider_type = f"provider-{db_id}"
|
||||
upstream.supports_ehbp = False
|
||||
upstream.prepare_headers = MagicMock(side_effect=lambda h: h)
|
||||
upstream.on_upstream_error_redirect = AsyncMock()
|
||||
upstream.forward_request = AsyncMock(
|
||||
return_value=MagicMock(status_code=status_code, body=b"{}")
|
||||
)
|
||||
return upstream
|
||||
|
||||
|
||||
def _make_request(headers: dict[str, str], body: bytes) -> MagicMock:
|
||||
request = MagicMock()
|
||||
request.method = "POST"
|
||||
request.headers = headers
|
||||
request.body = AsyncMock(return_value=body)
|
||||
request.state = MagicMock()
|
||||
request.state.request_id = "req-model-path"
|
||||
return request
|
||||
|
||||
|
||||
async def _run_proxy(
|
||||
request: MagicMock,
|
||||
candidates: list[tuple[Any, Any]],
|
||||
path: str = "v1/chat/completions",
|
||||
) -> Any:
|
||||
key = ApiKey(hashed_key="mpkey", balance=10_000)
|
||||
reservation = ReservationSnapshot(
|
||||
release_id="model-path-release",
|
||||
key_hash=key.hashed_key,
|
||||
billing_key_hash=key.hashed_key,
|
||||
reserved_msats=1_000,
|
||||
)
|
||||
with (
|
||||
patch.object(proxy_module, "get_candidates", return_value=candidates),
|
||||
patch.object(
|
||||
proxy_module, "get_max_cost_for_model", AsyncMock(return_value=1_000)
|
||||
),
|
||||
patch.object(
|
||||
proxy_module,
|
||||
"calculate_discounted_max_cost",
|
||||
AsyncMock(return_value=1_000),
|
||||
),
|
||||
patch.object(proxy_module, "check_token_balance", MagicMock()),
|
||||
patch.object(proxy_module, "get_bearer_token_key", AsyncMock(return_value=key)),
|
||||
patch.object(proxy_module, "pay_for_request", AsyncMock(return_value=1_000)),
|
||||
patch.object(
|
||||
proxy_module,
|
||||
"get_reservation_snapshot",
|
||||
AsyncMock(return_value=reservation),
|
||||
),
|
||||
patch.object(proxy_module, "revert_pay_for_request", AsyncMock()),
|
||||
):
|
||||
return await proxy_module.proxy(request, path, session=MagicMock())
|
||||
|
||||
|
||||
def test_decode_model_path_round_trips_encode() -> None:
|
||||
selector = decode_model_path(
|
||||
encode_model_path("https://openrouter.ai/api/v1", 7, MODEL_ID, "deepinfra/fp8")
|
||||
)
|
||||
assert selector is not None
|
||||
assert selector.base_url == "https://openrouter.ai/api/v1"
|
||||
assert selector.provider_id == 7
|
||||
assert selector.model_id == MODEL_ID
|
||||
assert selector.endpoint_tag == "deepinfra/fp8"
|
||||
|
||||
|
||||
def test_decode_model_path_without_endpoint_has_no_tag() -> None:
|
||||
selector = decode_model_path(encode_model_path("http://localhost", 1, MODEL_ID))
|
||||
assert selector is not None
|
||||
assert selector.endpoint_tag is None
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"path",
|
||||
[
|
||||
"",
|
||||
"url=http://localhost&model-id=test-model",
|
||||
"url=http://localhost&provider-id=abc&model-id=test-model",
|
||||
"provider-id=1&model-id=test-model",
|
||||
"url=http://localhost&provider-id=1",
|
||||
],
|
||||
)
|
||||
def test_decode_model_path_rejects_malformed_selectors(path: str) -> None:
|
||||
assert decode_model_path(path) is None
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_model_path_routes_to_the_selected_provider() -> None:
|
||||
first, selected = _make_upstream(1), _make_upstream(2)
|
||||
request = _make_request(
|
||||
{
|
||||
"authorization": "Bearer sk-mpkey",
|
||||
"x-routstr-model-path": encode_model_path("http://localhost", 2, MODEL_ID),
|
||||
},
|
||||
json.dumps({"model": MODEL_ID}).encode(),
|
||||
)
|
||||
|
||||
await _run_proxy(request, [(MagicMock(), first), (MagicMock(), selected)])
|
||||
|
||||
selected.forward_request.assert_awaited_once()
|
||||
first.forward_request.assert_not_awaited()
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@pytest.mark.parametrize("status_code", [400, 429, 500, 502, 503])
|
||||
async def test_model_path_failure_is_returned_without_falling_back(
|
||||
status_code: int,
|
||||
) -> None:
|
||||
selected = _make_upstream(1, status_code=status_code)
|
||||
fallback = _make_upstream(2)
|
||||
request = _make_request(
|
||||
{
|
||||
"authorization": "Bearer sk-mpkey",
|
||||
"x-routstr-model-path": encode_model_path("http://localhost", 1, MODEL_ID),
|
||||
},
|
||||
json.dumps({"model": MODEL_ID}).encode(),
|
||||
)
|
||||
|
||||
response = await _run_proxy(
|
||||
request, [(MagicMock(), selected), (MagicMock(), fallback)]
|
||||
)
|
||||
|
||||
assert response.status_code == status_code
|
||||
selected.forward_request.assert_awaited_once()
|
||||
fallback.forward_request.assert_not_awaited()
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_unknown_provider_in_model_path_is_rejected() -> None:
|
||||
upstream = _make_upstream(1)
|
||||
request = _make_request(
|
||||
{
|
||||
"authorization": "Bearer sk-mpkey",
|
||||
"x-routstr-model-path": encode_model_path("http://localhost", 99, MODEL_ID),
|
||||
},
|
||||
json.dumps({"model": MODEL_ID}).encode(),
|
||||
)
|
||||
|
||||
response = await _run_proxy(request, [(MagicMock(), upstream)])
|
||||
|
||||
assert response.status_code == 404
|
||||
assert json.loads(bytes(response.body))["error"]["type"] == "invalid_model_path"
|
||||
upstream.forward_request.assert_not_awaited()
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_model_path_disagreeing_with_the_body_model_is_rejected() -> None:
|
||||
upstream = _make_upstream(1)
|
||||
request = _make_request(
|
||||
{
|
||||
"authorization": "Bearer sk-mpkey",
|
||||
"x-routstr-model-path": encode_model_path(
|
||||
"http://localhost", 1, "other-model"
|
||||
),
|
||||
},
|
||||
json.dumps({"model": MODEL_ID}).encode(),
|
||||
)
|
||||
|
||||
response = await _run_proxy(request, [(MagicMock(), upstream)])
|
||||
|
||||
assert response.status_code == 400
|
||||
upstream.forward_request.assert_not_awaited()
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_malformed_model_path_header_is_rejected() -> None:
|
||||
upstream = _make_upstream(1)
|
||||
request = _make_request(
|
||||
{
|
||||
"authorization": "Bearer sk-mpkey",
|
||||
"x-routstr-model-path": "not-a-model-path",
|
||||
},
|
||||
json.dumps({"model": MODEL_ID}).encode(),
|
||||
)
|
||||
|
||||
response = await _run_proxy(request, [(MagicMock(), upstream)])
|
||||
|
||||
assert response.status_code == 400
|
||||
upstream.forward_request.assert_not_awaited()
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_endpoint_tag_pins_the_upstream_subprovider() -> None:
|
||||
upstream = _make_upstream(1)
|
||||
upstream.base_url = "https://openrouter.ai/api/v1"
|
||||
request = _make_request(
|
||||
{
|
||||
"authorization": "Bearer sk-mpkey",
|
||||
"x-routstr-model-path": encode_model_path(
|
||||
"https://openrouter.ai/api/v1", 1, MODEL_ID, "deepinfra/fp8"
|
||||
),
|
||||
},
|
||||
json.dumps({"model": MODEL_ID}).encode(),
|
||||
)
|
||||
|
||||
await _run_proxy(request, [(MagicMock(), upstream)])
|
||||
|
||||
forwarded_body = upstream.forward_request.await_args.args[3]
|
||||
assert json.loads(forwarded_body)["provider"] == {
|
||||
"order": ["deepinfra/fp8"],
|
||||
"allow_fallbacks": False,
|
||||
}
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@pytest.mark.parametrize(
|
||||
"raw",
|
||||
["", " ", "url=http://localhost&provider-id=1&model-id=test-model&provider-id=2"],
|
||||
)
|
||||
async def test_ambiguous_headers_do_not_route(raw: str) -> None:
|
||||
upstream = _make_upstream(1)
|
||||
request = _make_request(
|
||||
{"authorization": "Bearer key", "x-routstr-model-path": raw},
|
||||
json.dumps({"model": MODEL_ID}).encode(),
|
||||
)
|
||||
response = await _run_proxy(request, [(MagicMock(), upstream)])
|
||||
assert response.status_code == 400
|
||||
upstream.forward_request.assert_not_awaited()
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_selector_url_must_match_configured_provider() -> None:
|
||||
upstream = _make_upstream(1)
|
||||
request = _make_request(
|
||||
{
|
||||
"authorization": "Bearer key",
|
||||
"x-routstr-model-path": encode_model_path(
|
||||
"http://169.254.169.254", 1, MODEL_ID
|
||||
),
|
||||
},
|
||||
json.dumps({"model": MODEL_ID}).encode(),
|
||||
)
|
||||
response = await _run_proxy(request, [(MagicMock(), upstream)])
|
||||
assert response.status_code == 404
|
||||
upstream.forward_request.assert_not_awaited()
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_endpoint_pin_cannot_be_stripped_on_retry() -> None:
|
||||
upstream = _make_upstream(1, 400)
|
||||
upstream.base_url = "https://openrouter.ai/api/v1"
|
||||
upstream.forward_request.return_value.body = (
|
||||
b'{"error":{"message":"provider is not supported"}}'
|
||||
)
|
||||
request = _make_request(
|
||||
{
|
||||
"authorization": "Bearer key",
|
||||
"x-routstr-model-path": encode_model_path(
|
||||
upstream.base_url, 1, MODEL_ID, "deepinfra/fp8"
|
||||
),
|
||||
},
|
||||
json.dumps({"model": MODEL_ID}).encode(),
|
||||
)
|
||||
response = await _run_proxy(request, [(MagicMock(), upstream)])
|
||||
assert response.status_code == 400
|
||||
upstream.forward_request.assert_awaited_once()
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@pytest.mark.parametrize(
|
||||
"path,handler",
|
||||
[
|
||||
("v1/chat/completions", "handle_x_cashu"),
|
||||
("v1/responses", "handle_x_cashu_responses"),
|
||||
],
|
||||
)
|
||||
async def test_cashu_receives_endpoint_pinned_body(path: str, handler: str) -> None:
|
||||
upstream = _make_upstream(1)
|
||||
upstream.base_url = "https://openrouter.ai/api/v1"
|
||||
captured: dict[str, Any] = {}
|
||||
|
||||
async def handle(request: Any, *args: Any, **kwargs: Any) -> Any:
|
||||
captured.update(json.loads(kwargs.get("request_body") or await request.body()))
|
||||
return MagicMock(status_code=200)
|
||||
|
||||
setattr(upstream, handler, AsyncMock(side_effect=handle))
|
||||
request = _make_request(
|
||||
{
|
||||
"x-cashu": "test-token",
|
||||
"x-routstr-model-path": encode_model_path(
|
||||
upstream.base_url, 1, MODEL_ID, "deepinfra/fp8"
|
||||
),
|
||||
},
|
||||
json.dumps(
|
||||
{
|
||||
"model": MODEL_ID,
|
||||
"provider": {
|
||||
"order": ["other"],
|
||||
"allow_fallbacks": True,
|
||||
"data_collection": "deny",
|
||||
},
|
||||
}
|
||||
).encode(),
|
||||
)
|
||||
await _run_proxy(request, [(MagicMock(), upstream)], path)
|
||||
assert captured["provider"] == {
|
||||
"order": ["deepinfra/fp8"],
|
||||
"allow_fallbacks": False,
|
||||
"data_collection": "deny",
|
||||
}
|
||||
|
||||
|
||||
def test_model_path_header_is_not_forwarded() -> None:
|
||||
from routstr.upstream.base import BaseUpstreamProvider
|
||||
|
||||
upstream = BaseUpstreamProvider("http://localhost", "upstream-key")
|
||||
headers = upstream.prepare_headers({"x-routstr-model-path": "private-routing-data"})
|
||||
assert "x-routstr-model-path" not in headers
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@pytest.mark.parametrize("path", ["v1/chat/completions", "v1/responses"])
|
||||
@pytest.mark.parametrize("status_code", [200, 429, 502])
|
||||
async def test_cashu_pin_reaches_http_transport(path: str, status_code: int) -> None:
|
||||
import httpx
|
||||
from fastapi.responses import Response
|
||||
|
||||
from routstr.upstream.openrouter import OpenRouterUpstreamProvider
|
||||
|
||||
upstream = OpenRouterUpstreamProvider(api_key="upstream-key")
|
||||
upstream.db_id = 1
|
||||
fallback = _make_upstream(2)
|
||||
fallback.handle_x_cashu = AsyncMock()
|
||||
fallback.handle_x_cashu_responses = AsyncMock()
|
||||
sent: list[httpx.Request] = []
|
||||
|
||||
def respond(request: httpx.Request) -> httpx.Response:
|
||||
sent.append(request)
|
||||
return httpx.Response(status_code, json={"error": "test"})
|
||||
|
||||
client = httpx.AsyncClient(transport=httpx.MockTransport(respond))
|
||||
model = MagicMock(id=MODEL_ID, forwarded_model_id=None, canonical_slug=None)
|
||||
request = _make_request(
|
||||
{
|
||||
"x-cashu": "test-token",
|
||||
"x-routstr-model-path": encode_model_path(
|
||||
upstream.base_url, 1, MODEL_ID, "deepinfra/fp8"
|
||||
),
|
||||
},
|
||||
json.dumps({"model": MODEL_ID, "provider": {"allow_fallbacks": True}}).encode(),
|
||||
)
|
||||
request.query_params = {}
|
||||
with (
|
||||
patch("routstr.upstream.base.httpx.AsyncClient", return_value=client),
|
||||
patch(
|
||||
"routstr.upstream.base.recieve_token",
|
||||
AsyncMock(return_value=(1000, "msat", "https://mint.test")),
|
||||
) as redeem,
|
||||
patch("routstr.upstream.base.store_cashu_transaction", AsyncMock()),
|
||||
patch.object(upstream, "send_refund", AsyncMock(return_value="refund")),
|
||||
patch.object(
|
||||
upstream,
|
||||
"handle_x_cashu_chat_completion",
|
||||
AsyncMock(return_value=Response(status_code=200)),
|
||||
),
|
||||
patch.object(
|
||||
upstream,
|
||||
"handle_x_cashu_responses_completion",
|
||||
AsyncMock(return_value=Response(status_code=200)),
|
||||
),
|
||||
):
|
||||
response = await _run_proxy(
|
||||
request, [(model, upstream), (model, fallback)], path
|
||||
)
|
||||
assert response.status_code == status_code
|
||||
redeem.assert_awaited_once()
|
||||
assert len(sent) == 1
|
||||
assert sent[0].url.host == "openrouter.ai"
|
||||
assert json.loads(sent[0].content)["provider"] == {
|
||||
"order": ["deepinfra/fp8"],
|
||||
"allow_fallbacks": False,
|
||||
}
|
||||
assert "x-routstr-model-path" not in sent[0].headers
|
||||
fallback.handle_x_cashu.assert_not_awaited()
|
||||
fallback.handle_x_cashu_responses.assert_not_awaited()
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_unpinned_requests_still_fall_back() -> None:
|
||||
first, fallback = _make_upstream(1, 502), _make_upstream(2)
|
||||
request = _make_request(
|
||||
{"authorization": "Bearer key"}, json.dumps({"model": MODEL_ID}).encode()
|
||||
)
|
||||
response = await _run_proxy(
|
||||
request, [(MagicMock(), first), (MagicMock(), fallback)]
|
||||
)
|
||||
assert response.status_code == 200
|
||||
first.forward_request.assert_awaited_once()
|
||||
fallback.forward_request.assert_awaited_once()
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_pinned_exception_does_not_fall_back() -> None:
|
||||
from routstr.core.exceptions import UpstreamError
|
||||
|
||||
first, fallback = _make_upstream(1), _make_upstream(2)
|
||||
first.forward_request.side_effect = UpstreamError("unavailable", status_code=503)
|
||||
request = _make_request(
|
||||
{
|
||||
"authorization": "Bearer key",
|
||||
"x-routstr-model-path": encode_model_path(first.base_url, 1, MODEL_ID),
|
||||
},
|
||||
json.dumps({"model": MODEL_ID}).encode(),
|
||||
)
|
||||
response = await _run_proxy(
|
||||
request, [(MagicMock(), first), (MagicMock(), fallback)]
|
||||
)
|
||||
assert response.status_code == 503
|
||||
first.forward_request.assert_awaited_once()
|
||||
fallback.forward_request.assert_not_awaited()
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@pytest.mark.parametrize(
|
||||
"path,is_ehbp", [("v1/messages", False), ("v1/chat/completions", True)]
|
||||
)
|
||||
async def test_unsupported_endpoint_pins_fail_before_payment(
|
||||
path: str, is_ehbp: bool
|
||||
) -> None:
|
||||
upstream = _make_upstream(1)
|
||||
upstream.base_url = "https://openrouter.ai/api/v1"
|
||||
headers = {
|
||||
"x-cashu": "test-token",
|
||||
"x-routstr-model-path": encode_model_path(
|
||||
upstream.base_url, 1, MODEL_ID, "deepinfra/fp8"
|
||||
),
|
||||
}
|
||||
if is_ehbp:
|
||||
headers.update({"ehbp-encapsulated-key": "sealed", "x-routstr-model": MODEL_ID})
|
||||
request = _make_request(headers, json.dumps({"model": MODEL_ID}).encode())
|
||||
with (
|
||||
patch.object(proxy_module, "check_token_balance") as payment,
|
||||
patch.object(
|
||||
proxy_module, "get_candidates", return_value=[(MagicMock(), upstream)]
|
||||
),
|
||||
):
|
||||
response = await proxy_module.proxy(request, path, MagicMock())
|
||||
assert response.status_code == 400
|
||||
assert json.loads(response.body)["error"]["type"] == "unsupported_request"
|
||||
payment.assert_not_called()
|
||||
upstream.forward_request.assert_not_awaited()
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@pytest.mark.parametrize("cashu", [False, True])
|
||||
async def test_ehbp_pin_does_not_fall_back(cashu: bool) -> None:
|
||||
from routstr.core.exceptions import UpstreamError
|
||||
|
||||
selected, fallback = _make_upstream(1), _make_upstream(2)
|
||||
selected.supports_ehbp = fallback.supports_ehbp = True
|
||||
headers = {
|
||||
"ehbp-encapsulated-key": "sealed",
|
||||
"x-routstr-model": MODEL_ID,
|
||||
"x-routstr-model-path": encode_model_path(selected.base_url, 1, MODEL_ID),
|
||||
}
|
||||
headers.update({"x-cashu": "token"} if cashu else {"authorization": "Bearer key"})
|
||||
request = _make_request(headers, b"encrypted-body")
|
||||
handler = "forward_ehbp_x_cashu_request" if cashu else "forward_ehbp_request"
|
||||
with patch.object(
|
||||
proxy_module,
|
||||
handler,
|
||||
AsyncMock(side_effect=UpstreamError("unavailable", status_code=503)),
|
||||
) as forward:
|
||||
response = await _run_proxy(
|
||||
request, [(MagicMock(), selected), (MagicMock(), fallback)]
|
||||
)
|
||||
assert response.status_code == 503
|
||||
forward.assert_awaited_once()
|
||||
assert forward.await_args is not None
|
||||
assert forward.await_args.kwargs["upstream"] is selected
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_duplicate_header_fields_are_rejected() -> None:
|
||||
from starlette.datastructures import Headers
|
||||
|
||||
selected = _make_upstream(1)
|
||||
route = encode_model_path(selected.base_url, 1, MODEL_ID).encode()
|
||||
request = _make_request({}, json.dumps({"model": MODEL_ID}).encode())
|
||||
request.headers = Headers(
|
||||
raw=[
|
||||
(b"authorization", b"Bearer key"),
|
||||
(b"x-routstr-model-path", route),
|
||||
(b"x-routstr-model-path", route),
|
||||
]
|
||||
)
|
||||
response = await _run_proxy(request, [(MagicMock(), selected)])
|
||||
assert response.status_code == 400
|
||||
selected.forward_request.assert_not_awaited()
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_attestation_does_not_ignore_model_path() -> None:
|
||||
request = _make_request(
|
||||
{"x-routstr-model-path": encode_model_path("http://localhost", 1, MODEL_ID)},
|
||||
b"",
|
||||
)
|
||||
request.method = "GET"
|
||||
with patch.object(proxy_module, "_select_unauthenticated_get_upstreams") as select:
|
||||
response = await _run_proxy(request, [], "attestation")
|
||||
assert response.status_code == 400
|
||||
select.assert_not_called()
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_model_fallback_list_is_rejected_when_pinned() -> None:
|
||||
selected = _make_upstream(1)
|
||||
request = _make_request(
|
||||
{
|
||||
"authorization": "Bearer key",
|
||||
"x-routstr-model-path": encode_model_path(selected.base_url, 1, MODEL_ID),
|
||||
},
|
||||
json.dumps({"model": MODEL_ID, "models": ["other-model"]}).encode(),
|
||||
)
|
||||
response = await _run_proxy(request, [(MagicMock(), selected)])
|
||||
assert response.status_code == 400
|
||||
selected.forward_request.assert_not_awaited()
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@pytest.mark.parametrize("endpoint", [None, "deepinfra/fp8"])
|
||||
@pytest.mark.parametrize("path", ["v1/chat/completions", "v1/responses"])
|
||||
@pytest.mark.parametrize("final_status", [200, 429])
|
||||
async def test_pinned_recovery_stays_on_selected_provider(
|
||||
endpoint: str | None, path: str, final_status: int
|
||||
) -> None:
|
||||
selected, fallback = _make_upstream(1), _make_upstream(2)
|
||||
selected.base_url = "https://openrouter.ai/api/v1"
|
||||
handler = (
|
||||
"forward_responses_request" if path == "v1/responses" else "forward_request"
|
||||
)
|
||||
forward = AsyncMock(
|
||||
side_effect=[
|
||||
MagicMock(
|
||||
status_code=400,
|
||||
body=b'{"error":{"message":"temperature is deprecated"}}',
|
||||
),
|
||||
MagicMock(status_code=final_status, body=b"{}"),
|
||||
]
|
||||
)
|
||||
setattr(selected, handler, forward)
|
||||
setattr(fallback, handler, AsyncMock())
|
||||
request = _make_request(
|
||||
{
|
||||
"authorization": "Bearer key",
|
||||
"x-routstr-model-path": encode_model_path(
|
||||
selected.base_url, 1, MODEL_ID, endpoint
|
||||
),
|
||||
},
|
||||
json.dumps(
|
||||
{
|
||||
"model": MODEL_ID,
|
||||
"temperature": 0.7,
|
||||
"provider": {"data_collection": "deny"},
|
||||
}
|
||||
).encode(),
|
||||
)
|
||||
|
||||
response = await _run_proxy(
|
||||
request, [(MagicMock(), selected), (MagicMock(), fallback)], path
|
||||
)
|
||||
|
||||
assert response.status_code == final_status
|
||||
assert forward.await_count == 2
|
||||
before, after = [json.loads(call.args[3]) for call in forward.await_args_list]
|
||||
assert "temperature" in before
|
||||
assert "temperature" not in after
|
||||
assert after["model"] == before["model"] == MODEL_ID
|
||||
assert after["provider"] == before["provider"]
|
||||
if endpoint:
|
||||
assert after["provider"]["order"] == [endpoint]
|
||||
assert after["provider"]["allow_fallbacks"] is False
|
||||
getattr(fallback, handler).assert_not_awaited()
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@pytest.mark.parametrize("field", ["model", "provider"])
|
||||
@pytest.mark.parametrize("endpoint", [None, "deepinfra/fp8"])
|
||||
async def test_pinned_recovery_preserves_routing_fields(
|
||||
field: str, endpoint: str | None
|
||||
) -> None:
|
||||
selected, fallback = _make_upstream(1, 400), _make_upstream(2)
|
||||
selected.base_url = "https://openrouter.ai/api/v1"
|
||||
selected.forward_request.return_value.body = json.dumps(
|
||||
{"error": {"message": f"{field} is not supported"}}
|
||||
).encode()
|
||||
request = _make_request(
|
||||
{
|
||||
"authorization": "Bearer key",
|
||||
"x-routstr-model-path": encode_model_path(
|
||||
selected.base_url, 1, MODEL_ID, endpoint
|
||||
),
|
||||
},
|
||||
json.dumps(
|
||||
{"model": MODEL_ID, "provider": {"data_collection": "deny"}}
|
||||
).encode(),
|
||||
)
|
||||
|
||||
response = await _run_proxy(
|
||||
request, [(MagicMock(), selected), (MagicMock(), fallback)]
|
||||
)
|
||||
|
||||
assert response.status_code == 400
|
||||
selected.forward_request.assert_awaited_once()
|
||||
fallback.forward_request.assert_not_awaited()
|
||||
+279
-13
@@ -28,6 +28,7 @@ os.environ.setdefault("UPSTREAM_BASE_URL", "http://test")
|
||||
os.environ.setdefault("UPSTREAM_API_KEY", "test")
|
||||
|
||||
from routstr.core.db import ModelRow, UpstreamProviderRow # noqa: E402
|
||||
from routstr.payment import price as price_module # noqa: E402
|
||||
from routstr.payment.models import models_router # noqa: E402
|
||||
from routstr.upstream import model_paths as mp # noqa: E402
|
||||
from routstr.upstream.base import BaseUpstreamProvider # noqa: E402
|
||||
@@ -202,6 +203,75 @@ async def patched_session(
|
||||
await engine.dispose()
|
||||
|
||||
|
||||
# 1 sat = $0.00005, the quote the path pricing converts with in these tests.
|
||||
_QUOTE = 5.0e-5
|
||||
_DEFAULT_FEE = 1.01
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def sats_quote(monkeypatch: pytest.MonkeyPatch) -> float:
|
||||
monkeypatch.setattr(price_module, "sats_usd_price", lambda: _QUOTE)
|
||||
return _QUOTE
|
||||
|
||||
|
||||
def _priced_endpoints_response() -> httpx.Response:
|
||||
"""Two endpoints for one model, priced and sized differently."""
|
||||
return httpx.Response(
|
||||
200,
|
||||
json={
|
||||
"data": {
|
||||
"id": "anthropic/claude-opus-4.6",
|
||||
"name": "Claude Opus 4.6",
|
||||
"description": "Anthropic's most capable model",
|
||||
"architecture": {
|
||||
"input_modalities": ["text", "image"],
|
||||
"output_modalities": ["text"],
|
||||
"tokenizer": "Claude",
|
||||
"instruct_type": None,
|
||||
},
|
||||
"endpoints": [
|
||||
{
|
||||
"provider_name": "Anthropic",
|
||||
"tag": "anthropic",
|
||||
"context_length": 200_000,
|
||||
"pricing": {
|
||||
"prompt": "0.000005",
|
||||
"completion": "0.000025",
|
||||
},
|
||||
},
|
||||
{
|
||||
"provider_name": "Google",
|
||||
"tag": "google-vertex/us",
|
||||
"context_length": 128_000,
|
||||
"pricing": {
|
||||
"prompt": "0.000003",
|
||||
"completion": "0.000015",
|
||||
},
|
||||
},
|
||||
],
|
||||
}
|
||||
},
|
||||
)
|
||||
|
||||
|
||||
def _models_by_endpoint(payload: dict, model_id: str) -> dict[str, dict]:
|
||||
entry = next(item for item in payload["data"] if item["id"] == model_id)
|
||||
return {
|
||||
path["endpoint"]["tag"]: path["model"]
|
||||
for path in entry["paths"]
|
||||
if path["endpoint"] is not None
|
||||
}
|
||||
|
||||
|
||||
async def _set_provider_fee(engine: AsyncEngine, provider_id: int, fee: float) -> None:
|
||||
async with AsyncSession(engine) as session:
|
||||
provider = await session.get(UpstreamProviderRow, provider_id)
|
||||
assert provider is not None
|
||||
provider.provider_fee = fee
|
||||
session.add(provider)
|
||||
await session.commit()
|
||||
|
||||
|
||||
def _paths_of(payload: dict, model_id: str) -> set[str]:
|
||||
for entry in payload["data"]:
|
||||
if entry["id"] == model_id:
|
||||
@@ -244,6 +314,12 @@ def _path_entry(
|
||||
or ("anthropic" if provider_id == 1 else "openrouter"),
|
||||
},
|
||||
"endpoint": endpoint,
|
||||
"model": {
|
||||
"id": model_id,
|
||||
"forwarded_model_id": None,
|
||||
"canonical_slug": None,
|
||||
"enabled": True,
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
@@ -252,10 +328,22 @@ def _path_entry(
|
||||
# --------------------------------------------------------------------------- #
|
||||
|
||||
|
||||
def test_is_openrouter_base_url() -> None:
|
||||
assert mp.is_openrouter_base_url("https://openrouter.ai/api/v1") is True
|
||||
assert mp.is_openrouter_base_url("https://api.anthropic.com") is False
|
||||
assert mp.is_openrouter_base_url(None) is False
|
||||
@pytest.mark.parametrize(
|
||||
"url, expected",
|
||||
[
|
||||
("https://openrouter.ai/api/v1", True),
|
||||
("https://OPENROUTER.AI/api/v1", True),
|
||||
("https://api.anthropic.com", False),
|
||||
("https://openrouter.ai.evil.test/api/v1", False),
|
||||
("https://evil.test/openrouter.ai", False),
|
||||
("https://openrouter.ai@evil.test/api/v1", False),
|
||||
("https://[invalid", False),
|
||||
("", False),
|
||||
(None, False),
|
||||
],
|
||||
)
|
||||
def test_is_openrouter_base_url(url: str | None, expected: bool) -> None:
|
||||
assert mp.is_openrouter_base_url(url) is expected
|
||||
|
||||
|
||||
def test_native_anthropic_not_openrouter() -> None:
|
||||
@@ -367,6 +455,37 @@ async def test_direct_provider_single_path_uses_provider_type(
|
||||
assert payload["updated_at"] is not None
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_get_all_model_paths_includes_details_for_each_path(
|
||||
patched_session: AsyncEngine,
|
||||
) -> None:
|
||||
model = _model("claude-opus-4.6")
|
||||
model.name = "Claude Opus 4.6"
|
||||
model.description = "Anthropic's most capable model"
|
||||
model.pricing = {"prompt": 0.000001, "completion": 0.000002}
|
||||
provider = _FakeProvider(
|
||||
provider_type="anthropic",
|
||||
base_url="https://api.anthropic.com/v1",
|
||||
models=[model],
|
||||
db_id=1,
|
||||
)
|
||||
await mp.refresh_model_paths([provider])
|
||||
|
||||
payload = await mp.get_all_model_paths()
|
||||
|
||||
expected_path = _path_entry(1, "claude-opus-4.6")
|
||||
expected_path["model"] = {
|
||||
"id": "claude-opus-4.6",
|
||||
"forwarded_model_id": None,
|
||||
"canonical_slug": None,
|
||||
"enabled": True,
|
||||
"name": "Claude Opus 4.6",
|
||||
"description": "Anthropic's most capable model",
|
||||
"pricing": {"prompt": 0.000001, "completion": 0.000002},
|
||||
}
|
||||
assert payload["data"] == [{"id": "claude-opus-4.6", "paths": [expected_path]}]
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_direct_path_masks_private_configured_provider_url(
|
||||
patched_session: AsyncEngine,
|
||||
@@ -564,9 +683,12 @@ async def test_refresh_model_paths_uses_db_forwarded_alias(
|
||||
|
||||
await mp.refresh_model_paths([provider])
|
||||
|
||||
assert (await mp.get_all_model_paths())["data"] == [
|
||||
{"id": "public-alias", "paths": [_path_entry(1, "public-alias")]}
|
||||
]
|
||||
payload = await mp.get_all_model_paths()
|
||||
assert _paths_of(payload, "public-alias") == {_expected_path(1, "public-alias")}
|
||||
model = payload["data"][0]["paths"][0]["model"]
|
||||
assert model["id"] == "public-alias"
|
||||
assert model["description"] == "test model"
|
||||
assert model["pricing"] == {"prompt": 0.000001, "completion": 0.000002}
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@@ -586,12 +708,14 @@ async def test_refresh_model_paths_includes_enabled_db_override_missing_from_cac
|
||||
|
||||
await mp.refresh_model_paths([provider])
|
||||
|
||||
assert (await mp.get_all_model_paths())["data"] == [
|
||||
{
|
||||
"id": "public-deployment",
|
||||
"paths": [_path_entry(1, "public-deployment")],
|
||||
}
|
||||
]
|
||||
payload = await mp.get_all_model_paths()
|
||||
assert _paths_of(payload, "public-deployment") == {
|
||||
_expected_path(1, "public-deployment")
|
||||
}
|
||||
model = payload["data"][0]["paths"][0]["model"]
|
||||
assert model["id"] == "public-deployment"
|
||||
assert model["description"] == "test model"
|
||||
assert model["context_length"] == 8192
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@@ -765,6 +889,148 @@ async def test_openrouter_provider_adds_endpoint_paths(
|
||||
assert {item["provider"]["id"] for item in payload["data"]} == {2}
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_openrouter_paths_include_endpoint_specific_model_prices(
|
||||
patched_session: AsyncEngine, monkeypatch: pytest.MonkeyPatch, sats_quote: float
|
||||
) -> None:
|
||||
provider = _FakeOpenRouterProvider(
|
||||
models=[_model("claude-opus-4.6", canonical_slug="anthropic/claude-opus-4.6")],
|
||||
db_id=2,
|
||||
)
|
||||
_mock_transport(monkeypatch, lambda request: _priced_endpoints_response())
|
||||
|
||||
await mp.refresh_model_paths([provider])
|
||||
|
||||
payload = await mp.get_all_model_paths()
|
||||
assert payload["data"][0]["id"] == "claude-opus-4.6"
|
||||
models = _models_by_endpoint(payload, "claude-opus-4.6")
|
||||
|
||||
anthropic = models["anthropic"]
|
||||
google = models["google-vertex/us"]
|
||||
assert anthropic["description"] == "Anthropic's most capable model"
|
||||
assert google["description"] == "Anthropic's most capable model"
|
||||
assert anthropic["pricing"]["prompt"] == pytest.approx(0.000005 * _DEFAULT_FEE)
|
||||
assert anthropic["pricing"]["completion"] == pytest.approx(0.000025 * _DEFAULT_FEE)
|
||||
assert google["pricing"]["prompt"] == pytest.approx(0.000003 * _DEFAULT_FEE)
|
||||
assert google["pricing"]["completion"] == pytest.approx(0.000015 * _DEFAULT_FEE)
|
||||
assert anthropic["context_length"] == 200_000
|
||||
assert google["context_length"] == 128_000
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_endpoint_paths_are_priced_in_sats_from_their_own_rates(
|
||||
patched_session: AsyncEngine, monkeypatch: pytest.MonkeyPatch, sats_quote: float
|
||||
) -> None:
|
||||
provider = _FakeOpenRouterProvider(
|
||||
models=[_model("claude-opus-4.6", canonical_slug="anthropic/claude-opus-4.6")],
|
||||
db_id=2,
|
||||
)
|
||||
_mock_transport(monkeypatch, lambda request: _priced_endpoints_response())
|
||||
|
||||
await mp.refresh_model_paths([provider])
|
||||
|
||||
models = _models_by_endpoint(await mp.get_all_model_paths(), "claude-opus-4.6")
|
||||
anthropic = models["anthropic"]["sats_pricing"]
|
||||
google = models["google-vertex/us"]["sats_pricing"]
|
||||
|
||||
assert anthropic["prompt"] == pytest.approx(0.000005 * _DEFAULT_FEE / sats_quote)
|
||||
assert anthropic["completion"] == pytest.approx(
|
||||
0.000025 * _DEFAULT_FEE / sats_quote
|
||||
)
|
||||
assert google["prompt"] == pytest.approx(0.000003 * _DEFAULT_FEE / sats_quote)
|
||||
assert google["completion"] == pytest.approx(0.000015 * _DEFAULT_FEE / sats_quote)
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_endpoint_max_costs_use_that_endpoint_context_length(
|
||||
patched_session: AsyncEngine, monkeypatch: pytest.MonkeyPatch, sats_quote: float
|
||||
) -> None:
|
||||
provider = _FakeOpenRouterProvider(
|
||||
models=[_model("claude-opus-4.6", canonical_slug="anthropic/claude-opus-4.6")],
|
||||
db_id=2,
|
||||
)
|
||||
_mock_transport(monkeypatch, lambda request: _priced_endpoints_response())
|
||||
|
||||
await mp.refresh_model_paths([provider])
|
||||
|
||||
models = _models_by_endpoint(await mp.get_all_model_paths(), "claude-opus-4.6")
|
||||
# Max cost is the context window billed at the dearer of the two rates.
|
||||
assert models["anthropic"]["sats_pricing"]["max_cost"] == pytest.approx(
|
||||
200_000 * 0.000025 * _DEFAULT_FEE / sats_quote
|
||||
)
|
||||
assert models["google-vertex/us"]["sats_pricing"]["max_cost"] == pytest.approx(
|
||||
128_000 * 0.000015 * _DEFAULT_FEE / sats_quote
|
||||
)
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_path_pricing_uses_the_provider_fee_of_its_own_provider(
|
||||
patched_session: AsyncEngine, monkeypatch: pytest.MonkeyPatch, sats_quote: float
|
||||
) -> None:
|
||||
await _set_provider_fee(patched_session, 2, 1.5)
|
||||
provider = _FakeOpenRouterProvider(
|
||||
models=[_model("claude-opus-4.6", canonical_slug="anthropic/claude-opus-4.6")],
|
||||
db_id=2,
|
||||
)
|
||||
_mock_transport(monkeypatch, lambda request: _priced_endpoints_response())
|
||||
|
||||
await mp.refresh_model_paths([provider])
|
||||
|
||||
models = _models_by_endpoint(await mp.get_all_model_paths(), "claude-opus-4.6")
|
||||
anthropic = models["anthropic"]
|
||||
assert anthropic["pricing"]["prompt"] == pytest.approx(0.000005 * 1.5)
|
||||
assert anthropic["sats_pricing"]["prompt"] == pytest.approx(
|
||||
0.000005 * 1.5 / sats_quote
|
||||
)
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_paths_keep_upstream_pricing_when_the_quote_is_unavailable(
|
||||
patched_session: AsyncEngine, monkeypatch: pytest.MonkeyPatch
|
||||
) -> None:
|
||||
def _no_quote() -> float:
|
||||
raise ValueError("SATS price not initialized")
|
||||
|
||||
monkeypatch.setattr(price_module, "sats_usd_price", _no_quote)
|
||||
provider = _FakeOpenRouterProvider(
|
||||
models=[_model("claude-opus-4.6", canonical_slug="anthropic/claude-opus-4.6")],
|
||||
db_id=2,
|
||||
)
|
||||
_mock_transport(monkeypatch, lambda request: _priced_endpoints_response())
|
||||
|
||||
await mp.refresh_model_paths([provider])
|
||||
|
||||
models = _models_by_endpoint(await mp.get_all_model_paths(), "claude-opus-4.6")
|
||||
anthropic = models["anthropic"]
|
||||
assert "sats_pricing" not in anthropic
|
||||
assert anthropic["pricing"] == {
|
||||
"prompt": "0.000005",
|
||||
"completion": "0.000025",
|
||||
}
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_already_priced_metadata_is_not_priced_again(
|
||||
patched_session: AsyncEngine, sats_quote: float
|
||||
) -> None:
|
||||
model = _model("claude-opus-4.6")
|
||||
model.pricing = {"prompt": 0.000001, "completion": 0.000002}
|
||||
model.sats_pricing = {"prompt": 0.02, "completion": 0.04}
|
||||
provider = _FakeProvider(
|
||||
provider_type="anthropic",
|
||||
base_url="https://api.anthropic.com/v1",
|
||||
models=[model],
|
||||
db_id=1,
|
||||
)
|
||||
|
||||
await mp.refresh_model_paths([provider])
|
||||
|
||||
payload = await mp.get_all_model_paths()
|
||||
priced = payload["data"][0]["paths"][0]["model"]
|
||||
assert priced["sats_pricing"] == {"prompt": 0.02, "completion": 0.04}
|
||||
assert priced["pricing"] == {"prompt": 0.000001, "completion": 0.000002}
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_openrouter_uses_exact_tag_even_when_display_name_is_router(
|
||||
patched_session: AsyncEngine, monkeypatch: pytest.MonkeyPatch
|
||||
|
||||
@@ -288,3 +288,519 @@ async def test_discount_cannot_be_dodged_by_hiding_prompt_in_tools() -> None:
|
||||
# Same prompt weight → at least the same reservation, never the floor.
|
||||
assert cost >= cost_messages, where
|
||||
assert cost > 1000, where
|
||||
|
||||
|
||||
async def test_discounted_max_cost_counts_responses_input_images() -> None:
|
||||
import base64
|
||||
from io import BytesIO
|
||||
|
||||
from PIL import Image
|
||||
|
||||
from routstr.payment.helpers import calculate_discounted_max_cost
|
||||
|
||||
pricing = Mock()
|
||||
pricing.prompt = 0.001
|
||||
pricing.completion = 0.001
|
||||
pricing.max_prompt_cost = 100.0
|
||||
pricing.max_completion_cost = 0.0
|
||||
|
||||
model_obj = Mock()
|
||||
model_obj.sats_pricing = pricing
|
||||
model_obj.top_provider = None
|
||||
model_obj.context_length = None
|
||||
|
||||
image = Image.new("RGB", (512, 512), "red")
|
||||
buffer = BytesIO()
|
||||
image.save(buffer, format="JPEG")
|
||||
data_url = "data:image/jpeg;base64," + base64.b64encode(buffer.getvalue()).decode()
|
||||
|
||||
no_image = {
|
||||
"model": "test-model",
|
||||
"input": [{"role": "user", "content": "hi"}],
|
||||
}
|
||||
with_image = {
|
||||
"model": "test-model",
|
||||
"input": [
|
||||
{
|
||||
"role": "user",
|
||||
"content": [
|
||||
{"type": "input_text", "text": "hi"},
|
||||
{"type": "input_image", "image_url": data_url, "detail": "high"},
|
||||
],
|
||||
}
|
||||
],
|
||||
}
|
||||
|
||||
with (
|
||||
patch.object(settings, "fixed_pricing", False),
|
||||
patch.object(settings, "tolerance_percentage", 0),
|
||||
patch.object(settings, "min_request_msat", 1000),
|
||||
):
|
||||
cost_no_image = await calculate_discounted_max_cost(
|
||||
100_000, no_image, model_obj
|
||||
)
|
||||
cost_with_image = await calculate_discounted_max_cost(
|
||||
100_000, with_image, model_obj
|
||||
)
|
||||
|
||||
# The 512x512 high-detail image (85 + 170 = 255 tokens) is billed as prompt
|
||||
# weight, so it reserves strictly more than the identical text-only body.
|
||||
assert cost_with_image > cost_no_image
|
||||
|
||||
|
||||
def _responses_image(url: str, detail: str | None = "original") -> list[dict[str, Any]]:
|
||||
return [
|
||||
{
|
||||
"role": "user",
|
||||
"content": [{"type": "input_image", "image_url": url, "detail": detail}],
|
||||
}
|
||||
]
|
||||
|
||||
|
||||
async def _responses_image_tokens(input_data: list[dict[str, Any]]) -> int:
|
||||
from routstr.payment.helpers import estimate_image_tokens_in_messages
|
||||
from routstr.payment.responses_input import responses_input_to_messages
|
||||
|
||||
messages = responses_input_to_messages(input_data)
|
||||
assert messages is not None
|
||||
return await estimate_image_tokens_in_messages(messages)
|
||||
|
||||
|
||||
async def test_remote_original_image_is_fetched_not_worst_cased() -> None:
|
||||
from io import BytesIO
|
||||
|
||||
from PIL import Image
|
||||
|
||||
image = Image.new("RGB", (512, 512), "red")
|
||||
buffer = BytesIO()
|
||||
image.save(buffer, format="JPEG")
|
||||
image_bytes = buffer.getvalue()
|
||||
|
||||
with patch(
|
||||
"routstr.payment.helpers._fetch_image_from_url",
|
||||
new=AsyncMock(return_value=image_bytes),
|
||||
):
|
||||
# 256 patches * 1.2
|
||||
assert (
|
||||
await _responses_image_tokens(_responses_image("https://x.test/i.jpg"))
|
||||
== 308
|
||||
)
|
||||
|
||||
with patch(
|
||||
"routstr.payment.helpers._fetch_image_from_url",
|
||||
new=AsyncMock(return_value=None),
|
||||
):
|
||||
assert (
|
||||
await _responses_image_tokens(_responses_image("https://x.test/i.jpg"))
|
||||
== 36_000
|
||||
)
|
||||
|
||||
|
||||
async def test_broken_data_url_reserves_declared_detail_worst_case() -> None:
|
||||
from routstr.payment.helpers import estimate_image_tokens_in_messages
|
||||
|
||||
broken = "data:image/jpeg;base64,!!!"
|
||||
assert await _responses_image_tokens(_responses_image(broken)) == 36_000
|
||||
assert await _responses_image_tokens(_responses_image(broken, "high")) == 85 + (
|
||||
170 * 4
|
||||
)
|
||||
|
||||
chat = [
|
||||
{
|
||||
"role": "user",
|
||||
"content": [
|
||||
{
|
||||
"type": "image_url",
|
||||
"image_url": {"url": broken, "detail": "original"},
|
||||
}
|
||||
],
|
||||
}
|
||||
]
|
||||
assert await estimate_image_tokens_in_messages(chat) == 36_000
|
||||
|
||||
|
||||
async def test_chat_original_image_fetch_failure_reserves_worst_case() -> None:
|
||||
from routstr.payment.helpers import estimate_image_tokens_in_messages
|
||||
|
||||
chat = [
|
||||
{
|
||||
"role": "user",
|
||||
"content": [
|
||||
{
|
||||
"type": "image_url",
|
||||
"image_url": {"url": "https://x.test/i.jpg", "detail": "original"},
|
||||
}
|
||||
],
|
||||
}
|
||||
]
|
||||
with patch(
|
||||
"routstr.payment.helpers._fetch_image_from_url",
|
||||
new=AsyncMock(return_value=None),
|
||||
):
|
||||
assert await estimate_image_tokens_in_messages(chat) == 36_000
|
||||
|
||||
|
||||
async def test_responses_images_share_per_request_fetch_cap() -> None:
|
||||
from routstr.payment.helpers import IMAGE_FETCH_MAX_PER_REQUEST
|
||||
|
||||
input_data = [
|
||||
{
|
||||
"role": "user",
|
||||
"content": [
|
||||
{
|
||||
"type": "input_image",
|
||||
"image_url": f"https://x.test/{i}.jpg",
|
||||
"detail": "original",
|
||||
}
|
||||
for i in range(IMAGE_FETCH_MAX_PER_REQUEST + 1)
|
||||
],
|
||||
}
|
||||
]
|
||||
fetch = AsyncMock(return_value=None)
|
||||
with patch("routstr.payment.helpers._fetch_image_from_url", new=fetch):
|
||||
tokens = await _responses_image_tokens(input_data)
|
||||
|
||||
assert fetch.await_count == IMAGE_FETCH_MAX_PER_REQUEST
|
||||
assert tokens == 36_000 * (IMAGE_FETCH_MAX_PER_REQUEST + 1)
|
||||
|
||||
|
||||
async def test_responses_transform_failure_falls_back_to_worst_case() -> None:
|
||||
from routstr.payment.helpers import calculate_discounted_max_cost
|
||||
|
||||
pricing = Mock()
|
||||
pricing.prompt = 0.001
|
||||
pricing.completion = 0.001
|
||||
pricing.max_prompt_cost = 100.0
|
||||
pricing.max_completion_cost = 0.0
|
||||
|
||||
model_obj = Mock()
|
||||
model_obj.sats_pricing = pricing
|
||||
model_obj.top_provider = None
|
||||
model_obj.context_length = None
|
||||
|
||||
body = {
|
||||
"model": "test-model",
|
||||
"input": [
|
||||
{
|
||||
"role": "user",
|
||||
"content": [
|
||||
{"type": "input_image", "image_url": "https://x.test/a.jpg"},
|
||||
{"type": "input_image", "image_url": "https://x.test/b.jpg"},
|
||||
],
|
||||
}
|
||||
],
|
||||
}
|
||||
fetch = AsyncMock(return_value=None)
|
||||
with (
|
||||
patch.object(settings, "fixed_pricing", False),
|
||||
patch.object(settings, "tolerance_percentage", 0),
|
||||
patch.object(settings, "min_request_msat", 1000),
|
||||
patch("routstr.payment.helpers._fetch_image_from_url", new=fetch),
|
||||
patch(
|
||||
"routstr.payment.responses_input.LiteLLMCompletionResponsesConfig."
|
||||
"transform_responses_api_input_to_messages",
|
||||
side_effect=RuntimeError("boom"),
|
||||
),
|
||||
):
|
||||
cost = await calculate_discounted_max_cost(100_000, body, model_obj)
|
||||
|
||||
fetch.assert_not_awaited()
|
||||
# 2 * 36,000 tokens * 0.001 sats = 72 sats reserved
|
||||
assert 72_000 <= cost < 100_000
|
||||
|
||||
|
||||
async def test_discounted_max_cost_body_max_output_tokens_fallback() -> None:
|
||||
"""Body ``max_output_tokens`` (Responses API) is honored as a completion cap."""
|
||||
from routstr.payment.helpers import calculate_discounted_max_cost
|
||||
|
||||
pricing = Mock()
|
||||
pricing.prompt = 0.001
|
||||
pricing.completion = 0.001
|
||||
pricing.max_prompt_cost = 0.0
|
||||
pricing.max_completion_cost = 100.0
|
||||
|
||||
model_obj = Mock()
|
||||
model_obj.sats_pricing = pricing
|
||||
model_obj.top_provider = None
|
||||
model_obj.context_length = None
|
||||
|
||||
body = {"max_output_tokens": 80_000}
|
||||
|
||||
with (
|
||||
patch.object(settings, "fixed_pricing", False),
|
||||
patch.object(settings, "tolerance_percentage", 0),
|
||||
patch.object(settings, "min_request_msat", 1000),
|
||||
):
|
||||
cost = await calculate_discounted_max_cost(100_000, body, model_obj)
|
||||
|
||||
assert cost == 80_000
|
||||
|
||||
|
||||
async def test_discounted_max_cost_body_max_completion_tokens_fallback() -> None:
|
||||
"""Body ``max_completion_tokens`` (modern chat) is honored as a completion cap."""
|
||||
from routstr.payment.helpers import calculate_discounted_max_cost
|
||||
|
||||
pricing = Mock()
|
||||
pricing.prompt = 0.001
|
||||
pricing.completion = 0.001
|
||||
pricing.max_prompt_cost = 0.0
|
||||
pricing.max_completion_cost = 100.0
|
||||
|
||||
model_obj = Mock()
|
||||
model_obj.sats_pricing = pricing
|
||||
model_obj.top_provider = None
|
||||
model_obj.context_length = None
|
||||
|
||||
body = {"max_completion_tokens": 80_000}
|
||||
|
||||
with (
|
||||
patch.object(settings, "fixed_pricing", False),
|
||||
patch.object(settings, "tolerance_percentage", 0),
|
||||
patch.object(settings, "min_request_msat", 1000),
|
||||
):
|
||||
cost = await calculate_discounted_max_cost(100_000, body, model_obj)
|
||||
|
||||
assert cost == 80_000
|
||||
|
||||
|
||||
async def test_discounted_max_cost_uses_largest_completion_cap() -> None:
|
||||
"""With several completion caps declared, the largest bounds the reservation.
|
||||
|
||||
Upstream precedence between ``max_tokens`` / ``max_completion_tokens`` /
|
||||
``max_output_tokens`` varies by provider, so reserving against anything
|
||||
but the largest could under-cover what the upstream bills.
|
||||
"""
|
||||
from routstr.payment.helpers import calculate_discounted_max_cost
|
||||
|
||||
pricing = Mock()
|
||||
pricing.prompt = 0.001
|
||||
pricing.completion = 0.001
|
||||
pricing.max_prompt_cost = 0.0
|
||||
pricing.max_completion_cost = 100.0
|
||||
|
||||
model_obj = Mock()
|
||||
model_obj.sats_pricing = pricing
|
||||
model_obj.top_provider = None
|
||||
model_obj.context_length = None
|
||||
|
||||
body = {
|
||||
"max_tokens": 50_000,
|
||||
"max_completion_tokens": 10_000,
|
||||
"max_output_tokens": 80_000,
|
||||
}
|
||||
|
||||
with (
|
||||
patch.object(settings, "fixed_pricing", False),
|
||||
patch.object(settings, "tolerance_percentage", 0),
|
||||
patch.object(settings, "min_request_msat", 1000),
|
||||
):
|
||||
cost = await calculate_discounted_max_cost(100_000, body, model_obj)
|
||||
|
||||
# 80_000 is the largest declared cap: 100.0 - 80.0 = 20 sats discount.
|
||||
assert cost == 80_000
|
||||
|
||||
|
||||
async def test_discounted_max_cost_invalid_completion_cap_ignored() -> None:
|
||||
"""Unparseable caps yield no completion discount rather than under-reserving."""
|
||||
from routstr.payment.helpers import calculate_discounted_max_cost
|
||||
|
||||
pricing = Mock()
|
||||
pricing.prompt = 0.001
|
||||
pricing.completion = 0.001
|
||||
pricing.max_prompt_cost = 0.0
|
||||
pricing.max_completion_cost = 100.0
|
||||
|
||||
model_obj = Mock()
|
||||
model_obj.sats_pricing = pricing
|
||||
model_obj.top_provider = None
|
||||
model_obj.context_length = None
|
||||
|
||||
with (
|
||||
patch.object(settings, "fixed_pricing", False),
|
||||
patch.object(settings, "tolerance_percentage", 0),
|
||||
patch.object(settings, "min_request_msat", 1000),
|
||||
):
|
||||
# No valid cap at all -> no completion discount.
|
||||
cost = await calculate_discounted_max_cost(
|
||||
100_000, {"max_completion_tokens": "sixty-four-k"}, model_obj
|
||||
)
|
||||
assert cost == 100_000
|
||||
|
||||
# An invalid sibling does not poison a valid cap on another field.
|
||||
cost = await calculate_discounted_max_cost(
|
||||
100_000,
|
||||
{"max_tokens": "bad", "max_completion_tokens": 80_000},
|
||||
model_obj,
|
||||
)
|
||||
assert cost == 80_000
|
||||
|
||||
|
||||
def _responses_file_image(detail: str | None) -> list[dict[str, Any]]:
|
||||
part: dict[str, Any] = {"type": "input_image", "file_id": "file-1"}
|
||||
if detail is not None:
|
||||
part["detail"] = detail
|
||||
return [{"role": "user", "content": [part]}]
|
||||
|
||||
|
||||
async def test_responses_input_detail_and_file_id() -> None:
|
||||
import base64
|
||||
from io import BytesIO
|
||||
|
||||
from PIL import Image
|
||||
|
||||
# file_id: dimensions can't be fetched, so use a conservative max-size
|
||||
# estimate (4 tiles for auto/high) and honor the detail sibling for low.
|
||||
fetch = AsyncMock(return_value=None)
|
||||
with patch("routstr.payment.helpers._fetch_image_from_url", new=fetch):
|
||||
assert await _responses_image_tokens(_responses_file_image(None)) == 85 + (
|
||||
170 * 4
|
||||
)
|
||||
assert await _responses_image_tokens(_responses_file_image("low")) == 85
|
||||
fetch.assert_not_awaited()
|
||||
|
||||
# image_url honors the sibling detail instead of always defaulting to auto.
|
||||
image = Image.new("RGB", (512, 512), "red")
|
||||
buffer = BytesIO()
|
||||
image.save(buffer, format="JPEG")
|
||||
data_url = "data:image/jpeg;base64," + base64.b64encode(buffer.getvalue()).decode()
|
||||
|
||||
assert await _responses_image_tokens(_responses_image(data_url, "low")) == 85
|
||||
assert (
|
||||
await _responses_image_tokens(_responses_image(data_url, "high")) == 85 + 170
|
||||
) # 512x512 = 1 tile
|
||||
|
||||
|
||||
def test_calculate_image_tokens_original_detail() -> None:
|
||||
from routstr.payment.helpers import _calculate_image_tokens
|
||||
|
||||
# Patch-based pricing: ceil(patches * 1.2) tokens at 32x32px patches.
|
||||
assert _calculate_image_tokens(640, 640, "original") == 480 # 400 patches
|
||||
assert _calculate_image_tokens(2048, 2048, "original") == 4_916 # 4,096 patches
|
||||
# The same image on the tiled high-detail path caps at 765 tokens.
|
||||
assert _calculate_image_tokens(2048, 2048, "high") == 765
|
||||
# Above the 30,000-patch rejection limit the estimate is capped at
|
||||
# 36,000 tokens (30,000 patches * 1.2).
|
||||
assert _calculate_image_tokens(10_000, 10_000, "original") == 36_000
|
||||
|
||||
|
||||
async def test_responses_input_original_detail() -> None:
|
||||
import base64
|
||||
from io import BytesIO
|
||||
|
||||
from PIL import Image
|
||||
|
||||
image = Image.new("RGB", (2048, 2048), "red")
|
||||
buffer = BytesIO()
|
||||
image.save(buffer, format="JPEG")
|
||||
data_url = "data:image/jpeg;base64," + base64.b64encode(buffer.getvalue()).decode()
|
||||
|
||||
# image_url: billed at the decoded original resolution (4,096 patches),
|
||||
# not the 765-token tile cap.
|
||||
assert await _responses_image_tokens(_responses_image(data_url)) == 4_916
|
||||
|
||||
# file_id: dimensions unknown, so use the 30,000-patch worst case.
|
||||
assert await _responses_image_tokens(_responses_file_image("original")) == 36_000
|
||||
|
||||
# Explicit null detail behaves like the auto default (tiled math).
|
||||
assert await _responses_image_tokens(_responses_image(data_url, None)) == 85 + (
|
||||
170 * 4
|
||||
)
|
||||
|
||||
|
||||
def test_responses_input_to_messages_shapes() -> None:
|
||||
from routstr.payment.responses_input import (
|
||||
FILE_ID_URL_PREFIX,
|
||||
count_input_images,
|
||||
responses_input_to_messages,
|
||||
)
|
||||
|
||||
input_data = [
|
||||
{
|
||||
"role": "user",
|
||||
"content": [
|
||||
{"type": "input_text", "text": "hi"},
|
||||
{"type": "input_image", "file_id": "file-1", "detail": "original"},
|
||||
],
|
||||
},
|
||||
{"type": "function_call_output", "call_id": "c1", "output": "out"},
|
||||
]
|
||||
messages = responses_input_to_messages(input_data)
|
||||
assert messages is not None
|
||||
assert messages[0]["role"] == "user"
|
||||
parts = messages[0]["content"]
|
||||
assert parts[0] == {"type": "text", "text": "hi"}
|
||||
assert parts[1]["type"] == "image_url"
|
||||
assert parts[1]["image_url"] == {
|
||||
"url": f"{FILE_ID_URL_PREFIX}file-1",
|
||||
"detail": "original",
|
||||
}
|
||||
assert messages[1]["role"] == "tool"
|
||||
|
||||
# dict-form image_url: litellm nests it verbatim, so it is flattened first.
|
||||
nested = responses_input_to_messages(
|
||||
[
|
||||
{
|
||||
"type": "message",
|
||||
"role": "user",
|
||||
"content": [
|
||||
{
|
||||
"type": "input_image",
|
||||
"image_url": {
|
||||
"url": "https://x.test/a.jpg",
|
||||
"detail": "original",
|
||||
},
|
||||
}
|
||||
],
|
||||
}
|
||||
]
|
||||
)
|
||||
assert nested is not None
|
||||
assert nested[0]["content"][0]["image_url"] == {
|
||||
"url": "https://x.test/a.jpg",
|
||||
"detail": "original",
|
||||
}
|
||||
|
||||
assert responses_input_to_messages("plain") == [
|
||||
{"role": "user", "content": "plain"}
|
||||
]
|
||||
assert responses_input_to_messages(None) == []
|
||||
assert count_input_images(input_data) == 1
|
||||
|
||||
|
||||
async def test_estimate_image_tokens_in_messages_original_detail() -> None:
|
||||
"""Chat Completions also accepts original detail via the nested dict."""
|
||||
import base64
|
||||
from io import BytesIO
|
||||
|
||||
from PIL import Image
|
||||
|
||||
from routstr.payment.helpers import estimate_image_tokens_in_messages
|
||||
|
||||
image = Image.new("RGB", (640, 640), "blue")
|
||||
buffer = BytesIO()
|
||||
image.save(buffer, format="JPEG")
|
||||
data_url = "data:image/jpeg;base64," + base64.b64encode(buffer.getvalue()).decode()
|
||||
|
||||
messages = [
|
||||
{
|
||||
"role": "user",
|
||||
"content": [
|
||||
{
|
||||
"type": "image_url",
|
||||
"image_url": {"url": data_url, "detail": "original"},
|
||||
}
|
||||
],
|
||||
}
|
||||
]
|
||||
# 640x640 -> 20x20 = 400 patches -> ceil(400 * 1.2) = 480 tokens.
|
||||
assert await estimate_image_tokens_in_messages(messages) == 480
|
||||
|
||||
messages = [
|
||||
{
|
||||
"role": "user",
|
||||
"content": [
|
||||
{"type": "input_image", "file_id": "file-1", "detail": "original"}
|
||||
],
|
||||
}
|
||||
]
|
||||
assert await estimate_image_tokens_in_messages(messages) == 36_000
|
||||
|
||||
Reference in New Issue
Block a user