update RIP-04 into auditing

RIP-04.md

@@ -1,80 +1,39 @@
-# RIP-04: Evaluations & Quality Control
+# RIP-04: Auditing
 
-Specifies how independent evaluation (eval) services scan Nostr for all advertised LLM providers, run anonymized and randomized evals, and publish benchmark reports for each provider.
+This document defines the process for auditing Routstr providers to ensure reliability, compliance, and honesty. Auditing is a lightweight, continuous verification process performed by independent entities or clients themselves to maintain network health.
 
-## Goals
+## 1. Uptime & Reliability Monitoring
 
-- Prevent provider bias by making evals indistinguishable from real inference calls.
-- Collect unbiased metrics on quality, latency, and cost.
-- Establish eval services as the primary source of trust and ranking in the marketplace.
+Auditors periodically check the availability and responsiveness of providers.
 
-## Eval Service Model
+- **Ping Checks**: Auditors send lightweight requests (e.g., `GET /v1/info`) to verify the provider is online.
+- **Latency Tracking**: Response times are measured and logged. High latency or timeouts negatively impact the provider's score.
+- **Uptime Score**: A rolling average of successful checks over time (e.g., 99.9% uptime over 24h) is calculated and may be published via Nostr.
 
-- Evals are separate, continuously running services that monitor Nostr for all LLM providers advertising their capabilities.
-- For each discovered provider, the eval service runs anonymized, randomized, and proxied evaluation jobs to ensure providers cannot distinguish eval requests from normal user traffic.
-- Evals are designed to prevent providers from treating evals differently than real users, using strategies such as randomization, proxying, and ephemeral keys.
-- After running evals, the service publishes a report for each provider, containing benchmark scores and detailed metrics.
+## 2. Feature Verification
 
-## Report Tiers & Monetization
+Auditors verify that providers actually support the features and models they advertise in their Kind 38421 announcements.
 
-- There are multiple tiers of reports:
-  - **Free Report:** All providers receive a basic report at no cost, summarizing essential metrics.
-  - **Paid Reports:** Providers can pay the eval service (in sats) to receive more advanced, detailed reports. These paid reports offer deeper insights, improve user trust, and help the provider rank higher in the marketplace.
-- Eval services are a monetizable component of the ecosystem, earning fees for advanced reporting and acting as a trusted third party.
+- **Model Verification**: Auditors query the `/v1/models` endpoint to confirm the list of supported models matches the advertisement.
+- **Capabilities Check**: Simple inference requests (with minimal tokens) are sent to specific models to ensure they are functional and not just placeholders.
+- **API Compliance**: Checks are performed to ensure the provider adheres to the standard Routstr/OpenAI API schema (correct error codes, header formats, etc.).
 
-## Eval Flow
+## 3. Price Integrity Checks
 
-1. Eval service scans Nostr for all advertised providers.
-2. For each provider:
-   - Generate randomized inference parameters.
-   - Send anonymized, proxied inference requests.
-   - Measure latency, output quality against ground truth.
-3. Assemble eval result and prepare Nostr event:
-   - Use new random ephemeral Nostr key per eval.
-   - Publish Kind 31555 with standard rating tags:
-     - `d`: provider node-id
-     - `rating` categories: `quality`, `latency`, `value`
-   - No direct link to eval service's main pubkey.
-4. Publish report (free or paid tier) with benchmark scores.
+Auditors ensure that the prices charged by the provider match their advertised rates.
 
-## Tags
+- **Advertised vs. Actual**: The auditor compares the pricing information returned in `/v1/models` (or the advertisement event) with the actual cost deducted during a test transaction.
+- **Refund Verification**: The auditor verifies that the `/v1/balance/refund` endpoint correctly returns the expected remaining balance after a transaction.
+- **Overcharge Detection**: If a provider consistently charges more than advertised, they are flagged.
 
-- `d`: provider node-id
-- `rating`:
-  - `quality`: 0–1 match to ground truth
-  - `latency`: normalized inverse latency score
-  - `value`: cost vs performance
-- `content`: optional detailed notes
+## 4. Reporting
 
-## Anonymity & Randomization
+Audit results are published to the Nostr network to inform clients and other network participants.
 
-- Ephemeral keys rotated per session.
-- Randomized send times & intervals.
-- All requests are proxied and randomized to prevent provider detection.
-
-## Aggregation
-
-- Eval services can batch multiple evaluations in one event by repeating `rating` tags.
-
-## Frequency
-
-- Limit eval jobs to ≤5% of overall requests to avoid load spikes.
-
-## Trust & Benchmarking
-
-- Eval services are the source of trust in the marketplace, providing objective, third-party benchmarks.
-- Each report results in a set of benchmark scores, enabling transparent provider comparison and ranking.
-
----
-
-## TODO
-
-- [ ] Implement independent evaluation (eval) services for LLM providers
-- [ ] Scan Nostr for all advertised providers
-- [ ] Run anonymized, randomized, proxied evaluation jobs
-- [ ] Publish benchmark reports for each provider (Kind 31555)
-- [ ] Support free and paid report tiers
-- [ ] Measure and report quality, latency, and cost metrics
-- [ ] Publish standard ratings to Nostr
-- [ ] Limit eval jobs to ≤5% of overall requests
-- [ ] Make public evals modular component of the protocol
+- **Event Kind**: (To be defined, e.g., Kind 3xxxx)
+- **Content**:
+  - `provider_pubkey`: The pubkey of the audited provider.
+  - `timestamp`: Time of the audit.
+  - `status`: `pass`, `fail`, `warning`.
+  - `details`: Specifics of failures (e.g., "Model 'gpt-4' not found", "High latency > 2s").
+- **Client Usage**: Clients ingest these audit events to filter out unreliable or dishonest providers from their local registry.