mirror of
https://github.com/Routstr/protocol.git
synced 2026-07-30 19:36:15 +00:00
move todos
This commit is contained in:
@@ -1,15 +1,5 @@
|
||||
# RIP-04: Evaluations & Quality Control
|
||||
|
||||
- [ ] Implement independent evaluation (eval) services for LLM providers
|
||||
- [ ] Scan Nostr for all advertised providers
|
||||
- [ ] Run anonymized, randomized, proxied evaluation jobs
|
||||
- [ ] Publish benchmark reports for each provider (Kind 31555)
|
||||
- [ ] Support free and paid report tiers
|
||||
- [ ] Measure and report quality, latency, and cost metrics
|
||||
- [ ] Publish standard ratings to Nostr
|
||||
- [ ] Limit eval jobs to ≤5% of overall requests
|
||||
- [ ] Make public evals modular component of the protocol
|
||||
|
||||
Specifies how independent evaluation (eval) services scan Nostr for all advertised LLM providers, run anonymized and randomized evals, and publish benchmark reports for each provider.
|
||||
|
||||
## Goals
|
||||
@@ -74,3 +64,17 @@ Specifies how independent evaluation (eval) services scan Nostr for all advertis
|
||||
|
||||
- Eval services are the source of trust in the marketplace, providing objective, third-party benchmarks.
|
||||
- Each report results in a set of benchmark scores, enabling transparent provider comparison and ranking.
|
||||
|
||||
---
|
||||
|
||||
## TODO
|
||||
|
||||
- [ ] Implement independent evaluation (eval) services for LLM providers
|
||||
- [ ] Scan Nostr for all advertised providers
|
||||
- [ ] Run anonymized, randomized, proxied evaluation jobs
|
||||
- [ ] Publish benchmark reports for each provider (Kind 31555)
|
||||
- [ ] Support free and paid report tiers
|
||||
- [ ] Measure and report quality, latency, and cost metrics
|
||||
- [ ] Publish standard ratings to Nostr
|
||||
- [ ] Limit eval jobs to ≤5% of overall requests
|
||||
- [ ] Make public evals modular component of the protocol
|
||||
|
||||
@@ -1,24 +1,3 @@
|
||||
# TODO
|
||||
|
||||
- [ ] Develop multi-language SDKs (Python, TypeScript/JavaScript, Go, Rust, etc.)
|
||||
- [ ] Implement full-stack components: client, server, discovery, evaluation server, chat API
|
||||
- [ ] Provide unified interfaces for Cashu wallet management, API key rotation, proxy/anonymity
|
||||
- [ ] Expose high-level abstractions in SDKs (token management, API key rotation, provider selection, proxy/Tor integration, secure flows)
|
||||
- [ ] Smart clients: automated token cycling/top-up, per-request API key rotation, proxy/Tor circuit selection
|
||||
- [ ] Dynamic provider discovery and scoring (price, latency, rating, reliability)
|
||||
- [ ] Pluggable architecture for custom scoring, proxy, or wallet backends
|
||||
- [ ] Reference implementations for server, discovery, and evaluation endpoints in each language
|
||||
- [ ] Standardized APIs for provider registration, health, and metrics
|
||||
- [ ] Modular evaluation server for benchmarking and rating providers
|
||||
- [ ] Chat API reference implementations (real-time and batch)
|
||||
- [ ] Built-in proxy and Tor routing in all SDKs and clients
|
||||
- [ ] Automated rotation of API keys and tokens
|
||||
- [ ] Rate limiting and circuit rotation to prevent fingerprinting
|
||||
- [ ] Define core protocol interfaces and data models in each language
|
||||
- [ ] Maintain open-source SDKs and reference servers
|
||||
- [ ] Ensure feature parity and cross-language test suites
|
||||
- [ ] Document integration patterns and privacy best practices
|
||||
|
||||
# RIP-05: SDKs & Smart Clients
|
||||
|
||||
This RIP proposes a comprehensive suite of SDKs and Smart Client libraries across multiple programming languages, enabling seamless integration with the protocol and ensuring robust privacy, payment, and provider management.
|
||||
@@ -77,3 +56,24 @@ This RIP proposes a comprehensive suite of SDKs and Smart Client libraries acros
|
||||
- Document integration patterns and privacy best practices.
|
||||
|
||||
---
|
||||
|
||||
## TODO
|
||||
|
||||
- [ ] Develop multi-language SDKs (Python, TypeScript/JavaScript, Go, Rust, etc.)
|
||||
- [ ] Implement full-stack components: client, server, discovery, evaluation server, chat API
|
||||
- [ ] Provide unified interfaces for Cashu wallet management, API key rotation, proxy/anonymity
|
||||
- [ ] Expose high-level abstractions in SDKs (token management, API key rotation, provider selection, proxy/Tor integration, secure flows)
|
||||
- [ ] Smart clients: automated token cycling/top-up, per-request API key rotation, proxy/Tor circuit selection
|
||||
- [ ] Dynamic provider discovery and scoring (price, latency, rating, reliability)
|
||||
- [ ] Pluggable architecture for custom scoring, proxy, or wallet backends
|
||||
- [ ] Reference implementations for server, discovery, and evaluation endpoints in each language
|
||||
- [ ] Standardized APIs for provider registration, health, and metrics
|
||||
- [ ] Modular evaluation server for benchmarking and rating providers
|
||||
- [ ] Chat API reference implementations (real-time and batch)
|
||||
- [ ] Built-in proxy and Tor routing in all SDKs and clients
|
||||
- [ ] Automated rotation of API keys and tokens
|
||||
- [ ] Rate limiting and circuit rotation to prevent fingerprinting
|
||||
- [ ] Define core protocol interfaces and data models in each language
|
||||
- [ ] Maintain open-source SDKs and reference servers
|
||||
- [ ] Ensure feature parity and cross-language test suites
|
||||
- [ ] Document integration patterns and privacy best practices
|
||||
|
||||
Reference in New Issue
Block a user