@@ -0,0 +1,391 @@
# Negentropy (NIP-77) Implementation Plan — nostr_core_lib
## Goal
Implement [NIP-77 ](https://github.com/nostr-protocol/nips/blob/master/77.md )
Negentropy / **Range-Based Set Reconciliation (RBSR) ** as a reusable C module
inside [`nostr_core_lib` ](../nostr_core ), so that both
[`c-relay-pg` ](../../c-relay-pg ) (server side) and its
[caching daemon ](../../c-relay-pg/caching ) (client side) can use it for
bandwidth-optimal, bidirectional relay-to-relay event sync.
This plan covers **only the work that lives in `nostr_core_lib` ** : the RBSR
algorithm, the NIP-77 wire framing, and unit tests. The downstream relay and
caching-daemon integration is tracked separately in
[`c-relay-pg/plans/relay_to_relay_sync_plan.md` ](../../c-relay-pg/plans/relay_to_relay_sync_plan.md )
(Phases 2–
---
## Background
### What RBSR does
Two parties each hold a set of items (here: Nostr event IDs). They want to
learn * exactly * which IDs the other side has that they lack, and vice versa,
without transferring the full sets. RBSR does this by:
1. Sorting items into a sequence ordered by `(timestamp, id)` .
2. Recursively splitting the range tree and comparing small **fingerprints **
(a 64-bit hash of the IDs in a subrange) at each split.
3. When fingerprints match → both sides have the same IDs in that subrange,
skip it. When they differ → recurse / drill down until the differing
individual IDs are isolated.
4. Output: two lists — `have` (IDs the peer has that we lack) and
`need` (IDs we have that the peer lacks).
When sets overlap ~99%, only a few KB of fingerprints are exchanged instead of
megabytes of `REQ` paging. See the reference implementation
[hoytech/negentropy ](https://github.com/hoytech/negentropy ) (C++) and the C port
[mcl/negentropy-c ](https://github.com/mcl/negentropy-c ).
### NIP-77 wire framing
NIP-77 wraps RBSR messages over a Nostr WebSocket:
| Message | Direction | Purpose |
|-------------|------------------|------------------------------------------------------|
| `NEG-OPEN` | client → relay | open a reconciliation session: subscription id, NIP-01 filter, hex `initialMessage` (first RBSR frame) |
| `NEG-MSG` | both directions | carry subsequent RBSR frames (hex-encoded payload) |
| `NEG-CLOSE` | client → relay | terminate the session |
| `NEG-ERR` | relay → client | error (e.g. set too large, malformed) |
The relay builds its negentropy set from `events` matching the filter, runs the
RBSR state machine against the client's frames, and replies with `NEG-MSG`
until convergence.
---
## Design Decisions
### D1: Port, don't vendor as a submodule
**Decision: ** Port the RBSR algorithm directly into `nostr_core_lib` as a new
NIP module (`nip077` ), rather than adding `mcl/negentropy-c` as a git
submodule.
**Rationale: **
- The library already follows a "port the algorithm, no extra runtime deps"
pattern (see [`nip013.c` ](../nostr_core/nip013.c ), [`nip044.c` ](../nostr_core/nip044.c )).
- The RBSR core is ~600 lines of C++ in the reference; a C port is tractable
and keeps the static MUSL build self-contained (the relay's
[`build_static.sh` ](../../c-relay-pg/build_static.sh ) must not gain a new
submodule link step).
- Avoids submodule drift / version skew between `nostr_core_lib` and an
external repo.
**Caveat: ** We * study * `mcl/negentropy-c` and `hoytech/negentropy` for
correctness and test vectors, and credit them in the header. If the port turns
out to be risky, the fallback is to vendor `mcl/negentropy-c` under
`nostr_core/negentropy/` and wrap it — see the fallback note in Phase 1.
### D2: Record model = `(created_at, 32-byte id)`
RBSR items are `(timestamp, id)` pairs:
- `timestamp` → `events.created_at` (BIGINT → `uint64_t` ).
- `id` → 32-byte raw event ID (hex-decoded from `events.id` ).
The fingerprint is computed over the sorted `(timestamp || id)` byte sequence.
This matches the reference implementation's `Record` type exactly.
### D3: No DB code in the library
The library module is **pure algorithm + wire framing ** . It operates on
in-memory arrays of `(timestamp, id)` records and byte buffers. The caller
(the relay or caching daemon) is responsible for:
- Running the `SELECT id, created_at FROM events WHERE <filter>` query.
- Populating the record array.
- Sending/receiving the `NEG-*` WebSocket frames.
This keeps the module portable (no libpq dependency in `nostr_core_lib` ) and
testable in isolation.
### D4: Fingerprint = 64-bit, frame size configurable
Match the reference: 8-byte fingerprints, 16-byte frame size (min/max range
boundaries per level). Make frame size a parameter so callers can trade
bandwidth for round-trips.
### D5: Memory ownership
The library owns the RBSR state machine (`negentropy_t` ) and the record array
it's given a reference to (borrowed, not copied — the caller owns the query
result lifetime). Output `have` /`need` ID lists are allocated by the library
and freed by the caller via a provided `negentropy_id_list_free()` .
---
## File Layout
All new files under [`nostr_core_lib/nostr_core/` ](../nostr_core ):
```
nostr_core/
├── nip077.h # Public API: RBSR state machine + NIP-77 framing
├── nip077.c # Implementation
├── negentropy.h # Internal: RBSR core (records, ranges, fingerprints)
├── negentropy.c # Internal: RBSR core implementation
└── negentropy_msg.h # Internal: varint + message encode/decode (optional split)
```
Tests:
```
tests/
└── nip077_test.c # Round-trip + known-vector reconciliation tests
```
---
## Public API Sketch (`nip077.h`)
``` c
# ifndef NIP077_H
# define NIP077_H
# include <stdint.h>
# include <stddef.h>
/* A single record in the reconcilable set. */
typedef struct {
uint64_t created_at ; /* events.created_at */
uint8_t id [ 32 ] ; /* raw 32-byte event id */
} negentropy_record_t ;
/* Opaque RBSR state machine. */
typedef struct negentropy_ctx negentropy_ctx_t ;
/* Output of a reconciliation step: the IDs each side is missing. */
typedef struct {
uint8_t ( * have ) [ 32 ] ; /* IDs the peer has that we lack */
size_t have_count ;
uint8_t ( * need ) [ 32 ] ; /* IDs we have that the peer lacks */
size_t need_count ;
} negentropy_reconcile_result_t ;
/* Create a context over a sorted record array (borrowed, not copied).
* records must remain valid for the lifetime of the ctx.
* Returns NULL on error. */
negentropy_ctx_t * negentropy_new ( const negentropy_record_t * records ,
size_t record_count ,
uint64_t frame_size_bits /* default 16 */ ) ;
void negentropy_free ( negentropy_ctx_t * ctx ) ;
/* Produce the initial RBSR message (client side sends this in NEG-OPEN).
* Writes a hex string into out_hex (caller-allocated, capacity out_hex_cap).
* Returns bytes written, or -1 on error. */
int negentropy_initiate ( const negentropy_ctx_t * ctx ,
char * out_hex , size_t out_hex_cap ) ;
/* Feed an incoming RBSR message (hex) and produce the next outbound message.
* If the protocol has converged, returns 0 and sets *converged=1 (out_hex
* may be empty). Otherwise returns bytes written and *converged=0.
* On error returns -1. */
int negentropy_reconcile ( negentropy_ctx_t * ctx ,
const char * in_hex ,
char * out_hex , size_t out_hex_cap ,
int * converged ,
negentropy_reconcile_result_t * result /* optional */ ) ;
void negentropy_result_free ( negentropy_reconcile_result_t * r ) ;
/* ---- NIP-77 wire framing helpers (JSON array parse/build) ---- */
/* Parse a NEG-OPEN frame:
* ["NEG-OPEN", subscriptionId, filterJson, initialMessageHex]
* Returns 0 on success. Caller frees *subscription_id and *filter_json. */
int nip77_parse_neg_open ( const char * frame_json ,
char * * subscription_id ,
char * * filter_json ,
char * * initial_message_hex ) ;
/* Build a NEG-MSG frame:
* ["NEG-MSG", subscriptionId, payloadHex]
* into out (caller-allocated). */
int nip77_build_neg_msg ( const char * subscription_id ,
const char * payload_hex ,
char * out , size_t out_cap ) ;
/* Build a NEG-ERR frame:
* ["NEG-ERR", subscriptionId, reason] */
int nip77_build_neg_err ( const char * subscription_id ,
const char * reason ,
char * out , size_t out_cap ) ;
# endif /* NIP077_H */
```
---
## Implementation Todo List
### Phase 1: RBSR Core (`negentropy.c` / `.h`)
- [ ] **1.1 ** Port the data structures from `hoytech/negentropy` :
`Record` (timestamp + id), `Range` (lower/upper bound + mode:
`bounded` /`unbounded` /`fingerprint` ), `Fingerprint` (8 bytes).
Use `uint64_t` timestamps and fixed 32-byte ids.
- [ ] **1.2 ** Implement **varint ** encode/decode (LEB128-style as in the
reference). Unit-test edge cases (0, 127, 128, 2^32, 2^64).
- [ ] **1.3 ** Implement **fingerprint ** computation: a 64-bit hash over the
concatenated `(timestamp || id)` bytes of all records in a range. Use
SipHash-2-4 (or the reference's `Fingerprint` algorithm) — match the
reference exactly so frames interoperate with other NIP-77 clients.
* Decision needed: * confirm the exact hash the reference uses (it's a
custom 64-bit accumulator, not SipHash) and replicate it byte-for-byte.
- [ ] **1.4 ** Implement **set construction ** : take a sorted `(timestamp, id)`
array and build the initial range tree / flat sorted view the algorithm
operates on. Records **must ** be sorted by `(created_at, id)` — document
this contract and provide a `negentropy_sort_records()` helper using
`qsort` .
- [ ] **1.5 ** Implement **message encode ** : serialize the current protocol
state into a frame (varint-prefixed ranges with bounds + fingerprints).
Match the reference's `encodeBound` /`encodeFingerprint` byte layout.
- [ ] **1.6 ** Implement **message decode ** : parse an incoming frame into a
list of peer ranges. Validate varint bounds and reject malformed frames
(return -1, never crash).
- [ ] **1.7 ** Implement **reconcile step ** (the core loop):
- Walk our ranges vs the peer's ranges.
- Where fingerprints match → skip (both sides equal).
- Where they differ → split into sub-ranges and emit a response frame
drilling down. Use the `frameSizeBits` parameter to decide when to
stop splitting and instead request the full ID list for a small range.
- When a range is small enough, emit `have/need` IDs directly.
- Track `converged` state: no more differing ranges to investigate.
- [ ] **1.8 ** Implement `negentropy_initiate()` — the first frame a client
sends (a single unbounded range with the whole-set fingerprint).
- [ ] **1.9 ** Memory: all intermediate range lists are freed at the end of
each `reconcile` call; the ctx holds only the borrowed record array +
config. No leaks across round-trips.
### Phase 2: NIP-77 Wire Framing (`nip077.c` / `.h`)
- [ ] **2.1 ** Implement `nip77_parse_neg_open()` using
[`cJSON` ](../cjson/cJSON.c ): validate the 4-element array, extract
`subscriptionId` (string), `filter` (object, returned as a string for
the caller to re-parse with the relay's existing filter logic), and
`initialMessage` (hex string).
- [ ] **2.2 ** Implement `nip77_build_neg_msg()` and `nip77_build_neg_err()`
as simple cJSON array serializations.
- [ ] **2.3 ** Add hex encode/decode helpers (32-byte id ↔ 64-char hex) —
reuse [`utils.c` ](../nostr_core/utils.c ) if a hex helper already
exists there; otherwise add `negentropy_hex_encode/decode` locally.
- [ ] **2.4 ** Wire the RBSR core to the framing: a helper that takes a
parsed `NEG-OPEN` (filter + initialMessage), builds a
`negentropy_ctx_t` from a caller-provided record array, runs
`negentropy_reconcile()` , and returns the `NEG-MSG` /`NEG-ERR` response
string. This is the convenience function the relay's
[`websockets.c` ](../../c-relay-pg/src/websockets.c ) handler will call.
### Phase 3: Build System Integration
- [ ] **3.1 ** Add `nostr_core/nip077.c` and `nostr_core/negentropy.c` to the
`SOURCES` list in [`build.sh` ](../build.sh ) (the auto-detect loop at
~line 521 and the `--nips=all` list at ~line 201).
- [ ] **3.2 ** Add the same two files to the `add_library(nostr_core STATIC ...)`
list in [`CMakeLists.txt` ](../CMakeLists.txt ) (~line 40).
- [ ] **3.3 ** Add `077` to the `NIP_DESCRIPTIONS` case statement in
`build.sh` (~line 526): `077) NIP_DESCRIPTIONS="$NIP_DESCRIPTIONS NIP-077(Negentropy)" ;;`
- [ ] **3.4 ** Add `NIP-077(Negentropy)` to the NIP status table in
[`README.md` ](../README.md ) and [`nostr_core.h` ](../nostr_core/nostr_core.h )
quick-reference section.
- [ ] **3.5 ** Bump [`VERSION` ](../VERSION ) per the library's
[`increment_and_push.sh` ](../increment_and_push.sh ) convention once
Phase 1–
### Phase 4: Unit Tests (`tests/nip077_test.c`)
Follow the existing test style ([`nip13_test.c` ](tests/nip13_test.c )):
`print_test_header` / `print_test_result` , expected-vs-actual, exit code =
fail count.
- [ ] **4.1 ** Varint round-trip: encode/decode 0, 1, 127, 128, 300,
0xFFFFFFFF, 0xFFFFFFFFFFFFFFFF.
- [ ] **4.2 ** Fingerprint determinism: same input → same 8 bytes; different
input → different 8 bytes (with high probability).
- [ ] **4.3 ** Hex encode/decode round-trip for 32-byte ids.
- [ ] **4.4 ** **Known reconciliation vector ** (the critical test): two
contexts A and B with known record sets (use the NIP-77 appendix
example or a vector lifted from `hoytech/negentropy` 's test suite).
Run `initiate` → `reconcile` → ... → `converged` , and assert:
- A's `need` == IDs B has that A lacks.
- B's `need` == IDs A has that B lacks.
- Both `have` /`need` lists match the expected sets exactly.
- [ ] **4.5 ** **Identical sets ** : A and B hold the same 1000 records →
converge in one round-trip, empty `have` /`need` , minimal bytes.
- [ ] **4.6 ** **Disjoint sets ** : A and B share zero records → converge,
full ID lists exchanged.
- [ ] **4.7 ** **Large overlap ** : A and B share 9990/10000 records → assert
total bytes transferred is small (< a few KB), and the 10 differing
IDs are correctly reconciled. This is the headline bandwidth test.
- [ ] **4.8 ** **Malformed input ** : feed truncated/garbage hex to
`negentropy_reconcile` → returns -1, no crash.
- [ ] **4.9 ** **NIP-77 framing ** : parse a known `NEG-OPEN` JSON, build a
`NEG-MSG` , round-trip through cJSON, assert byte-exact output.
- [ ] **4.10 ** Register the test in the test build loop in `build.sh`
(auto-discovered via `find tests/ -maxdepth 1 -name "*.c"` ).
### Phase 5: Downstream Integration Hooks (reference only — implemented in c-relay-pg)
These are **not ** done in `nostr_core_lib` ; they're listed here so the API
contract is clear. Tracked in
[`relay_to_relay_sync_plan.md` ](../../c-relay-pg/plans/relay_to_relay_sync_plan.md )
Phases 2–
- [ ] **5.1 (relay) ** `NEG-OPEN` /`NEG-MSG` /`NEG-CLOSE` dispatch in
[`websockets.c` ](../../c-relay-pg/src/websockets.c ): build record
array from `SELECT id, created_at FROM events WHERE <filter>` via
[`db_ops_postgres.h` ](../../c-relay-pg/src/db_ops_postgres.h ), call
the Phase 2.4 helper, send `NEG-MSG` /`NEG-ERR` .
- [ ] **5.2 (relay) ** Per-session negentropy state on the `pss` struct,
`negentropy_enabled` + `negentropy_max_records` config keys in the
[`config` ](../../c-relay-pg/src/pg_schema.sql ) table.
- [ ] **5.3 (caching) ** `caching/src/negentropy_sync.c` : client-side
reconciliation loop against peer relays, replacing the
`since` -cursor backfill in [`backfill.c` ](../../c-relay-pg/caching/src/backfill.c )
for negentropy-capable peers.
- [ ] **5.4 (both) ** Integration test: two local relays converge to the
same event set; bandwidth comparison vs old backfill.
---
## Open Questions
1. **Exact fingerprint algorithm. ** The reference uses a custom 64-bit
accumulator, not a standard SipHash. Must confirm the exact byte layout
from `hoytech/negentropy` `Fingerprint` implementation so our frames
interoperate with `strfry` , `nak` , and other NIP-77 clients. **Blocker
for Phase 1.3.**
2. **Frame size default. ** Reference defaults to 16 bits (65536-way split).
Confirm this is a good default for relay-scale sets (100k–
or whether we should default larger (e.g. 20 bits) to reduce round-trips
on big relays.
3. **Max-records policy. ** Should the library enforce a hard cap on record
array size, or leave that entirely to the caller (relay config
`negentropy_max_records` )? Current design: caller enforces — library is
unbounded. Revisit if we want a built-in safety valve.
4. **Streaming / incremental set loading. ** For very large relays, building
the full sorted record array in memory may be heavy (10M records ×
bytes ≈ 400MB). Do we need a streaming/cursor-based variant that loads
records on demand from the DB cursor during reconcile? Defer to a
follow-up; the initial port assumes an in-memory array.
5. **NIP-77 version negotiation. ** NIP-77 has a `version` field in
`NEG-OPEN` (0 = negentropy, 1 = negentropy with timestamps). We need to
support both, or pick one. Reference relays support version 1. **Decide
before Phase 2.1.**
---
## References
- [NIP-77 ](https://github.com/nostr-protocol/nips/blob/master/77.md )
- [hoytech/negentropy ](https://github.com/hoytech/negentropy ) (C++ reference)
- [mcl/negentropy-c ](https://github.com/mcl/negentropy-c ) (C port, study
reference)
- [`c-relay-pg/plans/relay_to_relay_sync_plan.md` ](../../c-relay-pg/plans/relay_to_relay_sync_plan.md )
(downstream integration plan, Option C)
- Existing NIP module conventions: [`nip013.c` ](../nostr_core/nip013.c ),
[`nip044.c` ](../nostr_core/nip044.c )
- Build system: [`build.sh` ](../build.sh ), [`CMakeLists.txt` ](../CMakeLists.txt )
- Test conventions: [`tests/nip13_test.c` ](tests/nip13_test.c )
Laan Tungir