laantungir/fips
1
0
Fork 0
mirror of https://github.com/jmcorgan/fips.git synced 2026-07-30 19:46:15 +00:00
Code Issues Packages Projects Releases Wiki Activity

Redesign bloom filter parameters with mathematical justification

Browse Source 
Analysis revealed original parameters (4KB, k=7) were oversized:
- Expected filter occupancy ~250-800 entries, not ~4,096
- Original FPR estimates in docs were incorrect (5-7x optimistic)
- d^(2K) formula overcounted by assuming mesh vs tree structure

New v1 parameters:
- Filter size: 1 KB (was 4 KB) - 75% bandwidth reduction
- Hash functions: k=5 (was 7) - optimal for 800-1600 entries
- K-hop scope: 2 (unchanged)

Added forward compatibility via size_class field:
- Power-of-2 sizes (512B, 1KB, 2KB, 4KB) enable folding
- v1 requires size_class=1; future versions can negotiate larger
- Receivers can fold larger filters down to preferred size

Updated docs:
- fips-routing.md: Part 1 rewritten with math foundation
- fips-gossip-protocol.md: §3 and Appendix A.2 wire format
- fips-architecture.md: Configuration parameters
This commit is contained in:
Johnathan Corgan
2026-02-02 14:41:28 +00:00
parent 021bf69d73
commit 7d5d1adedc
3 changed files with 168 additions and 36 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/design/fips-architecture.md
+5 -2
View File
@@ -906,13 +906,16 @@ the limit but need not enforce it strictly.
| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `filter.size` | u32 | 32768 | Filter size in bits (4KB) |
| `filter.hash_count` | u8 | 7 | Number of hash functions |
| `filter.size_class` | u8 | 1 | Size class: 0=512B, 1=1KB, 2=2KB, 3=4KB |
| `filter.hash_count` | u8 | 5 | Number of hash functions |
| `filter.scope` | u8 | 2 | TTL for filter propagation (K) |
| `filter.refresh.interval` | duration | 60s | Periodic FilterAnnounce refresh |
| `filter.update.debounce` | duration | 500ms | Min interval between updates |
| `filter.stale.threshold` | duration | 300s | Consider peer's filter stale |
v1 protocol requires `size_class=1` (1 KB filters). The size_class field is
present in the wire format for forward compatibility with larger filters.
### Discovery Protocol
| Parameter | Type | Default | Description |
docs/design/fips-gossip-protocol.md
+66 -22
View File
@@ -135,27 +135,57 @@ node's filter indicates which destinations are reachable through it.
```text
FilterAnnounce {
filter: BloomFilter, // 4096 bytes (32,768 bits)
ttl: u8, // Remaining propagation hops
sequence: u64, // For freshness/deduplication
ttl: u8, // Remaining propagation hops
filter: BloomFilter, // Variable size based on size_class
}
BloomFilter {
bits: [u8; 4096], // Bit array
hash_count: u8, // Number of hash functions (typically 7)
hash_count: u8, // Number of hash functions (5 for v1)
size_class: u8, // Filter size: bytes = 512 << size_class
bits: [u8; 512 << size_class], // Bit array (1024 bytes for v1)
}
```
### 3.2 Field Semantics
### 3.2 Size Classes
**filter**: Contains node_ids reachable through this peer. Uses k=7 hash
functions for near-optimal false positive rate at expected fill levels.
Filter sizes are powers of 2 to enable folding (shrinking by ORing halves):
| size_class | Bits | Bytes | Status |
|------------|--------|-------|---------------------|
| 0 | 4,096 | 512 | Reserved (future) |
| 1 | 8,192 | 1,024 | **v1 default** |
| 2 | 16,384 | 2,048 | Reserved (future) |
| 3 | 32,768 | 4,096 | Reserved (future) |
**v1 protocol**: All nodes MUST use size_class=1 (1 KB filters). Nodes MUST
reject FilterAnnounce with size_class ≠ 1.
**Future versions**: Nodes may negotiate larger filters via capability exchange.
Receivers can fold larger filters down to their preferred size.
### 3.3 Field Semantics
**sequence**: Monotonic counter for this node's filter. Allows receivers to
detect stale or duplicate announcements.
**ttl**: Remaining propagation depth. Starts at K (typically 2), decremented
each hop. At TTL=0, entries are not propagated further.
**sequence**: Monotonic counter for this node's filter. Allows receivers to
detect stale or duplicate announcements.
**hash_count**: Number of hash functions used. v1 uses k=5, which is optimal
for 800-1,600 entries in a 1 KB filter.
**size_class**: Indicates filter size as `512 << size_class` bytes. Allows
forward-compatible extension to larger filters.
**bits**: The Bloom filter bit array. To test membership:
```text
for i in 0..hash_count:
bit_index = hash(node_id, i) % (8 * bits.len())
if !bits[bit_index]: return false
return true // "maybe present"
```
### 3.3 Filter Contents
@@ -486,43 +516,57 @@ Propagates Bloom filter reachability information.
│ │ 0 │ msg_type │ 1 byte │ 0x11 │ │
│ │ 1 │ sequence │ 8 bytes │ u64 LE, monotonic counter │ │
│ │ 9 │ ttl │ 1 byte │ Remaining propagation hops │ │
│ │ 10 │ hash_count │ 1 byte │ Number of hash functions (7) │ │
│ │ 11 │ filter_bits │ 4096 bytes│ Bloom filter bit array │ │
│ │ 10 │ hash_count │ 1 byte │ Number of hash functions (5) │ │
│ │ 11 │ size_class │ 1 byte │ Filter size: 512 << class │ │
│ │ 12 │ filter_bits │ variable │ 512 << size_class bytes │ │
│ └────────┴──────────────────┴───────────┴───────────────────────────────┘ │
│ │
│ Total payload: 4107 bytes (fixed size) │
│ With link overhead: 4136 bytes │
│ Size classes (powers of 2 for foldability): │
│ 0 = 512 bytes (4,096 bits) - Reserved for future │
│ 1 = 1,024 bytes (8,192 bits) - v1 default │
│ 2 = 2,048 bytes (16,384 bits) - Reserved for future │
│ 3 = 4,096 bytes (32,768 bits) - Reserved for future │
│ │
│ v1 total payload: 1 + 8 + 1 + 1 + 1 + 1024 = 1036 bytes │
│ With link overhead: 1065 bytes │
│ │
├─────────────────────────────────────────────────────────────────────────────┤
│ BLOOM FILTER STRUCTURE │
├─────────────────────────────────────────────────────────────────────────────┤
│ │
│ filter_bits[4096]: │
│ filter_bits[1024] (v1, size_class=1): │
│ ┌─────────────────────────────────────────────────────────────────────┐ │
│ │ Byte 0 │ Byte 1 │ ... │ Byte 4095 │ │
│ │ bits 0-7 │ bits 8-15 │ │ bits 32760-32767 │ │
│ │ Byte 0 │ Byte 1 │ ... │ Byte 1023 │ │
│ │ bits 0-7 │ bits 8-15 │ │ bits 8184-8191 │ │
│ └─────────────────────────────────────────────────────────────────────┘ │
│ │
│ To test membership of node_id: │
│ filter_bits = 8 * (512 << size_class) // 8192 for v1 │
│ for i in 0..hash_count: │
│ bit_index = hash(node_id, i) % 32768 │
│ if !filter_bits[bit_index]: return false │
│ bit_index = hash(node_id, i) % filter_bits │
│ if !bits[bit_index]: return false │
│ return true // "maybe present" │
│ │
│ Folding (for future heterogeneous sizes): │
│ To shrink a filter by half, OR its two halves: │
│ small[i] = large[i] | large[i + small.len()] │
│ This increases FPR but preserves correctness (no false negatives). │
│ │
└─────────────────────────────────────────────────────────────────────────────┘
```
**Concrete example**:
**Concrete example** (v1 with size_class=1):
```text
PLAINTEXT BYTES:
11 ← msg_type = FilterAnnounce
2A 00 00 00 00 00 00 00 ← sequence = 42
02 ← ttl = 2 (will propagate 2 more hops)
07 ← hash_count = 7
[4096 bytes of filter bits] ← Bloom filter
05 ← hash_count = 5
01 ← size_class = 1 (1 KB filter)
[1024 bytes of filter bits] ← Bloom filter
Total: 4107 bytes
Total: 1036 bytes
```
### A.3 LookupRequest (0x12)
docs/design/fips-routing.md
+97 -12
View File
@@ -66,20 +66,105 @@ traffic tunnels through that peer.
| Parameter | Value | Rationale |
|-----------|-------|-----------|
| Filter size | 4 KB (32,768 bits) | Balances accuracy vs. memory |
| Hash functions | 7 | Near-optimal for expected fill ratio |
| Filter size | 1 KB (8,192 bits) | Sized for expected occupancy with margin |
| Hash functions | 5 | Optimal for 800-1,600 entries at this size |
| Scope (K) | 2 | Effective ~4-hop reach with TTL propagation |
### False Positive Rates
### Mathematical Foundation
| Nodes in Filter | FPR |
|-----------------|-----|
| 1,000 | ~0.05% |
| 2,000 | ~0.5% |
| 5,000 | ~1.3% |
| 10,000 | ~8% |
**False Positive Rate (FPR):**
With K=2 and average degree d=8, expected nodes in scope ≈ d^(2K) ≈ 4,096.
```text
FPR = (1 - e^(-kn/m))^k
```
Where m = bits, n = entries, k = hash functions.
**Optimal hash count:**
```text
k_opt = (m/n) × ln(2) ≈ 0.693 × (m/n)
```
For m=8,192 and expected n=800: k_opt ≈ 7. We use k=5 to accommodate higher
occupancy scenarios (up to ~1,600 entries) while maintaining acceptable FPR.
**Required bits for target FPR:**
```text
m = -1.44 × n × ln(p)
```
For 1% FPR: m ≈ 9.6n bits. For 5% FPR: m ≈ 6.2n bits.
### Expected Filter Occupancy
Filter occupancy depends on K-hop scope and node degree, **not** total network
size. The TTL mechanism bounds entries regardless of network scale.
**Nodes within h hops in a tree (branching factor b = d-1):**
```text
nodes_within_h_hops = (b^(h+1) - 1) / (b - 1)
```
For d=8 (b=7), K=2: each peer's 2-hop neighborhood ≈ 57 nodes.
**Outgoing filter to peer Q contains:**
- Self (1 entry)
- Entries from (d-1) other peers' filters, with overlap
**Expected occupancy by node degree:**
| Degree (d) | Expected Entries | Notes |
|------------|------------------|-------|
| 5 | 100-200 | Constrained/IoT |
| 8 | 250-400 | Typical node |
| 12 | 500-800 | Well-connected |
| 20+ | 1,200-1,800 | Hub node |
### False Positive Rates (1 KB filter, k=5)
| Entries | FPR | Scenario |
|---------|-----|----------|
| 200 | 0.02% | Low-degree node |
| 400 | 0.3% | Typical node |
| 800 | 2.4% | Well-connected |
| 1,200 | 7.5% | Hub node |
| 1,600 | 15% | Heavily loaded hub |
FPR above 5% triggers more LookupRequests but the discovery protocol handles
this gracefully. Hub nodes may benefit from larger filters in future protocol
versions (see §1.6).
### Size Classes (Forward Compatibility)
Filter sizes are powers of 2 to enable **folding** — a technique for shrinking
filters by ORing halves:
```rust
fn fold(filter: &[u8]) -> Vec<u8> {
let half = filter.len() / 2;
(0..half).map(|i| filter[i] | filter[i + half]).collect()
}
```
Folding preserves correctness (no false negatives) but increases FPR.
| size_class | Bits | Bytes | Status |
|------------|------|-------|--------|
| 0 | 4,096 | 512 | Reserved (future) |
| 1 | 8,192 | 1,024 | **Current default** |
| 2 | 16,384 | 2,048 | Reserved (future) |
| 3 | 32,768 | 4,096 | Reserved (future) |
**v1 protocol**: All nodes MUST use size_class=1. The field is present in the
wire format for forward compatibility.
**Future versions**: Nodes may negotiate larger filters. Receivers fold down
to their preferred size if sender's filter is larger. This allows hub nodes
to maintain higher precision while constrained nodes use smaller filters.
### Filter Contents
@@ -470,7 +555,7 @@ enum RouteState {
| Type | Purpose | Size | When Used |
|------|---------|------|-----------|
| FilterAnnounce | Bloom filter propagation | ~4.1 KB | Topology changes |
| FilterAnnounce | Bloom filter propagation | ~1 KB | Topology changes |
| LookupRequest | Discover coordinates | ~300 bytes | First contact with distant node |
| LookupResponse | Return coordinates | ~400 bytes | Reply to discovery |
| SessionSetup | Warm router caches + crypto init | ~400-700 bytes | Before data transfer |
@@ -506,7 +591,7 @@ When nodes join/leave:
| Resource | Full Participant | Leaf-Only |
|----------|------------------|-----------|
| Bloom filter storage | d × 4 KB (d = peer count) | None |
| Bloom filter storage | d × 1 KB (d = peer count) | None |
| Coordinate cache | 10K-100K entries | None |
| Route cache | 1K-10K entries | Minimal |
| Bandwidth (idle) | < 1 KB/sec | Near zero |