# Event Tags Denormalization Plan

## Problem Statement

The relay is at 99% CPU with 1,178 WebSocket connections and ~120 new REQ subscriptions per minute. The root cause is that every tag-filtered REQ query (e.g., `#g`, `#e`, `#p`) uses `json_each(json(tags))` with `json_extract()` — a correlated subquery that parses the JSON tags column for every candidate row. With geohash-based subscriptions from a location app generating constant connection churn, this creates unsustainable CPU load.

### Evidence

- `EXPLAIN QUERY PLAN` shows: `CORRELATED SCALAR SUBQUERY → SCAN json_each VIRTUAL TABLE`
- 8,077 subscription creates/hour but only 442 active at any time (connection churn)
- Only 56 events stored/hour — the load is entirely from **read queries**, not writes
- 14,819 events match the kind filter; each query parses JSON tags on candidate rows

## Solution: Denormalized `event_tags` Table

Replace `json_each()` queries with indexed lookups on a separate `event_tags` table. This is the standard approach used by strfry, nostream, and other production Nostr relays.

### Performance Impact

| Metric | Before | After |
|--------|--------|-------|
| Tag query type | `json_each()` correlated subquery | Indexed JOIN/subquery |
| Complexity per query | O(rows × tags_per_event) JSON parsing | O(log n) B-tree lookup |
| Expected CPU reduction | 99% | <10% for same traffic |

## Files to Modify

| File | Changes |
|------|---------|
| `src/sql_schema.h` | Add `event_tags` table, indexes, cascade triggers; bump to schema v12 |
| `src/main.c` | Add `store_event_tags()`, update `handle_req_message()` tag filter SQL, move config reads out of row loop, add startup tag population |
| `src/nip009.c` | Add `DELETE FROM event_tags` alongside event deletions |
| `src/dm_admin.c` | Downgrade NIP-17 decryption failure log level |
| `src/websockets.c` | Downgrade NIP-17 error log to DEBUG at lines 780 and 1488 |

## Detailed Changes

### 1. Schema: `src/sql_schema.h`

Add the `event_tags` table after the events table definition. Bump schema version to 12.

```sql
-- Denormalized event tags for fast indexed lookups
-- Replaces json_each(json(tags)) queries which cause full JSON parsing per row
CREATE TABLE event_tags (
    event_id TEXT NOT NULL,
    tag_name TEXT NOT NULL,
    tag_value TEXT NOT NULL,
    tag_index INTEGER NOT NULL DEFAULT 0,
    FOREIGN KEY (event_id) REFERENCES events(id) ON DELETE CASCADE
);

-- Primary lookup index: find events by tag name + value
CREATE INDEX idx_event_tags_lookup ON event_tags(tag_name, tag_value);

-- Reverse lookup: find all tags for an event (for cleanup)
CREATE INDEX idx_event_tags_event ON event_tags(event_id);

-- Composite index for common query pattern: tag + kind (via join)
CREATE INDEX idx_event_tags_value_name ON event_tags(tag_value, tag_name);
```

**Key design decisions:**
- `ON DELETE CASCADE` handles cleanup when events are deleted (NIP-09, replaceable events)
- `tag_index` stores the position within the tags array (useful for ordered tag access)
- Only the first two elements of each tag are indexed (`tag_name` = `$[0]`, `tag_value` = `$[1]`) — this covers all standard Nostr tag filters (`#e`, `#p`, `#t`, `#g`, `#d`, etc.)
- `PRAGMA foreign_keys = ON` is already in the schema, so CASCADE will work

### 2. Store Tags: `src/main.c` — New `store_event_tags()` Function

Add a new function called after successful event INSERT in `store_event()`:

```c
// Insert denormalized tags into event_tags table for fast indexed lookups
int store_event_tags(const char* event_id, cJSON* tags) {
    if (!g_db || !event_id || !tags || !cJSON_IsArray(tags)) {
        return 0; // Not an error if no tags
    }

    const char* sql = "INSERT INTO event_tags (event_id, tag_name, tag_value, tag_index) VALUES (?, ?, ?, ?)";
    sqlite3_stmt* stmt;
    int rc = sqlite3_prepare_v2(g_db, sql, -1, &stmt, NULL);
    if (rc != SQLITE_OK) {
        DEBUG_ERROR("Failed to prepare event_tags insert: %s", sqlite3_errmsg(g_db));
        return -1;
    }

    int tag_index = 0;
    cJSON* tag = NULL;
    cJSON_ArrayForEach(tag, tags) {
        if (cJSON_IsArray(tag) && cJSON_GetArraySize(tag) >= 2) {
            cJSON* name = cJSON_GetArrayItem(tag, 0);
            cJSON* value = cJSON_GetArrayItem(tag, 1);

            if (cJSON_IsString(name) && cJSON_IsString(value)) {
                sqlite3_reset(stmt);
                sqlite3_bind_text(stmt, 1, event_id, -1, SQLITE_STATIC);
                sqlite3_bind_text(stmt, 2, cJSON_GetStringValue(name), -1, SQLITE_STATIC);
                sqlite3_bind_text(stmt, 3, cJSON_GetStringValue(value), -1, SQLITE_STATIC);
                sqlite3_bind_int(stmt, 4, tag_index);

                rc = sqlite3_step(stmt);
                if (rc != SQLITE_DONE) {
                    DEBUG_ERROR("Failed to insert event tag: %s", sqlite3_errmsg(g_db));
                }
            }
        }
        tag_index++;
    }

    sqlite3_finalize(stmt);
    return 0;
}
```

**Call site** — in `store_event()` at line ~784, after `monitoring_on_event_stored()`:

```c
// After successful event storage, insert denormalized tags
store_event_tags(cJSON_GetStringValue(id), tags);
```

### 3. Update REQ Query Builder: `src/main.c` — `handle_req_message()`

Replace the `json_each()` tag filter at lines 1243-1270 with an `event_tags` subquery:

**Before** (current code at line 1244):
```c
snprintf(sql_ptr, remaining, 
    " AND EXISTS (SELECT 1 FROM json_each(json(tags)) "
    "WHERE json_extract(value, '$[0]') = ? "
    "AND json_extract(value, '$[1]') IN (");
```

**After** (new code):
```c
snprintf(sql_ptr, remaining,
    " AND id IN (SELECT event_id FROM event_tags "
    "WHERE tag_name = ? AND tag_value IN (");
```

The rest of the parameter binding code (lines 1248-1270) stays the same — the bind params are identical (`tag_name` then `tag_value(s)`). Only the SQL template changes.

The closing parenthesis changes from `"))` to `"))` — same syntax, just different semantics.

### 4. NIP-09 Delete Cascade: `src/nip009.c`

The `ON DELETE CASCADE` foreign key handles this automatically. When `DELETE FROM events WHERE id = ?` executes, SQLite will automatically delete matching rows from `event_tags`. **No code changes needed in nip009.c** as long as `PRAGMA foreign_keys = ON` is set (it already is in the schema).

However, verify the replaceable/addressable event triggers in the schema also cascade properly. The existing triggers at schema lines 97-119 use `DELETE FROM events WHERE ...` which will trigger the CASCADE.

### 5. Performance Fix: Move Config Reads Out of Row Loop

In `handle_req_message()` at lines 1400-1401, `get_config_bool()` is called **inside the `while (sqlite3_step())` loop** — meaning it executes a SQLite query for every row returned. Move these before the loop:

**Before** (inside loop):
```c
while (sqlite3_step(stmt) == SQLITE_ROW) {
    // ...
    int expiration_enabled = get_config_bool("expiration_enabled", 1);  // SQLite query per row!
    int filter_responses = get_config_bool("expiration_filter", 1);     // SQLite query per row!
```

**After** (before loop):
```c
int expiration_enabled = get_config_bool("expiration_enabled", 1);
int filter_responses = get_config_bool("expiration_filter", 1);

while (sqlite3_step(stmt) == SQLITE_ROW) {
    // ... use cached values
```

### 6. Log Level Fixes

#### NIP-17 Decryption Failure — `src/websockets.c` lines 780 and 1488

Change from `DEBUG_ERROR` to `DEBUG_INFO`:
```c
// Before:
DEBUG_ERROR("NIP-17 admin message processing failed");
// After:
DEBUG_INFO("NIP-17 admin message processing failed");
```

This is expected behavior when non-admin gift wraps arrive — not an error condition.

#### Duplicate Event INSERT — `src/main.c` line 741

The `DEBUG_ERROR` at line 741 fires before the CONSTRAINT check at line 746. Suppress it for constraint violations:

```c
// Before (line 738-741):
if (rc != SQLITE_DONE) {
    const char* err_msg = sqlite3_errmsg(g_db);
    int extended_errcode = sqlite3_extended_errcode(g_db);
    DEBUG_ERROR("INSERT failed: rc=%d, extended_errcode=%d, msg=%s", rc, extended_errcode, err_msg);
}

// After:
if (rc != SQLITE_DONE) {
    const char* err_msg = sqlite3_errmsg(g_db);
    int extended_errcode = sqlite3_extended_errcode(g_db);
    if (rc != SQLITE_CONSTRAINT) {
        DEBUG_ERROR("INSERT failed: rc=%d, extended_errcode=%d, msg=%s", rc, extended_errcode, err_msg);
    }
}
```

### 7. Startup Tag Population

Since you said no migration code is needed (fresh deploy), the `event_tags` table will start empty and populate as new events arrive. Existing events won't have tags in the lookup table.

However, to avoid a period where old events don't appear in tag-filtered queries, add a one-time population function that runs at startup:

```c
// Populate event_tags from existing events (run once at startup)
int populate_event_tags_from_existing(void) {
    if (!g_db) return -1;

    // Check if event_tags is already populated
    sqlite3_stmt* check_stmt;
    sqlite3_prepare_v2(g_db, "SELECT COUNT(*) FROM event_tags", -1, &check_stmt, NULL);
    if (sqlite3_step(check_stmt) == SQLITE_ROW && sqlite3_column_int(check_stmt, 0) > 0) {
        sqlite3_finalize(check_stmt);
        DEBUG_INFO("event_tags already populated, skipping");
        return 0;
    }
    sqlite3_finalize(check_stmt);

    DEBUG_INFO("Populating event_tags from existing events...");

    const char* sql = "SELECT id, tags FROM events WHERE tags != '[]'";
    sqlite3_stmt* stmt;
    int rc = sqlite3_prepare_v2(g_db, sql, -1, &stmt, NULL);
    if (rc != SQLITE_OK) return -1;

    // Use a transaction for bulk insert performance
    sqlite3_exec(g_db, "BEGIN TRANSACTION", NULL, NULL, NULL);

    int event_count = 0;
    while (sqlite3_step(stmt) == SQLITE_ROW) {
        const char* event_id = (const char*)sqlite3_column_text(stmt, 0);
        const char* tags_json = (const char*)sqlite3_column_text(stmt, 1);

        if (event_id && tags_json) {
            cJSON* tags = cJSON_Parse(tags_json);
            if (tags) {
                store_event_tags(event_id, tags);
                cJSON_Delete(tags);
                event_count++;
            }
        }
    }

    sqlite3_finalize(stmt);
    sqlite3_exec(g_db, "COMMIT", NULL, NULL, NULL);

    DEBUG_INFO("Populated event_tags for %d events", event_count);
    return 0;
}
```

Call this during startup, after database initialization but before accepting connections.

## Architecture Diagram

```mermaid
flowchart TD
    subgraph "Current: O(n) per query"
        A1["REQ with #g filter"] --> B1["SQL: SELECT ... WHERE EXISTS"]
        B1 --> C1["json_each(json(tags))"]
        C1 --> D1["json_extract per row"]
        D1 --> E1["🔴 Full JSON parse per candidate row"]
    end

    subgraph "New: O(log n) per query"
        A2["REQ with #g filter"] --> B2["SQL: SELECT ... WHERE id IN"]
        B2 --> C2["event_tags table"]
        C2 --> D2["idx_event_tags_lookup"]
        D2 --> E2["🟢 B-tree index lookup"]
    end

    subgraph "Write Path (unchanged speed)"
        F["EVENT arrives"] --> G["store_event()"]
        G --> H["INSERT INTO events"]
        H --> I["store_event_tags()"]
        I --> J["INSERT INTO event_tags\n(one row per tag)"]
    end

    subgraph "Delete Path (automatic)"
        K["NIP-09 DELETE"] --> L["DELETE FROM events"]
        L --> M["ON DELETE CASCADE"]
        M --> N["event_tags rows auto-deleted"]
    end
```

## Testing Strategy

1. Run existing test suite: `tests/run_all_tests.sh`
2. Specifically run NIP tests that use tag filters: `tests/run_nip_tests.sh`
3. Verify tag-filtered REQ queries return correct results
4. Verify NIP-09 deletion cascades to event_tags
5. Verify replaceable event triggers cascade to event_tags
6. Monitor CPU usage after deployment with same traffic pattern

## Risk Assessment

- **Low risk**: The `event_tags` table is additive — it doesn't change the events table structure
- **Low risk**: `ON DELETE CASCADE` is a well-tested SQLite feature
- **Low risk**: The SQL change in `handle_req_message()` is a drop-in replacement (same bind params)
- **Medium risk**: Startup population on a large database could take a few seconds — but with only 55K events, this should complete in under 1 second
- **Write overhead**: Each event INSERT now also inserts ~5 rows into event_tags — negligible at 56 events/hour
