sovereign_browser/plans/download-website-offline.md

Plan: Download Website for Offline Viewing

Goal

Add a "Download Website" option to the tab right-click context menu. It crawls the current site (same-origin, bounded by a page cap), saves all pages and assets to a per-profile directory, rewrites links to relative paths, and opens the offline copy in a new tab via file://.

User decisions

• Depth: Entire site — follow all same-origin links, no depth limit, bounded by a max-page-count safety cap (default 500).
• Storage: Per-profile: ~/.sovereign_browser/profiles/<pubkey>/offline_sites/<hostname>/ (falls back to ~/.sovereign_browser/offline_sites/<hostname>/ when not logged in).
• Feedback: Small modal progress dialog with pages/files counts, a progress bar, and a Cancel button.

Trigger

Right-click on a tab → "Download Website" menu item. Added to both tab context menus in src/tab_manager.c:

• on_tab_label_button_press menu (line ~828)
• on_notebook_button_press menu (line ~751)

The callback reads the right-clicked tab's current_url and is only active for http:// / https:// URLs (greyed out / error dialog otherwise).

Architecture

flowchart TD
    A[Right-click tab] --> B[Download Website menu item]
    B --> C{URL is http/https?}
    C -- No --> D[Show error, abort]
    C -- Yes --> E[Show progress dialog]
    E --> F[Start worker thread BFS crawl]
    F --> G[Fetch page via libsoup]
    G --> H[Save HTML to offline_sites/host/path]
    H --> I[Extract same-origin links + assets]
    I --> J{Cancel clicked?}
    J -- Yes --> K[Stop crawl]
    J -- No --> L{More URLs and under cap?}
    L -- Yes --> G
    L -- No --> M[Close dialog]
    K --> M
    M --> N[Open file://offline_sites/host/index.html in new tab]

New files

src/site_downloader.h

Public API (self-contained, called from the tab context menu):

/* Start downloading the website at start_url. Shows a modal progress
 * dialog parented to `parent`. On completion (or cancel) opens the
 * offline copy in a new tab. No-op for non-http(s) URLs. */
void site_downloader_start(const char *start_url, GtkWindow *parent);

src/site_downloader.c

Implementation:

1. Progress dialog — GtkDialog with:

   • GtkLabel showing hostname + "Pages: N / Files: M"
   • GtkProgressBar (pulse mode; or fraction = pages / cap)
   • Cancel GtkButton → sets g_atomic_int_set(&cancel, 1)

2. Worker thread — g_thread_new("site-dl", ...) running a BFS crawl:

   • SoupSession (libsoup-3.0; pattern from src/agent_llm.c and src/search.c).
   • Queue (GQueue) of URLs to fetch + a GHashTable visited-set.
   • Politeness delay between requests (default 100ms).

3. Fetch + save — for each URL:

   • soup_session_send_and_read() (sync; we're on a worker thread).
   • Map URL → local path: mirror the URL path under the site root; directory-style URLs → index.html; sanitize illegal filename chars.
   • Write bytes to disk (g_file_set_contents for text, manual write for binary).
   • For HTML responses: extract links, rewrite them to relative paths, then save the rewritten HTML.

4. Link extraction & rewriting (no HTML parser dep in the project, so use regex over href=, src=, srcset=, poster=, and CSS url(...)):

   • Resolve each link against the page URL (g_uri_resolve_relative).
   • Same-origin filter: keep only links whose host matches the start URL's host (via g_uri_parse, pattern from src/tor_scheme.c). Enqueue same-origin HTML pages; download all assets (CSS/JS/images/fonts/etc.) regardless of origin but don't crawl cross-origin pages.
   • Rewrite kept links to relative file paths so the offline copy works from file://.

5. Completion — g_idle_add a callback on the main thread that:

   • Destroys the progress dialog.
   • Opens file://<offline_dir>/index.html in a new tab via tab_manager_new_tab().

Modified files

• src/tab_manager.c — add "Download Website" item to both context menus; new callback on_tab_download_website calling site_downloader_start(tab->current_url, g_window).
• Makefile — add src/site_downloader.c to SRC.
• src/settings.h / src/settings.c — add to browser_settings_t:
  • offline_site_max_pages (default 500)
  • offline_site_request_delay_ms (default 100)
  • load/save in settings_load() / settings_save().

Edge cases

• Non-http(s) URLs → menu item shows an error dialog, no crawl.
• Not logged in → fall back to global ~/.sovereign_browser/offline_sites/.
• Redirects → follow libsoup redirects; use the final URL for same-origin checks and path mapping.
• Binary assets → write raw bytes, never parse/rewrite.
• Duplicate URLs → visited-set prevents re-fetching.
• Very large sites → max-page cap stops the crawl; dialog shows "cap reached".
• Filename sanitization → replace illegal chars (/, ?, :, etc.) with _.
• Cancel → worker checks the cancel flag between requests; partial downloads are kept and the new tab opens to whatever was fetched.

Implementation order

1. Add settings fields + load/save.
2. Create site_downloader.h / site_downloader.c skeleton: progress dialog
   • thread stub + completion callback.
3. Implement libsoup fetch + file write + URL→path mapping.
4. Implement HTML link extraction, same-origin filter, link rewriting.
5. Implement BFS crawl loop: visited-set, cap, cancel flag, progress updates.
6. Implement completion callback: open new tab with file:// URL.
7. Wire "Download Website" into both tab context menus in tab_manager.c.
8. Add site_downloader.c to Makefile.
9. Build (make) and test with tests/local-site/ served via python3 -m http.server.

Testing

• Serve tests/local-site/ with python3 -m http.server 8000 from the tests/ directory.
• Open http://localhost:8000/local-site/index.html in the browser.
• Right-click the tab → Download Website.
• Verify the progress dialog appears, files are saved under ~/.sovereign_browser/profiles/<pubkey>/offline_sites/localhost/, and a new tab opens the offline index.html with working relative links and assets.
• Test Cancel mid-download.
• Test a non-http URL (e.g. sovereign://profile) → error dialog.