diff --git a/CHANGELOG.md b/CHANGELOG.md index b07afcb..a4fd96d 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -9,6 +9,12 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Breaking changes +- Version 3 performs an automatic, one-way migration of `NGIT_GIT_DATA_PATH` + from complete per-owner and `/prs/` repositories to thin views backed by + shared identifier families. Version 2 does not coordinate reads and writes + through this layout and must not be run against migrated Git data. Rollback + requires restoring the Git and relay-data snapshot taken before the v3 + launch, not only downgrading the binary. - Added `trusted_proxy_cidrs` to the public `Config` struct. Rust consumers that construct `Config` with a struct literal must provide it. - Added `base_path` to the public `Config` struct. Rust consumers that @@ -84,12 +90,14 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 a warning, backup-directory removals are fsynced before their journals are removed, and completed installations whose backups were already removed manually start unchanged. -- Stop treating server-side shallow repositories as valid family storage. - Migration preserves a legacy `shallow` marker on the installed thin view - and retains that backup until integrity repair recovers the complete - reachable closure from accepted clone sources; the family is reported as - unresolved until then, after which the marker is removed and the backup - retires. Client-requested shallow clones remain supported. +- Reconcile server-side shallow repositories left by untagged development + builds between 2026-01-05 (`623cae5`) and 2026-01-12 (`f25eea8`). No tagged + v1 or v2 release shipped the depth-one fallback fetch. The affected + non-happy path fetched missing state or PR data from another listed Git + server when the objects had not arrived by push. V3 preserves existing + shallow clone behavior and the legacy backup, automatically requests the + complete closure from accepted clone sources, and logs any family that + remains shallow without making it less available. - Scope bare log levels to ngit-grasp while keeping dependencies at warnings; explicit tracing filter expressions remain unchanged. - Keep per-event discovery and validation details at debug, retain aggregate diff --git a/docs/explanation/git-family-object-storage.md b/docs/explanation/git-family-object-storage.md index e0f7fb9..b3788b6 100644 --- a/docs/explanation/git-family-object-storage.md +++ b/docs/explanation/git-family-object-storage.md @@ -249,6 +249,42 @@ view whose alternate is the identifier family. Therefore: Mirroring a PR into an owner view becomes a ref update after an object availability check. It no longer copies the object graph. +## V2 to v3 migration boundary + +V3 changes the on-disk meaning of every served Git repository. Owner and +`/prs/` paths become thin views whose objects and write serialization belong to +the identifier family under `.grasp`. Migration happens automatically on the +first v3 launch, but the result is not safe for v2: v2 has no family lock, +retained-root, or shared-object write model. A software rollback therefore +requires restoring the Git and relay-data snapshot taken before v3; changing +only the binary is not supported. + +One pre-release development interval can leave server-side shallow +repositories for v3 to reconcile. State Git-data fallback was introduced by +commit `623cae5` on 2026-01-05 with `git fetch --depth=1`, carried into the +rewritten purgatory sync path, and changed to a full fetch by commit `f25eea8` +on 2026-01-12. The first tagged release was v1.0.0 on 2026-02-26 and already +contained the fix, so no tagged v1 or v2 release shipped the shallow-fetch +behavior. Only operators who deployed an untagged source revision from that +seven-day interval can have repositories created by this bug. + +The affected path was the non-happy-path Git-data fallback. It ran only when a +pending state or PR event named OIDs which had not arrived through an ordinary +push, then fetched those OIDs into an existing local repository from another +announcement or PR clone URL. A repository whose required data arrived through +the normal push path did not invoke this fallback. + +V3 treats a legacy `shallow` marker as compatibility state rather than a reason +to delete or disable the repository. Migration preserves the marker on the thin +view so its shallow clones and current tree continue to work, and retains the +legacy backup. After the listener starts, the ordinary integrity worker +requests the complete closure from other accepted clone servers. On success it +removes the marker; the next v3 launch retires the now-redundant backup. On +failure the existing shallow clone and current tree remain available, the +backup remains, and an `ERROR` log identifies the family with a non-zero +`shallow_views` count. The operator does not need to take a separate action for +the v3 upgrade. + ## Startup migration Migration runs after configuration validation and before purgatory restoration, @@ -302,19 +338,18 @@ for them; they are moved into content-addressed directories beneath migrations preserve different payloads even when their original pack filenames match, while an identical payload converges on the same quarantine path. -A server-side `shallow` marker means the legacy repository is a deliberate -truncation, and GRASP does not treat shallow repositories as valid family -storage. Migration copies the marker onto the installed thin view so the view -keeps serving what the legacy repository served, and its backup is excluded -from retirement until the family holds the complete reachable closure of the -backup's refs. Closure recovery is the ordinary integrity repair: the family -reports the truncated parents as missing objects and the marked views as +A server-side `shallow` marker is a compatibility boundary, not valid final +family storage. Migration copies it onto the installed thin view so the view +keeps serving exactly what the legacy repository served, and excludes its +backup from retirement until the family holds the complete reachable closure +of the backup's refs. Closure recovery is the ordinary integrity repair: the +family reports truncated parents as missing objects and marked views as `shallow_views`, repair fetches the closure from accepted clone sources, and -once complete the marker is removed and the backup retires on a later launch. -An unrecoverable closure retains the backup and keeps the family reported as -unresolved; this never blocks startup because recovery depends on remote -servers. Client-requested shallow clones and fetches are a protocol feature -of upload-pack and remain supported throughout. +once complete removes the marker. The backup retires on the next launch. An +unrecoverable closure retains both marker and backup and keeps the family +reported as unresolved; it never blocks startup because recovery depends on +remote servers. Client-requested shallow clones and fetches remain supported +throughout. Retirement failures during an active migration are fail-closed and keep the backup. Journals from a migration that already committed are handled diff --git a/docs/how-to/README.md b/docs/how-to/README.md index 404294d..ef59732 100644 --- a/docs/how-to/README.md +++ b/docs/how-to/README.md @@ -24,16 +24,17 @@ How-to guides are **recipes** that show you how to solve specific problems or ac ## Available How-To Guides -### [Upgrade Git family storage](upgrade-git-family-storage.md) -**Problem:** Deduplicate existing repository objects during a server upgrade +### [Upgrade from v2 to v3 Git family storage](upgrade-git-family-storage.md) + +**Problem:** Perform the one-way identifier-family storage migration safely **Difficulty:** Advanced **You'll learn:** -- Prepare capacity and a release rollback point -- Run the automatic crash-safe launch migration -- Verify owner and `/prs/` repository views -- Check or repair one identifier family on demand -- Recover safely from an interrupted launch + +- Prepare capacity and a snapshot-based rollback point +- Run the automatic crash-safe v3 launch migration +- Interpret migration and integrity summary logs +- Restore v2 safely when a release rollback is required --- diff --git a/docs/how-to/upgrade-git-family-storage.md b/docs/how-to/upgrade-git-family-storage.md index 94b4d21..1c975d6 100644 --- a/docs/how-to/upgrade-git-family-storage.md +++ b/docs/how-to/upgrade-git-family-storage.md @@ -1,148 +1,64 @@ -# Upgrade a server to identifier-family Git storage +# Upgrade from v2 to v3 Git family storage -This procedure upgrades existing owner repositories and `/prs/` repositories -to ref-only views backed by one local object family per repository identifier. -It requires no new configuration and keeps all durable Git storage local. +Version 3 replaces complete per-owner and `/prs/` bare repositories with thin +views backed by one local object family per repository identifier. The upgrade +requires no new configuration and runs automatically before the server accepts +traffic. -The upgrader runs automatically on every relay launch after configuration has -been validated and before purgatory restoration, background sync, or HTTP -request handling. A failed migration stops startup; restarting resumes from its -fsynced journal. +This is a one-way Git-data migration. A v2 binary does not understand the +family locking and write model and must not be run against a migrated +`NGIT_GIT_DATA_PATH`. Rolling back means restoring a complete pre-upgrade +snapshot, not only installing the older binary. -After runtime database initialization, a non-blocking integrity pass checks -the resulting identifier families and views. It attempts to fetch missing -objects from clone URLs in accepted repository announcements and emits an -`ERROR` log for any family that remains unhealthy. Network repair never holds -up the listening service or changes whether the structural migration commits. - -## Before deploying +## Before upgrading 1. Stop writes to the relay and take a filesystem snapshot of both the Git and - relay data directories. An older binary cannot serve the new thin views, - and legacy backups are retired during migration, so this external snapshot - is the rollback boundary for the release. -2. Check free space on `NGIT_GIT_DATA_PATH`. Migration processes one - identifier family at a time and retires that family's legacy backups after - verification before starting the next family. Plan for the current - footprint plus roughly one extra copy of the largest identifier family - (all owner and `/prs/` repositories sharing one identifier), not the whole - dataset. Deduplication reduces the final family size but should not be - assumed for preflight capacity planning. -3. Keep Git and relay data on durable storage. -4. Do not configure Git GC for this rollout. Unreachable objects intentionally - preserve delete-state rollback material inside each family. + relay data directories. Keep it for the rollback window. +2. Check free space on `NGIT_GIT_DATA_PATH`. Plan for the current footprint + plus roughly one extra copy of the largest identifier family: every owner + and `/prs/` repository sharing one identifier. The migration retires each + healthy family's legacy repositories before processing the next family. +3. Keep Git and relay data on durable storage. Do not enable Git garbage + collection for this rollout; retained objects preserve recovery from + deletion-state mistakes. -## Perform the upgrade +## Upgrade -Deploy the new release with the existing configuration and start the relay. -For every identifier, launch migration inventories all objects from all -matching owner and `/prs/` paths, including unreachable objects, verifies the -union, then atomically replaces each repository with a thin view. While a -family converts, its original repositories are held under: +Deploy v3 with the existing configuration and start the relay. The launch +migration is journaled, crash-safe, and safe to resume by restarting the same +v3 release. It: -```text -/.grasp/migration/backups/ -``` +1. Builds and verifies one identifier family at a time, including unreachable + objects retained for rollback. +2. Replaces each owner and `/prs/` repository with a thin view that preserves + its refs, `HEAD`, configuration, and observable Git behavior. +3. Deletes a legacy backup only after its object superset, view wiring, packs, + and ordinary family integrity report all verify. +4. Keeps any unhealthy family's backup while the running server automatically + attempts repair from accepted clone URLs. -Once every view of a family is installed, each of that family's backups is -verified — every Git-readable backup object present in the family, the thin -views wired and matching their journal snapshot, and every family pack -indexed and valid — and the ordinary family integrity report must be healthy. -Only then is the backup deleted before the next family is migrated. Peak -migration overhead is therefore bounded by the family currently in flight, -apart from families retained for automatic repair. A pack without an index is -invisible to Git and cannot be vouched for by that verification; such packs -are preserved by content under `.grasp/migration/unindexed-packs/` for manual -`git index-pack` recovery. +Progress is stored below `.grasp/migration/`; the completed layout is marked by +`.grasp/storage-version`. Do not edit these files while the service is running. -Progress and restart state live under `.grasp/migration/journal/`; each -journal is removed after its backup retires. The global -`.grasp/storage-version` marker is written only after every eligible view has -been reconciled and verified. Do not edit these files while the service is -running. - -Confirm the service reaches its normal listening state and exercise a clone, -fetch, and push for both a normal repository and a `/prs/` route. Retain the -external pre-upgrade snapshot for the rollback window; the retired migration -backups are gone once their family verifies. - -A legacy repository carrying a server-side `shallow` marker is a deliberate -truncation and is not valid family storage. Migration installs its thin view -with the marker intact, so clients keep seeing exactly what the legacy -repository served, and retains its backup with an `ERROR` log naming the -unresolved family. The integrity pass then tries to fetch the complete -reachable closure from accepted clone sources; once the family is complete, a -later launch removes the marker and retires the backup. If the closure cannot -be recovered, the backup stays and the family keeps being reported as -unresolved. Shallow clones and fetches requested by clients are unaffected -and remain supported. - -Installations that migrated under a release which kept backups indefinitely -retire them on the next launch after the same verification, except that the -stale journal ref snapshot is no longer enforced against views whose refs -have legitimately changed in service. A backup that cannot be verified — for -example its migrated view was later deleted by repository lifecycle, or the -family cannot prove it contains every backup object — is kept with a warning -and remains operator-managed. Installations whose backups were already -verified and deleted manually start normally; their completed journals are -compacted away. - -Watch for the terminal startup-pass summary: +Confirm that the service reaches its normal listening state, then check the +terminal integrity summary: ```text Git identifier-family integrity startup pass completed ``` -An `unresolved` or `failed` count above zero is accompanied by an `ERROR` log -for each affected identifier. This reports pre-existing missing data without -putting startup into a network-dependent restart loop. +An `unresolved` or `failed` count above zero has a corresponding `ERROR` naming +the identifier. The server has already attempted automatic repair. A retained +backup protects an unhealthy family's legacy data, and a legacy shallow view +continues serving at its pre-upgrade level rather than being made less usable. -## Check or repair one identifier on demand +## Roll back -Queue a read-only check for every object format and owner/`/prs/` view sharing -an identifier: +Stop v3 and restore the pre-upgrade Git and relay-data snapshot together before +starting v2. Do not point v2 at thin family views, even if ordinary clones +appear to work: v2 does not coordinate writes through the shared family or +preserve its durability invariants. -```console -ngit-grasp integrity-check \ - --git-data-path /var/lib/ngit-grasp/git \ - --identifier example -``` - -Add `--repair` to repair alternate wiring and try accepted clone servers for -missing OIDs: - -```console -ngit-grasp integrity-check \ - --git-data-path /var/lib/ngit-grasp/git \ - --identifier example \ - --repair -``` - -The command queues a durable request for the running relay rather than opening -or mutating a family from a second process. The worker normally consumes it -within five seconds and writes the result to the service log. A request queued -while the relay is stopped is processed after its next startup integrity pass. -Repeated requests for the same identifier and mode safely coalesce. - -## Failure and rollback - -- A migration or backup-retirement failure during an active migration is - fail-closed: the family's legacy backup is kept, startup stops, and a - restart resumes from the fsynced journal. A backup is never deleted before - its family verifies. -- Pre-existing damage does not block retirement. An object that was already - absent from the legacy repository cannot be preserved by keeping its backup, - so retirement proceeds; the condition stays visible through the integrity - startup pass and `integrity-check --repair`. -- A post-migration integrity repair failure is fail-open because the damage - predates conversion or arose after it. Inspect the identifier's `ERROR` log, - repair or update its listed clone sources, and queue `integrity-check - --repair` again. -- To roll the software release back, stop the service and restore the complete - pre-upgrade Git and relay-data snapshot together. Do not point an older - binary at migrated thin views. - -Backup retirement only removes verified legacy copies. Unreachable-object -pruning inside families remains deferred until rollback and delete-state -retention have an explicit policy. S3 adoption is a separate optional rollout -and is not required for this local storage model. +The storage model, automatic shallow-history compatibility repair, and detailed +retirement guarantees are described in +[Identifier-family Git object storage](../explanation/git-family-object-storage.md#v2-to-v3-migration-boundary).