laantungir/ngit-grasp
1
0
Fork 0
mirror of https://relay.ngit.dev/npub15qydau2hjma6ngxkl2cyar74wzyjshvl65za5k5rl69264ar2exs5cyejr/ngit-grasp.git synced 2026-10-05 15:08:24 +00:00
Code Issues Packages Projects Releases Wiki Activity

docs(storage): define the v2 to v3 migration boundary

Browse Source 
Explain that identifier-family storage is an automatic, one-way on-disk migration and that rollback requires the pre-upgrade Git and relay-data snapshot. Keep the operator procedure focused on actions they must actually take.

Document the precise untagged development window that could create server-side shallow repositories, the missing-push fallback that triggered it, and v3's non-degrading automatic repair behavior. No tagged v1 or v2 release shipped that fetch behavior.

This change deliberately leaves on-demand integrity commands in the technical explanation rather than making them a required upgrade step. Validation: git diff --check and historical commit/tag inspection.
This commit is contained in:
DanConwayDev
2026-08-19 07:50:25 +00:00
parent 56d705e2bb
commit eca818041c
4 changed files with 115 additions and 155 deletions
Download Patch File Download Diff File Expand all files Collapse all files
CHANGELOG.md
+14 -6
View File
@@ -9,6 +9,12 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
### Breaking changes
- Version 3 performs an automatic, one-way migration of `NGIT_GIT_DATA_PATH`
from complete per-owner and `/prs/` repositories to thin views backed by
shared identifier families. Version 2 does not coordinate reads and writes
through this layout and must not be run against migrated Git data. Rollback
requires restoring the Git and relay-data snapshot taken before the v3
launch, not only downgrading the binary.
- Added `trusted_proxy_cidrs` to the public `Config` struct. Rust consumers
that construct `Config` with a struct literal must provide it.
- Added `base_path` to the public `Config` struct. Rust consumers that
@@ -84,12 +90,14 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
a warning, backup-directory removals are fsynced before their journals are
removed, and completed installations whose backups were already removed
manually start unchanged.
- Stop treating server-side shallow repositories as valid family storage.
Migration preserves a legacy `shallow` marker on the installed thin view
and retains that backup until integrity repair recovers the complete
reachable closure from accepted clone sources; the family is reported as
unresolved until then, after which the marker is removed and the backup
retires. Client-requested shallow clones remain supported.
- Reconcile server-side shallow repositories left by untagged development
builds between 2026-01-05 (`623cae5`) and 2026-01-12 (`f25eea8`). No tagged
v1 or v2 release shipped the depth-one fallback fetch. The affected
non-happy path fetched missing state or PR data from another listed Git
server when the objects had not arrived by push. V3 preserves existing
shallow clone behavior and the legacy backup, automatically requests the
complete closure from accepted clone sources, and logs any family that
remains shallow without making it less available.
- Scope bare log levels to ngit-grasp while keeping dependencies at warnings;
explicit tracing filter expressions remain unchanged.
- Keep per-event discovery and validation details at debug, retain aggregate
docs/explanation/git-family-object-storage.md
+47 -12
View File
@@ -249,6 +249,42 @@ view whose alternate is the identifier family. Therefore:
Mirroring a PR into an owner view becomes a ref update after an object
availability check. It no longer copies the object graph.
## V2 to v3 migration boundary
V3 changes the on-disk meaning of every served Git repository. Owner and
`/prs/` paths become thin views whose objects and write serialization belong to
the identifier family under `.grasp`. Migration happens automatically on the
first v3 launch, but the result is not safe for v2: v2 has no family lock,
retained-root, or shared-object write model. A software rollback therefore
requires restoring the Git and relay-data snapshot taken before v3; changing
only the binary is not supported.
One pre-release development interval can leave server-side shallow
repositories for v3 to reconcile. State Git-data fallback was introduced by
commit `623cae5` on 2026-01-05 with `git fetch --depth=1`, carried into the
rewritten purgatory sync path, and changed to a full fetch by commit `f25eea8`
on 2026-01-12. The first tagged release was v1.0.0 on 2026-02-26 and already
contained the fix, so no tagged v1 or v2 release shipped the shallow-fetch
behavior. Only operators who deployed an untagged source revision from that
seven-day interval can have repositories created by this bug.
The affected path was the non-happy-path Git-data fallback. It ran only when a
pending state or PR event named OIDs which had not arrived through an ordinary
push, then fetched those OIDs into an existing local repository from another
announcement or PR clone URL. A repository whose required data arrived through
the normal push path did not invoke this fallback.
V3 treats a legacy `shallow` marker as compatibility state rather than a reason
to delete or disable the repository. Migration preserves the marker on the thin
view so its shallow clones and current tree continue to work, and retains the
legacy backup. After the listener starts, the ordinary integrity worker
requests the complete closure from other accepted clone servers. On success it
removes the marker; the next v3 launch retires the now-redundant backup. On
failure the existing shallow clone and current tree remain available, the
backup remains, and an `ERROR` log identifies the family with a non-zero
`shallow_views` count. The operator does not need to take a separate action for
the v3 upgrade.
## Startup migration
Migration runs after configuration validation and before purgatory restoration,
@@ -302,19 +338,18 @@ for them; they are moved into content-addressed directories beneath
migrations preserve different payloads even when their original pack filenames
match, while an identical payload converges on the same quarantine path.
A server-side `shallow` marker means the legacy repository is a deliberate
truncation, and GRASP does not treat shallow repositories as valid family
storage. Migration copies the marker onto the installed thin view so the view
keeps serving what the legacy repository served, and its backup is excluded
from retirement until the family holds the complete reachable closure of the
backup's refs. Closure recovery is the ordinary integrity repair: the family
reports the truncated parents as missing objects and the marked views as
A server-side `shallow` marker is a compatibility boundary, not valid final
family storage. Migration copies it onto the installed thin view so the view
keeps serving exactly what the legacy repository served, and excludes its
backup from retirement until the family holds the complete reachable closure
of the backup's refs. Closure recovery is the ordinary integrity repair: the
family reports truncated parents as missing objects and marked views as
`shallow_views`, repair fetches the closure from accepted clone sources, and
once complete the marker is removed and the backup retires on a later launch.
An unrecoverable closure retains the backup and keeps the family reported as
unresolved; this never blocks startup because recovery depends on remote
servers. Client-requested shallow clones and fetches are a protocol feature
of upload-pack and remain supported throughout.
once complete removes the marker. The backup retires on the next launch. An
unrecoverable closure retains both marker and backup and keeps the family
reported as unresolved; it never blocks startup because recovery depends on
remote servers. Client-requested shallow clones and fetches remain supported
throughout.
Retirement failures during an active migration are fail-closed and keep the
backup. Journals from a migration that already committed are handled
docs/how-to/README.md
+8 -7
View File
@@ -24,16 +24,17 @@ How-to guides are **recipes** that show you how to solve specific problems or ac
## Available How-To Guides
### [Upgrade Git family storage](upgrade-git-family-storage.md)
**Problem:** Deduplicate existing repository objects during a server upgrade
### [Upgrade from v2 to v3 Git family storage](upgrade-git-family-storage.md)
**Problem:** Perform the one-way identifier-family storage migration safely
**Difficulty:** Advanced
**You'll learn:**
- Prepare capacity and a release rollback point
- Run the automatic crash-safe launch migration
- Verify owner and `/prs/` repository views
- Check or repair one identifier family on demand
- Recover safely from an interrupted launch
- Prepare capacity and a snapshot-based rollback point
- Run the automatic crash-safe v3 launch migration
- Interpret migration and integrity summary logs
- Restore v2 safely when a release rollback is required
---
docs/how-to/upgrade-git-family-storage.md
+46 -130
View File
@@ -1,148 +1,64 @@
# Upgrade a server to identifier-family Git storage
# Upgrade from v2 to v3 Git family storage
This procedure upgrades existing owner repositories and `/prs/` repositories
to ref-only views backed by one local object family per repository identifier.
It requires no new configuration and keeps all durable Git storage local.
Version 3 replaces complete per-owner and `/prs/` bare repositories with thin
views backed by one local object family per repository identifier. The upgrade
requires no new configuration and runs automatically before the server accepts
traffic.
The upgrader runs automatically on every relay launch after configuration has
been validated and before purgatory restoration, background sync, or HTTP
request handling. A failed migration stops startup; restarting resumes from its
fsynced journal.
This is a one-way Git-data migration. A v2 binary does not understand the
family locking and write model and must not be run against a migrated
`NGIT_GIT_DATA_PATH`. Rolling back means restoring a complete pre-upgrade
snapshot, not only installing the older binary.
After runtime database initialization, a non-blocking integrity pass checks
the resulting identifier families and views. It attempts to fetch missing
objects from clone URLs in accepted repository announcements and emits an
`ERROR` log for any family that remains unhealthy. Network repair never holds
up the listening service or changes whether the structural migration commits.
## Before deploying
## Before upgrading
1. Stop writes to the relay and take a filesystem snapshot of both the Git and
relay data directories. An older binary cannot serve the new thin views,
and legacy backups are retired during migration, so this external snapshot
is the rollback boundary for the release.
2. Check free space on `NGIT_GIT_DATA_PATH`. Migration processes one
identifier family at a time and retires that family's legacy backups after
verification before starting the next family. Plan for the current
footprint plus roughly one extra copy of the largest identifier family
(all owner and `/prs/` repositories sharing one identifier), not the whole
dataset. Deduplication reduces the final family size but should not be
assumed for preflight capacity planning.
3. Keep Git and relay data on durable storage.
4. Do not configure Git GC for this rollout. Unreachable objects intentionally
preserve delete-state rollback material inside each family.
relay data directories. Keep it for the rollback window.
2. Check free space on `NGIT_GIT_DATA_PATH`. Plan for the current footprint
plus roughly one extra copy of the largest identifier family: every owner
and `/prs/` repository sharing one identifier. The migration retires each
healthy family's legacy repositories before processing the next family.
3. Keep Git and relay data on durable storage. Do not enable Git garbage
collection for this rollout; retained objects preserve recovery from
deletion-state mistakes.
## Perform the upgrade
## Upgrade
Deploy the new release with the existing configuration and start the relay.
For every identifier, launch migration inventories all objects from all
matching owner and `/prs/` paths, including unreachable objects, verifies the
union, then atomically replaces each repository with a thin view. While a
family converts, its original repositories are held under:
Deploy v3 with the existing configuration and start the relay. The launch
migration is journaled, crash-safe, and safe to resume by restarting the same
v3 release. It:
```text
<git-data>/.grasp/migration/backups/
```
1. Builds and verifies one identifier family at a time, including unreachable
objects retained for rollback.
2. Replaces each owner and `/prs/` repository with a thin view that preserves
its refs, `HEAD`, configuration, and observable Git behavior.
3. Deletes a legacy backup only after its object superset, view wiring, packs,
and ordinary family integrity report all verify.
4. Keeps any unhealthy family's backup while the running server automatically
attempts repair from accepted clone URLs.
Once every view of a family is installed, each of that family's backups is
verified — every Git-readable backup object present in the family, the thin
views wired and matching their journal snapshot, and every family pack
indexed and valid — and the ordinary family integrity report must be healthy.
Only then is the backup deleted before the next family is migrated. Peak
migration overhead is therefore bounded by the family currently in flight,
apart from families retained for automatic repair. A pack without an index is
invisible to Git and cannot be vouched for by that verification; such packs
are preserved by content under `.grasp/migration/unindexed-packs/` for manual
`git index-pack` recovery.
Progress is stored below `.grasp/migration/`; the completed layout is marked by
`.grasp/storage-version`. Do not edit these files while the service is running.
Progress and restart state live under `.grasp/migration/journal/`; each
journal is removed after its backup retires. The global
`.grasp/storage-version` marker is written only after every eligible view has
been reconciled and verified. Do not edit these files while the service is
running.
Confirm the service reaches its normal listening state and exercise a clone,
fetch, and push for both a normal repository and a `/prs/` route. Retain the
external pre-upgrade snapshot for the rollback window; the retired migration
backups are gone once their family verifies.
A legacy repository carrying a server-side `shallow` marker is a deliberate
truncation and is not valid family storage. Migration installs its thin view
with the marker intact, so clients keep seeing exactly what the legacy
repository served, and retains its backup with an `ERROR` log naming the
unresolved family. The integrity pass then tries to fetch the complete
reachable closure from accepted clone sources; once the family is complete, a
later launch removes the marker and retires the backup. If the closure cannot
be recovered, the backup stays and the family keeps being reported as
unresolved. Shallow clones and fetches requested by clients are unaffected
and remain supported.
Installations that migrated under a release which kept backups indefinitely
retire them on the next launch after the same verification, except that the
stale journal ref snapshot is no longer enforced against views whose refs
have legitimately changed in service. A backup that cannot be verified — for
example its migrated view was later deleted by repository lifecycle, or the
family cannot prove it contains every backup object — is kept with a warning
and remains operator-managed. Installations whose backups were already
verified and deleted manually start normally; their completed journals are
compacted away.
Watch for the terminal startup-pass summary:
Confirm that the service reaches its normal listening state, then check the
terminal integrity summary:
```text
Git identifier-family integrity startup pass completed
```
An `unresolved` or `failed` count above zero is accompanied by an `ERROR` log
for each affected identifier. This reports pre-existing missing data without
putting startup into a network-dependent restart loop.
An `unresolved` or `failed` count above zero has a corresponding `ERROR` naming
the identifier. The server has already attempted automatic repair. A retained
backup protects an unhealthy family's legacy data, and a legacy shallow view
continues serving at its pre-upgrade level rather than being made less usable.
## Check or repair one identifier on demand
## Roll back
Queue a read-only check for every object format and owner/`/prs/` view sharing
an identifier:
Stop v3 and restore the pre-upgrade Git and relay-data snapshot together before
starting v2. Do not point v2 at thin family views, even if ordinary clones
appear to work: v2 does not coordinate writes through the shared family or
preserve its durability invariants.
```console
ngit-grasp integrity-check \
--git-data-path /var/lib/ngit-grasp/git \
--identifier example
```
Add `--repair` to repair alternate wiring and try accepted clone servers for
missing OIDs:
```console
ngit-grasp integrity-check \
--git-data-path /var/lib/ngit-grasp/git \
--identifier example \
--repair
```
The command queues a durable request for the running relay rather than opening
or mutating a family from a second process. The worker normally consumes it
within five seconds and writes the result to the service log. A request queued
while the relay is stopped is processed after its next startup integrity pass.
Repeated requests for the same identifier and mode safely coalesce.
## Failure and rollback
- A migration or backup-retirement failure during an active migration is
fail-closed: the family's legacy backup is kept, startup stops, and a
restart resumes from the fsynced journal. A backup is never deleted before
its family verifies.
- Pre-existing damage does not block retirement. An object that was already
absent from the legacy repository cannot be preserved by keeping its backup,
so retirement proceeds; the condition stays visible through the integrity
startup pass and `integrity-check --repair`.
- A post-migration integrity repair failure is fail-open because the damage
predates conversion or arose after it. Inspect the identifier's `ERROR` log,
repair or update its listed clone sources, and queue `integrity-check
--repair` again.
- To roll the software release back, stop the service and restore the complete
pre-upgrade Git and relay-data snapshot together. Do not point an older
binary at migrated thin views.
Backup retirement only removes verified legacy copies. Unreachable-object
pruning inside families remains deferred until rollback and delete-state
retention have an explicit policy. S3 adoption is a separate optional rollout
and is not required for this local storage model.
The storage model, automatic shallow-history compatibility repair, and detailed
retirement guarantees are described in
[Identifier-family Git object storage](../explanation/git-family-object-storage.md#v2-to-v3-migration-boundary).