docs(storage): distinguish legacy incomplete histories

Motivation: Production v3 rehearsal found unmarked missing ancestors that the previous shallow-fetch wording could misattribute to a narrow pre-release bug.

Approach: Describe incomplete legacy object graphs without inferring provenance, retain the shallow marker as a specific historical case, and document a minimal maintainer-bundle recovery procedure. Remove stale test comments claiming current purgatory fetches are shallow.

Correctness: Current fetches have no depth limit, migration logged the affected graphs as pre-existing, and an unmarked missing parent is not evidence of the marked January fallback behavior.

Excluded scope: No integrity policy, object import automation, migration behavior, or production data is changed.

Validation: Ran git diff --check and reviewed the production sweep evidence; tests were intentionally skipped because this commit changes documentation and comments only.
This commit is contained in:
DanConwayDev
2026-08-20 20:16:32 +00:00
parent 47bfc2015d
commit 527bfeced5
5 changed files with 86 additions and 40 deletions
+11 -8
View File
@@ -170,14 +170,17 @@ Expect a bit of downtime as a git data migraiton is performed on startup. The la
a warning, backup-directory removals are fsynced before their journals are
removed, and completed installations whose backups were already removed
manually start unchanged.
- Reconcile server-side shallow repositories left by untagged development
builds between 2026-01-05 (`623cae5`) and 2026-01-12 (`f25eea8`). No tagged
v1 or v2 release shipped the depth-one fallback fetch. The affected
non-happy path fetched missing state or PR data from another listed Git
server when the objects had not arrived by push. V3 preserves existing
shallow clone behavior and the legacy backup, automatically requests the
complete closure from accepted clone sources, and logs any family that
remains shallow without making it less available.
- Detect and reconcile incomplete object graphs inherited from legacy storage.
The v3 migration preserves the legacy repository and its backup; the online
integrity worker then requests missing objects from every accepted clone
source and logs any family that remains incomplete without making it less
available. A server-side
`shallow` marker specifically identifies repositories created by the
depth-one fallback in untagged development builds between 2026-01-05
(`623cae5`) and 2026-01-12 (`f25eea8`); no tagged v1 or v2 release shipped
that behavior. Missing ancestors without a `shallow` marker are not
attributed to that bug and can require a complete maintainer clone or bundle
when every announced server inherited the same incomplete history.
- Scope bare log levels to ngit-grasp while keeping dependencies at warnings;
explicit tracing filter expressions remain unchanged.
- Keep per-event discovery and validation details at debug, retain aggregate
+33 -23
View File
@@ -259,31 +259,41 @@ retained-root, or shared-object write model. A software rollback therefore
requires restoring the Git and relay-data snapshot taken before v3; changing
only the binary is not supported.
One pre-release development interval can leave server-side shallow
repositories for v3 to reconcile. State Git-data fallback was introduced by
commit `623cae5` on 2026-01-05 with `git fetch --depth=1`, carried into the
rewritten purgatory sync path, and changed to a full fetch by commit `f25eea8`
on 2026-01-12. The first tagged release was v1.0.0 on 2026-02-26 and already
contained the fix, so no tagged v1 or v2 release shipped the shallow-fetch
behavior. Only operators who deployed an untagged source revision from that
seven-day interval can have repositories created by this bug.
V3 can encounter an object graph which was already incomplete in legacy
storage. Migration copies every Git-readable object and preserves the backup;
it does not manufacture a missing ancestor or infer its provenance. Earlier
manual copies or data migrations, an incomplete source repository, and other
pre-existing storage damage can all produce the same unmarked symptom. The
integrity report therefore describes the evidence (`missing_oids`, broken
links, and `shallow_views`) rather than assigning a cause from a missing object
alone.
The affected path was the non-happy-path Git-data fallback. It ran only when a
pending state or PR event named OIDs which had not arrived through an ordinary
push, then fetched those OIDs into an existing local repository from another
announcement or PR clone URL. A repository whose required data arrived through
the normal push path did not invoke this fallback.
One known and narrower source has an unambiguous marker. State Git-data fallback
was introduced by commit `623cae5` on 2026-01-05 with
`git fetch --depth=1`, carried into the rewritten purgatory sync path, and
changed to a full fetch by commit `f25eea8` on 2026-01-12. The first tagged
release was v1.0.0 on 2026-02-26 and already contained the fix, so no tagged v1
or v2 release shipped the shallow-fetch behavior. Only operators who deployed
an untagged source revision from that seven-day interval can have repositories
created by this bug. The affected fallback ran only when a pending State or PR
event named OIDs which had not arrived through an ordinary push.
V3 treats a legacy `shallow` marker as compatibility state rather than a reason
to delete or disable the repository. Migration preserves the marker on the thin
view so its shallow clones and current tree continue to work, and retains the
legacy backup. After the listener starts, the ordinary integrity worker
requests the complete closure from other accepted clone servers. On success it
removes the marker; the next v3 launch retires the now-redundant backup. On
failure the existing shallow clone and current tree remain available, the
backup remains, and an `ERROR` log identifies the family with a non-zero
`shallow_views` count. The operator does not need to take a separate action for
the v3 upgrade.
V3 treats that legacy `shallow` marker as compatibility state rather than a
reason to delete or disable the repository. Migration preserves the marker on
the thin view and retains the legacy backup. After the listener starts, the
ordinary integrity worker requests the complete closure from other accepted
clone servers. On success it removes the marker; the next v3 launch retires the
now-redundant backup. On failure the existing shallow repository and backup
remain, and an `ERROR` identifies the family with a non-zero `shallow_views`
count.
An incomplete graph with `shallow_views=0` is a different case. V3 still tries
every accepted clone source, but all listed servers may share the same missing
history or the complete source may no longer be announced. Such a family stays
unresolved until an operator obtains the missing closure from a known-complete
maintainer clone or bundle. Matching a signed State event is not enough to
prove this closure: State authorizes ref tips, while storage integrity also
checks every object reachable behind those tips.
## Startup migration
+38 -5
View File
@@ -122,9 +122,42 @@ Only the second command applies safe fixes. Re-run the check-only command after
repair; `repair_needed`, `manual_inspection`, and `failed` must all be zero for
that identifier to be considered clean.
A retained backup protects an unhealthy family's legacy data, and a legacy
shallow view continues serving at its pre-upgrade level rather than being made
less usable.
A retained backup protects an unhealthy family's legacy data. A non-zero
`shallow_views` count identifies the marked depth-one fallback used only by a
seven-day untagged development interval; no tagged v1 or v2 release shipped
that behavior. Do not attribute an unmarked `missing_oids` result to that bug.
It means the incomplete graph predated the v3 conversion, but its exact origin
cannot be inferred from the missing object alone. Earlier manual copies or data
migrations, an incomplete source, and other legacy storage damage can have the
same result.
### Recover an unresolved object graph
Automatic repair can only use objects available from the accepted clone URLs.
If all announced servers inherited the same gap, ask a maintainer with a
known-complete local clone to run the following, substituting the OIDs from the
relay's `ERROR` log:
```bash
git fsck --full
git cat-file -e '<missing-oid>^{commit}'
git cat-file -e '<child-oid>^{commit}'
git cat-file -p '<child-oid>'
git update-ref refs/integrity-repair/<child-short-id> '<child-oid>'
git bundle create repository-integrity-repair.bundle \
refs/integrity-repair/<child-short-id>
git update-ref -d refs/integrity-repair/<child-short-id>
git bundle verify repository-integrity-repair.bundle
sha256sum repository-integrity-repair.bundle
```
The `fsck` and both `cat-file` checks must succeed. Transfer the bundle and its
SHA-256 digest through agreed channels; an ordinary push or `ngit sync` may be
a no-op when the relay already advertises the authorized tip. Import the bundle
under a temporary ref, remove that ref after the objects are present, queue the
identifier with `--repair`, and finish with a check-only request. Do not delete
the retained migration backup until storage reports `unresolved=0` and
`failed=0`.
## Roll back
@@ -133,6 +166,6 @@ starting v2. Do not point v2 at thin family views, even if ordinary clones
appear to work: v2 does not coordinate writes through the shared family or
preserve its durability invariants.
The storage model, automatic shallow-history compatibility repair, and detailed
retirement guarantees are described in
The storage model, legacy incomplete-history repair, and detailed retirement
guarantees are described in
[Identifier-family Git object storage](../explanation/git-family-object-storage.md#v2-to-v3-migration-boundary).
+3 -3
View File
@@ -43,8 +43,8 @@
//! - **SimpleGitServer**: Fast, lightweight, good for basic `git fetch` without depth limits
//! - **SmartGitServer**: Full protocol support, required for `--depth=1` shallow fetches
//!
//! The purgatory sync system uses `git fetch --depth=1`, so tests involving purgatory
//! sync should use `SmartGitServer`.
//! Tests which explicitly exercise shallow client fetches should use
//! `SmartGitServer`; ordinary purgatory sync performs full fetches.
use grasp_audit::git_command;
use std::net::SocketAddr;
@@ -525,7 +525,7 @@ mod tests {
/// - Shallow fetches (`git fetch --depth=1`)
/// - Full protocol negotiation
///
/// This is required for testing purgatory sync, which uses `git fetch --depth=1`.
/// This is required for tests which explicitly exercise shallow client fetches.
pub struct SmartGitServer {
/// Shutdown signal sender
shutdown_tx: Option<oneshot::Sender<()>>,
+1 -1
View File
@@ -940,7 +940,7 @@ async fn test_pr_event_clone_tag_sync_with_partial_oid_aggregation_from_multiple
let mock_relay = MockRelay::start().await;
// 3. git_server - SmartGitServer with PR commit only
// Using SmartGitServer because purgatory sync uses `git fetch --depth=1`
// Using SmartGitServer because this fixture exercises smart HTTP fetches
// which requires the Git Smart HTTP protocol (not dumb HTTP)
let git_server = SmartGitServer::start(repo_b.path()).await;