docs(storage): distinguish legacy incomplete histories

Motivation: Production v3 rehearsal found unmarked missing ancestors that the previous shallow-fetch wording could misattribute to a narrow pre-release bug.

Approach: Describe incomplete legacy object graphs without inferring provenance, retain the shallow marker as a specific historical case, and document a minimal maintainer-bundle recovery procedure. Remove stale test comments claiming current purgatory fetches are shallow.

Correctness: Current fetches have no depth limit, migration logged the affected graphs as pre-existing, and an unmarked missing parent is not evidence of the marked January fallback behavior.

Excluded scope: No integrity policy, object import automation, migration behavior, or production data is changed.

Validation: Ran git diff --check and reviewed the production sweep evidence; tests were intentionally skipped because this commit changes documentation and comments only.
This commit is contained in:
DanConwayDev
2026-08-20 20:16:32 +00:00
parent 47bfc2015d
commit 527bfeced5
5 changed files with 86 additions and 40 deletions
+11 -8
View File
@@ -170,14 +170,17 @@ Expect a bit of downtime as a git data migraiton is performed on startup. The la
a warning, backup-directory removals are fsynced before their journals are a warning, backup-directory removals are fsynced before their journals are
removed, and completed installations whose backups were already removed removed, and completed installations whose backups were already removed
manually start unchanged. manually start unchanged.
- Reconcile server-side shallow repositories left by untagged development - Detect and reconcile incomplete object graphs inherited from legacy storage.
builds between 2026-01-05 (`623cae5`) and 2026-01-12 (`f25eea8`). No tagged The v3 migration preserves the legacy repository and its backup; the online
v1 or v2 release shipped the depth-one fallback fetch. The affected integrity worker then requests missing objects from every accepted clone
non-happy path fetched missing state or PR data from another listed Git source and logs any family that remains incomplete without making it less
server when the objects had not arrived by push. V3 preserves existing available. A server-side
shallow clone behavior and the legacy backup, automatically requests the `shallow` marker specifically identifies repositories created by the
complete closure from accepted clone sources, and logs any family that depth-one fallback in untagged development builds between 2026-01-05
remains shallow without making it less available. (`623cae5`) and 2026-01-12 (`f25eea8`); no tagged v1 or v2 release shipped
that behavior. Missing ancestors without a `shallow` marker are not
attributed to that bug and can require a complete maintainer clone or bundle
when every announced server inherited the same incomplete history.
- Scope bare log levels to ngit-grasp while keeping dependencies at warnings; - Scope bare log levels to ngit-grasp while keeping dependencies at warnings;
explicit tracing filter expressions remain unchanged. explicit tracing filter expressions remain unchanged.
- Keep per-event discovery and validation details at debug, retain aggregate - Keep per-event discovery and validation details at debug, retain aggregate
+33 -23
View File
@@ -259,31 +259,41 @@ retained-root, or shared-object write model. A software rollback therefore
requires restoring the Git and relay-data snapshot taken before v3; changing requires restoring the Git and relay-data snapshot taken before v3; changing
only the binary is not supported. only the binary is not supported.
One pre-release development interval can leave server-side shallow V3 can encounter an object graph which was already incomplete in legacy
repositories for v3 to reconcile. State Git-data fallback was introduced by storage. Migration copies every Git-readable object and preserves the backup;
commit `623cae5` on 2026-01-05 with `git fetch --depth=1`, carried into the it does not manufacture a missing ancestor or infer its provenance. Earlier
rewritten purgatory sync path, and changed to a full fetch by commit `f25eea8` manual copies or data migrations, an incomplete source repository, and other
on 2026-01-12. The first tagged release was v1.0.0 on 2026-02-26 and already pre-existing storage damage can all produce the same unmarked symptom. The
contained the fix, so no tagged v1 or v2 release shipped the shallow-fetch integrity report therefore describes the evidence (`missing_oids`, broken
behavior. Only operators who deployed an untagged source revision from that links, and `shallow_views`) rather than assigning a cause from a missing object
seven-day interval can have repositories created by this bug. alone.
The affected path was the non-happy-path Git-data fallback. It ran only when a One known and narrower source has an unambiguous marker. State Git-data fallback
pending state or PR event named OIDs which had not arrived through an ordinary was introduced by commit `623cae5` on 2026-01-05 with
push, then fetched those OIDs into an existing local repository from another `git fetch --depth=1`, carried into the rewritten purgatory sync path, and
announcement or PR clone URL. A repository whose required data arrived through changed to a full fetch by commit `f25eea8` on 2026-01-12. The first tagged
the normal push path did not invoke this fallback. release was v1.0.0 on 2026-02-26 and already contained the fix, so no tagged v1
or v2 release shipped the shallow-fetch behavior. Only operators who deployed
an untagged source revision from that seven-day interval can have repositories
created by this bug. The affected fallback ran only when a pending State or PR
event named OIDs which had not arrived through an ordinary push.
V3 treats a legacy `shallow` marker as compatibility state rather than a reason V3 treats that legacy `shallow` marker as compatibility state rather than a
to delete or disable the repository. Migration preserves the marker on the thin reason to delete or disable the repository. Migration preserves the marker on
view so its shallow clones and current tree continue to work, and retains the the thin view and retains the legacy backup. After the listener starts, the
legacy backup. After the listener starts, the ordinary integrity worker ordinary integrity worker requests the complete closure from other accepted
requests the complete closure from other accepted clone servers. On success it clone servers. On success it removes the marker; the next v3 launch retires the
removes the marker; the next v3 launch retires the now-redundant backup. On now-redundant backup. On failure the existing shallow repository and backup
failure the existing shallow clone and current tree remain available, the remain, and an `ERROR` identifies the family with a non-zero `shallow_views`
backup remains, and an `ERROR` log identifies the family with a non-zero count.
`shallow_views` count. The operator does not need to take a separate action for
the v3 upgrade. An incomplete graph with `shallow_views=0` is a different case. V3 still tries
every accepted clone source, but all listed servers may share the same missing
history or the complete source may no longer be announced. Such a family stays
unresolved until an operator obtains the missing closure from a known-complete
maintainer clone or bundle. Matching a signed State event is not enough to
prove this closure: State authorizes ref tips, while storage integrity also
checks every object reachable behind those tips.
## Startup migration ## Startup migration
+38 -5
View File
@@ -122,9 +122,42 @@ Only the second command applies safe fixes. Re-run the check-only command after
repair; `repair_needed`, `manual_inspection`, and `failed` must all be zero for repair; `repair_needed`, `manual_inspection`, and `failed` must all be zero for
that identifier to be considered clean. that identifier to be considered clean.
A retained backup protects an unhealthy family's legacy data, and a legacy A retained backup protects an unhealthy family's legacy data. A non-zero
shallow view continues serving at its pre-upgrade level rather than being made `shallow_views` count identifies the marked depth-one fallback used only by a
less usable. seven-day untagged development interval; no tagged v1 or v2 release shipped
that behavior. Do not attribute an unmarked `missing_oids` result to that bug.
It means the incomplete graph predated the v3 conversion, but its exact origin
cannot be inferred from the missing object alone. Earlier manual copies or data
migrations, an incomplete source, and other legacy storage damage can have the
same result.
### Recover an unresolved object graph
Automatic repair can only use objects available from the accepted clone URLs.
If all announced servers inherited the same gap, ask a maintainer with a
known-complete local clone to run the following, substituting the OIDs from the
relay's `ERROR` log:
```bash
git fsck --full
git cat-file -e '<missing-oid>^{commit}'
git cat-file -e '<child-oid>^{commit}'
git cat-file -p '<child-oid>'
git update-ref refs/integrity-repair/<child-short-id> '<child-oid>'
git bundle create repository-integrity-repair.bundle \
refs/integrity-repair/<child-short-id>
git update-ref -d refs/integrity-repair/<child-short-id>
git bundle verify repository-integrity-repair.bundle
sha256sum repository-integrity-repair.bundle
```
The `fsck` and both `cat-file` checks must succeed. Transfer the bundle and its
SHA-256 digest through agreed channels; an ordinary push or `ngit sync` may be
a no-op when the relay already advertises the authorized tip. Import the bundle
under a temporary ref, remove that ref after the objects are present, queue the
identifier with `--repair`, and finish with a check-only request. Do not delete
the retained migration backup until storage reports `unresolved=0` and
`failed=0`.
## Roll back ## Roll back
@@ -133,6 +166,6 @@ starting v2. Do not point v2 at thin family views, even if ordinary clones
appear to work: v2 does not coordinate writes through the shared family or appear to work: v2 does not coordinate writes through the shared family or
preserve its durability invariants. preserve its durability invariants.
The storage model, automatic shallow-history compatibility repair, and detailed The storage model, legacy incomplete-history repair, and detailed retirement
retirement guarantees are described in guarantees are described in
[Identifier-family Git object storage](../explanation/git-family-object-storage.md#v2-to-v3-migration-boundary). [Identifier-family Git object storage](../explanation/git-family-object-storage.md#v2-to-v3-migration-boundary).
+3 -3
View File
@@ -43,8 +43,8 @@
//! - **SimpleGitServer**: Fast, lightweight, good for basic `git fetch` without depth limits //! - **SimpleGitServer**: Fast, lightweight, good for basic `git fetch` without depth limits
//! - **SmartGitServer**: Full protocol support, required for `--depth=1` shallow fetches //! - **SmartGitServer**: Full protocol support, required for `--depth=1` shallow fetches
//! //!
//! The purgatory sync system uses `git fetch --depth=1`, so tests involving purgatory //! Tests which explicitly exercise shallow client fetches should use
//! sync should use `SmartGitServer`. //! `SmartGitServer`; ordinary purgatory sync performs full fetches.
use grasp_audit::git_command; use grasp_audit::git_command;
use std::net::SocketAddr; use std::net::SocketAddr;
@@ -525,7 +525,7 @@ mod tests {
/// - Shallow fetches (`git fetch --depth=1`) /// - Shallow fetches (`git fetch --depth=1`)
/// - Full protocol negotiation /// - Full protocol negotiation
/// ///
/// This is required for testing purgatory sync, which uses `git fetch --depth=1`. /// This is required for tests which explicitly exercise shallow client fetches.
pub struct SmartGitServer { pub struct SmartGitServer {
/// Shutdown signal sender /// Shutdown signal sender
shutdown_tx: Option<oneshot::Sender<()>>, shutdown_tx: Option<oneshot::Sender<()>>,
+1 -1
View File
@@ -940,7 +940,7 @@ async fn test_pr_event_clone_tag_sync_with_partial_oid_aggregation_from_multiple
let mock_relay = MockRelay::start().await; let mock_relay = MockRelay::start().await;
// 3. git_server - SmartGitServer with PR commit only // 3. git_server - SmartGitServer with PR commit only
// Using SmartGitServer because purgatory sync uses `git fetch --depth=1` // Using SmartGitServer because this fixture exercises smart HTTP fetches
// which requires the Git Smart HTTP protocol (not dumb HTTP) // which requires the Git Smart HTTP protocol (not dumb HTTP)
let git_server = SmartGitServer::start(repo_b.path()).await; let git_server = SmartGitServer::start(repo_b.path()).await;