ngit-grasp/bb46-git-http-protocol-test-suite.md

Git HTTP Smart Protocol Comprehensive Test Suite

ID: bb46

Problem

Production bug discovered: git clone failed because ngit-grasp wasn't decompressing gzip-encoded request bodies that modern git clients send. This reveals a critical gap in testing - we need comprehensive git HTTP smart protocol compliance tests that catch real-world client behavior before production.

Current testing gaps:

• No tests for Content-Encoding variations (gzip, deflate, none)
• No tests across git protocol versions (v0, v1, v2)
• No tests for edge cases (partial clones, large repos, concurrent requests)
• No integration tests with real git clients
• No conformance testing against git's own http-backend behavior

Plan

• Phase 1: Research existing test suites and specifications

  • Find git project's own http-backend tests
  • Survey cgit, gitea, gitlab test approaches
  • Identify RFC/specification compliance requirements
  • Document git HTTP smart protocol edge cases
  • Output: Research Report section in this issue

• Phase 2: Design comprehensive test matrix

  • Content-Encoding variations (gzip, deflate, identity, none)
  • Git protocol versions (v0, v1, v2)
  • Git operations (clone, fetch, push, ls-remote)
  • Repository states (empty, large, many refs, shallow)
  • Client configurations and versions
  • Error conditions (malformed requests, timeouts, auth failures)
  • Output: docs/reference/git-http-test-matrix.md

• Phase 3: Implementation strategy

  • Integration test approach using TestRelay fixture
  • Real git client testing framework
  • Conformance test suite structure
  • CI/CD integration plan
  • Output: docs/explanation/git-integration-test-framework.md

• Phase 4: Create test implementation in grasp-audit

  • Build git protocol utilities in grasp-audit/src/git/
  • Implement priority test cases in grasp-audit/src/specs/grasp01/
  • Add to existing grasp-audit test suite
  • Document test coverage
  • Output: grasp-audit/src/git/, grasp-audit/src/specs/grasp01/git_protocol.rs

• Phase 5: MVP - Single test following existing pattern

  • Pick minimal test requiring minimal GitClient wrapper
  • Follow pattern from git_clone.rs and how it's called in ngit-grasp tests
  • Validate architecture works end-to-end
  • Output: Working MVP test demonstrating the pattern
  • Completed: grasp-audit/src/git/client.rs, grasp-audit/src/specs/grasp01/git_http_protocol.rs, tests/git_http_protocol.rs

Vision

Build comprehensive git HTTP smart protocol compliance tests in grasp-audit that:

1. Validate GRASP conformance - Ensure all GRASP implementations fully comply with git HTTP protocol
2. Catch real-world bugs - Test with actual git client behavior (gzip, protocol versions, edge cases)
3. Reusable across implementations - Any GRASP implementation can use these tests for validation
4. Extend existing patterns - Build on grasp-audit's existing HTTP testing infrastructure

Architecture

grasp-audit/
├── src/
│   ├── git/                          # NEW: Git protocol utilities
│   │   ├── mod.rs
│   │   ├── pkt_line.rs               # Pkt-line parsing
│   │   ├── protocol.rs               # Protocol v2 support
│   │   └── client.rs                 # GitClient wrapper (uses relay_domain)
│   │
│   └── specs/grasp01/
│       ├── git_clone.rs              # EXISTING: Basic clone tests
│       ├── git_protocol.rs           # NEW: Protocol compliance tests
│       ├── git_fetch.rs              # NEW: Fetch protocol tests
│       └── git_push.rs               # NEW: Push protocol tests
│
ngit-grasp/
└── tests/
    └── git_http_protocol.rs          # Calls grasp-audit tests (uses TestRelay)

Integration Pattern

grasp-audit provides:

• Protocol utilities (pkt-line parsing, protocol v2)
• Conformance test functions that accept relay_domain parameter
• Reusable across any GRASP implementation

ngit-grasp provides:

• TestRelay fixture for convenience
• Integration tests that call grasp-audit functions
• Implementation-specific tests

MVP Approach

Start with ONE test that:

1. Requires minimal GitClient wrapper
2. Follows existing git_clone.rs pattern
3. Demonstrates end-to-end architecture
4. Validates the approach before building full suite

Progress

2026-01-21 [Session 18:00] - MVP COMPLETE ✅

• MVP Implementation Complete: All three architect agents completed their work
• Test Selection (Architect 1):
  • Selected: test_gzip_encoded_upload_pack_request
  • Rationale: Catches production bug, minimal wrapper needed, follows existing pattern
  • Uses existing OwnerStateDataPushed fixture
  • No new GitClient wrapper needed for MVP - uses reqwest directly
• GitClient Design (Architect 2):
  • Created minimal wrapper in grasp-audit/src/git/client.rs
  • Methods: get_info_refs(), upload_pack_gzip(), receive_pack_gzip()
  • Builder pattern: GitClient::builder(relay_domain).with_gzip(true).build()
  • Uses flate2 for gzip compression
  • Follows existing grasp-audit patterns (relay_domain parameter, reqwest client)
• Implementation (Architect 3):
  • Created: grasp-audit/src/git/mod.rs - Module exports
  • Created: grasp-audit/src/git/client.rs - GitClient wrapper (200+ lines)
  • Created: grasp-audit/src/specs/grasp01/git_http_protocol.rs - Test file
  • Created: tests/git_http_protocol.rs - Integration test in ngit-grasp
  • Modified: grasp-audit/Cargo.toml - Added flate2 dependency
  • Modified: grasp-audit/src/lib.rs - Exported git module
  • Modified: grasp-audit/src/specs/grasp01/mod.rs - Exported GitHttpProtocolTests
• Test Verification:
  • Test passes: cargo test --test git_http_protocol test_gzip_encoded_upload_pack
  • Validates gzip-encoded request handling (production bug scenario)
  • Follows existing pattern: isolated_test! macro, TestRelay fixture
• Architecture Validated:
  • grasp-audit provides test functions accepting relay_domain parameter
  • ngit-grasp integration tests use TestRelay and call grasp-audit functions
  • Pattern is reusable and extensible
• Next: Review MVP implementation, then expand to full test suite

2026-01-21 [Session 17:30]

• Updated: Issue with vision, architecture, and MVP approach
• Plan: Build MVP with single test following existing pattern
• Next: Architect to plan MVP test selection and implementation

2026-01-21 [Session 17:15] - REVISED DECISION

• DECISION REVERSED: Tests SHOULD move to grasp-audit
• User clarification: "We want other GRASP implementations to be fully compatible with the git HTTP protocol"
• Key insight: Git HTTP compliance IS part of GRASP conformance, not just implementation detail
• Architect-plan analysis revealed:
  • ✅ grasp-audit ALREADY tests HTTP endpoints extensively (CORS, git clone, capabilities)
  • ✅ Existing fixture system (relay_domain parameter) can support comprehensive git HTTP tests
  • ✅ Pattern already exists: git_clone.rs makes direct HTTP requests via reqwest
  • ✅ Shell git commands already used: clone_repo(), try_push() in fixtures
  • ❌ Previous recommendation was too quick to dismiss - architectural barrier is lower than assessed
• Technical findings:
  • grasp-audit's test-ngit-relay.sh already manages relay lifecycle
  • Tests receive relay_domain parameter for HTTP access
  • TestRelay is a convenience, not a requirement
  • What's missing: pkt-line parsing, protocol v2 support, comprehensive edge cases
• Proposed architecture:
  • Add grasp-audit/src/git/ module for protocol utilities (pkt_line.rs, protocol.rs, client.rs)
  • Expand existing specs/grasp01/git_clone.rs with protocol-level validation
  • Add new test files: git_fetch.rs, git_push.rs, git_protocol.rs
  • ngit-grasp keeps TestRelay for convenience, calls grasp-audit library functions
• Specification clarity needed:
  • GRASP-01 spec should explicitly require "Git Smart HTTP protocol as defined in git-scm.com/docs/http-protocol"
  • Should require protocol v2 support when client requests it
  • Should specify HTTP status codes for error conditions
• Migration path:
  • Phase 1: Add git protocol utilities to grasp-audit
  • Phase 2: Expand existing tests with protocol-level validation
  • Phase 3: ngit-grasp integration tests continue using TestRelay, call grasp-audit functions
• Next: Implement git protocol utilities in grasp-audit, migrate tests

2026-01-21 [Session 16:56]

• Fixed: Worktree setup to follow proper start-work conventions
• Issue: Original worktree was created with branch name bb46 instead of bb46-git-http-protocol-test-suite
• Action: Removed incorrect worktree and recreated with proper naming
• Current state:
  • ✅ Branch name: bb46-git-http-protocol-test-suite (matches issue filename without .md)
  • ✅ Worktree directory: worktrees/bb46-git-http-protocol-test-suite
  • ✅ dcdev-issues symlink: Properly configured and working
  • ❌ Test files: Lost during worktree recreation (were untracked)
• Next: Test files need to be recreated from previous session's work
  • tests/git_http_protocol.rs (18,844 bytes - 50 integration tests)
  • tests/common/git_client.rs (15,732 bytes - Git client wrapper)
  • docs/reference/git-http-test-matrix.md (20,169 bytes)
  • docs/explanation/git-integration-test-framework.md (22,329 bytes)
• Note: Files were created in Session 17:30 but not committed before worktree was incorrectly set up

2026-01-21 [Session 16:45] - SUPERSEDED BY SESSION 17:15

• Decision: Keep tests in ngit-grasp, do NOT move to grasp-audit
• Rationale:
  • grasp-audit is for GRASP compliance testing (Nostr events, black-box)
  • bb46 tests are implementation-specific (HTTP protocol, git client behavior)
  • grasp-audit already has sufficient git tests for GRASP-01 compliance
  • bb46 tests go beyond spec requirements (protocol versions, gzip, edge cases)
  • No other GRASP implementations exist yet to benefit from shared tests
• Architectural analysis:
  • grasp-audit scope: Nostr-level compliance via AuditClient
  • bb46 tests scope: HTTP-level implementation quality via GitClient
  • Scope mismatch would require new abstraction layer and maintenance burden
• Alternative approach: Document test patterns for other implementations to adapt
• Next: Commit tests to ngit-grasp, add reference note to grasp-audit README

2026-01-21 [Session 16:27]

• Started work: Created worktree for issue bb46
• Moved test implementation files from root repository to worktree:
  • tests/git_http_protocol.rs (18,844 bytes)
  • tests/common/git_client.rs (15,732 bytes)
  • docs/reference/git-http-test-matrix.md (20,169 bytes)
  • docs/explanation/git-integration-test-framework.md (22,329 bytes)
• Files are now in worktree as untracked files, ready to be committed
• Next: Review files, add to git, and commit to bb46 branch

2026-01-21 [Session 17:30]

• Completed: Phase 3 & 4 - Implementation strategy and test framework
• Created: docs/explanation/git-integration-test-framework.md - Comprehensive design document
• Created: tests/common/git_client.rs - Git client wrapper with protocol/compression control
• Created: tests/git_http_protocol.rs - 50 integration tests covering:
  • Protocol versions (v0, v1, v2)
  • Content encoding (gzip, uncompressed)
  • Git operations (clone, fetch, ls-remote, shallow clone)
  • Edge cases (nonexistent repos, timeouts, concurrent clones)
• All 50 tests passing
• Framework features:
  • Configurable git client wrapper (protocol version, gzip, timeout)
  • Proper test isolation using TestRelay fixture
  • Helper functions for repo setup with announcement + state events
  • Slow tests marked with #[ignore] for CI optimization

2026-01-21 [Session 16:00]

• Completed: Phase 1 - Research on existing test suites and specifications
• Analyzed: Git project's HTTP test suite (t55*.sh files)
• Reviewed: Gitea, GitLab test approaches
• Key findings from research:
  • Git's t5562-http-backend-content-length.sh is the key reference for Content-Encoding tests
  • Git supports both "gzip" and "x-gzip" Content-Encoding values
  • Protocol v2 is now default in modern git clients
  • Git tests use real Apache httpd via lib-httpd.sh
• Documented: 8 categories of edge cases, specification references
• Output: Research Report section added to this issue

2026-01-21 [Session 14:30]

• Completed: Phase 2 - Comprehensive test matrix designed
• Deliverable: docs/reference/git-http-test-matrix.md
• Coverage: 95 test scenarios across 10 categories
• Key findings:
  • Content-Encoding: 0% coverage (critical gap - caused production bug)
  • Protocol Versions: 0% coverage (v2 is default in modern git)
  • Git Operations: ~15% coverage (only basic clone tested)
  • Error Handling: ~20% coverage (minimal edge cases)
• Prioritization: 25 P0 tests, 40 P1 tests, 30 P2 tests
• Roadmap: 4-week implementation plan with weekly milestones
• Next: Phase 3 - Implementation strategy and test infrastructure

2026-01-21 [Session Start]

• Created: Issue created to track comprehensive git HTTP protocol testing
• Context: Discovered gzip decompression bug in production
• Goal: Build test suite that catches real-world client issues before production

Notes

• Root cause: Modern git clients send gzip-encoded request bodies by default
• Current tests use TestRelay fixture (tests/common/relay.rs:14)
• Need to test against REAL git clients, not just synthetic requests
• Consider using git's own test suite as reference implementation
• Must cover both happy path AND edge cases
• Integration tests should run against live ngit-grasp instance

Research Report (Phase 1)

1. Git Project's Own Test Suite

Key Test Files (in git/t/):

File	Purpose
t5551-http-fetch-smart.sh	Core smart HTTP fetch tests
t5541-http-push-smart.sh	Smart HTTP push tests
t5555-http-smart-common.sh	Protocol version tests (v0, v1, v2)
t5560-http-backend-noserver.sh	HTTP backend without server (unit-like)
t5561-http-backend.sh	Full HTTP backend integration
t5562-http-backend-content-length.sh	CRITICAL: Content-Length and gzip tests
t5539-fetch-http-shallow.sh	Shallow clone over HTTP
t5559-http-fetch-smart-http2.sh	HTTP/2 protocol tests
t556x_common	Shared test helpers
lib-httpd.sh	Apache httpd test infrastructure

What Git Tests:

• Reference discovery (info/refs?service=git-upload-pack)
• Smart service responses (git-upload-pack, git-receive-pack)
• Content-Type validation (application/x-git-upload-pack-advertisement)
• Protocol version negotiation (v0, v1, v2 via Git-Protocol header)
• Authentication (Basic auth, credential helpers)
• Redirects (301, 302)
• Chunked transfer encoding
• Large fetch-pack requests
• Cookie handling
• Error conditions (invalid Content-Type, server errors)

2. Content-Encoding Handling (Critical!)

From git's http-backend.c:

const char *encoding = getenv("HTTP_CONTENT_ENCODING");
if (encoding && (!strcmp(encoding, "gzip") || !strcmp(encoding, "x-gzip")))
    gzipped_request = 1;

From t5562-http-backend-content-length.sh:

# Tests gzip-encoded request bodies
test_expect_success GZIP 'fetch gzipped' '
    test_env HTTP_CONTENT_ENCODING="gzip" test_http_env upload fetch_body.gz &&
    verify_http_result "200 OK"
'

test_expect_success GZIP 'push gzipped' '
    test_env HTTP_CONTENT_ENCODING="gzip" test_http_env receive push_body.gz &&
    verify_http_result "200 OK"
'

MUST TEST:

• Content-Encoding: gzip - Most common, git client default
• Content-Encoding: x-gzip - Alternative gzip identifier
• Content-Encoding: identity - Explicit no encoding
• No Content-Encoding header - Implicit identity
• Truncated gzip streams (error handling)
• Empty gzip bodies (error handling)

3. Protocol Versions

Version 0 (Legacy):

• No Git-Protocol header
• Capabilities in first ref line after NUL byte
• symref=HEAD:refs/heads/main in capability line

Version 1:

• Git-Protocol: version=1 header
• Response starts with version 1 pkt-line
• Same capability format as v0

Version 2:

• Git-Protocol: version=2 header
• Response starts with version 2 pkt-line
• Capabilities as separate lines (agent, ls-refs, fetch, etc.)
• Stateless by default
• Commands: ls-refs, fetch, server-option, object-info

From t5555-http-smart-common.sh:

test_expect_success 'git upload-pack --advertise-refs: v2' '
    cat >expect <<-EOF &&
    version 2
    agent=FAKE
    ls-refs=unborn
    fetch=shallow wait-for-done
    server-option
    object-format=$(test_oid algo)
    0000
    EOF
    GIT_PROTOCOL=version=2 git upload-pack --advertise-refs . >out
'

4. HTTP Protocol Specification Requirements

From git-scm.com/docs/http-protocol:

Reference Discovery:

• GET $GIT_URL/info/refs?service=git-upload-pack
• Response Content-Type: application/x-git-upload-pack-advertisement
• Response must start with # service=git-upload-pack pkt-line
• Must include capability declarations on first ref

Smart Service (upload-pack):

• POST $GIT_URL/git-upload-pack
• Request Content-Type: application/x-git-upload-pack-request
• Response Content-Type: application/x-git-upload-pack-result
• Cache-Control: no-cache, max-age=0, must-revalidate

Smart Service (receive-pack):

• POST $GIT_URL/git-receive-pack
• Request Content-Type: application/x-git-receive-pack-request
• Response Content-Type: application/x-git-receive-pack-result

Error Handling:

• 403 Forbidden for disabled services
• 404 Not Found for missing repositories
• Must NOT return 200 OK for non-existent repos

5. Edge Cases That MUST Be Tested

Content-Encoding:

1. Gzip-encoded POST bodies (clone, fetch, push)
2. x-gzip encoding variant
3. Truncated gzip streams
4. Empty gzip bodies
5. Mixed encoding (gzip request, identity response)

Protocol Negotiation:

1. Client requests v2, server supports v2
2. Client requests v2, server only supports v0 (fallback)
3. Client requests v1
4. No version specified (default to v0)

Repository States:

1. Empty repository (no refs)
2. Repository with many refs (>2000 tags test in git)
3. Shallow repository
4. Repository with large objects
5. Repository with submodules

Operations:

1. Clone (full)
2. Clone (shallow with --depth)
3. Clone (partial with --filter)
4. Fetch (incremental)
5. Fetch (shallow-since, shallow-exclude)
6. Push (standard)
7. Push (atomic)
8. Push (chunked encoding for large packs)
9. ls-remote

HTTP Edge Cases:

1. Chunked Transfer-Encoding
2. Large Content-Length values
3. CONTENT_LENGTH overflow (ssize_t boundary)
4. Empty CONTENT_LENGTH
5. HTTP/1.0 vs HTTP/1.1
6. HTTP/2 (if supported)
7. Keep-alive connections
8. Redirects (301, 302)
9. Path traversal attempts (security)

Authentication:

1. No auth (public repo)
2. Basic auth required
3. Auth only for push
4. Auth only for objects (half-auth)
5. Invalid credentials
6. Expired credentials

6. Other Implementation Approaches

Gitea:

• Uses real git client integration tests
• Tests HTTP clone/push with actual git commands
• Tests LFS integration
• Tests SSH and HTTP in parallel

GitLab:

• Extensive RSpec tests for git HTTP
• Tests authentication edge cases
• Tests project path variations (.git suffix handling)
• Tests blocked users, expired passwords
• Tests namespace/project creation via push

7. Recommendations for Test Approach

Priority 1 - Critical (Would have caught our bug):

1. Content-Encoding: gzip for POST bodies
2. Content-Encoding: x-gzip variant
3. Truncated/malformed gzip handling

Priority 2 - Protocol Compliance:

1. Protocol version negotiation (v0, v1, v2)
2. Correct Content-Type headers
3. Correct pkt-line format
4. Capability advertisement

Priority 3 - Operations:

1. Clone (basic)
2. Clone (shallow)
3. Fetch (incremental)
4. Push (basic)
5. ls-remote

Priority 4 - Edge Cases:

1. Empty repository
2. Large repository (many refs)
3. Chunked encoding
4. HTTP/2 (if applicable)

Test Infrastructure:

• Use real git client (not mocked requests)
• Test with multiple git versions if possible
• Use TestRelay fixture for isolation
• Consider adapting git's t5562 test approach

8. Specification References

Document	URL
HTTP Protocol	https://git-scm.com/docs/http-protocol
Pack Protocol	https://git-scm.com/docs/pack-protocol
Protocol v2	https://git-scm.com/docs/protocol-v2
Git Test Suite	https://github.com/git/git/tree/master/t
RFC 2616 (HTTP/1.1)	https://www.ietf.org/rfc/rfc2616.txt
RFC 1952 (GZIP)	https://www.ietf.org/rfc/rfc1952.txt

9. Key Gotchas from Other Implementations

1. Git's gzip handling: Uses HTTP_CONTENT_ENCODING env var, supports both "gzip" and "x-gzip"
2. Protocol fallback: Client may request v2 but server can respond with v0 - client must handle gracefully
3. Half-auth: Some servers require auth only for objects, not refs - complex to test
4. Path normalization: Must handle /repo.git vs /repo vs /repo.git/ consistently
5. Chunked encoding: Large pushes use chunked encoding when http.postbuffer is small
6. Empty repos: Must handle repos with no refs (capabilities^{} line)
7. Peeled refs: Annotated tags must show both tag and peeled object
8. Symref: HEAD symref must be communicated in capabilities