laantungir/ngit-grasp

Fork 0

mirror of https://relay.ngit.dev/npub15qydau2hjma6ngxkl2cyar74wzyjshvl65za5k5rl69264ar2exs5cyejr/ngit-grasp.git synced 2026-10-05 15:08:24 +00:00

Files

T

DanConwayDev fbf18c9dd0 issue: reopen incomplete issues (b454, b905, bb46, ddbf, ff38)

2026-02-03 22:33:36 +00:00

33 KiB

Raw Blame History

Git HTTP Smart Protocol Comprehensive Test Suite

ID: bb46

Problem

Production bug discovered: git clone failed because ngit-grasp wasn't decompressing gzip-encoded request bodies that modern git clients send. This reveals a critical gap in testing - we need comprehensive git HTTP smart protocol compliance tests that catch real-world client behavior before production.

Current testing gaps:

No tests for Content-Encoding variations (gzip, deflate, none)
No tests across git protocol versions (v0, v1, v2)
No tests for edge cases (partial clones, large repos, concurrent requests)
No integration tests with real git clients
No conformance testing against git's own http-backend behavior

Plan

Completed Phases

Phase 1: Research existing test suites and specifications
- Find git project's own http-backend tests
- Survey cgit, gitea, gitlab test approaches
- Identify RFC/specification compliance requirements
- Document git HTTP smart protocol edge cases
- Output: Research Report section in this issue
Phase 2: Design comprehensive test matrix
- Content-Encoding variations (gzip, deflate, identity, none)
- Git protocol versions (v0, v1, v2)
- Git operations (clone, fetch, push, ls-remote)
- Repository states (empty, large, many refs, shallow)
- Client configurations and versions
- Error conditions (malformed requests, timeouts, auth failures)
- Output: Research Report section (test matrix doc not committed)
Phase 3: Implementation strategy
- Integration test approach using TestRelay fixture
- Real git client testing framework
- Conformance test suite structure
- CI/CD integration plan
- Output: Architecture documented in Vision section
Phase 4: Create test implementation in grasp-audit
- Build git protocol utilities in grasp-audit/src/git/
- Implement priority test cases in grasp-audit/src/specs/grasp01/
- Add to existing grasp-audit test suite
- Output: grasp-audit/src/git/client.rs, grasp-audit/src/specs/grasp01/git_http_protocol.rs
Phase 5: MVP - Single test following existing pattern
- Pick minimal test requiring minimal GitClient wrapper
- Follow pattern from git_clone.rs and how it's called in ngit-grasp tests
- Validate architecture works end-to-end
- Output: Working MVP test demonstrating the pattern
- Completed: grasp-audit/src/git/client.rs, grasp-audit/src/specs/grasp01/git_http_protocol.rs, tests/git_http_protocol.rs

Refined Implementation Plan (Post-MVP)

MVP Status: ✅ COMPLETE

GitClient wrapper: grasp-audit/src/git/client.rs (238 lines)
Test function: grasp-audit/src/specs/grasp01/git_http_protocol.rs (187 lines)
Integration test: tests/git_http_protocol.rs using isolated_test! macro
Test passes in ~1.5 seconds

Key Learnings:

Pattern works well - grasp-audit provides test functions, ngit-grasp calls them with TestRelay
Using real git fetch validates actual client behavior (not synthetic requests)
OwnerStateDataPushed fixture provides repo with git data ready for testing
The isolated_test! macro pattern is clean and reusable

Refined Test Matrix (65 scenarios, reduced from 95)

P0 - Critical (15 tests)

ID	Test	Status	Rationale
P0-01	`test_gzip_encoded_upload_pack`	✅ DONE	Caught production bug
P0-02	`test_gzip_encoded_receive_pack`	TODO	Push path validation
P0-03	`test_x_gzip_encoding_variant`	TODO	Git supports "x-gzip"
P0-04	`test_identity_encoding_explicit`	TODO	Explicit no-compression
P0-05	`test_no_encoding_header`	TODO	No header (implicit)
P0-06	`test_truncated_gzip_stream`	TODO	Error handling
P0-07	`test_protocol_v2_negotiation`	TODO	Modern git default
P0-08	`test_protocol_v0_fallback`	TODO	Legacy support
P0-09	`test_clone_basic`	TODO	Core operation
P0-10	`test_fetch_incremental`	TODO	Most common op
P0-11	`test_push_basic`	TODO	Write path
P0-12	`test_ls_remote`	TODO	Ref discovery
P0-13	`test_404_nonexistent_repo`	TODO	Must not return 200
P0-14	`test_403_unauthorized_push`	TODO	Auth enforcement
P0-15	`test_correct_content_types`	TODO	Protocol compliance

P1 - Important (25 tests)

Protocol: v1 negotiation, version header parsing
Operations: shallow clone, partial clone, atomic push
Repo States: empty repo, many refs (>100), large objects
HTTP: chunked transfer, large Content-Length, keep-alive
Errors: malformed pkt-line, invalid Content-Type, timeout
Concurrent: parallel clones, parallel fetches

P2 - Nice to Have (25 tests)

Advanced: HTTP/2, redirects, cookies
Security: path traversal, oversized requests
Performance: large packs, many small fetches
Edge Cases: peeled refs, symref, capabilities

Removed from original 95:

HTTP Basic auth tests (ngit-grasp uses Nostr auth)
Half-auth scenarios (not applicable to GRASP)
Credential helper tests (not applicable)
Submodule tests (out of scope)

Fixture Optimization: `GitRepoFixture`

Problem: Each test creates new repo (~1-2 seconds). 65 tests = 100+ seconds.

Solution: Shared fixture struct:

pub struct GitRepoFixture {
    pub npub: String,           // Owner's bech32 pubkey
    pub identifier: String,     // Repo d-tag
    pub relay_domain: String,   // HTTP endpoint
    pub commit_hash: String,    // Existing commit
    pub clone_url: String,      // Full clone URL
    pub announcement: Event,    // Repo announcement
    pub state_event: Event,     // State event
}

Performance Impact:

Approach	Time for 65 tests
Per-test fixture	~100 seconds
Shared fixture	~15 seconds
Parallel + shared	~8 seconds

Remaining Implementation Phases

Phase 6: P0 Content-Encoding Tests
- test_gzip_encoded_receive_pack - Push with gzip
- test_x_gzip_encoding_variant - Alternative gzip header
- test_identity_encoding_explicit - Explicit identity
- test_no_encoding_header - No header (implicit)
- test_truncated_gzip_stream - Error handling
- Add GitClient::with_encoding() method
Phase 7: Protocol Version Tests
- Create grasp-audit/src/git/pkt_line.rs module
- test_protocol_v2_negotiation
- test_protocol_v0_fallback
- test_protocol_v1_negotiation
- Add GitClient::with_protocol_version() method
Phase 8: Core Operations Tests
- Implement GitRepoFixture struct
- test_clone_basic
- test_fetch_incremental
- test_push_basic
- test_ls_remote
- test_shallow_clone
Phase 9: Error Handling Tests
- test_404_nonexistent_repo
- test_403_unauthorized_push
- test_correct_content_types
- test_malformed_pkt_line
- test_empty_repo_handling
Phase 10: P1 Tests & Polish
- Implement shared fixture pattern with OnceCell
- Complete remaining P1 tests (20)
- Performance optimization
- Create docs/reference/git-http-test-matrix.md

Infrastructure Needed

GitClient Enhancements:

impl GitClient {
    pub fn builder(relay_domain: &str) -> GitClientBuilder;
    pub fn with_protocol_version(self, version: u8) -> Self;
    pub fn with_encoding(self, encoding: Option<&str>) -> Self;
    pub async fn raw_request(&self, method: Method, path: &str,
                             body: Option<&[u8]>, headers: HeaderMap) -> Result<Response>;
}

New Modules:

Module	Purpose
`grasp-audit/src/git/pkt_line.rs`	Parse pkt-line format
`grasp-audit/src/git/protocol.rs`	Protocol version handling
`grasp-audit/src/git/fixture.rs`	GitRepoFixture struct

Success Metrics

Metric	Target
P0 tests passing	15/15 (100%)
P1 tests passing	25/25 (100%)
Single test time	< 2 seconds
Full suite time	< 2 minutes
CI integration	< 5 minutes

Definition of Done

All P0 tests passing
All P1 tests passing
Test matrix documented
Tests run in CI
Full suite < 2 minutes
grasp-audit git module reusable

Vision

Build comprehensive git HTTP smart protocol compliance tests in grasp-audit that:

Validate GRASP conformance - Ensure all GRASP implementations fully comply with git HTTP protocol
Catch real-world bugs - Test with actual git client behavior (gzip, protocol versions, edge cases)
Reusable across implementations - Any GRASP implementation can use these tests for validation
Extend existing patterns - Build on grasp-audit's existing HTTP testing infrastructure

Architecture

grasp-audit/
├── src/
│   ├── git/                          # NEW: Git protocol utilities
│   │   ├── mod.rs
│   │   ├── pkt_line.rs               # Pkt-line parsing
│   │   ├── protocol.rs               # Protocol v2 support
│   │   └── client.rs                 # GitClient wrapper (uses relay_domain)
│   │
│   └── specs/grasp01/
│       ├── git_clone.rs              # EXISTING: Basic clone tests
│       ├── git_protocol.rs           # NEW: Protocol compliance tests
│       ├── git_fetch.rs              # NEW: Fetch protocol tests
│       └── git_push.rs               # NEW: Push protocol tests
│
ngit-grasp/
└── tests/
    └── git_http_protocol.rs          # Calls grasp-audit tests (uses TestRelay)

Integration Pattern

grasp-audit provides:

Protocol utilities (pkt-line parsing, protocol v2)
Conformance test functions that accept relay_domain parameter
Reusable across any GRASP implementation

ngit-grasp provides:

TestRelay fixture for convenience
Integration tests that call grasp-audit functions
Implementation-specific tests

MVP Approach

Start with ONE test that:

Requires minimal GitClient wrapper
Follows existing git_clone.rs pattern
Demonstrates end-to-end architecture
Validates the approach before building full suite

Progress

2026-01-23 [Session End] - KNOWLEDGE PRESERVED & READY FOR IMPLEMENTATION

Session Goals Achieved:
- ✅ Researched existing test infrastructure (grasp-audit patterns)
- ✅ Designed comprehensive test matrix (95 scenarios)
- ✅ Made critical architectural decision (tests in grasp-audit, not ngit-grasp)
- ✅ Built and validated MVP (gzip upload-pack test)
- ✅ Preserved all knowledge to work/ directory for next session
- ✅ Refined implementation plan based on MVP learnings
Preserved to work/:
- work/2026-01-23-bb46-session-summary.md - Comprehensive session notes
- work/2026-01-23-bb46-mvp-code.md - Complete MVP implementation details
- work/2026-01-23-bb46-implementation-plan.md - Refined next steps
- All files contain: context, code, rationale, next steps
MVP Validation:
- Test passes: cargo test --test git_http_protocol test_gzip_encoded_upload_pack
- Architecture proven: grasp-audit provides reusable conformance tests
- Pattern established: relay_domain parameter, follows existing fixtures
- GitClient wrapper validated: minimal, extensible, idiomatic Rust
Key Learnings:
- grasp-audit ALREADY has extensive git HTTP testing infrastructure
- TestRelay is convenience (ngit-grasp), not requirement (grasp-audit uses relay_domain)
- Existing patterns (git_clone.rs, OwnerStateDataPushed) are perfect foundation
- Shell git commands already used in fixtures - no new precedent needed
Current Status:
- Phase 1-3, 5: COMPLETE (Research, Design, Strategy, MVP)
- Phase 4: READY TO START (Full test suite implementation)
- No blockers - clear path forward
Next Steps:
1. Review preserved knowledge in work/ directory
2. Implement remaining 94 test scenarios (prioritize P0 → P1 → P2)
3. Add pkt-line parsing utilities if needed for protocol v2 tests
4. Document test coverage in grasp-audit README
5. Update GRASP-01 spec to require git HTTP protocol compliance
Notes:
- Work can resume immediately from preserved files
- All architectural decisions documented and justified
- MVP proves the approach - scaling to full suite is mechanical
- Fixture optimization opportunity: OwnerStateDataPushed could be parameterized for better reuse

2026-01-21 [Session 18:00] - MVP COMPLETE ✅

MVP Implementation Complete: All three architect agents completed their work
Test Selection (Architect 1):
- Selected: test_gzip_encoded_upload_pack_request
- Rationale: Catches production bug, minimal wrapper needed, follows existing pattern
- Uses existing OwnerStateDataPushed fixture
- No new GitClient wrapper needed for MVP - uses reqwest directly
GitClient Design (Architect 2):
- Created minimal wrapper in grasp-audit/src/git/client.rs
- Methods: get_info_refs(), upload_pack_gzip(), receive_pack_gzip()
- Builder pattern: GitClient::builder(relay_domain).with_gzip(true).build()
- Uses flate2 for gzip compression
- Follows existing grasp-audit patterns (relay_domain parameter, reqwest client)
Implementation (Architect 3):
- Created: grasp-audit/src/git/mod.rs - Module exports
- Created: grasp-audit/src/git/client.rs - GitClient wrapper (200+ lines)
- Created: grasp-audit/src/specs/grasp01/git_http_protocol.rs - Test file
- Created: tests/git_http_protocol.rs - Integration test in ngit-grasp
- Modified: grasp-audit/Cargo.toml - Added flate2 dependency
- Modified: grasp-audit/src/lib.rs - Exported git module
- Modified: grasp-audit/src/specs/grasp01/mod.rs - Exported GitHttpProtocolTests
Test Verification:
- Test passes: cargo test --test git_http_protocol test_gzip_encoded_upload_pack
- Validates gzip-encoded request handling (production bug scenario)
- Follows existing pattern: isolated_test! macro, TestRelay fixture
Architecture Validated:
- grasp-audit provides test functions accepting relay_domain parameter
- ngit-grasp integration tests use TestRelay and call grasp-audit functions
- Pattern is reusable and extensible
Next: Review MVP implementation, then expand to full test suite

2026-01-21 [Session 17:30]

Updated: Issue with vision, architecture, and MVP approach
Plan: Build MVP with single test following existing pattern
Next: Architect to plan MVP test selection and implementation

2026-01-21 [Session 17:15] - REVISED DECISION

DECISION REVERSED: Tests SHOULD move to grasp-audit
User clarification: "We want other GRASP implementations to be fully compatible with the git HTTP protocol"
Key insight: Git HTTP compliance IS part of GRASP conformance, not just implementation detail
Architect-plan analysis revealed:
- ✅ grasp-audit ALREADY tests HTTP endpoints extensively (CORS, git clone, capabilities)
- ✅ Existing fixture system (relay_domain parameter) can support comprehensive git HTTP tests
- ✅ Pattern already exists: git_clone.rs makes direct HTTP requests via reqwest
- ✅ Shell git commands already used: clone_repo(), try_push() in fixtures
- ❌ Previous recommendation was too quick to dismiss - architectural barrier is lower than assessed
Technical findings:
- grasp-audit's test-ngit-relay.sh already manages relay lifecycle
- Tests receive relay_domain parameter for HTTP access
- TestRelay is a convenience, not a requirement
- What's missing: pkt-line parsing, protocol v2 support, comprehensive edge cases
Proposed architecture:
- Add grasp-audit/src/git/ module for protocol utilities (pkt_line.rs, protocol.rs, client.rs)
- Expand existing specs/grasp01/git_clone.rs with protocol-level validation
- Add new test files: git_fetch.rs, git_push.rs, git_protocol.rs
- ngit-grasp keeps TestRelay for convenience, calls grasp-audit library functions
Specification clarity needed:
- GRASP-01 spec should explicitly require "Git Smart HTTP protocol as defined in git-scm.com/docs/http-protocol"
- Should require protocol v2 support when client requests it
- Should specify HTTP status codes for error conditions
Migration path:
- Phase 1: Add git protocol utilities to grasp-audit
- Phase 2: Expand existing tests with protocol-level validation
- Phase 3: ngit-grasp integration tests continue using TestRelay, call grasp-audit functions
Next: Implement git protocol utilities in grasp-audit, migrate tests

2026-01-21 [Session 16:56]

Fixed: Worktree setup to follow proper start-work conventions
Issue: Original worktree was created with branch name bb46 instead of bb46-git-http-protocol-test-suite
Action: Removed incorrect worktree and recreated with proper naming
Current state:
- ✅ Branch name: bb46-git-http-protocol-test-suite (matches issue filename without .md)
- ✅ Worktree directory: worktrees/bb46-git-http-protocol-test-suite
- ✅ dcdev-issues symlink: Properly configured and working
- ❌ Test files: Lost during worktree recreation (were untracked)
Next: Test files need to be recreated from previous session's work
- tests/git_http_protocol.rs (18,844 bytes - 50 integration tests)
- tests/common/git_client.rs (15,732 bytes - Git client wrapper)
- docs/reference/git-http-test-matrix.md (20,169 bytes)
- docs/explanation/git-integration-test-framework.md (22,329 bytes)
Note: Files were created in Session 17:30 but not committed before worktree was incorrectly set up

2026-01-21 [Session 16:45] - SUPERSEDED BY SESSION 17:15

Decision: Keep tests in ngit-grasp, do NOT move to grasp-audit
Rationale:
- grasp-audit is for GRASP compliance testing (Nostr events, black-box)
- bb46 tests are implementation-specific (HTTP protocol, git client behavior)
- grasp-audit already has sufficient git tests for GRASP-01 compliance
- bb46 tests go beyond spec requirements (protocol versions, gzip, edge cases)
- No other GRASP implementations exist yet to benefit from shared tests
Architectural analysis:
- grasp-audit scope: Nostr-level compliance via AuditClient
- bb46 tests scope: HTTP-level implementation quality via GitClient
- Scope mismatch would require new abstraction layer and maintenance burden
Alternative approach: Document test patterns for other implementations to adapt
Next: Commit tests to ngit-grasp, add reference note to grasp-audit README

2026-01-21 [Session 16:27]

Started work: Created worktree for issue bb46
Moved test implementation files from root repository to worktree:
- tests/git_http_protocol.rs (18,844 bytes)
- tests/common/git_client.rs (15,732 bytes)
- docs/reference/git-http-test-matrix.md (20,169 bytes)
- docs/explanation/git-integration-test-framework.md (22,329 bytes)
Files are now in worktree as untracked files, ready to be committed
Next: Review files, add to git, and commit to bb46 branch

2026-01-21 [Session 17:30]

Completed: Phase 3 & 4 - Implementation strategy and test framework
Created: docs/explanation/git-integration-test-framework.md - Comprehensive design document
Created: tests/common/git_client.rs - Git client wrapper with protocol/compression control
Created: tests/git_http_protocol.rs - 50 integration tests covering:
- Protocol versions (v0, v1, v2)
- Content encoding (gzip, uncompressed)
- Git operations (clone, fetch, ls-remote, shallow clone)
- Edge cases (nonexistent repos, timeouts, concurrent clones)
All 50 tests passing
Framework features:
- Configurable git client wrapper (protocol version, gzip, timeout)
- Proper test isolation using TestRelay fixture
- Helper functions for repo setup with announcement + state events
- Slow tests marked with #[ignore] for CI optimization

2026-01-21 [Session 16:00]

Completed: Phase 1 - Research on existing test suites and specifications
Analyzed: Git project's HTTP test suite (t55*.sh files)
Reviewed: Gitea, GitLab test approaches
Key findings from research:
- Git's t5562-http-backend-content-length.sh is the key reference for Content-Encoding tests
- Git supports both "gzip" and "x-gzip" Content-Encoding values
- Protocol v2 is now default in modern git clients
- Git tests use real Apache httpd via lib-httpd.sh
Documented: 8 categories of edge cases, specification references
Output: Research Report section added to this issue

2026-01-21 [Session 14:30]

Completed: Phase 2 - Comprehensive test matrix designed
Deliverable: docs/reference/git-http-test-matrix.md
Coverage: 95 test scenarios across 10 categories
Key findings:
- Content-Encoding: 0% coverage (critical gap - caused production bug)
- Protocol Versions: 0% coverage (v2 is default in modern git)
- Git Operations: ~15% coverage (only basic clone tested)
- Error Handling: ~20% coverage (minimal edge cases)
Prioritization: 25 P0 tests, 40 P1 tests, 30 P2 tests
Roadmap: 4-week implementation plan with weekly milestones
Next: Phase 3 - Implementation strategy and test infrastructure

2026-01-21 [Session Start]

Created: Issue created to track comprehensive git HTTP protocol testing
Context: Discovered gzip decompression bug in production
Goal: Build test suite that catches real-world client issues before production

Notes

General

Root cause: Modern git clients send gzip-encoded request bodies by default
Current tests use TestRelay fixture (tests/common/relay.rs:14)
Need to test against REAL git clients, not just synthetic requests
Consider using git's own test suite as reference implementation
Must cover both happy path AND edge cases
Integration tests should run against live ngit-grasp instance

Key Learnings from MVP (2026-01-23)

Architecture Proven: grasp-audit pattern works perfectly for git HTTP tests
- Tests accept relay_domain parameter (implementation-agnostic)
- ngit-grasp integration tests use TestRelay for convenience
- Pattern is reusable by any GRASP implementation
Existing Infrastructure: grasp-audit already has comprehensive HTTP testing
- See: src/specs/grasp01/git_clone.rs - Makes direct HTTP requests via reqwest
- See: src/fixtures/relay.rs - OwnerStateDataPushed sets up test repos
- See: Shell commands in fixtures - clone_repo(), try_push() already used
Minimal Wrapper Sufficient: GitClient doesn't need to be complex
- Builder pattern: GitClient::builder(relay_domain).with_gzip(true).build()
- Methods: get_info_refs(), upload_pack_gzip(), receive_pack_gzip()
- Uses reqwest (already dependency), flate2 for gzip (minimal addition)
- ~200 lines for MVP - can grow incrementally as tests need more features
Fixture Optimization Opportunity:
- Current: Each test scenario needs separate fixture
- Future: Could parameterize OwnerStateDataPushed for different repo states
- Example: OwnerStateDataPushed::with_commits(5) vs ::with_many_refs(2000)
- Tradeoff: Fixture complexity vs test boilerplate

Preserved Knowledge (2026-01-23)

Session Summary: work/2026-01-23-bb46-session-summary.md
- Complete timeline of research → design → decision → MVP
- Rationale for architectural choices
- Links to all relevant files and discussions
MVP Implementation: work/2026-01-23-bb46-mvp-code.md
- Full code for GitClient wrapper
- Test implementation details
- Build and test commands
- Architecture diagram
Implementation Plan: work/2026-01-23-bb46-implementation-plan.md
- Week-by-week roadmap for remaining 94 tests
- Prioritization: P0 (gzip, protocol versions) → P1 (operations) → P2 (edge cases)
- Fixture design patterns
- CI/CD integration strategy

Next Session Quick Start

START HERE: work/bb46-next-steps.md - Read this first for complete action plan and roadmap

Then review the proposed plan:

work/bb46-detailed-test-plan.md - Review the 65-test matrix and priorities
work/bb46-fixture-optimization.md - Review proposed GitRepoFixture design
work/bb46-architecture.md - Understand the architecture and patterns

Then review the MVP implementation:

grasp-audit/src/git/client.rs - GitClient wrapper (238 lines)
grasp-audit/src/specs/grasp01/git_http_protocol.rs - Test pattern to replicate (187 lines)
tests/git_http_protocol.rs - Integration test example

Reference materials:

work/bb46-test-reference-libraries.md - Git's test suite, specs, edge cases

Research Report (Phase 1)

1. Git Project's Own Test Suite

Key Test Files (in git/t/):

File	Purpose
`t5551-http-fetch-smart.sh`	Core smart HTTP fetch tests
`t5541-http-push-smart.sh`	Smart HTTP push tests
`t5555-http-smart-common.sh`	Protocol version tests (v0, v1, v2)
`t5560-http-backend-noserver.sh`	HTTP backend without server (unit-like)
`t5561-http-backend.sh`	Full HTTP backend integration
`t5562-http-backend-content-length.sh`	CRITICAL: Content-Length and gzip tests
`t5539-fetch-http-shallow.sh`	Shallow clone over HTTP
`t5559-http-fetch-smart-http2.sh`	HTTP/2 protocol tests
`t556x_common`	Shared test helpers
`lib-httpd.sh`	Apache httpd test infrastructure

What Git Tests:

Reference discovery (info/refs?service=git-upload-pack)
Smart service responses (git-upload-pack, git-receive-pack)
Content-Type validation (application/x-git-upload-pack-advertisement)
Protocol version negotiation (v0, v1, v2 via Git-Protocol header)
Authentication (Basic auth, credential helpers)
Redirects (301, 302)
Chunked transfer encoding
Large fetch-pack requests
Cookie handling
Error conditions (invalid Content-Type, server errors)

2. Content-Encoding Handling (Critical!)

From git's http-backend.c:

const char *encoding = getenv("HTTP_CONTENT_ENCODING");
if (encoding && (!strcmp(encoding, "gzip") || !strcmp(encoding, "x-gzip")))
    gzipped_request = 1;

From t5562-http-backend-content-length.sh:

# Tests gzip-encoded request bodies
test_expect_success GZIP 'fetch gzipped' '
    test_env HTTP_CONTENT_ENCODING="gzip" test_http_env upload fetch_body.gz &&
    verify_http_result "200 OK"
'

test_expect_success GZIP 'push gzipped' '
    test_env HTTP_CONTENT_ENCODING="gzip" test_http_env receive push_body.gz &&
    verify_http_result "200 OK"
'

MUST TEST:

Content-Encoding: gzip - Most common, git client default
Content-Encoding: x-gzip - Alternative gzip identifier
Content-Encoding: identity - Explicit no encoding
No Content-Encoding header - Implicit identity
Truncated gzip streams (error handling)
Empty gzip bodies (error handling)

3. Protocol Versions

Version 0 (Legacy):

No Git-Protocol header
Capabilities in first ref line after NUL byte
symref=HEAD:refs/heads/main in capability line

Version 1:

Git-Protocol: version=1 header
Response starts with version 1 pkt-line
Same capability format as v0

Version 2:

Git-Protocol: version=2 header
Response starts with version 2 pkt-line
Capabilities as separate lines (agent, ls-refs, fetch, etc.)
Stateless by default
Commands: ls-refs, fetch, server-option, object-info

From t5555-http-smart-common.sh:

test_expect_success 'git upload-pack --advertise-refs: v2' '
    cat >expect <<-EOF &&
    version 2
    agent=FAKE
    ls-refs=unborn
    fetch=shallow wait-for-done
    server-option
    object-format=$(test_oid algo)
    0000
    EOF
    GIT_PROTOCOL=version=2 git upload-pack --advertise-refs . >out
'

4. HTTP Protocol Specification Requirements

From git-scm.com/docs/http-protocol:

Reference Discovery:

GET $GIT_URL/info/refs?service=git-upload-pack
Response Content-Type: application/x-git-upload-pack-advertisement
Response must start with # service=git-upload-pack pkt-line
Must include capability declarations on first ref

Smart Service (upload-pack):

POST $GIT_URL/git-upload-pack
Request Content-Type: application/x-git-upload-pack-request
Response Content-Type: application/x-git-upload-pack-result
Cache-Control: no-cache, max-age=0, must-revalidate

Smart Service (receive-pack):

POST $GIT_URL/git-receive-pack
Request Content-Type: application/x-git-receive-pack-request
Response Content-Type: application/x-git-receive-pack-result

Error Handling:

403 Forbidden for disabled services
404 Not Found for missing repositories
Must NOT return 200 OK for non-existent repos

5. Edge Cases That MUST Be Tested

Content-Encoding:

Gzip-encoded POST bodies (clone, fetch, push)
x-gzip encoding variant
Truncated gzip streams
Empty gzip bodies
Mixed encoding (gzip request, identity response)

Protocol Negotiation:

Client requests v2, server supports v2
Client requests v2, server only supports v0 (fallback)
Client requests v1
No version specified (default to v0)

Repository States:

Empty repository (no refs)
Repository with many refs (>2000 tags test in git)
Shallow repository
Repository with large objects
Repository with submodules

Operations:

Clone (full)
Clone (shallow with --depth)
Clone (partial with --filter)
Fetch (incremental)
Fetch (shallow-since, shallow-exclude)
Push (standard)
Push (atomic)
Push (chunked encoding for large packs)
ls-remote

HTTP Edge Cases:

Chunked Transfer-Encoding
Large Content-Length values
CONTENT_LENGTH overflow (ssize_t boundary)
Empty CONTENT_LENGTH
HTTP/1.0 vs HTTP/1.1
HTTP/2 (if supported)
Keep-alive connections
Redirects (301, 302)
Path traversal attempts (security)

Authentication:

No auth (public repo)
Basic auth required
Auth only for push
Auth only for objects (half-auth)
Invalid credentials
Expired credentials

6. Other Implementation Approaches

Gitea:

Uses real git client integration tests
Tests HTTP clone/push with actual git commands
Tests LFS integration
Tests SSH and HTTP in parallel

GitLab:

Extensive RSpec tests for git HTTP
Tests authentication edge cases
Tests project path variations (.git suffix handling)
Tests blocked users, expired passwords
Tests namespace/project creation via push

7. Recommendations for Test Approach

Priority 1 - Critical (Would have caught our bug):

Content-Encoding: gzip for POST bodies
Content-Encoding: x-gzip variant
Truncated/malformed gzip handling

Priority 2 - Protocol Compliance:

Protocol version negotiation (v0, v1, v2)
Correct Content-Type headers
Correct pkt-line format
Capability advertisement

Priority 3 - Operations:

Clone (basic)
Clone (shallow)
Fetch (incremental)
Push (basic)
ls-remote

Priority 4 - Edge Cases:

Empty repository
Large repository (many refs)
Chunked encoding
HTTP/2 (if applicable)

Test Infrastructure:

Use real git client (not mocked requests)
Test with multiple git versions if possible
Use TestRelay fixture for isolation
Consider adapting git's t5562 test approach

8. Specification References

Document	URL
HTTP Protocol	https://git-scm.com/docs/http-protocol
Pack Protocol	https://git-scm.com/docs/pack-protocol
Protocol v2	https://git-scm.com/docs/protocol-v2
Git Test Suite	https://github.com/git/git/tree/master/t
RFC 2616 (HTTP/1.1)	https://www.ietf.org/rfc/rfc2616.txt
RFC 1952 (GZIP)	https://www.ietf.org/rfc/rfc1952.txt

9. Key Gotchas from Other Implementations

Git's gzip handling: Uses HTTP_CONTENT_ENCODING env var, supports both "gzip" and "x-gzip"
Protocol fallback: Client may request v2 but server can respond with v0 - client must handle gracefully
Half-auth: Some servers require auth only for objects, not refs - complex to test
Path normalization: Must handle /repo.git vs /repo vs /repo.git/ consistently
Chunked encoding: Large pushes use chunked encoding when http.postbuffer is small
Empty repos: Must handle repos with no refs (capabilities^{} line)
Peeled refs: Annotated tags must show both tag and peeled object
Symref: HEAD symref must be communicated in capabilities

2026-01-23 [Session 10:00] - Plan Refinement

Reviewed: MVP implementation in bb46 worktree
Validated: Test passes (cargo test --test git_http_protocol ~1.5s)
Refined: Test matrix reduced from 95 to 65 scenarios
- Removed: HTTP Basic auth, half-auth, credential helpers, submodules (not applicable to GRASP)
- Kept: All content-encoding, protocol version, core operations, error handling tests
Designed: GitRepoFixture struct for fixture optimization
- Estimated performance improvement: 100s → 15s for full suite
Created: Detailed implementation phases 6-10
- Phase 6: P0 Content-Encoding tests
- Phase 7: Protocol version tests (requires pkt-line parser)
- Phase 8: Core operations tests (requires GitRepoFixture)
- Phase 9: Error handling tests
- Phase 10: P1 tests and polish
Defined: Success metrics and definition of done
Next: Merge MVP to main, then implement Phase 6

33 KiB Raw Blame History

Git HTTP Smart Protocol Comprehensive Test Suite

Problem

Plan

Completed Phases

Refined Implementation Plan (Post-MVP)

Refined Test Matrix (65 scenarios, reduced from 95)

P0 - Critical (15 tests)

P1 - Important (25 tests)

P2 - Nice to Have (25 tests)

Fixture Optimization: GitRepoFixture

Remaining Implementation Phases

Infrastructure Needed

Success Metrics

Definition of Done

Vision

Architecture

Integration Pattern

MVP Approach

Progress

2026-01-23 [Session End] - KNOWLEDGE PRESERVED & READY FOR IMPLEMENTATION

2026-01-21 [Session 18:00] - MVP COMPLETE ✅

2026-01-21 [Session 17:30]

2026-01-21 [Session 17:15] - REVISED DECISION

2026-01-21 [Session 16:56]

2026-01-21 [Session 16:45] - SUPERSEDED BY SESSION 17:15

2026-01-21 [Session 16:27]

2026-01-21 [Session 17:30]

2026-01-21 [Session 16:00]

2026-01-21 [Session 14:30]

2026-01-21 [Session Start]

Notes

General

Key Learnings from MVP (2026-01-23)

Preserved Knowledge (2026-01-23)

Next Session Quick Start

Research Report (Phase 1)

1. Git Project's Own Test Suite

2. Content-Encoding Handling (Critical!)

3. Protocol Versions

4. HTTP Protocol Specification Requirements

5. Edge Cases That MUST Be Tested

6. Other Implementation Approaches

7. Recommendations for Test Approach

8. Specification References

9. Key Gotchas from Other Implementations

2026-01-23 [Session 10:00] - Plan Refinement

33 KiB

Raw Blame History

Fixture Optimization: `GitRepoFixture`