poimen-memory/tasks/M7.4-git-repo-connector.md

M7.4 — Git repository connector

Field	Value
Phase	M7 — Source connectors
Size	M — 1–3 days
Status	⬜ Not started
Flags	—
Spec	inlined below
Blocks	M7.10
Depends	M7.1

Goal

Implement a SourceConnector that clones or pulls a git repository and exposes its documents for ingestion — so documentation repos, wikis, and runbook collections tracked in git become searchable through the memory service.

Facts (inlined — no spec read needed)

Many knowledge sources are already in git: internal wikis, Forgejo repos, infrastructure documentation, README collections. This connector pulls them without requiring manual export.

Configuration:

connectors:
  - kind: git_repo
    name: infra-docs
    config:
      repo_url: https://forgejo.riotpiao.com/rock/homelab-docs.git
      branch: main
      clone_dir: /tmp/mem-connectors/infra-docs  # local checkout
      paths: [docs/, runbooks/]        # only scan these dirs
      extensions: [md, txt, rst]
      auth_secret: forgejo-token       # k8s secret for private repos
      sync_depth: 1                    # shallow clone

Clone-then-walk model. The connector clones (or pulls) the repo to a local directory, then walks the filesystem like the Obsidian connector. This reuses DocCorpusSource internals and avoids git-specific content access APIs.

Change detection uses git. git diff --name-only HEAD@{1}..HEAD after a pull tells the sync framework exactly which files changed — more efficient than re-hashing every file.

Branch tracking. Connector watches one branch. Branch changes (main → prod) require config update. No multi-branch support — each branch is a separate connector instance.

Steps

1. Implement GitRepoConnector in mem-ingest/src/connectors/git_repo.rs.
2. On first sync: git clone --depth N --branch B <url> <clone_dir>.
3. On subsequent syncs: git -C <clone_dir> pull --ff-only.
4. list_documents() — walk clone_dir filtered by paths and extensions, compute sha256 per file.
5. fetch_document() — read file from clone_dir, return content with git metadata (last commit sha, author, date for that file via git log -1).
6. health_check() — verify clone_dir is a valid git repo, check remote connectivity via git ls-remote.
7. Handle auth for private repos (token from k8s secret → git credential helper or URL embedding).
8. Register "git_repo" kind in connector registry factory.
9. source_type() returns Reference.

Acceptance

• Clone + walk produces correct document list for fixture repo.
• Pull detects changed files without re-processing unchanged ones.
• Path filtering limits scan to configured subdirectories.
• Private repo auth works (token in URL or credential helper).
• Health check distinguishes: valid repo, invalid remote, auth failure.

Verify

Harness: fixture git repo created in temp dir with known content.

Integration test — tests/it_git_repo_connector.rs:

1. a1_clone_and_list — init fixture repo, connector clones it; assert list_documents() returns expected files.
2. a2_path_filtering — fixture with docs/ and src/; config says paths: [docs/]; assert only docs/ files returned.
3. a3_extension_filtering — fixture with .md, .rs, .txt; assert only .md/.txt.
4. a4_fetch_returns_content — fetch a doc; assert content matches fixture file.
5. a5_pull_detects_changes — add a commit to fixture repo; pull; assert changed file appears in list with new hash.
6. a6_health_check_valid_repo — valid clone_dir; assert reachable: true.
7. a7_health_check_no_clone — no clone_dir; assert reachable: false with message indicating clone needed.
8. a8_shallow_clone — assert clone depth matches config (git rev-list --count HEAD).
9. a9_config_from_yaml — parse connector from YAML; assert fields match.

Command: cargo test --test it_git_repo_connector

False pass:

• Testing with a local repo path instead of a clone. The clone/pull machinery is the whole point — a connector that reads a pre-existing checkout is just the Obsidian connector.
• Not testing pull-after-change. First sync always works; second sync is where change detection matters.

Traps

• Running git clone on every sync. Check if clone_dir already has a valid checkout first; clone only on first run.
• Not cleaning up failed clones. A partial clone leaves a directory that is neither valid nor absent — subsequent runs fail on both clone (dir exists) and pull (not a repo).
• Force-push upstream breaks --ff-only. Detect non-fast-forward, delete clone_dir, re-clone. Log a warning — this means all docs are re-processed.
• Symlinks across the repo boundary. walkdir follows symlinks by default; a symlink to /etc/passwd is a real concern in a cluster-wide service.

---

Background: DESIGN.md — source connectors section