poimen-memory/tasks/M7.4-git-repo-connector.md

# M7.4 — Git repository connector

| Field | Value |
|---|---|
| Phase | M7 — Source connectors |
| Size | M — 1–3 days |
| Status | ⬜ Not started |
| Flags | — |
| Spec | inlined below |
| Blocks | M7.10 |
| Depends | M7.1 |

## Goal

Implement a `SourceConnector` that clones or pulls a git repository and exposes
its documents for ingestion — so documentation repos, wikis, and runbook
collections tracked in git become searchable through the memory service.

## Facts (inlined — no spec read needed)

Many knowledge sources are already in git: internal wikis, Forgejo repos,
infrastructure documentation, README collections. This connector pulls them
without requiring manual export.

**Configuration:**
```yaml
connectors:
  - kind: git_repo
    name: infra-docs
    config:
      repo_url: https://forgejo.riotpiao.com/rock/homelab-docs.git
      branch: main
      clone_dir: /tmp/mem-connectors/infra-docs  # local checkout
      paths: [docs/, runbooks/]        # only scan these dirs
      extensions: [md, txt, rst]
      auth_secret: forgejo-token       # k8s secret for private repos
      sync_depth: 1                    # shallow clone
```

**Clone-then-walk model.** The connector clones (or pulls) the repo to a local
directory, then walks the filesystem like the Obsidian connector. This reuses
`DocCorpusSource` internals and avoids git-specific content access APIs.

**Change detection uses git.** `git diff --name-only HEAD@{1}..HEAD` after a pull
tells the sync framework exactly which files changed — more efficient than
re-hashing every file.

**Branch tracking.** Connector watches one branch. Branch changes (main → prod)
require config update. No multi-branch support — each branch is a separate
connector instance.

## Steps

1. Implement `GitRepoConnector` in `mem-ingest/src/connectors/git_repo.rs`.
2. On first sync: `git clone --depth N --branch B <url> <clone_dir>`.
3. On subsequent syncs: `git -C <clone_dir> pull --ff-only`.
4. `list_documents()` — walk `clone_dir` filtered by `paths` and `extensions`,
   compute sha256 per file.
5. `fetch_document()` — read file from `clone_dir`, return content with git
   metadata (last commit sha, author, date for that file via `git log -1`).
6. `health_check()` — verify `clone_dir` is a valid git repo, check remote
   connectivity via `git ls-remote`.
7. Handle auth for private repos (token from k8s secret → git credential helper
   or URL embedding).
8. Register `"git_repo"` kind in connector registry factory.
9. `source_type()` returns `Reference`.

## Acceptance

- Clone + walk produces correct document list for fixture repo.
- Pull detects changed files without re-processing unchanged ones.
- Path filtering limits scan to configured subdirectories.
- Private repo auth works (token in URL or credential helper).
- Health check distinguishes: valid repo, invalid remote, auth failure.

## Verify

**Harness:** fixture git repo created in temp dir with known content.

**Integration test** — `tests/it_git_repo_connector.rs`:
1. `a1_clone_and_list` — init fixture repo, connector clones it; assert
   `list_documents()` returns expected files.
2. `a2_path_filtering` — fixture with `docs/` and `src/`; config says `paths: [docs/]`;
   assert only `docs/` files returned.
3. `a3_extension_filtering` — fixture with .md, .rs, .txt; assert only .md/.txt.
4. `a4_fetch_returns_content` — fetch a doc; assert content matches fixture file.
5. `a5_pull_detects_changes` — add a commit to fixture repo; pull; assert changed
   file appears in list with new hash.
6. `a6_health_check_valid_repo` — valid clone_dir; assert `reachable: true`.
7. `a7_health_check_no_clone` — no clone_dir; assert `reachable: false` with
   message indicating clone needed.
8. `a8_shallow_clone` — assert clone depth matches config (`git rev-list --count HEAD`).
9. `a9_config_from_yaml` — parse connector from YAML; assert fields match.

**Command:** `cargo test --test it_git_repo_connector`

**False pass:**
- Testing with a local repo path instead of a clone. The clone/pull machinery
  is the whole point — a connector that reads a pre-existing checkout is just
  the Obsidian connector.
- Not testing pull-after-change. First sync always works; second sync is where
  change detection matters.

## Traps

- Running `git clone` on every sync. Check if `clone_dir` already has a valid
  checkout first; clone only on first run.
- Not cleaning up failed clones. A partial clone leaves a directory that is
  neither valid nor absent — subsequent runs fail on both clone (dir exists)
  and pull (not a repo).
- Force-push upstream breaks `--ff-only`. Detect non-fast-forward, delete
  `clone_dir`, re-clone. Log a warning — this means all docs are re-processed.
- Symlinks across the repo boundary. `walkdir` follows symlinks by default;
  a symlink to `/etc/passwd` is a real concern in a cluster-wide service.

---

Background: [DESIGN.md](../DESIGN.md) — source connectors section