docs: Add M7 source connectors (10 tasks), M3.5.10 auth integration, remove Kong refs
Build and Push / Test (push) Successful in 3m36s
Build and Push / Build and push image (push) Successful in 20s

- M7.1-M7.10: Extensible SourceConnector trait, Obsidian/paperless/git/S3
  connectors, sync framework, CLI, HTTP endpoints, health monitoring, gate
- M3.5.10: Auth integration with Authentik OIDC → Vault token validation
- DESIGN.md: Add source connectors architecture, update auth to
  Authentik/Vault (Kong removed from cluster)
- INDEX.md: 75 tasks, 11 gates
- Fix all Kong references in M3.5.1 task
This commit is contained in:
Story Crater Bot
2026-08-26 16:56:39 -07:00
parent c8d754b0ba
commit 71ecf482e7
14 changed files with 1560 additions and 13 deletions
+118
View File
@@ -0,0 +1,118 @@
# M7.4 — Git repository connector
| Field | Value |
|---|---|
| Phase | M7 — Source connectors |
| Size | M — 13 days |
| Status | ⬜ Not started |
| Flags | — |
| Spec | inlined below |
| Blocks | M7.10 |
| Depends | M7.1 |
## Goal
Implement a `SourceConnector` that clones or pulls a git repository and exposes
its documents for ingestion — so documentation repos, wikis, and runbook
collections tracked in git become searchable through the memory service.
## Facts (inlined — no spec read needed)
Many knowledge sources are already in git: internal wikis, Forgejo repos,
infrastructure documentation, README collections. This connector pulls them
without requiring manual export.
**Configuration:**
```yaml
connectors:
- kind: git_repo
name: infra-docs
config:
repo_url: https://forgejo.riotpiao.com/rock/homelab-docs.git
branch: main
clone_dir: /tmp/mem-connectors/infra-docs # local checkout
paths: [docs/, runbooks/] # only scan these dirs
extensions: [md, txt, rst]
auth_secret: forgejo-token # k8s secret for private repos
sync_depth: 1 # shallow clone
```
**Clone-then-walk model.** The connector clones (or pulls) the repo to a local
directory, then walks the filesystem like the Obsidian connector. This reuses
`DocCorpusSource` internals and avoids git-specific content access APIs.
**Change detection uses git.** `git diff --name-only HEAD@{1}..HEAD` after a pull
tells the sync framework exactly which files changed — more efficient than
re-hashing every file.
**Branch tracking.** Connector watches one branch. Branch changes (main → prod)
require config update. No multi-branch support — each branch is a separate
connector instance.
## Steps
1. Implement `GitRepoConnector` in `mem-ingest/src/connectors/git_repo.rs`.
2. On first sync: `git clone --depth N --branch B <url> <clone_dir>`.
3. On subsequent syncs: `git -C <clone_dir> pull --ff-only`.
4. `list_documents()` — walk `clone_dir` filtered by `paths` and `extensions`,
compute sha256 per file.
5. `fetch_document()` — read file from `clone_dir`, return content with git
metadata (last commit sha, author, date for that file via `git log -1`).
6. `health_check()` — verify `clone_dir` is a valid git repo, check remote
connectivity via `git ls-remote`.
7. Handle auth for private repos (token from k8s secret → git credential helper
or URL embedding).
8. Register `"git_repo"` kind in connector registry factory.
9. `source_type()` returns `Reference`.
## Acceptance
- Clone + walk produces correct document list for fixture repo.
- Pull detects changed files without re-processing unchanged ones.
- Path filtering limits scan to configured subdirectories.
- Private repo auth works (token in URL or credential helper).
- Health check distinguishes: valid repo, invalid remote, auth failure.
## Verify
**Harness:** fixture git repo created in temp dir with known content.
**Integration test**`tests/it_git_repo_connector.rs`:
1. `a1_clone_and_list` — init fixture repo, connector clones it; assert
`list_documents()` returns expected files.
2. `a2_path_filtering` — fixture with `docs/` and `src/`; config says `paths: [docs/]`;
assert only `docs/` files returned.
3. `a3_extension_filtering` — fixture with .md, .rs, .txt; assert only .md/.txt.
4. `a4_fetch_returns_content` — fetch a doc; assert content matches fixture file.
5. `a5_pull_detects_changes` — add a commit to fixture repo; pull; assert changed
file appears in list with new hash.
6. `a6_health_check_valid_repo` — valid clone_dir; assert `reachable: true`.
7. `a7_health_check_no_clone` — no clone_dir; assert `reachable: false` with
message indicating clone needed.
8. `a8_shallow_clone` — assert clone depth matches config (`git rev-list --count HEAD`).
9. `a9_config_from_yaml` — parse connector from YAML; assert fields match.
**Command:** `cargo test --test it_git_repo_connector`
**False pass:**
- Testing with a local repo path instead of a clone. The clone/pull machinery
is the whole point — a connector that reads a pre-existing checkout is just
the Obsidian connector.
- Not testing pull-after-change. First sync always works; second sync is where
change detection matters.
## Traps
- Running `git clone` on every sync. Check if `clone_dir` already has a valid
checkout first; clone only on first run.
- Not cleaning up failed clones. A partial clone leaves a directory that is
neither valid nor absent — subsequent runs fail on both clone (dir exists)
and pull (not a repo).
- Force-push upstream breaks `--ff-only`. Detect non-fast-forward, delete
`clone_dir`, re-clone. Log a warning — this means all docs are re-processed.
- Symlinks across the repo boundary. `walkdir` follows symlinks by default;
a symlink to `/etc/passwd` is a real concern in a cluster-wide service.
---
Background: [DESIGN.md](../DESIGN.md) — source connectors section