chore(llm-serving): park clef-serve to free a V100 for candle-vllm #91

2026-11-01T02:40:40Z

rock commented

2026-11-01 02:40:40 +00:00

Blocker this fixes

candle-vllm cannot start anywhere. worker-1 has four V100-32GB and all four are allocated.

Workload	GPU	Serves	In ingest path
`coder-predictor`	2	`coder`, `reasoning`	no
`reasoning-predictor`	1	`reasoning`	yes
`clef-serve`	1	nothing	no
free	0

clef-serve is the only GPU consumer in the namespace that is not vLLM, so it is the only card that can be released while still keeping vLLM healthy as instructed. Measured over its last 5,000 log lines since start (2026-10-30T06:23:40Z): 4,994 GET /health probes, 2 GET /v1/models, 1 GET /v1/completions, 1 GET /v1/chat/completions, 1 GET /docs, 1 GET /. No successful inference: the four inference-path requests are GET and those routes answer POST. Nothing in git selects its Service.

Teardown plan — staged, one gate per stage

Stage	Action	Where	Gate before the next stage	Rollback
0	Park `clef-serve` (`replicas: 0`) — this PR	git	allocated drops to 3, one card schedulable	`replicas: 1`
1	memory runs candle-vllm, Qwen3-30B-A3B 64K, 1 card	memory	their readiness command plus a real completion served from that pod	none needed, vLLM untouched
2	Cut ingest client to candle-vllm	poimen-memory config	one ingest run reaches and returns from the LLM	repoint `LLM_CHAT_BASE` back
3	Remove `reasoning.yaml` + `coder.yaml` from `kustomization.yaml`, merge, then `kubectl delete`	git + cluster	2 InferenceServices gone, candle still serving	re-add to kustomization
4	Coder-Next TP=2 on 2 cards, restore `clef-serve` on the 4th	memory + this repo	3 cards used, 1 free	park clef again

GPU accounting through the plan

After stage	candle	vLLM reasoning	vLLM coder	clef	free
0	0	1	2	0	1
1	1	1	2	0	0
3	1	0	0	0	3
4	3	0	0	1	0

Stage 0 and stage 3 both release exactly one card at a time relative to what candle needs, so ingest never has to stop.

Hard constraints on how the teardown may happen

It must be a merged git change. llm-serving runs selfHeal: true, so kubectl scale and kubectl delete are reverted within minutes. This is not a preference, it is measured: a live patch to the ingress policy and to the served-model-name both reverted before their new pods finished loading.
prune: false. After removing a file from kustomization.yaml, ArgoCD stops managing the object but does not delete it. Stage 3 therefore needs an explicit kubectl delete after the merge, or the pods keep holding cards while no longer being reconciled.
No stage is gated on entity persistence. Ingest currently reaches status: done and extraction runs, but every save fails with no unique or exclusion constraint matching the ON CONFLICT specification because the partial unique index on memory_entity is missing while migration 017 is recorded applied. If the handoff gate used "entity count rose" it could never pass. The gate is that the extraction LLM call succeeds and returns.

What breaks, and when

pi loses its coding model at stage 3. ~/.pi/agent/models.json defines coder-vllm at http://localhost:8000/v1, a kubectl port-forward onto coder-predictor. Deleting that InferenceService ends the port-forward and pi has no coder model until candle-vllm serves one. Expected, given vLLM is retired, but it is the day-to-day model, so the timing matters.
clef-flash is offline from stage 0 to stage 4. If clef-finetune needs it live, stage 0 must wait or clef must move to networked hardware.
reasoning stays ambiguous only until stage 3. After the vLLM teardown only one backend remains, so the duplicate name resolves itself.

Open questions sent to the memory terminal

GO or NO-GO on parking clef-serve; if NO, which workload instead.
Phase 0 needs 1 card or 2.
candle-vllm's k8s Service name, port, and served model id. Recommended: serve reasoning-long so the client name survives the handoff.
The exact readiness command and expected output I must run before stage 3.
Whether clef-flash must be live.

Unrelated bug found while checking: poimen-memory/k8s/app/config.yaml is modified but uncommitted with LLM_CHAT_BASE: http://reasoning-long-predictor.llm-serving.svc.cluster.local — no such Service exists. Committing and applying it would kill ingest on DNS failure. Live ConfigMap still points at reasoning-predictor, which is why ingest works today.

## Blocker this fixes `candle-vllm` cannot start anywhere. `worker-1` has four V100-32GB and all four are allocated. | Workload | GPU | Serves | In ingest path | |---|---|---|---| | `coder-predictor` | 2 | `coder`, `reasoning` | no | | `reasoning-predictor` | 1 | `reasoning` | **yes** | | `clef-serve` | 1 | nothing | no | | **free** | **0** | | | `clef-serve` is the only GPU consumer in the namespace that is not vLLM, so it is the only card that can be released while still keeping vLLM healthy as instructed. Measured over its last 5,000 log lines since start (`2026-10-30T06:23:40Z`): 4,994 `GET /health` probes, 2 `GET /v1/models`, 1 `GET /v1/completions`, 1 `GET /v1/chat/completions`, 1 `GET /docs`, 1 `GET /`. No successful inference: the four inference-path requests are GET and those routes answer POST. Nothing in git selects its Service. ## Teardown plan — staged, one gate per stage | Stage | Action | Where | Gate before the next stage | Rollback | |---|---|---|---|---| | **0** | Park `clef-serve` (`replicas: 0`) — **this PR** | git | allocated drops to 3, one card schedulable | `replicas: 1` | | **1** | memory runs candle-vllm, Qwen3-30B-A3B 64K, 1 card | memory | their readiness command **plus** a real completion served from that pod | none needed, vLLM untouched | | **2** | Cut ingest client to candle-vllm | poimen-memory config | one ingest run reaches and returns from the LLM | repoint `LLM_CHAT_BASE` back | | **3** | Remove `reasoning.yaml` + `coder.yaml` from `kustomization.yaml`, merge, then `kubectl delete` | git + cluster | 2 InferenceServices gone, candle still serving | re-add to kustomization | | **4** | Coder-Next TP=2 on 2 cards, restore `clef-serve` on the 4th | memory + this repo | 3 cards used, 1 free | park clef again | ### GPU accounting through the plan | After stage | candle | vLLM reasoning | vLLM coder | clef | free | |---|---|---|---|---|---| | 0 | 0 | 1 | 2 | 0 | **1** | | 1 | 1 | 1 | 2 | 0 | 0 | | 3 | 1 | 0 | 0 | 0 | **3** | | 4 | 3 | 0 | 0 | 1 | 0 | Stage 0 and stage 3 both release exactly one card at a time relative to what candle needs, so ingest never has to stop. ## Hard constraints on how the teardown may happen - **It must be a merged git change.** `llm-serving` runs `selfHeal: true`, so `kubectl scale` and `kubectl delete` are reverted within minutes. This is not a preference, it is measured: a live patch to the ingress policy and to the served-model-name both reverted before their new pods finished loading. - **`prune: false`.** After removing a file from `kustomization.yaml`, ArgoCD stops managing the object but does not delete it. Stage 3 therefore needs an explicit `kubectl delete` *after* the merge, or the pods keep holding cards while no longer being reconciled. - **No stage is gated on entity persistence.** Ingest currently reaches `status: done` and extraction runs, but every save fails with `no unique or exclusion constraint matching the ON CONFLICT specification` because the partial unique index on `memory_entity` is missing while migration 017 is recorded applied. If the handoff gate used "entity count rose" it could never pass. The gate is that the extraction LLM call succeeds and returns. ## What breaks, and when - **pi loses its coding model at stage 3.** `~/.pi/agent/models.json` defines `coder-vllm` at `http://localhost:8000/v1`, a `kubectl port-forward` onto `coder-predictor`. Deleting that InferenceService ends the port-forward and pi has no coder model until candle-vllm serves one. Expected, given vLLM is retired, but it is the day-to-day model, so the timing matters. - **clef-flash is offline from stage 0 to stage 4.** If `clef-finetune` needs it live, stage 0 must wait or clef must move to networked hardware. - **`reasoning` stays ambiguous only until stage 3.** After the vLLM teardown only one backend remains, so the duplicate name resolves itself. ## Open questions sent to the memory terminal 1. GO or NO-GO on parking `clef-serve`; if NO, which workload instead. 2. Phase 0 needs 1 card or 2. 3. candle-vllm's k8s Service name, port, and served model id. Recommended: serve `reasoning-long` so the client name survives the handoff. 4. The exact readiness command and expected output I must run before stage 3. 5. Whether clef-flash must be live. Unrelated bug found while checking: `poimen-memory/k8s/app/config.yaml` is modified but uncommitted with `LLM_CHAT_BASE: http://reasoning-long-predictor.llm-serving.svc.cluster.local` — no such Service exists. Committing and applying it would kill ingest on DNS failure. Live ConfigMap still points at `reasoning-predictor`, which is why ingest works today.

rock added 1 commit 2026-11-01 02:40:57 +00:00

chore(llm-serving): park clef-serve to free a V100 for candle-vllm d821718375

candle-vllm cannot start. worker-1 has four V100-32GB and all four are
allocated: coder 2, reasoning 1, clef-serve 1. Free is 0, so a new GPU pod
sits Pending no matter what it is.

This pod is the only GPU consumer in the namespace that is not vLLM, so it is
the only card that can be released without breaking the instruction to keep
vLLM healthy until candle-vllm answers. Ingest keeps running on
reasoning-predictor, and pi keeps its coder model on coder-predictor.

Measured activity over the last 5,000 log lines, since it started on
2026-10-30T06:23:40Z:

  4,994  GET /health        (kubelet probes)
      2  GET /v1/models
      1  GET /v1/completions
      1  GET /v1/chat/completions
      1  GET /docs
      1  GET /

No successful inference. The four inference-path requests are GET, and these
routes answer POST, so they cannot have produced a completion. It held a card
for a day to answer liveness probes, which is the same shape as the comfyui
retirement. Nothing in git selects this Service.

Reversible: set replicas back to 1. It should return once the rollout settles,
because 30B-A3B takes 1 card and Coder-Next TP=2 takes 2, which leaves card 4
for this workload.

Not applied. ArgoCD runs selfHeal for llm-serving, so a live scale to 0 is
reverted within minutes; this needs the merge to stick. Apply through
kustomize, not `kubectl apply -f clef-serve.yaml`: the Deployment mounts the
generated ConfigMap by its hashed name, and a raw apply drops the hash.

rock force-pushed chore/park-clef-for-candle from 44fbf071e1 to d821718375

2026-11-01 02:40:57 +00:00

Compare

rock referenced this pull request

2026-11-01 02:43:20 +00:00

feat(llm-serving): poimen ingress, 256K coder, and a stable name for the candle-vllm handoff #90

You are not authorized to merge this pull request.

This pull request can be merged automatically.

This branch is out-of-date with the base branch

View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.

git fetch -u origin chore/park-clef-for-candle:chore/park-clef-for-candle

git checkout chore/park-clef-for-candle

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: riotpiao-poimen/homelab#91