fix(agent-pod): force judge to actually call tools instead of narrating

Observed live: phase-judge (on homelab-reasoning) wrote a full page of
'I should check X, then Y' reasoning, declared VERDICT: PASS, and showed
the touch command as a fenced code block in its own text -- never ran
git diff, never wrote the result file, never touched the sentinel.
Coordinator timed out waiting on a file that was never going to appear.
This commit is contained in:
Story Crater Bot
2026-08-19 17:13:49 -07:00
parent d1b650041e
commit e7da32843e
+20 -4
View File
@@ -107,6 +107,22 @@ data:
"Read and follow ~/.pi/agent/skills/karpathy-guidelines/SKILL.md and " +
"~/.pi/agent/skills/caveman/SKILL.md as hard rules for this entire task, before anything else. ";
// judge (routed to homelab-reasoning) has been observed narrating an
// entire review in prose -- "I should run git diff, then check X..." --
// and then writing a verdict based on that narration without ever calling
// a real tool. Live example: a phase-judge call produced a page of
// "I would check..." reasoning, declared VERDICT: PASS, and showed the
// touch command as a fenced code block IN ITS OWN TEXT rather than
// executing it. Coordinator just timed out waiting on a sentinel that was
// never going to appear, since nothing was ever actually run. Spelled out
// explicitly since "use the judge skill" alone apparently isn't enough to
// rule this out.
const REQUIRE_REAL_TOOL_CALLS =
"Do not narrate what you would check -- actually run the commands via a real tool call and read their real " +
"output before writing anything. A verdict based on describing checks instead of executing them is invalid. " +
"Writing the verdict file and touching the sentinel are themselves tool calls you must execute, not text to " +
"display in your response. ";
function parseVerdictLine(text, label) {
if (!text) return null;
const re = new RegExp(`${label}:\\s*(\\w+)`, "i");
@@ -331,9 +347,9 @@ data:
function judgePrompt(task, baseBranch, attempt, cwd) {
return (
`Use the judge skill to review the diff against ${baseBranch}...HEAD for task ${task}. Write your verdict to ` +
`${path.join(cwd, `.task-result-${task}`)} as a single "VERDICT: PASS" or "VERDICT: FAIL" line plus one line of rationale, then ` +
`run: touch ${path.join(cwd, `.stage-done-${task}-judge-${attempt}`)}`
`${REQUIRE_REAL_TOOL_CALLS}Use the judge skill to review the diff against ${baseBranch}...HEAD for task ${task}. ` +
`Write your verdict to ${path.join(cwd, `.task-result-${task}`)} as a single "VERDICT: PASS" or "VERDICT: FAIL" ` +
`line plus one line of rationale, then run: touch ${path.join(cwd, `.stage-done-${task}-judge-${attempt}`)}`
);
}
@@ -694,7 +710,7 @@ data:
cwd,
repoId,
"judge",
`Use the judge skill to review the full phase diff for phase ${phaseLabel} against ` +
`${REQUIRE_REAL_TOOL_CALLS}Use the judge skill to review the full phase diff for phase ${phaseLabel} against ` +
`${baseBranch}...HEAD (covers every task in this phase: ${[...phaseTaskIds].join(", ")}). Every ` +
`individual task already passed its own judge review -- your job here is different: confirm the ` +
`tasks integrate correctly as one coherent narrative, and that real integration tests (not just ` +