Skip to content

Write eval context to file before passing to Claude#488

Open
jdcmsd wants to merge 1 commit into
mainfrom
fix-eval-stdin
Open

Write eval context to file before passing to Claude#488
jdcmsd wants to merge 1 commit into
mainfrom
fix-eval-stdin

Conversation

@jdcmsd

@jdcmsd jdcmsd commented Jun 16, 2026

Copy link
Copy Markdown
Member

Claude CLI timed out waiting for stdin because run_eval.py was still fetching from GitHub. Write output to a temp file first, then feed it to Claude via stdin redirect.

@jdcmsd jdcmsd requested a review from a team as a code owner June 16, 2026 16:19
@github-actions

Copy link
Copy Markdown

❌ Issue Linking Required

This PR could not be linked to an issue. All PRs must be linked to an issue for tracking purposes.

How to fix this:

Option 1: Add keyword to PR body (Recommended - auto-removes this comment)
Edit this PR description and add one of these lines:

  • This PR fixes #123 or Fixes: #123

  • This PR closes #123 or Closes: #123

  • This PR resolves #123 or Resolves: #123

  • Other supported keywords: fix, fixed, close, closed, resolve, resolved
    Option 2: Link via GitHub UI (Note: won't clear the failed check)

  1. Go to the PR → Development section (right sidebar)

  2. Click "Link issue" and select an existing issue

  3. Push a new commit or re-run the workflow to clear the failed check
    Option 3: Use branch naming
    Create a new branch with one of these patterns:

  • 123-feature-description (number at start)

  • issue-123-feature-description (issue-number at start)

  • feature-issue-123 (issue-number anywhere)

Why is this required?

Issue linking ensures proper tracking, documentation, and helps maintain project history. It connects your code changes to the problem they solve.---

This comment was automatically generated by the issue linking workflow

@claude

claude Bot commented Jun 16, 2026

Copy link
Copy Markdown

Claude finished @jdcmsd's task in 41s —— View job


PR Review: Write eval context to file before passing to Claude

  • Read the diff and surrounding workflow context
  • Analyze for bugs, risks, and edge cases

The change is small and sensible — decoupling run_eval.py from claude via a temp file is reasonable. A few things worth a second look:

Improvement (positive side effect): GitHub Actions runs run: blocks with bash -e but not pipefail. In the old run_eval.py | claude form, a failure in run_eval.py was masked by claude's exit code. With the split into two statements, a non-zero exit from run_eval.py now fails the step before claude runs. That's a real robustness gain, not just a stdin fix.

Worth a second look — empty/partial output: If run_eval.py exits 0 but writes little or nothing to /tmp/eval_context.md (e.g. a silent GitHub fetch issue), claude --print will now run against an empty/garbage prompt instead of failing fast. The old pipe had the same exposure, so this isn't a regression — but since the stated motivation is flaky GitHub fetching, a guard would harden it:

uv run python scripts/run_eval.py ... > /tmp/eval_context.md
[ -s /tmp/eval_context.md ] || { echo "eval context is empty"; exit 1; }
claude --print --allowedTools Bash,Write < /tmp/eval_context.md

Minor — diagnosis vs. fix: A normal pipe shouldn't make claude "time out waiting for stdin" — it just blocks until data arrives. The more likely culprit is claude's own stdin/TTY handling under a slow producer. The file-redirect fix is fine regardless, but if the timeout recurs, the root cause may be elsewhere (e.g. a claude stdin read timeout), not the pipe itself.

Nit — fixed temp path: /tmp/eval_context.md is a hardcoded path left on the runner. Fine on an ephemeral CI runner; $RUNNER_TEMP/eval_context.md would be marginally cleaner and auto-scoped.

Note: I can't push changes here — .github/workflows/ files can't be modified by the GitHub App (workflow scope). The snippet above is for you to apply if you'd like the empty-output guard.

Overall: clean, low-risk change that also quietly improves error propagation.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant