Skip to content

fix(run-queue): handle worker spawn failure gracefully and avoid raw …#39

Open
Adityakk9031 wants to merge 1 commit into
AlmanacCode:mainfrom
Adityakk9031:#38
Open

fix(run-queue): handle worker spawn failure gracefully and avoid raw …#39
Adityakk9031 wants to merge 1 commit into
AlmanacCode:mainfrom
Adityakk9031:#38

Conversation

@Adityakk9031

@Adityakk9031 Adityakk9031 commented Jul 16, 2026

Copy link
Copy Markdown
Contributor

close #38

Description

This PR fixes a critical bug where worker subprocess spawn failures (e.g. OSError raised by subprocess.Popen) during manual lifecycle operations (init, ingest, garden) propagate as raw uncaught Python tracebacks.

We now catch spawn errors safely, wrap them in a product-level ExecutionFailed exception, and present a clean user-facing error message with attach instructions.

Detailed Changes:

  • Run Queue Service: Added a safe helper method _spawn_worker_safely in src/codealmanac/workflows/run_queue/service.py to wrap the spawn_worker calls. If spawning fails, it raises an ExecutionFailed exception containing the orphaned run_id and recovery instructions (codealmanac jobs attach <id>).
  • Code Size Optimization: Using the helper method avoids adding duplicate try/except blocks across all three triggers, successfully keeping the service line count under the strict architecture test limit of 200 lines.
  • Unit Testing: Added test_run_queue_start_handles_worker_spawn_failure in tests/test_run_queue_workflow.py to assert that spawn failures raise ExecutionFailed and the run is kept persisted as QUEUED in the database.

Verification

Automated Tests

Ran the pytest suite for architecture invariants and run queue workflows:

.venv/Scripts/pytest tests/test_architecture.py
.venv/Scripts/pytest tests/test_run_queue_workflow.py

@Adityakk9031

Copy link
Copy Markdown
Contributor Author

@divitsheth have a look

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

bug: worker spawn OSError crashes CLI with raw traceback after run is durably committed to the queue

1 participant