Skip to content

feat: run scheduled maintenance endpoints in self-hosted deployments - #2299

Open
fancyHeat wants to merge 1 commit into
CapSoftware:mainfrom
fancyHeat:feat/self-hosted-cron
Open

fancyHeat wants to merge 1 commit into
CapSoftware:mainfrom
fancyHeat:feat/self-hosted-cron

Conversation

@fancyHeat

@fancyHeat fancyHeat commented Sep 15, 2026

Copy link
Copy Markdown

Problem

apps/web/vercel.json schedules the recovery endpoints as Vercel Cron Jobs. Self-hosted deployments have nothing that calls them, and CRON_SECRET isn't set in either compose file, so on every Docker Compose or Coolify install:

  • uploads that failed with a transient media server error (Server is busy, processVideoOnMediaServer … exceeded max retries) stay in phase = error permanently
  • media processing, transcription and AI generation that stall for over an hour are never restarted
  • segmented desktop recordings whose upload never completed are never finalized
  • expired Agent API records and media processing budgets are never cleaned up

Change

  • A cap-cron service in docker-compose.yml and docker-compose.coolify.yml (curlimages/curl, no build) calls the same endpoints on the same UTC schedules as vercel.json, with Authorization: Bearer $CRON_SECRET:
    • recover-failed-video-processing: 7,22,37,52 * * * *
    • finalize-stale-desktop-segments: */15 * * * *
    • cleanup-agent-api: 17 3 * * *
  • /api/cron/sync-loops is deliberately left out. It syncs contacts to Loops for Cap.so's own email and has no purpose on a self-hosted instance.
  • The script is inline rather than a mounted file, because the Coolify deployment doesn't have a repo checkout. That matches the existing minio-setup service. It handles SIGTERM, skips a route whose previous run is still going, reaps finished background jobs, and writes a heartbeat file for its healthcheck.
  • CRON_SECRET is set on cap-web and cap-cron:
    • docker-compose.yml: a placeholder default, like the other secrets.
    • Coolify: SERVICE_HEX_32_CRONSECRET.
  • Self-hosting docs: new "Scheduled Maintenance" section, and CRON_SECRET added to the production checklist.
  • test-self-hosting.yml: waits for cap-cron to be healthy, asserts each endpoint returns 200 using the container's secret, and asserts a wrong secret gets 401.

Testing

Ran docker compose up -d from this branch against the published ghcr.io/capsoftware/cap-web:latest:

  • All services became healthy.
  • The new CI step passed: 200 from each of the three endpoints, 401 for a wrong secret.
  • The scheduled 21:00 UTC call fired from inside the container and returned 200.
  • docker stop cap-cron exited 0 immediately.

The same design has been running in production on a self-hosted instance, where the first scheduled calls returned 200.

Heads up: MinIO images (not in this PR)

minio/minio:latest and minio/mc:latest can no longer be pulled from Docker Hub (pull access denied); quay.io/minio/minio and quay.io/minio/mc still work. So test-self-hosting.yml will likely fail on this PR, and on any other, at the image pull, independent of this change. I tested locally with a compose override pointing MinIO at quay.io. I kept that out of this PR to keep scope tight and am happy to send a separate fix.

RetriggerConfidence Score: 4/5

The scheduler implementation appears functionally safe, but the explicit repository comments requirement must be satisfied before merging; scheduler-path test coverage is a non-blocking follow-up.

Findings

  1. P2 Routine comments violate policy
  2. P2 Scheduler path remains untested
Fix with agent prompt
### Issue 1
docker-compose.coolify.env.example:23-24
These comments only restate the variable’s purpose and its standard generation command. This violates the repository directive to avoid code comments unless they preserve non-obvious context from a bug or complex investigation, so this requirement must be satisfied before merging.

```suggestion

```

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

### Issue 2
.github/workflows/test-self-hosting.yml:102-103
The workflow invokes each endpoint directly with `docker exec`, bypassing the new scheduler’s timing, route interpolation, and locking logic. A regression that leaves `cap-cron` healthy but prevents scheduled requests from launching would therefore still pass CI. Exercise at least one request through the scheduler path or make that scheduling logic independently testable.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Summary

This PR adds scheduled maintenance to Docker Compose and Coolify deployments through a lightweight cap-cron service.

  • Configures shared cron authorization and UTC schedules for three existing maintenance endpoints.
  • Documents scheduler operation and production secret setup.
  • Adds self-hosting workflow checks for endpoint availability and authorization.
  • The implementation is coherent, but CI bypasses the scheduler behavior it is intended to validate, and one changed configuration comment violates the repository’s comments policy.

Reviews (1) · Last reviewed commit: "feat: run scheduled maintenance endpoint..."

@superagent-security superagent-security Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Superagent found 1 security concern(s).

Comment thread docker-compose.yml
DATABASE_ENCRYPTION_KEY: ${DATABASE_ENCRYPTION_KEY:-0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef}
NEXTAUTH_SECRET: ${NEXTAUTH_SECRET:-abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123456789}
MEDIA_SERVER_WEBHOOK_SECRET: ${MEDIA_SERVER_WEBHOOK_SECRET:-fedcba9876543210fedcba9876543210fedcba9876543210fedcba9876543210}
CRON_SECRET: ${CRON_SECRET:-0f1e2d3c4b5a69780f1e2d3c4b5a69780f1e2d3c4b5a69780f1e2d3c4b5a6978}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2: Do not ship a predictable fallback secret for externally reachable maintenance endpoints

A predictable compose fallback authenticates the new recovery and deletion endpoints if operators omit CRON_SECRET.

Require a unique CRON_SECRET; do not provide a production compose fallback for this endpoint-authentication secret.

AI prompt
Check if this security scanner issue is valid. If so, understand the root cause and fix it. If appropriate, update or add tests. Keep the change focused and preserve intended behavior.

<file name="docker-compose.yml">
<violation number="1" location="docker-compose.yml:20">
<priority>P2</priority>
<title>Do not ship a predictable fallback secret for externally reachable maintenance endpoints</title>
<evidence>The newly added CRON_SECRET configuration supplies a deterministic built-in fallback instead of requiring an operator-provided secret. The same fallback is passed to cap-cron, so anyone who knows the published compose file can authenticate to the maintenance routes when this default is left unchanged. These routes perform recovery and deletion operations, expanding the impact beyond a normal local-development placeholder.</evidence>
<recommendation>Remove the CRON_SECRET fallback and fail the deployment or disable cap-cron when CRON_SECRET is unset. If a local-development default is required, make it opt-in through a clearly named development override and ensure production compose validation rejects it; keep the production and Coolify paths dependent on a unique generated secret.</recommendation>
</violation>
</file>

Comment on lines +23 to +24
# Secret the cap-cron service uses to call Cap Web's scheduled maintenance endpoints
# Generate with: openssl rand -hex 32

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Routine comments violate policy

These comments only restate the variable’s purpose and its standard generation command. This violates the repository directive to avoid code comments unless they preserve non-obvious context from a bug or complex investigation, so this requirement must be satisfied before merging.

Suggested change
# Secret the cap-cron service uses to call Cap Web's scheduled maintenance endpoints
# Generate with: openssl rand -hex 32

Context Used: AGENTS.md (source)

Prompt To Fix With AI
This is a comment left during a code review.
Path: docker-compose.coolify.env.example
Line: 23-24

Comment:
**Routine comments violate policy**

These comments only restate the variable’s purpose and its standard generation command. This violates the repository directive to avoid code comments unless they preserve non-obvious context from a bug or complex investigation, so this requirement must be satisfied before merging.

```suggestion

```

**Context Used:** AGENTS.md ([source](https://github.com/capsoftware/cap/blob/main/AGENTS.md))

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

Comment on lines +102 to +103
for route in recover-failed-video-processing finalize-stale-desktop-segments cleanup-agent-api; do
code=$(docker exec cap-cron sh -c 'curl -s -o /dev/null -w "%{http_code}" -H "Authorization: Bearer $CRON_SECRET" "$CAP_WEB_INTERNAL_URL/api/cron/'"$route"'"')

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Scheduler path remains untested

The workflow invokes each endpoint directly with docker exec, bypassing the new scheduler’s timing, route interpolation, and locking logic. A regression that leaves cap-cron healthy but prevents scheduled requests from launching would therefore still pass CI. Exercise at least one request through the scheduler path or make that scheduling logic independently testable.

Prompt To Fix With AI
This is a comment left during a code review.
Path: .github/workflows/test-self-hosting.yml
Line: 102-103

Comment:
**Scheduler path remains untested**

The workflow invokes each endpoint directly with `docker exec`, bypassing the new scheduler’s timing, route interpolation, and locking logic. A regression that leaves `cap-cron` healthy but prevents scheduled requests from launching would therefore still pass CI. Exercise at least one request through the scheduler path or make that scheduling logic independently testable.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant