From 9055e43d04f41ba59f062aa5806afcc89580524b Mon Sep 17 00:00:00 2001 From: HardMax71 Date: Fri, 23 Oct 2026 20:06:24 +0200 Subject: [PATCH] refactor: deploy.sh goes, each task runs through its own tool, and the Mongo backup is a compose service Compose starts and stops the stack with its own profiles and flags, pre-commit runs the lint and type checks (bandit added, with the same command as CI), and npm run generate:api builds the client. The spec export becomes backend/scripts/export_openapi.py. The nightly Mongo backup, a host cron on production only, is now the mongo-backup service: mongodump from the mongo image every 24 hours into the mongo_backups volume, 14 days kept, the same locally and in production. CI selects the test config with BACKEND_CONFIG in .env instead of copying it over config.toml. --- .env.example | 2 +- .github/ISSUE_TEMPLATE/bug_report.yml | 4 +- .github/ISSUE_TEMPLATE/feature_request.yml | 2 +- .github/PULL_REQUEST_TEMPLATE.md | 2 +- .github/actions/e2e-ready/action.yml | 3 +- .github/labeler.yml | 1 - .github/labels.yml | 2 +- .github/workflows/ci.yml | 11 +- .pre-commit-config.yaml | 8 + AGENTS.md | 34 +- CONTRIBUTING.md | 9 +- README.md | 23 +- backend/scripts/export_openapi.py | 18 + deploy.sh | 410 --------------------- docker-compose.yaml | 35 +- docs/architecture/frontend-build.md | 4 +- docs/components/workers/index.md | 2 +- docs/getting-started.md | 17 +- docs/index.md | 3 +- docs/operations/cicd.md | 22 +- docs/operations/deployment.md | 43 +-- docs/operations/metrics-reference.md | 2 +- docs/reference/api-reference.md | 2 +- docs/reference/configuration.md | 4 +- docs/testing/frontend-testing.md | 2 +- 25 files changed, 147 insertions(+), 518 deletions(-) create mode 100644 backend/scripts/export_openapi.py delete mode 100755 deploy.sh diff --git a/.env.example b/.env.example index 8ff46830..79650fd9 100644 --- a/.env.example +++ b/.env.example @@ -1,4 +1,4 @@ -# Local development values. ./deploy.sh dev copies this file to .env, which compose reads. +# Local development values. Copy this file to .env once (cp .env.example .env); compose reads .env. # Production never sees this file: its deploy writes .env from the repository secrets. SECRET_KEY=CHANGE_ME_min_32_chars_long_!!!! MONGO_USER=root diff --git a/.github/ISSUE_TEMPLATE/bug_report.yml b/.github/ISSUE_TEMPLATE/bug_report.yml index 9bd06057..647db6c0 100644 --- a/.github/ISSUE_TEMPLATE/bug_report.yml +++ b/.github/ISSUE_TEMPLATE/bug_report.yml @@ -49,7 +49,7 @@ body: - Kafka events (consumers, DLQ, replay) - Live updates and notifications - Metrics, dashboards and logs - - Deployment (compose, deploy.sh, production) + - Deployment (compose, production) - CI and tooling - Documentation - Not sure @@ -62,7 +62,7 @@ body: label: Environment description: Whatever applies. placeholder: | - - Where: local dev (./deploy.sh dev) or production + - Where: local dev (docker compose up) or production - Version: release tag or commit, e.g. 2026.10.3 or 7b9f060 - Script language and version, e.g. Go 1.27 - Browser, e.g. Firefox 145 diff --git a/.github/ISSUE_TEMPLATE/feature_request.yml b/.github/ISSUE_TEMPLATE/feature_request.yml index 22d14887..4e586654 100644 --- a/.github/ISSUE_TEMPLATE/feature_request.yml +++ b/.github/ISSUE_TEMPLATE/feature_request.yml @@ -45,7 +45,7 @@ body: - Kafka events (consumers, DLQ, replay) - Live updates and notifications - Metrics, dashboards and logs - - Deployment (compose, deploy.sh, production) + - Deployment (compose, production) - CI and tooling - Documentation - Not sure diff --git a/.github/PULL_REQUEST_TEMPLATE.md b/.github/PULL_REQUEST_TEMPLATE.md index a96d289b..ffaebb41 100644 --- a/.github/PULL_REQUEST_TEMPLATE.md +++ b/.github/PULL_REQUEST_TEMPLATE.md @@ -31,7 +31,7 @@ Fixes # - [ ] Frontend unit tests pass (`cd frontend && npm run test`) - [ ] Backend E2E tests pass (`docker compose exec -T backend sh -c 'uv sync --group test --no-dev --frozen --no-install-project --no-build && pytest tests/e2e'`) - [ ] Frontend E2E tests pass (`cd frontend && npm run test:e2e`) -- [ ] API spec and client regenerated if routes or schemas changed (`./deploy.sh openapi`, then `./deploy.sh types`), never edited by hand +- [ ] API spec and client regenerated if routes or schemas changed (`uv run python -m scripts.export_openapi` in backend/, then `npm run generate:api` in frontend/), never edited by hand - [ ] Manual testing performed **Manual test steps:** diff --git a/.github/actions/e2e-ready/action.yml b/.github/actions/e2e-ready/action.yml index b5e81e73..2d7c1ee3 100644 --- a/.github/actions/e2e-ready/action.yml +++ b/.github/actions/e2e-ready/action.yml @@ -36,7 +36,8 @@ runs: shell: bash run: | set -e - cp backend/config.test.toml backend/config.toml + # compose reads .env for every later command, so the backend and the worker both run with the test settings + echo BACKEND_CONFIG=config.test.toml >> .env timeout 120 tail --pid="$(cat /tmp/infra-pull.pid)" -f /dev/null 2>/dev/null || true cat /tmp/infra-pull.log 2>/dev/null || true diff --git a/.github/labeler.yml b/.github/labeler.yml index 46ac01cb..65a208fc 100644 --- a/.github/labeler.yml +++ b/.github/labeler.yml @@ -77,7 +77,6 @@ - changed-files: - any-glob-to-any-file: - docker-compose.yaml - - deploy.sh - "**/Dockerfile*" - .env.example - .github/workflows/docker.yml diff --git a/.github/labels.yml b/.github/labels.yml index 5aaabf9e..e354e68f 100644 --- a/.github/labels.yml +++ b/.github/labels.yml @@ -9,7 +9,7 @@ description: "GitHub workflows, test infrastructure, repo tooling" - name: "area: deploy" color: "#c5def5" - description: "Compose, deploy.sh, images, release workflow, production host" + description: "Compose, images, release workflow, production host" - name: "area: events" color: "#c5def5" description: "Kafka consumers and middlewares, producer, idempotency, DLQ, replay" diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index cdd4d56f..c1aeb83b 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -53,7 +53,6 @@ jobs: - *workflow - '.env.example' - 'docker-compose.yaml' - - 'deploy.sh' - '.github/actions/**' # The configs of the observability services, each in the folder named after its compose service observability: &observability @@ -67,7 +66,6 @@ jobs: api: - *workflow - 'backend/**' - - 'deploy.sh' - 'docs/reference/openapi.json' - 'frontend/src/lib/api/**' - 'frontend/openapi-ts.config.ts' @@ -82,7 +80,6 @@ jobs: - 'backend/**' - 'frontend/**' - 'docker-compose.yaml' - - 'deploy.sh' ruff: name: Ruff Linting @@ -176,7 +173,7 @@ jobs: cache-dependency-glob: "backend/uv.lock" - name: Run bandit - run: uvx --no-build --python 3.14 bandit==1.9.4 -r backend/ -x backend/tests/ -ll + run: uvx --no-build --python 3.14 bandit==1.9.4 -r backend/ -x backend/tests/,backend/.venv/ -ll frontend: name: Frontend Lint & Type Check @@ -250,14 +247,14 @@ jobs: - name: Regenerate the spec and the client run: | - ./deploy.sh openapi + (cd backend && uv run --no-sync --no-build python -m scripts.export_openapi) cd frontend && npm run generate:api - name: Fail on any difference run: | changes=$(git status --porcelain -- docs/reference/openapi.json frontend/src/lib/api) if [ -n "$changes" ]; then - echo "::error::docs/reference/openapi.json and frontend/src/lib/api must come from the generators, never from hand edits. Run ./deploy.sh openapi, then ./deploy.sh types, and commit the result." + echo "::error::docs/reference/openapi.json and frontend/src/lib/api must come from the generators, never from hand edits. Run uv run python -m scripts.export_openapi in backend/, then npm run generate:api in frontend/, and commit the result." echo "$changes" git diff --stat -- docs/reference/openapi.json frontend/src/lib/api exit 1 @@ -281,7 +278,7 @@ jobs: run: cd backend && uv sync --frozen --no-dev --no-install-project --no-build - name: Generate OpenAPI spec - run: ./deploy.sh openapi + run: cd backend && uv run --no-sync --no-build python -m scripts.export_openapi - name: Build documentation run: >- diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml index 33c0191f..e29afc74 100644 --- a/.pre-commit-config.yaml +++ b/.pre-commit-config.yaml @@ -38,6 +38,14 @@ repos: files: ^backend/.*\.py$ pass_filenames: false + # Bandit - matches CI: uvx --no-build --python 3.14 bandit==1.9.4 -r backend/ -x backend/tests/,backend/.venv/ -ll + - id: bandit-backend + name: bandit (backend) + entry: uvx --no-build --python 3.14 bandit==1.9.4 -r backend/ -x backend/tests/,backend/.venv/ -ll + language: system + files: ^backend/.*\.py$ + pass_filenames: false + # ESLint - matches CI: cd frontend && npm run lint - id: eslint-frontend name: eslint (frontend) diff --git a/AGENTS.md b/AGENTS.md index 0c1a03ed..b29e7aec 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -10,7 +10,7 @@ Platform for running scripts online in isolated Kubernetes pods. Users write Pyt | Backend | FastAPI, Dishka DI, Beanie ODM (MongoDB), Python 3.14 (backend/.python-version), async throughout, httpx2 for HTTP | | Events | Kafka (KRaft, no ZooKeeper) + FastStream (Pydantic JSON) | | Worker | one process (`app/worker.py`): execution scheduler, k8s worker, pod monitor, result processor, DLQ retries | -| Infra | MongoDB 9, Redis 8; optional Grafana, Jaeger, Victoria Metrics, Loki, OpenSearch, OTel Collector plus its host-metrics collector (`--observability`) and Kafdrop (`--debug`) | +| Infra | MongoDB 9 with a daily mongodump (`mongo-backup`), Redis 8; optional Grafana, Jaeger, Victoria Metrics, Loki, OpenSearch, OTel Collector plus its host-metrics collector (compose profile `observability`) and Kafdrop (profile `debug`) | | Runtimes | Python 3.11 to 3.14, Node 22/24/26, Ruby 3.3/3.4/4.0, Bash 5.3, Go 1.26/1.27 (app/runtime_registry.py) | ## Key Directory Map @@ -87,17 +87,20 @@ npm run test:e2e:browsers # once: Playwright Chromium npm run test:e2e # Playwright E2E, needs the running stack # Full stack (from project root) -./deploy.sh dev # start -./deploy.sh dev --build # rebuild and start -./deploy.sh dev --wait # start and wait for healthy -./deploy.sh dev --observability # plus Grafana, Jaeger, Victoria Metrics, Loki, OpenSearch, OTel Collector, host-metrics -./deploy.sh dev --debug # observability plus Kafdrop -./deploy.sh down # stop -./deploy.sh openapi # regenerate docs/reference/openapi.json -./deploy.sh types # regenerate frontend/src/lib/api/ from that spec +cp .env.example .env # once: the local secrets +docker compose up -d # start +docker compose up -d --build # rebuild and start +docker compose up -d --wait # start and wait for healthy +docker compose --profile observability up -d # plus Grafana, Jaeger, Victoria Metrics, Loki, OpenSearch, OTel Collector, host-metrics +docker compose --profile debug up -d # plus Kafdrop +docker compose down # stop + +# Generated API client +cd backend && uv run python -m scripts.export_openapi # regenerate docs/reference/openapi.json +cd frontend && npm run generate:api # regenerate frontend/src/lib/api/ from that spec ``` -`npm run dev` serves HTTPS when ./certs/server.key and server.crt exist and proxies /api to `https://backend:443` with the certificate checked, so it needs `NODE_EXTRA_CA_CERTS` pointing at the backend's `server.crt` (the `backend_tls_cert` volume). `backend` only resolves inside the compose network, so on the host the API calls fail. The compose `frontend` service already serves a production build on :5001; rebuild it with `./deploy.sh dev --build` after frontend changes. +`npm run dev` serves HTTPS when ./certs/server.key and server.crt exist and proxies /api to `https://backend:443` with the certificate checked, so it needs `NODE_EXTRA_CA_CERTS` pointing at the backend's `server.crt` (the `backend_tls_cert` volume). `backend` only resolves inside the compose network, so on the host the API calls fail. The compose `frontend` service already serves a production build on :5001; rebuild it with `docker compose up -d --build frontend` after frontend changes. Plain `uv run pytest` also collects tests/e2e, which needs the full stack, so always pass a path. E2E tests run inside the backend container against a running stack, the way CI does it (CI copies config.test.toml over config.toml before starting the stack): @@ -504,7 +507,7 @@ settings: FromDishka[Settings] Config load order (each overrides the previous): 1. `config.toml`: base settings (committed) 2. `config.worker.toml`: the worker's override (optional) -3. Secrets: `SECRET_KEY`, `MONGO_USER`, `MONGO_PASSWORD`, `REDIS_PASSWORD`. Each one is read from the env var first, then the TOML value, then a dev default, so a missing secret never raises in `Settings`. Compose has no defaults for them: `./deploy.sh dev` copies `.env.example` to `.env`, the deploy writes `.env` from the GitHub secrets, and compose refuses to start while one is empty. `MONGODB_URL` is built from the Mongo values +3. Secrets: `SECRET_KEY`, `MONGO_USER`, `MONGO_PASSWORD`, `REDIS_PASSWORD`. Each one is read from the env var first, then the TOML value, then a dev default, so a missing secret never raises in `Settings`. Compose has no defaults for them: locally `.env` is a copy of `.env.example`, the deploy writes `.env` from the GitHub secrets, and compose refuses to start while one is empty. `MONGODB_URL` is built from the Mongo values ```python Settings() # config.toml + secrets @@ -656,11 +659,10 @@ Use `interface` for object shapes, `type` for unions/intersections/aliases. Defi Hard rule: `docs/reference/openapi.json` and everything under `frontend/src/lib/api/` come from the generators only. Never edit them by hand, not even a one-line fix; change the backend (routes, schemas) and regenerate. CI's `Spec and client match the code` job regenerates both on every PR that touches the backend, the spec or the client, and fails on any difference, so a hand edit can't merge. -`npm run generate:api` reads `../docs/reference/openapi.json`, and `./deploy.sh types` regenerates the spec only when that file is missing. After any backend API change, regenerate the spec first: +`npm run generate:api` reads `../docs/reference/openapi.json`. After any backend API change, regenerate the spec first: ```bash -# From project root -./deploy.sh openapi # writes docs/reference/openapi.json from create_app() -./deploy.sh types # runs npm run generate:api into frontend/src/lib/api/ +cd backend && uv run python -m scripts.export_openapi # writes docs/reference/openapi.json from create_app() +cd frontend && npm run generate:api # writes frontend/src/lib/api/ from that spec ``` Function names are derived from URL paths: `getUserRateLimitsApiV1AdminRateLimitsUserIdGet`. @@ -881,7 +883,7 @@ npm run build npm run test:coverage # Vitest with coverage thresholds # Generated API (from the project root): must leave no diff -./deploy.sh openapi && (cd frontend && npm run generate:api) +(cd backend && uv run python -m scripts.export_openapi) && (cd frontend && npm run generate:api) git status --porcelain -- docs/reference/openapi.json frontend/src/lib/api ``` diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 45c01a95..e69752f1 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -25,7 +25,8 @@ uv tool install pre-commit pre-commit install # Start the stack -./deploy.sh dev +cp .env.example .env +docker compose up -d ``` ### Pre-commit Hooks @@ -73,7 +74,7 @@ uv run pytest tests/unit Always pass a path to pytest. A plain `uv run pytest` also collects tests/e2e, which needs the running stack and Kubernetes. The E2E suite runs inside the backend container, the same way CI runs it: ```bash -./deploy.sh dev --wait +docker compose up -d --wait docker compose exec -T backend sh -c 'uv sync --group test --no-dev --frozen --no-install-project && pytest tests/e2e' ``` @@ -90,13 +91,13 @@ npm run test npm run check npm run lint -# E2E tests (Playwright, needs the stack from ./deploy.sh dev) +# E2E tests (Playwright, needs the running stack) npm run test:e2e:deps # once, installs system deps for Chromium npm run test:e2e:browsers # once, installs Chromium npm run test:e2e ``` -The stack serves a production build of the frontend on https://localhost:5001, so rebuild it with `./deploy.sh dev --build` after frontend changes. `npm run dev` only works inside the compose network, since it proxies to the backend container and needs the shared CA. +The stack serves a production build of the frontend on https://localhost:5001, so rebuild it with `docker compose up -d --build frontend` after frontend changes. `npm run dev` only works inside the compose network, since it proxies to the backend container and needs the shared CA. ## Code Style diff --git a/README.md b/README.md index f2170850..cbd6aa48 100644 --- a/README.md +++ b/README.md @@ -66,10 +66,11 @@ ```bash git clone https://github.com/HardMax71/Integr8sCode.git cd Integr8sCode -./deploy.sh dev +cp .env.example .env +docker compose up -d ``` -`./deploy.sh dev` copies `.env.example` to `.env` on its first run, so local development needs no config. Compose has no defaults for secrets: anywhere else, write a `.env` with your own values, and compose refuses to start while one is missing. +`.env.example` holds the local secrets, so local development needs no other config. Compose has no defaults for secrets: anywhere else, write a `.env` with your own values, and compose refuses to start while one is missing. ### Verify @@ -79,15 +80,15 @@ curl -k https://localhost/api/v1/health/live ### Access -| Service | URL | Notes | -|--------------------|--------------------------------------------------|-----------------------------------------------| -| Frontend | [https://localhost:5001](https://localhost:5001) | `user` / `user123` or `admin` / `admin123` | -| Backend API | [https://localhost:443](https://localhost:443) | | -| Grafana | [http://localhost:3000](http://localhost:3000) | needs `--observability`; `admin` / `admin123` | -| Jaeger (Tracing) | [http://localhost:16686](http://localhost:16686) | needs `--observability` | -| Kafdrop (Kafka UI) | [http://localhost:9000](http://localhost:9000) | needs `--debug` | +| Service | URL | Notes | +|--------------------|--------------------------------------------------|---------------------------------------------------------| +| Frontend | [https://localhost:5001](https://localhost:5001) | `user` / `user123` or `admin` / `admin123` | +| Backend API | [https://localhost:443](https://localhost:443) | | +| Grafana | [http://localhost:3000](http://localhost:3000) | needs the `observability` profile; `admin` / `admin123` | +| Jaeger (Tracing) | [http://localhost:16686](http://localhost:16686) | needs the `observability` profile | +| Kafdrop (Kafka UI) | [http://localhost:9000](http://localhost:9000) | needs the `debug` profile | -`./deploy.sh dev --observability` adds Grafana, Jaeger, Victoria Metrics, Loki, OpenSearch and the OTel Collector. `--debug` adds all of those plus Kafdrop. +`docker compose --profile observability up -d` adds Grafana, Jaeger, Victoria Metrics, Loki, OpenSearch and the OTel Collector, and `--profile debug` adds Kafdrop. nginx and the backend each make a self-signed certificate when they first start, so accept the browser warning. nginx verifies the backend's certificate on every proxied request. A certificate mounted at `/etc/nginx/certs` in the frontend container is used as is. @@ -105,7 +106,7 @@ The E2E suites need the full stack; [CONTRIBUTING.md](CONTRIBUTING.md) has the c ### Stop ```bash -./deploy.sh down +docker compose down ``` See the [full deployment guide](https://hardmax71.github.io/Integr8sCode/operations/deployment/) for Docker build strategy, troubleshooting, pre-built images, and more. diff --git a/backend/scripts/export_openapi.py b/backend/scripts/export_openapi.py new file mode 100644 index 00000000..b74efe34 --- /dev/null +++ b/backend/scripts/export_openapi.py @@ -0,0 +1,18 @@ +import json +from pathlib import Path + +from app.main import create_app +from app.settings import Settings + +SPEC_PATH = Path(__file__).resolve().parents[2] / "docs" / "reference" / "openapi.json" + + +def export_openapi() -> None: + """Write the API's OpenAPI spec to docs/reference/openapi.json, where the frontend's client generator reads it.""" + # config.toml's collector address resolves only inside compose; exporting there adds 11 s of failed retries + app = create_app(settings=Settings().model_copy(update={"OTEL_EXPORTER_OTLP_ENDPOINT": None})) + SPEC_PATH.write_text(json.dumps(app.openapi(), indent=2) + "\n") + + +if __name__ == "__main__": + export_openapi() diff --git a/deploy.sh b/deploy.sh deleted file mode 100755 index 7b0c4674..00000000 --- a/deploy.sh +++ /dev/null @@ -1,410 +0,0 @@ -#!/bin/bash -# ============================================================================= -# Integr8sCode Deployment Script (Docker Compose) -# ============================================================================= -# -# Usage: -# ./deploy.sh dev # Start local development (docker-compose) -# ./deploy.sh dev --build # Rebuild and start local development -# ./deploy.sh down # Stop local development -# ./deploy.sh check # Run local quality checks (lint, type, security) -# ./deploy.sh test # Run full test suite locally -# ./deploy.sh logs [service] # View logs (dev mode) -# ./deploy.sh status # Show status of running services -# ./deploy.sh backup [dir] # Dump MongoDB, keep 14 days -# ./deploy.sh openapi [path] # Generate OpenAPI spec from backend -# ./deploy.sh types # Generate TypeScript types for frontend -# -# ============================================================================= - -set -e - -SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)" -cd "$SCRIPT_DIR" - -# Colors -RED='\033[0;31m' -GREEN='\033[0;32m' -YELLOW='\033[1;33m' -BLUE='\033[0;34m' -NC='\033[0m' - -print_header() { - echo -e "${BLUE}" - echo "===========================================================================" - echo " $1" - echo "===========================================================================" - echo -e "${NC}" -} - -print_success() { echo -e "${GREEN}✓ $1${NC}"; } -print_warning() { echo -e "${YELLOW}⚠ $1${NC}"; } -print_error() { echo -e "${RED}✗ $1${NC}"; } -print_info() { echo -e "${BLUE}→ $1${NC}"; } - -# --8<-- [start:usage] -show_help() { - echo "Integr8sCode Deployment Script" - echo "" - echo "Usage: ./deploy.sh [options]" - echo "" - echo "Commands:" - echo " dev [options] Start full stack (docker-compose)" - echo " --build Rebuild images locally" - echo " --no-build Use pre-built images only (no build fallback)" - echo " --wait Wait for services to be healthy" - echo " --timeout Health check timeout (default: 300)" - echo " --observability Include Grafana, Jaeger, etc." - echo " --debug Include observability + Kafdrop" - echo " infra [options] Start infrastructure only (mongo, redis, kafka, etc.)" - echo " --wait Wait for services to be healthy" - echo " --timeout Health check timeout (default: 120)" - echo " down Stop all services" - echo " check Run the static CI gates (ruff, mypy --strict, orphan modules, bandit)" - echo " test Run unit and contract tests, then E2E on a stack using config.test.toml" - echo " logs [service] View logs (defaults to all services)" - echo " status Show status of running services" - echo " backup [dir] Dump MongoDB to dir (default: /var/backups/mongo), keep 14 days" - echo " openapi [path] Generate OpenAPI spec (default: docs/reference/openapi.json)" - echo " types Generate TypeScript types for frontend from OpenAPI spec" - echo " help Show this help message" - echo "" - echo "Configuration:" - echo " Settings come from backend/config.toml, secrets from .env" - echo " ./deploy.sh test starts the stack with backend/config.test.toml instead" - echo "" - echo "Examples:" - echo " ./deploy.sh dev # Start dev environment" - echo " ./deploy.sh dev --build # Rebuild and start" - echo " ./deploy.sh dev --wait # Start and wait for healthy" - echo " ./deploy.sh logs backend # View backend logs" -} -# --8<-- [end:usage] - -# ============================================================================= -# LOCAL DEVELOPMENT (docker-compose) -# ============================================================================= -# compose reads .env for every command; production writes its own from the deploy's secrets -ensure_env_file() { - if [[ ! -f .env ]]; then - cp .env.example .env - print_info "Created .env from .env.example (local development values)" - fi -} - -cmd_dev() { - print_header "Starting Local Development Environment" - - local BUILD_FLAG="" - local NO_BUILD_FLAG="" - local WAIT_FLAG="" - local WAIT_TIMEOUT="300" - local PROFILE_FLAGS="" - - while [[ $# -gt 0 ]]; do - case "$1" in - --build) - BUILD_FLAG="--build" - print_info "Rebuilding images..." - ;; - --no-build) - NO_BUILD_FLAG="--no-build" - print_info "Using pre-built images (skipping build)..." - ;; - --wait) - WAIT_FLAG="--wait" - ;; - --timeout) - shift - WAIT_TIMEOUT="$1" - ;; - --observability) - PROFILE_FLAGS="--profile observability" - print_info "Including observability stack (Grafana, Jaeger, etc.)" - ;; - --debug) - PROFILE_FLAGS="--profile observability --profile debug" - print_info "Including observability + debug tools (Kafdrop, etc.)" - ;; - esac - shift - done - - local WAIT_TIMEOUT_FLAG="" - if [[ -n "$WAIT_FLAG" ]]; then - WAIT_TIMEOUT_FLAG="--wait-timeout $WAIT_TIMEOUT" - fi - - ensure_env_file - docker compose $PROFILE_FLAGS up -d $BUILD_FLAG $NO_BUILD_FLAG $WAIT_FLAG $WAIT_TIMEOUT_FLAG - - echo "" - print_success "Development environment started!" - echo "" - echo "Services:" - echo " Backend: https://localhost:443" - echo " Frontend: https://localhost:5001" - if [[ "$PROFILE_FLAGS" == *"debug"* ]]; then - echo " Kafdrop: http://localhost:9000" - fi - if [[ "$PROFILE_FLAGS" == *"observability"* ]]; then - echo " Jaeger: http://localhost:16686" - echo " Grafana: http://localhost:3000" - fi - echo "" - echo "Commands:" - echo " ./deploy.sh logs # View all logs" - echo " ./deploy.sh logs backend # View backend logs" - echo " ./deploy.sh down # Stop all services" -} - -cmd_down() { - print_header "Stopping Local Development Environment" - docker compose down - print_success "All services stopped" -} - -cmd_infra() { - print_header "Starting Infrastructure Services Only" - - local WAIT_FLAG="" - local WAIT_TIMEOUT="120" - - while [[ $# -gt 0 ]]; do - case "$1" in - --wait) - WAIT_FLAG="--wait" - ;; - --timeout) - shift - WAIT_TIMEOUT="$1" - ;; - esac - shift - done - - local WAIT_TIMEOUT_FLAG="" - if [[ -n "$WAIT_FLAG" ]]; then - WAIT_TIMEOUT_FLAG="--wait-timeout $WAIT_TIMEOUT" - fi - - # Start only infrastructure services (no app, no workers, no observability) - ensure_env_file - docker compose up -d mongo redis kafka $WAIT_FLAG $WAIT_TIMEOUT_FLAG - - print_success "Infrastructure services started" - docker compose ps -} - -cmd_logs() { - local SERVICE="$1" - if [[ -n "$SERVICE" ]]; then - docker compose logs -f "$SERVICE" - else - docker compose logs -f - fi -} - -cmd_status() { - print_header "Service Status" - - echo "" - echo "Docker Compose Services:" - docker compose ps 2>/dev/null || echo " No docker-compose services running" -} - -cmd_backup() { - local DIR="${1:-/var/backups/mongo}" - umask 077 - mkdir -p "$DIR" - if (( $(df --output=avail -k "$DIR" | tail -1) < 5 * 1024 * 1024 )); then - print_error "Less than 5G free in $DIR, no backup taken" - exit 1 - fi - local OUT="$DIR/mongo-$(date +%F).archive.gz" - docker compose exec -T mongo sh -c 'mongodump --quiet -u "$MONGO_INITDB_ROOT_USERNAME" -p "$MONGO_INITDB_ROOT_PASSWORD" --authenticationDatabase admin --archive --gzip' > "$OUT.part" - mv "$OUT.part" "$OUT" - find "$DIR" -name 'mongo-*' -mtime +13 -delete - print_success "Backup written to $OUT ($(du -h "$OUT" | cut -f1))" -} - -# ============================================================================= -# QUALITY CHECKS -# ============================================================================= -cmd_check() { - print_header "Running Quality Checks" - - cd backend - - print_info "Installing dependencies without building any package..." - uv sync --frozen --no-install-project --no-build - - print_info "Running Ruff linting..." - if uv run --no-sync --no-build ruff check . --config pyproject.toml; then - print_success "Ruff linting passed" - else - print_error "Ruff linting failed" - exit 1 - fi - - print_info "Running MyPy type checking..." - if uv run --no-sync --no-build mypy --config-file pyproject.toml --strict .; then - print_success "MyPy type checking passed" - else - print_error "MyPy type checking failed" - exit 1 - fi - - print_info "Running orphan module check..." - if uv run --no-sync --no-build python -m scripts.check_orphan_modules; then - print_success "Orphan module check passed" - else - print_error "Orphan module check failed" - exit 1 - fi - - print_info "Running Bandit security scan..." - if uvx --no-build --python 3.14 bandit==1.9.4 -r . -x ./tests,./.venv -ll; then - print_success "Security scan passed" - else - print_error "Security scan failed" - exit 1 - fi - - cd .. - print_success "All quality checks passed!" -} - -# ============================================================================= -# TEST SUITE -# ============================================================================= -cmd_test() { - print_header "Running Test Suite" - - print_info "Running unit and contract tests..." - if ! (cd backend && uv sync --frozen --no-install-project --no-build \ - && uv run --no-sync --no-build pytest tests/unit \ - && uv run --no-sync --no-build pytest tests/contract -m grafana_contract); then - print_error "Unit or contract tests failed" - exit 1 - fi - - print_info "Starting full stack with config.test.toml..." - export BACKEND_CONFIG=config.test.toml - cmd_dev --build --wait - - print_info "Running E2E tests inside Docker..." - if docker compose exec -T backend \ - sh -c 'uv sync --group test --no-dev --frozen --no-install-project --no-build && pytest tests/e2e'; then - print_success "All tests passed!" - TEST_RESULT=0 - else - print_error "Tests failed" - TEST_RESULT=1 - fi - - print_info "Cleaning up..." - docker compose down - - exit $TEST_RESULT -} - -# ============================================================================= -# OPENAPI SPEC GENERATION -# ============================================================================= -cmd_openapi() { - print_header "Generating OpenAPI Spec" - - local OUTPUT="${1:-docs/reference/openapi.json}" - - cd backend - print_info "Extracting schema from FastAPI app..." - - local tmp - tmp=$(mktemp) - if ! uv run --no-sync --no-build python -c " -import json -from app.main import create_app -from app.settings import Settings -app = create_app(settings=Settings().model_copy(update={'OTEL_EXPORTER_OTLP_ENDPOINT': None})) -schema = app.openapi() -print(json.dumps(schema, indent=2)) -" > "$tmp"; then - rm -f "$tmp" - cd .. - print_error "OpenAPI generation failed, $OUTPUT left unchanged" - exit 1 - fi - chmod 644 "$tmp" - mv "$tmp" "../$OUTPUT" - - cd .. - print_success "OpenAPI spec written to $OUTPUT" -} - -# ============================================================================= -# TYPESCRIPT API CLIENT GENERATION -# ============================================================================= -cmd_types() { - print_header "Generating TypeScript API Client" - - # Ensure OpenAPI spec exists - if [[ ! -f "docs/reference/openapi.json" ]]; then - print_info "OpenAPI spec not found, generating first..." - cmd_openapi - fi - - cd frontend - print_info "Generating typed API client from OpenAPI spec..." - npm run generate:api - cd .. - - print_success "Typed API client generated in frontend/src/lib/api/" -} - -# ============================================================================= -# MAIN -# ============================================================================= -case "${1:-help}" in - dev) - shift - cmd_dev "$@" - ;; - infra) - shift - cmd_infra "$@" - ;; - down) - cmd_down - ;; - logs) - cmd_logs "$2" - ;; - status) - cmd_status - ;; - backup) - cmd_backup "$2" - ;; - check) - cmd_check - ;; - test) - cmd_test - ;; - openapi) - cmd_openapi "$2" - ;; - types) - cmd_types - ;; - help|--help|-h) - show_help - ;; - *) - print_error "Unknown command: $1" - echo "" - show_help - exit 1 - ;; -esac diff --git a/docker-compose.yaml b/docker-compose.yaml index 3dd0b294..ae7f5508 100644 --- a/docker-compose.yaml +++ b/docker-compose.yaml @@ -9,7 +9,7 @@ x-logging: &logging cache-max-size: "10m" cache-max-file: "3" -# Secrets have no defaults: ./deploy.sh dev copies .env.example to .env, and the production deploy writes .env from its secrets +# Secrets have no defaults: locally .env is a copy of .env.example, and the production deploy writes .env from its secrets x-backend-secrets: &backend-secrets SECRET_KEY: ${SECRET_KEY:?see .env.example} MONGO_USER: ${MONGO_USER:?see .env.example} @@ -45,6 +45,36 @@ services: retries: 15 start_period: 5s + # Dumps MongoDB when it starts and every 24 hours after, and keeps 14 days of dumps + mongo-backup: + logging: *logging + image: mongo:9.0.4 + container_name: mongo-backup + depends_on: + mongo: + condition: service_healthy + init: true + entrypoint: + - sh + - -c + - | + umask 077 + while true; do + f=/backup/mongo-$$(date +%F).archive.gz + mongodump --quiet --host mongo -u "$$MONGO_USER" -p "$$MONGO_PASSWORD" --authenticationDatabase admin --archive="$$f.part" --gzip && mv "$$f.part" "$$f" && echo "Backup written to $$f" + find /backup -name 'mongo-*' -mtime +13 -delete + sleep 86400 + done + environment: + MONGO_USER: ${MONGO_USER:?see .env.example} + MONGO_PASSWORD: ${MONGO_PASSWORD:?see .env.example} + volumes: + - mongo_backups:/backup + networks: + - app-network + mem_limit: 256m + restart: unless-stopped + redis: logging: *logging image: redis:8.10.2-alpine @@ -83,7 +113,7 @@ services: - ./backend/scripts:/app/scripts - ./backend/tests:/app/tests:ro - ./backend/config.test.toml:/app/config.test.toml:ro - # ./deploy.sh test sets BACKEND_CONFIG=config.test.toml, the way CI copies it over config.toml + # BACKEND_CONFIG=config.test.toml in .env runs the stack with the test settings, as CI does - ./backend/${BACKEND_CONFIG:-config.toml}:/app/config.toml:ro - backend_tls_key:/app/certs/key - backend_tls_cert:/app/certs/cert @@ -491,6 +521,7 @@ services: # --8<-- [start:dev_volumes] volumes: mongo_data: + mongo_backups: redis_data: backend_tls_key: backend_tls_cert: diff --git a/docs/architecture/frontend-build.md b/docs/architecture/frontend-build.md index 3f6d3f14..c39d08a3 100644 --- a/docs/architecture/frontend-build.md +++ b/docs/architecture/frontend-build.md @@ -10,7 +10,7 @@ SvelteKit owns the entry point. `src/app.html` is the page template, and every r `src/routes/+layout.ts` asks `/api/v1/auth/me` who is signed in before any page renders. `src/routes/+layout.svelte` imports `app.css`, provides the TanStack Query client, applies the theme from the user settings query once signed in, and renders the header, the toaster, the page and the footer. [Routing](../frontend/routing.md) covers the routes and their guards. -`src/lib/api/` is generated by `@hey-api/openapi-ts` from `docs/reference/openapi.json` with Prettier as a post-processor. The `typescript` plugin emits types and, with `enums: 'javascript'`, runtime objects for every enum such as `UserRole`. The `sdk` and `client-fetch` plugins emit the request functions and the client. Responses stay plain JSON with no transformer, so a `date-time` field is the ISO string the API sends and the types say `string`; the formatters parse it only to display it. The `zod` plugin emits a schema per model, such as `zUserCreate`, with the spec's validation rules and defaults. The schemas use `zod/mini`: classic Zod builds its faster parser with `new Function`, which the CSP blocks, while mini never tries. The `@tanstack/svelte-query` plugin emits query and mutation options for every operation, covered in [Server state](../frontend/server-state.md). The client, the schemas and those options are re-exported from the index, so everything imports from `#lib/api/index.ts`, and Zod with every schema loads with the first page. Don't edit any of it by hand. After a backend change, `./deploy.sh openapi` rewrites the spec from `create_app()` and `./deploy.sh types` runs `npm run generate:api`. +`src/lib/api/` is generated by `@hey-api/openapi-ts` from `docs/reference/openapi.json` with Prettier as a post-processor. The `typescript` plugin emits types and, with `enums: 'javascript'`, runtime objects for every enum such as `UserRole`. The `sdk` and `client-fetch` plugins emit the request functions and the client. Responses stay plain JSON with no transformer, so a `date-time` field is the ISO string the API sends and the types say `string`; the formatters parse it only to display it. The `zod` plugin emits a schema per model, such as `zUserCreate`, with the spec's validation rules and defaults. The schemas use `zod/mini`: classic Zod builds its faster parser with `new Function`, which the CSP blocks, while mini never tries. The `@tanstack/svelte-query` plugin emits query and mutation options for every operation, covered in [Server state](../frontend/server-state.md). The client, the schemas and those options are re-exported from the index, so everything imports from `#lib/api/index.ts`, and Zod with every schema loads with the first page. Don't edit any of it by hand. After a backend change, `uv run python -m scripts.export_openapi` in backend/ rewrites the spec from `create_app()`, and `npm run generate:api` rewrites the client from it. Tailwind v4 is configured in CSS through the `@tailwindcss/vite` plugin, and there is no `tailwind.config.js`. `src/app.css` imports `tailwindcss`, `styles/components.css` (the shared classes: buttons, badges, cards, alerts, form controls, the page column, fields, label and value lists, stats, links, tables, overlays), `styles/pages.css` (editor and home page layout) and `tw-animate-css` (the open and close animations of the bits-ui overlays), loads `@tailwindcss/forms` with the `class` strategy and `@tailwindcss/typography` for the privacy page's `prose`, and declares the color tokens. Each token is one CSS variable: `:root` holds its light value and `.dark` (which mode-watcher toggles on ``) its dark one, and `@theme inline` turns it into utilities. So `bg-surface` or `text-fg-muted` is right in both modes, and markup never needs a `dark:` variant. @@ -40,6 +40,6 @@ Imports use package.json subpath imports instead of the `$lib` alias that Svelte nginx listens on 5001 with TLS. Its certificate is the one mounted at `/etc/nginx/certs`, or, when nothing is mounted, a self-signed one that a script in the image's `/docker-entrypoint.d/` makes before nginx starts. It proxies `/api/` to `BACKEND_URL` (`https://backend:443` in compose), verifying the backend against the certificate the backend keeps in the `backend_tls_cert` volume, and gives `/api/v1/events/` SSE settings: no buffering and 24 hour timeouts. `/grafana` is a 302 redirect to `GRAFANA_PUBLIC_URL`. Static assets get a one year `expires`, which is safe since their names carry content hashes, and HTML gets `expires -1` so browsers revalidate it. Any path that isn't a file gets `200.html`, so deep links and reloads work. [Nginx configuration](../operations/nginx-configuration.md) walks through the whole file. -`./deploy.sh dev` starts the compose stack, and the `frontend` service there is the same nginx image on `https://localhost:5001`. It has no source mount, so a frontend change needs `./deploy.sh dev --build` to show up. +`docker compose up -d` starts the compose stack, and the `frontend` service there is the same nginx image on `https://localhost:5001`. It has no source mount, so a frontend change needs `docker compose up -d --build frontend` to show up. `npm run dev` runs the Vite dev server on port 5001 with hot module replacement. It uses HTTPS when `./certs/server.key` and `./certs/server.crt` exist (or `SSL_KEY_FILE` and `SSL_CERT_FILE`), and proxies `/api` to `https://backend:443` with the certificate checked, so it needs `NODE_EXTRA_CA_CERTS` pointing at the backend's `server.crt`. The `backend` hostname only resolves inside compose, so API calls only work from a container on `app-network`, with the compose `frontend` service stopped to free the port. diff --git a/docs/components/workers/index.md b/docs/components/workers/index.md index 96a1d6a9..e803d471 100644 --- a/docs/components/workers/index.md +++ b/docs/components/workers/index.md @@ -30,7 +30,7 @@ The entry point is [`backend/app/worker.py`](https://github.com/HardMax71/Integr ## Running locally -`./deploy.sh dev` starts it. To debug it from your shell, stop the container and run it with uv from `backend/`: +`docker compose up -d` starts it. To debug it from your shell, stop the container and run it with uv from `backend/`: ```bash docker compose stop worker diff --git a/docs/getting-started.md b/docs/getting-started.md index c4ffb675..994b1d1e 100644 --- a/docs/getting-started.md +++ b/docs/getting-started.md @@ -19,10 +19,11 @@ That's all the cluster needs. The worker reads k3s's own kubeconfig from `/etc/r ```bash git clone https://github.com/HardMax71/Integr8sCode.git cd Integr8sCode -./deploy.sh dev +cp .env.example .env +docker compose up -d ``` -The first start pulls or builds the images and takes a few minutes. On its first run `./deploy.sh dev` copies `.env.example` to `.env`, which holds the local secrets (JWT key, MongoDB and Redis passwords, the two logins), so there's nothing to fill in. Add `--wait` to block until every service is running, and healthy where it has a healthcheck. +The first start pulls or builds the images and takes a few minutes. `.env` holds the local secrets (JWT key, MongoDB and Redis passwords, the two logins), and `.env.example` has working values, so there's nothing to fill in. Add `--wait` to block until every service is running, and healthy where it has a healthcheck. Check the backend: @@ -56,26 +57,26 @@ Limits are system-wide, not per execution. An execution request carries only the ## Look around -Kafdrop, Grafana and Jaeger aren't part of the default stack. `./deploy.sh dev --observability` adds Grafana, Jaeger, Victoria Metrics, Loki, OpenSearch and the OTel collector, and `--debug` adds Kafdrop on top. All of them listen on localhost only. +Kafdrop, Grafana and Jaeger aren't part of the default stack. `docker compose --profile observability up -d` adds Grafana, Jaeger, Victoria Metrics, Loki, OpenSearch and the OTel collector, and `--profile debug` adds Kafdrop. All of them listen on localhost only. | Service | URL | Needs | |---------|-----|-------| -| Grafana | [http://localhost:3000](http://localhost:3000) (admin / admin123) | `--observability` or `--debug` | -| Jaeger | [http://localhost:16686](http://localhost:16686) | `--observability` or `--debug` | -| Kafdrop | [http://localhost:9000](http://localhost:9000) | `--debug` | +| Grafana | [http://localhost:3000](http://localhost:3000) (admin / admin123) | `--profile observability` | +| Jaeger | [http://localhost:16686](http://localhost:16686) | `--profile observability` | +| Kafdrop | [http://localhost:9000](http://localhost:9000) | `--profile debug` | ## Troubleshooting If scripts stay queued, check `docker compose logs worker`, which logs the Kubernetes error when it can't reach k3s. A k3s that stopped after a VPN or IP change is covered in [Deployment](operations/deployment.md). -If every service times out talking to mongo or kafka right after k3s started or restarted, k3s's policy sync probably wiped Docker's iptables rules ([kube-router#1248](https://github.com/cloudnativelabs/kube-router/issues/1248)). `docker compose down` followed by `./deploy.sh dev` recreates the network and its rules. +If every service times out talking to mongo or kafka right after k3s started or restarted, k3s's policy sync probably wiped Docker's iptables rules ([kube-router#1248](https://github.com/cloudnativelabs/kube-router/issues/1248)). `docker compose down` followed by `docker compose up -d` recreates the network and its rules. If a service fails to start, `docker compose logs ` usually says why. The usual causes are a port already in use (443, 5001, or 27017, 6379 and 9092 on localhost) or Docker running out of memory. ## Stop the stack ```bash -./deploy.sh down +docker compose down ``` To delete the MongoDB, Redis and Kafka data as well: diff --git a/docs/index.md b/docs/index.md index 4fa48296..9514baec 100644 --- a/docs/index.md +++ b/docs/index.md @@ -18,7 +18,8 @@ You need Docker with Compose and a reachable Kubernetes cluster. [Getting starte ```bash git clone https://github.com/HardMax71/Integr8sCode.git cd Integr8sCode -./deploy.sh dev +cp .env.example .env +docker compose up -d ``` Then open [https://localhost:5001](https://localhost:5001) and log in as `user` / `user123`. diff --git a/docs/operations/cicd.md b/docs/operations/cicd.md index 4ea5ab16..c4272ac8 100644 --- a/docs/operations/cicd.md +++ b/docs/operations/cicd.md @@ -34,15 +34,15 @@ graph LR | Label Sync | label-sync.yml | push and PRs to `main` touching the label files or issue forms; Mondays at 03:00; manual | | Dependabot Auto-merge | dependabot-auto-merge.yml | Dependabot's PRs to `main` | -Besides the tests, CI runs one job per check. Ruff runs `ruff check`. MyPy runs `mypy --strict` with `SECRET_KEY` from the `TEST_SECRET_KEY` secret. Grimp runs `python -m scripts.check_orphan_modules`, which fails on any module nothing imports except `app.main` and `__init__.py` files that hold only a docstring. Security Scanning runs `uvx --no-build --python 3.14 bandit==1.9.4 -r backend/ -x backend/tests/ -ll`. Bandit parses with the Python it runs on, so on anything older it skips the files that use 3.14 syntax such as `except A, B:` without failing. The frontend job runs ESLint, Stylelint, the Prettier check, `svelte-check --fail-on-warnings` and a production build. The SBOM workflow generates SPDX SBOMs for backend and frontend with `anchore/sbom-action`, scans them with Grype through `anchore/scan-action` (cutoff `high`, `fail-build: false`) and uploads SARIF to the Security tab. +Besides the tests, CI runs one job per check. Ruff runs `ruff check`. MyPy runs `mypy --strict` with `SECRET_KEY` from the `TEST_SECRET_KEY` secret. Grimp runs `python -m scripts.check_orphan_modules`, which fails on any module nothing imports except `app.main` and `__init__.py` files that hold only a docstring. Security Scanning runs `uvx --no-build --python 3.14 bandit==1.9.4 -r backend/ -x backend/tests/,backend/.venv/ -ll`. Bandit parses with the Python it runs on, so on anything older it skips the files that use 3.14 syntax such as `except A, B:` without failing. The frontend job runs ESLint, Stylelint, the Prettier check, `svelte-check --fail-on-warnings` and a production build. The SBOM workflow generates SPDX SBOMs for backend and frontend with `anchore/sbom-action`, scans them with Grype through `anchore/scan-action` (cutoff `high`, `fail-build: false`) and uploads SARIF to the Security tab. -The docs job installs the backend dependencies, runs `./deploy.sh openapi` and builds with `mkdocs build --strict`. On pushes to `main`, Deploy Docs publishes the result to GitHub Pages and then deletes the `github-pages` deployments before it, so the repository lists only the live one. +The docs job installs the backend dependencies, runs `python -m scripts.export_openapi` and builds with `mkdocs build --strict`. On pushes to `main`, Deploy Docs publishes the result to GitHub Pages and then deletes the `github-pages` deployments before it, so the repository lists only the live one. Labels adds `area:` labels. On a PR, `actions/labeler` matches the changed paths against `.github/labeler.yml`; it runs on `pull_request_target` so fork PRs can be labeled too, and it never checks out the PR's code. On an issue, `github/issue-labeler` matches the Component answer of the issue form against `.github/issue-labeler.yml` and adds the label. It never removes one, so labels added by hand stay. Label Sync keeps labels as code. `.github/labels.yml` lists every label in the repo, and `crazy-max/ghaction-github-labeler` makes the repo match it: it creates and updates labels, renames one that has `from_name`, and deletes any label the file doesn't list, which also takes it off every issue. On a PR it only prints what it would change. Before that, a `yq` step fails the run when `.github/labeler.yml`, `.github/issue-labeler.yml` or an issue form uses a label the file doesn't declare. A new area needs its label in `.github/labels.yml`, a path rule in `.github/labeler.yml`, an option in both issue forms and a pattern in `.github/issue-labeler.yml`. -The `Spec and client match the code` job enforces that `docs/reference/openapi.json` and `frontend/src/lib/api/` only ever come from the generators. It runs `./deploy.sh openapi` and `npm run generate:api` on the checked-out code and fails if either leaves a difference, which catches a hand edit and a spec or client someone forgot to regenerate. +The `Spec and client match the code` job enforces that `docs/reference/openapi.json` and `frontend/src/lib/api/` only ever come from the generators. It runs `python -m scripts.export_openapi` and `npm run generate:api` on the checked-out code and fails if either leaves a difference, which catches a hand edit and a spec or client someone forgot to regenerate. ## Pull requests and merging @@ -53,7 +53,7 @@ CI is the only workflow a PR has to pass, and it runs only what the PR can affec | `backend/` | ruff, mypy, grimp, bandit, backend unit tests, Grafana contract tests | | `grafana/`, `host-metrics/`, `loki/`, `opensearch/`, `otel-collector/`, `victoria-metrics/` | Grafana contract tests, Docs Build | | `frontend/` | frontend lint and type check, frontend unit tests | -| `backend/`, `frontend/`, `.env.example`, compose, deploy.sh or the composite actions | the images and frontend E2E; backend E2E unless only `frontend/` changed | +| `backend/`, `frontend/`, `.env.example`, compose or the composite actions | the images and frontend E2E; backend E2E unless only `frontend/` changed | | `backend/`, the spec, the generated client or its generator config | Spec and client match the code | | `docs/`, mkdocs.yml, or `backend/`, `frontend/` and the service config folders, since the docs include snippets of them | Docs Build | @@ -137,30 +137,22 @@ Dependabot Auto-merge turns on auto-merge for every Dependabot PR, majors includ ## Running checks locally +`pre-commit run --all-files` runs CI's lint and type checks with the same commands: ruff, `mypy --strict`, the orphan module check and bandit on the backend, ESLint, Stylelint, Prettier and svelte-check on the frontend. The unit and contract tests: + ```bash cd backend -uv run ruff check . --config pyproject.toml -uv run mypy --config-file pyproject.toml --strict . -uv run python -m scripts.check_orphan_modules -uvx --no-build --python 3.14 bandit==1.9.4 -r . -x ./tests,./.venv -ll uv run pytest tests/unit uv run pytest tests/contract -m grafana_contract ``` ```bash cd frontend -npm run lint -npm run stylelint -npm run format:check -npm run check npm run test ``` -For E2E, do what CI does: copy `backend/config.test.toml` over `backend/config.toml`, start the stack with `./deploy.sh dev --wait`, then run the suites. +For E2E, do what CI does: add `BACKEND_CONFIG=config.test.toml` to `.env`, so the backend and the worker mount config.test.toml as their config.toml, start the stack with `docker compose up -d --wait`, then run the suites. ```bash docker compose exec -T backend sh -c 'uv sync --group test --no-dev --frozen --no-install-project --no-build && pytest tests/e2e' cd frontend && npm run test:e2e ``` - -`./deploy.sh test` and `./deploy.sh check` don't work right now ([#292](https://github.com/HardMax71/Integr8sCode/issues/292)). diff --git a/docs/operations/deployment.md b/docs/operations/deployment.md index 7506ccd8..7b65cac6 100644 --- a/docs/operations/deployment.md +++ b/docs/operations/deployment.md @@ -4,25 +4,18 @@ Everything runs from one docker-compose.yaml: the backend, the frontend, the wor `backend/Dockerfile` builds on `python:3.14-slim` in three stages. The builder stage runs `uv sync --locked --no-dev --no-install-project --no-build`, the `base` stage copies only the virtualenv plus a few runtime libraries, and the last stage adds the application code. On first start the container makes a self-signed certificate and keeps it in the `backend_tls_key` and `backend_tls_cert` volumes, then it seeds the default users and starts gunicorn with uvicorn workers on port 443 over TLS. nginx mounts the certificate volume and verifies the backend against it. The frontend image builds the Svelte app with `node:26-alpine` and serves it from `nginx:1.31-alpine`, see [Nginx configuration](nginx-configuration.md). -## deploy.sh - -[deploy.sh](https://github.com/HardMax71/Integr8sCode/blob/main/deploy.sh) is a thin wrapper around Docker Compose: - -```bash ---8<-- "deploy.sh:usage" -``` - -`check` runs the static CI gates on the host: ruff, `mypy --strict`, the orphan module check and bandit. `test` runs the unit and Grafana contract tests on the host, then starts the stack with `BACKEND_CONFIG=config.test.toml` and runs the e2e suite inside the backend container. That variable makes the backend and the worker mount config.test.toml as their config.toml, the way CI copies it over. [CI/CD](cicd.md#running-checks-locally) lists the same commands one by one. - ## Local stack ```bash -./deploy.sh dev # core stack -./deploy.sh dev --observability # plus Grafana, Jaeger, Loki, Victoria Metrics, OTel Collector, host-metrics -./deploy.sh dev --debug # all of the above plus Kafdrop +cp .env.example .env # once +docker compose up -d # core stack +docker compose --profile observability up -d # plus Grafana, Jaeger, Loki, OpenSearch, Victoria Metrics, OTel Collector, host-metrics +docker compose --profile debug up -d # plus Kafdrop ``` -The core stack is MongoDB, Redis, Kafka (KRaft, no ZooKeeper), the backend, the frontend and the worker. No container runs once and exits. Secrets come from `.env`, which `./deploy.sh dev` copies from `.env.example` on its first run; compose has no defaults for them and refuses to start while one is empty. Kafka creates topics on first use (`KAFKA_AUTO_CREATE_TOPICS_ENABLE`). Topic names are the event types themselves, such as `execution_requested`, with no prefix. +`--build` rebuilds the images first and `--wait` returns once every service is running, and healthy where it has a healthcheck. `docker compose logs -f ` follows one service and `docker compose ps` lists them. + +The core stack is MongoDB with its backup, Redis, Kafka (KRaft, no ZooKeeper), the backend, the frontend and the worker. No container runs once and exits. Secrets come from `.env`, your copy of `.env.example`; compose has no defaults for them and refuses to start while one is empty. Kafka creates topics on first use (`KAFKA_AUTO_CREATE_TOPICS_ENABLE`). Topic names are the event types themselves, such as `execution_requested`, with no prefix. | Service | URL | Needs | |--------------------|-------------------------|-------------------------------| @@ -31,11 +24,11 @@ The core stack is MongoDB, Redis, Kafka (KRaft, no ZooKeeper), the backend, the | Kafka (host) | localhost:9092 | | | MongoDB | 127.0.0.1:27017 | | | Redis | 127.0.0.1:6379 | | -| Grafana | http://localhost:3000 | `--observability` or `--debug` | -| Jaeger UI | http://localhost:16686 | `--observability` or `--debug` | -| Victoria Metrics | http://localhost:8428 | `--observability` or `--debug` | -| Loki | http://localhost:3100 | `--observability` or `--debug` | -| Kafdrop | http://localhost:9000 | `--debug` | +| Grafana | http://localhost:3000 | `--profile observability` | +| Jaeger UI | http://localhost:16686 | `--profile observability` | +| Victoria Metrics | http://localhost:8428 | `--profile observability` | +| Loki | http://localhost:3100 | `--profile observability` | +| Kafdrop | http://localhost:9000 | `--profile debug` | Inside the compose network Kafka is `kafka:29092`. From the host use `localhost:9092`, since that's the listener advertised for host clients. @@ -49,7 +42,7 @@ Data lives in named volumes: --8<-- "docker-compose.yaml:dev_volumes" ``` -`./deploy.sh down` stops the stack and keeps them. `docker compose down -v` deletes them too. +`docker compose down` stops the stack and keeps them, and `docker compose down -v` deletes them too. ## Healthchecks @@ -197,14 +190,8 @@ The images on GHCR are public, so the server pulls them without logging in. To s Every service logs through Docker's `fluentd` driver to the collector, which passes the lines to Loki ([Log aggregation](loki.md)), set once as `x-logging` at the top of docker-compose.yaml. Docker keeps a local copy for `docker logs` of at most 3 files of 10MB per container (`cache-max-size`, `cache-max-file`), and the collector itself uses the `local` driver with the same limits. The default `json-file` driver never rotates, and its logs once filled the production disk. -`./deploy.sh backup` dumps MongoDB as a gzip archive into /var/backups/mongo (or the directory you pass), skips the run when less than 5G is free there, and deletes dumps older than 14 days. Production runs it every night from /etc/cron.d/integr8scode-backup: - -```text -30 3 * * * root /opt/Integr8sCode/deploy.sh backup >> /var/log/integr8scode-backup.log 2>&1 -``` - -The dumps sit on the same disk as the database, so they cover a bad write or a failed upgrade, not a lost server. To restore one: +The `mongo-backup` service dumps MongoDB with `mongodump` into the `mongo_backups` volume when it starts and every 24 hours after, one `mongo-.archive.gz` a day, and deletes dumps older than 14 days. It runs on the same mongo image as the database, the same way locally and in production. The dumps sit on the same disk as the database, so they cover a bad write or a failed upgrade, not a lost server. `docker compose exec mongo-backup ls /backup` lists them, and this restores one: ```bash -docker compose exec -T mongo sh -c 'mongorestore -u "$MONGO_INITDB_ROOT_USERNAME" -p "$MONGO_INITDB_ROOT_PASSWORD" --authenticationDatabase admin --archive --gzip --drop' < /var/backups/mongo/mongo-2026-10-13.archive.gz +docker compose exec -T mongo-backup sh -c 'mongorestore --host mongo -u "$MONGO_USER" -p "$MONGO_PASSWORD" --authenticationDatabase admin --archive=/backup/mongo-2026-10-24.archive.gz --gzip --drop' ``` diff --git a/docs/operations/metrics-reference.md b/docs/operations/metrics-reference.md index 88bceacd..87b00474 100644 --- a/docs/operations/metrics-reference.md +++ b/docs/operations/metrics-reference.md @@ -17,7 +17,7 @@ flowchart LR `create_app` and the worker's `main()` each call `setup_metrics` (`app/core/metrics/base.py`) once. It installs a global `MeterProvider` whose `PeriodicExportingMetricReader` exports every 60 000 ms through an OTLP gRPC exporter. The exporter targets `OTEL_EXPORTER_OTLP_ENDPOINT`, which `config.toml` sets to `http://otel-collector:4317`. It also starts the system metrics instrumentation for the process's own CPU and memory. Without an endpoint it returns early and the no-op provider stays. The OpenTelemetry instrumentations take their meters from that provider. The app's own metrics have no meter at all: they come from spans, described next. -The collector's metrics pipeline (`otel-collector/otel-collector-config.yaml`) reads the `otlp`, `kafka_metrics` and `mongodb` receivers, runs `memory_limiter`, `batch` and `resource`, and exports with `prometheus_remote_write` to `http://victoria-metrics:8428/api/v1/write`. VictoriaMetrics keeps 30 days. Host metrics come from a second collector, the `host-metrics` service (`host-metrics/otel-host-metrics.yaml`). It runs only the `host_metrics` receiver in the host's network namespace, since in a container the network scraper sees only that container's `eth0` and `lo`, and sends to the main collector's OTLP port on `127.0.0.1:4317`. Both collectors, VictoriaMetrics and Grafana sit in the `observability` compose profile, so none of this runs unless you start the stack with `./deploy.sh dev --observability`. Counters and histograms made from spans take another way, described next. +The collector's metrics pipeline (`otel-collector/otel-collector-config.yaml`) reads the `otlp`, `kafka_metrics` and `mongodb` receivers, runs `memory_limiter`, `batch` and `resource`, and exports with `prometheus_remote_write` to `http://victoria-metrics:8428/api/v1/write`. VictoriaMetrics keeps 30 days. Host metrics come from a second collector, the `host-metrics` service (`host-metrics/otel-host-metrics.yaml`). It runs only the `host_metrics` receiver in the host's network namespace, since in a container the network scraper sees only that container's `eth0` and `lo`, and sends to the main collector's OTLP port on `127.0.0.1:4317`. Both collectors, VictoriaMetrics and Grafana sit in the `observability` compose profile, so none of this runs unless you start the stack with `docker compose --profile observability up -d`. Counters and histograms made from spans take another way, described next. ## Metrics made from spans diff --git a/docs/reference/api-reference.md b/docs/reference/api-reference.md index f8edb2d1..885cb5e0 100644 --- a/docs/reference/api-reference.md +++ b/docs/reference/api-reference.md @@ -2,6 +2,6 @@ Every route lives under `/api/v1`. Login sets a JWT in the httpOnly `access_token` cookie and a readable `csrf_token` cookie. Any request other than GET, HEAD or OPTIONS from a logged-in user must echo that token in an `X-CSRF-Token` header; login and register are exempt. -The page below renders `docs/reference/openapi.json`, which `./deploy.sh openapi` regenerates from the FastAPI app. The running backend serves its own `/docs` only when `DEVELOPMENT_MODE` is on. +The page below renders `docs/reference/openapi.json`, which `uv run python -m scripts.export_openapi` in backend/ regenerates from the FastAPI app. The running backend serves its own `/docs` only when `DEVELOPMENT_MODE` is on. diff --git a/docs/reference/configuration.md b/docs/reference/configuration.md index 9b1ea646..9a358661 100644 --- a/docs/reference/configuration.md +++ b/docs/reference/configuration.md @@ -18,7 +18,7 @@ In the tables, the value is what applies with the shipped `config.toml`. The sou ## Secrets -For each secret, `Settings` takes the environment variable if it's set and non-empty, then the TOML value, then a dev default. Compose passes all four to every backend service through the `x-backend-secrets` anchor and has no defaults of its own: the values come from `.env` (`./deploy.sh dev` copies `.env.example`, the production deploy writes it from the repository secrets), and compose refuses to start while one is empty. +For each secret, `Settings` takes the environment variable if it's set and non-empty, then the TOML value, then a dev default. Compose passes all four to every backend service through the `x-backend-secrets` anchor and has no defaults of its own: the values come from `.env` (locally a copy of `.env.example`, in production written by the deploy from the repository secrets), and compose refuses to start while one is empty. | Variable | Purpose | Dev default | |----------|---------|-------------| @@ -111,7 +111,7 @@ A language can raise its own pod's limits above the system settings. Go gets at | `HOSTNAME` | the machine's hostname | default | `service.instance.id` on logs and spans; inside a container that's the container ID | | `LOG_LEVEL` | `INFO` | config.toml | structlog level | -Jaeger and the OTel collector only run with `./deploy.sh dev --observability` or `--debug`. Without them the exporters log connection errors and the app keeps working. +Jaeger and the OTel collector only run with the `observability` compose profile. Without them the exporters log connection errors and the app keeps working. ## Server diff --git a/docs/testing/frontend-testing.md b/docs/testing/frontend-testing.md index bb5e170f..171b7f03 100644 --- a/docs/testing/frontend-testing.md +++ b/docs/testing/frontend-testing.md @@ -79,7 +79,7 @@ When a `vi.mock` factory refers to mock functions, create them in `vi.hoisted()` --8<-- "frontend/playwright.config.ts:config" ``` -The suite runs against the stack on `https://localhost:5001`, which is the compose frontend that `./deploy.sh dev` starts. `npm run dev` can't stand in for it, since its proxy sends `/api` to a hostname that only resolves inside the compose network. The only account the suite expects is the seeded admin, whose password it reads from `ADMIN_USER_PASSWORD` like seed_users.py does, falling back to `admin123`. +The suite runs against the stack on `https://localhost:5001`, which is the compose frontend that `docker compose up -d` starts. `npm run dev` can't stand in for it, since its proxy sends `/api` to a hostname that only resolves inside the compose network. The only account the suite expects is the seeded admin, whose password it reads from `ADMIN_USER_PASSWORD` like seed_users.py does, falling back to `admin123`. [`e2e/fixtures.ts`](https://github.com/HardMax71/Integr8sCode/blob/main/frontend/e2e/fixtures.ts) registers one account per worker through the API and signs it and the seeded admin in once per worker, so two workers never share a user's settings, scripts or notifications. Every test gets a fresh browser context: `page` for a guest, and `userPage` or `adminPage` starting with the worker's session cookies, so nothing one test leaves in localStorage reaches the next. The `api` fixture is an API client signed in as the worker's user, and `execution` starts an execution through it for the tests that need one, such as the notification and admin table tests.