diff --git a/.github/workflows/release-deploy.yml b/.github/workflows/release-deploy.yml index 45437a8f..13514373 100644 --- a/.github/workflows/release-deploy.yml +++ b/.github/workflows/release-deploy.yml @@ -200,7 +200,6 @@ jobs: umask 077 cat > .env <` follows one service and `docker compose ps` lists them. -The core stack is MongoDB with its backup, Redis, Kafka (KRaft, no ZooKeeper), the backend, the frontend and the worker. No container runs once and exits. Secrets come from `.env`, your copy of `.env.example`; compose has no defaults for them and refuses to start while one is empty. Kafka creates topics on first use (`KAFKA_AUTO_CREATE_TOPICS_ENABLE`). Topic names are the event types themselves, such as `execution_requested`, with no prefix. +The stack is MongoDB with its backup, Redis, Kafka (KRaft, no ZooKeeper), the backend, the frontend, the worker and the observability services, the same locally and in production. No container runs once and exits. Secrets come from `.env`, your copy of `.env.example`; compose has no defaults for them and refuses to start while one is empty. Kafka creates topics on first use (`KAFKA_AUTO_CREATE_TOPICS_ENABLE`). Topic names are the event types themselves, such as `execution_requested`, with no prefix. | Service | URL | Needs | |--------------------|-------------------------|-------------------------------| @@ -24,10 +23,10 @@ The core stack is MongoDB with its backup, Redis, Kafka (KRaft, no ZooKeeper), t | Kafka (host) | localhost:9092 | | | MongoDB | 127.0.0.1:27017 | | | Redis | 127.0.0.1:6379 | | -| Grafana | http://localhost:3000 | `--profile observability` | -| Jaeger UI | http://localhost:16686 | `--profile observability` | -| Victoria Metrics | http://localhost:8428 | `--profile observability` | -| Loki | http://localhost:3100 | `--profile observability` | +| Grafana | http://localhost:3000 | | +| Jaeger UI | http://localhost:16686 | | +| Victoria Metrics | http://localhost:8428 | | +| Loki | http://localhost:3100 | | | Kafdrop | http://localhost:9000 | `--profile debug` | Inside the compose network Kafka is `kafka:29092`. From the host use `localhost:9092`, since that's the listener advertised for host clients. @@ -174,7 +173,7 @@ IMAGE_TAG=sha-abc1234 docker compose up -d --no-build ## Production -Production gets one release a day. Every merge to `main` goes through CI and Docker Scan & Promote, which moves `latest`; at 23:00 Europe/Berlin [Release & Deploy](cicd.md#release-deploy) releases whatever `latest` is and deploys it, and skips the night when nothing new passed. The deploy job SSHes into the server, removes images no container uses, checks out the release's git tag in `/opt/Integr8sCode` (so the compose file and config match the images) and writes `/opt/Integr8sCode/.env` (mode 600) from the repo's secrets (`SECRET_KEY`, `MONGO_USER`, `MONGO_PASSWORD`, `MONGO_MONITOR_PASSWORD`, `REDIS_PASSWORD`, `ADMIN_USER_PASSWORD`, `DEFAULT_USER_PASSWORD`, the Grafana and Mailjet values) together with `IMAGE_TAG`, `COMPOSE_PROFILES=observability` and `GRAFANA_ROOT_URL=https://grafana.integr8scode.cc/`. Compose reads that file for every command, so the backup cron and commands run by hand on the server use the same values, and a secret that came through empty stops compose before anything starts. The values sit in single quotes, so a secret can't contain one. Then it runs: +Production gets one release a day. Every merge to `main` goes through CI and Docker Scan & Promote, which moves `latest`; at 23:00 Europe/Berlin [Release & Deploy](cicd.md#release-deploy) releases whatever `latest` is and deploys it, and skips the night when nothing new passed. The deploy job SSHes into the server, removes images no container uses, checks out the release's git tag in `/opt/Integr8sCode` (so the compose file and config match the images) and writes `/opt/Integr8sCode/.env` (mode 600) from the repo's secrets (`SECRET_KEY`, `MONGO_USER`, `MONGO_PASSWORD`, `MONGO_MONITOR_PASSWORD`, `REDIS_PASSWORD`, `ADMIN_USER_PASSWORD`, `DEFAULT_USER_PASSWORD`, the Grafana and Mailjet values) together with `IMAGE_TAG` and `GRAFANA_ROOT_URL=https://grafana.integr8scode.cc/`. Compose reads that file for every command, so the backup cron and commands run by hand on the server use the same values, and a secret that came through empty stops compose before anything starts. The values sit in single quotes, so a secret can't contain one. Then it runs: ```bash docker compose pull diff --git a/docs/operations/grafana-integration.md b/docs/operations/grafana-integration.md index c21122d6..78e8e132 100644 --- a/docs/operations/grafana-integration.md +++ b/docs/operations/grafana-integration.md @@ -1,6 +1,6 @@ # Grafana integration -Grafana runs under the `observability` compose profile. Everything it shows is provisioned from [`grafana/`](https://github.com/HardMax71/Integr8sCode/tree/main/grafana): three datasources, thirteen dashboards, thirteen alert rules, an email contact point and a heartbeat webhook. Locally it's at `http://localhost:3000`; in production nginx serves it on `grafana.integr8scode.cc` (see [Nginx configuration](nginx-configuration.md#grafana-server)). Anonymous access is off. The admin login comes from `GRAFANA_ADMIN_USER` and `GRAFANA_ADMIN_PASSWORD` (default `admin` / `admin123`), and the entrypoint resets the password to that value on every start. +Everything Grafana shows is provisioned from [`grafana/`](https://github.com/HardMax71/Integr8sCode/tree/main/grafana): three datasources, thirteen dashboards, thirteen alert rules, an email contact point and a heartbeat webhook. Locally it's at `http://localhost:3000`; in production nginx serves it on `grafana.integr8scode.cc` (see [Nginx configuration](nginx-configuration.md#grafana-server)). Anonymous access is off. The admin login comes from `GRAFANA_ADMIN_USER` and `GRAFANA_ADMIN_PASSWORD` (default `admin` / `admin123`), and the entrypoint resets the password to that value on every start. ## Datasources diff --git a/docs/operations/logging.md b/docs/operations/logging.md index a30f6998..eae48b1c 100644 --- a/docs/operations/logging.md +++ b/docs/operations/logging.md @@ -78,6 +78,6 @@ That keeps fields queryable, and it stops log injection. If user input goes into The browser logs through [LogTape](https://logtape.org), configured once in `init` in `frontend/src/hooks.client.ts`, under the `integr8scode` category plus an area (`api`, `auth`, `editor`, `notifications`, `replay`, `app`). Values travel as properties named by `{placeholders}` in the message, the same rule as the backend's keyword arguments, and ESLint's `no-console` keeps every other way of logging out of `src/`. -Nothing goes to the browser console. Every build sends every record to the collector the way the backend sends its logs: LogTape's OpenTelemetry sink ([@logtape/otel](https://github.com/dahlia/logtape/tree/main/packages/otel)) hands them to an OTel `LoggerProvider` with a `BatchLogRecordProcessor` and an `OTLPLogExporter`, which posts OTLP JSON to `/otlp/v1/logs`. nginx passes that to the collector ([Nginx](nginx-configuration.md#browser-logs)), and Loki labels the lines `service_name="browser"`. What gets dropped is the collector's call, not the browser's: `filter/browser` drops lines below `warning` when `ENVIRONMENT` is `production` and keeps everything in any other environment ([Loki](loki.md#collector)). Without the observability profile nginx answers 502 and the records are lost. Besides the errors the app logs itself, `handleError` logs what breaks while a page loads or renders, and two window listeners log an uncaught error and an unhandled promise rejection. An `error` event without an error object, like Chromium's "ResizeObserver loop completed with undelivered notifications", is skipped. +Nothing goes to the browser console. Every build sends every record to the collector the way the backend sends its logs: LogTape's OpenTelemetry sink ([@logtape/otel](https://github.com/dahlia/logtape/tree/main/packages/otel)) hands them to an OTel `LoggerProvider` with a `BatchLogRecordProcessor` and an `OTLPLogExporter`, which posts OTLP JSON to `/otlp/v1/logs`. nginx passes that to the collector ([Nginx](nginx-configuration.md#browser-logs)), and Loki labels the lines `service_name="browser"`. What gets dropped is the collector's call, not the browser's: `filter/browser` drops lines below `warning` when `ENVIRONMENT` is `production` and keeps everything in any other environment ([Loki](loki.md#collector)). While the collector is down nginx answers 502 and the records are lost. Besides the errors the app logs itself, `handleError` logs what breaks while a page loads or renders, and two window listeners log an uncaught error and an unhandled promise rejection. An `error` event without an error object, like Chromium's "ResizeObserver loop completed with undelivered notifications", is skipped. Three things in that setup are there for the browser. The sink's default setup loads its exporter with an `import()` a browser can't resolve, so `init` builds the `LoggerProvider` itself, as [LogTape #166](https://github.com/dahlia/logtape/issues/166) suggests. Its default object renderer falls back to `JSON.stringify` in a browser, and an Error comes out as `{}`, so the sink uses `objectRenderer: 'json'`, which writes the error's name, message and stack into the line. And @logtape/otel 2.4 asks for the OTel 0.208 packages, whose `@opentelemetry/core` carries [GHSA-8988-4f7v-96qf](https://github.com/advisories/GHSA-8988-4f7v-96qf), so the `overrides` in package.json move its SDK and exporters to the 0.223 line. `@opentelemetry/api-logs` stays at 0.208 for it, since it imports `NOOP_LOGGER` and later versions dropped that export. diff --git a/docs/operations/loki.md b/docs/operations/loki.md index cc77710e..978b3326 100644 --- a/docs/operations/loki.md +++ b/docs/operations/loki.md @@ -1,6 +1,6 @@ # Log aggregation (Loki) -Loki stores the logs of the backend and the worker, and the output of every container, so you can query them in Grafana across processes and time. It runs only with the `observability` compose profile. Without it, `docker compose logs` still shows each container's output from Docker's local cache. +Loki stores the logs of the backend and the worker, and the output of every container, so you can query them in Grafana across processes and time. Without it, `docker compose logs` still shows each container's output from Docker's local cache. ```mermaid flowchart LR @@ -31,7 +31,7 @@ Every compose service logs through Docker's `fluentd` log driver (the `x-logging Docker's lines arrive without a level, and Loki's own guess marks Mongo's and Redis's info lines as errors, so `transform/level` gives each one from its format before Loki sees it: a JSON or logfmt `level`, Mongo's `"s"`, a zap tab-separated level, nginx's `[error]`, then a level word such as log4j's or gunicorn's `ERROR`, a traceback's first line and Redis's marker after the time, and only then a plain `error` or `warn` word. Records that already have a level, like the backend's and worker's over OTLP, keep theirs. The [Errors dashboard](grafana-integration.md#dashboards) and the error-lines alert read the result through `detected_level`. -With `fluentd-async` a container starts and buffers its lines while the collector is down or not in the profile, and `docker logs` reads Docker's local copy (3 files of 10 MB per container). The collector's own container keeps Docker's `local` driver, since its debug exporter would otherwise print its own lines back to itself. +With `fluentd-async` a container starts and buffers its lines while the collector is down, and `docker logs` reads Docker's local copy (3 files of 10 MB per container). The collector's own container keeps Docker's `local` driver, since its debug exporter would otherwise print its own lines back to itself. Loki 3 accepts OTLP natively. Resource attributes such as `service.name` and `service.namespace` become stream labels (`service_name`, `service_namespace`). The record's `trace_id` and `span_id` become structured metadata. For our own events the log line is the JSON that structlog rendered. Third-party library records arrive as plain text, so `| json` can't pull fields out of them. diff --git a/docs/operations/metrics-reference.md b/docs/operations/metrics-reference.md index 87b00474..9bd23f67 100644 --- a/docs/operations/metrics-reference.md +++ b/docs/operations/metrics-reference.md @@ -17,7 +17,7 @@ flowchart LR `create_app` and the worker's `main()` each call `setup_metrics` (`app/core/metrics/base.py`) once. It installs a global `MeterProvider` whose `PeriodicExportingMetricReader` exports every 60 000 ms through an OTLP gRPC exporter. The exporter targets `OTEL_EXPORTER_OTLP_ENDPOINT`, which `config.toml` sets to `http://otel-collector:4317`. It also starts the system metrics instrumentation for the process's own CPU and memory. Without an endpoint it returns early and the no-op provider stays. The OpenTelemetry instrumentations take their meters from that provider. The app's own metrics have no meter at all: they come from spans, described next. -The collector's metrics pipeline (`otel-collector/otel-collector-config.yaml`) reads the `otlp`, `kafka_metrics` and `mongodb` receivers, runs `memory_limiter`, `batch` and `resource`, and exports with `prometheus_remote_write` to `http://victoria-metrics:8428/api/v1/write`. VictoriaMetrics keeps 30 days. Host metrics come from a second collector, the `host-metrics` service (`host-metrics/otel-host-metrics.yaml`). It runs only the `host_metrics` receiver in the host's network namespace, since in a container the network scraper sees only that container's `eth0` and `lo`, and sends to the main collector's OTLP port on `127.0.0.1:4317`. Both collectors, VictoriaMetrics and Grafana sit in the `observability` compose profile, so none of this runs unless you start the stack with `docker compose --profile observability up -d`. Counters and histograms made from spans take another way, described next. +The collector's metrics pipeline (`otel-collector/otel-collector-config.yaml`) reads the `otlp`, `kafka_metrics` and `mongodb` receivers, runs `memory_limiter`, `batch` and `resource`, and exports with `prometheus_remote_write` to `http://victoria-metrics:8428/api/v1/write`. VictoriaMetrics keeps 30 days. Host metrics come from a second collector, the `host-metrics` service (`host-metrics/otel-host-metrics.yaml`). It runs only the `host_metrics` receiver in the host's network namespace, since in a container the network scraper sees only that container's `eth0` and `lo`, and sends to the main collector's OTLP port on `127.0.0.1:4317`. Counters and histograms made from spans take another way, described next. ## Metrics made from spans diff --git a/docs/operations/nginx-configuration.md b/docs/operations/nginx-configuration.md index 69260385..77c783c9 100644 --- a/docs/operations/nginx-configuration.md +++ b/docs/operations/nginx-configuration.md @@ -69,7 +69,7 @@ The SSE location under `/api/v1/events/` speaks HTTP/1.1 with an empty `Connecti --8<-- "frontend/nginx.conf.template:browser_logs" ``` -The app sends its log records here as OTLP JSON, at most one batch a second, and nginx passes them to the collector's `otlp/browser` receiver ([Loki](loki.md#collector) has the rest of the path). Anyone can post to this path, so it takes POST only, bodies up to 64 KB, and 30 requests a minute per address with a burst of 30, from the `limit_req_zone` at the top of the file; past that nginx answers 429. The upstream goes through a variable like the backend's, so nginx starts without the observability profile and this path returns 502 instead. +The app sends its log records here as OTLP JSON, at most one batch a second, and nginx passes them to the collector's `otlp/browser` receiver ([Loki](loki.md#collector) has the rest of the path). Anyone can post to this path, so it takes POST only, bodies up to 64 KB, and 30 requests a minute per address with a burst of 30, from the `limit_req_zone` at the top of the file; past that nginx answers 429. The upstream goes through a variable like the backend's, so nginx starts while the collector is down and this path returns 502 until it's back. ### Static files and SPA routing @@ -89,7 +89,7 @@ SvelteKit writes the JavaScript and CSS under `_app/immutable/` with content has Grafana gets its own hostname, `grafana.integr8scode.cc`, and is proxied at the root. grafana.ini sets no `root_url`; compose passes `GF_SERVER_ROOT_URL` from `GRAFANA_ROOT_URL`. `/api/live/` is the Grafana Live websocket, so it gets the `Upgrade` and `Connection` headers. -The upstream goes through a variable with Docker's resolver (`127.0.0.11`), so nginx resolves `grafana` per request instead of at startup. Without that, nginx would refuse to start when the observability profile is off. In that case the Grafana host just returns 502. This server block sets no security headers. +The upstream goes through a variable with Docker's resolver (`127.0.0.11`), so nginx resolves `grafana` per request instead of at startup. Without that, nginx would refuse to start while Grafana is down. In that case the Grafana host just returns 502. This server block sets no security headers. ## Environment variables diff --git a/docs/operations/opensearch.md b/docs/operations/opensearch.md index 8a9d2eb3..120e69cb 100644 --- a/docs/operations/opensearch.md +++ b/docs/operations/opensearch.md @@ -1,6 +1,6 @@ # Log search (OpenSearch) -OpenSearch holds the same log lines as [Loki](loki.md), with every field indexed, so a search by execution id, user id or error text over 30 days answers in milliseconds instead of reading every chunk. Loki stays for now: the plan in [#373](https://github.com/HardMax71/Integr8sCode/issues/373) is to run both on real traffic, compare, and then drop one. OpenSearch runs with the `observability` compose profile, next to Loki. +OpenSearch holds the same log lines as [Loki](loki.md), with every field indexed, so a search by execution id, user id or error text over 30 days answers in milliseconds instead of reading every chunk. Loki stays for now: the plan in [#373](https://github.com/HardMax71/Integr8sCode/issues/373) is to run both on real traffic, compare, and then drop one. ```mermaid flowchart LR diff --git a/docs/operations/tracing.md b/docs/operations/tracing.md index 66e59be9..6008227f 100644 --- a/docs/operations/tracing.md +++ b/docs/operations/tracing.md @@ -61,6 +61,6 @@ The API stores the event in MongoDB, publishes `execution_requested` and answers ## Jaeger -Jaeger runs as `jaegertracing/jaeger:2.20.0` under the `observability` profile, with in-memory storage capped at 50 000 traces (see [Retention and sampling](deployment.md#retention-and-sampling)) and the UI on `http://localhost:16686`. OTLP ports 4317 and 4318 stay on the compose network. Grafana also has a Jaeger datasource. Jaeger is pinned below 2.21 because 2.21 removed the search API that datasource calls ([grafana/grafana#112336](https://github.com/grafana/grafana/issues/112336)), so trace search in Grafana would stop working. +Jaeger runs as `jaegertracing/jaeger:2.20.0`, with in-memory storage capped at 50 000 traces (see [Retention and sampling](deployment.md#retention-and-sampling)) and the UI on `http://localhost:16686`. OTLP ports 4317 and 4318 stay on the compose network. Grafana also has a Jaeger datasource. Jaeger is pinned below 2.21 because 2.21 removed the search API that datasource calls ([grafana/grafana#112336](https://github.com/grafana/grafana/issues/112336)), so trace search in Grafana would stop working. To turn tracing off, set `TRACING_SAMPLING_RATE = 0`. Leaving `OTLP_TRACES_ENDPOINT` empty only stops the export; spans are still recorded in process, so log lines keep their `trace_id`. diff --git a/docs/reference/configuration.md b/docs/reference/configuration.md index 9a358661..4651208f 100644 --- a/docs/reference/configuration.md +++ b/docs/reference/configuration.md @@ -111,7 +111,7 @@ A language can raise its own pod's limits above the system settings. Go gets at | `HOSTNAME` | the machine's hostname | default | `service.instance.id` on logs and spans; inside a container that's the container ID | | `LOG_LEVEL` | `INFO` | config.toml | structlog level | -Jaeger and the OTel collector only run with the `observability` compose profile. Without them the exporters log connection errors and the app keeps working. +While the OTel collector is down, the exporters log connection errors and the app keeps working. ## Server