diff --git a/.github/PULL_REQUEST_TEMPLATE.md b/.github/PULL_REQUEST_TEMPLATE.md index 91a9787a..5996a3a5 100644 --- a/.github/PULL_REQUEST_TEMPLATE.md +++ b/.github/PULL_REQUEST_TEMPLATE.md @@ -1,5 +1,5 @@ # Servicename: context - + ## Description @@ -17,7 +17,9 @@ - [ ] I have performed a self-review of my code and followed the [templates](https://github.com/tailscale-dev/ScaleTail/tree/main/templates/service-template) structure. - [ ] I have added verification that the stack works as expected. - [ ] I have updated necessary documentation (e.g. frontpage [README.md](https://github.com/tailscale-dev/ScaleTail/blob/main/README.md) ). -- [ ] I have selected the correct label(s) for this PR. +- [ ] I ran Compose validation with dummy values for required variables. +- [ ] I ran rumdl on the whole repository. +- [ ] I committed no working credential. ## Additional Context diff --git a/AGENTS.md b/AGENTS.md index 97bf57b4..d6f08e23 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -1,26 +1,62 @@ # Agent guidance -Follow [CONTRIBUTING.md](CONTRIBUTING.md) and the current service template. +ScaleTail is a collection of Docker Compose stacks. Each directory in `services/` is one stack, which usually runs a self-hosted application behind a Tailscale sidecar container. + +## Where the rules live + +- [CONTRIBUTING.md](CONTRIBUTING.md): how to add, update, verify, and submit a service. Follow it. +- [templates/service-template/](templates/service-template/): the starting point for every service. Its comments explain each setting. +- [documentation/standard-setup.md](documentation/standard-setup.md): the shared setup. A service README lists only what differs, under "Deviations from the standard setup". +- [.github/workflows/](.github/workflows/): the Compose validation and Markdown lint that CI runs. ## Before you edit -- Inspect the target service and its README. -- Confirm ports, commands, volumes, and variables in the image's upstream docs. -- Preserve service-specific exceptions that still apply. +- Read the service's `compose.yaml`, `.env`, and README, including "Deviations from the standard setup". +- Confirm ports, commands, volumes, variables, and health checks in the image's upstream docs. +- Preserve documented exceptions. Some stacks use other Compose service keys or container names, and other containers or `.env` values refer to those names. - Keep changes focused on the requested service or documentation. -- Never commit working auth keys, passwords, tokens, or other credentials. +- Apply the rules to new stacks and to the parts of an existing stack that you change. Existing stacks are brought in line over time, so do not fix the rest of a stack unless that is the task. + +## Files that change together + +- Add, rename, or remove a service: the `services//` directory and its row in the root `README.md`. Insert a new row in alphabetical order by the bold name, and leave the other rows alone. +- Make existing users act, such as with a new required secret, a moved volume, or a changed `SERVICE`: add an "Upgrading" section to the service README. `SERVICE` also names the Tailnet device, the containers, and most data folders. +- Change the template: also update `documentation/standard-setup.md` and `CONTRIBUTING.md` where they describe the same setting. Validate it from `templates/service-template/` with `SERVICE=dummy IMAGE_URL=dummy docker compose config --quiet`, because the template leaves both values empty. + +## Secrets and local data + +- Every `.env` file is tracked and published. Never commit an auth key, password, token, or other credential. +- Leave each secret empty in `.env`. When the stack cannot start without it, read it in `compose.yaml` as `${VAR:?Set VAR in .env}`, and add a `# Required:` comment above it in `.env` that says what to set, such as how to generate it. Keep `TS_AUTHKEY` optional, because only the first start needs it. +- Start test stacks from a copy outside the repository, such as `cp -R services/ /tmp/`, and put real values only in the copy. A running stack writes `./config`, `./ts/state` (with the device key), and its data folders next to `compose.yaml`. +- Before you commit, run `git status` and stage only the files you meant to change. ## Verification -Run `docker compose config --quiet` from each changed service directory. When -possible, start the stack and test its main function through the Tailnet. +From each changed service directory, validate Compose. The command gives each required variable a dummy value, as CI does, so you do not edit `.env`: -From the repository root, lint the changed Markdown files. Replace the example paths below as needed: +```sh +env $(grep -ohE '\$\{[A-Za-z0-9_]+:\?' compose.yaml .env | sed -E 's/^\$\{//; s/:\?$/=dummy/' | sort -u) docker compose config --quiet +``` + +It must exit with status 0. CI fails on "variable is not set" warnings. + +From the repository root, lint all Markdown with the rumdl version that `.github/workflows/linting.yml` pins in its `version:` input, and check your changes for whitespace errors: ```sh -rumdl check --config .markdownlint.yml AGENTS.md CONTRIBUTING.md -git diff --check +version=$(sed -nE 's/.*version: "([0-9.]+)".*/\1/p' .github/workflows/linting.yml) +uvx "rumdl@$version" check --config .markdownlint.yml . +git diff --check origin/main ``` -Report the checks you ran. State any checks that credentials, hardware, -permissions, or unavailable tools prevented. +Without `uv`, download that version from the [rumdl releases](https://github.com/rvben/rumdl/releases) and run `rumdl check --config .markdownlint.yml .`. + +Lint the whole tree, because renaming a service breaks a link in the root `README.md`. `git diff --check origin/main` also covers staged and committed changes, but not untracked files, so stage new files first. In a fork, compare with the `main` branch of this repository instead, for example `upstream/main`. + +When you have an auth key, start the copied stack and test its main function through the Tailnet. Run each health check command in its container with `docker exec`. + +Report the checks you ran. State any checks that credentials, hardware, permissions, or unavailable tools prevented. + +## Pull requests + +- Title the pull request `Service: what changes`, such as `Umami: new service`. For several services, list them or use `All services:`. For other areas, use `CI:`, `Docs:`, or `Template:`. +- Fill in the [pull request template](.github/PULL_REQUEST_TEMPLATE.md). Maintainers apply the labels. diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 889d1afd..9258cdbf 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -2,6 +2,10 @@ Thanks for helping improve these Tailscale sidecar examples. +The rules on this page apply to new stacks and to the parts of an existing +stack that you change. Existing stacks are brought in line over time, so you +do not have to fix the rest of a stack that you touch. + ## Add a service 1. Copy the service template from the repository root. Replace `my-service` with your service name: @@ -12,33 +16,64 @@ Thanks for helping improve these Tailscale sidecar examples. This command includes the hidden `.env` file. Use a lowercase directory name. -2. Update `.env` with safe example values. +2. Update `.env`. + + Set `SERVICE`, usually to the directory name, and set `IMAGE_URL`, + `SERVICEPORT`, and the application variables. Define every variable that + `compose.yaml` uses, unless it has a default such as `${VAR:-value}`. Put + comments on their own line: `VAR= # note` sets `VAR` to `# note`. + + `.env` is published with the repository. Never commit a working auth key, + password, token, or other credential. Leave each secret empty. When the stack + cannot start without it, require it in `compose.yaml`: + + ```yaml + - DB_PASSWORD=${DB_PASSWORD:?Set DB_PASSWORD in .env} + ``` - Set `SERVICE`, `IMAGE_URL`, `SERVICEPORT`, and the application variables. - Never commit a working auth key, password, token, or other credential. + Compose then stops with an error until the user sets a value. Above the + empty variable in `.env`, add a comment that starts with `# Required:` and + says what to set, such as + `# Required: signs the sessions. Generate it with: openssl rand -hex 32`. + Keep `TS_AUTHKEY` and optional secrets without `:?`. 3. Adapt `compose.yaml`. - Keep the Compose service keys `tailscale` and `application`. - - Name the containers `tailscale-${SERVICE}` and `app-${SERVICE}`. + - Name the containers `tailscale-${SERVICE}` and `app-${SERVICE}`, unless the + stack documents a deviation under "Deviations from the standard setup". - Keep `network_mode: service:tailscale` on the application. - Keep the application's health-based dependency on `tailscale`. - Add all required persistent volumes. - Add required devices and capabilities explicitly. + - When the stack needs another Compose service key or container name, more + than one application container, or none, list the difference and the + reason under "Deviations from the standard setup". Prefer + `app-${SERVICE}-` for extra containers, such as `app-${SERVICE}-db`. + - Pass `PUID` and `PGID` only when the image reads them, such as LinuxServer.io + images. Keep `TZ`, as the template does. + - Write the image tag explicitly and use the tag upstream recommends. Pin + databases, caches, and search engines to a major or major.minor tag. When + you pin an exact version, say why in a comment. + - Keep application data under `./${SERVICE}-data/`, so that `.gitignore` + covers it. A file that the repository must track inside such a folder + needs its own exception in `.gitignore`, as for Donetick and Dozzle. 4. Set the Serve proxy to the application's internal port. - The Serve JSON does not read `SERVICEPORT` from `.env`. Keep runtime variables - escaped, such as `$${TS_CERT_DOMAIN}`. + Hard-code the port in the Serve JSON. Do not use `${SERVICEPORT}`: it is + meant for the optional LAN `ports` block and can differ from the port that + Serve must proxy to. Keep runtime variables escaped, such as + `$${TS_CERT_DOMAIN}`. Keep the `ports` block commented for Tailnet-only access. Document any LAN - port you expose. Remove the Serve configuration when the service does not use - Tailscale Serve. + port you expose. When nothing in the stack uses Tailscale Serve, remove the + Serve configuration: the `TS_SERVE_CONFIG` line, the `configs` entry of the + `tailscale` service, and the top-level `configs` block. 5. Configure the application's health check. - Use the first option that the image supports. The template lists the same - options in the same order. + Use the first option that the image supports. 1. When the image defines its own `HEALTHCHECK`, omit the block and add a `# Healthcheck: defined by the image (...)` comment that names the @@ -54,42 +89,37 @@ Thanks for helping improve these Tailscale sidecar examples. is possible. Omit the block and add a `# Healthcheck: none possible ...` comment that names the reason. - Run the command inside the running container before you commit it. Avoid - `pgrep -f ${SERVICE}`, which breaks when `SERVICE` is renamed. - - A container that runs once and exits needs no check. Omit the block and - add a `# Healthcheck: none needed ...` comment that names the reason. - - For a database, connect over TCP, for example with - `pg_isready -h 127.0.0.1` or `mariadb-admin ping -h 127.0.0.1`. During the - first start, the image runs a temporary server that accepts only socket - connections, so a socket check reports ready too early. When another - service depends on the database, wait with `condition: service_healthy`. - A database image's own `HEALTHCHECK` still comes first. When that check - connects over the socket, say so in the comment. - - Keep the Tailscale health check. + Use the comment text from the template's `compose.yaml`, which covers the + same options, the database case, and the no-check cases. Run the command + inside the running container before you commit it. Keep the Tailscale health + check. 6. Complete the service README. Every service README uses the headings of the template, in the same order. - Do not rename them and do not add others, so that every service reads the - same way. Replace the placeholders in capitals. + Do not rename them, and add no headings other than the optional sections + below, so that every service reads the same way. Replace the placeholders in + capitals. - - **Introduction.** Say what the service does in one or two sentences and - link to the upstream project. Leave feature lists to upstream. + - **Introduction.** The paragraph under the title. Say what the service does + in one or two sentences and link to the upstream project. Leave feature + lists to upstream. - **At a glance.** Give the Tailnet address, the port that Tailscale Serve forwards to, the image, and the data paths on the host. Add a row for - each further port that users connect to, such as DNS or SMTP. + each further port that users connect to, such as DNS or SMTP. Write `None` + for the web interface when there is none, and add an `API` row when Serve + publishes an API. - **Before you start.** List only what the Quick Start does not cover: values that must change in `.env`, secrets to generate, folders to - create, and required host groups or devices. + create, and required host groups or devices. List every required secret + and say that Compose stops with an error until it is set. - **Deviations from the standard setup.** List every difference from [the standard setup](documentation/standard-setup.md) and give the reason. Examples are extra containers, published host ports, a changed or removed Serve configuration, DNS settings, and added capabilities. - **First run.** Describe what the user does after the first start, such - as creating the first account or finding a generated password. + as creating the first account or finding a generated password. When there + is nothing to do, write `Nothing to set up. Open the web interface.` - **Links.** Link to the upstream documentation and source code. When a section has nothing to report, keep the heading and the sentence @@ -101,7 +131,9 @@ Thanks for helping improve these Tailscale sidecar examples. - **Configuration.** Optional settings that users commonly change. - **Troubleshooting.** Known errors and their solutions. - - **Upgrading.** Steps for users of an older version of the stack. + - **Upgrading.** Required when a change makes existing users act, such as a + new required secret, a moved data folder, or a changed `SERVICE`. Say what + to do, as in [FossFLOW](services/fossflow/README.md#upgrading). State only what you confirmed in the Compose file, the upstream documentation, or a running stack. Put guidance that applies to more than @@ -109,7 +141,7 @@ Thanks for helping improve these Tailscale sidecar examples. 7. Add the service to the correct category in the root `README.md`. - Keep the entries in that category alphabetized. + Insert the row in alphabetical order of the bold name, ignoring case. ## Update a service @@ -119,20 +151,55 @@ Thanks for helping improve these Tailscale sidecar examples. - Use `${VARIABLE}` for Compose interpolation, not `$(VARIABLE)`. - Update the service README when ports, paths, setup, or behavior change. - Update the root service list when you add, remove, or rename a service. +- Changing `SERVICE` renames the Tailnet device and its address, the + containers, and the `./${SERVICE}-data` folder. Treat it like a volume change + and add an "Upgrading" section. +- Do not rename existing containers or service keys. Other containers and + `.env` values may refer to them. Preserve valid service-specific exceptions. ## Verify your change -Run Compose validation from each changed service directory: +Run Compose validation from each changed service directory. Stacks that +require your own secrets stop with "required variable ... is missing a value". +Give those variables a dummy value on the command line, as CI does, instead of +editing `.env`: ```sh -docker compose config --quiet +env $(grep -ohE '\$\{[A-Za-z0-9_]+:\?' compose.yaml .env | sed -E 's/^\$\{//; s/:\?$/=dummy/' | sort -u) docker compose config --quiet ``` -This command does not prove that the application works. +The command must exit with status 0 and add no "variable is not set" +warnings. It does not prove that the application works. + +The template leaves `SERVICE` and `IMAGE_URL` empty. To validate it, run +`SERVICE=dummy IMAGE_URL=dummy docker compose config --quiet` from +`templates/service-template/`. + +From the repository root, lint all Markdown with the rumdl version that +`.github/workflows/linting.yml` pins in its `version:` input: + +```sh +version=$(sed -nE 's/.*version: "([0-9.]+)".*/\1/p' .github/workflows/linting.yml) +uvx "rumdl@$version" check --config .markdownlint.yml . +``` + +Without `uv`, download that version from the +[rumdl releases](https://github.com/rvben/rumdl/releases) and run +`rumdl check --config .markdownlint.yml .`. + +After you stage new files, check whitespace with `git diff --check origin/main`. +In a fork, compare with the `main` branch of this repository instead, for +example `upstream/main`. + +When possible, start the stack from a copy outside the repository, such as +`cp -R services/my-service /tmp/my-service`, and put real values only in the +copy. A running stack writes `./config`, `./ts/state` (with the Tailscale +device key), and its data folders next to `compose.yaml`. Before you commit, +run `git status` and stage only the files you changed on purpose. -When possible, start the stack and confirm: +Then confirm: - Tailscale becomes healthy and joins the Tailnet. - The application starts and is reachable through the Tailnet. @@ -141,5 +208,8 @@ When possible, start the stack and confirm: ## Submit a pull request -Follow the pull request template. Report the checks you ran and any checks you -could not run. +Title the pull request `Service: what changes`, such as `Umami: new service`. +For several services, list them or use `All services:`. For other areas, use +`CI:`, `Docs:`, or `Template:`. Fill in the pull request template, and report +the checks you ran and any checks you could not run. Maintainers apply the +labels. diff --git a/templates/service-template/.env b/templates/service-template/.env index 2b2ad66b..d6a9c053 100644 --- a/templates/service-template/.env +++ b/templates/service-template/.env @@ -1,27 +1,27 @@ -#version=1.1 #URL=https://github.com/tailscale-dev/ScaleTail -#COMPOSE_PROJECT_NAME= # Optional: only use when running multiple deployments on the same infrastructure. +# To run a second copy, copy the folder under another name and change SERVICE. # Service Configuration # Service name (e.g., adguard). Used as hostname in Tailscale and for container naming (app-${SERVICE}). SERVICE= -# Docker image URL from container registry (e.g., adguard/adguard-home). +# Docker image URL from container registry (e.g., adguard/adguardhome:latest). IMAGE_URL= # Network Configuration # Port to expose to local network. Uncomment the "ports:" section in compose.yaml to enable. SERVICEPORT= -DNS_SERVER=9.9.9.9 # Preferred DNS server for Tailscale. Uncomment the "dns:" section in compose.yaml to enable. +# Preferred DNS server for Tailscale. Uncomment the "dns:" section in compose.yaml to enable. +DNS_SERVER=9.9.9.9 # Tailscale Configuration -# Auth key from https://tailscale.com/admin/authkeys. See: https://tailscale.com/kb/1085/auth-keys#generate-an-auth-key for instructions. +# Auth key from https://console.tailscale.com/admin/settings/keys. See: https://tailscale.com/kb/1085/auth-keys#generate-an-auth-key for instructions. TS_AUTHKEY= # Optional Service variables # PUID=1000 -# Time Zone setting for containers -TZ=Europe/Amsterdam # See: https://en.wikipedia.org/wiki/List_of_tz_database_time_zones +# Time Zone setting for containers. See: https://en.wikipedia.org/wiki/List_of_tz_database_time_zones +TZ=Europe/Amsterdam # Any Container environment variables are declared below. See https://docs.docker.com/compose/how-tos/environment-variables/ diff --git a/templates/service-template/compose.yaml b/templates/service-template/compose.yaml index 33734d16..dff85a0a 100644 --- a/templates/service-template/compose.yaml +++ b/templates/service-template/compose.yaml @@ -9,7 +9,7 @@ configs: services: # Make sure you have updated/checked the .env file with the correct variables. -# Every variable used in this file must be defined there. +# Every variable used in this file must be defined there, unless it has a default such as ${VAR:-value}. # Tailscale Sidecar Configuration tailscale: image: tailscale/tailscale:latest # Image to be used @@ -53,8 +53,9 @@ services: network_mode: service:tailscale # Sidecar configuration to route the service through Tailscale container_name: app-${SERVICE} # Name for local container management environment: # Variables are declared in .env file. - - PUID=1000 - - PGID=1000 + # Set PUID and PGID only for images that read them, such as LinuxServer.io images. Many images ignore them. + #- PUID=1000 + #- PGID=1000 - TZ=${TZ} #- EXAMPLE_VAR=${EXAMPLE_VAR} volumes: @@ -76,7 +77,10 @@ services: # "# Healthcheck: none possible. The image contains only , with no shell, HTTP client, or health command." # A container that runs once and exits needs no check. Remove this block and replace it with # "# Healthcheck: none needed. ." - # Avoid `pgrep -f ${SERVICE}`: it breaks when SERVICE is renamed, and many images do not include pgrep. + # Database: check over TCP (for example pg_isready -h 127.0.0.1), because the first-start server accepts only socket connections. + # A database image's own HEALTHCHECK still comes first - when it connects over the socket, say so in the comment. + # When another service depends on the database, let it wait with `condition: service_healthy`. + # Avoid `pgrep -f` with the service name: it breaks when SERVICE is renamed, and many images do not include pgrep. healthcheck: test: ["CMD", "wget", "--spider", "-q", "http://127.0.0.1:80/"] # Check if the service is responding interval: 1m # How often to perform the check