Sync Funnel documentation from develop

.github/workflows/funnel-docs-check.yml

@@ -0,0 +1,29 @@
+name: Check Funnel documentation sync
+
+on:
+  pull_request:
+  push:
+    branches:
+      - main
+  workflow_dispatch:
+
+permissions:
+  contents: read
+
+jobs:
+  check:
+    name: Test sync and build site
+    runs-on: ubuntu-latest
+    steps:
+      - name: Check out CALYPR docs
+        uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
+      - name: Set up Python
+        uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5.6.0
+        with:
+          python-version: '3.12'
+      - name: Test sync behavior
+        run: python -m unittest discover -s tests -v
+      - name: Install website dependencies
+        run: python -m pip install -r requirements.txt
+      - name: Build website
+        run: zensical build --clean

.github/workflows/sync-funnel-docs.yml

@@ -0,0 +1,77 @@
+name: Update Funnel documentation
+
+on:
+  schedule:
+    - cron: '17 6 * * *'
+  workflow_dispatch:
+
+permissions:
+  contents: write
+  pull-requests: write
+
+concurrency:
+  group: sync-funnel-docs
+  cancel-in-progress: false
+
+jobs:
+  sync:
+    name: Prepare Funnel docs update
+    runs-on: ubuntu-latest
+    steps:
+      - name: Check out CALYPR docs
+        uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
+        with:
+          persist-credentials: false
+
+      - name: Check out Funnel develop
+        uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
+        with:
+          repository: calypr/funnel
+          ref: develop
+          path: .funnel-upstream
+          persist-credentials: false
+
+      - name: Check Funnel revision
+        id: revision
+        run: |
+          revision=$(git -C .funnel-upstream rev-parse HEAD)
+          previous=$(python3 -c 'import json; print(json.load(open("docs/tools/funnel/.funnel-docs.json"))["revision"])')
+          echo "revision=$revision" >> "$GITHUB_OUTPUT"
+          if [ "$revision" != "$previous" ]; then
+            echo "changed=true" >> "$GITHUB_OUTPUT"
+          fi
+
+      - name: Set up Python
+        if: steps.revision.outputs.changed == 'true'
+        uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5.6.0
+        with:
+          python-version: '3.12'
+
+      - name: Test sync behavior
+        if: steps.revision.outputs.changed == 'true'
+        run: python -m unittest discover -s tests -v
+
+      - name: Sync Funnel documentation
+        if: steps.revision.outputs.changed == 'true'
+        run: python scripts/sync_funnel_docs.py .funnel-upstream --revision "${{ steps.revision.outputs.revision }}"
+
+      - name: Install website dependencies
+        if: steps.revision.outputs.changed == 'true'
+        run: python -m pip install -r requirements.txt
+
+      - name: Build the website
+        if: steps.revision.outputs.changed == 'true'
+        run: zensical build --clean
+
+      - name: Create or update pull request
+        if: steps.revision.outputs.changed == 'true'
+        uses: peter-evans/create-pull-request@22a9089034f40e5a961c8808d113e2c98fb63676 # v7
+        with:
+          branch: automation/sync-funnel-docs
+          commit-message: 'docs: sync Funnel develop documentation'
+          title: 'docs: sync Funnel develop documentation'
+          body: |
+            Update the CALYPR site from [Funnel develop](https://github.com/calypr/funnel/tree/${{ steps.revision.outputs.revision }}) at `${{ steps.revision.outputs.revision }}`.
+
+            Review the generated content and navigation before merging. This PR was prepared and built by the scheduled documentation sync.
+          delete-branch: true

.gitignore

@@ -4,3 +4,5 @@ venv
 site/
 .idea/
 attic/
+
+/.funnel-upstream/

DEVELOPING.md

@@ -1,176 +1,57 @@
 # Developing CALYPR Docs
 
-This repo now uses a Zensical-only build pipeline. MkDocs, `.nav.yml`, and the old multirepo plugin are gone.
+The site is built with Zensical from the content in `docs/`. Navigation is
+configured in `zensical.toml`.
 
-The main thing to understand is that the published site is built from a generated docs tree, not directly from `docs/`.
+## Build locally
 
-## Mental Model
-
-- `docs/` contains repo-authored source content.
-- `.generated/docs/` is the staged build input that Zensical actually reads.
-- `site/` is the rendered output from `zensical build`.
-- `zensical.toml` is the only navigation source of truth.
-
-The build pipeline is:
-
-1. Copy repo-authored docs from `docs/` into `.generated/docs/`
-2. Import selected upstream docs from `git-drs` and `syfon`
-3. Apply a few link rewrites needed for imported pages
-4. Add generated assets like `termynal.css` and `termynal.js`
-5. Run Zensical against `.generated/docs`
-
-## Core Commands
-
-From the repo root:
-
-```bash
+```sh
 python3 -m venv venv
 source venv/bin/activate
 pip install -r requirements.txt
-make serve
+make build
 ```
 
-Useful commands:
-
-- `make prepare`: rebuild `.generated/docs`
-- `make build`: run a clean Zensical build
-- `make serve`: prepare docs and start the local server
-- `make update-branches`: update GitHub source links in authored docs to the preferred upstream branches
-
-If you want the raw commands:
-
-```bash
-python scripts/prepare_docs.py
-zensical build --clean
-zensical serve
+Use `make serve` for local development. The rendered site is written to `site/`.
+The [GitHub Pages workflow](.github/workflows/publish.yml) builds and publishes
+when changes reach `main`.
+
+## Funnel documentation
+
+The canonical Funnel reference pages live in the `website/content/docs/`
+directory on [Funnel's `develop` branch](https://github.com/calypr/funnel/tree/develop/website/content/docs).
+A scheduled workflow checks that branch daily and opens or updates a pull request
+with changed pages and generated navigation. Review and merge the pull request
+to publish the update. The workflow also runs the sync tests and builds the site.
+
+The Funnel landing page, download page, and storage index remain CALYPR site
+content. The installation script is copied from Funnel's `install.sh`. The sync
+converts Funnel's Hugo shortcodes to Markdown compatible with this site and
+rejects unknown shortcodes rather than publishing them verbatim.
+
+Three pages are temporarily held at their existing CALYPR versions while newer
+site guidance is reconciled with Funnel: the GCP Batch, Kubernetes, and S3
+guides. Their expected upstream versions are recorded in
+`docs/tools/funnel/.funnel-docs.json`. Once those exact versions reach `develop`,
+the sync imports them and removes their exceptions automatically. If the
+upstream content changes during review, update the expected hashes after
+reconciling the differences.
+
+To run the sync locally against a Funnel checkout:
+
+```sh
+python scripts/sync_funnel_docs.py /path/to/funnel --revision "$(git -C /path/to/funnel rev-parse HEAD)"
+python -m unittest discover -s tests -v
+make build
 ```
 
-## Where Navigation Lives
-
-Navigation is fully defined in [zensical.toml](./zensical.toml).
-
-If you want to:
-
-- rename a sidebar label
-- move a section
-- add a new page to the nav
-- change the Tools grouping
-
-edit the `nav = [...]` structure in `zensical.toml`.
-
-There are no `.nav.yml` files anymore. If you add a page under `docs/` and do not add it to `zensical.toml`, it will not appear in the intended nav structure.
-
-## Source of Truth Rules
-
-### Repo-authored pages
-
-These live in `docs/` and should be edited directly there.
-
-Examples:
-
-- `docs/index.md`
-- `docs/calypr/...`
-- `docs/tools/index.md`
-- `docs/tools/funnel/...`
-- `docs/tools/grip/...`
-
-### Imported tool pages
-
-These are not authored in this repo anymore.
-
-Current imported repos:
-
-- `git-drs`
-- `syfon`
-
-Their imported file mappings live in [scripts/prepare_docs.py](./scripts/prepare_docs.py) under `REPO_IMPORTS`.
-
-If you need to:
-
-- add a new imported page
-- stop importing a page
-- change where an imported page lands under `tools/...`
-
-update `REPO_IMPORTS` and usually also update `zensical.toml`.
-
-Do not hand-edit imported pages under `.generated/docs`. They are build artifacts and will be replaced on the next `make prepare`.
-
-## How Upstream Imports Work
-
-The importer prefers local sibling repos by default:
-
-- `../git-drs`
-- `../syfon`
-
-So on a typical local machine, the imported content comes from whatever is currently checked out in those repos.
-
-If local sibling repos are unavailable, the importer falls back to GitHub and picks the first available branch from [scripts/branch_config.json](./scripts/branch_config.json).
-
-Current branch preference order:
-
-- `development`
-- `develop`
-- `main`
-- `master`
-
-The last import decision is written to `.generated/imports.json`.
-
-Environment overrides:
-
-- `CALYPR_IMPORT_PREFER_LOCAL=0`: force remote import mode
-- `CALYPR_IMPORT_GIT_DRS_DIR=/path/to/git-drs`: override local `git-drs` source
-- `CALYPR_IMPORT_SYFON_DIR=/path/to/syfon`: override local `syfon` source
-
-## When You Add or Change Pages
-
-### Adding a repo-authored page
-
-1. Create the file under `docs/`
-2. Add it to `zensical.toml`
-3. Run `make serve` or `make build`
-
-### Adding an imported tool page
-
-1. Add the file mapping in `scripts/prepare_docs.py`
-2. Add the page to `zensical.toml`
-3. If the imported markdown links break after relocation, add a targeted rewrite in `MARKDOWN_REWRITES`
-4. Run `make prepare`
-5. Run `make build`
-
-## Styling and Overrides
-
-Site-level assets still come from the repo:
-
-- custom CSS: `docs/stylesheets/extra.css`
-- custom JS: `docs/javascripts/nav-open-level1.js`
-- theme overrides: `overrides/`
-
-Those are copied into `.generated/docs` during preparation and then used by Zensical.
-
-## Deployment Behavior
-
-Deployment uses the same pipeline as local builds:
-
-- GitHub Pages runs `python scripts/prepare_docs.py` and then `zensical build --clean`
-- Netlify does the same
-
-That means deployment now depends on the import step succeeding. In CI, the importer will usually use remote GitHub repos rather than local sibling repos.
-
-## Files You Should Usually Ignore
-
-- `.generated/`
-- `site/`
-- `.cache/`
-
-These are build/runtime artifacts.
-
-## Known Caveat
-
-Zensical currently reports some warnings from imported and legacy docs content, especially around bracketed literals like:
-
-- `[SIGNED]`
-- `[0]`
-- `[]string`
-- `map[string]string`
+Generated Funnel pages and the marked Funnel navigation block should be changed
+in the Funnel repository, not edited manually here. The sync branch is reused
+while an update PR is open. If the Funnel revision is unchanged, no PR is created.
 
-Those are content/rendering warnings, not a sign that the Zensical migration is broken.
+The workflow needs both the organization policy and the repository setting that
+allow GitHub Actions to create pull requests, plus `contents: write` and
+`pull-requests: write` job permissions.
+GitHub places PR workflows created with `GITHUB_TOKEN` in an approval-required
+state. The sync workflow runs its own tests and site build before creating the
+PR; a repository writer can approve the PR workflows separately.

README.md

@@ -15,16 +15,14 @@ make serve
 
 The local build contract is:
 
-- `make prepare` stages repo-authored docs plus imported upstream tool docs into `.generated/docs`
 - `make build` runs a clean Zensical build
-- `make serve` stages the docs and starts the local Zensical server
+- `make serve` starts the local Zensical server
 
 For repo maintenance details, navigation ownership, and import behavior, see [DEVELOPING.md](./DEVELOPING.md).
 
 If you prefer the raw commands:
 
 ```sh
-python scripts/prepare_docs.py
 zensical serve
 ```
 

docs/tools/funnel/.funnel-docs.json

@@ -0,0 +1,11 @@
+{
+  "source": "https://github.com/calypr/funnel",
+  "branch": "develop",
+  "revision": "a3be3b208e0f6202bc2a33d1946fb56a8e850431",
+  "files": [],
+  "pending_reconciliation": {
+    "compute/gcp-batch.md": "a1ad1e72c4b7b8befd4ec98814855aa43fb48f8b76b6b12f9147ad64802f4d79",
+    "compute/kubernetes.md": "456c6cbed9d9d1687b717dde7e647daea896fd96b03662ec4022d862930ca400",
+    "storage/s3.md": "7624b293ea197bc68dfb01d06227b2c2da3b1ccd16762d2c00b3abfb01b4cf5b"
+  }
+}