From b0702ef08b726b2b6638b4ea93e0d2467de935b1 Mon Sep 17 00:00:00 2001 From: Audric Ackermann Date: Thu, 24 Sep 2026 10:08:12 +1000 Subject: [PATCH 001/101] feat: daily digest of contributor pull requests MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit One message each weekday morning listing the open PRs across session-foundation's own repositories whose author is not a maintainer and which have moved in the last three days. A single search fetches every open PR in the org and the window is applied to the result, so the header can carry the total contributor backlog for the cost of one query. Forks and archived repos are excluded by checking results against the org's repository list rather than by name, and bot accounts on GitHub's own account type. Weekdays means Monday has to cover the weekend, so the window is 72h and overlaps itself by two days on every run. A state file absorbs the overlap: 🟒 is a PR never reported, ✏️ one that has moved since it was, and a PR that has not moved is left out. Only what Discord accepted is recorded, so a run that fails partway re-reports that message's PRs rather than losing them, and every way of failing to read the file treats the window as new β€” noisy once, never wrong. Dedup is keyed on updated_at, which moves on any change at all, so an edit touching several PRs at once resurfaces all of them. The accurate alternative needs a request per PR that moved; this is the cheaper half of that trade. Maintainers are a hand-written list. Org membership covers six accounts, two of which are not in the review loop, and push access is held by a dozen more as outside collaborators β€” several of them contractors whose PRs are the point of the digest. Both signals would get it wrong in both directions. Runs on the box that already hosts the Zendesk digest, under its own user, env file and venv β€” the venv because the two jobs pin requests differently. --- README.md | 88 +++++ deploy/README.md | 102 +++++- deploy/alert.py | 6 +- deploy/github-prs-alert@.service | 22 ++ deploy/github-prs-digest.service | 36 ++ deploy/github-prs-digest.timer | 20 ++ github_prs/digest.py | 592 +++++++++++++++++++++++++++++++ github_prs/maintainers.txt | 20 ++ github_prs/requirements.txt | 2 + github_prs/test_digest.py | 412 +++++++++++++++++++++ 10 files changed, 1290 insertions(+), 10 deletions(-) create mode 100644 deploy/github-prs-alert@.service create mode 100644 deploy/github-prs-digest.service create mode 100644 deploy/github-prs-digest.timer create mode 100755 github_prs/digest.py create mode 100644 github_prs/maintainers.txt create mode 100644 github_prs/requirements.txt create mode 100644 github_prs/test_digest.py diff --git a/README.md b/README.md index 80e770d..7deedf0 100644 --- a/README.md +++ b/README.md @@ -653,6 +653,94 @@ tried. `session_util` grew a `blind15_id` covering this in [`7e8d126`](https://github.com/session-foundation/libsession-python/commit/7e8d126), which no packaged release carries yet. +## Contributor Pull Requests + +[`github_prs/digest.py`](github_prs/digest.py) posts one message each weekday morning +listing the open pull requests in session-foundation's repositories whose author is not +a maintainer and which have moved in the last three days β€” the ones nobody on the team +has a reason to already know about: + +``` +**Contributor pull requests** Β· last 3 days +🟒 **2** new Β· ✏️ **1** updated +**36** open from contributors across the org. + +**session-desktop** +🟒 [#1958](…) @KyWB Β· 12h Β· πŸ’¬1 Β· Fix issue #563 +✏️ [#1904](…) @scrense-hash Β· 3h Β· πŸ’¬2 Β· feat: add SOCKS5 proxy support +``` + +🟒 is a PR the digest has never reported; ✏️ is one it has, which has moved since. A PR +that has not moved is left out entirely, however wide the window. The backlog line +counts every open contributor PR regardless, so a quiet day still says how much is +waiting. + +### Weekdays, and the state file + +The timer runs `Mon..Fri`, so Monday's run has to cover the weekend β€” hence a 72-hour +window rather than a daily one. That window overlaps itself by two days on every run, +and [`--state`](github_prs/digest.py) is what stops the overlap being noise: it records +which PRs reached Discord and what each one's `updated_at` was at the time. + +`updated_at` moves on *any* change, including one that touches several PRs at once β€” a +label sweep, a base branch renamed β€” so those resurface once even though nobody worked +on them. The accurate alternative is the head SHA and the comment counts, which are not +in the search result and cost a request per PR that moved; this is the cheaper half of +that trade, taken deliberately. + +Only what Discord accepted is recorded, so a run that fails on its second message +re-reports that message's PRs tomorrow rather than losing them. Every way of failing to +read the state file β€” missing, unreadable, written by another version β€” treats every PR +in the window as new: noisy once, never wrong, which is what makes it a cache rather +than something to back up. + +One search fetches every open PR in the org and the window is applied to the result +here rather than in the query β€” that is what buys the backlog count for the cost of a +single query. Past GitHub's 1000-result search ceiling the digest says the counts are a +floor instead of failing. + +### Who is a maintainer + +[`github_prs/maintainers.txt`](github_prs/maintainers.txt), one login per line, matched +case-insensitively. Bot accounts need no entry β€” every account GitHub types as a `Bot` +is dropped, so a renamed Dependabot stays out on its own. + +Neither of the two things GitHub could answer this with is a substitute. Org membership +covers six accounts, two of which are not in the review loop; push access is held by a +dozen more as outside collaborators, several of them contractors whose PRs are exactly +what the digest is for. Both would get it wrong in both directions, so the list is +written by hand β€” and goes stale silently, since a new maintainer's PRs are reported as +a stranger's until someone adds them. + +### What is left out + +Forks and archived repositories, by checking the search results against the org's +repository list rather than by name β€” so a repository created today is covered today, +and `session-pysogs` or `session-android-private` never are. `--include-forks`, +`--include-archived` and `--exclude-repo` override that per run. + +| flag | | +| --- | --- | +| `--window-hours N` | how far back a PR must have moved to be considered (default 72) | +| `--state PATH` | dedup state; without it every PR in the window is new | +| `--state-retention-days N` | drop state entries older than this (default 30) | +| `--dry-run` | print the Discord payload, post nothing | +| `--org`, `--token`, `--webhook` | override the environment | +| `--maintainers PATH` | a different list | + +| env var | | +| --- | --- | +| `GITHUB_PRS_TOKEN` | read-only token; no scope at all is needed for the public repos, `repo` to also see the org's private ones | +| `GITHUB_PRS_DISCORD_WEBHOOK_URL` | the channel it posts to (not needed with `--dry-run`) | +| `GITHUB_PRS_ORG` | optional; defaults to `session-foundation` | + +It runs on the same box as the Zendesk digest, under its own user and its own +environment file β€” see [deploy/README.md](deploy/README.md). + +```sh +cd github_prs && python -m unittest discover +``` + ## Workflow Failure Notificaiton If a workflow fails and is in the list of workflows monitored by the failure notificaiton workflow, the failure notificaiton workflow will send a message to a discord webhook. diff --git a/deploy/README.md b/deploy/README.md index 323af01..3174827 100644 --- a/deploy/README.md +++ b/deploy/README.md @@ -1,17 +1,20 @@ # Self-hosted deployment -The Zendesk triage digest and the `claude:` note webhook, both running on one -machine. - -Two things run here: +The Zendesk triage digest, the `claude:` note webhook and the contributor pull request +digest, all on one machine. | Unit | What it is | | --- | --- | | `zendesk-relay.service` | Always on. The HTTPS endpoint Zendesk posts note webhooks to. | | `zendesk-digest.timer` β†’ `.service` | Weekday mornings. Resolves positive reviews, then posts the digest. | +| `github-prs-digest.timer` β†’ `.service` | Weekday mornings. Posts the contributor pull request digest. | + +`zendesk-alert@.service` and `github-prs-alert@.service` are pulled in by `OnFailure=` +and report the failed unit to the channel that job posts to. -`zendesk-alert@.service` is pulled in by `OnFailure=` on both, and reports the failed -unit to the triage channel. +One clone at `/opt/zendesk` holds all of it β€” the directory is named after its first +tenant, not its contents. The venvs are separate, because the two jobs pin `requests` +differently and a shared one would silently be whichever was installed last. ## Host requirements @@ -30,7 +33,8 @@ unit to the triage channel. - **nginx already installed**, with certbot managing its certificates. This adds one server block to it rather than a second web server; two would fight over :443 and take the host's other sites down with them. -- Persistent `/var/lib/zendesk` β€” it holds the dedup state, the only thing on disk. +- Persistent `/var/lib/zendesk` and `/var/lib/github-prs` β€” they hold the two digests' + dedup state, the only thing on disk. `StateDirectory=` creates the second one. Note who else holds root. This box becomes custodian of a Zendesk API token that can write a public comment to any ticket, and of a logged-in Claude Code session. @@ -107,6 +111,39 @@ nginx -t && systemctl reload nginx # validate what certbot wrote > nginx -t && systemctl reload nginx > ``` +### Adding the pull request digest + +Its own user, its own environment file and its own venv, out of the same clone. A +token that can read the org's repositories has no business in the environment of the +relay, which is the one process here reachable from the internet. + +The account needs no home of its own: nothing in this job shells out to the Claude +CLI, so the `$HOME` that the Zendesk units bend over backwards to preserve is not +wanted here at all. + +```bash +useradd --system --no-create-home --home /nonexistent --shell /usr/sbin/nologin ghdigest +python3 -m venv /opt/github-prs/venv +/opt/github-prs/venv/bin/pip install -r /opt/zendesk/github_prs/requirements.txt +chown -R ghdigest:ghdigest /opt/github-prs + +install -d -m 750 -o root -g ghdigest /etc/github-prs +[ -e /etc/github-prs/env ] || install -m 640 -o root -g ghdigest /dev/null /etc/github-prs/env +"${EDITOR:-nano}" /etc/github-prs/env # contents under Secrets, below + +cp /opt/zendesk/deploy/github-prs-*.service /opt/zendesk/deploy/github-prs-*.timer \ + /etc/systemd/system/ +systemctl daemon-reload +systemctl enable --now github-prs-digest.timer +``` + +The clone stays owned by `zendesk` and world-readable, which is what lets `ghdigest` +run out of it. Nothing secret lives there β€” every secret is under `/etc`. + +`/var/lib/github-prs` needs no `install` step: `StateDirectory=github-prs` on the unit +creates it with the right owner on first start. It holds the dedup state that keeps the +72-hour window from re-reporting the same PR every weekday morning. + ## Secrets `/etc/zendesk/env`, mode `640`, `root:zendesk` β€” readable by the service, not by @@ -173,6 +210,26 @@ Set `RELAY_DRY_RUN=1` for the first deployment. The whole `claude:` note path ru webhook, command parsing, composing, translation β€” and the Zendesk writes are skipped. +### `/etc/github-prs/env` + +Mode `640`, `root:ghdigest`, and separate from the Zendesk file rather than merged +into it β€” see above. + +```sh +# Read-only. No scope at all is needed for the org's public repositories; add `repo` +# to have the digest also see the private ones. Nothing here ever writes to GitHub. +GITHUB_PRS_TOKEN= + +# The channel the digest posts to. A webhook is bound to the channel it was created +# in, so this one value decides where the digest goes. +GITHUB_PRS_DISCORD_WEBHOOK_URL= + +# Required, and normally the same webhook: without it alert.py falls back to +# ZENDESK_DISCORD_WEBHOOK_URL, which is not in this file, and the failure notifier +# fails instead of reporting. +ALERT_DISCORD_WEBHOOK_URL= +``` + ## Verifying, in order **1. Locally, before Zendesk knows the address.** An unsigned request must be refused: @@ -255,12 +312,39 @@ else: the alert still sends, with the message it always sent. ```sh cd deploy && python -m unittest discover # the alert's own tests ``` +**6. The pull request digest.** A dry run under the unit's own confinement renders the +digest and posts nothing. `systemd-run` rather than `runuser` because the token then +comes from the environment file rather than an argument every process on the box can +read out of `ps`: + +```bash +systemd-run --pty --uid=ghdigest -p EnvironmentFile=/etc/github-prs/env \ + -p WorkingDirectory=/opt/zendesk/github_prs \ + /opt/github-prs/venv/bin/python digest.py --dry-run +``` + +Then for real: `systemctl start github-prs-digest.service`, +`systemctl list-timers github-prs-digest` (expect the next weekday, not tomorrow), and +`systemctl start github-prs-alert@test.service` for its failure path. + +Start it twice. The second run is the one that proves the dedup state: it should report +everything, then nothing, and say so. + +```bash +journalctl -u github-prs-digest -n 5 --no-pager # "N new, 0 changed, N unchanged" +ls -l /var/lib/github-prs/seen.json +``` + +A second run that reports everything again means the state was not written β€” check that +`StateDirectory=` reached systemd with +`systemctl show github-prs-digest -p StateDirectory`. ## Updating ```bash runuser -u zendesk -- git -C /opt/zendesk pull /opt/zendesk/venv/bin/pip install -r /opt/zendesk/zendesk_triage/requirements.txt +/opt/github-prs/venv/bin/pip install -r /opt/zendesk/github_prs/requirements.txt systemctl restart zendesk-relay ``` @@ -321,8 +405,10 @@ What that costs, in order of how much it matters: - **The digest is late, not lost.** `Persistent=yes` on the timer means a host that was down at 10:00 runs the digest once when it comes back, and the 72-hour window covers the gap. -- **The dedup state may be stale.** Losing `/var/lib/zendesk/seen.json` re-reports the +- **The dedup state may be stale.** Losing either `seen.json` re-reports that digest's window once: noisy, never wrong. +- **The pull request digest is late, not lost**, on the same `Persistent=yes` as the + Zendesk one, and its 72-hour window already covers a weekend's gap. Failures that are not a whole-host outage report themselves β€” `OnFailure=` on both units posts the failed unit and a `journalctl` line to the triage channel. \ No newline at end of file diff --git a/deploy/alert.py b/deploy/alert.py index d01db3b..545ab0b 100644 --- a/deploy/alert.py +++ b/deploy/alert.py @@ -9,7 +9,8 @@ Posts over ZENDESK_DISCORD_WEBHOOK_URL rather than the bot token, deliberately. This is one line of text needing no components, and a failure notifier should depend on as -little as possible of whatever just broke. +little as possible of whatever just broke. ALERT_DISCORD_WEBHOOK_URL overrides it, so +a job that posts to a channel of its own reports its failures there too. The failed unit's last journal line comes with it, so the channel says what broke rather than only that something did. Reading the journal needs the unit to carry @@ -102,7 +103,8 @@ def main(): args = [arg.strip() for arg in sys.argv[1:]] if not args or len(args) > 2 or not args[0]: sys.exit("usage: alert.py [journal-unit]") - webhook = triage.get_env("ZENDESK_DISCORD_WEBHOOK_URL") + webhook = (os.environ.get("ALERT_DISCORD_WEBHOOK_URL") + or triage.get_env("ZENDESK_DISCORD_WEBHOOK_URL")) detail = last_job_line(journal_tail(args[-1])) message = build_message(args[0], socket.gethostname(), *args[1:], detail=detail) # A fresh session, never a Zendesk one β€” that carries the API-token auth header, diff --git a/deploy/github-prs-alert@.service b/deploy/github-prs-alert@.service new file mode 100644 index 0000000..8103550 --- /dev/null +++ b/deploy/github-prs-alert@.service @@ -0,0 +1,22 @@ +[Unit] +Description=Report a failed pull request digest unit to Discord +Documentation=https://github.com/session-foundation/session-shared-scripts + +[Service] +Type=oneshot +User=ghdigest +Group=ghdigest +EnvironmentFile=/etc/github-prs/env + +# Same script as zendesk-alert@, run under this job's environment so the failure +# lands in the channel the job posts to rather than the triage one. +ExecStart=/opt/github-prs/venv/bin/python /opt/zendesk/deploy/alert.py %i + +# See zendesk-alert@.service: read-only access to the failed unit's journal, which is +# the line the alert quotes. +SupplementaryGroups=systemd-journal + +NoNewPrivileges=yes +PrivateTmp=yes +ProtectSystem=strict +ProtectHome=yes diff --git a/deploy/github-prs-digest.service b/deploy/github-prs-digest.service new file mode 100644 index 0000000..0de28b9 --- /dev/null +++ b/deploy/github-prs-digest.service @@ -0,0 +1,36 @@ +[Unit] +Description=Daily digest of contributor pull requests +Documentation=https://github.com/session-foundation/session-shared-scripts +After=network-online.target +Wants=network-online.target +OnFailure=github-prs-alert@%n.service + +[Service] +Type=oneshot +User=ghdigest +Group=ghdigest +WorkingDirectory=/opt/zendesk/github_prs +EnvironmentFile=/etc/github-prs/env + +ExecStart=/opt/github-prs/venv/bin/python digest.py --window-hours 72 \ + --state /var/lib/github-prs/seen.json + +# A dozen API calls. Anything near this is GitHub rate-limiting the run rather than +# a large result set. +TimeoutStartSec=15min + +NoNewPrivileges=yes +PrivateTmp=yes +PrivateDevices=yes +ProtectSystem=strict +ProtectHome=yes +ProtectKernelTunables=yes +ProtectKernelModules=yes +ProtectControlGroups=yes +RestrictAddressFamilies=AF_INET AF_INET6 +RestrictNamespaces=yes +LockPersonality=yes +# The dedup state, and the only thing this job writes anywhere. Losing it re-reports +# the window once: noisy, never wrong, so it needs persisting rather than backing up. +ReadWritePaths=/var/lib/github-prs +StateDirectory=github-prs diff --git a/deploy/github-prs-digest.timer b/deploy/github-prs-digest.timer new file mode 100644 index 0000000..ca94aa7 --- /dev/null +++ b/deploy/github-prs-digest.timer @@ -0,0 +1,20 @@ +[Unit] +Description=Post the contributor pull request digest +Documentation=https://github.com/session-foundation/session-shared-scripts + +[Timer] +# Half an hour before the Zendesk digest, where the people reading it are, tracking +# daylight saving. Weekdays only, which is what makes the 72h window load-bearing: +# Monday's run has to reach back over the weekend. +# +# The timezone belongs in the expression: there is no Timezone= key in [Timer], and +# systemd *ignores* one silently. Check any change with: +# systemd-analyze calendar "Mon..Fri 09:30 Australia/Melbourne" +OnCalendar=Mon..Fri 09:30 Australia/Melbourne +Persistent=yes +# The window overlaps by two days and the state file absorbs the overlap, so drift +# here costs nothing at all. +RandomizedDelaySec=2min + +[Install] +WantedBy=timers.target diff --git a/github_prs/digest.py b/github_prs/digest.py new file mode 100755 index 0000000..a03b439 --- /dev/null +++ b/github_prs/digest.py @@ -0,0 +1,592 @@ +#!/usr/bin/env python3 +""" +Daily digest of contributor pull requests across the session-foundation org. + +Lists every open PR in the org's own repositories whose author is not a maintainer, +grouped by repository, marking the ones never reported before and the ones that have +moved since they were. Posted to a Discord webhook of its own. + +--state is what makes the second of those answerable: it records what reached Discord +and what each PR's updated_at was at the time, so a PR the digest already showed stays +out until something happens to it. Without it every PR in the window reads as new. + +Who is a maintainer comes from maintainers.txt, one login per line; bot accounts are +dropped on GitHub's own account type rather than by name. Forks and archived repos +are excluded by checking the search results against the org's repository list, so a +repo created today is covered today. + +One search fetches every open PR in the org, and the window is applied to the result +here rather than in the query. That is what lets the header carry the total open +contributor backlog alongside the day's changes for the cost of a single query. + +The helpers here deliberately duplicate their zendesk_triage counterparts rather +than importing them: the two jobs run under different users from different env +files, and a shared module would make either one's dependencies the other's. + +Config (env vars, or flags for local runs): + GITHUB_PRS_TOKEN GitHub token, read-only. Needs no scope at all for the + public repos; add `repo` if the digest should also see + the org's private ones. + GITHUB_PRS_DISCORD_WEBHOOK_URL + Discord incoming webhook for the channel this posts to + (not needed with --dry-run) + GITHUB_PRS_ORG (optional) org to scan; defaults to session-foundation + +Usage: + # real run (what the timer does) + python digest.py + + # fetch and render, print the payload, post nothing + python digest.py --dry-run + + # what the weekday timer does: a window covering the weekend, deduped + python digest.py --window-hours 72 --state /var/lib/github-prs/seen.json +""" +import argparse +import json +import math +import os +import sys +import time +from datetime import datetime, timedelta, timezone +from operator import itemgetter +from urllib.parse import parse_qsl, urlencode, urlsplit, urlunsplit + +import requests + +API = "https://api.github.com" +DEFAULT_ORG = "session-foundation" +# Three days, because the timer runs on weekdays: Monday's window has to reach back +# over the weekend. Overlap between consecutive runs is what --state absorbs. +DEFAULT_WINDOW_HOURS = 72 +DEFAULT_RETENTION_DAYS = 30 +STATE_VERSION = 1 +# The Search API caps a query at 1000 results and returns 422 for any page past it +# (at per_page=100 that is page 11). Past the cap the digest reports truncation +# rather than failing the run. +# https://docs.github.com/en/rest/search#about-search +SEARCH_RESULT_LIMIT = 1000 +PER_PAGE = 100 +MAINTAINERS_FILE = os.path.join(os.path.dirname(os.path.abspath(__file__)), + "maintainers.txt") + + +def get_env(name, cli_value=None, required=True): + if cli_value: + return cli_value + value = os.environ.get(name) + if value: + return value + if required: + sys.exit(f"Missing required config: set the {name} environment variable (or pass the matching flag).") + return None + + +def load_maintainers(path): + """Logins from maintainers.txt, lowercased. `#` comments and blanks ignored.""" + logins = set() + with open(path, encoding="utf-8") as handle: + for line in handle: + login = line.split("#", 1)[0].strip() + if login: + logins.add(login.lower()) + return frozenset(logins) + + +def github_session(token): + session = requests.Session() + session.headers.update({ + "Authorization": f"Bearer {token}", + "Accept": "application/vnd.github+json", + "X-GitHub-Api-Version": "2022-11-28", + }) + return session + + +def retry_after_seconds(resp, default): + """Seconds to wait per the response, falling back to `default`. + + GitHub answers a secondary rate limit with Retry-After, and a primary one with + x-ratelimit-reset as an epoch second and no Retry-After at all, so both are + read. Anything unparseable, negative or infinite falls back rather than taking + the run down inside time.sleep(). + """ + raw = resp.headers.get("retry-after") + if raw is None: + reset = resp.headers.get("x-ratelimit-reset") + if reset is None: + return default + try: + return max(0.0, float(reset) - time.time()) + except (TypeError, ValueError): + return default + try: + seconds = float(raw) + except (TypeError, ValueError): + return default + if not math.isfinite(seconds) or seconds < 0: + return default + return seconds + + +def request_with_retry(session, method, url, attempts=5, **kwargs): + """GET/POST with backoff on 429 and 5xx.""" + if attempts <= 0: + raise ValueError("attempts must be at least 1") + + delay = 1.0 + last_exc = None + resp = None + for attempt in range(attempts): + final = attempt == attempts - 1 + try: + resp = session.request(method, url, timeout=30, **kwargs) + except requests.RequestException as exc: + last_exc = exc + if final: + break + time.sleep(delay) + delay = min(delay * 2, 30) + continue + if resp.status_code == 429 or resp.status_code >= 500: + if final: + break + time.sleep(min(retry_after_seconds(resp, delay), 60)) + delay = min(delay * 2, 30) + continue + return resp + if last_exc: + raise last_exc + return resp + + +def fetch_json(session, url, **kwargs): + resp = request_with_retry(session, "GET", url, **kwargs) + if resp.status_code >= 400: + sys.exit(f"GitHub {resp.status_code} on {url}: {resp.text[:300]}") + return resp.json() + + +def fetch_repos(session, org, include_forks=False, include_archived=False): + """Names of the org's repositories the digest is willing to report on.""" + names, page = set(), 1 + while True: + batch = fetch_json(session, f"{API}/orgs/{org}/repos", + params={"per_page": PER_PAGE, "page": page, "type": "all"}) + for repo in batch: + if repo.get("fork") and not include_forks: + continue + if repo.get("archived") and not include_archived: + continue + names.add(repo["name"]) + if len(batch) < PER_PAGE: + return names + page += 1 + + +def search_open_prs(session, org, max_results=SEARCH_RESULT_LIMIT): + """Every open PR in the org, newest activity first. + + Returns (items, truncated). `truncated` is the caller's cue that the counts are + a floor rather than a total. + """ + items, page = [], 1 + limit = min(max_results, SEARCH_RESULT_LIMIT) + while len(items) < limit: + payload = fetch_json(session, f"{API}/search/issues", params={ + "q": f"org:{org} is:pr is:open", + "sort": "updated", + "order": "desc", + "per_page": PER_PAGE, + "page": page, + # The legacy issue-search syntax is gone; without this the endpoint + # rejects the query rather than falling back to it. + "advanced_search": "true", + }) + batch = payload.get("items", []) + items.extend(batch) + total = payload.get("total_count", len(items)) + if len(batch) < PER_PAGE or len(items) >= total: + return items[:limit], total > len(items[:limit]) + page += 1 + return items[:limit], True + + +def repo_name(item): + """Repo name for a search result β€” the API gives only the repository's API URL.""" + return item.get("repository_url", "").rsplit("/", 1)[-1] + + +def is_bot(item): + return (item.get("user") or {}).get("type") == "Bot" + + +def author(item): + return (item.get("user") or {}).get("login") or "?" + + +def parse_time(value): + return datetime.fromisoformat(value) + + +def contributor_prs(items, repos, maintainers): + """The open PRs worth reporting: org repos, human authors, not maintainers.""" + keep = [] + for item in items: + if repo_name(item) not in repos or is_bot(item): + continue + if author(item).lower() in maintainers: + continue + keep.append(item) + return keep + + +def in_window(prs, cutoff): + """The PRs that have moved at all since the cutoff. A PR nobody has touched in + three days is not news, whatever its state here.""" + return [pr for pr in prs if parse_time(pr["updated_at"]) >= cutoff] + + +def pr_id(pr): + return str(pr.get("id")) + + +def activity_key(pr): + """The value a re-report is judged against. + + `updated_at` moves on any change at all, so an edit that touches several PRs at + once β€” a label sweep, a base branch renamed β€” resurfaces every one of them. The + accurate alternative is the head SHA and the comment counts, which are not in the + search result and cost a request per PR that moved; this is the deliberate cheaper + half of that trade. + """ + return pr.get("updated_at") + + +# ---- Dedup state ----------------------------------------------------------- +# +# Which PRs have already been reported, and what their activity looked like then. +# Every failure to read it is a cache miss rather than an error: losing the file +# re-reports the window once, which is noisy and never wrong, and that is what makes +# it a cache rather than something to back up. + + +def empty_state(): + return {"version": STATE_VERSION, "seen": {}} + + +def load_state(path): + if not path: + return empty_state() + if not os.path.exists(path): + print(f"No state file at {path}; treating every PR in the window as new.") + return empty_state() + try: + with open(path, encoding="utf-8") as handle: + data = json.load(handle) + except (OSError, json.JSONDecodeError) as exc: + print(f"Note: unreadable state file {path} ({exc}); treating every PR as new.") + return empty_state() + if not isinstance(data, dict) or not isinstance(data.get("seen"), dict): + print(f"Note: unexpected shape in {path}; treating every PR as new.") + return empty_state() + # A file written by another schema version cannot be trusted field by field, so + # it is a cache miss rather than something to misread. + if data.get("version") != STATE_VERSION: + print(f"Note: {path} is version {data.get('version')!r}, expected " + f"{STATE_VERSION}; treating every PR as new.") + return empty_state() + print(f"Loaded state for {len(data['seen'])} previously reported PRs.") + return data + + +def partition_by_state(prs, state): + """Split into (new, changed, unchanged) against what was last reported.""" + seen = state.get("seen", {}) + new, changed, unchanged = [], [], [] + for pr in prs: + previous = seen.get(pr_id(pr)) + if previous is None: + new.append(pr) + elif previous.get("updated_at") != activity_key(pr): + changed.append(pr) + else: + unchanged.append(pr) + return new, changed, unchanged + + +def save_state(path, state, reported, retention_days=DEFAULT_RETENTION_DAYS): + """Record `reported` as seen, prune old entries, write atomically. + + Returns (kept, pruned). + """ + now = datetime.now(timezone.utc) + stamp = now.strftime("%Y-%m-%dT%H:%M:%SZ") + seen = dict(state.get("seen", {})) + for pr in reported: + seen[pr_id(pr)] = { + "updated_at": activity_key(pr), + "last_reported": stamp, + # Not read back. The file is the first thing anyone opens when the digest + # reports the wrong thing, and an id alone identifies nothing. + "pr": f"{repo_name(pr)}#{pr.get('number')}", + } + + cutoff = now - timedelta(days=retention_days) + kept = {} + for key, record in seen.items(): + try: + last = datetime.strptime(record.get("last_reported", ""), + "%Y-%m-%dT%H:%M:%SZ").replace(tzinfo=timezone.utc) + except (TypeError, ValueError): + continue # malformed entry β€” drop it rather than keep it forever + if last >= cutoff: + kept[key] = record + + directory = os.path.dirname(path) + if directory: + os.makedirs(directory, exist_ok=True) + temporary = f"{path}.tmp" + with open(temporary, "w", encoding="utf-8") as handle: + json.dump({"version": STATE_VERSION, "updated_at": stamp, "seen": kept}, + handle, indent=2) + os.replace(temporary, path) # atomic: a crash mid-write cannot corrupt the state + return len(kept), len(seen) - len(kept) + + +# ---- Discord rendering ----------------------------------------------------- +# +# A header, then one block per repository whose PRs changed. Components V2 so that +# each repository is its own component: a long day splits between repositories +# rather than mid-list. +COMPONENTS_V2_FLAG = 1 << 15 +CONTAINER = 17 +TEXT_DISPLAY = 10 +SEPARATOR = 14 +# Discord's ceiling on all the text in one Components V2 message. +MAX_MESSAGE_TEXT_CHARS = 4000 +MAX_COMPONENTS_PER_MESSAGE = 10 +TITLE_CHARS = 90 + +NEW_MARKER = "🟒" +UPDATED_MARKER = "✏️" + + +def clip(text, limit): + text = (text or "").strip() + return text if len(text) <= limit else text[: limit - 1] + "…" + + +def age(then, now): + """Compact age, coarsening as it grows: 40m, 6h, 3d, 5w.""" + minutes = max(0, int((now - then).total_seconds() // 60)) + if minutes < 60: + return f"{minutes}m" + hours = minutes // 60 + if hours < 48: + return f"{hours}h" + days = hours // 24 + if days < 14: + return f"{days}d" + return f"{days // 7}w" + + +def build_pr_line(pr, now, is_new): + marker = NEW_MARKER if is_new else UPDATED_MARKER + stamp = age(parse_time(pr["created_at"] if is_new else pr["updated_at"]), now) + title = clip(pr.get("title"), TITLE_CHARS) + if pr.get("draft"): + title = f"[draft] {title}" + comments = pr.get("comments") or 0 + replies = f" Β· πŸ’¬{comments}" if comments else "" + return (f"{marker} [#{pr['number']}]({pr['html_url']}) @{author(pr)} Β· " + f"{stamp}{replies} Β· {title}") + + +def group_by_repo(new, updated, now): + """(text, ids) per repository, new PRs above updated ones. + + Repositories are ordered by how much changed, so the busiest is read first. + Within a group each PR sorts on the timestamp its line shows β€” the search + returns them in update order, which reads as no order at all next to an age + taken from the creation date. + """ + blocks = [] + for repo in sorted({repo_name(pr) for pr in new + updated}): + lines, ids = [f"**{repo}**"], set() + for prs, field, is_new in ((new, "created_at", True), + (updated, "updated_at", False)): + group = sorted((pr for pr in prs if repo_name(pr) == repo), + key=itemgetter(field), reverse=True) + lines += [build_pr_line(pr, now, is_new) for pr in group] + ids |= {pr_id(pr) for pr in group} + blocks.append(("\n".join(lines), ids)) + blocks.sort(key=lambda block: -len(block[1])) + return blocks + + +def window_label(hours): + if hours % 24 == 0 and hours >= 24: + days = hours // 24 + return f"{days} day{'s' if days > 1 else ''}" + return f"{hours}h" + + +def build_header(new, updated, backlog, window_hours, truncated): + lines = [f"**Contributor pull requests** Β· last {window_label(window_hours)}"] + if new or updated: + lines.append(f"{NEW_MARKER} **{len(new)}** new Β· " + f"{UPDATED_MARKER} **{len(updated)}** updated") + else: + lines.append("Nothing opened or updated.") + lines.append(f"**{backlog}** open from contributors across the org.") + if truncated: + lines.append("_GitHub capped the search at 1000 results; the counts are a floor._") + return "\n".join(lines) + + +def chunk_blocks(blocks, first_used=0, + max_items=MAX_COMPONENTS_PER_MESSAGE, + max_chars=MAX_MESSAGE_TEXT_CHARS): + """Group rendered blocks into messages within Discord's budgets. + + `first_used` is what the header has already spent on the first message; without + it a busy day's first message goes over on the header alone. + + A single block over the character budget still gets its own message rather than + being dropped β€” clipped titles should keep that from arising. + """ + chunks, current, used = [], [], first_used + for block in blocks: + cost = len(block[0]) if isinstance(block, tuple) else len(block) + if current and (len(current) >= max_items or used + cost > max_chars): + chunks.append(current) + current, used = [], 0 + current.append(block) + used += cost + if current: + chunks.append(current) + return chunks + + +def build_messages(new, updated, backlog, window_hours, now, truncated=False): + """Return (messages, coverage). + + coverage[i] is the set of PR ids message i accounts for, so a run that fails + partway through can still record exactly what reached Discord β€” otherwise a + failure on the last message re-posts the first ones tomorrow. + """ + header = build_header(new, updated, backlog, window_hours, truncated) + blocks = group_by_repo(new, updated, now) + messages, coverage = [], [] + # A quiet day still owes the channel its header, so seed one empty chunk. + for index, chunk in enumerate(chunk_blocks(blocks, first_used=len(header)) or [[]]): + components = [] + if index == 0: + components.append({"type": TEXT_DISPLAY, "content": header}) + if chunk: + components.append({"type": SEPARATOR}) + components += [{"type": TEXT_DISPLAY, "content": text} for text, _ in chunk] + messages.append({ + "flags": COMPONENTS_V2_FLAG, + "components": [{"type": CONTAINER, "components": components}], + }) + coverage.append(set().union(*(ids for _, ids in chunk)) if chunk else set()) + return messages, coverage + + +def components_webhook_url(webhook_url): + """The webhook, told to respect the components field, which it ignores without.""" + parts = urlsplit(webhook_url) + query = dict(parse_qsl(parts.query)) + query["with_components"] = "true" + return urlunsplit(parts._replace(query=urlencode(query))) + + +def post_to_discord(session, url, messages): + """POST each message in order; return how many Discord accepted.""" + for index, payload in enumerate(messages): + try: + resp = request_with_retry(session, "POST", url, json=payload) + except requests.RequestException as exc: + print(f"Discord unreachable on message {index + 1}/{len(messages)} ({exc}).") + return index + if resp.status_code >= 400: + print(f"Discord rejected message {index + 1}/{len(messages)} " + f"({resp.status_code}): {resp.text[:300]}") + return index + return len(messages) + + +def main(): + parser = argparse.ArgumentParser( + description="Post a daily digest of contributor pull requests to Discord.") + parser.add_argument("--org", help=f"GitHub org to scan (else GITHUB_PRS_ORG, default {DEFAULT_ORG}).") + parser.add_argument("--token", help="GitHub token (else GITHUB_PRS_TOKEN).") + parser.add_argument("--webhook", help="Discord webhook URL (else GITHUB_PRS_DISCORD_WEBHOOK_URL).") + parser.add_argument("--maintainers", default=MAINTAINERS_FILE, + help="Logins to treat as maintainers, one per line.") + parser.add_argument("--window-hours", type=int, default=DEFAULT_WINDOW_HOURS, + help=f"How far back a PR must have moved to be considered " + f"(default {DEFAULT_WINDOW_HOURS}).") + parser.add_argument("--state", metavar="PATH", + help="Dedup state: without it every PR in the window is new.") + parser.add_argument("--state-retention-days", type=int, + default=DEFAULT_RETENTION_DAYS, metavar="N", + help=f"Drop state entries older than this " + f"(default {DEFAULT_RETENTION_DAYS}).") + parser.add_argument("--include-forks", action="store_true", + help="Also report the org's forks of upstream projects.") + parser.add_argument("--include-archived", action="store_true", + help="Also report archived repositories.") + parser.add_argument("--exclude-repo", action="append", default=[], + help="Skip this repository (repeatable).") + parser.add_argument("--dry-run", action="store_true", + help="Print the Discord payload instead of posting it.") + args = parser.parse_args() + + if args.window_hours < 1: + sys.exit("--window-hours must be at least 1.") + + org = args.org or os.environ.get("GITHUB_PRS_ORG") or DEFAULT_ORG + token = get_env("GITHUB_PRS_TOKEN", args.token) + webhook = get_env("GITHUB_PRS_DISCORD_WEBHOOK_URL", args.webhook, + required=not args.dry_run) + maintainers = load_maintainers(args.maintainers) + + session = github_session(token) + repos = fetch_repos(session, org, args.include_forks, args.include_archived) + repos -= set(args.exclude_repo) + items, truncated = search_open_prs(session, org) + prs = contributor_prs(items, repos, maintainers) + + now = datetime.now(timezone.utc) + state = load_state(args.state) + new, changed, unchanged = partition_by_state( + in_window(prs, now - timedelta(hours=args.window_hours)), state) + print(f"{len(items)} open PRs in {org}, {len(prs)} from contributors across " + f"{len(repos)} repos: {len(new)} new, {len(changed)} changed since last " + f"reported, {len(unchanged)} unchanged (skipped).") + + messages, coverage = build_messages(new, changed, len(prs), args.window_hours, + now, truncated) + if args.dry_run: + print(json.dumps(messages, indent=2, ensure_ascii=False)) + return + + posted = post_to_discord(requests.Session(), + components_webhook_url(webhook), messages) + # Only what Discord accepted. A PR in a message that never landed stays eligible. + if args.state: + landed = set().union(*coverage[:posted]) if posted else set() + reported = [pr for pr in new + changed if pr_id(pr) in landed] + kept, pruned = save_state(args.state, state, reported, + args.state_retention_days) + print(f"State: {len(reported)} recorded, {kept} tracked " + f"({pruned} pruned beyond {args.state_retention_days} days).") + if posted < len(messages): + sys.exit(f"Posted {posted} of {len(messages)} messages.") + + +if __name__ == "__main__": + main() diff --git a/github_prs/maintainers.txt b/github_prs/maintainers.txt new file mode 100644 index 0000000..b9c0ac5 --- /dev/null +++ b/github_prs/maintainers.txt @@ -0,0 +1,20 @@ +# GitHub logins whose pull requests the digest does not report: the people already in +# the review loop. Everyone else opening a PR in session-foundation's repos is a +# contributor and gets a line. +# +# Matched case-insensitively against the PR author's login. Bot accounts need no +# entry β€” the digest drops every account GitHub types as a Bot. +# +# Neither org membership nor push access is a usable substitute for this list. The org +# has six members, two of whom are not in the review loop, and push access is held by +# a dozen more as outside collaborators. Both would report a maintainer's PR as a +# stranger's, or leave a contractor's PR out of the digest entirely. +# +# The list goes stale silently: until a new maintainer is added, their PRs are +# reported as if they came from an outside contributor. That is the failure mode to +# watch for when someone joins. + +Bilb +jagerman +mpretty-cyro +stfsession diff --git a/github_prs/requirements.txt b/github_prs/requirements.txt new file mode 100644 index 0000000..70429a6 --- /dev/null +++ b/github_prs/requirements.txt @@ -0,0 +1,2 @@ +# What the deployment runs. The suite is stdlib unittest and imports nothing else. +requests==2.34.2 diff --git a/github_prs/test_digest.py b/github_prs/test_digest.py new file mode 100644 index 0000000..ec42cfe --- /dev/null +++ b/github_prs/test_digest.py @@ -0,0 +1,412 @@ +""" + python -m unittest discover # from github_prs/ +""" +import contextlib +import io +import json +import os +import tempfile +import unittest +from datetime import datetime, timedelta, timezone +from unittest import mock + +import digest + +NOW = datetime(2026, 9, 24, 12, 0, tzinfo=timezone.utc) +CUTOFF = NOW - timedelta(hours=25) + + +def pr(number=1, login="octocat", repo="session-android", created="2026-09-24T08:00:00Z", + updated=None, title="Fix a thing", user_type="User", **extra): + item = { + "id": 10_000 + number, + "number": number, + "title": title, + "html_url": f"https://github.com/session-foundation/{repo}/pull/{number}", + "repository_url": f"https://api.github.com/repos/session-foundation/{repo}", + "user": {"login": login, "type": user_type}, + "created_at": created, + "updated_at": updated or created, + } + item.update(extra) + return item + + +class TestMaintainers(unittest.TestCase): + def test_comments_and_blanks_are_ignored(self): + with mock.patch("builtins.open", mock.mock_open(read_data= + "# a comment\n\nBilb\njagerman # trailing\n")): + self.assertEqual(digest.load_maintainers("x"), {"bilb", "jagerman"}) + + def test_the_shipped_list_parses_and_is_unique(self): + with open(digest.MAINTAINERS_FILE, encoding="utf-8") as handle: + lines = [line.split("#", 1)[0].strip() for line in handle] + logins = [line for line in lines if line] + self.assertTrue(logins) + self.assertEqual(len(logins), len(set(login.lower() for login in logins))) + + +class TestSelection(unittest.TestCase): + repos = {"session-android", "session-desktop"} + maintainers = frozenset({"bilb", "mpretty-cyro"}) + + def select(self, items): + return digest.contributor_prs(items, self.repos, self.maintainers) + + def test_maintainers_are_dropped_whatever_the_case(self): + self.assertEqual(self.select([pr(login="BiLb")]), []) + + def test_bots_are_dropped_on_account_type(self): + self.assertEqual(self.select([pr(login="dependabot[bot]", user_type="Bot")]), []) + + def test_repos_outside_the_allowed_set_are_dropped(self): + self.assertEqual(self.select([pr(repo="session-pysogs")]), []) + + def test_a_contributor_pr_is_kept(self): + self.assertEqual(len(self.select([pr(login="someone")])), 1) + + +class TestWindow(unittest.TestCase): + def test_only_prs_that_moved_since_the_cutoff_are_considered(self): + fresh = pr(1, updated="2026-09-24T09:00:00Z") + stale = pr(2, created="2026-08-01T08:00:00Z", updated="2026-08-02T08:00:00Z") + self.assertEqual([p["number"] for p in digest.in_window([fresh, stale], CUTOFF)], + [1]) + + def test_the_cutoff_itself_counts_as_inside_the_window(self): + edge = pr(1, updated=CUTOFF.strftime("%Y-%m-%dT%H:%M:%SZ")) + self.assertEqual(len(digest.in_window([edge], CUTOFF)), 1) + + def test_an_old_pr_that_just_moved_is_in(self): + """New means never reported, not recently opened β€” a year-old PR that picks up + a comment is exactly what the digest is for.""" + old = pr(1, created="2025-01-01T00:00:00Z", updated="2026-09-24T09:00:00Z") + self.assertEqual(len(digest.in_window([old], CUTOFF)), 1) + + +class TestPartitionByState(unittest.TestCase): + def state(self, *records): + return {"version": digest.STATE_VERSION, + "seen": {key: {"updated_at": stamp} for key, stamp in records}} + + def test_a_pr_never_reported_is_new(self): + new, changed, unchanged = digest.partition_by_state([pr(1)], digest.empty_state()) + self.assertEqual([p["number"] for p in new], [1]) + self.assertEqual((changed, unchanged), ([], [])) + + def test_a_pr_that_moved_since_it_was_reported_is_changed(self): + item = pr(1, updated="2026-09-24T09:00:00Z") + new, changed, unchanged = digest.partition_by_state( + [item], self.state((digest.pr_id(item), "2026-09-20T09:00:00Z"))) + self.assertEqual([p["number"] for p in changed], [1]) + self.assertEqual((new, unchanged), ([], [])) + + def test_a_pr_that_has_not_moved_is_dropped(self): + item = pr(1, updated="2026-09-24T09:00:00Z") + new, changed, unchanged = digest.partition_by_state( + [item], self.state((digest.pr_id(item), "2026-09-24T09:00:00Z"))) + self.assertEqual([p["number"] for p in unchanged], [1]) + self.assertEqual((new, changed), ([], [])) + + +class TestStateFile(unittest.TestCase): + def setUp(self): + self.directory = tempfile.TemporaryDirectory() + self.addCleanup(self.directory.cleanup) + self.path = os.path.join(self.directory.name, "sub", "seen.json") + + def save(self, *prs, **kwargs): + with contextlib.redirect_stdout(io.StringIO()): + return digest.save_state(self.path, digest.empty_state(), list(prs), **kwargs) + + def load(self): + with contextlib.redirect_stdout(io.StringIO()): + return digest.load_state(self.path) + + def test_a_reported_pr_comes_back_unchanged_next_run(self): + item = pr(1, updated="2026-09-24T09:00:00Z") + self.save(item) + _, _, unchanged = digest.partition_by_state([item], self.load()) + self.assertEqual(len(unchanged), 1) + + def test_the_same_pr_moved_comes_back_changed(self): + self.save(pr(1, updated="2026-09-24T09:00:00Z")) + _, changed, _ = digest.partition_by_state( + [pr(1, updated="2026-09-24T11:00:00Z")], self.load()) + self.assertEqual(len(changed), 1) + + def test_the_file_names_the_pr_for_whoever_opens_it(self): + self.save(pr(1958, repo="session-desktop")) + with open(self.path, encoding="utf-8") as handle: + self.assertIn("session-desktop#1958", handle.read()) + + def test_no_path_means_no_state_and_no_complaint(self): + self.assertEqual(digest.load_state(None), digest.empty_state()) + + def test_every_unreadable_state_is_a_cache_miss_not_an_error(self): + """Losing it re-reports the window once. Failing the run instead would mean a + corrupt cache stops the digest entirely.""" + for content in ("{ not json", '{"version": 1}', "[]", + '{"version": 99, "seen": {}}'): + with self.subTest(content=content): + with open(self.path.replace("sub/", ""), "w", encoding="utf-8") as fh: + fh.write(content) + with contextlib.redirect_stdout(io.StringIO()): + state = digest.load_state(self.path.replace("sub/", "")) + self.assertEqual(state, digest.empty_state()) + + def test_a_missing_file_is_a_cache_miss(self): + self.assertEqual(self.load(), digest.empty_state()) + + def test_entries_are_pruned_past_the_retention(self): + self.save(pr(1)) + state = self.load() + for record in state["seen"].values(): + record["last_reported"] = "2026-01-01T00:00:00Z" + with contextlib.redirect_stdout(io.StringIO()): + kept, pruned = digest.save_state(self.path, state, [], retention_days=30) + self.assertEqual((kept, pruned), (0, 1)) + + def test_a_malformed_entry_is_dropped_rather_than_kept_forever(self): + state = {"version": digest.STATE_VERSION, + "seen": {"1": {"updated_at": "x", "last_reported": "never"}}} + with contextlib.redirect_stdout(io.StringIO()): + kept, _ = digest.save_state(self.path, state, []) + self.assertEqual(kept, 0) + + def test_nothing_is_left_behind_by_the_atomic_write(self): + self.save(pr(1)) + self.assertEqual(sorted(os.listdir(os.path.dirname(self.path))), ["seen.json"]) + + +class TestAge(unittest.TestCase): + def test_coarsens_as_it_grows(self): + for delta, expected in [(timedelta(minutes=40), "40m"), + (timedelta(hours=6), "6h"), + (timedelta(hours=47), "47h"), + (timedelta(days=3), "3d"), + (timedelta(days=35), "5w")]: + self.assertEqual(digest.age(NOW - delta, NOW), expected) + + def test_a_clock_skewed_future_timestamp_reads_as_zero(self): + self.assertEqual(digest.age(NOW + timedelta(minutes=5), NOW), "0m") + + +class TestLines(unittest.TestCase): + def test_a_new_pr_line_carries_the_link_author_and_age(self): + line = digest.build_pr_line(pr(2151, login="octocat"), NOW, True) + self.assertIn(digest.NEW_MARKER, line) + self.assertIn("[#2151](https://github.com/session-foundation/session-android/pull/2151)", line) + self.assertIn("@octocat", line) + self.assertIn("4h", line) + + def test_an_updated_line_ages_from_the_update_not_the_creation(self): + item = pr(7, created="2026-09-01T12:00:00Z", updated="2026-09-24T09:00:00Z") + line = digest.build_pr_line(item, NOW, False) + self.assertIn(digest.UPDATED_MARKER, line) + self.assertIn("3h", line) + + def test_drafts_and_comment_counts_are_marked(self): + line = digest.build_pr_line(pr(9, draft=True, comments=3), NOW, True) + self.assertIn("[draft]", line) + self.assertIn("πŸ’¬3", line) + + def test_a_pr_with_no_comments_says_nothing_about_them(self): + self.assertNotIn("πŸ’¬", digest.build_pr_line(pr(9, comments=0), NOW, True)) + + def test_long_titles_are_clipped(self): + line = digest.build_pr_line(pr(9, title="x" * 200), NOW, True) + self.assertIn("…", line) + self.assertLess(len(line), 200) + + +class TestGrouping(unittest.TestCase): + def test_the_busiest_repo_comes_first_and_new_leads_its_block(self): + new = [pr(1, repo="session-ios"), pr(2, repo="session-android"), + pr(3, repo="session-android")] + updated = [pr(4, repo="session-android")] + blocks = digest.group_by_repo(new, updated, NOW) + self.assertTrue(blocks[0][0].startswith("**session-android**")) + android = blocks[0][0].splitlines() + self.assertEqual(len(android), 4) + self.assertEqual(android.count(""), 0) + self.assertIn(digest.NEW_MARKER, android[1]) + self.assertIn(digest.UPDATED_MARKER, android[3]) + + def test_each_group_sorts_on_the_age_it_shows(self): + new = [pr(1, created="2026-09-01T00:00:00Z", updated="2026-09-24T00:00:00Z"), + pr(2, created="2026-09-20T00:00:00Z", updated="2026-09-21T00:00:00Z")] + updated = [pr(3, created="2026-01-01T00:00:00Z", updated="2026-09-22T00:00:00Z"), + pr(4, created="2026-01-01T00:00:00Z", updated="2026-09-23T00:00:00Z")] + lines = digest.group_by_repo(new, updated, NOW)[0][0].splitlines() + self.assertEqual([line.split("[#")[1].split("]")[0] for line in lines[1:]], + ["2", "1", "4", "3"]) + + def test_a_repo_appears_once_however_many_prs_it_has(self): + blocks = digest.group_by_repo([pr(1), pr(2), pr(3)], [], NOW) + self.assertEqual(len(blocks), 1) + + def test_a_block_accounts_for_every_pr_it_shows(self): + """What reached Discord is recorded per message, so a partial post cannot + suppress the PRs that never went out.""" + prs = [pr(1), pr(2)] + _, ids = digest.group_by_repo(prs, [], NOW)[0] + self.assertEqual(ids, {digest.pr_id(p) for p in prs}) + + +class TestHeader(unittest.TestCase): + def test_counts_both_kinds_and_the_backlog(self): + header = digest.build_header([pr(1)], [pr(2), pr(3)], 12, 25, False) + self.assertIn("**1** new", header) + self.assertIn("**2** updated", header) + self.assertIn("**12** open from contributors", header) + self.assertIn("last 25h", header) + + def test_a_quiet_day_says_so(self): + header = digest.build_header([], [], 12, 25, False) + self.assertIn("Nothing opened or updated.", header) + self.assertIn("**12** open", header) + + def test_a_long_window_reads_in_days(self): + self.assertIn("last 3 days", digest.build_header([], [], 0, 72, False)) + self.assertIn("last 1 day", digest.build_header([], [], 0, 24, False)) + self.assertIn("last 25h", digest.build_header([], [], 0, 25, False)) + + def test_truncation_is_declared(self): + self.assertIn("floor", digest.build_header([], [], 1000, 25, True)) + + +class TestChunking(unittest.TestCase): + def test_splits_on_the_component_count(self): + blocks = ["x"] * (digest.MAX_COMPONENTS_PER_MESSAGE + 1) + chunks = digest.chunk_blocks(blocks) + self.assertEqual([len(chunk) for chunk in chunks], + [digest.MAX_COMPONENTS_PER_MESSAGE, 1]) + + def test_splits_on_the_character_budget(self): + blocks = ["x" * 2500, "y" * 2500] + self.assertEqual(len(digest.chunk_blocks(blocks)), 2) + + def test_the_header_is_charged_to_the_first_message_only(self): + block = "x" * 1900 + chunks = digest.chunk_blocks([block, block], first_used=0) + self.assertEqual(len(chunks), 1) + chunks = digest.chunk_blocks([block, block], first_used=1000) + self.assertEqual(len(chunks), 2) + + def test_an_oversized_block_still_gets_a_message(self): + blocks = digest.chunk_blocks(["x" * (digest.MAX_MESSAGE_TEXT_CHARS + 100)]) + self.assertEqual(len(blocks), 1) + + +class TestMessages(unittest.TestCase): + def test_a_quiet_day_still_posts_the_header(self): + messages, coverage = digest.build_messages([], [], 4, 25, NOW) + self.assertEqual(len(messages), 1) + self.assertEqual(coverage, [set()]) + blocks = messages[0]["components"][0]["components"] + self.assertEqual(len(blocks), 1) + self.assertIn("Nothing opened", blocks[0]["content"]) + + def test_the_payload_is_components_v2(self): + message = digest.build_messages([pr(1)], [], 1, 25, NOW)[0][0] + self.assertEqual(message["flags"], digest.COMPONENTS_V2_FLAG) + self.assertEqual(message["components"][0]["type"], digest.CONTAINER) + + def test_only_the_first_message_carries_the_header(self): + new = [pr(n, repo=f"repo-{n}", title="y" * 80) for n in range(40)] + messages, coverage = digest.build_messages(new, [], 40, 25, NOW) + self.assertGreater(len(messages), 1) + self.assertEqual(set().union(*coverage), {digest.pr_id(p) for p in new}) + for message in messages[1:]: + rendered = json.dumps(message, ensure_ascii=False) + self.assertNotIn("Contributor pull requests", rendered) + self.assertLess(len(rendered), digest.MAX_MESSAGE_TEXT_CHARS * 2) + + +class TestWebhookUrl(unittest.TestCase): + def test_components_are_requested(self): + url = digest.components_webhook_url("https://discord.com/api/webhooks/1/tok") + self.assertIn("with_components=true", url) + + def test_an_existing_query_survives(self): + url = digest.components_webhook_url("https://discord.com/api/webhooks/1/tok?wait=true") + self.assertIn("wait=true", url) + self.assertIn("with_components=true", url) + + +class TestFetching(unittest.TestCase): + def test_forks_and_archived_repos_are_left_out(self): + page = [{"name": "session-android", "fork": False, "archived": False}, + {"name": "session-pysogs", "fork": True, "archived": False}, + {"name": "session-ios-private", "fork": False, "archived": True}] + with mock.patch.object(digest, "fetch_json", return_value=page): + self.assertEqual(digest.fetch_repos(None, "org"), {"session-android"}) + + def test_forks_can_be_asked_for(self): + page = [{"name": "session-pysogs", "fork": True, "archived": False}] + with mock.patch.object(digest, "fetch_json", return_value=page): + self.assertEqual(digest.fetch_repos(None, "org", include_forks=True), + {"session-pysogs"}) + + def test_repo_pagination_follows_full_pages(self): + pages = [[{"name": f"r{n}", "fork": False, "archived": False} + for n in range(digest.PER_PAGE)], + [{"name": "last", "fork": False, "archived": False}]] + with mock.patch.object(digest, "fetch_json", side_effect=pages): + self.assertEqual(len(digest.fetch_repos(None, "org")), digest.PER_PAGE + 1) + + def test_search_stops_when_the_results_run_out(self): + payload = {"total_count": 2, "items": [pr(1), pr(2)]} + with mock.patch.object(digest, "fetch_json", return_value=payload) as fetch: + items, truncated = digest.search_open_prs(None, "org") + self.assertEqual(len(items), 2) + self.assertFalse(truncated) + self.assertEqual(fetch.call_count, 1) + + def test_search_stops_at_the_result_ceiling_rather_than_erroring(self): + full = {"total_count": 1500, + "items": [pr(n) for n in range(digest.PER_PAGE)]} + with mock.patch.object(digest, "fetch_json", return_value=full) as fetch: + items, truncated = digest.search_open_prs(None, "org") + self.assertEqual(len(items), digest.SEARCH_RESULT_LIMIT) + self.assertTrue(truncated) + self.assertEqual(fetch.call_count, digest.SEARCH_RESULT_LIMIT // digest.PER_PAGE) + + +class TestRetryAfter(unittest.TestCase): + def response(self, headers): + return mock.Mock(headers=headers) + + def test_retry_after_seconds_is_used(self): + self.assertEqual(digest.retry_after_seconds(self.response({"retry-after": "12"}), 1), 12) + + def test_a_primary_limit_falls_back_to_the_reset_epoch(self): + import time as time_module + reset = str(int(time_module.time()) + 30) + seconds = digest.retry_after_seconds( + self.response({"x-ratelimit-reset": reset}), 1) + self.assertTrue(25 <= seconds <= 31, seconds) + + def test_nonsense_falls_back_to_the_caller_default(self): + for headers in ({"retry-after": "soon"}, {"retry-after": "-30"}, + {"retry-after": "inf"}, {"x-ratelimit-reset": "?"}, {}): + self.assertEqual(digest.retry_after_seconds(self.response(headers), 7), 7) + + +class TestPosting(unittest.TestCase): + def test_a_rejection_reports_what_landed_before_it(self): + ok, bad = mock.Mock(status_code=204), mock.Mock(status_code=400, text="no") + session = mock.Mock() + session.request.side_effect = [ok, bad] + with contextlib.redirect_stdout(io.StringIO()): + self.assertEqual(digest.post_to_discord(session, "url", [{}, {}, {}]), 1) + + def test_all_accepted(self): + session = mock.Mock() + session.request.return_value = mock.Mock(status_code=204) + self.assertEqual(digest.post_to_discord(session, "url", [{}, {}]), 2) + + +if __name__ == "__main__": + unittest.main() From 2e7a1621270e567206cc60fbb01757fc8908f6e0 Mon Sep 17 00:00:00 2001 From: Audric Ackermann Date: Thu, 24 Sep 2026 12:29:22 +1000 Subject: [PATCH 002/101] fix: split a repository block that outgrows a Discord message A block over the text budget was posted whole and rejected, and since nothing in a rejected message is recorded, the same block was rebuilt on every run until the PRs aged out of the window. Blocks are now split under a repeated heading, each sized to fit beside the header. --- github_prs/digest.py | 41 ++++++++++++++++++++++++++++----------- github_prs/test_digest.py | 31 +++++++++++++++++++++++++++++ 2 files changed, 61 insertions(+), 11 deletions(-) diff --git a/github_prs/digest.py b/github_prs/digest.py index a03b439..559cf20 100755 --- a/github_prs/digest.py +++ b/github_prs/digest.py @@ -358,7 +358,7 @@ def save_state(path, state, reported, retention_days=DEFAULT_RETENTION_DAYS): # # A header, then one block per repository whose PRs changed. Components V2 so that # each repository is its own component: a long day splits between repositories -# rather than mid-list. +# rather than mid-list, unless one repository alone outgrows a message. COMPONENTS_V2_FLAG = 1 << 15 CONTAINER = 17 TEXT_DISPLAY = 10 @@ -403,7 +403,25 @@ def build_pr_line(pr, now, is_new): f"{stamp}{replies} Β· {title}") -def group_by_repo(new, updated, now): +def split_block(heading, entries, max_chars): + """(text, ids) blocks of at most `max_chars`, each under its own copy of `heading`. + + A block over the budget is rejected by Discord, and since nothing in it is then + recorded, the same block would be rebuilt every run until the window moved on. + """ + blocks, lines, ids, used = [], [heading], set(), len(heading) + for line, key in entries: + if ids and used + 1 + len(line) > max_chars: + blocks.append(("\n".join(lines), ids)) + lines, ids, used = [heading], set(), len(heading) + lines.append(line) + ids.add(key) + used += 1 + len(line) + blocks.append(("\n".join(lines), ids)) + return blocks + + +def group_by_repo(new, updated, now, max_chars=MAX_MESSAGE_TEXT_CHARS): """(text, ids) per repository, new PRs above updated ones. Repositories are ordered by how much changed, so the busiest is read first. @@ -411,18 +429,17 @@ def group_by_repo(new, updated, now): returns them in update order, which reads as no order at all next to an age taken from the creation date. """ - blocks = [] + repos = [] for repo in sorted({repo_name(pr) for pr in new + updated}): - lines, ids = [f"**{repo}**"], set() + entries = [] for prs, field, is_new in ((new, "created_at", True), (updated, "updated_at", False)): group = sorted((pr for pr in prs if repo_name(pr) == repo), key=itemgetter(field), reverse=True) - lines += [build_pr_line(pr, now, is_new) for pr in group] - ids |= {pr_id(pr) for pr in group} - blocks.append(("\n".join(lines), ids)) - blocks.sort(key=lambda block: -len(block[1])) - return blocks + entries += [(build_pr_line(pr, now, is_new), pr_id(pr)) for pr in group] + repos.append((split_block(f"**{repo}**", entries, max_chars), len(entries))) + repos.sort(key=lambda item: -item[1]) + return [block for blocks, _ in repos for block in blocks] def window_label(hours): @@ -454,7 +471,7 @@ def chunk_blocks(blocks, first_used=0, it a busy day's first message goes over on the header alone. A single block over the character budget still gets its own message rather than - being dropped β€” clipped titles should keep that from arising. + being dropped; group_by_repo splits blocks to keep that from arising. """ chunks, current, used = [], [], first_used for block in blocks: @@ -477,7 +494,9 @@ def build_messages(new, updated, backlog, window_hours, now, truncated=False): failure on the last message re-posts the first ones tomorrow. """ header = build_header(new, updated, backlog, window_hours, truncated) - blocks = group_by_repo(new, updated, now) + # Every block is sized to fit beside the header, though only the first message + # carries it: simpler than sizing the first block differently. + blocks = group_by_repo(new, updated, now, MAX_MESSAGE_TEXT_CHARS - len(header)) messages, coverage = [], [] # A quiet day still owes the channel its header, so seed one empty chunk. for index, chunk in enumerate(chunk_blocks(blocks, first_used=len(header)) or [[]]): diff --git a/github_prs/test_digest.py b/github_prs/test_digest.py index ec42cfe..c2c550c 100644 --- a/github_prs/test_digest.py +++ b/github_prs/test_digest.py @@ -246,6 +246,27 @@ def test_a_repo_appears_once_however_many_prs_it_has(self): blocks = digest.group_by_repo([pr(1), pr(2), pr(3)], [], NOW) self.assertEqual(len(blocks), 1) + def test_a_repo_that_outgrows_a_message_is_split_under_repeated_headings(self): + prs = [pr(n, title="t" * 80) for n in range(60)] + blocks = digest.group_by_repo(prs, [], NOW, max_chars=1000) + self.assertGreater(len(blocks), 1) + for text, ids in blocks: + self.assertLessEqual(len(text), 1000) + self.assertTrue(text.startswith("**session-android**\n")) + self.assertEqual(len(text.splitlines()) - 1, len(ids)) + self.assertEqual(sum(len(ids) for _, ids in blocks), len(prs)) + self.assertEqual(set().union(*(ids for _, ids in blocks)), + {digest.pr_id(p) for p in prs}) + + def test_a_split_repo_keeps_its_blocks_together_and_first(self): + busy = [pr(n, repo="busy", title="t" * 80) for n in range(30)] + quiet = [pr(100, repo="quiet")] + headings = [text.splitlines()[0] + for text, _ in digest.group_by_repo(busy + quiet, [], NOW, max_chars=1000)] + self.assertGreater(len(headings), 2) + self.assertEqual(headings[-1], "**quiet**") + self.assertEqual(set(headings[:-1]), {"**busy**"}) + def test_a_block_accounts_for_every_pr_it_shows(self): """What reached Discord is recorded per message, so a partial post cannot suppress the PRs that never went out.""" @@ -313,6 +334,16 @@ def test_the_payload_is_components_v2(self): self.assertEqual(message["flags"], digest.COMPONENTS_V2_FLAG) self.assertEqual(message["components"][0]["type"], digest.CONTAINER) + def test_one_busy_repo_never_exceeds_the_message_budget(self): + new = [pr(n, title="y" * 80) for n in range(120)] + messages, coverage = digest.build_messages(new, [], 120, 72, NOW) + self.assertGreater(len(messages), 1) + for message in messages: + text = sum(len(c.get("content", "")) + for c in message["components"][0]["components"]) + self.assertLessEqual(text, digest.MAX_MESSAGE_TEXT_CHARS) + self.assertEqual(set().union(*coverage), {digest.pr_id(p) for p in new}) + def test_only_the_first_message_carries_the_header(self): new = [pr(n, repo=f"repo-{n}", title="y" * 80) for n in range(40)] messages, coverage = digest.build_messages(new, [], 40, 25, NOW) From 7e2c46e0cc08aee145a15cddea78143d58dacc14 Mon Sep 17 00:00:00 2001 From: Audric Ackermann Date: Thu, 24 Sep 2026 13:01:48 +1000 Subject: [PATCH 003/101] fix: never report private repositories, and stop naming excluded repos in the docs Exclusion is by repository property, so nothing private is named anywhere in the repo. The token guidance drops the option of a `repo` scope with it: the digest posts to Discord, and nothing about a private repository belongs there. --- README.md | 11 ++++++----- deploy/README.md | 4 ++-- github_prs/digest.py | 19 ++++++++++++------- github_prs/test_digest.py | 11 +++++++++-- 4 files changed, 29 insertions(+), 16 deletions(-) diff --git a/README.md b/README.md index 7deedf0..a657c95 100644 --- a/README.md +++ b/README.md @@ -714,10 +714,11 @@ a stranger's until someone adds them. ### What is left out -Forks and archived repositories, by checking the search results against the org's -repository list rather than by name β€” so a repository created today is covered today, -and `session-pysogs` or `session-android-private` never are. `--include-forks`, -`--include-archived` and `--exclude-repo` override that per run. +Forks, archived repositories and private repositories, by checking the search results +against the org's repository list rather than by name β€” so a repository created today +is covered today and a fork of an upstream project never is. `--include-forks`, +`--include-archived` and `--exclude-repo` override the first two per run; private +repositories stay out whatever the token can see. | flag | | | --- | --- | @@ -730,7 +731,7 @@ and `session-pysogs` or `session-android-private` never are. `--include-forks`, | env var | | | --- | --- | -| `GITHUB_PRS_TOKEN` | read-only token; no scope at all is needed for the public repos, `repo` to also see the org's private ones | +| `GITHUB_PRS_TOKEN` | read-only token; no scope at all is needed, the digest reads public repositories only | | `GITHUB_PRS_DISCORD_WEBHOOK_URL` | the channel it posts to (not needed with `--dry-run`) | | `GITHUB_PRS_ORG` | optional; defaults to `session-foundation` | diff --git a/deploy/README.md b/deploy/README.md index 3174827..4dd3608 100644 --- a/deploy/README.md +++ b/deploy/README.md @@ -216,8 +216,8 @@ Mode `640`, `root:ghdigest`, and separate from the Zendesk file rather than merg into it β€” see above. ```sh -# Read-only. No scope at all is needed for the org's public repositories; add `repo` -# to have the digest also see the private ones. Nothing here ever writes to GitHub. +# Read-only, and no scope at all: the digest reads public repositories only. Nothing +# here ever writes to GitHub. GITHUB_PRS_TOKEN= # The channel the digest posts to. A webhook is bound to the channel it was created diff --git a/github_prs/digest.py b/github_prs/digest.py index 559cf20..477ca89 100755 --- a/github_prs/digest.py +++ b/github_prs/digest.py @@ -11,9 +11,9 @@ out until something happens to it. Without it every PR in the window reads as new. Who is a maintainer comes from maintainers.txt, one login per line; bot accounts are -dropped on GitHub's own account type rather than by name. Forks and archived repos -are excluded by checking the search results against the org's repository list, so a -repo created today is covered today. +dropped on GitHub's own account type rather than by name. Forks, archived and private +repos are excluded by checking the search results against the org's repository list, +so a repo created today is covered today. One search fetches every open PR in the org, and the window is applied to the result here rather than in the query. That is what lets the header carry the total open @@ -24,9 +24,8 @@ files, and a shared module would make either one's dependencies the other's. Config (env vars, or flags for local runs): - GITHUB_PRS_TOKEN GitHub token, read-only. Needs no scope at all for the - public repos; add `repo` if the digest should also see - the org's private ones. + GITHUB_PRS_TOKEN GitHub token, read-only. Needs no scope at all: the + digest reads public repositories only. GITHUB_PRS_DISCORD_WEBHOOK_URL Discord incoming webhook for the channel this posts to (not needed with --dry-run) @@ -168,12 +167,18 @@ def fetch_json(session, url, **kwargs): def fetch_repos(session, org, include_forks=False, include_archived=False): - """Names of the org's repositories the digest is willing to report on.""" + """Names of the org's repositories the digest is willing to report on. + + Private repositories are never reported, whatever the token can see: the digest + posts to Discord, and nothing about them belongs there. + """ names, page = set(), 1 while True: batch = fetch_json(session, f"{API}/orgs/{org}/repos", params={"per_page": PER_PAGE, "page": page, "type": "all"}) for repo in batch: + if repo.get("private"): + continue if repo.get("fork") and not include_forks: continue if repo.get("archived") and not include_archived: diff --git a/github_prs/test_digest.py b/github_prs/test_digest.py index c2c550c..dda1ed9 100644 --- a/github_prs/test_digest.py +++ b/github_prs/test_digest.py @@ -367,13 +367,20 @@ def test_an_existing_query_survives(self): class TestFetching(unittest.TestCase): - def test_forks_and_archived_repos_are_left_out(self): + def test_forks_archived_and_private_repos_are_left_out(self): page = [{"name": "session-android", "fork": False, "archived": False}, {"name": "session-pysogs", "fork": True, "archived": False}, - {"name": "session-ios-private", "fork": False, "archived": True}] + {"name": "retired", "fork": False, "archived": True}, + {"name": "internal", "fork": False, "archived": False, "private": True}] with mock.patch.object(digest, "fetch_json", return_value=page): self.assertEqual(digest.fetch_repos(None, "org"), {"session-android"}) + def test_private_repos_cannot_be_asked_for(self): + page = [{"name": "internal", "fork": True, "archived": True, "private": True}] + with mock.patch.object(digest, "fetch_json", return_value=page): + self.assertEqual(digest.fetch_repos(None, "org", include_forks=True, + include_archived=True), set()) + def test_forks_can_be_asked_for(self): page = [{"name": "session-pysogs", "fork": True, "archived": False}] with mock.patch.object(digest, "fetch_json", return_value=page): From ad2aa8d0667f92a10d571fc8888b8deb6100faa9 Mon Sep 17 00:00:00 2001 From: Audric Ackermann Date: Thu, 24 Sep 2026 13:11:01 +1000 Subject: [PATCH 004/101] refactor: drop the flags that widened which repositories the digest reads Forks, archived and private repositories are always left out; there was no run that wanted them back in. --- README.md | 5 ++--- github_prs/digest.py | 19 ++++--------------- github_prs/test_digest.py | 12 ------------ 3 files changed, 6 insertions(+), 30 deletions(-) diff --git a/README.md b/README.md index a657c95..eea595c 100644 --- a/README.md +++ b/README.md @@ -716,9 +716,8 @@ a stranger's until someone adds them. Forks, archived repositories and private repositories, by checking the search results against the org's repository list rather than by name β€” so a repository created today -is covered today and a fork of an upstream project never is. `--include-forks`, -`--include-archived` and `--exclude-repo` override the first two per run; private -repositories stay out whatever the token can see. +is covered today and a fork of an upstream project never is. There is no flag to widen +that: private repositories stay out whatever the token can see. | flag | | | --- | --- | diff --git a/github_prs/digest.py b/github_prs/digest.py index 477ca89..5db6c0d 100755 --- a/github_prs/digest.py +++ b/github_prs/digest.py @@ -166,8 +166,8 @@ def fetch_json(session, url, **kwargs): return resp.json() -def fetch_repos(session, org, include_forks=False, include_archived=False): - """Names of the org's repositories the digest is willing to report on. +def fetch_repos(session, org): + """Names of the org's own, live, public repositories. Private repositories are never reported, whatever the token can see: the digest posts to Discord, and nothing about them belongs there. @@ -177,11 +177,7 @@ def fetch_repos(session, org, include_forks=False, include_archived=False): batch = fetch_json(session, f"{API}/orgs/{org}/repos", params={"per_page": PER_PAGE, "page": page, "type": "all"}) for repo in batch: - if repo.get("private"): - continue - if repo.get("fork") and not include_forks: - continue - if repo.get("archived") and not include_archived: + if repo.get("private") or repo.get("fork") or repo.get("archived"): continue names.add(repo["name"]) if len(batch) < PER_PAGE: @@ -559,12 +555,6 @@ def main(): default=DEFAULT_RETENTION_DAYS, metavar="N", help=f"Drop state entries older than this " f"(default {DEFAULT_RETENTION_DAYS}).") - parser.add_argument("--include-forks", action="store_true", - help="Also report the org's forks of upstream projects.") - parser.add_argument("--include-archived", action="store_true", - help="Also report archived repositories.") - parser.add_argument("--exclude-repo", action="append", default=[], - help="Skip this repository (repeatable).") parser.add_argument("--dry-run", action="store_true", help="Print the Discord payload instead of posting it.") args = parser.parse_args() @@ -579,8 +569,7 @@ def main(): maintainers = load_maintainers(args.maintainers) session = github_session(token) - repos = fetch_repos(session, org, args.include_forks, args.include_archived) - repos -= set(args.exclude_repo) + repos = fetch_repos(session, org) items, truncated = search_open_prs(session, org) prs = contributor_prs(items, repos, maintainers) diff --git a/github_prs/test_digest.py b/github_prs/test_digest.py index dda1ed9..49d2fda 100644 --- a/github_prs/test_digest.py +++ b/github_prs/test_digest.py @@ -375,18 +375,6 @@ def test_forks_archived_and_private_repos_are_left_out(self): with mock.patch.object(digest, "fetch_json", return_value=page): self.assertEqual(digest.fetch_repos(None, "org"), {"session-android"}) - def test_private_repos_cannot_be_asked_for(self): - page = [{"name": "internal", "fork": True, "archived": True, "private": True}] - with mock.patch.object(digest, "fetch_json", return_value=page): - self.assertEqual(digest.fetch_repos(None, "org", include_forks=True, - include_archived=True), set()) - - def test_forks_can_be_asked_for(self): - page = [{"name": "session-pysogs", "fork": True, "archived": False}] - with mock.patch.object(digest, "fetch_json", return_value=page): - self.assertEqual(digest.fetch_repos(None, "org", include_forks=True), - {"session-pysogs"}) - def test_repo_pagination_follows_full_pages(self): pages = [[{"name": f"r{n}", "fork": False, "archived": False} for n in range(digest.PER_PAGE)], From 9b5929941ca98c24471a43cae9b059c89f0120f9 Mon Sep 17 00:00:00 2001 From: Audric Ackermann Date: Thu, 24 Sep 2026 15:46:15 +1000 Subject: [PATCH 005/101] fix: name the CLI's reason when a zero exit still reports is_error The envelope's `result` carries the reason whichever way the CLI exits, and the exit-1 path already reads it. The exit-0 path reported only the subtype and the status code, which does not say what went wrong. --- zendesk_triage/test_triage.py | 11 +++++++++++ zendesk_triage/triage.py | 4 +++- 2 files changed, 14 insertions(+), 1 deletion(-) diff --git a/zendesk_triage/test_triage.py b/zendesk_triage/test_triage.py index 1eb854c..f2dd801 100644 --- a/zendesk_triage/test_triage.py +++ b/zendesk_triage/test_triage.py @@ -1952,6 +1952,17 @@ def test_a_reported_failure_stops_the_run(self): with self.assertRaises(SystemExit): self.run_cli(response=response) + def test_a_zero_exit_that_reports_is_error_names_the_reason(self): + """The envelope's `result` is where the CLI puts the reason, whatever the exit + code; the subtype and status alone do not say what went wrong.""" + with self.assertRaises(SystemExit) as caught: + self.run_cli(response={"subtype": "success", "is_error": True, + "api_error_status": 404, + "result": "There's an issue with the selected model."}) + message = str(caught.exception) + self.assertIn("api_error_status=404", message) + self.assertIn("selected model", message) + def test_a_non_zero_exit_prefers_stderr_over_stdout(self): """A non-zero exit means there is no JSON to read, and the CLI could echo the prompt back β€” which this repo's public run logs must not carry.""" diff --git a/zendesk_triage/triage.py b/zendesk_triage/triage.py index 6edbd7d..b2f8463 100644 --- a/zendesk_triage/triage.py +++ b/zendesk_triage/triage.py @@ -1469,9 +1469,11 @@ def claude_cli_json(model, effort, system_prompt, schema, prompt, timeout, label # is not β€” a successful structured-output run reports "tool_use", because that is # how the schema is enforced underneath. if response.get("is_error") or response.get("subtype") != "success": + detail = cli_failure_detail(done.stdout, done.stderr) sys.exit(f"{CLAUDE_CLI} reported failure on {label} " f"(subtype={response.get('subtype')!r}, " - f"api_error_status={response.get('api_error_status')!r}).") + f"api_error_status={response.get('api_error_status')!r})" + f"{': ' + detail if detail else '.'}") # stop_reason is worth reading for this one value. There is no --max-tokens to # raise, so an answer too long to finish comes back as JSON that stops mid-object, # and the parse below would report a baffling syntax error for something whose From a2ad6c7ebad8e60efc2f5a978705201adf48f68f Mon Sep 17 00:00:00 2001 From: Audric Ackermann Date: Thu, 24 Sep 2026 15:50:10 +1000 Subject: [PATCH 006/101] refactor: move the retry, Discord transport, dedup state and env helpers into shared/ The pull request digest was about to carry its own copy of each; crowdin's report already does. One copy, imported by every script out of the same clone. triage.py keeps get_env, request_with_retry, clip and post_to_discord in its namespace: note_reply, resolve_reviews and the alert reach them there, and the tests patch them there. --- shared/__init__.py | 6 + shared/discord.py | 124 +++++++++++++++ shared/env.py | 13 ++ shared/retry.py | 70 +++++++++ shared/state.py | 73 +++++++++ shared/test_discord.py | 130 ++++++++++++++++ shared/test_retry.py | 153 +++++++++++++++++++ shared/test_state.py | 85 +++++++++++ shared/testing.py | 88 +++++++++++ zendesk_triage/test_triage.py | 274 ++-------------------------------- zendesk_triage/triage.py | 268 +++------------------------------ 11 files changed, 776 insertions(+), 508 deletions(-) create mode 100644 shared/__init__.py create mode 100644 shared/discord.py create mode 100644 shared/env.py create mode 100644 shared/retry.py create mode 100644 shared/state.py create mode 100644 shared/test_discord.py create mode 100644 shared/test_retry.py create mode 100644 shared/test_state.py create mode 100644 shared/testing.py diff --git a/shared/__init__.py b/shared/__init__.py new file mode 100644 index 0000000..cd9e60c --- /dev/null +++ b/shared/__init__.py @@ -0,0 +1,6 @@ +"""Helpers shared by the scripts in this repository. + +A script inserts the repository root on sys.path and imports from here. Every job +runs out of the same clone, so no install step is involved; the only dependency +outside the standard library is `requests`, which every consumer already pins. +""" diff --git a/shared/discord.py b/shared/discord.py new file mode 100644 index 0000000..d9662e1 --- /dev/null +++ b/shared/discord.py @@ -0,0 +1,124 @@ +"""Components V2 messages over a plain incoming webhook. + +A digest is a Container of Text Displays: one block per entry, so a reader skims +lines rather than a wall, and each message records which entries it accounts for. + +Every component here is non-interactive, which is what lets a plain incoming +webhook carry it: Discord allows a webhook that no application owns only those. +Adding an interactive one would need the transport moved to a bot token. +https://docs.discord.com/developers/components/reference +""" +from urllib.parse import parse_qsl, urlencode, urlsplit, urlunsplit + +import requests + +from shared.retry import request_with_retry + +COMPONENTS_V2_FLAG = 1 << 15 +CONTAINER = 17 +TEXT_DISPLAY = 10 +SEPARATOR = 14 +# Discord's ceiling on all the text in one Components V2 message. +MAX_MESSAGE_TEXT_CHARS = 4000 + + +def clip(text, limit): + text = (text or "").strip() + return text if len(text) <= limit else text[: limit - 1] + "…" + + +def text_display(content): + return {"type": TEXT_DISPLAY, "content": content} + + +def separator(): + return {"type": SEPARATOR} + + +def container_message(blocks): + return {"flags": COMPONENTS_V2_FLAG, + "components": [{"type": CONTAINER, "components": blocks}]} + + +def chunk_entries(entries, max_items, max_chars=MAX_MESSAGE_TEXT_CHARS, first_used=0): + """Group (text, ids) pairs into messages within Discord's budgets. + + Whichever limit binds first splits the message. Entries are separate components + rather than joined text, so nothing is spent on the newlines between them. + + `first_used` is what the caller has already spent on the first message before + any entry goes in: the header. Without it the header rides on top of a full + budget of entries and a busy day's first message goes over. + + An entry longer than the character budget still gets its own message rather + than being dropped; callers pre-clip so that should not arise. + + Entries are passed through, not rebuilt, so a caller can still find one by + identity inside a chunk. + """ + chunks, current, current_chars = [], [], first_used + for entry in entries: + text, _ = entry + if current and (len(current) >= max_items + or current_chars + len(text) > max_chars): + chunks.append(current) + current, current_chars = [], 0 + current.append(entry) + current_chars += len(text) + if current: + chunks.append(current) + return chunks + + +def messages_from_entries(header, entries, max_items, max_chars=MAX_MESSAGE_TEXT_CHARS): + """Return (messages, coverage): the header leads message one, entries follow. + + coverage[i] is the union of the ids message i carries, so a run that fails + partway can record exactly what reached Discord. Message one's header is not + credited with any ids; a caller whose header accounts for entries adds them. + + A quiet day still owes the channel its header, so no entries yields one + message. + """ + messages, coverage = [], [] + chunks = chunk_entries(entries, max_items, max_chars, first_used=len(header)) or [[]] + for index, chunk in enumerate(chunks): + blocks = [] + if index == 0: + blocks.append(text_display(header)) + if chunk: + blocks.append(separator()) + blocks += [text_display(text) for text, _ in chunk] + messages.append(container_message(blocks)) + coverage.append(set().union(*(ids for _, ids in chunk)) if chunk else set()) + return messages, coverage + + +def components_webhook_url(webhook_url): + """The webhook, told to respect the components field, which it ignores without.""" + parts = urlsplit(webhook_url) + query = dict(parse_qsl(parts.query)) + query["with_components"] = "true" + return urlunsplit(parts._replace(query=urlencode(query))) + + +def post_to_discord(session, url, messages): + """POST each message in order; return how many Discord accepted. + + Stops at the first failure and returns the count instead of exiting, so the + caller can record what did land before signalling the failure. Letting the + exhausted retry propagate would skip that recording, and the messages that + landed would be reposted on the next run. + """ + for index, payload in enumerate(messages): + try: + resp = request_with_retry(session, "POST", url, json=payload) + except requests.RequestException as exc: + print(f"Discord unreachable on message {index + 1}/{len(messages)} " + f"({exc}).") + return index + if resp.status_code >= 400: + print(f"Discord rejected message {index + 1}/{len(messages)} " + f"({resp.status_code}): {resp.text[:300]}") + return index + return len(messages) diff --git a/shared/env.py b/shared/env.py new file mode 100644 index 0000000..4d6fcbf --- /dev/null +++ b/shared/env.py @@ -0,0 +1,13 @@ +import os +import sys + + +def get_env(name, cli_value=None, required=True): + if cli_value: + return cli_value + value = os.environ.get(name) + if value: + return value + if required: + sys.exit(f"Missing required config: set the {name} environment variable (or pass the matching flag).") + return None diff --git a/shared/retry.py b/shared/retry.py new file mode 100644 index 0000000..981d075 --- /dev/null +++ b/shared/retry.py @@ -0,0 +1,70 @@ +import math +import time + +import requests + + +def retry_after_seconds(resp, default): + """Seconds to wait per the response's rate-limit headers, else `default`. + + Retry-After first. GitHub answers a primary rate limit with x-ratelimit-reset + as an epoch second and no Retry-After at all, so that is read when the header + is absent. + + RFC 9110 allows Retry-After to be an HTTP-date; float() on the date form + raises, so anything unparseable falls back rather than crashing the run. + Negative, NaN and infinite values fall back too: time.sleep() rejects the first + two outright, so a hostile or buggy proxy sending `Retry-After: -30` would + otherwise take the run down with a ValueError. + """ + raw = resp.headers.get("retry-after") + if raw is None: + reset = resp.headers.get("x-ratelimit-reset") + if reset is None: + return default + try: + return max(0.0, float(reset) - time.time()) + except (TypeError, ValueError): + return default + try: + seconds = float(raw) + except (TypeError, ValueError): + return default + if not math.isfinite(seconds) or seconds < 0: + return default + return seconds + + +def request_with_retry(session, method, url, attempts=6, **kwargs): + """GET/POST with backoff on 429 and 5xx. + + Lower `attempts` for calls whose result is nice-to-have: the full budget can + burn ~60s of backoff, which is not worth spending on optional data. + """ + if attempts <= 0: + raise ValueError("attempts must be at least 1") + + delay = 1.0 + last_exc = None + resp = None + for attempt in range(attempts): + final = attempt == attempts - 1 + try: + resp = session.request(method, url, timeout=30, **kwargs) + except requests.RequestException as exc: + last_exc = exc + if final: + break + time.sleep(delay) + delay = min(delay * 2, 30) + continue + if resp.status_code == 429 or resp.status_code >= 500: + if final: + break + time.sleep(min(retry_after_seconds(resp, delay), 60)) + delay = min(delay * 2, 30) + continue + return resp + if last_exc: + raise last_exc + return resp diff --git a/shared/state.py b/shared/state.py new file mode 100644 index 0000000..90a8bb5 --- /dev/null +++ b/shared/state.py @@ -0,0 +1,73 @@ +"""The dedup state a digest keeps between runs: what it reported, and when. + +Every failure to read it is a cache miss rather than an error. Losing the file +re-reports one window, which is noisy and never wrong, and that is what makes it +a cache rather than something to back up. +""" +import json +import os +from datetime import datetime, timedelta, timezone + +STAMP = "%Y-%m-%dT%H:%M:%SZ" + + +def empty_state(version): + return {"version": version, "seen": {}} + + +def load_state(path, version, noun): + """The state at `path`, or an empty one for any file that cannot be trusted. + + `noun` names what the caller dedups, for the log line. + """ + if not path: + return empty_state(version) + if not os.path.exists(path): + print(f"No state file at {path}; treating every {noun} in the window as new.") + return empty_state(version) + try: + with open(path, encoding="utf-8") as handle: + data = json.load(handle) + except (OSError, json.JSONDecodeError) as exc: + print(f"Note: unreadable state file {path} ({exc}); treating every {noun} as new.") + return empty_state(version) + if not isinstance(data, dict) or not isinstance(data.get("seen"), dict): + print(f"Note: unexpected shape in {path}; treating every {noun} as new.") + return empty_state(version) + # A file written by another schema version cannot be trusted field by field. + if data.get("version") != version: + print(f"Note: {path} is version {data.get('version')!r}, expected {version}; " + f"treating every {noun} as new.") + return empty_state(version) + print(f"Loaded state for {len(data['seen'])} previously reported {noun}s.") + return data + + +def save_state(path, state, records, retention_days, version): + """Merge `records` ({key: fields}) into the state as reported now, prune entries + older than `retention_days`, write atomically. Returns (kept, pruned).""" + now = datetime.now(timezone.utc) + stamp = now.strftime(STAMP) + seen = dict(state.get("seen", {})) + for key, fields in records.items(): + seen[key] = {**fields, "last_reported": stamp} + + cutoff = now - timedelta(days=retention_days) + kept = {} + for key, record in seen.items(): + try: + last = datetime.strptime(record.get("last_reported", ""), STAMP) \ + .replace(tzinfo=timezone.utc) + except (TypeError, ValueError): + continue # malformed entry: drop it rather than keep it forever + if last >= cutoff: + kept[key] = record + + directory = os.path.dirname(path) + if directory: + os.makedirs(directory, exist_ok=True) + temporary = f"{path}.tmp" + with open(temporary, "w", encoding="utf-8") as handle: + json.dump({"version": version, "updated_at": stamp, "seen": kept}, handle, indent=2) + os.replace(temporary, path) # atomic: a crash mid-write cannot corrupt the state + return len(kept), len(seen) - len(kept) diff --git a/shared/test_discord.py b/shared/test_discord.py new file mode 100644 index 0000000..41330c4 --- /dev/null +++ b/shared/test_discord.py @@ -0,0 +1,130 @@ +import contextlib +import io +import os +import sys +import unittest +from urllib.parse import parse_qsl, urlsplit + +sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__)))) +from shared import discord # noqa: E402 +from shared.testing import FakeResponse, FakeSession, NoSleep # noqa: E402 + + +class TestComponentsWebhookUrl(unittest.TestCase): + """Discord ignores `components` on a webhook post without this param, and a + digest is nothing but components.""" + + def test_adds_the_param(self): + self.assertEqual( + discord.components_webhook_url("https://discord.com/api/webhooks/1/tok"), + "https://discord.com/api/webhooks/1/tok?with_components=true") + + def test_keeps_an_existing_query(self): + url = discord.components_webhook_url( + "https://discord.com/api/webhooks/1/tok?thread_id=42") + self.assertEqual(dict(parse_qsl(urlsplit(url).query)), + {"thread_id": "42", "with_components": "true"}) + + def test_does_not_duplicate_the_param(self): + once = discord.components_webhook_url("https://discord.com/api/webhooks/1/tok") + self.assertEqual(discord.components_webhook_url(once), once) + + +class TestPostToDiscord(unittest.TestCase): + """Returns the accepted count rather than exiting, so a caller can record exactly + what landed before signalling the failure.""" + + def post(self, session, messages): + with contextlib.redirect_stdout(io.StringIO()): + return discord.post_to_discord(session, "https://hook", messages) + + def test_all_accepted(self): + session = FakeSession([FakeResponse({}, status_code=204)] * 3) + self.assertEqual(self.post(session, [{}, {}, {}]), 3) + self.assertEqual(len(session.calls), 3) + + def test_stops_at_the_first_failure_and_reports_the_prefix(self): + session = FakeSession([ + FakeResponse({}, status_code=204), + FakeResponse({}, status_code=400), + ]) + self.assertEqual(self.post(session, [{}, {}, {}]), 1) + + def test_does_not_post_after_a_failure(self): + session = FakeSession([FakeResponse({}, status_code=404)]) + self.post(session, [{}, {}, {}]) + self.assertEqual(len(session.calls), 1) + + def test_first_message_failing_reports_zero(self): + session = FakeSession([FakeResponse({}, status_code=500)] * 6) + with NoSleep(): + self.assertEqual(self.post(session, [{}]), 0) + + def test_an_unreachable_webhook_reports_the_prefix_rather_than_raising(self): + import requests + session = FakeSession([FakeResponse({}, status_code=204)] + + [requests.ConnectionError("down")] * 6) + with NoSleep(): + self.assertEqual(self.post(session, [{}, {}]), 1) + + def test_no_messages_is_zero(self): + self.assertEqual(self.post(FakeSession([]), []), 0) + + +class TestChunkEntries(unittest.TestCase): + def test_splits_on_the_entry_count(self): + entries = [("x", {i}) for i in range(11)] + self.assertEqual([len(c) for c in discord.chunk_entries(entries, 10)], [10, 1]) + + def test_splits_on_the_character_budget(self): + entries = [("x" * 2500, {1}), ("y" * 2500, {2})] + self.assertEqual(len(discord.chunk_entries(entries, 10)), 2) + + def test_the_header_is_charged_to_the_first_message_only(self): + entries = [("x" * 1900, {1}), ("x" * 1900, {2})] + self.assertEqual(len(discord.chunk_entries(entries, 10, first_used=0)), 1) + self.assertEqual(len(discord.chunk_entries(entries, 10, first_used=1000)), 2) + + def test_an_oversized_entry_still_gets_a_message(self): + entries = [("x" * (discord.MAX_MESSAGE_TEXT_CHARS + 100), {1})] + self.assertEqual(len(discord.chunk_entries(entries, 10)), 1) + + def test_entries_come_back_by_identity(self): + entry = ("x", {1}) + self.assertIs(discord.chunk_entries([entry], 10)[0][0], entry) + + +class TestMessagesFromEntries(unittest.TestCase): + def texts(self, message): + return [c["content"] for c in message["components"][0]["components"] + if c["type"] == discord.TEXT_DISPLAY] + + def test_no_entries_still_yields_the_header(self): + messages, coverage = discord.messages_from_entries("header", [], 10) + self.assertEqual(len(messages), 1) + self.assertEqual(self.texts(messages[0]), ["header"]) + self.assertEqual(coverage, [set()]) + self.assertEqual(len(messages[0]["components"][0]["components"]), 1) # no separator + + def test_the_payload_is_a_components_v2_container(self): + message = discord.messages_from_entries("h", [("a", {1})], 10)[0][0] + self.assertEqual(message["flags"], discord.COMPONENTS_V2_FLAG) + self.assertEqual(message["components"][0]["type"], discord.CONTAINER) + types = [c["type"] for c in message["components"][0]["components"]] + self.assertEqual(types, [discord.TEXT_DISPLAY, discord.SEPARATOR, discord.TEXT_DISPLAY]) + + def test_only_the_first_message_carries_the_header(self): + entries = [(f"e{i}", {i}) for i in range(3)] + messages, coverage = discord.messages_from_entries("header", entries, 2) + self.assertEqual(self.texts(messages[0]), ["header", "e0", "e1"]) + self.assertEqual(self.texts(messages[1]), ["e2"]) + self.assertEqual(coverage, [{0, 1}, {2}]) + + def test_the_header_counts_against_the_first_budget(self): + entries = [("x" * 2000, {1}), ("y" * 1500, {2})] + messages, _ = discord.messages_from_entries("h" * 1000, entries, 10) + self.assertEqual(len(messages), 2) + + +if __name__ == "__main__": + unittest.main() diff --git a/shared/test_retry.py b/shared/test_retry.py new file mode 100644 index 0000000..fef3ec6 --- /dev/null +++ b/shared/test_retry.py @@ -0,0 +1,153 @@ +""" + cd shared && python -m unittest discover +""" +import os +import sys +import time +import unittest + +import requests + +sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__)))) +from shared import retry # noqa: E402 +from shared.testing import FakeResponse, FakeSession, NoSleep # noqa: E402 + + +class TestRequestWithRetry(unittest.TestCase): + def test_returns_the_first_success_without_retrying(self): + session = FakeSession([FakeResponse({"ok": True})]) + resp = retry.request_with_retry(session, "GET", "https://x") + self.assertEqual(resp.json(), {"ok": True}) + self.assertEqual(len(session.calls), 1) + + def test_retries_a_server_error_then_succeeds(self): + session = FakeSession([ + FakeResponse({}, status_code=500), + FakeResponse({"ok": True}), + ]) + resp = retry.request_with_retry(session, "GET", "https://x", attempts=3) + self.assertEqual(resp.json(), {"ok": True}) + self.assertEqual(len(session.calls), 2) + + def test_does_not_retry_a_client_error(self): + session = FakeSession([FakeResponse({}, status_code=404)]) + resp = retry.request_with_retry(session, "GET", "https://x") + self.assertEqual(resp.status_code, 404) + self.assertEqual(len(session.calls), 1) + + def test_gives_up_after_the_attempt_budget(self): + session = FakeSession([FakeResponse({}, status_code=503)] * 3) + resp = retry.request_with_retry(session, "GET", "https://x", attempts=3) + self.assertEqual(resp.status_code, 503) + self.assertEqual(len(session.calls), 3) + + def test_transport_failures_retry_then_raise_when_exhausted(self): + session = FakeSession([requests.ConnectionError("boom")] * 3) + with NoSleep(): + with self.assertRaises(requests.ConnectionError): + retry.request_with_retry(session, "GET", "https://x", attempts=3) + self.assertEqual(len(session.calls), 3) + + def test_a_transport_failure_can_recover_on_a_later_attempt(self): + session = FakeSession([requests.Timeout("slow"), FakeResponse({"ok": True})]) + with NoSleep(): + resp = retry.request_with_retry(session, "GET", "https://x", attempts=3) + self.assertEqual(resp.json(), {"ok": True}) + self.assertEqual(len(session.calls), 2) + + def test_no_sleep_after_the_final_attempt(self): + """Sleeping after the last try only delays the caller β€” nothing follows it.""" + session = FakeSession([FakeResponse({}, status_code=503)] * 3) + with NoSleep() as clock: + retry.request_with_retry(session, "GET", "https://x", attempts=3) + self.assertEqual(len(clock.slept), 2) # 3 attempts, 2 gaps + + def test_numeric_retry_after_is_honoured(self): + session = FakeSession([ + FakeResponse({}, status_code=429, retry_after="7"), + FakeResponse({"ok": True}), + ]) + with NoSleep() as clock: + retry.request_with_retry(session, "GET", "https://x", attempts=3) + self.assertEqual(clock.slept, [7.0]) + + def test_retry_after_is_capped(self): + session = FakeSession([ + FakeResponse({}, status_code=429, retry_after="9999"), + FakeResponse({"ok": True}), + ]) + with NoSleep() as clock: + retry.request_with_retry(session, "GET", "https://x", attempts=3) + self.assertEqual(clock.slept, [60]) + + def test_http_date_retry_after_falls_back_instead_of_crashing(self): + """RFC 9110 allows an HTTP-date here; float() on it used to raise ValueError.""" + session = FakeSession([ + FakeResponse({}, status_code=503, retry_after="Wed, 21 Oct 2026 07:28:00 GMT"), + FakeResponse({"ok": True}), + ]) + with NoSleep() as clock: + resp = retry.request_with_retry(session, "GET", "https://x", attempts=3) + self.assertEqual(resp.json(), {"ok": True}) + self.assertEqual(clock.slept, [1.0]) # fell back to the backoff delay + + def test_zero_attempts_is_rejected_rather_than_unbound(self): + with self.assertRaises(ValueError): + retry.request_with_retry(FakeSession([]), "GET", "https://x", attempts=0) + + def test_a_negative_retry_after_does_not_crash_a_real_retry_loop(self): + session = FakeSession([ + FakeResponse({}, status_code=503, retry_after="-30"), + FakeResponse({"ok": True}), + ]) + with NoSleep() as clock: + resp = retry.request_with_retry(session, "GET", "https://x", attempts=3) + self.assertEqual(resp.json(), {"ok": True}) + self.assertTrue(all(s >= 0 for s in clock.slept), clock.slept) + + +class TestRetryAfterSeconds(unittest.TestCase): + def seconds(self, headers, default=4.0): + resp = FakeResponse({}, retry_after=None) + resp.headers = headers + return retry.retry_after_seconds(resp, default) + + def test_missing_header_uses_the_default(self): + self.assertEqual(self.seconds({}), 4.0) + + def test_numeric_header_wins(self): + self.assertEqual(self.seconds({"retry-after": "12"}), 12.0) + + def test_unparseable_header_uses_the_default(self): + for raw in ("Wed, 21 Oct 2026 07:28:00 GMT", "", "soon", "12s"): + self.assertEqual(self.seconds({"retry-after": raw}), 4.0) + + def test_negative_and_non_finite_values_use_the_default(self): + """time.sleep() rejects a negative or NaN duration, so passing one through + would crash the run on a hostile or buggy Retry-After header.""" + for raw in ("-30", "-0.5", "nan", "inf", "-inf"): + self.assertEqual(self.seconds({"retry-after": raw}), 4.0, msg=f"retry-after={raw!r}") + + def test_zero_is_honoured_rather_than_replaced(self): + """Zero is a valid instruction to retry immediately, not a missing value.""" + self.assertEqual(self.seconds({"retry-after": "0"}), 0.0) + + def test_a_primary_rate_limit_falls_back_to_the_reset_epoch(self): + """GitHub's primary limit sends x-ratelimit-reset and no Retry-After.""" + reset = str(int(time.time()) + 30) + seconds = self.seconds({"x-ratelimit-reset": reset}) + self.assertTrue(25 <= seconds <= 31, seconds) + + def test_a_reset_epoch_in_the_past_means_no_wait(self): + self.assertEqual(self.seconds({"x-ratelimit-reset": "1"}), 0.0) + + def test_retry_after_beats_the_reset_epoch(self): + reset = str(int(time.time()) + 3000) + self.assertEqual(self.seconds({"retry-after": "5", "x-ratelimit-reset": reset}), 5.0) + + def test_an_unparseable_reset_epoch_uses_the_default(self): + self.assertEqual(self.seconds({"x-ratelimit-reset": "?"}), 4.0) + + +if __name__ == "__main__": + unittest.main() diff --git a/shared/test_state.py b/shared/test_state.py new file mode 100644 index 0000000..1097085 --- /dev/null +++ b/shared/test_state.py @@ -0,0 +1,85 @@ +import contextlib +import io +import json +import os +import sys +import tempfile +import unittest + +sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__)))) +from shared import state as dedup # noqa: E402 + +VERSION = 7 + + +class TestStateFile(unittest.TestCase): + def setUp(self): + self.directory = tempfile.TemporaryDirectory() + self.addCleanup(self.directory.cleanup) + self.path = os.path.join(self.directory.name, "sub", "seen.json") + + def load(self, path=None): + with contextlib.redirect_stdout(io.StringIO()): + return dedup.load_state(path or self.path, VERSION, "thing") + + def save(self, records, state=None, retention_days=30): + with contextlib.redirect_stdout(io.StringIO()): + return dedup.save_state(self.path, state or dedup.empty_state(VERSION), + records, retention_days, VERSION) + + def test_a_record_comes_back_with_its_fields_and_a_stamp(self): + self.save({"1": {"updated_at": "A"}}) + record = self.load()["seen"]["1"] + self.assertEqual(record["updated_at"], "A") + self.assertIn("last_reported", record) + + def test_a_later_save_replaces_the_record(self): + self.save({"1": {"updated_at": "A"}}) + self.save({"1": {"updated_at": "B"}}, state=self.load()) + self.assertEqual(self.load()["seen"]["1"]["updated_at"], "B") + + def test_no_path_means_no_state_and_no_complaint(self): + self.assertEqual(dedup.load_state(None, VERSION, "thing"), dedup.empty_state(VERSION)) + + def test_a_missing_file_is_a_cache_miss(self): + self.assertEqual(self.load(), dedup.empty_state(VERSION)) + + def test_every_unreadable_state_is_a_cache_miss_not_an_error(self): + """Losing it re-reports the window once. Failing the run instead would mean a + corrupt cache stops the digest entirely.""" + path = os.path.join(self.directory.name, "flat.json") + for content in ("{ not json", '{"version": %d}' % VERSION, "[]", + '{"version": 99, "seen": {}}', '{"seen": []}'): + with self.subTest(content=content): + with open(path, "w", encoding="utf-8") as handle: + handle.write(content) + self.assertEqual(self.load(path), dedup.empty_state(VERSION)) + + def test_the_right_version_is_loaded(self): + path = os.path.join(self.directory.name, "flat.json") + with open(path, "w", encoding="utf-8") as handle: + json.dump({"version": VERSION, "seen": {"1": {"updated_at": "A"}}}, handle) + self.assertEqual(list(self.load(path)["seen"]), ["1"]) + + def test_entries_are_pruned_past_the_retention(self): + self.save({"1": {}}) + state = self.load() + for record in state["seen"].values(): + record["last_reported"] = "2026-01-01T00:00:00Z" + self.assertEqual(self.save({}, state=state), (0, 1)) + + def test_a_fresh_entry_is_kept(self): + self.assertEqual(self.save({"1": {}, "2": {}}), (2, 0)) + + def test_a_malformed_entry_is_dropped_rather_than_kept_forever(self): + state = {"version": VERSION, "seen": {"1": {"last_reported": "never"}}} + kept, _ = self.save({}, state=state) + self.assertEqual(kept, 0) + + def test_the_directory_is_created_and_the_write_is_atomic(self): + self.save({"1": {}}) + self.assertEqual(sorted(os.listdir(os.path.dirname(self.path))), ["seen.json"]) + + +if __name__ == "__main__": + unittest.main() diff --git a/shared/testing.py b/shared/testing.py new file mode 100644 index 0000000..c5e3677 --- /dev/null +++ b/shared/testing.py @@ -0,0 +1,88 @@ +"""Fakes for the tests of every script that talks HTTP through `shared`.""" +import json +import time + +import requests + + +class FakeResponse: + # retry-after: 0 keeps the retry tests instant instead of sleeping through + # the real backoff, and exercises the header-honoring path while it's at it. + def __init__(self, payload, status_code=200, retry_after="0"): + self._payload = payload + self.status_code = status_code + self.headers = {"retry-after": retry_after} + self.text = json.dumps(payload) + + def json(self): + return self._payload + + +class NonJsonResponse(FakeResponse): + """A 200 whose body isn't JSON β€” a proxy error page, say.""" + + def __init__(self): + super().__init__({}) + self.text = "maintenance" + + def json(self): + raise requests.exceptions.JSONDecodeError("Expecting value", self.text, 0) + + +class FakeSession: + """Returns queued responses in order and records the requests made. + + A queued Exception is raised instead of returned, so transport failures can be + exercised alongside HTTP status codes. + """ + + def __init__(self, responses): + self._responses = list(responses) + self.calls = [] + + def request(self, method, url, **kwargs): + self.calls.append((method, url, kwargs)) + item = self._responses.pop(0) + if isinstance(item, Exception): + raise item + return item + + +class Patched: + """Swap module attributes for the duration of a block, then put them back.""" + + def __init__(self, module, **attrs): + self.module, self.attrs, self.saved = module, attrs, {} + + def __enter__(self): + for name, value in self.attrs.items(): + try: + self.saved[name] = getattr(self.module, name) + except AttributeError: + # Roll back what is already swapped. Without this, a typo'd or + # since-removed attribute leaves earlier patches applied and + # __exit__ never runs β€” every later test in the file then fails + # against a module the failing test quietly rewrote. + self.__exit__() + raise + setattr(self.module, name, value) + return self + + def __exit__(self, *exc): + for name, value in self.saved.items(): + setattr(self.module, name, value) + return False + + +class NoSleep: + """Patch out time.sleep so retry tests assert on delays without waiting.""" + + def __enter__(self): + self.slept = [] + self._real = time.sleep + time.sleep = self.slept.append + return self + + def __exit__(self, *exc): + time.sleep = self._real + return False diff --git a/zendesk_triage/test_triage.py b/zendesk_triage/test_triage.py index f2dd801..f9e3151 100644 --- a/zendesk_triage/test_triage.py +++ b/zendesk_triage/test_triage.py @@ -18,12 +18,15 @@ import tempfile import unittest from datetime import datetime, timedelta, timezone -from urllib.parse import parse_qsl, urlsplit import requests sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))) +sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__)))) import triage # noqa: E402 (needs the path insert above) +from shared import discord # noqa: E402 +from shared.testing import ( # noqa: E402 + FakeResponse, FakeSession, NoSleep, NonJsonResponse, Patched) STAMP = "%Y-%m-%dT%H:%M:%SZ" @@ -74,7 +77,7 @@ def text_displays(node): return [t for item in node for t in text_displays(item)] if not isinstance(node, dict): return [] - found = [node["content"]] if node.get("type") == triage.TEXT_DISPLAY else [] + found = [node["content"]] if node.get("type") == discord.TEXT_DISPLAY else [] for key in ("components", "accessory"): found += text_displays(node.get(key)) return found @@ -99,94 +102,6 @@ def all_text(message): return sum(len(t) for t in text_displays(message["components"])) -class FakeResponse: - # retry-after: 0 keeps the retry tests instant instead of sleeping through - # the real backoff, and exercises the header-honoring path while it's at it. - def __init__(self, payload, status_code=200, retry_after="0"): - self._payload = payload - self.status_code = status_code - self.headers = {"retry-after": retry_after} - self.text = json.dumps(payload) - - def json(self): - return self._payload - - -class NonJsonResponse(FakeResponse): - """A 200 whose body isn't JSON β€” a proxy error page, say.""" - - def __init__(self): - super().__init__({}) - self.text = "maintenance" - - def json(self): - raise requests.exceptions.JSONDecodeError("Expecting value", self.text, 0) - - -class FakeSession: - """Returns queued responses in order and records the requests made. - - A queued Exception is raised instead of returned, so transport failures can be - exercised alongside HTTP status codes. - """ - - def __init__(self, responses): - self._responses = list(responses) - self.calls = [] - - def request(self, method, url, **kwargs): - self.calls.append((method, url, kwargs)) - item = self._responses.pop(0) - if isinstance(item, Exception): - raise item - return item - - -class Patched: - """Swap module attributes for the duration of a block, then put them back. - - Lives here rather than in each test file because test_relay and test_reply both - need it, and two copies of a helper that restores state is two chances for one of - them to stop doing so. - """ - - def __init__(self, module, **attrs): - self.module, self.attrs, self.saved = module, attrs, {} - - def __enter__(self): - for name, value in self.attrs.items(): - try: - self.saved[name] = getattr(self.module, name) - except AttributeError: - # Roll back what is already swapped. Without this, a typo'd or - # since-removed attribute leaves earlier patches applied and - # __exit__ never runs β€” every later test in the file then fails - # against a module the failing test quietly rewrote. - self.__exit__() - raise - setattr(self.module, name, value) - return self - - def __exit__(self, *exc): - for name, value in self.saved.items(): - setattr(self.module, name, value) - return False - - -class NoSleep: - """Patch out time.sleep so retry tests assert on delays without waiting.""" - - def __enter__(self): - self.slept = [] - self._real = triage.time.sleep - triage.time.sleep = self.slept.append - return self - - def __exit__(self, *exc): - triage.time.sleep = self._real - return False - - # ---- Window construction --------------------------------------------------- @@ -687,10 +602,10 @@ def test_messages_are_components_v2(self): """content and embeds stop working once the flag is set, so a payload still carrying either would be silently rendered empty.""" for message in build_messages([finding(1)], "acme"): - self.assertEqual(message["flags"], triage.COMPONENTS_V2_FLAG) + self.assertEqual(message["flags"], discord.COMPONENTS_V2_FLAG) self.assertNotIn("content", message) self.assertNotIn("embeds", message) - self.assertEqual(message["components"][0]["type"], triage.CONTAINER) + self.assertEqual(message["components"][0]["type"], discord.CONTAINER) def test_each_ticket_gets_its_own_block(self): """One Text Display per ticket, so a reader skims lines rather than a wall.""" @@ -1178,7 +1093,7 @@ def test_a_full_message_stays_inside_the_character_ceiling(self): component rather than three.""" findings = [self.fat(i) for i in range(triage.MAX_HIGHLIGHTS)] for message in build_messages(findings, "acme"): - self.assertLessEqual(all_text(message), triage.MAX_MESSAGE_TEXT_CHARS) + self.assertLessEqual(all_text(message), discord.MAX_MESSAGE_TEXT_CHARS) def test_the_card_count_splits_a_busy_day(self): """Short lines never reach the character budget, so without the component @@ -1196,12 +1111,13 @@ def test_chunking_splits_on_whichever_limit_binds_first(self): # Together these overrun MAX_COMPONENT_CHARS, so characters bind before the # entry count does. entries = [("x" * 2100, {1}), ("y" * 2100, {2})] - self.assertEqual(len(triage.chunk_entries(entries)), 2) # characters + self.assertEqual(len(discord.chunk_entries(entries, triage.MAX_ENTRIES_PER_MESSAGE)), 2) # characters lean = [("x", {i}) for i in range(triage.MAX_ENTRIES_PER_MESSAGE + 1)] - self.assertEqual(len(triage.chunk_entries(lean)), 2) # card count + self.assertEqual(len(discord.chunk_entries(lean, triage.MAX_ENTRIES_PER_MESSAGE)), 2) # card count def test_an_oversized_entry_still_gets_a_message(self): - chunks = triage.chunk_entries([("x" * (triage.MAX_COMPONENT_CHARS + 50), {1})]) + chunks = discord.chunk_entries([("x" * (triage.MAX_COMPONENT_CHARS + 50), {1})], + triage.MAX_ENTRIES_PER_MESSAGE) self.assertEqual(len(chunks), 1) def test_a_full_digest_of_fat_lines_and_abuse_reports_stays_within_the_limit(self): @@ -1284,56 +1200,6 @@ def test_abuse_reports_dropped_from_the_link_list_are_still_covered(self): self.assertIn("more", digest_text(messages)) -class TestDigestWebhookUrl(unittest.TestCase): - """Discord ignores `components` on a webhook post without this param, and the - digest is nothing but components.""" - - def test_adds_the_param(self): - self.assertEqual( - triage.digest_webhook_url("https://discord.com/api/webhooks/1/tok"), - "https://discord.com/api/webhooks/1/tok?with_components=true") - - def test_keeps_an_existing_query(self): - url = triage.digest_webhook_url( - "https://discord.com/api/webhooks/1/tok?thread_id=42") - self.assertEqual(dict(parse_qsl(urlsplit(url).query)), - {"thread_id": "42", "with_components": "true"}) - - def test_does_not_duplicate_the_param(self): - once = triage.digest_webhook_url("https://discord.com/api/webhooks/1/tok") - self.assertEqual(triage.digest_webhook_url(once), once) - - -class TestPostToDiscord(unittest.TestCase): - """Returns the accepted count rather than exiting, so main can record exactly the - tickets that landed before signalling the failure.""" - - def test_all_accepted(self): - session = FakeSession([FakeResponse({}, status_code=204)] * 3) - self.assertEqual(triage.post_to_discord(session, "https://hook", [{}, {}, {}]), 3) - self.assertEqual(len(session.calls), 3) - - def test_stops_at_the_first_failure_and_reports_the_prefix(self): - session = FakeSession([ - FakeResponse({}, status_code=204), - FakeResponse({}, status_code=400), - ]) - self.assertEqual(triage.post_to_discord(session, "https://hook", [{}, {}, {}]), 1) - - def test_does_not_post_after_a_failure(self): - session = FakeSession([FakeResponse({}, status_code=404)]) - triage.post_to_discord(session, "https://hook", [{}, {}, {}]) - self.assertEqual(len(session.calls), 1) - - def test_first_message_failing_reports_zero(self): - session = FakeSession([FakeResponse({}, status_code=500)] * 6) - with NoSleep(): - self.assertEqual(triage.post_to_discord(session, "https://hook", [{}]), 0) - - def test_no_messages_is_zero(self): - self.assertEqual(triage.post_to_discord(FakeSession([]), "https://hook", []), 0) - - class TestSelectHighlights(unittest.TestCase): def test_splits_at_the_display_cap(self): findings = [finding(i, priority_rank=i) for i in range(triage.MAX_HIGHLIGHTS + 4)] @@ -1702,122 +1568,6 @@ def test_a_non_json_body_is_non_fatal(self): self.assertIsNone(triage.fetch_total_unsolved(session, "acme")) -class TestRequestWithRetry(unittest.TestCase): - def test_returns_the_first_success_without_retrying(self): - session = FakeSession([FakeResponse({"ok": True})]) - resp = triage.request_with_retry(session, "GET", "https://x") - self.assertEqual(resp.json(), {"ok": True}) - self.assertEqual(len(session.calls), 1) - - def test_retries_a_server_error_then_succeeds(self): - session = FakeSession([ - FakeResponse({}, status_code=500), - FakeResponse({"ok": True}), - ]) - resp = triage.request_with_retry(session, "GET", "https://x", attempts=3) - self.assertEqual(resp.json(), {"ok": True}) - self.assertEqual(len(session.calls), 2) - - def test_does_not_retry_a_client_error(self): - session = FakeSession([FakeResponse({}, status_code=404)]) - resp = triage.request_with_retry(session, "GET", "https://x") - self.assertEqual(resp.status_code, 404) - self.assertEqual(len(session.calls), 1) - - def test_gives_up_after_the_attempt_budget(self): - session = FakeSession([FakeResponse({}, status_code=503)] * 3) - resp = triage.request_with_retry(session, "GET", "https://x", attempts=3) - self.assertEqual(resp.status_code, 503) - self.assertEqual(len(session.calls), 3) - - def test_transport_failures_retry_then_raise_when_exhausted(self): - session = FakeSession([requests.ConnectionError("boom")] * 3) - with NoSleep(): - with self.assertRaises(requests.ConnectionError): - triage.request_with_retry(session, "GET", "https://x", attempts=3) - self.assertEqual(len(session.calls), 3) - - def test_a_transport_failure_can_recover_on_a_later_attempt(self): - session = FakeSession([requests.Timeout("slow"), FakeResponse({"ok": True})]) - with NoSleep(): - resp = triage.request_with_retry(session, "GET", "https://x", attempts=3) - self.assertEqual(resp.json(), {"ok": True}) - self.assertEqual(len(session.calls), 2) - - def test_no_sleep_after_the_final_attempt(self): - """Sleeping after the last try only delays the caller β€” nothing follows it.""" - session = FakeSession([FakeResponse({}, status_code=503)] * 3) - with NoSleep() as clock: - triage.request_with_retry(session, "GET", "https://x", attempts=3) - self.assertEqual(len(clock.slept), 2) # 3 attempts, 2 gaps - - def test_numeric_retry_after_is_honoured(self): - session = FakeSession([ - FakeResponse({}, status_code=429, retry_after="7"), - FakeResponse({"ok": True}), - ]) - with NoSleep() as clock: - triage.request_with_retry(session, "GET", "https://x", attempts=3) - self.assertEqual(clock.slept, [7.0]) - - def test_retry_after_is_capped(self): - session = FakeSession([ - FakeResponse({}, status_code=429, retry_after="9999"), - FakeResponse({"ok": True}), - ]) - with NoSleep() as clock: - triage.request_with_retry(session, "GET", "https://x", attempts=3) - self.assertEqual(clock.slept, [60]) - - def test_http_date_retry_after_falls_back_instead_of_crashing(self): - """RFC 9110 allows an HTTP-date here; float() on it used to raise ValueError.""" - session = FakeSession([ - FakeResponse({}, status_code=503, retry_after="Wed, 21 Oct 2026 07:28:00 GMT"), - FakeResponse({"ok": True}), - ]) - with NoSleep() as clock: - resp = triage.request_with_retry(session, "GET", "https://x", attempts=3) - self.assertEqual(resp.json(), {"ok": True}) - self.assertEqual(clock.slept, [1.0]) # fell back to the backoff delay - - def test_zero_attempts_is_rejected_rather_than_unbound(self): - with self.assertRaises(ValueError): - triage.request_with_retry(FakeSession([]), "GET", "https://x", attempts=0) - - -class TestRetryAfterSeconds(unittest.TestCase): - def test_missing_header_uses_the_default(self): - self.assertEqual(triage.retry_after_seconds(FakeResponse({}, retry_after=None), 4.0), 4.0) - - def test_numeric_header_wins(self): - self.assertEqual(triage.retry_after_seconds(FakeResponse({}, retry_after="12"), 4.0), 12.0) - - def test_unparseable_header_uses_the_default(self): - for raw in ("Wed, 21 Oct 2026 07:28:00 GMT", "", "soon", "12s"): - self.assertEqual(triage.retry_after_seconds(FakeResponse({}, retry_after=raw), 4.0), 4.0) - - def test_negative_and_non_finite_values_use_the_default(self): - """time.sleep() rejects a negative or NaN duration, so passing one through - would crash the run on a hostile or buggy Retry-After header.""" - for raw in ("-30", "-0.5", "nan", "inf", "-inf"): - self.assertEqual(triage.retry_after_seconds(FakeResponse({}, retry_after=raw), 4.0), - 4.0, msg=f"retry-after={raw!r}") - - def test_zero_is_honoured_rather_than_replaced(self): - """Zero is a valid instruction to retry immediately, not a missing value.""" - self.assertEqual(triage.retry_after_seconds(FakeResponse({}, retry_after="0"), 4.0), 0.0) - - def test_a_negative_retry_after_does_not_crash_a_real_retry_loop(self): - session = FakeSession([ - FakeResponse({}, status_code=503, retry_after="-30"), - FakeResponse({"ok": True}), - ]) - with NoSleep() as clock: - resp = triage.request_with_retry(session, "GET", "https://x", attempts=3) - self.assertEqual(resp.json(), {"ok": True}) - self.assertTrue(all(s >= 0 for s in clock.slept), clock.slept) - - # ---- The Claude Code CLI --------------------------------------------------- diff --git a/zendesk_triage/triage.py b/zendesk_triage/triage.py index b2f8463..9956dce 100644 --- a/zendesk_triage/triage.py +++ b/zendesk_triage/triage.py @@ -71,19 +71,22 @@ """ import argparse import json -import math import os import re import subprocess import sys import textwrap -import time from datetime import datetime, timedelta, timezone from functools import partial -from urllib.parse import parse_qsl, urlencode, urlsplit, urlunsplit import requests +sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__)))) +from shared import discord, state as dedup # noqa: E402 +from shared.discord import clip, post_to_discord # noqa: E402 +from shared.env import get_env # noqa: E402 +from shared.retry import request_with_retry # noqa: E402 + # The channel AppFollow imports app-store reviews on. Identified reviews with no # false positives in a 3,662-ticket sample; tags did not (only 287 carried one). REVIEW_CHANNEL = "any_channel" @@ -384,17 +387,6 @@ def window_label(hours): SYSTEM_PROMPT = _SYSTEM_PROMPT_TEMPLATE.replace("__CATEGORIES__", CATEGORY_GUIDANCE) -def get_env(name, cli_value=None, required=True): - if cli_value: - return cli_value - value = os.environ.get(name) - if value: - return value - if required: - sys.exit(f"Missing required config: set the {name} environment variable (or pass the matching flag).") - return None - - def zendesk_session(email, token): session = requests.Session() # Zendesk API-token auth: username is "{email}/token", password is the token. @@ -403,63 +395,6 @@ def zendesk_session(email, token): return session -def retry_after_seconds(resp, default): - """Seconds to wait per the Retry-After header, falling back to `default`. - - RFC 9110 allows either a delay in seconds or an HTTP-date; float() on the date - form raises, so anything unparseable falls back rather than crashing the run. - - Negative, NaN, and infinite values fall back too: time.sleep() rejects the first - two outright, so a hostile or buggy proxy sending `Retry-After: -30` would - otherwise take the run down with a ValueError. - """ - raw = resp.headers.get("retry-after") - if raw is None: - return default - try: - seconds = float(raw) - except (TypeError, ValueError): - return default - if not math.isfinite(seconds) or seconds < 0: - return default - return seconds - - -def request_with_retry(session, method, url, attempts=6, **kwargs): - """GET/POST with backoff on 429 and 5xx. - - Lower `attempts` for calls whose result is nice-to-have: the full budget can - burn ~60s of backoff, which is not worth spending on optional data. - """ - if attempts <= 0: - raise ValueError("attempts must be at least 1") - - delay = 1.0 - last_exc = None - resp = None - for attempt in range(attempts): - final = attempt == attempts - 1 - try: - resp = session.request(method, url, timeout=30, **kwargs) - except requests.RequestException as exc: - last_exc = exc - if final: - break - time.sleep(delay) - delay = min(delay * 2, 30) - continue - if resp.status_code == 429 or resp.status_code >= 500: - if final: - break - time.sleep(min(retry_after_seconds(resp, delay), 60)) - delay = min(delay * 2, 30) - continue - return resp - if last_exc: - raise last_exc - return resp - - def fetch_every_ticket(session, subdomain, query, max_tickets): """Fetch past the Search API's 1000-result ceiling, in created_at slices. @@ -606,30 +541,11 @@ def fetch_total_unsolved(session, subdomain, query=BACKLOG_QUERY): def empty_state(): - return {"version": STATE_VERSION, "seen": {}} + return dedup.empty_state(STATE_VERSION) def load_state(path): - if not os.path.exists(path): - print(f"No state file at {path}; treating every ticket in the window as new.") - return empty_state() - try: - with open(path, encoding="utf-8") as fh: - data = json.load(fh) - except (OSError, json.JSONDecodeError) as exc: - print(f"Note: unreadable state file {path} ({exc}); treating every ticket as new.") - return empty_state() - if not isinstance(data, dict) or not isinstance(data.get("seen"), dict): - print(f"Note: unexpected shape in {path}; treating every ticket as new.") - return empty_state() - # A state file written by a different schema version can't be trusted field by - # field, so treat it as a cache miss rather than misreading it. - if data.get("version") != STATE_VERSION: - print(f"Note: {path} is version {data.get('version')!r}, expected {STATE_VERSION}; " - f"treating every ticket as new.") - return empty_state() - print(f"Loaded state for {len(data['seen'])} previously reported tickets.") - return data + return dedup.load_state(path, STATE_VERSION, "ticket") def activity_key(ticket): @@ -702,40 +618,10 @@ def partition_by_state(tickets, state): def save_state(path, state, reported, retention_days): - """Record `reported` as seen, prune old entries, write atomically. - - Returns (kept, pruned). - """ - now = datetime.now(timezone.utc) - stamp = now.strftime("%Y-%m-%dT%H:%M:%SZ") - seen = dict(state.get("seen", {})) - for ticket in reported: - seen[str(ticket.get("id"))] = { - "requester_updated_at": activity_key(ticket), - "last_reported": stamp, - } - - # Bound the file: the window is 72h, so anything older than retention is moot. - cutoff = now - timedelta(days=retention_days) - kept = {} - for ticket_id, record in seen.items(): - try: - last = datetime.strptime( - record.get("last_reported", ""), "%Y-%m-%dT%H:%M:%SZ" - ).replace(tzinfo=timezone.utc) - except (TypeError, ValueError): - continue # malformed entry β€” drop it rather than keep it forever - if last >= cutoff: - kept[ticket_id] = record - - directory = os.path.dirname(path) - if directory: - os.makedirs(directory, exist_ok=True) - temporary = f"{path}.tmp" - with open(temporary, "w", encoding="utf-8") as fh: - json.dump({"version": STATE_VERSION, "updated_at": stamp, "seen": kept}, fh, indent=2) - os.replace(temporary, path) # atomic: a crash mid-write can't corrupt the state - return len(kept), len(seen) - len(kept) + """Record `reported` as seen. Returns (kept, pruned).""" + records = {str(t.get("id")): {"requester_updated_at": activity_key(t)} + for t in reported} + return dedup.save_state(path, state, records, retention_days, STATE_VERSION) # ---- App-store review filtering -------------------------------------------- @@ -1557,40 +1443,23 @@ def analyze(model, effort, compact_tickets): # ---- Components V2 --------------------------------------------------------- # -# The digest is a Container of Text Displays: one block per ticket, so a reader skims -# lines rather than a wall, and each message records which ticket ids it accounts for. -# -# Every component here is non-interactive, which is what lets a plain incoming webhook -# carry it: Discord allows a webhook that no application owns only those. Adding an -# interactive one would need the transport moved back to a bot token β€” see -# test_the_digest_carries_no_interactive_components. -# -# https://docs.discord.com/developers/components/reference -COMPONENTS_V2_FLAG = 1 << 15 -CONTAINER = 17 -TEXT_DISPLAY = 10 -SEPARATOR = 14 +# The digest is a Container of Text Displays, one per ticket, built by +# shared.discord β€” which also documents why a plain webhook can carry it. # Discord allows 40 components in one message, and a ticket now costs one Text # Display, so that ceiling no longer binds β€” the character budget below does. Ten is # kept because it is a readable message, not because it is the limit. MAX_ENTRIES_PER_MESSAGE = 10 -# Discord's ceiling on all the text in one Components V2 message, and the constraint -# that actually binds. Ten clipped ticket lines plus a header come to roughly 3,500, +# Discord's ceiling on all the text in one message, and the constraint that +# actually binds. Ten clipped ticket lines plus a header come to roughly 3,500, # so this is a guard rather than a routine constraint. -MAX_MESSAGE_TEXT_CHARS = 4000 -MAX_COMPONENT_CHARS = MAX_MESSAGE_TEXT_CHARS +MAX_COMPONENT_CHARS = discord.MAX_MESSAGE_TEXT_CHARS def ticket_url(subdomain, ticket_id): return f"https://{subdomain}.zendesk.com/agent/tickets/{ticket_id}" -def clip(text, limit): - text = (text or "").strip() - return text if len(text) <= limit else text[: limit - 1] + "…" - - def is_urgent(finding): return finding.get("category") in URGENT_CATEGORIES @@ -1737,42 +1606,6 @@ def build_header(findings, highlights, stats=None): return "\n".join(lines) -def chunk_entries(entries, max_items=MAX_ENTRIES_PER_MESSAGE, - max_chars=MAX_COMPONENT_CHARS, first_used=0): - """Group (line, ticket_ids) pairs into messages within Discord's budgets. - - Two limits rather than one, and whichever binds first splits the message: a - Components V2 message allows 40 components, of which a ticket costs three, and a - character budget that clipped lines rarely approach. Sections are separate - components rather than joined text, so unlike the old plain-content digest - nothing is spent on the newlines between them. - - `first_used` is what the caller has already spent on the first message before any - ticket goes in β€” the header. Without it the header rides on top of a full budget - of ticket lines, and a busy day's accounting lines are enough to put message one - over the limit. - - An entry longer than the character budget still gets its own message rather than - being dropped; the pieces are pre-clipped so that shouldn't arise. - - Entries are passed through, not rebuilt, so the caller can still tell which one - it is looking at by identity β€” build_messages needs that to find the collapsed - line again once its entry is somewhere inside a chunk. - """ - chunks, current, current_chars = [], [], first_used - for entry in entries: - text, _ = entry - if current and (len(current) >= max_items - or current_chars + len(text) > max_chars): - chunks.append(current) - current, current_chars = [], 0 - current.append(entry) - current_chars += len(text) - if current: - chunks.append(current) - return chunks - - def select_highlights(findings): """Ordered highlights split into (shown, omitted) by the display cap. @@ -1822,70 +1655,12 @@ def build_messages(findings, subdomain, stats=None, updated_ids=None): if collapsed: entries.append((build_collapsed_line(collapsed, subdomain), collapsed_ids)) - messages, coverage = [], [] - # A quiet day still owes the channel the header β€” chunk_entries has nothing to - # chunk when no ticket is worth looking into, so seed one empty chunk. - # The header only lands on message one, so only message one's budget pays for - # it. chunk_entries resets to zero for every chunk after the first. - chunks = chunk_entries(entries, first_used=len(header)) or [[]] - for index, chunk in enumerate(chunks): - blocks = [] - covered = set() - if index == 0: - blocks.append({"type": TEXT_DISPLAY, "content": header}) - if chunk: - blocks.append({"type": SEPARATOR}) - # The header accounts for every classified ticket except the highlights - # that didn't fit; those are covered by no message and stay eligible. - covered |= header_ids - for text, ids in chunk: - blocks.append({"type": TEXT_DISPLAY, "content": text}) - covered |= ids - messages.append({ - "flags": COMPONENTS_V2_FLAG, - "components": [{"type": CONTAINER, "components": blocks}], - }) - coverage.append(covered) + messages, coverage = discord.messages_from_entries( + header, entries, MAX_ENTRIES_PER_MESSAGE, MAX_COMPONENT_CHARS) + coverage[0] |= header_ids return messages, coverage -def digest_webhook_url(webhook_url): - """The webhook, told to respect the components field. - - Discord ignores `components` on a webhook post without it, and the digest is - nothing but components. - """ - parts = urlsplit(webhook_url) - query = dict(parse_qsl(parts.query)) - query["with_components"] = "true" - return urlunsplit(parts._replace(query=urlencode(query))) - - -def post_to_discord(session, url, messages): - """POST each message in order; return how many Discord accepted. - - Stops at the first failure and returns the accepted count instead of exiting, so - the caller can record the tickets that did land before signalling the failure β€” - otherwise a failure on message 3 of 3 reposts messages 1 and 2 on the next run. - """ - for index, payload in enumerate(messages): - try: - resp = request_with_retry(session, "POST", url, json=payload) - except requests.RequestException as exc: - # request_with_retry re-raises once its budget is spent. Letting that - # propagate would skip save_state entirely, so the messages that already - # landed would be reposted on the next run β€” the exact thing returning a - # count exists to prevent. - print(f"Discord unreachable on message {index + 1}/{len(messages)} " - f"({exc}).") - return index - if resp.status_code >= 400: - print(f"Discord rejected message {index + 1}/{len(messages)} " - f"({resp.status_code}): {resp.text[:300]}") - return index - return len(messages) - - def main(): parser = argparse.ArgumentParser(description="Triage open Zendesk tickets with Claude and post a Discord summary.") parser.add_argument("--subdomain", help="Zendesk subdomain (else ZENDESK_SUBDOMAIN).") @@ -2116,7 +1891,8 @@ def main(): return # A fresh session, never the Zendesk one: that carries the API-token auth header. - posted = post_to_discord(requests.Session(), digest_webhook_url(webhook), messages) + posted = post_to_discord(requests.Session(), discord.components_webhook_url(webhook), + messages) print(f"Posted {posted} of {len(messages)} Discord message(s).") # Record only tickets covered by messages Discord actually accepted, so a partial From 3c8d8920e25b1b69ff47b4ec91b57633b31bd306 Mon Sep 17 00:00:00 2001 From: Audric Ackermann Date: Thu, 24 Sep 2026 15:52:20 +1000 Subject: [PATCH 007/101] refactor: build the pull request digest on the shared helpers The digest carried its own copy of the retry loop, the dedup state file, the Components V2 constants, the chunker and the webhook posting. It now imports them from shared/, and keeps only what is its own: the record it stores per PR, its per-message block cap, and the rendering. The tests that covered the copies move with the code; what stays here covers the wrappers and the rendering. --- README.md | 5 +- github_prs/digest.py | 240 ++++---------------------------------- github_prs/test_digest.py | 116 ++---------------- 3 files changed, 32 insertions(+), 329 deletions(-) diff --git a/README.md b/README.md index 3bfc869..4a3e689 100644 --- a/README.md +++ b/README.md @@ -735,10 +735,13 @@ that: private repositories stay out whatever the token can see. | `GITHUB_PRS_ORG` | optional; defaults to `session-foundation` | It runs on the same box as the Zendesk digest, under its own user and its own -environment file β€” see [deploy/README.md](deploy/README.md). +environment file β€” see [deploy/README.md](deploy/README.md). Its HTTP retries, +Discord posting and dedup state are the same code the Zendesk digest uses, in +[shared/](shared/). ```sh cd github_prs && python -m unittest discover +cd shared && python -m unittest discover ``` ## Workflow Failure Notificaiton diff --git a/github_prs/digest.py b/github_prs/digest.py index 5db6c0d..e6be7e8 100755 --- a/github_prs/digest.py +++ b/github_prs/digest.py @@ -19,10 +19,6 @@ here rather than in the query. That is what lets the header carry the total open contributor backlog alongside the day's changes for the cost of a single query. -The helpers here deliberately duplicate their zendesk_triage counterparts rather -than importing them: the two jobs run under different users from different env -files, and a shared module would make either one's dependencies the other's. - Config (env vars, or flags for local runs): GITHUB_PRS_TOKEN GitHub token, read-only. Needs no scope at all: the digest reads public repositories only. @@ -43,16 +39,19 @@ """ import argparse import json -import math import os import sys -import time from datetime import datetime, timedelta, timezone from operator import itemgetter -from urllib.parse import parse_qsl, urlencode, urlsplit, urlunsplit import requests +sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__)))) +from shared import discord, state as dedup # noqa: E402 +from shared.discord import MAX_MESSAGE_TEXT_CHARS, clip # noqa: E402 +from shared.env import get_env # noqa: E402 +from shared.retry import request_with_retry # noqa: E402 + API = "https://api.github.com" DEFAULT_ORG = "session-foundation" # Three days, because the timer runs on weekdays: Monday's window has to reach back @@ -70,17 +69,6 @@ "maintainers.txt") -def get_env(name, cli_value=None, required=True): - if cli_value: - return cli_value - value = os.environ.get(name) - if value: - return value - if required: - sys.exit(f"Missing required config: set the {name} environment variable (or pass the matching flag).") - return None - - def load_maintainers(path): """Logins from maintainers.txt, lowercased. `#` comments and blanks ignored.""" logins = set() @@ -102,63 +90,6 @@ def github_session(token): return session -def retry_after_seconds(resp, default): - """Seconds to wait per the response, falling back to `default`. - - GitHub answers a secondary rate limit with Retry-After, and a primary one with - x-ratelimit-reset as an epoch second and no Retry-After at all, so both are - read. Anything unparseable, negative or infinite falls back rather than taking - the run down inside time.sleep(). - """ - raw = resp.headers.get("retry-after") - if raw is None: - reset = resp.headers.get("x-ratelimit-reset") - if reset is None: - return default - try: - return max(0.0, float(reset) - time.time()) - except (TypeError, ValueError): - return default - try: - seconds = float(raw) - except (TypeError, ValueError): - return default - if not math.isfinite(seconds) or seconds < 0: - return default - return seconds - - -def request_with_retry(session, method, url, attempts=5, **kwargs): - """GET/POST with backoff on 429 and 5xx.""" - if attempts <= 0: - raise ValueError("attempts must be at least 1") - - delay = 1.0 - last_exc = None - resp = None - for attempt in range(attempts): - final = attempt == attempts - 1 - try: - resp = session.request(method, url, timeout=30, **kwargs) - except requests.RequestException as exc: - last_exc = exc - if final: - break - time.sleep(delay) - delay = min(delay * 2, 30) - continue - if resp.status_code == 429 or resp.status_code >= 500: - if final: - break - time.sleep(min(retry_after_seconds(resp, delay), 60)) - delay = min(delay * 2, 30) - continue - return resp - if last_exc: - raise last_exc - return resp - - def fetch_json(session, url, **kwargs): resp = request_with_retry(session, "GET", url, **kwargs) if resp.status_code >= 400: @@ -265,40 +196,14 @@ def activity_key(pr): # ---- Dedup state ----------------------------------------------------------- -# -# Which PRs have already been reported, and what their activity looked like then. -# Every failure to read it is a cache miss rather than an error: losing the file -# re-reports the window once, which is noisy and never wrong, and that is what makes -# it a cache rather than something to back up. def empty_state(): - return {"version": STATE_VERSION, "seen": {}} + return dedup.empty_state(STATE_VERSION) def load_state(path): - if not path: - return empty_state() - if not os.path.exists(path): - print(f"No state file at {path}; treating every PR in the window as new.") - return empty_state() - try: - with open(path, encoding="utf-8") as handle: - data = json.load(handle) - except (OSError, json.JSONDecodeError) as exc: - print(f"Note: unreadable state file {path} ({exc}); treating every PR as new.") - return empty_state() - if not isinstance(data, dict) or not isinstance(data.get("seen"), dict): - print(f"Note: unexpected shape in {path}; treating every PR as new.") - return empty_state() - # A file written by another schema version cannot be trusted field by field, so - # it is a cache miss rather than something to misread. - if data.get("version") != STATE_VERSION: - print(f"Note: {path} is version {data.get('version')!r}, expected " - f"{STATE_VERSION}; treating every PR as new.") - return empty_state() - print(f"Loaded state for {len(data['seen'])} previously reported PRs.") - return data + return dedup.load_state(path, STATE_VERSION, "PR") def partition_by_state(prs, state): @@ -317,42 +222,14 @@ def partition_by_state(prs, state): def save_state(path, state, reported, retention_days=DEFAULT_RETENTION_DAYS): - """Record `reported` as seen, prune old entries, write atomically. - - Returns (kept, pruned). - """ - now = datetime.now(timezone.utc) - stamp = now.strftime("%Y-%m-%dT%H:%M:%SZ") - seen = dict(state.get("seen", {})) - for pr in reported: - seen[pr_id(pr)] = { - "updated_at": activity_key(pr), - "last_reported": stamp, - # Not read back. The file is the first thing anyone opens when the digest - # reports the wrong thing, and an id alone identifies nothing. - "pr": f"{repo_name(pr)}#{pr.get('number')}", - } - - cutoff = now - timedelta(days=retention_days) - kept = {} - for key, record in seen.items(): - try: - last = datetime.strptime(record.get("last_reported", ""), - "%Y-%m-%dT%H:%M:%SZ").replace(tzinfo=timezone.utc) - except (TypeError, ValueError): - continue # malformed entry β€” drop it rather than keep it forever - if last >= cutoff: - kept[key] = record - - directory = os.path.dirname(path) - if directory: - os.makedirs(directory, exist_ok=True) - temporary = f"{path}.tmp" - with open(temporary, "w", encoding="utf-8") as handle: - json.dump({"version": STATE_VERSION, "updated_at": stamp, "seen": kept}, - handle, indent=2) - os.replace(temporary, path) # atomic: a crash mid-write cannot corrupt the state - return len(kept), len(seen) - len(kept) + """Record `reported` as seen. Returns (kept, pruned).""" + records = {pr_id(pr): {"updated_at": activity_key(pr), + # Not read back. The file is the first thing anyone opens + # when the digest reports the wrong thing, and an id + # alone identifies nothing. + "pr": f"{repo_name(pr)}#{pr.get('number')}"} + for pr in reported} + return dedup.save_state(path, state, records, retention_days, STATE_VERSION) # ---- Discord rendering ----------------------------------------------------- @@ -360,12 +237,6 @@ def save_state(path, state, reported, retention_days=DEFAULT_RETENTION_DAYS): # A header, then one block per repository whose PRs changed. Components V2 so that # each repository is its own component: a long day splits between repositories # rather than mid-list, unless one repository alone outgrows a message. -COMPONENTS_V2_FLAG = 1 << 15 -CONTAINER = 17 -TEXT_DISPLAY = 10 -SEPARATOR = 14 -# Discord's ceiling on all the text in one Components V2 message. -MAX_MESSAGE_TEXT_CHARS = 4000 MAX_COMPONENTS_PER_MESSAGE = 10 TITLE_CHARS = 90 @@ -373,11 +244,6 @@ def save_state(path, state, reported, retention_days=DEFAULT_RETENTION_DAYS): UPDATED_MARKER = "✏️" -def clip(text, limit): - text = (text or "").strip() - return text if len(text) <= limit else text[: limit - 1] + "…" - - def age(then, now): """Compact age, coarsening as it grows: 40m, 6h, 3d, 5w.""" minutes = max(0, int((now - then).total_seconds() // 60)) @@ -463,79 +329,13 @@ def build_header(new, updated, backlog, window_hours, truncated): return "\n".join(lines) -def chunk_blocks(blocks, first_used=0, - max_items=MAX_COMPONENTS_PER_MESSAGE, - max_chars=MAX_MESSAGE_TEXT_CHARS): - """Group rendered blocks into messages within Discord's budgets. - - `first_used` is what the header has already spent on the first message; without - it a busy day's first message goes over on the header alone. - - A single block over the character budget still gets its own message rather than - being dropped; group_by_repo splits blocks to keep that from arising. - """ - chunks, current, used = [], [], first_used - for block in blocks: - cost = len(block[0]) if isinstance(block, tuple) else len(block) - if current and (len(current) >= max_items or used + cost > max_chars): - chunks.append(current) - current, used = [], 0 - current.append(block) - used += cost - if current: - chunks.append(current) - return chunks - - def build_messages(new, updated, backlog, window_hours, now, truncated=False): - """Return (messages, coverage). - - coverage[i] is the set of PR ids message i accounts for, so a run that fails - partway through can still record exactly what reached Discord β€” otherwise a - failure on the last message re-posts the first ones tomorrow. - """ + """Return (messages, coverage), as shared.discord.messages_from_entries does.""" header = build_header(new, updated, backlog, window_hours, truncated) # Every block is sized to fit beside the header, though only the first message # carries it: simpler than sizing the first block differently. blocks = group_by_repo(new, updated, now, MAX_MESSAGE_TEXT_CHARS - len(header)) - messages, coverage = [], [] - # A quiet day still owes the channel its header, so seed one empty chunk. - for index, chunk in enumerate(chunk_blocks(blocks, first_used=len(header)) or [[]]): - components = [] - if index == 0: - components.append({"type": TEXT_DISPLAY, "content": header}) - if chunk: - components.append({"type": SEPARATOR}) - components += [{"type": TEXT_DISPLAY, "content": text} for text, _ in chunk] - messages.append({ - "flags": COMPONENTS_V2_FLAG, - "components": [{"type": CONTAINER, "components": components}], - }) - coverage.append(set().union(*(ids for _, ids in chunk)) if chunk else set()) - return messages, coverage - - -def components_webhook_url(webhook_url): - """The webhook, told to respect the components field, which it ignores without.""" - parts = urlsplit(webhook_url) - query = dict(parse_qsl(parts.query)) - query["with_components"] = "true" - return urlunsplit(parts._replace(query=urlencode(query))) - - -def post_to_discord(session, url, messages): - """POST each message in order; return how many Discord accepted.""" - for index, payload in enumerate(messages): - try: - resp = request_with_retry(session, "POST", url, json=payload) - except requests.RequestException as exc: - print(f"Discord unreachable on message {index + 1}/{len(messages)} ({exc}).") - return index - if resp.status_code >= 400: - print(f"Discord rejected message {index + 1}/{len(messages)} " - f"({resp.status_code}): {resp.text[:300]}") - return index - return len(messages) + return discord.messages_from_entries(header, blocks, MAX_COMPONENTS_PER_MESSAGE) def main(): @@ -587,8 +387,8 @@ def main(): print(json.dumps(messages, indent=2, ensure_ascii=False)) return - posted = post_to_discord(requests.Session(), - components_webhook_url(webhook), messages) + posted = discord.post_to_discord(requests.Session(), + discord.components_webhook_url(webhook), messages) # Only what Discord accepted. A PR in a message that never landed stays eligible. if args.state: landed = set().union(*coverage[:posted]) if posted else set() diff --git a/github_prs/test_digest.py b/github_prs/test_digest.py index 49d2fda..0cc2b82 100644 --- a/github_prs/test_digest.py +++ b/github_prs/test_digest.py @@ -5,12 +5,15 @@ import io import json import os +import sys import tempfile import unittest from datetime import datetime, timedelta, timezone from unittest import mock -import digest +sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__)))) +import digest # noqa: E402 +from shared import discord # noqa: E402 NOW = datetime(2026, 9, 24, 12, 0, tzinfo=timezone.utc) CUTOFF = NOW - timedelta(hours=25) @@ -143,41 +146,6 @@ def test_the_file_names_the_pr_for_whoever_opens_it(self): def test_no_path_means_no_state_and_no_complaint(self): self.assertEqual(digest.load_state(None), digest.empty_state()) - def test_every_unreadable_state_is_a_cache_miss_not_an_error(self): - """Losing it re-reports the window once. Failing the run instead would mean a - corrupt cache stops the digest entirely.""" - for content in ("{ not json", '{"version": 1}', "[]", - '{"version": 99, "seen": {}}'): - with self.subTest(content=content): - with open(self.path.replace("sub/", ""), "w", encoding="utf-8") as fh: - fh.write(content) - with contextlib.redirect_stdout(io.StringIO()): - state = digest.load_state(self.path.replace("sub/", "")) - self.assertEqual(state, digest.empty_state()) - - def test_a_missing_file_is_a_cache_miss(self): - self.assertEqual(self.load(), digest.empty_state()) - - def test_entries_are_pruned_past_the_retention(self): - self.save(pr(1)) - state = self.load() - for record in state["seen"].values(): - record["last_reported"] = "2026-01-01T00:00:00Z" - with contextlib.redirect_stdout(io.StringIO()): - kept, pruned = digest.save_state(self.path, state, [], retention_days=30) - self.assertEqual((kept, pruned), (0, 1)) - - def test_a_malformed_entry_is_dropped_rather_than_kept_forever(self): - state = {"version": digest.STATE_VERSION, - "seen": {"1": {"updated_at": "x", "last_reported": "never"}}} - with contextlib.redirect_stdout(io.StringIO()): - kept, _ = digest.save_state(self.path, state, []) - self.assertEqual(kept, 0) - - def test_nothing_is_left_behind_by_the_atomic_write(self): - self.save(pr(1)) - self.assertEqual(sorted(os.listdir(os.path.dirname(self.path))), ["seen.json"]) - class TestAge(unittest.TestCase): def test_coarsens_as_it_grows(self): @@ -297,29 +265,6 @@ def test_truncation_is_declared(self): self.assertIn("floor", digest.build_header([], [], 1000, 25, True)) -class TestChunking(unittest.TestCase): - def test_splits_on_the_component_count(self): - blocks = ["x"] * (digest.MAX_COMPONENTS_PER_MESSAGE + 1) - chunks = digest.chunk_blocks(blocks) - self.assertEqual([len(chunk) for chunk in chunks], - [digest.MAX_COMPONENTS_PER_MESSAGE, 1]) - - def test_splits_on_the_character_budget(self): - blocks = ["x" * 2500, "y" * 2500] - self.assertEqual(len(digest.chunk_blocks(blocks)), 2) - - def test_the_header_is_charged_to_the_first_message_only(self): - block = "x" * 1900 - chunks = digest.chunk_blocks([block, block], first_used=0) - self.assertEqual(len(chunks), 1) - chunks = digest.chunk_blocks([block, block], first_used=1000) - self.assertEqual(len(chunks), 2) - - def test_an_oversized_block_still_gets_a_message(self): - blocks = digest.chunk_blocks(["x" * (digest.MAX_MESSAGE_TEXT_CHARS + 100)]) - self.assertEqual(len(blocks), 1) - - class TestMessages(unittest.TestCase): def test_a_quiet_day_still_posts_the_header(self): messages, coverage = digest.build_messages([], [], 4, 25, NOW) @@ -331,8 +276,8 @@ def test_a_quiet_day_still_posts_the_header(self): def test_the_payload_is_components_v2(self): message = digest.build_messages([pr(1)], [], 1, 25, NOW)[0][0] - self.assertEqual(message["flags"], digest.COMPONENTS_V2_FLAG) - self.assertEqual(message["components"][0]["type"], digest.CONTAINER) + self.assertEqual(message["flags"], discord.COMPONENTS_V2_FLAG) + self.assertEqual(message["components"][0]["type"], discord.CONTAINER) def test_one_busy_repo_never_exceeds_the_message_budget(self): new = [pr(n, title="y" * 80) for n in range(120)] @@ -341,7 +286,7 @@ def test_one_busy_repo_never_exceeds_the_message_budget(self): for message in messages: text = sum(len(c.get("content", "")) for c in message["components"][0]["components"]) - self.assertLessEqual(text, digest.MAX_MESSAGE_TEXT_CHARS) + self.assertLessEqual(text, discord.MAX_MESSAGE_TEXT_CHARS) self.assertEqual(set().union(*coverage), {digest.pr_id(p) for p in new}) def test_only_the_first_message_carries_the_header(self): @@ -352,18 +297,7 @@ def test_only_the_first_message_carries_the_header(self): for message in messages[1:]: rendered = json.dumps(message, ensure_ascii=False) self.assertNotIn("Contributor pull requests", rendered) - self.assertLess(len(rendered), digest.MAX_MESSAGE_TEXT_CHARS * 2) - - -class TestWebhookUrl(unittest.TestCase): - def test_components_are_requested(self): - url = digest.components_webhook_url("https://discord.com/api/webhooks/1/tok") - self.assertIn("with_components=true", url) - - def test_an_existing_query_survives(self): - url = digest.components_webhook_url("https://discord.com/api/webhooks/1/tok?wait=true") - self.assertIn("wait=true", url) - self.assertIn("with_components=true", url) + self.assertLess(len(rendered), discord.MAX_MESSAGE_TEXT_CHARS * 2) class TestFetching(unittest.TestCase): @@ -400,39 +334,5 @@ def test_search_stops_at_the_result_ceiling_rather_than_erroring(self): self.assertEqual(fetch.call_count, digest.SEARCH_RESULT_LIMIT // digest.PER_PAGE) -class TestRetryAfter(unittest.TestCase): - def response(self, headers): - return mock.Mock(headers=headers) - - def test_retry_after_seconds_is_used(self): - self.assertEqual(digest.retry_after_seconds(self.response({"retry-after": "12"}), 1), 12) - - def test_a_primary_limit_falls_back_to_the_reset_epoch(self): - import time as time_module - reset = str(int(time_module.time()) + 30) - seconds = digest.retry_after_seconds( - self.response({"x-ratelimit-reset": reset}), 1) - self.assertTrue(25 <= seconds <= 31, seconds) - - def test_nonsense_falls_back_to_the_caller_default(self): - for headers in ({"retry-after": "soon"}, {"retry-after": "-30"}, - {"retry-after": "inf"}, {"x-ratelimit-reset": "?"}, {}): - self.assertEqual(digest.retry_after_seconds(self.response(headers), 7), 7) - - -class TestPosting(unittest.TestCase): - def test_a_rejection_reports_what_landed_before_it(self): - ok, bad = mock.Mock(status_code=204), mock.Mock(status_code=400, text="no") - session = mock.Mock() - session.request.side_effect = [ok, bad] - with contextlib.redirect_stdout(io.StringIO()): - self.assertEqual(digest.post_to_discord(session, "url", [{}, {}, {}]), 1) - - def test_all_accepted(self): - session = mock.Mock() - session.request.return_value = mock.Mock(status_code=204) - self.assertEqual(digest.post_to_discord(session, "url", [{}, {}]), 2) - - if __name__ == "__main__": unittest.main() From 061e608c33ba33e6d1c4a626cae555c76f08c021 Mon Sep 17 00:00:00 2001 From: Audric Ackermann Date: Thu, 24 Sep 2026 16:01:48 +1000 Subject: [PATCH 008/101] feat: let the shared retry read an HTTP-date Retry-After and take a timeout RFC 9110 allows Retry-After in either form. Crowdin's report already parsed the date form, so the shared loop has to before it can replace that copy; a date already gone reads as no header, since time.sleep() rejects a negative. The timeout was fixed at 30s; Crowdin's scan runs on 60. --- shared/retry.py | 36 +++++++++++++++++++++++++----------- shared/test_retry.py | 30 ++++++++++++++++++++++++------ 2 files changed, 49 insertions(+), 17 deletions(-) diff --git a/shared/retry.py b/shared/retry.py index 981d075..82059f3 100644 --- a/shared/retry.py +++ b/shared/retry.py @@ -1,5 +1,7 @@ +import email.utils import math import time +from datetime import datetime, timezone import requests @@ -7,14 +9,14 @@ def retry_after_seconds(resp, default): """Seconds to wait per the response's rate-limit headers, else `default`. - Retry-After first. GitHub answers a primary rate limit with x-ratelimit-reset - as an epoch second and no Retry-After at all, so that is read when the header - is absent. + Retry-After first, in either form RFC 9110 allows: a delay in seconds or an + HTTP-date. GitHub answers a primary rate limit with x-ratelimit-reset as an + epoch second and no Retry-After at all, so that is read when the header is + absent. - RFC 9110 allows Retry-After to be an HTTP-date; float() on the date form - raises, so anything unparseable falls back rather than crashing the run. - Negative, NaN and infinite values fall back too: time.sleep() rejects the first - two outright, so a hostile or buggy proxy sending `Retry-After: -30` would + Anything unparseable falls back rather than crashing the run. Negative, NaN + and infinite values fall back too: time.sleep() rejects the first two + outright, so a hostile or buggy proxy sending `Retry-After: -30` would otherwise take the run down with a ValueError. """ raw = resp.headers.get("retry-after") @@ -29,14 +31,26 @@ def retry_after_seconds(resp, default): try: seconds = float(raw) except (TypeError, ValueError): - return default + seconds = _seconds_until_http_date(raw) + if seconds is None: + return default if not math.isfinite(seconds) or seconds < 0: return default return seconds -def request_with_retry(session, method, url, attempts=6, **kwargs): - """GET/POST with backoff on 429 and 5xx. +def _seconds_until_http_date(value): + try: + when = email.utils.parsedate_to_datetime(value) + except (TypeError, ValueError): + return None + if when.tzinfo is None: + when = when.replace(tzinfo=timezone.utc) + return (when - datetime.now(timezone.utc)).total_seconds() + + +def request_with_retry(session, method, url, attempts=6, timeout=30, **kwargs): + """GET/POST with backoff on 429 and 5xx. Returns the response, whatever its status. Lower `attempts` for calls whose result is nice-to-have: the full budget can burn ~60s of backoff, which is not worth spending on optional data. @@ -50,7 +64,7 @@ def request_with_retry(session, method, url, attempts=6, **kwargs): for attempt in range(attempts): final = attempt == attempts - 1 try: - resp = session.request(method, url, timeout=30, **kwargs) + resp = session.request(method, url, timeout=timeout, **kwargs) except requests.RequestException as exc: last_exc = exc if final: diff --git a/shared/test_retry.py b/shared/test_retry.py index fef3ec6..f1fa3df 100644 --- a/shared/test_retry.py +++ b/shared/test_retry.py @@ -5,6 +5,8 @@ import sys import time import unittest +from email.utils import format_datetime +from datetime import datetime, timedelta, timezone import requests @@ -80,16 +82,24 @@ def test_retry_after_is_capped(self): retry.request_with_retry(session, "GET", "https://x", attempts=3) self.assertEqual(clock.slept, [60]) - def test_http_date_retry_after_falls_back_instead_of_crashing(self): - """RFC 9110 allows an HTTP-date here; float() on it used to raise ValueError.""" + def test_an_http_date_retry_after_is_honoured(self): + """RFC 9110 allows an HTTP-date here as well as a count of seconds.""" + soon = format_datetime(datetime.now(timezone.utc) + timedelta(seconds=20), usegmt=True) session = FakeSession([ - FakeResponse({}, status_code=503, retry_after="Wed, 21 Oct 2026 07:28:00 GMT"), + FakeResponse({}, status_code=503, retry_after=soon), FakeResponse({"ok": True}), ]) with NoSleep() as clock: resp = retry.request_with_retry(session, "GET", "https://x", attempts=3) self.assertEqual(resp.json(), {"ok": True}) - self.assertEqual(clock.slept, [1.0]) # fell back to the backoff delay + self.assertEqual(len(clock.slept), 1) + self.assertTrue(15 <= clock.slept[0] <= 21, clock.slept) + + def test_the_timeout_is_forwarded_and_defaults_to_thirty_seconds(self): + session = FakeSession([FakeResponse({}), FakeResponse({})]) + retry.request_with_retry(session, "GET", "https://x") + retry.request_with_retry(session, "GET", "https://x", timeout=60) + self.assertEqual([kw["timeout"] for _, _, kw in session.calls], [30, 60]) def test_zero_attempts_is_rejected_rather_than_unbound(self): with self.assertRaises(ValueError): @@ -119,8 +129,16 @@ def test_numeric_header_wins(self): self.assertEqual(self.seconds({"retry-after": "12"}), 12.0) def test_unparseable_header_uses_the_default(self): - for raw in ("Wed, 21 Oct 2026 07:28:00 GMT", "", "soon", "12s"): - self.assertEqual(self.seconds({"retry-after": raw}), 4.0) + for raw in ("", "soon", "12s", "Wed, 32 Oct 2026 07:28:00 GMT"): + self.assertEqual(self.seconds({"retry-after": raw}), 4.0, msg=f"retry-after={raw!r}") + + def test_an_http_date_in_the_past_uses_the_default(self): + """A date already gone means a negative wait, which time.sleep() rejects.""" + self.assertEqual(self.seconds({"retry-after": "Wed, 21 Oct 2015 07:28:00 GMT"}), 4.0) + + def test_an_http_date_is_the_seconds_until_it(self): + soon = format_datetime(datetime.now(timezone.utc) + timedelta(seconds=30), usegmt=True) + self.assertTrue(25 <= self.seconds({"retry-after": soon}) <= 31) def test_negative_and_non_finite_values_use_the_default(self): """time.sleep() rejects a negative or NaN duration, so passing one through From 8e30ddd2d6b3b85e42fc69a24b33f863d54c5342 Mon Sep 17 00:00:00 2001 From: Audric Ackermann Date: Thu, 24 Sep 2026 16:01:48 +1000 Subject: [PATCH 009/101] refactor: post the Crowdin report through the shared retry and webhook code The third copy of the retry loop and the Discord posting. What stays is Crowdin's own contract: ten attempts at a 60s timeout, a 4xx that raises so callers can read the body unchecked, and the plain-text warning posted before the run exits on a rejected embed. --- crowdin/report_multiple_translations.py | 93 +++++--------------- crowdin/test_report_multiple_translations.py | 75 ++++++++++++++++ 2 files changed, 97 insertions(+), 71 deletions(-) create mode 100644 crowdin/test_report_multiple_translations.py diff --git a/crowdin/report_multiple_translations.py b/crowdin/report_multiple_translations.py index 43da1e9..18e1c37 100644 --- a/crowdin/report_multiple_translations.py +++ b/crowdin/report_multiple_translations.py @@ -40,16 +40,17 @@ import collections import concurrent.futures import datetime as dt -import email.utils import json import os import subprocess import sys import threading -import time import requests +sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__)))) +from shared import discord, retry # noqa: E402 + API = "https://api.crowdin.com/api/v2" DEFAULT_PROJECT = "618696" KEYRING_ATTRS = ["service", "crowdin", "key", "translation-api-token"] @@ -81,49 +82,10 @@ def get_token(cli_token): "or store it via secret-tool).") -def parse_retry_after(value, fallback, cap=30): - """Seconds to wait for a Retry-After header, parsed defensively. - - Supports both numeric-seconds and HTTP-date forms (RFC 7231); falls back to - `fallback` when the header is missing or unparseable, and clamps the result - to `cap` so a bogus/huge value can't stall the run.""" - wait = fallback - if value is not None: - try: - wait = float(value) - except (TypeError, ValueError): - try: - when = email.utils.parsedate_to_datetime(value) - if when.tzinfo is None: - when = when.replace(tzinfo=dt.timezone.utc) - wait = (when - dt.datetime.now(dt.timezone.utc)).total_seconds() - except (TypeError, ValueError): - wait = fallback - if wait < 0: - wait = fallback - return min(wait, cap) - - -def request_with_retry(session, method, url, max_retries=10, **kw): - """Request with backoff on 429/5xx AND on network errors (flaky DNS/connection).""" - delay = 0.5 - last_exc = None - for _ in range(max_retries): - try: - r = session.request(method, url, timeout=60, **kw) - except requests.exceptions.RequestException as e: - last_exc = e - time.sleep(delay) - delay = min(delay * 2, 30) - continue - if r.status_code == 429 or r.status_code >= 500: - time.sleep(parse_retry_after(r.headers.get("Retry-After"), delay)) - delay = min(delay * 2, 30) - continue - r.raise_for_status() - return r - if last_exc: - raise last_exc +def request_with_retry(session, method, url, **kw): + """A Crowdin API call. A non-retryable 4xx raises, so a caller can read the + body of what it asked for without checking the status first.""" + r = retry.request_with_retry(session, method, url, attempts=10, timeout=60, **kw) r.raise_for_status() return r @@ -366,33 +328,22 @@ def pack_embeds(embeds): def post_to_discord(webhook_url, messages): - # Use a fresh, unauthenticated session -- the Crowdin Bearer token must never - # be sent to Discord. request_with_retry raises on any non-retryable 4xx, so - # we translate that into the concise failure message here. + # A fresh, unauthenticated session: the Crowdin Bearer token must never be + # sent to Discord. with requests.Session() as webhook_session: - for payload in messages: - try: - request_with_retry(webhook_session, "POST", webhook_url, json=payload) - except requests.exceptions.RequestException as e: - resp = getattr(e, "response", None) - if resp is not None: - detail = f"Discord webhook failed ({resp.status_code}): {resp.text[:300]}" - else: - detail = f"Discord webhook failed: {e}" - # A rich payload can be rejected outright (e.g. an embed exceeded - # Discord's size limits). Before crashing, best-effort post a plain - # warning so the failure is at least visible in the channel; if even - # that fails, fall through to the sys.exit below. - try: - request_with_retry(webhook_session, "POST", webhook_url, json={ - "content": "⚠️ Crowdin multiple-translations report failed to post " - "its results (a message was rejected by Discord). " - "Re-run `report_multiple_translations.py --json` for the " - "full list.", - }) - except requests.exceptions.RequestException: - pass - sys.exit(detail) + posted = discord.post_to_discord(webhook_session, webhook_url, messages) + if posted == len(messages): + return + # A rich payload can be rejected outright, an embed over Discord's size + # limits say. A plain warning at least makes the failure visible in the + # channel; if that fails too, the exit below still says so. + discord.post_to_discord(webhook_session, webhook_url, [{ + "content": "⚠️ Crowdin multiple-translations report failed to post " + "its results (a message was rejected by Discord). " + "Re-run `report_multiple_translations.py --json` for the " + "full list.", + }]) + sys.exit(f"Discord accepted {posted} of {len(messages)} messages.") # --------------------------------------------------------------------------- # diff --git a/crowdin/test_report_multiple_translations.py b/crowdin/test_report_multiple_translations.py new file mode 100644 index 0000000..50a0381 --- /dev/null +++ b/crowdin/test_report_multiple_translations.py @@ -0,0 +1,75 @@ +""" + cd crowdin && python -m unittest discover +""" +import contextlib +import io +import os +import sys +import unittest + +import requests + +sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))) +sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__)))) +import report_multiple_translations as report # noqa: E402 +from shared.testing import FakeResponse, FakeSession, NoSleep, Patched # noqa: E402 + + +class ApiResponse(FakeResponse): + def raise_for_status(self): + if self.status_code >= 400: + raise requests.HTTPError(f"{self.status_code}", response=self) + + +class WebhookSession(FakeSession): + def __enter__(self): + return self + + def __exit__(self, *exc): + return False + + +class TestRequestWithRetry(unittest.TestCase): + def test_a_client_error_raises_rather_than_returning(self): + """Callers read the body straight off the response, so a 4xx has to stop them.""" + with self.assertRaises(requests.HTTPError): + report.request_with_retry(FakeSession([ApiResponse({}, status_code=404)]), + "GET", "https://x") + + def test_crowdins_budget_is_ten_attempts_at_sixty_seconds(self): + session = FakeSession([ApiResponse({}, status_code=503)] * 9 + [ApiResponse({"ok": 1})]) + with NoSleep(): + self.assertEqual(report.request_with_retry(session, "GET", "https://x").json(), {"ok": 1}) + self.assertEqual(len(session.calls), 10) + self.assertEqual({kw["timeout"] for _, _, kw in session.calls}, {60}) + + +class TestPostToDiscord(unittest.TestCase): + def post(self, responses, messages): + session = WebhookSession(responses) + with Patched(report.requests, Session=lambda: session), \ + contextlib.redirect_stdout(io.StringIO()): + report.post_to_discord("https://hook", messages) + return session + + def test_every_message_accepted_posts_nothing_else(self): + session = self.post([FakeResponse({}, status_code=204)] * 2, [{"embeds": []}] * 2) + self.assertEqual(len(session.calls), 2) + + def test_a_rejection_posts_a_plain_warning_then_exits(self): + responses = [FakeResponse({}, status_code=204), FakeResponse({}, status_code=400), + FakeResponse({}, status_code=204)] + with self.assertRaises(SystemExit) as caught: + self.post(responses, [{"embeds": []}] * 3) + self.assertIn("1 of 3", str(caught.exception)) + + def test_the_warning_is_plain_content_the_webhook_cannot_reject_for_size(self): + session = WebhookSession([FakeResponse({}, status_code=400), FakeResponse({}, status_code=204)]) + with Patched(report.requests, Session=lambda: session), \ + contextlib.redirect_stdout(io.StringIO()), self.assertRaises(SystemExit): + report.post_to_discord("https://hook", [{"embeds": [{"title": "x" * 9000}]}]) + self.assertEqual(list(session.calls[1][2]["json"]), ["content"]) + + +if __name__ == "__main__": + unittest.main() From da30f818348310155af023d74af5751f4e775e03 Mon Sep 17 00:00:00 2001 From: Audric Ackermann Date: Fri, 25 Sep 2026 12:46:09 +1000 Subject: [PATCH 010/101] test: pin the digest and Crowdin outputs as goldens over recorded responses Each golden replays a recording of API responses through the unchanged script and compares its output byte for byte, so the packaging and transport changes that follow cannot alter a payload unnoticed. - digest: a live --dry-run over a 720 h window (2026-09-25), trimmed to the fields the digest reads; its replay is byte-identical to the live output. A second case adds a state file, covering the new/changed/unchanged split. - report: --locales de over a synthetic project with duplicates in plain and plural slots, both the Discord payload and the --json findings. - download: the export payloads and the files written, invoked as the sync workflow invokes it. RecordedSession matches requests by method, URL, query and body rather than order, because both Crowdin scripts fan requests out across threads. --- crowdin/test_goldens.py | 75 ++++ github_prs/test_golden.py | 44 ++ shared/testing.py | 98 +++++ tests/goldens/README.md | 22 + tests/goldens/download/output.json | 36 ++ tests/goldens/download/responses.json | 192 +++++++++ tests/goldens/report/dry-run.json | 23 ++ tests/goldens/report/findings.json | 181 +++++++++ tests/goldens/report/responses.json | 562 ++++++++++++++++++++++++++ 9 files changed, 1233 insertions(+) create mode 100644 crowdin/test_goldens.py create mode 100644 github_prs/test_golden.py create mode 100644 tests/goldens/README.md create mode 100644 tests/goldens/download/output.json create mode 100644 tests/goldens/download/responses.json create mode 100644 tests/goldens/report/dry-run.json create mode 100644 tests/goldens/report/findings.json create mode 100644 tests/goldens/report/responses.json diff --git a/crowdin/test_goldens.py b/crowdin/test_goldens.py new file mode 100644 index 0000000..51d4b51 --- /dev/null +++ b/crowdin/test_goldens.py @@ -0,0 +1,75 @@ +""" + python -m unittest discover # from crowdin/ + UPDATE_GOLDENS=1 python -m unittest test_goldens # accept new output + +The recordings are shaped like Crowdin's API responses rather than captured live. +""" +import contextlib +import io +import json +import os +import runpy +import sys +import tempfile +import unittest +from unittest import mock + +import colorama +import requests + +HERE = os.path.dirname(os.path.abspath(__file__)) +sys.path.insert(0, HERE) +sys.path.insert(0, os.path.dirname(HERE)) +import report_multiple_translations as report # noqa: E402 +from shared.testing import RecordedSession, assert_golden, load_golden_json # noqa: E402 + + +def argv_for(recording, tmp): + return [arg.replace("{tmp}", tmp) for arg in recording["argv"]] + + +class TestReportGolden(unittest.TestCase): + def test_dry_run_for_one_locale(self): + recording = load_golden_json("report/responses.json") + session = RecordedSession(recording["exchanges"]) + out = io.StringIO() + with tempfile.TemporaryDirectory() as tmp: + with mock.patch.object(report.requests, "Session", lambda: session), \ + mock.patch.object(sys, "argv", ["report", *argv_for(recording, tmp)]), \ + contextlib.redirect_stdout(out), contextlib.redirect_stderr(io.StringIO()): + report.main() + with open(os.path.join(tmp, "findings.json"), encoding="utf-8") as handle: + findings = handle.read() + assert_golden(self, "report/dry-run.json", out.getvalue()) + assert_golden(self, "report/findings.json", findings + "\n") + + +class TestDownloadGolden(unittest.TestCase): + def test_export_payloads_and_written_files(self): + recording = load_golden_json("download/responses.json") + session = RecordedSession(recording["exchanges"]) + with tempfile.TemporaryDirectory() as tmp: + with mock.patch.object(requests, "get", session.get), \ + mock.patch.object(requests, "post", session.post), \ + mock.patch.object(sys, "argv", ["download", *argv_for(recording, tmp)]), \ + contextlib.redirect_stdout(io.StringIO()) as out: + try: + runpy.run_path(os.path.join(HERE, "download_translations_from_crowdin.py"), + run_name="__main__") + except SystemExit as exc: + self.fail(f"download exited {exc.code}:\n{out.getvalue()}") + finally: + colorama.deinit() + files = {} + for name in sorted(os.listdir(tmp)): + with open(os.path.join(tmp, name), encoding="utf-8") as handle: + files[name] = handle.read() + exports = sorted((json.loads(data) for method, _, _, data in session.calls + if method == "POST"), key=lambda body: body["targetLanguageId"]) + assert_golden(self, "download/output.json", + json.dumps({"exports": exports, "files": files}, indent=2, + ensure_ascii=False) + "\n") + + +if __name__ == "__main__": + unittest.main() diff --git a/github_prs/test_golden.py b/github_prs/test_golden.py new file mode 100644 index 0000000..62a5468 --- /dev/null +++ b/github_prs/test_golden.py @@ -0,0 +1,44 @@ +""" + python -m unittest discover # from github_prs/ + UPDATE_GOLDENS=1 python -m unittest test_golden # accept a new payload +""" +import contextlib +import io +import os +import sys +import unittest +from datetime import datetime +from unittest import mock + +sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__)))) +import digest # noqa: E402 +from shared.testing import (GOLDENS_DIR, RecordedSession, assert_golden, # noqa: E402 + frozen_datetime, load_golden_json) + + +class TestDigestGolden(unittest.TestCase): + """The dry-run output of a recorded live run over a 720-hour window.""" + + def run_digest(self, *extra): + recording = load_golden_json("digest/responses.json") + session = RecordedSession(recording["exchanges"]) + out = io.StringIO() + with mock.patch.object(digest, "github_session", lambda token: session), \ + mock.patch.object(digest, "datetime", + frozen_datetime(datetime.fromisoformat(recording["now"]))), \ + mock.patch.object(sys, "argv", ["digest.py", *recording["argv"], *extra]), \ + mock.patch.dict(os.environ, {"GITHUB_PRS_TOKEN": "t"}), \ + contextlib.redirect_stdout(out): + digest.main() + return out.getvalue() + + def test_dry_run_matches_the_recorded_run(self): + assert_golden(self, "digest/dry-run.txt", self.run_digest()) + + def test_state_splits_the_window_into_new_changed_and_unchanged(self): + state = os.path.join(GOLDENS_DIR, "digest", "state.json") + assert_golden(self, "digest/dry-run-with-state.txt", self.run_digest("--state", state)) + + +if __name__ == "__main__": + unittest.main() diff --git a/shared/testing.py b/shared/testing.py index c5e3677..277076d 100644 --- a/shared/testing.py +++ b/shared/testing.py @@ -1,6 +1,8 @@ """Fakes for the tests of every script that talks HTTP through `shared`.""" import json +import os import time +from datetime import datetime import requests @@ -86,3 +88,99 @@ def __enter__(self): def __exit__(self, *exc): time.sleep = self._real return False + + +GOLDENS_DIR = os.path.join(os.path.dirname(os.path.dirname(os.path.abspath(__file__))), + "tests", "goldens") + + +def load_golden_json(relpath): + with open(os.path.join(GOLDENS_DIR, relpath), encoding="utf-8") as handle: + return json.load(handle) + + +def assert_golden(case, relpath, actual): + """Compare `actual` with tests/goldens/; UPDATE_GOLDENS=1 rewrites it.""" + path = os.path.join(GOLDENS_DIR, relpath) + if os.environ.get("UPDATE_GOLDENS"): + os.makedirs(os.path.dirname(path), exist_ok=True) + with open(path, "w", encoding="utf-8") as handle: + handle.write(actual) + return + with open(path, encoding="utf-8") as handle: + expected = handle.read() + case.maxDiff = None + case.assertEqual(expected, actual, f"{relpath} differs (UPDATE_GOLDENS=1 to accept)") + + +def request_key(method, url, params=None, data=None, json_body=None): + """What identifies a request in a recording: method, URL, query and body.""" + if json_body is not None: + data = json.dumps(json_body, sort_keys=True) + elif isinstance(data, (str, bytes)): + try: + data = json.dumps(json.loads(data), sort_keys=True) + except ValueError: + data = data.decode() if isinstance(data, bytes) else data + query = sorted((str(k), str(v)) for k, v in (params or {}).items()) + return json.dumps([method.upper(), url, query, data]) + + +class RecordedResponse(FakeResponse): + def __init__(self, recorded): + super().__init__(recorded.get("json"), recorded.get("status", 200)) + if "text" in recorded: + self.text = recorded["text"] + self.content = self.text.encode() + + def json(self): + if self._payload is None: + raise requests.exceptions.JSONDecodeError("Expecting value", self.text, 0) + return self._payload + + def raise_for_status(self): + if self.status_code >= 400: + raise requests.HTTPError(str(self.status_code), response=self) + + def iter_content(self, chunk_size=1): + for start in range(0, len(self.content), chunk_size): + yield self.content[start:start + chunk_size] + + +class RecordedSession: + """Answers each request from a recording, matched by request rather than order. + + Order-independent so that a caller fanning requests across threads replays + deterministically. A request missing from the recording fails the test by name. + """ + + def __init__(self, exchanges): + self.headers = {} + self._responses = { + request_key(ex["method"], ex["url"], ex.get("params"), ex.get("data")): + ex["response"] + for ex in exchanges + } + self.calls = [] + + def request(self, method, url, params=None, data=None, json=None, **kwargs): + self.calls.append((method, url, params, data if json is None else json)) + key = request_key(method, url, params, data, json) + if key not in self._responses: + raise AssertionError(f"request not in the recording: {key}") + return RecordedResponse(self._responses[key]) + + def get(self, url, **kwargs): + return self.request("GET", url, **kwargs) + + def post(self, url, **kwargs): + return self.request("POST", url, **kwargs) + + +def frozen_datetime(now): + """A datetime class whose now() is `now`, for patching over a module's import.""" + class Frozen(datetime): + @classmethod + def now(cls, tz=None): + return now.astimezone(tz) if tz else now.replace(tzinfo=None) + return Frozen diff --git a/tests/goldens/README.md b/tests/goldens/README.md new file mode 100644 index 0000000..82b1c21 --- /dev/null +++ b/tests/goldens/README.md @@ -0,0 +1,22 @@ +# Golden outputs + +Each directory holds what a job printed for a recorded set of API responses. A test replays +`responses.json` through the job and compares its output byte for byte, so a refactor that +changes a payload fails. + +| directory | job | recording | +| ----------- | ------------------------------------------- | ------------------------------------------ | +| `digest/` | `github_prs/digest.py --dry-run`, 720 h | live org, 2026-09-25, trimmed to read fields | +| `report/` | `report_multiple_translations.py --locales de` | synthetic, shaped like Crowdin's API | +| `download/` | `download_translations_from_crowdin.py` as the sync runs it | synthetic | + +Requests are matched by method, URL, query and body rather than by order, because the +Crowdin scripts fan out across threads. A request the recording lacks fails the test and +names it. + +To accept a deliberate change, rerun the suite with `UPDATE_GOLDENS=1` and review the diff: + +```sh +cd github_prs && UPDATE_GOLDENS=1 python -m unittest test_golden +cd crowdin && UPDATE_GOLDENS=1 python -m unittest test_goldens +``` diff --git a/tests/goldens/download/output.json b/tests/goldens/download/output.json new file mode 100644 index 0000000..0f998d4 --- /dev/null +++ b/tests/goldens/download/output.json @@ -0,0 +1,36 @@ +{ + "exports": [ + { + "targetLanguageId": "de", + "format": "xliff", + "skipUntranslatedStrings": true, + "exportApprovedOnly": true + }, + { + "targetLanguageId": "en", + "format": "xliff", + "skipUntranslatedStrings": false, + "exportApprovedOnly": false + }, + { + "targetLanguageId": "es-ES", + "format": "xliff", + "skipUntranslatedStrings": true, + "exportApprovedOnly": true + }, + { + "targetLanguageId": "fr", + "format": "xliff", + "skipUntranslatedStrings": true, + "exportApprovedOnly": true + } + ], + "files": { + "_non_translatable_strings.json": "{\n \"data\": [\n {\n \"data\": {\n \"id\": 7,\n \"text\": \"Session\",\n \"conceptId\": 36\n }\n },\n {\n \"data\": {\n \"id\": 8,\n \"text\": \"Lokinet\",\n \"conceptId\": 36\n }\n }\n ],\n \"pagination\": {\n \"offset\": 0,\n \"limit\": 500\n }\n}", + "_project_info.json": "{\n \"data\": {\n \"id\": 618696,\n \"identifier\": \"session-crossplatform-strings\",\n \"sourceLanguageId\": \"en\",\n \"sourceLanguage\": {\n \"id\": \"en\",\n \"name\": \"English\",\n \"editorCode\": \"en\",\n \"locale\": \"en-US\"\n },\n \"targetLanguageIds\": [\n \"de\",\n \"fr\",\n \"es-ES\"\n ],\n \"targetLanguages\": [\n {\n \"id\": \"de\",\n \"name\": \"German\",\n \"editorCode\": \"de\",\n \"locale\": \"de-DE\"\n },\n {\n \"id\": \"fr\",\n \"name\": \"French\",\n \"editorCode\": \"fr\",\n \"locale\": \"fr-FR\"\n },\n {\n \"id\": \"es-ES\",\n \"name\": \"Spanish\",\n \"editorCode\": \"es\",\n \"locale\": \"es-ES\"\n }\n ]\n }\n}", + "de-DE.xliff": "\n\n", + "en-US.xliff": "\n\n", + "es-ES.xliff": "\n\n", + "fr-FR.xliff": "\n\n" + } +} diff --git a/tests/goldens/download/responses.json b/tests/goldens/download/responses.json new file mode 100644 index 0000000..11ffb72 --- /dev/null +++ b/tests/goldens/download/responses.json @@ -0,0 +1,192 @@ +{ + "argv": [ + "t", + "618696", + "{tmp}", + "--glossary_id", + "407522", + "--concept_id", + "36", + "--skip-untranslated-strings" + ], + "exchanges": [ + { + "method": "GET", + "url": "https://api.crowdin.com/api/v2/projects/618696", + "params": null, + "data": null, + "response": { + "status": 200, + "json": { + "data": { + "id": 618696, + "identifier": "session-crossplatform-strings", + "sourceLanguageId": "en", + "sourceLanguage": { + "id": "en", + "name": "English", + "editorCode": "en", + "locale": "en-US" + }, + "targetLanguageIds": [ + "de", + "fr", + "es-ES" + ], + "targetLanguages": [ + { + "id": "de", + "name": "German", + "editorCode": "de", + "locale": "de-DE" + }, + { + "id": "fr", + "name": "French", + "editorCode": "fr", + "locale": "fr-FR" + }, + { + "id": "es-ES", + "name": "Spanish", + "editorCode": "es", + "locale": "es-ES" + } + ] + } + } + } + }, + { + "method": "POST", + "url": "https://api.crowdin.com/api/v2/projects/618696/translations/exports", + "params": null, + "data": "{\"targetLanguageId\": \"en\", \"format\": \"xliff\", \"skipUntranslatedStrings\": false, \"exportApprovedOnly\": false}", + "response": { + "status": 200, + "json": { + "data": { + "url": "https://production-enterprise-exports.example/en.xliff?sig=x", + "expireIn": "2026-09-25T12:00:00+00:00" + } + } + } + }, + { + "method": "GET", + "url": "https://production-enterprise-exports.example/en.xliff?sig=x", + "params": null, + "data": null, + "response": { + "status": 200, + "text": "\n\n" + } + }, + { + "method": "POST", + "url": "https://api.crowdin.com/api/v2/projects/618696/translations/exports", + "params": null, + "data": "{\"targetLanguageId\": \"de\", \"format\": \"xliff\", \"skipUntranslatedStrings\": true, \"exportApprovedOnly\": true}", + "response": { + "status": 200, + "json": { + "data": { + "url": "https://production-enterprise-exports.example/de.xliff?sig=x", + "expireIn": "2026-09-25T12:00:00+00:00" + } + } + } + }, + { + "method": "GET", + "url": "https://production-enterprise-exports.example/de.xliff?sig=x", + "params": null, + "data": null, + "response": { + "status": 200, + "text": "\n\n" + } + }, + { + "method": "POST", + "url": "https://api.crowdin.com/api/v2/projects/618696/translations/exports", + "params": null, + "data": "{\"targetLanguageId\": \"fr\", \"format\": \"xliff\", \"skipUntranslatedStrings\": true, \"exportApprovedOnly\": true}", + "response": { + "status": 200, + "json": { + "data": { + "url": "https://production-enterprise-exports.example/fr.xliff?sig=x", + "expireIn": "2026-09-25T12:00:00+00:00" + } + } + } + }, + { + "method": "GET", + "url": "https://production-enterprise-exports.example/fr.xliff?sig=x", + "params": null, + "data": null, + "response": { + "status": 200, + "text": "\n\n" + } + }, + { + "method": "POST", + "url": "https://api.crowdin.com/api/v2/projects/618696/translations/exports", + "params": null, + "data": "{\"targetLanguageId\": \"es-ES\", \"format\": \"xliff\", \"skipUntranslatedStrings\": true, \"exportApprovedOnly\": true}", + "response": { + "status": 200, + "json": { + "data": { + "url": "https://production-enterprise-exports.example/es-ES.xliff?sig=x", + "expireIn": "2026-09-25T12:00:00+00:00" + } + } + } + }, + { + "method": "GET", + "url": "https://production-enterprise-exports.example/es-ES.xliff?sig=x", + "params": null, + "data": null, + "response": { + "status": 200, + "text": "\n\n" + } + }, + { + "method": "GET", + "url": "https://api.crowdin.com/api/v2/glossaries/407522/terms?conceptId=36&limit=500", + "params": null, + "data": null, + "response": { + "status": 200, + "json": { + "data": [ + { + "data": { + "id": 7, + "text": "Session", + "conceptId": 36 + } + }, + { + "data": { + "id": 8, + "text": "Lokinet", + "conceptId": 36 + } + } + ], + "pagination": { + "offset": 0, + "limit": 500 + } + } + } + } + ] +} \ No newline at end of file diff --git a/tests/goldens/report/dry-run.json b/tests/goldens/report/dry-run.json new file mode 100644 index 0000000..e4791bf --- /dev/null +++ b/tests/goldens/report/dry-run.json @@ -0,0 +1,23 @@ +[ + { + "embeds": [ + { + "title": "🈳 Crowdin: strings with multiple translations", + "description": "Scanned **1** locale(s). Found **5** string/plural slot(s) with 2+ translations across **1** locale(s).\nEach slot needs one translation chosen as the keeper; the rest should be deleted so exactly one translation (per plural form) is exported.", + "color": 15105570, + "fields": [ + { + "name": "Slots per locale", + "value": "`de` β€” **5**", + "inline": false + } + ] + }, + { + "title": "de β€” 5 slot(s)", + "description": "β€’ [string 107](https://crowdin.com/editor/session-crossplatform-strings/all/en-de#107) β€” **2** translations\nβ€’ [accept](https://crowdin.com/editor/session-crossplatform-strings/all/en-de#101) β€” **2** translations\nβ€’ [deleteMessage](https://crowdin.com/editor/session-crossplatform-strings/all/en-de#105) `[one]` β€” **2** translations\nβ€’ [deleteMessage](https://crowdin.com/editor/session-crossplatform-strings/all/en-de#105) β€” **3** translations\nβ€’ [messageNew](https://crowdin.com/editor/session-crossplatform-strings/all/en-de#104) β€” **2** translations", + "color": 15105570 + } + ] + } +] diff --git a/tests/goldens/report/findings.json b/tests/goldens/report/findings.json new file mode 100644 index 0000000..f0dec74 --- /dev/null +++ b/tests/goldens/report/findings.json @@ -0,0 +1,181 @@ +[ + { + "locale": "de", + "status": "multiple-translations", + "stringId": 107, + "identifier": null, + "webUrl": "https://crowdin.com/editor/session-crossplatform-strings/all/en-de#107", + "pluralCategory": null, + "count": 2, + "sourceText": "A string with no identifier and a long source text that goes on and on", + "translations": [ + { + "translationId": 14, + "user": "1:alice", + "createdAt": "2026-09-01T10:00:00+00:00", + "approved": false, + "rating": 0, + "isPreTranslated": false, + "provider": null, + "text": "Eins" + }, + { + "translationId": 15, + "user": "2:Bob B", + "createdAt": "2026-09-01T10:00:00+00:00", + "approved": false, + "rating": 0, + "isPreTranslated": false, + "provider": null, + "text": "Zwei" + } + ] + }, + { + "locale": "de", + "status": "multiple-translations", + "stringId": 101, + "identifier": "accept", + "webUrl": "https://crowdin.com/editor/session-crossplatform-strings/all/en-de#101", + "pluralCategory": null, + "count": 2, + "sourceText": "Accept", + "translations": [ + { + "translationId": 2, + "user": "2:Bob B", + "createdAt": "2026-09-01T10:00:00+00:00", + "approved": false, + "rating": 2, + "isPreTranslated": false, + "provider": null, + "text": "Annehmen" + }, + { + "translationId": 1, + "user": "1:alice", + "createdAt": "2026-09-02T10:00:00+00:00", + "approved": true, + "rating": 0, + "isPreTranslated": false, + "provider": null, + "text": "Akzeptieren" + } + ] + }, + { + "locale": "de", + "status": "multiple-translations", + "stringId": 105, + "identifier": "deleteMessage", + "webUrl": "https://crowdin.com/editor/session-crossplatform-strings/all/en-de#105", + "pluralCategory": "one", + "count": 2, + "sourceText": { + "one": "Delete message", + "other": "Delete messages" + }, + "translations": [ + { + "translationId": 9, + "user": "1:alice", + "createdAt": "2026-09-01T10:00:00+00:00", + "approved": false, + "rating": 0, + "isPreTranslated": false, + "provider": null, + "text": "Nachricht lΓΆschen" + }, + { + "translationId": 10, + "user": "2:Bob B", + "createdAt": "2026-09-01T10:00:00+00:00", + "approved": false, + "rating": 0, + "isPreTranslated": false, + "provider": null, + "text": "Nachricht entfernen" + } + ] + }, + { + "locale": "de", + "status": "multiple-translations", + "stringId": 105, + "identifier": "deleteMessage", + "webUrl": "https://crowdin.com/editor/session-crossplatform-strings/all/en-de#105", + "pluralCategory": "other", + "count": 3, + "sourceText": { + "one": "Delete message", + "other": "Delete messages" + }, + "translations": [ + { + "translationId": 11, + "user": "1:alice", + "createdAt": "2026-09-01T10:00:00+00:00", + "approved": true, + "rating": 0, + "isPreTranslated": false, + "provider": null, + "text": "Nachrichten lΓΆschen" + }, + { + "translationId": 12, + "user": "2:Bob B", + "createdAt": "2026-09-01T10:00:00+00:00", + "approved": false, + "rating": 0, + "isPreTranslated": false, + "provider": null, + "text": "Nachrichten entfernen" + }, + { + "translationId": 13, + "user": "2:Bob B", + "createdAt": "2026-09-04T10:00:00+00:00", + "approved": false, + "rating": 0, + "isPreTranslated": false, + "provider": null, + "text": "Nachrichten wegmachen" + } + ] + }, + { + "locale": "de", + "status": "multiple-translations", + "stringId": 104, + "identifier": "messageNew", + "webUrl": "https://crowdin.com/editor/session-crossplatform-strings/all/en-de#104", + "pluralCategory": "other", + "count": 2, + "sourceText": { + "one": "New message", + "other": "New messages" + }, + "translations": [ + { + "translationId": 7, + "user": "1:alice", + "createdAt": "2026-09-01T10:00:00+00:00", + "approved": false, + "rating": 0, + "isPreTranslated": false, + "provider": null, + "text": "Neue Nachrichten" + }, + { + "translationId": 8, + "user": "", + "createdAt": "2026-09-03T10:00:00+00:00", + "approved": false, + "rating": 0, + "isPreTranslated": true, + "provider": "mt", + "text": "Neue Mitteilungen" + } + ] + } +] diff --git a/tests/goldens/report/responses.json b/tests/goldens/report/responses.json new file mode 100644 index 0000000..a38e6e3 --- /dev/null +++ b/tests/goldens/report/responses.json @@ -0,0 +1,562 @@ +{ + "argv": [ + "--api-token", + "t", + "--locales", + "de", + "--dry-run", + "--json", + "{tmp}/findings.json" + ], + "exchanges": [ + { + "method": "GET", + "url": "https://api.crowdin.com/api/v2/projects/618696", + "params": null, + "data": null, + "response": { + "status": 200, + "json": { + "data": { + "id": 618696, + "identifier": "session-crossplatform-strings", + "sourceLanguageId": "en", + "sourceLanguage": { + "id": "en", + "name": "English", + "editorCode": "en", + "locale": "en-US" + }, + "targetLanguageIds": [ + "de", + "fr", + "es-ES" + ], + "targetLanguages": [ + { + "id": "de", + "name": "German", + "editorCode": "de", + "locale": "de-DE" + }, + { + "id": "fr", + "name": "French", + "editorCode": "fr", + "locale": "fr-FR" + }, + { + "id": "es-ES", + "name": "Spanish", + "editorCode": "es", + "locale": "es-ES" + } + ] + } + } + } + }, + { + "method": "GET", + "url": "https://api.crowdin.com/api/v2/projects/618696/strings", + "params": { + "limit": 500, + "offset": 0 + }, + "data": null, + "response": { + "status": 200, + "json": { + "data": [ + { + "data": { + "id": 101, + "identifier": "accept", + "text": "Accept" + } + }, + { + "data": { + "id": 102, + "identifier": "attachmentsSaveError", + "text": "Unable to save file." + } + }, + { + "data": { + "id": 103, + "identifier": "memberCount", + "text": { + "one": "{count} member", + "other": "{count} members" + } + } + }, + { + "data": { + "id": 104, + "identifier": "messageNew", + "text": { + "one": "New message", + "other": "New messages" + } + } + }, + { + "data": { + "id": 105, + "identifier": "deleteMessage", + "text": { + "one": "Delete message", + "other": "Delete messages" + } + } + }, + { + "data": { + "id": 106, + "identifier": "untranslatedYet", + "text": "Nothing here yet" + } + }, + { + "data": { + "id": 107, + "identifier": null, + "text": "A string with no identifier and a long source text that goes on and on" + } + } + ], + "pagination": { + "offset": 0, + "limit": 500 + } + } + } + }, + { + "method": "GET", + "url": "https://api.crowdin.com/api/v2/projects/618696/approvals", + "params": { + "languageId": "de", + "limit": 500, + "offset": 0 + }, + "data": null, + "response": { + "status": 200, + "json": { + "data": [ + { + "data": { + "id": 900, + "stringId": 101, + "translationId": 1 + } + }, + { + "data": { + "id": 901, + "stringId": 105, + "translationId": 11 + } + } + ], + "pagination": { + "offset": 0, + "limit": 500 + } + } + } + }, + { + "method": "GET", + "url": "https://api.crowdin.com/api/v2/projects/618696/translations", + "params": { + "stringId": 101, + "languageId": "de", + "limit": 500, + "offset": 0 + }, + "data": null, + "response": { + "status": 200, + "json": { + "data": [ + { + "data": { + "id": 1, + "text": "Akzeptieren", + "pluralCategoryName": null, + "user": { + "id": 1, + "username": "alice", + "fullName": "Alice A" + }, + "rating": 0, + "isPreTranslated": false, + "provider": null, + "createdAt": "2026-09-02T10:00:00+00:00" + } + }, + { + "data": { + "id": 2, + "text": "Annehmen", + "pluralCategoryName": null, + "user": { + "id": 2, + "username": null, + "fullName": "Bob B" + }, + "rating": 2, + "isPreTranslated": false, + "provider": null, + "createdAt": "2026-09-01T10:00:00+00:00" + } + } + ], + "pagination": { + "offset": 0, + "limit": 500 + } + } + } + }, + { + "method": "GET", + "url": "https://api.crowdin.com/api/v2/projects/618696/translations", + "params": { + "stringId": 102, + "languageId": "de", + "limit": 500, + "offset": 0 + }, + "data": null, + "response": { + "status": 200, + "json": { + "data": [ + { + "data": { + "id": 3, + "text": "Datei konnte nicht gespeichert werden.", + "pluralCategoryName": null, + "user": { + "id": 1, + "username": "alice", + "fullName": "Alice A" + }, + "rating": 0, + "isPreTranslated": false, + "provider": null, + "createdAt": "2026-09-01T10:00:00+00:00" + } + } + ], + "pagination": { + "offset": 0, + "limit": 500 + } + } + } + }, + { + "method": "GET", + "url": "https://api.crowdin.com/api/v2/projects/618696/translations", + "params": { + "stringId": 103, + "languageId": "de", + "limit": 500, + "offset": 0 + }, + "data": null, + "response": { + "status": 200, + "json": { + "data": [ + { + "data": { + "id": 4, + "text": "{count} Mitglied", + "pluralCategoryName": "one", + "user": { + "id": 1, + "username": "alice", + "fullName": "Alice A" + }, + "rating": 0, + "isPreTranslated": false, + "provider": null, + "createdAt": "2026-09-01T10:00:00+00:00" + } + }, + { + "data": { + "id": 5, + "text": "{count} Mitglieder", + "pluralCategoryName": "other", + "user": { + "id": 1, + "username": "alice", + "fullName": "Alice A" + }, + "rating": 0, + "isPreTranslated": false, + "provider": null, + "createdAt": "2026-09-01T10:00:00+00:00" + } + } + ], + "pagination": { + "offset": 0, + "limit": 500 + } + } + } + }, + { + "method": "GET", + "url": "https://api.crowdin.com/api/v2/projects/618696/translations", + "params": { + "stringId": 104, + "languageId": "de", + "limit": 500, + "offset": 0 + }, + "data": null, + "response": { + "status": 200, + "json": { + "data": [ + { + "data": { + "id": 6, + "text": "Neue Nachricht", + "pluralCategoryName": "one", + "user": { + "id": 1, + "username": "alice", + "fullName": "Alice A" + }, + "rating": 0, + "isPreTranslated": false, + "provider": null, + "createdAt": "2026-09-01T10:00:00+00:00" + } + }, + { + "data": { + "id": 7, + "text": "Neue Nachrichten", + "pluralCategoryName": "other", + "user": { + "id": 1, + "username": "alice", + "fullName": "Alice A" + }, + "rating": 0, + "isPreTranslated": false, + "provider": null, + "createdAt": "2026-09-01T10:00:00+00:00" + } + }, + { + "data": { + "id": 8, + "text": "Neue Mitteilungen", + "pluralCategoryName": "other", + "user": null, + "rating": 0, + "isPreTranslated": true, + "provider": "mt", + "createdAt": "2026-09-03T10:00:00+00:00" + } + } + ], + "pagination": { + "offset": 0, + "limit": 500 + } + } + } + }, + { + "method": "GET", + "url": "https://api.crowdin.com/api/v2/projects/618696/translations", + "params": { + "stringId": 105, + "languageId": "de", + "limit": 500, + "offset": 0 + }, + "data": null, + "response": { + "status": 200, + "json": { + "data": [ + { + "data": { + "id": 9, + "text": "Nachricht lΓΆschen", + "pluralCategoryName": "one", + "user": { + "id": 1, + "username": "alice", + "fullName": "Alice A" + }, + "rating": 0, + "isPreTranslated": false, + "provider": null, + "createdAt": "2026-09-01T10:00:00+00:00" + } + }, + { + "data": { + "id": 10, + "text": "Nachricht entfernen", + "pluralCategoryName": "one", + "user": { + "id": 2, + "username": null, + "fullName": "Bob B" + }, + "rating": 0, + "isPreTranslated": false, + "provider": null, + "createdAt": "2026-09-01T10:00:00+00:00" + } + }, + { + "data": { + "id": 11, + "text": "Nachrichten lΓΆschen", + "pluralCategoryName": "other", + "user": { + "id": 1, + "username": "alice", + "fullName": "Alice A" + }, + "rating": 0, + "isPreTranslated": false, + "provider": null, + "createdAt": "2026-09-01T10:00:00+00:00" + } + }, + { + "data": { + "id": 12, + "text": "Nachrichten entfernen", + "pluralCategoryName": "other", + "user": { + "id": 2, + "username": null, + "fullName": "Bob B" + }, + "rating": 0, + "isPreTranslated": false, + "provider": null, + "createdAt": "2026-09-01T10:00:00+00:00" + } + }, + { + "data": { + "id": 13, + "text": "Nachrichten wegmachen", + "pluralCategoryName": "other", + "user": { + "id": 2, + "username": null, + "fullName": "Bob B" + }, + "rating": 0, + "isPreTranslated": false, + "provider": null, + "createdAt": "2026-09-04T10:00:00+00:00" + } + } + ], + "pagination": { + "offset": 0, + "limit": 500 + } + } + } + }, + { + "method": "GET", + "url": "https://api.crowdin.com/api/v2/projects/618696/translations", + "params": { + "stringId": 106, + "languageId": "de", + "limit": 500, + "offset": 0 + }, + "data": null, + "response": { + "status": 200, + "json": { + "data": [], + "pagination": { + "offset": 0, + "limit": 500 + } + } + } + }, + { + "method": "GET", + "url": "https://api.crowdin.com/api/v2/projects/618696/translations", + "params": { + "stringId": 107, + "languageId": "de", + "limit": 500, + "offset": 0 + }, + "data": null, + "response": { + "status": 200, + "json": { + "data": [ + { + "data": { + "id": 14, + "text": "Eins", + "pluralCategoryName": null, + "user": { + "id": 1, + "username": "alice", + "fullName": "Alice A" + }, + "rating": 0, + "isPreTranslated": false, + "provider": null, + "createdAt": "2026-09-01T10:00:00+00:00" + } + }, + { + "data": { + "id": 15, + "text": "Zwei", + "pluralCategoryName": null, + "user": { + "id": 2, + "username": null, + "fullName": "Bob B" + }, + "rating": 0, + "isPreTranslated": false, + "provider": null, + "createdAt": "2026-09-01T10:00:00+00:00" + } + } + ], + "pagination": { + "offset": 0, + "limit": 500 + } + } + } + } + ] +} \ No newline at end of file From c7aff0b1909c569972d9e2a7b87cf25b03a6ebdc Mon Sep 17 00:00:00 2001 From: Audric Ackermann Date: Fri, 25 Sep 2026 12:46:10 +1000 Subject: [PATCH 011/101] ci: run every unittest suite and ruff on pull requests One job per suite, since two requirements files pin requests differently. sogs_moderation runs on the system interpreter with a venv that can see python3-session-util, which ships as a deb rather than a wheel. ruff is limited to pyflakes and syntax errors, the class of mistake a large refactor introduces. The one finding, an unused import in relay.py, is removed. --- .github/workflows/tests.yml | 85 +++++++++++++++++++++++++++++++++++++ ruff.toml | 5 +++ zendesk_triage/relay.py | 3 -- 3 files changed, 90 insertions(+), 3 deletions(-) create mode 100644 .github/workflows/tests.yml create mode 100644 ruff.toml diff --git a/.github/workflows/tests.yml b/.github/workflows/tests.yml new file mode 100644 index 0000000..1f8d167 --- /dev/null +++ b/.github/workflows/tests.yml @@ -0,0 +1,85 @@ +name: Tests + +on: + push: + branches: [main] + pull_request: + +permissions: + contents: read + +jobs: + lint: + runs-on: ubuntu-latest + timeout-minutes: 5 + steps: + - uses: actions/checkout@v7 + + - uses: actions/setup-python@v6 + with: + python-version: "3.12" + + - run: pip install ruff==0.16.9 + + - run: ruff check --output-format=github . + + unittest: + runs-on: ubuntu-latest + timeout-minutes: 10 + strategy: + fail-fast: false + matrix: + include: + - suite: shared + requirements: github_prs/requirements.txt + - suite: github_prs + requirements: github_prs/requirements.txt + - suite: zendesk_triage + requirements: zendesk_triage/requirements.txt zendesk_triage/requirements-dev.txt + - suite: deploy + requirements: github_prs/requirements.txt + - suite: crowdin + requirements: crowdin/requirements.txt + steps: + - uses: actions/checkout@v7 + + - uses: actions/setup-python@v6 + with: + python-version: "3.12" + cache: pip + cache-dependency-path: "**/requirements*.txt" + + - name: Install dependencies + run: | + for file in ${{ matrix.requirements }}; do + pip install -r "$file" + done + + - name: Run ${{ matrix.suite }} tests + working-directory: ${{ matrix.suite }} + run: python -m unittest discover -v + + sogs_moderation: + runs-on: ubuntu-latest + timeout-minutes: 10 + steps: + - uses: actions/checkout@v7 + + # session_util is published as a deb built against the system interpreter, so + # this suite runs on that interpreter with a venv that can see it. + - name: Install python3-session-util + run: | + sudo curl -so /usr/share/keyrings/session-foundation.gpg https://deb.session.foundation/pub.gpg + printf 'Types: deb\nURIs: https://deb.session.foundation\nSuites: %s\nComponents: main\nSigned-By: /usr/share/keyrings/session-foundation.gpg\n' \ + "$(lsb_release -sc)" | sudo tee /etc/apt/sources.list.d/session.sources + sudo apt-get update + sudo apt-get install -y python3-session-util + + - name: Install dependencies + run: | + python3 -m venv --system-site-packages .venv + .venv/bin/pip install -r sogs_moderation/requirements.txt + + - name: Run sogs_moderation tests + working-directory: sogs_moderation + run: ../.venv/bin/python -m unittest discover -v diff --git a/ruff.toml b/ruff.toml new file mode 100644 index 0000000..7ffe240 --- /dev/null +++ b/ruff.toml @@ -0,0 +1,5 @@ +# Pyflakes and syntax errors only: what catches a broken import or name mid-refactor. +target-version = "py312" + +[lint] +select = ["F", "E9"] diff --git a/zendesk_triage/relay.py b/zendesk_triage/relay.py index 75196fd..2a29120 100644 --- a/zendesk_triage/relay.py +++ b/zendesk_triage/relay.py @@ -54,9 +54,6 @@ from fastapi import BackgroundTasks, FastAPI, Request, Response from starlette.concurrency import run_in_threadpool -sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))) -import triage # noqa: E402 (needs the path insert above) - NOTE_SCRIPT = os.path.join(os.path.dirname(os.path.abspath(__file__)), "note_reply.py") # How stale a signed request may be. The signature covers the timestamp, so this From f0ca8c233a1d821988df6ca17812b63aecc01760 Mon Sep 17 00:00:00 2001 From: Audric Ackermann Date: Fri, 25 Sep 2026 12:49:44 +1000 Subject: [PATCH 012/101] feat: alert when a scheduled job stops succeeding, not only when it fails OnFailure= reports a run that failed and nothing about one that never happened: a timer left disabled, a unit renamed, a host down through a whole schedule. Each digest unit now touches /var/lib/session-ops/stamps/ from a `+` ExecStartPost=, which a oneshot runs only after every ExecStart= succeeded and which runs outside the sandbox, so neither job gains a write path. silence.py, on an hourly timer under an account of its own, compares each stamp's age with deploy/jobs.toml and posts one message for every job past its max_age_hours, repeated daily while it stays quiet. A missing stamp is timed from the first check that found it missing, so installing the checker alerts on nothing. The stamps are root's and world-readable, so the checker needs no access to the Zendesk state directory and the ticket data in it. test_silence.py fails when a shipped timer has no registry entry or no stamp line. --- deploy/README.md | 56 ++++++++- deploy/github-prs-digest.service | 3 + deploy/jobs.toml | 15 +++ deploy/session-ops-alert@.service | 17 +++ deploy/session-ops-silence.service | 30 +++++ deploy/session-ops-silence.timer | 10 ++ deploy/session-ops.tmpfiles | 3 + deploy/silence.py | 178 +++++++++++++++++++++++++++++ deploy/test_silence.py | 100 ++++++++++++++++ deploy/zendesk-digest.service | 3 + 10 files changed, 413 insertions(+), 2 deletions(-) create mode 100644 deploy/jobs.toml create mode 100644 deploy/session-ops-alert@.service create mode 100644 deploy/session-ops-silence.service create mode 100644 deploy/session-ops-silence.timer create mode 100644 deploy/session-ops.tmpfiles create mode 100644 deploy/silence.py create mode 100644 deploy/test_silence.py diff --git a/deploy/README.md b/deploy/README.md index 4adf7b6..c96b7a7 100644 --- a/deploy/README.md +++ b/deploy/README.md @@ -8,9 +8,18 @@ digest, all on one machine. | `zendesk-relay.service` | Always on. The HTTPS endpoint Zendesk posts note webhooks to. | | `zendesk-digest.timer` β†’ `.service` | Weekday mornings. Resolves positive reviews, then posts the digest. | | `github-prs-digest.timer` β†’ `.service` | Weekday mornings. Posts the contributor pull request digest. | +| `session-ops-silence.timer` β†’ `.service` | Hourly. Alerts when a job in `jobs.toml` has not succeeded within its `max_age_hours`. | -`zendesk-alert@.service` and `github-prs-alert@.service` are pulled in by `OnFailure=` -and report the failed unit to the channel that job posts to. +`zendesk-alert@.service`, `github-prs-alert@.service` and `session-ops-alert@.service` +are pulled in by `OnFailure=` and report the failed unit to the channel that job posts +to. + +`OnFailure=` only sees a run that failed. A run that never happened (a timer left +disabled, a host down through a whole schedule) is what the silence checker is for: +every scheduled unit touches `/var/lib/session-ops/stamps/` on success, and +`silence.py` compares each stamp's age with `jobs.toml`. A new scheduled job needs both +its `ExecStartPost=` line and a `jobs.toml` entry; `test_silence.py` fails without +either. One clone at `/opt/zendesk` holds all of it β€” the directory is named after its first tenant, not its contents. The venvs are separate, because the two jobs pin `requests` @@ -144,6 +153,29 @@ run out of it. Nothing secret lives there β€” every secret is under `/etc`. creates it with the right owner on first start. It holds the dedup state that keeps the 72-hour window from re-reporting the same PR every weekday morning. +### Adding the silence checker + +Its own account, which reads the stamps and nothing else: they are root's and +world-readable, so the account needs no access to any job's state. + +```bash +useradd --system --no-create-home --home /nonexistent --shell /usr/sbin/nologin sessionops + +install -d -m 750 -o root -g sessionops /etc/session-ops +[ -e /etc/session-ops/env ] || install -m 640 -o root -g sessionops /dev/null /etc/session-ops/env +"${EDITOR:-nano}" /etc/session-ops/env # contents under Secrets, below + +cp /opt/zendesk/deploy/session-ops.tmpfiles /etc/tmpfiles.d/session-ops.conf +systemd-tmpfiles --create session-ops.conf +cp /opt/zendesk/deploy/*.service /opt/zendesk/deploy/*.timer /etc/systemd/system/ +systemctl daemon-reload +systemctl enable --now session-ops-silence.timer +``` + +The `cp` of every unit is the point: the two digests gain the `ExecStartPost=` that +writes their stamp. Until each has run once, the checker counts its silence from the +first check that found the stamp missing, so it does not alert on install. + ## Secrets `/etc/zendesk/env`, mode `640`, `root:zendesk` β€” readable by the service, not by @@ -240,6 +272,15 @@ GITHUB_PRS_DISCORD_WEBHOOK_URL= ALERT_DISCORD_WEBHOOK_URL= ``` +### `/etc/session-ops/env` + +Mode `640`, `root:sessionops`. + +```sh +# Where silence alerts go. Any channel whose readers can act on a job that stopped. +ALERT_DISCORD_WEBHOOK_URL= +``` + ## Verifying, in order **1. Locally, before Zendesk knows the address.** An unsigned request must be refused: @@ -365,6 +406,17 @@ A second run that reports everything again means the state was not written β€” c `StateDirectory=` reached systemd with `systemctl show github-prs-digest -p StateDirectory`. +**7. The silence checker.** A dry run prints every job's last success and the alert +it would post, and writes no state: + +```bash +systemd-run --pipe --wait --uid=sessionops -p EnvironmentFile=/etc/session-ops/env \ + /opt/github-prs/venv/bin/python /opt/zendesk/deploy/silence.py --dry-run +ls -l /var/lib/session-ops/stamps/ # one file per job that has succeeded since +``` + +`systemctl start session-ops-alert@test.service` checks its failure path. + ## Updating ```bash diff --git a/deploy/github-prs-digest.service b/deploy/github-prs-digest.service index 0de28b9..b940c63 100644 --- a/deploy/github-prs-digest.service +++ b/deploy/github-prs-digest.service @@ -14,6 +14,9 @@ EnvironmentFile=/etc/github-prs/env ExecStart=/opt/github-prs/venv/bin/python digest.py --window-hours 72 \ --state /var/lib/github-prs/seen.json +# The silence checker's evidence that this ran (deploy/silence.py). `+` runs it +# outside the sandbox, so the job itself gets no new write path. +ExecStartPost=+/usr/bin/touch /var/lib/session-ops/stamps/github-prs-digest # A dozen API calls. Anything near this is GitHub rate-limiting the run rather than # a large result set. diff --git a/deploy/jobs.toml b/deploy/jobs.toml new file mode 100644 index 0000000..2210c2a --- /dev/null +++ b/deploy/jobs.toml @@ -0,0 +1,15 @@ +# Every scheduled job on this host. `name` is the unit's name without .service/.timer, +# and the stamp its ExecStartPost= touches under /var/lib/session-ops/stamps/. +# +# max_age_hours is the longest a healthy schedule goes between successes, plus slack +# for RandomizedDelaySec and the run itself. For a weekday job that is the weekend. + +[[job]] +name = "github-prs-digest" +# Mon..Fri 09:30: Friday's run to Monday's is 72 h. +max_age_hours = 80 + +[[job]] +name = "zendesk-digest" +# Mon..Fri 10:00, and a full backlog run can take 90 min. +max_age_hours = 80 diff --git a/deploy/session-ops-alert@.service b/deploy/session-ops-alert@.service new file mode 100644 index 0000000..8e14507 --- /dev/null +++ b/deploy/session-ops-alert@.service @@ -0,0 +1,17 @@ +[Unit] +Description=Report a failed silence check to Discord +Documentation=https://github.com/session-foundation/session-shared-scripts + +[Service] +Type=oneshot +User=sessionops +Group=sessionops +EnvironmentFile=/etc/session-ops/env +ExecStart=/opt/github-prs/venv/bin/python /opt/zendesk/deploy/alert.py %i +# See zendesk-alert@.service: the failed unit's journal is the line the alert quotes. +SupplementaryGroups=systemd-journal + +NoNewPrivileges=yes +PrivateTmp=yes +ProtectSystem=strict +ProtectHome=yes diff --git a/deploy/session-ops-silence.service b/deploy/session-ops-silence.service new file mode 100644 index 0000000..e359803 --- /dev/null +++ b/deploy/session-ops-silence.service @@ -0,0 +1,30 @@ +[Unit] +Description=Alert when a scheduled job has stopped succeeding +Documentation=https://github.com/session-foundation/session-shared-scripts +After=network-online.target +Wants=network-online.target +OnFailure=session-ops-alert@%n.service + +[Service] +Type=oneshot +User=sessionops +Group=sessionops +EnvironmentFile=/etc/session-ops/env +ExecStart=/opt/github-prs/venv/bin/python /opt/zendesk/deploy/silence.py \ + --state /var/lib/session-ops/silence/state.json +TimeoutStartSec=5min + +NoNewPrivileges=yes +PrivateTmp=yes +PrivateDevices=yes +ProtectSystem=strict +ProtectHome=yes +ProtectKernelTunables=yes +ProtectKernelModules=yes +ProtectControlGroups=yes +RestrictAddressFamilies=AF_INET AF_INET6 +RestrictNamespaces=yes +LockPersonality=yes +# The stamps it reads are root's and world-readable; the only thing it writes is +# when it last alerted, and when it first found each stamp missing. +StateDirectory=session-ops/silence diff --git a/deploy/session-ops-silence.timer b/deploy/session-ops-silence.timer new file mode 100644 index 0000000..2022e69 --- /dev/null +++ b/deploy/session-ops-silence.timer @@ -0,0 +1,10 @@ +[Unit] +Description=Check hourly that every scheduled job is still succeeding +Documentation=https://github.com/session-foundation/session-shared-scripts + +[Timer] +OnCalendar=hourly +RandomizedDelaySec=5min + +[Install] +WantedBy=timers.target diff --git a/deploy/session-ops.tmpfiles b/deploy/session-ops.tmpfiles new file mode 100644 index 0000000..3d85c02 --- /dev/null +++ b/deploy/session-ops.tmpfiles @@ -0,0 +1,3 @@ +# Installed as /etc/tmpfiles.d/session-ops.conf. Root's, because every job's unit +# writes its stamp here from a `+` ExecStartPost=, and world-readable for silence.py. +d /var/lib/session-ops/stamps 0755 root root - diff --git a/deploy/silence.py b/deploy/silence.py new file mode 100644 index 0000000..2219ca4 --- /dev/null +++ b/deploy/silence.py @@ -0,0 +1,178 @@ +#!/usr/bin/env python3 +""" +Alert when a scheduled job has stopped succeeding. + +OnFailure= reports a run that failed. It says nothing about a run that never +happened: a timer left disabled after an update, a unit renamed, a host that was +down through a whole schedule. This catches those, by age rather than by event. + +Each job's unit touches /var/lib/session-ops/stamps/ when it succeeds +(ExecStartPost=, which a oneshot runs only after every ExecStart= exited 0). This +runs hourly on a timer of its own and posts one message listing every job in +jobs.toml whose stamp is older than its `max_age_hours`, then repeats it once a day +while the job stays silent. A job with no stamp at all is measured from the first +check that found it missing, so installing this does not alert on jobs that simply +have not run since. + +Success is quiet: nothing is posted when every job is on time. + +Config: + ALERT_DISCORD_WEBHOOK_URL where the alert goes (not needed with --dry-run) + +Usage: + silence.py --state /var/lib/session-ops/silence/state.json + silence.py --dry-run # print each job's age and the alert, post nothing +""" +import argparse +import json +import os +import socket +import sys +import time +import tomllib +from datetime import datetime, timezone + +import requests + +sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__)))) +from shared import discord # noqa: E402 +from shared.env import get_env # noqa: E402 + +REGISTRY = os.path.join(os.path.dirname(os.path.abspath(__file__)), "jobs.toml") +STAMPS_DIR = "/var/lib/session-ops/stamps" +REMIND_SECONDS = 24 * 3600 + + +def load_jobs(path): + with open(path, "rb") as handle: + jobs = tomllib.load(handle).get("job", []) + for job in jobs: + if not job.get("name") or not isinstance(job.get("max_age_hours"), (int, float)): + sys.exit(f"{path}: every [[job]] needs a name and a numeric max_age_hours: {job}") + return jobs + + +def stamp_time(stamps_dir, name): + try: + return os.stat(os.path.join(stamps_dir, name)).st_mtime + except FileNotFoundError: + return None + + +def load_state(path): + """Per-job bookkeeping, or {} for anything unreadable: the worst a lost file + costs is one alert repeated early, or a missing stamp's clock restarting.""" + if not path or not os.path.exists(path): + return {} + try: + with open(path, encoding="utf-8") as handle: + data = json.load(handle) + except (OSError, ValueError): + return {} + return data if isinstance(data, dict) else {} + + +def save_state(path, state): + temporary = f"{path}.tmp" + with open(temporary, "w", encoding="utf-8") as handle: + json.dump(state, handle, indent=2, sort_keys=True) + os.replace(temporary, path) + + +def evaluate(jobs, stamps_dir, state, now): + """Return (silent, due): every job past its max age, and the ones to post now. + + Each entry is (job, last_success, silent_since), `last_success` None when no + stamp exists. `state` is updated in place except for `alerted_at`, which only + a delivered alert may set. + """ + silent, due = [], [] + for job in jobs: + entry = state.setdefault(job["name"], {}) + last = stamp_time(stamps_dir, job["name"]) + if last is None: + since = entry.setdefault("missing_since", now) + else: + entry.pop("missing_since", None) + since = last + if now - since <= job["max_age_hours"] * 3600: + entry.pop("alerted_at", None) + continue + silent.append((job, last, since)) + if now - entry.get("alerted_at", 0) >= REMIND_SECONDS: + due.append((job, last, since)) + known = {job["name"] for job in jobs} + for name in [name for name in state if name not in known]: + del state[name] + return silent, due + + +def duration(seconds): + hours = int(seconds // 3600) + if hours < 48: + return f"{hours}h" + return f"{hours // 24}d {hours % 24}h" + + +def utc(epoch): + return datetime.fromtimestamp(epoch, timezone.utc).strftime("%Y-%m-%d %H:%M UTC") + + +def build_message(due, host, now): + lines = [f"πŸ”• **Scheduled jobs gone quiet** on `{host}`"] + for job, last, since in due: + name = job["name"] + seen = (f"last success {utc(last)}" if last is not None + else f"no success recorded since {utc(since)}") + lines.append(f"β€’ **{name}**: silent for **{duration(now - since)}** " + f"(allowed {job['max_age_hours']}h), {seen}.") + lines.append(f" `systemctl list-timers {name}.timer` Β· " + f"`journalctl -u {name}.service -n 50 --no-pager`") + return "\n".join(lines) + + +def main(): + parser = argparse.ArgumentParser(description="Alert on jobs that stopped succeeding.") + parser.add_argument("--registry", default=REGISTRY) + parser.add_argument("--stamps", default=STAMPS_DIR) + parser.add_argument("--state", metavar="PATH", + help="Alert bookkeeping; without it every run re-alerts.") + parser.add_argument("--webhook", help="Discord webhook URL (else ALERT_DISCORD_WEBHOOK_URL).") + parser.add_argument("--dry-run", action="store_true", + help="Print the alert instead of posting it; write no state.") + args = parser.parse_args() + + webhook = get_env("ALERT_DISCORD_WEBHOOK_URL", args.webhook, required=not args.dry_run) + jobs = load_jobs(args.registry) + now = time.time() + state = load_state(args.state) + silent, due = evaluate(jobs, args.stamps, state, now) + + for job in jobs: + last = stamp_time(args.stamps, job["name"]) + age = "never" if last is None else f"{duration(now - last)} ago" + print(f"{job['name']}: last success {age} (allowed {job['max_age_hours']}h)") + if not due: + print(f"{len(silent)} silent, none due an alert." if silent else "All jobs on time.") + if args.state and not args.dry_run: + save_state(args.state, state) + return + + message = build_message(due, socket.gethostname(), now) + if args.dry_run: + print(message) + return + if not discord.post_to_discord(requests.Session(), webhook, [{"content": message}]): + # State still saved: a missing stamp's clock must survive a failed post. + if args.state: + save_state(args.state, state) + sys.exit("Could not post the silence alert to Discord.") + for job, _, _ in due: + state[job["name"]]["alerted_at"] = now + if args.state: + save_state(args.state, state) + print(f"Alerted on {len(due)} job(s).") + + +if __name__ == "__main__": + main() diff --git a/deploy/test_silence.py b/deploy/test_silence.py new file mode 100644 index 0000000..d5b547d --- /dev/null +++ b/deploy/test_silence.py @@ -0,0 +1,100 @@ +""" + python -m unittest discover # from deploy/ +""" +import os +import tempfile +import unittest + +import silence + +HOUR = 3600 +NOW = 1_790_000_000.0 +JOB = {"name": "github-prs-digest", "max_age_hours": 80} + + +class TestEvaluate(unittest.TestCase): + def setUp(self): + self.stamps = tempfile.mkdtemp() + + def stamp(self, name, age_hours): + path = os.path.join(self.stamps, name) + open(path, "w").close() + os.utime(path, (NOW - age_hours * HOUR, NOW - age_hours * HOUR)) + + def test_a_stamp_within_its_max_age_is_quiet(self): + self.stamp(JOB["name"], 79) + self.assertEqual(silence.evaluate([JOB], self.stamps, {}, NOW), ([], [])) + + def test_a_stale_stamp_is_due(self): + self.stamp(JOB["name"], 81) + silent, due = silence.evaluate([JOB], self.stamps, {}, NOW) + self.assertEqual(due, silent) + self.assertEqual(due[0][1], NOW - 81 * HOUR) + + def test_an_alerted_job_is_repeated_once_a_day_not_every_hour(self): + self.stamp(JOB["name"], 100) + state = {JOB["name"]: {"alerted_at": NOW - 23 * HOUR}} + silent, due = silence.evaluate([JOB], self.stamps, state, NOW) + self.assertEqual((len(silent), due), (1, [])) + state[JOB["name"]]["alerted_at"] = NOW - 24 * HOUR + self.assertEqual(len(silence.evaluate([JOB], self.stamps, state, NOW)[1]), 1) + + def test_recovery_clears_the_alert_so_the_next_silence_alerts_at_once(self): + self.stamp(JOB["name"], 1) + state = {JOB["name"]: {"alerted_at": NOW - HOUR}} + silence.evaluate([JOB], self.stamps, state, NOW) + self.assertNotIn("alerted_at", state[JOB["name"]]) + + def test_a_missing_stamp_is_timed_from_when_it_was_first_found_missing(self): + """Installing the checker must not alert on jobs that have not run since.""" + state = {} + self.assertEqual(silence.evaluate([JOB], self.stamps, state, NOW), ([], [])) + self.assertEqual(state[JOB["name"]]["missing_since"], NOW) + later = NOW + 81 * HOUR + silent, due = silence.evaluate([JOB], self.stamps, state, later) + self.assertEqual(due, [(JOB, None, NOW)]) + + def test_the_first_success_clears_the_missing_clock(self): + state = {JOB["name"]: {"missing_since": NOW - 10 * HOUR}} + self.stamp(JOB["name"], 0) + silence.evaluate([JOB], self.stamps, state, NOW) + self.assertEqual(state[JOB["name"]], {}) + + def test_a_job_dropped_from_the_registry_is_forgotten(self): + state = {"retired-job": {"missing_since": NOW}} + silence.evaluate([JOB], self.stamps, state, NOW) + self.assertNotIn("retired-job", state) + + +class TestMessage(unittest.TestCase): + def test_names_the_job_its_silence_and_where_to_look(self): + message = silence.build_message([(JOB, NOW - 100 * HOUR, NOW - 100 * HOUR)], + "box", NOW) + self.assertIn("`box`", message) + self.assertIn("**github-prs-digest**: silent for **4d 4h** (allowed 80h)", message) + self.assertIn("last success 2026-09-", message) + self.assertIn("systemctl list-timers github-prs-digest.timer", message) + self.assertIn("journalctl -u github-prs-digest.service", message) + + def test_a_job_that_never_succeeded_says_since_when_it_was_watched(self): + message = silence.build_message([(JOB, None, NOW - 90 * HOUR)], "box", NOW) + self.assertIn("no success recorded since", message) + + +class TestRegistry(unittest.TestCase): + def test_the_shipped_registry_matches_the_shipped_units(self): + """A job listed without a unit, or a timer missing from the list, is silence + nobody would detect.""" + here = os.path.dirname(os.path.abspath(__file__)) + names = {job["name"] for job in silence.load_jobs(silence.REGISTRY)} + timers = {f[:-len(".timer")] for f in os.listdir(here) + if f.endswith(".timer") and f != "session-ops-silence.timer"} + self.assertEqual(names, timers) + for name in names: + with open(os.path.join(here, f"{name}.service"), encoding="utf-8") as unit: + self.assertIn(f"ExecStartPost=+/usr/bin/touch {silence.STAMPS_DIR}/{name}\n", + unit.read()) + + +if __name__ == "__main__": + unittest.main() diff --git a/deploy/zendesk-digest.service b/deploy/zendesk-digest.service index 2354534..4a1ed0c 100644 --- a/deploy/zendesk-digest.service +++ b/deploy/zendesk-digest.service @@ -25,6 +25,9 @@ EnvironmentFile=/etc/zendesk/env ExecStart=/bin/sh -c '/opt/zendesk/venv/bin/python resolve_reviews.py --apply || /opt/zendesk/venv/bin/python /opt/zendesk/deploy/alert.py resolve_reviews.py zendesk-digest.service' ExecStart=/opt/zendesk/venv/bin/python triage.py --window-hours 72 \ --state /var/lib/zendesk/seen.json +# The silence checker's evidence that this ran (deploy/silence.py). `+` runs it +# outside the sandbox, so the job itself gets no new write path. +ExecStartPost=+/usr/bin/touch /var/lib/session-ops/stamps/zendesk-digest # Covers the worst case rather than the normal one. The resolver polls each bulk job # for up to JOB_TIMEOUT_SECONDS (300s) and a full 1000-ticket run is ten batches, so From 2f995e8875b0adbba7e515e7d5b336f7dca1e4a9 Mon Sep 17 00:00:00 2001 From: Audric Ackermann Date: Fri, 25 Sep 2026 12:58:00 +1000 Subject: [PATCH 013/101] build: one session_ops package, one lockfile, one venv pyproject.toml and uv.lock replace the five requirements files, which pinned requests two different ways and so needed a venv each. The code moves to src/session_ops/{shared,github_prs,zendesk,crowdin,sogs,monitor} with absolute imports, so no module inserts anything on sys.path; tests move to tests/ in the same shape and run as one discover. Every script is a console entry point, and the units call those out of one venv at /opt/zendesk/.venv, built with the system Python: a uv-managed interpreter would sit under /home/zendesk, which ProtectHome=yes hides from the units that do not run as zendesk. test_units.py fails when a unit names an entry point the package does not declare. The relay runs note_reply with -m rather than by path. download_translations_from_crowdin.py parsed its arguments at import; that moves into main(argv) so it can be an entry point. babel stays pinned exactly, since its CLDR data is what the generated language names come from. CI runs one locked environment instead of a matrix. tests/sogs skips itself without session_util; its own job checks the import first, so the skip cannot hide a broken suite. The Crowdin workflows install the package and run modules until they move to the host. All 586 tests and the four goldens pass unchanged, and a live digest dry run through the new entry point matches its golden apart from PR ages. --- .../workflows/check_for_crowdin_updates.yml | 16 +- .../crowdin_multiple_translations_report.yml | 4 +- .github/workflows/tests.yml | 47 +- .gitignore | 4 +- README.md | 75 ++- actions/setup_shared/action.yml | 2 +- crowdin/requirements.txt | 7 - deploy/README.md | 35 +- deploy/github-prs-alert@.service | 2 +- deploy/github-prs-digest.service | 5 +- deploy/session-ops-alert@.service | 2 +- deploy/session-ops-silence.service | 2 +- deploy/session-ops.tmpfiles | 2 +- deploy/zendesk-alert@.service | 2 +- deploy/zendesk-digest.service | 7 +- deploy/zendesk-relay.service | 3 +- github_prs/requirements.txt | 2 - github_prs/test_golden.py | 44 -- pyproject.toml | 51 ++ ruff.toml | 5 - shared/__init__.py | 6 - sogs_moderation/requirements.txt | 14 - src/session_ops/__init__.py | 0 src/session_ops/crowdin/__init__.py | 0 .../session_ops/crowdin}/approve_strings.py | 6 +- .../crowdin}/codegen_localization.py | 8 +- .../download_translations_from_crowdin.py | 73 +- .../crowdin}/generate_android_strings.py | 8 +- .../crowdin}/generate_ios_strings.py | 8 +- .../crowdin}/generate_language_list.py | 8 +- .../session_ops/crowdin}/generate_shared.py | 0 .../session_ops/crowdin}/parse_xliff.py | 2 +- .../crowdin}/report_multiple_translations.py | 9 +- src/session_ops/github_prs/__init__.py | 0 .../session_ops/github_prs}/digest.py | 15 +- .../session_ops/github_prs}/maintainers.txt | 0 src/session_ops/monitor/__init__.py | 0 {deploy => src/session_ops/monitor}/alert.py | 10 +- {deploy => src/session_ops/monitor}/jobs.toml | 0 .../session_ops/monitor}/silence.py | 9 +- src/session_ops/shared/__init__.py | 1 + {shared => src/session_ops/shared}/discord.py | 2 +- {shared => src/session_ops/shared}/env.py | 0 {shared => src/session_ops/shared}/retry.py | 0 {shared => src/session_ops/shared}/state.py | 0 {shared => src/session_ops/shared}/testing.py | 24 - src/session_ops/sogs/__init__.py | 0 .../session_ops/sogs}/ban.py | 16 +- .../session_ops/sogs}/perms.py | 12 +- src/session_ops/zendesk/__init__.py | 0 .../session_ops/zendesk}/note_reply.py | 3 +- .../session_ops/zendesk}/relay.py | 4 +- .../session_ops/zendesk}/resolve_reviews.py | 12 +- .../session_ops/zendesk}/triage.py | 23 +- tests/__init__.py | 0 tests/crowdin/__init__.py | 0 .../crowdin/test_crowdin_goldens.py | 17 +- .../crowdin}/test_language_list.py | 11 +- .../test_report_multiple_translations.py | 10 +- tests/github_prs/__init__.py | 0 .../github_prs}/test_digest.py | 8 +- tests/golden.py | 25 + tests/goldens/README.md | 10 +- tests/monitor/__init__.py | 0 {deploy => tests/monitor}/test_alert.py | 4 +- {deploy => tests/monitor}/test_silence.py | 7 +- tests/monitor/test_units.py | 26 + tests/shared/__init__.py | 0 {shared => tests/shared}/test_discord.py | 7 +- {shared => tests/shared}/test_retry.py | 9 +- {shared => tests/shared}/test_state.py | 4 +- tests/sogs/__init__.py | 7 + {sogs_moderation => tests/sogs}/test_ban.py | 4 +- {sogs_moderation => tests/sogs}/test_perms.py | 6 +- tests/zendesk/__init__.py | 0 .../zendesk}/test_note_reply.py | 10 +- .../zendesk}/test_relay.py | 8 +- .../zendesk}/test_resolve_reviews.py | 10 +- .../zendesk}/test_triage.py | 19 +- uv.lock | 633 ++++++++++++++++++ zendesk_triage/requirements-dev.txt | 9 - zendesk_triage/requirements.txt | 7 - 82 files changed, 1030 insertions(+), 411 deletions(-) delete mode 100644 crowdin/requirements.txt delete mode 100644 github_prs/requirements.txt delete mode 100644 github_prs/test_golden.py create mode 100644 pyproject.toml delete mode 100644 ruff.toml delete mode 100644 shared/__init__.py delete mode 100644 sogs_moderation/requirements.txt create mode 100644 src/session_ops/__init__.py create mode 100644 src/session_ops/crowdin/__init__.py rename {crowdin => src/session_ops/crowdin}/approve_strings.py (97%) rename {crowdin => src/session_ops/crowdin}/codegen_localization.py (99%) rename {crowdin => src/session_ops/crowdin}/download_translations_from_crowdin.py (78%) rename {crowdin => src/session_ops/crowdin}/generate_android_strings.py (98%) rename {crowdin => src/session_ops/crowdin}/generate_ios_strings.py (99%) rename {crowdin => src/session_ops/crowdin}/generate_language_list.py (98%) rename {crowdin => src/session_ops/crowdin}/generate_shared.py (100%) rename {crowdin => src/session_ops/crowdin}/parse_xliff.py (99%) rename {crowdin => src/session_ops/crowdin}/report_multiple_translations.py (98%) create mode 100644 src/session_ops/github_prs/__init__.py rename {github_prs => src/session_ops/github_prs}/digest.py (97%) rename {github_prs => src/session_ops/github_prs}/maintainers.txt (100%) create mode 100644 src/session_ops/monitor/__init__.py rename {deploy => src/session_ops/monitor}/alert.py (94%) rename {deploy => src/session_ops/monitor}/jobs.toml (100%) rename {deploy => src/session_ops/monitor}/silence.py (95%) create mode 100644 src/session_ops/shared/__init__.py rename {shared => src/session_ops/shared}/discord.py (98%) rename {shared => src/session_ops/shared}/env.py (100%) rename {shared => src/session_ops/shared}/retry.py (100%) rename {shared => src/session_ops/shared}/state.py (100%) rename {shared => src/session_ops/shared}/testing.py (86%) create mode 100644 src/session_ops/sogs/__init__.py rename {sogs_moderation => src/session_ops/sogs}/ban.py (98%) rename {sogs_moderation => src/session_ops/sogs}/perms.py (96%) create mode 100644 src/session_ops/zendesk/__init__.py rename {zendesk_triage => src/session_ops/zendesk}/note_reply.py (99%) rename {zendesk_triage => src/session_ops/zendesk}/relay.py (98%) rename {zendesk_triage => src/session_ops/zendesk}/resolve_reviews.py (98%) rename {zendesk_triage => src/session_ops/zendesk}/triage.py (99%) create mode 100644 tests/__init__.py create mode 100644 tests/crowdin/__init__.py rename crowdin/test_goldens.py => tests/crowdin/test_crowdin_goldens.py (79%) rename {crowdin => tests/crowdin}/test_language_list.py (92%) rename {crowdin => tests/crowdin}/test_report_multiple_translations.py (88%) create mode 100644 tests/github_prs/__init__.py rename {github_prs => tests/github_prs}/test_digest.py (98%) create mode 100644 tests/golden.py create mode 100644 tests/monitor/__init__.py rename {deploy => tests/monitor}/test_alert.py (98%) rename {deploy => tests/monitor}/test_silence.py (94%) create mode 100644 tests/monitor/test_units.py create mode 100644 tests/shared/__init__.py rename {shared => tests/shared}/test_discord.py (96%) rename {shared => tests/shared}/test_retry.py (96%) rename {shared => tests/shared}/test_state.py (96%) create mode 100644 tests/sogs/__init__.py rename {sogs_moderation => tests/sogs}/test_ban.py (99%) rename {sogs_moderation => tests/sogs}/test_perms.py (93%) create mode 100644 tests/zendesk/__init__.py rename {zendesk_triage => tests/zendesk}/test_note_reply.py (99%) rename {zendesk_triage => tests/zendesk}/test_relay.py (98%) rename {zendesk_triage => tests/zendesk}/test_resolve_reviews.py (98%) rename {zendesk_triage => tests/zendesk}/test_triage.py (99%) create mode 100644 uv.lock delete mode 100644 zendesk_triage/requirements-dev.txt delete mode 100644 zendesk_triage/requirements.txt diff --git a/.github/workflows/check_for_crowdin_updates.yml b/.github/workflows/check_for_crowdin_updates.yml index 80d6e81..2388a45 100644 --- a/.github/workflows/check_for_crowdin_updates.yml +++ b/.github/workflows/check_for_crowdin_updates.yml @@ -56,12 +56,12 @@ jobs: - name: Install Dependencies shell: bash run: | - pip install -r ${{ github.workspace }}/scripts/crowdin/requirements.txt + pip install ${{ github.workspace }}/scripts - name: Download Translations env: CROWDIN_API_TOKEN: ${{ secrets.CROWDIN_API_TOKEN }} run: | - python "${{ github.workspace }}/scripts/crowdin/download_translations_from_crowdin.py" \ + python -m session_ops.crowdin.download_translations_from_crowdin \ "$CROWDIN_API_TOKEN" \ 618696 \ "${{ github.workspace }}/raw_translations" \ @@ -99,7 +99,7 @@ jobs: - name: Install Dependencies shell: bash run: | - pip install -r ${{ github.workspace }}/scripts/crowdin/requirements.txt + pip install ${{ github.workspace }}/scripts - name: Download raw translations uses: actions/download-artifact@v8 @@ -109,7 +109,7 @@ jobs: - name: Parse and validate XLIFF files run: | - python "${{ github.workspace }}/scripts/crowdin/parse_xliff.py" \ + python -m session_ops.crowdin.parse_xliff \ "${{ github.workspace }}/raw_translations" \ "${{ github.workspace }}/parsed_translations.json" \ ${{ inputs.SKIP_VALIDATION_ERRORS != true && '--error-on-validation-failure' || '' }} \ @@ -150,7 +150,7 @@ jobs: - name: Prepare iOS Strings run: | - python "${{ github.workspace }}/scripts/crowdin/generate_ios_strings.py" \ + python -m session_ops.crowdin.generate_ios_strings \ "${{ github.workspace }}/parsed_translations.json" \ "${{ github.workspace }}/ios/Session/Meta/Translations" \ "${{ github.workspace }}/ios/SessionUIKit/Style Guide/Constants.swift" @@ -186,13 +186,13 @@ jobs: - name: Generate TypeScript run: | - python "${{ github.workspace }}/scripts/crowdin/codegen_localization.py" \ + python -m session_ops.crowdin.codegen_localization \ "${{ github.workspace }}/parsed_translations.json" \ "${{ github.workspace }}/output/generated" - name: Generate the language list run: | - python "${{ github.workspace }}/scripts/crowdin/generate_language_list.py" \ + python -m session_ops.crowdin.generate_language_list \ "${{ github.workspace }}/parsed_translations.json" \ "${{ github.workspace }}/output/generated" @@ -248,7 +248,7 @@ jobs: - name: Prepare Android Strings run: | rm -rf ${{ github.workspace }}/android/app/src/main/res/values*/strings.xml - python "${{ github.workspace }}/scripts/crowdin/generate_android_strings.py" \ + python -m session_ops.crowdin.generate_android_strings \ "${{ github.workspace }}/parsed_translations.json" \ "${{ github.workspace }}/android/app/src/main/res" \ "${{ github.workspace }}/android/app/src/main/java/org/session/libsession/utilities/NonTranslatableStringConstants.kt" diff --git a/.github/workflows/crowdin_multiple_translations_report.yml b/.github/workflows/crowdin_multiple_translations_report.yml index 1ac6fe1..19cf71f 100644 --- a/.github/workflows/crowdin_multiple_translations_report.yml +++ b/.github/workflows/crowdin_multiple_translations_report.yml @@ -55,7 +55,7 @@ jobs: cache: "pip" - name: Install dependencies - run: pip install -r crowdin/requirements.txt + run: pip install . - name: Report strings with multiple translations env: @@ -73,7 +73,7 @@ jobs: read -ra locale_args <<< "$LOCALES" args+=(--locales "${locale_args[@]}") fi - python crowdin/report_multiple_translations.py \ + python -m session_ops.crowdin.report_multiple_translations \ "${args[@]}" \ ${{ github.event.inputs.all_locales == 'true' && '--all-locales' || '' }} \ ${{ github.event.inputs.dry_run == 'true' && '--dry-run' || '' }} diff --git a/.github/workflows/tests.yml b/.github/workflows/tests.yml index 1f8d167..0a2eae7 100644 --- a/.github/workflows/tests.yml +++ b/.github/workflows/tests.yml @@ -19,45 +19,26 @@ jobs: with: python-version: "3.12" - - run: pip install ruff==0.16.9 + - run: pip install uv==0.12.19 - - run: ruff check --output-format=github . + - run: uv run --locked --only-dev ruff check --output-format=github . unittest: runs-on: ubuntu-latest timeout-minutes: 10 - strategy: - fail-fast: false - matrix: - include: - - suite: shared - requirements: github_prs/requirements.txt - - suite: github_prs - requirements: github_prs/requirements.txt - - suite: zendesk_triage - requirements: zendesk_triage/requirements.txt zendesk_triage/requirements-dev.txt - - suite: deploy - requirements: github_prs/requirements.txt - - suite: crowdin - requirements: crowdin/requirements.txt steps: - uses: actions/checkout@v7 - uses: actions/setup-python@v6 with: python-version: "3.12" - cache: pip - cache-dependency-path: "**/requirements*.txt" - - name: Install dependencies - run: | - for file in ${{ matrix.requirements }}; do - pip install -r "$file" - done + - run: pip install uv==0.12.19 + + - run: uv sync --locked - - name: Run ${{ matrix.suite }} tests - working-directory: ${{ matrix.suite }} - run: python -m unittest discover -v + # tests/sogs skips itself here: session_util is not installable from PyPI. + - run: uv run --locked python -m unittest discover -s tests -t . -v sogs_moderation: runs-on: ubuntu-latest @@ -75,11 +56,15 @@ jobs: sudo apt-get update sudo apt-get install -y python3-session-util + # pipx, since the system pip refuses to install outside a venv (PEP 668). + - run: pipx install uv==0.12.19 + - name: Install dependencies run: | - python3 -m venv --system-site-packages .venv - .venv/bin/pip install -r sogs_moderation/requirements.txt + uv venv --system-site-packages --python /usr/bin/python3 + uv sync --locked + + # Fails here rather than letting tests/sogs skip itself. + - run: uv run --locked python -c "import session_util" - - name: Run sogs_moderation tests - working-directory: sogs_moderation - run: ../.venv/bin/python -m unittest discover -v + - run: uv run --locked python -m unittest discover -s tests/sogs -t . -v diff --git a/.gitignore b/.gitignore index 792c7cf..f0edb99 100644 --- a/.gitignore +++ b/.gitignore @@ -14,9 +14,11 @@ do_not_commit.sh .crowdin_audit/ # Local Zendesk triage debugging artifacts β€” these contain ticket content -zendesk_triage/*.json +/*.json # Zendesk triage dedup state (ticket ids + timestamps; restored from CI cache) .triage-state/ .claude/ + +.venv/ diff --git a/README.md b/README.md index 4a3e689..290b20e 100644 --- a/README.md +++ b/README.md @@ -2,6 +2,20 @@ This repo houses scripts which are shared between the different platform repos for Session, it also contains a number of Actions used to automatically sync some shared elements across the repos. +## Development + +One package, `session_ops` under [src/](src/), with one lockfile. Every job is a console +script declared in [pyproject.toml](pyproject.toml). + +```sh +uv sync # .venv with every dependency +uv run python -m unittest discover -s tests -t . # every suite +uv run ruff check . +``` + +`tests/sogs` skips itself unless `session_util` is importable; see +[Community Bans](#dependencies) for why it is not a PyPI dependency. + ## Crowdin Translation Workflow Automated workflow that downloads translations from Crowdin, validates them, and creates PRs for iOS and Android platforms and for the Typescript Localization Module for Desktop and QA. @@ -67,7 +81,7 @@ Claude reviews the Zendesk tickets awaiting a reply β€” `new` and `open`, no app ### Categories -`CATEGORY_SPECS` in [triage.py](zendesk_triage/triage.py) is the single source of truth β€” the schema enum, the Discord labels and emoji, which categories count as urgent, and the prompt guidance are all derived from it, so adding a category is one edit. +`CATEGORY_SPECS` in [triage.py](src/session_ops/zendesk/triage.py) is the single source of truth β€” the schema enum, the Discord labels and emoji, which categories count as urgent, and the prompt guidance are all derived from it, so adding a category is one edit. | Category | Notes | | --- | --- | @@ -104,7 +118,7 @@ The star-floor machinery (`partition_reviews`, `--review-star-floor`, `--include Twitter DM tickets arrive with `description` identical to `subject` β€” both just `"Conversation with "` β€” which is 15% of non-review tickets and unclassifiable as fetched. For those only, `hydrate_descriptions` fetches a page of up to 10 comments and joins every body that differs from the subject into the description; later replies often carry the actual detail. Hydration is an enrichment, so an HTTP error or an unreachable endpoint leaves the ticket as-is rather than failing the run (`--no-hydrate` to skip it entirely). -The script (`zendesk_triage/triage.py`) fetches the tickets in a rolling time window, classifies the whole batch in one schema-enforced request through the Claude Code CLI, and posts a Discord digest: a short header, then one line per ticket worth looking into. +The script (`zendesk-triage`, [triage.py](src/session_ops/zendesk/triage.py)) fetches the tickets in a rolling time window, classifies the whole batch in one schema-enforced request through the Claude Code CLI, and posts a Discord digest: a short header, then one line per ticket worth looking into. Each line leads with a severity marker, a category emoji and a platform icon, links the ticket id, and carries the model's one-line summary plus its root-cause guess: @@ -152,7 +166,7 @@ Two caveats worth knowing: - The comparison is on `requester_updated_at`, from the ticket's metric set, **not** `updated_at`. `updated_at` moves on any change β€” our own replies, a tag edit, and in this account an hourly automation that bumps tickets at :01 past the hour β€” so deduping on it re-reports the same ticket every run. Measured on a real window: an automation pass over 18 tickets produced 18 re-reports under `updated_at` and 0 under `requester_updated_at`. The metric sets are sideloaded through `show_many`, one request per 100 tickets, and a failed sideload falls back to `updated_at` β€” noisy, never silent. - Unchanged tickets are filtered out *before* the model call, which is what makes the dedup free. The trade-off is that duplicate-cluster detection only sees the new and changed tickets in a given run, not the whole window. -> **Note:** This repo is public, so ticket content is never written to the run logs or the job summary β€” ticket detail goes only to the Discord webhook (a private channel), and the links require Zendesk auth to open. The one exception is the local `--dump-batch` debugging flag, which writes ticket content to a file you name; `zendesk_triage/*.json` is gitignored to keep those out of the repo. +> **Note:** This repo is public, so ticket content is never written to the run logs or the job summary β€” ticket detail goes only to the Discord webhook (a private channel), and the links require Zendesk auth to open. The one exception is the local `--dump-batch` debugging flag, which writes ticket content to a file you name; `*.json` at the checkout root is gitignored to keep those out of the repo. ### Required Secrets @@ -191,7 +205,7 @@ The trade is process startup, a few seconds per call, against holding an API cre **Opus**, because the hard part of this job isn't per-ticket classification β€” enum-constrained categories with prompt guidance is squarely mid-tier work. It's the two batch-wide fields: `cluster` has to spot that a German app-store review and an English bug report describe one root cause, and `priority_rank` has to stay consistent across the whole batch. Those need the model to hold ~45 heterogeneous tickets in mind at once. The exact-transcription requirement (a 66-character Session ID copied verbatim) points the same way. And the entire job costs **single-digit dollars a month** on any current model β€” roughly $10 on Opus 5 against $6 on Sonnet 5 and $2 on Haiku 4.5 β€” so trading classification quality for a few dollars would be optimising the wrong thing when the cost of a miss is an unseen security report or a crash cluster nobody grouped. -**Pinned to an id rather than the `opus` alias**, because this is an unattended digest. An alias resolves to the newest Opus the credential allows, so severity calibration and cluster labels would shift on someone else's release schedule, with no run in between to notice it. Bumping the pin is a deliberate one-line change in [triage.py](zendesk_triage/triage.py) (`DEFAULT_MODEL`). +**Pinned to an id rather than the `opus` alias**, because this is an unattended digest. An alias resolves to the newest Opus the credential allows, so severity calibration and cluster labels would shift on someone else's release schedule, with no run in between to notice it. Bumping the pin is a deliberate one-line change in [triage.py](src/session_ops/zendesk/triage.py) (`DEFAULT_MODEL`). Two cases for overriding it: @@ -238,12 +252,11 @@ The file is written atomically (`os.replace`) so a crash mid-write cannot corrup ### Tests ``` -pip install -r zendesk_triage/requirements-dev.txt -python -m unittest discover -s zendesk_triage -v +uv run python -m unittest discover -s tests/zendesk -t . -v ``` -`requirements-dev.txt` is the test-only half: `test_relay.py` drives the relay through -starlette's `TestClient`, which needs an HTTP client the deployment does not. +The `dev` dependency group carries what only the tests need: `test_relay.py` drives the +relay through starlette's `TestClient`, which needs an HTTP client the deployment does not. Offline tests covering the window arithmetic, dedup partitioning, state round-trip and pruning, corrupt-state degradation, Discord card rendering and message chunking, defensive JSON parsing, and the retry/pagination behaviour with a stub session. No secrets or network access needed. @@ -252,29 +265,29 @@ Offline tests covering the window arithmetic, dedup partitioning, state round-tr Local runs need the `claude` CLI on `PATH` and logged in (`claude --version`), alongside the Zendesk credentials. `--dry-run` prints the Discord payload instead of posting, so no webhook is needed. Keep it to local runs: it prints ticket content. `--no-discord` prints counts only: ``` -pip install -r zendesk_triage/requirements.txt +uv sync export ZENDESK_SUBDOMAIN=... ZENDESK_EMAIL=... ZENDESK_API_TOKEN=... # what the unit runs, minus the Discord post and the state file -python zendesk_triage/triage.py --window-hours 72 --dry-run +uv run zendesk-triage --window-hours 72 --dry-run # keep it cheap while iterating on the rendering -python zendesk_triage/triage.py --window-hours 12 --max-tickets 5 --dry-run +uv run zendesk-triage --window-hours 12 --max-tickets 5 --dry-run # same run without the payload dump: fetches, classifies, posts nothing -python zendesk_triage/triage.py --window-hours 72 --no-discord +uv run zendesk-triage --window-hours 72 --no-discord # or take the model out of the loop: dump the batch, classify it by hand, # and feed the findings back in to render -python zendesk_triage/triage.py --dump-batch /tmp/batch.json --window-hours 48 -python zendesk_triage/triage.py --findings /tmp/findings.json --dry-run +uv run zendesk-triage --dump-batch /tmp/batch.json --window-hours 48 +uv run zendesk-triage --findings /tmp/findings.json --dry-run ``` ## Zendesk Resolve Positive Reviews The triage's opening act: it solves the 4-5β˜… AppFollow reviews that were never going to be actioned, so the unsolved backlog reflects work that actually exists. When this was written **5,253** reviews were unsolved β€” **4,812** of them still `new` β€” against **428** non-review unsolved tickets. Solving reviews was already being done by hand: **4,959** were already solved or closed. The job has since solved **3,882**, and the reviews it now finds are `open` rather than `new` β€” see the status bullet below. -> ⚠️ **This writes to Zendesk.** The scheduled run always applies. Run by hand it is a **dry run** unless you pass `--apply`, so nothing can bulk-edit tickets by accident. Read the warning at the top of [resolve_reviews.py](zendesk_triage/resolve_reviews.py) before the first applied run. +> ⚠️ **This writes to Zendesk.** The scheduled run always applies. Run by hand it is a **dry run** unless you pass `--apply`, so nothing can bulk-edit tickets by accident. Read the warning at the top of [resolve_reviews.py](src/session_ops/zendesk/resolve_reviews.py) before the first applied run. ### What it will and will not touch @@ -322,7 +335,7 @@ Silence would be indistinguishable from a job that has quietly stopped working **A run that died reports too**, from the unit rather than the script β€” a Zendesk `4xx`, a bulk job that never completes, a host that rebooted all exit before a message exists: -> ❌ **resolve_reviews.py** failed on `angus`, as part of zendesk-digest.service. +> ❌ **zendesk-resolve-reviews** failed on `angus`, as part of zendesk-digest.service. > `journalctl -u zendesk-digest.service -n 50 --no-pager` It says nothing about counts, because it also fires after the script has already posted a tally alongside per-ticket failures, and nothing about the cause, because the run may have died before it had one β€” it points at the journal instead of guessing. @@ -482,7 +495,7 @@ written. ``` # what the webhook does, against a real ticket, writing nothing -python zendesk_triage/note_reply.py --ticket 27603 --dry-run +uv run zendesk-note-reply --ticket 27603 --dry-run ``` A ticket with no command note prints `no command note to act on` and stops, so this @@ -490,13 +503,13 @@ is safe to point at anything. ## Community Bans -Abuse reports arrive through Zendesk with a Session ID. [`sogs_moderation/ban.py`](sogs_moderation/ban.py) +Abuse reports arrive through Zendesk with a Session ID. [`ban.py`](src/session_ops/sogs/ban.py) bans those IDs from the whole SOGS we run, not room by room, and prints what the server answered at each step, so the reply to the ticket can say what happened: ```sh set -a && . ./.env && set +a # SOGS_MOD_SEED -python sogs_moderation/ban.py 05abc...def +uv run sogs-ban 05abc...def ``` The ban is server-wide, and the deletion follows once it has landed. The order matters: @@ -574,13 +587,13 @@ attempt answers and the loop never reaches the rest. It is harmless until then. Our rooms are read-only to everyone but moderators, so a test account has nothing for the deletion step to delete and the run proves nothing. -[`sogs_moderation/perms.py`](sogs_moderation/perms.py) grants it write permission in +[`perms.py`](src/session_ops/sogs/perms.py) grants it write permission in one room, and takes it back afterwards: ```sh -python sogs_moderation/perms.py --room session-updates --write on --upload on 05 +uv run sogs-perms --room session-updates --write on --upload on 05 # ... post from that account, then ban it, then: -python sogs_moderation/perms.py --room session-updates --write default --upload default 05 +uv run sogs-perms --room session-updates --write default --upload default 05 ``` `on` grants, `off` denies (muting one account without banning it), `default` drops the @@ -623,8 +636,8 @@ yet. Deleting messages is the step that needs the blinded form β€” see ```sh sudo apt install python3-session-util # see "Dependencies" below -pip install -r sogs_moderation/requirements.txt -python -m unittest discover -s sogs_moderation -v +uv venv --system-site-packages --python /usr/bin/python3 && uv sync +uv run python -m unittest discover -s tests/sogs -t . -v ``` The blinded signature is checked by verifying it under the blinded pubkey rather than @@ -645,7 +658,7 @@ sudo apt install python3-session-util It is a compiled extension built per Python minor version, so a Python upgrade needs a matching package, and a virtualenv needs `--system-site-packages` to see it. This is why -`ban.py` is run from a checkout by hand rather than deployed anywhere. +`sogs-ban` is run from a checkout by hand rather than deployed anywhere. pynacl stays for the blinding factor and for `blinded_ids`, which the deletion step walks: a Session ID does not carry the sign of the key behind it, so both candidates have to be @@ -655,7 +668,7 @@ no packaged release carries yet. ## Contributor Pull Requests -[`github_prs/digest.py`](github_prs/digest.py) posts one message each weekday morning +[`digest.py`](src/session_ops/github_prs/digest.py) posts one message each weekday morning listing the open pull requests in session-foundation's repositories whose author is not a maintainer and which have moved in the last three days β€” the ones nobody on the team has a reason to already know about: @@ -679,7 +692,7 @@ waiting. The timer runs `Mon..Fri`, so Monday's run has to cover the weekend β€” hence a 72-hour window rather than a daily one. That window overlaps itself by two days on every run, -and [`--state`](github_prs/digest.py) is what stops the overlap being noise: it records +and [`--state`](src/session_ops/github_prs/digest.py) is what stops the overlap being noise: it records which PRs reached Discord and what each one's `updated_at` was at the time. `updated_at` moves on *any* change, including one that touches several PRs at once β€” a @@ -701,7 +714,7 @@ floor instead of failing. ### Who is a maintainer -[`github_prs/maintainers.txt`](github_prs/maintainers.txt), one login per line, matched +[`maintainers.txt`](src/session_ops/github_prs/maintainers.txt), one login per line, matched case-insensitively. Bot accounts need no entry β€” every account GitHub types as a `Bot` is dropped, so a renamed Dependabot stays out on its own. @@ -737,11 +750,11 @@ that: private repositories stay out whatever the token can see. It runs on the same box as the Zendesk digest, under its own user and its own environment file β€” see [deploy/README.md](deploy/README.md). Its HTTP retries, Discord posting and dedup state are the same code the Zendesk digest uses, in -[shared/](shared/). +[shared/](src/session_ops/shared/). ```sh -cd github_prs && python -m unittest discover -cd shared && python -m unittest discover +uv run python -m unittest discover -s tests/github_prs -t . +uv run python -m unittest discover -s tests/shared -t . ``` ## Workflow Failure Notificaiton diff --git a/actions/setup_shared/action.yml b/actions/setup_shared/action.yml index 357bdbb..3d529d5 100644 --- a/actions/setup_shared/action.yml +++ b/actions/setup_shared/action.yml @@ -11,7 +11,7 @@ runs: - name: Install Dependencies shell: bash run: | - pip install -r ${{ github.workspace }}/scripts/crowdin/requirements.txt + pip install ${{ github.workspace }}/scripts - uses: actions/download-artifact@v8 with: diff --git a/crowdin/requirements.txt b/crowdin/requirements.txt deleted file mode 100644 index 6808702..0000000 --- a/crowdin/requirements.txt +++ /dev/null @@ -1,7 +0,0 @@ -babel==2.17.0 -certifi==2024.7.4 -charset-normalizer==3.3.2 -colorama==0.4.6 -idna==3.7 -requests==2.32.3 -urllib3==2.2.2 \ No newline at end of file diff --git a/deploy/README.md b/deploy/README.md index c96b7a7..380b673 100644 --- a/deploy/README.md +++ b/deploy/README.md @@ -17,17 +17,21 @@ to. `OnFailure=` only sees a run that failed. A run that never happened (a timer left disabled, a host down through a whole schedule) is what the silence checker is for: every scheduled unit touches `/var/lib/session-ops/stamps/` on success, and -`silence.py` compares each stamp's age with `jobs.toml`. A new scheduled job needs both -its `ExecStartPost=` line and a `jobs.toml` entry; `test_silence.py` fails without +`session-ops-silence` compares each stamp's age with `jobs.toml`. A new scheduled job needs both +its `ExecStartPost=` line and an entry in +[`jobs.toml`](../src/session_ops/monitor/jobs.toml); `tests/monitor/test_silence.py` fails without either. One clone at `/opt/zendesk` holds all of it β€” the directory is named after its first -tenant, not its contents. The venvs are separate, because the two jobs pin `requests` -differently and a shared one would silently be whichever was installed last. +tenant, not its contents. One venv, `/opt/zendesk/.venv`, built from `uv.lock`, serves +every unit: each job is a console script in it. ## Host requirements - Linux with systemd 252 or newer (the timer needs a timezone in `OnCalendar=`), and Python 3.12+ +- [uv](https://docs.astral.sh/uv/) on root's `PATH`. The venv must use the system + Python, not one uv downloads: that would live under `/home/zendesk`, which + `ProtectHome=yes` hides from every unit not running as `zendesk`. - **The Claude Code CLI installed and logged in as the service user.** Both Claude calls go through it β€” the digest's classification and the reply flow's translation β€” so its login is the only Claude credential this box holds. It must be @@ -71,10 +75,11 @@ runuser -u zendesk -- env HOME=/home/zendesk sh -c 'curl -fsSL https://claude.ai runuser -u zendesk -- env HOME=/home/zendesk /home/zendesk/.local/bin/claude # 2. The code and its venv +curl -LsSf https://astral.sh/uv/install.sh | env UV_INSTALL_DIR=/usr/local/bin sh git clone https://github.com/session-foundation/session-shared-scripts /opt/zendesk -python3 -m venv /opt/zendesk/venv -/opt/zendesk/venv/bin/pip install -r /opt/zendesk/zendesk_triage/requirements.txt chown -R zendesk:zendesk /opt/zendesk +runuser -u zendesk -- env HOME=/home/zendesk UV_PYTHON_DOWNLOADS=never \ + uv sync --locked --no-dev --python /usr/bin/python3 --directory /opt/zendesk # 3. State install -d -o zendesk -g zendesk -m 750 /var/lib/zendesk @@ -122,7 +127,7 @@ nginx -t && systemctl reload nginx # validate what certbot wrote ### Adding the pull request digest -Its own user, its own environment file and its own venv, out of the same clone. A +Its own user and its own environment file, out of the same clone and venv. A token that can read the org's repositories has no business in the environment of the relay, which is the one process here reachable from the internet. @@ -132,9 +137,6 @@ wanted here at all. ```bash useradd --system --no-create-home --home /nonexistent --shell /usr/sbin/nologin ghdigest -python3 -m venv /opt/github-prs/venv -/opt/github-prs/venv/bin/pip install -r /opt/zendesk/github_prs/requirements.txt -chown -R ghdigest:ghdigest /opt/github-prs install -d -m 750 -o root -g ghdigest /etc/github-prs [ -e /etc/github-prs/env ] || install -m 640 -o root -g ghdigest /dev/null /etc/github-prs/env @@ -266,7 +268,7 @@ GITHUB_PRS_TOKEN= # in, so this one value decides where the digest goes. GITHUB_PRS_DISCORD_WEBHOOK_URL= -# Required, and normally the same webhook: without it alert.py falls back to +# Required, and normally the same webhook: without it session-ops-alert falls back to # ZENDESK_DISCORD_WEBHOOK_URL, which is not in this file, and the failure notifier # fails instead of reporting. ALERT_DISCORD_WEBHOOK_URL= @@ -377,7 +379,7 @@ Updating. A permission error from the second command costs the excerpt and nothi else: the alert still sends, with the message it always sent. ```sh -cd deploy && python -m unittest discover # the alert's own tests +uv run python -m unittest discover -s tests/monitor -t . # the alert's own tests ``` **6. The pull request digest.** A dry run under the unit's own confinement renders the digest and posts nothing. `systemd-run` rather than `runuser` because the token then @@ -386,8 +388,7 @@ read out of `ps`: ```bash systemd-run --pty --uid=ghdigest -p EnvironmentFile=/etc/github-prs/env \ - -p WorkingDirectory=/opt/zendesk/github_prs \ - /opt/github-prs/venv/bin/python digest.py --dry-run + /opt/zendesk/.venv/bin/github-prs-digest --dry-run ``` Then for real: `systemctl start github-prs-digest.service`, @@ -411,7 +412,7 @@ it would post, and writes no state: ```bash systemd-run --pipe --wait --uid=sessionops -p EnvironmentFile=/etc/session-ops/env \ - /opt/github-prs/venv/bin/python /opt/zendesk/deploy/silence.py --dry-run + /opt/zendesk/.venv/bin/session-ops-silence --dry-run ls -l /var/lib/session-ops/stamps/ # one file per job that has succeeded since ``` @@ -421,8 +422,8 @@ ls -l /var/lib/session-ops/stamps/ # one file per job that has succeeded sin ```bash runuser -u zendesk -- git -C /opt/zendesk pull -/opt/zendesk/venv/bin/pip install -r /opt/zendesk/zendesk_triage/requirements.txt -/opt/github-prs/venv/bin/pip install -r /opt/zendesk/github_prs/requirements.txt +runuser -u zendesk -- env HOME=/home/zendesk UV_PYTHON_DOWNLOADS=never \ + uv sync --locked --no-dev --python /usr/bin/python3 --directory /opt/zendesk systemctl restart zendesk-relay ``` diff --git a/deploy/github-prs-alert@.service b/deploy/github-prs-alert@.service index 8103550..567f076 100644 --- a/deploy/github-prs-alert@.service +++ b/deploy/github-prs-alert@.service @@ -10,7 +10,7 @@ EnvironmentFile=/etc/github-prs/env # Same script as zendesk-alert@, run under this job's environment so the failure # lands in the channel the job posts to rather than the triage one. -ExecStart=/opt/github-prs/venv/bin/python /opt/zendesk/deploy/alert.py %i +ExecStart=/opt/zendesk/.venv/bin/session-ops-alert %i # See zendesk-alert@.service: read-only access to the failed unit's journal, which is # the line the alert quotes. diff --git a/deploy/github-prs-digest.service b/deploy/github-prs-digest.service index b940c63..95c3dea 100644 --- a/deploy/github-prs-digest.service +++ b/deploy/github-prs-digest.service @@ -9,12 +9,11 @@ OnFailure=github-prs-alert@%n.service Type=oneshot User=ghdigest Group=ghdigest -WorkingDirectory=/opt/zendesk/github_prs EnvironmentFile=/etc/github-prs/env -ExecStart=/opt/github-prs/venv/bin/python digest.py --window-hours 72 \ +ExecStart=/opt/zendesk/.venv/bin/github-prs-digest --window-hours 72 \ --state /var/lib/github-prs/seen.json -# The silence checker's evidence that this ran (deploy/silence.py). `+` runs it +# The silence checker's evidence that this ran (session-ops-silence). `+` runs it # outside the sandbox, so the job itself gets no new write path. ExecStartPost=+/usr/bin/touch /var/lib/session-ops/stamps/github-prs-digest diff --git a/deploy/session-ops-alert@.service b/deploy/session-ops-alert@.service index 8e14507..bca5e02 100644 --- a/deploy/session-ops-alert@.service +++ b/deploy/session-ops-alert@.service @@ -7,7 +7,7 @@ Type=oneshot User=sessionops Group=sessionops EnvironmentFile=/etc/session-ops/env -ExecStart=/opt/github-prs/venv/bin/python /opt/zendesk/deploy/alert.py %i +ExecStart=/opt/zendesk/.venv/bin/session-ops-alert %i # See zendesk-alert@.service: the failed unit's journal is the line the alert quotes. SupplementaryGroups=systemd-journal diff --git a/deploy/session-ops-silence.service b/deploy/session-ops-silence.service index e359803..7eb6e19 100644 --- a/deploy/session-ops-silence.service +++ b/deploy/session-ops-silence.service @@ -10,7 +10,7 @@ Type=oneshot User=sessionops Group=sessionops EnvironmentFile=/etc/session-ops/env -ExecStart=/opt/github-prs/venv/bin/python /opt/zendesk/deploy/silence.py \ +ExecStart=/opt/zendesk/.venv/bin/session-ops-silence \ --state /var/lib/session-ops/silence/state.json TimeoutStartSec=5min diff --git a/deploy/session-ops.tmpfiles b/deploy/session-ops.tmpfiles index 3d85c02..208d9d7 100644 --- a/deploy/session-ops.tmpfiles +++ b/deploy/session-ops.tmpfiles @@ -1,3 +1,3 @@ # Installed as /etc/tmpfiles.d/session-ops.conf. Root's, because every job's unit -# writes its stamp here from a `+` ExecStartPost=, and world-readable for silence.py. +# writes its stamp here from a `+` ExecStartPost=, and world-readable for session-ops-silence. d /var/lib/session-ops/stamps 0755 root root - diff --git a/deploy/zendesk-alert@.service b/deploy/zendesk-alert@.service index cef435a..851d158 100644 --- a/deploy/zendesk-alert@.service +++ b/deploy/zendesk-alert@.service @@ -12,7 +12,7 @@ EnvironmentFile=/etc/zendesk/env # one-liner because the message needs JSON encoding, and quoting that through # systemd -> sh -> jq is the kind of thing that works until a unit name has a # character nobody thought about. -ExecStart=/opt/zendesk/venv/bin/python /opt/zendesk/deploy/alert.py %i +ExecStart=/opt/zendesk/.venv/bin/session-ops-alert %i # Read-only access to the failed unit's journal, which is where the alert gets the # line it quotes. Without it journalctl exits non-zero, the excerpt is dropped and diff --git a/deploy/zendesk-digest.service b/deploy/zendesk-digest.service index 4a1ed0c..878dc11 100644 --- a/deploy/zendesk-digest.service +++ b/deploy/zendesk-digest.service @@ -9,7 +9,6 @@ OnFailure=zendesk-alert@%n.service Type=oneshot User=zendesk Group=zendesk -WorkingDirectory=/opt/zendesk/zendesk_triage EnvironmentFile=/etc/zendesk/env # Two ExecStart lines, in this order, rather than two units: a solved review leaves @@ -22,10 +21,10 @@ EnvironmentFile=/etc/zendesk/env # looks exactly like one with nothing to do. The `||` reports it and still lets the # digest run. If the alert itself also fails, the shell exits non-zero and the unit's # own OnFailure catches that. -ExecStart=/bin/sh -c '/opt/zendesk/venv/bin/python resolve_reviews.py --apply || /opt/zendesk/venv/bin/python /opt/zendesk/deploy/alert.py resolve_reviews.py zendesk-digest.service' -ExecStart=/opt/zendesk/venv/bin/python triage.py --window-hours 72 \ +ExecStart=/bin/sh -c '/opt/zendesk/.venv/bin/zendesk-resolve-reviews --apply || /opt/zendesk/.venv/bin/session-ops-alert zendesk-resolve-reviews zendesk-digest.service' +ExecStart=/opt/zendesk/.venv/bin/zendesk-triage --window-hours 72 \ --state /var/lib/zendesk/seen.json -# The silence checker's evidence that this ran (deploy/silence.py). `+` runs it +# The silence checker's evidence that this ran (session-ops-silence). `+` runs it # outside the sandbox, so the job itself gets no new write path. ExecStartPost=+/usr/bin/touch /var/lib/session-ops/stamps/zendesk-digest diff --git a/deploy/zendesk-relay.service b/deploy/zendesk-relay.service index ec71985..594b2ac 100644 --- a/deploy/zendesk-relay.service +++ b/deploy/zendesk-relay.service @@ -11,12 +11,11 @@ OnFailure=zendesk-alert@%n.service Type=exec User=zendesk Group=zendesk -WorkingDirectory=/opt/zendesk/zendesk_triage EnvironmentFile=/etc/zendesk/env # Bound to loopback: nginx terminates TLS and is the only thing that should be able # to reach this. Exposing it directly would put an unauthenticated port on the box β€” # the HMAC check is the only gate, and it should not be the only barrier too. -ExecStart=/opt/zendesk/venv/bin/uvicorn relay:app --host 127.0.0.1 --port 8080 \ +ExecStart=/opt/zendesk/.venv/bin/uvicorn session_ops.zendesk.relay:app --host 127.0.0.1 --port 8080 \ --no-access-log --proxy-headers --forwarded-allow-ips 127.0.0.1 Restart=always RestartSec=2 diff --git a/github_prs/requirements.txt b/github_prs/requirements.txt deleted file mode 100644 index 70429a6..0000000 --- a/github_prs/requirements.txt +++ /dev/null @@ -1,2 +0,0 @@ -# What the deployment runs. The suite is stdlib unittest and imports nothing else. -requests==2.34.2 diff --git a/github_prs/test_golden.py b/github_prs/test_golden.py deleted file mode 100644 index 62a5468..0000000 --- a/github_prs/test_golden.py +++ /dev/null @@ -1,44 +0,0 @@ -""" - python -m unittest discover # from github_prs/ - UPDATE_GOLDENS=1 python -m unittest test_golden # accept a new payload -""" -import contextlib -import io -import os -import sys -import unittest -from datetime import datetime -from unittest import mock - -sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__)))) -import digest # noqa: E402 -from shared.testing import (GOLDENS_DIR, RecordedSession, assert_golden, # noqa: E402 - frozen_datetime, load_golden_json) - - -class TestDigestGolden(unittest.TestCase): - """The dry-run output of a recorded live run over a 720-hour window.""" - - def run_digest(self, *extra): - recording = load_golden_json("digest/responses.json") - session = RecordedSession(recording["exchanges"]) - out = io.StringIO() - with mock.patch.object(digest, "github_session", lambda token: session), \ - mock.patch.object(digest, "datetime", - frozen_datetime(datetime.fromisoformat(recording["now"]))), \ - mock.patch.object(sys, "argv", ["digest.py", *recording["argv"], *extra]), \ - mock.patch.dict(os.environ, {"GITHUB_PRS_TOKEN": "t"}), \ - contextlib.redirect_stdout(out): - digest.main() - return out.getvalue() - - def test_dry_run_matches_the_recorded_run(self): - assert_golden(self, "digest/dry-run.txt", self.run_digest()) - - def test_state_splits_the_window_into_new_changed_and_unchanged(self): - state = os.path.join(GOLDENS_DIR, "digest", "state.json") - assert_golden(self, "digest/dry-run-with-state.txt", self.run_digest("--state", state)) - - -if __name__ == "__main__": - unittest.main() diff --git a/pyproject.toml b/pyproject.toml new file mode 100644 index 0000000..3c02d13 --- /dev/null +++ b/pyproject.toml @@ -0,0 +1,51 @@ +[project] +name = "session-ops" +version = "0.1.0" +description = "Session Foundation's scheduled jobs and webhooks: Zendesk, GitHub, Crowdin, SOGS." +readme = "README.md" +requires-python = ">=3.12" +dependencies = [ + # CLDR data: a new babel can rename a language in the generated language lists. + "babel==2.17.0", + "colorama>=0.4.6", + "fastapi>=0.141.1", + "pynacl>=1.6.2", + "requests>=2.32.3", + "uvicorn>=0.52.4", +] + +[project.scripts] +github-prs-digest = "session_ops.github_prs.digest:main" +zendesk-triage = "session_ops.zendesk.triage:main" +zendesk-resolve-reviews = "session_ops.zendesk.resolve_reviews:main" +zendesk-note-reply = "session_ops.zendesk.note_reply:main" +crowdin-download = "session_ops.crowdin.download_translations_from_crowdin:main" +crowdin-parse-xliff = "session_ops.crowdin.parse_xliff:main" +crowdin-generate-ios = "session_ops.crowdin.generate_ios_strings:cli" +crowdin-generate-android = "session_ops.crowdin.generate_android_strings:cli" +crowdin-codegen-localization = "session_ops.crowdin.codegen_localization:cli" +crowdin-generate-language-list = "session_ops.crowdin.generate_language_list:cli" +crowdin-report-duplicates = "session_ops.crowdin.report_multiple_translations:main" +crowdin-approve-strings = "session_ops.crowdin.approve_strings:main" +sogs-ban = "session_ops.sogs.ban:cli" +sogs-perms = "session_ops.sogs.perms:cli" +session-ops-alert = "session_ops.monitor.alert:main" +session-ops-silence = "session_ops.monitor.silence:main" + +[dependency-groups] +dev = [ + # starlette's TestClient ships no HTTP client, and warns on httpx in favour of this. + "httpx2>=2.12.0", + "ruff==0.16.9", +] + +[build-system] +requires = ["uv_build>=0.12,<0.13"] +build-backend = "uv_build" + +# Pyflakes and syntax errors only: what catches a broken import or name mid-refactor. +[tool.ruff] +target-version = "py312" + +[tool.ruff.lint] +select = ["F", "E9"] diff --git a/ruff.toml b/ruff.toml deleted file mode 100644 index 7ffe240..0000000 --- a/ruff.toml +++ /dev/null @@ -1,5 +0,0 @@ -# Pyflakes and syntax errors only: what catches a broken import or name mid-refactor. -target-version = "py312" - -[lint] -select = ["F", "E9"] diff --git a/shared/__init__.py b/shared/__init__.py deleted file mode 100644 index cd9e60c..0000000 --- a/shared/__init__.py +++ /dev/null @@ -1,6 +0,0 @@ -"""Helpers shared by the scripts in this repository. - -A script inserts the repository root on sys.path and imports from here. Every job -runs out of the same clone, so no install step is involved; the only dependency -outside the standard library is `requests`, which every consumer already pins. -""" diff --git a/sogs_moderation/requirements.txt b/sogs_moderation/requirements.txt deleted file mode 100644 index 8951225..0000000 --- a/sogs_moderation/requirements.txt +++ /dev/null @@ -1,14 +0,0 @@ -# What ban.py needs from PyPI. The suite is stdlib unittest and imports nothing else. -requests==2.34.2 - -# The blinding factor and blinded_ids are still derived here: the scalar arithmetic for -# those reaches session_util only in a release later than 0.1.0. -pynacl==1.6.2 - -# NOT INSTALLABLE FROM HERE. The blinded request signatures come from session_util, -# libsession-util's Python binding, which is published as a deb rather than a wheel: -# -# https://deb.oxen.io -> sudo apt install python3-session-util -# -# It is a compiled extension built per Python minor version, so a virtualenv needs -# --system-site-packages to see it. diff --git a/src/session_ops/__init__.py b/src/session_ops/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/src/session_ops/crowdin/__init__.py b/src/session_ops/crowdin/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/crowdin/approve_strings.py b/src/session_ops/crowdin/approve_strings.py similarity index 97% rename from crowdin/approve_strings.py rename to src/session_ops/crowdin/approve_strings.py index e5613cc..f7a932e 100644 --- a/crowdin/approve_strings.py +++ b/src/session_ops/crowdin/approve_strings.py @@ -14,13 +14,13 @@ Usage: # inspect first: list every translation + who submitted it (no changes) - python approve_strings.py --list ongoingAppeal ongoingAppealDescription + crowdin-approve-strings --list ongoingAppeal ongoingAppealDescription # approve ONLY translations submitted by trusted users (by username or id): - python approve_strings.py --by-user alice --by-user 12345 ongoingAppeal ongoingAppealDescription + crowdin-approve-strings --by-user alice --by-user 12345 ongoingAppeal ongoingAppealDescription # dry-run of the above (shows what would be approved, changes nothing): - python approve_strings.py --dry-run --by-user alice ongoingAppeal ongoingAppealDescription + crowdin-approve-strings --dry-run --by-user alice ongoingAppeal ongoingAppealDescription Exactly one of --by-user or --all-users is required unless --list is given, so approval is always an explicit choice -- a forgotten filter is an error, never diff --git a/crowdin/codegen_localization.py b/src/session_ops/crowdin/codegen_localization.py similarity index 99% rename from crowdin/codegen_localization.py rename to src/session_ops/crowdin/codegen_localization.py index 632d9a4..b28132e 100644 --- a/crowdin/codegen_localization.py +++ b/src/session_ops/crowdin/codegen_localization.py @@ -13,7 +13,7 @@ import re import os from typing import Dict, List, Any, Tuple -from generate_shared import ( +from session_ops.crowdin.generate_shared import ( DISCLAIMER_GENERATED, get_locale_key, load_parsed_translations, @@ -654,5 +654,9 @@ def main(): print_success("All files generated successfully!") -if __name__ == "__main__": +def cli(): run_main(main) + + +if __name__ == "__main__": + cli() diff --git a/crowdin/download_translations_from_crowdin.py b/src/session_ops/crowdin/download_translations_from_crowdin.py similarity index 78% rename from crowdin/download_translations_from_crowdin.py rename to src/session_ops/crowdin/download_translations_from_crowdin.py index 033607b..9b7c6bf 100644 --- a/crowdin/download_translations_from_crowdin.py +++ b/src/session_ops/crowdin/download_translations_from_crowdin.py @@ -9,35 +9,40 @@ from colorama import Fore, Style, init -# Initialize colorama -init(autoreset=True) - -# Parse command-line arguments -parser = argparse.ArgumentParser(description='Download translations from Crowdin.') -parser.add_argument('api_token', help='Crowdin API token') -parser.add_argument('project_id', help='Crowdin project ID') -parser.add_argument('download_directory', help='Directory to save the initial downloaded files') -parser.add_argument('--glossary_id', help='Crowdin glossary ID (optional)', default=None) -parser.add_argument('--concept_id', help='Crowdin non-translatable terms concept ID (optional)', default=None) -parser.add_argument('--skip-untranslated-strings', action='store_true', help='Exclude strings which have not been translated from the translation files') -parser.add_argument('--force-allow-unapproved', action='store_true', help='Include unapproved translations in the translation files') -parser.add_argument('--max-workers', type=int, default=10, help='Maximum number of parallel downloads (default: 10, max: 20 due to Crowdin API limits)') -parser.add_argument('-v', '--verbose', action='store_true', help='Enable verbose output') -args = parser.parse_args() - CROWDIN_API_BASE_URL = "https://api.crowdin.com/api/v2" -CROWDIN_API_TOKEN = args.api_token -CROWDIN_PROJECT_ID = args.project_id -CROWDIN_GLOSSARY_ID = args.glossary_id -CROWDIN_CONCEPT_ID = args.concept_id -DOWNLOAD_DIRECTORY = args.download_directory -SKIP_UNTRANSLATED_STRINGS = args.skip_untranslated_strings -FORCE_ALLOW_UNAPPROVED = args.force_allow_unapproved -VERBOSE = args.verbose -# Crowdin API limit is 20 simultaneous requests per account -MAX_WORKERS = min(args.max_workers, 20) -# Semaphore ensures we don't exceed the concurrent requests limit -api_semaphore = Semaphore(MAX_WORKERS) + + +def parse_args(argv=None): + parser = argparse.ArgumentParser(description='Download translations from Crowdin.') + parser.add_argument('api_token', help='Crowdin API token') + parser.add_argument('project_id', help='Crowdin project ID') + parser.add_argument('download_directory', help='Directory to save the initial downloaded files') + parser.add_argument('--glossary_id', help='Crowdin glossary ID (optional)', default=None) + parser.add_argument('--concept_id', help='Crowdin non-translatable terms concept ID (optional)', default=None) + parser.add_argument('--skip-untranslated-strings', action='store_true', help='Exclude strings which have not been translated from the translation files') + parser.add_argument('--force-allow-unapproved', action='store_true', help='Include unapproved translations in the translation files') + parser.add_argument('--max-workers', type=int, default=10, help='Maximum number of parallel downloads (default: 10, max: 20 due to Crowdin API limits)') + parser.add_argument('-v', '--verbose', action='store_true', help='Enable verbose output') + return parser.parse_args(argv) + + +def configure(args): + global CROWDIN_API_TOKEN, CROWDIN_PROJECT_ID, CROWDIN_GLOSSARY_ID, CROWDIN_CONCEPT_ID + global DOWNLOAD_DIRECTORY, SKIP_UNTRANSLATED_STRINGS, FORCE_ALLOW_UNAPPROVED, VERBOSE + global MAX_WORKERS, api_semaphore + CROWDIN_API_TOKEN = args.api_token + CROWDIN_PROJECT_ID = args.project_id + CROWDIN_GLOSSARY_ID = args.glossary_id + CROWDIN_CONCEPT_ID = args.concept_id + DOWNLOAD_DIRECTORY = args.download_directory + SKIP_UNTRANSLATED_STRINGS = args.skip_untranslated_strings + FORCE_ALLOW_UNAPPROVED = args.force_allow_unapproved + VERBOSE = args.verbose + # Crowdin API limit is 20 simultaneous requests per account + MAX_WORKERS = min(args.max_workers, 20) + # Semaphore ensures we don't exceed the concurrent requests limit + api_semaphore = Semaphore(MAX_WORKERS) + REQUEST_TIMEOUT_S = 30 MAX_RETRIES = 5 @@ -136,7 +141,7 @@ def export_and_download_language(language: dict, is_source: bool = False) -> str return lang_locale -def main(): +def download(): global total_count, completed_count # Retrieve the list of languages print(f"{Fore.WHITE}⏳ Retrieving project details...{Style.RESET_ALL}", end='\r') @@ -218,12 +223,18 @@ def main(): print(f"\033[2K{Fore.GREEN}βœ… Downloading non-translatable complete{Style.RESET_ALL}") -if __name__ == "__main__": +def main(argv=None): + init(autoreset=True) + configure(parse_args(argv)) try: - main() + download() except KeyboardInterrupt: print(f"\n{Fore.RED}Process interrupted by user{Style.RESET_ALL}") sys.exit(0) except Exception as e: print(f"\033[2K{Fore.RED}❌ An error occurred: {e}{Style.RESET_ALL}") sys.exit(1) + + +if __name__ == "__main__": + main() diff --git a/crowdin/generate_android_strings.py b/src/session_ops/crowdin/generate_android_strings.py similarity index 98% rename from crowdin/generate_android_strings.py rename to src/session_ops/crowdin/generate_android_strings.py index e6311ff..c90c728 100644 --- a/crowdin/generate_android_strings.py +++ b/src/session_ops/crowdin/generate_android_strings.py @@ -3,7 +3,7 @@ import argparse from pathlib import Path from typing import Dict, Any -from generate_shared import ( +from session_ops.crowdin.generate_shared import ( load_parsed_translations, clean_string, print_progress, @@ -189,5 +189,9 @@ def main(): print_success("All conversions complete") -if __name__ == "__main__": +def cli(): run_main(main) + + +if __name__ == "__main__": + cli() diff --git a/crowdin/generate_ios_strings.py b/src/session_ops/crowdin/generate_ios_strings.py similarity index 99% rename from crowdin/generate_ios_strings.py rename to src/session_ops/crowdin/generate_ios_strings.py index 3649b86..0041cdb 100644 --- a/crowdin/generate_ios_strings.py +++ b/src/session_ops/crowdin/generate_ios_strings.py @@ -4,7 +4,7 @@ from pathlib import Path from datetime import datetime from typing import Dict, Any -from generate_shared import ( +from session_ops.crowdin.generate_shared import ( load_parsed_translations, clean_string, print_progress, @@ -225,5 +225,9 @@ def main(): print_success("All conversions complete") -if __name__ == "__main__": +def cli(): run_main(main) + + +if __name__ == "__main__": + cli() diff --git a/crowdin/generate_language_list.py b/src/session_ops/crowdin/generate_language_list.py similarity index 98% rename from crowdin/generate_language_list.py rename to src/session_ops/crowdin/generate_language_list.py index 254a432..49fc9bc 100755 --- a/crowdin/generate_language_list.py +++ b/src/session_ops/crowdin/generate_language_list.py @@ -17,7 +17,7 @@ from babel import Locale from babel.core import get_global -from generate_shared import ( +from session_ops.crowdin.generate_shared import ( DISCLAIMER_GENERATED, cldr_code, cldr_locale, @@ -179,5 +179,9 @@ def main(): generate_languages_ts(parsed_data, os.path.join(args.output_directory, 'languages.ts')) -if __name__ == "__main__": +def cli(): run_main(main) + + +if __name__ == "__main__": + cli() diff --git a/crowdin/generate_shared.py b/src/session_ops/crowdin/generate_shared.py similarity index 100% rename from crowdin/generate_shared.py rename to src/session_ops/crowdin/generate_shared.py diff --git a/crowdin/parse_xliff.py b/src/session_ops/crowdin/parse_xliff.py similarity index 99% rename from crowdin/parse_xliff.py rename to src/session_ops/crowdin/parse_xliff.py index b538156..232ffbf 100644 --- a/crowdin/parse_xliff.py +++ b/src/session_ops/crowdin/parse_xliff.py @@ -7,7 +7,7 @@ from typing import Dict, List, Any, Set, Tuple from dataclasses import dataclass, field from colorama import Fore, Style -from generate_shared import setup_generation, load_glossary_dict +from session_ops.crowdin.generate_shared import setup_generation, load_glossary_dict XLIFF_NAMESPACE = {'ns': 'urn:oasis:names:tc:xliff:document:1.2'} diff --git a/crowdin/report_multiple_translations.py b/src/session_ops/crowdin/report_multiple_translations.py similarity index 98% rename from crowdin/report_multiple_translations.py rename to src/session_ops/crowdin/report_multiple_translations.py index 18e1c37..eb56660 100644 --- a/crowdin/report_multiple_translations.py +++ b/src/session_ops/crowdin/report_multiple_translations.py @@ -32,9 +32,9 @@ --dry-run the payloads are printed instead of posted (no webhook required). Usage: - python report_multiple_translations.py --dry-run # scan + print, post nothing - python report_multiple_translations.py # scan + post to Discord - python report_multiple_translations.py --json multi.json # also write findings JSON + crowdin-report-duplicates --dry-run # scan + print, post nothing + crowdin-report-duplicates # scan + post to Discord + crowdin-report-duplicates --json multi.json # also write findings JSON """ import argparse import collections @@ -48,8 +48,7 @@ import requests -sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__)))) -from shared import discord, retry # noqa: E402 +from session_ops.shared import discord, retry API = "https://api.crowdin.com/api/v2" DEFAULT_PROJECT = "618696" diff --git a/src/session_ops/github_prs/__init__.py b/src/session_ops/github_prs/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/github_prs/digest.py b/src/session_ops/github_prs/digest.py similarity index 97% rename from github_prs/digest.py rename to src/session_ops/github_prs/digest.py index e6be7e8..e648951 100755 --- a/github_prs/digest.py +++ b/src/session_ops/github_prs/digest.py @@ -29,13 +29,13 @@ Usage: # real run (what the timer does) - python digest.py + github-prs-digest # fetch and render, print the payload, post nothing - python digest.py --dry-run + github-prs-digest --dry-run # what the weekday timer does: a window covering the weekend, deduped - python digest.py --window-hours 72 --state /var/lib/github-prs/seen.json + github-prs-digest --window-hours 72 --state /var/lib/github-prs/seen.json """ import argparse import json @@ -46,11 +46,10 @@ import requests -sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__)))) -from shared import discord, state as dedup # noqa: E402 -from shared.discord import MAX_MESSAGE_TEXT_CHARS, clip # noqa: E402 -from shared.env import get_env # noqa: E402 -from shared.retry import request_with_retry # noqa: E402 +from session_ops.shared import discord, state as dedup +from session_ops.shared.discord import MAX_MESSAGE_TEXT_CHARS, clip +from session_ops.shared.env import get_env +from session_ops.shared.retry import request_with_retry API = "https://api.github.com" DEFAULT_ORG = "session-foundation" diff --git a/github_prs/maintainers.txt b/src/session_ops/github_prs/maintainers.txt similarity index 100% rename from github_prs/maintainers.txt rename to src/session_ops/github_prs/maintainers.txt diff --git a/src/session_ops/monitor/__init__.py b/src/session_ops/monitor/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/deploy/alert.py b/src/session_ops/monitor/alert.py similarity index 94% rename from deploy/alert.py rename to src/session_ops/monitor/alert.py index e8254b7..99fe974 100644 --- a/deploy/alert.py +++ b/src/session_ops/monitor/alert.py @@ -18,17 +18,15 @@ is what it always was, rather than failing. Usage: - alert.py [journal-unit] + session-ops-alert [journal-unit] """ import os import socket import subprocess import sys -sys.path.insert(0, os.path.join(os.path.dirname(os.path.dirname( - os.path.abspath(__file__))), "zendesk_triage")) -import requests # noqa: E402 -import triage # noqa: E402 +import requests +from session_ops.zendesk import triage # A dead login reads as a broken job unless the alert names it: the job is fine and @@ -103,7 +101,7 @@ def build_message(unit, host, journal_unit=None, detail=""): def main(): args = [arg.strip() for arg in sys.argv[1:]] if not args or len(args) > 2 or not args[0]: - sys.exit("usage: alert.py [journal-unit]") + sys.exit("usage: session-ops-alert [journal-unit]") webhook = (os.environ.get("ALERT_DISCORD_WEBHOOK_URL") or triage.get_env("ZENDESK_DISCORD_WEBHOOK_URL")) detail = last_job_line(journal_tail(args[-1])) diff --git a/deploy/jobs.toml b/src/session_ops/monitor/jobs.toml similarity index 100% rename from deploy/jobs.toml rename to src/session_ops/monitor/jobs.toml diff --git a/deploy/silence.py b/src/session_ops/monitor/silence.py similarity index 95% rename from deploy/silence.py rename to src/session_ops/monitor/silence.py index 2219ca4..b55580a 100644 --- a/deploy/silence.py +++ b/src/session_ops/monitor/silence.py @@ -20,8 +20,8 @@ ALERT_DISCORD_WEBHOOK_URL where the alert goes (not needed with --dry-run) Usage: - silence.py --state /var/lib/session-ops/silence/state.json - silence.py --dry-run # print each job's age and the alert, post nothing + session-ops-silence --state /var/lib/session-ops/silence/state.json + session-ops-silence --dry-run # print each job's age and the alert, post nothing """ import argparse import json @@ -34,9 +34,8 @@ import requests -sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__)))) -from shared import discord # noqa: E402 -from shared.env import get_env # noqa: E402 +from session_ops.shared import discord +from session_ops.shared.env import get_env REGISTRY = os.path.join(os.path.dirname(os.path.abspath(__file__)), "jobs.toml") STAMPS_DIR = "/var/lib/session-ops/stamps" diff --git a/src/session_ops/shared/__init__.py b/src/session_ops/shared/__init__.py new file mode 100644 index 0000000..e9a0de1 --- /dev/null +++ b/src/session_ops/shared/__init__.py @@ -0,0 +1 @@ +"""Helpers every job shares: HTTP retry, Discord posting, dedup state, env config.""" diff --git a/shared/discord.py b/src/session_ops/shared/discord.py similarity index 98% rename from shared/discord.py rename to src/session_ops/shared/discord.py index d9662e1..90c3a12 100644 --- a/shared/discord.py +++ b/src/session_ops/shared/discord.py @@ -12,7 +12,7 @@ import requests -from shared.retry import request_with_retry +from session_ops.shared.retry import request_with_retry COMPONENTS_V2_FLAG = 1 << 15 CONTAINER = 17 diff --git a/shared/env.py b/src/session_ops/shared/env.py similarity index 100% rename from shared/env.py rename to src/session_ops/shared/env.py diff --git a/shared/retry.py b/src/session_ops/shared/retry.py similarity index 100% rename from shared/retry.py rename to src/session_ops/shared/retry.py diff --git a/shared/state.py b/src/session_ops/shared/state.py similarity index 100% rename from shared/state.py rename to src/session_ops/shared/state.py diff --git a/shared/testing.py b/src/session_ops/shared/testing.py similarity index 86% rename from shared/testing.py rename to src/session_ops/shared/testing.py index 277076d..a855f05 100644 --- a/shared/testing.py +++ b/src/session_ops/shared/testing.py @@ -1,6 +1,5 @@ """Fakes for the tests of every script that talks HTTP through `shared`.""" import json -import os import time from datetime import datetime @@ -90,29 +89,6 @@ def __exit__(self, *exc): return False -GOLDENS_DIR = os.path.join(os.path.dirname(os.path.dirname(os.path.abspath(__file__))), - "tests", "goldens") - - -def load_golden_json(relpath): - with open(os.path.join(GOLDENS_DIR, relpath), encoding="utf-8") as handle: - return json.load(handle) - - -def assert_golden(case, relpath, actual): - """Compare `actual` with tests/goldens/; UPDATE_GOLDENS=1 rewrites it.""" - path = os.path.join(GOLDENS_DIR, relpath) - if os.environ.get("UPDATE_GOLDENS"): - os.makedirs(os.path.dirname(path), exist_ok=True) - with open(path, "w", encoding="utf-8") as handle: - handle.write(actual) - return - with open(path, encoding="utf-8") as handle: - expected = handle.read() - case.maxDiff = None - case.assertEqual(expected, actual, f"{relpath} differs (UPDATE_GOLDENS=1 to accept)") - - def request_key(method, url, params=None, data=None, json_body=None): """What identifies a request in a recording: method, URL, query and body.""" if json_body is not None: diff --git a/src/session_ops/sogs/__init__.py b/src/session_ops/sogs/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/sogs_moderation/ban.py b/src/session_ops/sogs/ban.py similarity index 98% rename from sogs_moderation/ban.py rename to src/session_ops/sogs/ban.py index 78c6597..24b8974 100644 --- a/sogs_moderation/ban.py +++ b/src/session_ops/sogs/ban.py @@ -91,11 +91,11 @@ ## Usage - python ban.py 05abc...def # one ID - python ban.py 05abc...def 05123...456 # several - python ban.py --from-file ids.txt # one ID per line, # comments allowed - python ban.py --dry-run 05abc...def # print the steps, send no ban or deletion - python ban.py --unban 05abc...def # undo (does not restore messages) + sogs-ban 05abc...def # one ID + sogs-ban 05abc...def 05123...456 # several + sogs-ban --from-file ids.txt # one ID per line, # comments allowed + sogs-ban --dry-run 05abc...def # print the steps, send no ban or deletion + sogs-ban --unban 05abc...def # undo (does not restore messages) """ import argparse @@ -505,7 +505,7 @@ def main(): return 1 if failures else 0 -if __name__ == '__main__': +def cli(): try: sys.exit(main()) except SogsError as e: @@ -513,3 +513,7 @@ def main(): sys.exit(2) except KeyboardInterrupt: sys.exit(130) + + +if __name__ == '__main__': + cli() diff --git a/sogs_moderation/perms.py b/src/session_ops/sogs/perms.py similarity index 96% rename from sogs_moderation/perms.py rename to src/session_ops/sogs/perms.py index a384e96..8170478 100644 --- a/sogs_moderation/perms.py +++ b/src/session_ops/sogs/perms.py @@ -8,8 +8,8 @@ It is also the way to mute one account in a room (`--write off`) without banning it. - python perms.py --room session-updates --write on --upload on 05 - python perms.py --room session-updates --write default --upload default 05 + sogs-perms --room session-updates --write on --upload on 05 + sogs-perms --room session-updates --write default --upload default 05 `on` grants, `off` denies, `default` removes the override and returns the account to the room's own default. A room with `default_write` false is read-only, so `off` and @@ -38,7 +38,7 @@ import argparse import sys -from ban import ( +from session_ops.sogs.ban import ( SESSION_ID_RE, SOGS_PUBKEY, SOGS_URL, @@ -156,7 +156,7 @@ def main(): return 1 if failures else 0 -if __name__ == '__main__': +def cli(): try: sys.exit(main()) except SogsError as e: @@ -164,3 +164,7 @@ def main(): sys.exit(2) except KeyboardInterrupt: sys.exit(130) + + +if __name__ == '__main__': + cli() diff --git a/src/session_ops/zendesk/__init__.py b/src/session_ops/zendesk/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/zendesk_triage/note_reply.py b/src/session_ops/zendesk/note_reply.py similarity index 99% rename from zendesk_triage/note_reply.py rename to src/session_ops/zendesk/note_reply.py index 6d9f7a9..5083751 100644 --- a/zendesk_triage/note_reply.py +++ b/src/session_ops/zendesk/note_reply.py @@ -67,8 +67,7 @@ import sys import textwrap -sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))) -import triage # noqa: E402 (needs the path insert above) +from session_ops.zendesk import triage DEFAULT_MODEL = "claude-sonnet-5" COMPOSE_TIMEOUT_SECONDS = 240 diff --git a/zendesk_triage/relay.py b/src/session_ops/zendesk/relay.py similarity index 98% rename from zendesk_triage/relay.py rename to src/session_ops/zendesk/relay.py index 2a29120..41032f7 100644 --- a/zendesk_triage/relay.py +++ b/src/session_ops/zendesk/relay.py @@ -54,7 +54,7 @@ from fastapi import BackgroundTasks, FastAPI, Request, Response from starlette.concurrency import run_in_threadpool -NOTE_SCRIPT = os.path.join(os.path.dirname(os.path.abspath(__file__)), "note_reply.py") +NOTE_MODULE = "session_ops.zendesk.note_reply" # How stale a signed request may be. The signature covers the timestamp, so this # cannot be forged β€” it bounds *replay* of a request that was genuinely signed. @@ -115,7 +115,7 @@ def run_note_reply(ticket_id): ticket, not at an open dialog β€” so the outcome goes to the journal and, where note_reply.py got far enough to write one, to the ticket itself. """ - command = [sys.executable, NOTE_SCRIPT, "--ticket", str(ticket_id)] + command = [sys.executable, "-m", NOTE_MODULE, "--ticket", str(ticket_id)] if dry_run_requested(): command.append("--dry-run") try: diff --git a/zendesk_triage/resolve_reviews.py b/src/session_ops/zendesk/resolve_reviews.py similarity index 98% rename from zendesk_triage/resolve_reviews.py rename to src/session_ops/zendesk/resolve_reviews.py index b3fffa7..4a48c7f 100644 --- a/zendesk_triage/resolve_reviews.py +++ b/src/session_ops/zendesk/resolve_reviews.py @@ -59,19 +59,18 @@ Usage: # report what would be solved, touch nothing (the default) - python resolve_reviews.py + zendesk-resolve-reviews # actually solve them - python resolve_reviews.py --apply + zendesk-resolve-reviews --apply # first real run: small, observable - python resolve_reviews.py --apply --max-tickets 5 + zendesk-resolve-reviews --apply --max-tickets 5 # solve them without telling the channel - python resolve_reviews.py --apply --no-discord + zendesk-resolve-reviews --apply --no-discord """ import argparse -import os import sys import time from datetime import datetime, timedelta, timezone @@ -79,8 +78,7 @@ import requests -sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))) -import triage # noqa: E402 (needs the path insert above) +from session_ops.zendesk import triage # Reviews at or above this rating carry nothing to act on. Fixed rather than a flag: # 3β˜… and below are what the triage treats as bug reports in disguise, so lowering the diff --git a/zendesk_triage/triage.py b/src/session_ops/zendesk/triage.py similarity index 99% rename from zendesk_triage/triage.py rename to src/session_ops/zendesk/triage.py index 9956dce..b867a40 100644 --- a/zendesk_triage/triage.py +++ b/src/session_ops/zendesk/triage.py @@ -50,24 +50,24 @@ Usage: # real run (CI): reads everything from the environment - python triage.py + zendesk-triage # local dry run: fetch + analyze, print the Discord payload, post nothing - python triage.py --dry-run + zendesk-triage --dry-run # exercise the whole job without posting, and without printing tickets - python triage.py --no-discord + zendesk-triage --no-discord # what the scheduled weekday run does: 72h window, skipping unchanged repeats - python triage.py --window-hours 72 --state .triage-state/seen.json + zendesk-triage --window-hours 72 --state .triage-state/seen.json # or an explicit query, which overrides --window-hours - python triage.py --query "type:ticket status:open tags:bug" --max-tickets 50 + zendesk-triage --query "type:ticket status:open tags:bug" --max-tickets 50 # split classification out entirely: dump the batch, classify it by hand, # feed the findings back in to render - python triage.py --dump-batch /tmp/batch.json --max-tickets 20 - python triage.py --findings /tmp/findings.json --dry-run + zendesk-triage --dump-batch /tmp/batch.json --max-tickets 20 + zendesk-triage --findings /tmp/findings.json --dry-run """ import argparse import json @@ -81,11 +81,10 @@ import requests -sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__)))) -from shared import discord, state as dedup # noqa: E402 -from shared.discord import clip, post_to_discord # noqa: E402 -from shared.env import get_env # noqa: E402 -from shared.retry import request_with_retry # noqa: E402 +from session_ops.shared import discord, state as dedup +from session_ops.shared.discord import clip, post_to_discord +from session_ops.shared.env import get_env +from session_ops.shared.retry import request_with_retry # The channel AppFollow imports app-store reviews on. Identified reviews with no # false positives in a 3,662-ticket sample; tags did not (only 287 carried one). diff --git a/tests/__init__.py b/tests/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/tests/crowdin/__init__.py b/tests/crowdin/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/crowdin/test_goldens.py b/tests/crowdin/test_crowdin_goldens.py similarity index 79% rename from crowdin/test_goldens.py rename to tests/crowdin/test_crowdin_goldens.py index 51d4b51..7f8e08b 100644 --- a/crowdin/test_goldens.py +++ b/tests/crowdin/test_crowdin_goldens.py @@ -1,6 +1,5 @@ """ - python -m unittest discover # from crowdin/ - UPDATE_GOLDENS=1 python -m unittest test_goldens # accept new output + UPDATE_GOLDENS=1 uv run python -m unittest tests.crowdin.test_crowdin_goldens # accept new output The recordings are shaped like Crowdin's API responses rather than captured live. """ @@ -8,7 +7,6 @@ import io import json import os -import runpy import sys import tempfile import unittest @@ -17,11 +15,10 @@ import colorama import requests -HERE = os.path.dirname(os.path.abspath(__file__)) -sys.path.insert(0, HERE) -sys.path.insert(0, os.path.dirname(HERE)) -import report_multiple_translations as report # noqa: E402 -from shared.testing import RecordedSession, assert_golden, load_golden_json # noqa: E402 +from session_ops.crowdin import download_translations_from_crowdin as download +from session_ops.crowdin import report_multiple_translations as report +from session_ops.shared.testing import RecordedSession +from tests.golden import assert_golden, load_golden_json def argv_for(recording, tmp): @@ -51,11 +48,9 @@ def test_export_payloads_and_written_files(self): with tempfile.TemporaryDirectory() as tmp: with mock.patch.object(requests, "get", session.get), \ mock.patch.object(requests, "post", session.post), \ - mock.patch.object(sys, "argv", ["download", *argv_for(recording, tmp)]), \ contextlib.redirect_stdout(io.StringIO()) as out: try: - runpy.run_path(os.path.join(HERE, "download_translations_from_crowdin.py"), - run_name="__main__") + download.main(argv_for(recording, tmp)) except SystemExit as exc: self.fail(f"download exited {exc.code}:\n{out.getvalue()}") finally: diff --git a/crowdin/test_language_list.py b/tests/crowdin/test_language_list.py similarity index 92% rename from crowdin/test_language_list.py rename to tests/crowdin/test_language_list.py index 5a3be06..469ac75 100644 --- a/crowdin/test_language_list.py +++ b/tests/crowdin/test_language_list.py @@ -3,18 +3,15 @@ Stdlib unittest so the repo needs no test dependency. Run from anywhere: - python -m unittest discover -s crowdin -v + uv run python -m unittest tests.crowdin.test_language_list Offline: CLDR ships with Babel, so nothing here talks to Crowdin. """ -import os -import sys import unittest -sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))) -from babel import Locale # noqa: E402 (needs the path insert above) -from generate_shared import cldr_locale, is_rtl # noqa: E402 -from generate_language_list import ( # noqa: E402 +from babel import Locale +from session_ops.crowdin.generate_shared import cldr_locale, is_rtl +from session_ops.crowdin.generate_language_list import ( build_language_data, display_name, locale_keys, diff --git a/crowdin/test_report_multiple_translations.py b/tests/crowdin/test_report_multiple_translations.py similarity index 88% rename from crowdin/test_report_multiple_translations.py rename to tests/crowdin/test_report_multiple_translations.py index 50a0381..70dd844 100644 --- a/crowdin/test_report_multiple_translations.py +++ b/tests/crowdin/test_report_multiple_translations.py @@ -1,18 +1,14 @@ """ - cd crowdin && python -m unittest discover + uv run python -m unittest tests.crowdin.test_report_multiple_translations """ import contextlib import io -import os -import sys import unittest import requests -sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))) -sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__)))) -import report_multiple_translations as report # noqa: E402 -from shared.testing import FakeResponse, FakeSession, NoSleep, Patched # noqa: E402 +from session_ops.crowdin import report_multiple_translations as report +from session_ops.shared.testing import FakeResponse, FakeSession, NoSleep, Patched class ApiResponse(FakeResponse): diff --git a/tests/github_prs/__init__.py b/tests/github_prs/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/github_prs/test_digest.py b/tests/github_prs/test_digest.py similarity index 98% rename from github_prs/test_digest.py rename to tests/github_prs/test_digest.py index 0cc2b82..7d82e0d 100644 --- a/github_prs/test_digest.py +++ b/tests/github_prs/test_digest.py @@ -1,19 +1,17 @@ """ - python -m unittest discover # from github_prs/ + uv run python -m unittest tests.github_prs.test_digest """ import contextlib import io import json import os -import sys import tempfile import unittest from datetime import datetime, timedelta, timezone from unittest import mock -sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__)))) -import digest # noqa: E402 -from shared import discord # noqa: E402 +from session_ops.github_prs import digest +from session_ops.shared import discord NOW = datetime(2026, 9, 24, 12, 0, tzinfo=timezone.utc) CUTOFF = NOW - timedelta(hours=25) diff --git a/tests/golden.py b/tests/golden.py new file mode 100644 index 0000000..52891c5 --- /dev/null +++ b/tests/golden.py @@ -0,0 +1,25 @@ +"""Byte-for-byte comparison of a job's output with tests/goldens/.""" +import json +import os + +GOLDENS_DIR = os.path.join(os.path.dirname(os.path.abspath(__file__)), "goldens") + + +def load_golden_json(relpath): + with open(os.path.join(GOLDENS_DIR, relpath), encoding="utf-8") as handle: + return json.load(handle) + + +def assert_golden(case, relpath, actual): + """Compare `actual` with tests/goldens/; UPDATE_GOLDENS=1 rewrites it.""" + path = os.path.join(GOLDENS_DIR, relpath) + if os.environ.get("UPDATE_GOLDENS"): + os.makedirs(os.path.dirname(path), exist_ok=True) + with open(path, "w", encoding="utf-8") as handle: + handle.write(actual) + return + with open(path, encoding="utf-8") as handle: + expected = handle.read() + case.maxDiff = None + case.assertEqual(expected, actual, f"{relpath} differs (UPDATE_GOLDENS=1 to accept)") + diff --git a/tests/goldens/README.md b/tests/goldens/README.md index 82b1c21..d776128 100644 --- a/tests/goldens/README.md +++ b/tests/goldens/README.md @@ -6,9 +6,9 @@ changes a payload fails. | directory | job | recording | | ----------- | ------------------------------------------- | ------------------------------------------ | -| `digest/` | `github_prs/digest.py --dry-run`, 720 h | live org, 2026-09-25, trimmed to read fields | -| `report/` | `report_multiple_translations.py --locales de` | synthetic, shaped like Crowdin's API | -| `download/` | `download_translations_from_crowdin.py` as the sync runs it | synthetic | +| `digest/` | `github-prs-digest --dry-run`, 720 h | live org, 2026-09-25, trimmed to read fields | +| `report/` | `crowdin-report-duplicates --locales de` | synthetic, shaped like Crowdin's API | +| `download/` | `crowdin-download` as the sync runs it | synthetic | Requests are matched by method, URL, query and body rather than by order, because the Crowdin scripts fan out across threads. A request the recording lacks fails the test and @@ -17,6 +17,6 @@ names it. To accept a deliberate change, rerun the suite with `UPDATE_GOLDENS=1` and review the diff: ```sh -cd github_prs && UPDATE_GOLDENS=1 python -m unittest test_golden -cd crowdin && UPDATE_GOLDENS=1 python -m unittest test_goldens +UPDATE_GOLDENS=1 uv run python -m unittest tests.github_prs.test_digest_golden +UPDATE_GOLDENS=1 uv run python -m unittest tests.crowdin.test_crowdin_goldens ``` diff --git a/tests/monitor/__init__.py b/tests/monitor/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/deploy/test_alert.py b/tests/monitor/test_alert.py similarity index 98% rename from deploy/test_alert.py rename to tests/monitor/test_alert.py index 45bdd66..fba7b54 100644 --- a/deploy/test_alert.py +++ b/tests/monitor/test_alert.py @@ -1,11 +1,11 @@ """ - python -m unittest discover # from deploy/ + uv run python -m unittest tests.monitor.test_alert """ import subprocess import unittest from unittest import mock -import alert +from session_ops.monitor import alert # What the digest itself left in the journal on 2026-09-23. JOURNAL = """\ diff --git a/deploy/test_silence.py b/tests/monitor/test_silence.py similarity index 94% rename from deploy/test_silence.py rename to tests/monitor/test_silence.py index d5b547d..5a37458 100644 --- a/deploy/test_silence.py +++ b/tests/monitor/test_silence.py @@ -1,11 +1,11 @@ """ - python -m unittest discover # from deploy/ + uv run python -m unittest tests.monitor.test_silence """ import os import tempfile import unittest -import silence +from session_ops.monitor import silence HOUR = 3600 NOW = 1_790_000_000.0 @@ -85,7 +85,8 @@ class TestRegistry(unittest.TestCase): def test_the_shipped_registry_matches_the_shipped_units(self): """A job listed without a unit, or a timer missing from the list, is silence nobody would detect.""" - here = os.path.dirname(os.path.abspath(__file__)) + here = os.path.join(os.path.dirname(os.path.dirname(os.path.dirname( + os.path.abspath(__file__)))), "deploy") names = {job["name"] for job in silence.load_jobs(silence.REGISTRY)} timers = {f[:-len(".timer")] for f in os.listdir(here) if f.endswith(".timer") and f != "session-ops-silence.timer"} diff --git a/tests/monitor/test_units.py b/tests/monitor/test_units.py new file mode 100644 index 0000000..8e1e483 --- /dev/null +++ b/tests/monitor/test_units.py @@ -0,0 +1,26 @@ +import glob +import os +import re +import tomllib +import unittest + +ROOT = os.path.dirname(os.path.dirname(os.path.dirname(os.path.abspath(__file__)))) +VENV_BIN = "/opt/zendesk/.venv/bin/" + + +class TestUnits(unittest.TestCase): + def test_every_venv_command_a_unit_runs_is_a_declared_entry_point(self): + """A unit only runs on the host, so a renamed entry point surfaces as a failed + timer rather than at review time.""" + with open(os.path.join(ROOT, "pyproject.toml"), "rb") as handle: + declared = set(tomllib.load(handle)["project"]["scripts"]) | {"uvicorn"} + used = set() + for unit in glob.glob(os.path.join(ROOT, "deploy", "*.service")): + with open(unit, encoding="utf-8") as handle: + used |= set(re.findall(re.escape(VENV_BIN) + r"([\w-]+)", handle.read())) + self.assertTrue(used) + self.assertEqual(used - declared, set()) + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/shared/__init__.py b/tests/shared/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/shared/test_discord.py b/tests/shared/test_discord.py similarity index 96% rename from shared/test_discord.py rename to tests/shared/test_discord.py index 41330c4..b96a475 100644 --- a/shared/test_discord.py +++ b/tests/shared/test_discord.py @@ -1,13 +1,10 @@ import contextlib import io -import os -import sys import unittest from urllib.parse import parse_qsl, urlsplit -sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__)))) -from shared import discord # noqa: E402 -from shared.testing import FakeResponse, FakeSession, NoSleep # noqa: E402 +from session_ops.shared import discord +from session_ops.shared.testing import FakeResponse, FakeSession, NoSleep class TestComponentsWebhookUrl(unittest.TestCase): diff --git a/shared/test_retry.py b/tests/shared/test_retry.py similarity index 96% rename from shared/test_retry.py rename to tests/shared/test_retry.py index f1fa3df..d9ccef5 100644 --- a/shared/test_retry.py +++ b/tests/shared/test_retry.py @@ -1,8 +1,6 @@ """ - cd shared && python -m unittest discover + uv run python -m unittest tests.shared.test_retry """ -import os -import sys import time import unittest from email.utils import format_datetime @@ -10,9 +8,8 @@ import requests -sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__)))) -from shared import retry # noqa: E402 -from shared.testing import FakeResponse, FakeSession, NoSleep # noqa: E402 +from session_ops.shared import retry +from session_ops.shared.testing import FakeResponse, FakeSession, NoSleep class TestRequestWithRetry(unittest.TestCase): diff --git a/shared/test_state.py b/tests/shared/test_state.py similarity index 96% rename from shared/test_state.py rename to tests/shared/test_state.py index 1097085..a47d810 100644 --- a/shared/test_state.py +++ b/tests/shared/test_state.py @@ -2,12 +2,10 @@ import io import json import os -import sys import tempfile import unittest -sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__)))) -from shared import state as dedup # noqa: E402 +from session_ops.shared import state as dedup VERSION = 7 diff --git a/tests/sogs/__init__.py b/tests/sogs/__init__.py new file mode 100644 index 0000000..817e422 --- /dev/null +++ b/tests/sogs/__init__.py @@ -0,0 +1,7 @@ +import importlib.util +import unittest + +# session_util is a deb, not a wheel (see README, Community Bans). CI's sogs job +# installs it and checks the import first, so this skip cannot hide a broken suite. +if importlib.util.find_spec("session_util") is None: + raise unittest.SkipTest("session_util is not installed") diff --git a/sogs_moderation/test_ban.py b/tests/sogs/test_ban.py similarity index 99% rename from sogs_moderation/test_ban.py rename to tests/sogs/test_ban.py index f4727d8..94f475b 100644 --- a/sogs_moderation/test_ban.py +++ b/tests/sogs/test_ban.py @@ -1,5 +1,5 @@ """ - python -m unittest discover # from sogs_moderation/ + uv run python -m unittest tests.sogs.test_ban The signing vectors are pysogs' own: they are the headers printed by its contrib/auth-example.py for the seed, server pubkey, nonce and timestamp below. Any @@ -16,7 +16,7 @@ from nacl.exceptions import BadSignatureError from nacl.signing import SigningKey, VerifyKey -import ban +from session_ops.sogs import ban SEED = 'c010d89eccbaf5d1c6d19df766c6eedf965d4a28a56f87c9fc819edb59896dd9' SERVER_PUBKEY = 'c3b3c6f32f0ab5a57f853cc4f30f5da7fda5624b0c77b3fb0829de562ada081d' diff --git a/sogs_moderation/test_perms.py b/tests/sogs/test_perms.py similarity index 93% rename from sogs_moderation/test_perms.py rename to tests/sogs/test_perms.py index 0b9ca1e..8af4d9c 100644 --- a/sogs_moderation/test_perms.py +++ b/tests/sogs/test_perms.py @@ -1,11 +1,11 @@ """ - python -m unittest discover # from sogs_moderation/ + uv run python -m unittest tests.sogs.test_perms """ import unittest -import perms -from test_ban import BLINDED_ABS, BLINDED_NEG, SERVER_PUBKEY, SESSION_ID +from session_ops.sogs import perms +from tests.sogs.test_ban import BLINDED_ABS, BLINDED_NEG, SERVER_PUBKEY, SESSION_ID def args(**kwargs): diff --git a/tests/zendesk/__init__.py b/tests/zendesk/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/zendesk_triage/test_note_reply.py b/tests/zendesk/test_note_reply.py similarity index 99% rename from zendesk_triage/test_note_reply.py rename to tests/zendesk/test_note_reply.py index 54abf11..0f80624 100644 --- a/zendesk_triage/test_note_reply.py +++ b/tests/zendesk/test_note_reply.py @@ -3,7 +3,7 @@ Stdlib unittest, offline, same stubs as the rest. Run from anywhere: - python -m unittest discover -s zendesk_triage -v + uv run python -m unittest tests.zendesk.test_note_reply This publishes comments that email a customer, so the tests are about the guards rather than the happy path. Four of them exist because getting them wrong is how this @@ -18,13 +18,11 @@ import html import os import re -import sys import unittest -sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))) -import note_reply # noqa: E402 -import triage # noqa: E402 -from test_triage import FakeResponse, FakeSession, Patched # noqa: E402 +from session_ops.zendesk import note_reply +from session_ops.zendesk import triage +from tests.zendesk.test_triage import FakeResponse, FakeSession, Patched API_USER = 901790886886 AGENT = 555 diff --git a/zendesk_triage/test_relay.py b/tests/zendesk/test_relay.py similarity index 98% rename from zendesk_triage/test_relay.py rename to tests/zendesk/test_relay.py index 936593e..c925350 100644 --- a/zendesk_triage/test_relay.py +++ b/tests/zendesk/test_relay.py @@ -5,7 +5,7 @@ actually executed, and requests go through FastAPI's TestClient rather than a socket. Run from anywhere: - python -m unittest discover -s zendesk_triage -v + uv run python -m unittest tests.zendesk.test_relay This is the front door to something that writes public comments to customer tickets, so the tests are about the gate: that an unsigned, tampered, stale or wrongly-signed @@ -18,14 +18,12 @@ import hmac import json import os -import sys import unittest from fastapi.testclient import TestClient -sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))) -import relay # noqa: E402 -from test_triage import Patched # noqa: E402 +from session_ops.zendesk import relay +from tests.zendesk.test_triage import Patched # The relay only ever compares this against what it computes, so its value is # arbitrary β€” but it must not be empty, since an unset secret refuses everything. diff --git a/zendesk_triage/test_resolve_reviews.py b/tests/zendesk/test_resolve_reviews.py similarity index 98% rename from zendesk_triage/test_resolve_reviews.py rename to tests/zendesk/test_resolve_reviews.py index 5f9e749..f8e0aa6 100644 --- a/zendesk_triage/test_resolve_reviews.py +++ b/tests/zendesk/test_resolve_reviews.py @@ -4,7 +4,7 @@ Stdlib unittest, same as test_triage.py. Everything is offline β€” the Zendesk calls run against a stub session. Run from anywhere: - python -m unittest discover -s zendesk_triage -v + uv run python -m unittest tests.zendesk.test_resolve_reviews This script writes to Zendesk, so the tests lean on the guards rather than the happy path: that a dry run cannot PUT, that only positive app-store reviews are selected, @@ -13,15 +13,13 @@ import inspect import os import re -import sys import unittest from datetime import datetime, timezone from urllib.parse import quote -sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))) -import resolve_reviews # noqa: E402 -import triage # noqa: E402 -from test_triage import ( # noqa: E402 +from session_ops.zendesk import resolve_reviews +from session_ops.zendesk import triage +from tests.zendesk.test_triage import ( ROOT, FakeResponse, FakeSession, NoSleep, unit_commands) diff --git a/zendesk_triage/test_triage.py b/tests/zendesk/test_triage.py similarity index 99% rename from zendesk_triage/test_triage.py rename to tests/zendesk/test_triage.py index f9e3151..4667ddc 100644 --- a/zendesk_triage/test_triage.py +++ b/tests/zendesk/test_triage.py @@ -3,7 +3,7 @@ Stdlib unittest so the repo needs no test dependency. Run from anywhere: - python -m unittest discover -s zendesk_triage -v + uv run python -m unittest tests.zendesk.test_triage Everything here is offline β€” no Zendesk, Claude, or Discord calls. The fetch tests drive fetch_tickets with a stub session instead. @@ -14,18 +14,15 @@ import json import os import re -import sys import tempfile import unittest from datetime import datetime, timedelta, timezone import requests -sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))) -sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__)))) -import triage # noqa: E402 (needs the path insert above) -from shared import discord # noqa: E402 -from shared.testing import ( # noqa: E402 +from session_ops.zendesk import triage +from session_ops.shared import discord +from session_ops.shared.testing import ( FakeResponse, FakeSession, NoSleep, NonJsonResponse, Patched) @@ -1812,7 +1809,7 @@ def test_a_wedged_cli_does_not_hang_the_run(self): # ---- Workflow wiring ------------------------------------------------------- -ROOT = os.path.dirname(os.path.dirname(os.path.abspath(__file__))) +ROOT = os.path.dirname(os.path.dirname(os.path.dirname(os.path.abspath(__file__)))) def unit_commands(unit="zendesk-digest.service"): @@ -1840,8 +1837,8 @@ class TestDigestOrdering(unittest.TestCase): def test_the_resolver_runs_before_the_digest(self): commands = unit_commands() self.assertEqual(len(commands), 2, "expected exactly resolve then triage") - self.assertIn("resolve_reviews.py", commands[0]) - self.assertIn("triage.py", commands[1]) + self.assertIn("zendesk-resolve-reviews", commands[0]) + self.assertIn("zendesk-triage", commands[1]) def test_a_failed_resolve_does_not_cost_the_digest(self): """Type=oneshot stops at the first failing ExecStart, which would make @@ -1856,7 +1853,7 @@ def test_a_failed_resolve_is_still_reported(self): resolver = unit_commands()[0] self.assertFalse(resolver.startswith("-"), "a - prefix swallows the failure instead of reporting it") - self.assertIn("alert.py", resolver) + self.assertIn("session-ops-alert", resolver) def test_the_digest_is_not_prevented_from_failing_loudly(self): """The reverse for triage itself: a swallowed failure there is a silent day diff --git a/uv.lock b/uv.lock new file mode 100644 index 0000000..1d484e0 --- /dev/null +++ b/uv.lock @@ -0,0 +1,633 @@ +version = 1 +revision = 3 +requires-python = ">=3.12" + +[[package]] +name = "annotated-doc" +version = "0.0.5" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/5a/8e/38aa427ed5402449e226975b649c5dc73ccadfefeb95e6aecb8f8ea4b6b6/annotated_doc-0.0.5.tar.gz", hash = "sha256:c7e58ce09192557605d8bbd92836d7e1d520ac9580096042c0bfd197efacf1bb", size = 10758, upload-time = "2026-07-28T13:50:58.129Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/3e/30/e900b21425a860e195f32e37657aa1f7c7f2b1bfb26f03ca209b90933c06/annotated_doc-0.0.5-py3-none-any.whl", hash = "sha256:117bac03a25ede5df5440e855b32d556049ca169ead221505badf432fed4b101", size = 5302, upload-time = "2026-07-28T13:50:57.239Z" }, +] + +[[package]] +name = "annotated-types" +version = "0.8.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/5f/56/a8120250d128bed162cd73c76d45f6ef9991f3e068f62a8ee060afa3104a/annotated_types-0.8.0.tar.gz", hash = "sha256:13b2beaad985e05e2d6407ee4c4f35590b11f8d693a258a561055cac8f64cab7", size = 15893, upload-time = "2026-07-23T20:16:13.995Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/99/91/8acff4f5e50511b911bbccb72b8628a49c68ce14148cd9f6431094859a90/annotated_types-0.8.0-py3-none-any.whl", hash = "sha256:f072f4d804ea359e4eaf198b1af7a8b0943881a87f31bb764f8bf219bb9419e0", size = 13427, upload-time = "2026-07-23T20:16:12.938Z" }, +] + +[[package]] +name = "anyio" +version = "4.15.1" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "idna" }, + { name = "typing-extensions", marker = "python_full_version < '3.15'" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/a9/d2/f4d173e22df740bc37b1db102b386ba719b66e95b0f0d751f556b387e6d2/anyio-4.15.1.tar.gz", hash = "sha256:9f28306018cbd6d329e64a36d58256edff76dd996fe423bc957326e578b82a94", size = 276966, upload-time = "2026-09-05T10:42:39.44Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/12/b8/4bd346e22b28902df4d651910f5242c28d84e4a5c2435ca5c3f797ed7e2e/anyio-4.15.1-py3-none-any.whl", hash = "sha256:6152fdbbf9a77fdec97731721bebf7c4c44f7c29b424b0065826173efc7ed101", size = 132079, upload-time = "2026-09-05T10:42:37.923Z" }, +] + +[[package]] +name = "babel" +version = "2.17.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/7d/6b/d52e42361e1aa00709585ecc30b3f9684b3ab62530771402248b1b1d6240/babel-2.17.0.tar.gz", hash = "sha256:0c54cffb19f690cdcc52a3b50bcbf71e07a808d1c80d549f2459b9d2cf0afb9d", size = 9951852, upload-time = "2025-02-01T15:17:41.026Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/b7/b8/3fe70c75fe32afc4bb507f75563d39bc5642255d1d94f1f23604725780bf/babel-2.17.0-py3-none-any.whl", hash = "sha256:4d0b53093fdfb4b21c92b5213dba5a1b23885afa8383709427046b21c366e5f2", size = 10182537, upload-time = "2025-02-01T15:17:37.39Z" }, +] + +[[package]] +name = "certifi" +version = "2026.7.22" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/a3/c2/24167ea9858356b47a87a50d39908bfdb72ceeefe0041586e704e5376b3a/certifi-2026.7.22.tar.gz", hash = "sha256:741e2c3b351ddf169a738da9f2c048608ff7f2c5cc02f1ebc6b118bb090d5d55", size = 138112, upload-time = "2026-07-22T03:35:12.644Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/0b/a7/71ac2cff56fec219ed242bb11b8efb69fcc4bec75db06fb7bfe35de520e6/certifi-2026.7.22-py3-none-any.whl", hash = "sha256:62f22742b58a1a33014a2b6b706588a8d7e2a88ae7bd1a6ebe8c992928483775", size = 136983, upload-time = "2026-07-22T03:35:11.276Z" }, +] + +[[package]] +name = "cffi" +version = "2.1.1" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "pycparser", marker = "implementation_name != 'PyPy'" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/9e/ef/008a1939e372c06329a3fce4279c02f328488f3526744906eeec3da7ad5f/cffi-2.1.1.tar.gz", hash = "sha256:dd31f52ea1086513bb9df30f8fcee9b8918323ae067a3d5b78bc826a000712be", size = 530807, upload-time = "2026-08-03T21:21:18.939Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/10/69/43965eccfdead3b9220015fd1320e117be8c6ed01a62ffab76eeb752f5d5/cffi-2.1.1-cp312-cp312-macosx_10_15_x86_64.whl", hash = "sha256:c8c69575568085ba0b1b10c0249d779a214aea6f6522e949a0fc9fb0fcb449d0", size = 184821, upload-time = "2026-08-03T21:19:44.887Z" }, + { url = "https://files.pythonhosted.org/packages/54/7d/16e5a096677b5e313ca80cd5e5170efa3ea44624a82bb111925522da64b1/cffi-2.1.1-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:f81b3b8f3d4e343550fa4baa0e479bba9f2d29ce9c2e9b51d1ce1718d7442fcf", size = 184719, upload-time = "2026-08-03T21:19:46.129Z" }, + { url = "https://files.pythonhosted.org/packages/56/e6/8941622732edec876dd17d0453dce07317ae96db34f2ec1436c9d3785986/cffi-2.1.1-cp312-cp312-manylinux1_i686.manylinux2014_i686.manylinux_2_17_i686.manylinux_2_5_i686.whl", hash = "sha256:811bd1e21d32de12efca32393a0ab3f5133b54fce9bd44b8bd77ab07da14bf6a", size = 214799, upload-time = "2026-08-03T21:19:47.218Z" }, + { url = "https://files.pythonhosted.org/packages/44/de/f98430906df1545ffde0d543dd124a7a439bc2cd32b36b9c53f805df7333/cffi-2.1.1-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:68e62fe11f30d5ca8289242866f0a5291402d8529ca2178ab8afc5c9694ae890", size = 222389, upload-time = "2026-08-03T21:19:48.331Z" }, + { url = "https://files.pythonhosted.org/packages/6a/5b/717f1526b9957b34456313c31645c5b82b8fb5c3fe9e4752999be7128bfc/cffi-2.1.1-cp312-cp312-manylinux2014_ppc64le.manylinux_2_17_ppc64le.whl", hash = "sha256:4a7c934f7360e8cd64fe9efadcbd10c7c6364f531e432b9a4bf5ccbc9e0e8b50", size = 210249, upload-time = "2026-08-03T21:19:49.543Z" }, + { url = "https://files.pythonhosted.org/packages/64/b3/f8aa4f3e34986c7e4ec45072d1b1b9dd295b6b18007b45518d79726dd725/cffi-2.1.1-cp312-cp312-manylinux2014_s390x.manylinux_2_17_s390x.whl", hash = "sha256:3143d81e29e1e20a9ce10901ec369012947876596f75a222235965f2b7ae832e", size = 208775, upload-time = "2026-08-03T21:19:50.918Z" }, + { url = "https://files.pythonhosted.org/packages/b1/db/dceb9dd5b231e1da801793f8acc9f3c52a7e1afe40bb1aae37e02b0faad5/cffi-2.1.1-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:c1453022f490d2459a11819d83ad1d586e9ff65a12ac3e705ffebd46d3685dcf", size = 221822, upload-time = "2026-08-03T21:19:52.054Z" }, + { url = "https://files.pythonhosted.org/packages/a0/d2/6cd24ae3be000a634109c247d1475d62e5616d0dc78c82770942ec384248/cffi-2.1.1-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:208f941bb9d18e768138677f0a6d2ce01f590df56043dda1df1535ac57c88517", size = 225232, upload-time = "2026-08-03T21:19:53.109Z" }, + { url = "https://files.pythonhosted.org/packages/cb/52/3fa190537004dd7f0ab860a6dc7c0175b8667f68d1e618a46f5498d30250/cffi-2.1.1-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:210019b6c7cf07f081b4c54635c8cf744377001350e29cc0f81c4377b4797735", size = 223597, upload-time = "2026-08-03T21:19:54.515Z" }, + { url = "https://files.pythonhosted.org/packages/80/fb/0bb75b7039588c074b37ae99f40d9bfddf990ecb2fbc346ebccd2e56b9be/cffi-2.1.1-cp312-cp312-win32.whl", hash = "sha256:046bfc24911b37851ee1b51aab8bffe713d89c68c6a057b09484ce9fd5f69b4e", size = 175292, upload-time = "2026-08-03T21:19:55.566Z" }, + { url = "https://files.pythonhosted.org/packages/d9/79/615cc094e2fb508cade7de88d3b4f6c4ec2bab695c97bce9153dc65aadf5/cffi-2.1.1-cp312-cp312-win_amd64.whl", hash = "sha256:f53e442b08449d42821fa4a4fba000095af9f62742a500f978a9f557ec44339a", size = 185919, upload-time = "2026-08-03T21:19:56.89Z" }, + { url = "https://files.pythonhosted.org/packages/70/c6/d0ea84713fe46b243a436a18fcd47d639732747e21635c8a27191b06dc30/cffi-2.1.1-cp312-cp312-win_arm64.whl", hash = "sha256:7bde5e4cc5c10140859842b9d383af292b22639a4dffb725314baf45968cef80", size = 180093, upload-time = "2026-08-03T21:19:58.155Z" }, + { url = "https://files.pythonhosted.org/packages/9d/f4/035513d4117049066b4779dc3b7c0c0fdad175fa13731c9f4003f1cd1478/cffi-2.1.1-cp313-cp313-ios_13_0_arm64_iphoneos.whl", hash = "sha256:b5bdfd1c873d4e093aabc0ca84c4ca6dbc4f752afb5c86f146d9742580c9da2e", size = 194248, upload-time = "2026-08-03T21:19:59.399Z" }, + { url = "https://files.pythonhosted.org/packages/76/af/2aeb4dbb5fc41a04161ae9ff1518de7cec08e164f44a8ce6a4cf7fd2cd1d/cffi-2.1.1-cp313-cp313-ios_13_0_arm64_iphonesimulator.whl", hash = "sha256:31348097ff5bbe827ccc41795d4dd099d9f0625e7def00ee653c137a490c2a6c", size = 196908, upload-time = "2026-08-03T21:20:00.746Z" }, + { url = "https://files.pythonhosted.org/packages/a7/46/2e5fdde8555706dd98139a910ca11be02809f3f605ce956f655d0214e100/cffi-2.1.1-cp313-cp313-macosx_10_15_x86_64.whl", hash = "sha256:9d2055050ea716bd38b7f7f1579c275386646b4894c155a3e2f3cd62ed41b7c6", size = 184805, upload-time = "2026-08-03T21:20:02.02Z" }, + { url = "https://files.pythonhosted.org/packages/55/41/4c7042f317b9217502988f0873af87e16ad606dc20f84e546e3e6ce9764c/cffi-2.1.1-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:19ee6127ee34de7d83ce3d371ebc5ed91addbdcc39f9ab15ce4eb35a4e534971", size = 184764, upload-time = "2026-08-03T21:20:03.141Z" }, + { url = "https://files.pythonhosted.org/packages/43/1f/1c3d90d91811c8f86ced9ed637956c54bfe5b79ca98fe976d7f8c8979f6b/cffi-2.1.1-cp313-cp313-manylinux1_i686.manylinux2014_i686.manylinux_2_17_i686.manylinux_2_5_i686.whl", hash = "sha256:6a8dddef476fab96d066d578fc88526767b836ab5ab21754e1d5bf3879c31c7c", size = 214722, upload-time = "2026-08-03T21:20:04.377Z" }, + { url = "https://files.pythonhosted.org/packages/37/6f/3b5ce4c3b2192d250f04908f2bfd91ef34552ec8f7716a5d4abdb8d67bb2/cffi-2.1.1-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:f16c709686a78c727bbbf059f92b0bf41c6fc60deec706d2dc19f529175a6125", size = 222369, upload-time = "2026-08-03T21:20:05.544Z" }, + { url = "https://files.pythonhosted.org/packages/02/10/4b3c75dde3d9663c9e02ba05c2668b954f671d4bbe346413ca8c696b295a/cffi-2.1.1-cp313-cp313-manylinux2014_ppc64le.manylinux_2_17_ppc64le.whl", hash = "sha256:fcd22650c908d7b7da162bbfaab594a1227a15d1643a98c68b122ac642fa2264", size = 210175, upload-time = "2026-08-03T21:20:06.75Z" }, + { url = "https://files.pythonhosted.org/packages/df/62/14f74b9543e605d17701dc797b815958b8bb70b7624ce1b832ddad48ed6c/cffi-2.1.1-cp313-cp313-manylinux2014_s390x.manylinux_2_17_s390x.whl", hash = "sha256:aa9511c62d14da7aacc9b4bf51f3f697a621e83b2d6919008243c3aad168eea3", size = 208670, upload-time = "2026-08-03T21:20:08.04Z" }, + { url = "https://files.pythonhosted.org/packages/95/95/86342356ff5953b3fb06f7ef7c5bee212d45e770abc7218d451b9148313c/cffi-2.1.1-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:a931079504ecc49efed7744c476a5c343a92fabf66dec2db95edb1b2fdc770e2", size = 221824, upload-time = "2026-08-03T21:20:09.274Z" }, + { url = "https://files.pythonhosted.org/packages/eb/ff/7b3429ff53aafe931ed8a5fc69f481bbef7ba6de87ddcbb63d08f483f613/cffi-2.1.1-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:a2d7755bef5a12ed488f4ef1f1b69ee9191d7396083b755a5d2295f6edb4768b", size = 225148, upload-time = "2026-08-03T21:20:10.7Z" }, + { url = "https://files.pythonhosted.org/packages/34/34/a95870b9221e09cf4f2ce3178b1a210abdfe63a1bd357da940418d7b8d15/cffi-2.1.1-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:e0bcb7e0f677f543555d2adff3bf19c05f66cdb4796e5ff602442ab2fe3c4ef7", size = 223564, upload-time = "2026-08-03T21:20:12.165Z" }, + { url = "https://files.pythonhosted.org/packages/70/ea/839b50531021a647fb5e929f72cf97bc1ff702b5472166164b5b6e76b851/cffi-2.1.1-cp313-cp313-win32.whl", hash = "sha256:334644fbac4eff73d985a17a91226df55d0f394160c4cfb880e084c8f7161cac", size = 175263, upload-time = "2026-08-03T21:20:13.559Z" }, + { url = "https://files.pythonhosted.org/packages/60/a6/8b149b2c3f2e11aaa1618ef64500b45f50f22c57a977a4dff1aff1f91042/cffi-2.1.1-cp313-cp313-win_amd64.whl", hash = "sha256:1aa5645c30469b09530c4ebca77ebf8f17618293c58f8549cb1a543a50236e7d", size = 185688, upload-time = "2026-08-03T21:20:14.69Z" }, + { url = "https://files.pythonhosted.org/packages/01/9a/11f687cb39d6a3504060d5242f04f48c735afb4d3d533958a20594890cb2/cffi-2.1.1-cp313-cp313-win_arm64.whl", hash = "sha256:63bbfd5ded17c4840ac07cd8f1c21ba9d9708141f840b324f422f41b207e3973", size = 180078, upload-time = "2026-08-03T21:20:15.917Z" }, + { url = "https://files.pythonhosted.org/packages/d3/7b/d6bbf82b8b96e7391438898c42f5bd96dd02030fd5b64937d248220003e2/cffi-2.1.1-cp314-cp314-ios_13_0_arm64_iphoneos.whl", hash = "sha256:7dbb61fe3a7699468030f71bbe5f8a0e326a151daa91beb11a6fc1f980c55e1c", size = 194064, upload-time = "2026-08-03T21:20:17.148Z" }, + { url = "https://files.pythonhosted.org/packages/94/e6/bcc91b283be94735e268487a054004f0aa19947b6348fa367db53230abc8/cffi-2.1.1-cp314-cp314-ios_13_0_arm64_iphonesimulator.whl", hash = "sha256:f24fb43132a4c6b4cb4eb029492919b2db645be6808d738f244fd146c03c32cb", size = 196720, upload-time = "2026-08-03T21:20:18.268Z" }, + { url = "https://files.pythonhosted.org/packages/d9/99/c4b0c17cacdc9c3b8f280026286a9826d6a208c0f047591a3c3ce99b91fd/cffi-2.1.1-cp314-cp314-macosx_10_15_x86_64.whl", hash = "sha256:d28630f5854ab07ab1fd4aba756de52326c82e6be15d414b12793f1975048b54", size = 184964, upload-time = "2026-08-03T21:20:19.708Z" }, + { url = "https://files.pythonhosted.org/packages/b3/a9/9db617d05d7367c1ad0ab00b3aa6e6f9281edd689b4ee9ea0e5a84e89c97/cffi-2.1.1-cp314-cp314-macosx_11_0_arm64.whl", hash = "sha256:661c298b4821edebead0c91edd2b00374d67ad7c5a1f7a91d4442633b79d6a72", size = 184962, upload-time = "2026-08-03T21:20:20.833Z" }, + { url = "https://files.pythonhosted.org/packages/67/b8/b42132ca113dc567d37684437b46ca1dafc885902b02a110a02d5b511857/cffi-2.1.1-cp314-cp314-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:58acb8ab8e295e6c5ea12f888cbb13cf21511ef2a3303a23f4325c29d17fe5c1", size = 222328, upload-time = "2026-08-03T21:20:22.118Z" }, + { url = "https://files.pythonhosted.org/packages/80/10/c5c0cbf0a657aecf59ef511409734230bf556f05a0d6c9eed7aa5c0a0166/cffi-2.1.1-cp314-cp314-manylinux2014_ppc64le.manylinux_2_17_ppc64le.whl", hash = "sha256:456a61fa52d579ebf9df2e9552ead5129855dbaff6c1e5a9b1bc408809bdc062", size = 209985, upload-time = "2026-08-03T21:20:23.401Z" }, + { url = "https://files.pythonhosted.org/packages/d5/6c/bfa0b87b03b9238148beca990292843c9396ba069b54496596594173de7b/cffi-2.1.1-cp314-cp314-manylinux2014_s390x.manylinux_2_17_s390x.whl", hash = "sha256:a4f00aa42f75d6e4595e8866e748cc1705adc0cddfeb2ca86d0d03993d63ba03", size = 208530, upload-time = "2026-08-03T21:20:24.628Z" }, + { url = "https://files.pythonhosted.org/packages/e9/02/4e7d553a7ac4b4238b38b3c1b80d486e9d4436f8d2acbf87a0997fe3f402/cffi-2.1.1-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:b0431303acaea1089ad4b3e9ce4e6518193def1118d4073ca848635ee4ea2e96", size = 221525, upload-time = "2026-08-03T21:20:25.758Z" }, + { url = "https://files.pythonhosted.org/packages/82/1d/a4aaf9babd75acb4d5f223bff71533bee748dd770a382619a798960ee9ba/cffi-2.1.1-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:64faea20f4e2613363a1a9b9c7dd73058f3ecd00133a511e72ad7c511658f527", size = 225053, upload-time = "2026-08-03T21:20:26.985Z" }, + { url = "https://files.pythonhosted.org/packages/81/10/5dc0e7bdd18e22107054288283380fc97a06ae3f1656a106908d666a3c88/cffi-2.1.1-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:5c58fe613dc5e5336357eff555824a314d8e43282600435c8d1cb6a7a2fedd13", size = 223213, upload-time = "2026-08-03T21:20:28.277Z" }, + { url = "https://files.pythonhosted.org/packages/0b/e9/d0061c364cde06ee43168a0d076ac1da512cbc380d44767b844ba34fe2b6/cffi-2.1.1-cp314-cp314-win32.whl", hash = "sha256:1a18a57b58cfb21fc28d72e876acf10eaed67a1ed96226f92af4df681d571c4c", size = 177682, upload-time = "2026-08-03T21:20:44.288Z" }, + { url = "https://files.pythonhosted.org/packages/a7/06/1c3e01e3ba14c39f6d10bfbac52753b7e22259e38088e5cfe1d704918690/cffi-2.1.1-cp314-cp314-win_amd64.whl", hash = "sha256:3222ba5d678f80a030e6afbcc33dc1ae5cb45facabb61cee2c7016b8432fde48", size = 187949, upload-time = "2026-08-03T21:20:45.623Z" }, + { url = "https://files.pythonhosted.org/packages/87/5b/da4e39efe18eeb89cf580ea9cfc66b6a7c3eadb808fc0cc1d3a295cb5a5d/cffi-2.1.1-cp314-cp314-win_arm64.whl", hash = "sha256:ab36d55f9ed2d067327667c2fea18dda018eb628dd6347aa01dda6cf1f5d3836", size = 182947, upload-time = "2026-08-03T21:20:46.955Z" }, + { url = "https://files.pythonhosted.org/packages/23/59/40338bf421c5accea1d45158170c87006ef1cd371b05c077e76476949728/cffi-2.1.1-cp314-cp314t-macosx_10_15_x86_64.whl", hash = "sha256:7750c6449dff7864bb9bb27ddfb0267756189201a3afc911d82b3caacd70dfc3", size = 188504, upload-time = "2026-08-03T21:20:29.495Z" }, + { url = "https://files.pythonhosted.org/packages/7d/47/5ecf1023850036e674c77ec4de86182d309ae344e39e7cba984b7df5d647/cffi-2.1.1-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:0beceaabe56af686895136a2de78db54ecd8e4046b236b8fd6d6cb61389e9bf2", size = 188259, upload-time = "2026-08-03T21:20:31.291Z" }, + { url = "https://files.pythonhosted.org/packages/2a/9c/92934c3bea9f785b23eba304538c0b4d37a2a96d2431eb3a1bc87a11aa19/cffi-2.1.1-cp314-cp314t-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:49cbc70e6542d4ccccb936558d1064a8012541e78f821f955cff24e357776c94", size = 223864, upload-time = "2026-08-03T21:20:32.571Z" }, + { url = "https://files.pythonhosted.org/packages/4d/45/ba4c93527bc38616a8bd36488acb69a2212d60486794f0c1f318949bbb76/cffi-2.1.1-cp314-cp314t-manylinux2014_ppc64le.manylinux_2_17_ppc64le.whl", hash = "sha256:e2d65b31f36619cda3999b78b2aa9632e76b78448e7a56fc4240824200e7c4fc", size = 211538, upload-time = "2026-08-03T21:20:33.808Z" }, + { url = "https://files.pythonhosted.org/packages/80/e9/b6ef565e452acb932fb0cb5443f44a78efbd1233e566f02b5a83855e9115/cffi-2.1.1-cp314-cp314t-manylinux2014_s390x.manylinux_2_17_s390x.whl", hash = "sha256:28907ab9bfb6aa13184cfc17c6b8e1023c5ab6fd7076d8c20a35e59fe04f8f29", size = 210688, upload-time = "2026-08-03T21:20:34.974Z" }, + { url = "https://files.pythonhosted.org/packages/9a/95/eff5f0cee78d2eabc7eebffec40d3fc1876b5f3c95582e018bb4b99601f2/cffi-2.1.1-cp314-cp314t-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:51b31d1c98274844cfd7838ce00bfc27c7423a4dc00fc0772fc3331c2cc90676", size = 223803, upload-time = "2026-08-03T21:20:36.564Z" }, + { url = "https://files.pythonhosted.org/packages/fa/01/579d39fb8bef00a335a23d83757b44feb24cd6345a2c451b64cb67b9c362/cffi-2.1.1-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:5e7cecbaadb83884793e05828cee59b210b24583b9c7425d0ba6a754fe22eb4e", size = 226763, upload-time = "2026-08-03T21:20:37.816Z" }, + { url = "https://files.pythonhosted.org/packages/8d/b0/0b44f47c60b01b57b6e2bbd92343f13a85a1d93bc46ccf6e47e244acd99c/cffi-2.1.1-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:25792eac27877609e7bb06d42ff88278a6624fff2ba9bbb523c09616b117e80f", size = 225688, upload-time = "2026-08-03T21:20:38.959Z" }, + { url = "https://files.pythonhosted.org/packages/eb/d2/3b7176cb570a1d3e27faf67b72f591af508036e0d8b2be2ef9af9e8c84bb/cffi-2.1.1-cp314-cp314t-win32.whl", hash = "sha256:8ef53b2de9bcb9197d31854256575d59dbac0cba72ac627bb291ef5eceb74be4", size = 182868, upload-time = "2026-08-03T21:20:40.388Z" }, + { url = "https://files.pythonhosted.org/packages/56/78/31f00c1bcd97c9bbf55f1bfdf5bc809a5de8887473e90bb9960dca825e80/cffi-2.1.1-cp314-cp314t-win_amd64.whl", hash = "sha256:616f097f2fe415bc92a247f02e11f634e1f9e9a83d327e3c915c15089c87869e", size = 194104, upload-time = "2026-08-03T21:20:41.725Z" }, + { url = "https://files.pythonhosted.org/packages/7b/1b/58496f2ed0a35de575250c02a43ab3cc2c04d494a88fed31c1cabc0fd176/cffi-2.1.1-cp314-cp314t-win_arm64.whl", hash = "sha256:ad2c86c495b899d862ea0f4b42891b8713a3bd45dd4105c7fd51c2a72f39f3a5", size = 186402, upload-time = "2026-08-03T21:20:43.042Z" }, + { url = "https://files.pythonhosted.org/packages/c1/8f/9ebe220eab48a093d1a5a5e339ab0dc7316eef3bb04d63c42f0251b61f50/cffi-2.1.1-cp315-cp315-ios_13_0_arm64_iphoneos.whl", hash = "sha256:dddad92b554513a31f272570678ba307fb9f618f05e3d4a5eacafff9eae03e1d", size = 194043, upload-time = "2026-08-03T21:20:48.179Z" }, + { url = "https://files.pythonhosted.org/packages/ff/69/844bad3ece306c4782c2ecb93597035b6690d48704b803914c199da1e8b3/cffi-2.1.1-cp315-cp315-ios_13_0_arm64_iphonesimulator.whl", hash = "sha256:da0e573f9f97159390c89d9f1a9e41908b66d408cc5b58d08cf3847d844c531b", size = 196737, upload-time = "2026-08-03T21:20:49.457Z" }, + { url = "https://files.pythonhosted.org/packages/1b/8a/af668013284634733f02d683458a0728739c7d6ddb5e14cb0c20832266fe/cffi-2.1.1-cp315-cp315-macosx_10_15_x86_64.whl", hash = "sha256:fb92203a88b3d3053034db775110081c49d28be6551923805e039924093761e4", size = 184933, upload-time = "2026-08-03T21:20:50.639Z" }, + { url = "https://files.pythonhosted.org/packages/0c/75/2f5207ff6d1a613133b23a5203cc0c2a628313b5eb3974d7956ae3c57950/cffi-2.1.1-cp315-cp315-macosx_11_0_arm64.whl", hash = "sha256:2ae64be792b8966f2c69538199728b290e34726562896df1e5dc8ffd8d8188e8", size = 185002, upload-time = "2026-08-03T21:20:52.173Z" }, + { url = "https://files.pythonhosted.org/packages/e2/31/9e1313b0a6e30e91b3b3d3fff51ae99c857c07738e3afcce1f7334e1b7ab/cffi-2.1.1-cp315-cp315-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:507a24c282e0f42f8ed737cf048572cbf580468da5555764a8331735e9c736b6", size = 222271, upload-time = "2026-08-03T21:20:53.462Z" }, + { url = "https://files.pythonhosted.org/packages/50/e3/f6234a833e6e08c7007003074723c406559eecf9b48dfc97471e5a8eb7a0/cffi-2.1.1-cp315-cp315-manylinux2014_ppc64le.manylinux_2_17_ppc64le.whl", hash = "sha256:246fa40ce8645a614ff682e0b70f37134e460eaf93a775e0cbe3cca585a67a80", size = 209919, upload-time = "2026-08-03T21:20:54.783Z" }, + { url = "https://files.pythonhosted.org/packages/0d/fc/5f74e293fced6edb51af3a46c4ccf6c23c9943774ecb375ddbd522c76add/cffi-2.1.1-cp315-cp315-manylinux2014_s390x.manylinux_2_17_s390x.whl", hash = "sha256:471cee653ae88de62096552e6d24ccb4a5adb8c8c9f10b5054d0122c15bf2779", size = 208529, upload-time = "2026-08-03T21:20:56.066Z" }, + { url = "https://files.pythonhosted.org/packages/44/16/29e6d01b388bef055ecd6ca8244b3f4d336bd09e92d5d892187b9601084e/cffi-2.1.1-cp315-cp315-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:aeae0e330c9f6acd681f647d46cefd30c29f93e3392882e792e82080c9691399", size = 221630, upload-time = "2026-08-03T21:20:57.336Z" }, + { url = "https://files.pythonhosted.org/packages/a4/18/fa7f1f6857d5eb88a4ca99ffcbfb7c387a287ccc154c64a73e86314745d7/cffi-2.1.1-cp315-cp315-musllinux_1_2_aarch64.whl", hash = "sha256:42a494cee34437f05546455144f2b5d9ac09b1face62bcfce597d2e521066688", size = 225134, upload-time = "2026-08-03T21:20:58.675Z" }, + { url = "https://files.pythonhosted.org/packages/e0/9f/e8e3dfa04a1b4c241f8c91faacad872b4d4efd051d49764ad4e2fd4b9fea/cffi-2.1.1-cp315-cp315-musllinux_1_2_x86_64.whl", hash = "sha256:cc572dace3f60ef98d7b12ff411d20f5362feb31a0439eab0085bbfd349982d7", size = 223197, upload-time = "2026-08-03T21:20:59.968Z" }, + { url = "https://files.pythonhosted.org/packages/f8/7e/8debeb04f1ab9fe2a6963964cd6f1aaf7192627b83926586a6a4e089c9fa/cffi-2.1.1-cp315-cp315-win32.whl", hash = "sha256:4f42141fc14250de6dde5ee7ea4432be017252d91f19c5ad043c084cea629cac", size = 177683, upload-time = "2026-08-03T21:21:14.901Z" }, + { url = "https://files.pythonhosted.org/packages/e0/31/5158704cc474ab65c1647932e88be78dc0873f47130e253be38bcaf13d01/cffi-2.1.1-cp315-cp315-win_amd64.whl", hash = "sha256:e6e8cff14d6fb0be70a09c0bdc58096f501952d04624ebf867e0e56da2df8960", size = 187897, upload-time = "2026-08-03T21:21:16.108Z" }, + { url = "https://files.pythonhosted.org/packages/cc/4b/b3a2da8570c704ffc0f9762cdc3ec0f02c8573798e0b5cf7f11c82bbb70f/cffi-2.1.1-cp315-cp315-win_arm64.whl", hash = "sha256:27350daa11d4f10c540e6e89dada4c54feb7256ad03e9a4dc075ebad7ba360d1", size = 182935, upload-time = "2026-08-03T21:21:17.271Z" }, + { url = "https://files.pythonhosted.org/packages/d0/ef/5443574510a1207e6f6bc38ba6e1f1de36cb48fef07b2728bb896a21f430/cffi-2.1.1-cp315-cp315t-macosx_10_15_x86_64.whl", hash = "sha256:c26608d2222fb1e94487e4a387d85f13eb55d5ed725cb25a0c589ac4ee60e7bc", size = 188464, upload-time = "2026-08-03T21:21:01.163Z" }, + { url = "https://files.pythonhosted.org/packages/7e/ae/a56fa8c4686ad50e148fcbc8d3ae0d03915ff5c30d795058988c24118cef/cffi-2.1.1-cp315-cp315t-macosx_11_0_arm64.whl", hash = "sha256:4be96343e422f2dfcd12ab5c9f5aebe03f82f737c6bffeca6830b3875cb44aab", size = 188262, upload-time = "2026-08-03T21:21:02.382Z" }, + { url = "https://files.pythonhosted.org/packages/53/b2/6187f46f2912276a3ae284076109cc5c8680482f11f766ccf26db4a86427/cffi-2.1.1-cp315-cp315t-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:937c0052c05a31ca1daf18de3158eed4dbfcb9cc107adbea227728d647be701e", size = 223779, upload-time = "2026-08-03T21:21:03.553Z" }, + { url = "https://files.pythonhosted.org/packages/8a/f6/c3ad28bd19f77047a03084424fbd4cbe997303267c14423737324be0385d/cffi-2.1.1-cp315-cp315t-manylinux2014_ppc64le.manylinux_2_17_ppc64le.whl", hash = "sha256:df423d40ee8654634421812bc3b196da3f9bd7d32929da813f8394c4348a5358", size = 211520, upload-time = "2026-08-03T21:21:04.863Z" }, + { url = "https://files.pythonhosted.org/packages/a0/cd/ccac9013a5bd9fd764de118674ab9c805b5ca10c19270d90ee273f8b2240/cffi-2.1.1-cp315-cp315t-manylinux2014_s390x.manylinux_2_17_s390x.whl", hash = "sha256:a730a083190634c65cca36ba5f489531576ebd79bcd5c8e172130f6453127231", size = 210673, upload-time = "2026-08-03T21:21:06.223Z" }, + { url = "https://files.pythonhosted.org/packages/52/86/2976131c639aead931c5bee5aba67e4b09fbeb8018b6f282f70803f923a7/cffi-2.1.1-cp315-cp315t-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:363e05fa78e15116c3c32c210ee36884fd6b9afa6d440e47112c3bd511d64cb6", size = 223835, upload-time = "2026-08-03T21:21:07.539Z" }, + { url = "https://files.pythonhosted.org/packages/ac/0c/33a7aeab2f9c76918c52e084beb39c570db3588133412929e8ec06fab90b/cffi-2.1.1-cp315-cp315t-musllinux_1_2_aarch64.whl", hash = "sha256:770de9db11e84213beec501cfcaa013b019820ca881e03344dea5844f7876d94", size = 226705, upload-time = "2026-08-03T21:21:08.774Z" }, + { url = "https://files.pythonhosted.org/packages/e3/26/2cde30fdde421130bfc18f70395731a6e6b2053c6a1978a5258ff04e72fa/cffi-2.1.1-cp315-cp315t-musllinux_1_2_x86_64.whl", hash = "sha256:7da0c5eff80f0197f3b3d1232ec5a682a9325f4ae9016a78f5f5ca35f9ced1f5", size = 225539, upload-time = "2026-08-03T21:21:09.911Z" }, + { url = "https://files.pythonhosted.org/packages/6d/cd/a361394c94b2129d604bb846f624a8e88255a3ee33129c434a00d715e64f/cffi-2.1.1-cp315-cp315t-win32.whl", hash = "sha256:06c72bb76605a4b0cd0aad6930b69d4baf7dd5d806cfc409b824191099700e66", size = 182707, upload-time = "2026-08-03T21:21:11.226Z" }, + { url = "https://files.pythonhosted.org/packages/9b/b5/ba2b299993c26577d529b6ae29841f9e15b9fcf004d65f423f4fcf94ade9/cffi-2.1.1-cp315-cp315t-win_amd64.whl", hash = "sha256:d9c275eaacd24aa73f94ffd6de08fc3f932424d8b6c376f4bed7cde376fe7bc3", size = 193772, upload-time = "2026-08-03T21:21:12.39Z" }, + { url = "https://files.pythonhosted.org/packages/aa/29/35e016098c814cd93de9cd320c66b5bfba14dc6ecedd3cb518fa7c408c69/cffi-2.1.1-cp315-cp315t-win_arm64.whl", hash = "sha256:d18e5ac0f2f03f4f518d3e23db0f0cad7faa1da8620e9c09461d443bbf6e6692", size = 186360, upload-time = "2026-08-03T21:21:13.636Z" }, +] + +[[package]] +name = "charset-normalizer" +version = "3.5.1" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/e5/3f/143b048436775b0f76ac3eec145c019e8173ccc2885c8f20319b996d5e83/charset_normalizer-3.5.1.tar.gz", hash = "sha256:6117b84ea48435e5356dc737f5121485c30920ba43375fa7b434fd753df0eac3", size = 171764, upload-time = "2026-08-15T08:20:44.807Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/30/27/78873dc8b6a56357517b74b6bb9568b80450e7bb4f6ef7e3fa9d22aa0bd7/charset_normalizer-3.5.1-cp312-cp312-macosx_10_13_universal2.whl", hash = "sha256:5b6d1386bf0096d26d3a863dc0a487a5b4eb9aa93cf5ba69683d29dde6b9d60f", size = 344456, upload-time = "2026-08-15T08:17:10.072Z" }, + { url = "https://files.pythonhosted.org/packages/9a/4c/be49ada26b1f0232d57aa89bbebf997a5cc2332a5616b6eca26ff680044d/charset_normalizer-3.5.1-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:4582c27e8c889d64811987b5967fbd3ae0c823fe1fd933b543d55ac20bb475fa", size = 238530, upload-time = "2026-08-15T08:17:11.563Z" }, + { url = "https://files.pythonhosted.org/packages/76/84/6f1290fa07ae6978d3960caa3eb1b8019bf9284ab7c2297b00c099ef4250/charset_normalizer-3.5.1-cp312-cp312-manylinux2014_armv7l.manylinux_2_17_armv7l.manylinux_2_31_armv7l.whl", hash = "sha256:1d1c7a53a6c2103925cdd6d7229f8c567379f211c869793df679f2e9f738c369", size = 230200, upload-time = "2026-08-15T08:17:12.919Z" }, + { url = "https://files.pythonhosted.org/packages/e7/a0/47b18adeed31c8f16ba9700f32c1b18594cfa09f47eb672a488c273c22bf/charset_normalizer-3.5.1-cp312-cp312-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:e6621fb2a4988d6e53eedc455e5903e2679f3967b8acb3d639f1b63c14a2e893", size = 262222, upload-time = "2026-08-15T08:17:14.571Z" }, + { url = "https://files.pythonhosted.org/packages/38/fe/341861ac118dae06f3ec0eb487488af52128f2ef2faf0b11003944d22259/charset_normalizer-3.5.1-cp312-cp312-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:7c0c10730342b0c9b35dd1d619beb8214e520bd96a1f870f452680b238aab3e0", size = 258951, upload-time = "2026-08-15T08:17:16.158Z" }, + { url = "https://files.pythonhosted.org/packages/6f/89/bb5108dc6c3651dca963f2b0a3ba19bbcb370c94e1b6d3e0e844a58e6dca/charset_normalizer-3.5.1-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:b9af956078716df40d985fb0dfeb2c2120c5ca92ba4ff4b388acfd01cdc14d08", size = 248801, upload-time = "2026-08-15T08:17:17.683Z" }, + { url = "https://files.pythonhosted.org/packages/b1/ba/ef83ae3aca816393decfa3530976f38a79812d707b80b580ac33b83f9877/charset_normalizer-3.5.1-cp312-cp312-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:f9f8405c2c758532c74fed975dbee57be1f31a6e865c031870c79a6ed3212ada", size = 244070, upload-time = "2026-08-15T08:17:19.191Z" }, + { url = "https://files.pythonhosted.org/packages/f6/0b/c5292a2462d69b7378ea89793bbb5b2b6fcf6f7dd6d1667f9619094ad553/charset_normalizer-3.5.1-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:96fef3e886d6a9874b14f27fc193fbdc69d5d8035783d86aa4e1cea594e695f9", size = 240110, upload-time = "2026-08-15T08:17:20.547Z" }, + { url = "https://files.pythonhosted.org/packages/46/22/111e5be3b740d5c2a5bfcedb3d237b6591e5c2e82ae9d6ffcb121fe0909c/charset_normalizer-3.5.1-cp312-cp312-musllinux_1_2_armv7l.whl", hash = "sha256:5d8531a6569d025f68e2321e7638fb7978f23db58e5f69f56913837aae03816e", size = 232836, upload-time = "2026-08-15T08:17:21.895Z" }, + { url = "https://files.pythonhosted.org/packages/f9/d2/d2aad6fe0dbb44b194bf3becb60f5a0ac48446ade999a47fe7bb41eb09a7/charset_normalizer-3.5.1-cp312-cp312-musllinux_1_2_ppc64le.whl", hash = "sha256:aae2ee51122d3ae968a3837d97dc24a0aeebb0dea23694422cd172bd30017cd6", size = 262712, upload-time = "2026-08-15T08:17:23.727Z" }, + { url = "https://files.pythonhosted.org/packages/35/5a/337e4663a5eae6de99db940ee8066d4145caafb61327db62deda15313cce/charset_normalizer-3.5.1-cp312-cp312-musllinux_1_2_riscv64.whl", hash = "sha256:7235dc28fc6dd9d832ac7c7bce95367dedb85929f17368a0c2bee1e080b9acbf", size = 242977, upload-time = "2026-08-15T08:17:25.157Z" }, + { url = "https://files.pythonhosted.org/packages/ca/85/f82f8a92e31c7519410e2e1afdc630f28ec47490ce2c09a11c1a43cbb459/charset_normalizer-3.5.1-cp312-cp312-musllinux_1_2_s390x.whl", hash = "sha256:4abdc5f9ad448c1ecbfae2974b820535d6bc6e7eef63babbab3d81cf46968c71", size = 260207, upload-time = "2026-08-15T08:17:26.602Z" }, + { url = "https://files.pythonhosted.org/packages/b7/52/643d11ffd60e9ac2fd1fb87e167a19285b9eefeff4a40e63c87cbfbeab36/charset_normalizer-3.5.1-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:ba501e667c17d8411f98e67a022d9604ef179aff0e459b7e292c796837c13573", size = 250562, upload-time = "2026-08-15T08:17:27.971Z" }, + { url = "https://files.pythonhosted.org/packages/62/16/46556278c2168d12df9da7fede5dc6fc70e60301b26a82bbeec238c9cfe3/charset_normalizer-3.5.1-cp312-cp312-win32.whl", hash = "sha256:cfa1c0cc3a8f9f53f1243a5a99ac36fd003880199383b37672e86ddda9cb07e2", size = 178507, upload-time = "2026-08-15T08:17:29.277Z" }, + { url = "https://files.pythonhosted.org/packages/9d/7a/4c6c298171e6b3e745633180ff59350fc0ca0db1ffd28df1e369e0579f71/charset_normalizer-3.5.1-cp312-cp312-win_amd64.whl", hash = "sha256:3617ac3cfd8b9888f145ad89dd6e692285834b0201c6074a5eeaad3fd4d668c2", size = 200551, upload-time = "2026-08-15T08:17:30.668Z" }, + { url = "https://files.pythonhosted.org/packages/cd/d7/eb95a042f0dd22e304b0b6472b154f3546a1a039a9ee89ccb2a7f61591fc/charset_normalizer-3.5.1-cp312-cp312-win_arm64.whl", hash = "sha256:88e85ab89cb822c1e635f51d6d32e488f94e002e70e2f492bdb8b945543f345a", size = 180700, upload-time = "2026-08-15T08:17:32.028Z" }, + { url = "https://files.pythonhosted.org/packages/bc/61/2cb6ad133dbbb449fa2d37ccae973232f4827e799af258d15e589a3d1e9e/charset_normalizer-3.5.1-cp313-cp313-android_24_arm64_v8a.whl", hash = "sha256:4f298bdadb8f0b9e5672877f647d1be9373ef5320c9e2f049795e26cad28b6a9", size = 211584, upload-time = "2026-08-15T08:17:33.597Z" }, + { url = "https://files.pythonhosted.org/packages/18/57/a305c968be1ca13f3dd1b32f445877e97addf55d80b65c7cb35fac82b777/charset_normalizer-3.5.1-cp313-cp313-android_24_x86_64.whl", hash = "sha256:88ca277405c2d3b71c4e1c2ee0e7966e807bcba86a69d11e19ba199d18ae4491", size = 223359, upload-time = "2026-08-15T08:17:35.022Z" }, + { url = "https://files.pythonhosted.org/packages/09/0a/d3646670292ce8d8f8cc11ac067d44885e697a5591f57a9221128da5e7b3/charset_normalizer-3.5.1-cp313-cp313-ios_13_0_arm64_iphoneos.whl", hash = "sha256:9362dd90aa7dab48c0054a21187791ccf05473f7dba5d92b8033ae62164675e7", size = 194464, upload-time = "2026-08-15T08:17:36.452Z" }, + { url = "https://files.pythonhosted.org/packages/de/93/d51ec556e01042fed6f993ea859311bc7917b466684182fbbceb6ca24762/charset_normalizer-3.5.1-cp313-cp313-ios_13_0_arm64_iphonesimulator.whl", hash = "sha256:977cdbd483a9cff38179bea4fd754289a6f2195c7abd414aba85410b3e66cc5e", size = 197676, upload-time = "2026-08-15T08:17:37.819Z" }, + { url = "https://files.pythonhosted.org/packages/a4/a0/562247944386f7d4ef94467e84876600cc1e0f1b93239aaa9213d2bc3cbd/charset_normalizer-3.5.1-cp313-cp313-macosx_10_13_universal2.whl", hash = "sha256:e90251c0c7bdd54a100a0dce3c07b7e637278c93af29dbf78ebb89a58c4bac7d", size = 340473, upload-time = "2026-08-15T08:17:39.303Z" }, + { url = "https://files.pythonhosted.org/packages/31/e7/1d994be1b93d41e9502b8b0460eaa88a1dd8df335df415db87d6c3e91ab2/charset_normalizer-3.5.1-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:94d78ecec2605a8d0398b0f365d5f12a63248438516f5dac536a5eff7337df4a", size = 240156, upload-time = "2026-08-15T08:17:40.66Z" }, + { url = "https://files.pythonhosted.org/packages/09/53/27923ce5cc6cbccb832037b27dca98882d9c53e9b69e866bbbef4aae7fc8/charset_normalizer-3.5.1-cp313-cp313-manylinux2014_armv7l.manylinux_2_17_armv7l.manylinux_2_31_armv7l.whl", hash = "sha256:d59b75732e9b6f27388e10c14b0259cc5f2e48c78627d185e6a177b58ad3cffe", size = 228246, upload-time = "2026-08-15T08:17:42.003Z" }, + { url = "https://files.pythonhosted.org/packages/ce/48/5a97e84d63af1d55c07439cb80e56d99a8efb4295700eb4e18c0d1615d2c/charset_normalizer-3.5.1-cp313-cp313-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:0d929fc574b4d6fd9e7c0f5c2ede8716a41911923aa7fa5fce38e0818aa4a1ac", size = 263660, upload-time = "2026-08-15T08:17:43.627Z" }, + { url = "https://files.pythonhosted.org/packages/7a/c2/071575791dcc88316c0a9a65ce38897a82e4cfe4a325f0f7fe1b1ac47bcf/charset_normalizer-3.5.1-cp313-cp313-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:394fea06235c8543390050ed5f529187074b029fb027213f6c46ac11ab5d950e", size = 260354, upload-time = "2026-08-15T08:17:45.094Z" }, + { url = "https://files.pythonhosted.org/packages/fb/af/63240b0c0248c075c2535a1f1bd992821d8251b9f173abc13329661d09e4/charset_normalizer-3.5.1-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:62b55f6722735a6c472f88361cde6640608773d9443cebdbb51abf436a1fcdd3", size = 250638, upload-time = "2026-08-15T08:17:46.496Z" }, + { url = "https://files.pythonhosted.org/packages/4d/66/70dfad64f15be09c15ccfee81330a7e515895dbe296dd23114e9a231268a/charset_normalizer-3.5.1-cp313-cp313-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:fa48b1b63d639f9483e0633e092f5851e2348c352f1f9bb6c8182f87884ef876", size = 244583, upload-time = "2026-08-15T08:17:47.963Z" }, + { url = "https://files.pythonhosted.org/packages/c0/24/ef36367d38b9ddd4bccbf72888c342e8de1f5ae506fa0b2dcf970e2732a1/charset_normalizer-3.5.1-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:c71fb0d56c920c269cd3e2e3fe7c610e3f1fdb21a6ce60efa6430ff63676cea6", size = 242038, upload-time = "2026-08-15T08:17:49.481Z" }, + { url = "https://files.pythonhosted.org/packages/db/ab/55e683ba0fff2e43adafc10daa3001eac90fdaa419a97227d5a7067eedde/charset_normalizer-3.5.1-cp313-cp313-musllinux_1_2_armv7l.whl", hash = "sha256:485a0d363cafefcd2538a73c7c838daa2035f09b2c9f9b5e3133f80c6aeb84c2", size = 233677, upload-time = "2026-08-15T08:17:50.845Z" }, + { url = "https://files.pythonhosted.org/packages/bd/67/0f40eaf8d1b6e7cf15e82382a2965efaca787fc1c2794b7021d37aaf5036/charset_normalizer-3.5.1-cp313-cp313-musllinux_1_2_ppc64le.whl", hash = "sha256:5c0ea61a470e070686aa30892fed79e297d2c8d0ab46b8bcdf027d38c51da591", size = 264491, upload-time = "2026-08-15T08:17:52.61Z" }, + { url = "https://files.pythonhosted.org/packages/5c/64/12b4c2a11ee8df4fcc518c78b0d93e3a92bd3d5253d1617ce74ff0e8c7ef/charset_normalizer-3.5.1-cp313-cp313-musllinux_1_2_riscv64.whl", hash = "sha256:90b7481fb62fbe172c558bc6fd1c4c98d82004a54a7551f20e11ac9bf0b8708c", size = 245196, upload-time = "2026-08-15T08:17:54.023Z" }, + { url = "https://files.pythonhosted.org/packages/37/2e/651d910af6d0fba325eee1cda37ec5443462ed25360e666c144166eb6091/charset_normalizer-3.5.1-cp313-cp313-musllinux_1_2_s390x.whl", hash = "sha256:35fe081843b35aad20ffeccec3eeffbe637b15d14f3fb22cc1b59cd8ec17e93c", size = 261660, upload-time = "2026-08-15T08:17:55.491Z" }, + { url = "https://files.pythonhosted.org/packages/90/c6/b09e05e6db7f64338e0dc067c79577b1138da86c1e38369096851d96be88/charset_normalizer-3.5.1-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:fd0350afdc3aabd5576f60ea109228bd5538139713c7b094c5cd27c73a98bc6f", size = 252618, upload-time = "2026-08-15T08:17:57.025Z" }, + { url = "https://files.pythonhosted.org/packages/76/4e/362d4f9fdcdf5556fb2aa3ce7d4a58ebce03ed1ff03aa1d9aca8d02f13f3/charset_normalizer-3.5.1-cp313-cp313-pyemscripten_2025_0_wasm32.whl", hash = "sha256:9d9a0dc7cbe9bec24c3f767c9122c41fe5a1bc43f47cd099d00d393e09769de4", size = 140362, upload-time = "2026-08-15T08:17:58.425Z" }, + { url = "https://files.pythonhosted.org/packages/b4/d4/703be739b26acce318bd29eb3b25b7209e1b1f527f9eae3d1f1f01fdde2b/charset_normalizer-3.5.1-cp313-cp313-win32.whl", hash = "sha256:d63600d620ad0064c3a748b950ac5ea38a80190e5498532efefa4b7b3f1da1f3", size = 177755, upload-time = "2026-08-15T08:18:00.037Z" }, + { url = "https://files.pythonhosted.org/packages/8a/33/56d97ade41c8db611e727168c52ae46c9224c362ec28d4b65d7e9869e8da/charset_normalizer-3.5.1-cp313-cp313-win_amd64.whl", hash = "sha256:aea996a6aba25260827c9ea511d1addfde2da9eb686ac961838509086188b7e6", size = 199295, upload-time = "2026-08-15T08:18:01.506Z" }, + { url = "https://files.pythonhosted.org/packages/5b/75/5b20dd1e6573a01a08158fe104104fa2c8abf941745596954185726cd46c/charset_normalizer-3.5.1-cp313-cp313-win_arm64.whl", hash = "sha256:fd0a274c0e5f9a21565cd9d3dd749b61f96b7aa1e20a93aa1ba4029518f2e5c0", size = 179856, upload-time = "2026-08-15T08:18:02.929Z" }, + { url = "https://files.pythonhosted.org/packages/29/cd/2b812ce5e888f1ce69a5350281e58aab07ae64a958ecae8912f30865718e/charset_normalizer-3.5.1-cp314-cp314-android_24_arm64_v8a.whl", hash = "sha256:774d157f112367ff4abd29019f38f023c24e00e56edc7829c20e358a5a913ad8", size = 212318, upload-time = "2026-08-15T08:18:04.403Z" }, + { url = "https://files.pythonhosted.org/packages/9e/4a/a6ee107430768a5334e6d63f31f148a04a1a491ef161a1ac9415a73f2fa8/charset_normalizer-3.5.1-cp314-cp314-android_24_x86_64.whl", hash = "sha256:26422d45fd13551cf564c58932f7d72b4f58b93b0fcf18c35ba6be12b46bb102", size = 224897, upload-time = "2026-08-15T08:18:05.997Z" }, + { url = "https://files.pythonhosted.org/packages/c3/d9/35ae3f64f29d0179c35c3baefe575904df2913dde519129c7f75995a2b1d/charset_normalizer-3.5.1-cp314-cp314-ios_13_0_arm64_iphoneos.whl", hash = "sha256:09a7bba9f739468c8e78c36a75c33768e53cb1959fc638f510454c14683f00d5", size = 194848, upload-time = "2026-08-15T08:18:07.397Z" }, + { url = "https://files.pythonhosted.org/packages/74/76/f2fc7380f056cc273a53af37f50d08ad54b2c59f61078f31432edcf1c2bd/charset_normalizer-3.5.1-cp314-cp314-ios_13_0_arm64_iphonesimulator.whl", hash = "sha256:4c9548dc78002099910abaebc0a72ac58b7d30931869e0351c09b507dff4ece3", size = 198163, upload-time = "2026-08-15T08:18:08.989Z" }, + { url = "https://files.pythonhosted.org/packages/e9/40/095ce62fa078483cccc1fa2b36e6bc9580b85422a20ee9f925341c50e44f/charset_normalizer-3.5.1-cp314-cp314-macosx_10_15_universal2.whl", hash = "sha256:c428c6c31eb5f4277d7f8eccaf767fbd548ddd5ce3c8b4f4cbbfab3d96b5904c", size = 341823, upload-time = "2026-08-15T08:18:10.458Z" }, + { url = "https://files.pythonhosted.org/packages/f1/5a/0e58b1c04a1596e0256f407274a92d5fb2ee21324409d1fab1da48a65b5b/charset_normalizer-3.5.1-cp314-cp314-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:2f06b7eae9dbe77fe1d644ca244dad508de8d302870a43f3c559b521270938a0", size = 242458, upload-time = "2026-08-15T08:18:11.989Z" }, + { url = "https://files.pythonhosted.org/packages/22/95/b4618ce912e6db0b1aae89ba788e38e8a7eba0f3025cc66e8c0699f977b2/charset_normalizer-3.5.1-cp314-cp314-manylinux2014_armv7l.manylinux_2_17_armv7l.manylinux_2_31_armv7l.whl", hash = "sha256:6b7430cf5728e68f6c462254009a6ef4086e1bea43cf2f57aa9c55fb4f50ff96", size = 226717, upload-time = "2026-08-15T08:18:13.401Z" }, + { url = "https://files.pythonhosted.org/packages/8a/76/c681192bbda3d55356db5dadd64381d5202b37c6b598fcda5282e88b5d3d/charset_normalizer-3.5.1-cp314-cp314-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:ab743e9bc90c1f73552ec33e10e3331315acd2c397b36065b591b0181de533cc", size = 266111, upload-time = "2026-08-15T08:18:14.961Z" }, + { url = "https://files.pythonhosted.org/packages/88/be/55127bfca72c0cff6c022488d140d7c5b04c771e3b72e9bdb4836d54979d/charset_normalizer-3.5.1-cp314-cp314-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:f6f7deae3feb4edfa2efaf7c574fe88cbf055038a6abdb40188e4fff66d5699f", size = 263128, upload-time = "2026-08-15T08:18:16.515Z" }, + { url = "https://files.pythonhosted.org/packages/e0/91/39c3af510b0aa32bbda03374259200f28430febfd1bf5e511fe765282ce5/charset_normalizer-3.5.1-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:15f024313246a4ed976c60f440bb8d257815513a681d212ff74fd46f7d715a90", size = 251240, upload-time = "2026-08-15T08:18:18.127Z" }, + { url = "https://files.pythonhosted.org/packages/1c/a5/cbe418bbc6ecdfc3e05a0116002897c4b403a5e838d697e64c78e9f0190d/charset_normalizer-3.5.1-cp314-cp314-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:823f82903d189af463d7df250ef1f7f696f3cee08cc8d91deb565e8d425f6506", size = 245282, upload-time = "2026-08-15T08:18:19.625Z" }, + { url = "https://files.pythonhosted.org/packages/cc/a4/689bb42e8e7cd492f3cb64907c6bc00ad247ec9a3628cd3f8eed126e8ae1/charset_normalizer-3.5.1-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:01e93745f7f219b703b60ba7afead36cfc4242782be5af484673fc500df12da5", size = 244597, upload-time = "2026-08-15T08:18:21.121Z" }, + { url = "https://files.pythonhosted.org/packages/c1/ce/9962938e179cf9f699d3f1e7b3114b5d7642dee6a893745229f9dd04f274/charset_normalizer-3.5.1-cp314-cp314-musllinux_1_2_armv7l.whl", hash = "sha256:329fc3ccb63ad22d867d84c2adea759a64079a37ba4a343433b02c7a2816871e", size = 231376, upload-time = "2026-08-15T08:18:22.57Z" }, + { url = "https://files.pythonhosted.org/packages/85/54/46000450ada53bd9eac5429a2c8c54cd2d9b39c0c255f229aea9af0948a5/charset_normalizer-3.5.1-cp314-cp314-musllinux_1_2_ppc64le.whl", hash = "sha256:bb57753e36e4855b8ca375069482250a6246372331a3e4f3407eaebb007443f5", size = 266715, upload-time = "2026-08-15T08:18:24.235Z" }, + { url = "https://files.pythonhosted.org/packages/3d/bb/618749d70f792b44252a777bf89bfb86823b9bbc1ea13fe8ce759b07f38a/charset_normalizer-3.5.1-cp314-cp314-musllinux_1_2_riscv64.whl", hash = "sha256:fce8cbd4997efeb450bd298b54f755dcdff18d496f7a5ddbb4867c6d7c88fdc3", size = 245848, upload-time = "2026-08-15T08:18:25.726Z" }, + { url = "https://files.pythonhosted.org/packages/7e/3f/ffb64458527c7668031d5eb095d978de561958dc9f5b53f8e488a533e603/charset_normalizer-3.5.1-cp314-cp314-musllinux_1_2_s390x.whl", hash = "sha256:6c9cdde8becb25a7fde49924511aa2644d6f8081cc8df8e9452724303348d8e3", size = 264521, upload-time = "2026-08-15T08:18:27.193Z" }, + { url = "https://files.pythonhosted.org/packages/4f/ab/74a55fd803916a35ac461daf002708191aac19b546b80dc8cabfedc63d98/charset_normalizer-3.5.1-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:9ac4444d8d4fd4c4bd08bf451ed3167aa9e7ec6cdb41b648794f1d1103652e36", size = 253054, upload-time = "2026-08-15T08:18:28.568Z" }, + { url = "https://files.pythonhosted.org/packages/a0/2a/6a9034b7d3c60b17499afb482df5878bf9fa20b50cc3887d5ef017a833db/charset_normalizer-3.5.1-cp314-cp314-pyemscripten_2026_0_wasm32.whl", hash = "sha256:f03ac127268b43ef4fe9e6ab6794a6794b49485a0cc0c1db79876d2f33f75bc7", size = 140580, upload-time = "2026-08-15T08:18:30.214Z" }, + { url = "https://files.pythonhosted.org/packages/f3/46/1d362e1a00d035d66b9869e1281eee115907f7e390a16a07824ab5737360/charset_normalizer-3.5.1-cp314-cp314-win32.whl", hash = "sha256:1f5883d77fd409a261abb5dc8ccbe335720d798b1de4abb3b1d47ccbbc76b53b", size = 180325, upload-time = "2026-08-15T08:18:31.877Z" }, + { url = "https://files.pythonhosted.org/packages/7a/7c/4938c329b6a9d446f6a59aa2092ff7118f274209b5ed0e26893d1d30a63c/charset_normalizer-3.5.1-cp314-cp314-win_amd64.whl", hash = "sha256:c658c50ac0c98cd755a2dd50b7977d3bca7df401dcc47fbdfa87db53ef7d4e8b", size = 204175, upload-time = "2026-08-15T08:18:33.466Z" }, + { url = "https://files.pythonhosted.org/packages/ac/33/eeb384dbd8dec570661354592f4f2e1b2fcc92585624d146a000caf53841/charset_normalizer-3.5.1-cp314-cp314-win_arm64.whl", hash = "sha256:4bea7f8ebe90bbd7f0e4a2de42ca6924ba23e3e76418c408ff82f1d46fabd687", size = 184123, upload-time = "2026-08-15T08:18:34.913Z" }, + { url = "https://files.pythonhosted.org/packages/1c/6c/c73fa9d5a85f6ab05395de61c5f6984e0a9ff40bb5ff888d46dff02526c6/charset_normalizer-3.5.1-cp314-cp314t-macosx_10_15_universal2.whl", hash = "sha256:fbc597639158fd7c14d55e808718848319540f51b0e6746e3eefa59723a4a348", size = 381682, upload-time = "2026-08-15T08:18:36.349Z" }, + { url = "https://files.pythonhosted.org/packages/30/c7/63565f860921457feba93bae6c86fb7746deb4cffeed2f375cb845318146/charset_normalizer-3.5.1-cp314-cp314t-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:e71c909f353863b2b89c83de2ebed71ea6d0df8a6ef65a128193c5e650766bef", size = 240826, upload-time = "2026-08-15T08:18:37.887Z" }, + { url = "https://files.pythonhosted.org/packages/06/ae/7ae8807410dfa33f8e6f1715740adeaafa8a816cc4cb33508f54b1f7c896/charset_normalizer-3.5.1-cp314-cp314t-manylinux2014_armv7l.manylinux_2_17_armv7l.manylinux_2_31_armv7l.whl", hash = "sha256:7ac76cf9afd34929d76eb7fcb63be476a4853d8a96f0dcf2d0db68a0cbdf9885", size = 227861, upload-time = "2026-08-15T08:18:39.315Z" }, + { url = "https://files.pythonhosted.org/packages/e9/a3/887c1642f0da26000b0e0652d91071113c0e72cea33952e225cf589f49a9/charset_normalizer-3.5.1-cp314-cp314t-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:a3a370082ce34d0612f421e15fe011c53bb1feff21a26d06ad4fb244dab5a375", size = 260758, upload-time = "2026-08-15T08:18:40.88Z" }, + { url = "https://files.pythonhosted.org/packages/3e/11/e6f5b9a3d0e55b0ef7505cd3765cdd48f22db89994c947b316f52f801fd8/charset_normalizer-3.5.1-cp314-cp314t-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:256dd4d85d9e4dc595e2bc983c980e73f62ddeb3165c58b4c3dfe78c5c8548c1", size = 259950, upload-time = "2026-08-15T08:18:42.351Z" }, + { url = "https://files.pythonhosted.org/packages/1b/ee/e4e10a94d51cd1ee638aa7e00b65399e6b2a4e8376ab6d2eac9f95586671/charset_normalizer-3.5.1-cp314-cp314t-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:58d4aa13a59c969dbfdf9e6a9560e242cbfd9e8a8f50c2747714df1a423adf65", size = 249329, upload-time = "2026-08-15T08:18:43.914Z" }, + { url = "https://files.pythonhosted.org/packages/c4/25/d5f4198819e6059735a84e8d0bfb72dc33976da67b97adcd3fb5a5e07ec6/charset_normalizer-3.5.1-cp314-cp314t-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:0c6dfb5ca6723eeed15aa8e564a014d69fcb8812f94eef11fe3631e0508199f5", size = 243137, upload-time = "2026-08-15T08:18:45.368Z" }, + { url = "https://files.pythonhosted.org/packages/a5/e9/e925ca7569cf9fb9701fd82503fee73eea5268fdb856bdd64947092d3daa/charset_normalizer-3.5.1-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:c010f5581d9c612804cc59fcf7b524b707fbcb72828551237ab545bb5c7034af", size = 242820, upload-time = "2026-08-15T08:18:46.842Z" }, + { url = "https://files.pythonhosted.org/packages/34/17/672c251a888ed2aebcdd2fe830ad0104e25ff83c43f5c4f9c15e9fc6853c/charset_normalizer-3.5.1-cp314-cp314t-musllinux_1_2_armv7l.whl", hash = "sha256:52ec005752a56ae79547a05c0139ca2501a0c866390b6115008456b9f0e7cde1", size = 230504, upload-time = "2026-08-15T08:18:48.353Z" }, + { url = "https://files.pythonhosted.org/packages/3f/fc/f6a85abebd42ce4da2f1db0aa56cc6a0df1995e318b3875d14401b8381d1/charset_normalizer-3.5.1-cp314-cp314t-musllinux_1_2_ppc64le.whl", hash = "sha256:2bced4061f000f7187254a02ad3433ae17eaf991747ceea2f478422590a5bba9", size = 263087, upload-time = "2026-08-15T08:18:49.859Z" }, + { url = "https://files.pythonhosted.org/packages/98/66/7c42677e739ba66746b297e2046918d793078094dc239e1e72768cffccc6/charset_normalizer-3.5.1-cp314-cp314t-musllinux_1_2_riscv64.whl", hash = "sha256:9eea3ab2597a5e65fe65296e2d6a84570845a6b55532d90333d740d48bbc850a", size = 243269, upload-time = "2026-08-15T08:18:51.601Z" }, + { url = "https://files.pythonhosted.org/packages/de/d8/a50b79237f417af10f8c2a501ce8d1ca87829a22e69117891ca4ba20a69e/charset_normalizer-3.5.1-cp314-cp314t-musllinux_1_2_s390x.whl", hash = "sha256:496846868fea80e479324862fa877f02411f2fd0f83b79ccee2607aa68b2a032", size = 258766, upload-time = "2026-08-15T08:18:53.23Z" }, + { url = "https://files.pythonhosted.org/packages/2e/1d/0fc91aeaeb3c83b748f532399ce67cf84604b48297405d740000f7a9e786/charset_normalizer-3.5.1-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:85d5855daafc240cc045c026d7a15fd198a09b0fc8ff6f5ecbb5297b509cb11e", size = 250814, upload-time = "2026-08-15T08:18:54.768Z" }, + { url = "https://files.pythonhosted.org/packages/ae/10/3d8c777cf9024615295aa1b808324ad5b4a77855869c00824bad74ffaf8a/charset_normalizer-3.5.1-cp314-cp314t-win32.whl", hash = "sha256:58d3e12c88e0950bca850ae1f7c256055c097639c2edb9eb123af9807d8b15e4", size = 191074, upload-time = "2026-08-15T08:18:56.305Z" }, + { url = "https://files.pythonhosted.org/packages/4d/81/ae557d3c44d1a1d688696d60563413a0866a91b7ebc50f20df838be3d8c8/charset_normalizer-3.5.1-cp314-cp314t-win_amd64.whl", hash = "sha256:acaf604462bf330b0d07e7a07c1d6e4adac79e5fb13e9c5140590542cafacc00", size = 216476, upload-time = "2026-08-15T08:18:57.889Z" }, + { url = "https://files.pythonhosted.org/packages/27/e9/61c01fb8b804692569c036b3fc50495814502dcf13a60649c6055390b02c/charset_normalizer-3.5.1-cp314-cp314t-win_arm64.whl", hash = "sha256:fdb8a068947befafba9952162645dc2fecaeb400e64584829ed5e9b2fbe21a7f", size = 194115, upload-time = "2026-08-15T08:18:59.418Z" }, + { url = "https://files.pythonhosted.org/packages/4a/4e/8544831ef59d8f27ce92c80871380fdacc8076a8a56ed62f82e54f991333/charset_normalizer-3.5.1-cp315-cp315-macosx_10_15_universal2.whl", hash = "sha256:9085f87b0e38a2b92b8923059b4e8789fe40d9279712d15dcc670048d77079af", size = 342048, upload-time = "2026-08-15T08:19:01.054Z" }, + { url = "https://files.pythonhosted.org/packages/7f/a6/e3b46852424246065355644f4fb6dbccc0239a42a2eee27ecfc8957f0bcd/charset_normalizer-3.5.1-cp315-cp315-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:2679de311c7946dde5d3b6f44941844133ff5c7cb86099c0061ab1e8901c20a8", size = 242997, upload-time = "2026-08-15T08:19:02.492Z" }, + { url = "https://files.pythonhosted.org/packages/03/3b/0cc9a26777334ab2f2e3089b948bbf4e4fe72ea70b897715ef6415043ec8/charset_normalizer-3.5.1-cp315-cp315-manylinux2014_armv7l.manylinux_2_17_armv7l.manylinux_2_31_armv7l.whl", hash = "sha256:baf3775a2635e5a11fbd5e4e64ee69c7e86875d224a5c72aca4c141064589a90", size = 237014, upload-time = "2026-08-15T08:19:03.943Z" }, + { url = "https://files.pythonhosted.org/packages/8c/c2/027335f0aa337a2a2e121bac1ad88c4f02ba6053ea0926802784f3db11af/charset_normalizer-3.5.1-cp315-cp315-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:8ac8c94b6539074e0f40899301273ac8402b9b3e01c7b7ba269ff30340aaaf20", size = 266174, upload-time = "2026-08-15T08:19:05.598Z" }, + { url = "https://files.pythonhosted.org/packages/86/d3/e367787febe4e74769dec0f406f2c3c8d1b955fce5aee1fd0f94e8367a45/charset_normalizer-3.5.1-cp315-cp315-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:8fe532b3c966d1fb794e0698e4589d0444017ae77fc0b31edea13c0e35bcc449", size = 263361, upload-time = "2026-08-15T08:19:07.251Z" }, + { url = "https://files.pythonhosted.org/packages/af/3d/391b193eb9f3e84b02f9314088c386debdc0debee843535aaea2e2c6715d/charset_normalizer-3.5.1-cp315-cp315-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:5c84bec0ab5ae0c64bfe73a7d2adcb5ce73b467523fc27fd6a28ab2aa6cbe35a", size = 252143, upload-time = "2026-08-15T08:19:08.816Z" }, + { url = "https://files.pythonhosted.org/packages/2e/57/de221f1745a90d418199761967e2776bfe2c275a1194220985e8c1d37833/charset_normalizer-3.5.1-cp315-cp315-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:854066be00447fa8de2ccbbe893e2ffc4b123ef16d897af794c1e18bd4a714b0", size = 252086, upload-time = "2026-08-15T08:19:10.255Z" }, + { url = "https://files.pythonhosted.org/packages/c8/e3/d119f86a01f9331e8186175f24873b1d74a7ee9e2e4b4d68f9947dae5afd/charset_normalizer-3.5.1-cp315-cp315-musllinux_1_2_aarch64.whl", hash = "sha256:21b82d8082f6f5e7f456ef0bd16323d08de1266efbfeb476e64b2a91d1471a4e", size = 245231, upload-time = "2026-08-15T08:19:11.807Z" }, + { url = "https://files.pythonhosted.org/packages/26/de/d8e48c135ae480879539cdb179c8d3b50c7879497d75dd899b5763b69cee/charset_normalizer-3.5.1-cp315-cp315-musllinux_1_2_armv7l.whl", hash = "sha256:838648accb3a7fd9803fd45c87bce8509648eb0c11bc34e216141300977244f2", size = 241546, upload-time = "2026-08-15T08:19:13.416Z" }, + { url = "https://files.pythonhosted.org/packages/67/c4/217755fd1abc50d326c252922cd642002758095a81ff45010337b8b3ef65/charset_normalizer-3.5.1-cp315-cp315-musllinux_1_2_ppc64le.whl", hash = "sha256:195ce897c6153c0700078142cf8efe3e6454ca4cf4357499e4078dfd83396626", size = 267033, upload-time = "2026-08-15T08:19:14.981Z" }, + { url = "https://files.pythonhosted.org/packages/b8/d7/34d8e404e358d2adcc5a228c2134643af00104c8fb0bf525f3688d756f05/charset_normalizer-3.5.1-cp315-cp315-musllinux_1_2_riscv64.whl", hash = "sha256:978eab16f55b4ab2c2a745be9a0a840bf8f09a7f227d9c76eb30214d078865a5", size = 252045, upload-time = "2026-08-15T08:19:16.618Z" }, + { url = "https://files.pythonhosted.org/packages/5e/fa/40414471acf0aa0692ca77305aa00e434fcd8288f0941c93c30e9a5f8f2f/charset_normalizer-3.5.1-cp315-cp315-musllinux_1_2_s390x.whl", hash = "sha256:cc0329df4caaceb950d2f580b5ac716a377f7059624a0bafaeaf8a218c6ed774", size = 264866, upload-time = "2026-08-15T08:19:18.101Z" }, + { url = "https://files.pythonhosted.org/packages/32/90/fcc850bae791abd2e0c041847f13e270aa08692a79f3e00de6d2dce1cb50/charset_normalizer-3.5.1-cp315-cp315-musllinux_1_2_x86_64.whl", hash = "sha256:687c9ca3035544b113bea2055e180af96fb63c0c476e22a9180f51925186e7b7", size = 253932, upload-time = "2026-08-15T08:19:19.734Z" }, + { url = "https://files.pythonhosted.org/packages/af/af/53afe99068b3c10b4cbae592a52ef72a7c92c0188440e83ee3a078fd8f75/charset_normalizer-3.5.1-cp315-cp315-win32.whl", hash = "sha256:706bfd38730a5ac7a365793269a00f4e988178cec121391f4248d84ad8c972e9", size = 180320, upload-time = "2026-08-15T08:19:21.37Z" }, + { url = "https://files.pythonhosted.org/packages/c9/bc/f46a132041b29e4a8779ed712d3df1bf112e94ca8de58b66d7ec2c0cf8b9/charset_normalizer-3.5.1-cp315-cp315-win_amd64.whl", hash = "sha256:92caef967d287a407085d61176fce4012b1dd62daed4eb6d5ceb26d3d2538712", size = 204174, upload-time = "2026-08-15T08:19:23.088Z" }, + { url = "https://files.pythonhosted.org/packages/a1/5d/9ed554480eda8e447b673648628fdc29574d23dbad01fe11837adedd1cae/charset_normalizer-3.5.1-cp315-cp315-win_arm64.whl", hash = "sha256:5fc45d653ea8c9a20479167e11d4a0f8cb2fa3470737ab6f9c827532313187b7", size = 184126, upload-time = "2026-08-15T08:19:24.471Z" }, + { url = "https://files.pythonhosted.org/packages/3b/32/9b8929bf384061ee1fe5d9c27c6f9776d3d824039ad4e14c88ec00c7808e/charset_normalizer-3.5.1-cp315-cp315t-macosx_10_15_universal2.whl", hash = "sha256:59171c6e45bf07d0d5cab3b0bf81d945035530f6873398b3b531c31184d46663", size = 381441, upload-time = "2026-08-15T08:19:26.038Z" }, + { url = "https://files.pythonhosted.org/packages/96/10/e9aa7923d3ddac652c99a1c5f7be494e737e151566a44abe018daf757f2c/charset_normalizer-3.5.1-cp315-cp315t-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:9dbdd9205662134957cf0c324f639bdc5031c0ca056e2369e238db75187c0f11", size = 241742, upload-time = "2026-08-15T08:19:27.532Z" }, + { url = "https://files.pythonhosted.org/packages/28/53/a2d249ebddf47b889a100c0bdcb61a2f9dbb8bc24ef325cc062e4f476877/charset_normalizer-3.5.1-cp315-cp315t-manylinux2014_armv7l.manylinux_2_17_armv7l.manylinux_2_31_armv7l.whl", hash = "sha256:e4b018dc5a0eee4676e38fe84a47a427816c590b93b55d9025274ec4d6ffc2dc", size = 235298, upload-time = "2026-08-15T08:19:29.274Z" }, + { url = "https://files.pythonhosted.org/packages/7d/07/469f78af590f7d5cd48e20d8dbfa3d66deeff9ba37768c04d886b5afd45c/charset_normalizer-3.5.1-cp315-cp315t-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:ced3fdd71aaa83ce593746c2edb42b7a59cb4c19c8b5c407781c72e493aae55a", size = 262500, upload-time = "2026-08-15T08:19:30.955Z" }, + { url = "https://files.pythonhosted.org/packages/55/66/3bb56a47f7dcba014055b1a1d33c6f08bbe9c1e74dba154cfa25f90ae885/charset_normalizer-3.5.1-cp315-cp315t-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:19a3dd5aa73cef1c99687c4fc57db016a9c17104ae1185da88ba566a5d3bebe4", size = 258888, upload-time = "2026-08-15T08:19:32.458Z" }, + { url = "https://files.pythonhosted.org/packages/ff/c1/2adc2800903fb013210349313b710a5376856578d9e33e6b9a1d8b36714a/charset_normalizer-3.5.1-cp315-cp315t-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:cc5d36d96478aa9c60654bd932525bf32964c62a7281eafdf16d85003a8d6004", size = 250243, upload-time = "2026-08-15T08:19:33.94Z" }, + { url = "https://files.pythonhosted.org/packages/95/b5/a18d0dd1157ab655cc2cb14a545f4a4784bbad70ab3502412e36097502d9/charset_normalizer-3.5.1-cp315-cp315t-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:04368edf83514385ffc3e1cfd4546e595f4f1272dd23ba437a93a9cc3741d47b", size = 249871, upload-time = "2026-08-15T08:19:35.413Z" }, + { url = "https://files.pythonhosted.org/packages/ad/c3/525f508cd1e58d0450ac55ed40ac75bc3a97482c59def5278456a5fbf03c/charset_normalizer-3.5.1-cp315-cp315t-musllinux_1_2_aarch64.whl", hash = "sha256:9b5db6052055d34d41230fb78d7c439c23dc536a9896f6cb039e8dd92cfc1263", size = 243580, upload-time = "2026-08-15T08:19:36.886Z" }, + { url = "https://files.pythonhosted.org/packages/7c/c1/49a91fe7e97c8140094ca5c64161ab623a70d9f636bf834eace14048acb5/charset_normalizer-3.5.1-cp315-cp315t-musllinux_1_2_armv7l.whl", hash = "sha256:252d099029bcbea642f2a06c4ed5046bdf8b5a8150b64afa5e027e88b106e5ee", size = 239807, upload-time = "2026-08-15T08:19:38.392Z" }, + { url = "https://files.pythonhosted.org/packages/d3/58/56a48c296601274c4689b864a8e2dfb209b81dfcb39472753ce95eea662b/charset_normalizer-3.5.1-cp315-cp315t-musllinux_1_2_ppc64le.whl", hash = "sha256:6199d5606e2bbf2b096cf64d03f8b6790c91081d5ac866b8e7bb6422738cc60c", size = 264083, upload-time = "2026-08-15T08:19:39.856Z" }, + { url = "https://files.pythonhosted.org/packages/10/4c/dc48409274a1817ff349711d26c62aa0c597df865d4d69ef79160c859193/charset_normalizer-3.5.1-cp315-cp315t-musllinux_1_2_riscv64.whl", hash = "sha256:77efcff2b23071c349402ac1066667a3d011f62398d81408c9b88ad991747c9e", size = 250317, upload-time = "2026-08-15T08:19:41.53Z" }, + { url = "https://files.pythonhosted.org/packages/81/58/d325912115caec62d6bdd77bbab5e0b7da5d234a9f20affdffcbcb530d0b/charset_normalizer-3.5.1-cp315-cp315t-musllinux_1_2_s390x.whl", hash = "sha256:a5cbd90ecf0fc62e64726917ad083b73001f0563657a87ec3c0b504e277dc90d", size = 258173, upload-time = "2026-08-15T08:19:43.07Z" }, + { url = "https://files.pythonhosted.org/packages/34/f7/b13b1ccae2c8ec63980d13be1890eb73f8aeabbfce02a24aabc0908788f5/charset_normalizer-3.5.1-cp315-cp315t-musllinux_1_2_x86_64.whl", hash = "sha256:4d26f14f041e83dd8edfd61f4cd4fa7285d31798b5bf1f28e70c367ba6c41d61", size = 251960, upload-time = "2026-08-15T08:19:44.587Z" }, + { url = "https://files.pythonhosted.org/packages/1e/25/ed3f9919c5aef8cc818be1f972f565f7610d7b2076b8ebb98839516ffc3c/charset_normalizer-3.5.1-cp315-cp315t-win32.whl", hash = "sha256:ac13b004224fb341e1e25a1ed5e19d32f57cdb2a403e01f003b46f051a550f6f", size = 191186, upload-time = "2026-08-15T08:19:46.293Z" }, + { url = "https://files.pythonhosted.org/packages/69/d5/43c2b3e9d8267092b913eb8b0603f0f71993c395632886bd37a7223f96cf/charset_normalizer-3.5.1-cp315-cp315t-win_amd64.whl", hash = "sha256:35aea775dc2bd5f54cd84a1cd2696cc3207c479cb9cf0bd346f0d343e4300ddb", size = 215947, upload-time = "2026-08-15T08:19:47.853Z" }, + { url = "https://files.pythonhosted.org/packages/a8/76/9aad3e9c8865e5e0efa9a7f6f81c37a67635a985145ecd44528a81e088ee/charset_normalizer-3.5.1-cp315-cp315t-win_arm64.whl", hash = "sha256:fb78f6e7fcd8ad785d28cd577168bc1aaee827b25bb8755638f694794ea98f0a", size = 193909, upload-time = "2026-08-15T08:19:49.383Z" }, + { url = "https://files.pythonhosted.org/packages/5b/97/fb4e82231aba271ffd775a1b4993b0defc4e3059f286ae41d9433409fe85/charset_normalizer-3.5.1-cp37-abi3-macosx_10_9_universal2.whl", hash = "sha256:41876ee62a3dddf48ff1121ad8f0798032aa03f2fd35f21f34a4cab14f18d8d2", size = 331467, upload-time = "2026-08-15T08:19:50.959Z" }, + { url = "https://files.pythonhosted.org/packages/9f/2f/fe3f187327aac18e2d54e9d2b08e15d27bf9b642d9e51c219f130fc34d1a/charset_normalizer-3.5.1-cp37-abi3-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl", hash = "sha256:a6dac12ff6b846103483683f60c5f8fee205121adc58ffd87e90a90a3af69e99", size = 253057, upload-time = "2026-08-15T08:19:52.654Z" }, + { url = "https://files.pythonhosted.org/packages/d7/c7/9e48cee5c161fe24da823b61bf381921d77cb994a0a4de148e95018c1984/charset_normalizer-3.5.1-cp37-abi3-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:cee5dd7c6fb5dd52a0fe2a740f9bc6e3593f5f8b1788bde49de02086f30182b2", size = 240930, upload-time = "2026-08-15T08:19:54.163Z" }, + { url = "https://files.pythonhosted.org/packages/49/e0/716601f3cc69be7b198951150c75ead1ece33c3c8036ff6ffa46029659a0/charset_normalizer-3.5.1-cp37-abi3-manylinux2014_armv7l.manylinux_2_17_armv7l.manylinux_2_31_armv7l.whl", hash = "sha256:343fb4f2821043bd87095f7b08a1a181febc8e36ac64212143bbfd0a0e1bc235", size = 230822, upload-time = "2026-08-15T08:19:55.807Z" }, + { url = "https://files.pythonhosted.org/packages/d3/05/71bfc5caa0abcc45aea1f6a4d50ac68e59605ddc7666fe8494f4cd229665/charset_normalizer-3.5.1-cp37-abi3-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:ae4a097991662cd4fff0ddc74e0fe7874f82e00042fa0ea00855645ed0c79598", size = 260037, upload-time = "2026-08-15T08:19:57.312Z" }, + { url = "https://files.pythonhosted.org/packages/c3/92/de7e32ed05341e7a9c4c877c318418197b7f2d66a3b68d561bf2ac57ca3e/charset_normalizer-3.5.1-cp37-abi3-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:4b599739b93b2cbeded49645ae3c8d1405c29ddfbceac1545c87a3f9580a9e96", size = 255097, upload-time = "2026-08-15T08:19:59.056Z" }, + { url = "https://files.pythonhosted.org/packages/f5/7b/ade0a122600319dfa0b1000ab0f9731c94a817904cf3c5de408c73a4ede7/charset_normalizer-3.5.1-cp37-abi3-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:b39b69b347e5e47a3b5b8cfc005c68c1ba347474e3960236c4944a8ecd174962", size = 250166, upload-time = "2026-08-15T08:20:00.612Z" }, + { url = "https://files.pythonhosted.org/packages/75/9c/019fbb9f4834491a160951349b1a3714439376f66e5f7cf18b4f18f0c7aa/charset_normalizer-3.5.1-cp37-abi3-musllinux_1_2_aarch64.whl", hash = "sha256:a2028475ba855475b8b4d3cfeb4994269c967aea8b9892dfba907f4263a863a3", size = 241821, upload-time = "2026-08-15T08:20:02.321Z" }, + { url = "https://files.pythonhosted.org/packages/2b/b8/11d4840bfc99330cc7fbcc2681ee5a044553a6e77655508d8f9b2bff7b34/charset_normalizer-3.5.1-cp37-abi3-musllinux_1_2_armv7l.whl", hash = "sha256:36047af20e17097c3bb9476c2b7655f2f7aa51322c0ba58c07695bedf755a950", size = 232529, upload-time = "2026-08-15T08:20:04.008Z" }, + { url = "https://files.pythonhosted.org/packages/18/96/2b3a21492d9f65171ac75d872f5018260013d00bfa0ff70ec9f179148cbd/charset_normalizer-3.5.1-cp37-abi3-musllinux_1_2_ppc64le.whl", hash = "sha256:4c4fb141a727957c93edfe5c32a26ceb6b5f6461d67146e2d39f51e16170bea8", size = 260348, upload-time = "2026-08-15T08:20:05.877Z" }, + { url = "https://files.pythonhosted.org/packages/d6/aa/a69a2028e8bd052476c245460ab19d7de595de084dd968f2d75cd50c3e25/charset_normalizer-3.5.1-cp37-abi3-musllinux_1_2_riscv64.whl", hash = "sha256:2f293479cce755c75f1697e87c409b7ae4c555c7dfecb6e988ad13abba943031", size = 247234, upload-time = "2026-08-15T08:20:07.487Z" }, + { url = "https://files.pythonhosted.org/packages/35/8a/3d130aeabcaf3d2466af76b7b141c08d9e89c9016ab4b7cdd0f7dc2d1c62/charset_normalizer-3.5.1-cp37-abi3-musllinux_1_2_s390x.whl", hash = "sha256:3588e376b3ea2eea84976f67273d679f229e24c66dce7b82ae45aef04ff6e072", size = 256917, upload-time = "2026-08-15T08:20:09.142Z" }, + { url = "https://files.pythonhosted.org/packages/80/c2/a7379b840292d0c1ab9fbd17d1f3967aa81794dc95bc74be8999d7fedcf7/charset_normalizer-3.5.1-cp37-abi3-musllinux_1_2_x86_64.whl", hash = "sha256:e199fb99720074809a7720f1c0b4d919eea8b87e88713e0f8f602f7bef543d9d", size = 254846, upload-time = "2026-08-15T08:20:10.727Z" }, + { url = "https://files.pythonhosted.org/packages/01/65/d43b714731bb2f40d4053dfa00ecfc1c5a301f8e3316c5db3a09af59fe94/charset_normalizer-3.5.1-cp37-abi3-win32.whl", hash = "sha256:dd732602a7009217f658d5863d12d79d373a4de0eebc111094bcdd3bb8e0a6cc", size = 174216, upload-time = "2026-08-15T08:20:12.334Z" }, + { url = "https://files.pythonhosted.org/packages/35/4f/b911ed898b26a09789eba9c9200c999aff6c61b4bafaf4838e56d1a1e1a3/charset_normalizer-3.5.1-cp37-abi3-win_amd64.whl", hash = "sha256:70055ff39b97c99e7ae40ea3e393fb62aa2e44dbd9b29f8d14f42fb0025c3959", size = 199764, upload-time = "2026-08-15T08:20:13.908Z" }, + { url = "https://files.pythonhosted.org/packages/f0/a7/920baf467bfd9bf689f3b318340f37aee4572a71f162bd8db51da55ba4fa/charset_normalizer-3.5.1-cp37-abi3-win_arm64.whl", hash = "sha256:87e4f41d375c0b9be2fb5251aee4b8a689169e134535aed81bf085c3b647451e", size = 287318, upload-time = "2026-08-15T08:20:15.551Z" }, + { url = "https://files.pythonhosted.org/packages/cc/61/d01fc49b8dea277640b55a9e15960dbca9fdc8c9fde18e572d39c59f4019/charset_normalizer-3.5.1-py3-none-any.whl", hash = "sha256:6df0ec430f9a831772c23ca5a224cba36517a58a84bb32c32bb59a9fa67c47f6", size = 68658, upload-time = "2026-08-15T08:20:43.306Z" }, +] + +[[package]] +name = "click" +version = "8.5.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/c7/0e/7fa0ef50764b67090eca4114772a2abf8b6148198475e54c660b97caeee6/click-8.5.0.tar.gz", hash = "sha256:ba0d2089de75ea0310e2dde03160e6ca10009947fb95a182f9b54021bb272e34", size = 382235, upload-time = "2026-08-26T13:33:14.56Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/58/50/6c0d534c5f134586a8e1ba4e330569e32f057e33372ae556463212fb4cd3/click-8.5.0-py3-none-any.whl", hash = "sha256:255bc9599cf7748b4b1a446ccc735421bd08a2ae529a8b88597d3de5664ee360", size = 125251, upload-time = "2026-08-26T13:33:12.928Z" }, +] + +[[package]] +name = "colorama" +version = "0.4.6" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/d8/53/6f443c9a4a8358a93a6792e2acffb9d9d5cb0a5cfd8802644b7b1c9a02e4/colorama-0.4.6.tar.gz", hash = "sha256:08695f5cb7ed6e0531a20572697297273c47b8cae5a63ffc6d6ed5c201be6e44", size = 27697, upload-time = "2022-10-25T02:36:22.414Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/d1/d6/3965ed04c63042e047cb6a3e6ed1a63a35087b6a609aa3a15ed8ac56c221/colorama-0.4.6-py2.py3-none-any.whl", hash = "sha256:4f1d9991f5acc0ca119f9d443620b77f9d6b33703e51011c16baf57afb285fc6", size = 25335, upload-time = "2022-10-25T02:36:20.889Z" }, +] + +[[package]] +name = "fastapi" +version = "0.141.1" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "annotated-doc" }, + { name = "pydantic" }, + { name = "starlette" }, + { name = "typing-extensions" }, + { name = "typing-inspection" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/8a/02/91e3416a8fdd715abb903a952a6bec7cdd8d14eed55d415fc8595524c319/fastapi-0.141.1.tar.gz", hash = "sha256:e8822fc40db1e1858054d7a949a888695bc9bdce70139178e33bd2871a453ca1", size = 425799, upload-time = "2026-07-29T17:18:05.568Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/cb/03/10388a42375ee7e4ac9b94eb2c5c569c8b5795e377e701c9ac3ad63de890/fastapi-0.141.1-py3-none-any.whl", hash = "sha256:bfb91aa2d334c61cb35ba9a116fc123b3d3df31640b801cf57a7a78ec3f603b3", size = 131954, upload-time = "2026-07-29T17:18:04.364Z" }, +] + +[[package]] +name = "h11" +version = "0.16.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/01/ee/02a2c011bdab74c6fb3c75474d40b3052059d95df7e73351460c8588d963/h11-0.16.0.tar.gz", hash = "sha256:4e35b956cf45792e4caa5885e69fba00bdbc6ffafbfa020300e549b208ee5ff1", size = 101250, upload-time = "2025-04-24T03:35:25.427Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/04/4b/29cac41a4d98d144bf5f6d33995617b185d14b22401f75ca86f384e87ff1/h11-0.16.0-py3-none-any.whl", hash = "sha256:63cf8bbe7522de3bf65932fda1d9c2772064ffb3dae62d55932da54b31cb6c86", size = 37515, upload-time = "2025-04-24T03:35:24.344Z" }, +] + +[[package]] +name = "httpcore2" +version = "2.13.1" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "h11" }, + { name = "truststore" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/cb/f3/1db7aa2bc2524062192bb0e0323969492d1883152a232fe36eea65f4e35c/httpcore2-2.13.1.tar.gz", hash = "sha256:e0aa977abe17e69a3b820a24542a6fa88702676d83880b8d194dcd18408e5103", size = 68071, upload-time = "2026-09-23T07:47:22.372Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/09/ba/a4568248771ce81957bfb7cc600264a40fbcda092391ee1c415c50be4bea/httpcore2-2.13.1-py3-none-any.whl", hash = "sha256:e1e05d4f25f7d7d496bfb96748f6f4b67657b03da069b3a68c36069f3db73d0a", size = 83423, upload-time = "2026-09-23T07:47:19.365Z" }, +] + +[[package]] +name = "httpx2" +version = "2.13.1" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "anyio", marker = "sys_platform != 'emscripten'" }, + { name = "httpcore2", marker = "sys_platform != 'emscripten'" }, + { name = "httpx2-jsfetch", marker = "sys_platform == 'emscripten'" }, + { name = "idna" }, + { name = "truststore", marker = "sys_platform != 'emscripten'" }, + { name = "typing-extensions", marker = "python_full_version < '3.13'" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/d5/44/474bef2a0e9d90f1715d32cb98b0738695ca17ba324095fb2497ed7fbd59/httpx2-2.13.1.tar.gz", hash = "sha256:e48744a19e3af5ee48313d0ce5fe941d5422fae5705ea922a4aabf94d7800dfa", size = 100405, upload-time = "2026-09-23T07:47:23.052Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/d8/9c/6fe8931fd9f381042a9e4c7d5a7b4cbf7016b252bec0c99a49fce42c3326/httpx2-2.13.1-py3-none-any.whl", hash = "sha256:6dff50fabc270ee5fd25d845d0b078ed20564579744d6d962850975996d2f9a4", size = 95597, upload-time = "2026-09-23T07:47:20.995Z" }, +] + +[[package]] +name = "httpx2-jsfetch" +version = "1.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/cd/c4/0e5636363151a2a1795e0a77617168b9ca438e1748ec05fc9b5687f93d64/httpx2_jsfetch-1.0.tar.gz", hash = "sha256:70a0e3eabfef7cce5ad9c629f7d01ca05e418f586646f4ddf14782e4c1454c60", size = 6872, upload-time = "2026-08-07T00:13:07.492Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/9b/43/832f631d32e4f1211caa2ba368317739fe71f0b8530e4c9d15dc454bac2a/httpx2_jsfetch-1.0-py3-none-any.whl", hash = "sha256:cb916b707601e69a07721aabc8f3f6659be3a6893bc1ff5c6f9e02241df2da32", size = 6382, upload-time = "2026-08-07T00:13:06.567Z" }, +] + +[[package]] +name = "idna" +version = "3.20" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/f5/08/8eea9d4b8302028f3abb2c0813953f7aec26d33b7a8960ed760e65ff29fa/idna-3.20.tar.gz", hash = "sha256:a7db850025b95ded1eae8a46181a1a6c56c92c96f0e2b005d9ff8dc0210cab44", size = 216463, upload-time = "2026-09-17T14:11:04.752Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/58/a2/bb081bab032533a855d44de1d56f8e8426114ff1ba5d1f07a438a0a654f8/idna-3.20-py3-none-any.whl", hash = "sha256:ab7ae7122974553370f0bdb919e1a960b2cd1bc1ef0276416d896db81c14582c", size = 69583, upload-time = "2026-09-17T14:11:03.168Z" }, +] + +[[package]] +name = "pycparser" +version = "3.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/1b/7d/92392ff7815c21062bea51aa7b87d45576f649f16458d78b7cf94b9ab2e6/pycparser-3.0.tar.gz", hash = "sha256:600f49d217304a5902ac3c37e1281c9fe94e4d0489de643a9504c5cdfdfc6b29", size = 103492, upload-time = "2026-01-21T14:26:51.89Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/0c/c3/44f3fbbfa403ea2a7c779186dc20772604442dde72947e7d01069cbe98e3/pycparser-3.0-py3-none-any.whl", hash = "sha256:b727414169a36b7d524c1c3e31839a521725078d7b2ff038656844266160a992", size = 48172, upload-time = "2026-01-21T14:26:50.693Z" }, +] + +[[package]] +name = "pydantic" +version = "2.13.5" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "annotated-types" }, + { name = "pydantic-core" }, + { name = "typing-extensions" }, + { name = "typing-inspection" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/53/ef/fc4f868f4e2cee79f863883abffceff107875f569b848507319842d2a681/pydantic-2.13.5.tar.gz", hash = "sha256:51a9c5f7b2f8e636f04c6cada605d9b6a3bf1348fdf945a3d8869b19bba0ee08", size = 845750, upload-time = "2026-08-28T14:04:00.916Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/eb/47/c95ffc2009878c7aac0c5e08528022dcb885933252a88b5f170058014464/pydantic-2.13.5-py3-none-any.whl", hash = "sha256:346a034f080da3755d8e9cb5e00e8b07de1d39e4f6e2c87d8ab7cafa0b269a73", size = 472589, upload-time = "2026-08-28T14:03:59.136Z" }, +] + +[[package]] +name = "pydantic-core" +version = "2.46.5" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "typing-extensions" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/af/f9/8a06bea35ef8daf588f707784c973a7046e0034c8d8cfb08828eeffb8b75/pydantic_core-2.46.5.tar.gz", hash = "sha256:10416c15b8839ecc4ef4d0885da76da6fd0f67333a0eb8aff6d93c4b8f2910fc", size = 472262, upload-time = "2026-08-28T10:01:31.677Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/82/3f/76358795aa7a8c6d4f36e2cb828ad1c90ee118e1393a9281664f5aade9d4/pydantic_core-2.46.5-cp312-cp312-macosx_10_12_x86_64.whl", hash = "sha256:b9fe6fb92520e3fd61f2e49000b6911b188824f089b75973ea06d6267f0b476d", size = 2076516, upload-time = "2026-08-28T09:58:21.576Z" }, + { url = "https://files.pythonhosted.org/packages/db/50/26b091836076ce4cb2fac264186936acc069e0595772cfd02a563bc4761a/pydantic_core-2.46.5-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:a39ac25a9a2fa4072efdb429833c4a4c8009a51ff9eea3eeae131713cd27991e", size = 1922874, upload-time = "2026-08-28T09:58:23.766Z" }, + { url = "https://files.pythonhosted.org/packages/09/f0/2a8ce3849e299d44e2d2c196b6082643a3235565a735cb51db7a6261f614/pydantic_core-2.46.5-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:4fdc8b93a41521988916eeaa271173fcca7fa0803d62f87675aac8dcec1c8e29", size = 1951772, upload-time = "2026-08-28T09:58:25.435Z" }, + { url = "https://files.pythonhosted.org/packages/87/46/ac0dc8bdd9e6048183a14eb127764e7ad9240021c17513074a4711b0e31e/pydantic_core-2.46.5-cp312-cp312-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:b98134087d9de723658d17a42c7d0da8d6e2ef08015dee7dc93889047315f5e4", size = 2031832, upload-time = "2026-08-28T09:58:27.102Z" }, + { url = "https://files.pythonhosted.org/packages/c4/c2/339de5bef7be36301a2231eaa52e62163742c2281f11b5f4892bc79785cd/pydantic_core-2.46.5-cp312-cp312-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:e652ab17569c94bff5475520f907b7148b8c24036a8ebbe5cf7cf7493d28579a", size = 2208645, upload-time = "2026-08-28T09:58:28.948Z" }, + { url = "https://files.pythonhosted.org/packages/7b/a0/9ff22b797724262da14427abaed4dd1d864a139693fc5e7809114376a716/pydantic_core-2.46.5-cp312-cp312-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:d925f3d9afd05a8c0fb3a1031463a8d59ebe5e2afad297e29c78be19e13b4e62", size = 2265935, upload-time = "2026-08-28T09:58:30.625Z" }, + { url = "https://files.pythonhosted.org/packages/c0/a4/eb9409ec0736e50aa70a412f16c204ed149516846912f7e6724d4c73ee53/pydantic_core-2.46.5-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:0fc5be0abd4a407e200d844b404e33639a554e7bd0d448e7b9ae181be4789ac2", size = 2066284, upload-time = "2026-08-28T09:58:32.289Z" }, + { url = "https://files.pythonhosted.org/packages/c0/02/7f6156ffc926857f1c37c07d9a388682865a81830ab6a1b637082c25e399/pydantic_core-2.46.5-cp312-cp312-manylinux_2_31_riscv64.whl", hash = "sha256:816ff0a6550ffc06c098ccd2e0698600f9aa7da192a79eaa6f9af504a35db869", size = 2105889, upload-time = "2026-08-28T09:58:33.986Z" }, + { url = "https://files.pythonhosted.org/packages/92/b1/e781d357ebe09fc929f995700f1b3503e8897f1cece183ecb1300d4d67e9/pydantic_core-2.46.5-cp312-cp312-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:c7ea57fc63aa7da93a1bd2d644e6577befae10c52c4e36377635eea1056a74f5", size = 2158006, upload-time = "2026-08-28T09:58:35.647Z" }, + { url = "https://files.pythonhosted.org/packages/70/0a/644597d84ab400e50609c192120b85c9681c22d3a20461b9060a79be0a7a/pydantic_core-2.46.5-cp312-cp312-musllinux_1_1_aarch64.whl", hash = "sha256:efd62a42486f1bda5d24cb4f63d15a3c7768375fe83d36f9417b4ad7a2fb20b3", size = 2158408, upload-time = "2026-08-28T09:58:37.38Z" }, + { url = "https://files.pythonhosted.org/packages/1e/ee/ca3b7b3a4b3769ffe9ce9432a7c9be755de9593a46d3b0d54d0409323e44/pydantic_core-2.46.5-cp312-cp312-musllinux_1_1_armv7l.whl", hash = "sha256:2bc9419666990c06d7397831f2126a1ecc3594aaa3ff7de5bf2d066802f4e07b", size = 2309609, upload-time = "2026-08-28T09:58:39.22Z" }, + { url = "https://files.pythonhosted.org/packages/ce/52/39fa1f451486019524ca685020390e7ca351832fd874530ba30c8628e6dc/pydantic_core-2.46.5-cp312-cp312-musllinux_1_1_x86_64.whl", hash = "sha256:18a09e1e1011b462f2e32774f25859ef1223d5c2b0546a633cf56654710721e0", size = 2342618, upload-time = "2026-08-28T09:58:40.89Z" }, + { url = "https://files.pythonhosted.org/packages/81/5e/468fc630568c61dcef3cd47ad32ffbeed9af643f49208d1ea86ab4f890c4/pydantic_core-2.46.5-cp312-cp312-win32.whl", hash = "sha256:5cb482e9e84c851f4e623fe4acc1ced89168cf1fe18f7089db4548c8f5bbb65b", size = 1939475, upload-time = "2026-08-28T09:58:42.591Z" }, + { url = "https://files.pythonhosted.org/packages/cf/c9/4c19f41b84cf6b622a72fbeed7665b25d47a187d68d47d0d430c07f23268/pydantic_core-2.46.5-cp312-cp312-win_amd64.whl", hash = "sha256:5e81740c09e310f5aa5cbd3e434a01c154d4bef93241c7877b39f211d2b78ba8", size = 2043140, upload-time = "2026-08-28T09:58:44.272Z" }, + { url = "https://files.pythonhosted.org/packages/af/dd/0c1a050299147c746e5256db16d645ab5efd4f78c59937d581a0524e74a2/pydantic_core-2.46.5-cp312-cp312-win_arm64.whl", hash = "sha256:f7b0ec93a2893de856652154d73b7ba622f26fa97726487dcac373de5f4c6084", size = 1997729, upload-time = "2026-08-28T09:58:46.13Z" }, + { url = "https://files.pythonhosted.org/packages/f5/37/5abe39a8372a61d3dc3c1338fc504281c01b32fdb3169cd7187153b56d3e/pydantic_core-2.46.5-cp313-cp313-macosx_10_12_x86_64.whl", hash = "sha256:b7ca9034437b6022f941f4857459562ee00a560b97e7cce8a0ec5a74fc6766e0", size = 2075885, upload-time = "2026-08-28T09:58:47.856Z" }, + { url = "https://files.pythonhosted.org/packages/21/43/6323b1f8b217780454c61304bcd2b38ae4762f50754414124603ccc90bb2/pydantic_core-2.46.5-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:f332f0e72a5a0400141f830744e141bf9f97917878dbe968669e8a7fefea78ff", size = 1922768, upload-time = "2026-08-28T09:58:49.58Z" }, + { url = "https://files.pythonhosted.org/packages/0f/a3/c05ca796e1197618a774b01e596aeedfefc2f7d8c01ae3054e910b120e8a/pydantic_core-2.46.5-cp313-cp313-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:193375f3548919d3f0b60936ca113ada3e38f264f91b9b8e0508efaad57be931", size = 1951241, upload-time = "2026-08-28T09:58:51.511Z" }, + { url = "https://files.pythonhosted.org/packages/68/32/33bc39ac705c52cffc908e8389f9754fdb208aea5c69cceddf4eb3ce99af/pydantic_core-2.46.5-cp313-cp313-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:79bdfa52f843137045b2d081cc05c120ba6665d29b7559c2c47690906f39279f", size = 2031975, upload-time = "2026-08-28T09:58:53.166Z" }, + { url = "https://files.pythonhosted.org/packages/b0/70/2333e885c0f6a67bc105c5916965dac9b57f2718ee20d81d1a06a4ebdc13/pydantic_core-2.46.5-cp313-cp313-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:24922243639cbdac66c75fcb6fd6495a9cb52b213d62f9a0d16f0310b1ff8038", size = 2208542, upload-time = "2026-08-28T09:58:55.017Z" }, + { url = "https://files.pythonhosted.org/packages/f7/ea/296debfb4264207bbda5936133892e027c0a58875ad53ebd512fba8ec3a2/pydantic_core-2.46.5-cp313-cp313-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:c76fe65e607be28c7fd4d56fc3c42b1583aa058ce3408b7ad0fd540171d31f9f", size = 2264692, upload-time = "2026-08-28T09:58:56.767Z" }, + { url = "https://files.pythonhosted.org/packages/d3/f2/9e4de77a6271e07a76d2d58b11c091a979c191ed2939bf80067568b369d2/pydantic_core-2.46.5-cp313-cp313-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:6f7b393a8b3da82f5c1fc0751e6d01ac6c55b93c18226a60bdfba4a724efafd1", size = 2066633, upload-time = "2026-08-28T09:58:58.531Z" }, + { url = "https://files.pythonhosted.org/packages/8d/db/f9e9d0c97445987b2084823d5c240de88087338f04fc2cfaa2df186b8049/pydantic_core-2.46.5-cp313-cp313-manylinux_2_31_riscv64.whl", hash = "sha256:7ac031912d54f3d83ef3b3eb98dfabc1608802e2202263d25957eeed40b94761", size = 2105235, upload-time = "2026-08-28T09:59:00.421Z" }, + { url = "https://files.pythonhosted.org/packages/07/c5/79169b047b3b2c3e99e04bc76372af9637e0bf6db638274fa927df96369e/pydantic_core-2.46.5-cp313-cp313-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:837b396ca3d7b74091ca623f6cbd8351bd42d670a79c2683e79fb089f06a2de5", size = 2157367, upload-time = "2026-08-28T09:59:02.442Z" }, + { url = "https://files.pythonhosted.org/packages/26/b5/ba6057afb7c291bd449f51b867f95aef2072941c4ce4e5c31d6ffd132d3b/pydantic_core-2.46.5-cp313-cp313-musllinux_1_1_aarch64.whl", hash = "sha256:5ee239d575f80b08eca11f6e20f90c4c695de7825c67eefe6091fbf20dda648e", size = 2158420, upload-time = "2026-08-28T09:59:04.2Z" }, + { url = "https://files.pythonhosted.org/packages/6e/28/2057abecaafdc22912afa819603a51f0a62d40643b7c4871c51721fea9be/pydantic_core-2.46.5-cp313-cp313-musllinux_1_1_armv7l.whl", hash = "sha256:e80675d75ae2cd14372cb65cad5400d9347a3d3f6c13000183f22dfd027283ed", size = 2309588, upload-time = "2026-08-28T09:59:06.048Z" }, + { url = "https://files.pythonhosted.org/packages/71/9d/881156dc404e27479c4246128d73538464cab4a239bec61995e227644c30/pydantic_core-2.46.5-cp313-cp313-musllinux_1_1_x86_64.whl", hash = "sha256:9c4b71f10dd532fb7a5cbc8f58707779e64f03a258c2bf8bfbaecfcd9970b519", size = 2341866, upload-time = "2026-08-28T09:59:08.539Z" }, + { url = "https://files.pythonhosted.org/packages/5a/38/d66f443a259f84d13babdceae568e572b0ed26da17ca5d0a649ebb110a67/pydantic_core-2.46.5-cp313-cp313-win32.whl", hash = "sha256:97bf8de4d541598c94a59344eeb988a94c08ff76b5723c41f6567ec18c7892ea", size = 1938580, upload-time = "2026-08-28T09:59:10.402Z" }, + { url = "https://files.pythonhosted.org/packages/2c/1e/1d5371213f4cc9a7ed70c0bfcc7911de22311ee99a662a56077d7292d2ac/pydantic_core-2.46.5-cp313-cp313-win_amd64.whl", hash = "sha256:15f4a94963c95accac15b7b657bb177d3ad82bb90b0d0526d9a9b85079925db5", size = 2041980, upload-time = "2026-08-28T09:59:12.396Z" }, + { url = "https://files.pythonhosted.org/packages/5a/48/4222d90b1c67568bace4dec6dca6271449c66de3595d72b6d098f5fde597/pydantic_core-2.46.5-cp313-cp313-win_arm64.whl", hash = "sha256:d22a945598fb91236b4dd793a6e42e4f3dd7740bb5aace5ebd7d4c08d13bb575", size = 1997213, upload-time = "2026-08-28T09:59:14.245Z" }, + { url = "https://files.pythonhosted.org/packages/8e/8a/14596f2a8367da50cf7cbac48169ee5d9c8e11d486a3b527082384630c72/pydantic_core-2.46.5-cp314-cp314-macosx_10_12_x86_64.whl", hash = "sha256:c1c43ad4339643d70ebb8124e1305a7dab423001eff58bb41a0f731adbc98355", size = 2074081, upload-time = "2026-08-28T09:59:16.141Z" }, + { url = "https://files.pythonhosted.org/packages/ae/d5/d8a4eb6d6c7f66b91dd37c576d76e9e60fba900caf5372c17bcf949febc2/pydantic_core-2.46.5-cp314-cp314-macosx_11_0_arm64.whl", hash = "sha256:1a353f84de772f423b5ffb11d7ae352fbbef0f446f3c0b0af0f8236d7233606e", size = 1920497, upload-time = "2026-08-28T09:59:18.065Z" }, + { url = "https://files.pythonhosted.org/packages/8e/26/092079428f86e927e030b2c0ced87df69dbb1c875cdeaa67bf42ea2be746/pydantic_core-2.46.5-cp314-cp314-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:5086029a57366b8cf81b130a43908738095c270c21a8d7f0e8bdfdb89718e2f3", size = 1952130, upload-time = "2026-08-28T09:59:20.476Z" }, + { url = "https://files.pythonhosted.org/packages/08/c3/8ec0e290a9ebaebd64047bf5fda94be835c6b1551b02437e4b76778fbcd7/pydantic_core-2.46.5-cp314-cp314-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:46c25dda9d092a06c08db76ffe0a197107904d0dfac653f7d5306bbcd6d6119c", size = 2026371, upload-time = "2026-08-28T09:59:22.227Z" }, + { url = "https://files.pythonhosted.org/packages/01/72/4fd20ad520fb8da0157f95b27a7eb05a72790ef08138e7701ac972c342ea/pydantic_core-2.46.5-cp314-cp314-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:37ea7b83c935e5b0d68c9449b82651accf78a10828b2c02b2f2d9e9496446c21", size = 2202822, upload-time = "2026-08-28T09:59:24.277Z" }, + { url = "https://files.pythonhosted.org/packages/31/b0/d16e0771206b29314f0d52198b720be21e8a99ab2bf11e3bc0d7c9cebdff/pydantic_core-2.46.5-cp314-cp314-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:e64e88d5585bea9ce95861079de72006c7fa6d3df4e3a3b65ba31eb979c15c9f", size = 2262756, upload-time = "2026-08-28T09:59:26.608Z" }, + { url = "https://files.pythonhosted.org/packages/2c/9b/59634b7ac631c63b2a37760eb6943af3e29573d6b59a4abc5e7f019d4cee/pydantic_core-2.46.5-cp314-cp314-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:54d510bac3ee52247af28ed4bb18a1e799f040ac60fd2bf5ccd4c92f1fbe786f", size = 2068352, upload-time = "2026-08-28T09:59:29.044Z" }, + { url = "https://files.pythonhosted.org/packages/08/7c/570abb1ad2155348dc754ea91be22e5aaa18eb6d69a6068f7c6f2679a6ed/pydantic_core-2.46.5-cp314-cp314-manylinux_2_31_riscv64.whl", hash = "sha256:a2a5e1d0ff29adddc9f6d6821a66302e4493f8ca898b715b6b1182c2c201ea0a", size = 2104777, upload-time = "2026-08-28T09:59:30.95Z" }, + { url = "https://files.pythonhosted.org/packages/8e/25/5bf74adc65a1ac5b7be3f6cb0bcb5433615c1598a801c19d830d84c98ded/pydantic_core-2.46.5-cp314-cp314-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:03b9666e41e35d8909852ba191a0607520f81b74eaf12ccf8737005dbb313821", size = 2156312, upload-time = "2026-08-28T09:59:32.604Z" }, + { url = "https://files.pythonhosted.org/packages/90/6a/2ef38830675e050121040618135564ed56b860b45433b02d9b4ebece46f3/pydantic_core-2.46.5-cp314-cp314-musllinux_1_1_aarch64.whl", hash = "sha256:a91c17edf6eea2402cb5457b4c89e99bc5ed1004aa34c4adf1d4258c1a5c22c2", size = 2150067, upload-time = "2026-08-28T09:59:34.453Z" }, + { url = "https://files.pythonhosted.org/packages/90/ef/a7dbb03a14a64c2a4621f989c615ed9a892535a6cad938fc27079f919d80/pydantic_core-2.46.5-cp314-cp314-musllinux_1_1_armv7l.whl", hash = "sha256:b49924c73a235e969511bf2aabdff3beebf9820931f646c80274d5d780010c47", size = 2304516, upload-time = "2026-08-28T09:59:36.194Z" }, + { url = "https://files.pythonhosted.org/packages/68/f8/6bb4c4b80e8a6fde1904c64a51c62a1d04fcdfa3ea521a66b2ddefa1d885/pydantic_core-2.46.5-cp314-cp314-musllinux_1_1_x86_64.whl", hash = "sha256:2cbd9a5eff05e51c447c34dfa4632145b26b09120cf04bd0c871e44c1a5e1c9a", size = 2335223, upload-time = "2026-08-28T09:59:37.931Z" }, + { url = "https://files.pythonhosted.org/packages/2a/80/f46b8c681195190b2c1f1c7c0a81abce60663e987613e09ef64d433dd96b/pydantic_core-2.46.5-cp314-cp314-win32.whl", hash = "sha256:2d5d76654becf5efd62c9e51c3756c67b49498b0c9a40884934c40807adbd074", size = 1934827, upload-time = "2026-08-28T09:59:39.836Z" }, + { url = "https://files.pythonhosted.org/packages/f7/3c/60674207246bc0a4009d2391b7c7251c7159f279c8d2ab8aae8ef46f3dee/pydantic_core-2.46.5-cp314-cp314-win_amd64.whl", hash = "sha256:fa10ef4112775900e7a0661068635eb67b2ab824fbde764de6e0e21982a93db0", size = 2042648, upload-time = "2026-08-28T09:59:41.792Z" }, + { url = "https://files.pythonhosted.org/packages/69/0c/117c562c7c1babdf44576b72a5e496906506c93690387ecfbca7c729ae2e/pydantic_core-2.46.5-cp314-cp314-win_arm64.whl", hash = "sha256:045ab3b6d308439e32b81cc173bba5b9018bc6ed896afd0c65b3b009b1699af5", size = 1989652, upload-time = "2026-08-28T09:59:43.702Z" }, + { url = "https://files.pythonhosted.org/packages/e8/66/9336ae58f9eb68c41d121894e52c4c89eccb07eb8f602a04ee9c3f37736a/pydantic_core-2.46.5-cp314-cp314t-macosx_10_12_x86_64.whl", hash = "sha256:8816f3d218beb4b787de5c9759c259b8fa61f9dec42dc7811f320a33771778b7", size = 2065829, upload-time = "2026-08-28T09:59:45.364Z" }, + { url = "https://files.pythonhosted.org/packages/c5/02/bc19b47a96c2d3109760711acf22369e56bd7e405ca52f7ade164d2ead57/pydantic_core-2.46.5-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:bce57638e08ac148e5778cce7feb968307a727d66f8e2274a543d0cf0c9ad6a3", size = 1905716, upload-time = "2026-08-28T09:59:47.18Z" }, + { url = "https://files.pythonhosted.org/packages/52/a4/70b47c0509923dd98ccfed04fb3e32ea3849c82a0ff2205bb41009b43c00/pydantic_core-2.46.5-cp314-cp314t-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:976e1128455aa595ea04c79ccfedff1aaeab96ee013fcc916bed120c4f0ad94f", size = 1934216, upload-time = "2026-08-28T09:59:49.241Z" }, + { url = "https://files.pythonhosted.org/packages/52/ab/aa03b65f7bb198585edf806b906c3223ecf1795543e39e23aec4cce27ad2/pydantic_core-2.46.5-cp314-cp314t-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:e7b891faeedeafba41b2983e5001a81b6a915b69544c7e7570d1989ce1c36ac7", size = 2010635, upload-time = "2026-08-28T09:59:51.692Z" }, + { url = "https://files.pythonhosted.org/packages/3c/8b/0da06343f30b84ec549aafd309c6456223d5dc8bd36af504c573faad561d/pydantic_core-2.46.5-cp314-cp314t-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:5f194189415698233dd1114a093a9b56e61e2c57e11b469be3b0506f46f0771c", size = 2209369, upload-time = "2026-08-28T09:59:53.582Z" }, + { url = "https://files.pythonhosted.org/packages/d6/5b/844c4defaa34a3df66eb9257087d121d70c201298b96abdf9f492fc2f1bf/pydantic_core-2.46.5-cp314-cp314t-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:82a36973cf8a2ef5406f4fe2edbf8ed0c99629535d959e0b100c76a32535a111", size = 2253238, upload-time = "2026-08-28T09:59:55.484Z" }, + { url = "https://files.pythonhosted.org/packages/f4/64/a4e536cb16d7f61a7fd3120b46c577fc7fa7325992f69c4f52bc786d77d8/pydantic_core-2.46.5-cp314-cp314t-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:cdbb78909f52b981d3b2d56b97328d71eb0b974c36bd77c920123a7ebb192829", size = 2065740, upload-time = "2026-08-28T09:59:58.038Z" }, + { url = "https://files.pythonhosted.org/packages/5f/75/aaa38c6bc2d085f6605b34eabdc6a8a4e0b2e61fc9c8e6e52b28e97b3125/pydantic_core-2.46.5-cp314-cp314t-manylinux_2_31_riscv64.whl", hash = "sha256:52e24eacdb536cade636aa90fb851835222becff8484b7001fdc78cb0290f2aa", size = 2087425, upload-time = "2026-08-28T09:59:59.898Z" }, + { url = "https://files.pythonhosted.org/packages/55/ae/fcab4cfc39aba3689e1d20c8b5250ad280957022c09af2ed9cd585602a5e/pydantic_core-2.46.5-cp314-cp314t-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:37ae34309d7bd8c0d61ab839668058f2a7962ea1fc51d105d2db228fe0618034", size = 2139306, upload-time = "2026-08-28T10:00:03.057Z" }, + { url = "https://files.pythonhosted.org/packages/2d/f4/f1d03a4bc9d9acbc62f4d742b8a319af52f71885079868b2ff8e48a651ee/pydantic_core-2.46.5-cp314-cp314t-musllinux_1_1_aarch64.whl", hash = "sha256:0cdbada856a1c69a7624a64d3d9aefe79300bd6ef827b43a4f265010b9b55184", size = 2144589, upload-time = "2026-08-28T10:00:05.645Z" }, + { url = "https://files.pythonhosted.org/packages/83/f3/7a53bb1356de514a4cd295f25b6ac39237895620c0462d2592b76c16e114/pydantic_core-2.46.5-cp314-cp314t-musllinux_1_1_armv7l.whl", hash = "sha256:545f26c504b27c3758439a5e6d9349931f0a04f855668d5fe323c89e82300a38", size = 2288882, upload-time = "2026-08-28T10:00:07.931Z" }, + { url = "https://files.pythonhosted.org/packages/cd/94/5a81583660c175c59d49ffb09f4b3a44debeaf86a19fca664ae1cdd9ee32/pydantic_core-2.46.5-cp314-cp314t-musllinux_1_1_x86_64.whl", hash = "sha256:ff218293c9c806138dca139765e3b067621be52bcd93cdc14c7711be7ddc90a9", size = 2335210, upload-time = "2026-08-28T10:00:10.177Z" }, + { url = "https://files.pythonhosted.org/packages/5a/9f/5d685c2693b972d1a59c998586e8823712b66603aeff47ee60a4bdaafd37/pydantic_core-2.46.5-cp314-cp314t-win32.whl", hash = "sha256:97cf3eb53a8cccacf9d46686a0926186c9bfb5574f2ed66d3639d5fe117cd3a9", size = 1921180, upload-time = "2026-08-28T10:00:12.35Z" }, + { url = "https://files.pythonhosted.org/packages/70/12/5c94ee16d65a37a15f9e869f5e6256df111154491173801a4c5e800ab548/pydantic_core-2.46.5-cp314-cp314t-win_amd64.whl", hash = "sha256:d2f9fc07a8042a8f95925b35c4f04f469707c981fc33245b6ca187cf5d2dd290", size = 2020515, upload-time = "2026-08-28T10:00:14.774Z" }, + { url = "https://files.pythonhosted.org/packages/63/19/67830dda664e6bdf9285ee2e40f355d0d7d6b92aa0c42e8d217bb8d33d36/pydantic_core-2.46.5-cp314-cp314t-win_arm64.whl", hash = "sha256:acf8a67ba51f4ca9ddbd0e6b3000a65ac51ab734661778b3e7ba64d99a710f2f", size = 1989276, upload-time = "2026-08-28T10:00:16.984Z" }, + { url = "https://files.pythonhosted.org/packages/df/dd/053c2e4303f791f3b8f8a14ab0b22008e8eb21d868c0c90b4f9be705b76a/pydantic_core-2.46.5-graalpy312-graalpy250_312_native-macosx_10_12_x86_64.whl", hash = "sha256:013d6f3483d81e02e7c328831808f336c8596ee33b4bd4026b9ffb1e960b8942", size = 2062540, upload-time = "2026-08-28T10:01:00.318Z" }, + { url = "https://files.pythonhosted.org/packages/d7/dd/a18df751a5e37dd51bfad7f68e766999125bebe68c9e1d10a493ad01bd63/pydantic_core-2.46.5-graalpy312-graalpy250_312_native-macosx_11_0_arm64.whl", hash = "sha256:e9c134bb666dd54b778b9fc0d2b50cbb7f979b9e3716f26a88c9ab3b6fc1dd0f", size = 1902040, upload-time = "2026-08-28T10:01:02.529Z" }, + { url = "https://files.pythonhosted.org/packages/b7/13/01d40f9d07ce8a779fd6e0bd8ad4fba91309500dd67b869e2e219d261a6d/pydantic_core-2.46.5-graalpy312-graalpy250_312_native-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:347ec774390c87326a2e4929d58d3f7e8763a104d5d35f4cd595a4c952366433", size = 1967479, upload-time = "2026-08-28T10:01:05.004Z" }, + { url = "https://files.pythonhosted.org/packages/fa/04/c81d4841331c2178b6fb09ae225425e110ed72d990c9fe556c4ec03d1013/pydantic_core-2.46.5-graalpy312-graalpy250_312_native-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:8e24d8f05fa2d28513d94e877e9c75ad66175376209b3977f916e240e623193c", size = 2111034, upload-time = "2026-08-28T10:01:07.345Z" }, +] + +[[package]] +name = "pynacl" +version = "1.6.2" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "cffi", marker = "platform_python_implementation != 'PyPy'" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/d9/9a/4019b524b03a13438637b11538c82781a5eda427394380381af8f04f467a/pynacl-1.6.2.tar.gz", hash = "sha256:018494d6d696ae03c7e656e5e74cdfd8ea1326962cc401bcf018f1ed8436811c", size = 3511692, upload-time = "2026-01-01T17:48:10.851Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/4b/79/0e3c34dc3c4671f67d251c07aa8eb100916f250ee470df230b0ab89551b4/pynacl-1.6.2-cp314-cp314t-macosx_10_10_universal2.whl", hash = "sha256:622d7b07cc5c02c666795792931b50c91f3ce3c2649762efb1ef0d5684c81594", size = 390064, upload-time = "2026-01-01T17:31:57.264Z" }, + { url = "https://files.pythonhosted.org/packages/eb/1c/23a26e931736e13b16483795c8a6b2f641bf6a3d5238c22b070a5112722c/pynacl-1.6.2-cp314-cp314t-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:d071c6a9a4c94d79eb665db4ce5cedc537faf74f2355e4d502591d850d3913c0", size = 809370, upload-time = "2026-01-01T17:31:59.198Z" }, + { url = "https://files.pythonhosted.org/packages/87/74/8d4b718f8a22aea9e8dcc8b95deb76d4aae380e2f5b570cc70b5fd0a852d/pynacl-1.6.2-cp314-cp314t-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:fe9847ca47d287af41e82be1dd5e23023d3c31a951da134121ab02e42ac218c9", size = 1408304, upload-time = "2026-01-01T17:32:01.162Z" }, + { url = "https://files.pythonhosted.org/packages/fd/73/be4fdd3a6a87fe8a4553380c2b47fbd1f7f58292eb820902f5c8ac7de7b0/pynacl-1.6.2-cp314-cp314t-manylinux_2_26_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:04316d1fc625d860b6c162fff704eb8426b1a8bcd3abacea11142cbd99a6b574", size = 844871, upload-time = "2026-01-01T17:32:02.824Z" }, + { url = "https://files.pythonhosted.org/packages/55/ad/6efc57ab75ee4422e96b5f2697d51bbcf6cdcc091e66310df91fbdc144a8/pynacl-1.6.2-cp314-cp314t-manylinux_2_26_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:44081faff368d6c5553ccf55322ef2819abb40e25afaec7e740f159f74813634", size = 1446356, upload-time = "2026-01-01T17:32:04.452Z" }, + { url = "https://files.pythonhosted.org/packages/78/b7/928ee9c4779caa0a915844311ab9fb5f99585621c5d6e4574538a17dca07/pynacl-1.6.2-cp314-cp314t-manylinux_2_34_aarch64.whl", hash = "sha256:a9f9932d8d2811ce1a8ffa79dcbdf3970e7355b5c8eb0c1a881a57e7f7d96e88", size = 826814, upload-time = "2026-01-01T17:32:06.078Z" }, + { url = "https://files.pythonhosted.org/packages/f7/a9/1bdba746a2be20f8809fee75c10e3159d75864ef69c6b0dd168fc60e485d/pynacl-1.6.2-cp314-cp314t-manylinux_2_34_x86_64.whl", hash = "sha256:bc4a36b28dd72fb4845e5d8f9760610588a96d5a51f01d84d8c6ff9849968c14", size = 1411742, upload-time = "2026-01-01T17:32:07.651Z" }, + { url = "https://files.pythonhosted.org/packages/f3/2f/5e7ea8d85f9f3ea5b6b87db1d8388daa3587eed181bdeb0306816fdbbe79/pynacl-1.6.2-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:3bffb6d0f6becacb6526f8f42adfb5efb26337056ee0831fb9a7044d1a964444", size = 801714, upload-time = "2026-01-01T17:32:09.558Z" }, + { url = "https://files.pythonhosted.org/packages/06/ea/43fe2f7eab5f200e40fb10d305bf6f87ea31b3bbc83443eac37cd34a9e1e/pynacl-1.6.2-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:2fef529ef3ee487ad8113d287a593fa26f48ee3620d92ecc6f1d09ea38e0709b", size = 1372257, upload-time = "2026-01-01T17:32:11.026Z" }, + { url = "https://files.pythonhosted.org/packages/4d/54/c9ea116412788629b1347e415f72195c25eb2f3809b2d3e7b25f5c79f13a/pynacl-1.6.2-cp314-cp314t-win32.whl", hash = "sha256:a84bf1c20339d06dc0c85d9aea9637a24f718f375d861b2668b2f9f96fa51145", size = 231319, upload-time = "2026-01-01T17:32:12.46Z" }, + { url = "https://files.pythonhosted.org/packages/ce/04/64e9d76646abac2dccf904fccba352a86e7d172647557f35b9fe2a5ee4a1/pynacl-1.6.2-cp314-cp314t-win_amd64.whl", hash = "sha256:320ef68a41c87547c91a8b58903c9caa641ab01e8512ce291085b5fe2fcb7590", size = 244044, upload-time = "2026-01-01T17:32:13.781Z" }, + { url = "https://files.pythonhosted.org/packages/33/33/7873dc161c6a06f43cda13dec67b6fe152cb2f982581151956fa5e5cdb47/pynacl-1.6.2-cp314-cp314t-win_arm64.whl", hash = "sha256:d29bfe37e20e015a7d8b23cfc8bd6aa7909c92a1b8f41ee416bbb3e79ef182b2", size = 188740, upload-time = "2026-01-01T17:32:15.083Z" }, + { url = "https://files.pythonhosted.org/packages/be/7b/4845bbf88e94586ec47a432da4e9107e3fc3ce37eb412b1398630a37f7dd/pynacl-1.6.2-cp38-abi3-macosx_10_10_universal2.whl", hash = "sha256:c949ea47e4206af7c8f604b8278093b674f7c79ed0d4719cc836902bf4517465", size = 388458, upload-time = "2026-01-01T17:32:16.829Z" }, + { url = "https://files.pythonhosted.org/packages/1e/b4/e927e0653ba63b02a4ca5b4d852a8d1d678afbf69b3dbf9c4d0785ac905c/pynacl-1.6.2-cp38-abi3-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:8845c0631c0be43abdd865511c41eab235e0be69c81dc66a50911594198679b0", size = 800020, upload-time = "2026-01-01T17:32:18.34Z" }, + { url = "https://files.pythonhosted.org/packages/7f/81/d60984052df5c97b1d24365bc1e30024379b42c4edcd79d2436b1b9806f2/pynacl-1.6.2-cp38-abi3-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:22de65bb9010a725b0dac248f353bb072969c94fa8d6b1f34b87d7953cf7bbe4", size = 1399174, upload-time = "2026-01-01T17:32:20.239Z" }, + { url = "https://files.pythonhosted.org/packages/68/f7/322f2f9915c4ef27d140101dd0ed26b479f7e6f5f183590fd32dfc48c4d3/pynacl-1.6.2-cp38-abi3-manylinux_2_26_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:46065496ab748469cdd999246d17e301b2c24ae2fdf739132e580a0e94c94a87", size = 835085, upload-time = "2026-01-01T17:32:22.24Z" }, + { url = "https://files.pythonhosted.org/packages/3e/d0/f301f83ac8dbe53442c5a43f6a39016f94f754d7a9815a875b65e218a307/pynacl-1.6.2-cp38-abi3-manylinux_2_26_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:8a66d6fb6ae7661c58995f9c6435bda2b1e68b54b598a6a10247bfcdadac996c", size = 1437614, upload-time = "2026-01-01T17:32:23.766Z" }, + { url = "https://files.pythonhosted.org/packages/c4/58/fc6e649762b029315325ace1a8c6be66125e42f67416d3dbd47b69563d61/pynacl-1.6.2-cp38-abi3-manylinux_2_34_aarch64.whl", hash = "sha256:26bfcd00dcf2cf160f122186af731ae30ab120c18e8375684ec2670dccd28130", size = 818251, upload-time = "2026-01-01T17:32:25.69Z" }, + { url = "https://files.pythonhosted.org/packages/c9/a8/b917096b1accc9acd878819a49d3d84875731a41eb665f6ebc826b1af99e/pynacl-1.6.2-cp38-abi3-manylinux_2_34_x86_64.whl", hash = "sha256:c8a231e36ec2cab018c4ad4358c386e36eede0319a0c41fed24f840b1dac59f6", size = 1402859, upload-time = "2026-01-01T17:32:27.215Z" }, + { url = "https://files.pythonhosted.org/packages/85/42/fe60b5f4473e12c72f977548e4028156f4d340b884c635ec6b063fe7e9a5/pynacl-1.6.2-cp38-abi3-musllinux_1_2_aarch64.whl", hash = "sha256:68be3a09455743ff9505491220b64440ced8973fe930f270c8e07ccfa25b1f9e", size = 791926, upload-time = "2026-01-01T17:32:29.314Z" }, + { url = "https://files.pythonhosted.org/packages/fa/f9/e40e318c604259301cc091a2a63f237d9e7b424c4851cafaea4ea7c4834e/pynacl-1.6.2-cp38-abi3-musllinux_1_2_x86_64.whl", hash = "sha256:8b097553b380236d51ed11356c953bf8ce36a29a3e596e934ecabe76c985a577", size = 1363101, upload-time = "2026-01-01T17:32:31.263Z" }, + { url = "https://files.pythonhosted.org/packages/48/47/e761c254f410c023a469284a9bc210933e18588ca87706ae93002c05114c/pynacl-1.6.2-cp38-abi3-win32.whl", hash = "sha256:5811c72b473b2f38f7e2a3dc4f8642e3a3e9b5e7317266e4ced1fba85cae41aa", size = 227421, upload-time = "2026-01-01T17:32:33.076Z" }, + { url = "https://files.pythonhosted.org/packages/41/ad/334600e8cacc7d86587fe5f565480fde569dfb487389c8e1be56ac21d8ac/pynacl-1.6.2-cp38-abi3-win_amd64.whl", hash = "sha256:62985f233210dee6548c223301b6c25440852e13d59a8b81490203c3227c5ba0", size = 239754, upload-time = "2026-01-01T17:32:34.557Z" }, + { url = "https://files.pythonhosted.org/packages/29/7d/5945b5af29534641820d3bd7b00962abbbdfee84ec7e19f0d5b3175f9a31/pynacl-1.6.2-cp38-abi3-win_arm64.whl", hash = "sha256:834a43af110f743a754448463e8fd61259cd4ab5bbedcf70f9dabad1d28a394c", size = 184801, upload-time = "2026-01-01T17:32:36.309Z" }, +] + +[[package]] +name = "requests" +version = "2.34.2" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "certifi" }, + { name = "charset-normalizer" }, + { name = "idna" }, + { name = "urllib3" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/ac/c3/e2a2b89f2d3e2179abd6d00ebd70bff6273f37fb3e0cc209f48b39d00cbf/requests-2.34.2.tar.gz", hash = "sha256:f288924cae4e29463698d6d60bc6a4da69c89185ad1e0bcc4104f584e960b9ed", size = 142856, upload-time = "2026-05-14T19:25:27.735Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/a0/f4/c67b0b3f1b9245e8d266f0f112c500d50e5b4e83cb6f3b71b6528104182a/requests-2.34.2-py3-none-any.whl", hash = "sha256:2a0d60c172f83ac6ab31e4554906c0f3b3588d37b5cb939b1c061f4907e278e0", size = 73075, upload-time = "2026-05-14T19:25:26.443Z" }, +] + +[[package]] +name = "ruff" +version = "0.16.9" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/96/bf/c935ca98e73fe8ce65b87ef08a280c0c1e85295d569228c15e87d8fdfaf1/ruff-0.16.9.tar.gz", hash = "sha256:12b625c6cfba78d285d9f48eda5f053374f1e53cb10ef17342a383750db99161", size = 4948764, upload-time = "2026-09-24T20:37:49.416Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/0d/26/df51322b52ee1ada7eff2d071ea09d11d5c2d1dcc9f02594f5785c4d1635/ruff-0.16.9-py3-none-linux_armv6l.whl", hash = "sha256:95e6f022090368ab3b824c36276839c53b2adf1a3f4c09fefc33dfc400f6da96", size = 10082922, upload-time = "2026-09-24T20:37:13.045Z" }, + { url = "https://files.pythonhosted.org/packages/a5/27/7bf51f5a7aa375e9f339280a303aab44ca75dc1525f1cdc5991761685b0f/ruff-0.16.9-py3-none-macosx_10_12_x86_64.whl", hash = "sha256:a5f27be168556594a86d2f415db0cf43f5291917849318f873c7e2791f7a8c67", size = 10236360, upload-time = "2026-09-24T20:37:16.049Z" }, + { url = "https://files.pythonhosted.org/packages/b6/63/09659283f92f02dff45809da194a70da2f688d87c55d8875c4fae3536072/ruff-0.16.9-py3-none-macosx_11_0_arm64.whl", hash = "sha256:1632eb1d6197f33bd00b1acbc5b71009e89a8895c158e2d2b03a834fac964ab6", size = 9892940, upload-time = "2026-09-24T20:37:17.957Z" }, + { url = "https://files.pythonhosted.org/packages/24/58/98de1b72ec172f5f8f1731236fe21585b3998bf7dfe9fcc44ae9ba626012/ruff-0.16.9-py3-none-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:b3f951b14d865d5952c89d40a5ca07e87abe24fa5453299878411e127748fb1c", size = 10032114, upload-time = "2026-09-24T20:37:19.942Z" }, + { url = "https://files.pythonhosted.org/packages/c8/7d/f1e17c54ab59d4bad1dce8ee3e22a7a1d0ef4745240decacdcf3832b5bb2/ruff-0.16.9-py3-none-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:447fc07e1573afff7cb02803462b12b6c8ece7cf10e2cd78565fa6d7a1c0bf8d", size = 9910227, upload-time = "2026-09-24T20:37:21.872Z" }, + { url = "https://files.pythonhosted.org/packages/34/19/436f647a65075bbd3bab2668b3bdaa5120559b294694018cdcefabbbf30b/ruff-0.16.9-py3-none-manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:8a3e039a6a40ed976c491722b60e0ae4a4aa1a86057f540ee7a37a5d19ae9120", size = 10547484, upload-time = "2026-09-24T20:37:24.229Z" }, + { url = "https://files.pythonhosted.org/packages/03/59/38430a6bc2f6d8095447ac39625cf8b6e9344a47e6c26225c8ba1bff3ffb/ruff-0.16.9-py3-none-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:4684dded7db60aa57cb118fa158630f5feade4af5782903b6053484bdf9bd129", size = 11412367, upload-time = "2026-09-24T20:37:26.307Z" }, + { url = "https://files.pythonhosted.org/packages/c8/bd/bbb6d7fc7f208c8b8c50dd5dc8206e4cfdb1e7a8fb852606a3adf370c880/ruff-0.16.9-py3-none-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:d29c934357e45642fda2f34c0b1f4025b4a6c01e15e4bf0016879d60078a142c", size = 10869787, upload-time = "2026-09-24T20:37:28.35Z" }, + { url = "https://files.pythonhosted.org/packages/bc/b8/9c543074918061abbefc3bd139bee22de35abedb00dde0f2d27288838962/ruff-0.16.9-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:a21713e629d3e5bdb2f5c2def1cc7f04f47fa8e1a7eb0571b4a28e1da64bc728", size = 10406494, upload-time = "2026-09-24T20:37:30.624Z" }, + { url = "https://files.pythonhosted.org/packages/35/7a/5a8851bd146e7ccf8fd4b003f6c75c11f8fbdb0e60673b45097986b6bf41/ruff-0.16.9-py3-none-manylinux_2_31_riscv64.whl", hash = "sha256:7baa24ef5fc8e77aa93879e1d3f43754a01ae488e869f1ae30cf431afd4d2452", size = 10590083, upload-time = "2026-09-24T20:37:32.439Z" }, + { url = "https://files.pythonhosted.org/packages/87/f0/4c3467188f23f806960b46fa76575a7cd0514c9ba90562650b71efc96980/ruff-0.16.9-py3-none-musllinux_1_2_aarch64.whl", hash = "sha256:a41aac6230aadfaa133bdfa1614488531ffa3e0837567ae04c0da2058a9c0f9e", size = 10119151, upload-time = "2026-09-24T20:37:34.581Z" }, + { url = "https://files.pythonhosted.org/packages/15/34/5a4def5adea572ce6aea0bb64f21f928ee317b80e0db747d7979b01d7261/ruff-0.16.9-py3-none-musllinux_1_2_armv7l.whl", hash = "sha256:c2529fb5896d49115b0e9aa8f887490b34bbe76baf879ec2264ac59406869ce7", size = 9911544, upload-time = "2026-09-24T20:37:36.796Z" }, + { url = "https://files.pythonhosted.org/packages/62/5d/d15ebea7499eef9373318c0ee6ca127832927c6529731f6d48e18dce7ca9/ruff-0.16.9-py3-none-musllinux_1_2_i686.whl", hash = "sha256:41e3870277694177429b56406d65dfbdb2c2802c52b715edaf6a0b829c69d4ee", size = 10269884, upload-time = "2026-09-24T20:37:38.857Z" }, + { url = "https://files.pythonhosted.org/packages/d1/56/c5d3cd119ded7a3c7aba0e961b69cb3df701c91662c98ad694467d060ce1/ruff-0.16.9-py3-none-musllinux_1_2_x86_64.whl", hash = "sha256:8adbe4e58af167f767d7b2ba5e83c42e878350796cf78c2f5e14ab9903a92588", size = 10749366, upload-time = "2026-09-24T20:37:41.042Z" }, + { url = "https://files.pythonhosted.org/packages/ac/fe/734ec7527029ac757ecf821f143c9f3fcf69149c21f53a044a899b430f5a/ruff-0.16.9-py3-none-win32.whl", hash = "sha256:0e1dbc2073624dee6618d41d0098690a7244654af746704b64759e12b6b6b385", size = 10152355, upload-time = "2026-09-24T20:37:43.025Z" }, + { url = "https://files.pythonhosted.org/packages/14/21/26e4643629b3ebb44f0a06f9c9a53058d63d989415f63a9a3c28e2ee7f22/ruff-0.16.9-py3-none-win_amd64.whl", hash = "sha256:6bd40fec8cd4c8a3d4dd589bd8ad4e6320c13c29234159bfd959a40d529d597b", size = 10592965, upload-time = "2026-09-24T20:37:44.944Z" }, + { url = "https://files.pythonhosted.org/packages/51/60/5fb1a39dbb5ae314d5f59bc7348a63c1d5c20f3cd83914c4b5cb0be31d2d/ruff-0.16.9-py3-none-win_arm64.whl", hash = "sha256:ed1a252039200f57a59eebc063b54beabea67bfbaaca0eeaa7f54b5fbcda2284", size = 10458649, upload-time = "2026-09-24T20:37:46.882Z" }, +] + +[[package]] +name = "session-ops" +version = "0.1.0" +source = { editable = "." } +dependencies = [ + { name = "babel" }, + { name = "colorama" }, + { name = "fastapi" }, + { name = "pynacl" }, + { name = "requests" }, + { name = "uvicorn" }, +] + +[package.dev-dependencies] +dev = [ + { name = "httpx2" }, + { name = "ruff" }, +] + +[package.metadata] +requires-dist = [ + { name = "babel", specifier = "==2.17.0" }, + { name = "colorama", specifier = ">=0.4.6" }, + { name = "fastapi", specifier = ">=0.141.1" }, + { name = "pynacl", specifier = ">=1.6.2" }, + { name = "requests", specifier = ">=2.32.3" }, + { name = "uvicorn", specifier = ">=0.52.4" }, +] + +[package.metadata.requires-dev] +dev = [ + { name = "httpx2", specifier = ">=2.12.0" }, + { name = "ruff", specifier = "==0.16.9" }, +] + +[[package]] +name = "starlette" +version = "1.7.0" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "anyio" }, + { name = "typing-extensions", marker = "python_full_version < '3.13'" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/7b/2b/3850dc6bf7ef71b088962eba31dafc6cffd2f96e577ebb0bb316df96da3e/starlette-1.7.0.tar.gz", hash = "sha256:c79f74ea63cff761804fbbfb182f1e0b440c2d07b164d24700c5a1bab5d6ff5d", size = 2736246, upload-time = "2026-09-23T07:30:26.35Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/4e/d6/1ec1b290f9e0fb067899b61e1d37a30c923068bad260b216dbe37a7d2967/starlette-1.7.0-py3-none-any.whl", hash = "sha256:67f8e99895493dd2911a03f11314af6ceebeae4e704bb9f43dfc6a9db151c93e", size = 78980, upload-time = "2026-09-23T07:30:24.567Z" }, +] + +[[package]] +name = "truststore" +version = "0.10.4" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/53/a3/1585216310e344e8102c22482f6060c7a6ea0322b63e026372e6dcefcfd6/truststore-0.10.4.tar.gz", hash = "sha256:9d91bd436463ad5e4ee4aba766628dd6cd7010cf3e2461756b3303710eebc301", size = 26169, upload-time = "2025-08-12T18:49:02.73Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/19/97/56608b2249fe206a67cd573bc93cd9896e1efb9e98bce9c163bcdc704b88/truststore-0.10.4-py3-none-any.whl", hash = "sha256:adaeaecf1cbb5f4de3b1959b42d41f6fab57b2b1666adb59e89cb0b53361d981", size = 18660, upload-time = "2025-08-12T18:49:01.46Z" }, +] + +[[package]] +name = "typing-extensions" +version = "4.16.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/f6/cc/6253133b5bb138fc3306cebfbda2c520f545d36b5be2c7255cc528bb45d6/typing_extensions-4.16.0.tar.gz", hash = "sha256:dc983d19a509c94dba722ee6abd33940f7c05a89e243c47e907eb4db6f1a43e5", size = 113555, upload-time = "2026-07-02T08:40:05.92Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/49/d3/b8441a820a491ddfc024b0b0cf0393375b75ea13866d9c66727e54c2fc80/typing_extensions-4.16.0-py3-none-any.whl", hash = "sha256:481caa481374e813c1b176ada14e97f1f67a4539ce9cfeb3f350d78d6370c2e8", size = 45571, upload-time = "2026-07-02T08:40:04.659Z" }, +] + +[[package]] +name = "typing-inspection" +version = "0.4.4" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "typing-extensions" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/a3/26/b09b8010994eccc3c09092e6b34058f36a460eea2d4c3e8b910c695975a0/typing_inspection-0.4.4.tar.gz", hash = "sha256:547274fa6b0a561ccf549cc9524b999a578e737d015d8709d021f9d0d13bea47", size = 76928, upload-time = "2026-08-12T12:37:25.997Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/67/81/4add07e5172b7ac40d8ed5ff580409a7801a4fe26d529bdd915401dabfbe/typing_inspection-0.4.4-py3-none-any.whl", hash = "sha256:65b8397ba37ccbce054456aaccddfc91e6e3083c92824df348d96ca832f3f147", size = 14750, upload-time = "2026-08-12T12:37:24.648Z" }, +] + +[[package]] +name = "urllib3" +version = "2.8.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/e3/05/b17359e1cefb4f909b5e40b1b90a496d987258916dbbf88e842c729f510e/urllib3-2.8.0.tar.gz", hash = "sha256:63bf2ead4c879426ebf22ef2a781eeb4aa3b4ae798a0435506f8687fd5bb9b63", size = 458972, upload-time = "2026-09-15T19:29:36.253Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/92/9d/c4e665119135114480843e7ab388fa94d8480650450e6f8e26b70d323a4c/urllib3-2.8.0-py3-none-any.whl", hash = "sha256:0cf3cae568d36aa9576b28dfb35f11328f1cb974ca7647d9475ebb86c75ac6e3", size = 135717, upload-time = "2026-09-15T19:29:34.577Z" }, +] + +[[package]] +name = "uvicorn" +version = "0.53.0" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "click" }, + { name = "h11" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/5d/ad/04bbb797c84fc1f26cb171f7394716f4865ffb8d8c5e1eef42565c2dfa6b/uvicorn-0.53.0.tar.gz", hash = "sha256:a9356f0cb89b3b8621529c5d5eebd69bfe154f4c3f68b4cf2de47e45fa855c2e", size = 110881, upload-time = "2026-09-14T07:44:23.815Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/76/18/0eea75741ee812e9f598b687619ce2454f6c3a1c5cd21ea990ec6bd26f45/uvicorn-0.53.0-py3-none-any.whl", hash = "sha256:e8dca71ec86dce5f04e333f0d56cdedf942446e6643b9cea1af0d6d3a02cb03e", size = 87081, upload-time = "2026-09-14T07:44:22.179Z" }, +] diff --git a/zendesk_triage/requirements-dev.txt b/zendesk_triage/requirements-dev.txt deleted file mode 100644 index 46dec50..0000000 --- a/zendesk_triage/requirements-dev.txt +++ /dev/null @@ -1,9 +0,0 @@ -# Test-only dependencies. The suite is stdlib unittest, so this covers what only the -# tests import: -# -# python -m unittest discover # from zendesk_triage/ -# -# test_relay.py drives the FastAPI app through starlette's TestClient, which ships no -# HTTP client of its own. starlette still accepts httpx but warns that it is -# deprecated for this, so the pin is the one it asks for. -httpx2==2.12.0 diff --git a/zendesk_triage/requirements.txt b/zendesk_triage/requirements.txt deleted file mode 100644 index ee12b8e..0000000 --- a/zendesk_triage/requirements.txt +++ /dev/null @@ -1,7 +0,0 @@ -# What the deployment runs. Test-only dependencies live in requirements-dev.txt. -requests==2.32.3 - -# relay.py only: the endpoint Zendesk posts note webhooks to. Its signatures are -# HMAC-SHA256, which is stdlib hmac, so nothing here verifies them. -fastapi==0.141.1 -uvicorn==0.52.4 From 331b16c7ccf1e6ad6c7913026af69fa542cc7e6e Mon Sep 17 00:00:00 2001 From: Audric Ackermann Date: Fri, 25 Sep 2026 13:09:15 +1000 Subject: [PATCH 014/101] refactor: one HTTP transport under every job, and Crowdin on its SDK shared/http.py replaces shared/retry.py. Session is a requests.Session with a urllib3 Retry mounted, so retries happen below the caller, which sees only the final answer: 429 and every 5xx retried for all methods, 1 s doubling to 30 s, Retry-After in either form capped at a minute, GitHub's x-ratelimit-reset when Retry-After is absent, and an unusable header falling back to backoff. It adds a default timeout, an optional token bucket, and a per-call attempts= for optional data, carried to the adapter in a ContextVar because requests has no way to pass one down. Its tests run against a local server, so the retries tested are urllib3's own. The Crowdin scripts move to crowdin-api-client through crowdin/sdk.py. The SDK never retries a 429 and retries 5xx on a fixed 100 ms, so its session is swapped for per-thread http.Sessions drawing on one 30/s bucket, and its own loop is off. Its decoder also turns every ISO timestamp into a datetime, which json.dump cannot write: the download's project and glossary files would have failed on the first real payload. Responses stay plain JSON. The recordings now carry timestamps as Crowdin sends them, and approve_strings.py gains a golden. Every golden was regenerated by the previous code from the same recordings, and the new code matches it byte for byte; only request keys the SDK spells differently (offset=0, a query moved into params) were edited. A live digest dry run matches its golden apart from PR ages. Caller tests that queued a retry's worth of failures now queue one, and assert the budget they ask for instead. --- pyproject.toml | 1 + src/session_ops/crowdin/approve_strings.py | 55 +- .../download_translations_from_crowdin.py | 118 ++-- .../crowdin/report_multiple_translations.py | 72 +-- src/session_ops/crowdin/sdk.py | 93 ++++ src/session_ops/github_prs/digest.py | 10 +- src/session_ops/monitor/alert.py | 4 +- src/session_ops/monitor/silence.py | 5 +- src/session_ops/shared/discord.py | 3 +- src/session_ops/shared/http.py | 163 ++++++ src/session_ops/shared/retry.py | 84 --- src/session_ops/shared/testing.py | 12 +- src/session_ops/zendesk/note_reply.py | 6 +- src/session_ops/zendesk/resolve_reviews.py | 10 +- src/session_ops/zendesk/triage.py | 30 +- tests/crowdin/test_approve_golden.py | 43 ++ tests/crowdin/test_crowdin_goldens.py | 13 +- .../test_report_multiple_translations.py | 34 +- tests/crowdin/test_sdk.py | 82 +++ tests/goldens/approve/approve.txt | 13 + tests/goldens/approve/list.txt | 10 + tests/goldens/approve/responses.json | 516 ++++++++++++++++++ tests/goldens/download/output.json | 4 +- tests/goldens/download/responses.json | 18 +- tests/goldens/report/responses.json | 4 +- tests/shared/test_discord.py | 12 +- tests/shared/test_http.py | 248 +++++++++ tests/shared/test_retry.py | 168 ------ tests/zendesk/test_note_reply.py | 7 +- tests/zendesk/test_resolve_reviews.py | 6 +- tests/zendesk/test_triage.py | 32 +- uv.lock | 102 ++++ 32 files changed, 1452 insertions(+), 526 deletions(-) create mode 100644 src/session_ops/crowdin/sdk.py create mode 100644 src/session_ops/shared/http.py delete mode 100644 src/session_ops/shared/retry.py create mode 100644 tests/crowdin/test_approve_golden.py create mode 100644 tests/crowdin/test_sdk.py create mode 100644 tests/goldens/approve/approve.txt create mode 100644 tests/goldens/approve/list.txt create mode 100644 tests/goldens/approve/responses.json create mode 100644 tests/shared/test_http.py delete mode 100644 tests/shared/test_retry.py diff --git a/pyproject.toml b/pyproject.toml index 3c02d13..25b05fc 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -8,6 +8,7 @@ dependencies = [ # CLDR data: a new babel can rename a language in the generated language lists. "babel==2.17.0", "colorama>=0.4.6", + "crowdin-api-client>=1.29.0", "fastapi>=0.141.1", "pynacl>=1.6.2", "requests>=2.32.3", diff --git a/src/session_ops/crowdin/approve_strings.py b/src/session_ops/crowdin/approve_strings.py index f7a932e..b2306b4 100644 --- a/src/session_ops/crowdin/approve_strings.py +++ b/src/session_ops/crowdin/approve_strings.py @@ -32,11 +32,9 @@ import os import subprocess import sys -import time -import requests +from session_ops.crowdin import sdk -API = "https://api.crowdin.com/api/v2" DEFAULT_PROJECT = "618696" # libsecret attributes identifying the token in the keyring; must match the @@ -72,6 +70,10 @@ def get_token(): ) +def crowdin_client(token, project_id): + return sdk.client(token, project_id, timeout=30) + + def describe_user(u): """Human-readable submitter for a translation's user object (may be None).""" if not u: @@ -129,13 +131,7 @@ def main(): identifiers = args.identifiers pid = args.project_id - s = requests.Session() - s.headers.update({"Authorization": f"Bearer {token}"}) - - def get(path, **params): - r = s.get(f"{API}{path}", params=params, timeout=30) - r.raise_for_status() - return r.json() + client = crowdin_client(token, pid) # 1) Resolve identifiers -> string IDs string_ids = {} @@ -143,8 +139,8 @@ def get(path, **params): found = None offset = 0 while True: - data = get(f"/projects/{pid}/strings", filter=ident, - scope="identifier", limit=500, offset=offset)["data"] + data = client.source_strings.list_strings( + filter=ident, scope="identifier", limit=500, offset=offset)["data"] for row in data: if row["data"]["identifier"] == ident: found = row["data"]["id"] @@ -158,7 +154,7 @@ def get(path, **params): print(f" string '{ident}' -> id {found}") # 2) Target languages - proj = get(f"/projects/{pid}")["data"] + proj = client.projects.get_project()["data"] target_langs = proj["targetLanguageIds"] print(f"\n{len(target_langs)} target languages: {', '.join(target_langs)}\n") @@ -169,15 +165,15 @@ def get(path, **params): approved, skipped, already, errors = 0, 0, 0, 0 for ident, sid in string_ids.items(): for lang in target_langs: - raw = get(f"/projects/{pid}/translations", - stringId=sid, languageId=lang, limit=500)["data"] + raw = client.string_translations.list_string_translations( + stringId=sid, languageId=lang, limit=500)["data"] trans = [t["data"] for t in raw] if args.list: # Show only unapproved translations (i.e. what a run would approve), # honouring the --by-user filter if one was given. - approvals = get(f"/projects/{pid}/approvals", - stringId=sid, languageId=lang, limit=500)["data"] + approvals = client.string_translations.list_translation_approvals( + stringId=sid, languageId=lang, limit=500)["data"] approved_tids = {a["data"]["translationId"] for a in approvals} pending = [t for t in trans if t["id"] not in approved_tids @@ -209,18 +205,19 @@ def get(path, **params): approved += 1 continue - r = s.post(f"{API}/projects/{pid}/approvals", - json={"translationId": tid}, timeout=30) - if r.status_code == 201: - print(f" [ ok ] {ident} / {lang}: approved {info}") - approved += 1 - elif r.status_code == 400 and "already" in r.text.lower(): - print(f" [ -- ] {ident} / {lang}: already approved {info}") - already += 1 - else: - print(f" [ERR ] {ident} / {lang}: {r.status_code} {r.text}") - errors += 1 - time.sleep(0.1) # gentle on rate limits + try: + client.string_translations.add_approval(translationId=tid) + except sdk.APIException as exc: + body = sdk.response_text(exc) + if exc.http_status == 400 and "already" in body.lower(): + print(f" [ -- ] {ident} / {lang}: already approved {info}") + already += 1 + else: + print(f" [ERR ] {ident} / {lang}: {exc.http_status} {body}") + errors += 1 + continue + print(f" [ ok ] {ident} / {lang}: approved {info}") + approved += 1 if args.list: return diff --git a/src/session_ops/crowdin/download_translations_from_crowdin.py b/src/session_ops/crowdin/download_translations_from_crowdin.py index 9b7c6bf..cd71621 100644 --- a/src/session_ops/crowdin/download_translations_from_crowdin.py +++ b/src/session_ops/crowdin/download_translations_from_crowdin.py @@ -2,14 +2,13 @@ import json import sys import argparse -import requests -import time from concurrent.futures import ThreadPoolExecutor, as_completed -from threading import Lock, Semaphore +from threading import Lock from colorama import Fore, Style, init -CROWDIN_API_BASE_URL = "https://api.crowdin.com/api/v2" +from session_ops.crowdin import sdk +from session_ops.shared import http def parse_args(argv=None): @@ -26,10 +25,10 @@ def parse_args(argv=None): return parser.parse_args(argv) -def configure(args): +def configure(args, session=None): global CROWDIN_API_TOKEN, CROWDIN_PROJECT_ID, CROWDIN_GLOSSARY_ID, CROWDIN_CONCEPT_ID global DOWNLOAD_DIRECTORY, SKIP_UNTRANSLATED_STRINGS, FORCE_ALLOW_UNAPPROVED, VERBOSE - global MAX_WORKERS, api_semaphore + global MAX_WORKERS, CLIENT, FILES CROWDIN_API_TOKEN = args.api_token CROWDIN_PROJECT_ID = args.project_id CROWDIN_GLOSSARY_ID = args.glossary_id @@ -38,69 +37,32 @@ def configure(args): SKIP_UNTRANSLATED_STRINGS = args.skip_untranslated_strings FORCE_ALLOW_UNAPPROVED = args.force_allow_unapproved VERBOSE = args.verbose - # Crowdin API limit is 20 simultaneous requests per account + # Crowdin API limit is 20 simultaneous requests per account, and each worker + # has one in flight at a time. MAX_WORKERS = min(args.max_workers, 20) - # Semaphore ensures we don't exceed the concurrent requests limit - api_semaphore = Semaphore(MAX_WORKERS) + CLIENT = sdk.client(CROWDIN_API_TOKEN, CROWDIN_PROJECT_ID, attempts=MAX_ATTEMPTS, + timeout=REQUEST_TIMEOUT_S, session=session) + # The export URLs are presigned: the Crowdin token must not travel with them. + FILES = session or http.Session(attempts=MAX_ATTEMPTS, timeout=REQUEST_TIMEOUT_S) REQUEST_TIMEOUT_S = 30 -MAX_RETRIES = 5 -INITIAL_RETRY_DELAY_S = 0.5 +MAX_ATTEMPTS = 5 progress_lock = Lock() completed_count = 0 total_count = 0 -def make_request_with_retry(method: str, url: str, **kwargs) -> requests.Response: - last_exception = None - - for attempt in range(MAX_RETRIES): - try: - with api_semaphore: - if method.upper() == 'GET': - response = requests.get( - url, timeout=REQUEST_TIMEOUT_S, **kwargs) - elif method.upper() == 'POST': - response = requests.post( - url, timeout=REQUEST_TIMEOUT_S, **kwargs) - else: - raise ValueError(f"Unsupported HTTP method: {method}") - - # Handle rate limiting - if response.status_code == 429: - retry_after = int(response.headers.get( - 'Retry-After', INITIAL_RETRY_DELAY_S * (2 ** attempt))) - if VERBOSE: - print(f"\n{Fore.YELLOW}⚠️ Rate limited, waiting { - retry_after}s before retry...{Style.RESET_ALL}") - time.sleep(retry_after) - continue - - return response - - except requests.exceptions.RequestException as e: - last_exception = e - delay = INITIAL_RETRY_DELAY_S * (2 ** attempt) - if VERBOSE: - print(f"\n{Fore.YELLOW}⚠️ Request failed, retrying in { - delay}s... ({e}){Style.RESET_ALL}") - time.sleep(delay) - - raise last_exception or Exception( - f"Request failed after {MAX_RETRIES} retries") - - -def check_error(response, context=""): - if response.status_code != 200: - error_msg = response.json().get('error', {}).get('message', 'Unknown error') - raise Exception( - f"{context}: {error_msg} (Code: {response.status_code})") +def crowdin_call(context, call, **kwargs): + try: + return call(**kwargs) + except sdk.APIException as exc: + raise Exception(f"{context}: {sdk.error_message(exc)} (Code: {exc.http_status})") from exc def download_file(url: str, output_path: str): - response = requests.get(url, stream=True, timeout=REQUEST_TIMEOUT_S) + response = FILES.get(url, stream=True) response.raise_for_status() with open(output_path, 'wb') as f: @@ -120,16 +82,10 @@ def export_and_download_language(language: dict, is_source: bool = False) -> str "exportApprovedOnly": False if is_source else (not FORCE_ALLOW_UNAPPROVED) } - export_response = make_request_with_retry( - 'POST', - f"{CROWDIN_API_BASE_URL}/projects/{CROWDIN_PROJECT_ID}/translations/exports", - headers={"Authorization": f"Bearer {CROWDIN_API_TOKEN}", - "Content-Type": "application/json"}, - data=json.dumps(export_payload) - ) - check_error(export_response, f"Export failed for {lang_locale}") + exported = crowdin_call(f"Export failed for {lang_locale}", + CLIENT.translations.export_project_translation, **export_payload) - download_url = export_response.json()['data']['url'] + download_url = exported['data']['url'] download_path = os.path.join(DOWNLOAD_DIRECTORY, f"{lang_locale}.xliff") download_file(download_url, download_path) @@ -145,27 +101,23 @@ def download(): global total_count, completed_count # Retrieve the list of languages print(f"{Fore.WHITE}⏳ Retrieving project details...{Style.RESET_ALL}", end='\r') - project_response = make_request_with_retry( - 'GET', - f"{CROWDIN_API_BASE_URL}/projects/{CROWDIN_PROJECT_ID}", - headers={"Authorization": f"Bearer {CROWDIN_API_TOKEN}"} - ) - check_error(project_response, "Failed to retrieve project details") - project_details = project_response.json()['data'] + project_response = crowdin_call("Failed to retrieve project details", + CLIENT.projects.get_project) + project_details = project_response['data'] source_language = project_details['sourceLanguage'] target_languages = project_details['targetLanguages'] num_languages = len(target_languages) print(f"\033[2K{Fore.GREEN}βœ… Project details retrieved, found {num_languages} translations{Style.RESET_ALL}") if VERBOSE: - print(f"{Fore.BLUE}Response: {json.dumps(project_response.json(), indent=2)}{Style.RESET_ALL}") + print(f"{Fore.BLUE}Response: {json.dumps(project_response, indent=2)}{Style.RESET_ALL}") if not os.path.exists(DOWNLOAD_DIRECTORY): os.makedirs(DOWNLOAD_DIRECTORY) project_info_file = os.path.join(DOWNLOAD_DIRECTORY, "_project_info.json") with open(project_info_file, 'w', encoding='utf-8') as file: - json.dump(project_response.json(), file, indent=2) + json.dump(project_response, file, indent=2) all_languages = [{'language': source_language, 'is_source': True}] for lang in sorted(target_languages, key=lambda x: x['locale']): @@ -206,26 +158,24 @@ def download(): # Download non-translatable terms (if requested) if CROWDIN_GLOSSARY_ID is not None and CROWDIN_CONCEPT_ID is not None: print(f"{Fore.WHITE}⏳ Retrieving non-translatable strings...{Style.RESET_ALL}", end='\r') - static_string_response = make_request_with_retry( - 'GET', - f"{CROWDIN_API_BASE_URL}/glossaries/{CROWDIN_GLOSSARY_ID}/terms?conceptId={CROWDIN_CONCEPT_ID}&limit=500", - headers={"Authorization": f"Bearer {CROWDIN_API_TOKEN}"} - ) - check_error(static_string_response, "Failed to retrieve non-translatable strings") + static_string_response = crowdin_call( + "Failed to retrieve non-translatable strings", CLIENT.glossaries.list_terms, + glossaryId=CROWDIN_GLOSSARY_ID, conceptId=CROWDIN_CONCEPT_ID, limit=500) if VERBOSE: - print(f"{Fore.BLUE}Response: {json.dumps(static_string_response.json(), indent=2)}{Style.RESET_ALL}") + print(f"{Fore.BLUE}Response: {json.dumps(static_string_response, indent=2)}{Style.RESET_ALL}") non_translatable_strings_file = os.path.join(DOWNLOAD_DIRECTORY, "_non_translatable_strings.json") with open(non_translatable_strings_file, 'w', encoding='utf-8') as file: - json.dump(static_string_response.json(), file, indent=2) + json.dump(static_string_response, file, indent=2) print(f"\033[2K{Fore.GREEN}βœ… Downloading non-translatable complete{Style.RESET_ALL}") -def main(argv=None): +def main(argv=None, session=None): + """`session` answers every request instead of the network, for tests.""" init(autoreset=True) - configure(parse_args(argv)) + configure(parse_args(argv), session) try: download() except KeyboardInterrupt: diff --git a/src/session_ops/crowdin/report_multiple_translations.py b/src/session_ops/crowdin/report_multiple_translations.py index eb56660..4a4cdf4 100644 --- a/src/session_ops/crowdin/report_multiple_translations.py +++ b/src/session_ops/crowdin/report_multiple_translations.py @@ -44,13 +44,10 @@ import os import subprocess import sys -import threading -import requests +from session_ops.crowdin import sdk +from session_ops.shared import discord, http -from session_ops.shared import discord, retry - -API = "https://api.crowdin.com/api/v2" DEFAULT_PROJECT = "618696" KEYRING_ATTRS = ["service", "crowdin", "key", "translation-api-token"] @@ -81,27 +78,11 @@ def get_token(cli_token): "or store it via secret-tool).") -def request_with_retry(session, method, url, **kw): - """A Crowdin API call. A non-retryable 4xx raises, so a caller can read the - body of what it asked for without checking the status first.""" - r = retry.request_with_retry(session, method, url, attempts=10, timeout=60, **kw) - r.raise_for_status() - return r - - -def paged(session, path, **params): - """Yield every item from a paginated Crowdin list endpoint.""" - offset = 0 - limit = 500 - while True: - r = request_with_retry(session, "GET", f"{API}{path}", - params={**params, "limit": limit, "offset": offset}) - data = r.json()["data"] - for row in data: - yield row["data"] - if len(data) < limit: - return - offset += limit +def crowdin_client(token, project_id): + """Ten attempts at sixty seconds: a locale is ~1,400 requests near Crowdin's rate + limit, so a 429 partway through is expected rather than exceptional. A 4xx + raises, so a caller can read the body of what it asked for without checking.""" + return sdk.client(token, project_id, attempts=10, timeout=60) def user_label(u): @@ -132,30 +113,18 @@ def clip(text, n): # full 80-locale scan is ~1366*80 requests β‰ˆ 40 min. To keep a daily job fast we # scan a ROTATING shard of locales each run (see pick_locales), covering them all # over a cycle. Within a locale we fan strings out across a bounded thread pool. -def scan_locale(session, pid, lang, string_ids, strings, editor_url, max_workers): +def scan_locale(client, lang, string_ids, strings, editor_url, max_workers): """Return the finding dicts for one locale (one slot per 2+-translation group).""" approved_here = {} # stringId -> set(translationId) that are approved - for a in paged(session, f"/projects/{pid}/approvals", languageId=lang): + for a in sdk.fetch_all(client.string_translations, "list_translation_approvals", + languageId=lang): approved_here.setdefault(a["stringId"], set()).add(a["translationId"]) print(f"[{lang}] scanning {len(string_ids)} strings " f"({len(approved_here)} with approvals) …", file=sys.stderr) - # requests.Session is not guaranteed thread-safe, so instead of sharing the - # caller's session across the pool, give each worker its own (lazily created, - # carrying the same auth header). - tls = threading.local() - - def worker_session(): - s = getattr(tls, "session", None) - if s is None: - s = requests.Session() - s.headers.update(session.headers) - tls.session = s - return s - def process(sid): - trans = list(paged(worker_session(), f"/projects/{pid}/translations", - stringId=sid, languageId=lang)) + trans = sdk.fetch_all(client.string_translations, "list_string_translations", + stringId=sid, languageId=lang) approved_tids = approved_here.get(sid, set()) by_cat = collections.defaultdict(list) for t in trans: @@ -206,13 +175,13 @@ def process(sid): return found -def scan(session, pid, locales, strings, editor_url, max_workers): +def scan(client, locales, strings, editor_url, max_workers): """Return a flat list of finding dicts, one per slot with 2+ translations.""" string_ids = list(strings.keys()) findings = [] for lang in locales: - findings.extend(scan_locale(session, pid, lang, string_ids, strings, - editor_url, max_workers)) + findings.extend(scan_locale(client, lang, string_ids, strings, editor_url, + max_workers)) findings.sort(key=lambda f: (f["locale"], f["identifier"] or "", str(f["pluralCategory"]))) return findings @@ -329,7 +298,7 @@ def pack_embeds(embeds): def post_to_discord(webhook_url, messages): # A fresh, unauthenticated session: the Crowdin Bearer token must never be # sent to Discord. - with requests.Session() as webhook_session: + with http.Session() as webhook_session: posted = discord.post_to_discord(webhook_session, webhook_url, messages) if posted == len(messages): return @@ -387,13 +356,12 @@ def main(): sys.exit("No Discord webhook (pass --webhook, set DISCORD_WEBHOOK_URL, or use --dry-run).") pid = args.project_id - session = requests.Session() - session.headers.update({"Authorization": f"Bearer {token}"}) + client = crowdin_client(token, pid) # Project details: identifier + per-language editor codes, used to build # editor URLs that actually point at the right locale. (A string's own webUrl # always targets the first target language, regardless of the locale.) - proj = request_with_retry(session, "GET", f"{API}/projects/{pid}").json()["data"] + proj = client.projects.get_project()["data"] project_slug = proj["identifier"] src_code = proj["sourceLanguage"].get("editorCode") or proj["sourceLanguage"]["id"] editor_code = {l["id"]: (l.get("editorCode") or l["id"]) for l in proj["targetLanguages"]} @@ -420,12 +388,12 @@ def editor_url(lang, sid): print(f"Loading strings for project {pid} …", file=sys.stderr) strings = {} - for s in paged(session, f"/projects/{pid}/strings"): + for s in sdk.fetch_all(client.source_strings, "list_strings"): strings[s["id"]] = {"identifier": s.get("identifier"), "text": s.get("text")} print(f" {len(strings)} strings; scanning {len(locales)} locale(s): " f"{', '.join(locales)}", file=sys.stderr) - findings = scan(session, pid, locales, strings, editor_url, args.max_workers) + findings = scan(client, locales, strings, editor_url, args.max_workers) print("\n" + "=" * 80, file=sys.stderr) print(f"Found {len(findings)} slot(s) with 2+ translations across " diff --git a/src/session_ops/crowdin/sdk.py b/src/session_ops/crowdin/sdk.py new file mode 100644 index 0000000..9a4608d --- /dev/null +++ b/src/session_ops/crowdin/sdk.py @@ -0,0 +1,93 @@ +"""crowdin-api-client on this repo's transport. + +The SDK never retries a 429 (its should_retry is false for 300-499) and retries 5xx +with a fixed 100 ms sleep, while Crowdin throttles near 40 requests a second. So the +SDK supplies the endpoints, parameter names and error types, and shared.http the +retries and pacing, by standing in for the session its requester talks to. The SDK's +own loop is switched off with max_retries=1. +""" +import json +import threading + +import crowdin_api.requester +from crowdin_api import CrowdinClient +from crowdin_api.exceptions import APIException + +from session_ops.shared import http + +# The SDK decodes every ISO timestamp in a response into a datetime, which json.dump +# cannot write and which does not sort against a missing one. Responses stay JSON. +crowdin_api.requester.loads = json.loads + +# Shared by all of a client's threads, and under the ~40/s Crowdin throttles at, so a +# fan-out spends its budget on work rather than on 429s. +REQUESTS_PER_SECOND = 30 + +__all__ = ["APIException", "client", "error_message", "fetch_all"] + + +class _PerThreadSession: + """One http.Session per thread behind the SDK's single session: the SDK shares + it across every caller, and requests.Session is not guaranteed thread-safe.""" + + def __init__(self, headers, attempts, timeout, limiter): + self.headers = dict(headers) + self._make = lambda: http.Session(attempts=attempts, timeout=timeout, limiter=limiter) + self._local = threading.local() + + def request(self, method, url, **kwargs): + session = getattr(self._local, "session", None) + if session is None: + session = self._local.session = self._make() + session.headers.update(self.headers) + return session.request(method, url, **kwargs) + + def close(self): + session = getattr(self._local, "session", None) + if session is not None: + session.close() + + +def client(token, project_id, attempts=10, timeout=60, rate=REQUESTS_PER_SECOND, + session=None): + """A CrowdinClient whose requests go through shared.http. + + `session` replaces the transport outright, which is how a test hands in a fake; + it receives the SDK's auth headers like any other. + """ + crowdin = CrowdinClient(token=token, project_id=int(project_id), timeout=timeout, + max_retries=1) + requester = crowdin.get_api_requestor() + headers = requester.session.headers + requester.session.close() + if session is None: + session = _PerThreadSession(headers, attempts, timeout, + http.TokenBucket(rate) if rate else None) + else: + session.headers.update(headers) + requester._session = session + return crowdin + + +def error_message(exc): + """Crowdin's error message, or the start of the body when it is not that envelope.""" + body = response_text(exc) + try: + return json.loads(body).get("error", {}).get("message", "Unknown error") + except (ValueError, AttributeError): + return body[:200] or "Unknown error" + + +def response_text(exc): + body = exc.context or b"" + return body.decode("utf-8", "replace") if isinstance(body, bytes) else str(body) + + +def fetch_all(resource, method, **params): + """Every item of a paginated list endpoint, unwrapped from the SDK's envelopes. + + `resource` must be fresh from the client (client.source_strings, say): the SDK + keeps the fetch-all flag on the resource object, so a shared one races. + """ + listing = getattr(resource.with_fetch_all(), method) + return [row["data"] for row in listing(**params)["data"]] diff --git a/src/session_ops/github_prs/digest.py b/src/session_ops/github_prs/digest.py index e648951..2d65395 100755 --- a/src/session_ops/github_prs/digest.py +++ b/src/session_ops/github_prs/digest.py @@ -44,12 +44,10 @@ from datetime import datetime, timedelta, timezone from operator import itemgetter -import requests -from session_ops.shared import discord, state as dedup +from session_ops.shared import discord, http, state as dedup from session_ops.shared.discord import MAX_MESSAGE_TEXT_CHARS, clip from session_ops.shared.env import get_env -from session_ops.shared.retry import request_with_retry API = "https://api.github.com" DEFAULT_ORG = "session-foundation" @@ -80,7 +78,7 @@ def load_maintainers(path): def github_session(token): - session = requests.Session() + session = http.Session() session.headers.update({ "Authorization": f"Bearer {token}", "Accept": "application/vnd.github+json", @@ -90,7 +88,7 @@ def github_session(token): def fetch_json(session, url, **kwargs): - resp = request_with_retry(session, "GET", url, **kwargs) + resp = session.request("GET", url, **kwargs) if resp.status_code >= 400: sys.exit(f"GitHub {resp.status_code} on {url}: {resp.text[:300]}") return resp.json() @@ -386,7 +384,7 @@ def main(): print(json.dumps(messages, indent=2, ensure_ascii=False)) return - posted = discord.post_to_discord(requests.Session(), + posted = discord.post_to_discord(http.Session(), discord.components_webhook_url(webhook), messages) # Only what Discord accepted. A PR in a message that never landed stays eligible. if args.state: diff --git a/src/session_ops/monitor/alert.py b/src/session_ops/monitor/alert.py index 99fe974..3ea871a 100644 --- a/src/session_ops/monitor/alert.py +++ b/src/session_ops/monitor/alert.py @@ -25,7 +25,7 @@ import subprocess import sys -import requests +from session_ops.shared import http from session_ops.zendesk import triage @@ -108,7 +108,7 @@ def main(): message = build_message(args[0], socket.gethostname(), *args[1:], detail=detail) # A fresh session, never a Zendesk one β€” that carries the API-token auth header, # and Discord has no business receiving it. - if not triage.post_to_discord(requests.Session(), webhook, [{"content": message}]): + if not triage.post_to_discord(http.Session(), webhook, [{"content": message}]): sys.exit("Could not post the failure to Discord.") diff --git a/src/session_ops/monitor/silence.py b/src/session_ops/monitor/silence.py index b55580a..d3e80c5 100644 --- a/src/session_ops/monitor/silence.py +++ b/src/session_ops/monitor/silence.py @@ -32,9 +32,8 @@ import tomllib from datetime import datetime, timezone -import requests -from session_ops.shared import discord +from session_ops.shared import discord, http from session_ops.shared.env import get_env REGISTRY = os.path.join(os.path.dirname(os.path.abspath(__file__)), "jobs.toml") @@ -161,7 +160,7 @@ def main(): if args.dry_run: print(message) return - if not discord.post_to_discord(requests.Session(), webhook, [{"content": message}]): + if not discord.post_to_discord(http.Session(), webhook, [{"content": message}]): # State still saved: a missing stamp's clock must survive a failed post. if args.state: save_state(args.state, state) diff --git a/src/session_ops/shared/discord.py b/src/session_ops/shared/discord.py index 90c3a12..0a4aeb0 100644 --- a/src/session_ops/shared/discord.py +++ b/src/session_ops/shared/discord.py @@ -12,7 +12,6 @@ import requests -from session_ops.shared.retry import request_with_retry COMPONENTS_V2_FLAG = 1 << 15 CONTAINER = 17 @@ -112,7 +111,7 @@ def post_to_discord(session, url, messages): """ for index, payload in enumerate(messages): try: - resp = request_with_retry(session, "POST", url, json=payload) + resp = session.request("POST", url, json=payload) except requests.RequestException as exc: print(f"Discord unreachable on message {index + 1}/{len(messages)} " f"({exc}).") diff --git a/src/session_ops/shared/http.py b/src/session_ops/shared/http.py new file mode 100644 index 0000000..101e81a --- /dev/null +++ b/src/session_ops/shared/http.py @@ -0,0 +1,163 @@ +"""The one HTTP transport every job uses: a requests.Session that retries, paces and +times out. + +Retries happen in urllib3, below the session, so a caller sees only the final +answer. A 429 or 5xx that outlasts the budget comes back as a response, whatever its +status; a transport failure that outlasts it raises. Every method is retried, POST +included, which is what the callers here have always relied on. + + session = http.Session(attempts=10, timeout=60, rate=20) + session.get(url) # the session's budget + session.request("GET", url, attempts=2) # a shorter one for optional data +""" +import contextvars +import email.utils +import math +import threading +import time +from datetime import datetime, timezone + +import requests +from requests.adapters import HTTPAdapter +from urllib3.util.retry import Retry + +DEFAULT_ATTEMPTS = 6 +DEFAULT_TIMEOUT = 30 +MAX_BACKOFF = 30 +MAX_RETRY_AFTER = 60 +RETRY_STATUSES = frozenset({429, *range(500, 600)}) + +_attempts = contextvars.ContextVar("attempts", default=None) + + +def retry_after_seconds(resp, default): + """Seconds to wait per the response's rate-limit headers, else `default`. + + Retry-After first, in either form RFC 9110 allows: a delay in seconds or an + HTTP-date. GitHub answers a primary rate limit with x-ratelimit-reset as an + epoch second and no Retry-After at all, so that is read when the header is + absent. + + Anything unparseable falls back rather than crashing the run. Negative, NaN + and infinite values fall back too: time.sleep() rejects the first two + outright, so a hostile or buggy proxy sending `Retry-After: -30` would + otherwise take the run down with a ValueError. + """ + raw = resp.headers.get("retry-after") + if raw is None: + reset = resp.headers.get("x-ratelimit-reset") + if reset is None: + return default + try: + return max(0.0, float(reset) - time.time()) + except (TypeError, ValueError): + return default + try: + seconds = float(raw) + except (TypeError, ValueError): + seconds = _seconds_until_http_date(raw) + if seconds is None: + return default + if not math.isfinite(seconds) or seconds < 0: + return default + return seconds + + +def _seconds_until_http_date(value): + try: + when = email.utils.parsedate_to_datetime(value) + except (TypeError, ValueError): + return None + if when.tzinfo is None: + when = when.replace(tzinfo=timezone.utc) + return (when - datetime.now(timezone.utc)).total_seconds() + + +class _Retry(Retry): + """urllib3's Retry with this repo's waits: 1 s doubling to 30 s, or what the + server asked for, capped at a minute.""" + + def get_backoff_time(self): + return min(2.0 ** (len(self.history) - 1), MAX_BACKOFF) if self.history else 0 + + def get_retry_after(self, response): + seconds = retry_after_seconds(response, None) + return None if seconds is None else min(seconds, MAX_RETRY_AFTER) + + def sleep_for_retry(self, response): + # The base class treats Retry-After: 0 as absent and backs off instead. + seconds = self.get_retry_after(response) + if seconds is None: + return False + time.sleep(seconds) + return True + + +def _retry(attempts): + if attempts <= 0: + raise ValueError("attempts must be at least 1") + return _Retry(total=attempts - 1, status_forcelist=RETRY_STATUSES, allowed_methods=None, + raise_on_status=False, respect_retry_after_header=True) + + +class _Adapter(HTTPAdapter): + """Reads a per-call budget from the context HTTPAdapter.send() runs in, since + requests gives no way to pass one down.""" + + @property + def max_retries(self): + attempts = _attempts.get() + return self._max_retries if attempts is None else _retry(attempts) + + @max_retries.setter + def max_retries(self, value): + self._max_retries = value + + +class TokenBucket: + """At most `rate` requests per second on average, in bursts of up to `rate`.""" + + def __init__(self, rate): + self.rate = float(rate) + self.tokens = self.rate + self.updated = time.monotonic() + self._lock = threading.Lock() + + def acquire(self): + with self._lock: + now = time.monotonic() + self.tokens = min(self.rate, self.tokens + (now - self.updated) * self.rate) + self.updated = now + wait = 0.0 if self.tokens >= 1 else (1 - self.tokens) / self.rate + self.tokens -= 1 + if wait: + time.sleep(wait) + + +class Session(requests.Session): + """A session that retries 429 and 5xx, times out, and optionally paces itself. + + `limiter` may be shared between sessions, so that per-thread sessions against one + API still draw on one budget. It paces each call, not each retry inside it. + """ + + def __init__(self, attempts=DEFAULT_ATTEMPTS, timeout=DEFAULT_TIMEOUT, rate=None, + limiter=None): + super().__init__() + self.timeout = timeout + self.limiter = limiter or (TokenBucket(rate) if rate else None) + adapter = _Adapter(max_retries=_retry(attempts)) + self.mount("https://", adapter) + self.mount("http://", adapter) + + def request(self, method, url, *args, attempts=None, **kwargs): + if attempts is not None and attempts <= 0: + raise ValueError("attempts must be at least 1") + kwargs.setdefault("timeout", self.timeout) + if self.limiter: + self.limiter.acquire() + token = _attempts.set(attempts) + try: + return super().request(method, url, *args, **kwargs) + finally: + _attempts.reset(token) diff --git a/src/session_ops/shared/retry.py b/src/session_ops/shared/retry.py deleted file mode 100644 index 82059f3..0000000 --- a/src/session_ops/shared/retry.py +++ /dev/null @@ -1,84 +0,0 @@ -import email.utils -import math -import time -from datetime import datetime, timezone - -import requests - - -def retry_after_seconds(resp, default): - """Seconds to wait per the response's rate-limit headers, else `default`. - - Retry-After first, in either form RFC 9110 allows: a delay in seconds or an - HTTP-date. GitHub answers a primary rate limit with x-ratelimit-reset as an - epoch second and no Retry-After at all, so that is read when the header is - absent. - - Anything unparseable falls back rather than crashing the run. Negative, NaN - and infinite values fall back too: time.sleep() rejects the first two - outright, so a hostile or buggy proxy sending `Retry-After: -30` would - otherwise take the run down with a ValueError. - """ - raw = resp.headers.get("retry-after") - if raw is None: - reset = resp.headers.get("x-ratelimit-reset") - if reset is None: - return default - try: - return max(0.0, float(reset) - time.time()) - except (TypeError, ValueError): - return default - try: - seconds = float(raw) - except (TypeError, ValueError): - seconds = _seconds_until_http_date(raw) - if seconds is None: - return default - if not math.isfinite(seconds) or seconds < 0: - return default - return seconds - - -def _seconds_until_http_date(value): - try: - when = email.utils.parsedate_to_datetime(value) - except (TypeError, ValueError): - return None - if when.tzinfo is None: - when = when.replace(tzinfo=timezone.utc) - return (when - datetime.now(timezone.utc)).total_seconds() - - -def request_with_retry(session, method, url, attempts=6, timeout=30, **kwargs): - """GET/POST with backoff on 429 and 5xx. Returns the response, whatever its status. - - Lower `attempts` for calls whose result is nice-to-have: the full budget can - burn ~60s of backoff, which is not worth spending on optional data. - """ - if attempts <= 0: - raise ValueError("attempts must be at least 1") - - delay = 1.0 - last_exc = None - resp = None - for attempt in range(attempts): - final = attempt == attempts - 1 - try: - resp = session.request(method, url, timeout=timeout, **kwargs) - except requests.RequestException as exc: - last_exc = exc - if final: - break - time.sleep(delay) - delay = min(delay * 2, 30) - continue - if resp.status_code == 429 or resp.status_code >= 500: - if final: - break - time.sleep(min(retry_after_seconds(resp, delay), 60)) - delay = min(delay * 2, 30) - continue - return resp - if last_exc: - raise last_exc - return resp diff --git a/src/session_ops/shared/testing.py b/src/session_ops/shared/testing.py index a855f05..079453c 100644 --- a/src/session_ops/shared/testing.py +++ b/src/session_ops/shared/testing.py @@ -15,6 +15,10 @@ def __init__(self, payload, status_code=200, retry_after="0"): self.headers = {"retry-after": retry_after} self.text = json.dumps(payload) + @property + def content(self): + return self.text.encode() + def json(self): return self._payload @@ -40,6 +44,7 @@ class FakeSession: def __init__(self, responses): self._responses = list(responses) self.calls = [] + self.headers = {} def request(self, method, url, **kwargs): self.calls.append((method, url, kwargs)) @@ -48,6 +53,9 @@ def request(self, method, url, **kwargs): raise item return item + def close(self): + pass + class Patched: """Swap module attributes for the duration of a block, then put them back.""" @@ -107,7 +115,6 @@ def __init__(self, recorded): super().__init__(recorded.get("json"), recorded.get("status", 200)) if "text" in recorded: self.text = recorded["text"] - self.content = self.text.encode() def json(self): if self._payload is None: @@ -152,6 +159,9 @@ def get(self, url, **kwargs): def post(self, url, **kwargs): return self.request("POST", url, **kwargs) + def close(self): + pass + def frozen_datetime(now): """A datetime class whose now() is `now`, for patching over a module's import.""" diff --git a/src/session_ops/zendesk/note_reply.py b/src/session_ops/zendesk/note_reply.py index 5083751..44dbffb 100644 --- a/src/session_ops/zendesk/note_reply.py +++ b/src/session_ops/zendesk/note_reply.py @@ -204,7 +204,7 @@ def api_user_id(session, subdomain): this same user so a draft never fires the webhook at all β€” see the README. """ url = f"https://{subdomain}.zendesk.com/api/v2/users/me.json" - resp = triage.request_with_retry(session, "GET", url) + resp = session.request("GET", url) if resp.status_code >= 400: sys.exit(f"Zendesk refused to identify the API user ({resp.status_code}).") return ((resp.json() or {}).get("user") or {}).get("id") @@ -241,7 +241,7 @@ def change_tags(session, subdomain, ticket_id, add=(), drop=()): ("DELETE", [t for t in drop if t])): if not names: continue - resp = triage.request_with_retry(session, method, url, json={"tags": names}) + resp = session.request(method, url, json={"tags": names}) if resp.status_code >= 400: # Never worth failing a run over: tags are a dashboard light, not the work. print(f"Note: could not {method.lower()} tags on #{ticket_id} " @@ -353,7 +353,7 @@ def write_to_ticket(session, subdomain, ticket_id, body, public, if status: fields["status"] = status url = f"https://{subdomain}.zendesk.com/api/v2/tickets/{ticket_id}.json" - resp = triage.request_with_retry(session, "PUT", url, json={"ticket": fields}) + resp = session.request("PUT", url, json={"ticket": fields}) if resp.status_code >= 400: sys.exit(f"Zendesk rejected the {'reply' if public else 'note'} on " f"#{ticket_id} ({resp.status_code}).") diff --git a/src/session_ops/zendesk/resolve_reviews.py b/src/session_ops/zendesk/resolve_reviews.py index 4a48c7f..5bd74fc 100644 --- a/src/session_ops/zendesk/resolve_reviews.py +++ b/src/session_ops/zendesk/resolve_reviews.py @@ -76,8 +76,8 @@ from datetime import datetime, timedelta, timezone from urllib.parse import quote -import requests +from session_ops.shared import http from session_ops.zendesk import triage # Reviews at or above this rating carry nothing to act on. Fixed rather than a flag: @@ -186,9 +186,7 @@ def solve_batch(session, subdomain, ids, tag, note): payload = {"ticket": {"status": "solved", "additional_tags": [tag]}} if note: payload["ticket"]["comment"] = {"body": note, "public": False} - resp = triage.request_with_retry( - session, "PUT", url, params={"ids": ",".join(str(i) for i in ids)}, json=payload - ) + resp = session.request("PUT", url, params={"ids": ",".join(str(i) for i in ids)}, json=payload) if resp.status_code >= 400: sys.exit(f"update_many failed ({resp.status_code}): {resp.text[:300]}") job = (resp.json() or {}).get("job_status") or {} @@ -212,7 +210,7 @@ def wait_for_job(session, subdomain, job_id, timeout=JOB_TIMEOUT_SECONDS): url = f"https://{subdomain}.zendesk.com/api/v2/job_statuses/{job_id}.json" deadline = time.monotonic() + timeout while True: - resp = triage.request_with_retry(session, "GET", url) + resp = session.request("GET", url) if resp.status_code >= 400: sys.exit(f"could not read job {job_id} ({resp.status_code}): {resp.text[:200]}") job = (resp.json() or {}).get("job_status") or {} @@ -316,7 +314,7 @@ def post_summary(webhook_url, message): A fresh session, never the Zendesk one β€” that carries the API-token auth header, and Discord has no business receiving it. """ - return bool(triage.post_to_discord(requests.Session(), webhook_url, + return bool(triage.post_to_discord(http.Session(), webhook_url, [{"content": message}])) diff --git a/src/session_ops/zendesk/triage.py b/src/session_ops/zendesk/triage.py index b867a40..d4582e7 100644 --- a/src/session_ops/zendesk/triage.py +++ b/src/session_ops/zendesk/triage.py @@ -81,10 +81,9 @@ import requests -from session_ops.shared import discord, state as dedup +from session_ops.shared import discord, http, state as dedup from session_ops.shared.discord import clip, post_to_discord from session_ops.shared.env import get_env -from session_ops.shared.retry import request_with_retry # The channel AppFollow imports app-store reviews on. Identified reviews with no # false positives in a 3,662-ticket sample; tags did not (only 287 carried one). @@ -387,7 +386,7 @@ def window_label(hours): def zendesk_session(email, token): - session = requests.Session() + session = http.Session() # Zendesk API-token auth: username is "{email}/token", password is the token. session.auth = (f"{email}/token", token) session.headers["Accept"] = "application/json" @@ -450,7 +449,7 @@ def fetch_tickets(session, subdomain, query, max_tickets): # Whichever bites first: our own runaway guard or Zendesk's hard result limit. cap = min(max_tickets, SEARCH_RESULT_LIMIT) while url and len(tickets) < cap: - resp = request_with_retry(session, "GET", url, params=params) + resp = session.request("GET", url, params=params) params = None # next_page already carries the query if resp.status_code == 403: sys.exit("Zendesk returned 403 β€” the API token/email may lack search access.") @@ -479,7 +478,7 @@ def fetch_tickets(session, subdomain, query, max_tickets): def fetch_ticket(session, subdomain, ticket_id): url = f"https://{subdomain}.zendesk.com/api/v2/tickets/{ticket_id}.json" - resp = request_with_retry(session, "GET", url) + resp = session.request("GET", url) if resp.status_code == 404: sys.exit(f"Ticket #{ticket_id} does not exist.") if resp.status_code >= 400: @@ -498,8 +497,7 @@ def fetch_comments(session, subdomain, ticket_id): opening back-and-forth. """ url = f"https://{subdomain}.zendesk.com/api/v2/tickets/{ticket_id}/comments.json" - resp = request_with_retry( - session, "GET", url, params={"per_page": 100, "sort_order": "desc"}) + resp = session.request("GET", url, params={"per_page": 100, "sort_order": "desc"}) if resp.status_code >= 400: sys.exit(f"Could not read the comments on #{ticket_id} ({resp.status_code}).") return (resp.json() or {}).get("comments") or [] @@ -513,15 +511,13 @@ def fetch_total_unsolved(session, subdomain, query=BACKLOG_QUERY): """ url = f"https://{subdomain}.zendesk.com/api/v2/search/count.json" try: - resp = request_with_retry( - session, "GET", url, attempts=2, params={"query": query} - ) + resp = session.request("GET", url, attempts=2, params={"query": query}) if resp.status_code >= 400: print(f"Note: could not count the unsolved backlog ({resp.status_code}).") return None return resp.json().get("count") except (requests.RequestException, ValueError) as exc: - # request_with_retry re-raises the transport error once its (short) budget is + # The session re-raises the transport error once its (short) budget is # spent, and .json() raises on a non-JSON body β€” neither is a reason to lose # the digest over one context number, so both land on the documented None. print(f"Note: could not count the unsolved backlog ({exc}).") @@ -573,7 +569,7 @@ def hydrate_requester_activity(session, subdomain, tickets): chunk = ids[start : start + 100] url = f"https://{subdomain}.zendesk.com/api/v2/tickets/show_many.json" try: - resp = request_with_retry(session, "GET", url, attempts=2, params={ + resp = session.request("GET", url, attempts=2, params={ "ids": ",".join(str(i) for i in chunk), "include": "metric_sets"}) except requests.RequestException as exc: print(f"Note: could not fetch ticket metrics ({exc}); " @@ -695,7 +691,7 @@ def fetch_user(session, subdomain, user_id): failed lookup widens the sample rather than silencing it. """ url = f"https://{subdomain}.zendesk.com/api/v2/users/{user_id}.json" - resp = request_with_retry(session, "GET", url, attempts=2) + resp = session.request("GET", url, attempts=2) if resp.status_code >= 400: return {} return (resp.json() or {}).get("user") or {} @@ -865,7 +861,7 @@ def hydrate_descriptions(session, subdomain, tickets): continue url = f"https://{subdomain}.zendesk.com/api/v2/tickets/{ticket['id']}/comments.json" try: - resp = request_with_retry(session, "GET", url, attempts=2, params={"per_page": 10}) + resp = session.request("GET", url, attempts=2, params={"per_page": 10}) except requests.RequestException as exc: # Hydration is an enrichment, never a reason to abort the digest: an # unreachable comments endpoint just leaves the description as-is. @@ -982,7 +978,7 @@ def conversation_turns(session, subdomain, ticket): return None url = f"https://{subdomain}.zendesk.com/api/v2/tickets/{ticket['id']}/comments.json" try: - resp = request_with_retry(session, "GET", url, attempts=2, + resp = session.request("GET", url, attempts=2, params={"per_page": 100, "sort_order": "asc"}) except requests.RequestException as exc: print(f"Note: could not fetch comments for #{ticket['id']} ({exc}).") @@ -1059,7 +1055,7 @@ def write_english_field(session, subdomain, ticket_id, field_id, english): url = f"https://{subdomain}.zendesk.com/api/v2/tickets/{ticket_id}.json" payload = {"ticket": {"custom_fields": [{"id": field_id, "value": english}]}} try: - resp = request_with_retry(session, "PUT", url, attempts=2, json=payload) + resp = session.request("PUT", url, attempts=2, json=payload) except requests.RequestException as exc: print(f"Note: could not write the English transcript to #{ticket_id} ({exc}).") return False @@ -1890,7 +1886,7 @@ def main(): return # A fresh session, never the Zendesk one: that carries the API-token auth header. - posted = post_to_discord(requests.Session(), discord.components_webhook_url(webhook), + posted = post_to_discord(http.Session(), discord.components_webhook_url(webhook), messages) print(f"Posted {posted} of {len(messages)} Discord message(s).") diff --git a/tests/crowdin/test_approve_golden.py b/tests/crowdin/test_approve_golden.py new file mode 100644 index 0000000..240626c --- /dev/null +++ b/tests/crowdin/test_approve_golden.py @@ -0,0 +1,43 @@ +""" + UPDATE_GOLDENS=1 uv run python -m unittest tests.crowdin.test_approve_golden + +The recording is shaped like Crowdin's API responses rather than captured live. +""" +import contextlib +import io +import sys +import unittest +from unittest import mock + +from session_ops.crowdin import approve_strings as approve +from session_ops.crowdin import sdk +from session_ops.shared.testing import NoSleep, RecordedSession +from tests.golden import assert_golden, load_golden_json + + +class TestApproveGolden(unittest.TestCase): + def run_approve(self, name): + recording = load_golden_json("approve/responses.json") + session = RecordedSession(recording["exchanges"]) + out = io.StringIO() + with mock.patch.object(approve, "get_token", lambda: "t"), \ + mock.patch.object(approve, "crowdin_client", + lambda token, pid: sdk.client(token, pid, session=session)), \ + mock.patch.object(sys, "argv", ["approve", *recording["runs"][name]]), \ + NoSleep(), contextlib.redirect_stdout(out): + approve.main() + return out.getvalue(), [call for call in session.calls if call[0].upper() == "POST"] + + def test_approving_by_user(self): + out, posts = self.run_approve("approve") + self.assertEqual(len(posts), 4) + assert_golden(self, "approve/approve.txt", out) + + def test_listing_approves_nothing(self): + out, posts = self.run_approve("list") + self.assertEqual(posts, []) + assert_golden(self, "approve/list.txt", out) + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/crowdin/test_crowdin_goldens.py b/tests/crowdin/test_crowdin_goldens.py index 7f8e08b..f8b6487 100644 --- a/tests/crowdin/test_crowdin_goldens.py +++ b/tests/crowdin/test_crowdin_goldens.py @@ -13,10 +13,10 @@ from unittest import mock import colorama -import requests from session_ops.crowdin import download_translations_from_crowdin as download from session_ops.crowdin import report_multiple_translations as report +from session_ops.crowdin import sdk from session_ops.shared.testing import RecordedSession from tests.golden import assert_golden, load_golden_json @@ -31,7 +31,8 @@ def test_dry_run_for_one_locale(self): session = RecordedSession(recording["exchanges"]) out = io.StringIO() with tempfile.TemporaryDirectory() as tmp: - with mock.patch.object(report.requests, "Session", lambda: session), \ + with mock.patch.object(report, "crowdin_client", + lambda token, pid: sdk.client(token, pid, session=session)), \ mock.patch.object(sys, "argv", ["report", *argv_for(recording, tmp)]), \ contextlib.redirect_stdout(out), contextlib.redirect_stderr(io.StringIO()): report.main() @@ -46,11 +47,9 @@ def test_export_payloads_and_written_files(self): recording = load_golden_json("download/responses.json") session = RecordedSession(recording["exchanges"]) with tempfile.TemporaryDirectory() as tmp: - with mock.patch.object(requests, "get", session.get), \ - mock.patch.object(requests, "post", session.post), \ - contextlib.redirect_stdout(io.StringIO()) as out: + with contextlib.redirect_stdout(io.StringIO()) as out: try: - download.main(argv_for(recording, tmp)) + download.main(argv_for(recording, tmp), session=session) except SystemExit as exc: self.fail(f"download exited {exc.code}:\n{out.getvalue()}") finally: @@ -60,7 +59,7 @@ def test_export_payloads_and_written_files(self): with open(os.path.join(tmp, name), encoding="utf-8") as handle: files[name] = handle.read() exports = sorted((json.loads(data) for method, _, _, data in session.calls - if method == "POST"), key=lambda body: body["targetLanguageId"]) + if method.upper() == "POST"), key=lambda body: body["targetLanguageId"]) assert_golden(self, "download/output.json", json.dumps({"exports": exports, "files": files}, indent=2, ensure_ascii=False) + "\n") diff --git a/tests/crowdin/test_report_multiple_translations.py b/tests/crowdin/test_report_multiple_translations.py index 70dd844..f591c28 100644 --- a/tests/crowdin/test_report_multiple_translations.py +++ b/tests/crowdin/test_report_multiple_translations.py @@ -5,16 +5,11 @@ import io import unittest -import requests +from unittest import mock from session_ops.crowdin import report_multiple_translations as report -from session_ops.shared.testing import FakeResponse, FakeSession, NoSleep, Patched - - -class ApiResponse(FakeResponse): - def raise_for_status(self): - if self.status_code >= 400: - raise requests.HTTPError(f"{self.status_code}", response=self) +from session_ops.crowdin import sdk +from session_ops.shared.testing import FakeResponse, FakeSession, Patched class WebhookSession(FakeSession): @@ -25,25 +20,26 @@ def __exit__(self, *exc): return False -class TestRequestWithRetry(unittest.TestCase): +class TestCrowdinClient(unittest.TestCase): def test_a_client_error_raises_rather_than_returning(self): """Callers read the body straight off the response, so a 4xx has to stop them.""" - with self.assertRaises(requests.HTTPError): - report.request_with_retry(FakeSession([ApiResponse({}, status_code=404)]), - "GET", "https://x") + client = sdk.client("t", 1, session=FakeSession([FakeResponse({}, status_code=404)])) + with self.assertRaises(sdk.APIException): + client.projects.get_project() def test_crowdins_budget_is_ten_attempts_at_sixty_seconds(self): - session = FakeSession([ApiResponse({}, status_code=503)] * 9 + [ApiResponse({"ok": 1})]) - with NoSleep(): - self.assertEqual(report.request_with_retry(session, "GET", "https://x").json(), {"ok": 1}) - self.assertEqual(len(session.calls), 10) - self.assertEqual({kw["timeout"] for _, _, kw in session.calls}, {60}) + """A locale is ~1,400 requests near the rate limit: 429s are expected.""" + with mock.patch.object(report.sdk.http, "Session") as made: + client = report.crowdin_client("t", 1) + client.get_api_requestor().session.request("GET", "https://x") + self.assertEqual(made.call_args.kwargs["attempts"], 10) + self.assertEqual(made.call_args.kwargs["timeout"], 60) class TestPostToDiscord(unittest.TestCase): def post(self, responses, messages): session = WebhookSession(responses) - with Patched(report.requests, Session=lambda: session), \ + with Patched(report.http, Session=lambda: session), \ contextlib.redirect_stdout(io.StringIO()): report.post_to_discord("https://hook", messages) return session @@ -61,7 +57,7 @@ def test_a_rejection_posts_a_plain_warning_then_exits(self): def test_the_warning_is_plain_content_the_webhook_cannot_reject_for_size(self): session = WebhookSession([FakeResponse({}, status_code=400), FakeResponse({}, status_code=204)]) - with Patched(report.requests, Session=lambda: session), \ + with Patched(report.http, Session=lambda: session), \ contextlib.redirect_stdout(io.StringIO()), self.assertRaises(SystemExit): report.post_to_discord("https://hook", [{"embeds": [{"title": "x" * 9000}]}]) self.assertEqual(list(session.calls[1][2]["json"]), ["content"]) diff --git a/tests/crowdin/test_sdk.py b/tests/crowdin/test_sdk.py new file mode 100644 index 0000000..9115686 --- /dev/null +++ b/tests/crowdin/test_sdk.py @@ -0,0 +1,82 @@ +""" + uv run python -m unittest tests.crowdin.test_sdk +""" +import threading +import unittest +from unittest import mock + +from crowdin_api.exceptions import APIException + +from session_ops.crowdin import sdk +from session_ops.shared.testing import FakeResponse, FakeSession, RecordedSession + +API = "https://api.crowdin.com/api/v2" + + +class TestClient(unittest.TestCase): + def test_the_sdks_own_retry_loop_is_off(self): + """It retries 5xx on a fixed 100 ms and never a 429; shared.http does both.""" + self.assertEqual(sdk.client("t", 1).get_api_requestor()._max_retries, 1) + + def test_requests_carry_the_token_to_an_injected_session(self): + session = FakeSession([FakeResponse({"data": {"id": 1}})]) + sdk.client("tok", 1, session=session).projects.get_project() + self.assertEqual(session.headers["Authorization"], "Bearer tok") + method, url, _ = session.calls[0] + self.assertEqual((method, url), ("get", f"{API}/projects/1")) + + def test_each_thread_gets_its_own_session_with_the_same_headers(self): + """requests.Session is not guaranteed thread-safe, and the SDK shares one.""" + made = [] + + class Made(FakeSession): + def __init__(self, **kwargs): + super().__init__([FakeResponse({})] * 2) + made.append(self) + + with mock.patch.object(sdk.http, "Session", Made): + shared = sdk.client("tok", 1).get_api_requestor().session + shared.request("GET", "https://x") + shared.request("GET", "https://x") + other = threading.Thread(target=shared.request, args=("GET", "https://x")) + other.start() + other.join() + self.assertEqual(len(made), 2) + self.assertEqual({s.headers["Authorization"] for s in made}, {"Bearer tok"}) + + def test_every_thread_draws_on_one_rate_limit(self): + shared = sdk.client("tok", 1).get_api_requestor().session + self.assertIs(shared._make().limiter, shared._make().limiter) + + +class TestFetchAll(unittest.TestCase): + def test_pages_until_a_short_page_and_unwraps_each_row(self): + rows = [{"data": {"id": i}} for i in range(500)] + session = RecordedSession([ + {"method": "get", "url": f"{API}/projects/1/strings", + "params": {"limit": 500, "offset": 0}, "response": {"json": {"data": rows}}}, + {"method": "get", "url": f"{API}/projects/1/strings", + "params": {"limit": 500, "offset": 500}, + "response": {"json": {"data": [{"data": {"id": 500}}]}}}, + ]) + client = sdk.client("t", 1, session=session) + ids = [row["id"] for row in sdk.fetch_all(client.source_strings, "list_strings")] + self.assertEqual(ids, list(range(501))) + + +class TestErrorMessage(unittest.TestCase): + def error(self, body): + return APIException(http_status=400, context=body) + + def test_crowdins_envelope(self): + self.assertEqual(sdk.error_message(self.error(b'{"error": {"message": "Nope"}}')), "Nope") + + def test_a_body_that_is_not_the_envelope_is_quoted(self): + self.assertEqual(sdk.error_message(self.error(b"502")), "502") + + def test_an_empty_body(self): + self.assertEqual(sdk.error_message(self.error(b"")), "Unknown error") + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/goldens/approve/approve.txt b/tests/goldens/approve/approve.txt new file mode 100644 index 0000000..da91099 --- /dev/null +++ b/tests/goldens/approve/approve.txt @@ -0,0 +1,13 @@ + string 'ongoingAppeal' -> id 201 + string 'ongoingAppealDescription' -> id 202 + +3 target languages: de, fr, ja + + [ ok ] ongoingAppeal / de: approved tid=12 submitter=1:alice | Widerspruch + [skip] ongoingAppeal / fr: no translation from an allowed submitter (present: submitter=2:Bob B) + [skip] ongoingAppeal / ja: no translation + [ -- ] ongoingAppealDescription / de: already approved tid=31 submitter=1:alice | Einspruch lΓ€uft + [ERR ] ongoingAppealDescription / fr: 500 {"error": {"code": 500, "message": "Internal error"}} + [ ok ] ongoingAppealDescription / ja: approved tid=51 submitter=1:alice | 異議申し立て中 + +Done. approved: 2, already: 1, skipped: 2, errors: 1 diff --git a/tests/goldens/approve/list.txt b/tests/goldens/approve/list.txt new file mode 100644 index 0000000..06669f0 --- /dev/null +++ b/tests/goldens/approve/list.txt @@ -0,0 +1,10 @@ + string 'ongoingAppeal' -> id 201 + string 'ongoingAppealDescription' -> id 202 + +3 target languages: de, fr, ja + + ongoingAppeal / de: tid=12 submitter=1:alice | Widerspruch + ongoingAppeal / de: tid=13 submitter=2:Bob B | Beschwerde + ongoingAppeal / fr: tid=21 submitter=2:Bob B | Appel + ongoingAppealDescription / de: tid=31 submitter=1:alice | Einspruch lΓ€uft + ongoingAppealDescription / ja: tid=51 submitter=1:alice | 異議申し立て中 diff --git a/tests/goldens/approve/responses.json b/tests/goldens/approve/responses.json new file mode 100644 index 0000000..30b9ccb --- /dev/null +++ b/tests/goldens/approve/responses.json @@ -0,0 +1,516 @@ +{ + "runs": { + "approve": [ + "ongoingAppeal", + "ongoingAppealDescription", + "--by-user", + "alice" + ], + "list": [ + "ongoingAppeal", + "ongoingAppealDescription", + "--list" + ] + }, + "exchanges": [ + { + "method": "GET", + "url": "https://api.crowdin.com/api/v2/projects/618696/strings", + "params": { + "filter": "ongoingAppeal", + "scope": "identifier", + "limit": 500, + "offset": 0 + }, + "data": null, + "response": { + "status": 200, + "json": { + "data": [ + { + "data": { + "id": 200, + "identifier": "ongoingAppealTitle" + } + }, + { + "data": { + "id": 201, + "identifier": "ongoingAppeal" + } + } + ], + "pagination": { + "offset": 0, + "limit": 500 + } + } + } + }, + { + "method": "GET", + "url": "https://api.crowdin.com/api/v2/projects/618696/strings", + "params": { + "filter": "ongoingAppealDescription", + "scope": "identifier", + "limit": 500, + "offset": 0 + }, + "data": null, + "response": { + "status": 200, + "json": { + "data": [ + { + "data": { + "id": 202, + "identifier": "ongoingAppealDescription" + } + } + ], + "pagination": { + "offset": 0, + "limit": 500 + } + } + } + }, + { + "method": "GET", + "url": "https://api.crowdin.com/api/v2/projects/618696", + "params": null, + "data": null, + "response": { + "status": 200, + "json": { + "data": { + "id": 618696, + "identifier": "session-crossplatform-strings", + "targetLanguageIds": [ + "de", + "fr", + "ja" + ] + } + } + } + }, + { + "method": "GET", + "url": "https://api.crowdin.com/api/v2/projects/618696/translations", + "params": { + "stringId": 201, + "languageId": "de", + "limit": 500, + "offset": 0 + }, + "data": null, + "response": { + "status": 200, + "json": { + "data": [ + { + "data": { + "id": 11, + "text": "Einspruch", + "user": { + "id": 1, + "username": "alice", + "fullName": "Alice A" + } + } + }, + { + "data": { + "id": 12, + "text": "Widerspruch", + "user": { + "id": 1, + "username": "alice", + "fullName": "Alice A" + } + } + }, + { + "data": { + "id": 13, + "text": "Beschwerde", + "user": { + "id": 2, + "username": null, + "fullName": "Bob B" + } + } + } + ], + "pagination": { + "offset": 0, + "limit": 500 + } + } + } + }, + { + "method": "GET", + "url": "https://api.crowdin.com/api/v2/projects/618696/approvals", + "params": { + "stringId": 201, + "languageId": "de", + "limit": 500, + "offset": 0 + }, + "data": null, + "response": { + "status": 200, + "json": { + "data": [ + { + "data": { + "id": 900, + "translationId": 11 + } + } + ], + "pagination": { + "offset": 0, + "limit": 500 + } + } + } + }, + { + "method": "GET", + "url": "https://api.crowdin.com/api/v2/projects/618696/translations", + "params": { + "stringId": 201, + "languageId": "fr", + "limit": 500, + "offset": 0 + }, + "data": null, + "response": { + "status": 200, + "json": { + "data": [ + { + "data": { + "id": 21, + "text": "Appel", + "user": { + "id": 2, + "username": null, + "fullName": "Bob B" + } + } + } + ], + "pagination": { + "offset": 0, + "limit": 500 + } + } + } + }, + { + "method": "GET", + "url": "https://api.crowdin.com/api/v2/projects/618696/approvals", + "params": { + "stringId": 201, + "languageId": "fr", + "limit": 500, + "offset": 0 + }, + "data": null, + "response": { + "status": 200, + "json": { + "data": [], + "pagination": { + "offset": 0, + "limit": 500 + } + } + } + }, + { + "method": "GET", + "url": "https://api.crowdin.com/api/v2/projects/618696/translations", + "params": { + "stringId": 201, + "languageId": "ja", + "limit": 500, + "offset": 0 + }, + "data": null, + "response": { + "status": 200, + "json": { + "data": [], + "pagination": { + "offset": 0, + "limit": 500 + } + } + } + }, + { + "method": "GET", + "url": "https://api.crowdin.com/api/v2/projects/618696/approvals", + "params": { + "stringId": 201, + "languageId": "ja", + "limit": 500, + "offset": 0 + }, + "data": null, + "response": { + "status": 200, + "json": { + "data": [], + "pagination": { + "offset": 0, + "limit": 500 + } + } + } + }, + { + "method": "GET", + "url": "https://api.crowdin.com/api/v2/projects/618696/translations", + "params": { + "stringId": 202, + "languageId": "de", + "limit": 500, + "offset": 0 + }, + "data": null, + "response": { + "status": 200, + "json": { + "data": [ + { + "data": { + "id": 31, + "text": "Einspruch lΓ€uft", + "user": { + "id": 1, + "username": "alice", + "fullName": "Alice A" + } + } + } + ], + "pagination": { + "offset": 0, + "limit": 500 + } + } + } + }, + { + "method": "GET", + "url": "https://api.crowdin.com/api/v2/projects/618696/approvals", + "params": { + "stringId": 202, + "languageId": "de", + "limit": 500, + "offset": 0 + }, + "data": null, + "response": { + "status": 200, + "json": { + "data": [], + "pagination": { + "offset": 0, + "limit": 500 + } + } + } + }, + { + "method": "GET", + "url": "https://api.crowdin.com/api/v2/projects/618696/translations", + "params": { + "stringId": 202, + "languageId": "fr", + "limit": 500, + "offset": 0 + }, + "data": null, + "response": { + "status": 200, + "json": { + "data": [ + { + "data": { + "id": 41, + "text": "Appel en cours", + "user": { + "id": 1, + "username": "alice", + "fullName": "Alice A" + } + } + } + ], + "pagination": { + "offset": 0, + "limit": 500 + } + } + } + }, + { + "method": "GET", + "url": "https://api.crowdin.com/api/v2/projects/618696/approvals", + "params": { + "stringId": 202, + "languageId": "fr", + "limit": 500, + "offset": 0 + }, + "data": null, + "response": { + "status": 200, + "json": { + "data": [ + { + "data": { + "id": 901, + "translationId": 41 + } + } + ], + "pagination": { + "offset": 0, + "limit": 500 + } + } + } + }, + { + "method": "GET", + "url": "https://api.crowdin.com/api/v2/projects/618696/translations", + "params": { + "stringId": 202, + "languageId": "ja", + "limit": 500, + "offset": 0 + }, + "data": null, + "response": { + "status": 200, + "json": { + "data": [ + { + "data": { + "id": 51, + "text": "異議申し立て中", + "user": { + "id": 1, + "username": "alice", + "fullName": "Alice A" + } + } + } + ], + "pagination": { + "offset": 0, + "limit": 500 + } + } + } + }, + { + "method": "GET", + "url": "https://api.crowdin.com/api/v2/projects/618696/approvals", + "params": { + "stringId": 202, + "languageId": "ja", + "limit": 500, + "offset": 0 + }, + "data": null, + "response": { + "status": 200, + "json": { + "data": [], + "pagination": { + "offset": 0, + "limit": 500 + } + } + } + }, + { + "method": "POST", + "url": "https://api.crowdin.com/api/v2/projects/618696/approvals", + "params": null, + "data": "{\"translationId\": 12}", + "response": { + "status": 201, + "json": { + "data": { + "id": 950, + "translationId": 12 + } + } + } + }, + { + "method": "POST", + "url": "https://api.crowdin.com/api/v2/projects/618696/approvals", + "params": null, + "data": "{\"translationId\": 31}", + "response": { + "status": 400, + "json": { + "errors": [ + { + "error": { + "key": "translationId", + "errors": [ + { + "code": "exists", + "message": "Translation is already approved" + } + ] + } + } + ] + } + } + }, + { + "method": "POST", + "url": "https://api.crowdin.com/api/v2/projects/618696/approvals", + "params": null, + "data": "{\"translationId\": 41}", + "response": { + "status": 500, + "json": { + "error": { + "code": 500, + "message": "Internal error" + } + } + } + }, + { + "method": "POST", + "url": "https://api.crowdin.com/api/v2/projects/618696/approvals", + "params": null, + "data": "{\"translationId\": 51}", + "response": { + "status": 201, + "json": { + "data": { + "id": 951, + "translationId": 51 + } + } + } + } + ] +} \ No newline at end of file diff --git a/tests/goldens/download/output.json b/tests/goldens/download/output.json index 0f998d4..0641eda 100644 --- a/tests/goldens/download/output.json +++ b/tests/goldens/download/output.json @@ -26,8 +26,8 @@ } ], "files": { - "_non_translatable_strings.json": "{\n \"data\": [\n {\n \"data\": {\n \"id\": 7,\n \"text\": \"Session\",\n \"conceptId\": 36\n }\n },\n {\n \"data\": {\n \"id\": 8,\n \"text\": \"Lokinet\",\n \"conceptId\": 36\n }\n }\n ],\n \"pagination\": {\n \"offset\": 0,\n \"limit\": 500\n }\n}", - "_project_info.json": "{\n \"data\": {\n \"id\": 618696,\n \"identifier\": \"session-crossplatform-strings\",\n \"sourceLanguageId\": \"en\",\n \"sourceLanguage\": {\n \"id\": \"en\",\n \"name\": \"English\",\n \"editorCode\": \"en\",\n \"locale\": \"en-US\"\n },\n \"targetLanguageIds\": [\n \"de\",\n \"fr\",\n \"es-ES\"\n ],\n \"targetLanguages\": [\n {\n \"id\": \"de\",\n \"name\": \"German\",\n \"editorCode\": \"de\",\n \"locale\": \"de-DE\"\n },\n {\n \"id\": \"fr\",\n \"name\": \"French\",\n \"editorCode\": \"fr\",\n \"locale\": \"fr-FR\"\n },\n {\n \"id\": \"es-ES\",\n \"name\": \"Spanish\",\n \"editorCode\": \"es\",\n \"locale\": \"es-ES\"\n }\n ]\n }\n}", + "_non_translatable_strings.json": "{\n \"data\": [\n {\n \"data\": {\n \"id\": 7,\n \"text\": \"Session\",\n \"conceptId\": 36,\n \"createdAt\": \"2023-11-02T01:17:09+00:00\"\n }\n },\n {\n \"data\": {\n \"id\": 8,\n \"text\": \"Lokinet\",\n \"conceptId\": 36,\n \"createdAt\": \"2023-11-02T01:17:09+00:00\"\n }\n }\n ],\n \"pagination\": {\n \"offset\": 0,\n \"limit\": 500\n }\n}", + "_project_info.json": "{\n \"data\": {\n \"id\": 618696,\n \"identifier\": \"session-crossplatform-strings\",\n \"sourceLanguageId\": \"en\",\n \"sourceLanguage\": {\n \"id\": \"en\",\n \"name\": \"English\",\n \"editorCode\": \"en\",\n \"locale\": \"en-US\"\n },\n \"targetLanguageIds\": [\n \"de\",\n \"fr\",\n \"es-ES\"\n ],\n \"targetLanguages\": [\n {\n \"id\": \"de\",\n \"name\": \"German\",\n \"editorCode\": \"de\",\n \"locale\": \"de-DE\"\n },\n {\n \"id\": \"fr\",\n \"name\": \"French\",\n \"editorCode\": \"fr\",\n \"locale\": \"fr-FR\"\n },\n {\n \"id\": \"es-ES\",\n \"name\": \"Spanish\",\n \"editorCode\": \"es\",\n \"locale\": \"es-ES\"\n }\n ],\n \"createdAt\": \"2021-03-04T05:26:03+00:00\",\n \"lastActivity\": \"2026-09-24T22:10:41+00:00\"\n }\n}", "de-DE.xliff": "\n\n", "en-US.xliff": "\n\n", "es-ES.xliff": "\n\n", diff --git a/tests/goldens/download/responses.json b/tests/goldens/download/responses.json index 11ffb72..95321ce 100644 --- a/tests/goldens/download/responses.json +++ b/tests/goldens/download/responses.json @@ -52,7 +52,9 @@ "editorCode": "es", "locale": "es-ES" } - ] + ], + "createdAt": "2021-03-04T05:26:03+00:00", + "lastActivity": "2026-09-24T22:10:41+00:00" } } } @@ -159,8 +161,12 @@ }, { "method": "GET", - "url": "https://api.crowdin.com/api/v2/glossaries/407522/terms?conceptId=36&limit=500", - "params": null, + "url": "https://api.crowdin.com/api/v2/glossaries/407522/terms", + "params": { + "conceptId": 36, + "limit": 500, + "offset": 0 + }, "data": null, "response": { "status": 200, @@ -170,14 +176,16 @@ "data": { "id": 7, "text": "Session", - "conceptId": 36 + "conceptId": 36, + "createdAt": "2023-11-02T01:17:09+00:00" } }, { "data": { "id": 8, "text": "Lokinet", - "conceptId": 36 + "conceptId": 36, + "createdAt": "2023-11-02T01:17:09+00:00" } } ], diff --git a/tests/goldens/report/responses.json b/tests/goldens/report/responses.json index a38e6e3..d9cac28 100644 --- a/tests/goldens/report/responses.json +++ b/tests/goldens/report/responses.json @@ -51,7 +51,9 @@ "editorCode": "es", "locale": "es-ES" } - ] + ], + "createdAt": "2021-03-04T05:26:03+00:00", + "lastActivity": "2026-09-24T22:10:41+00:00" } } } diff --git a/tests/shared/test_discord.py b/tests/shared/test_discord.py index b96a475..cfd4019 100644 --- a/tests/shared/test_discord.py +++ b/tests/shared/test_discord.py @@ -4,7 +4,7 @@ from urllib.parse import parse_qsl, urlsplit from session_ops.shared import discord -from session_ops.shared.testing import FakeResponse, FakeSession, NoSleep +from session_ops.shared.testing import FakeResponse, FakeSession class TestComponentsWebhookUrl(unittest.TestCase): @@ -53,16 +53,14 @@ def test_does_not_post_after_a_failure(self): self.assertEqual(len(session.calls), 1) def test_first_message_failing_reports_zero(self): - session = FakeSession([FakeResponse({}, status_code=500)] * 6) - with NoSleep(): - self.assertEqual(self.post(session, [{}]), 0) + session = FakeSession([FakeResponse({}, status_code=500)]) + self.assertEqual(self.post(session, [{}]), 0) def test_an_unreachable_webhook_reports_the_prefix_rather_than_raising(self): import requests session = FakeSession([FakeResponse({}, status_code=204)] - + [requests.ConnectionError("down")] * 6) - with NoSleep(): - self.assertEqual(self.post(session, [{}, {}]), 1) + + [requests.ConnectionError("down")]) + self.assertEqual(self.post(session, [{}, {}]), 1) def test_no_messages_is_zero(self): self.assertEqual(self.post(FakeSession([]), []), 0) diff --git a/tests/shared/test_http.py b/tests/shared/test_http.py new file mode 100644 index 0000000..ad86e71 --- /dev/null +++ b/tests/shared/test_http.py @@ -0,0 +1,248 @@ +""" + uv run python -m unittest tests.shared.test_http + +Against a real server on localhost, so the retries under test are urllib3's own. +""" +import json +import socket +import threading +import time +import unittest +from datetime import datetime, timedelta, timezone +from email.utils import format_datetime +from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer +from unittest import mock + +import requests + +from session_ops.shared import http +from session_ops.shared.testing import FakeResponse, NoSleep + +DROP = "drop" + + +class Server: + """Answers each request with the next scripted (status, headers) or drops the + connection for DROP, and records what it was asked.""" + + def __init__(self, *script): + self.script, self.seen = list(script), [] + outer = self + + class Handler(BaseHTTPRequestHandler): + def log_message(self, *args): + pass + + def answer(self): + length = int(self.headers.get("content-length") or 0) + outer.seen.append((self.command, self.path, self.rfile.read(length))) + step = outer.script.pop(0) + if step == DROP: + self.close_connection = True + self.connection.shutdown(socket.SHUT_RDWR) + return + status, headers = step + body = json.dumps({"status": status}).encode() + self.send_response(status) + for name, value in headers.items(): + self.send_header(name, value) + self.send_header("content-length", str(len(body))) + self.end_headers() + self.wfile.write(body) + + do_GET = do_POST = do_PUT = answer + + self.httpd = ThreadingHTTPServer(("127.0.0.1", 0), Handler) + self.url = f"http://127.0.0.1:{self.httpd.server_port}/x" + threading.Thread(target=self.httpd.serve_forever, args=(0.01,), daemon=True).start() + + def close(self): + self.httpd.shutdown() + self.httpd.server_close() + + +def ok(**headers): + return (200, headers) + + +def status(code, **headers): + return (code, {k.replace("_", "-"): v for k, v in headers.items()}) + + +class TestSession(unittest.TestCase): + def serve(self, *script): + server = Server(*script) + self.addCleanup(server.close) + return server + + def test_returns_the_first_success_without_retrying(self): + server = self.serve(ok()) + self.assertEqual(http.Session().get(server.url).status_code, 200) + self.assertEqual(len(server.seen), 1) + + def test_retries_a_server_error_then_succeeds(self): + server = self.serve(status(500), ok()) + with NoSleep(): + self.assertEqual(http.Session(attempts=3).get(server.url).status_code, 200) + self.assertEqual(len(server.seen), 2) + + def test_does_not_retry_a_client_error(self): + server = self.serve(status(404)) + self.assertEqual(http.Session().get(server.url).status_code, 404) + self.assertEqual(len(server.seen), 1) + + def test_an_exhausted_budget_returns_the_last_response(self): + server = self.serve(*[status(503)] * 3) + with NoSleep(): + self.assertEqual(http.Session(attempts=3).get(server.url).status_code, 503) + self.assertEqual(len(server.seen), 3) + + def test_post_is_retried_too(self): + server = self.serve(status(502), ok()) + with NoSleep(): + resp = http.Session(attempts=3).post(server.url, json={"a": 1}) + self.assertEqual(resp.status_code, 200) + self.assertEqual([body for _, _, body in server.seen], [b'{"a": 1}'] * 2) + + def test_a_per_call_budget_overrides_the_sessions(self): + server = self.serve(*[status(503)] * 6) + session = http.Session(attempts=6) + with NoSleep(): + self.assertEqual(session.request("GET", server.url, attempts=2).status_code, 503) + self.assertEqual(len(server.seen), 2) + with NoSleep(): + session.request("GET", server.url, attempts=4) + self.assertEqual(len(server.seen), 6) + + def test_a_dropped_connection_is_retried(self): + server = self.serve(DROP, ok()) + with NoSleep(): + self.assertEqual(http.Session(attempts=3).get(server.url).status_code, 200) + self.assertEqual(len(server.seen), 2) + + def test_transport_failures_raise_once_the_budget_is_spent(self): + with socket.socket() as sock: + sock.bind(("127.0.0.1", 0)) + port = sock.getsockname()[1] + with NoSleep() as clock, self.assertRaises(requests.ConnectionError): + http.Session(attempts=3).get(f"http://127.0.0.1:{port}/") + self.assertEqual(clock.slept, [1.0, 2.0]) + + def test_backoff_doubles_from_one_second_and_never_sleeps_after_the_last_try(self): + server = self.serve(*[status(500)] * 8) + with NoSleep() as clock: + http.Session(attempts=8).get(server.url) + self.assertEqual(clock.slept, [1.0, 2.0, 4.0, 8.0, 16.0, 30.0, 30.0]) + + def test_numeric_retry_after_is_honoured(self): + server = self.serve(status(429, retry_after="7"), ok()) + with NoSleep() as clock: + http.Session().get(server.url) + self.assertEqual(clock.slept, [7.0]) + + def test_retry_after_zero_means_now_not_backoff(self): + server = self.serve(status(429, retry_after="0"), ok()) + with NoSleep() as clock: + http.Session().get(server.url) + self.assertEqual(clock.slept, [0.0]) + + def test_retry_after_is_capped_at_a_minute(self): + server = self.serve(status(429, retry_after="9999"), ok()) + with NoSleep() as clock: + http.Session().get(server.url) + self.assertEqual(clock.slept, [60]) + + def test_an_http_date_retry_after_is_honoured(self): + soon = format_datetime(datetime.now(timezone.utc) + timedelta(seconds=20), usegmt=True) + server = self.serve(status(503, retry_after=soon), ok()) + with NoSleep() as clock: + http.Session().get(server.url) + self.assertTrue(15 <= clock.slept[0] <= 21, clock.slept) + + def test_a_negative_retry_after_falls_back_to_backoff(self): + server = self.serve(status(503, retry_after="-30"), ok()) + with NoSleep() as clock: + self.assertEqual(http.Session().get(server.url).status_code, 200) + self.assertEqual(clock.slept, [1.0]) + + def test_githubs_reset_epoch_stands_in_for_a_missing_retry_after(self): + reset = str(int(time.time()) + 30) + server = self.serve(status(429, x_ratelimit_reset=reset), ok()) + with NoSleep() as clock: + http.Session().get(server.url) + self.assertTrue(25 <= clock.slept[0] <= 31, clock.slept) + + def test_the_timeout_defaults_to_the_sessions_and_can_be_overridden(self): + with mock.patch.object(requests.Session, "request") as sent: + session = http.Session(timeout=45) + session.get("https://x") + session.get("https://x", timeout=5) + self.assertEqual([call.kwargs["timeout"] for call in sent.call_args_list], [45, 5]) + + def test_zero_attempts_is_rejected(self): + with self.assertRaises(ValueError): + http.Session(attempts=0) + with self.assertRaises(ValueError): + http.Session().request("GET", "https://x", attempts=0) + + def test_the_limiter_paces_every_call(self): + limiter = mock.Mock() + with mock.patch.object(requests.Session, "request"): + session = http.Session(limiter=limiter) + session.get("https://x") + session.post("https://x") + self.assertEqual(limiter.acquire.call_count, 2) + + +class TestTokenBucket(unittest.TestCase): + def test_a_burst_up_to_the_rate_is_free_then_calls_wait(self): + bucket = http.TokenBucket(4) + with NoSleep() as clock: + for _ in range(4): + bucket.acquire() + self.assertEqual(clock.slept, []) + bucket.acquire() + self.assertEqual(len(clock.slept), 1) + self.assertAlmostEqual(clock.slept[0], 0.25, places=2) + + +class TestRetryAfterSeconds(unittest.TestCase): + def seconds(self, headers, default=4.0): + resp = FakeResponse({}, retry_after=None) + resp.headers = headers + return http.retry_after_seconds(resp, default) + + def test_missing_header_uses_the_default(self): + self.assertEqual(self.seconds({}), 4.0) + + def test_numeric_header_wins(self): + self.assertEqual(self.seconds({"retry-after": "12"}), 12.0) + + def test_unparseable_header_uses_the_default(self): + for raw in ("", "soon", "12s", "Wed, 32 Oct 2026 07:28:00 GMT"): + self.assertEqual(self.seconds({"retry-after": raw}), 4.0, msg=f"retry-after={raw!r}") + + def test_an_http_date_in_the_past_uses_the_default(self): + """A date already gone means a negative wait, which time.sleep() rejects.""" + self.assertEqual(self.seconds({"retry-after": "Wed, 21 Oct 2015 07:28:00 GMT"}), 4.0) + + def test_negative_and_non_finite_values_use_the_default(self): + for raw in ("-30", "-0.5", "nan", "inf", "-inf"): + self.assertEqual(self.seconds({"retry-after": raw}), 4.0, msg=f"retry-after={raw!r}") + + def test_zero_is_honoured_rather_than_replaced(self): + self.assertEqual(self.seconds({"retry-after": "0"}), 0.0) + + def test_a_reset_epoch_in_the_past_means_no_wait(self): + self.assertEqual(self.seconds({"x-ratelimit-reset": "1"}), 0.0) + + def test_retry_after_beats_the_reset_epoch(self): + reset = str(int(time.time()) + 3000) + self.assertEqual(self.seconds({"retry-after": "5", "x-ratelimit-reset": reset}), 5.0) + + def test_an_unparseable_reset_epoch_uses_the_default(self): + self.assertEqual(self.seconds({"x-ratelimit-reset": "?"}), 4.0) + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/shared/test_retry.py b/tests/shared/test_retry.py deleted file mode 100644 index d9ccef5..0000000 --- a/tests/shared/test_retry.py +++ /dev/null @@ -1,168 +0,0 @@ -""" - uv run python -m unittest tests.shared.test_retry -""" -import time -import unittest -from email.utils import format_datetime -from datetime import datetime, timedelta, timezone - -import requests - -from session_ops.shared import retry -from session_ops.shared.testing import FakeResponse, FakeSession, NoSleep - - -class TestRequestWithRetry(unittest.TestCase): - def test_returns_the_first_success_without_retrying(self): - session = FakeSession([FakeResponse({"ok": True})]) - resp = retry.request_with_retry(session, "GET", "https://x") - self.assertEqual(resp.json(), {"ok": True}) - self.assertEqual(len(session.calls), 1) - - def test_retries_a_server_error_then_succeeds(self): - session = FakeSession([ - FakeResponse({}, status_code=500), - FakeResponse({"ok": True}), - ]) - resp = retry.request_with_retry(session, "GET", "https://x", attempts=3) - self.assertEqual(resp.json(), {"ok": True}) - self.assertEqual(len(session.calls), 2) - - def test_does_not_retry_a_client_error(self): - session = FakeSession([FakeResponse({}, status_code=404)]) - resp = retry.request_with_retry(session, "GET", "https://x") - self.assertEqual(resp.status_code, 404) - self.assertEqual(len(session.calls), 1) - - def test_gives_up_after_the_attempt_budget(self): - session = FakeSession([FakeResponse({}, status_code=503)] * 3) - resp = retry.request_with_retry(session, "GET", "https://x", attempts=3) - self.assertEqual(resp.status_code, 503) - self.assertEqual(len(session.calls), 3) - - def test_transport_failures_retry_then_raise_when_exhausted(self): - session = FakeSession([requests.ConnectionError("boom")] * 3) - with NoSleep(): - with self.assertRaises(requests.ConnectionError): - retry.request_with_retry(session, "GET", "https://x", attempts=3) - self.assertEqual(len(session.calls), 3) - - def test_a_transport_failure_can_recover_on_a_later_attempt(self): - session = FakeSession([requests.Timeout("slow"), FakeResponse({"ok": True})]) - with NoSleep(): - resp = retry.request_with_retry(session, "GET", "https://x", attempts=3) - self.assertEqual(resp.json(), {"ok": True}) - self.assertEqual(len(session.calls), 2) - - def test_no_sleep_after_the_final_attempt(self): - """Sleeping after the last try only delays the caller β€” nothing follows it.""" - session = FakeSession([FakeResponse({}, status_code=503)] * 3) - with NoSleep() as clock: - retry.request_with_retry(session, "GET", "https://x", attempts=3) - self.assertEqual(len(clock.slept), 2) # 3 attempts, 2 gaps - - def test_numeric_retry_after_is_honoured(self): - session = FakeSession([ - FakeResponse({}, status_code=429, retry_after="7"), - FakeResponse({"ok": True}), - ]) - with NoSleep() as clock: - retry.request_with_retry(session, "GET", "https://x", attempts=3) - self.assertEqual(clock.slept, [7.0]) - - def test_retry_after_is_capped(self): - session = FakeSession([ - FakeResponse({}, status_code=429, retry_after="9999"), - FakeResponse({"ok": True}), - ]) - with NoSleep() as clock: - retry.request_with_retry(session, "GET", "https://x", attempts=3) - self.assertEqual(clock.slept, [60]) - - def test_an_http_date_retry_after_is_honoured(self): - """RFC 9110 allows an HTTP-date here as well as a count of seconds.""" - soon = format_datetime(datetime.now(timezone.utc) + timedelta(seconds=20), usegmt=True) - session = FakeSession([ - FakeResponse({}, status_code=503, retry_after=soon), - FakeResponse({"ok": True}), - ]) - with NoSleep() as clock: - resp = retry.request_with_retry(session, "GET", "https://x", attempts=3) - self.assertEqual(resp.json(), {"ok": True}) - self.assertEqual(len(clock.slept), 1) - self.assertTrue(15 <= clock.slept[0] <= 21, clock.slept) - - def test_the_timeout_is_forwarded_and_defaults_to_thirty_seconds(self): - session = FakeSession([FakeResponse({}), FakeResponse({})]) - retry.request_with_retry(session, "GET", "https://x") - retry.request_with_retry(session, "GET", "https://x", timeout=60) - self.assertEqual([kw["timeout"] for _, _, kw in session.calls], [30, 60]) - - def test_zero_attempts_is_rejected_rather_than_unbound(self): - with self.assertRaises(ValueError): - retry.request_with_retry(FakeSession([]), "GET", "https://x", attempts=0) - - def test_a_negative_retry_after_does_not_crash_a_real_retry_loop(self): - session = FakeSession([ - FakeResponse({}, status_code=503, retry_after="-30"), - FakeResponse({"ok": True}), - ]) - with NoSleep() as clock: - resp = retry.request_with_retry(session, "GET", "https://x", attempts=3) - self.assertEqual(resp.json(), {"ok": True}) - self.assertTrue(all(s >= 0 for s in clock.slept), clock.slept) - - -class TestRetryAfterSeconds(unittest.TestCase): - def seconds(self, headers, default=4.0): - resp = FakeResponse({}, retry_after=None) - resp.headers = headers - return retry.retry_after_seconds(resp, default) - - def test_missing_header_uses_the_default(self): - self.assertEqual(self.seconds({}), 4.0) - - def test_numeric_header_wins(self): - self.assertEqual(self.seconds({"retry-after": "12"}), 12.0) - - def test_unparseable_header_uses_the_default(self): - for raw in ("", "soon", "12s", "Wed, 32 Oct 2026 07:28:00 GMT"): - self.assertEqual(self.seconds({"retry-after": raw}), 4.0, msg=f"retry-after={raw!r}") - - def test_an_http_date_in_the_past_uses_the_default(self): - """A date already gone means a negative wait, which time.sleep() rejects.""" - self.assertEqual(self.seconds({"retry-after": "Wed, 21 Oct 2015 07:28:00 GMT"}), 4.0) - - def test_an_http_date_is_the_seconds_until_it(self): - soon = format_datetime(datetime.now(timezone.utc) + timedelta(seconds=30), usegmt=True) - self.assertTrue(25 <= self.seconds({"retry-after": soon}) <= 31) - - def test_negative_and_non_finite_values_use_the_default(self): - """time.sleep() rejects a negative or NaN duration, so passing one through - would crash the run on a hostile or buggy Retry-After header.""" - for raw in ("-30", "-0.5", "nan", "inf", "-inf"): - self.assertEqual(self.seconds({"retry-after": raw}), 4.0, msg=f"retry-after={raw!r}") - - def test_zero_is_honoured_rather_than_replaced(self): - """Zero is a valid instruction to retry immediately, not a missing value.""" - self.assertEqual(self.seconds({"retry-after": "0"}), 0.0) - - def test_a_primary_rate_limit_falls_back_to_the_reset_epoch(self): - """GitHub's primary limit sends x-ratelimit-reset and no Retry-After.""" - reset = str(int(time.time()) + 30) - seconds = self.seconds({"x-ratelimit-reset": reset}) - self.assertTrue(25 <= seconds <= 31, seconds) - - def test_a_reset_epoch_in_the_past_means_no_wait(self): - self.assertEqual(self.seconds({"x-ratelimit-reset": "1"}), 0.0) - - def test_retry_after_beats_the_reset_epoch(self): - reset = str(int(time.time()) + 3000) - self.assertEqual(self.seconds({"retry-after": "5", "x-ratelimit-reset": reset}), 5.0) - - def test_an_unparseable_reset_epoch_uses_the_default(self): - self.assertEqual(self.seconds({"x-ratelimit-reset": "?"}), 4.0) - - -if __name__ == "__main__": - unittest.main() diff --git a/tests/zendesk/test_note_reply.py b/tests/zendesk/test_note_reply.py index 0f80624..354dda5 100644 --- a/tests/zendesk/test_note_reply.py +++ b/tests/zendesk/test_note_reply.py @@ -497,12 +497,7 @@ def test_a_dry_run_clears_nothing(self): def test_a_failure_to_clear_is_not_fatal(self): """The tag is a dashboard light, not the work.""" - session = fake_session(*[FakeResponse({}, status_code=500), - FakeResponse({}, status_code=500), - FakeResponse({}, status_code=500), - FakeResponse({}, status_code=500), - FakeResponse({}, status_code=500), - FakeResponse({}, status_code=500)]) + session = fake_session(FakeResponse({}, status_code=500)) note_reply.clear_queued(session, "sub", 7) # must not raise diff --git a/tests/zendesk/test_resolve_reviews.py b/tests/zendesk/test_resolve_reviews.py index f8e0aa6..77db79e 100644 --- a/tests/zendesk/test_resolve_reviews.py +++ b/tests/zendesk/test_resolve_reviews.py @@ -19,8 +19,8 @@ from session_ops.zendesk import resolve_reviews from session_ops.zendesk import triage -from tests.zendesk.test_triage import ( - ROOT, FakeResponse, FakeSession, NoSleep, unit_commands) +from session_ops.shared.testing import FakeResponse, FakeSession, NoSleep +from tests.zendesk.test_triage import ROOT, unit_commands def review(ticket_id, stars=5, channel="any_channel", subject=None): @@ -229,7 +229,7 @@ def test_a_job_that_never_finishes_exits(self): resolve_reviews.wait_for_job(session, "acme", "job1", timeout=0) def test_an_http_error_exits(self): - session = FakeSession([FakeResponse({}, status_code=500)] * 8) + session = FakeSession([FakeResponse({}, status_code=500)]) with NoSleep(): with self.assertRaises(SystemExit): resolve_reviews.wait_for_job(session, "acme", "job1") diff --git a/tests/zendesk/test_triage.py b/tests/zendesk/test_triage.py index 4667ddc..53c14ce 100644 --- a/tests/zendesk/test_triage.py +++ b/tests/zendesk/test_triage.py @@ -23,7 +23,7 @@ from session_ops.zendesk import triage from session_ops.shared import discord from session_ops.shared.testing import ( - FakeResponse, FakeSession, NoSleep, NonJsonResponse, Patched) + FakeResponse, FakeSession, NonJsonResponse, Patched) STAMP = "%Y-%m-%dT%H:%M:%SZ" @@ -306,9 +306,7 @@ def test_a_failed_sideload_leaves_the_ticket_alone(self): for response in (FakeResponse({}, status_code=500), NonJsonResponse(), requests.ConnectionError("unreachable")): with self.subTest(response=type(response).__name__): - with NoSleep(): - triage.hydrate_requester_activity( - FakeSession([response] * 2), "acme", tickets) + triage.hydrate_requester_activity(FakeSession([response]), "acme", tickets) self.assertNotIn("requester_updated_at", tickets[0]) self.assertEqual(triage.activity_key(tickets[0]), "X") @@ -977,9 +975,8 @@ def test_hydration_survives_an_api_failure(self): def test_hydration_survives_a_transport_failure(self): """An unreachable comments endpoint must not abort the whole digest.""" row = ticket(1, subject="Conversation with x", description="Conversation with x") - session = FakeSession([requests.ConnectionError("unreachable")] * 2) - with NoSleep(): - self.assertEqual(triage.hydrate_descriptions(session, "acme", [row]), 0) + session = FakeSession([requests.ConnectionError("unreachable")]) + self.assertEqual(triage.hydrate_descriptions(session, "acme", [row]), 0) self.assertEqual(row["description"], "Conversation with x") def test_hydration_survives_a_non_json_body(self): @@ -995,11 +992,10 @@ def test_one_unreachable_ticket_does_not_block_the_next(self): ticket(2, subject="Conversation with b", description="Conversation with b"), ] session = FakeSession([ - requests.ConnectionError("unreachable"), requests.ConnectionError("unreachable"), + requests.ConnectionError("unreachable"), FakeResponse({"comments": [{"body": "Cannot log in since the update"}]}), ]) - with NoSleep(): - self.assertEqual(triage.hydrate_descriptions(session, "acme", rows), 1) + self.assertEqual(triage.hydrate_descriptions(session, "acme", rows), 1) self.assertIn("Cannot log in", rows[1]["description"]) def test_hydration_joins_every_informative_comment(self): @@ -1549,15 +1545,14 @@ def test_failure_is_non_fatal(self): def test_uses_a_short_retry_budget(self): """A full 6-attempt backoff would stall the digest ~60s for optional data.""" - session = FakeSession([FakeResponse({}, status_code=500)] * 6) + session = FakeSession([FakeResponse({}, status_code=500)]) triage.fetch_total_unsolved(session, "acme") - self.assertEqual(len(session.calls), 2) + self.assertEqual(session.calls[0][2]["attempts"], 2) def test_a_transport_failure_is_non_fatal_too(self): - """request_with_retry re-raises once its budget is spent; None is documented.""" - session = FakeSession([requests.ConnectionError("no route")] * 2) - with NoSleep(): - self.assertIsNone(triage.fetch_total_unsolved(session, "acme")) + """The session re-raises once its budget is spent; None is documented.""" + session = FakeSession([requests.ConnectionError("no route")]) + self.assertIsNone(triage.fetch_total_unsolved(session, "acme")) def test_a_non_json_body_is_non_fatal(self): """A 200 with an HTML error page (proxy, maintenance) must not abort the run.""" @@ -1921,9 +1916,8 @@ class Resp: def json(): return {"comments": comments} - with Patched(triage, request_with_retry=lambda *a, **k: Resp()): - return triage.conversation_turns(object(), "acme", - {"id": 1, "requester_id": requester_id}) + return triage.conversation_turns(FakeSession([Resp()]), "acme", + {"id": 1, "requester_id": requester_id}) def comment(self, body, author_id=5, public=True, created_at="2026-08-28T00:22:38Z"): return {"body": body, "author_id": author_id, "public": public, diff --git a/uv.lock b/uv.lock index 1d484e0..1b85f17 100644 --- a/uv.lock +++ b/uv.lock @@ -285,6 +285,31 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/d1/d6/3965ed04c63042e047cb6a3e6ed1a63a35087b6a609aa3a15ed8ac56c221/colorama-0.4.6-py2.py3-none-any.whl", hash = "sha256:4f1d9991f5acc0ca119f9d443620b77f9d6b33703e51011c16baf57afb285fc6", size = 25335, upload-time = "2022-10-25T02:36:20.889Z" }, ] +[[package]] +name = "crowdin-api-client" +version = "1.29.0" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "deprecated" }, + { name = "requests" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/2c/a3/b357800b5996d5475de875187327f917b00fd34f614b428ca0d88bcf58e2/crowdin_api_client-1.29.0.tar.gz", hash = "sha256:becdb4103e0d3d6e2a5b5c129ad7850f0f8631da24bb46703c2b2d7befa4a5d7", size = 74666, upload-time = "2026-09-03T08:15:13.213Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/e0/23/184b601011cb42d8a740eb6174406234d3d4fa40bf8aceec3155464c7a64/crowdin_api_client-1.29.0-py3-none-any.whl", hash = "sha256:2f16a27ff71a518b35f0b0209336f1a044aef26001c8bbcd546a6d18d2d9e3c2", size = 115738, upload-time = "2026-09-03T08:15:11.968Z" }, +] + +[[package]] +name = "deprecated" +version = "1.3.1" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "wrapt" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/49/85/12f0a49a7c4ffb70572b6c2ef13c90c88fd190debda93b23f026b25f9634/deprecated-1.3.1.tar.gz", hash = "sha256:b1b50e0ff0c1fddaa5708a2c6b0a6588bb09b892825ab2b214ac9ea9d92a5223", size = 2932523, upload-time = "2025-10-30T08:19:02.757Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/84/d0/205d54408c08b13550c733c4b85429e7ead111c7f0014309637425520a9a/deprecated-1.3.1-py2.py3-none-any.whl", hash = "sha256:597bfef186b6f60181535a29fbe44865ce137a5079f295b479886c82729d5f3f", size = 11298, upload-time = "2025-10-30T08:19:00.758Z" }, +] + [[package]] name = "fastapi" version = "0.141.1" @@ -539,6 +564,7 @@ source = { editable = "." } dependencies = [ { name = "babel" }, { name = "colorama" }, + { name = "crowdin-api-client" }, { name = "fastapi" }, { name = "pynacl" }, { name = "requests" }, @@ -555,6 +581,7 @@ dev = [ requires-dist = [ { name = "babel", specifier = "==2.17.0" }, { name = "colorama", specifier = ">=0.4.6" }, + { name = "crowdin-api-client", specifier = ">=1.29.0" }, { name = "fastapi", specifier = ">=0.141.1" }, { name = "pynacl", specifier = ">=1.6.2" }, { name = "requests", specifier = ">=2.32.3" }, @@ -631,3 +658,78 @@ sdist = { url = "https://files.pythonhosted.org/packages/5d/ad/04bbb797c84fc1f26 wheels = [ { url = "https://files.pythonhosted.org/packages/76/18/0eea75741ee812e9f598b687619ce2454f6c3a1c5cd21ea990ec6bd26f45/uvicorn-0.53.0-py3-none-any.whl", hash = "sha256:e8dca71ec86dce5f04e333f0d56cdedf942446e6643b9cea1af0d6d3a02cb03e", size = 87081, upload-time = "2026-09-14T07:44:22.179Z" }, ] + +[[package]] +name = "wrapt" +version = "2.4.1" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/42/a6/6375d56c44d590ef24acf0f8f5bf7ed768ff7a510b959306ec412611e90f/wrapt-2.4.1.tar.gz", hash = "sha256:fd6390aab9e8aa40c52eff3c180f098e8d9f5894b1fd4c4fd2c207067b33ed16", size = 164597, upload-time = "2026-09-10T23:12:16.811Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/9f/1f/a2f3225c5ecf522684c1d051aea8ce8253f240e55be75826b787777afd6c/wrapt-2.4.1-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:7e86fbc2ac8a363ea04abf631fad82720e16b17a25020f32dbe9b24a2ed2b0e3", size = 98890, upload-time = "2026-09-10T23:10:07.876Z" }, + { url = "https://files.pythonhosted.org/packages/68/6c/eb45660fd4d92cce11ec923f55bb2e647a6c18d30e53734eb07a3c530e31/wrapt-2.4.1-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:24389748f0b9d5b67e478fad4fc8b3f1108422ef80716e48eead6cebcebbff08", size = 98742, upload-time = "2026-09-10T23:10:09.236Z" }, + { url = "https://files.pythonhosted.org/packages/a3/4f/17a89a580cb0082e61b8375074d5c9e5d38e4aa83ee19b6174ee472d17c2/wrapt-2.4.1-cp312-cp312-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl", hash = "sha256:30d11c289b013bf384ff1a1a6553f150d0b855901708a9bef667a5680f8247c9", size = 236301, upload-time = "2026-09-10T23:10:11.039Z" }, + { url = "https://files.pythonhosted.org/packages/01/ca/4700eb008a34bf02de328806ddde15fc84c8d1e65d3dcafb92a935a50319/wrapt-2.4.1-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:9356dbb59199a0e4709de35fa4a1ac1a88ef6da99711a397f5b009233faff326", size = 237805, upload-time = "2026-09-10T23:10:12.594Z" }, + { url = "https://files.pythonhosted.org/packages/75/5d/26c1740299b29e190d5f4b4a99eb001401042a9d9ab338e3f8e1ce140ecb/wrapt-2.4.1-cp312-cp312-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:8342f332dada211f64b74609e332d727b13315e9a83177f7918bf68c59f815f2", size = 217037, upload-time = "2026-09-10T23:10:14.028Z" }, + { url = "https://files.pythonhosted.org/packages/3a/55/ec72991153a2ae8b40238bc44cec7c3ddf7706ef6e2d314b0c6f5c7febce/wrapt-2.4.1-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:2f86e328c482bc5383b4eda5094be0bed3617fc3076aa9225ff1a9eb6372de9b", size = 234659, upload-time = "2026-09-10T23:10:15.384Z" }, + { url = "https://files.pythonhosted.org/packages/51/32/7cfa1e070dcda76ea56a3e252341cba1cd1e9412baf23cd7adfebf1114e2/wrapt-2.4.1-cp312-cp312-musllinux_1_2_riscv64.whl", hash = "sha256:4dc92697444ee380544fbb43c86612d8486529aadf917c22b5524141d4af074c", size = 214590, upload-time = "2026-09-10T23:10:16.744Z" }, + { url = "https://files.pythonhosted.org/packages/79/10/248841cb30107f6f32c53a02662e1c3e0c7c06bea0b8ebfaaee94885dcee/wrapt-2.4.1-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:edd03758a7578526642508b8833d43496fdfba0f64e0025dfca153a7c1777735", size = 225103, upload-time = "2026-09-10T23:10:18.346Z" }, + { url = "https://files.pythonhosted.org/packages/90/02/5b2bf7b35b008a39939a2908e85dba3b867596e535fc9f12ddf3ba1fcaf6/wrapt-2.4.1-cp312-cp312-win32.whl", hash = "sha256:5d83e412665aeb1e854eefbf1564d0d67872d9994b502a0bce96e6ff7f4970b7", size = 93441, upload-time = "2026-09-10T23:10:19.81Z" }, + { url = "https://files.pythonhosted.org/packages/d4/2a/47be56772bfb07ef242d6e924049688e38384af2b2fc99b0f31180988448/wrapt-2.4.1-cp312-cp312-win_amd64.whl", hash = "sha256:b4e7efdd476ac631a0181551fd9aace844765ea3ce2b5133b194fae4421e8ad0", size = 98808, upload-time = "2026-09-10T23:10:21.084Z" }, + { url = "https://files.pythonhosted.org/packages/d7/ec/40e4c9735626afd1dc0eeb310310278361f192800503cda0f5b3d8d24db4/wrapt-2.4.1-cp312-cp312-win_arm64.whl", hash = "sha256:38819761401baa2d11916d7265b82f23265f8fe5a31c431dd7c24a8863c65f88", size = 95240, upload-time = "2026-09-10T23:10:22.39Z" }, + { url = "https://files.pythonhosted.org/packages/5c/1a/9b5aa3c2391aa6d00fffa085c2219e8fc91cd4d2b9b080d2b4e4b6b93f42/wrapt-2.4.1-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:55f36bb1461f93beaf18d818e568b5de343dd8fade7456773d201a38ee723bd3", size = 98597, upload-time = "2026-09-10T23:10:23.723Z" }, + { url = "https://files.pythonhosted.org/packages/fe/07/dc98150c2f9ee5b5fcbd841765e178ea6cc6c43733ab8d1f5181a4fb9f3d/wrapt-2.4.1-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:53e15cd74bd6b84d7fa90b93dda7334d85f4641fae97632de8aa61d268dfd145", size = 98842, upload-time = "2026-09-10T23:10:25.109Z" }, + { url = "https://files.pythonhosted.org/packages/08/c2/0e772a570e8d75c1b3ec45930a3023492fa68223f8f5e3485af4844c10c3/wrapt-2.4.1-cp313-cp313-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl", hash = "sha256:be6cdd7121adc89a6f52e3c2f4e26a2d4dcdc1c0fde47e3156234db8939e4cdc", size = 234642, upload-time = "2026-09-10T23:10:26.644Z" }, + { url = "https://files.pythonhosted.org/packages/76/25/4ce4d02dd95ff9ed972a2fc396d04fec636daa5a4ac18cc73a3dc20aa6c3/wrapt-2.4.1-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:03b5598edd435373278731d0d53449ce7a9626bc48d5548e4a71124ee3e526a1", size = 235484, upload-time = "2026-09-10T23:10:28.098Z" }, + { url = "https://files.pythonhosted.org/packages/fd/80/436ef1df620ef8edd9ef057b4d55a0cd704435ff66852a0ef26cbaa02a58/wrapt-2.4.1-cp313-cp313-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:fdc997819a6df4c65bdb0a1c5601c98b479ee34cc2f94ffa98a767034ad6366d", size = 214371, upload-time = "2026-09-10T23:10:29.841Z" }, + { url = "https://files.pythonhosted.org/packages/6a/2c/cc5b7503843399087db3106a7cf60d60d0900f69539e96148b9fff116291/wrapt-2.4.1-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:3bfc6907ebed560d2d3f677c3b17bf6199679163b6c6e475035c9dca497d1697", size = 232347, upload-time = "2026-09-10T23:10:31.511Z" }, + { url = "https://files.pythonhosted.org/packages/86/fb/17668b1ca572c44b458c09d76064d35f5f57d5ac7629248871604bef816c/wrapt-2.4.1-cp313-cp313-musllinux_1_2_riscv64.whl", hash = "sha256:a0c3b217332cf0c4df085fec41c126bf7507d6c1ca0efb3ba8d2b3fd234d4e73", size = 212792, upload-time = "2026-09-10T23:10:32.968Z" }, + { url = "https://files.pythonhosted.org/packages/b8/2b/0c5de10d7259e07c478c44bf2fbe179c6878727d09edf0632b97884801db/wrapt-2.4.1-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:a23b89621cfeb3329b1a290596bf402e61d7d5a647a65ca8ea735e48771b71d4", size = 223585, upload-time = "2026-09-10T23:10:34.43Z" }, + { url = "https://files.pythonhosted.org/packages/aa/36/013dce1c687f8f1c87b1e78f403d04c80ae9b2ae77de4ddba16069a3e7d1/wrapt-2.4.1-cp313-cp313-win32.whl", hash = "sha256:bc67d4872af5ab2dc1b88904097b92ac00e7658fdf010dee36807807fe882ac4", size = 93425, upload-time = "2026-09-10T23:10:35.901Z" }, + { url = "https://files.pythonhosted.org/packages/b7/1d/d53bcf5910209a45191c8bc173ff0e00830416547d8038772dc444932ee0/wrapt-2.4.1-cp313-cp313-win_amd64.whl", hash = "sha256:1fe758b9c2d49138231ec3efabd106fec665f86fec50b3d02d7edd75f08a69ca", size = 98569, upload-time = "2026-09-10T23:10:37.316Z" }, + { url = "https://files.pythonhosted.org/packages/e8/1f/759d0c522918f9dfb620136622d8e1ce619570adda0de8fa2cfbb55cbb86/wrapt-2.4.1-cp313-cp313-win_arm64.whl", hash = "sha256:0c974c36e8205255a3947dad9c2fe000431b57dd522946e56c192174a7f92a0f", size = 95272, upload-time = "2026-09-10T23:10:38.657Z" }, + { url = "https://files.pythonhosted.org/packages/08/05/ed5aa8991c5e9969e2ca17f0e44eb324a9757f44fa982bfc13844cfe9e1d/wrapt-2.4.1-cp314-cp314-macosx_10_15_x86_64.whl", hash = "sha256:c6c35541cc729964c65c2b9b1f9cf317811039abc2da13b4557f20f21cdc292b", size = 98876, upload-time = "2026-09-10T23:10:40.078Z" }, + { url = "https://files.pythonhosted.org/packages/f1/5f/f8bf07c3b9a2ad01d28d5ca419e28819bb16937d129167d97c446c05875d/wrapt-2.4.1-cp314-cp314-macosx_11_0_arm64.whl", hash = "sha256:6b9df84f0a96763159cccb8e3b0ed83cc950c7c8bf82d6e45428372a805b3224", size = 99054, upload-time = "2026-09-10T23:10:41.484Z" }, + { url = "https://files.pythonhosted.org/packages/f5/7a/518e5f3ebd18472652e5332bf37728cd6634ff81a0aa568969d05cc66099/wrapt-2.4.1-cp314-cp314-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl", hash = "sha256:78af62413095d0a57077606654ce85e273deda8e2bfa28fdb04257b962d94ef2", size = 237430, upload-time = "2026-09-10T23:10:43.007Z" }, + { url = "https://files.pythonhosted.org/packages/8a/2b/503835d183c1ca99268e0b5a98af8f487d528ebabb124f4003cd96e73b1f/wrapt-2.4.1-cp314-cp314-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:d60702ebc914d0bb01aa48f5c1785ceaaaa505e0a841b444a69d6ceb5de4097e", size = 237988, upload-time = "2026-09-10T23:10:44.466Z" }, + { url = "https://files.pythonhosted.org/packages/fa/9d/a11a3e1eb18ae1d9adef1c39f8e8e36fd5b86db33938bebf8e6bc5fd9b06/wrapt-2.4.1-cp314-cp314-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:8c4b44e4be7680fc496816e824b6ed134d781c4da296e44b75399196e5c248f1", size = 218554, upload-time = "2026-09-10T23:10:45.969Z" }, + { url = "https://files.pythonhosted.org/packages/e1/61/61a6f983737ba81008561ab634373bba25b986761d3a0f0aded8352ae3e8/wrapt-2.4.1-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:fb5b3f94258bcf71db902795f4a71151c7a74d9fe21fffa00752d2bd286866f8", size = 235476, upload-time = "2026-09-10T23:10:47.528Z" }, + { url = "https://files.pythonhosted.org/packages/42/68/acb7cba46e0f662eaae421487807d2fed28ba52df9a00a3f04db16675e38/wrapt-2.4.1-cp314-cp314-musllinux_1_2_riscv64.whl", hash = "sha256:f1556a96b20d5bfdc9cb5dcd0a3ec65cbbac8a4eebcd3cd34efdc91c9519f660", size = 216454, upload-time = "2026-09-10T23:10:49.126Z" }, + { url = "https://files.pythonhosted.org/packages/8b/52/3a4dab8d4803df595de82f19775535d28c036746340c1b498fc8ecba39aa/wrapt-2.4.1-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:094a606d0bf1c4b847b3a743d22fc69cb164015b8c70cf6f20a534d30889ed47", size = 225346, upload-time = "2026-09-10T23:10:51.052Z" }, + { url = "https://files.pythonhosted.org/packages/5a/48/d72d051757fb7955021d8174014679f951fd5067187d2faed95520e1f8d6/wrapt-2.4.1-cp314-cp314-win32.whl", hash = "sha256:b0ee076be124406a7f97ca663c4a3ba32b6bcdd9102ca82497feaca79e9cb33c", size = 93870, upload-time = "2026-09-10T23:10:52.495Z" }, + { url = "https://files.pythonhosted.org/packages/a1/00/1a1bf4d8b60b9e7ac009969595438549ae6e33b2e3e174b78d26a833aad3/wrapt-2.4.1-cp314-cp314-win_amd64.whl", hash = "sha256:d2d6f9abaa52de05090a2b4a4c1d0e858a268c4de69eec277506af9fbcccff91", size = 98910, upload-time = "2026-09-10T23:10:53.878Z" }, + { url = "https://files.pythonhosted.org/packages/1d/59/b3169c3bcdcff70a6d02ed2b6366097b083d31391d616c407efe9a943a82/wrapt-2.4.1-cp314-cp314-win_arm64.whl", hash = "sha256:3152b2e94d733a9bd70dbd1c95f148266f079a70e9ffca2a9c5dc421ade1b4e6", size = 96013, upload-time = "2026-09-10T23:10:55.607Z" }, + { url = "https://files.pythonhosted.org/packages/b1/91/d29a063c2ca6e779305cd444dd5acaac60833d73b2d94c7b1d0820c27e3e/wrapt-2.4.1-cp314-cp314t-macosx_10_15_x86_64.whl", hash = "sha256:37ff91b390400463ecd4d080ea823314510b6843ac53e6e35cc09bba1805d466", size = 102069, upload-time = "2026-09-10T23:10:57.179Z" }, + { url = "https://files.pythonhosted.org/packages/72/26/7435de516099b528e2232770b459d6706002736f41d8f092bbb11ea01575/wrapt-2.4.1-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:e0a518cac3e789443af54fc77f23e66f17d80192c281b160b42058f11fabccc5", size = 102647, upload-time = "2026-09-10T23:10:58.852Z" }, + { url = "https://files.pythonhosted.org/packages/7b/a7/cca7fe0f26aa11cbbf09b2e6f4f774db167feeca43ad22fa1772afdccbad/wrapt-2.4.1-cp314-cp314t-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl", hash = "sha256:9ecdeb8a1ec13397421e6f925412186bd87ab86d63517e6825b6d7bccf781f26", size = 275614, upload-time = "2026-09-10T23:11:00.464Z" }, + { url = "https://files.pythonhosted.org/packages/77/22/f3b5f4428ae679b1e2c0d4833519045d8983a9df0ad24df96b195d44dd1b/wrapt-2.4.1-cp314-cp314t-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:c15c4ede3fde08723cabab0a892d4b75d35b5a6f0a51c84e6542ac0bdc0507aa", size = 287064, upload-time = "2026-09-10T23:11:02.111Z" }, + { url = "https://files.pythonhosted.org/packages/c2/b8/24274ec01b6ab6d0672f4312bfa29b5f1ffd9d56f16c8da67d5683883956/wrapt-2.4.1-cp314-cp314t-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:e32c5951c36fed88b6c603dc0bab209a62dc3217bd8762413634725fe28f2f3c", size = 255440, upload-time = "2026-09-10T23:11:03.746Z" }, + { url = "https://files.pythonhosted.org/packages/8b/df/c63a36f0f03d70c7b92f60ec1ad3757088167914e1a262f5dcc7df233d51/wrapt-2.4.1-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:898513db90d55a4ed3009312d41c12932b8163edbae3535280046d47aaff774f", size = 281681, upload-time = "2026-09-10T23:11:05.41Z" }, + { url = "https://files.pythonhosted.org/packages/bc/c7/4930b6a369d9b3da59f5ada4ae9e659e2d4a68fe72c5d1ce15e5bead2d24/wrapt-2.4.1-cp314-cp314t-musllinux_1_2_riscv64.whl", hash = "sha256:42d01574bd4bcafc3476e77a95c6c0dd6167101991401325fc5591e2db21a91d", size = 252409, upload-time = "2026-09-10T23:11:06.867Z" }, + { url = "https://files.pythonhosted.org/packages/68/d7/f6bc5703061598c4a2ad4f7a2efce696782be6e9d5afed714a59f5902ead/wrapt-2.4.1-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:68a403adbeb4dd2654d6d108e43e69f0f90e6d34cb7588e0e6589109f6985e67", size = 270559, upload-time = "2026-09-10T23:11:08.385Z" }, + { url = "https://files.pythonhosted.org/packages/68/a3/952337a153e634f530079334c47dcc935b20f8ea22f364729fc2a8cf1821/wrapt-2.4.1-cp314-cp314t-win32.whl", hash = "sha256:7cb3035b332bc9d21478600ba82c7c71e2d074ed9b4e76364297f54598792227", size = 96241, upload-time = "2026-09-10T23:11:09.862Z" }, + { url = "https://files.pythonhosted.org/packages/09/28/48f0651547a125dd84e312b86268d4150c565a29f81ff9ca6fd81eba4be4/wrapt-2.4.1-cp314-cp314t-win_amd64.whl", hash = "sha256:707d2bef68deddd0fc74a81103d91b286a69ac5a9ff0f0a6dad66f8787e86697", size = 102655, upload-time = "2026-09-10T23:11:11.388Z" }, + { url = "https://files.pythonhosted.org/packages/21/a6/977441bed5e5afbffe7a789067ac2f5fd2e9f848884122a23db9ccae6a70/wrapt-2.4.1-cp314-cp314t-win_arm64.whl", hash = "sha256:28cb1c2713b4377bf03ddcb3e76d8216e4bb334199066c6122be7eed2f72de8c", size = 98495, upload-time = "2026-09-10T23:11:12.952Z" }, + { url = "https://files.pythonhosted.org/packages/7c/1c/03bcbc3dbf6ac11231f0434f2494aa7b80657b130be1a1a639192a3e44c9/wrapt-2.4.1-cp315-cp315-macosx_10_15_x86_64.whl", hash = "sha256:96a5023fa63ca2f095f7776c8bfaa1694547577f762646c375abafd8f8ae649b", size = 98878, upload-time = "2026-09-10T23:11:14.506Z" }, + { url = "https://files.pythonhosted.org/packages/87/26/5ad9bd421b8cdce0b2227a61f09586423765da3c284f8bbf0f69fc149fdf/wrapt-2.4.1-cp315-cp315-macosx_11_0_arm64.whl", hash = "sha256:2b21924949dedc3ac63725e09b9b0a130e771975f03432789344ed3422c46008", size = 99090, upload-time = "2026-09-10T23:11:16.101Z" }, + { url = "https://files.pythonhosted.org/packages/a1/c7/fa768261966d587650004ac37bef541540014b1fe23fbf4919c497ebc98c/wrapt-2.4.1-cp315-cp315-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl", hash = "sha256:2f725af353bb3319c528ee69dbff838599426974288b281a3258d5397b18cb63", size = 237803, upload-time = "2026-09-10T23:11:17.738Z" }, + { url = "https://files.pythonhosted.org/packages/0f/37/9c1c876bd88fef8d065b3e8bf125cad9291d70ed3695cedb41b36b2453ec/wrapt-2.4.1-cp315-cp315-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:20327e162ef7953fae56b31a43ca457f94ed1fc7a206d01e3d5c8412d1b91572", size = 238338, upload-time = "2026-09-10T23:11:19.742Z" }, + { url = "https://files.pythonhosted.org/packages/91/5b/47b2f2f08b90d9d3b4e1790ccae0d4e4de40e73f6614c9a52465a04f02a5/wrapt-2.4.1-cp315-cp315-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:2cedb743dbdfb9b6d4f11acd8cb6329460264429777e81ea2e86b1b72ea503af", size = 220668, upload-time = "2026-09-10T23:11:21.422Z" }, + { url = "https://files.pythonhosted.org/packages/43/1f/da7844c7a3f7c01f3496e690e15ce9d01c0949278c64d4287ec472c90f72/wrapt-2.4.1-cp315-cp315-musllinux_1_2_aarch64.whl", hash = "sha256:5a54744b1193505f19016194979b773ee3754a199e9ad90db18f2e9a18fffa16", size = 235812, upload-time = "2026-09-10T23:11:23.211Z" }, + { url = "https://files.pythonhosted.org/packages/69/af/ac4aa9f795721a54a21bac329201f7df1b199b71ef064187a266d4d29836/wrapt-2.4.1-cp315-cp315-musllinux_1_2_riscv64.whl", hash = "sha256:b0d38d9cc23e9781e6584f303e5c5a9f0d45b85de92ad45678c4dde8d49d9535", size = 218195, upload-time = "2026-09-10T23:11:24.983Z" }, + { url = "https://files.pythonhosted.org/packages/78/aa/cff7670ec1cfa75b951171cfb41d45c7bdb94e928620a1599ec2b00cd2dd/wrapt-2.4.1-cp315-cp315-musllinux_1_2_x86_64.whl", hash = "sha256:a524ca32f0bcdde2b728d9f81f9527d4dd24b13f15d180f05d08cdc01d1cebe0", size = 225713, upload-time = "2026-09-10T23:11:26.576Z" }, + { url = "https://files.pythonhosted.org/packages/77/c6/64f138aef50b5ea1dfacca2a023b32ba3e41ef087bbd92e8997a95f4a0b8/wrapt-2.4.1-cp315-cp315-win32.whl", hash = "sha256:47c267617551e906de72f6e7265aa3bce84c63d44513d2d2e735e943422aa0a2", size = 93880, upload-time = "2026-09-10T23:11:28.253Z" }, + { url = "https://files.pythonhosted.org/packages/8d/aa/afe7484a0f7232e87f2ebb65286c1025d43cbe1de4e4051b127b024a92f7/wrapt-2.4.1-cp315-cp315-win_amd64.whl", hash = "sha256:ee437bd7fd050823ef731aad20e968ca8fe670b95e1841f5d201bfdbad4a4e96", size = 98919, upload-time = "2026-09-10T23:11:29.832Z" }, + { url = "https://files.pythonhosted.org/packages/83/21/6865f976c6a1082ca61e2792bc6a2d6a0ed03cd750a46f837753127191f8/wrapt-2.4.1-cp315-cp315-win_arm64.whl", hash = "sha256:ebf3b703752b53366fd02b7fbd8c447428e0349c9af3fc17c7a05076dbdd749a", size = 96021, upload-time = "2026-09-10T23:11:31.448Z" }, + { url = "https://files.pythonhosted.org/packages/8d/48/baf784cc9f1fe554f96daf15af06c7c466d76469d5540e357cf564755606/wrapt-2.4.1-cp315-cp315t-macosx_10_15_x86_64.whl", hash = "sha256:0f5990f5db090f069fcd577cc61cc8d463db73cde132abba9033b0a264fc06d0", size = 102028, upload-time = "2026-09-10T23:11:33.102Z" }, + { url = "https://files.pythonhosted.org/packages/82/35/fb809c95bf5512196aeda4c640aabf4e92f607340b8b2556555690a52b7a/wrapt-2.4.1-cp315-cp315t-macosx_11_0_arm64.whl", hash = "sha256:ef19a2590b195ac294deff8ee350a027a479afdd2ef2170ce3900af92406e110", size = 102640, upload-time = "2026-09-10T23:11:35.145Z" }, + { url = "https://files.pythonhosted.org/packages/a1/3f/bd462010ccbbe298479f320bf2bb02996706003e9c20a15790b65a186f8b/wrapt-2.4.1-cp315-cp315t-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl", hash = "sha256:432f402f9b6014403cacf9fd18a6bf089c77964330eae951a155131ab0414f5d", size = 275999, upload-time = "2026-09-10T23:11:36.863Z" }, + { url = "https://files.pythonhosted.org/packages/56/1e/bd043b2bad2943336e090583ff6cea2f3e8776bde02f073ccf760b16eee1/wrapt-2.4.1-cp315-cp315t-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:eb7c0f8bdd21e954bad89d554cfb7431c2f8179842853f199841ab90cbebe914", size = 287576, upload-time = "2026-09-10T23:11:38.79Z" }, + { url = "https://files.pythonhosted.org/packages/1c/d9/9937bbd61ca4e7f58a7e8a38f5ff3562334f741de51b87f0e031df441254/wrapt-2.4.1-cp315-cp315t-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:8e82a1669d63b79a2b53041bbb6ea096b5763cab3ca698ed7ac244b3acb7cd51", size = 256763, upload-time = "2026-09-10T23:11:40.668Z" }, + { url = "https://files.pythonhosted.org/packages/aa/a8/0f78cc7e7fc6313cdba32df06f4b29a45a4a2aed3499040d6c1628d3f339/wrapt-2.4.1-cp315-cp315t-musllinux_1_2_aarch64.whl", hash = "sha256:b19e71c914c435d2c5652caea386c9bf2807979e957183f5bb06d5ed5fa8b55e", size = 282244, upload-time = "2026-09-10T23:11:42.483Z" }, + { url = "https://files.pythonhosted.org/packages/be/0d/30317d738b9898a7fbedc193657f151a468b0fa7235f4abd58a646a99cd8/wrapt-2.4.1-cp315-cp315t-musllinux_1_2_riscv64.whl", hash = "sha256:a1e4870d3368c6c918f38e308d5dcea970ea5b908ce889c21412f3a517ebf0c3", size = 254153, upload-time = "2026-09-10T23:11:44.357Z" }, + { url = "https://files.pythonhosted.org/packages/12/53/fe1f251a89f471ca7c5e17f2edbd8efd48fea6f1c08430dcf60986ea4183/wrapt-2.4.1-cp315-cp315t-musllinux_1_2_x86_64.whl", hash = "sha256:2286e8e4a937966706463d1bcea30dfefee529e6e73e097a18e9e066a07ae6c2", size = 271234, upload-time = "2026-09-10T23:11:46.132Z" }, + { url = "https://files.pythonhosted.org/packages/bd/7c/d63758cd7fa0d18c415a87312a07cdd9f0102eb824aaab5c7450b3ec08e5/wrapt-2.4.1-cp315-cp315t-win32.whl", hash = "sha256:80afa3b7010e82899044a2a189c468a0cd8c89980398a59ba3b96b451a6dc5e9", size = 96236, upload-time = "2026-09-10T23:11:47.942Z" }, + { url = "https://files.pythonhosted.org/packages/f7/2e/1785e6d2696db08b07c053b1f4ccb2b5e8f9d12e9bf2c7497693be6b4504/wrapt-2.4.1-cp315-cp315t-win_amd64.whl", hash = "sha256:3593b43fabab6b59fe77e38f7e40974aae120c30cea3fd1ceaea6621ee96be00", size = 102662, upload-time = "2026-09-10T23:11:49.831Z" }, + { url = "https://files.pythonhosted.org/packages/2f/fd/3f4ac5c4754948355e050f952ae7a64cdca94891e49fe9927c3f90a73334/wrapt-2.4.1-cp315-cp315t-win_arm64.whl", hash = "sha256:ac1939ccf3e1c33f463706fbf52db5075f5eefb6042bcc1f9cf48c2c20ef478c", size = 98500, upload-time = "2026-09-10T23:11:51.501Z" }, + { url = "https://files.pythonhosted.org/packages/4f/a2/edcfc8d9a30375791b775715f8501364588f65b494ec4f6930568a19e765/wrapt-2.4.1-py3-none-any.whl", hash = "sha256:1e84ec5d89a0a07a0ef6bcd343f5c8ecdc95601d71de3058cdc63274e86c193c", size = 75317, upload-time = "2026-09-10T23:12:14.82Z" }, +] From 7bc559f84616e3dfeaeba7b73107d1474a976e26 Mon Sep 17 00:00:00 2001 From: Audric Ackermann Date: Fri, 25 Sep 2026 13:16:00 +1000 Subject: [PATCH 015/101] feat: report Crowdin duplicate translations from webhooks, reconciled daily The sharded daily scan saw each locale once every eight days and could say nothing about slots that had been resolved. This keeps the open slots, keyed (string, locale, plural category), and posts only what changed. - crowdin-relay.service takes Crowdin's suggestion events, acknowledges at once and re-checks the one (string, locale) named. Crowdin signs nothing, so the secret is the last path segment, and a wrong or unset one is a 404. It is a process and account of its own rather than a route on the Zendesk relay, which can write public comments and has no business holding the Crowdin token. - crowdin-reconcile-duplicates, daily, judges every string of every locale and posts new and resolved slots, or nothing. Crowdin never retries a webhook, so this is what keeps the state correct. --croql narrows each locale to the strings CroQL counts two translations for; it stays off until that query is checked live. - Both write the state under a file lock. The relay records when it checked each (string, locale), and a scan's older view of that scope is ignored, so a scan that began before a suggestion landed cannot resolve what the event opened. - The state is written only once every message landed; a run that fails to post repeats its whole diff next time. --seed records the current slots without posting them. report_multiple_translations.py and its workflow stay until a reconciliation cycle has run clean on the host; the two now share the slot logic, and its golden is unchanged. --- deploy/README.md | 79 +++++ deploy/crowdin-duplicates.service | 33 +++ deploy/crowdin-duplicates.timer | 13 + deploy/crowdin-relay.service | 39 +++ deploy/nginx-webhooks.conf | 14 + pyproject.toml | 1 + src/session_ops/crowdin/duplicates.py | 266 +++++++++++++++++ src/session_ops/crowdin/reconcile.py | 151 ++++++++++ src/session_ops/crowdin/relay.py | 135 +++++++++ .../crowdin/report_multiple_translations.py | 44 +-- src/session_ops/monitor/jobs.toml | 5 + tests/crowdin/test_duplicates.py | 270 ++++++++++++++++++ tests/goldens/README.md | 3 + tests/goldens/reconcile/dry-run.json | 21 ++ 14 files changed, 1035 insertions(+), 39 deletions(-) create mode 100644 deploy/crowdin-duplicates.service create mode 100644 deploy/crowdin-duplicates.timer create mode 100644 deploy/crowdin-relay.service create mode 100644 src/session_ops/crowdin/duplicates.py create mode 100644 src/session_ops/crowdin/reconcile.py create mode 100644 src/session_ops/crowdin/relay.py create mode 100644 tests/crowdin/test_duplicates.py create mode 100644 tests/goldens/reconcile/dry-run.json diff --git a/deploy/README.md b/deploy/README.md index 380b673..7bdcdf4 100644 --- a/deploy/README.md +++ b/deploy/README.md @@ -9,6 +9,8 @@ digest, all on one machine. | `zendesk-digest.timer` β†’ `.service` | Weekday mornings. Resolves positive reviews, then posts the digest. | | `github-prs-digest.timer` β†’ `.service` | Weekday mornings. Posts the contributor pull request digest. | | `session-ops-silence.timer` β†’ `.service` | Hourly. Alerts when a job in `jobs.toml` has not succeeded within its `max_age_hours`. | +| `crowdin-relay.service` | Always on. The endpoint Crowdin posts suggestion webhooks to. | +| `crowdin-duplicates.timer` β†’ `.service` | Daily. Reconciles the open duplicate-translation slots and posts what changed. | `zendesk-alert@.service`, `github-prs-alert@.service` and `session-ops-alert@.service` are pulled in by `OnFailure=` and report the failed unit to the channel that job posts @@ -178,6 +180,43 @@ The `cp` of every unit is the point: the two digests gain the `ExecStartPost=` t writes their stamp. Until each has run once, the checker counts its silence from the first check that found the stamp missing, so it does not alert on install. +### Adding the Crowdin duplicate-translation report + +Its own account and environment file, shared by the relay and the daily +reconciliation, which both read and write the open slots in +`/var/lib/session-ops/crowdin/duplicates.json` under a file lock. Neither needs a home. + +```bash +useradd --system --no-create-home --home /nonexistent --shell /usr/sbin/nologin crowdin +[ -e /etc/session-ops/crowdin.env ] || install -m 640 -o root -g crowdin /dev/null /etc/session-ops/crowdin.env +"${EDITOR:-nano}" /etc/session-ops/crowdin.env # contents under Secrets, below + +cp /opt/zendesk/deploy/crowdin-* /etc/systemd/system/ +systemctl daemon-reload + +# Seed once: records every open slot and posts nothing. About an hour, read-only. +systemd-run --pipe --wait --uid=crowdin -p EnvironmentFile=/etc/session-ops/crowdin.env \ + -p StateDirectory=session-ops/crowdin \ + /opt/zendesk/.venv/bin/crowdin-reconcile-duplicates --seed \ + --state /var/lib/session-ops/crowdin/duplicates.json + +systemctl enable --now crowdin-relay.service crowdin-duplicates.timer +``` + +Then the route, in the live nginx file rather than by re-copying it (see the note +under Install): add the `location ^~ /crowdin/suggestions/` block from +`nginx-webhooks.conf`, then `nginx -t && systemctl reload nginx`. + +Last, in Crowdin: project β†’ Integrations β†’ Webhooks β†’ Add, URL +`https://webhooks.session.codes/crowdin/suggestions/`, request +type POST, content type `application/json`, events *Suggestion added, updated, +deleted, approved and disapproved*. + +The relay does nothing until the state exists, and a wrong secret gets a 404. Crowdin +never retries a delivery, so what the relay misses the next reconciliation posts, at +most a day late. The old sharded report keeps running from GitHub Actions until a full +reconciliation cycle has run clean here. + ## Secrets `/etc/zendesk/env`, mode `640`, `root:zendesk` β€” readable by the service, not by @@ -283,6 +322,26 @@ Mode `640`, `root:sessionops`. ALERT_DISCORD_WEBHOOK_URL= ``` +### `/etc/session-ops/crowdin.env` + +Mode `640`, `root:crowdin`. + +```sh +# Read-only: the relay is reachable from the internet, and nothing here writes to +# Crowdin. Scopes: Projects, Source files & strings, Translations (read). +CROWDIN_API_TOKEN= + +# The channel new and resolved slots go to. +CROWDIN_DISCORD_WEBHOOK_URL= + +# The last path segment of the webhook URL given to Crowdin, which signs nothing. +# Empty refuses every delivery. Generate with: openssl rand -hex 32 +CROWDIN_WEBHOOK_SECRET= + +# Uncomment for the first deliveries: the relay prints what it would post. +#CROWDIN_RELAY_DRY_RUN=1 +``` + ## Verifying, in order **1. Locally, before Zendesk knows the address.** An unsigned request must be refused: @@ -418,6 +477,26 @@ ls -l /var/lib/session-ops/stamps/ # one file per job that has succeeded sin `systemctl start session-ops-alert@test.service` checks its failure path. +**8. The Crowdin relay and reconciliation.** Without the secret the route must not +exist, and with it an empty delivery is acknowledged: + +```bash +curl -sS -o /dev/null -w '%{http_code}\n' -X POST 127.0.0.1:8081/crowdin/suggestions/wrong \ + -H 'Content-Type: application/json' -d '{}' # expect 404 +. /etc/session-ops/crowdin.env; curl -sS -X POST \ + "127.0.0.1:8081/crowdin/suggestions/$CROWDIN_WEBHOOK_SECRET" \ + -H 'Content-Type: application/json' -d '{}' # expect "checks":0 +``` + +A suggestion typed into the Crowdin editor should then log a line in +`journalctl -fu crowdin-relay`. A dry run of reconciliation for one locale: + +```bash +systemd-run --pipe --wait --uid=crowdin -p EnvironmentFile=/etc/session-ops/crowdin.env \ + -p StateDirectory=session-ops/crowdin /opt/zendesk/.venv/bin/crowdin-reconcile-duplicates \ + --dry-run --locales de --state /var/lib/session-ops/crowdin/duplicates.json +``` + ## Updating ```bash diff --git a/deploy/crowdin-duplicates.service b/deploy/crowdin-duplicates.service new file mode 100644 index 0000000..d7aba0e --- /dev/null +++ b/deploy/crowdin-duplicates.service @@ -0,0 +1,33 @@ +[Unit] +Description=Reconcile Crowdin duplicate translations and post what changed +Documentation=https://github.com/session-foundation/session-shared-scripts +After=network-online.target +Wants=network-online.target +OnFailure=session-ops-alert@%n.service + +[Service] +Type=oneshot +User=crowdin +Group=crowdin +EnvironmentFile=/etc/session-ops/crowdin.env +ExecStart=/opt/zendesk/.venv/bin/crowdin-reconcile-duplicates \ + --state /var/lib/session-ops/crowdin/duplicates.json +# The silence checker's evidence that this ran (session-ops-silence). `+` runs it +# outside the sandbox, so the job itself gets no new write path. +ExecStartPost=+/usr/bin/touch /var/lib/session-ops/stamps/crowdin-duplicates + +# ~110k requests at 30/s without --croql. Twice that is Crowdin throttling the run. +TimeoutStartSec=3h + +NoNewPrivileges=yes +PrivateTmp=yes +PrivateDevices=yes +ProtectSystem=strict +ProtectHome=yes +ProtectKernelTunables=yes +ProtectKernelModules=yes +ProtectControlGroups=yes +RestrictAddressFamilies=AF_INET AF_INET6 +RestrictNamespaces=yes +LockPersonality=yes +StateDirectory=session-ops/crowdin diff --git a/deploy/crowdin-duplicates.timer b/deploy/crowdin-duplicates.timer new file mode 100644 index 0000000..fdc0b2c --- /dev/null +++ b/deploy/crowdin-duplicates.timer @@ -0,0 +1,13 @@ +[Unit] +Description=Reconcile Crowdin duplicate translations daily +Documentation=https://github.com/session-foundation/session-shared-scripts + +[Timer] +# Webhooks carry the day's changes; this catches whatever Crowdin failed to deliver, +# at most a day late. +OnCalendar=*-*-* 03:00 UTC +Persistent=yes +RandomizedDelaySec=10min + +[Install] +WantedBy=timers.target diff --git a/deploy/crowdin-relay.service b/deploy/crowdin-relay.service new file mode 100644 index 0000000..f1a6a29 --- /dev/null +++ b/deploy/crowdin-relay.service @@ -0,0 +1,39 @@ +[Unit] +Description=Crowdin suggestion webhook endpoint (duplicate translations) +Documentation=https://github.com/session-foundation/session-shared-scripts +After=network-online.target +Wants=network-online.target +OnFailure=session-ops-alert@%n.service + +[Service] +Type=exec +User=crowdin +Group=crowdin +EnvironmentFile=/etc/session-ops/crowdin.env +Environment=CROWDIN_DUPLICATES_STATE=/var/lib/session-ops/crowdin/duplicates.json +# Loopback only: nginx terminates TLS and is the only thing that should reach it. +ExecStart=/opt/zendesk/.venv/bin/uvicorn session_ops.crowdin.relay:app \ + --host 127.0.0.1 --port 8081 --no-access-log --proxy-headers \ + --forwarded-allow-ips 127.0.0.1 +Restart=always +RestartSec=2 + +NoNewPrivileges=yes +PrivateTmp=yes +PrivateDevices=yes +ProtectSystem=strict +ProtectHome=yes +ProtectKernelTunables=yes +ProtectKernelModules=yes +ProtectControlGroups=yes +RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX +RestrictNamespaces=yes +LockPersonality=yes +MemoryDenyWriteExecute=yes +SystemCallFilter=@system-service +SystemCallErrorNumber=EPERM +# The open slots, shared with crowdin-duplicates.service under a file lock. +StateDirectory=session-ops/crowdin + +[Install] +WantedBy=multi-user.target diff --git a/deploy/nginx-webhooks.conf b/deploy/nginx-webhooks.conf index 7a9c96c..aa445e2 100644 --- a/deploy/nginx-webhooks.conf +++ b/deploy/nginx-webhooks.conf @@ -80,6 +80,20 @@ server { proxy_connect_timeout 5s; } + # Crowdin suggestion events, for crowdin-relay.service. The last path segment is + # the secret, since Crowdin signs nothing: access_log off above is what keeps it + # out of the logs. + location ^~ /crowdin/suggestions/ { + proxy_pass http://127.0.0.1:8081; + proxy_http_version 1.1; + proxy_set_header Host $host; + proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; + proxy_set_header X-Forwarded-Proto $scheme; + # Crowdin drops a delivery not answered within 30 s; the relay acks at once. + proxy_read_timeout 10s; + proxy_connect_timeout 5s; + } + location = /healthz { proxy_pass http://127.0.0.1:8080; proxy_set_header Host $host; diff --git a/pyproject.toml b/pyproject.toml index 25b05fc..622cf10 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -28,6 +28,7 @@ crowdin-codegen-localization = "session_ops.crowdin.codegen_localization:cli" crowdin-generate-language-list = "session_ops.crowdin.generate_language_list:cli" crowdin-report-duplicates = "session_ops.crowdin.report_multiple_translations:main" crowdin-approve-strings = "session_ops.crowdin.approve_strings:main" +crowdin-reconcile-duplicates = "session_ops.crowdin.reconcile:main" sogs-ban = "session_ops.sogs.ban:cli" sogs-perms = "session_ops.sogs.perms:cli" session-ops-alert = "session_ops.monitor.alert:main" diff --git a/src/session_ops/crowdin/duplicates.py b/src/session_ops/crowdin/duplicates.py new file mode 100644 index 0000000..d5ed2c4 --- /dev/null +++ b/src/session_ops/crowdin/duplicates.py @@ -0,0 +1,266 @@ +"""Crowdin string slots holding more than one translation: finding them, remembering +which are open, and telling Discord what changed. + +A slot is (string, locale, plural category). Exactly one translation per slot is the +goal, so a slot with two or more needs someone to choose the keeper. + +The state is the set of open slots. Two writers keep it current: the relay, one +(string, locale) at a time as Crowdin reports suggestions, and reconciliation, over +every string of every locale it scans. Crowdin never retries a webhook, so +reconciliation is what makes the state correct; the relay only makes it prompt. +Each writer records when it checked each (string, locale), and the newer check wins, +so a scan that began before an event cannot undo what the event found. +""" +import collections +import contextlib +import fcntl +import json +import os +import time + +from session_ops.crowdin import sdk +from session_ops.shared.discord import clip + +STATE_VERSION = 1 +MAX_EMBEDS_PER_MESSAGE = 10 +MAX_DESC_CHARS = 3800 +MAX_MESSAGE_CHARS = 6000 +OPEN_COLOR, RESOLVED_COLOR = 0xE67E22, 0x2ECC71 + + +def user_label(u): + if not u: + return "" + return f"{u.get('id')}:{u.get('username') or u.get('fullName') or '?'}" + + +def slots_for_string(translations, approved_ids, lang, sid, meta, web_url): + """The finding for every plural category of one string holding 2+ translations.""" + by_cat = collections.defaultdict(list) + for t in translations: + by_cat[t.get("pluralCategoryName")].append(t) + found = [] + for cat, ts in by_cat.items(): + if len(ts) < 2: + continue + ts.sort(key=lambda t: t.get("createdAt") or "") + found.append({ + "locale": lang, + "status": "multiple-translations", + "stringId": sid, + "identifier": meta.get("identifier"), + "webUrl": web_url, + "pluralCategory": cat, + "count": len(ts), + "sourceText": meta.get("text"), + "translations": [{ + "translationId": t["id"], + "user": user_label(t.get("user")), + "createdAt": t.get("createdAt"), + "approved": t["id"] in approved_ids, + "rating": t.get("rating"), + "isPreTranslated": t.get("isPreTranslated"), + "provider": t.get("provider"), + "text": t.get("text"), + } for t in ts], + }) + return found + + +def check_string(client, sid, lang, meta, web_url, approved_ids=None): + """Fetch one (string, locale) and return its open slots. `approved_ids` saves a + request when the caller already listed the locale's approvals.""" + translations = sdk.fetch_all(client.string_translations, "list_string_translations", + stringId=sid, languageId=lang) + if approved_ids is None: + approved_ids = {a["translationId"] for a in sdk.fetch_all( + client.string_translations, "list_translation_approvals", + stringId=sid, languageId=lang)} + return slots_for_string(translations, approved_ids, lang, sid, meta, web_url) + + +class Project: + """What building editor links needs: the project slug and per-locale editor codes. + A string's own webUrl always targets the first target language.""" + + def __init__(self, details): + self.slug = details["identifier"] + source = details["sourceLanguage"] + self.source_code = source.get("editorCode") or source["id"] + self.editor_code = {lang["id"]: lang.get("editorCode") or lang["id"] + for lang in details["targetLanguages"]} + self.locales = details["targetLanguageIds"] + + def editor_url(self, lang, sid): + return (f"https://crowdin.com/editor/{self.slug}/all/" + f"{self.source_code}-{self.editor_code.get(lang, lang)}#{sid}") + + +# ---- State ------------------------------------------------------------------- + + +def slot_key(sid, lang, category): + return f"{sid}:{lang}:{category or ''}" + + +def scope_key(sid, lang): + return f"{sid}:{lang}" + + +def empty_state(): + return {"version": STATE_VERSION, "slots": {}, "checked": {}} + + +def load(path): + """The state at `path`. Unlike a digest's dedup file, a lost one is not harmless: + every open slot would be reported as new, so an unreadable file stops the run.""" + if not os.path.exists(path): + return empty_state() + with open(path, encoding="utf-8") as handle: + data = json.load(handle) + if data.get("version") != STATE_VERSION: + raise SystemExit(f"{path} is version {data.get('version')!r}, expected " + f"{STATE_VERSION}; move it aside and --seed again.") + return data + + +def save(path, state): + temporary = f"{path}.tmp" + with open(temporary, "w", encoding="utf-8") as handle: + json.dump(state, handle, indent=1, sort_keys=True) + os.replace(temporary, path) + + +@contextlib.contextmanager +def locked(path): + """Hold the state's lock: the relay and reconciliation run as separate processes.""" + with open(f"{path}.lock", "w") as handle: + fcntl.flock(handle, fcntl.LOCK_EX) + try: + yield + finally: + fcntl.flock(handle, fcntl.LOCK_UN) + + +def record(finding, now): + return {"stringId": finding["stringId"], "locale": finding["locale"], + "pluralCategory": finding["pluralCategory"], + "identifier": finding["identifier"], "count": finding["count"], + "opened_at": now} + + +def apply(state, findings, checked, now, remember): + """Merge a check into the state. Returns (opened, resolved) slot records. + + `checked` maps scope_key -> when that (string, locale) was fetched. Only those + scopes are judged, and only where the state holds no newer check: a slot outside + them, or checked later by someone else, is left exactly as it is. + + `remember` keeps the check times, which only matters while a reconciliation that + started earlier may still be scanning: the relay's checks, not a scan's own. + """ + newer = state["checked"] + judged = {scope for scope, at in checked.items() if at >= newer.get(scope, 0)} + current = {slot_key(f["stringId"], f["locale"], f["pluralCategory"]): f + for f in findings if scope_key(f["stringId"], f["locale"]) in judged} + opened, resolved = [], [] + for key, slot in list(state["slots"].items()): + if scope_key(slot["stringId"], slot["locale"]) in judged and key not in current: + resolved.append(state["slots"].pop(key)) + for key, finding in current.items(): + if key in state["slots"]: + state["slots"][key]["count"] = finding["count"] + else: + state["slots"][key] = record(finding, now) + opened.append(state["slots"][key]) + if remember: + for scope in judged: + newer[scope] = checked[scope] + return opened, resolved + + +def forget_checks_before(state, cutoff): + """Drop the check times no scan still running can predate: a reconciliation that + started at `cutoff` has applied, and the next starts later.""" + state["checked"] = {k: at for k, at in state["checked"].items() if at > cutoff} + + +# ---- Discord ----------------------------------------------------------------- + + +def slot_line(slot, project, suffix): + cat = slot["pluralCategory"] + cat_txt = f" `[{cat}]`" if cat and cat != "other" else "" + ident = clip(slot["identifier"] or f"string {slot['stringId']}", 90) + url = project.editor_url(slot["locale"], slot["stringId"]) + return f"β€’ [{ident}]({url}){cat_txt}{suffix(slot)}" + + +def section_embeds(slots, project, title, color, suffix): + """One embed per locale, split when a locale outgrows a description.""" + embeds = [] + by_locale = collections.defaultdict(list) + for slot in slots: + by_locale[slot["locale"]].append(slot) + for lang in sorted(by_locale, key=lambda lang: (-len(by_locale[lang]), lang)): + items = sorted(by_locale[lang], key=lambda s: (s["identifier"] or "", + str(s["pluralCategory"]))) + lines, used, first = [], 0, True + for slot in items: + line = slot_line(slot, project, suffix) + if lines and used + len(line) + 1 > MAX_DESC_CHARS: + embeds.append({"title": title(lang, len(items)) if first else f"{lang} (cont.)", + "description": "\n".join(lines), "color": color}) + lines, used, first = [], 0, False + lines.append(line) + used += len(line) + 1 + embeds.append({"title": title(lang, len(items)) if first else f"{lang} (cont.)", + "description": "\n".join(lines), "color": color}) + return embeds + + +def build_messages(opened, resolved, still_open, project): + """Discord payloads for what changed, or [] when nothing did.""" + if not opened and not resolved: + return [] + summary = { + "title": "🈳 Crowdin: translations to choose between", + "description": ( + f"**{len(opened)}** slot(s) newly holding 2+ translations, " + f"**{len(resolved)}** resolved. **{still_open}** open in total.\n" + "Each open slot needs one translation kept and the rest deleted, so that " + "exactly one translation (per plural form) is exported."), + "color": OPEN_COLOR if opened else RESOLVED_COLOR, + } + embeds = [summary] + embeds += section_embeds(opened, project, lambda lang, n: f"{lang} β€” {n} new", + OPEN_COLOR, lambda s: f" β€” **{s['count']}** translations") + embeds += section_embeds(resolved, project, lambda lang, n: f"{lang} β€” {n} resolved", + RESOLVED_COLOR, lambda s: "") + return pack_embeds(embeds) + + +def embed_len(e): + total = len(e.get("title") or "") + len(e.get("description") or "") + for fld in e.get("fields", []): + total += len(fld.get("name") or "") + len(fld.get("value") or "") + return total + + +def pack_embeds(embeds): + """Messages within Discord's 10 embeds and 6000 characters apiece.""" + messages, chunk, used = [], [], 0 + for e in embeds: + size = embed_len(e) + if chunk and (len(chunk) >= MAX_EMBEDS_PER_MESSAGE or used + size > MAX_MESSAGE_CHARS): + messages.append({"embeds": chunk}) + chunk, used = [], 0 + chunk.append(e) + used += size + if chunk: + messages.append({"embeds": chunk}) + return messages + + +def now(): + return time.time() diff --git a/src/session_ops/crowdin/reconcile.py b/src/session_ops/crowdin/reconcile.py new file mode 100644 index 0000000..c55b782 --- /dev/null +++ b/src/session_ops/crowdin/reconcile.py @@ -0,0 +1,151 @@ +#!/usr/bin/env python3 +""" +Reconcile the open duplicate-translation slots with Crowdin, and post what changed. + +Every string of every scanned locale is judged against the state: slots newly holding +2+ translations are posted as new, slots no longer holding them as resolved, and +nothing is posted when nothing changed. This is what makes the state correct; the +relay only makes it prompt, and Crowdin drops any webhook it fails to deliver. + +With --croql, a locale costs one query for the strings holding 2+ translations in it +at all, then one request per candidate. Without it, one request per string. A plural +string with one translation per category is a candidate too; CroQL cannot tell plural +categories apart. + +The state is written only after every message landed, so a failed post is repeated in +full on the next run rather than lost. + +Config (env vars): + CROWDIN_API_TOKEN read-only Crowdin token + CROWDIN_DISCORD_WEBHOOK_URL where changes go (not needed with --dry-run or --seed) + +Usage: + crowdin-reconcile-duplicates --seed --state PATH # first run: record, post nothing + crowdin-reconcile-duplicates --state PATH # what the timer does + crowdin-reconcile-duplicates --dry-run --state PATH --locales de +""" +import argparse +import concurrent.futures +import json +import sys + +from session_ops.crowdin import duplicates, sdk +from session_ops.shared import discord, http +from session_ops.shared.env import get_env + +DEFAULT_PROJECT = "618696" +CROQL = 'count of translations where ( language = @language:"{lang}" ) > 1' + + +def crowdin_client(token, project_id): + return sdk.client(token, project_id, attempts=10, timeout=60) + + +def candidates(client, lang, string_ids, use_croql): + if not use_croql: + return list(string_ids) + rows = sdk.fetch_all(client.source_strings, "list_strings", croql=CROQL.format(lang=lang)) + return [row["id"] for row in rows] + + +def scan_locale(client, project, lang, strings, use_croql, max_workers): + """(findings, checked) for one locale. A string that failed is left unjudged.""" + started = duplicates.now() + approved = {} + for a in sdk.fetch_all(client.string_translations, "list_translation_approvals", + languageId=lang): + approved.setdefault(a["stringId"], set()).add(a["translationId"]) + todo = candidates(client, lang, strings, use_croql) + # Everything CroQL left out holds at most one translation here, as of its query. + checked = {duplicates.scope_key(sid, lang): started for sid in strings} + + def check(sid): + at = duplicates.now() + return at, duplicates.check_string(client, sid, lang, strings.get(sid, {}), + project.editor_url(lang, sid), + approved.get(sid, set())) + + findings, failed = [], 0 + with concurrent.futures.ThreadPoolExecutor(max_workers=max_workers) as pool: + futures = {pool.submit(check, sid): sid for sid in todo} + for future in concurrent.futures.as_completed(futures): + sid = futures[future] + try: + at, found = future.result() + except Exception as exc: + failed += 1 + checked.pop(duplicates.scope_key(sid, lang), None) + print(f"[{lang}] string {sid} failed, left unjudged: {exc!r}", file=sys.stderr) + continue + checked[duplicates.scope_key(sid, lang)] = at + findings.extend(found) + print(f"[{lang}] {len(todo)} checked, {failed} failed, {len(findings)} slot(s) open", + file=sys.stderr) + return findings, checked + + +def main(argv=None): + parser = argparse.ArgumentParser(description="Reconcile Crowdin duplicate translations.") + parser.add_argument("--project-id", default=DEFAULT_PROJECT) + parser.add_argument("--state", required=True, metavar="PATH", + help="The open slots; the relay reads and writes the same file.") + parser.add_argument("--locales", nargs="+", help="Only these (default: every target).") + parser.add_argument("--croql", action="store_true", + help="Narrow each locale with a CroQL query before checking strings.") + parser.add_argument("--max-workers", type=int, default=16) + parser.add_argument("--seed", action="store_true", + help="Record what is open without posting it.") + parser.add_argument("--dry-run", action="store_true", + help="Print what would be posted; write nothing.") + args = parser.parse_args(argv) + + token = get_env("CROWDIN_API_TOKEN") + webhook = get_env("CROWDIN_DISCORD_WEBHOOK_URL", + required=not (args.dry_run or args.seed)) + client = crowdin_client(token, args.project_id) + project = duplicates.Project(client.projects.get_project()["data"]) + locales = args.locales or project.locales + strings = {s["id"]: {"identifier": s.get("identifier"), "text": s.get("text")} + for s in sdk.fetch_all(client.source_strings, "list_strings")} + print(f"{len(strings)} strings, {len(locales)} locale(s)", file=sys.stderr) + + started = duplicates.now() + findings, checked = [], {} + for lang in locales: + try: + found, judged = scan_locale(client, project, lang, strings, args.croql, + args.max_workers) + except sdk.APIException as exc: + print(f"[{lang}] skipped, left unjudged: {exc.http_status} " + f"{sdk.error_message(exc)}", file=sys.stderr) + continue + findings += found + checked.update(judged) + if not checked: + sys.exit("No locale could be scanned.") + + with duplicates.locked(args.state): + state = duplicates.load(args.state) + opened, resolved = duplicates.apply(state, findings, checked, duplicates.now(), + remember=False) + duplicates.forget_checks_before(state, started) + print(f"{len(opened)} opened, {len(resolved)} resolved, " + f"{len(state['slots'])} open", file=sys.stderr) + if args.dry_run: + messages = duplicates.build_messages(opened, resolved, len(state["slots"]), + project) + print(json.dumps(messages, indent=2, ensure_ascii=False)) + return + if not args.seed: + messages = duplicates.build_messages(opened, resolved, len(state["slots"]), + project) + posted = discord.post_to_discord(http.Session(), webhook, messages) \ + if messages else 0 + if posted < len(messages): + sys.exit(f"Posted {posted} of {len(messages)} messages; state not written, " + f"so the next run repeats them.") + duplicates.save(args.state, state) + + +if __name__ == "__main__": + main() diff --git a/src/session_ops/crowdin/relay.py b/src/session_ops/crowdin/relay.py new file mode 100644 index 0000000..2876f42 --- /dev/null +++ b/src/session_ops/crowdin/relay.py @@ -0,0 +1,135 @@ +""" +The endpoint Crowdin posts suggestion webhooks to. Each event names a string and a +locale; the relay acknowledges at once, then re-checks that one (string, locale) and +posts any slot it opened or resolved. + +Crowdin gives up on a delivery that is not answered 2xx within 30 seconds and never +retries it, so the check runs after the response. What it misses anyway, the +reconciliation timer finds. + +Crowdin signs nothing, so the secret is in the URL: the webhook is configured as +https:///crowdin/suggestions/. A wrong or unset secret +gets a 404, the same as a route that does not exist. + +Its own process and account rather than a route on the Zendesk relay: that one can +write public comments on any ticket, and the Crowdin token does not belong beside it. + +Config (env vars): + CROWDIN_API_TOKEN read-only Crowdin token + CROWDIN_DISCORD_WEBHOOK_URL where changes go + CROWDIN_WEBHOOK_SECRET the last path segment Crowdin posts to + CROWDIN_DUPLICATES_STATE the reconciliation's --state file + CROWDIN_PROJECT_ID (optional) defaults to 618696 + CROWDIN_RELAY_DRY_RUN (optional) "1" prints what it would post, writes nothing + + uvicorn session_ops.crowdin.relay:app --host 127.0.0.1 --port 8081 +""" +import functools +import hmac +import json +import os +import sys + +from fastapi import BackgroundTasks, FastAPI, Request, Response +from starlette.concurrency import run_in_threadpool + +from session_ops.crowdin import duplicates, sdk +from session_ops.crowdin.reconcile import DEFAULT_PROJECT, crowdin_client +from session_ops.shared import discord, http + +EVENTS = frozenset({"suggestion.added", "suggestion.updated", "suggestion.deleted", + "suggestion.approved", "suggestion.disapproved"}) + +app = FastAPI(docs_url=None, redoc_url=None, openapi_url=None) + + +def scopes_from(payload): + """The (string id, locale) pairs a delivery names, batched or not. + + Payload fields are read defensively: an event this cannot place is dropped, and + reconciliation covers it. + """ + events = payload.get("events") if isinstance(payload, dict) and "events" in payload \ + else [payload] + scopes = set() + for event in events if isinstance(events, list) else []: + if not isinstance(event, dict) or event.get("event") not in EVENTS: + continue + translation = event.get("translation") or {} + string = translation.get("sourceString") or translation.get("string") or {} + lang = (translation.get("targetLanguage") or {}).get("id") + try: + sid = int(string.get("id")) + except (TypeError, ValueError): + continue + if lang: + scopes.add((sid, str(lang))) + return scopes + + +@functools.cache +def crowdin(): + client = crowdin_client(os.environ["CROWDIN_API_TOKEN"], + os.environ.get("CROWDIN_PROJECT_ID") or DEFAULT_PROJECT) + return client, duplicates.Project(client.projects.get_project()["data"]) + + +def check(sid, lang): + """Re-check one (string, locale) and post what changed. Never raises: nothing is + waiting on it, so the journal is where a failure goes.""" + state_path = os.environ["CROWDIN_DUPLICATES_STATE"] + dry_run = os.environ.get("CROWDIN_RELAY_DRY_RUN") == "1" + try: + if not os.path.exists(state_path): + print(f"No state at {state_path}: seed it before the relay acts.", flush=True) + return + client, project = crowdin() + string = client.source_strings.get_string(stringId=sid)["data"] + at = duplicates.now() + found = duplicates.check_string(client, sid, lang, string, + project.editor_url(lang, sid)) + with duplicates.locked(state_path): + state = duplicates.load(state_path) + opened, resolved = duplicates.apply( + state, found, {duplicates.scope_key(sid, lang): at}, duplicates.now(), + remember=True) + messages = duplicates.build_messages(opened, resolved, len(state["slots"]), + project) + if dry_run: + print(json.dumps(messages, ensure_ascii=False), flush=True) + return + webhook = os.environ["CROWDIN_DISCORD_WEBHOOK_URL"] + if messages and discord.post_to_discord(http.Session(), webhook, + messages) < len(messages): + print(f"{sid}/{lang}: Discord refused; reconciliation will repeat it.", + flush=True) + return + duplicates.save(state_path, state) + print(f"{sid}/{lang}: {len(opened)} opened, {len(resolved)} resolved", flush=True) + except sdk.APIException as exc: + print(f"{sid}/{lang}: Crowdin {exc.http_status} {sdk.error_message(exc)}", + flush=True) + except BaseException as exc: # SystemExit included: this runs inside a server + print(f"{sid}/{lang}: {exc!r}", file=sys.stderr, flush=True) + + +@app.post("/crowdin/suggestions/{secret}") +async def suggestions(secret: str, request: Request, background: BackgroundTasks): + expected = os.environ.get("CROWDIN_WEBHOOK_SECRET", "") + if not expected or not hmac.compare_digest(secret.encode(), expected.encode()): + return Response(status_code=404) + try: + payload = await request.json() + except ValueError: + return Response(status_code=400) + scopes = scopes_from(payload) + if not scopes: + print("A delivery named no suggestion this relay handles.", flush=True) + for sid, lang in sorted(scopes): + background.add_task(run_in_threadpool, check, sid, lang) + return {"ok": True, "checks": len(scopes)} + + +@app.get("/healthz") +async def healthz(): + return {"ok": True} diff --git a/src/session_ops/crowdin/report_multiple_translations.py b/src/session_ops/crowdin/report_multiple_translations.py index 4a4cdf4..4618da9 100644 --- a/src/session_ops/crowdin/report_multiple_translations.py +++ b/src/session_ops/crowdin/report_multiple_translations.py @@ -45,7 +45,7 @@ import subprocess import sys -from session_ops.crowdin import sdk +from session_ops.crowdin import duplicates, sdk from session_ops.shared import discord, http DEFAULT_PROJECT = "618696" @@ -85,12 +85,6 @@ def crowdin_client(token, project_id): return sdk.client(token, project_id, attempts=10, timeout=60) -def user_label(u): - if not u: - return "" - return f"{u.get('id')}:{u.get('username') or u.get('fullName') or '?'}" - - def snippet(text, n=70): text = (text or "").replace("\n", " ") return text[:n] + ("…" if len(text) > n else "") @@ -125,36 +119,8 @@ def scan_locale(client, lang, string_ids, strings, editor_url, max_workers): def process(sid): trans = sdk.fetch_all(client.string_translations, "list_string_translations", stringId=sid, languageId=lang) - approved_tids = approved_here.get(sid, set()) - by_cat = collections.defaultdict(list) - for t in trans: - by_cat[t.get("pluralCategoryName")].append(t) - local = [] - for cat, ts in by_cat.items(): - if len(ts) < 2: - continue # sole translation in its category -> used as-is, nothing to review - ts.sort(key=lambda t: t.get("createdAt") or "") - local.append({ - "locale": lang, - "status": "multiple-translations", - "stringId": sid, - "identifier": strings.get(sid, {}).get("identifier"), - "webUrl": editor_url(lang, sid), - "pluralCategory": cat, - "count": len(ts), - "sourceText": strings.get(sid, {}).get("text"), - "translations": [{ - "translationId": t["id"], - "user": user_label(t.get("user")), - "createdAt": t.get("createdAt"), - "approved": t["id"] in approved_tids, - "rating": t.get("rating"), - "isPreTranslated": t.get("isPreTranslated"), - "provider": t.get("provider"), - "text": t.get("text"), - } for t in ts], - }) - return local + return duplicates.slots_for_string(trans, approved_here.get(sid, set()), lang, sid, + strings.get(sid, {}), editor_url(lang, sid)) found = [] failed = 0 @@ -259,7 +225,7 @@ def loc_title(): embeds.append({ "title": "…and more", "description": (f"Only the first **{MAX_SLOTS_LISTED}** slots are listed here. " - f"Run `report_multiple_translations.py --json` for the full set."), + f"Run `crowdin-report-duplicates --json` for the full set."), "color": 0x95A5A6, }) @@ -308,7 +274,7 @@ def post_to_discord(webhook_url, messages): discord.post_to_discord(webhook_session, webhook_url, [{ "content": "⚠️ Crowdin multiple-translations report failed to post " "its results (a message was rejected by Discord). " - "Re-run `report_multiple_translations.py --json` for the " + "Re-run `crowdin-report-duplicates --json` for the " "full list.", }]) sys.exit(f"Discord accepted {posted} of {len(messages)} messages.") diff --git a/src/session_ops/monitor/jobs.toml b/src/session_ops/monitor/jobs.toml index 2210c2a..fb753b2 100644 --- a/src/session_ops/monitor/jobs.toml +++ b/src/session_ops/monitor/jobs.toml @@ -13,3 +13,8 @@ max_age_hours = 80 name = "zendesk-digest" # Mon..Fri 10:00, and a full backlog run can take 90 min. max_age_hours = 80 + +[[job]] +name = "crowdin-duplicates" +# Daily at 03:00 UTC; a full scan without --croql runs about an hour. +max_age_hours = 28 diff --git a/tests/crowdin/test_duplicates.py b/tests/crowdin/test_duplicates.py new file mode 100644 index 0000000..dd79172 --- /dev/null +++ b/tests/crowdin/test_duplicates.py @@ -0,0 +1,270 @@ +""" + uv run python -m unittest tests.crowdin.test_duplicates +""" +import contextlib +import copy +import io +import json +import os +import tempfile +import unittest +from unittest import mock + +from fastapi.testclient import TestClient + +from session_ops.crowdin import duplicates, reconcile, relay, sdk +from session_ops.shared.testing import FakeResponse, FakeSession, RecordedSession +from tests.golden import assert_golden, load_golden_json + +API = "https://api.crowdin.com/api/v2" +PROJECT = duplicates.Project({ + "identifier": "p", "sourceLanguage": {"id": "en"}, "targetLanguageIds": ["de"], + "targetLanguages": [{"id": "de", "editorCode": "de"}]}) + + +def finding(sid, lang="de", cat=None, count=2, identifier=None): + return {"stringId": sid, "locale": lang, "pluralCategory": cat, "count": count, + "identifier": identifier or f"s{sid}"} + + +class TestApply(unittest.TestCase): + def test_a_new_slot_opens_and_a_gone_one_resolves(self): + state = duplicates.empty_state() + opened, _ = duplicates.apply(state, [finding(1)], {"1:de": 10, "2:de": 10}, 10, False) + self.assertEqual([s["stringId"] for s in opened], [1]) + opened, resolved = duplicates.apply(state, [finding(2)], {"1:de": 20, "2:de": 20}, + 20, False) + self.assertEqual(([s["stringId"] for s in opened], [s["stringId"] for s in resolved]), + ([2], [1])) + self.assertEqual(list(state["slots"]), ["2:de:"]) + + def test_an_open_slot_seen_again_is_not_news(self): + state = duplicates.empty_state() + duplicates.apply(state, [finding(1)], {"1:de": 10}, 10, False) + self.assertEqual(duplicates.apply(state, [finding(1, count=3)], {"1:de": 20}, 20, False), + ([], [])) + self.assertEqual(state["slots"]["1:de:"]["count"], 3) + + def test_only_the_checked_scopes_are_judged(self): + """A relay check of one string must not resolve every other open slot.""" + state = duplicates.empty_state() + duplicates.apply(state, [finding(1), finding(2)], {"1:de": 10, "2:de": 10}, 10, False) + _, resolved = duplicates.apply(state, [], {"2:de": 20}, 20, True) + self.assertEqual([s["stringId"] for s in resolved], [2]) + self.assertIn("1:de:", state["slots"]) + + def test_each_plural_category_is_its_own_slot(self): + state = duplicates.empty_state() + duplicates.apply(state, [finding(1, cat="one"), finding(1, cat="few")], + {"1:de": 10}, 10, False) + _, resolved = duplicates.apply(state, [finding(1, cat="one")], {"1:de": 20}, 20, False) + self.assertEqual([s["pluralCategory"] for s in resolved], ["few"]) + + def test_a_scan_older_than_an_event_check_leaves_that_scope_alone(self): + """The scan fetched before the suggestion landed; the event saw it.""" + state = duplicates.empty_state() + duplicates.apply(state, [finding(1)], {"1:de": 50}, 50, remember=True) + opened, resolved = duplicates.apply(state, [], {"1:de": 40, "2:de": 40}, 60, False) + self.assertEqual((opened, resolved), ([], [])) + self.assertIn("1:de:", state["slots"]) + + def test_forgetting_keeps_only_checks_newer_than_the_scan(self): + state = duplicates.empty_state() + state["checked"] = {"1:de": 10, "2:de": 30} + duplicates.forget_checks_before(state, 20) + self.assertEqual(state["checked"], {"2:de": 30}) + + +class TestMessages(unittest.TestCase): + def test_nothing_changed_posts_nothing(self): + self.assertEqual(duplicates.build_messages([], [], 7, PROJECT), []) + + def test_new_and_resolved_are_listed_with_editor_links(self): + messages = duplicates.build_messages( + [finding(1, cat="few", count=3)], [finding(2)], 4, PROJECT) + embeds = messages[0]["embeds"] + self.assertIn("**1** slot(s) newly holding 2+ translations, **1** resolved. " + "**4** open in total.", embeds[0]["description"]) + self.assertEqual(embeds[1]["title"], "de β€” 1 new") + self.assertIn("[s1](https://crowdin.com/editor/p/all/en-de#1) `[few]` β€” **3**", + embeds[1]["description"]) + self.assertEqual(embeds[2]["title"], "de β€” 1 resolved") + + def test_a_long_locale_splits_across_embeds_and_messages(self): + opened = [finding(i, identifier="x" * 80) for i in range(400)] + messages = duplicates.build_messages(opened, [], 400, PROJECT) + embeds = [e for m in messages for e in m["embeds"]] + self.assertTrue(all(len(e["description"]) <= duplicates.MAX_DESC_CHARS + for e in embeds if "description" in e)) + self.assertTrue(all(sum(duplicates.embed_len(e) for e in m["embeds"]) + <= duplicates.MAX_MESSAGE_CHARS for m in messages)) + self.assertEqual(sum(e["description"].count("\nβ€’ ") + 1 for e in embeds[1:]), 400) + + +def recording(changes=None): + """The report's recording, with some strings' German translations replaced.""" + exchanges = copy.deepcopy(load_golden_json("report/responses.json")["exchanges"]) + for ex in exchanges: + sid = (ex.get("params") or {}).get("stringId") + if sid in (changes or {}): + ex["response"]["json"]["data"] = [{"data": t} for t in changes[sid]] + return exchanges + + +def translation(tid, text, cat=None): + return {"id": tid, "text": text, "pluralCategoryName": cat, "user": None, + "createdAt": "2026-09-10T00:00:00+00:00"} + + +class TestReconcile(unittest.TestCase): + def setUp(self): + self.dir = tempfile.mkdtemp() + self.state = os.path.join(self.dir, "duplicates.json") + + def run_reconcile(self, exchanges, *flags): + session = RecordedSession(exchanges) + out = io.StringIO() + with mock.patch.object(reconcile, "crowdin_client", + lambda token, pid: sdk.client(token, pid, session=session)), \ + mock.patch.dict(os.environ, {"CROWDIN_API_TOKEN": "t"}), \ + contextlib.redirect_stdout(out), contextlib.redirect_stderr(io.StringIO()): + reconcile.main(["--state", self.state, "--locales", "de", *flags]) + return out.getvalue() + + def slots(self): + with open(self.state, encoding="utf-8") as handle: + return sorted(json.load(handle)["slots"]) + + def test_seeding_records_every_open_slot_and_posts_nothing(self): + self.assertEqual(self.run_reconcile(recording(), "--seed"), "") + self.assertEqual(self.slots(), ["101:de:", "104:de:other", "105:de:one", + "105:de:other", "107:de:"]) + + def test_a_later_run_posts_only_the_difference(self): + self.run_reconcile(recording(), "--seed") + changed = recording({ + 101: [translation(1, "Akzeptieren")], + 102: [translation(3, "Datei konnte nicht gespeichert werden."), + translation(16, "Speichern fehlgeschlagen.")], + }) + assert_golden(self, "reconcile/dry-run.json", self.run_reconcile(changed, "--dry-run")) + self.assertIn("101:de:", self.slots(), "a dry run writes nothing") + + def test_an_unchanged_run_posts_nothing_at_all(self): + self.run_reconcile(recording(), "--seed") + with mock.patch.object(reconcile.discord, "post_to_discord") as post: + with mock.patch.dict(os.environ, {"CROWDIN_DISCORD_WEBHOOK_URL": "https://hook"}): + self.run_reconcile(recording()) + post.assert_not_called() + + def test_croql_checks_only_its_candidates_and_resolves_the_rest(self): + self.run_reconcile(recording(), "--seed") + exchanges = recording() + [{ + "method": "GET", "url": f"{API}/projects/618696/strings", + "params": {"croql": reconcile.CROQL.format(lang="de"), "limit": 500, "offset": 0}, + "response": {"json": {"data": [{"data": {"id": 104}}, {"data": {"id": 105}}]}}}] + session = RecordedSession(exchanges) + with mock.patch.object(reconcile, "crowdin_client", + lambda token, pid: sdk.client(token, pid, session=session)), \ + mock.patch.dict(os.environ, {"CROWDIN_API_TOKEN": "t"}), \ + contextlib.redirect_stdout(io.StringIO()), \ + contextlib.redirect_stderr(io.StringIO()): + reconcile.main(["--state", self.state, "--locales", "de", "--croql", "--seed"]) + checked = {params.get("stringId") for method, url, params, _ in session.calls + if url.endswith("/translations")} + self.assertEqual(checked, {104, 105}) + self.assertEqual(self.slots(), ["104:de:other", "105:de:one", "105:de:other"]) + + def test_a_failed_post_writes_no_state(self): + self.run_reconcile(recording(), "--seed") + with open(self.state, encoding="utf-8") as handle: + before = handle.read() + changed = recording({101: [translation(1, "Akzeptieren")]}) + with mock.patch.object(reconcile.discord, "post_to_discord", return_value=0), \ + mock.patch.dict(os.environ, {"CROWDIN_DISCORD_WEBHOOK_URL": "https://hook"}), \ + self.assertRaises(SystemExit): + self.run_reconcile(changed) + with open(self.state, encoding="utf-8") as handle: + self.assertEqual(handle.read(), before) + + +class TestRelay(unittest.TestCase): + SECRET = "s3cret" + + def setUp(self): + self.client = TestClient(relay.app) + self.checked = [] + patcher = mock.patch.object(relay, "check", lambda sid, lang: self.checked.append((sid, lang))) + patcher.start() + self.addCleanup(patcher.stop) + + def post(self, secret, payload, env_secret=SECRET): + with mock.patch.dict(os.environ, {"CROWDIN_WEBHOOK_SECRET": env_secret}): + return self.client.post(f"/crowdin/suggestions/{secret}", json=payload) + + def event(self, sid=12, lang="uk", name="suggestion.added"): + return {"event": name, "translation": {"sourceString": {"id": str(sid)}, + "targetLanguage": {"id": lang}}} + + def test_the_right_secret_acks_and_checks_the_slot(self): + resp = self.post(self.SECRET, self.event()) + self.assertEqual(resp.status_code, 200) + self.assertEqual(self.checked, [(12, "uk")]) + + def test_a_wrong_secret_is_indistinguishable_from_no_route(self): + self.assertEqual(self.post("guess", self.event()).status_code, 404) + self.assertEqual(self.checked, []) + + def test_an_unset_secret_refuses_everything(self): + self.assertEqual(self.post("", self.event(), env_secret="").status_code, 404) + self.assertEqual(self.post("x", self.event(), env_secret="").status_code, 404) + + def test_a_batched_delivery_checks_each_scope_once(self): + payload = {"events": [self.event(12, "uk"), self.event(12, "uk", "suggestion.deleted"), + self.event(13, "de"), {"event": "file.added"}]} + self.post(self.SECRET, payload) + self.assertEqual(sorted(self.checked), [(12, "uk"), (13, "de")]) + + def test_the_older_string_key_is_read_too(self): + event = {"event": "suggestion.updated", + "translation": {"string": {"id": 7}, "targetLanguage": {"id": "fr"}}} + self.assertEqual(relay.scopes_from(event), {(7, "fr")}) + + def test_an_unplaceable_event_is_acked_and_left_to_reconciliation(self): + resp = self.post(self.SECRET, {"event": "suggestion.added", "translation": {}}) + self.assertEqual(resp.status_code, 200) + self.assertEqual(self.checked, []) + + +class TestRelayCheck(unittest.TestCase): + def test_a_check_posts_what_changed_and_records_it(self): + directory = tempfile.mkdtemp() + state_path = os.path.join(directory, "duplicates.json") + duplicates.save(state_path, duplicates.empty_state()) + api = FakeSession([ + FakeResponse({"data": {"id": 12, "identifier": "greeting", "text": "Hi"}}), + FakeResponse({"data": [{"data": translation(1, "Hallo")}, + {"data": translation(2, "Servus")}]}), + FakeResponse({"data": []}), + ]) + webhook = FakeSession([FakeResponse({}, status_code=204)]) + env = {"CROWDIN_DUPLICATES_STATE": state_path, "CROWDIN_DISCORD_WEBHOOK_URL": "https://hook"} + with mock.patch.object(relay, "crowdin", + lambda: (sdk.client("t", 1, session=api), PROJECT)), \ + mock.patch.object(relay.http, "Session", lambda: webhook), \ + mock.patch.dict(os.environ, env), contextlib.redirect_stdout(io.StringIO()): + relay.check(12, "de") + self.assertIn("greeting", webhook.calls[0][2]["json"]["embeds"][1]["description"]) + self.assertEqual(list(duplicates.load(state_path)["slots"]), ["12:de:"]) + self.assertIn("12:de", duplicates.load(state_path)["checked"]) + + def test_an_unseeded_state_makes_the_relay_do_nothing(self): + env = {"CROWDIN_DUPLICATES_STATE": "/nonexistent/duplicates.json"} + with mock.patch.object(relay, "crowdin") as crowdin, mock.patch.dict(os.environ, env), \ + contextlib.redirect_stdout(io.StringIO()): + relay.check(12, "de") + crowdin.assert_not_called() + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/goldens/README.md b/tests/goldens/README.md index d776128..40cf7ac 100644 --- a/tests/goldens/README.md +++ b/tests/goldens/README.md @@ -9,6 +9,8 @@ changes a payload fails. | `digest/` | `github-prs-digest --dry-run`, 720 h | live org, 2026-09-25, trimmed to read fields | | `report/` | `crowdin-report-duplicates --locales de` | synthetic, shaped like Crowdin's API | | `download/` | `crowdin-download` as the sync runs it | synthetic | +| `approve/` | `crowdin-approve-strings`, approving and `--list` | synthetic | +| `reconcile/` | `crowdin-reconcile-duplicates --dry-run` after a seed | the report's, edited | Requests are matched by method, URL, query and body rather than by order, because the Crowdin scripts fan out across threads. A request the recording lacks fails the test and @@ -19,4 +21,5 @@ To accept a deliberate change, rerun the suite with `UPDATE_GOLDENS=1` and revie ```sh UPDATE_GOLDENS=1 uv run python -m unittest tests.github_prs.test_digest_golden UPDATE_GOLDENS=1 uv run python -m unittest tests.crowdin.test_crowdin_goldens +UPDATE_GOLDENS=1 uv run python -m unittest tests.crowdin.test_approve_golden tests.crowdin.test_duplicates ``` diff --git a/tests/goldens/reconcile/dry-run.json b/tests/goldens/reconcile/dry-run.json new file mode 100644 index 0000000..d5d1a6f --- /dev/null +++ b/tests/goldens/reconcile/dry-run.json @@ -0,0 +1,21 @@ +[ + { + "embeds": [ + { + "title": "🈳 Crowdin: translations to choose between", + "description": "**1** slot(s) newly holding 2+ translations, **1** resolved. **5** open in total.\nEach open slot needs one translation kept and the rest deleted, so that exactly one translation (per plural form) is exported.", + "color": 15105570 + }, + { + "title": "de β€” 1 new", + "description": "β€’ [attachmentsSaveError](https://crowdin.com/editor/session-crossplatform-strings/all/en-de#102) β€” **2** translations", + "color": 15105570 + }, + { + "title": "de β€” 1 resolved", + "description": "β€’ [accept](https://crowdin.com/editor/session-crossplatform-strings/all/en-de#101)", + "color": 3066993 + } + ] + } +] From 31fe9841a58bf289bd6fa44bde7bad379977dc0b Mon Sep 17 00:00:00 2001 From: Audric Ackermann Date: Fri, 25 Sep 2026 13:19:41 +1000 Subject: [PATCH 016/101] docs: one page per job, and a token of its own for the one Crowdin write Each job's section of the README moves to docs/jobs/.md, opening with where it runs, which secrets it reads, how to dry-run it, how to re-run it and where it logs. The text moves as it was; only headings and relative links change. The README becomes the index, and tests/test_docs.py fails when a job in the registry has no page. crowdin-duplicates and session-ops-silence are new pages; the SOGS tools, which nothing schedules, move to docs/tools/. crowdin-approve-strings is the only script that writes to Crowdin, and it read the same token as every read-only job. It now reads CROWDIN_PROOFREADER_TOKEN, or the keyring's proofreader-api-token, and never falls back to CROWDIN_API_TOKEN, which can therefore be read-only everywhere else. --- README.md | 767 +-------------------- deploy/README.md | 2 +- docs/jobs/crowdin-duplicates.md | 66 ++ docs/jobs/crowdin-sync.md | 62 ++ docs/jobs/github-prs-digest.md | 98 +++ docs/jobs/session-ops-silence.md | 27 + docs/jobs/zendesk-digest.md | 286 ++++++++ docs/jobs/zendesk-relay.md | 154 +++++ docs/tools/sogs-ban.md | 170 +++++ src/session_ops/crowdin/approve_strings.py | 21 +- src/session_ops/zendesk/note_reply.py | 2 +- tests/crowdin/test_approve_golden.py | 14 + tests/sogs/__init__.py | 2 +- tests/test_docs.py | 14 + 14 files changed, 927 insertions(+), 758 deletions(-) create mode 100644 docs/jobs/crowdin-duplicates.md create mode 100644 docs/jobs/crowdin-sync.md create mode 100644 docs/jobs/github-prs-digest.md create mode 100644 docs/jobs/session-ops-silence.md create mode 100644 docs/jobs/zendesk-digest.md create mode 100644 docs/jobs/zendesk-relay.md create mode 100644 docs/tools/sogs-ban.md create mode 100644 tests/test_docs.py diff --git a/README.md b/README.md index 290b20e..f3f183b 100644 --- a/README.md +++ b/README.md @@ -1,11 +1,28 @@ # Session Shared Scripts -This repo houses scripts which are shared between the different platform repos for Session, it also contains a number of Actions used to automatically sync some shared elements across the repos. +Session Foundation's scheduled jobs and webhooks, and the scripts the platform repos +share for translations. One package, `session_ops`, deployed to one self-hosted box: +[deploy/README.md](deploy/README.md) installs it. + +## Jobs + +| Job | What it does | Doc | +| --- | --- | --- | +| `zendesk-digest` | Weekday Discord digest of the Zendesk tickets awaiting a reply, after closing positive app-store reviews | [zendesk-digest](docs/jobs/zendesk-digest.md) | +| `zendesk-relay` | Drafts and sends Zendesk replies from `claude:` private notes | [zendesk-relay](docs/jobs/zendesk-relay.md) | +| `github-prs-digest` | Weekday Discord digest of open pull requests from outside contributors | [github-prs-digest](docs/jobs/github-prs-digest.md) | +| `crowdin-duplicates` | Crowdin string slots holding more than one translation, as they open and close | [crowdin-duplicates](docs/jobs/crowdin-duplicates.md) | +| `crowdin-sync` | Weekly: Crowdin translations into pull requests on iOS, Android and the localization module | [crowdin-sync](docs/jobs/crowdin-sync.md) | +| `session-ops-silence` | Discord alerts for a job that failed, or stopped running | [session-ops-silence](docs/jobs/session-ops-silence.md) | + +Run by hand, not scheduled: [`sogs-ban` and `sogs-perms`](docs/tools/sogs-ban.md) for +community bans, and `crowdin-approve-strings`, described with +[the duplicate report](docs/jobs/crowdin-duplicates.md#approving-by-hand). ## Development -One package, `session_ops` under [src/](src/), with one lockfile. Every job is a console -script declared in [pyproject.toml](pyproject.toml). +Every job is a console script declared in [pyproject.toml](pyproject.toml), and one +lockfile pins what all of them run. ```sh uv sync # .venv with every dependency @@ -14,748 +31,8 @@ uv run ruff check . ``` `tests/sogs` skips itself unless `session_util` is importable; see -[Community Bans](#dependencies) for why it is not a PyPI dependency. - -## Crowdin Translation Workflow - -Automated workflow that downloads translations from Crowdin, validates them, and creates PRs for iOS and Android platforms and for the Typescript Localization Module for Desktop and QA. - -### Required Secrets - -| Secret | Description | -| ------------------- | ------------------------------------------------------- | -| `CROWDIN_API_TOKEN` | Crowdin personal access token (see scopes below) | -| `CROWDIN_PR_TOKEN` | GitHub token with PR creation permissions | - -#### Crowdin token scopes - -Crowdin scopes personal access tokens per endpoint family, so a token missing one -scope returns `403 Forbidden` on just those endpoints while every other call keeps -working. The scripts in this repo need: - -| Scope | Value | Needed for | -| ---------------------- | -------------------- | ----------------------------------------------------------------------- | -| Projects | `project` | Project details and the target-language list | -| Source files & strings | `project.source` | Listing source strings (`approve_strings.py`, multiple-translations report) | -| Translations | `project.translation` | Translation exports, plus reading/adding approvals and translations | -| Glossaries | `glossary` | Non-translatable strings (glossary terms) | - -> **Note:** Scopes only cap what a token may do β€” they don't grant anything the -> token's Crowdin account can't already do, so the account also needs a project -> role that allows it (manager/proofreader for anything that writes, e.g. the -> approvals `POST` in `approve_strings.py`). - -### Workflow Inputs - -| Input | Default | Description | -| ------------------------ | ------- | ---------------------------------------- | -| `UPDATE_PULL_REQUESTS` | `true` | Create/update PRs for all platforms | -| `SKIP_VALIDATION_ERRORS` | `false` | Continue even if string validation fails | - -### Schedule - -Runs automatically every Monday at 00:00 UTC. - -### Validation Rules - -#### All Strings (including plurals) - -- **Valid `{variable}` syntax** - No broken braces (`{`, `}`, `{}`, `{ space }`) -- **Allowed HTML tags only** - Only ``, `
`, `` -- **Valid tag syntax** - No malformed `<` (e.g., `