Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions .env.sample
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,8 @@ PORT=3000
ALLOW_PRIVATE_ADDRESS=false

REMOTE_ACTOR_FETCH_POSTS=10
# Document-loader call budget per remote context/replies backfill attempt:
REMOTE_REPLIES_SCRAPE_MAX_REQUESTS=200

# File uploads and media storage:
DRIVE_DISK=
Expand Down
13 changes: 12 additions & 1 deletion CHANGES.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ Version 0.10.0

To be released.

- Upgraded Fedify to 2.4.1.
- Upgraded Fedify to 2.4.2.

- Quotes awaiting [FEP-044f] approval now expose `quoteUrl` for compatibility
with older software when the original public or unlisted post explicitly
Expand Down Expand Up @@ -73,6 +73,14 @@ To be released.
after expiry. Delayed poll schedules no longer allocate worker timers.
[[#653], [#663]]

- Remote conversation backfill now reads [FEP-f228] `context` collections
through *@fedify/backfill* before falling back to `replies`. Collection
aliases share durable jobs, and each seed's replies wait for the context
result. Backfill preserves existing posts, restores available parent
links, and shares a per-attempt document-loader budget across traversal
and persistence. Set `REMOTE_REPLIES_SCRAPE_MAX_REQUESTS` to tune that
budget (default: `200`). [[#661], [#664]]

- Replaced the undocumented `SEONBI_URL` integration with the embedded
[Gukhanmun] Node-API binding. Set `GUKHANMUN` to a comma-separated list of
locale patterns, such as `ko,ko-*`, to add Hangul readings to Hanja in
Expand Down Expand Up @@ -140,6 +148,7 @@ To be released.

[FEP-044f]: https://w3id.org/fep/044f
[FEP-c0e0]: https://w3id.org/fep/c0e0
[FEP-f228]: https://w3id.org/fep/f228
[Gukhanmun]: https://gukhanmun.org/
[RFC 6749]: https://datatracker.ietf.org/doc/html/rfc6749#section-3.3
[GHSA-2g3p-m8c9-hhwh]: https://github.com/MasterKale/SimpleWebAuthn/security/advisories/GHSA-2g3p-m8c9-hhwh
Expand All @@ -163,7 +172,9 @@ To be released.
[#650]: https://github.com/fedify-dev/hollo/pull/650
[#653]: https://github.com/fedify-dev/hollo/issues/653
[#660]: https://github.com/fedify-dev/hollo/pull/660
[#661]: https://github.com/fedify-dev/hollo/issues/661
[#663]: https://github.com/fedify-dev/hollo/pull/663
[#664]: https://github.com/fedify-dev/hollo/pull/664


Version 0.9.22
Expand Down
1 change: 1 addition & 0 deletions compose-fs.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@ services:
SECRET_KEY: "${SECRET_KEY}"
LOG_LEVEL: "${LOG_LEVEL}"
BEHIND_PROXY: "${BEHIND_PROXY}"
# REMOTE_REPLIES_SCRAPE_MAX_REQUESTS: "200"
# GUKHANMUN: "ko,ko-*"
DRIVE_DISK: fs
STORAGE_URL_BASE: http://localhost:3000/assets/
Expand Down
1 change: 1 addition & 0 deletions compose.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@ services:
SECRET_KEY: "${SECRET_KEY}"
LOG_LEVEL: "${LOG_LEVEL}"
BEHIND_PROXY: "${BEHIND_PROXY}"
# REMOTE_REPLIES_SCRAPE_MAX_REQUESTS: "200"
# GUKHANMUN: "ko,ko-*"
DRIVE_DISK: s3
STORAGE_URL_BASE: http://localhost:9000/hollo/
Expand Down
30 changes: 24 additions & 6 deletions docs/src/content/docs/install/env.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -158,18 +158,35 @@ encountered first time.

### `REMOTE_REPLIES_SCRAPE_DEPTH` <Badge text="Optional" />

The number of remote reply levels to scrape in background worker jobs.
Set this to `0` to disable remote replies scraping.
The number of remote reply levels to scrape in background jobs. Hollo first
reads a remote post’s FEP-f228 `context` collection, when present, and uses
`replies` collections as a fallback. Set this to `0` to disable both.
Missing ancestors outside the context collection are not fetched. Posts
with missing parents are stored, but remain disconnected from those parents
in the status-context reply tree.

`2` by default.

### `REMOTE_REPLIES_SCRAPE_MAX_ITEMS` <Badge text="Optional" />

The maximum number of reply items to persist from a single remote replies
scraping job.
The maximum number of items yielded by one context backfill attempt,
including already stored posts. For a `replies` job, this limits persisted
reply items. Reaching the cap leaves the context partial and enables the
seed posts’ `replies` fallback.

`100` by default.

### `REMOTE_REPLIES_SCRAPE_MAX_REQUESTS` <Badge text="Optional" />

The maximum number of document-loader calls in one backfill attempt, shared
by collection traversal and post/actor persistence. Cached documents and
built-in JSON-LD contexts do not count; custom JSON-LD contexts do.
This does not limit redirects or signature retries inside the loader,
WebFinger, NodeInfo, media downloads, or preview-card requests. Reaching the
cap leaves the context partial and enables the `replies` fallback.

`200` by default.

### `REMOTE_EMOJI_REACTIONS_FETCH_MAX_ITEMS` <Badge text="Optional" />

The maximum number of emoji reaction items to import from a remote
Expand All @@ -193,8 +210,9 @@ The fallback delay before retrying a remote replies scraping job after an HTTP

### `REMOTE_REPLIES_SCRAPE_COOLDOWN_SECONDS` <Badge text="Optional" />

The time window during which completed remote replies scraping jobs suppress
duplicate jobs for the same replies collection.
The time window during which completed remote scraping jobs suppress
duplicates for the same collection. Failed context jobs also observe this
cooldown; their seeds can still fall back to `replies` collections.

`300` by default.

Expand Down
33 changes: 25 additions & 8 deletions docs/src/content/docs/ja/install/env.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -155,16 +155,32 @@ Gukhanmunの読み込みまたは変換に失敗すると、投稿リクエス

### `REMOTE_REPLIES_SCRAPE_DEPTH` <Badge text="オプション" />

バックグラウンドワーカージョブで取得するリモート返信の階層数。
`0`に設定すると、リモート返信の取得を無効にします。
バックグラウンドで取得するリモート返信の階層数です。投稿に FEP-f228 の
`context` コレクションがあれば先に読み、必要に応じて `replies` に
フォールバックします。`0` にすると両方を無効にします。コンテキスト
コレクションに含まれない祖先の投稿は取得しません。親がない投稿は保存しますが、
ステータスコンテキストの返信ツリーではその親に接続されません。

デフォルトは`2`です。
デフォルトは `2` です。

### `REMOTE_REPLIES_SCRAPE_MAX_ITEMS` <Badge text="オプション" />

1つのリモート返信取得ジョブから保存する返信アイテムの最大数。
コンテキスト取得の各試行で走査する項目数の上限です。保存済みの投稿も
含みます。`replies` ジョブでは保存する返信数を制限します。上限に達すると
部分取得として記録し、起点となる投稿の `replies` にフォールバックします。

デフォルトは`100`です。
デフォルトは `100` です。

### `REMOTE_REPLIES_SCRAPE_MAX_REQUESTS` <Badge text="オプション" />

取得の各試行でのドキュメントローダー呼び出し数の上限です。コレクションの
走査と投稿・アクターの保存で共有します。キャッシュと組み込み JSON-LD
コンテキストは除外し、独自の JSON-LD コンテキストは含めます。ローダー
内部のリダイレクト・署名再試行、WebFinger、NodeInfo、メディア取得、
プレビューカードのリクエストは制限しません。上限に達すると部分取得として
記録し、`replies` にフォールバックします。

デフォルトは `200` です。

### `REMOTE_EMOJI_REACTIONS_FETCH_MAX_ITEMS` <Badge text="オプション" />

Expand All @@ -189,10 +205,11 @@ Gukhanmunの読み込みまたは変換に失敗すると、投稿リクエス

### `REMOTE_REPLIES_SCRAPE_COOLDOWN_SECONDS` <Badge text="オプション" />

完了したリモート返信取得ジョブが同じ返信コレクションの重複ジョブを
抑制する時間。
完了したリモート取得ジョブが同じコレクションの重複ジョブを抑制する時間です。
失敗したコンテキストジョブにも適用しますが、起点となる投稿は `replies` に
フォールバックできます。

デフォルトは`300`です。
デフォルトは `300` 秒です。

### `MEDIA_PROXY` <Badge text="オプション" />

Expand Down
29 changes: 24 additions & 5 deletions docs/src/content/docs/ko/install/env.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -153,17 +153,35 @@ Hollo가 L7 로드 밸런서 뒤에 위치할 경우 (일반적으로 그래야

### `REMOTE_REPLIES_SCRAPE_DEPTH` <Badge text="선택" />

백그라운드 워커 작업에서 가져올 원격 답글의 계층 수입니다.
`0`으로 설정하면 원격 답글 가져오기를 비활성화합니다.
백그라운드에서 가져올 원격 답글의 깊이입니다. 원격 게시물에 FEP-f228
`context` 컬렉션이 있으면 먼저 읽고, 필요하면 `replies` 컬렉션으로
대체합니다. `0`으로 설정하면 두 수집 방식 모두 비활성화됩니다.
컨텍스트 컬렉션 밖의 누락된 조상 게시물은 가져오지 않습니다. 부모가 없는
게시물은 저장하지만 상태 컨텍스트의 답글 트리에서 해당 부모에 연결되지
않습니다.

기본값은 `2`입니다.

### `REMOTE_REPLIES_SCRAPE_MAX_ITEMS` <Badge text="선택" />

원격 답글 가져오기 작업 하나에서 저장할 답글 항목의 최대 수입니다.
컨텍스트 수집 시도 하나에서 순회할 항목의 최대 수입니다. 이미 저장된
게시물도 포함합니다. `replies` 작업에서는 저장할 답글 항목 수를 제한합니다.
상한에 도달하면 컨텍스트를 부분 수집으로 기록하고 각 시작 게시물의
`replies` 컬렉션을 대신 수집합니다.

기본값은 `100`입니다.

### `REMOTE_REPLIES_SCRAPE_MAX_REQUESTS` <Badge text="선택" />

수집 시도 하나에서 문서 로더를 호출할 수 있는 최대 횟수입니다. 컬렉션
순회와 게시물·작성자 저장이 같은 한도를 공유합니다. 캐시된 문서와 내장
JSON-LD 컨텍스트는 제외하고 사용자 정의 JSON-LD 컨텍스트는 포함합니다.
로더 내부의 리디렉션·서명 재시도, WebFinger, NodeInfo, 미디어 다운로드,
미리보기 카드 요청은 제한하지 않습니다. 한도에 도달하면 컨텍스트를
부분 수집으로 기록하고 `replies` 컬렉션을 대신 수집합니다.

기본값은 `200`입니다.

### `REMOTE_EMOJI_REACTIONS_FETCH_MAX_ITEMS` <Badge text="선택" />

원격 게시물을 저장할 때 원격 `emojiReactions` 컬렉션에서 가져와 저장할
Expand All @@ -187,8 +205,9 @@ Hollo가 L7 로드 밸런서 뒤에 위치할 경우 (일반적으로 그래야

### `REMOTE_REPLIES_SCRAPE_COOLDOWN_SECONDS` <Badge text="선택" />

완료된 원격 답글 가져오기 작업이 같은 답글 컬렉션의 중복 작업을
억제하는 시간(초)입니다.
완료된 원격 수집 작업이 같은 컬렉션의 중복 작업을 억제하는 시간(초)입니다.
실패한 컨텍스트 작업에도 적용되지만, 각 시작 게시물은 `replies` 컬렉션을
대신 수집할 수 있습니다.

기본값은 `300`입니다.

Expand Down
27 changes: 21 additions & 6 deletions docs/src/content/docs/zh-cn/install/env.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -142,15 +142,29 @@ openssl rand -hex 32

### `REMOTE_REPLIES_SCRAPE_DEPTH` <Badge text="可选" />

后台工作器作业抓取远程回复的层级数。设置为`0`可禁用远程回复抓取。
后台获取远程回复的层数。如果帖子有 FEP-f228 `context` 集合,Hollo 会
先读取该集合,并在需要时回退到 `replies` 集合。设为 `0` 可禁用这两种
收集方式。不会获取上下文集合中缺失的祖先帖子。缺少父帖的帖子会保存,
但在状态上下文的回复树中不会连接到该父帖。

默认为`2`。
默认值为 `2`。

### `REMOTE_REPLIES_SCRAPE_MAX_ITEMS` <Badge text="可选" />

单个远程回复抓取作业最多保存的回复项目数。
每次上下文收集尝试遍历的项目数上限,包括已保存的帖子。对于 `replies`
任务,此值限制保存的回复数。达到上限后会记录为部分收集,并回退到
各起始帖子的 `replies` 集合。

默认为`100`。
默认值为 `100`。

### `REMOTE_REPLIES_SCRAPE_MAX_REQUESTS` <Badge text="可选" />

每次收集尝试调用文档加载器的次数上限,由集合遍历和帖子、作者保存共享。
缓存文档和内置 JSON-LD 上下文不计入,自定义 JSON-LD 上下文计入。
不限制加载器内部的重定向、签名重试,以及 WebFinger、NodeInfo、媒体下载
或预览卡片请求。达到上限后会记录为部分收集,并回退到 `replies` 集合。

默认值为 `200`。

### `REMOTE_EMOJI_REACTIONS_FETCH_MAX_ITEMS` <Badge text="可选" />

Expand All @@ -174,9 +188,10 @@ openssl rand -hex 32

### `REMOTE_REPLIES_SCRAPE_COOLDOWN_SECONDS` <Badge text="可选" />

已完成的远程回复抓取作业抑制同一回复集合重复作业的时间窗口秒数。
完成的远程收集任务抑制同一集合重复任务的时间。失败的上下文任务也遵守
此冷却时间,但各起始帖子仍可回退到 `replies` 集合。

默认为`300`。
默认值为 `300` 秒。

### `MEDIA_PROXY` <Badge text="可选" />

Expand Down
27 changes: 21 additions & 6 deletions docs/src/content/docs/zh-tw/install/env.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -142,15 +142,29 @@ openssl rand -hex 32

### `REMOTE_REPLIES_SCRAPE_DEPTH` <Badge text="可選" />

後台工作器作業抓取遠端回覆的層級數。設定為 `0` 可停用遠端回覆抓取。
背景取得遠端回覆的層數。如果貼文有 FEP-f228 `context` 集合,Hollo 會
先讀取該集合,並在需要時回退至 `replies` 集合。設為 `0` 可停用這兩種
收集方式。不會取得上下文集合中缺少的祖先貼文。缺少父貼文的貼文會儲存,
但在狀態上下文的回覆樹中不會連接至該父貼文。

預設為 `2`。
預設值為 `2`。

### `REMOTE_REPLIES_SCRAPE_MAX_ITEMS` <Badge text="可選" />

單個遠端回覆抓取作業最多儲存的回覆項目數。
每次上下文收集嘗試遍歷的項目數上限,包含已儲存的貼文。對於 `replies`
工作,此值限制儲存的回覆數。達到上限後會記錄為部分收集,並回退至
各起始貼文的 `replies` 集合。

預設為 `100`。
預設值為 `100`。

### `REMOTE_REPLIES_SCRAPE_MAX_REQUESTS` <Badge text="可選" />

每次收集嘗試呼叫文件載入器的次數上限,由集合遍歷與貼文、作者儲存共用。
快取文件與內建 JSON-LD 上下文不計入,自訂 JSON-LD 上下文計入。
不限制載入器內部的重新導向、簽章重試,以及 WebFinger、NodeInfo、媒體下載
或預覽卡片請求。達到上限後會記錄為部分收集,並回退至 `replies` 集合。

預設值為 `200`。

### `REMOTE_EMOJI_REACTIONS_FETCH_MAX_ITEMS` <Badge text="可選" />

Expand All @@ -174,9 +188,10 @@ openssl rand -hex 32

### `REMOTE_REPLIES_SCRAPE_COOLDOWN_SECONDS` <Badge text="可選" />

已完成的遠端回覆抓取作業抑制同一回覆集合重複作業的時間窗口秒數。
完成的遠端收集工作抑制同一集合重複工作的時間。失敗的上下文工作也遵守
此冷卻時間,但各起始貼文仍可回退至 `replies` 集合。

預設為 `300`。
預設值為 `300` 秒。

### `MEDIA_PROXY` <Badge text="可選" />

Expand Down
21 changes: 21 additions & 0 deletions drizzle/20261008183749_context_backfill/migration.sql
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
CREATE TYPE "remote_reply_scrape_kind" AS ENUM('replies', 'context');--> statement-breakpoint
ALTER TYPE "remote_reply_scrape_job_status" ADD VALUE 'waiting';--> statement-breakpoint
CREATE TABLE "remote_context_scrape_aliases" (
"iri" text PRIMARY KEY,
"job_id" uuid NOT NULL
);
--> statement-breakpoint
ALTER TABLE "remote_reply_scrape_jobs" DROP CONSTRAINT "remote_reply_scrape_jobs_replies_iri_unique";--> statement-breakpoint
ALTER TABLE "remote_reply_scrape_jobs" ADD COLUMN "kind" "remote_reply_scrape_kind" DEFAULT 'replies'::"remote_reply_scrape_kind" NOT NULL;--> statement-breakpoint
ALTER TABLE "remote_reply_scrape_jobs" ADD COLUMN "blocked_by_job_id" uuid;--> statement-breakpoint
ALTER TABLE "remote_reply_scrape_jobs" ADD COLUMN "partial" boolean DEFAULT false NOT NULL;--> statement-breakpoint
ALTER TABLE "remote_reply_scrape_jobs" ADD COLUMN "yielded_items" integer DEFAULT 0 NOT NULL;--> statement-breakpoint
ALTER TABLE "remote_reply_scrape_jobs" ADD COLUMN "request_count" integer DEFAULT 0 NOT NULL;--> statement-breakpoint
ALTER TABLE "remote_reply_scrape_jobs" ADD COLUMN "skipped_items" integer DEFAULT 0 NOT NULL;--> statement-breakpoint
ALTER TABLE "remote_reply_scrape_jobs" ADD COLUMN "host_requeues" integer DEFAULT 0 NOT NULL;--> statement-breakpoint
ALTER TABLE "remote_reply_scrape_origins" ADD COLUMN "cooldown_until" timestamp with time zone DEFAULT 'epoch' NOT NULL;--> statement-breakpoint
CREATE INDEX "remote_context_scrape_aliases_job_id_index" ON "remote_context_scrape_aliases" ("job_id");--> statement-breakpoint
CREATE UNIQUE INDEX "remote_reply_scrape_jobs_kind_collection_unique" ON "remote_reply_scrape_jobs" ("kind","replies_iri");--> statement-breakpoint
CREATE INDEX "remote_reply_scrape_jobs_blocked_by_job_id_index" ON "remote_reply_scrape_jobs" ("blocked_by_job_id");--> statement-breakpoint
ALTER TABLE "remote_context_scrape_aliases" ADD CONSTRAINT "remote_context_scrape_aliases_t0fqdOkRvVea_fkey" FOREIGN KEY ("job_id") REFERENCES "remote_reply_scrape_jobs"("id") ON DELETE CASCADE;--> statement-breakpoint
ALTER TABLE "remote_reply_scrape_jobs" ADD CONSTRAINT "remote_reply_scrape_jobs_FpYh7OnicGJo_fkey" FOREIGN KEY ("blocked_by_job_id") REFERENCES "remote_reply_scrape_jobs"("id") ON DELETE SET NULL;
Loading
Loading