From 8a093bf90d9ec539b7c5f2be16b824286d2e28fe Mon Sep 17 00:00:00 2001 From: John Payne Date: Wed, 28 Oct 2026 09:59:00 -0400 Subject: [PATCH 01/37] docs: design collected Micro.blog clippings posts --- ...6-10-28-microblog-clippings-post-design.md | 69 +++++++++++++++++++ 1 file changed, 69 insertions(+) create mode 100644 docs/superpowers/specs/2026-10-28-microblog-clippings-post-design.md diff --git a/docs/superpowers/specs/2026-10-28-microblog-clippings-post-design.md b/docs/superpowers/specs/2026-10-28-microblog-clippings-post-design.md new file mode 100644 index 00000000..bf94297e --- /dev/null +++ b/docs/superpowers/specs/2026-10-28-microblog-clippings-post-design.md @@ -0,0 +1,69 @@ +# Micro.blog clippings posts + +## Intent and scope + +Let readers publish selected clippings to Micro.blog, or opt into automatic +publication, with one collected post per book. Each post links to the matched +book's Micro.blog page. This replaces the original bookmark-highlights proposal. +Existing reading-progress and bookshelf behavior continues independently. + +## Approaches + +- Bookmark highlights: originally requested, but requires offsets into archived + bookmark content and does not fit clippings from an ebook. +- A separate post per clipping: straightforward, but automatic sync would create + many posts for the same book. +- One post per book, updated as clippings arrive: selected by the user. Retains + a single shareable collection and avoids repeated publication per clipping. + +## Behavior + +- When Micro.blog is configured, each clipping on the clippings page offers + **Add highlight to Micro.blog**. It selects only that clipping for publication + and schedules creation or updating of the book's collected post. +- The plugin offers **Automatically sync highlights**, disabled by default. + Enabling it makes existing and future live clippings eligible for publication. +- A missing book match never blocks reader sync or creates a book solely to + publish clippings. Persist publication intent even while unmatched. Once a + match is available, the next device sync or plugin **Sync now** includes + eligible existing clippings, even when they were not changed on that sync. +- Manual selection works with automatic publication disabled. Disabling the + automatic option stops automatic additions; explicitly selected clippings + remain eligible. Already published content is retained. +- Successful unchanged syncs do not create posts or rewrite existing posts. + Normal transient failures are retried using the connector queue. + +## Implementation choices + +These details are proposed defaults for implementation: + +- Publish a normal blog post through Micro.blog's Micropub endpoint, using the + existing plugin credential. Save its returned URL and update that same URL. +- Title the post **Highlights from {book title}**. Include a book link, followed + by eligible passages as quotations with their notes and chapter labels. + Use a stable reading order so unchanged content has a stable fingerprint. +- Persist per-clipping publication intent separately from queue attempts, and + persist the post URL and last successful content fingerprint per user/book. + Resolve document aliases using the existing merge model. +- Use a per-book queue key so multiple new clippings coalesce into one update. + Read current eligible clippings when processing the job rather than publishing + an obsolete snapshot. Preserve selected content when automatic sync is disabled. +- Treat the collected post as managed by crosspoint-sync: later syncs replace its + generated body. Do not delete remote posts on unlink or local clipping deletion. +- Do not blindly retry a creation with an ambiguous outcome: recover the existing + post through Micropub source lookup before allowing another creation. If a saved + post is missing or cannot be recovered, surface an actionable error instead of + silently creating a duplicate. + +## Verification + +Cover default-off behavior, manual selection, automatic backfill, unmatched books, +manual matching followed by an otherwise unchanged sync, one post per book, +unchanged-content deduplication, updates, retry/recovery, per-user isolation, and +canonical document aliases. Run relevant backend tests, the backend build, and +frontend checks available in this repository. + +## References + +- [Micro.blog Micropub creation and updates](https://microblog.dev/api/posting/) +- [Micro.blog Micropub guide](https://book.micro.blog/micropub/) From dbeeb827814a6728a949376a389b224d162fa6ff Mon Sep 17 00:00:00 2001 From: John Payne Date: Wed, 28 Oct 2026 10:24:20 -0400 Subject: [PATCH 02/37] feat: persist Micro.blog clipping selections and publication state --- .../2026-10-28-microblog-clippings-posts.md | 78 +++++++++++ migrations/0040_microblog_clippings.sql | 18 +++ src/connectors/microblog-clippings.ts | 126 ++++++++++++++++++ src/models/merge.ts | 2 + test/microblog-clippings.test.ts | 73 ++++++++++ 5 files changed, 297 insertions(+) create mode 100644 docs/superpowers/plans/2026-10-28-microblog-clippings-posts.md create mode 100644 migrations/0040_microblog_clippings.sql create mode 100644 src/connectors/microblog-clippings.ts create mode 100644 test/microblog-clippings.test.ts diff --git a/docs/superpowers/plans/2026-10-28-microblog-clippings-posts.md b/docs/superpowers/plans/2026-10-28-microblog-clippings-posts.md new file mode 100644 index 00000000..333a4a75 --- /dev/null +++ b/docs/superpowers/plans/2026-10-28-microblog-clippings-posts.md @@ -0,0 +1,78 @@ +# Micro.blog Clippings Posts Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Publish eligible clippings as one maintained Micro.blog post per book, with reliable catch-up after matching. + +**Architecture:** Keep publication intent and remote post state in dedicated tables, independent of queue attempts. A Micro.blog-specific module selects canonical clippings and schedules one coalesced job per book; the connector publishes the latest selected content via Micropub. Extend the current settings UI and clippings hub without changing other connectors' highlight behavior. + +**Tech Stack:** TypeScript, Hono, SQLite, Vitest, React, existing HTTP transport. + +**Spec:** `docs/superpowers/specs/2026-10-28-microblog-clippings-post-design.md` + +## Global Constraints + +- **Automatically sync highlights**, disabled by default. +- A missing book match never blocks reader sync or creates a book solely to publish clippings. +- Manual selection works with automatic publication disabled. +- Do not delete remote posts on unlink or local clipping deletion. +- Do not blindly retry a creation with an ambiguous outcome. +- Leave unrelated work untouched; use an isolated worktree for implementation. + +## Review Focus + +- Failed creation after remote acceptance must recover the original post rather than duplicate it (Task 2). +- Disabling automatic publication must preserve already published content and manual selections (Tasks 1–2). +- Alias merges must preserve selections and avoid a second post (Tasks 1–2). +- An unchanged device sync after a manual match must publish earlier eligible clippings (Task 3). +- Unlinked, disabled, or another user's account must never publish a requested clipping (Task 3). + +### Task 1: Durable selection and post state + +**Files:** New migration after the current migration sequence; new `src/connectors/microblog-clippings.ts`; modify `src/models/merge.ts`; new `test/microblog-clippings.test.ts`. + +**Interfaces:** Produce `selectMicroblogClipping(db: DB, userId: number, document: string, clippingId: string): void`, `scheduleMicroblogClippings(db: DB, userId: number, document?: string): number`, and `microblogClippingsContent(db: DB, userId: number, document: string, bookUrl: string): { title: string; content: string; fingerprint: string } | null`. + +- [ ] Write failing tests: manual selection includes only the requested live clipping; automatic selection includes all existing live clippings; stable reading order produces an unchanged fingerprint; disabling automatic publication retains previously published passages; alias selection resolves to the canonical book; users cannot share state. +- [ ] Run `npm test -- test/microblog-clippings.test.ts`; confirm failures are missing behavior. +- [ ] Add selection rows with manual/automatic intent and published content snapshots; add per-book remote URL, fingerprint, and creation-attempt/recovery state. Use existing database migration and transaction patterns. Schedule only enabled accounts, eligible content, and usable existing matches; coalesce with `microblog:clippings` as the queue key and `highlight` as the event kind. +- [ ] Implement the interfaces above using the merged clipping view. Render **Highlights from {book title}**, the matched book link, quoted passages, notes, and chapters; escape user text appropriately. Carry selections/state through document merges. If both merged documents already have remote posts, retain one authoritative URL and surface the conflict without deleting either post. +- [ ] Re-run the new tests plus existing merge tests; commit the tested deliverable. + +### Task 2: Micropub creation, updates, and recovery + +**Files:** Modify `src/connectors/microblog.ts`, `src/connectors/runner.ts`; add publication functions to `src/connectors/microblog-clippings.ts`; extend `test/microblog-helpers.ts`; add `test/microblog-clippings-post.test.ts`. + +**Interfaces:** Consume Task 1's content function. Produce `publishMicroblogClippings(cred: Credential, match: Match, event: OutboundEvent, http: HttpTransport, ctx: ConnectorContext): Promise`; dispatch Micro.blog highlight jobs to it before shelf reconciliation. + +- [ ] Write failing transport tests for first creation returning `Location`, subsequent JSON Micropub update to the saved URL, no request for an unchanged fingerprint, and no shelf operations for highlight jobs. +- [ ] Add failing tests for 401/403 reauthentication, retryable 429/5xx, malformed or absent creation responses, ambiguous network completion, paginated source recovery finding a unique owned post, ambiguous recovery requiring attention, and a missing saved post requiring attention rather than recreation. Verify automatic-off updates retain snapshots of published passages. +- [ ] Run `npm test -- test/microblog-clippings-post.test.ts`; verify expected failures. +- [ ] Implement JSON Micropub requests without the current form-only helper overriding content type. Persist the creation URL before marking success. Save a stable recovery identity before creation, query the account's Micropub source with pagination after uncertain outcomes, and adopt only one unambiguously identified result; otherwise fail with an actionable message and do not create again. +- [ ] Guard `shouldPush` so highlight events do not inherit percentage filtering. In the runner, Micro.blog highlights require an existing match and never invoke `createBook`; preserve other connectors' lifecycle behavior. Resolve ISBN/book-page identity from verified book metadata or the shelf inventory instead of assuming an internal shelf ID is an ISBN. +- [ ] Run new tests and existing Micro.blog unit/integration tests; commit the tested deliverable. + +### Task 3: Settings, manual API, and catch-up triggers + +**Files:** Modify `src/routes/v1/connectors.ts`, `src/routes/v1/clippings.ts`, `src/connectors/fanout.ts`, `src/connectors/microblog.ts`, and the shared progress refresh entry point `src/connectors/refresh.ts`; extend backend integration tests. + +**Interfaces:** Add `POST /api/v1/connectors/microblog/clippings/:document/:clippingId`, returning `{ queued: boolean, waiting_for_match: boolean }`. Consume Task 1's selection and scheduling functions. Add connector toggle `sync_highlights`, label **Automatically sync highlights**, default `false`. + +- [ ] Write failing endpoint tests for authenticated manual selection, missing/deleted clipping, cross-user requests, unlinked/disabled accounts, and idempotent repeated requests. Assert disabled automatic sync still allows manual publication intent. +- [ ] Write failing catch-up tests: select or automatically include clippings while unmatched, manually match the book, perform an unchanged device progress GET or clipping sync, drain the queue, and observe one collected post. Cover legacy and v1 progress paths, plugin **Sync now**, enabling/disabling the option, and zero shelf creation for unmatched highlights. +- [ ] Run the new integration tests and confirm expected failures. +- [ ] Register the default-off option; schedule automatic selection on clipping sync and option enable. Schedule eligible book jobs on shared progress refresh, clipping sync, and plugin backfill, even without changed clippings. Persist unmatched intent without dead-letter churn. Bypass generic per-clipping fanout/backfill for Micro.blog so the single per-book job is authoritative. +- [ ] Implement the manual endpoint with canonical resolution, ownership checks, configured/enabled-account checks, and durable selection before scheduling. On unlink, clear local publication credentials/state according to the existing unlink lifecycle while leaving remote posts intact; ensure a replacement account cannot inherit an old account's remote post URL. +- [ ] Run backend integration and existing connector/queue tests; commit the tested deliverable. + +### Task 4: Clippings action and final verification + +**Files:** Modify `app/src/Clippings.jsx`, `app/src/api.js`; add `app/src/Clippings.test.jsx`; update Micro.blog help copy where appropriate. + +**Interfaces:** Add `api.addMicroblogHighlight(session, document, clippingId)` calling Task 3's endpoint. Use existing connector listing and settings rendering. + +- [ ] Write failing UI tests for an action shown only with configured Micro.blog, selecting an individual clipping, waiting-for-match feedback, request failure feedback, and preventing repeated clicks while a request is pending. +- [ ] Run `npm --prefix app test -- src/Clippings.test.jsx`; confirm expected failures. +- [ ] Add **Add highlight to Micro.blog** to clipping cards, including the daily clipping. Explain successful scheduling or waiting for a book match without claiming the post was already published. Display that the optional sync publishes a managed post per book; preserve existing share/delete behavior. +- [ ] Run the UI tests, `npm test`, `npm run build`, `npm --prefix app test`, and `npm --prefix app run build`. Report any failures by name. Check the complete diff and obtain an independent code review; address verified findings with regression tests. +- [ ] Commit the tested implementation and report changed behavior, checks, and any remaining limitations. Do not publish real posts during development or deploy without a user request. diff --git a/migrations/0040_microblog_clippings.sql b/migrations/0040_microblog_clippings.sql new file mode 100644 index 00000000..77f8e394 --- /dev/null +++ b/migrations/0040_microblog_clippings.sql @@ -0,0 +1,18 @@ +CREATE TABLE microblog_clipping_selections ( + user_id INTEGER NOT NULL REFERENCES users(id) ON DELETE CASCADE, + document TEXT NOT NULL, + clipping_id TEXT NOT NULL, + manual INTEGER NOT NULL DEFAULT 0, + published_snapshot TEXT, + PRIMARY KEY (user_id, document, clipping_id) +); +CREATE TABLE microblog_clipping_posts ( + user_id INTEGER NOT NULL REFERENCES users(id) ON DELETE CASCADE, + document TEXT NOT NULL, + identity TEXT NOT NULL, + url TEXT, + fingerprint TEXT, + attempted INTEGER NOT NULL DEFAULT 0, + error TEXT, + PRIMARY KEY (user_id, document) +); diff --git a/src/connectors/microblog-clippings.ts b/src/connectors/microblog-clippings.ts new file mode 100644 index 00000000..0d239544 --- /dev/null +++ b/src/connectors/microblog-clippings.ts @@ -0,0 +1,126 @@ +import { createHash, randomUUID } from 'node:crypto'; +import { withTransaction, type DB } from '../db/db.js'; +import { clippingDocuments, mergedClippingsCte, resolveDocument } from '../models/merge.js'; +import { documentMeta, getAccount, getMatch } from './store.js'; +import { enqueue } from './queue.js'; +import { nowSeconds } from '../models/sync.js'; + +export interface ClippingsPost { + identity: string; + url: string | null; + fingerprint: string | null; + attempted: number; + error: string | null; +} +interface Passage { + id: string; text: string; note: string | null; chapter_title: string; + spine_index: number; start_offset: number | null; created_at: number; +} + +function automatic(db: DB, userId: number): boolean { + const row = db.prepare("SELECT options FROM connector_accounts WHERE user_id=? AND connector_id='microblog'") + .get(userId) as { options: string | null } | undefined; + try { return JSON.parse(row?.options || '{}').sync_highlights === true; } catch { return false; } +} + +function liveClippings(db: DB, userId: number, document: string): Passage[] { + const docs = clippingDocuments(db, userId, document); + return db.prepare(`${mergedClippingsCte(docs.length)} SELECT id,text,note,chapter_title,spine_index,start_offset,created_at + FROM merged WHERE document=? AND deleted=0`).all(userId, ...docs, document) as unknown as Passage[]; +} + +export function clippingsPost(db: DB, userId: number, document: string): ClippingsPost { + db.prepare('INSERT OR IGNORE INTO microblog_clipping_posts(user_id,document,identity) VALUES(?,?,?)') + .run(userId, document, randomUUID()); + return db.prepare('SELECT identity,url,fingerprint,attempted,error FROM microblog_clipping_posts WHERE user_id=? AND document=?') + .get(userId, document) as unknown as ClippingsPost; +} + +export function selectMicroblogClipping(db: DB, userId: number, document: string, clippingId: string): void { + document = resolveDocument(db, userId, document); + if (!liveClippings(db, userId, document).some((c) => c.id === clippingId)) throw new Error('Clipping not found'); + db.prepare(`INSERT INTO microblog_clipping_selections(user_id,document,clipping_id,manual) VALUES(?,?,?,1) + ON CONFLICT(user_id,document,clipping_id) DO UPDATE SET manual=1`).run(userId, document, clippingId); +} + +/** Include published snapshots even after deletion or disabling automatic additions. */ +function passages(db: DB, userId: number, document: string): Passage[] { + const live = liveClippings(db, userId, document); + const enabled = automatic(db, userId); + const selections = db.prepare('SELECT clipping_id,manual,published_snapshot FROM microblog_clipping_selections WHERE user_id=? AND document=?') + .all(userId, document) as { clipping_id: string; manual: number; published_snapshot: string | null }[]; + return selections.flatMap((s) => { + const current = live.find((c) => c.id === s.clipping_id); + if ((enabled || s.manual) && current) return [current]; + return s.published_snapshot ? [JSON.parse(s.published_snapshot) as Passage] : []; + }).sort((a, b) => a.spine_index - b.spine_index || (a.start_offset ?? 0) - (b.start_offset ?? 0) + || a.created_at - b.created_at || a.id.localeCompare(b.id)); +} + +function escapeHtml(s: string): string { + return s.replace(/&/g, '&').replace(//g, '>').replace(/"/g, '"'); +} +function paragraph(s: string): string { return escapeHtml(s).replace(/\n/g, '
'); } + +export function microblogClippingsContent(db: DB, userId: number, document: string, bookUrl: string): { title: string; content: string; fingerprint: string } | null { + document = resolveDocument(db, userId, document); + const selected = passages(db, userId, document); + if (!selected.length) return null; + const meta = documentMeta(db, userId, document); + const title = `Highlights from ${meta.title || meta.filename || 'Untitled book'}`; + const post = clippingsPost(db, userId, document); + const content = [`

${escapeHtml(meta.title || 'View book on Micro.blog')}${meta.author ? ` by ${escapeHtml(meta.author)}` : ''}

`, + ...selected.map((c) => `

${paragraph(c.text)}

${c.note ? `

${paragraph(c.note)}

` : ''}${c.chapter_title ? `

${paragraph(c.chapter_title)}

` : ''}`), + ``].join('\n\n'); + return { title, content, fingerprint: createHash('sha256').update(JSON.stringify([title, content])).digest('hex') }; +} + +export function savePublishedClippings(db: DB, userId: number, document: string, selected: Passage[]): void { + for (const c of selected) db.prepare('UPDATE microblog_clipping_selections SET published_snapshot=? WHERE user_id=? AND document=? AND clipping_id=?') + .run(JSON.stringify(c), userId, document, c.id); +} + +export function scheduleMicroblogClippings(db: DB, userId: number, document?: string): number { + const account = getAccount(db, userId, 'microblog'); + if (!account?.enabled) return 0; + const docs = document ? [resolveDocument(db, userId, document)] : [...new Set((db.prepare(` + SELECT document FROM clippings WHERE user_id=? UNION SELECT document FROM microblog_clipping_selections WHERE user_id=?`) + .all(userId, userId) as { document: string }[]).map((r) => resolveDocument(db, userId, r.document)))]; + let queued = 0; + for (const doc of docs) { + if (automatic(db, userId)) { + const insert = db.prepare('INSERT OR IGNORE INTO microblog_clipping_selections(user_id,document,clipping_id) VALUES(?,?,?)'); + for (const c of liveClippings(db, userId, doc)) insert.run(userId, doc, c.id); + } + if (!getMatch(db, userId, 'microblog', doc)?.external_id || !passages(db, userId, doc).length) continue; + const state = clippingsPost(db, userId, doc); + if (state.error) continue; + const existing = db.prepare("SELECT status FROM connector_queue WHERE user_id=? AND connector_id='microblog' AND document=? AND kind='microblog:clippings'") + .get(userId, doc) as { status: string } | undefined; + if (existing?.status === 'pending') continue; // Preserve backoff after a transient failure. + enqueue(db, userId, 'microblog', { kind: 'highlight', document: doc, timestamp: nowSeconds() }, 'microblog:clippings'); + queued++; + } + return queued; +} + +/** Migrate publication state before the ordinary document side tables move. */ +export function mergeMicroblogClippings(db: DB, userId: number, from: string, into: string): void { + const source = db.prepare('SELECT * FROM microblog_clipping_posts WHERE user_id=? AND document=?').get(userId, from) as unknown as ClippingsPost | undefined; + const target = db.prepare('SELECT * FROM microblog_clipping_posts WHERE user_id=? AND document=?').get(userId, into) as unknown as ClippingsPost | undefined; + if (source && target && (source.url || source.attempted) && (target.url || target.attempted)) { + db.prepare('UPDATE microblog_clipping_posts SET error=? WHERE user_id=? AND document=?') + .run('Merged books both have Micro.blog publication history; review their posts before syncing.', userId, into); + } else if (source && (!target || source.url || source.attempted)) { + db.prepare('DELETE FROM microblog_clipping_posts WHERE user_id=? AND document=?').run(userId, into); + db.prepare('UPDATE microblog_clipping_posts SET document=? WHERE user_id=? AND document=?').run(into, userId, from); + } + db.prepare('DELETE FROM microblog_clipping_posts WHERE user_id=? AND document=?').run(userId, from); + db.prepare(`INSERT INTO microblog_clipping_selections(user_id,document,clipping_id,manual,published_snapshot) + SELECT user_id,?,clipping_id,manual,published_snapshot FROM microblog_clipping_selections WHERE user_id=? AND document=? + ON CONFLICT(user_id,document,clipping_id) DO UPDATE SET manual=MAX(manual,excluded.manual), + published_snapshot=COALESCE(published_snapshot,excluded.published_snapshot)`).run(into, userId, from); + db.prepare('DELETE FROM microblog_clipping_selections WHERE user_id=? AND document=?').run(userId, from); +} + +export const _microblogClippings = { passages }; diff --git a/src/models/merge.ts b/src/models/merge.ts index 1c5bf829..ac109870 100644 --- a/src/models/merge.ts +++ b/src/models/merge.ts @@ -2,6 +2,7 @@ import { withTransaction, type DB } from '../db/db.js'; import { nextChangeSeq } from './changes.js'; import { md5Hex } from '../auth/password.js'; import { autoUnpause } from './pause.js'; +import { mergeMicroblogClippings } from '../connectors/microblog-clippings.js'; import { MANUAL_COLUMNS, mergeManualSql } from './manual-details.js'; /** @@ -173,6 +174,7 @@ export function mergeDocuments( const last = db.prepare('SELECT MAX(at) AS at FROM progress_log WHERE user_id = ? AND document = ?').get(userId, into) as { at: number | null }; if (last.at !== null) autoUnpause(db, userId, into, last.at); + mergeMicroblogClippings(db, userId, from, into); // Uniquely-keyed side tables: move what fits, drop the (rare) conflicts. for (const table of ['bookmarks', 'clippings', 'connector_matches', 'connector_queue']) { db.prepare(`UPDATE OR IGNORE ${table} SET document = ? WHERE user_id = ? AND document = ?`).run( diff --git a/test/microblog-clippings.test.ts b/test/microblog-clippings.test.ts new file mode 100644 index 00000000..702483ea --- /dev/null +++ b/test/microblog-clippings.test.ts @@ -0,0 +1,73 @@ +import { afterEach, beforeEach, expect, it } from 'vitest'; +import { makeTestApp, registerUser, DOC } from './helpers.js'; +import { resetEncryptionKeyCache } from '../src/crypto/secrets.js'; +import { upsertAccount, saveMatch } from '../src/connectors/store.js'; +import { selectMicroblogClipping, scheduleMicroblogClippings, microblogClippingsContent } from '../src/connectors/microblog-clippings.js'; +import { mergeDocuments } from '../src/models/merge.js'; + +beforeEach(() => { process.env.TOKEN_ENC_KEY = 'a'.repeat(64); resetEncryptionKeyCache(); }); +afterEach(() => { delete process.env.TOKEN_ENC_KEY; resetEncryptionKeyCache(); }); + +async function setup() { + const { app, db } = makeTestApp(); + const { username } = await registerUser(app); + const uid = (db.prepare('SELECT id FROM users WHERE username=?').get(username) as { id: number }).id; + upsertAccount(db, uid, 'microblog', { token: 'token' }, null); + db.prepare('INSERT INTO documents(user_id,document,title,author,updated_at) VALUES(?,?,?,?,0)').run(uid, DOC, 'Test book', 'Author'); + for (const [id, text, spine] of [['one', 'First passage', 1], ['two', 'Second passage', 2]] as const) { + db.prepare('INSERT INTO clippings(user_id,document,id,text,spine_index,created_at,updated_at) VALUES(?,?,?,?,?,1,1)').run(uid, DOC, id, text, spine); + } + return { db, uid }; +} + +it('persists manual intent without a match and queues one book after matching', async () => { + const { db, uid } = await setup(); + selectMicroblogClipping(db, uid, DOC, 'two'); + expect(scheduleMicroblogClippings(db, uid, DOC)).toBe(0); + saveMatch(db, uid, 'microblog', DOC, { externalId: '123', externalEdition: '9781786487849', confidence: 1 }, 'manual'); + expect(scheduleMicroblogClippings(db, uid, DOC)).toBe(1); + scheduleMicroblogClippings(db, uid, DOC); + expect(db.prepare("SELECT COUNT(*) n FROM connector_queue WHERE connector_id='microblog'").get()).toEqual({ n: 1 }); + const content = microblogClippingsContent(db, uid, DOC, 'https://micro.blog/books/9781786487849'); + expect(content?.title).toBe('Highlights from Test book'); + expect(content?.content).toContain('Second passage'); + expect(content?.content).not.toContain('First passage'); + expect(microblogClippingsContent(db, uid, DOC, 'https://micro.blog/books/9781786487849')?.fingerprint).toBe(content?.fingerprint); + db.close(); +}); + +it('automatic mode includes existing clippings in reading order and isolates users', async () => { + const { db, uid } = await setup(); + db.prepare("UPDATE connector_accounts SET options=? WHERE user_id=? AND connector_id='microblog'").run('{"sync_highlights":true}', uid); + scheduleMicroblogClippings(db, uid, DOC); + const content = microblogClippingsContent(db, uid, DOC, 'https://micro.blog/books/9781786487849'); + expect(content?.content.indexOf('First passage')).toBeLessThan(content!.content.indexOf('Second passage')); + expect(microblogClippingsContent(db, uid + 1, DOC, 'https://micro.blog/books/9781786487849')).toBeNull(); + db.close(); +}); + +it('automatic-off excludes unpublished automatic selections but preserves manual intent', async () => { + const { db, uid } = await setup(); + db.prepare("UPDATE connector_accounts SET options=? WHERE user_id=?").run('{"sync_highlights":true}', uid); + scheduleMicroblogClippings(db, uid, DOC); + selectMicroblogClipping(db, uid, DOC, 'two'); + db.prepare("UPDATE connector_accounts SET options='{}' WHERE user_id=?").run(uid); + expect(microblogClippingsContent(db, uid, DOC, 'https://micro.blog/books/9781786487849')?.content).not.toContain('First passage'); + expect(microblogClippingsContent(db, uid, DOC, 'https://micro.blog/books/9781786487849')?.content).toContain('Second passage'); + db.close(); +}); + +it('keeps selected clippings and the remote post when merging books', async () => { + const { db, uid } = await setup(); + const target = 'b'.repeat(32); + selectMicroblogClipping(db, uid, DOC, 'one'); + microblogClippingsContent(db, uid, DOC, 'https://micro.blog/books/9781786487849'); + db.prepare('UPDATE microblog_clipping_posts SET url=?,attempted=1 WHERE user_id=?').run('https://example.micro.blog/highlights', uid); + mergeDocuments(db, uid, DOC, target, 10); + expect(microblogClippingsContent(db, uid, DOC, 'https://micro.blog/books/9781786487849')?.content).toContain('First passage'); + selectMicroblogClipping(db, uid, DOC, 'two'); + expect(microblogClippingsContent(db, uid, target, 'https://micro.blog/books/9781786487849')?.content).toContain('Second passage'); + expect(db.prepare('SELECT url,document FROM microblog_clipping_posts WHERE user_id=?').get(uid)) + .toEqual({ url: 'https://example.micro.blog/highlights', document: target }); + db.close(); +}); From 7b4bf70df241530afbaa5cb16c91fc538f729344 Mon Sep 17 00:00:00 2001 From: John Payne Date: Wed, 28 Oct 2026 10:26:43 -0400 Subject: [PATCH 03/37] feat: publish and recover collected Micro.blog clippings posts --- src/connectors/microblog-clippings.ts | 115 +++++++++++++++++- src/connectors/microblog.ts | 13 +- src/connectors/registry.ts | 1 + src/connectors/runner.ts | 6 + test/microblog-clippings-post.test.ts | 164 ++++++++++++++++++++++++++ 5 files changed, 297 insertions(+), 2 deletions(-) create mode 100644 test/microblog-clippings-post.test.ts diff --git a/src/connectors/microblog-clippings.ts b/src/connectors/microblog-clippings.ts index 0d239544..0175c03f 100644 --- a/src/connectors/microblog-clippings.ts +++ b/src/connectors/microblog-clippings.ts @@ -4,6 +4,7 @@ import { clippingDocuments, mergedClippingsCte, resolveDocument } from '../model import { documentMeta, getAccount, getMatch } from './store.js'; import { enqueue } from './queue.js'; import { nowSeconds } from '../models/sync.js'; +import { ConnectorOperationError, type Credential, type Match, type OutboundEvent, type HttpTransport, type ConnectorContext, type PushResult } from './types.js'; export interface ClippingsPost { identity: string; @@ -123,4 +124,116 @@ export function mergeMicroblogClippings(db: DB, userId: number, from: string, in db.prepare('DELETE FROM microblog_clipping_selections WHERE user_id=? AND document=?').run(userId, from); } -export const _microblogClippings = { passages }; +/** Only publish/recover one book at a time, even when two drains overlap. */ +const publishing = new WeakMap>(); + +async function micropub(http: HttpTransport, token: string, path: string, body?: unknown) { + let res; + try { + res = await http(`https://micro.blog${path}`, { + method: body ? 'POST' : 'GET', + headers: { authorization: `Bearer ${token}`, ...(body ? { 'content-type': 'application/json' } : {}) }, + ...(body ? { body: JSON.stringify(body) } : {}), + }); + } catch { throw new ConnectorOperationError('Micro.blog Micropub connection failed; the next attempt will check for an existing post.', true); } + if (res.status >= 300) { + throw new ConnectorOperationError(`Micro.blog Micropub failed (${res.status})`, res.status === 429 || res.status >= 500, + res.status === 401 || res.status === 403); + } + return res; +} + +function postUrl(value: unknown): string | null { + if (typeof value !== 'string') return null; + try { const u = new URL(value); return u.protocol === 'https:' && !u.username && !u.password ? u.href : null; } catch { return null; } +} + +async function recoverPost(http: HttpTransport, token: string, identity: string): Promise { + const matches = new Set(); + // Bound the scan; an incomplete scan never permits creating a replacement. + for (let offset = 0; offset < 10000; offset += 100) { + const res = await micropub(http, token, `/micropub?q=source&limit=100&offset=${offset}`); + const payload = await res.json() as { items?: { properties?: { content?: unknown[]; url?: unknown[] } }[] }; + if (!Array.isArray(payload.items)) throw new ConnectorOperationError('Micro.blog post recovery returned an invalid source feed.', true); + for (const item of payload.items) { + const content = JSON.stringify(item.properties?.content ?? []); + if (!content.includes(`crosspoint-sync:${identity}`)) continue; + const url = postUrl(item.properties?.url?.[0]); + if (url) matches.add(url); + } + if (payload.items.length < 100) return matches.size === 1 ? [...matches][0] : null; + } + return null; +} + +export async function publishMicroblogClippings( + cred: Credential, match: Match, event: OutboundEvent, http: HttpTransport, ctx: ConnectorContext +): Promise { + const { db, userId } = ctx; + const document = resolveDocument(db, userId, event.document); + const key = `${userId}:${document}`; + let locks = publishing.get(db); + if (!locks) { locks = new Set(); publishing.set(db, locks); } + if (locks.has(key)) return { ok: false, retryable: true, error: 'Micro.blog book publication is already in progress' }; + locks.add(key); + const permanent = (error: string): PushResult => { + db.prepare('UPDATE microblog_clipping_posts SET error=? WHERE user_id=? AND document=?').run(error, userId, document); + return { ok: false, retryable: false, error }; + }; + try { + const token = typeof cred.token === 'string' ? cred.token.trim() : ''; + if (!token) throw new ConnectorOperationError('Missing Micro.blog token', false, true); + // Caller resolves a verified ISBN; internal bookshelf IDs are never used as ISBNs. + const isbn = match.externalEdition; + if (!isbn || !/^(?:\d{13}|\d{9}[\dXx])$/.test(isbn)) return permanent('Micro.blog book ISBN is unavailable; choose a book match with an ISBN.'); + const selected = passages(db, userId, document); + const rendered = microblogClippingsContent(db, userId, document, `https://micro.blog/books/${encodeURIComponent(isbn)}`); + if (!rendered) return { ok: true }; + const state = clippingsPost(db, userId, document); + if (state.error) return { ok: false, retryable: false, error: state.error }; + if (rendered.fingerprint === state.fingerprint && state.url) return { ok: true }; + let url = state.url; + if (!url && state.attempted) { + url = await recoverPost(http, token, state.identity); + if (!url) return permanent('Micro.blog post creation could not be resolved uniquely. Review your blog posts before retrying; no duplicate was created.'); + db.prepare('UPDATE microblog_clipping_posts SET url=? WHERE user_id=? AND document=?').run(url, userId, document); + } + if (url) { + try { + // Detect deleted posts before replacing generated content. + await micropub(http, token, `/micropub?q=source&url=${encodeURIComponent(url)}`); + await micropub(http, token, '/micropub', { action: 'update', url, + replace: { name: [rendered.title], content: [{ html: rendered.content }] } }); + } catch (err) { + if (err instanceof ConnectorOperationError && !err.retryable && !err.needsReauth) return permanent(`${err.message}. Review the saved Micro.blog post before syncing again.`); + throw err; + } + } else { + db.prepare('UPDATE microblog_clipping_posts SET attempted=1 WHERE user_id=? AND document=?').run(userId, document); + let res; + try { + res = await micropub(http, token, '/micropub', { type: ['h-entry'], properties: { + name: [rendered.title], content: [{ html: rendered.content }], 'post-status': ['published'], + 'mp-slug': [`crosspoint-highlights-${state.identity}`], + } }); + } catch (err) { + if (err instanceof ConnectorOperationError && (err.needsReauth || err.message.includes('(429)'))) { + db.prepare('UPDATE microblog_clipping_posts SET attempted=0 WHERE user_id=? AND document=?').run(userId, document); + } else if (err instanceof ConnectorOperationError && !err.retryable) return permanent(err.message); + throw err; + } + url = postUrl(res.headers?.get('location')); + if (!url) { + throw new ConnectorOperationError('Micro.blog accepted the post without its URL; the next attempt will recover it.', true); + } + // Persist Location immediately, before any other work can fail. + db.prepare('UPDATE microblog_clipping_posts SET url=? WHERE user_id=? AND document=?').run(url, userId, document); + } + withTransaction(db, () => { + savePublishedClippings(db, userId, document, selected); + db.prepare('UPDATE microblog_clipping_posts SET fingerprint=?,error=NULL WHERE user_id=? AND document=?') + .run(rendered.fingerprint, userId, document); + }); + return { ok: true }; + } finally { locks.delete(key); } +} diff --git a/src/connectors/microblog.ts b/src/connectors/microblog.ts index b41e210e..0c91c18c 100644 --- a/src/connectors/microblog.ts +++ b/src/connectors/microblog.ts @@ -1,4 +1,5 @@ import { decideMatch, type Candidate } from './matching.js'; +import { publishMicroblogClippings } from './microblog-clippings.js'; import { ConnectorOperationError, type Credential, @@ -361,6 +362,7 @@ async function reconcileBook( } function shouldPush(ev: OutboundEvent, canonicalPercentage?: number | null): boolean { + if (ev.kind === 'highlight') return true; if (ev.kind === 'progress' && (ev.percentage ?? 0) <= 0) return false; if (canonicalPercentage == null) return true; if (canonicalPercentage <= 0) return false; @@ -382,7 +384,16 @@ export const microblogConnector: Connector = { search: searchBooks, match: matchBook, createBook, - push: reconcileBook, + async push(cred, match, ev, http, ctx) { + if (ev.kind !== 'highlight') return reconcileBook(cred, match, ev, http); + if (!ctx) return { ok: false, retryable: false, error: 'Micro.blog publication requires local storage' }; + let isbn = match.externalEdition; + if (!isbn) { + if (/^(?:\d{13}|\d{9}[\dXx])$/.test(match.externalId)) isbn = match.externalId; + else isbn = (await loadInventory(cred, http)).find((b) => b.externalId === match.externalId)?.isbn; + } + return publishMicroblogClippings(cred, { ...match, externalEdition: isbn }, ev, http, ctx); + }, }; export const _microblog = { diff --git a/src/connectors/registry.ts b/src/connectors/registry.ts index 3ccf39ac..aee3227e 100644 --- a/src/connectors/registry.ts +++ b/src/connectors/registry.ts @@ -44,6 +44,7 @@ export const fetchTransport: HttpTransport = async (url, init) => { const res = await fetch(url, init); return { status: res.status, + headers: res.headers, body: res.body, text: () => res.text(), json: () => res.json(), diff --git a/src/connectors/runner.ts b/src/connectors/runner.ts index 8a1e77c8..f14ea903 100644 --- a/src/connectors/runner.ts +++ b/src/connectors/runner.ts @@ -126,6 +126,12 @@ export async function processRow(db: DB, row: QueueRow, http: HttpTransport): Pr } const ev = JSON.parse(row.payload) as OutboundEvent; + if (row.connector_id === 'microblog' && ev.kind === 'highlight' + && !getMatch(db, row.user_id, 'microblog', row.document)?.external_id) { + // Publication intent remains durable; a later matched sync reschedules it. + markDone(db, row.id); + return; + } if (connector.shouldPush?.( ev, latestPercentage(db, row.user_id, row.document) diff --git a/test/microblog-clippings-post.test.ts b/test/microblog-clippings-post.test.ts new file mode 100644 index 00000000..0442ccd1 --- /dev/null +++ b/test/microblog-clippings-post.test.ts @@ -0,0 +1,164 @@ +import { afterEach, beforeEach, expect, it } from 'vitest'; +import { makeTestApp, registerUser, DOC } from './helpers.js'; +import { resetEncryptionKeyCache } from '../src/crypto/secrets.js'; +import { upsertAccount, saveMatch } from '../src/connectors/store.js'; +import { selectMicroblogClipping, scheduleMicroblogClippings } from '../src/connectors/microblog-clippings.js'; +import { drainQueue } from '../src/connectors/runner.js'; +import type { HttpTransport } from '../src/connectors/types.js'; + +beforeEach(() => { process.env.TOKEN_ENC_KEY = 'a'.repeat(64); resetEncryptionKeyCache(); }); +afterEach(() => { delete process.env.TOKEN_ENC_KEY; resetEncryptionKeyCache(); }); +const remoteUrl = 'https://example.micro.blog/2026/10/28/highlights.html'; +async function setup() { + const { app, db } = makeTestApp(); + const { username } = await registerUser(app); + const uid = (db.prepare('SELECT id FROM users WHERE username=?').get(username) as { id: number }).id; + upsertAccount(db, uid, 'microblog', { token: 'token' }, null); + db.prepare('INSERT INTO documents(user_id,document,title,author,updated_at) VALUES(?,?,?,?,0)').run(uid, DOC, 'Test book', 'Author'); + db.prepare('INSERT INTO clippings(user_id,document,id,text,spine_index,created_at,updated_at) VALUES(?,?,?,?,0,1,1)').run(uid, DOC, 'one', ''); + selectMicroblogClipping(db, uid, DOC, 'one'); + saveMatch(db, uid, 'microblog', DOC, { externalId: '123', externalEdition: '9781786487849', confidence: 1 }, 'manual'); + scheduleMicroblogClippings(db, uid, DOC); + return { db, uid }; +} +function response(status: number, body: unknown = {}, location?: string) { + return { status, json: async () => body, text: async () => JSON.stringify(body), headers: { get: (key: string) => key.toLowerCase() === 'location' ? location ?? null : null } }; +} + +it('publishes one escaped collected post, skips unchanged content, then updates the saved URL', async () => { + const { db, uid } = await setup(); + const writes: Record[] = []; + const http: HttpTransport = async (url, init) => { + expect(url).toContain('https://micro.blog/micropub'); + if (init.method === 'GET') return response(200, { properties: { url: [remoteUrl] } }); + expect(init.headers?.['content-type']).toBe('application/json'); + const body = JSON.parse(init.body!); writes.push(body); + return response(body.action ? 200 : 201, {}, body.action ? undefined : remoteUrl); + }; + await drainQueue(db, http); + expect(writes).toHaveLength(1); + expect(JSON.stringify(writes[0])).toContain('<First passage>'); + expect(JSON.stringify(writes[0])).toContain('https://micro.blog/books/9781786487849'); + scheduleMicroblogClippings(db, uid, DOC); + await drainQueue(db, http); + expect(writes).toHaveLength(1); + db.prepare('UPDATE clippings SET note=? WHERE user_id=?').run('A note', uid); + scheduleMicroblogClippings(db, uid, DOC); + await drainQueue(db, http); + expect(writes).toHaveLength(2); + expect(writes[1]).toMatchObject({ action: 'update', url: remoteUrl }); + expect(JSON.stringify(writes[1])).toContain('A note'); + db.close(); +}); + +it('recovers an uncertain creation by source identity without creating a second post', async () => { + const { db } = await setup(); + let creates = 0; + let saved: unknown; + const http: HttpTransport = async (url, init) => { + expect(url).toContain('/micropub'); + if (init.method === 'GET') return response(200, { items: [{ properties: { ...(saved as any), url: [remoteUrl] } }] }); + const body = JSON.parse(init.body!); + if (!body.action) { creates++; saved = body.properties; throw new Error('connection lost after acceptance'); } + return response(200); + }; + await drainQueue(db, http); + db.prepare("UPDATE connector_queue SET next_try_at=0 WHERE connector_id='microblog'").run(); + await drainQueue(db, http); + expect(creates).toBe(1); + expect(db.prepare('SELECT url FROM microblog_clipping_posts').get()).toEqual({ url: remoteUrl }); + expect(db.prepare('SELECT status FROM connector_queue').get()).toEqual({ status: 'done' }); + db.close(); +}); + +it.each([401, 403, 429, 500])('handles HTTP %s without marking an upload successful', async (status) => { + const { db } = await setup(); + await drainQueue(db, async () => response(status)); + const row = db.prepare('SELECT status,last_error FROM connector_queue').get() as { status: string; last_error: string }; + expect(row.status).toBe(status < 429 ? 'dead' : 'pending'); + expect(row.last_error).toContain(String(status)); + expect(db.prepare('SELECT fingerprint FROM microblog_clipping_posts').get()).toEqual({ fingerprint: null }); + if (status === 401 || status === 403) expect(db.prepare('SELECT status FROM connector_accounts').get()).toEqual({ status: 'needs_reauth' }); + db.close(); +}); + +it('does not recreate a missing published post', async () => { + const { db } = await setup(); + db.prepare('UPDATE microblog_clipping_posts SET url=?,attempted=1').run(remoteUrl); + let creates = 0; + await drainQueue(db, async (_url, init) => { if (init.method === 'POST' && !JSON.parse(init.body!).action) creates++; return response(404); }); + expect(creates).toBe(0); + expect(db.prepare('SELECT status FROM connector_queue').get()).toEqual({ status: 'dead' }); + db.close(); +}); + +it.each(['missing', 'ambiguous', 'malformed'])('requires attention after %s recovery instead of creating again', async (mode) => { + const { db } = await setup(); + let creates = 0; + let properties: any; + const http: HttpTransport = async (_url, init) => { + if (init.method === 'GET') { + if (mode === 'malformed') return response(200, {}); + const item = { properties: { ...properties, url: [remoteUrl] } }; + return response(200, { items: mode === 'missing' ? [] : [item, { properties: { ...properties, url: ['https://example.micro.blog/duplicate.html'] } }] }); + } + creates++; properties = JSON.parse(init.body!).properties; + return response(201); // Accepted, but Location was lost. + }; + await drainQueue(db, http); + db.prepare('UPDATE connector_queue SET next_try_at=0').run(); + await drainQueue(db, http); + expect(creates).toBe(1); + const state = db.prepare('SELECT url,error FROM microblog_clipping_posts').get() as { url: string | null; error: string | null }; + expect(state.url).toBeNull(); + if (mode !== 'malformed') expect(state.error).toContain('could not be resolved'); + db.close(); +}); + +it('recovers across source pagination', async () => { + const { db } = await setup(); + let properties: any; + let creates = 0; + const pages: string[] = []; + const http: HttpTransport = async (url, init) => { + if (init.method === 'GET') { + if (url.includes('url=')) return response(200, { properties }); + pages.push(url); + return response(200, { items: url.includes('offset=0') ? Array.from({ length: 100 }, () => ({ properties: { content: ['Other post'] } })) + : [{ properties: { ...properties, url: [remoteUrl] } }] }); + } + const body = JSON.parse(init.body!); + if (!body.action) { creates++; properties = body.properties; throw new Error('lost response'); } + return response(200); + }; + await drainQueue(db, http); + db.prepare('UPDATE connector_queue SET next_try_at=0').run(); + await drainQueue(db, http); + expect(creates).toBe(1); + expect(pages.some((p) => p.includes('offset=100'))).toBe(true); + expect(db.prepare('SELECT url FROM microblog_clipping_posts').get()).toEqual({ url: remoteUrl }); + db.close(); +}); + +it('retains published automatic passages when automatic sync is disabled and another is selected', async () => { + const { db, uid } = await setup(); + db.prepare("UPDATE microblog_clipping_selections SET manual=0").run(); + db.prepare("UPDATE connector_accounts SET options='{" + '"sync_highlights":true' + "}'").run(); + const posts: any[] = []; + const http: HttpTransport = async (_url, init) => { + if (init.method === 'GET') return response(200, {}); + const body = JSON.parse(init.body!); posts.push(body); + return response(body.action ? 200 : 201, {}, remoteUrl); + }; + await drainQueue(db, http); + db.prepare("UPDATE connector_accounts SET options='{}'").run(); + db.prepare("UPDATE clippings SET deleted=1").run(); + db.prepare('INSERT INTO clippings(user_id,document,id,text,spine_index,created_at,updated_at) VALUES(?,?,?,?,1,2,2)').run(uid, DOC, 'two', 'Second passage'); + selectMicroblogClipping(db, uid, DOC, 'two'); + scheduleMicroblogClippings(db, uid, DOC); + await drainQueue(db, http); + expect(posts).toHaveLength(2); + expect(JSON.stringify(posts[1])).toContain('<First passage>'); + expect(JSON.stringify(posts[1])).toContain('Second passage'); + db.close(); +}); From bc54d447d9008f52bf51e3221f5dfe2d70f077bb Mon Sep 17 00:00:00 2001 From: John Payne Date: Wed, 28 Oct 2026 10:30:03 -0400 Subject: [PATCH 04/37] feat: wire manual and automatic Micro.blog clipping sync --- src/connectors/fanout.ts | 6 +- src/connectors/microblog-clippings.ts | 2 +- src/connectors/microblog.ts | 3 +- src/connectors/refresh.ts | 2 + src/connectors/store.ts | 15 +++- src/routes/v1/clippings.ts | 3 + src/routes/v1/connectors.ts | 19 +++++ test/microblog-clippings-api.test.ts | 105 ++++++++++++++++++++++++++ 8 files changed, 151 insertions(+), 4 deletions(-) create mode 100644 test/microblog-clippings-api.test.ts diff --git a/src/connectors/fanout.ts b/src/connectors/fanout.ts index b6e5fbe3..5cbbf2a3 100644 --- a/src/connectors/fanout.ts +++ b/src/connectors/fanout.ts @@ -6,6 +6,7 @@ import { activeConnectorIds, carriesFor, documentMeta, getMatch } from './store. import { extractTitleAuthor } from './matching.js'; import { nowSeconds } from '../models/sync.js'; import type { Connector, OutboundEvent } from './types.js'; +import { scheduleMicroblogClippings } from './microblog-clippings.js'; /** * Whether it's worth attempting a document on this connector. Metadata-matched @@ -40,6 +41,7 @@ function fanOut( if (!secretsEnabled()) return; try { for (const connectorId of activeConnectorIds(db, userId)) { + if (connectorId === 'microblog' && ev.kind === 'highlight') continue; if (connectorId === exceptConnectorId) continue; // loop suppression: don't echo to the source const conn = getConnector(connectorId); if (!conn || !conn.capabilities.write || !carriesFor(db, userId, conn, ev.kind)) continue; @@ -63,6 +65,7 @@ export function fanOutProgress( positionJson?: string | null, exceptConnectorId?: string ): void { + scheduleMicroblogClippings(db, userId, document); const finished = percentage >= 0.98; let position: Record | null = null; if (positionJson) { @@ -191,7 +194,8 @@ export function backfillConnector(db: DB, userId: number, connectorId: string, d } } - if (carriesFor(db, userId, conn, 'highlight')) { + if (connectorId === 'microblog') queued += scheduleMicroblogClippings(db, userId); + if (connectorId !== 'microblog' && carriesFor(db, userId, conn, 'highlight')) { const rows = db .prepare( `SELECT c.id, c.document, c.text, c.note, c.created_at, c.spine_index, c.start_offset, c.end_offset, diff --git a/src/connectors/microblog-clippings.ts b/src/connectors/microblog-clippings.ts index 0175c03f..bc594cea 100644 --- a/src/connectors/microblog-clippings.ts +++ b/src/connectors/microblog-clippings.ts @@ -185,7 +185,7 @@ export async function publishMicroblogClippings( if (!token) throw new ConnectorOperationError('Missing Micro.blog token', false, true); // Caller resolves a verified ISBN; internal bookshelf IDs are never used as ISBNs. const isbn = match.externalEdition; - if (!isbn || !/^(?:\d{13}|\d{9}[\dXx])$/.test(isbn)) return permanent('Micro.blog book ISBN is unavailable; choose a book match with an ISBN.'); + if (!isbn || !/^(?:\d{13}|\d{9}[\dXx])$/.test(isbn)) return { ok: false, retryable: false, error: 'Micro.blog book ISBN is unavailable; choose a book match with an ISBN.' }; const selected = passages(db, userId, document); const rendered = microblogClippingsContent(db, userId, document, `https://micro.blog/books/${encodeURIComponent(isbn)}`); if (!rendered) return { ok: true }; diff --git a/src/connectors/microblog.ts b/src/connectors/microblog.ts index 0c91c18c..40fe6a3b 100644 --- a/src/connectors/microblog.ts +++ b/src/connectors/microblog.ts @@ -375,7 +375,8 @@ export const microblogConnector: Connector = { displayName: 'Micro.blog', tier: 1, capabilities: { read: false, write: true }, - carries: ['progress', 'finished'], + carries: ['progress', 'finished', 'highlight'], + options: [{ key: 'sync_highlights', label: 'Automatically sync highlights', default: false }], credentialKind: 'token', beta: false, matchBy: 'metadata', diff --git a/src/connectors/refresh.ts b/src/connectors/refresh.ts index b4ee13ac..228f6c2b 100644 --- a/src/connectors/refresh.ts +++ b/src/connectors/refresh.ts @@ -5,6 +5,7 @@ import { resolveMatch } from './runner.js'; import { getAccount, getMatch } from './store.js'; import type { HttpTransport } from './types.js'; import { importBookorbitStats } from './bookorbit-stats.js'; +import { scheduleMicroblogClippings } from './microblog-clippings.js'; export type ProgressRefresh = (userId: number, document: string) => Promise; @@ -39,6 +40,7 @@ const PER_BOOK_PULLERS: { id: string; strict: boolean; sidecarOnly: boolean; mat export function createProgressRefresh(db: DB, http: HttpTransport = fetchTransport): ProgressRefresh { const pending = new Map>(); return async (userId, document) => { + scheduleMicroblogClippings(db, userId, document); const pullers: typeof PER_BOOK_PULLERS = []; for (const p of PER_BOOK_PULLERS) { const account = getAccount(db, userId, p.id); diff --git a/src/connectors/store.ts b/src/connectors/store.ts index 2596f270..4c9caf96 100644 --- a/src/connectors/store.ts +++ b/src/connectors/store.ts @@ -25,7 +25,12 @@ export function upsertAccount( now = nowSeconds() ): void { const enc = encryptSecret(JSON.stringify(cred)); - const previous = connectorId === 'bookorbit' ? getAccount(db, userId, connectorId) : null; + const previous = connectorId === 'bookorbit' || connectorId === 'microblog' ? getAccount(db, userId, connectorId) : null; + let changedMicroblogAccount = false; + if (connectorId === 'microblog' && previous) { + try { changedMicroblogAccount = decryptCredential(previous).token !== cred.token; } + catch { changedMicroblogAccount = true; } + } let sameOrbitAccount = false; if (previous) { try { @@ -34,6 +39,11 @@ export function upsertAccount( } catch { /* Reconnecting also recovers an unreadable old credential. */ } } withTransaction(db, () => { + if (changedMicroblogAccount) { + for (const table of ['microblog_clipping_selections', 'microblog_clipping_posts']) db.prepare(`DELETE FROM ${table} WHERE user_id=?`).run(userId); + for (const table of ['connector_matches', 'connector_queue']) db.prepare(`DELETE FROM ${table} WHERE user_id=? AND connector_id='microblog'`).run(userId); + db.prepare("UPDATE connector_accounts SET options='{}' WHERE user_id=? AND connector_id='microblog'").run(userId); + } db.prepare( `INSERT INTO connector_accounts (user_id, connector_id, cred_enc, account_label, status, enabled, created_at, updated_at) @@ -90,6 +100,9 @@ export function decryptCredential(row: AccountRow, db?: DB): Credential { } export function deleteAccount(db: DB, userId: number, connectorId: string): void { + if (connectorId === 'microblog') { + for (const table of ['microblog_clipping_selections', 'microblog_clipping_posts']) db.prepare(`DELETE FROM ${table} WHERE user_id=?`).run(userId); + } db.prepare('DELETE FROM connector_accounts WHERE user_id = ? AND connector_id = ?').run( userId, connectorId diff --git a/src/routes/v1/clippings.ts b/src/routes/v1/clippings.ts index 4f1b83db..29eb7d19 100644 --- a/src/routes/v1/clippings.ts +++ b/src/routes/v1/clippings.ts @@ -6,6 +6,7 @@ import { isItemId, nowSeconds, parseListParams } from '../../models/sync.js'; import { fanOutHighlight, highlightFromRow, type ClippingHighlightRow } from '../../connectors/fanout.js'; import { documentMeta } from '../../connectors/store.js'; import { clippingDocuments, mergedClippingsCte, resolveDocument } from '../../models/merge.js'; +import { scheduleMicroblogClippings } from '../../connectors/microblog-clippings.js'; const MAX_BATCH = 50; const MAX_TEXT = 4096; // full firmware clipping, not the text-export cap @@ -81,6 +82,7 @@ export function clippingRoutes(db: DB): Hono { if (!Number.isSafeInteger(cursor) || cursor < 0) return kosyncError(c, 403, 2003, 'Invalid cursor'); const revisionMode = cursorRaw !== undefined; const canonical = resolveDocument(db, user.id, document); + scheduleMicroblogClippings(db, user.id, canonical); const documents = clippingDocuments(db, user.id, canonical); const rows = db .prepare( @@ -278,6 +280,7 @@ export function clippingRoutes(db: DB): Hono { ); } } + scheduleMicroblogClippings(db, user.id, canonical); return c.json({ until: now, accepted: ops.length }); }); diff --git a/src/routes/v1/connectors.ts b/src/routes/v1/connectors.ts index 5674a9c9..5adf2fe9 100644 --- a/src/routes/v1/connectors.ts +++ b/src/routes/v1/connectors.ts @@ -35,6 +35,8 @@ import { ConnectorOperationError, type HttpTransport } from '../../connectors/ty import { importBookorbitStats } from '../../connectors/bookorbit-stats.js'; import { resolveDocument } from '../../models/merge.js'; import { hasDocumentData } from '../../models/document.js'; +import { selectMicroblogClipping, scheduleMicroblogClippings } from '../../connectors/microblog-clippings.js'; +import { isItemId } from '../../models/sync.js'; /** Where a provider sends the browser back: one https URL for web and native (an app link there). */ export const oauthCallbackPath = (id: string) => `/connectors/${id}/callback`; @@ -458,6 +460,22 @@ export function connectorRoutes( } }); + app.post('/connectors/microblog/clippings/:document/:clippingId', (c) => { + const user = c.get('user'); + const account = getAccount(db, user.id, 'microblog'); + if (!account?.enabled) return c.json({ message: 'Micro.blog is not linked or enabled' }, 400); + const document = c.req.param('document'); + const clippingId = c.req.param('clippingId'); + if (!isValidDocument(document) || !isItemId(clippingId)) return c.json({ message: 'Invalid clipping' }, 400); + const canonical = resolveDocument(db, user.id, document); + try { selectMicroblogClipping(db, user.id, canonical, clippingId); } + catch { return c.json({ message: 'Clipping not found' }, 404); } + const state = db.prepare('SELECT error FROM microblog_clipping_posts WHERE user_id=? AND document=?').get(user.id, canonical) as { error: string | null } | undefined; + if (state?.error) return c.json({ message: state.error }, 409); + const queued = scheduleMicroblogClippings(db, user.id, canonical) > 0; + return c.json({ queued, waiting_for_match: !getMatch(db, user.id, 'microblog', canonical)?.external_id }); + }); + // Change a linked connector's options. Unknown keys and bad values are ignored. app.put('/connectors/:id/options', async (c) => { const conn = getConnector(c.req.param('id')); @@ -483,6 +501,7 @@ export function connectorRoutes( } } }); + if (conn.id === 'microblog') scheduleMicroblogClippings(db, user.id); return c.json({ id: conn.id, option_values: next }); }); diff --git a/test/microblog-clippings-api.test.ts b/test/microblog-clippings-api.test.ts new file mode 100644 index 00000000..141e87ed --- /dev/null +++ b/test/microblog-clippings-api.test.ts @@ -0,0 +1,105 @@ +import { afterEach, beforeEach, expect, it } from 'vitest'; +import { makeTestApp, registerUser, DOC } from './helpers.js'; +import { resetEncryptionKeyCache } from '../src/crypto/secrets.js'; +import { upsertAccount } from '../src/connectors/store.js'; +import { drainQueue } from '../src/connectors/runner.js'; +import type { HttpTransport } from '../src/connectors/types.js'; + +beforeEach(() => { process.env.TOKEN_ENC_KEY = 'a'.repeat(64); resetEncryptionKeyCache(); }); +afterEach(() => { delete process.env.TOKEN_ENC_KEY; resetEncryptionKeyCache(); }); +async function setup() { + const posts: unknown[] = []; + const http: HttpTransport = async (url, init) => { + if (url.endsWith('/micropub') && init.method === 'POST') posts.push(JSON.parse(init.body!)); + else if (!url.includes('/micropub?')) throw new Error(`Unexpected network request: ${url}`); + return { status: 201, headers: { get: () => 'https://example.micro.blog/highlights.html' }, json: async () => ({}), text: async () => '' }; + }; + const { app, db } = makeTestApp({}, { connectorTransport: http }); + const { username, headers } = await registerUser(app); + const uid = (db.prepare('SELECT id FROM users WHERE username=?').get(username) as { id: number }).id; + upsertAccount(db, uid, 'microblog', { token: 'token' }, null); + const put = (path: string, body: unknown) => app.request(path, { method: 'PUT', headers, body: JSON.stringify(body) }); + await put('/api/v1/documents', { items: [{ document: DOC, title: 'Test book', author: 'Author' }] }); + const clipping = () => put(`/api/v1/clippings/${DOC}`, { items: [{ id: 'one', spine: 1, text: 'A passage', created_at: 1 }] }); + await clipping(); + const select = (id = 'one', auth = headers) => app.request(`/api/v1/connectors/microblog/clippings/${DOC}/${id}`, { method: 'POST', headers: auth }); + const match = () => put(`/api/v1/connectors/microblog/matches/${DOC}`, { external_id: '123', external_edition: '9781786487849' }); + return { app, db, uid, headers, posts, http, put, clipping, select, match }; +} + +it('defaults automatic publication off and remembers a manually selected unmatched clipping', async () => { + const s = await setup(); + expect(s.db.prepare("SELECT COUNT(*) n FROM connector_queue WHERE kind='microblog:clippings'").get()).toEqual({ n: 0 }); + const selected = await s.select(); + expect(selected.status).toBe(200); + expect(await selected.json()).toEqual({ queued: false, waiting_for_match: true }); + expect(s.db.prepare('SELECT manual FROM microblog_clipping_selections').get()).toEqual({ manual: 1 }); + expect(s.posts).toHaveLength(0); + s.db.close(); +}); + +it.each(['legacy', 'v1', 'clippings', 'sync-now'])('catches up selected clippings after matching on %s sync', async (trigger) => { + const s = await setup(); + await s.select(); + await s.match(); + if (trigger === 'legacy') await s.app.request(`/syncs/progress/${DOC}`, { headers: s.headers }); + if (trigger === 'v1') await s.app.request(`/api/v1/progress/${DOC}`, { headers: s.headers }); + if (trigger === 'clippings') await s.clipping(); + if (trigger === 'sync-now') await s.app.request('/api/v1/connectors/microblog/sync', { method: 'POST', headers: s.headers }); + await drainQueue(s.db, s.http); + expect(s.posts).toHaveLength(1); + expect(JSON.stringify(s.posts[0])).toContain('A passage'); + s.db.close(); +}); + +it('enabling automatic publication backfills existing highlights and waits for matching', async () => { + const s = await setup(); + const changed = await s.put('/api/v1/connectors/microblog/options', { sync_highlights: true }); + expect(changed.status).toBe(200); + expect(s.db.prepare('SELECT manual FROM microblog_clipping_selections').get()).toEqual({ manual: 0 }); + expect(s.db.prepare("SELECT COUNT(*) n FROM connector_queue WHERE kind='microblog:clippings'").get()).toEqual({ n: 0 }); + await s.match(); await s.clipping(); await drainQueue(s.db, s.http); + expect(s.posts).toHaveLength(1); + s.db.close(); +}); + +it('rejects missing, deleted, foreign, unlinked, disabled and unauthenticated selections', async () => { + const s = await setup(); + expect((await s.select('missing')).status).toBe(404); + const other = await registerUser(s.app); + expect((await s.select('one', other.headers)).status).toBe(400); + const otherId = (s.db.prepare('SELECT id FROM users WHERE username=?').get(other.username) as { id: number }).id; + upsertAccount(s.db, otherId, 'microblog', { token: 'other' }, null); + expect((await s.select('one', other.headers)).status).toBe(404); + expect((await s.select('one', {})).status).toBe(401); + s.db.prepare('UPDATE clippings SET deleted=1').run(); + expect((await s.select()).status).toBe(404); + s.db.prepare('UPDATE clippings SET deleted=0').run(); + s.db.prepare('UPDATE connector_accounts SET enabled=0').run(); + expect((await s.select()).status).toBe(400); + s.db.prepare('DELETE FROM connector_accounts').run(); + expect((await s.select()).status).toBe(400); + s.db.close(); +}); + +it('a replacement token cannot inherit another account’s published post or automatic preference', async () => { + const s = await setup(); await s.match(); await s.select(); await drainQueue(s.db, s.http); + s.db.prepare("UPDATE connector_accounts SET options='{" + '"sync_highlights":true' + "}' WHERE user_id=?").run(s.uid); + upsertAccount(s.db, s.uid, 'microblog', { token: 'replacement' }, null); + expect(s.db.prepare('SELECT COUNT(*) n FROM microblog_clipping_posts').get()).toEqual({ n: 0 }); + expect(s.db.prepare("SELECT options FROM connector_accounts WHERE user_id=?").get(s.uid)).toEqual({ options: '{}' }); + expect(s.db.prepare('SELECT COUNT(*) n FROM connector_matches WHERE user_id=?').get(s.uid)).toEqual({ n: 0 }); + s.db.close(); +}); + +it('coalesces repeated selections and clears local publication state on unlink', async () => { + const s = await setup(); await s.match(); + await s.select(); await s.select(); + expect(s.db.prepare("SELECT COUNT(*) n FROM connector_queue WHERE kind='microblog:clippings'").get()).toEqual({ n: 1 }); + await drainQueue(s.db, s.http); + await s.app.request('/api/v1/connectors/microblog', { method: 'DELETE', headers: s.headers }); + expect(s.db.prepare('SELECT COUNT(*) n FROM microblog_clipping_posts').get()).toEqual({ n: 0 }); + expect(s.db.prepare('SELECT COUNT(*) n FROM microblog_clipping_selections').get()).toEqual({ n: 0 }); + expect(s.posts).toHaveLength(1); + s.db.close(); +}); From 4e6e79e74071b75bc71cec6ef8c4dca5a2d98638 Mon Sep 17 00:00:00 2001 From: John Payne Date: Wed, 28 Oct 2026 10:44:46 -0400 Subject: [PATCH 05/37] fix: preserve Micro.blog publication identity across concurrent syncs and reauthentication --- .../0041_microblog_clippings_destination.sql | 2 + src/connectors/microblog-clippings.ts | 38 +++++++-- src/connectors/microblog.ts | 14 ++++ src/connectors/runner.ts | 14 ++++ src/connectors/store.ts | 23 +++++- src/models/document.ts | 2 + src/routes/account.ts | 2 + test/microblog-clippings-api.test.ts | 16 +++- test/microblog-clippings-post.test.ts | 82 ++++++++++++++++++- test/microblog.test.ts | 15 ++++ 10 files changed, 197 insertions(+), 11 deletions(-) create mode 100644 migrations/0041_microblog_clippings_destination.sql diff --git a/migrations/0041_microblog_clippings_destination.sql b/migrations/0041_microblog_clippings_destination.sql new file mode 100644 index 00000000..e6037306 --- /dev/null +++ b/migrations/0041_microblog_clippings_destination.sql @@ -0,0 +1,2 @@ +ALTER TABLE microblog_clipping_posts ADD COLUMN book_url TEXT; +ALTER TABLE microblog_clipping_posts ADD COLUMN matched_book TEXT; diff --git a/src/connectors/microblog-clippings.ts b/src/connectors/microblog-clippings.ts index bc594cea..4281c1a2 100644 --- a/src/connectors/microblog-clippings.ts +++ b/src/connectors/microblog-clippings.ts @@ -1,7 +1,7 @@ import { createHash, randomUUID } from 'node:crypto'; import { withTransaction, type DB } from '../db/db.js'; import { clippingDocuments, mergedClippingsCte, resolveDocument } from '../models/merge.js'; -import { documentMeta, getAccount, getMatch } from './store.js'; +import { decryptCredential, documentMeta, getAccount, getMatch } from './store.js'; import { enqueue } from './queue.js'; import { nowSeconds } from '../models/sync.js'; import { ConnectorOperationError, type Credential, type Match, type OutboundEvent, type HttpTransport, type ConnectorContext, type PushResult } from './types.js'; @@ -10,6 +10,8 @@ export interface ClippingsPost { identity: string; url: string | null; fingerprint: string | null; + book_url: string | null; + matched_book: string | null; attempted: number; error: string | null; } @@ -33,7 +35,7 @@ function liveClippings(db: DB, userId: number, document: string): Passage[] { export function clippingsPost(db: DB, userId: number, document: string): ClippingsPost { db.prepare('INSERT OR IGNORE INTO microblog_clipping_posts(user_id,document,identity) VALUES(?,?,?)') .run(userId, document, randomUUID()); - return db.prepare('SELECT identity,url,fingerprint,attempted,error FROM microblog_clipping_posts WHERE user_id=? AND document=?') + return db.prepare('SELECT identity,url,fingerprint,book_url,matched_book,attempted,error FROM microblog_clipping_posts WHERE user_id=? AND document=?') .get(userId, document) as unknown as ClippingsPost; } @@ -93,9 +95,14 @@ export function scheduleMicroblogClippings(db: DB, userId: number, document?: st const insert = db.prepare('INSERT OR IGNORE INTO microblog_clipping_selections(user_id,document,clipping_id) VALUES(?,?,?)'); for (const c of liveClippings(db, userId, doc)) insert.run(userId, doc, c.id); } - if (!getMatch(db, userId, 'microblog', doc)?.external_id || !passages(db, userId, doc).length) continue; + const match = getMatch(db, userId, 'microblog', doc); + if (!match?.external_id || !passages(db, userId, doc).length) continue; const state = clippingsPost(db, userId, doc); if (state.error) continue; + const isbn = match.external_edition ?? (/^(?:\d{13}|\d{9}[\dXx])$/.test(match.external_id) ? match.external_id : null); + const bookUrl = isbn ? `https://micro.blog/books/${encodeURIComponent(isbn)}` : state.book_url; + if (state.url && bookUrl && state.matched_book === match.external_id + && state.fingerprint === microblogClippingsContent(db, userId, doc, bookUrl)?.fingerprint) continue; const existing = db.prepare("SELECT status FROM connector_queue WHERE user_id=? AND connector_id='microblog' AND document=? AND kind='microblog:clippings'") .get(userId, doc) as { status: string } | undefined; if (existing?.status === 'pending') continue; // Preserve backoff after a transient failure. @@ -176,13 +183,25 @@ export async function publishMicroblogClippings( if (!locks) { locks = new Set(); publishing.set(db, locks); } if (locks.has(key)) return { ok: false, retryable: true, error: 'Micro.blog book publication is already in progress' }; locks.add(key); + const account = getAccount(db, userId, 'microblog'); + let identity: string | undefined; + const assertCurrent = () => { + const current = getAccount(db, userId, 'microblog'); + const saved = db.prepare('SELECT identity FROM microblog_clipping_posts WHERE user_id=? AND document=?').get(userId, document) as { identity: string } | undefined; + if (!account?.enabled || !current?.enabled || current.cred_enc !== account.cred_enc || (identity && saved?.identity !== identity)) { + throw new ConnectorOperationError('Micro.blog account or book changed during publication', false); + } + }; const permanent = (error: string): PushResult => { + assertCurrent(); db.prepare('UPDATE microblog_clipping_posts SET error=? WHERE user_id=? AND document=?').run(error, userId, document); return { ok: false, retryable: false, error }; }; try { const token = typeof cred.token === 'string' ? cred.token.trim() : ''; if (!token) throw new ConnectorOperationError('Missing Micro.blog token', false, true); + assertCurrent(); + if (decryptCredential(account!).token !== cred.token) throw new ConnectorOperationError('Micro.blog credential changed before publication', false); // Caller resolves a verified ISBN; internal bookshelf IDs are never used as ISBNs. const isbn = match.externalEdition; if (!isbn || !/^(?:\d{13}|\d{9}[\dXx])$/.test(isbn)) return { ok: false, retryable: false, error: 'Micro.blog book ISBN is unavailable; choose a book match with an ISBN.' }; @@ -190,21 +209,26 @@ export async function publishMicroblogClippings( const rendered = microblogClippingsContent(db, userId, document, `https://micro.blog/books/${encodeURIComponent(isbn)}`); if (!rendered) return { ok: true }; const state = clippingsPost(db, userId, document); + identity = state.identity; if (state.error) return { ok: false, retryable: false, error: state.error }; if (rendered.fingerprint === state.fingerprint && state.url) return { ok: true }; let url = state.url; if (!url && state.attempted) { url = await recoverPost(http, token, state.identity); - if (!url) return permanent('Micro.blog post creation could not be resolved uniquely. Review your blog posts before retrying; no duplicate was created.'); + assertCurrent(); + if (!url) return permanent('Micro.blog post creation could not be resolved uniquely. Review your blog posts before retrying; no duplicate was created. If switching blogs, unlink Micro.blog and connect the new blog.'); db.prepare('UPDATE microblog_clipping_posts SET url=? WHERE user_id=? AND document=?').run(url, userId, document); } if (url) { try { // Detect deleted posts before replacing generated content. await micropub(http, token, `/micropub?q=source&url=${encodeURIComponent(url)}`); + assertCurrent(); await micropub(http, token, '/micropub', { action: 'update', url, replace: { name: [rendered.title], content: [{ html: rendered.content }] } }); + assertCurrent(); } catch (err) { + assertCurrent(); if (err instanceof ConnectorOperationError && !err.retryable && !err.needsReauth) return permanent(`${err.message}. Review the saved Micro.blog post before syncing again.`); throw err; } @@ -217,11 +241,13 @@ export async function publishMicroblogClippings( 'mp-slug': [`crosspoint-highlights-${state.identity}`], } }); } catch (err) { + assertCurrent(); if (err instanceof ConnectorOperationError && (err.needsReauth || err.message.includes('(429)'))) { db.prepare('UPDATE microblog_clipping_posts SET attempted=0 WHERE user_id=? AND document=?').run(userId, document); } else if (err instanceof ConnectorOperationError && !err.retryable) return permanent(err.message); throw err; } + assertCurrent(); url = postUrl(res.headers?.get('location')); if (!url) { throw new ConnectorOperationError('Micro.blog accepted the post without its URL; the next attempt will recover it.', true); @@ -231,8 +257,8 @@ export async function publishMicroblogClippings( } withTransaction(db, () => { savePublishedClippings(db, userId, document, selected); - db.prepare('UPDATE microblog_clipping_posts SET fingerprint=?,error=NULL WHERE user_id=? AND document=?') - .run(rendered.fingerprint, userId, document); + db.prepare('UPDATE microblog_clipping_posts SET fingerprint=?,book_url=?,matched_book=?,error=NULL WHERE user_id=? AND document=?') + .run(rendered.fingerprint, `https://micro.blog/books/${encodeURIComponent(isbn)}`, match.externalId, userId, document); }); return { ok: true }; } finally { locks.delete(key); } diff --git a/src/connectors/microblog.ts b/src/connectors/microblog.ts index 40fe6a3b..d6fbe6b4 100644 --- a/src/connectors/microblog.ts +++ b/src/connectors/microblog.ts @@ -199,6 +199,20 @@ async function loadInventory(cred: Credential, http: HttpTransport): Promise { try { await loadShelves(cred, http); + // Never trust a destination supplied alongside a pasted token. Shelf-only + // accounts still link when Micropub configuration is unavailable. + delete cred.microblogDestination; + try { + const config = await request(http, tokenOf(cred), '/micropub?q=config', { method: 'GET' }) as { + destination?: { uid?: unknown; 'microblog-default'?: boolean }[]; + }; + const destinations = Array.isArray(config.destination) ? config.destination : []; + const chosen = destinations.find((d) => d['microblog-default'] === true) ?? (destinations.length === 1 ? destinations[0] : null); + if (typeof chosen?.uid === 'string') { + const url = new URL(chosen.uid); + if (url.protocol === 'https:' && !url.username && !url.password) cred.microblogDestination = url.href; + } + } catch { /* Older servers/tokens can continue syncing bookshelves. */ } return { ok: true }; } catch (error) { if (error instanceof ConnectorOperationError && error.needsReauth) { diff --git a/src/connectors/runner.ts b/src/connectors/runner.ts index f14ea903..5d4e47eb 100644 --- a/src/connectors/runner.ts +++ b/src/connectors/runner.ts @@ -20,6 +20,7 @@ import { setMatchNote, } from './store.js'; import { decideMatch, extractTitleAuthor } from './matching.js'; +import { scheduleMicroblogClippings } from './microblog-clippings.js'; import { ConnectorOperationError, type HttpTransport, @@ -126,6 +127,13 @@ export async function processRow(db: DB, row: QueueRow, http: HttpTransport): Pr } const ev = JSON.parse(row.payload) as OutboundEvent; + const publication = row.connector_id === 'microblog' && ev.kind === 'highlight'; + const originalPost = publication ? db.prepare('SELECT identity FROM microblog_clipping_posts WHERE user_id=? AND document=?') + .get(row.user_id, row.document) as { identity: string } | undefined : undefined; + const stillCurrent = () => row.connector_id !== 'microblog' || ( + getAccount(db, row.user_id, row.connector_id)?.cred_enc === account.cred_enc && (!originalPost || + (db.prepare('SELECT identity FROM microblog_clipping_posts WHERE user_id=? AND document=?').get(row.user_id, row.document) as { identity: string } | undefined)?.identity === originalPost.identity) + ); if (row.connector_id === 'microblog' && ev.kind === 'highlight' && !getMatch(db, row.user_id, 'microblog', row.document)?.external_id) { // Publication intent remains durable; a later matched sync reschedules it. @@ -143,7 +151,9 @@ export async function processRow(db: DB, row: QueueRow, http: HttpTransport): Pr let match: Match | null; try { match = await resolveMatch(db, row.connector_id, row.user_id, row.document, http, ev); + if (!stillCurrent()) return; } catch (err) { + if (!stillCurrent()) return; failOperation(db, row, err, 'match failed'); return; } @@ -157,10 +167,13 @@ export async function processRow(db: DB, row: QueueRow, http: HttpTransport): Pr const cred = decryptCredential(account, db); try { const result = await connector.push(cred, match, ev, http, { db, userId: row.user_id }); + if (!stillCurrent()) return; if (result.ok) { // Persist (or clear) the per-book condition so the review UI can show it. setMatchNote(db, row.user_id, row.connector_id, row.document, result.note ?? null); markDone(db, row.id); + // A clipping may have arrived while the network request was in flight. + if (publication) scheduleMicroblogClippings(db, row.user_id, row.document); return; } if (result.needsReauth) { @@ -169,6 +182,7 @@ export async function processRow(db: DB, row: QueueRow, http: HttpTransport): Pr logPushFailure(row, result.error, result.retryable); markFailed(db, row, result.error, result.retryable); } catch (err) { + if (!stillCurrent()) return; failOperation(db, row, err, 'push failed'); } } diff --git a/src/connectors/store.ts b/src/connectors/store.ts index 4c9caf96..91a5dd37 100644 --- a/src/connectors/store.ts +++ b/src/connectors/store.ts @@ -27,9 +27,21 @@ export function upsertAccount( const enc = encryptSecret(JSON.stringify(cred)); const previous = connectorId === 'bookorbit' || connectorId === 'microblog' ? getAccount(db, userId, connectorId) : null; let changedMicroblogAccount = false; + let verifyMicroblogPosts = false; if (connectorId === 'microblog' && previous) { - try { changedMicroblogAccount = decryptCredential(previous).token !== cred.token; } - catch { changedMicroblogAccount = true; } + try { + const old = decryptCredential(previous); + if (old.token !== cred.token) { + // Existing installs may have published before destination discovery was + // added. Location URLs provide a conservative same-blog identity hint. + const urls = db.prepare('SELECT url FROM microblog_clipping_posts WHERE user_id=? AND url IS NOT NULL') + .all(userId) as { url: string }[]; + const origins = new Set(urls.flatMap((r) => { try { return [new URL(r.url).origin + '/']; } catch { return []; } })); + const oldDestination = old.microblogDestination ?? (origins.size === 1 ? [...origins][0] : null); + if (oldDestination && cred.microblogDestination) changedMicroblogAccount = oldDestination !== cred.microblogDestination; + else verifyMicroblogPosts = true; + } + } catch { verifyMicroblogPosts = true; } } let sameOrbitAccount = false; if (previous) { @@ -39,6 +51,13 @@ export function upsertAccount( } catch { /* Reconnecting also recovers an unreadable old credential. */ } } withTransaction(db, () => { + if (verifyMicroblogPosts) { + // Unknown ownership: preserve intent/identity, but recover each existing + // post under the new token before reusing a URL or creating anything. + db.prepare(`UPDATE microblog_clipping_posts SET attempted=CASE WHEN url IS NOT NULL THEN 1 ELSE attempted END, + url=NULL,fingerprint=NULL,error=NULL WHERE user_id=?`).run(userId); + db.prepare("DELETE FROM connector_queue WHERE user_id=? AND connector_id='microblog'").run(userId); + } if (changedMicroblogAccount) { for (const table of ['microblog_clipping_selections', 'microblog_clipping_posts']) db.prepare(`DELETE FROM ${table} WHERE user_id=?`).run(userId); for (const table of ['connector_matches', 'connector_queue']) db.prepare(`DELETE FROM ${table} WHERE user_id=? AND connector_id='microblog'`).run(userId); diff --git a/src/models/document.ts b/src/models/document.ts index 9e37a458..39c295c4 100644 --- a/src/models/document.ts +++ b/src/models/document.ts @@ -8,6 +8,8 @@ import { withTransaction, type DB } from '../db/db.js'; * intent obvious next to the account-level delete in routes/account.ts. */ const DOCUMENT_TABLES = [ + 'microblog_clipping_selections', + 'microblog_clipping_posts', 'connector_queue', 'connector_matches', 'stats_device_book', diff --git a/src/routes/account.ts b/src/routes/account.ts index ab378820..bc1eca3b 100644 --- a/src/routes/account.ts +++ b/src/routes/account.ts @@ -14,6 +14,8 @@ import { USERNAME_RE } from './kosync.js'; export function deleteKosyncUserData(db: DB, userId: number, username: string, { keepUser = false } = {}): void { withTransaction(db, () => { for (const table of [ + 'microblog_clipping_selections', + 'microblog_clipping_posts', 'bookorbit_session_exports', 'bookorbit_session_destination', 'reading_sessions', diff --git a/test/microblog-clippings-api.test.ts b/test/microblog-clippings-api.test.ts index 141e87ed..eb0332a4 100644 --- a/test/microblog-clippings-api.test.ts +++ b/test/microblog-clippings-api.test.ts @@ -4,6 +4,8 @@ import { resetEncryptionKeyCache } from '../src/crypto/secrets.js'; import { upsertAccount } from '../src/connectors/store.js'; import { drainQueue } from '../src/connectors/runner.js'; import type { HttpTransport } from '../src/connectors/types.js'; +import { deleteDocumentData } from '../src/models/document.js'; +import { deleteKosyncUserData } from '../src/routes/account.js'; beforeEach(() => { process.env.TOKEN_ENC_KEY = 'a'.repeat(64); resetEncryptionKeyCache(); }); afterEach(() => { delete process.env.TOKEN_ENC_KEY; resetEncryptionKeyCache(); }); @@ -17,7 +19,7 @@ async function setup() { const { app, db } = makeTestApp({}, { connectorTransport: http }); const { username, headers } = await registerUser(app); const uid = (db.prepare('SELECT id FROM users WHERE username=?').get(username) as { id: number }).id; - upsertAccount(db, uid, 'microblog', { token: 'token' }, null); + upsertAccount(db, uid, 'microblog', { token: 'token', microblogDestination: 'https://example.micro.blog/' }, null); const put = (path: string, body: unknown) => app.request(path, { method: 'PUT', headers, body: JSON.stringify(body) }); await put('/api/v1/documents', { items: [{ document: DOC, title: 'Test book', author: 'Author' }] }); const clipping = () => put(`/api/v1/clippings/${DOC}`, { items: [{ id: 'one', spine: 1, text: 'A passage', created_at: 1 }] }); @@ -85,13 +87,23 @@ it('rejects missing, deleted, foreign, unlinked, disabled and unauthenticated se it('a replacement token cannot inherit another account’s published post or automatic preference', async () => { const s = await setup(); await s.match(); await s.select(); await drainQueue(s.db, s.http); s.db.prepare("UPDATE connector_accounts SET options='{" + '"sync_highlights":true' + "}' WHERE user_id=?").run(s.uid); - upsertAccount(s.db, s.uid, 'microblog', { token: 'replacement' }, null); + upsertAccount(s.db, s.uid, 'microblog', { token: 'replacement', microblogDestination: 'https://replacement.micro.blog/' }, null); expect(s.db.prepare('SELECT COUNT(*) n FROM microblog_clipping_posts').get()).toEqual({ n: 0 }); expect(s.db.prepare("SELECT options FROM connector_accounts WHERE user_id=?").get(s.uid)).toEqual({ options: '{}' }); expect(s.db.prepare('SELECT COUNT(*) n FROM connector_matches WHERE user_id=?').get(s.uid)).toEqual({ n: 0 }); s.db.close(); }); +it.each(['book', 'account'])('clears local publication data on %s removal without deleting the remote post', async (scope) => { + const s = await setup(); await s.match(); await s.select(); await drainQueue(s.db, s.http); + if (scope === 'book') deleteDocumentData(s.db, s.uid, DOC); + else deleteKosyncUserData(s.db, s.uid, s.headers['x-auth-user'], { keepUser: true }); + expect(s.db.prepare('SELECT COUNT(*) n FROM microblog_clipping_posts').get()).toEqual({ n: 0 }); + expect(s.db.prepare('SELECT COUNT(*) n FROM microblog_clipping_selections').get()).toEqual({ n: 0 }); + expect(s.posts).toHaveLength(1); + s.db.close(); +}); + it('coalesces repeated selections and clears local publication state on unlink', async () => { const s = await setup(); await s.match(); await s.select(); await s.select(); diff --git a/test/microblog-clippings-post.test.ts b/test/microblog-clippings-post.test.ts index 0442ccd1..2d5a78a1 100644 --- a/test/microblog-clippings-post.test.ts +++ b/test/microblog-clippings-post.test.ts @@ -13,7 +13,7 @@ async function setup() { const { app, db } = makeTestApp(); const { username } = await registerUser(app); const uid = (db.prepare('SELECT id FROM users WHERE username=?').get(username) as { id: number }).id; - upsertAccount(db, uid, 'microblog', { token: 'token' }, null); + upsertAccount(db, uid, 'microblog', { token: 'token', microblogDestination: 'https://example.micro.blog/' }, null); db.prepare('INSERT INTO documents(user_id,document,title,author,updated_at) VALUES(?,?,?,?,0)').run(uid, DOC, 'Test book', 'Author'); db.prepare('INSERT INTO clippings(user_id,document,id,text,spine_index,created_at,updated_at) VALUES(?,?,?,?,0,1,1)').run(uid, DOC, 'one', ''); selectMicroblogClipping(db, uid, DOC, 'one'); @@ -162,3 +162,83 @@ it('retains published automatic passages when automatic sync is disabled and ano expect(JSON.stringify(posts[1])).toContain('Second passage'); db.close(); }); + +it('queues a follow-up when a clipping is selected while the first post is being published', async () => { + const { db, uid } = await setup(); + let release!: () => void; + let started!: () => void; + const accepted = new Promise((r) => { started = r; }); + const held = new Promise((r) => { release = r; }); + const writes: any[] = []; + const http: HttpTransport = async (_url, init) => { + if (init.method === 'GET') return response(200, {}); + const body = JSON.parse(init.body!); writes.push(body); + if (!body.action) { started(); await held; } + return response(body.action ? 200 : 201, {}, remoteUrl); + }; + const running = drainQueue(db, http); + await accepted; + db.prepare('INSERT INTO clippings(user_id,document,id,text,spine_index,created_at,updated_at) VALUES(?,?,?,?,1,2,2)').run(uid, DOC, 'two', 'Selected during upload'); + selectMicroblogClipping(db, uid, DOC, 'two'); + scheduleMicroblogClippings(db, uid, DOC); + release(); await running; + await drainQueue(db, http); + expect(writes).toHaveLength(2); + expect(JSON.stringify(writes[1])).toContain('Selected during upload'); + db.close(); +}); + +it('an in-flight old-account upload cannot overwrite replacement-account state or acknowledge its queue row', async () => { + const { db, uid } = await setup(); + let release!: () => void; + let started!: () => void; + const accepted = new Promise((r) => { started = r; }); + const held = new Promise((r) => { release = r; }); + const http: HttpTransport = async (_url, init) => { + if (init.headers?.authorization === 'Bearer token') { started(); await held; return response(201, {}, remoteUrl); } + return response(201, {}, 'https://replacement.micro.blog/highlights.html'); + }; + const running = drainQueue(db, http); await accepted; + upsertAccount(db, uid, 'microblog', { token: 'replacement', microblogDestination: 'https://replacement.micro.blog/' }, null); + saveMatch(db, uid, 'microblog', DOC, { externalId: '456', externalEdition: '9781786487849', confidence: 1 }, 'manual'); + selectMicroblogClipping(db, uid, DOC, 'one'); scheduleMicroblogClippings(db, uid, DOC); + release(); await running; + expect(db.prepare('SELECT url,fingerprint FROM microblog_clipping_posts').get()).toEqual({ url: null, fingerprint: null }); + expect(db.prepare('SELECT status FROM connector_queue').get()).toEqual({ status: 'pending' }); + await drainQueue(db, http); + expect(db.prepare('SELECT url FROM microblog_clipping_posts').get()).toEqual({ url: 'https://replacement.micro.blog/highlights.html' }); + db.close(); +}); + +it('rotating a token for the same verified blog preserves the collected post and manual selections', async () => { + const { db, uid } = await setup(); + let creates = 0; + const http: HttpTransport = async (_url, init) => { + if (init.method === 'GET') return response(200, {}); + if (!JSON.parse(init.body!).action) creates++; + return response(201, {}, remoteUrl); + }; + await drainQueue(db, http); + upsertAccount(db, uid, 'microblog', { token: 'rotated', microblogDestination: 'https://example.micro.blog/' }, null); + expect(db.prepare('SELECT url FROM microblog_clipping_posts').get()).toEqual({ url: remoteUrl }); + expect(db.prepare('SELECT manual FROM microblog_clipping_selections').get()).toEqual({ manual: 1 }); + scheduleMicroblogClippings(db, uid, DOC); await drainQueue(db, http); + expect(creates).toBe(1); + db.close(); +}); + +it('updates the book link on the existing post after correcting a manual match', async () => { + const { db, uid } = await setup(); + const writes: any[] = []; + const http: HttpTransport = async (_url, init) => { + if (init.method === 'GET') return response(200, {}); + writes.push(JSON.parse(init.body!)); return response(201, {}, remoteUrl); + }; + await drainQueue(db, http); + saveMatch(db, uid, 'microblog', DOC, { externalId: '456', externalEdition: '9780143127796', confidence: 1 }, 'manual'); + scheduleMicroblogClippings(db, uid, DOC); await drainQueue(db, http); + expect(writes).toHaveLength(2); + expect(writes[1]).toMatchObject({ action: 'update', url: remoteUrl }); + expect(JSON.stringify(writes[1])).toContain('https://micro.blog/books/9780143127796'); + db.close(); +}); diff --git a/test/microblog.test.ts b/test/microblog.test.ts index aeeec0df..9715a324 100644 --- a/test/microblog.test.ts +++ b/test/microblog.test.ts @@ -11,6 +11,21 @@ const DOC = { const EV = { kind: 'progress' as const, document: 'd', percentage: 0.4, timestamp: 1 }; describe('Micro.blog shelf lookup', () => { + it('records a server-verified default blog so rotating a token preserves publication identity', async () => { + const cred: Record = { token: 'mb-token', microblogDestination: 'https://untrusted.example/' }; + const fake = makeMicroblogTransport(); + const http: HttpTransport = async (url, init) => { + if (url === 'https://micro.blog/micropub?q=config') return { + status: 200, text: async () => '', json: async () => ({ destination: [ + { uid: 'https://other.micro.blog/', 'microblog-default': false }, + { uid: 'https://example.micro.blog/', 'microblog-default': true }, + ] }), + }; + return fake.transport(url, init); + }; + expect((await _microblog.validateCredential(cred, http)).ok).toBe(true); + expect(cred.microblogDestination).toBe('https://example.micro.blog/'); + }); it('validates an app token against the bookshelves endpoint', async () => { const fake = makeMicroblogTransport(); expect(await _microblog.validateCredential(CRED, fake.transport)).toEqual({ ok: true }); From 2f8bc2b0acd4599413dffde88a889156cf102052 Mon Sep 17 00:00:00 2001 From: John Payne Date: Wed, 28 Oct 2026 10:44:46 -0400 Subject: [PATCH 06/37] feat: add Micro.blog clipping publishing controls and local testing guide --- README.md | 20 ++++++++ app/src/Clippings.jsx | 43 +++++++++++++--- app/src/Clippings.test.jsx | 49 +++++++++++++++++++ app/src/Settings.jsx | 2 +- app/src/api.js | 1 + ...6-10-28-microblog-clippings-post-design.md | 19 +++++++ src/routes/web.ts | 2 +- test/web-microblog.test.ts | 2 +- 8 files changed, 129 insertions(+), 9 deletions(-) create mode 100644 app/src/Clippings.test.jsx diff --git a/README.md b/README.md index 4353a50e..4f043869 100644 --- a/README.md +++ b/README.md @@ -161,6 +161,26 @@ Native app association files, when enabled, remain at domain-root `/.well-known/ Treat the token like a password: Micro.blog app tokens have full account access. CrossPoint Sync encrypts the token at rest using `TOKEN_ENC_KEY`. +With Micro.blog linked, each clipping on the **Clippings** page has an +**Add highlight to Micro.blog** action. Selected highlights are collected into +one published post per book, linking to its matched Micro.blog book page. +Further selections update that post. CrossPoint Sync manages its generated +content, so later updates replace edits made directly to that content on Micro.blog. + +**Automatically sync highlights** in the Micro.blog settings is off by default. +Enabling it includes existing and future clippings. Unmatched books wait without +interrupting reader sync; after matching a book, its eligible highlights are sent +on the next reader sync or **Sync now**. Turning automatic sync off stops automatic +additions while retaining published passages and manually selected highlights. + +For local testing, use Node 22.13 or newer, build the frontend with +`npm run build:frontend`, and copy `.env.example` to `.env`. Set a separate +`DATABASE_PATH` (such as `./data/microblog-test.db`), `PORT=8081`, and a persistent +`TOKEN_ENC_KEY` generated with `openssl rand -hex 32`. Run `npm run dev` and open +[the local app](http://localhost:8081/app/). Create a local account, sync a book +and its clippings from a reader, link Micro.blog, and match the book before trying +the clipping action or automatic sync. These actions publish to the linked blog. + ## Point your reader at it - **KOReader:** Tools → Progress sync → Custom sync server → `http://your-host:8080` diff --git a/app/src/Clippings.jsx b/app/src/Clippings.jsx index 6a7af78c..c3fbfd50 100644 --- a/app/src/Clippings.jsx +++ b/app/src/Clippings.jsx @@ -1,4 +1,4 @@ -import { useMemo, useState } from 'react' +import { useMemo, useRef, useState } from 'react' import { Copy, FileDown, Quote, Search } from 'lucide-react' import { api, cached, isApp } from './api.js' import { saveFile } from './shareCard.js' @@ -25,7 +25,7 @@ function markdown(groups) { .join('\n\n---\n\n') } -function Clip({ session, clip, book, onShare, onDelete, showBook }) { +function Clip({ session, clip, book, onShare, onDelete, showBook, onMicroblog, sending }) { return (
)} -
+
+ {onMicroblog && ( + + )}

{[clip.chapter, date(clip.created_at)].filter(Boolean).join(' · ')}

@@ -54,11 +59,16 @@ function Clip({ session, clip, book, onShare, onDelete, showBook }) { export default function Clippings({ session, books }) { const [items, error, reload] = useLoad(() => api.allClippings(session, books), [session], `clippings:${session.username}`, () => cached.clippings(session)) + const [connectors] = useLoad(() => api.connectors(session).then((r) => r.connectors), [session], `clipping-connectors:${session.username}`) + const microblog = connectors?.some((c) => c.id === 'microblog' && c.linked) const [query, setQuery] = useState('') const [only, setOnly] = useState(null) // document const [sharing, setSharing] = useState(null) const [deleting, setDeleting] = useState(null) const [note, setNote] = useState(null) + const [sending, setSending] = useState(false) + const pending = useRef(false) + const [sendError, setSendError] = useState(null) const byDoc = useMemo(() => new Map(books.map((b) => [b.document, b])), [books]) // Only clippings of books the library shows (metadata-less ones are hidden everywhere). @@ -74,6 +84,26 @@ export default function Clippings({ session, books }) { const counts = all.reduce((m, c) => m.set(c.document, (m.get(c.document) ?? 0) + 1), new Map()) const daily = all.length ? ofTheDay(all) : null + async function addToMicroblog(clip) { + if (pending.current) return + pending.current = true + setSending(true) + setNote(null) + setSendError(null) + try { + const result = await api.addMicroblogHighlight(session, clip.document, clip.id) + setNote(result.waiting_for_match + ? 'Highlight saved for Micro.blog. Match this book in Settings; it will be included on the next sync.' + : result.queued ? 'Highlight selected for the book’s Micro.blog post. The update is queued.' + : 'Highlight is selected for the book’s Micro.blog post.') + } catch (error) { + setSendError(error.message || 'Could not add this highlight to Micro.blog.') + } finally { + pending.current = false + setSending(false) + } + } + async function exportMd(save, e) { // A
menu stays open until toggled; close it so the note shows. e.currentTarget.closest('details').open = false @@ -90,6 +120,7 @@ export default function Clippings({ session, books }) { return (

Clippings

+ {sendError &&

{sendError}

} {error && !items ? ( @@ -107,7 +138,7 @@ export default function Clippings({ session, books }) {

Highlight of the day

- +
)} @@ -140,7 +171,7 @@ export default function Clippings({ session, books }) {
- {note &&

{note}

} + {note &&

{note}

}
{onMicroblog && ( )}

{[clip.chapter, date(clip.created_at)].filter(Boolean).join(' · ')}

@@ -61,6 +61,9 @@ export default function Clippings({ session, books }) { const [items, error, reload] = useLoad(() => api.allClippings(session, books), [session], `clippings:${session.username}`, () => cached.clippings(session)) const [connectors] = useLoad(() => api.connectors(session).then((r) => r.connectors), [session], `clipping-connectors:${session.username}`) const microblog = connectors?.some((c) => c.id === 'microblog' && c.linked) + const [postData, postError, reloadPosts] = useLoad(() => microblog ? api.microblogClippingPosts(session) : Promise.resolve({ items: [] }), [session, microblog]) + const postByDocument = new Map((postData?.items || []).map((p) => [p.document, p])) + const [recreating, setRecreating] = useState(null) const [query, setQuery] = useState('') const [only, setOnly] = useState(null) // document const [sharing, setSharing] = useState(null) @@ -91,13 +94,13 @@ export default function Clippings({ session, books }) { setNote(null) setSendError(null) try { - const result = await api.addMicroblogHighlight(session, clip.document, clip.id) + const result = await api.addMicroblogClipping(session, clip.document, clip.id) setNote(result.waiting_for_match - ? 'Highlight saved for Micro.blog. Match this book in Settings; it will be included on the next sync.' - : result.queued ? 'Highlight selected for the book’s Micro.blog post. The update is queued.' - : 'Highlight is selected for the book’s Micro.blog post.') + ? 'Clipping saved for Micro.blog. Match this book in Settings; it will be included on the next sync.' + : result.queued ? 'Clipping selected for the book’s Micro.blog post. The update is queued.' + : 'Clipping is selected for the book’s Micro.blog post.') } catch (error) { - setSendError(error.message || 'Could not add this highlight to Micro.blog.') + setSendError(error.message || 'Could not add this clipping to Micro.blog.') } finally { pending.current = false setSending(false) @@ -121,6 +124,7 @@ export default function Clippings({ session, books }) {

Clippings

{sendError &&

{sendError}

} + {postError && microblog &&

Could not load Micro.blog post status: {postError.message}

} {error && !items ? ( @@ -129,14 +133,14 @@ export default function Clippings({ session, books }) { ) : all.length === 0 ? (
- Highlight a passage on your reader with clipping sync on, and it lands here, ready to search and share. + Clippings saved on your reader land here, ready to search and share.
) : ( <> {daily && !q && !only && (
-

Highlight of the day

+

Clipping of the day

@@ -207,6 +211,10 @@ export default function Clippings({ session, books }) {

{book.author}

+ {postByDocument.get(book.document)?.missing &&
+

The Micro.blog clippings post was deleted. Clippings are saved here while publishing is paused.

+ +
}
{clips.map((c) => ( @@ -217,6 +225,13 @@ export default function Clippings({ session, books }) { )} + {recreating && setRecreating(null)} onConfirm={async () => { + await api.recreateMicroblogPost(session, recreating.document) + reloadPosts() + setNote('Replacement clippings post queued for Micro.blog.') + }}> + This will republish this book’s collected clippings and resume updates on its original blog: {recreating.destination || recreating.url}. + } {sharing && setSharing(null)} />} {deleting && ( { window.matchMedia = vi.fn(() => ({ matches: false, addEventListener() {}, removeEventListener() {} })) }) -vi.mock('./api.js', () => ({ isApp: false, cached: { clippings: () => null }, api: { allClippings: vi.fn(), connectors: vi.fn(), addMicroblogHighlight: vi.fn() } })) +vi.hoisted(() => { window.scrollTo = vi.fn(); window.matchMedia = vi.fn(() => ({ matches: false, addEventListener() {}, removeEventListener() {} })) }) +vi.mock('./api.js', () => ({ isApp: false, cached: { clippings: () => null }, api: { allClippings: vi.fn(), connectors: vi.fn(), addMicroblogClipping: vi.fn(), microblogClippingPosts: vi.fn(), recreateMicroblogPost: vi.fn() } })) vi.mock('./tabbar.js', () => ({ useSheet() {}, useNativeBack() {}, useNativeButton() {} })) import Clippings from './Clippings.jsx' import { api } from './api.js' @@ -12,38 +12,68 @@ const books = [{ document: 'book', title: 'Test book', author: 'An author', cove beforeEach(() => { api.allClippings.mockResolvedValue([{ id: 'one', document: 'book', text: 'A passage', created_at: 1 }]) api.connectors.mockResolvedValue({ connectors: [{ id: 'microblog', linked: true }] }) - api.addMicroblogHighlight.mockResolvedValue({ queued: true, waiting_for_match: false }) + api.addMicroblogClipping.mockResolvedValue({ queued: true, waiting_for_match: false }) + api.microblogClippingPosts.mockResolvedValue({ items: [] }) + api.recreateMicroblogPost.mockResolvedValue({ queued: true }) }) afterEach(() => { cleanup(); vi.clearAllMocks() }) it('shows the action on both daily and book clippings with configured Micro.blog', async () => { render() - expect(await screen.findAllByRole('button', { name: 'Add highlight to Micro.blog' })).toHaveLength(2) + expect(await screen.findAllByRole('button', { name: 'Add clipping to Micro.blog' })).toHaveLength(2) }) it('hides the action when Micro.blog is unlinked', async () => { api.connectors.mockResolvedValue({ connectors: [{ id: 'microblog', linked: false }] }) render() await screen.findAllByText('A passage') - expect(screen.queryByRole('button', { name: 'Add highlight to Micro.blog' })).toBeNull() + expect(screen.queryByRole('button', { name: 'Add clipping to Micro.blog' })).toBeNull() }) it('selects a clipping, prevents repeat requests, and explains waiting for a match', async () => { let finish - api.addMicroblogHighlight.mockImplementation(() => new Promise((resolve) => { finish = resolve })) + api.addMicroblogClipping.mockImplementation(() => new Promise((resolve) => { finish = resolve })) render() - const buttons = await screen.findAllByRole('button', { name: 'Add highlight to Micro.blog' }) + const buttons = await screen.findAllByRole('button', { name: 'Add clipping to Micro.blog' }) fireEvent.click(buttons[0]) expect(buttons[1].disabled).toBe(true) fireEvent.click(buttons[1]) - expect(api.addMicroblogHighlight).toHaveBeenCalledTimes(1) - expect(api.addMicroblogHighlight).toHaveBeenCalledWith(session, 'book', 'one') + expect(api.addMicroblogClipping).toHaveBeenCalledTimes(1) + expect(api.addMicroblogClipping).toHaveBeenCalledWith(session, 'book', 'one') finish({ queued: false, waiting_for_match: true }) await waitFor(() => expect(screen.getByRole('status').textContent).toMatch(/match.*next sync/i)) }) it('shows failures and allows another attempt without claiming publication succeeded', async () => { - api.addMicroblogHighlight.mockRejectedValue(new Error('Micro.blog is unavailable')) + api.addMicroblogClipping.mockRejectedValue(new Error('Micro.blog is unavailable')) render() - fireEvent.click((await screen.findAllByRole('button', { name: 'Add highlight to Micro.blog' }))[0]) + fireEvent.click((await screen.findAllByRole('button', { name: 'Add clipping to Micro.blog' }))[0]) expect((await screen.findByRole('alert')).textContent).toContain('Micro.blog is unavailable') expect(screen.queryByRole('status')).toBeNull() - expect(screen.getAllByRole('button', { name: 'Add highlight to Micro.blog' })[0].disabled).toBe(false) + expect(screen.getAllByRole('button', { name: 'Add clipping to Micro.blog' })[0].disabled).toBe(false) +}) + +const deletedPost = { document: 'book', missing: true, destination: 'https://first.micro.blog/', url: 'https://first.micro.blog/deleted.html' } +it('offers recreation once per missing book and cancels without sending a request', async () => { + api.microblogClippingPosts.mockResolvedValue({ items: [deletedPost] }) + render() + fireEvent.click(await screen.findByRole('button', { name: 'Recreate clippings post' })) + expect(screen.getByRole('dialog').textContent).toContain('https://first.micro.blog/') + fireEvent.click(screen.getByRole('button', { name: 'Cancel' })) + expect(api.recreateMicroblogPost).not.toHaveBeenCalled() +}) +it('queues explicit recreation and refreshes the book status', async () => { + api.microblogClippingPosts.mockResolvedValueOnce({ items: [deletedPost] }).mockResolvedValue({ items: [] }) + render() + fireEvent.click(await screen.findByRole('button', { name: 'Recreate clippings post' })) + fireEvent.click(screen.getByRole('button', { name: 'Recreate post' })) + await waitFor(() => expect(api.recreateMicroblogPost).toHaveBeenCalledWith(session, 'book')) + await waitFor(() => expect(screen.queryByRole('dialog')).toBeNull()) + expect(screen.getByRole('status').textContent).toContain('queued') +}) +it('shows recreation errors and allows retry', async () => { + api.microblogClippingPosts.mockResolvedValue({ items: [deletedPost] }) + api.recreateMicroblogPost.mockRejectedValue(new Error('Original blog is unavailable')) + render() + fireEvent.click(await screen.findByRole('button', { name: 'Recreate clippings post' })) + fireEvent.click(screen.getByRole('button', { name: 'Recreate post' })) + expect((await screen.findByRole('alert')).textContent).toContain('Original blog is unavailable') + expect(screen.getByRole('button', { name: 'Recreate post' }).disabled).toBe(false) }) diff --git a/app/src/MicroblogPosting.jsx b/app/src/MicroblogPosting.jsx new file mode 100644 index 00000000..1d95d2ef --- /dev/null +++ b/app/src/MicroblogPosting.jsx @@ -0,0 +1,40 @@ +import { useRef, useState } from 'react' +import { api } from './api.js' +import { useLoad } from './ui.jsx' + +export default function MicroblogPosting({ session, onChange }) { + const [data, loadError, reload] = useLoad(() => api.microblogDestinations(session), [session]) + const [override, setOverride] = useState(undefined) + const [busy, setBusy] = useState(false) + const pending = useRef(false) + const [error, setError] = useState(null) + const value = override === undefined ? data?.selected || '' : override + async function change(destination) { + if (pending.current) return + pending.current = true + setBusy(true) + setError(null) + try { + await api.setMicroblogDestination(session, destination) + setOverride(destination) + onChange?.() + } catch (e) { setError(e.message || 'Could not save blog') } + finally { pending.current = false; setBusy(false) } + } + return ( +
+ +

For new collections. Existing posts and later clippings stay on their original blog.

+ {data?.items?.length === 0 &&

No blogs are available to this token.

} + {(error || loadError) &&

{error || loadError.message}

} + {loadError && } +
+ ) +} diff --git a/app/src/MicroblogPosting.test.jsx b/app/src/MicroblogPosting.test.jsx new file mode 100644 index 00000000..34eec772 --- /dev/null +++ b/app/src/MicroblogPosting.test.jsx @@ -0,0 +1,67 @@ +// @vitest-environment jsdom +import { afterEach, beforeEach, expect, it, vi } from 'vitest' +import { cleanup, fireEvent, render, screen, waitFor } from '@testing-library/react' +vi.mock('./api.js', () => ({ api: { microblogDestinations: vi.fn(), setMicroblogDestination: vi.fn() } })) +import MicroblogPosting from './MicroblogPosting.jsx' +import { api } from './api.js' +const a = 'https://first.micro.blog/' +const b = 'https://second.micro.blog/' +beforeEach(() => { + api.microblogDestinations.mockResolvedValue({ items: [{ uid: a, name: 'First blog' }, { uid: b, name: 'Second blog' }], selected: null }) + api.setMicroblogDestination.mockResolvedValue({ destination: b }) +}) +afterEach(() => { cleanup(); vi.clearAllMocks() }) +it('requires a choice among multiple blogs and saves it', async () => { + const onChange = vi.fn() + render() + const select = await screen.findByLabelText('Post clippings to') + await waitFor(() => expect(select.options.length).toBe(3)) + expect(select.value).toBe('') + fireEvent.change(select, { target: { value: b } }) + await waitFor(() => expect(onChange).toHaveBeenCalledOnce()) + expect(api.setMicroblogDestination).toHaveBeenCalledWith({}, b) + expect(select.value).toBe(b) +}) +it('shows the sole automatically selected blog without issuing a save', async () => { + api.microblogDestinations.mockResolvedValue({ items: [{ uid: a, name: 'First blog' }], selected: a }) + render() + await waitFor(() => expect(screen.getByLabelText('Post clippings to').value).toBe(a)) + expect(api.setMicroblogDestination).not.toHaveBeenCalled() +}) +it('keeps an unavailable saved choice visible rather than silently choosing another blog', async () => { + api.microblogDestinations.mockResolvedValue({ items: [{ uid: a, name: 'First blog' }], selected: b }) + render() + expect(await screen.findByText(/Unavailable.*second.micro.blog/)).toBeTruthy() + expect(screen.getByLabelText('Post clippings to').value).toBe(b) +}) +it('restores the saved value and allows retry after a failed save', async () => { + api.setMicroblogDestination.mockRejectedValue(new Error('Could not save blog')) + render() + const select = await screen.findByLabelText('Post clippings to') + await waitFor(() => expect(select.options.length).toBe(3)) + fireEvent.change(select, { target: { value: b } }) + expect((await screen.findByRole('alert')).textContent).toContain('Could not save blog') + expect(select.value).toBe('') + expect(select.disabled).toBe(false) +}) +it('shows discovery errors and retries loading', async () => { + api.microblogDestinations.mockRejectedValueOnce(new Error('Discovery unavailable')) + render() + expect((await screen.findByRole('alert')).textContent).toContain('Discovery unavailable') + fireEvent.click(screen.getByRole('button', { name: 'Retry loading blogs' })) + expect(await screen.findByRole('option', { name: 'First blog' })).toBeTruthy() +}) +it('disables the selector until a save completes', async () => { + let finish + api.setMicroblogDestination.mockImplementation(() => new Promise((resolve) => { finish = resolve })) + render() + const select = await screen.findByLabelText('Post clippings to') + await waitFor(() => expect(select.options.length).toBe(3)) + fireEvent.change(select, { target: { value: b } }) + expect(select.disabled).toBe(true) + fireEvent.change(select, { target: { value: a } }) + expect(api.setMicroblogDestination).toHaveBeenCalledTimes(1) + finish({ destination: b }) + await waitFor(() => expect(select.disabled).toBe(false)) + expect(select.value).toBe(b) +}) diff --git a/app/src/Settings.jsx b/app/src/Settings.jsx index e13250da..6d95fde9 100644 --- a/app/src/Settings.jsx +++ b/app/src/Settings.jsx @@ -1,4 +1,5 @@ import { useEffect, useRef, useState } from 'react' +import MicroblogPosting from './MicroblogPosting.jsx' import { openUrl } from '@tauri-apps/plugin-opener' import { onOpenUrl } from '@tauri-apps/plugin-deep-link' import { ArrowLeft, ChevronRight, Copy, KeyRound, ListChecks, Loader2, LogOut, Monitor, Moon, RefreshCw, Search, Server, Sun, Trash2, UserX } from 'lucide-react' @@ -38,7 +39,7 @@ function Appearance({ theme: [pref, setPref] }) { // What each service does, as on the web dashboard. const HINTS = { hardcover: 'Syncs your reading progress and shelf status to Hardcover, and your highlights if you turn them on.', - microblog: 'Keeps your Currently reading and Finished reading bookshelves in sync. Optional highlight sync publishes one post per book and updates its generated content on later syncs, replacing edits made on Micro.blog.', + microblog: 'Keeps your Currently reading and Finished reading bookshelves in sync. Optional clipping sync publishes one post per book and updates its generated content on later syncs, replacing edits made on Micro.blog.', readwise: 'Syncs your highlights to Readwise.', 'readwise-reader': 'Archives books in Reader when you finish them, and brings your Reader progress back.', kosync: 'Mirrors your reading progress to another KOReader-compatible (KOSync) server. For Calibre-Web, use your library URL ending in /kosync.', @@ -578,6 +579,7 @@ function Service({ session, conn, onChange }) { {conn.linked && !needsRelink && conn.options && ( act('options', () => api.setConnectorOptions(session, conn.id, v))} /> )} + {conn.id === 'microblog' && conn.linked && !needsRelink && } {openForm && (!conn.linked || needsRelink) && ( call(s, `/api/v1/connectors/${id}`, { method: 'PUT', body: JSON.stringify({ credential }) }), unlinkConnector: (s, id) => call(s, `/api/v1/connectors/${id}`, { method: 'DELETE' }), syncConnector: (s, id) => call(s, `/api/v1/connectors/${id}/sync`, { method: 'POST' }), - addMicroblogHighlight: (s, doc, clippingId) => call(s, `/api/v1/connectors/microblog/clippings/${encodeURIComponent(doc)}/${encodeURIComponent(clippingId)}`, { method: 'POST' }), + addMicroblogClipping: (s, doc, clippingId) => call(s, `/api/v1/connectors/microblog/clippings/${encodeURIComponent(doc)}/${encodeURIComponent(clippingId)}`, { method: 'POST' }), + microblogDestinations: (s) => call(s, '/api/v1/connectors/microblog/destinations'), + setMicroblogDestination: (s, destination) => call(s, '/api/v1/connectors/microblog/destination', { method: 'PUT', body: JSON.stringify({ destination }) }), + microblogClippingPosts: (s) => call(s, '/api/v1/connectors/microblog/clipping-posts'), + recreateMicroblogPost: (s, doc) => call(s, `/api/v1/connectors/microblog/clipping-posts/${encodeURIComponent(doc)}/recreate`, { method: 'POST' }), syncBook: (s, id, doc) => call(s, `/api/v1/connectors/${id}/sync/${doc}`, { method: 'POST' }), setConnectorOptions: (s, id, options) => call(s, `/api/v1/connectors/${id}/options`, { method: 'PUT', body: JSON.stringify(options) }), beginLink: (s, id) => call(s, `/api/v1/connectors/${id}/link/begin`, { method: 'POST' }), diff --git a/docs/superpowers/plans/2026-10-28-microblog-destinations.md b/docs/superpowers/plans/2026-10-28-microblog-destinations.md index fa139330..db875a53 100644 --- a/docs/superpowers/plans/2026-10-28-microblog-destinations.md +++ b/docs/superpowers/plans/2026-10-28-microblog-destinations.md @@ -2,7 +2,7 @@ > **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. -**Goal:** Select the blog for new highlight collections, keep existing collections on their original blogs, and explicitly recreate deleted posts. +**Goal:** Select the blog for new clipping collections, keep existing collections on their original blogs, and explicitly recreate deleted posts. **Architecture:** A focused destination module reads and validates the token's available blogs. Persist each collection's destination independently of the plugin-wide choice, and route every Micropub request explicitly. Extend the existing clippings publication state with confirmed deletion and a guarded recreation operation; reuse the settings and action-sheet patterns. @@ -12,9 +12,9 @@ ## Global Constraints -- **Post highlights to** chooses the destination only for new collections. +- **Post clippings to** chooses the destination only for new collections. - Existing collections stay on their original blog, including subsequent clippings and explicit recreation. -- Title collection posts **📚 Highlights from {book title}** on creation, recreation, and next update. +- Title collection posts **📚 Clippings from {book title}** on creation, recreation, and next update. - Require a selection when multiple blogs are available; automatically use the sole available blog. - Do not move, copy, or delete existing collections when changing the selected blog. - Never automatically recreate a deleted post or treat a network/authorization failure as confirmed deletion. @@ -48,7 +48,7 @@ **Interfaces:** Consume Task 1 discovery and choice functions. Extend `ClippingsPost` with `destination: string | null` and `missing: number`. Export `recreateMicroblogClippings(db: DB, userId: number, document: string, http: HttpTransport): Promise<{ queued: boolean }>`; add `GET /api/v1/connectors/microblog/clipping-posts` returning this user's post status by canonical document, and `POST /api/v1/connectors/microblog/clipping-posts/:document/recreate`. -- [ ] Write failing publication tests for the **📚 Highlights from Test book** title, selected non-default blog on every Micropub request, sole-blog fallback, no creation without a multi-blog choice, changed selection leaving existing posts pinned, an in-flight selection change, and revoked access refusing to publish rather than falling back. +- [ ] Write failing publication tests for the **📚 Clippings from Test book** title, selected non-default blog on every Micropub request, sole-blog fallback, no creation without a multi-blog choice, changed selection leaving existing posts pinned, an in-flight selection change, and revoked access refusing to publish rather than falling back. - [ ] Add failing legacy tests using an existing post URL: infer the unique matching available destination, prefer the longest valid blog URL prefix, and pause when no unique destination is provable. Test credential rotation with multiple accessible original destinations. - [ ] Add failing deletion/recreation tests: a confirmed missing source pauses publication and retains clippings; 401/403/429/5xx do not set `missing`; explicit recreation verifies absence, stays on the original blog, includes saved and new eligible passages, and preserves normal uncertain-creation recovery. Repeated/concurrent requests create at most one replacement; an existing original post resumes updates without creating another. - [ ] Run `npm test -- test/microblog-clippings-post.test.ts test/microblog-clippings-api.test.ts test/microblog-clippings.test.ts`; confirm new tests fail before implementation. @@ -64,10 +64,10 @@ **Interfaces:** Add `api.microblogDestinations(session)`, `api.setMicroblogDestination(session, uid)`, `api.microblogClippingPosts(session)`, and `api.recreateMicroblogPost(session, document)` for Task 1/2 endpoints. `MicroblogPosting({ session, onChange })` owns destination loading and saving. -- [ ] Write failing UI tests for **Post highlights to**, sole-blog selection, a required choice with multiple blogs, saved-choice restoration, loading/saving errors, unavailable selected blogs, and preventing overlapping saves. Keep **Automatically sync highlights** and its default unchanged. -- [ ] Write failing clipping tests for a per-book **Recreate highlight post** action shown only for missing collections, the confirmation explaining publication on the original blog, cancel without requests, success/refetch, failure feedback, and disabled repeated clicks. +- [ ] Write failing UI tests for **Post clippings to**, sole-blog selection, a required choice with multiple blogs, saved-choice restoration, loading/saving errors, unavailable selected blogs, and preventing overlapping saves. Keep **Automatically sync clippings** and its default unchanged. +- [ ] Write failing clipping tests for a per-book **Recreate clipping post** action shown only for missing collections, the confirmation explaining publication on the original blog, cancel without requests, success/refetch, failure feedback, and disabled repeated clicks. - [ ] Run `npm --prefix app test -- src/MicroblogPosting.test.jsx src/Clippings.test.jsx`; verify the new behaviors fail first. - [ ] Implement the plugin selector using the existing app form style. Explain that it affects new collections while existing posts stay on their original blog. Include an unavailable saved value rather than silently displaying a different blog; show load/save errors and allow retry. -- [ ] Show the recreation action once per affected book on the clippings hub, with an action sheet confirming that eligible collected highlights will be republished to the original blog. Refresh post status after the request and report queuing without claiming publication is already complete. Wire the API methods and update setup documentation, including the emoji title. +- [ ] Show the recreation action once per affected book on the clippings hub, with an action sheet confirming that eligible collected clippings will be republished to the original blog. Refresh post status after the request and report queuing without claiming publication is already complete. Wire the API methods and update setup documentation, including the emoji title. - [ ] Run `npm test`, `npm run build`, `npm --prefix app test`, and `npm --prefix app run build`. Use bundled Node entrypoints if the execution shell lacks npm; report failures by name. Request one independent review of the full change, reproduce important findings with failing tests, and complete one fix pass. - [ ] Commit the tested result on `codex/microblog-clippings`. Keep the worktree available for local Docker testing and report the new settings/recreation behavior, verification, and material limitations. Do not create real Micro.blog posts or alter the user's running container during development. diff --git a/docs/superpowers/specs/2026-10-28-microblog-destinations-design.md b/docs/superpowers/specs/2026-10-28-microblog-destinations-design.md index 16199726..082b7859 100644 --- a/docs/superpowers/specs/2026-10-28-microblog-destinations-design.md +++ b/docs/superpowers/specs/2026-10-28-microblog-destinations-design.md @@ -3,7 +3,7 @@ ## Intent A Micro.blog token may access several blogs. Let the user choose the blog for -new highlight collections, preserve existing collections where they were first +new clipping collections, preserve existing collections where they were first published, and provide an explicit way to resume publication after deleting a remote collection post. @@ -14,19 +14,19 @@ remote collection post. - Choose a blog per clipping or per book: unnecessary complexity for this request. - One plugin-wide choice for new collections, with existing collections pinned to their original blog: the selected approach. The user also approved an - explicit **Recreate highlight post** action after remote deletion. + explicit **Recreate clipping post** action after remote deletion. ## Agreed behavior -- Title collected clipping posts **📚 Highlights from {book title}**, as requested +- Title collected clipping posts **📚 Clippings from {book title}**, as requested for Micro.blog discovery. Apply this to new and recreated posts and to existing collection posts when they are next updated. -- Show **Post highlights to** under the linked Micro.blog plugin. Populate it +- Show **Post clippings to** under the linked Micro.blog plugin. Populate it from the token's authenticated Micropub `q=config` destination list. - Automatically select the sole available blog. When multiple blogs are available, require a selection before creating a new collection post. - Use the selected destination for both manually selected and automatically - included highlights. Bookshelves and book matching remain account-level. + included clippings. Bookshelves and book matching remain account-level. - Once a book's collection post has been created, persist its destination. Additional clippings continue updating that post on its original blog, regardless of subsequent changes to the plugin-wide selection. @@ -34,9 +34,9 @@ remote collection post. - If a collection post is deleted on Micro.blog, keep its clippings locally, report the missing post, and pause publication for that book. Reader sync and other books continue normally. Do not automatically recreate it. -- Offer **Recreate highlight post** for an affected book. Explicit recreation +- Offer **Recreate clipping post** for an affected book. Explicit recreation creates one replacement collection on that book's original blog and resumes - updates there. Include its eligible saved highlights and any new clippings. + updates there. Include its eligible saved clippings and any new clippings. ## Implementation choices @@ -53,9 +53,9 @@ remote collection post. resume updates to it rather than create a duplicate. Treat authorization or network failures separately from deletion; do not permit recreation when deletion is unconfirmed or access to the original blog has been revoked. -- Keep ordinary **Sync now** and **Add highlight to Micro.blog** from silently +- Keep ordinary **Sync now** and **Add clipping to Micro.blog** from silently clearing a missing-post pause. Use the existing action-sheet pattern to explain - that recreation republishes the collected highlights to the original blog. + that recreation republishes the collected clippings to the original blog. - Refresh destination choices from the linked token without exposing credentials; preserve the saved choice across same-account token rotation when accessible. diff --git a/src/connectors/store.ts b/src/connectors/store.ts index deafc030..20cbe474 100644 --- a/src/connectors/store.ts +++ b/src/connectors/store.ts @@ -34,12 +34,12 @@ export function upsertAccount( if (old.token !== cred.token) { // Existing installs may have published before destination discovery was // added. Location URLs provide a conservative same-blog identity hint. - const urls = db.prepare('SELECT url FROM microblog_clipping_posts WHERE user_id=? AND url IS NOT NULL') - .all(userId) as { url: string }[]; - const origins = new Set(urls.flatMap((r) => { try { return [new URL(r.url).origin + '/']; } catch { return []; } })); + const posts = db.prepare('SELECT url,destination FROM microblog_clipping_posts WHERE user_id=?') + .all(userId) as { url: string | null; destination: string | null }[]; + const origins = new Set(posts.flatMap((r) => { try { return r.url ? [new URL(r.url).origin + '/'] : []; } catch { return []; } })); const oldDestination = old.microblogDestination ?? (origins.size === 1 ? [...origins][0] : null); const available = Array.isArray(cred.microblogDestinations) ? cred.microblogDestinations as { uid?: string }[] : []; - if (oldDestination && available.some((d) => d.uid === oldDestination)) changedMicroblogAccount = false; + if (available.some((d) => d.uid === oldDestination || posts.some((p) => p.destination === d.uid))) changedMicroblogAccount = false; else if (oldDestination && cred.microblogDestination) changedMicroblogAccount = oldDestination !== cred.microblogDestination; else verifyMicroblogPosts = true; } @@ -54,14 +54,13 @@ export function upsertAccount( } withTransaction(db, () => { if (verifyMicroblogPosts) { - // Unknown ownership: preserve intent/identity, but recover each existing - // post under the new token before reusing a URL or creating anything. - db.prepare(`UPDATE microblog_clipping_posts SET attempted=CASE WHEN url IS NOT NULL THEN 1 ELSE attempted END, - url=NULL,fingerprint=NULL,error=NULL WHERE user_id=?`).run(userId); + // Retain URL/destination/history. Publication validates access to each + // original blog and its source URL before any further update. db.prepare("DELETE FROM connector_queue WHERE user_id=? AND connector_id='microblog'").run(userId); } if (changedMicroblogAccount) { - for (const table of ['microblog_clipping_selections', 'microblog_clipping_posts']) db.prepare(`DELETE FROM ${table} WHERE user_id=?`).run(userId); + // A new token's default blog cannot prove that older collections should + // be forgotten. Only explicit unlink erases their publication history. for (const table of ['connector_matches', 'connector_queue']) db.prepare(`DELETE FROM ${table} WHERE user_id=? AND connector_id='microblog'`).run(userId); db.prepare("UPDATE connector_accounts SET options='{}' WHERE user_id=? AND connector_id='microblog'").run(userId); } diff --git a/src/routes/web.ts b/src/routes/web.ts index 1abc9a91..1d5e08ae 100644 --- a/src/routes/web.ts +++ b/src/routes/web.ts @@ -613,7 +613,7 @@ async function linkCredential(body) { const HINTS = { hardcover: 'Syncs your reading progress and shelf status to Hardcover.', - microblog: 'Connect an app token to keep your Currently reading and Finished reading bookshelves in sync. Optional highlight sync publishes one post per book and updates its generated content on later syncs, replacing edits made on Micro.blog.', + microblog: 'Connect an app token to keep your Currently reading and Finished reading bookshelves in sync. Optional clipping sync publishes one post per book and updates its generated content on later syncs, replacing edits made on Micro.blog.', readwise: 'Paste your Readwise access token from readwise.io/access_token. Syncs your highlights.', 'readwise-reader': 'Paste your Readwise access token from readwise.io/access_token. Archives books in Reader when you finish them, and brings your Reader progress back to your device.', kosync: 'Mirror your reading progress to another KOReader-compatible (KOSync) server, so your other devices see it too.', @@ -637,7 +637,8 @@ const TOKEN_HELP = { + '
  • Open Account → App tokens.
    • ' + '
  • Create a separate token for CrossPoint Sync.
    • ' + '
  • Copy the new token and paste it below.
    • ' - + '

    Treat this token like a password: it has full access to your Micro.blog account. CrossPoint Sync encrypts it before storing it.

    ', + + '

    Choose the posting blog and automatic clipping sync in app Settings.

    ' + + '

    Treat this token like a password: it has full access to your Micro.blog account. CrossPoint Sync encrypts it before storing it.

    ', kindle: '
    ' + '

    Set up with the CrossPoint Kindle Link extension (recommended):

    ' + '
      ' diff --git a/test/microblog-clippings-api.test.ts b/test/microblog-clippings-api.test.ts index d7813624..4ccf8d15 100644 --- a/test/microblog-clippings-api.test.ts +++ b/test/microblog-clippings-api.test.ts @@ -85,11 +85,11 @@ it('rejects missing, deleted, foreign, unlinked, disabled and unauthenticated se s.db.close(); }); -it('a replacement token cannot inherit another account’s published post or automatic preference', async () => { +it('a replacement token preserves original post history while resetting inaccessible-account preferences', async () => { const s = await setup(); await s.match(); await s.select(); await drainQueue(s.db, s.http); s.db.prepare("UPDATE connector_accounts SET options='{" + '"sync_highlights":true' + "}' WHERE user_id=?").run(s.uid); upsertAccount(s.db, s.uid, 'microblog', { token: 'replacement', microblogDestination: 'https://replacement.micro.blog/' }, null); - expect(s.db.prepare('SELECT COUNT(*) n FROM microblog_clipping_posts').get()).toEqual({ n: 0 }); + expect(s.db.prepare('SELECT url FROM microblog_clipping_posts').get()).toEqual({ url: 'https://example.micro.blog/highlights.html' }); expect(s.db.prepare("SELECT options FROM connector_accounts WHERE user_id=?").get(s.uid)).toEqual({ options: '{}' }); expect(s.db.prepare('SELECT COUNT(*) n FROM connector_matches WHERE user_id=?').get(s.uid)).toEqual({ n: 0 }); s.db.close(); diff --git a/test/microblog-clippings-post.test.ts b/test/microblog-clippings-post.test.ts index f3a26fd7..bddd4828 100644 --- a/test/microblog-clippings-post.test.ts +++ b/test/microblog-clippings-post.test.ts @@ -213,7 +213,7 @@ it('an in-flight old-account upload cannot overwrite replacement-account state o expect(db.prepare('SELECT url,fingerprint FROM microblog_clipping_posts').get()).toEqual({ url: null, fingerprint: null }); expect(db.prepare('SELECT status FROM connector_queue').get()).toEqual({ status: 'pending' }); await drainQueue(db, http); - expect(db.prepare('SELECT url FROM microblog_clipping_posts').get()).toEqual({ url: 'https://replacement.micro.blog/highlights.html' }); + expect(db.prepare('SELECT url,destination FROM microblog_clipping_posts').get()).toEqual({ url: null, destination: 'https://example.micro.blog/' }); db.close(); }); diff --git a/test/microblog-destinations.test.ts b/test/microblog-destinations.test.ts index 4dbcac5e..2f908909 100644 --- a/test/microblog-destinations.test.ts +++ b/test/microblog-destinations.test.ts @@ -62,3 +62,11 @@ it('identifies URL path boundaries and the longest matching blog prefix', () => expect(destinationForPost('https://shared.test/blogging/post.html', blogs)).toBeNull(); expect(destinationForPost('https://shared.test.evil/blog/post.html', blogs)).toBeNull(); }); +it.each([b, 'https://third.micro.blog/'])('never discards original collections when rotating to access %s', async (accessible) => { + const s = await setup([{ uid: a }, { uid: b }]); + upsertAccount(s.db, s.uid, 'microblog', { token: 'old', microblogDestination: a }, null); + s.db.prepare('INSERT INTO microblog_clipping_posts(user_id,document,identity,url,destination) VALUES(?,?,?,?,?)').run(s.uid, 'book', 'identity', `${b}post.html`, b); + upsertAccount(s.db, s.uid, 'microblog', { token: 'new', microblogDestination: accessible, microblogDestinations: [{ uid: accessible }] }, null); + expect(s.db.prepare('SELECT identity,url,destination FROM microblog_clipping_posts').get()).toEqual({ identity: 'identity', url: `${b}post.html`, destination: b }); + s.db.close(); +}); From 36fcb4fb25ff41c26f4ba9197ffe23441487684e Mon Sep 17 00:00:00 2001 From: John Payne Date: Wed, 28 Oct 2026 12:16:13 -0400 Subject: [PATCH 14/37] Allow choosing previous or current blog when recreating clippings posts --- README.md | 15 ++++--- app/src/Clippings.jsx | 36 +++++++++++++--- app/src/Clippings.test.jsx | 27 +++++++++++- app/src/api.js | 2 +- app/src/api.test.js | 8 ++++ ...026-10-28-microblog-destinations-design.md | 22 ++++++---- src/connectors/microblog-clippings.ts | 24 ++++++++--- src/routes/v1/connectors.ts | 5 ++- test/microblog-destination-posts.test.ts | 41 ++++++++++++++++++- 9 files changed, 151 insertions(+), 29 deletions(-) diff --git a/README.md b/README.md index 915c37cf..440e4055 100644 --- a/README.md +++ b/README.md @@ -185,9 +185,9 @@ to that content on Micro.blog. Choose **Post clippings to** in Micro.blog settings when the token accesses multiple blogs. A sole available blog is selected automatically. The choice -applies to new collections: existing posts, new clippings for those books, and -explicitly recreated posts stay on the book's original blog. Revoked access pauses -posting to that blog; selecting a different one never moves existing collections. +applies to new collections: existing posts and new clippings for those books stay +on the book's original blog. Revoked access pauses posting to that blog; selecting +a different one never moves existing collections automatically. **Automatically sync clippings** is off by default. Enabling it includes existing and future clippings. Unmatched books wait without interrupting reader sync; @@ -198,8 +198,13 @@ passages and manually selected clippings. Deleting a post on Micro.blog pauses publication for that book while preserving its clippings locally. After the missing post is detected on an update, the book's section on the Clippings page offers **Recreate clippings post**. This explicitly -republishes the collected clippings to the original blog after verifying the post -is missing. Network or permission errors do not trigger recreation. +republishes the collected clippings to either the previous blog (the default) or +the current clippings blog selected in Settings. Future updates for this book +follow the replacement post. If the previous blog is accessible, Sync rechecks +that the old post is still missing; a restored post can be resumed on the previous +blog instead. A previously confirmed deletion can also be recreated on the current +blog after access to the previous blog is lost. Network or permission errors do +not trigger automatic recreation. For local testing, use Node 22.13 or newer, build the frontend with `npm run build:frontend`, and copy `.env.example` to `.env`. Set a separate diff --git a/app/src/Clippings.jsx b/app/src/Clippings.jsx index c82f255a..d7896f72 100644 --- a/app/src/Clippings.jsx +++ b/app/src/Clippings.jsx @@ -225,13 +225,10 @@ export default function Clippings({ session, books }) { )} - {recreating && setRecreating(null)} onConfirm={async () => { - await api.recreateMicroblogPost(session, recreating.document) + {recreating && setRecreating(null)} onDone={() => { reloadPosts() setNote('Replacement clippings post queued for Micro.blog.') - }}> - This will republish this book’s collected clippings and resume updates on its original blog: {recreating.destination || recreating.url}. - } + }} />} {sharing && setSharing(null)} />} {deleting && ( ) } + +function RecreateClippingsPost({ session, post, onClose, onDone }) { + const [blogs, error, reload] = useLoad(() => api.microblogDestinations(session), [session]) + const [target, setTarget] = useState('original') + const [busy, setBusy] = useState(false) + const currentBlog = blogs?.items?.find((b) => b.uid === blogs.selected) + const original = post.destination || post.url + return ( + { + setBusy(true) + try { + await api.recreateMicroblogPost(session, post.document, target) + onDone() + } finally { setBusy(false) } + }}> + This republishes this book’s collected clippings. Future clippings for this book will update the replacement on the chosen blog. + + {currentBlog?.uid !== post.destination && } + {currentBlog?.uid === post.destination && The current clippings blog is also the previous blog.} + {error && Could not load the current clippings blog. } + + ) +} diff --git a/app/src/Clippings.test.jsx b/app/src/Clippings.test.jsx index 2426a47f..b0756ac0 100644 --- a/app/src/Clippings.test.jsx +++ b/app/src/Clippings.test.jsx @@ -2,7 +2,7 @@ import { afterEach, beforeEach, expect, it, vi } from 'vitest' import { cleanup, fireEvent, render, screen, waitFor } from '@testing-library/react' vi.hoisted(() => { window.scrollTo = vi.fn(); window.matchMedia = vi.fn(() => ({ matches: false, addEventListener() {}, removeEventListener() {} })) }) -vi.mock('./api.js', () => ({ isApp: false, cached: { clippings: () => null }, api: { allClippings: vi.fn(), connectors: vi.fn(), addMicroblogClipping: vi.fn(), microblogClippingPosts: vi.fn(), recreateMicroblogPost: vi.fn() } })) +vi.mock('./api.js', () => ({ isApp: false, cached: { clippings: () => null }, api: { allClippings: vi.fn(), connectors: vi.fn(), addMicroblogClipping: vi.fn(), microblogClippingPosts: vi.fn(), recreateMicroblogPost: vi.fn(), microblogDestinations: vi.fn() } })) vi.mock('./tabbar.js', () => ({ useSheet() {}, useNativeBack() {}, useNativeButton() {} })) import Clippings from './Clippings.jsx' import { api } from './api.js' @@ -15,6 +15,7 @@ beforeEach(() => { api.addMicroblogClipping.mockResolvedValue({ queued: true, waiting_for_match: false }) api.microblogClippingPosts.mockResolvedValue({ items: [] }) api.recreateMicroblogPost.mockResolvedValue({ queued: true }) + api.microblogDestinations.mockResolvedValue({ items: [{ uid: 'https://first.micro.blog/', name: 'Previous' }, { uid: 'https://new.micro.blog/', name: 'New' }], selected: 'https://new.micro.blog/' }) }) afterEach(() => { cleanup(); vi.clearAllMocks() }) @@ -64,7 +65,7 @@ it('queues explicit recreation and refreshes the book status', async () => { render() fireEvent.click(await screen.findByRole('button', { name: 'Recreate clippings post' })) fireEvent.click(screen.getByRole('button', { name: 'Recreate post' })) - await waitFor(() => expect(api.recreateMicroblogPost).toHaveBeenCalledWith(session, 'book')) + await waitFor(() => expect(api.recreateMicroblogPost).toHaveBeenCalledWith(session, 'book', 'original')) await waitFor(() => expect(screen.queryByRole('dialog')).toBeNull()) expect(screen.getByRole('status').textContent).toContain('queued') }) @@ -77,3 +78,25 @@ it('shows recreation errors and allows retry', async () => { expect((await screen.findByRole('alert')).textContent).toContain('Original blog is unavailable') expect(screen.getByRole('button', { name: 'Recreate post' }).disabled).toBe(false) }) +it('lets the reader choose the current clippings blog when recreating a post', async () => { + api.microblogClippingPosts.mockResolvedValue({ items: [deletedPost] }) + render() + fireEvent.click(await screen.findByRole('button', { name: 'Recreate clippings post' })) + const choice = await screen.findByLabelText(/Current clippings blog/) + await waitFor(() => expect(choice.disabled).toBe(false)) + expect(screen.getByLabelText(/Previous blog/).checked).toBe(true) + fireEvent.click(choice) + fireEvent.click(screen.getByRole('button', { name: 'Recreate post' })) + await waitFor(() => expect(api.recreateMicroblogPost).toHaveBeenCalledWith(session, 'book', 'current')) + await waitFor(() => expect(screen.queryByRole('dialog')).toBeNull()) +}) +it('keeps the previous blog usable when the current clippings blog is unavailable', async () => { + api.microblogClippingPosts.mockResolvedValue({ items: [deletedPost] }) + api.microblogDestinations.mockResolvedValue({ items: [], selected: 'https://unavailable.micro.blog/' }) + render() + fireEvent.click(await screen.findByRole('button', { name: 'Recreate clippings post' })) + await screen.findByText(/Choose an available blog in Settings/) + expect(screen.getByLabelText(/Current clippings blog/).disabled).toBe(true) + fireEvent.click(screen.getByRole('button', { name: 'Recreate post' })) + await waitFor(() => expect(api.recreateMicroblogPost).toHaveBeenCalledWith(session, 'book', 'original')) +}) diff --git a/app/src/api.js b/app/src/api.js index f5a46a75..224f7d3e 100644 --- a/app/src/api.js +++ b/app/src/api.js @@ -239,7 +239,7 @@ export const api = { microblogDestinations: (s) => call(s, '/api/v1/connectors/microblog/destinations'), setMicroblogDestination: (s, destination) => call(s, '/api/v1/connectors/microblog/destination', { method: 'PUT', body: JSON.stringify({ destination }) }), microblogClippingPosts: (s) => call(s, '/api/v1/connectors/microblog/clipping-posts'), - recreateMicroblogPost: (s, doc) => call(s, `/api/v1/connectors/microblog/clipping-posts/${encodeURIComponent(doc)}/recreate`, { method: 'POST' }), + recreateMicroblogPost: (s, doc, target = 'original') => call(s, `/api/v1/connectors/microblog/clipping-posts/${encodeURIComponent(doc)}/recreate`, { method: 'POST', body: JSON.stringify({ target }) }), syncBook: (s, id, doc) => call(s, `/api/v1/connectors/${id}/sync/${doc}`, { method: 'POST' }), setConnectorOptions: (s, id, options) => call(s, `/api/v1/connectors/${id}/options`, { method: 'PUT', body: JSON.stringify(options) }), beginLink: (s, id) => call(s, `/api/v1/connectors/${id}/link/begin`, { method: 'POST' }), diff --git a/app/src/api.test.js b/app/src/api.test.js index b6420e61..f8491c96 100644 --- a/app/src/api.test.js +++ b/app/src/api.test.js @@ -19,6 +19,14 @@ beforeEach(() => { fetchMock.mockReset() }) +it('serializes the explicit clippings recreation destination choice', async () => { + fetchMock.mockResolvedValue(response({ queued: true })) + await api.recreateMicroblogPost(session('reader'), 'book', 'current') + expect(fetchMock).toHaveBeenCalledWith('https://sync.test/api/v1/connectors/microblog/clipping-posts/book/recreate', expect.objectContaining({ + method: 'POST', body: '{"target":"current"}', + })) +}) + describe('prefixed server URLs', () => { it.each(['', '/reading', '/services/reading'])('detects the hosted server at %j', (basePath) => { expect(hostedServerUrl(new URL(`https://sync.test${basePath}/app/`))).toBe(`https://sync.test${basePath}`) diff --git a/docs/superpowers/specs/2026-10-28-microblog-destinations-design.md b/docs/superpowers/specs/2026-10-28-microblog-destinations-design.md index 082b7859..25406077 100644 --- a/docs/superpowers/specs/2026-10-28-microblog-destinations-design.md +++ b/docs/superpowers/specs/2026-10-28-microblog-destinations-design.md @@ -34,9 +34,11 @@ remote collection post. - If a collection post is deleted on Micro.blog, keep its clippings locally, report the missing post, and pause publication for that book. Reader sync and other books continue normally. Do not automatically recreate it. -- Offer **Recreate clipping post** for an affected book. Explicit recreation - creates one replacement collection on that book's original blog and resumes - updates there. Include its eligible saved clippings and any new clippings. +- Offer **Recreate clippings post** for an affected book. Explicit recreation + offers its previous blog (default) or the current clippings blog. Create one + replacement collection and pin later updates to the chosen blog. Include its + eligible saved clippings and any new clippings. This destination choice was + requested after local testing on October 28. ## Implementation choices @@ -49,13 +51,17 @@ remote collection post. collection with an actionable error rather than choose a different blog. - Pin the destination before the first creation attempt so retries and uncertain creation recovery cannot be redirected by a concurrent settings change. -- Recreation must verify the original post is missing. If it is still present, - resume updates to it rather than create a duplicate. Treat authorization or - network failures separately from deletion; do not permit recreation when - deletion is unconfirmed or access to the original blog has been revoked. +- Recreation requires a confirmed missing post and a currently accessible target + blog. When the previous blog is accessible, recheck its post; if restored, only + resume updates there rather than create a duplicate elsewhere. With an existing + confirmed deletion, allow explicit recreation on the current blog after access + to the previous blog is revoked. Network and authorization failures during a + source recheck still stop recreation. - Keep ordinary **Sync now** and **Add clipping to Micro.blog** from silently clearing a missing-post pause. Use the existing action-sheet pattern to explain - that recreation republishes the collected clippings to the original blog. + that recreation republishes the collected clippings to the chosen blog. Show + addresses for both choices; disable the current choice if no available blog is + selected and show only the previous choice when both destinations are the same. - Refresh destination choices from the linked token without exposing credentials; preserve the saved choice across same-account token rotation when accessible. diff --git a/src/connectors/microblog-clippings.ts b/src/connectors/microblog-clippings.ts index 5ed19ed3..b2f2a2d1 100644 --- a/src/connectors/microblog-clippings.ts +++ b/src/connectors/microblog-clippings.ts @@ -293,7 +293,7 @@ export async function publishMicroblogClippings( } finally { locks.delete(key); } } -export async function recreateMicroblogClippings(db: DB, userId: number, doc: string, http: HttpTransport): Promise<{ queued: boolean }> { +export async function recreateMicroblogClippings(db: DB, userId: number, doc: string, http: HttpTransport, target: 'original' | 'current' = 'original'): Promise<{ queued: boolean }> { const document = resolveDocument(db, userId, doc); const account = getAccount(db, userId, 'microblog'); if (!account?.enabled) throw new ConnectorOperationError('Micro.blog is not linked or enabled', false); @@ -310,12 +310,24 @@ export async function recreateMicroblogClippings(db: DB, userId: number, doc: st try { const cred = decryptCredential(account, db); const blogs = await listMicroblogDestinations(cred, http); current(); - const destination = state.destination ?? destinationForPost(state.url, blogs); - if (!destination || !blogs.some((b) => b.uid === destination)) throw new ConnectorOperationError('Restore access to the original Micro.blog blog before recreating this post.', false); - let missing = false; - try { await micropub(http, String(cred.token), destination, `/micropub?q=source&url=${encodeURIComponent(state.url)}`, undefined, true); } - catch (error) { if (error instanceof MissingMicroblogPost) missing = true; else throw error; } + const original = state.destination ?? destinationForPost(state.url, blogs); + const destination = target === 'current' + ? selectedMicroblogDestination(db, userId) ?? (blogs.length === 1 ? blogs[0].uid : null) + : original; + if (!destination || !blogs.some((b) => b.uid === destination)) throw new ConnectorOperationError(target === 'current' + ? 'Choose an available clippings blog in Micro.blog settings before recreating this post.' + : 'Restore access to the original Micro.blog blog or choose the current clippings blog.', false); + // The durable missing flag permits an explicit move after original-blog access is lost. + // When accessible, recheck the original to avoid duplicating a restored post. + let missing = true; + if (original && blogs.some((b) => b.uid === original)) { + try { + await micropub(http, String(cred.token), original, `/micropub?q=source&url=${encodeURIComponent(state.url)}`, undefined, true); + missing = false; + } catch (error) { if (!(error instanceof MissingMicroblogPost)) throw error; } + } current(); + if (!missing && destination !== original) throw new ConnectorOperationError('The previous post exists again. Choose the previous blog to resume updating it.', false); withTransaction(db, () => { if (missing) db.prepare(`UPDATE microblog_clipping_posts SET identity=?,url=NULL,fingerprint=NULL,attempted=0,error=NULL,missing=0,destination=? WHERE user_id=? AND document=?`) .run(randomUUID(), destination, userId, document); diff --git a/src/routes/v1/connectors.ts b/src/routes/v1/connectors.ts index 69f77094..d8ed1088 100644 --- a/src/routes/v1/connectors.ts +++ b/src/routes/v1/connectors.ts @@ -486,7 +486,10 @@ export function connectorRoutes( if (!credentialRequestIsSecure(c, trustProxy, allowHttpLocalhost)) return c.json({ message: CREDENTIAL_HTTPS_REQUIRED }, 400); const document = c.req.param('document'); if (!isValidDocument(document)) return c.json({ message: 'Invalid book' }, 400); - try { return c.json(await recreateMicroblogClippings(db, c.get('user').id, document, transport)); } + const body = await c.req.json().catch(() => null) as { target?: unknown } | null; + const target = body?.target ?? 'original'; + if (target !== 'original' && target !== 'current') return c.json({ message: 'Choose the previous or current clippings blog' }, 400); + try { return c.json(await recreateMicroblogClippings(db, c.get('user').id, document, transport, target)); } catch (error) { return c.json({ message: error instanceof Error ? error.message : 'Could not recreate post' }, error instanceof ConnectorOperationError && !error.retryable ? 409 : 502); } }); diff --git a/test/microblog-destination-posts.test.ts b/test/microblog-destination-posts.test.ts index 144c2b14..8bf7a574 100644 --- a/test/microblog-destination-posts.test.ts +++ b/test/microblog-destination-posts.test.ts @@ -39,7 +39,7 @@ async function setup() { selectMicroblogClipping(db, uid, DOC, 'one'); const choose = (destination: string) => app.request('/api/v1/connectors/microblog/destination', { method: 'PUT', headers, body: JSON.stringify({ destination }) }); const sync = async () => { scheduleMicroblogClippings(db, uid, DOC); await drainQueue(db, http); }; - const recreate = () => app.request(`/api/v1/connectors/microblog/clipping-posts/${DOC}/recreate`, { method: 'POST', headers }); + const recreate = (target?: string) => app.request(`/api/v1/connectors/microblog/clipping-posts/${DOC}/recreate`, { method: 'POST', headers, ...(target ? { body: JSON.stringify({ target }) } : {}) }); const status = async () => (await app.request('/api/v1/connectors/microblog/clipping-posts', { headers })).json(); return { db, app, uid, headers, http, writes, requests, choose, sync, recreate, status, setBlogs: (value: string[]) => { blogs = value; }, missing: (code = 404) => { sourceStatus = code; }, @@ -88,6 +88,45 @@ it.each([401, 403, 429, 500])('does not offer recreation for an unconfirmed dele expect(await s.status()).toMatchObject({ items: [{ missing: false }] }); expect((await s.recreate()).status).not.toBe(200); expect(s.writes).toHaveLength(1); s.db.close(); }); +it('recreates on the current clippings blog when explicitly chosen and pins later updates there', async () => { + const s = await setup(); await s.choose(b); await s.sync(); s.missing(); s.change(); await s.sync(); + await s.choose(a); + expect((await s.recreate('current')).status).toBe(200); + await drainQueue(s.db, s.http); + expect(s.writes[1]).toMatchObject({ destination: a, body: { properties: { name: ['📚 Clippings from Test book'] } } }); + expect(JSON.stringify(s.writes[1])).toContain('First passage'); + await s.choose(b); s.missing(200); + s.db.prepare('UPDATE clippings SET note=? WHERE user_id=?').run('Later annotation', s.uid); + await s.sync(); + expect(s.writes[2]).toMatchObject({ destination: a, body: { action: 'update', url: `${a}post.html` } }); + s.db.close(); +}); +it('allows explicit recreation on the current blog after access to the previous blog is lost', async () => { + const s = await setup(); await s.choose(b); await s.sync(); s.missing(); s.change(); await s.sync(); + await s.choose(a); s.setBlogs([a]); + expect((await s.recreate('current')).status).toBe(200); await drainQueue(s.db, s.http); + expect(s.writes[1]?.destination).toBe(a); s.db.close(); +}); +it('rejects an invalid recreation choice without changing the paused post', async () => { + const s = await setup(); await s.choose(b); await s.sync(); s.missing(); s.change(); await s.sync(); + expect((await s.recreate('elsewhere')).status).toBe(400); + expect(await s.status()).toMatchObject({ items: [{ missing: true, destination: b }] }); s.db.close(); +}); +it.each(['unselected', 'revoked'])('does not fall back to another blog when the current blog is %s', async (mode) => { + const s = await setup(); await s.choose(b); await s.sync(); s.missing(); s.change(); await s.sync(); + if (mode === 'unselected') s.db.prepare("UPDATE connector_accounts SET options='{}' WHERE user_id=?").run(s.uid); + else { await s.choose(a); s.setBlogs([b]); } + expect((await s.recreate('current')).status).toBe(409); + expect(await s.status()).toMatchObject({ items: [{ missing: true, destination: b }] }); + expect(s.writes).toHaveLength(1); s.db.close(); +}); +it('refuses to move a post that has been restored on the previous blog', async () => { + const s = await setup(); await s.choose(b); await s.sync(); s.missing(); s.change(); await s.sync(); + await s.choose(a); s.missing(200); + expect((await s.recreate('current')).status).toBe(409); + expect(await s.status()).toMatchObject({ items: [{ missing: true, destination: b }] }); + expect(s.writes).toHaveLength(1); s.db.close(); +}); it('resumes an original post that has been restored instead of creating a duplicate', async () => { const s = await setup(); await s.choose(b); await s.sync(); s.missing(); s.change(); await s.sync(); s.missing(200); expect((await s.recreate()).status).toBe(200); await drainQueue(s.db, s.http); From 1567a45e971e1bcdd45d11688f2789866ecb3381 Mon Sep 17 00:00:00 2001 From: John Payne Date: Wed, 28 Oct 2026 12:22:47 -0400 Subject: [PATCH 15/37] Use string content for Micro.blog clippings updates --- src/connectors/microblog-clippings.ts | 8 +++++--- test/microblog-clippings-post.test.ts | 20 ++++++++++++++++++++ 2 files changed, 25 insertions(+), 3 deletions(-) diff --git a/src/connectors/microblog-clippings.ts b/src/connectors/microblog-clippings.ts index b2f2a2d1..2ee79938 100644 --- a/src/connectors/microblog-clippings.ts +++ b/src/connectors/microblog-clippings.ts @@ -142,6 +142,8 @@ class MissingMicroblogPost extends ConnectorOperationError { } async function micropub(http: HttpTransport, token: string, destination: string, path: string, body?: unknown, sourceLookup = false) { + const operation = body ? (body as { action?: string }).action === 'update' ? 'update' : 'create' + : sourceLookup ? 'source lookup' : 'post recovery'; let res; try { const url = new URL(path, 'https://micro.blog'); @@ -151,14 +153,14 @@ async function micropub(http: HttpTransport, token: string, destination: string, headers: { authorization: `Bearer ${token}`, ...(body ? { 'content-type': 'application/json' } : {}) }, ...(body ? { body: JSON.stringify(body) } : {}), }); - } catch { throw new ConnectorOperationError('Micro.blog Micropub connection failed; the next attempt will check for an existing post.', true); } + } catch { throw new ConnectorOperationError(`Micro.blog Micropub ${operation} connection failed; the next attempt will check for an existing post.`, true); } if (sourceLookup && res.status === 404) throw new MissingMicroblogPost(); if (sourceLookup && res.status === 400) { const failure = await res.json().catch(() => null) as { error?: string; error_description?: string } | null; if (failure?.error === 'invalid_request' && /no post|not found|does not exist|could not find|unknown.*url/i.test(failure.error_description || '')) throw new MissingMicroblogPost(); } if (res.status >= 300) { - throw new ConnectorOperationError(`Micro.blog Micropub failed (${res.status})`, res.status === 429 || res.status >= 500, + throw new ConnectorOperationError(`Micro.blog Micropub ${operation} failed (${res.status})`, res.status === 429 || res.status >= 500, res.status === 401 || res.status === 403); } return res; @@ -250,7 +252,7 @@ export async function publishMicroblogClippings( await micropub(http, token, destination, `/micropub?q=source&url=${encodeURIComponent(url)}`, undefined, true); assertCurrent(); await micropub(http, token, destination, '/micropub', { action: 'update', url, - replace: { name: [rendered.title], content: [{ html: rendered.content }] } }); + replace: { name: [rendered.title], content: [rendered.content] } }); assertCurrent(); } catch (err) { assertCurrent(); diff --git a/test/microblog-clippings-post.test.ts b/test/microblog-clippings-post.test.ts index bddd4828..e9259b3b 100644 --- a/test/microblog-clippings-post.test.ts +++ b/test/microblog-clippings-post.test.ts @@ -54,10 +54,30 @@ it('publishes one escaped collected post, skips unchanged content, then updates await drainQueue(db, http); expect(writes).toHaveLength(2); expect(writes[1]).toMatchObject({ action: 'update', url: remoteUrl }); + // Micro.blog's update API accepts the generated HTML as a string, unlike + // the html-value object supported on initial creation. + expect((writes[1] as any).replace.content).toEqual([expect.stringContaining('<First passage>')]); expect(JSON.stringify(writes[1])).toContain('A note'); db.close(); }); +it.each(['source lookup', 'update'])('identifies a failed %s and preserves the existing post for retry', async (operation) => { + const { db, uid } = await setup(); + const http: HttpTransport = async (_url, init) => { + if (init.method === 'GET') return response(operation === 'source lookup' ? 500 : 200); + return response(500); + }; + db.prepare('UPDATE microblog_clipping_posts SET url=?,attempted=1').run(remoteUrl); + await drainQueue(db, http); + expect(db.prepare('SELECT status,last_error FROM connector_queue').get()).toMatchObject({ + status: 'pending', last_error: expect.stringContaining(`Micropub ${operation} failed (500)`), + }); + expect(db.prepare('SELECT url,missing,error,fingerprint FROM microblog_clipping_posts WHERE user_id=?').get(uid)).toEqual({ + url: remoteUrl, missing: 0, error: null, fingerprint: null, + }); + db.close(); +}); + it('recovers an uncertain creation by source identity without creating a second post', async () => { const { db } = await setup(); let creates = 0; From 23a183f47f9e2b9d8701619f184fd2317444ff06 Mon Sep 17 00:00:00 2001 From: John Payne Date: Wed, 28 Oct 2026 12:51:15 -0400 Subject: [PATCH 16/37] Prevent Micro.blog publication lock leaks on account read failures --- src/connectors/microblog-clippings.ts | 2 +- test/microblog-clippings-post.test.ts | 17 ++++++++++++++++- 2 files changed, 17 insertions(+), 2 deletions(-) diff --git a/src/connectors/microblog-clippings.ts b/src/connectors/microblog-clippings.ts index 2ee79938..58eeabc1 100644 --- a/src/connectors/microblog-clippings.ts +++ b/src/connectors/microblog-clippings.ts @@ -198,7 +198,6 @@ export async function publishMicroblogClippings( let locks = publishing.get(db); if (!locks) { locks = new Set(); publishing.set(db, locks); } if (locks.has(key)) return { ok: false, retryable: true, error: 'Micro.blog book publication is already in progress' }; - locks.add(key); const account = getAccount(db, userId, 'microblog'); let identity: string | undefined; const assertCurrent = () => { @@ -213,6 +212,7 @@ export async function publishMicroblogClippings( db.prepare('UPDATE microblog_clipping_posts SET error=? WHERE user_id=? AND document=?').run(error, userId, document); return { ok: false, retryable: false, error }; }; + locks.add(key); try { const token = typeof cred.token === 'string' ? cred.token.trim() : ''; if (!token) throw new ConnectorOperationError('Missing Micro.blog token', false, true); diff --git a/test/microblog-clippings-post.test.ts b/test/microblog-clippings-post.test.ts index e9259b3b..b92970a2 100644 --- a/test/microblog-clippings-post.test.ts +++ b/test/microblog-clippings-post.test.ts @@ -2,7 +2,7 @@ import { afterEach, beforeEach, expect, it } from 'vitest'; import { makeTestApp, registerUser, DOC } from './helpers.js'; import { resetEncryptionKeyCache } from '../src/crypto/secrets.js'; import { upsertAccount, saveMatch } from '../src/connectors/store.js'; -import { selectMicroblogClipping, scheduleMicroblogClippings } from '../src/connectors/microblog-clippings.js'; +import { publishMicroblogClippings, selectMicroblogClipping, scheduleMicroblogClippings } from '../src/connectors/microblog-clippings.js'; import { drainQueue as runQueue } from '../src/connectors/runner.js'; import type { HttpTransport } from '../src/connectors/types.js'; @@ -32,6 +32,21 @@ function response(status: number, body: unknown = {}, location?: string) { return { status, json: async () => body, text: async () => JSON.stringify(body), headers: { get: (key: string) => key.toLowerCase() === 'location' ? location ?? null : null } }; } +it('allows retry after an account read fails before publication starts', async () => { + const { db, uid } = await setup(); + db.exec('ALTER TABLE connector_accounts RENAME TO unavailable_accounts'); + try { + await expect(publishMicroblogClippings( + { token: 'token' }, { externalId: '123', externalEdition: '9781786487849', confidence: 1 }, + { kind: 'highlight', document: DOC, timestamp: 1 }, async () => response(201, {}, remoteUrl), { db, userId: uid }, + )).rejects.toThrow('no such table'); + } finally { db.exec('ALTER TABLE unavailable_accounts RENAME TO connector_accounts'); } + await drainQueue(db, async () => response(201, {}, remoteUrl)); + expect(db.prepare('SELECT url FROM microblog_clipping_posts').get()).toEqual({ url: remoteUrl }); + expect(db.prepare('SELECT status FROM connector_queue').get()).toEqual({ status: 'done' }); + db.close(); +}); + it('publishes one escaped collected post, skips unchanged content, then updates the saved URL', async () => { const { db, uid } = await setup(); const writes: Record[] = []; From 238de639c2a2b99733e44f7284758988bdec8d41 Mon Sep 17 00:00:00 2001 From: John Payne Date: Wed, 28 Oct 2026 12:52:14 -0400 Subject: [PATCH 17/37] Keep reader sync working when Micro.blog scheduling fails --- src/connectors/fanout.ts | 7 +++- src/connectors/refresh.ts | 7 +++- src/routes/v1/clippings.ts | 14 ++++++-- test/microblog-clippings-api.test.ts | 48 ++++++++++++++++++++++++++++ 4 files changed, 72 insertions(+), 4 deletions(-) diff --git a/src/connectors/fanout.ts b/src/connectors/fanout.ts index 5cbbf2a3..d7a4ddad 100644 --- a/src/connectors/fanout.ts +++ b/src/connectors/fanout.ts @@ -65,7 +65,12 @@ export function fanOutProgress( positionJson?: string | null, exceptConnectorId?: string ): void { - scheduleMicroblogClippings(db, userId, document); + try { + scheduleMicroblogClippings(db, userId, document); + } catch (err) { + console.error(JSON.stringify({ msg: 'microblog schedule failed', user_id: userId, document, + error: err instanceof Error ? err.message : String(err) })); + } const finished = percentage >= 0.98; let position: Record | null = null; if (positionJson) { diff --git a/src/connectors/refresh.ts b/src/connectors/refresh.ts index 228f6c2b..b44999e5 100644 --- a/src/connectors/refresh.ts +++ b/src/connectors/refresh.ts @@ -40,7 +40,12 @@ const PER_BOOK_PULLERS: { id: string; strict: boolean; sidecarOnly: boolean; mat export function createProgressRefresh(db: DB, http: HttpTransport = fetchTransport): ProgressRefresh { const pending = new Map>(); return async (userId, document) => { - scheduleMicroblogClippings(db, userId, document); + try { + scheduleMicroblogClippings(db, userId, document); + } catch (err) { + console.error(JSON.stringify({ msg: 'microblog schedule failed', user_id: userId, document, + error: err instanceof Error ? err.message : String(err) })); + } const pullers: typeof PER_BOOK_PULLERS = []; for (const p of PER_BOOK_PULLERS) { const account = getAccount(db, userId, p.id); diff --git a/src/routes/v1/clippings.ts b/src/routes/v1/clippings.ts index 29eb7d19..823662ae 100644 --- a/src/routes/v1/clippings.ts +++ b/src/routes/v1/clippings.ts @@ -82,7 +82,12 @@ export function clippingRoutes(db: DB): Hono { if (!Number.isSafeInteger(cursor) || cursor < 0) return kosyncError(c, 403, 2003, 'Invalid cursor'); const revisionMode = cursorRaw !== undefined; const canonical = resolveDocument(db, user.id, document); - scheduleMicroblogClippings(db, user.id, canonical); + try { + scheduleMicroblogClippings(db, user.id, canonical); + } catch (err) { + console.error(JSON.stringify({ msg: 'microblog schedule failed', user_id: user.id, document: canonical, + error: err instanceof Error ? err.message : String(err) })); + } const documents = clippingDocuments(db, user.id, canonical); const rows = db .prepare( @@ -280,7 +285,12 @@ export function clippingRoutes(db: DB): Hono { ); } } - scheduleMicroblogClippings(db, user.id, canonical); + try { + scheduleMicroblogClippings(db, user.id, canonical); + } catch (err) { + console.error(JSON.stringify({ msg: 'microblog schedule failed', user_id: user.id, document: canonical, + error: err instanceof Error ? err.message : String(err) })); + } return c.json({ until: now, accepted: ops.length }); }); diff --git a/test/microblog-clippings-api.test.ts b/test/microblog-clippings-api.test.ts index 4ccf8d15..c1318510 100644 --- a/test/microblog-clippings-api.test.ts +++ b/test/microblog-clippings-api.test.ts @@ -6,6 +6,8 @@ import { drainQueue } from '../src/connectors/runner.js'; import type { HttpTransport } from '../src/connectors/types.js'; import { deleteDocumentData } from '../src/models/document.js'; import { deleteKosyncUserData } from '../src/routes/account.js'; +import { fanOutProgress } from '../src/connectors/fanout.js'; +import { createProgressRefresh } from '../src/connectors/refresh.js'; beforeEach(() => { process.env.TOKEN_ENC_KEY = 'a'.repeat(64); resetEncryptionKeyCache(); }); afterEach(() => { delete process.env.TOKEN_ENC_KEY; resetEncryptionKeyCache(); }); @@ -30,6 +32,52 @@ async function setup() { return { app, db, uid, headers, posts, http, put, clipping, select, match }; } +async function schedulingFailure() { + const s = await setup(); + s.db.prepare('UPDATE connector_accounts SET options=? WHERE user_id=? AND connector_id=?') + .run(JSON.stringify({ sync_highlights: true }), s.uid, 'microblog'); + s.db.exec(`CREATE TEMP TRIGGER fail_microblog_selection BEFORE INSERT ON microblog_clipping_selections + BEGIN SELECT RAISE(FAIL, 'simulated scheduling database failure'); END`); + return s; +} + +it('still fans out progress to other connectors when Micro.blog scheduling fails', async () => { + const s = await schedulingFailure(); + upsertAccount(s.db, s.uid, 'kosync', { server: 'mirror.test', username: 'reader', password: 'test' }, null); + expect(() => fanOutProgress(s.db, s.uid, DOC, 0.4, 100)).not.toThrow(); + expect(s.db.prepare("SELECT kind FROM connector_queue WHERE connector_id='kosync'").get()).toEqual({ kind: 'progress' }); + s.db.close(); +}); + +it('does not reject progress refresh when Micro.blog scheduling fails', async () => { + const s = await schedulingFailure(); + await expect(createProgressRefresh(s.db, s.http)(s.uid, DOC)).resolves.toBeUndefined(); + s.db.close(); +}); + +it('still returns clippings when Micro.blog scheduling fails', async () => { + const s = await schedulingFailure(); + const res = await s.app.request(`/api/v1/clippings/${DOC}`, { headers: s.headers }); + expect(res.status).toBe(200); + expect(await res.json()).toMatchObject({ items: [{ id: 'one', text: 'A passage' }] }); + s.db.close(); +}); + +it('acknowledges saved clippings and can schedule them on a later sync after a database failure', async () => { + const s = await schedulingFailure(); + const res = await s.put(`/api/v1/clippings/${DOC}`, { items: [{ id: 'two', spine: 2, text: 'Another passage', created_at: 2 }] }); + expect(res.status).toBe(200); + expect(await res.json()).toMatchObject({ accepted: 1 }); + expect(s.db.prepare('SELECT text FROM clippings WHERE user_id=? AND id=?').get(s.uid, 'two')).toEqual({ text: 'Another passage' }); + s.db.exec('DROP TRIGGER fail_microblog_selection'); + await s.match(); + expect((await s.app.request(`/api/v1/clippings/${DOC}`, { headers: s.headers })).status).toBe(200); + await drainQueue(s.db, s.http); + expect(s.posts).toHaveLength(1); + expect(JSON.stringify(s.posts[0])).toContain('Another passage'); + s.db.close(); +}); + it('defaults automatic publication off and remembers a manually selected unmatched clipping', async () => { const s = await setup(); expect(s.db.prepare("SELECT COUNT(*) n FROM connector_queue WHERE kind='microblog:clippings'").get()).toEqual({ n: 0 }); From b94a1d2de2a5d1814bd2320e13b3741c7dd7a1b0 Mon Sep 17 00:00:00 2001 From: John Payne Date: Wed, 28 Oct 2026 12:56:20 -0400 Subject: [PATCH 18/37] Acknowledge Micro.blog rematches when post content is unchanged --- src/connectors/microblog-clippings.ts | 6 +++++- test/microblog-destination-posts.test.ts | 11 +++++++++++ 2 files changed, 16 insertions(+), 1 deletion(-) diff --git a/src/connectors/microblog-clippings.ts b/src/connectors/microblog-clippings.ts index 58eeabc1..92df3bc4 100644 --- a/src/connectors/microblog-clippings.ts +++ b/src/connectors/microblog-clippings.ts @@ -238,7 +238,11 @@ export async function publishMicroblogClippings( const rendered = microblogClippingsContent(db, userId, document, `https://micro.blog/books/${encodeURIComponent(isbn)}`); if (!rendered) return { ok: true }; db.prepare('UPDATE microblog_clipping_posts SET destination=? WHERE user_id=? AND document=?').run(destination, userId, document); - if (rendered.fingerprint === state.fingerprint && state.url) return { ok: true }; + if (rendered.fingerprint === state.fingerprint && state.url) { + db.prepare('UPDATE microblog_clipping_posts SET book_url=?,matched_book=? WHERE user_id=? AND document=?') + .run(`https://micro.blog/books/${encodeURIComponent(isbn)}`, match.externalId, userId, document); + return { ok: true }; + } let url = state.url; if (!url && state.attempted) { url = await recoverPost(http, token, destination, state.identity); diff --git a/test/microblog-destination-posts.test.ts b/test/microblog-destination-posts.test.ts index 8bf7a574..fb3db2e0 100644 --- a/test/microblog-destination-posts.test.ts +++ b/test/microblog-destination-posts.test.ts @@ -59,6 +59,17 @@ it('creates 📚 collections on the selected blog and keeps updates there after expect(s.requests.every((url) => new URL(url).searchParams.get('mp-destination') === b)).toBe(true); s.db.close(); }); +it('acknowledges a rematch with unchanged ISBN and content without requeuing publication', async () => { + const s = await setup(); await s.choose(b); await s.sync(); + saveMatch(s.db, s.uid, 'microblog', DOC, { externalId: '456', externalEdition: '9781786487849', confidence: 1 }, 'manual'); + await s.sync(); + expect(s.db.prepare('SELECT matched_book,book_url FROM microblog_clipping_posts WHERE user_id=? AND document=?').get(s.uid, DOC)) + .toEqual({ matched_book: '456', book_url: 'https://micro.blog/books/9781786487849' }); + expect(s.db.prepare("SELECT status FROM connector_queue WHERE kind='microblog:clippings'").get()).toEqual({ status: 'done' }); + expect(scheduleMicroblogClippings(s.db, s.uid, DOC)).toBe(0); + expect(s.writes).toHaveLength(1); + s.db.close(); +}); it('pins creation before a concurrent change of the plugin-wide blog choice', async () => { const s = await setup(); await s.choose(b); s.onCreate(async () => { await s.choose(a); }); From 73d136e2628584e0168260eb09a271f1150ae619 Mon Sep 17 00:00:00 2001 From: John Payne Date: Wed, 28 Oct 2026 12:57:24 -0400 Subject: [PATCH 19/37] Allow recreation when a Micro.blog post disappears during update --- src/connectors/microblog-clippings.ts | 5 +++-- test/microblog-destination-posts.test.ts | 24 ++++++++++++++++++++++++ 2 files changed, 27 insertions(+), 2 deletions(-) diff --git a/src/connectors/microblog-clippings.ts b/src/connectors/microblog-clippings.ts index 92df3bc4..b9345bcd 100644 --- a/src/connectors/microblog-clippings.ts +++ b/src/connectors/microblog-clippings.ts @@ -154,8 +154,9 @@ async function micropub(http: HttpTransport, token: string, destination: string, ...(body ? { body: JSON.stringify(body) } : {}), }); } catch { throw new ConnectorOperationError(`Micro.blog Micropub ${operation} connection failed; the next attempt will check for an existing post.`, true); } - if (sourceLookup && res.status === 404) throw new MissingMicroblogPost(); - if (sourceLookup && res.status === 400) { + const existingPost = sourceLookup || operation === 'update'; + if (existingPost && res.status === 404) throw new MissingMicroblogPost(); + if (existingPost && res.status === 400) { const failure = await res.json().catch(() => null) as { error?: string; error_description?: string } | null; if (failure?.error === 'invalid_request' && /no post|not found|does not exist|could not find|unknown.*url/i.test(failure.error_description || '')) throw new MissingMicroblogPost(); } diff --git a/test/microblog-destination-posts.test.ts b/test/microblog-destination-posts.test.ts index fb3db2e0..bb1be6f3 100644 --- a/test/microblog-destination-posts.test.ts +++ b/test/microblog-destination-posts.test.ts @@ -10,6 +10,7 @@ afterEach(() => { delete process.env.TOKEN_ENC_KEY; resetEncryptionKeyCache(); } const a = 'https://first.micro.blog/'; const b = 'https://second.micro.blog/'; async function setup() { let blogs = [a, b]; let sourceStatus = 200; + let updateStatus = 200; let updateDescription = 'No post found for this URL.'; const writes: { destination: string | null; body: any }[] = []; const requests: string[] = []; let onCreate: (() => Promise) | undefined; @@ -25,6 +26,7 @@ async function setup() { else { const data = JSON.parse(init.body!); writes.push({ destination, body: data }); if (!data.action) { await onCreate?.(); status = 201; location = `${destination || a}post.html`; } + else if (data.action === 'update') { status = updateStatus; body = { error: 'invalid_request', error_description: updateDescription }; } } } return { status, headers: { get: () => location }, json: async () => body, text: async () => JSON.stringify(body) }; @@ -43,6 +45,7 @@ async function setup() { const status = async () => (await app.request('/api/v1/connectors/microblog/clipping-posts', { headers })).json(); return { db, app, uid, headers, http, writes, requests, choose, sync, recreate, status, setBlogs: (value: string[]) => { blogs = value; }, missing: (code = 404) => { sourceStatus = code; }, + updateFailure: (code: number, description = 'No post found for this URL.') => { updateStatus = code; updateDescription = description; }, onCreate: (fn: () => Promise) => { onCreate = fn; }, onDiscover: (fn: () => void) => { onDiscover = fn; }, change: () => db.prepare('UPDATE clippings SET note=? WHERE user_id=?').run('New annotation', uid) }; @@ -99,6 +102,27 @@ it.each([401, 403, 429, 500])('does not offer recreation for an unconfirmed dele expect(await s.status()).toMatchObject({ items: [{ missing: false }] }); expect((await s.recreate()).status).not.toBe(200); expect(s.writes).toHaveLength(1); s.db.close(); }); +it.each([404, 400])('offers explicit recreation when the update reports a deleted post after a successful lookup (%s)', async (code) => { + const s = await setup(); await s.choose(b); await s.sync(); + s.updateFailure(code); s.change(); await s.sync(); + expect(await s.status()).toMatchObject({ items: [{ document: DOC, missing: true, destination: b, url: `${b}post.html` }] }); + await s.sync(); expect(s.writes).toHaveLength(2); // No automatic recreation. + s.missing(); + expect((await s.recreate()).status).toBe(200); + await drainQueue(s.db, s.http); + expect(s.writes.filter((w) => !w.body.action)).toHaveLength(2); + expect(JSON.stringify(s.writes[2])).toContain('New annotation'); + expect(await s.status()).toMatchObject({ items: [{ missing: false, error: null }] }); + s.db.close(); +}); +it.each([400, 401, 403, 429, 500])('does not mark an unrelated update failure as deletion (%s)', async (code) => { + const s = await setup(); await s.choose(b); await s.sync(); + s.updateFailure(code, 'Unsupported content format'); s.change(); await s.sync(); + expect(await s.status()).toMatchObject({ items: [{ missing: false, url: `${b}post.html` }] }); + expect((await s.recreate()).status).toBe(409); + expect(s.writes.filter((w) => !w.body.action)).toHaveLength(1); + s.db.close(); +}); it('recreates on the current clippings blog when explicitly chosen and pins later updates there', async () => { const s = await setup(); await s.choose(b); await s.sync(); s.missing(); s.change(); await s.sync(); await s.choose(a); From 977bbe6914afb607b29be2870f07f751b83c22bc Mon Sep 17 00:00:00 2001 From: jcapayne <7683622+jcapayne@users.noreply.github.com> Date: Wed, 28 Oct 2026 21:13:53 -0400 Subject: [PATCH 20/37] Queue saved Micro.blog clippings after progress establishes a match --- src/connectors/runner.ts | 5 +++-- test/microblog-clippings-api.test.ts | 33 ++++++++++++++++++++++++++-- 2 files changed, 34 insertions(+), 4 deletions(-) diff --git a/src/connectors/runner.ts b/src/connectors/runner.ts index 5d4e47eb..f749bf96 100644 --- a/src/connectors/runner.ts +++ b/src/connectors/runner.ts @@ -172,8 +172,9 @@ export async function processRow(db: DB, row: QueueRow, http: HttpTransport): Pr // Persist (or clear) the per-book condition so the review UI can show it. setMatchNote(db, row.user_id, row.connector_id, row.document, result.note ?? null); markDone(db, row.id); - // A clipping may have arrived while the network request was in flight. - if (publication) scheduleMicroblogClippings(db, row.user_id, row.document); + // A progress push may have established the first match, or a clipping + // may have arrived while publication was in flight. + if (row.connector_id === 'microblog') scheduleMicroblogClippings(db, row.user_id, row.document); return; } if (result.needsReauth) { diff --git a/test/microblog-clippings-api.test.ts b/test/microblog-clippings-api.test.ts index c1318510..26d6a28b 100644 --- a/test/microblog-clippings-api.test.ts +++ b/test/microblog-clippings-api.test.ts @@ -8,15 +8,19 @@ import { deleteDocumentData } from '../src/models/document.js'; import { deleteKosyncUserData } from '../src/routes/account.js'; import { fanOutProgress } from '../src/connectors/fanout.js'; import { createProgressRefresh } from '../src/connectors/refresh.js'; +import { makeMicroblogTransport } from './microblog-helpers.js'; beforeEach(() => { process.env.TOKEN_ENC_KEY = 'a'.repeat(64); resetEncryptionKeyCache(); }); afterEach(() => { delete process.env.TOKEN_ENC_KEY; resetEncryptionKeyCache(); }); -async function setup() { +async function setup(shelfTransport?: HttpTransport) { const posts: unknown[] = []; const http: HttpTransport = async (url, init) => { if (new URL(url).searchParams.get('q') === 'config') return { status: 200, json: async () => ({ destination: [{ uid: 'https://example.micro.blog/' }] }), text: async () => '' }; if (new URL(url).pathname === '/micropub' && init.method === 'POST') posts.push(JSON.parse(init.body!)); - else if (!url.includes('/micropub?')) throw new Error(`Unexpected network request: ${url}`); + else if (!url.includes('/micropub?')) { + if (shelfTransport) return shelfTransport(url, init); + throw new Error(`Unexpected network request: ${url}`); + } return { status: 201, headers: { get: () => 'https://example.micro.blog/highlights.html' }, json: async () => ({}), text: async () => '' }; }; const { app, db } = makeTestApp({}, { connectorTransport: http }); @@ -103,6 +107,31 @@ it.each(['legacy', 'v1', 'clippings', 'sync-now'])('catches up selected clipping s.db.close(); }); +it.each([ + ['reader progress', 0.4, false], ['reader finished', 0.99, false], + ['Sync now progress', 0.4, true], ['Sync now finished', 0.99, true], +] as const)('queues saved clippings when %s first establishes the match', async (_label, percentage, syncNow) => { + const shelves = makeMicroblogTransport({ shelves: { 'to-read': [{ + id: '123', title: 'Test book', author: 'Author', isbn: '9781786487849', + }] } }); + const s = await setup(shelves.transport); + expect(await (await s.select()).json()).toMatchObject({ waiting_for_match: true }); + const progress = await s.put('/syncs/progress', { document: DOC, device_id: 'reader', progress: '/body/p', percentage }); + expect(progress.status).toBe(200); + if (syncNow) { + s.db.prepare('DELETE FROM connector_queue').run(); + expect((await s.app.request('/api/v1/connectors/microblog/sync', { method: 'POST', headers: s.headers })).status).toBe(200); + } + expect(s.db.prepare("SELECT COUNT(*) n FROM connector_queue WHERE kind='microblog:clippings'").get()).toEqual({ n: 0 }); + await drainQueue(s.db, s.http); + expect(s.db.prepare("SELECT external_id FROM connector_matches WHERE connector_id='microblog'").get()).toEqual({ external_id: '123' }); + expect(s.db.prepare("SELECT status FROM connector_queue WHERE kind='microblog:clippings'").get()).toEqual({ status: 'pending' }); + await drainQueue(s.db, s.http); + expect(s.posts).toHaveLength(1); + expect(JSON.stringify(s.posts[0])).toContain('A passage'); + s.db.close(); +}); + it('enabling automatic publication backfills existing highlights and waits for matching', async () => { const s = await setup(); const changed = await s.put('/api/v1/connectors/microblog/options', { sync_highlights: true }); From 126cbf2f81a1a312d8ace38400ed64b5193181d5 Mon Sep 17 00:00:00 2001 From: jcapayne <7683622+jcapayne@users.noreply.github.com> Date: Wed, 28 Oct 2026 21:14:29 -0400 Subject: [PATCH 21/37] Preserve path-based legacy Micro.blog destinations during token rotation --- src/connectors/store.ts | 6 +++++- test/microblog-destinations.test.ts | 24 +++++++++++++++++++++++- 2 files changed, 28 insertions(+), 2 deletions(-) diff --git a/src/connectors/store.ts b/src/connectors/store.ts index 20cbe474..3d891d8f 100644 --- a/src/connectors/store.ts +++ b/src/connectors/store.ts @@ -3,6 +3,7 @@ import { withTransaction, type DB } from '../db/db.js'; import { nowSeconds } from '../models/sync.js'; import { decryptSecret, encryptSecret } from '../crypto/secrets.js'; import { SAVE_CREDENTIAL, type Connector, type Credential, type DataKind, type DocumentMeta, type Match, type SavableCredential } from './types.js'; +import { destinationForPost } from './microblog-destinations.js'; export interface AccountRow { user_id: number; @@ -39,7 +40,10 @@ export function upsertAccount( const origins = new Set(posts.flatMap((r) => { try { return r.url ? [new URL(r.url).origin + '/'] : []; } catch { return []; } })); const oldDestination = old.microblogDestination ?? (origins.size === 1 ? [...origins][0] : null); const available = Array.isArray(cred.microblogDestinations) ? cred.microblogDestinations as { uid?: string }[] : []; - if (available.some((d) => d.uid === oldDestination || posts.some((p) => p.destination === d.uid))) changedMicroblogAccount = false; + const availableBlogs = available.flatMap((d) => typeof d?.uid === 'string' + ? [{ uid: d.uid, name: d.uid, isDefault: false }] : []); + const accessibleLegacyPost = posts.some((p) => !p.destination && p.url && destinationForPost(p.url, availableBlogs)); + if (accessibleLegacyPost || available.some((d) => d.uid === oldDestination || posts.some((p) => p.destination === d.uid))) changedMicroblogAccount = false; else if (oldDestination && cred.microblogDestination) changedMicroblogAccount = oldDestination !== cred.microblogDestination; else verifyMicroblogPosts = true; } diff --git a/test/microblog-destinations.test.ts b/test/microblog-destinations.test.ts index 2f908909..c030610e 100644 --- a/test/microblog-destinations.test.ts +++ b/test/microblog-destinations.test.ts @@ -1,8 +1,9 @@ import { afterEach, beforeEach, expect, it } from 'vitest'; import { makeTestApp, registerUser } from './helpers.js'; import { resetEncryptionKeyCache } from '../src/crypto/secrets.js'; -import { upsertAccount } from '../src/connectors/store.js'; +import { saveMatch, setConnectorOptions, upsertAccount } from '../src/connectors/store.js'; import { destinationForPost } from '../src/connectors/microblog-destinations.js'; +import { enqueue } from '../src/connectors/queue.js'; beforeEach(() => { process.env.TOKEN_ENC_KEY = 'a'.repeat(64); resetEncryptionKeyCache(); }); afterEach(() => { delete process.env.TOKEN_ENC_KEY; resetEncryptionKeyCache(); }); @@ -70,3 +71,24 @@ it.each([b, 'https://third.micro.blog/'])('never discards original collections w expect(s.db.prepare('SELECT identity,url,destination FROM microblog_clipping_posts').get()).toEqual({ identity: 'identity', url: `${b}post.html`, destination: b }); s.db.close(); }); + +it.each([ + ['https://shared.test/blog/post.html', true], + ['https://shared.test/blog/reading/post.html', true], + ['https://shared.test/blogging/post.html', false], + ['https://shared.test.evil/blog/post.html', false], +] as const)('preserves legacy settings on rotation only when %s belongs to the available path-based blog', async (url, sameBlog) => { + const blog = 'https://shared.test/blog/'; + const s = await setup([{ uid: blog }]); + s.db.prepare('INSERT INTO microblog_clipping_posts(user_id,document,identity,url) VALUES(?,?,?,?)').run(s.uid, 'book', 'identity', url); + setConnectorOptions(s.db, s.uid, 'microblog', { sync_highlights: true, highlight_blog: blog }); + saveMatch(s.db, s.uid, 'microblog', 'book', { externalId: '123', confidence: 1 }, 'manual'); + enqueue(s.db, s.uid, 'microblog', { kind: 'progress', document: 'book', percentage: 0.4, timestamp: 1 }); + upsertAccount(s.db, s.uid, 'microblog', { token: 'rotated', microblogDestination: blog, microblogDestinations: [{ uid: blog }] }, null); + const account = s.db.prepare("SELECT options FROM connector_accounts WHERE user_id=? AND connector_id='microblog'").get(s.uid) as { options: string }; + expect(JSON.parse(account.options)).toEqual(sameBlog ? { sync_highlights: true, highlight_blog: blog } : {}); + expect(s.db.prepare("SELECT COUNT(*) n FROM connector_matches WHERE user_id=? AND connector_id='microblog'").get(s.uid)).toEqual({ n: sameBlog ? 1 : 0 }); + expect(s.db.prepare("SELECT COUNT(*) n FROM connector_queue WHERE user_id=? AND connector_id='microblog'").get(s.uid)).toEqual({ n: sameBlog ? 1 : 0 }); + expect(s.db.prepare('SELECT identity,url FROM microblog_clipping_posts WHERE user_id=?').get(s.uid)).toEqual({ identity: 'identity', url }); + s.db.close(); +}); From df157d0183e36c9ee5acbc291cf51ca11ea40afc Mon Sep 17 00:00:00 2001 From: jcapayne <7683622+jcapayne@users.noreply.github.com> Date: Wed, 28 Oct 2026 21:20:38 -0400 Subject: [PATCH 22/37] Initialize connector registry lazily to avoid circular import failures --- src/connectors/registry.ts | 42 ++++++++++++++++++++++---------------- 1 file changed, 24 insertions(+), 18 deletions(-) diff --git a/src/connectors/registry.ts b/src/connectors/registry.ts index aee3227e..57e976ee 100644 --- a/src/connectors/registry.ts +++ b/src/connectors/registry.ts @@ -12,31 +12,37 @@ import { bookorbitConnector } from './bookorbit.js'; import { spotifyConnector } from './spotify.js'; import { seekquelConnector } from './seekquel.js'; -/** All connectors known to this build. */ -const CONNECTORS: Connector[] = [ - kosyncConnector, - hardcoverConnector, - readwiseConnector, - readwiseReaderConnector, - bookfusionConnector, - audiobookshelfConnector, - jellyfinConnector, - bookorbitConnector, - spotifyConnector, - seekquelConnector, - microblogConnector, - kindleConnector, -]; +let byId: Map | undefined; -const byId = new Map(CONNECTORS.map((c) => [c.id, c])); +/** Storage and connectors can import each other; resolve exports on first use. */ +function connectorRegistry(): Map { + if (!byId) { + const connectors: Connector[] = [ + kosyncConnector, + hardcoverConnector, + readwiseConnector, + readwiseReaderConnector, + bookfusionConnector, + audiobookshelfConnector, + jellyfinConnector, + bookorbitConnector, + spotifyConnector, + seekquelConnector, + microblogConnector, + kindleConnector, + ]; + byId = new Map(connectors.map((c) => [c.id, c])); + } + return byId; +} /** Connectors shown in the UI/API (excludes hidden ones). */ export function listConnectors(): Connector[] { - return CONNECTORS.filter((c) => !c.hidden); + return [...connectorRegistry().values()].filter((c) => !c.hidden); } export function getConnector(id: string): Connector | undefined { - return byId.get(id); + return connectorRegistry().get(id); } /** Default transport: the platform fetch, adapted to HttpTransport. */ From 2de8618eb60443da405072d9bad0cc358052298d Mon Sep 17 00:00:00 2001 From: jcapayne <7683622+jcapayne@users.noreply.github.com> Date: Wed, 28 Oct 2026 21:31:57 -0400 Subject: [PATCH 23/37] Preserve selected Micro.blog destination across token rotations --- src/connectors/store.ts | 5 +++-- test/microblog-destinations.test.ts | 20 ++++++++++++++++++-- 2 files changed, 21 insertions(+), 4 deletions(-) diff --git a/src/connectors/store.ts b/src/connectors/store.ts index 3d891d8f..92747fa4 100644 --- a/src/connectors/store.ts +++ b/src/connectors/store.ts @@ -3,7 +3,7 @@ import { withTransaction, type DB } from '../db/db.js'; import { nowSeconds } from '../models/sync.js'; import { decryptSecret, encryptSecret } from '../crypto/secrets.js'; import { SAVE_CREDENTIAL, type Connector, type Credential, type DataKind, type DocumentMeta, type Match, type SavableCredential } from './types.js'; -import { destinationForPost } from './microblog-destinations.js'; +import { destinationForPost, selectedMicroblogDestination } from './microblog-destinations.js'; export interface AccountRow { user_id: number; @@ -43,7 +43,8 @@ export function upsertAccount( const availableBlogs = available.flatMap((d) => typeof d?.uid === 'string' ? [{ uid: d.uid, name: d.uid, isDefault: false }] : []); const accessibleLegacyPost = posts.some((p) => !p.destination && p.url && destinationForPost(p.url, availableBlogs)); - if (accessibleLegacyPost || available.some((d) => d.uid === oldDestination || posts.some((p) => p.destination === d.uid))) changedMicroblogAccount = false; + const selectedBlog = selectedMicroblogDestination(db, userId); + if (accessibleLegacyPost || available.some((d) => d.uid === oldDestination || d.uid === selectedBlog || posts.some((p) => p.destination === d.uid))) changedMicroblogAccount = false; else if (oldDestination && cred.microblogDestination) changedMicroblogAccount = oldDestination !== cred.microblogDestination; else verifyMicroblogPosts = true; } diff --git a/test/microblog-destinations.test.ts b/test/microblog-destinations.test.ts index c030610e..eec8bf69 100644 --- a/test/microblog-destinations.test.ts +++ b/test/microblog-destinations.test.ts @@ -72,6 +72,21 @@ it.each([b, 'https://third.micro.blog/'])('never discards original collections w s.db.close(); }); +it.each([b, 'https://third.micro.blog/'])('preserves an unpublished selected blog on rotation only when it remains available (%s)', async (accessible) => { + const s = await setup([{ uid: a }, { uid: b }]); + upsertAccount(s.db, s.uid, 'microblog', { token: 'old', microblogDestination: a }, null); + setConnectorOptions(s.db, s.uid, 'microblog', { sync_highlights: true, highlight_blog: b }); + saveMatch(s.db, s.uid, 'microblog', 'book', { externalId: '123', confidence: 1 }, 'manual'); + enqueue(s.db, s.uid, 'microblog', { kind: 'progress', document: 'book', percentage: 0.4, timestamp: 1 }); + upsertAccount(s.db, s.uid, 'microblog', { token: 'rotated', microblogDestination: accessible, microblogDestinations: [{ uid: accessible }] }, null); + const account = s.db.prepare("SELECT options FROM connector_accounts WHERE user_id=? AND connector_id='microblog'").get(s.uid) as { options: string }; + expect(JSON.parse(account.options)).toEqual(accessible === b ? { sync_highlights: true, highlight_blog: b } : {}); + expect(s.db.prepare("SELECT COUNT(*) n FROM connector_matches WHERE user_id=? AND connector_id='microblog'").get(s.uid)).toEqual({ n: accessible === b ? 1 : 0 }); + expect(s.db.prepare("SELECT COUNT(*) n FROM connector_queue WHERE user_id=? AND connector_id='microblog'").get(s.uid)).toEqual({ n: accessible === b ? 1 : 0 }); + expect(s.db.prepare('SELECT COUNT(*) n FROM microblog_clipping_posts WHERE user_id=?').get(s.uid)).toEqual({ n: 0 }); + s.db.close(); +}); + it.each([ ['https://shared.test/blog/post.html', true], ['https://shared.test/blog/reading/post.html', true], @@ -81,12 +96,13 @@ it.each([ const blog = 'https://shared.test/blog/'; const s = await setup([{ uid: blog }]); s.db.prepare('INSERT INTO microblog_clipping_posts(user_id,document,identity,url) VALUES(?,?,?,?)').run(s.uid, 'book', 'identity', url); - setConnectorOptions(s.db, s.uid, 'microblog', { sync_highlights: true, highlight_blog: blog }); + // Isolate legacy URL evidence from an independently saved blog selection. + setConnectorOptions(s.db, s.uid, 'microblog', { sync_highlights: true }); saveMatch(s.db, s.uid, 'microblog', 'book', { externalId: '123', confidence: 1 }, 'manual'); enqueue(s.db, s.uid, 'microblog', { kind: 'progress', document: 'book', percentage: 0.4, timestamp: 1 }); upsertAccount(s.db, s.uid, 'microblog', { token: 'rotated', microblogDestination: blog, microblogDestinations: [{ uid: blog }] }, null); const account = s.db.prepare("SELECT options FROM connector_accounts WHERE user_id=? AND connector_id='microblog'").get(s.uid) as { options: string }; - expect(JSON.parse(account.options)).toEqual(sameBlog ? { sync_highlights: true, highlight_blog: blog } : {}); + expect(JSON.parse(account.options)).toEqual(sameBlog ? { sync_highlights: true } : {}); expect(s.db.prepare("SELECT COUNT(*) n FROM connector_matches WHERE user_id=? AND connector_id='microblog'").get(s.uid)).toEqual({ n: sameBlog ? 1 : 0 }); expect(s.db.prepare("SELECT COUNT(*) n FROM connector_queue WHERE user_id=? AND connector_id='microblog'").get(s.uid)).toEqual({ n: sameBlog ? 1 : 0 }); expect(s.db.prepare('SELECT identity,url FROM microblog_clipping_posts WHERE user_id=?').get(s.uid)).toEqual({ identity: 'identity', url }); From 8a882dd06c93a9e0a3d7adacf4af31304e7317a7 Mon Sep 17 00:00:00 2001 From: jcapayne <7683622+jcapayne@users.noreply.github.com> Date: Wed, 28 Oct 2026 21:32:39 -0400 Subject: [PATCH 24/37] Isolate follow-up clipping scheduling from successful Micro.blog pushes --- src/connectors/runner.ts | 9 ++++++++- test/microblog-clippings-api.test.ts | 22 ++++++++++++++++++++++ 2 files changed, 30 insertions(+), 1 deletion(-) diff --git a/src/connectors/runner.ts b/src/connectors/runner.ts index f749bf96..2b6c6277 100644 --- a/src/connectors/runner.ts +++ b/src/connectors/runner.ts @@ -174,7 +174,14 @@ export async function processRow(db: DB, row: QueueRow, http: HttpTransport): Pr markDone(db, row.id); // A progress push may have established the first match, or a clipping // may have arrived while publication was in flight. - if (row.connector_id === 'microblog') scheduleMicroblogClippings(db, row.user_id, row.document); + if (row.connector_id === 'microblog') { + try { + scheduleMicroblogClippings(db, row.user_id, row.document); + } catch (err) { + console.error(JSON.stringify({ msg: 'microblog schedule failed', user_id: row.user_id, document: row.document, + error: err instanceof Error ? err.message : String(err) })); + } + } return; } if (result.needsReauth) { diff --git a/test/microblog-clippings-api.test.ts b/test/microblog-clippings-api.test.ts index 26d6a28b..e3b1f2aa 100644 --- a/test/microblog-clippings-api.test.ts +++ b/test/microblog-clippings-api.test.ts @@ -132,6 +132,28 @@ it.each([ s.db.close(); }); +it.each([0.4, 0.99])('keeps a successful shelf push done when follow-up clipping scheduling fails (%s)', async (percentage) => { + const shelves = makeMicroblogTransport({ shelves: { 'to-read': [{ + id: '123', title: 'Test book', author: 'Author', isbn: '9781786487849', + }] } }); + const s = await setup(shelves.transport); + expect((await s.put('/syncs/progress', { document: DOC, device_id: 'reader', progress: '/body/p', percentage })).status).toBe(200); + s.db.prepare('UPDATE connector_accounts SET options=? WHERE user_id=? AND connector_id=?') + .run(JSON.stringify({ sync_highlights: true }), s.uid, 'microblog'); + s.db.exec(`CREATE TEMP TRIGGER fail_post_push_schedule BEFORE INSERT ON microblog_clipping_selections + BEGIN SELECT RAISE(FAIL, 'simulated post-push scheduling failure'); END`); + await drainQueue(s.db, s.http); + expect(shelves.shelves.get(percentage < 0.98 ? 'reading' : 'finished')?.map((book) => book.id)).toEqual(['123']); + expect(s.db.prepare("SELECT status,attempts,last_error FROM connector_queue WHERE kind=?").get(percentage < 0.98 ? 'progress' : 'finished')) + .toEqual({ status: 'done', attempts: 0, last_error: null }); + s.db.exec('DROP TRIGGER fail_post_push_schedule'); + expect((await s.app.request(`/api/v1/clippings/${DOC}`, { headers: s.headers })).status).toBe(200); + await drainQueue(s.db, s.http); + expect(s.posts).toHaveLength(1); + expect(shelves.calls.filter((c) => c.method === 'POST' && c.url.endsWith('/assign'))).toHaveLength(1); + s.db.close(); +}); + it('enabling automatic publication backfills existing highlights and waits for matching', async () => { const s = await setup(); const changed = await s.put('/api/v1/connectors/microblog/options', { sync_highlights: true }); From 4aaf763a92a02013932da9cad0fcf654694b7ba0 Mon Sep 17 00:00:00 2001 From: jcapayne <7683622+jcapayne@users.noreply.github.com> Date: Wed, 28 Oct 2026 21:57:37 -0400 Subject: [PATCH 25/37] Limit per-book Micro.blog backfill to the requested book --- src/connectors/fanout.ts | 2 +- test/microblog-clippings-api.test.ts | 25 +++++++++++++++++++++++++ 2 files changed, 26 insertions(+), 1 deletion(-) diff --git a/src/connectors/fanout.ts b/src/connectors/fanout.ts index d7a4ddad..47451dbd 100644 --- a/src/connectors/fanout.ts +++ b/src/connectors/fanout.ts @@ -199,7 +199,7 @@ export function backfillConnector(db: DB, userId: number, connectorId: string, d } } - if (connectorId === 'microblog') queued += scheduleMicroblogClippings(db, userId); + if (connectorId === 'microblog') queued += scheduleMicroblogClippings(db, userId, document); if (connectorId !== 'microblog' && carriesFor(db, userId, conn, 'highlight')) { const rows = db .prepare( diff --git a/test/microblog-clippings-api.test.ts b/test/microblog-clippings-api.test.ts index e3b1f2aa..5d09f1ba 100644 --- a/test/microblog-clippings-api.test.ts +++ b/test/microblog-clippings-api.test.ts @@ -215,3 +215,28 @@ it('coalesces repeated selections and clears local publication state on unlink', expect(s.posts).toHaveLength(1); s.db.close(); }); + +it('limits per-book sync to that book while account-wide sync still queues other selected books', async () => { + const s = await setup(); + const other = 'b'.repeat(32); + await s.put('/api/v1/documents', { items: [{ document: other, title: 'Other book', author: 'Author' }] }); + await s.put(`/api/v1/clippings/${other}`, { items: [{ id: 'other', spine: 1, text: 'Other passage', created_at: 1 }] }); + await s.match(); + await s.put(`/api/v1/connectors/microblog/matches/${other}`, { external_id: '456', external_edition: '9780547951973' }); + await s.select(); + await s.app.request(`/api/v1/connectors/microblog/clippings/${other}/other`, { method: 'POST', headers: s.headers }); + s.db.prepare('DELETE FROM connector_queue').run(); + const res = await s.app.request(`/api/v1/connectors/microblog/sync/${DOC}`, { method: 'POST', headers: s.headers }); + expect(res.status).toBe(200); + expect(await res.json()).toMatchObject({ queued: 1 }); + expect(s.db.prepare("SELECT document FROM connector_queue WHERE kind='microblog:clippings'").all()).toEqual([{ document: DOC }]); + await drainQueue(s.db, s.http); + expect(s.posts).toHaveLength(1); + expect(JSON.stringify(s.posts[0])).not.toContain('Other passage'); + const all = await s.app.request('/api/v1/connectors/microblog/sync', { method: 'POST', headers: s.headers }); + expect(await all.json()).toMatchObject({ queued: 1 }); + await drainQueue(s.db, s.http); + expect(s.posts).toHaveLength(2); + expect(JSON.stringify(s.posts[1])).toContain('Other passage'); + s.db.close(); +}); From 3ebab7b26b46f6b95447ab80e49103d264c64429 Mon Sep 17 00:00:00 2001 From: jcapayne <7683622+jcapayne@users.noreply.github.com> Date: Wed, 28 Oct 2026 21:59:03 -0400 Subject: [PATCH 26/37] Allow corrected Micro.blog updates to retry the existing post --- src/connectors/microblog-clippings.ts | 13 ++++++++++-- test/microblog-destination-posts.test.ts | 26 ++++++++++++++++++++++++ 2 files changed, 37 insertions(+), 2 deletions(-) diff --git a/src/connectors/microblog-clippings.ts b/src/connectors/microblog-clippings.ts index b9345bcd..b17bd3ff 100644 --- a/src/connectors/microblog-clippings.ts +++ b/src/connectors/microblog-clippings.ts @@ -105,7 +105,14 @@ export function scheduleMicroblogClippings(db: DB, userId: number, document?: st const isbn = match.external_edition ?? (/^(?:\d{13}|\d{9}[\dXx])$/.test(match.external_id) ? match.external_id : null); const bookUrl = isbn ? `https://micro.blog/books/${encodeURIComponent(isbn)}` : state.book_url; if (state.url && bookUrl && state.matched_book === match.external_id - && state.fingerprint === microblogClippingsContent(db, userId, doc, bookUrl)?.fingerprint) continue; + && state.fingerprint === microblogClippingsContent(db, userId, doc, bookUrl)?.fingerprint) { + // Removing rejected unpublished content can restore the last successful + // post without another network update. Its old validation error is stale. + db.prepare(`UPDATE connector_queue SET status='done',last_error=NULL,updated_at=? + WHERE user_id=? AND connector_id='microblog' AND document=? AND kind='microblog:clippings' AND status='dead'`) + .run(nowSeconds(), userId, doc); + continue; + } const existing = db.prepare("SELECT status FROM connector_queue WHERE user_id=? AND connector_id='microblog' AND document=? AND kind='microblog:clippings'") .get(userId, doc) as { status: string } | undefined; if (existing?.status === 'pending') continue; // Preserve backoff after a transient failure. @@ -265,7 +272,9 @@ export async function publishMicroblogClippings( db.prepare('UPDATE microblog_clipping_posts SET missing=1 WHERE user_id=? AND document=?').run(userId, document); return permanent(err.message); } - if (err instanceof ConnectorOperationError && !err.retryable && !err.needsReauth) return permanent(`${err.message}. Review the saved Micro.blog post before syncing again.`); + if (err instanceof ConnectorOperationError && !err.retryable && !err.needsReauth) return { + ok: false, retryable: false, error: `${err.message}. Correct the clippings and sync this book again.`, + }; throw err; } } else { diff --git a/test/microblog-destination-posts.test.ts b/test/microblog-destination-posts.test.ts index bb1be6f3..7b65c090 100644 --- a/test/microblog-destination-posts.test.ts +++ b/test/microblog-destination-posts.test.ts @@ -123,6 +123,32 @@ it.each([400, 401, 403, 429, 500])('does not mark an unrelated update failure as expect(s.writes.filter((w) => !w.body.action)).toHaveLength(1); s.db.close(); }); +it.each([400, 413, 422])('can retry the existing post after correcting a rejected update (%s)', async (code) => { + const s = await setup(); await s.choose(b); await s.sync(); + s.updateFailure(code, 'Content is too large'); s.change(); await s.sync(); + expect(await s.status()).toMatchObject({ items: [{ missing: false, url: `${b}post.html`, error: null }] }); + expect(s.db.prepare("SELECT status FROM connector_queue WHERE kind='microblog:clippings'").get()).toEqual({ status: 'dead' }); + s.db.prepare('UPDATE clippings SET note=? WHERE user_id=?').run('Short corrected note', s.uid); + s.updateFailure(200); await s.sync(); + expect(s.writes).toHaveLength(3); + expect(s.writes[2]).toMatchObject({ destination: b, body: { action: 'update', url: `${b}post.html` } }); + expect(JSON.stringify(s.writes[2])).toContain('Short corrected note'); + expect(s.writes.filter((w) => !w.body.action)).toHaveLength(1); + expect(s.db.prepare("SELECT status FROM connector_queue WHERE kind='microblog:clippings'").get()).toEqual({ status: 'done' }); + await s.sync(); expect(s.writes).toHaveLength(3); + s.db.close(); +}); +it('clears a rejected update when removing an unpublished clipping restores the published content', async () => { + const s = await setup(); await s.choose(b); await s.sync(); + s.db.prepare('INSERT INTO clippings(user_id,document,id,text,spine_index,created_at,updated_at) VALUES(?,?,?,?,1,2,2)').run(s.uid, DOC, 'two', 'Rejected passage'); + selectMicroblogClipping(s.db, s.uid, DOC, 'two'); + s.updateFailure(413); await s.sync(); + s.db.prepare('UPDATE clippings SET deleted=1 WHERE user_id=? AND id=?').run(s.uid, 'two'); + await s.sync(); + expect(s.writes).toHaveLength(2); // The remote content is already correct. + expect(s.db.prepare("SELECT status,last_error FROM connector_queue WHERE kind='microblog:clippings'").get()).toEqual({ status: 'done', last_error: null }); + s.db.close(); +}); it('recreates on the current clippings blog when explicitly chosen and pins later updates there', async () => { const s = await setup(); await s.choose(b); await s.sync(); s.missing(); s.change(); await s.sync(); await s.choose(a); From eeece6b0523225c01b6ee901ba3bc9c27687d1dd Mon Sep 17 00:00:00 2001 From: jcapayne <7683622+jcapayne@users.noreply.github.com> Date: Wed, 28 Oct 2026 22:00:27 -0400 Subject: [PATCH 27/37] Preserve clipping reading order using positional fallbacks --- src/connectors/microblog-clippings.ts | 24 +++++++++++++++---- test/microblog-clippings-post.test.ts | 26 +++++++++++++++++++++ test/microblog-clippings.test.ts | 33 +++++++++++++++++++++++++++ 3 files changed, 79 insertions(+), 4 deletions(-) diff --git a/src/connectors/microblog-clippings.ts b/src/connectors/microblog-clippings.ts index b17bd3ff..5a1eb6f0 100644 --- a/src/connectors/microblog-clippings.ts +++ b/src/connectors/microblog-clippings.ts @@ -21,6 +21,8 @@ export interface ClippingsPost { interface Passage { id: string; text: string; note: string | null; chapter_title: string; spine_index: number; start_offset: number | null; created_at: number; + // Older published snapshots predate these fallback coordinates. + paragraph_index?: number | null; start_page?: number; start_word?: number; } function automatic(db: DB, userId: number): boolean { @@ -31,7 +33,7 @@ function automatic(db: DB, userId: number): boolean { function liveClippings(db: DB, userId: number, document: string): Passage[] { const docs = clippingDocuments(db, userId, document); - return db.prepare(`${mergedClippingsCte(docs.length)} SELECT id,text,note,chapter_title,spine_index,start_offset,created_at + return db.prepare(`${mergedClippingsCte(docs.length)} SELECT id,text,note,chapter_title,spine_index,start_offset,paragraph_index,start_page,start_word,created_at FROM merged WHERE document=? AND deleted=0`).all(userId, ...docs, document) as unknown as Passage[]; } @@ -55,12 +57,26 @@ function passages(db: DB, userId: number, document: string): Passage[] { const enabled = automatic(db, userId); const selections = db.prepare('SELECT clipping_id,manual,published_snapshot FROM microblog_clipping_selections WHERE user_id=? AND document=?') .all(userId, document) as { clipping_id: string; manual: number; published_snapshot: string | null }[]; - return selections.flatMap((s) => { + const selected = selections.flatMap((s) => { const current = live.find((c) => c.id === s.clipping_id); if ((enabled || s.manual) && current) return [current]; return s.published_snapshot ? [JSON.parse(s.published_snapshot) as Passage] : []; - }).sort((a, b) => a.spine_index - b.spine_index || (a.start_offset ?? 0) - (b.start_offset ?? 0) - || a.created_at - b.created_at || a.id.localeCompare(b.id)); + }); + const coordinates = ['start_offset', 'paragraph_index', 'start_page', 'start_word'] as const; + const common = new Map(); + // Use the same coordinate system throughout a spine. Mixed client versions + // must not compare a character offset to a paragraph or page number. + for (const c of selected) { + common.set(c.spine_index, (common.get(c.spine_index) ?? coordinates).filter((key) => c[key] != null)); + } + return selected.sort((a, b) => { + if (a.spine_index !== b.spine_index) return a.spine_index - b.spine_index; + for (const key of common.get(a.spine_index)!) { + const difference = a[key]! - b[key]!; + if (difference) return difference; + } + return a.created_at - b.created_at || a.id.localeCompare(b.id); + }); } function escapeHtml(s: string): string { diff --git a/test/microblog-clippings-post.test.ts b/test/microblog-clippings-post.test.ts index b92970a2..ee1b97ee 100644 --- a/test/microblog-clippings-post.test.ts +++ b/test/microblog-clippings-post.test.ts @@ -76,6 +76,32 @@ it('publishes one escaped collected post, skips unchanged content, then updates db.close(); }); +it('retains fallback positions in published snapshots and preserves reading order after local deletion', async () => { + const { db, uid } = await setup(); + db.prepare('UPDATE clippings SET paragraph_index=1,start_page=2,start_word=3,created_at=2 WHERE user_id=?').run(uid); + db.prepare('INSERT INTO clippings(user_id,document,id,text,spine_index,paragraph_index,start_page,start_word,created_at,updated_at) VALUES(?,?,?,?,0,2,4,10,1,1)') + .run(uid, DOC, 'two', 'Second passage'); + selectMicroblogClipping(db, uid, DOC, 'two'); + const writes: unknown[] = []; + const http: HttpTransport = async (_url, init) => { + if (init.method === 'GET') return response(200); + writes.push(JSON.parse(init.body!)); + return response(201, {}, remoteUrl); + }; + await drainQueue(db, http); + const snapshot = db.prepare('SELECT published_snapshot FROM microblog_clipping_selections WHERE user_id=? AND clipping_id=?').get(uid, 'one') as { published_snapshot: string }; + expect(JSON.parse(snapshot.published_snapshot)).toMatchObject({ paragraph_index: 1, start_page: 2, start_word: 3 }); + db.prepare('UPDATE clippings SET deleted=1 WHERE user_id=? AND id=?').run(uid, 'one'); + db.prepare('UPDATE clippings SET note=? WHERE user_id=? AND id=?').run('New note', uid, 'two'); + scheduleMicroblogClippings(db, uid, DOC); await drainQueue(db, http); + expect(writes).toHaveLength(2); + for (const write of writes) { + const content = JSON.stringify(write); + expect(content.indexOf('First passage')).toBeLessThan(content.indexOf('Second passage')); + } + db.close(); +}); + it.each(['source lookup', 'update'])('identifies a failed %s and preserves the existing post for retry', async (operation) => { const { db, uid } = await setup(); const http: HttpTransport = async (_url, init) => { diff --git a/test/microblog-clippings.test.ts b/test/microblog-clippings.test.ts index 04aeef93..8016bf4b 100644 --- a/test/microblog-clippings.test.ts +++ b/test/microblog-clippings.test.ts @@ -46,6 +46,39 @@ it('automatic mode includes existing clippings in reading order and isolates use db.close(); }); +it.each([ + ['text offsets', { start_offset: 10 }, { start_offset: 30 }], + ['paragraphs', { paragraph_index: 1 }, { paragraph_index: 2 }], + ['zero-based paragraphs', { paragraph_index: 0 }, { paragraph_index: 1 }], + ['pages', { start_page: 2 }, { start_page: 4 }], + ['words within a page', { start_page: 2, start_word: 3 }, { start_page: 2, start_word: 10 }], + ['mixed offset availability', { start_offset: 100, paragraph_index: 1 }, { paragraph_index: 2 }], +] as const)('uses %s before creation time when ordering clippings in the same spine', async (_label, first, second) => { + const { db, uid } = await setup(); + const update = db.prepare('UPDATE clippings SET spine_index=0,start_offset=?,paragraph_index=?,start_page=?,start_word=?,created_at=? WHERE user_id=? AND id=?'); + for (const [id, position, createdAt] of [['one', first, 2], ['two', second, 1]] as const) { + const p = { start_offset: null, paragraph_index: null, start_page: 0, start_word: 0, ...position }; + update.run(p.start_offset, p.paragraph_index, p.start_page, p.start_word, createdAt, uid, id); + selectMicroblogClipping(db, uid, DOC, id); + } + const content = microblogClippingsContent(db, uid, DOC, 'https://micro.blog/books/9781786487849')!.content; + expect(content.indexOf('First passage')).toBeLessThan(content.indexOf('Second passage')); + db.close(); +}); + +it('keeps legacy published snapshots without fallback positions readable', async () => { + const { db, uid } = await setup(); + for (const [id, text, created_at] of [['one', 'First passage', 1], ['two', 'Second passage', 2]] as const) { + selectMicroblogClipping(db, uid, DOC, id); + const snapshot = { id, text, note: null, chapter_title: '', spine_index: 0, start_offset: null, created_at }; + db.prepare('UPDATE microblog_clipping_selections SET manual=0,published_snapshot=? WHERE user_id=? AND clipping_id=?') + .run(JSON.stringify(snapshot), uid, id); + } + const content = microblogClippingsContent(db, uid, DOC, 'https://micro.blog/books/9781786487849')!.content; + expect(content.indexOf('First passage')).toBeLessThan(content.indexOf('Second passage')); + db.close(); +}); + it('automatic-off excludes unpublished automatic selections but preserves manual intent', async () => { const { db, uid } = await setup(); db.prepare("UPDATE connector_accounts SET options=? WHERE user_id=?").run('{"sync_highlights":true}', uid); From 527151d74f53f5de1da54554d978ff06b96f84a7 Mon Sep 17 00:00:00 2001 From: jcapayne <7683622+jcapayne@users.noreply.github.com> Date: Wed, 28 Oct 2026 22:20:06 -0400 Subject: [PATCH 28/37] Allow corrected initial Micro.blog posts after validation rejection --- src/connectors/microblog-clippings.ts | 16 +++++++++++++--- test/microblog-clippings-post.test.ts | 26 ++++++++++++++++++++++++++ 2 files changed, 39 insertions(+), 3 deletions(-) diff --git a/src/connectors/microblog-clippings.ts b/src/connectors/microblog-clippings.ts index 5a1eb6f0..7d48c5b8 100644 --- a/src/connectors/microblog-clippings.ts +++ b/src/connectors/microblog-clippings.ts @@ -164,6 +164,13 @@ class MissingMicroblogPost extends ConnectorOperationError { constructor() { super('The Micro.blog clippings post was deleted. Use Recreate clippings post to resume publishing.', false); } } +class MicropubResponseError extends ConnectorOperationError { + constructor(readonly status: number, operation: string) { + super(`Micro.blog Micropub ${operation} failed (${status})`, status === 429 || status >= 500, + status === 401 || status === 403); + } +} + async function micropub(http: HttpTransport, token: string, destination: string, path: string, body?: unknown, sourceLookup = false) { const operation = body ? (body as { action?: string }).action === 'update' ? 'update' : 'create' : sourceLookup ? 'source lookup' : 'post recovery'; @@ -184,8 +191,7 @@ async function micropub(http: HttpTransport, token: string, destination: string, if (failure?.error === 'invalid_request' && /no post|not found|does not exist|could not find|unknown.*url/i.test(failure.error_description || '')) throw new MissingMicroblogPost(); } if (res.status >= 300) { - throw new ConnectorOperationError(`Micro.blog Micropub ${operation} failed (${res.status})`, res.status === 429 || res.status >= 500, - res.status === 401 || res.status === 403); + throw new MicropubResponseError(res.status, operation); } return res; } @@ -303,8 +309,12 @@ export async function publishMicroblogClippings( } }); } catch (err) { assertCurrent(); - if (err instanceof ConnectorOperationError && (err.needsReauth || err.message.includes('(429)'))) { + // Only known request rejections prove that no post was created. + if (err instanceof MicropubResponseError && [400, 401, 403, 413, 415, 422, 429].includes(err.status)) { db.prepare('UPDATE microblog_clipping_posts SET attempted=0 WHERE user_id=? AND document=?').run(userId, document); + if (!err.retryable && !err.needsReauth) return { + ok: false, retryable: false, error: `${err.message}. Correct the clippings and sync this book again.`, + }; } else if (err instanceof ConnectorOperationError && !err.retryable) return permanent(err.message); throw err; } diff --git a/test/microblog-clippings-post.test.ts b/test/microblog-clippings-post.test.ts index ee1b97ee..041e4182 100644 --- a/test/microblog-clippings-post.test.ts +++ b/test/microblog-clippings-post.test.ts @@ -150,6 +150,32 @@ it.each([401, 403, 429, 500])('handles HTTP %s without marking an upload success db.close(); }); +it.each([400, 413, 422])('allows a corrected initial post after a definite validation rejection (%s)', async (status) => { + const { db, uid } = await setup(); + const creates: any[] = []; + let rejected = true; + const http: HttpTransport = async (_url, init) => { + if (init.method === 'GET') throw new Error('A rejected create must not need uncertain-creation recovery'); + const body = JSON.parse(init.body!); + expect(body.action).toBeUndefined(); + creates.push(body); + return rejected ? response(status, { error: 'invalid_request', error_description: 'Content is too large' }) + : response(201, {}, remoteUrl); + }; + await drainQueue(db, http); + expect(db.prepare('SELECT attempted,error,url FROM microblog_clipping_posts').get()).toEqual({ attempted: 0, error: null, url: null }); + expect(db.prepare('SELECT status FROM connector_queue').get()).toEqual({ status: 'dead' }); + db.prepare('UPDATE clippings SET text=? WHERE user_id=?').run('Short corrected passage', uid); + rejected = false; + expect(scheduleMicroblogClippings(db, uid, DOC)).toBe(1); + await drainQueue(db, http); + expect(creates).toHaveLength(2); + expect(JSON.stringify(creates[1])).toContain('Short corrected passage'); + expect(db.prepare('SELECT url,error FROM microblog_clipping_posts').get()).toEqual({ url: remoteUrl, error: null }); + expect(db.prepare('SELECT status FROM connector_queue').get()).toEqual({ status: 'done' }); + db.close(); +}); + it('does not recreate a missing published post', async () => { const { db } = await setup(); db.prepare('UPDATE microblog_clipping_posts SET url=?,attempted=1').run(remoteUrl); From 8a0ee920999838d545d6b2a84682e150ae5a65f7 Mon Sep 17 00:00:00 2001 From: jcapayne <7683622+jcapayne@users.noreply.github.com> Date: Wed, 28 Oct 2026 22:21:38 -0400 Subject: [PATCH 29/37] Schedule Micro.blog clippings after completing a document merge --- src/models/merge.ts | 10 +++++++- test/microblog-clippings-post.test.ts | 34 +++++++++++++++++++++++++++ 2 files changed, 43 insertions(+), 1 deletion(-) diff --git a/src/models/merge.ts b/src/models/merge.ts index ac109870..b6ea0044 100644 --- a/src/models/merge.ts +++ b/src/models/merge.ts @@ -2,7 +2,7 @@ import { withTransaction, type DB } from '../db/db.js'; import { nextChangeSeq } from './changes.js'; import { md5Hex } from '../auth/password.js'; import { autoUnpause } from './pause.js'; -import { mergeMicroblogClippings } from '../connectors/microblog-clippings.js'; +import { mergeMicroblogClippings, scheduleMicroblogClippings } from '../connectors/microblog-clippings.js'; import { MANUAL_COLUMNS, mergeManualSql } from './manual-details.js'; /** @@ -195,6 +195,14 @@ export function mergeDocuments( 'INSERT OR REPLACE INTO document_aliases (user_id, alias, document, created_at, merge_stats) VALUES (?, ?, ?, ?, ?)' ).run(userId, from, into, now, mergeStats ? 1 : 0); }); + // Selections from an unmatched duplicate may now belong to a matched book. + // Schedule only after all clippings, matches, and aliases have been settled. + try { + scheduleMicroblogClippings(db, userId, into); + } catch (err) { + console.error(JSON.stringify({ msg: 'microblog schedule failed', user_id: userId, document: into, + error: err instanceof Error ? err.message : String(err) })); + } } /** diff --git a/test/microblog-clippings-post.test.ts b/test/microblog-clippings-post.test.ts index 041e4182..b1266fc0 100644 --- a/test/microblog-clippings-post.test.ts +++ b/test/microblog-clippings-post.test.ts @@ -5,6 +5,7 @@ import { upsertAccount, saveMatch } from '../src/connectors/store.js'; import { publishMicroblogClippings, selectMicroblogClipping, scheduleMicroblogClippings } from '../src/connectors/microblog-clippings.js'; import { drainQueue as runQueue } from '../src/connectors/runner.js'; import type { HttpTransport } from '../src/connectors/types.js'; +import { mergeDocuments } from '../src/models/merge.js'; beforeEach(() => { process.env.TOKEN_ENC_KEY = 'a'.repeat(64); resetEncryptionKeyCache(); }); afterEach(() => { delete process.env.TOKEN_ENC_KEY; resetEncryptionKeyCache(); }); @@ -102,6 +103,39 @@ it('retains fallback positions in published snapshots and preserves reading orde db.close(); }); +it.each([false, true])('publishes selected clippings after merging an unmatched duplicate (scheduling failure: %s)', async (failScheduling) => { + const { db, uid } = await setup(); + const writes: any[] = []; + const http: HttpTransport = async (_url, init) => { + if (init.method === 'GET') return response(200); + const body = JSON.parse(init.body!); writes.push(body); + return response(body.action ? 200 : 201, {}, remoteUrl); + }; + await drainQueue(db, http); + const source = 'b'.repeat(32); + db.prepare('INSERT INTO documents(user_id,document,title,updated_at) VALUES(?,?,?,0)').run(uid, source, 'Duplicate'); + db.prepare('INSERT INTO clippings(user_id,document,id,text,spine_index,created_at,updated_at) VALUES(?,?,?,?,1,2,2)').run(uid, source, 'merged', 'Merged passage'); + selectMicroblogClipping(db, uid, source, 'merged'); + expect(scheduleMicroblogClippings(db, uid, source)).toBe(0); + if (failScheduling) db.exec(`CREATE TEMP TRIGGER fail_merge_schedule BEFORE INSERT ON connector_queue + WHEN NEW.connector_id='microblog' BEGIN SELECT RAISE(FAIL, 'simulated merge scheduling failure'); END`); + expect(() => mergeDocuments(db, uid, source, DOC, 10)).not.toThrow(); + expect(db.prepare('SELECT document FROM document_aliases WHERE user_id=? AND alias=?').get(uid, source)).toEqual({ document: DOC }); + expect(db.prepare('SELECT document FROM microblog_clipping_selections WHERE user_id=? AND clipping_id=?').get(uid, 'merged')).toEqual({ document: DOC }); + if (failScheduling) { + db.exec('DROP TRIGGER fail_merge_schedule'); + expect(scheduleMicroblogClippings(db, uid, DOC)).toBe(1); + } + expect(db.prepare("SELECT status FROM connector_queue WHERE kind='microblog:clippings'").get()).toEqual({ status: 'pending' }); + await drainQueue(db, http); + expect(writes).toHaveLength(2); + expect(writes[1]).toMatchObject({ action: 'update', url: remoteUrl }); + expect(JSON.stringify(writes[1])).toContain('First passage'); + expect(JSON.stringify(writes[1])).toContain('Merged passage'); + expect(writes.filter((w) => !w.action)).toHaveLength(1); + db.close(); +}); + it.each(['source lookup', 'update'])('identifies a failed %s and preserves the existing post for retry', async (operation) => { const { db, uid } = await setup(); const http: HttpTransport = async (_url, init) => { From 31c602a5aadf22ece74fc470a578979347403eb9 Mon Sep 17 00:00:00 2001 From: jcapayne <7683622+jcapayne@users.noreply.github.com> Date: Thu, 29 Oct 2026 08:08:42 -0400 Subject: [PATCH 30/37] Keep Micro.blog recreation available without live clippings --- README.md | 5 +++-- app/src/Clippings.jsx | 20 ++++++++++++++------ app/src/Clippings.test.jsx | 21 +++++++++++++++++++++ 3 files changed, 38 insertions(+), 8 deletions(-) diff --git a/README.md b/README.md index 440e4055..c9e14bdd 100644 --- a/README.md +++ b/README.md @@ -196,8 +196,9 @@ Turning automatic sync off stops automatic additions while retaining published passages and manually selected clippings. Deleting a post on Micro.blog pauses publication for that book while preserving -its clippings locally. After the missing post is detected on an update, the book's -section on the Clippings page offers **Recreate clippings post**. This explicitly +its clippings locally. After the missing post is detected on an update, the +**Micro.blog publishing paused** section on the Clippings page offers **Recreate +clippings post**, even if the book has no live local clippings. This explicitly republishes the collected clippings to either the previous blog (the default) or the current clippings blog selected in Settings. Future updates for this book follow the replacement post. If the previous blog is accessible, Sync rechecks diff --git a/app/src/Clippings.jsx b/app/src/Clippings.jsx index d7896f72..c04cfaee 100644 --- a/app/src/Clippings.jsx +++ b/app/src/Clippings.jsx @@ -62,7 +62,6 @@ export default function Clippings({ session, books }) { const [connectors] = useLoad(() => api.connectors(session).then((r) => r.connectors), [session], `clipping-connectors:${session.username}`) const microblog = connectors?.some((c) => c.id === 'microblog' && c.linked) const [postData, postError, reloadPosts] = useLoad(() => microblog ? api.microblogClippingPosts(session) : Promise.resolve({ items: [] }), [session, microblog]) - const postByDocument = new Map((postData?.items || []).map((p) => [p.document, p])) const [recreating, setRecreating] = useState(null) const [query, setQuery] = useState('') const [only, setOnly] = useState(null) // document @@ -73,6 +72,7 @@ export default function Clippings({ session, books }) { const pending = useRef(false) const [sendError, setSendError] = useState(null) const byDoc = useMemo(() => new Map(books.map((b) => [b.document, b])), [books]) + const missingPosts = microblog ? (postData?.items || []).filter((post) => post.missing && byDoc.has(post.document)) : [] // Only clippings of books the library shows (metadata-less ones are hidden everywhere). const all = (items ?? []).filter((c) => byDoc.has(c.document)) @@ -125,6 +125,19 @@ export default function Clippings({ session, books }) {

      Clippings

      {sendError &&

      {sendError}

      } {postError && microblog &&

      Could not load Micro.blog post status: {postError.message}

      } + {note &&

      {note}

      } + {missingPosts.length > 0 &&
      +

      Micro.blog publishing paused

      + {missingPosts.map((post) => { + const book = byDoc.get(post.document) + return
      + {book.title || book.filename} + {book.author &&

      {book.author}

      } +

      The Micro.blog post was deleted. Clippings are retained while publishing is paused.

      + +
      + })} +
      } {error && !items ? ( @@ -175,7 +188,6 @@ export default function Clippings({ session, books }) {
    - {note &&

    {note}

    }
    - {postByDocument.get(book.document)?.missing &&
    -

    The Micro.blog clippings post was deleted. Clippings are saved here while publishing is paused.

    - -
    }
    {clips.map((c) => ( diff --git a/app/src/Clippings.test.jsx b/app/src/Clippings.test.jsx index b0756ac0..f5882db4 100644 --- a/app/src/Clippings.test.jsx +++ b/app/src/Clippings.test.jsx @@ -52,6 +52,27 @@ it('shows failures and allows another attempt without claiming publication succe }) const deletedPost = { document: 'book', missing: true, destination: 'https://first.micro.blog/', url: 'https://first.micro.blog/deleted.html' } +it('recreates a missing post after its last live local clipping has been deleted', async () => { + api.allClippings.mockResolvedValue([]) + api.microblogClippingPosts.mockResolvedValueOnce({ items: [deletedPost] }).mockResolvedValue({ items: [] }) + render() + await screen.findByText('No clippings yet') + fireEvent.click(await screen.findByRole('button', { name: 'Recreate clippings post' })) + expect(screen.getByRole('dialog').textContent).toContain('https://first.micro.blog/') + fireEvent.click(screen.getByRole('button', { name: 'Recreate post' })) + await waitFor(() => expect(api.recreateMicroblogPost).toHaveBeenCalledWith(session, 'book', 'original')) + await waitFor(() => expect(screen.queryByRole('dialog')).toBeNull()) + expect(screen.getByRole('status').textContent).toContain('queued') + await waitFor(() => expect(screen.queryByRole('button', { name: 'Recreate clippings post' })).toBeNull()) +}) +it('keeps missing-post recovery available when search hides every live clipping group', async () => { + api.microblogClippingPosts.mockResolvedValue({ items: [deletedPost] }) + render() + await screen.findByRole('button', { name: 'Recreate clippings post' }) + fireEvent.change(screen.getByRole('textbox', { name: 'Search clippings' }), { target: { value: 'unmatched search' } }) + await screen.findByText('No clippings match') + expect(screen.getAllByRole('button', { name: 'Recreate clippings post' })).toHaveLength(1) +}) it('offers recreation once per missing book and cancels without sending a request', async () => { api.microblogClippingPosts.mockResolvedValue({ items: [deletedPost] }) render() From 2801b21b44be4fb352fd28787a71d62924135089 Mon Sep 17 00:00:00 2001 From: jcapayne <7683622+jcapayne@users.noreply.github.com> Date: Thu, 29 Oct 2026 08:32:29 -0400 Subject: [PATCH 31/37] Preserve paused Micro.blog posts until the book is matched --- src/connectors/microblog-clippings.ts | 7 +++++++ test/microblog-destination-posts.test.ts | 19 +++++++++++++++++++ 2 files changed, 26 insertions(+) diff --git a/src/connectors/microblog-clippings.ts b/src/connectors/microblog-clippings.ts index 7d48c5b8..e7f148fe 100644 --- a/src/connectors/microblog-clippings.ts +++ b/src/connectors/microblog-clippings.ts @@ -341,6 +341,12 @@ export async function recreateMicroblogClippings(db: DB, userId: number, doc: st if (!account?.enabled) throw new ConnectorOperationError('Micro.blog is not linked or enabled', false); const state = db.prepare('SELECT * FROM microblog_clipping_posts WHERE user_id=? AND document=?').get(userId, document) as unknown as ClippingsPost | undefined; if (!state?.missing || !state.url) throw new ConnectorOperationError('This book does not have a confirmed missing clippings post.', false); + const requireMatch = () => { + if (!getMatch(db, userId, 'microblog', document)?.external_id) { + throw new ConnectorOperationError('Match this book in Micro.blog before recreating this post.', false); + } + }; + requireMatch(); let locks = publishing.get(db); if (!locks) { locks = new Set(); publishing.set(db, locks); } const key = `${userId}:${document}`; if (locks.has(key)) throw new ConnectorOperationError('Micro.blog publication is already in progress', true); @@ -348,6 +354,7 @@ export async function recreateMicroblogClippings(db: DB, userId: number, doc: st const current = () => { const row = db.prepare('SELECT identity FROM microblog_clipping_posts WHERE user_id=? AND document=?').get(userId, document) as { identity: string } | undefined; if (getAccount(db, userId, 'microblog')?.cred_enc !== account.cred_enc || row?.identity !== state.identity) throw new ConnectorOperationError('Micro.blog account or book changed; reload before retrying.', false); + requireMatch(); }; try { const cred = decryptCredential(account, db); diff --git a/test/microblog-destination-posts.test.ts b/test/microblog-destination-posts.test.ts index 7b65c090..6e3eaf65 100644 --- a/test/microblog-destination-posts.test.ts +++ b/test/microblog-destination-posts.test.ts @@ -168,6 +168,25 @@ it('allows explicit recreation on the current blog after access to the previous expect((await s.recreate('current')).status).toBe(200); await drainQueue(s.db, s.http); expect(s.writes[1]?.destination).toBe(a); s.db.close(); }); +it.each(['before recreation', 'during discovery'])('preserves a paused post when its book match is removed %s', async (when) => { + const s = await setup(); await s.choose(b); await s.sync(); s.missing(); s.change(); await s.sync(); + await s.choose(a); + const before = s.db.prepare('SELECT * FROM microblog_clipping_posts').get(); + const removeMatch = () => { s.db.prepare("DELETE FROM connector_matches WHERE user_id=? AND connector_id='microblog'").run(s.uid); }; + if (when === 'before recreation') removeMatch(); + else s.onDiscover(removeMatch); + const result = await s.recreate('current'); + expect(result.status).toBe(409); + expect(await result.json()).toMatchObject({ message: expect.stringContaining('Match this book') }); + expect(s.db.prepare('SELECT * FROM microblog_clipping_posts').get()).toEqual(before); + expect(s.writes).toHaveLength(1); + s.onDiscover(() => {}); + saveMatch(s.db, s.uid, 'microblog', DOC, { externalId: '123', externalEdition: '9781786487849', confidence: 1 }, 'manual'); + expect((await s.recreate('current')).status).toBe(200); + await drainQueue(s.db, s.http); + expect(s.writes[1]?.destination).toBe(a); + s.db.close(); +}); it('rejects an invalid recreation choice without changing the paused post', async () => { const s = await setup(); await s.choose(b); await s.sync(); s.missing(); s.change(); await s.sync(); expect((await s.recreate('elsewhere')).status).toBe(400); From 00ba58a68e9f7fbc4a2d6d11e6fb8099e46c054a Mon Sep 17 00:00:00 2001 From: jcapayne <7683622+jcapayne@users.noreply.github.com> Date: Thu, 29 Oct 2026 08:33:08 -0400 Subject: [PATCH 32/37] Retry Micro.blog Micropub timeouts through post recovery --- src/connectors/microblog-clippings.ts | 2 +- test/microblog-clippings-post.test.ts | 29 ++++++++++++++++++++------- 2 files changed, 23 insertions(+), 8 deletions(-) diff --git a/src/connectors/microblog-clippings.ts b/src/connectors/microblog-clippings.ts index e7f148fe..d3210784 100644 --- a/src/connectors/microblog-clippings.ts +++ b/src/connectors/microblog-clippings.ts @@ -166,7 +166,7 @@ class MissingMicroblogPost extends ConnectorOperationError { class MicropubResponseError extends ConnectorOperationError { constructor(readonly status: number, operation: string) { - super(`Micro.blog Micropub ${operation} failed (${status})`, status === 429 || status >= 500, + super(`Micro.blog Micropub ${operation} failed (${status})`, status === 408 || status === 429 || status >= 500, status === 401 || status === 403); } } diff --git a/test/microblog-clippings-post.test.ts b/test/microblog-clippings-post.test.ts index b1266fc0..c52cb7a1 100644 --- a/test/microblog-clippings-post.test.ts +++ b/test/microblog-clippings-post.test.ts @@ -136,16 +136,16 @@ it.each([false, true])('publishes selected clippings after merging an unmatched db.close(); }); -it.each(['source lookup', 'update'])('identifies a failed %s and preserves the existing post for retry', async (operation) => { +it.each([500, 408].flatMap((status) => ['source lookup', 'update'].map((operation) => ({ status, operation }))))('identifies a failed $operation ($status) and preserves the existing post for retry', async ({ operation, status }) => { const { db, uid } = await setup(); const http: HttpTransport = async (_url, init) => { - if (init.method === 'GET') return response(operation === 'source lookup' ? 500 : 200); - return response(500); + if (init.method === 'GET') return response(operation === 'source lookup' ? status : 200); + return response(status); }; db.prepare('UPDATE microblog_clipping_posts SET url=?,attempted=1').run(remoteUrl); await drainQueue(db, http); expect(db.prepare('SELECT status,last_error FROM connector_queue').get()).toMatchObject({ - status: 'pending', last_error: expect.stringContaining(`Micropub ${operation} failed (500)`), + status: 'pending', last_error: expect.stringContaining(`Micropub ${operation} failed (${status})`), }); expect(db.prepare('SELECT url,missing,error,fingerprint FROM microblog_clipping_posts WHERE user_id=?').get(uid)).toEqual({ url: remoteUrl, missing: 0, error: null, fingerprint: null, @@ -153,20 +153,35 @@ it.each(['source lookup', 'update'])('identifies a failed %s and preserves the e db.close(); }); -it('recovers an uncertain creation by source identity without creating a second post', async () => { +it.each(['connection loss', 'create timeout', 'recovery timeout'])('recovers an uncertain creation after %s without creating a second post', async (failure) => { const { db } = await setup(); let creates = 0; let saved: unknown; + let timedOut = false; const http: HttpTransport = async (url, init) => { expect(url).toContain('/micropub'); - if (init.method === 'GET') return response(200, { items: [{ properties: { ...(saved as any), url: [remoteUrl] } }] }); + if (init.method === 'GET') { + if (failure === 'recovery timeout' && !timedOut) { timedOut = true; return response(408); } + return response(200, { items: [{ properties: { ...(saved as any), url: [remoteUrl] } }] }); + } const body = JSON.parse(init.body!); - if (!body.action) { creates++; saved = body.properties; throw new Error('connection lost after acceptance'); } + if (!body.action) { + creates++; saved = body.properties; + if (failure === 'create timeout') return response(408); + throw new Error('connection lost after acceptance'); + } return response(200); }; await drainQueue(db, http); + expect(db.prepare('SELECT attempted,error FROM microblog_clipping_posts').get()).toEqual({ attempted: 1, error: null }); + expect(db.prepare('SELECT status FROM connector_queue').get()).toEqual({ status: 'pending' }); db.prepare("UPDATE connector_queue SET next_try_at=0 WHERE connector_id='microblog'").run(); await drainQueue(db, http); + if (failure === 'recovery timeout') { + expect(db.prepare('SELECT status FROM connector_queue').get()).toEqual({ status: 'pending' }); + db.prepare("UPDATE connector_queue SET next_try_at=0 WHERE connector_id='microblog'").run(); + await drainQueue(db, http); + } expect(creates).toBe(1); expect(db.prepare('SELECT url FROM microblog_clipping_posts').get()).toEqual({ url: remoteUrl }); expect(db.prepare('SELECT status FROM connector_queue').get()).toEqual({ status: 'done' }); From 3d3e278ff01cfbb0197f4ad857d7ed107ee414a5 Mon Sep 17 00:00:00 2001 From: jcapayne <7683622+jcapayne@users.noreply.github.com> Date: Thu, 29 Oct 2026 08:34:43 -0400 Subject: [PATCH 33/37] Schedule Micro.blog collections after importing clippings --- src/connectors/fanin.ts | 9 +++++++ test/bookorbit-pull.test.ts | 51 ++++++++++++++++++++++++++++++++++++- 2 files changed, 59 insertions(+), 1 deletion(-) diff --git a/src/connectors/fanin.ts b/src/connectors/fanin.ts index d4521718..f36c5bef 100644 --- a/src/connectors/fanin.ts +++ b/src/connectors/fanin.ts @@ -3,6 +3,7 @@ import { withTransaction, type DB } from '../db/db.js'; import { nowSeconds } from '../models/sync.js'; import { getConnector, fetchTransport } from './registry.js'; import { fanOutHighlight, fanOutProgress } from './fanout.js'; +import { scheduleMicroblogClippings } from './microblog-clippings.js'; import { clippingDocuments } from '../models/merge.js'; import { nearestProgressSample, recordProgressSample, upsertProgress } from '../routes/kosync.js'; import { @@ -198,6 +199,14 @@ function importHighlights(db: DB, userId: number, document: string, items: Inbou endOffset: h.endOffset, chapter: h.chapter, }, now, source); } + if (added.length) { + try { + scheduleMicroblogClippings(db, userId, document); + } catch (err) { + console.error(JSON.stringify({ msg: 'microblog schedule failed', user_id: userId, document, + error: err instanceof Error ? err.message : String(err) })); + } + } } /** Poll library-wide providers; per-book providers refresh on progress requests. */ diff --git a/test/bookorbit-pull.test.ts b/test/bookorbit-pull.test.ts index 4100379a..d4420bb6 100644 --- a/test/bookorbit-pull.test.ts +++ b/test/bookorbit-pull.test.ts @@ -1,7 +1,8 @@ import { afterEach, beforeEach, expect, it, vi } from 'vitest'; import { pollConnector } from '../src/connectors/fanin.js'; import { bookorbitConnector } from '../src/connectors/bookorbit.js'; -import { deleteAccount, saveMatch, upsertAccount } from '../src/connectors/store.js'; +import { deleteAccount, saveMatch, setConnectorOptions, upsertAccount } from '../src/connectors/store.js'; +import { scheduleMicroblogClippings } from '../src/connectors/microblog-clippings.js'; import { resetEncryptionKeyCache } from '../src/crypto/secrets.js'; import { upsertProgress } from '../src/routes/kosync.js'; import type { HttpTransport, InboundHighlight } from '../src/connectors/types.js'; @@ -52,6 +53,54 @@ it('imports progress and clippings on demand, exposing progress to bulk sync', a expect(next.changes).toEqual([]); }); +it.each([ + { automatic: true, matched: true }, + { automatic: false, matched: true }, + { automatic: true, matched: false }, +])('schedules imported clippings without a progress change (automatic: $automatic, matched: $matched)', async ({ automatic, matched }) => { + const f = await fixture(); + vi.spyOn(bookorbitConnector, 'pullProgress').mockResolvedValue(null); + vi.spyOn(bookorbitConnector, 'pullHighlights').mockResolvedValue([highlight]); + upsertAccount(f.db, f.id, 'microblog', { token: 'token' }, null); + setConnectorOptions(f.db, f.id, 'microblog', { sync_highlights: automatic }); + const match = () => saveMatch(f.db, f.id, 'microblog', DOC, + { externalId: '123', externalEdition: '9781786487849', confidence: 1 }, 'manual'); + if (matched) match(); + expect(await pollConnector(f.db, f.id, 'bookorbit', f.http, { document: DOC })).toBe(0); + expect(f.db.prepare('SELECT text FROM clippings').get()).toEqual({ text: highlight.text }); + expect(f.db.prepare('SELECT COUNT(*) AS n FROM microblog_clipping_selections').get()).toEqual({ n: automatic ? 1 : 0 }); + const jobs = () => f.db.prepare("SELECT document,status FROM connector_queue WHERE kind='microblog:clippings'").all(); + expect(jobs()).toEqual(automatic && matched ? [{ document: DOC, status: 'pending' }] : []); + if (automatic && !matched) { + match(); + expect(scheduleMicroblogClippings(f.db, f.id, DOC)).toBe(1); + } + await pollConnector(f.db, f.id, 'bookorbit', f.http, { document: DOC }); + expect(jobs()).toHaveLength(automatic ? 1 : 0); + expect(f.db.prepare('SELECT COUNT(*) AS n FROM clippings').get()).toEqual({ n: 1 }); + expect(f.db.prepare("SELECT COUNT(*) AS n FROM connector_queue WHERE connector_id='bookorbit'").get()).toEqual({ n: 0 }); + f.db.close(); +}); + +it('keeps imported clippings and other fanout when Micro.blog scheduling fails', async () => { + const f = await fixture(); + vi.spyOn(bookorbitConnector, 'pullProgress').mockResolvedValue(null); + vi.spyOn(bookorbitConnector, 'pullHighlights').mockResolvedValue([highlight]); + upsertAccount(f.db, f.id, 'microblog', { token: 'token' }, null); + setConnectorOptions(f.db, f.id, 'microblog', { sync_highlights: true }); + saveMatch(f.db, f.id, 'microblog', DOC, { externalId: '123', externalEdition: '9781786487849', confidence: 1 }, 'manual'); + upsertAccount(f.db, f.id, 'seekquel', { token: 'token' }, null); + saveMatch(f.db, f.id, 'seekquel', DOC, { externalId: '456', confidence: 1 }, 'manual'); + f.db.exec(`CREATE TEMP TRIGGER fail_import_schedule BEFORE INSERT ON microblog_clipping_selections + BEGIN SELECT RAISE(FAIL, 'simulated import scheduling failure'); END`); + await expect(pollConnector(f.db, f.id, 'bookorbit', f.http, { document: DOC, throwOnError: true })).resolves.toBe(0); + expect(f.db.prepare('SELECT text FROM clippings').get()).toEqual({ text: highlight.text }); + expect(f.db.prepare("SELECT kind FROM connector_queue WHERE connector_id='seekquel'").get()).toEqual({ kind: expect.stringMatching(/^highlight:/) }); + f.db.exec('DROP TRIGGER fail_import_schedule'); + expect(scheduleMicroblogClippings(f.db, f.id, DOC)).toBe(1); + f.db.close(); +}); + it('manually syncs only the requested book and queues the pulled position, not the old local one', async () => { const f = await fixture(); const other = '1234567890abcdef1234567890abcdef'; From b19606dd23edffbb7724a765ea30e74751d46328 Mon Sep 17 00:00:00 2001 From: jcapayne <7683622+jcapayne@users.noreply.github.com> Date: Thu, 29 Oct 2026 08:52:19 -0400 Subject: [PATCH 34/37] Recheck Micro.blog book matches before publication writes --- src/connectors/microblog-clippings.ts | 9 +++++++ test/microblog-clippings-post.test.ts | 36 +++++++++++++++++++++++++++ 2 files changed, 45 insertions(+) diff --git a/src/connectors/microblog-clippings.ts b/src/connectors/microblog-clippings.ts index d3210784..266b1cee 100644 --- a/src/connectors/microblog-clippings.ts +++ b/src/connectors/microblog-clippings.ts @@ -229,6 +229,7 @@ export async function publishMicroblogClippings( if (!locks) { locks = new Set(); publishing.set(db, locks); } if (locks.has(key)) return { ok: false, retryable: true, error: 'Micro.blog book publication is already in progress' }; const account = getAccount(db, userId, 'microblog'); + const initialMatch = getMatch(db, userId, 'microblog', document); let identity: string | undefined; const assertCurrent = () => { const current = getAccount(db, userId, 'microblog'); @@ -236,6 +237,14 @@ export async function publishMicroblogClippings( if (!account?.enabled || !current?.enabled || current.cred_enc !== account.cred_enc || (identity && saved?.identity !== identity)) { throw new ConnectorOperationError('Micro.blog account or book changed during publication', false); } + const currentMatch = getMatch(db, userId, 'microblog', document); + // The ISBN may have been enriched from the shelf without changing the saved + // match. Compare saved editions as well as the match passed by the caller. + if (!initialMatch?.external_id || initialMatch.external_id !== match.externalId || + (initialMatch.external_edition && initialMatch.external_edition !== match.externalEdition) || + currentMatch?.external_id !== initialMatch.external_id || currentMatch?.external_edition !== initialMatch.external_edition) { + throw new ConnectorOperationError('Micro.blog book match changed during publication; sync again with the current match.', false); + } }; const permanent = (error: string): PushResult => { assertCurrent(); diff --git a/test/microblog-clippings-post.test.ts b/test/microblog-clippings-post.test.ts index c52cb7a1..263a932e 100644 --- a/test/microblog-clippings-post.test.ts +++ b/test/microblog-clippings-post.test.ts @@ -385,3 +385,39 @@ it('updates the book link on the existing post after correcting a manual match', expect(JSON.stringify(writes[1])).toContain('https://micro.blog/books/9780143127796'); db.close(); }); + +it.each(['config', 'source lookup', 'post recovery'].flatMap((stage) => + ['disabled', 'book', 'edition'].map((change) => ({ stage, change }))))('stops publication when the match changes during $stage ($change)', async ({ stage, change }) => { + const { db, uid } = await setup(); + if (stage === 'source lookup') db.prepare('UPDATE microblog_clipping_posts SET url=?,attempted=1').run(remoteUrl); + if (stage === 'post recovery') db.prepare('UPDATE microblog_clipping_posts SET attempted=1,destination=?').run('https://example.micro.blog/'); + const before = db.prepare('SELECT * FROM microblog_clipping_posts').get(); + let changed = false; + const writes: unknown[] = []; + const http: HttpTransport = async (url, init) => { + const parsed = new URL(url); + if (init.method === 'POST') { writes.push(JSON.parse(init.body!)); return response(201, {}, remoteUrl); } + const requestStage = parsed.searchParams.get('q') === 'config' ? 'config' + : parsed.searchParams.has('url') ? 'source lookup' : 'post recovery'; + if (requestStage === stage && !changed) { + changed = true; + saveMatch(db, uid, 'microblog', DOC, change === 'disabled' ? null : { + externalId: change === 'book' ? '456' : '123', externalEdition: '9780143127796', confidence: 1, + }, 'manual'); + } + if (requestStage === 'config') return response(200, { destination: [{ uid: 'https://example.micro.blog/' }] }); + if (requestStage === 'post recovery') return response(200, { items: [{ properties: { + content: [``], url: [remoteUrl], + } }] }); + return response(200); + }; + // Use the real transport directly so the discovery request can change the match. + await runQueue(db, http); + expect(changed).toBe(true); + expect(writes).toEqual([]); + expect(db.prepare('SELECT url,fingerprint,error,missing FROM microblog_clipping_posts').get()).toMatchObject({ + url: stage === 'source lookup' ? remoteUrl : null, fingerprint: null, error: null, missing: 0, + }); + expect(db.prepare('SELECT last_error FROM connector_queue').get()).toMatchObject({ last_error: expect.stringContaining('match changed') }); + db.close(); +}); From f571e21c7c4466735ff1c5616e04ae3b619c9eae Mon Sep 17 00:00:00 2001 From: jcapayne <7683622+jcapayne@users.noreply.github.com> Date: Thu, 29 Oct 2026 08:54:14 -0400 Subject: [PATCH 35/37] Surface paused Micro.blog collection conflicts in Settings --- README.md | 4 ++++ src/connectors/microblog-clippings.ts | 12 ++++++++++ test/microblog-clippings-api.test.ts | 32 +++++++++++++++++++++++++++ 3 files changed, 48 insertions(+) diff --git a/README.md b/README.md index c9e14bdd..6bbe38da 100644 --- a/README.md +++ b/README.md @@ -207,6 +207,10 @@ blog instead. A previously confirmed deletion can also be recreated on the curre blog after access to the previous blog is lost. Network or permission errors do not trigger automatic recreation. +Merging books that both have Micro.blog publication history pauses their combined +collection. Settings shows the conflict as a sync error so you can review the +existing posts before publishing further clippings. + For local testing, use Node 22.13 or newer, build the frontend with `npm run build:frontend`, and copy `.env.example` to `.env`. Set a separate `DATABASE_PATH` (such as `./data/microblog-test.db`), `PORT=8081`, and a persistent diff --git a/src/connectors/microblog-clippings.ts b/src/connectors/microblog-clippings.ts index 266b1cee..97d97ef9 100644 --- a/src/connectors/microblog-clippings.ts +++ b/src/connectors/microblog-clippings.ts @@ -114,6 +114,18 @@ export function scheduleMicroblogClippings(db: DB, userId: number, document?: st const insert = db.prepare('INSERT OR IGNORE INTO microblog_clipping_selections(user_id,document,clipping_id) VALUES(?,?,?)'); for (const c of liveClippings(db, userId, doc)) insert.run(userId, doc, c.id); } + const blocked = db.prepare('SELECT error FROM microblog_clipping_posts WHERE user_id=? AND document=?') + .get(userId, doc) as { error: string | null } | undefined; + if (blocked?.error) { + // A merge can pause publication without making a delivery attempt. + // Surface durable errors in Settings even if this book is now unmatched. + withTransaction(db, () => { + enqueue(db, userId, 'microblog', { kind: 'highlight', document: doc, timestamp: nowSeconds() }, 'microblog:clippings'); + db.prepare("UPDATE connector_queue SET status='dead',last_error=? WHERE user_id=? AND connector_id='microblog' AND document=? AND kind='microblog:clippings'") + .run(blocked.error, userId, doc); + }); + continue; + } const match = getMatch(db, userId, 'microblog', doc); if (!match?.external_id || !passages(db, userId, doc).length) continue; const state = clippingsPost(db, userId, doc); diff --git a/test/microblog-clippings-api.test.ts b/test/microblog-clippings-api.test.ts index 5d09f1ba..77fbac04 100644 --- a/test/microblog-clippings-api.test.ts +++ b/test/microblog-clippings-api.test.ts @@ -9,6 +9,8 @@ import { deleteKosyncUserData } from '../src/routes/account.js'; import { fanOutProgress } from '../src/connectors/fanout.js'; import { createProgressRefresh } from '../src/connectors/refresh.js'; import { makeMicroblogTransport } from './microblog-helpers.js'; +import { mergeDocuments } from '../src/models/merge.js'; +import { clippingsPost, scheduleMicroblogClippings } from '../src/connectors/microblog-clippings.js'; beforeEach(() => { process.env.TOKEN_ENC_KEY = 'a'.repeat(64); resetEncryptionKeyCache(); }); afterEach(() => { delete process.env.TOKEN_ENC_KEY; resetEncryptionKeyCache(); }); @@ -45,6 +47,36 @@ async function schedulingFailure() { return s; } +it.each(['attempted', 'published'].flatMap((history) => [false, true].map((matched) => ({ history, matched }))))('surfaces merged $history post conflicts in Settings (matched: $matched)', async ({ history, matched }) => { + const s = await setup(); + const duplicate = 'b'.repeat(32); + s.db.prepare('UPDATE connector_accounts SET options=? WHERE user_id=? AND connector_id=?') + .run(JSON.stringify({ sync_highlights: true }), s.uid, 'microblog'); + s.db.prepare('INSERT INTO documents(user_id,document,title,updated_at) VALUES(?,?,?,0)').run(s.uid, duplicate, 'Duplicate book'); + s.db.prepare('INSERT INTO clippings(user_id,document,id,text,spine_index,created_at,updated_at) VALUES(?,?,?,?,0,1,1)') + .run(s.uid, duplicate, 'two', 'Another passage'); + for (const document of [DOC, duplicate]) { + clippingsPost(s.db, s.uid, document); + s.db.prepare('UPDATE microblog_clipping_posts SET attempted=1,url=? WHERE user_id=? AND document=?') + .run(history === 'published' ? `https://example.micro.blog/${document}.html` : null, s.uid, document); + } + if (matched) { + await s.match(); + s.db.prepare("UPDATE connector_queue SET status='done',last_error=NULL").run(); + } + mergeDocuments(s.db, s.uid, duplicate, DOC, 10); + const list = await (await s.app.request('/api/v1/connectors', { headers: s.headers })).json(); + expect(list.connectors.find((c: { id: string }) => c.id === 'microblog')).toMatchObject({ + queue: { dead: 1 }, + queue_errors: [{ count: 1, titles: ['Test book'], message: expect.stringContaining('Merged books both have Micro.blog publication history') }], + }); + expect(scheduleMicroblogClippings(s.db, s.uid, DOC)).toBe(0); + await drainQueue(s.db, s.http); + expect(s.posts).toEqual([]); + expect(s.db.prepare('SELECT COUNT(*) AS n FROM clippings').get()).toEqual({ n: 2 }); + s.db.close(); +}); + it('still fans out progress to other connectors when Micro.blog scheduling fails', async () => { const s = await schedulingFailure(); upsertAccount(s.db, s.uid, 'kosync', { server: 'mirror.test', username: 'reader', password: 'test' }, null); From 33bd45eb8e09dadde1a28966d106e404f617ba06 Mon Sep 17 00:00:00 2001 From: jcapayne <7683622+jcapayne@users.noreply.github.com> Date: Thu, 29 Oct 2026 09:06:42 -0400 Subject: [PATCH 36/37] Preserve Micro.blog conflicts raised during publication --- src/connectors/microblog-clippings.ts | 5 +++- test/microblog-clippings-post.test.ts | 41 ++++++++++++++++++++++++++- 2 files changed, 44 insertions(+), 2 deletions(-) diff --git a/src/connectors/microblog-clippings.ts b/src/connectors/microblog-clippings.ts index 97d97ef9..1c5b6daf 100644 --- a/src/connectors/microblog-clippings.ts +++ b/src/connectors/microblog-clippings.ts @@ -245,10 +245,13 @@ export async function publishMicroblogClippings( let identity: string | undefined; const assertCurrent = () => { const current = getAccount(db, userId, 'microblog'); - const saved = db.prepare('SELECT identity FROM microblog_clipping_posts WHERE user_id=? AND document=?').get(userId, document) as { identity: string } | undefined; + const saved = db.prepare('SELECT identity,error FROM microblog_clipping_posts WHERE user_id=? AND document=?').get(userId, document) as { identity: string; error: string | null } | undefined; if (!account?.enabled || !current?.enabled || current.cred_enc !== account.cred_enc || (identity && saved?.identity !== identity)) { throw new ConnectorOperationError('Micro.blog account or book changed during publication', false); } + // A merge can pause this collection without changing its identity. An older + // in-flight attempt must not acknowledge the write or clear that new error. + if (identity && saved?.error) throw new ConnectorOperationError(saved.error, false); const currentMatch = getMatch(db, userId, 'microblog', document); // The ISBN may have been enriched from the shelf without changing the saved // match. Compare saved editions as well as the match passed by the caller. diff --git a/test/microblog-clippings-post.test.ts b/test/microblog-clippings-post.test.ts index 263a932e..6626e442 100644 --- a/test/microblog-clippings-post.test.ts +++ b/test/microblog-clippings-post.test.ts @@ -2,7 +2,7 @@ import { afterEach, beforeEach, expect, it } from 'vitest'; import { makeTestApp, registerUser, DOC } from './helpers.js'; import { resetEncryptionKeyCache } from '../src/crypto/secrets.js'; import { upsertAccount, saveMatch } from '../src/connectors/store.js'; -import { publishMicroblogClippings, selectMicroblogClipping, scheduleMicroblogClippings } from '../src/connectors/microblog-clippings.js'; +import { clippingsPost, publishMicroblogClippings, selectMicroblogClipping, scheduleMicroblogClippings } from '../src/connectors/microblog-clippings.js'; import { drainQueue as runQueue } from '../src/connectors/runner.js'; import type { HttpTransport } from '../src/connectors/types.js'; import { mergeDocuments } from '../src/models/merge.js'; @@ -136,6 +136,45 @@ it.each([false, true])('publishes selected clippings after merging an unmatched db.close(); }); +it.each(['source lookup', 'create', 'update'])('preserves a merge conflict raised during an in-flight %s', async (stage) => { + const { db, uid } = await setup(); + if (stage !== 'create') db.prepare('UPDATE microblog_clipping_posts SET url=?,attempted=1').run(remoteUrl); + const source = 'b'.repeat(32); + db.prepare('INSERT INTO documents(user_id,document,title,updated_at) VALUES(?,?,?,0)').run(uid, source, 'Duplicate'); + db.prepare('INSERT INTO clippings(user_id,document,id,text,spine_index,created_at,updated_at) VALUES(?,?,?,?,1,2,2)') + .run(uid, source, 'merged', 'Merged passage'); + selectMicroblogClipping(db, uid, source, 'merged'); + clippingsPost(db, uid, source); + db.prepare('UPDATE microblog_clipping_posts SET attempted=1,url=? WHERE user_id=? AND document=?') + .run('https://example.micro.blog/duplicate.html', uid, source); + let merged = false; + const writes: any[] = []; + const http: HttpTransport = async (_url, init) => { + const body = init.method === 'POST' ? JSON.parse(init.body!) : null; + if (body) writes.push(body); + const operation = body ? body.action === 'update' ? 'update' : 'create' : 'source lookup'; + if (!merged && operation === stage) { + merged = true; + mergeDocuments(db, uid, source, DOC, 10); + } + return response(body && !body.action ? 201 : 200, {}, remoteUrl); + }; + await drainQueue(db, http); + expect(merged).toBe(true); + expect(db.prepare('SELECT error,fingerprint FROM microblog_clipping_posts WHERE user_id=? AND document=?').get(uid, DOC)).toEqual({ + error: expect.stringContaining('Merged books both have Micro.blog publication history'), fingerprint: null, + }); + expect(db.prepare("SELECT status,last_error FROM connector_queue WHERE document=? AND kind='microblog:clippings'").get(DOC)).toEqual({ + status: 'dead', last_error: expect.stringContaining('Merged books both have Micro.blog publication history'), + }); + expect(db.prepare('SELECT COUNT(*) AS n FROM microblog_clipping_selections WHERE published_snapshot IS NOT NULL').get()).toEqual({ n: 0 }); + expect(writes).toHaveLength(stage === 'source lookup' ? 0 : 1); + expect(scheduleMicroblogClippings(db, uid, DOC)).toBe(0); + await drainQueue(db, http); + expect(writes).toHaveLength(stage === 'source lookup' ? 0 : 1); + db.close(); +}); + it.each([500, 408].flatMap((status) => ['source lookup', 'update'].map((operation) => ({ status, operation }))))('identifies a failed $operation ($status) and preserves the existing post for retry', async ({ operation, status }) => { const { db, uid } = await setup(); const http: HttpTransport = async (_url, init) => { From 933edf0b06bebe3cffcceb4a7feb2e6fe95386c0 Mon Sep 17 00:00:00 2001 From: jcapayne <7683622+jcapayne@users.noreply.github.com> Date: Thu, 29 Oct 2026 09:07:36 -0400 Subject: [PATCH 37/37] Recheck clipping eligibility after Micro.blog source lookup --- src/connectors/microblog-clippings.ts | 5 +++ test/microblog-clippings-post.test.ts | 44 +++++++++++++++++++++++++++ 2 files changed, 49 insertions(+) diff --git a/src/connectors/microblog-clippings.ts b/src/connectors/microblog-clippings.ts index 1c5b6daf..2018ad10 100644 --- a/src/connectors/microblog-clippings.ts +++ b/src/connectors/microblog-clippings.ts @@ -309,6 +309,11 @@ export async function publishMicroblogClippings( // Detect deleted posts before replacing generated content. await micropub(http, token, destination, `/micropub?q=source&url=${encodeURIComponent(url)}`, undefined, true); assertCurrent(); + // Options and selections can change while the source request is in + // flight. Retry with fresh content before publishing or snapshotting it. + if (microblogClippingsContent(db, userId, document, `https://micro.blog/books/${encodeURIComponent(isbn)}`)?.fingerprint !== rendered.fingerprint) { + throw new ConnectorOperationError('Micro.blog clipping selections changed during publication; the next attempt will use the current selections.', true); + } await micropub(http, token, destination, '/micropub', { action: 'update', url, replace: { name: [rendered.title], content: [rendered.content] } }); assertCurrent(); diff --git a/test/microblog-clippings-post.test.ts b/test/microblog-clippings-post.test.ts index 6626e442..9f641fd6 100644 --- a/test/microblog-clippings-post.test.ts +++ b/test/microblog-clippings-post.test.ts @@ -345,6 +345,50 @@ it('retains published automatic passages when automatic sync is disabled and ano db.close(); }); +it.each([false, true])('rechecks eligibility when automatic sync is disabled during source lookup (new manual clipping: %s)', async (manual) => { + const { db, uid } = await setup(); + const writes: any[] = []; + let disableOnLookup = false; + const http: HttpTransport = async (_url, init) => { + if (init.method === 'GET') { + if (disableOnLookup) { + disableOnLookup = false; + db.prepare("UPDATE connector_accounts SET options='{}' WHERE user_id=? AND connector_id='microblog'").run(uid); + } + return response(200); + } + const body = JSON.parse(init.body!); writes.push(body); + return response(body.action ? 200 : 201, {}, remoteUrl); + }; + await drainQueue(db, http); + db.prepare("UPDATE connector_accounts SET options=? WHERE user_id=? AND connector_id='microblog'").run('{"sync_highlights":true}', uid); + const insert = db.prepare('INSERT INTO clippings(user_id,document,id,text,spine_index,created_at,updated_at) VALUES(?,?,?,?,1,2,2)'); + insert.run(uid, DOC, 'two', 'Unpublished automatic passage'); + if (manual) { + insert.run(uid, DOC, 'three', 'New manually selected passage'); + selectMicroblogClipping(db, uid, DOC, 'three'); + } + scheduleMicroblogClippings(db, uid, DOC); + disableOnLookup = true; + await drainQueue(db, http); + expect(writes).toHaveLength(1); + expect(db.prepare('SELECT published_snapshot FROM microblog_clipping_selections WHERE clipping_id=?').get('two')).toEqual({ published_snapshot: null }); + expect(db.prepare('SELECT error FROM microblog_clipping_posts').get()).toEqual({ error: null }); + expect(db.prepare('SELECT status FROM connector_queue').get()).toEqual({ status: 'pending' }); + db.prepare('UPDATE connector_queue SET next_try_at=0').run(); + await drainQueue(db, http); + expect(writes).toHaveLength(manual ? 2 : 1); + expect(JSON.stringify(writes)).not.toContain('Unpublished automatic passage'); + if (manual) { + expect(writes[1]).toMatchObject({ action: 'update', url: remoteUrl }); + expect(JSON.stringify(writes[1])).toContain('New manually selected passage'); + expect(JSON.stringify(writes[1])).toContain('First passage'); + } + expect(db.prepare('SELECT status FROM connector_queue').get()).toEqual({ status: 'done' }); + expect(db.prepare('SELECT published_snapshot FROM microblog_clipping_selections WHERE clipping_id=?').get('two')).toEqual({ published_snapshot: null }); + db.close(); +}); + it('queues a follow-up when a clipping is selected while the first post is being published', async () => { const { db, uid } = await setup(); let release!: () => void;