diff --git a/.env.example b/.env.example index 7f7df13b..931f0f8a 100644 --- a/.env.example +++ b/.env.example @@ -8,6 +8,9 @@ PORT=8080 # openssl rand -hex 32 TOKEN_ENC_KEY= +# Docker localhost testing only: bind the published port to 127.0.0.1. +# ALLOW_HTTP_LOCALHOST=true + # Enable only behind an HTTPS reverse proxy that sets X-Forwarded-Proto, # with direct access to Sync blocked. See README.md for Caddy setup. # TRUST_PROXY=true diff --git a/README.md b/README.md index 4353a50e..6bbe38da 100644 --- a/README.md +++ b/README.md @@ -39,6 +39,20 @@ For local image builds from the checkout, use: docker compose -f docker-compose.dev.yml up -d --build ``` +For localhost testing with connector credentials, Docker can forward requests +from a bridge address instead of a loopback address. Put your persistent +`TOKEN_ENC_KEY` in `.env`, then run: + +```sh +HOST_BIND=127.0.0.1 HOST_PORT=8081 ALLOW_HTTP_LOCALHOST=true \ + docker compose -f docker-compose.dev.yml up -d --build +``` + +Open `http://localhost:8081/app/`. This opt-in allows credential entry over HTTP +only for localhost URLs; keep the host port bound to `127.0.0.1`. For `docker run`, +use `-p 127.0.0.1:8081:8080 -e ALLOW_HTTP_LOCALHOST=true` and your existing encryption +key and data volume. Public/self-hosted network deployments should use HTTPS. + To serve under a subpath, pass `-e BASE_PATH=/crosspoint-sync` to `docker run`, or run `BASE_PATH=/crosspoint-sync docker compose up -d`. The same image works at any prefix; no frontend rebuild is needed. @@ -137,6 +151,7 @@ DATABASE_PATH=./data/crosspoint.db PORT=8080 node dist/index.js | `REGISTRATION_DISABLED` | `false` | Set `true` to lock down a private instance | | `AUTH_RATE_LIMIT_PER_MINUTE` | `30` | Per-IP limit on registration (0 disables) | | `TOKEN_ENC_KEY` | _(unset)_ | Enables external-service connectors. 64 hex chars, a base64 32-byte key, or a ≥32-char passphrase. Encrypts stored connector credentials at rest; unset = connectors disabled. | +| `ALLOW_HTTP_LOCALHOST` | `false` | Local Docker testing only: allow connector credentials over HTTP at `localhost`, `127.0.0.1`, or `[::1]` when Docker forwards from a bridge peer. Bind the published port to `127.0.0.1`; leave disabled on network-facing deployments. | | `TRUST_PROXY` | `false` (`true` on Railway) | Set `true` only when direct access is blocked and a trusted reverse proxy overwrites any client-supplied `X-Forwarded-Proto`; permits connector linking through an HTTPS-terminating proxy. Defaults to `true` when `RAILWAY_ENVIRONMENT` is present, since Railway always fronts the service with its TLS-terminating edge; set `TRUST_PROXY=false` to override. | | `GOOGLE_BOOKS_API_KEY` | _(unset)_ | Optional. Fallback source for print page counts when Open Library has none (keyless Google Books has no quota). | | `HARDCOVER_API_KEY` | _(unset)_ | Optional. A Hardcover API key (`read:catalog` is enough) for book details from Hardcover's catalog: moods, genres, content warnings, rating, series, release year, and page counts/covers where other sources miss. One lookup per distinct book, cached for all users, throttled under Hardcover's free-plan limits. | @@ -161,6 +176,49 @@ Native app association files, when enabled, remain at domain-root `/.well-known/ Treat the token like a password: Micro.blog app tokens have full account access. CrossPoint Sync encrypts the token at rest using `TOKEN_ENC_KEY`. +With Micro.blog linked, each clipping on the **Clippings** page has an +**Add clipping to Micro.blog** action. Selected clippings are collected into +one post per book, titled **📚 Clippings from {book title}**, with a link to the +matched Micro.blog book page. Later clippings update the same post. CrossPoint +Sync manages its generated content, so later updates replace edits made directly +to that content on Micro.blog. + +Choose **Post clippings to** in Micro.blog settings when the token accesses +multiple blogs. A sole available blog is selected automatically. The choice +applies to new collections: existing posts and new clippings for those books stay +on the book's original blog. Revoked access pauses posting to that blog; selecting +a different one never moves existing collections automatically. + +**Automatically sync clippings** is off by default. Enabling it includes existing +and future clippings. Unmatched books wait without interrupting reader sync; +after matching, eligible clippings are sent on the next reader sync or **Sync now**. +Turning automatic sync off stops automatic additions while retaining published +passages and manually selected clippings. + +Deleting a post on Micro.blog pauses publication for that book while preserving +its clippings locally. After the missing post is detected on an update, the +**Micro.blog publishing paused** section on the Clippings page offers **Recreate +clippings post**, even if the book has no live local clippings. This explicitly +republishes the collected clippings to either the previous blog (the default) or +the current clippings blog selected in Settings. Future updates for this book +follow the replacement post. If the previous blog is accessible, Sync rechecks +that the old post is still missing; a restored post can be resumed on the previous +blog instead. A previously confirmed deletion can also be recreated on the current +blog after access to the previous blog is lost. Network or permission errors do +not trigger automatic recreation. + +Merging books that both have Micro.blog publication history pauses their combined +collection. Settings shows the conflict as a sync error so you can review the +existing posts before publishing further clippings. + +For local testing, use Node 22.13 or newer, build the frontend with +`npm run build:frontend`, and copy `.env.example` to `.env`. Set a separate +`DATABASE_PATH` (such as `./data/microblog-test.db`), `PORT=8081`, and a persistent +`TOKEN_ENC_KEY` generated with `openssl rand -hex 32`. Run `npm run dev` and open +[the local app](http://localhost:8081/app/). Create a local account, sync a book +and its clippings from a reader, link Micro.blog, and match the book before trying +the clipping action or automatic sync. These actions publish to the linked blog. + ## Point your reader at it - **KOReader:** Tools → Progress sync → Custom sync server → `http://your-host:8080` diff --git a/app/src/Clippings.jsx b/app/src/Clippings.jsx index 6a7af78c..c04cfaee 100644 --- a/app/src/Clippings.jsx +++ b/app/src/Clippings.jsx @@ -1,9 +1,9 @@ -import { useMemo, useState } from 'react' +import { useMemo, useRef, useState } from 'react' import { Copy, FileDown, Quote, Search } from 'lucide-react' import { api, cached, isApp } from './api.js' import { saveFile } from './shareCard.js' import { ClipMenu, ClipShare, DeleteClip, hold } from './Book.jsx' -import { Cover, EmptyState, ErrorNote, ShareIcon, Spinner, useLoad } from './ui.jsx' +import { ActionSheet, Cover, EmptyState, ErrorNote, ShareIcon, Spinner, useLoad } from './ui.jsx' const date = (unix) => new Date(unix * 1000).toLocaleDateString(undefined, { month: 'short', day: 'numeric', year: 'numeric' }) @@ -25,7 +25,7 @@ function markdown(groups) { .join('\n\n---\n\n') } -function Clip({ session, clip, book, onShare, onDelete, showBook }) { +function Clip({ session, clip, book, onShare, onDelete, showBook, onMicroblog, sending }) { return (
)} -
+
+ {onMicroblog && ( + + )}

{[clip.chapter, date(clip.created_at)].filter(Boolean).join(' · ')}

@@ -54,12 +59,20 @@ function Clip({ session, clip, book, onShare, onDelete, showBook }) { export default function Clippings({ session, books }) { const [items, error, reload] = useLoad(() => api.allClippings(session, books), [session], `clippings:${session.username}`, () => cached.clippings(session)) + const [connectors] = useLoad(() => api.connectors(session).then((r) => r.connectors), [session], `clipping-connectors:${session.username}`) + const microblog = connectors?.some((c) => c.id === 'microblog' && c.linked) + const [postData, postError, reloadPosts] = useLoad(() => microblog ? api.microblogClippingPosts(session) : Promise.resolve({ items: [] }), [session, microblog]) + const [recreating, setRecreating] = useState(null) const [query, setQuery] = useState('') const [only, setOnly] = useState(null) // document const [sharing, setSharing] = useState(null) const [deleting, setDeleting] = useState(null) const [note, setNote] = useState(null) + const [sending, setSending] = useState(false) + const pending = useRef(false) + const [sendError, setSendError] = useState(null) const byDoc = useMemo(() => new Map(books.map((b) => [b.document, b])), [books]) + const missingPosts = microblog ? (postData?.items || []).filter((post) => post.missing && byDoc.has(post.document)) : [] // Only clippings of books the library shows (metadata-less ones are hidden everywhere). const all = (items ?? []).filter((c) => byDoc.has(c.document)) @@ -74,6 +87,26 @@ export default function Clippings({ session, books }) { const counts = all.reduce((m, c) => m.set(c.document, (m.get(c.document) ?? 0) + 1), new Map()) const daily = all.length ? ofTheDay(all) : null + async function addToMicroblog(clip) { + if (pending.current) return + pending.current = true + setSending(true) + setNote(null) + setSendError(null) + try { + const result = await api.addMicroblogClipping(session, clip.document, clip.id) + setNote(result.waiting_for_match + ? 'Clipping saved for Micro.blog. Match this book in Settings; it will be included on the next sync.' + : result.queued ? 'Clipping selected for the book’s Micro.blog post. The update is queued.' + : 'Clipping is selected for the book’s Micro.blog post.') + } catch (error) { + setSendError(error.message || 'Could not add this clipping to Micro.blog.') + } finally { + pending.current = false + setSending(false) + } + } + async function exportMd(save, e) { // A
menu stays open until toggled; close it so the note shows. e.currentTarget.closest('details').open = false @@ -90,6 +123,21 @@ export default function Clippings({ session, books }) { return (

Clippings

+ {sendError &&

{sendError}

} + {postError && microblog &&

Could not load Micro.blog post status: {postError.message}

} + {note &&

{note}

} + {missingPosts.length > 0 &&
+

Micro.blog publishing paused

+ {missingPosts.map((post) => { + const book = byDoc.get(post.document) + return
+ {book.title || book.filename} + {book.author &&

{book.author}

} +

The Micro.blog post was deleted. Clippings are retained while publishing is paused.

+ +
+ })} +
} {error && !items ? ( @@ -98,16 +146,16 @@ export default function Clippings({ session, books }) { ) : all.length === 0 ? (
- Highlight a passage on your reader with clipping sync on, and it lands here, ready to search and share. + Clippings saved on your reader land here, ready to search and share.
) : ( <> {daily && !q && !only && (
-

Highlight of the day

+

Clipping of the day

- +
)} @@ -140,7 +188,6 @@ export default function Clippings({ session, books }) {
- {note &&

{note}

}
} + + ) +} diff --git a/app/src/Clippings.test.jsx b/app/src/Clippings.test.jsx new file mode 100644 index 00000000..f5882db4 --- /dev/null +++ b/app/src/Clippings.test.jsx @@ -0,0 +1,123 @@ +// @vitest-environment jsdom +import { afterEach, beforeEach, expect, it, vi } from 'vitest' +import { cleanup, fireEvent, render, screen, waitFor } from '@testing-library/react' +vi.hoisted(() => { window.scrollTo = vi.fn(); window.matchMedia = vi.fn(() => ({ matches: false, addEventListener() {}, removeEventListener() {} })) }) +vi.mock('./api.js', () => ({ isApp: false, cached: { clippings: () => null }, api: { allClippings: vi.fn(), connectors: vi.fn(), addMicroblogClipping: vi.fn(), microblogClippingPosts: vi.fn(), recreateMicroblogPost: vi.fn(), microblogDestinations: vi.fn() } })) +vi.mock('./tabbar.js', () => ({ useSheet() {}, useNativeBack() {}, useNativeButton() {} })) +import Clippings from './Clippings.jsx' +import { api } from './api.js' + +const session = { username: 'reader' } +const books = [{ document: 'book', title: 'Test book', author: 'An author', cover_url: '/cover.jpg' }] +beforeEach(() => { + api.allClippings.mockResolvedValue([{ id: 'one', document: 'book', text: 'A passage', created_at: 1 }]) + api.connectors.mockResolvedValue({ connectors: [{ id: 'microblog', linked: true }] }) + api.addMicroblogClipping.mockResolvedValue({ queued: true, waiting_for_match: false }) + api.microblogClippingPosts.mockResolvedValue({ items: [] }) + api.recreateMicroblogPost.mockResolvedValue({ queued: true }) + api.microblogDestinations.mockResolvedValue({ items: [{ uid: 'https://first.micro.blog/', name: 'Previous' }, { uid: 'https://new.micro.blog/', name: 'New' }], selected: 'https://new.micro.blog/' }) +}) +afterEach(() => { cleanup(); vi.clearAllMocks() }) + +it('shows the action on both daily and book clippings with configured Micro.blog', async () => { + render() + expect(await screen.findAllByRole('button', { name: 'Add clipping to Micro.blog' })).toHaveLength(2) +}) +it('hides the action when Micro.blog is unlinked', async () => { + api.connectors.mockResolvedValue({ connectors: [{ id: 'microblog', linked: false }] }) + render() + await screen.findAllByText('A passage') + expect(screen.queryByRole('button', { name: 'Add clipping to Micro.blog' })).toBeNull() +}) +it('selects a clipping, prevents repeat requests, and explains waiting for a match', async () => { + let finish + api.addMicroblogClipping.mockImplementation(() => new Promise((resolve) => { finish = resolve })) + render() + const buttons = await screen.findAllByRole('button', { name: 'Add clipping to Micro.blog' }) + fireEvent.click(buttons[0]) + expect(buttons[1].disabled).toBe(true) + fireEvent.click(buttons[1]) + expect(api.addMicroblogClipping).toHaveBeenCalledTimes(1) + expect(api.addMicroblogClipping).toHaveBeenCalledWith(session, 'book', 'one') + finish({ queued: false, waiting_for_match: true }) + await waitFor(() => expect(screen.getByRole('status').textContent).toMatch(/match.*next sync/i)) +}) +it('shows failures and allows another attempt without claiming publication succeeded', async () => { + api.addMicroblogClipping.mockRejectedValue(new Error('Micro.blog is unavailable')) + render() + fireEvent.click((await screen.findAllByRole('button', { name: 'Add clipping to Micro.blog' }))[0]) + expect((await screen.findByRole('alert')).textContent).toContain('Micro.blog is unavailable') + expect(screen.queryByRole('status')).toBeNull() + expect(screen.getAllByRole('button', { name: 'Add clipping to Micro.blog' })[0].disabled).toBe(false) +}) + +const deletedPost = { document: 'book', missing: true, destination: 'https://first.micro.blog/', url: 'https://first.micro.blog/deleted.html' } +it('recreates a missing post after its last live local clipping has been deleted', async () => { + api.allClippings.mockResolvedValue([]) + api.microblogClippingPosts.mockResolvedValueOnce({ items: [deletedPost] }).mockResolvedValue({ items: [] }) + render() + await screen.findByText('No clippings yet') + fireEvent.click(await screen.findByRole('button', { name: 'Recreate clippings post' })) + expect(screen.getByRole('dialog').textContent).toContain('https://first.micro.blog/') + fireEvent.click(screen.getByRole('button', { name: 'Recreate post' })) + await waitFor(() => expect(api.recreateMicroblogPost).toHaveBeenCalledWith(session, 'book', 'original')) + await waitFor(() => expect(screen.queryByRole('dialog')).toBeNull()) + expect(screen.getByRole('status').textContent).toContain('queued') + await waitFor(() => expect(screen.queryByRole('button', { name: 'Recreate clippings post' })).toBeNull()) +}) +it('keeps missing-post recovery available when search hides every live clipping group', async () => { + api.microblogClippingPosts.mockResolvedValue({ items: [deletedPost] }) + render() + await screen.findByRole('button', { name: 'Recreate clippings post' }) + fireEvent.change(screen.getByRole('textbox', { name: 'Search clippings' }), { target: { value: 'unmatched search' } }) + await screen.findByText('No clippings match') + expect(screen.getAllByRole('button', { name: 'Recreate clippings post' })).toHaveLength(1) +}) +it('offers recreation once per missing book and cancels without sending a request', async () => { + api.microblogClippingPosts.mockResolvedValue({ items: [deletedPost] }) + render() + fireEvent.click(await screen.findByRole('button', { name: 'Recreate clippings post' })) + expect(screen.getByRole('dialog').textContent).toContain('https://first.micro.blog/') + fireEvent.click(screen.getByRole('button', { name: 'Cancel' })) + expect(api.recreateMicroblogPost).not.toHaveBeenCalled() +}) +it('queues explicit recreation and refreshes the book status', async () => { + api.microblogClippingPosts.mockResolvedValueOnce({ items: [deletedPost] }).mockResolvedValue({ items: [] }) + render() + fireEvent.click(await screen.findByRole('button', { name: 'Recreate clippings post' })) + fireEvent.click(screen.getByRole('button', { name: 'Recreate post' })) + await waitFor(() => expect(api.recreateMicroblogPost).toHaveBeenCalledWith(session, 'book', 'original')) + await waitFor(() => expect(screen.queryByRole('dialog')).toBeNull()) + expect(screen.getByRole('status').textContent).toContain('queued') +}) +it('shows recreation errors and allows retry', async () => { + api.microblogClippingPosts.mockResolvedValue({ items: [deletedPost] }) + api.recreateMicroblogPost.mockRejectedValue(new Error('Original blog is unavailable')) + render() + fireEvent.click(await screen.findByRole('button', { name: 'Recreate clippings post' })) + fireEvent.click(screen.getByRole('button', { name: 'Recreate post' })) + expect((await screen.findByRole('alert')).textContent).toContain('Original blog is unavailable') + expect(screen.getByRole('button', { name: 'Recreate post' }).disabled).toBe(false) +}) +it('lets the reader choose the current clippings blog when recreating a post', async () => { + api.microblogClippingPosts.mockResolvedValue({ items: [deletedPost] }) + render() + fireEvent.click(await screen.findByRole('button', { name: 'Recreate clippings post' })) + const choice = await screen.findByLabelText(/Current clippings blog/) + await waitFor(() => expect(choice.disabled).toBe(false)) + expect(screen.getByLabelText(/Previous blog/).checked).toBe(true) + fireEvent.click(choice) + fireEvent.click(screen.getByRole('button', { name: 'Recreate post' })) + await waitFor(() => expect(api.recreateMicroblogPost).toHaveBeenCalledWith(session, 'book', 'current')) + await waitFor(() => expect(screen.queryByRole('dialog')).toBeNull()) +}) +it('keeps the previous blog usable when the current clippings blog is unavailable', async () => { + api.microblogClippingPosts.mockResolvedValue({ items: [deletedPost] }) + api.microblogDestinations.mockResolvedValue({ items: [], selected: 'https://unavailable.micro.blog/' }) + render() + fireEvent.click(await screen.findByRole('button', { name: 'Recreate clippings post' })) + await screen.findByText(/Choose an available blog in Settings/) + expect(screen.getByLabelText(/Current clippings blog/).disabled).toBe(true) + fireEvent.click(screen.getByRole('button', { name: 'Recreate post' })) + await waitFor(() => expect(api.recreateMicroblogPost).toHaveBeenCalledWith(session, 'book', 'original')) +}) diff --git a/app/src/MicroblogPosting.jsx b/app/src/MicroblogPosting.jsx new file mode 100644 index 00000000..1d95d2ef --- /dev/null +++ b/app/src/MicroblogPosting.jsx @@ -0,0 +1,40 @@ +import { useRef, useState } from 'react' +import { api } from './api.js' +import { useLoad } from './ui.jsx' + +export default function MicroblogPosting({ session, onChange }) { + const [data, loadError, reload] = useLoad(() => api.microblogDestinations(session), [session]) + const [override, setOverride] = useState(undefined) + const [busy, setBusy] = useState(false) + const pending = useRef(false) + const [error, setError] = useState(null) + const value = override === undefined ? data?.selected || '' : override + async function change(destination) { + if (pending.current) return + pending.current = true + setBusy(true) + setError(null) + try { + await api.setMicroblogDestination(session, destination) + setOverride(destination) + onChange?.() + } catch (e) { setError(e.message || 'Could not save blog') } + finally { pending.current = false; setBusy(false) } + } + return ( +
+ +

For new collections. Existing posts and later clippings stay on their original blog.

+ {data?.items?.length === 0 &&

No blogs are available to this token.

} + {(error || loadError) &&

{error || loadError.message}

} + {loadError && } +
+ ) +} diff --git a/app/src/MicroblogPosting.test.jsx b/app/src/MicroblogPosting.test.jsx new file mode 100644 index 00000000..34eec772 --- /dev/null +++ b/app/src/MicroblogPosting.test.jsx @@ -0,0 +1,67 @@ +// @vitest-environment jsdom +import { afterEach, beforeEach, expect, it, vi } from 'vitest' +import { cleanup, fireEvent, render, screen, waitFor } from '@testing-library/react' +vi.mock('./api.js', () => ({ api: { microblogDestinations: vi.fn(), setMicroblogDestination: vi.fn() } })) +import MicroblogPosting from './MicroblogPosting.jsx' +import { api } from './api.js' +const a = 'https://first.micro.blog/' +const b = 'https://second.micro.blog/' +beforeEach(() => { + api.microblogDestinations.mockResolvedValue({ items: [{ uid: a, name: 'First blog' }, { uid: b, name: 'Second blog' }], selected: null }) + api.setMicroblogDestination.mockResolvedValue({ destination: b }) +}) +afterEach(() => { cleanup(); vi.clearAllMocks() }) +it('requires a choice among multiple blogs and saves it', async () => { + const onChange = vi.fn() + render() + const select = await screen.findByLabelText('Post clippings to') + await waitFor(() => expect(select.options.length).toBe(3)) + expect(select.value).toBe('') + fireEvent.change(select, { target: { value: b } }) + await waitFor(() => expect(onChange).toHaveBeenCalledOnce()) + expect(api.setMicroblogDestination).toHaveBeenCalledWith({}, b) + expect(select.value).toBe(b) +}) +it('shows the sole automatically selected blog without issuing a save', async () => { + api.microblogDestinations.mockResolvedValue({ items: [{ uid: a, name: 'First blog' }], selected: a }) + render() + await waitFor(() => expect(screen.getByLabelText('Post clippings to').value).toBe(a)) + expect(api.setMicroblogDestination).not.toHaveBeenCalled() +}) +it('keeps an unavailable saved choice visible rather than silently choosing another blog', async () => { + api.microblogDestinations.mockResolvedValue({ items: [{ uid: a, name: 'First blog' }], selected: b }) + render() + expect(await screen.findByText(/Unavailable.*second.micro.blog/)).toBeTruthy() + expect(screen.getByLabelText('Post clippings to').value).toBe(b) +}) +it('restores the saved value and allows retry after a failed save', async () => { + api.setMicroblogDestination.mockRejectedValue(new Error('Could not save blog')) + render() + const select = await screen.findByLabelText('Post clippings to') + await waitFor(() => expect(select.options.length).toBe(3)) + fireEvent.change(select, { target: { value: b } }) + expect((await screen.findByRole('alert')).textContent).toContain('Could not save blog') + expect(select.value).toBe('') + expect(select.disabled).toBe(false) +}) +it('shows discovery errors and retries loading', async () => { + api.microblogDestinations.mockRejectedValueOnce(new Error('Discovery unavailable')) + render() + expect((await screen.findByRole('alert')).textContent).toContain('Discovery unavailable') + fireEvent.click(screen.getByRole('button', { name: 'Retry loading blogs' })) + expect(await screen.findByRole('option', { name: 'First blog' })).toBeTruthy() +}) +it('disables the selector until a save completes', async () => { + let finish + api.setMicroblogDestination.mockImplementation(() => new Promise((resolve) => { finish = resolve })) + render() + const select = await screen.findByLabelText('Post clippings to') + await waitFor(() => expect(select.options.length).toBe(3)) + fireEvent.change(select, { target: { value: b } }) + expect(select.disabled).toBe(true) + fireEvent.change(select, { target: { value: a } }) + expect(api.setMicroblogDestination).toHaveBeenCalledTimes(1) + finish({ destination: b }) + await waitFor(() => expect(select.disabled).toBe(false)) + expect(select.value).toBe(b) +}) diff --git a/app/src/Settings.jsx b/app/src/Settings.jsx index 9c4350df..6d95fde9 100644 --- a/app/src/Settings.jsx +++ b/app/src/Settings.jsx @@ -1,4 +1,5 @@ import { useEffect, useRef, useState } from 'react' +import MicroblogPosting from './MicroblogPosting.jsx' import { openUrl } from '@tauri-apps/plugin-opener' import { onOpenUrl } from '@tauri-apps/plugin-deep-link' import { ArrowLeft, ChevronRight, Copy, KeyRound, ListChecks, Loader2, LogOut, Monitor, Moon, RefreshCw, Search, Server, Sun, Trash2, UserX } from 'lucide-react' @@ -38,7 +39,7 @@ function Appearance({ theme: [pref, setPref] }) { // What each service does, as on the web dashboard. const HINTS = { hardcover: 'Syncs your reading progress and shelf status to Hardcover, and your highlights if you turn them on.', - microblog: 'Keeps your Currently reading and Finished reading bookshelves in sync.', + microblog: 'Keeps your Currently reading and Finished reading bookshelves in sync. Optional clipping sync publishes one post per book and updates its generated content on later syncs, replacing edits made on Micro.blog.', readwise: 'Syncs your highlights to Readwise.', 'readwise-reader': 'Archives books in Reader when you finish them, and brings your Reader progress back.', kosync: 'Mirrors your reading progress to another KOReader-compatible (KOSync) server. For Calibre-Web, use your library URL ending in /kosync.', @@ -578,6 +579,7 @@ function Service({ session, conn, onChange }) { {conn.linked && !needsRelink && conn.options && ( act('options', () => api.setConnectorOptions(session, conn.id, v))} /> )} + {conn.id === 'microblog' && conn.linked && !needsRelink && } {openForm && (!conn.linked || needsRelink) && ( call(s, `/api/v1/connectors/${id}`, { method: 'PUT', body: JSON.stringify({ credential }) }), unlinkConnector: (s, id) => call(s, `/api/v1/connectors/${id}`, { method: 'DELETE' }), syncConnector: (s, id) => call(s, `/api/v1/connectors/${id}/sync`, { method: 'POST' }), + addMicroblogClipping: (s, doc, clippingId) => call(s, `/api/v1/connectors/microblog/clippings/${encodeURIComponent(doc)}/${encodeURIComponent(clippingId)}`, { method: 'POST' }), + microblogDestinations: (s) => call(s, '/api/v1/connectors/microblog/destinations'), + setMicroblogDestination: (s, destination) => call(s, '/api/v1/connectors/microblog/destination', { method: 'PUT', body: JSON.stringify({ destination }) }), + microblogClippingPosts: (s) => call(s, '/api/v1/connectors/microblog/clipping-posts'), + recreateMicroblogPost: (s, doc, target = 'original') => call(s, `/api/v1/connectors/microblog/clipping-posts/${encodeURIComponent(doc)}/recreate`, { method: 'POST', body: JSON.stringify({ target }) }), syncBook: (s, id, doc) => call(s, `/api/v1/connectors/${id}/sync/${doc}`, { method: 'POST' }), setConnectorOptions: (s, id, options) => call(s, `/api/v1/connectors/${id}/options`, { method: 'PUT', body: JSON.stringify(options) }), beginLink: (s, id) => call(s, `/api/v1/connectors/${id}/link/begin`, { method: 'POST' }), diff --git a/app/src/api.test.js b/app/src/api.test.js index b6420e61..f8491c96 100644 --- a/app/src/api.test.js +++ b/app/src/api.test.js @@ -19,6 +19,14 @@ beforeEach(() => { fetchMock.mockReset() }) +it('serializes the explicit clippings recreation destination choice', async () => { + fetchMock.mockResolvedValue(response({ queued: true })) + await api.recreateMicroblogPost(session('reader'), 'book', 'current') + expect(fetchMock).toHaveBeenCalledWith('https://sync.test/api/v1/connectors/microblog/clipping-posts/book/recreate', expect.objectContaining({ + method: 'POST', body: '{"target":"current"}', + })) +}) + describe('prefixed server URLs', () => { it.each(['', '/reading', '/services/reading'])('detects the hosted server at %j', (basePath) => { expect(hostedServerUrl(new URL(`https://sync.test${basePath}/app/`))).toBe(`https://sync.test${basePath}`) diff --git a/docker-compose.dev.yml b/docker-compose.dev.yml index e8b22abf..cf252479 100644 --- a/docker-compose.dev.yml +++ b/docker-compose.dev.yml @@ -2,11 +2,14 @@ services: crosspoint-sync: build: . ports: - - "8080:8080" + - "${HOST_BIND:-0.0.0.0}:${HOST_PORT:-8080}:8080" volumes: - crosspoint-data:/data environment: BASE_PATH: "${BASE_PATH:-}" + TOKEN_ENC_KEY: "${TOKEN_ENC_KEY:-}" + # Local Docker testing only; set HOST_BIND=127.0.0.1 alongside this. + ALLOW_HTTP_LOCALHOST: "${ALLOW_HTTP_LOCALHOST:-false}" # Set to "true" on private instances once your accounts exist. REGISTRATION_DISABLED: "false" # Set Listen Address so the service is accessible from outside the container diff --git a/docs/superpowers/plans/2026-10-28-microblog-clippings-posts.md b/docs/superpowers/plans/2026-10-28-microblog-clippings-posts.md new file mode 100644 index 00000000..333a4a75 --- /dev/null +++ b/docs/superpowers/plans/2026-10-28-microblog-clippings-posts.md @@ -0,0 +1,78 @@ +# Micro.blog Clippings Posts Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Publish eligible clippings as one maintained Micro.blog post per book, with reliable catch-up after matching. + +**Architecture:** Keep publication intent and remote post state in dedicated tables, independent of queue attempts. A Micro.blog-specific module selects canonical clippings and schedules one coalesced job per book; the connector publishes the latest selected content via Micropub. Extend the current settings UI and clippings hub without changing other connectors' highlight behavior. + +**Tech Stack:** TypeScript, Hono, SQLite, Vitest, React, existing HTTP transport. + +**Spec:** `docs/superpowers/specs/2026-10-28-microblog-clippings-post-design.md` + +## Global Constraints + +- **Automatically sync highlights**, disabled by default. +- A missing book match never blocks reader sync or creates a book solely to publish clippings. +- Manual selection works with automatic publication disabled. +- Do not delete remote posts on unlink or local clipping deletion. +- Do not blindly retry a creation with an ambiguous outcome. +- Leave unrelated work untouched; use an isolated worktree for implementation. + +## Review Focus + +- Failed creation after remote acceptance must recover the original post rather than duplicate it (Task 2). +- Disabling automatic publication must preserve already published content and manual selections (Tasks 1–2). +- Alias merges must preserve selections and avoid a second post (Tasks 1–2). +- An unchanged device sync after a manual match must publish earlier eligible clippings (Task 3). +- Unlinked, disabled, or another user's account must never publish a requested clipping (Task 3). + +### Task 1: Durable selection and post state + +**Files:** New migration after the current migration sequence; new `src/connectors/microblog-clippings.ts`; modify `src/models/merge.ts`; new `test/microblog-clippings.test.ts`. + +**Interfaces:** Produce `selectMicroblogClipping(db: DB, userId: number, document: string, clippingId: string): void`, `scheduleMicroblogClippings(db: DB, userId: number, document?: string): number`, and `microblogClippingsContent(db: DB, userId: number, document: string, bookUrl: string): { title: string; content: string; fingerprint: string } | null`. + +- [ ] Write failing tests: manual selection includes only the requested live clipping; automatic selection includes all existing live clippings; stable reading order produces an unchanged fingerprint; disabling automatic publication retains previously published passages; alias selection resolves to the canonical book; users cannot share state. +- [ ] Run `npm test -- test/microblog-clippings.test.ts`; confirm failures are missing behavior. +- [ ] Add selection rows with manual/automatic intent and published content snapshots; add per-book remote URL, fingerprint, and creation-attempt/recovery state. Use existing database migration and transaction patterns. Schedule only enabled accounts, eligible content, and usable existing matches; coalesce with `microblog:clippings` as the queue key and `highlight` as the event kind. +- [ ] Implement the interfaces above using the merged clipping view. Render **Highlights from {book title}**, the matched book link, quoted passages, notes, and chapters; escape user text appropriately. Carry selections/state through document merges. If both merged documents already have remote posts, retain one authoritative URL and surface the conflict without deleting either post. +- [ ] Re-run the new tests plus existing merge tests; commit the tested deliverable. + +### Task 2: Micropub creation, updates, and recovery + +**Files:** Modify `src/connectors/microblog.ts`, `src/connectors/runner.ts`; add publication functions to `src/connectors/microblog-clippings.ts`; extend `test/microblog-helpers.ts`; add `test/microblog-clippings-post.test.ts`. + +**Interfaces:** Consume Task 1's content function. Produce `publishMicroblogClippings(cred: Credential, match: Match, event: OutboundEvent, http: HttpTransport, ctx: ConnectorContext): Promise`; dispatch Micro.blog highlight jobs to it before shelf reconciliation. + +- [ ] Write failing transport tests for first creation returning `Location`, subsequent JSON Micropub update to the saved URL, no request for an unchanged fingerprint, and no shelf operations for highlight jobs. +- [ ] Add failing tests for 401/403 reauthentication, retryable 429/5xx, malformed or absent creation responses, ambiguous network completion, paginated source recovery finding a unique owned post, ambiguous recovery requiring attention, and a missing saved post requiring attention rather than recreation. Verify automatic-off updates retain snapshots of published passages. +- [ ] Run `npm test -- test/microblog-clippings-post.test.ts`; verify expected failures. +- [ ] Implement JSON Micropub requests without the current form-only helper overriding content type. Persist the creation URL before marking success. Save a stable recovery identity before creation, query the account's Micropub source with pagination after uncertain outcomes, and adopt only one unambiguously identified result; otherwise fail with an actionable message and do not create again. +- [ ] Guard `shouldPush` so highlight events do not inherit percentage filtering. In the runner, Micro.blog highlights require an existing match and never invoke `createBook`; preserve other connectors' lifecycle behavior. Resolve ISBN/book-page identity from verified book metadata or the shelf inventory instead of assuming an internal shelf ID is an ISBN. +- [ ] Run new tests and existing Micro.blog unit/integration tests; commit the tested deliverable. + +### Task 3: Settings, manual API, and catch-up triggers + +**Files:** Modify `src/routes/v1/connectors.ts`, `src/routes/v1/clippings.ts`, `src/connectors/fanout.ts`, `src/connectors/microblog.ts`, and the shared progress refresh entry point `src/connectors/refresh.ts`; extend backend integration tests. + +**Interfaces:** Add `POST /api/v1/connectors/microblog/clippings/:document/:clippingId`, returning `{ queued: boolean, waiting_for_match: boolean }`. Consume Task 1's selection and scheduling functions. Add connector toggle `sync_highlights`, label **Automatically sync highlights**, default `false`. + +- [ ] Write failing endpoint tests for authenticated manual selection, missing/deleted clipping, cross-user requests, unlinked/disabled accounts, and idempotent repeated requests. Assert disabled automatic sync still allows manual publication intent. +- [ ] Write failing catch-up tests: select or automatically include clippings while unmatched, manually match the book, perform an unchanged device progress GET or clipping sync, drain the queue, and observe one collected post. Cover legacy and v1 progress paths, plugin **Sync now**, enabling/disabling the option, and zero shelf creation for unmatched highlights. +- [ ] Run the new integration tests and confirm expected failures. +- [ ] Register the default-off option; schedule automatic selection on clipping sync and option enable. Schedule eligible book jobs on shared progress refresh, clipping sync, and plugin backfill, even without changed clippings. Persist unmatched intent without dead-letter churn. Bypass generic per-clipping fanout/backfill for Micro.blog so the single per-book job is authoritative. +- [ ] Implement the manual endpoint with canonical resolution, ownership checks, configured/enabled-account checks, and durable selection before scheduling. On unlink, clear local publication credentials/state according to the existing unlink lifecycle while leaving remote posts intact; ensure a replacement account cannot inherit an old account's remote post URL. +- [ ] Run backend integration and existing connector/queue tests; commit the tested deliverable. + +### Task 4: Clippings action and final verification + +**Files:** Modify `app/src/Clippings.jsx`, `app/src/api.js`; add `app/src/Clippings.test.jsx`; update Micro.blog help copy where appropriate. + +**Interfaces:** Add `api.addMicroblogHighlight(session, document, clippingId)` calling Task 3's endpoint. Use existing connector listing and settings rendering. + +- [ ] Write failing UI tests for an action shown only with configured Micro.blog, selecting an individual clipping, waiting-for-match feedback, request failure feedback, and preventing repeated clicks while a request is pending. +- [ ] Run `npm --prefix app test -- src/Clippings.test.jsx`; confirm expected failures. +- [ ] Add **Add highlight to Micro.blog** to clipping cards, including the daily clipping. Explain successful scheduling or waiting for a book match without claiming the post was already published. Display that the optional sync publishes a managed post per book; preserve existing share/delete behavior. +- [ ] Run the UI tests, `npm test`, `npm run build`, `npm --prefix app test`, and `npm --prefix app run build`. Report any failures by name. Check the complete diff and obtain an independent code review; address verified findings with regression tests. +- [ ] Commit the tested implementation and report changed behavior, checks, and any remaining limitations. Do not publish real posts during development or deploy without a user request. diff --git a/docs/superpowers/plans/2026-10-28-microblog-destinations.md b/docs/superpowers/plans/2026-10-28-microblog-destinations.md new file mode 100644 index 00000000..db875a53 --- /dev/null +++ b/docs/superpowers/plans/2026-10-28-microblog-destinations.md @@ -0,0 +1,73 @@ +# Micro.blog Destinations and Recreation Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Select the blog for new clipping collections, keep existing collections on their original blogs, and explicitly recreate deleted posts. + +**Architecture:** A focused destination module reads and validates the token's available blogs. Persist each collection's destination independently of the plugin-wide choice, and route every Micropub request explicitly. Extend the existing clippings publication state with confirmed deletion and a guarded recreation operation; reuse the settings and action-sheet patterns. + +**Tech Stack:** Existing TypeScript/Hono/SQLite backend, React frontend, Vitest, injected HTTP transport. + +**Spec:** `docs/superpowers/specs/2026-10-28-microblog-destinations-design.md` + +## Global Constraints + +- **Post clippings to** chooses the destination only for new collections. +- Existing collections stay on their original blog, including subsequent clippings and explicit recreation. +- Title collection posts **📚 Clippings from {book title}** on creation, recreation, and next update. +- Require a selection when multiple blogs are available; automatically use the sole available blog. +- Do not move, copy, or delete existing collections when changing the selected blog. +- Never automatically recreate a deleted post or treat a network/authorization failure as confirmed deletion. +- Preserve manual/automatic eligibility, same-account token rotation, merged-book resolution, and per-user isolation. +- Continue in the existing `codex/microblog-clippings` worktree; retain Docker localhost support. + +## Review Focus + +- Micro.blog silently falls back when an invalid destination is supplied: validate availability before every publication/recreation (Task 1/2). +- Selection changes while a creation request is in flight must not retarget recovery or updates (Task 2). +- Legacy posts may use a custom domain or a blog URL prefix: identify a unique available destination or pause (Task 2). +- Repeated recreation requests must not duplicate a replacement after an uncertain creation response (Task 2). +- A token rotation/default-blog change must preserve collections whose original blog is still accessible (Task 1/2). + +### Task 1: Available blogs and validated selection + +**Files:** Create `src/connectors/microblog-destinations.ts` and `test/microblog-destinations.test.ts`; modify `src/connectors/microblog.ts`, `src/connectors/store.ts`, `src/routes/v1/connectors.ts`, and relevant Micro.blog transport helpers. + +**Interfaces:** Export `type MicroblogDestination = { uid: string; name: string; isDefault: boolean }`, `listMicroblogDestinations(cred: Credential, http: HttpTransport): Promise`, and `selectedMicroblogDestination(db: DB, userId: number): string | null`. Store the plugin choice as `highlight_blog` in account options. Add `GET /api/v1/connectors/microblog/destinations` returning `{ items, selected }` and `PUT /api/v1/connectors/microblog/destination` accepting `{ destination: string }`. + +- [ ] Write failing tests for several available blogs with no implicit selection, one available blog automatically chosen, duplicates/malformed configuration, saving an available UID, rejecting an unavailable UID, unlinked accounts, and another user's isolation. Assert selection changes leave existing post rows and snapshots unchanged. +- [ ] Add a failing token-rotation test where the default changes but the original blog remains available: preserve existing publication identity and selections. Validate that client-supplied destination metadata cannot be trusted. +- [ ] Run `npm test -- test/microblog-destinations.test.ts test/microblog.test.ts test/microblog-clippings-api.test.ts`; verify the new behavior fails first. +- [ ] Implement discovery using authenticated `/micropub?q=config`, HTTPS destination UIDs, and server-returned names. Select a sole blog without requiring a settings write. Persist only an explicitly validated choice; expose no token fields. Use the existing credential-entry transport guard for the new settings write. +- [ ] Extend validated credentials with server-discovered available destinations. Update relinking logic so a different default among still-accessible blogs does not wipe original collections. Retain conservative ownership recovery when destination discovery is unavailable; truly different accounts must not inherit another account's URLs. +- [ ] Re-run these tests plus existing connector-link tests; commit the tested task. + +### Task 2: Pinned publication destinations, deletion, and recreation + +**Files:** Create the next migration after the current sequence; modify `src/connectors/microblog-clippings.ts`, `src/routes/v1/connectors.ts`, and `src/connectors/runner.ts` as needed; extend `test/microblog-clippings-post.test.ts`, `test/microblog-clippings.test.ts`, and `test/microblog-clippings-api.test.ts`. + +**Interfaces:** Consume Task 1 discovery and choice functions. Extend `ClippingsPost` with `destination: string | null` and `missing: number`. Export `recreateMicroblogClippings(db: DB, userId: number, document: string, http: HttpTransport): Promise<{ queued: boolean }>`; add `GET /api/v1/connectors/microblog/clipping-posts` returning this user's post status by canonical document, and `POST /api/v1/connectors/microblog/clipping-posts/:document/recreate`. + +- [ ] Write failing publication tests for the **📚 Clippings from Test book** title, selected non-default blog on every Micropub request, sole-blog fallback, no creation without a multi-blog choice, changed selection leaving existing posts pinned, an in-flight selection change, and revoked access refusing to publish rather than falling back. +- [ ] Add failing legacy tests using an existing post URL: infer the unique matching available destination, prefer the longest valid blog URL prefix, and pause when no unique destination is provable. Test credential rotation with multiple accessible original destinations. +- [ ] Add failing deletion/recreation tests: a confirmed missing source pauses publication and retains clippings; 401/403/429/5xx do not set `missing`; explicit recreation verifies absence, stays on the original blog, includes saved and new eligible passages, and preserves normal uncertain-creation recovery. Repeated/concurrent requests create at most one replacement; an existing original post resumes updates without creating another. +- [ ] Run `npm test -- test/microblog-clippings-post.test.ts test/microblog-clippings-api.test.ts test/microblog-clippings.test.ts`; confirm new tests fail before implementation. +- [ ] Add destination/deletion state through a new migration. Pin the selected validated UID before the first creation attempt; preserve it through retries, merges, same-account relinks, and recreation. Infer legacy destinations only against the authenticated available list, using URL origin/path boundaries. Read the current state after asynchronous operations and retain account/identity race guards. +- [ ] Route every Micropub request with an explicit URL-encoded `mp-destination` query parameter, including source pagination, single-post lookup, creation, and updates. Revalidate that the pinned blog remains available before requests that could publish. Changing the plugin choice does not change a pinned UID. +- [ ] Distinguish a confirmed deleted post from permission/network errors using the documented source lookup response. Set `missing` only on confirmed absence, retain its old URL and destination, and expose the pause in status. Ordinary sync/manual additions cannot clear it. +- [ ] Implement recreation with account/ownership checks and compare-and-set identity protection after remote verification. For a confirmed missing original, reset creation state to a fresh recovery identity while preserving the pinned blog, selections, and snapshots, then enqueue one collection job. If the old post exists, clear only its pause and resume updates. Block recreation when ownership or deletion cannot be established. +- [ ] Run publication, merge, account-cleanup, and queue tests; commit the tested task. + +### Task 3: Settings selector, recreation action, and verification + +**Files:** Create `app/src/MicroblogPosting.jsx` and its test; modify `app/src/Settings.jsx`, `app/src/Clippings.jsx`, `app/src/Clippings.test.jsx`, `app/src/api.js`, README, and the legacy Micro.blog settings page in `src/routes/web.ts` if it remains user-accessible. + +**Interfaces:** Add `api.microblogDestinations(session)`, `api.setMicroblogDestination(session, uid)`, `api.microblogClippingPosts(session)`, and `api.recreateMicroblogPost(session, document)` for Task 1/2 endpoints. `MicroblogPosting({ session, onChange })` owns destination loading and saving. + +- [ ] Write failing UI tests for **Post clippings to**, sole-blog selection, a required choice with multiple blogs, saved-choice restoration, loading/saving errors, unavailable selected blogs, and preventing overlapping saves. Keep **Automatically sync clippings** and its default unchanged. +- [ ] Write failing clipping tests for a per-book **Recreate clipping post** action shown only for missing collections, the confirmation explaining publication on the original blog, cancel without requests, success/refetch, failure feedback, and disabled repeated clicks. +- [ ] Run `npm --prefix app test -- src/MicroblogPosting.test.jsx src/Clippings.test.jsx`; verify the new behaviors fail first. +- [ ] Implement the plugin selector using the existing app form style. Explain that it affects new collections while existing posts stay on their original blog. Include an unavailable saved value rather than silently displaying a different blog; show load/save errors and allow retry. +- [ ] Show the recreation action once per affected book on the clippings hub, with an action sheet confirming that eligible collected clippings will be republished to the original blog. Refresh post status after the request and report queuing without claiming publication is already complete. Wire the API methods and update setup documentation, including the emoji title. +- [ ] Run `npm test`, `npm run build`, `npm --prefix app test`, and `npm --prefix app run build`. Use bundled Node entrypoints if the execution shell lacks npm; report failures by name. Request one independent review of the full change, reproduce important findings with failing tests, and complete one fix pass. +- [ ] Commit the tested result on `codex/microblog-clippings`. Keep the worktree available for local Docker testing and report the new settings/recreation behavior, verification, and material limitations. Do not create real Micro.blog posts or alter the user's running container during development. diff --git a/docs/superpowers/specs/2026-10-28-microblog-clippings-post-design.md b/docs/superpowers/specs/2026-10-28-microblog-clippings-post-design.md new file mode 100644 index 00000000..a6e28951 --- /dev/null +++ b/docs/superpowers/specs/2026-10-28-microblog-clippings-post-design.md @@ -0,0 +1,88 @@ +# Micro.blog clippings posts + +## Intent and scope + +Let readers publish selected clippings to Micro.blog, or opt into automatic +publication, with one collected post per book. Each post links to the matched +book's Micro.blog page. This replaces the original bookmark-highlights proposal. +Existing reading-progress and bookshelf behavior continues independently. + +## Approaches + +- Bookmark highlights: originally requested, but requires offsets into archived + bookmark content and does not fit clippings from an ebook. +- A separate post per clipping: straightforward, but automatic sync would create + many posts for the same book. +- One post per book, updated as clippings arrive: selected by the user. Retains + a single shareable collection and avoids repeated publication per clipping. + +## Behavior + +- When Micro.blog is configured, each clipping on the clippings page offers + **Add highlight to Micro.blog**. It selects only that clipping for publication + and schedules creation or updating of the book's collected post. +- The plugin offers **Automatically sync highlights**, disabled by default. + Enabling it makes existing and future live clippings eligible for publication. +- A missing book match never blocks reader sync or creates a book solely to + publish clippings. Persist publication intent even while unmatched. Once a + match is available, the next device sync or plugin **Sync now** includes + eligible existing clippings, even when they were not changed on that sync. +- Manual selection works with automatic publication disabled. Disabling the + automatic option stops automatic additions; explicitly selected clippings + remain eligible. Already published content is retained. +- Successful unchanged syncs do not create posts or rewrite existing posts. + Normal transient failures are retried using the connector queue. + +## Implementation choices + +These details are proposed defaults for implementation: + +- Publish a normal blog post through Micro.blog's Micropub endpoint, using the + existing plugin credential. Save its returned URL and update that same URL. +- Title the post **Highlights from {book title}**. Include a book link, followed + by eligible passages as quotations with their notes and chapter labels. + Use a stable reading order so unchanged content has a stable fingerprint. +- Persist per-clipping publication intent separately from queue attempts, and + persist the post URL and last successful content fingerprint per user/book. + Resolve document aliases using the existing merge model. +- Use a per-book queue key so multiple new clippings coalesce into one update. + Read current eligible clippings when processing the job rather than publishing + an obsolete snapshot. Preserve selected content when automatic sync is disabled. +- Treat the collected post as managed by crosspoint-sync: later syncs replace its + generated body. Do not delete remote posts on unlink or local clipping deletion. +- Do not blindly retry a creation with an ambiguous outcome: recover the existing + post through Micropub source lookup before allowing another creation. If a saved + post is missing or cannot be recovered, surface an actionable error instead of + silently creating a duplicate. + +## Verification + +Cover default-off behavior, manual selection, automatic backfill, unmatched books, +manual matching followed by an otherwise unchanged sync, one post per book, +unchanged-content deduplication, updates, retry/recovery, per-user isolation, and +canonical document aliases. Run relevant backend tests, the backend build, and +frontend checks available in this repository. + +## Implementation notes + +- Quotes and notes are escaped HTML, so highlighted Markdown is displayed as + text rather than interpreted as new markup; reader typography may differ. +- Pending jobs preserve retry backoff. When content changes during a network + request, successful completion compares current content and queues a follow-up. +- Token validation discovers the default blog through Micropub configuration. + Rotating a token for the same verified blog preserves the post and selections. + A proven different blog resets local tracking and disables automatic sync. + If ownership is unknown, retained identities must be recovered through the new + token's source feed; switching blogs may require explicit unlink/reconnect. +- Missing ISBN errors can be retried after correcting the book match. A still + invalid match may produce a failed attempt on another sync. +- Deleting book/account data clears local publication records while retaining + remote posts. Reimporting a deleted book starts fresh tracking. +- Migration 0041 adds publication-link deduplication metadata separately so + databases already testing migration 0040 can upgrade normally. +- Verified with 820 backend tests, 232 frontend tests, and backend/frontend builds. + +## References + +- [Micro.blog Micropub creation and updates](https://microblog.dev/api/posting/) +- [Micro.blog Micropub guide](https://book.micro.blog/micropub/) diff --git a/docs/superpowers/specs/2026-10-28-microblog-destinations-design.md b/docs/superpowers/specs/2026-10-28-microblog-destinations-design.md new file mode 100644 index 00000000..25406077 --- /dev/null +++ b/docs/superpowers/specs/2026-10-28-microblog-destinations-design.md @@ -0,0 +1,81 @@ +# Micro.blog destination selection and post recreation + +## Intent + +A Micro.blog token may access several blogs. Let the user choose the blog for +new clipping collections, preserve existing collections where they were first +published, and provide an explicit way to resume publication after deleting a +remote collection post. + +## Approaches + +- Continue using the account's implicit default blog: rejected because a token + can access multiple blogs and the user needs to choose a destination. +- Choose a blog per clipping or per book: unnecessary complexity for this request. +- One plugin-wide choice for new collections, with existing collections pinned + to their original blog: the selected approach. The user also approved an + explicit **Recreate clipping post** action after remote deletion. + +## Agreed behavior + +- Title collected clipping posts **📚 Clippings from {book title}**, as requested + for Micro.blog discovery. Apply this to new and recreated posts and to existing + collection posts when they are next updated. +- Show **Post clippings to** under the linked Micro.blog plugin. Populate it + from the token's authenticated Micropub `q=config` destination list. +- Automatically select the sole available blog. When multiple blogs are + available, require a selection before creating a new collection post. +- Use the selected destination for both manually selected and automatically + included clippings. Bookshelves and book matching remain account-level. +- Once a book's collection post has been created, persist its destination. + Additional clippings continue updating that post on its original blog, + regardless of subsequent changes to the plugin-wide selection. +- Changing the selection never moves, copies, or deletes existing collections. +- If a collection post is deleted on Micro.blog, keep its clippings locally, + report the missing post, and pause publication for that book. Reader sync and + other books continue normally. Do not automatically recreate it. +- Offer **Recreate clippings post** for an affected book. Explicit recreation + offers its previous blog (default) or the current clippings blog. Create one + replacement collection and pin later updates to the chosen blog. Include its + eligible saved clippings and any new clippings. This destination choice was + requested after local testing on October 28. + +## Implementation choices + +- Send `mp-destination` explicitly on all creation, source lookup, recovery, and + update requests. Validate it against destinations returned for the token; + do not rely on Micro.blog's fallback to its default when a destination is invalid. +- Store the original blog alongside each collection's identity and post URL. + For previously created posts, infer the original destination only from a + matching available blog URL. If it cannot be identified uniquely, pause that + collection with an actionable error rather than choose a different blog. +- Pin the destination before the first creation attempt so retries and uncertain + creation recovery cannot be redirected by a concurrent settings change. +- Recreation requires a confirmed missing post and a currently accessible target + blog. When the previous blog is accessible, recheck its post; if restored, only + resume updates there rather than create a duplicate elsewhere. With an existing + confirmed deletion, allow explicit recreation on the current blog after access + to the previous blog is revoked. Network and authorization failures during a + source recheck still stop recreation. +- Keep ordinary **Sync now** and **Add clipping to Micro.blog** from silently + clearing a missing-post pause. Use the existing action-sheet pattern to explain + that recreation republishes the collected clippings to the chosen blog. Show + addresses for both choices; disable the current choice if no available blog is + selected and show only the previous choice when both destinations are the same. +- Refresh destination choices from the linked token without exposing credentials; + preserve the saved choice across same-account token rotation when accessible. + +## Verification + +Test the 📚 title prefix on creation, recreation, and updates. Test selection +among several blogs, sole-blog selection, invalid/revoked +destinations, manual and automatic posting, explicit destination on every request, +changing the choice while creation is in flight, existing collections staying on +their original blog, legacy post identification, same-account token rotation, +remote deletion, explicit recreation, duplicate prevention, and per-user access. +Verify the settings selector and recreation action in frontend tests. Run the +backend and frontend suites and builds before reporting completion. + +## Reference + +[Micro.blog Micropub destinations and source queries](https://microblog.dev/api/posting/) diff --git a/migrations/0040_microblog_clippings.sql b/migrations/0040_microblog_clippings.sql new file mode 100644 index 00000000..77f8e394 --- /dev/null +++ b/migrations/0040_microblog_clippings.sql @@ -0,0 +1,18 @@ +CREATE TABLE microblog_clipping_selections ( + user_id INTEGER NOT NULL REFERENCES users(id) ON DELETE CASCADE, + document TEXT NOT NULL, + clipping_id TEXT NOT NULL, + manual INTEGER NOT NULL DEFAULT 0, + published_snapshot TEXT, + PRIMARY KEY (user_id, document, clipping_id) +); +CREATE TABLE microblog_clipping_posts ( + user_id INTEGER NOT NULL REFERENCES users(id) ON DELETE CASCADE, + document TEXT NOT NULL, + identity TEXT NOT NULL, + url TEXT, + fingerprint TEXT, + attempted INTEGER NOT NULL DEFAULT 0, + error TEXT, + PRIMARY KEY (user_id, document) +); diff --git a/migrations/0041_microblog_clippings_destination.sql b/migrations/0041_microblog_clippings_destination.sql new file mode 100644 index 00000000..e6037306 --- /dev/null +++ b/migrations/0041_microblog_clippings_destination.sql @@ -0,0 +1,2 @@ +ALTER TABLE microblog_clipping_posts ADD COLUMN book_url TEXT; +ALTER TABLE microblog_clipping_posts ADD COLUMN matched_book TEXT; diff --git a/migrations/0042_microblog_post_destinations.sql b/migrations/0042_microblog_post_destinations.sql new file mode 100644 index 00000000..a3279a55 --- /dev/null +++ b/migrations/0042_microblog_post_destinations.sql @@ -0,0 +1,2 @@ +ALTER TABLE microblog_clipping_posts ADD COLUMN destination TEXT; +ALTER TABLE microblog_clipping_posts ADD COLUMN missing INTEGER NOT NULL DEFAULT 0; diff --git a/src/app.ts b/src/app.ts index 1a174ec5..0ee84607 100644 --- a/src/app.ts +++ b/src/app.ts @@ -89,7 +89,7 @@ export function createApp(db: DB, config: Config, opts: AppOptions = {}): Hono o.trim()).filter(Boolean) : '*', diff --git a/src/connectors/fanin.ts b/src/connectors/fanin.ts index d4521718..f36c5bef 100644 --- a/src/connectors/fanin.ts +++ b/src/connectors/fanin.ts @@ -3,6 +3,7 @@ import { withTransaction, type DB } from '../db/db.js'; import { nowSeconds } from '../models/sync.js'; import { getConnector, fetchTransport } from './registry.js'; import { fanOutHighlight, fanOutProgress } from './fanout.js'; +import { scheduleMicroblogClippings } from './microblog-clippings.js'; import { clippingDocuments } from '../models/merge.js'; import { nearestProgressSample, recordProgressSample, upsertProgress } from '../routes/kosync.js'; import { @@ -198,6 +199,14 @@ function importHighlights(db: DB, userId: number, document: string, items: Inbou endOffset: h.endOffset, chapter: h.chapter, }, now, source); } + if (added.length) { + try { + scheduleMicroblogClippings(db, userId, document); + } catch (err) { + console.error(JSON.stringify({ msg: 'microblog schedule failed', user_id: userId, document, + error: err instanceof Error ? err.message : String(err) })); + } + } } /** Poll library-wide providers; per-book providers refresh on progress requests. */ diff --git a/src/connectors/fanout.ts b/src/connectors/fanout.ts index b6e5fbe3..47451dbd 100644 --- a/src/connectors/fanout.ts +++ b/src/connectors/fanout.ts @@ -6,6 +6,7 @@ import { activeConnectorIds, carriesFor, documentMeta, getMatch } from './store. import { extractTitleAuthor } from './matching.js'; import { nowSeconds } from '../models/sync.js'; import type { Connector, OutboundEvent } from './types.js'; +import { scheduleMicroblogClippings } from './microblog-clippings.js'; /** * Whether it's worth attempting a document on this connector. Metadata-matched @@ -40,6 +41,7 @@ function fanOut( if (!secretsEnabled()) return; try { for (const connectorId of activeConnectorIds(db, userId)) { + if (connectorId === 'microblog' && ev.kind === 'highlight') continue; if (connectorId === exceptConnectorId) continue; // loop suppression: don't echo to the source const conn = getConnector(connectorId); if (!conn || !conn.capabilities.write || !carriesFor(db, userId, conn, ev.kind)) continue; @@ -63,6 +65,12 @@ export function fanOutProgress( positionJson?: string | null, exceptConnectorId?: string ): void { + try { + scheduleMicroblogClippings(db, userId, document); + } catch (err) { + console.error(JSON.stringify({ msg: 'microblog schedule failed', user_id: userId, document, + error: err instanceof Error ? err.message : String(err) })); + } const finished = percentage >= 0.98; let position: Record | null = null; if (positionJson) { @@ -191,7 +199,8 @@ export function backfillConnector(db: DB, userId: number, connectorId: string, d } } - if (carriesFor(db, userId, conn, 'highlight')) { + if (connectorId === 'microblog') queued += scheduleMicroblogClippings(db, userId, document); + if (connectorId !== 'microblog' && carriesFor(db, userId, conn, 'highlight')) { const rows = db .prepare( `SELECT c.id, c.document, c.text, c.note, c.created_at, c.spine_index, c.start_offset, c.end_offset, diff --git a/src/connectors/microblog-clippings.ts b/src/connectors/microblog-clippings.ts new file mode 100644 index 00000000..2018ad10 --- /dev/null +++ b/src/connectors/microblog-clippings.ts @@ -0,0 +1,418 @@ +import { createHash, randomUUID } from 'node:crypto'; +import { withTransaction, type DB } from '../db/db.js'; +import { clippingDocuments, mergedClippingsCte, resolveDocument } from '../models/merge.js'; +import { decryptCredential, documentMeta, getAccount, getMatch } from './store.js'; +import { enqueue } from './queue.js'; +import { nowSeconds } from '../models/sync.js'; +import { destinationForPost, listMicroblogDestinations, selectedMicroblogDestination } from './microblog-destinations.js'; +import { ConnectorOperationError, type Credential, type Match, type OutboundEvent, type HttpTransport, type ConnectorContext, type PushResult } from './types.js'; + +export interface ClippingsPost { + identity: string; + url: string | null; + fingerprint: string | null; + book_url: string | null; + matched_book: string | null; + attempted: number; + error: string | null; + destination: string | null; + missing: number; +} +interface Passage { + id: string; text: string; note: string | null; chapter_title: string; + spine_index: number; start_offset: number | null; created_at: number; + // Older published snapshots predate these fallback coordinates. + paragraph_index?: number | null; start_page?: number; start_word?: number; +} + +function automatic(db: DB, userId: number): boolean { + const row = db.prepare("SELECT options FROM connector_accounts WHERE user_id=? AND connector_id='microblog'") + .get(userId) as { options: string | null } | undefined; + try { return JSON.parse(row?.options || '{}').sync_highlights === true; } catch { return false; } +} + +function liveClippings(db: DB, userId: number, document: string): Passage[] { + const docs = clippingDocuments(db, userId, document); + return db.prepare(`${mergedClippingsCte(docs.length)} SELECT id,text,note,chapter_title,spine_index,start_offset,paragraph_index,start_page,start_word,created_at + FROM merged WHERE document=? AND deleted=0`).all(userId, ...docs, document) as unknown as Passage[]; +} + +export function clippingsPost(db: DB, userId: number, document: string): ClippingsPost { + db.prepare('INSERT OR IGNORE INTO microblog_clipping_posts(user_id,document,identity) VALUES(?,?,?)') + .run(userId, document, randomUUID()); + return db.prepare('SELECT identity,url,fingerprint,book_url,matched_book,attempted,error,destination,missing FROM microblog_clipping_posts WHERE user_id=? AND document=?') + .get(userId, document) as unknown as ClippingsPost; +} + +export function selectMicroblogClipping(db: DB, userId: number, document: string, clippingId: string): void { + document = resolveDocument(db, userId, document); + if (!liveClippings(db, userId, document).some((c) => c.id === clippingId)) throw new Error('Clipping not found'); + db.prepare(`INSERT INTO microblog_clipping_selections(user_id,document,clipping_id,manual) VALUES(?,?,?,1) + ON CONFLICT(user_id,document,clipping_id) DO UPDATE SET manual=1`).run(userId, document, clippingId); +} + +/** Include published snapshots even after deletion or disabling automatic additions. */ +function passages(db: DB, userId: number, document: string): Passage[] { + const live = liveClippings(db, userId, document); + const enabled = automatic(db, userId); + const selections = db.prepare('SELECT clipping_id,manual,published_snapshot FROM microblog_clipping_selections WHERE user_id=? AND document=?') + .all(userId, document) as { clipping_id: string; manual: number; published_snapshot: string | null }[]; + const selected = selections.flatMap((s) => { + const current = live.find((c) => c.id === s.clipping_id); + if ((enabled || s.manual) && current) return [current]; + return s.published_snapshot ? [JSON.parse(s.published_snapshot) as Passage] : []; + }); + const coordinates = ['start_offset', 'paragraph_index', 'start_page', 'start_word'] as const; + const common = new Map(); + // Use the same coordinate system throughout a spine. Mixed client versions + // must not compare a character offset to a paragraph or page number. + for (const c of selected) { + common.set(c.spine_index, (common.get(c.spine_index) ?? coordinates).filter((key) => c[key] != null)); + } + return selected.sort((a, b) => { + if (a.spine_index !== b.spine_index) return a.spine_index - b.spine_index; + for (const key of common.get(a.spine_index)!) { + const difference = a[key]! - b[key]!; + if (difference) return difference; + } + return a.created_at - b.created_at || a.id.localeCompare(b.id); + }); +} + +function escapeHtml(s: string): string { + return s.replace(/&/g, '&').replace(//g, '>').replace(/"/g, '"'); +} +function paragraph(s: string): string { return escapeHtml(s).replace(/\n/g, '
'); } + +export function microblogClippingsContent(db: DB, userId: number, document: string, bookUrl: string): { title: string; content: string; fingerprint: string } | null { + document = resolveDocument(db, userId, document); + const selected = passages(db, userId, document); + if (!selected.length) return null; + const meta = documentMeta(db, userId, document); + const title = `📚 Clippings from ${meta.title || meta.filename || 'Untitled book'}`; + const post = clippingsPost(db, userId, document); + const content = [`

${escapeHtml(meta.title || 'View book on Micro.blog')}${meta.author ? ` by ${escapeHtml(meta.author)}` : ''}

`, + ...selected.map((c) => `

${paragraph(c.text)}

${c.note ? `

${paragraph(c.note)}

` : ''}${c.chapter_title ? `

${paragraph(c.chapter_title)}

` : ''}`), + ``].join('\n\n'); + return { title, content, fingerprint: createHash('sha256').update(JSON.stringify([title, content])).digest('hex') }; +} + +export function savePublishedClippings(db: DB, userId: number, document: string, selected: Passage[]): void { + for (const c of selected) db.prepare('UPDATE microblog_clipping_selections SET published_snapshot=? WHERE user_id=? AND document=? AND clipping_id=?') + .run(JSON.stringify(c), userId, document, c.id); +} + +export function scheduleMicroblogClippings(db: DB, userId: number, document?: string): number { + const account = getAccount(db, userId, 'microblog'); + if (!account?.enabled) return 0; + const docs = document ? [resolveDocument(db, userId, document)] : [...new Set((db.prepare(` + SELECT document FROM clippings WHERE user_id=? UNION SELECT document FROM microblog_clipping_selections WHERE user_id=?`) + .all(userId, userId) as { document: string }[]).map((r) => resolveDocument(db, userId, r.document)))]; + let queued = 0; + for (const doc of docs) { + if (automatic(db, userId)) { + const insert = db.prepare('INSERT OR IGNORE INTO microblog_clipping_selections(user_id,document,clipping_id) VALUES(?,?,?)'); + for (const c of liveClippings(db, userId, doc)) insert.run(userId, doc, c.id); + } + const blocked = db.prepare('SELECT error FROM microblog_clipping_posts WHERE user_id=? AND document=?') + .get(userId, doc) as { error: string | null } | undefined; + if (blocked?.error) { + // A merge can pause publication without making a delivery attempt. + // Surface durable errors in Settings even if this book is now unmatched. + withTransaction(db, () => { + enqueue(db, userId, 'microblog', { kind: 'highlight', document: doc, timestamp: nowSeconds() }, 'microblog:clippings'); + db.prepare("UPDATE connector_queue SET status='dead',last_error=? WHERE user_id=? AND connector_id='microblog' AND document=? AND kind='microblog:clippings'") + .run(blocked.error, userId, doc); + }); + continue; + } + const match = getMatch(db, userId, 'microblog', doc); + if (!match?.external_id || !passages(db, userId, doc).length) continue; + const state = clippingsPost(db, userId, doc); + if (state.error) continue; + const isbn = match.external_edition ?? (/^(?:\d{13}|\d{9}[\dXx])$/.test(match.external_id) ? match.external_id : null); + const bookUrl = isbn ? `https://micro.blog/books/${encodeURIComponent(isbn)}` : state.book_url; + if (state.url && bookUrl && state.matched_book === match.external_id + && state.fingerprint === microblogClippingsContent(db, userId, doc, bookUrl)?.fingerprint) { + // Removing rejected unpublished content can restore the last successful + // post without another network update. Its old validation error is stale. + db.prepare(`UPDATE connector_queue SET status='done',last_error=NULL,updated_at=? + WHERE user_id=? AND connector_id='microblog' AND document=? AND kind='microblog:clippings' AND status='dead'`) + .run(nowSeconds(), userId, doc); + continue; + } + const existing = db.prepare("SELECT status FROM connector_queue WHERE user_id=? AND connector_id='microblog' AND document=? AND kind='microblog:clippings'") + .get(userId, doc) as { status: string } | undefined; + if (existing?.status === 'pending') continue; // Preserve backoff after a transient failure. + enqueue(db, userId, 'microblog', { kind: 'highlight', document: doc, timestamp: nowSeconds() }, 'microblog:clippings'); + queued++; + } + return queued; +} + +/** Migrate publication state before the ordinary document side tables move. */ +export function mergeMicroblogClippings(db: DB, userId: number, from: string, into: string): void { + const source = db.prepare('SELECT * FROM microblog_clipping_posts WHERE user_id=? AND document=?').get(userId, from) as unknown as ClippingsPost | undefined; + const target = db.prepare('SELECT * FROM microblog_clipping_posts WHERE user_id=? AND document=?').get(userId, into) as unknown as ClippingsPost | undefined; + if (source && target && (source.url || source.attempted) && (target.url || target.attempted)) { + db.prepare('UPDATE microblog_clipping_posts SET error=? WHERE user_id=? AND document=?') + .run('Merged books both have Micro.blog publication history; review their posts before syncing.', userId, into); + } else if (source && (!target || source.url || source.attempted)) { + db.prepare('DELETE FROM microblog_clipping_posts WHERE user_id=? AND document=?').run(userId, into); + db.prepare('UPDATE microblog_clipping_posts SET document=? WHERE user_id=? AND document=?').run(into, userId, from); + } + db.prepare('DELETE FROM microblog_clipping_posts WHERE user_id=? AND document=?').run(userId, from); + db.prepare(`INSERT INTO microblog_clipping_selections(user_id,document,clipping_id,manual,published_snapshot) + SELECT user_id,?,clipping_id,manual,published_snapshot FROM microblog_clipping_selections WHERE user_id=? AND document=? + ON CONFLICT(user_id,document,clipping_id) DO UPDATE SET manual=MAX(manual,excluded.manual), + published_snapshot=COALESCE(published_snapshot,excluded.published_snapshot)`).run(into, userId, from); + db.prepare('DELETE FROM microblog_clipping_selections WHERE user_id=? AND document=?').run(userId, from); +} + +/** Only publish/recover one book at a time, even when two drains overlap. */ +const publishing = new WeakMap>(); + +class MissingMicroblogPost extends ConnectorOperationError { + constructor() { super('The Micro.blog clippings post was deleted. Use Recreate clippings post to resume publishing.', false); } +} + +class MicropubResponseError extends ConnectorOperationError { + constructor(readonly status: number, operation: string) { + super(`Micro.blog Micropub ${operation} failed (${status})`, status === 408 || status === 429 || status >= 500, + status === 401 || status === 403); + } +} + +async function micropub(http: HttpTransport, token: string, destination: string, path: string, body?: unknown, sourceLookup = false) { + const operation = body ? (body as { action?: string }).action === 'update' ? 'update' : 'create' + : sourceLookup ? 'source lookup' : 'post recovery'; + let res; + try { + const url = new URL(path, 'https://micro.blog'); + url.searchParams.set('mp-destination', destination); + res = await http(url.href, { + method: body ? 'POST' : 'GET', + headers: { authorization: `Bearer ${token}`, ...(body ? { 'content-type': 'application/json' } : {}) }, + ...(body ? { body: JSON.stringify(body) } : {}), + }); + } catch { throw new ConnectorOperationError(`Micro.blog Micropub ${operation} connection failed; the next attempt will check for an existing post.`, true); } + const existingPost = sourceLookup || operation === 'update'; + if (existingPost && res.status === 404) throw new MissingMicroblogPost(); + if (existingPost && res.status === 400) { + const failure = await res.json().catch(() => null) as { error?: string; error_description?: string } | null; + if (failure?.error === 'invalid_request' && /no post|not found|does not exist|could not find|unknown.*url/i.test(failure.error_description || '')) throw new MissingMicroblogPost(); + } + if (res.status >= 300) { + throw new MicropubResponseError(res.status, operation); + } + return res; +} + +function postUrl(value: unknown): string | null { + if (typeof value !== 'string') return null; + try { const u = new URL(value); return u.protocol === 'https:' && !u.username && !u.password ? u.href : null; } catch { return null; } +} + +async function recoverPost(http: HttpTransport, token: string, destination: string, identity: string): Promise { + const matches = new Set(); + // Bound the scan; an incomplete scan never permits creating a replacement. + for (let offset = 0; offset < 10000; offset += 100) { + const res = await micropub(http, token, destination, `/micropub?q=source&limit=100&offset=${offset}`); + const payload = await res.json() as { items?: { properties?: { content?: unknown[]; url?: unknown[] } }[] }; + if (!Array.isArray(payload.items)) throw new ConnectorOperationError('Micro.blog post recovery returned an invalid source feed.', true); + for (const item of payload.items) { + const content = JSON.stringify(item.properties?.content ?? []); + if (!content.includes(`crosspoint-sync:${identity}`)) continue; + const url = postUrl(item.properties?.url?.[0]); + if (url) matches.add(url); + } + if (payload.items.length < 100) return matches.size === 1 ? [...matches][0] : null; + } + return null; +} + +export async function publishMicroblogClippings( + cred: Credential, match: Match, event: OutboundEvent, http: HttpTransport, ctx: ConnectorContext +): Promise { + const { db, userId } = ctx; + const document = resolveDocument(db, userId, event.document); + const key = `${userId}:${document}`; + let locks = publishing.get(db); + if (!locks) { locks = new Set(); publishing.set(db, locks); } + if (locks.has(key)) return { ok: false, retryable: true, error: 'Micro.blog book publication is already in progress' }; + const account = getAccount(db, userId, 'microblog'); + const initialMatch = getMatch(db, userId, 'microblog', document); + let identity: string | undefined; + const assertCurrent = () => { + const current = getAccount(db, userId, 'microblog'); + const saved = db.prepare('SELECT identity,error FROM microblog_clipping_posts WHERE user_id=? AND document=?').get(userId, document) as { identity: string; error: string | null } | undefined; + if (!account?.enabled || !current?.enabled || current.cred_enc !== account.cred_enc || (identity && saved?.identity !== identity)) { + throw new ConnectorOperationError('Micro.blog account or book changed during publication', false); + } + // A merge can pause this collection without changing its identity. An older + // in-flight attempt must not acknowledge the write or clear that new error. + if (identity && saved?.error) throw new ConnectorOperationError(saved.error, false); + const currentMatch = getMatch(db, userId, 'microblog', document); + // The ISBN may have been enriched from the shelf without changing the saved + // match. Compare saved editions as well as the match passed by the caller. + if (!initialMatch?.external_id || initialMatch.external_id !== match.externalId || + (initialMatch.external_edition && initialMatch.external_edition !== match.externalEdition) || + currentMatch?.external_id !== initialMatch.external_id || currentMatch?.external_edition !== initialMatch.external_edition) { + throw new ConnectorOperationError('Micro.blog book match changed during publication; sync again with the current match.', false); + } + }; + const permanent = (error: string): PushResult => { + assertCurrent(); + db.prepare('UPDATE microblog_clipping_posts SET error=? WHERE user_id=? AND document=?').run(error, userId, document); + return { ok: false, retryable: false, error }; + }; + locks.add(key); + try { + const token = typeof cred.token === 'string' ? cred.token.trim() : ''; + if (!token) throw new ConnectorOperationError('Missing Micro.blog token', false, true); + assertCurrent(); + if (decryptCredential(account!).token !== cred.token) throw new ConnectorOperationError('Micro.blog credential changed before publication', false); + // Caller resolves a verified ISBN; internal bookshelf IDs are never used as ISBNs. + const isbn = match.externalEdition; + if (!isbn || !/^(?:\d{13}|\d{9}[\dXx])$/.test(isbn)) return { ok: false, retryable: false, error: 'Micro.blog book ISBN is unavailable; choose a book match with an ISBN.' }; + const state = clippingsPost(db, userId, document); + identity = state.identity; + if (state.error) return { ok: false, retryable: false, error: state.error }; + const selectedBlog = selectedMicroblogDestination(db, userId); + const blogs = await listMicroblogDestinations(cred, http); + assertCurrent(); + const destination = state.destination ?? (state.url ? destinationForPost(state.url, blogs) + : state.attempted ? null : selectedBlog ?? (blogs.length === 1 ? blogs[0].uid : null)); + if (!destination || !blogs.some((b) => b.uid === destination)) return { + ok: false, retryable: false, error: state.url || state.attempted || state.destination + ? 'The original Micro.blog blog is unavailable or cannot be identified. Restore access to that blog before syncing.' + : 'Choose Post clippings to in Micro.blog settings before publishing.', + }; + const selected = passages(db, userId, document); + const rendered = microblogClippingsContent(db, userId, document, `https://micro.blog/books/${encodeURIComponent(isbn)}`); + if (!rendered) return { ok: true }; + db.prepare('UPDATE microblog_clipping_posts SET destination=? WHERE user_id=? AND document=?').run(destination, userId, document); + if (rendered.fingerprint === state.fingerprint && state.url) { + db.prepare('UPDATE microblog_clipping_posts SET book_url=?,matched_book=? WHERE user_id=? AND document=?') + .run(`https://micro.blog/books/${encodeURIComponent(isbn)}`, match.externalId, userId, document); + return { ok: true }; + } + let url = state.url; + if (!url && state.attempted) { + url = await recoverPost(http, token, destination, state.identity); + assertCurrent(); + if (!url) return permanent('Micro.blog post creation could not be resolved uniquely. Review your blog posts before retrying; no duplicate was created. If switching blogs, unlink Micro.blog and connect the new blog.'); + db.prepare('UPDATE microblog_clipping_posts SET url=? WHERE user_id=? AND document=?').run(url, userId, document); + } + if (url) { + try { + // Detect deleted posts before replacing generated content. + await micropub(http, token, destination, `/micropub?q=source&url=${encodeURIComponent(url)}`, undefined, true); + assertCurrent(); + // Options and selections can change while the source request is in + // flight. Retry with fresh content before publishing or snapshotting it. + if (microblogClippingsContent(db, userId, document, `https://micro.blog/books/${encodeURIComponent(isbn)}`)?.fingerprint !== rendered.fingerprint) { + throw new ConnectorOperationError('Micro.blog clipping selections changed during publication; the next attempt will use the current selections.', true); + } + await micropub(http, token, destination, '/micropub', { action: 'update', url, + replace: { name: [rendered.title], content: [rendered.content] } }); + assertCurrent(); + } catch (err) { + assertCurrent(); + if (err instanceof MissingMicroblogPost) { + db.prepare('UPDATE microblog_clipping_posts SET missing=1 WHERE user_id=? AND document=?').run(userId, document); + return permanent(err.message); + } + if (err instanceof ConnectorOperationError && !err.retryable && !err.needsReauth) return { + ok: false, retryable: false, error: `${err.message}. Correct the clippings and sync this book again.`, + }; + throw err; + } + } else { + db.prepare('UPDATE microblog_clipping_posts SET attempted=1 WHERE user_id=? AND document=?').run(userId, document); + let res; + try { + res = await micropub(http, token, destination, '/micropub', { type: ['h-entry'], properties: { + name: [rendered.title], content: [{ html: rendered.content }], 'post-status': ['published'], + 'mp-slug': [`crosspoint-clippings-${state.identity}`], + } }); + } catch (err) { + assertCurrent(); + // Only known request rejections prove that no post was created. + if (err instanceof MicropubResponseError && [400, 401, 403, 413, 415, 422, 429].includes(err.status)) { + db.prepare('UPDATE microblog_clipping_posts SET attempted=0 WHERE user_id=? AND document=?').run(userId, document); + if (!err.retryable && !err.needsReauth) return { + ok: false, retryable: false, error: `${err.message}. Correct the clippings and sync this book again.`, + }; + } else if (err instanceof ConnectorOperationError && !err.retryable) return permanent(err.message); + throw err; + } + assertCurrent(); + url = postUrl(res.headers?.get('location')); + if (!url) { + throw new ConnectorOperationError('Micro.blog accepted the post without its URL; the next attempt will recover it.', true); + } + // Persist Location immediately, before any other work can fail. + db.prepare('UPDATE microblog_clipping_posts SET url=? WHERE user_id=? AND document=?').run(url, userId, document); + } + withTransaction(db, () => { + savePublishedClippings(db, userId, document, selected); + db.prepare('UPDATE microblog_clipping_posts SET fingerprint=?,book_url=?,matched_book=?,error=NULL WHERE user_id=? AND document=?') + .run(rendered.fingerprint, `https://micro.blog/books/${encodeURIComponent(isbn)}`, match.externalId, userId, document); + }); + return { ok: true }; + } finally { locks.delete(key); } +} + +export async function recreateMicroblogClippings(db: DB, userId: number, doc: string, http: HttpTransport, target: 'original' | 'current' = 'original'): Promise<{ queued: boolean }> { + const document = resolveDocument(db, userId, doc); + const account = getAccount(db, userId, 'microblog'); + if (!account?.enabled) throw new ConnectorOperationError('Micro.blog is not linked or enabled', false); + const state = db.prepare('SELECT * FROM microblog_clipping_posts WHERE user_id=? AND document=?').get(userId, document) as unknown as ClippingsPost | undefined; + if (!state?.missing || !state.url) throw new ConnectorOperationError('This book does not have a confirmed missing clippings post.', false); + const requireMatch = () => { + if (!getMatch(db, userId, 'microblog', document)?.external_id) { + throw new ConnectorOperationError('Match this book in Micro.blog before recreating this post.', false); + } + }; + requireMatch(); + let locks = publishing.get(db); if (!locks) { locks = new Set(); publishing.set(db, locks); } + const key = `${userId}:${document}`; + if (locks.has(key)) throw new ConnectorOperationError('Micro.blog publication is already in progress', true); + locks.add(key); + const current = () => { + const row = db.prepare('SELECT identity FROM microblog_clipping_posts WHERE user_id=? AND document=?').get(userId, document) as { identity: string } | undefined; + if (getAccount(db, userId, 'microblog')?.cred_enc !== account.cred_enc || row?.identity !== state.identity) throw new ConnectorOperationError('Micro.blog account or book changed; reload before retrying.', false); + requireMatch(); + }; + try { + const cred = decryptCredential(account, db); + const blogs = await listMicroblogDestinations(cred, http); current(); + const original = state.destination ?? destinationForPost(state.url, blogs); + const destination = target === 'current' + ? selectedMicroblogDestination(db, userId) ?? (blogs.length === 1 ? blogs[0].uid : null) + : original; + if (!destination || !blogs.some((b) => b.uid === destination)) throw new ConnectorOperationError(target === 'current' + ? 'Choose an available clippings blog in Micro.blog settings before recreating this post.' + : 'Restore access to the original Micro.blog blog or choose the current clippings blog.', false); + // The durable missing flag permits an explicit move after original-blog access is lost. + // When accessible, recheck the original to avoid duplicating a restored post. + let missing = true; + if (original && blogs.some((b) => b.uid === original)) { + try { + await micropub(http, String(cred.token), original, `/micropub?q=source&url=${encodeURIComponent(state.url)}`, undefined, true); + missing = false; + } catch (error) { if (!(error instanceof MissingMicroblogPost)) throw error; } + } + current(); + if (!missing && destination !== original) throw new ConnectorOperationError('The previous post exists again. Choose the previous blog to resume updating it.', false); + withTransaction(db, () => { + if (missing) db.prepare(`UPDATE microblog_clipping_posts SET identity=?,url=NULL,fingerprint=NULL,attempted=0,error=NULL,missing=0,destination=? WHERE user_id=? AND document=?`) + .run(randomUUID(), destination, userId, document); + else db.prepare('UPDATE microblog_clipping_posts SET fingerprint=NULL,error=NULL,missing=0,destination=? WHERE user_id=? AND document=?').run(destination, userId, document); + // A previous missing-post job may still be pending in a concurrent drain. + db.prepare("DELETE FROM connector_queue WHERE user_id=? AND document=? AND connector_id='microblog' AND kind='microblog:clippings'").run(userId, document); + }); + } finally { locks.delete(key); } + return { queued: scheduleMicroblogClippings(db, userId, document) > 0 }; +} diff --git a/src/connectors/microblog-destinations.ts b/src/connectors/microblog-destinations.ts new file mode 100644 index 00000000..d0c1b212 --- /dev/null +++ b/src/connectors/microblog-destinations.ts @@ -0,0 +1,46 @@ +import type { DB } from '../db/db.js'; +import { ConnectorOperationError, type Credential, type HttpTransport } from './types.js'; + +export interface MicroblogDestination { uid: string; name: string; isDefault: boolean } + +export async function listMicroblogDestinations(cred: Credential, http: HttpTransport): Promise { + if (typeof cred.token !== 'string' || !cred.token.trim()) throw new ConnectorOperationError('Missing Micro.blog token', false, true); + let res; + try { res = await http('https://micro.blog/micropub?q=config', { method: 'GET', headers: { authorization: `Bearer ${cred.token.trim()}` } }); } + catch { throw new ConnectorOperationError('Could not load Micro.blog blogs', true); } + if (res.status >= 300) throw new ConnectorOperationError(`Micro.blog blog discovery failed (${res.status})`, res.status === 429 || res.status >= 500, res.status === 401 || res.status === 403); + let config: { destination?: unknown }; + try { config = await res.json() as typeof config; } catch { throw new ConnectorOperationError('Micro.blog returned invalid blog configuration', true); } + if (!config || !Array.isArray(config.destination)) throw new ConnectorOperationError('Micro.blog did not return available blogs', true); + const byUid = new Map(); + for (const item of config.destination) { + if (!item || typeof item !== 'object') continue; + const d = item as { uid?: unknown; name?: unknown; 'microblog-default'?: unknown }; + if (typeof d.uid !== 'string') continue; + try { + const url = new URL(d.uid); + if (url.protocol !== 'https:' || url.username || url.password || url.search || url.hash) continue; + const uid = url.href; + if (!byUid.has(uid)) byUid.set(uid, { uid, name: typeof d.name === 'string' && d.name.trim() ? d.name : uid, isDefault: d['microblog-default'] === true }); + } catch { /* Ignore malformed destination entries. */ } + } + return [...byUid.values()]; +} + +export function selectedMicroblogDestination(db: DB, userId: number): string | null { + const account = db.prepare("SELECT options FROM connector_accounts WHERE user_id=? AND connector_id='microblog'").get(userId) as { options: string | null } | undefined; + try { const value = JSON.parse(account?.options || '{}').highlight_blog; return typeof value === 'string' && value ? value : null; } + catch { return null; } +} + +/** URL origin and path boundaries prevent similar hostnames/prefixes matching. */ +export function destinationForPost(url: string, blogs: MicroblogDestination[]): string | null { + let post: URL; + try { post = new URL(url); } catch { return null; } + const matches = blogs.filter((blog) => { + const base = new URL(blog.uid); + const prefix = base.pathname.endsWith('/') ? base.pathname : `${base.pathname}/`; + return post.origin === base.origin && (post.pathname === base.pathname || post.pathname.startsWith(prefix)); + }).sort((a, b) => b.uid.length - a.uid.length); + return matches[0]?.uid ?? null; +} diff --git a/src/connectors/microblog.ts b/src/connectors/microblog.ts index b41e210e..0315e8da 100644 --- a/src/connectors/microblog.ts +++ b/src/connectors/microblog.ts @@ -1,4 +1,6 @@ import { decideMatch, type Candidate } from './matching.js'; +import { publishMicroblogClippings } from './microblog-clippings.js'; +import { listMicroblogDestinations } from './microblog-destinations.js'; import { ConnectorOperationError, type Credential, @@ -198,6 +200,16 @@ async function loadInventory(cred: Credential, http: HttpTransport): Promise { try { await loadShelves(cred, http); + // Never trust a destination supplied alongside a pasted token. Shelf-only + // accounts still link when Micropub configuration is unavailable. + delete cred.microblogDestination; + delete cred.microblogDestinations; + try { + const destinations = await listMicroblogDestinations(cred, http); + cred.microblogDestinations = destinations; + const chosen = destinations.find((d) => d.isDefault) ?? (destinations.length === 1 ? destinations[0] : null); + if (chosen) cred.microblogDestination = chosen.uid; + } catch { /* Older servers/tokens can continue syncing bookshelves. */ } return { ok: true }; } catch (error) { if (error instanceof ConnectorOperationError && error.needsReauth) { @@ -361,6 +373,7 @@ async function reconcileBook( } function shouldPush(ev: OutboundEvent, canonicalPercentage?: number | null): boolean { + if (ev.kind === 'highlight') return true; if (ev.kind === 'progress' && (ev.percentage ?? 0) <= 0) return false; if (canonicalPercentage == null) return true; if (canonicalPercentage <= 0) return false; @@ -373,7 +386,8 @@ export const microblogConnector: Connector = { displayName: 'Micro.blog', tier: 1, capabilities: { read: false, write: true }, - carries: ['progress', 'finished'], + carries: ['progress', 'finished', 'highlight'], + options: [{ key: 'sync_highlights', label: 'Automatically sync clippings', default: false }], credentialKind: 'token', beta: false, matchBy: 'metadata', @@ -382,7 +396,16 @@ export const microblogConnector: Connector = { search: searchBooks, match: matchBook, createBook, - push: reconcileBook, + async push(cred, match, ev, http, ctx) { + if (ev.kind !== 'highlight') return reconcileBook(cred, match, ev, http); + if (!ctx) return { ok: false, retryable: false, error: 'Micro.blog publication requires local storage' }; + let isbn = match.externalEdition; + if (!isbn) { + if (/^(?:\d{13}|\d{9}[\dXx])$/.test(match.externalId)) isbn = match.externalId; + else isbn = (await loadInventory(cred, http)).find((b) => b.externalId === match.externalId)?.isbn; + } + return publishMicroblogClippings(cred, { ...match, externalEdition: isbn }, ev, http, ctx); + }, }; export const _microblog = { diff --git a/src/connectors/refresh.ts b/src/connectors/refresh.ts index b4ee13ac..b44999e5 100644 --- a/src/connectors/refresh.ts +++ b/src/connectors/refresh.ts @@ -5,6 +5,7 @@ import { resolveMatch } from './runner.js'; import { getAccount, getMatch } from './store.js'; import type { HttpTransport } from './types.js'; import { importBookorbitStats } from './bookorbit-stats.js'; +import { scheduleMicroblogClippings } from './microblog-clippings.js'; export type ProgressRefresh = (userId: number, document: string) => Promise; @@ -39,6 +40,12 @@ const PER_BOOK_PULLERS: { id: string; strict: boolean; sidecarOnly: boolean; mat export function createProgressRefresh(db: DB, http: HttpTransport = fetchTransport): ProgressRefresh { const pending = new Map>(); return async (userId, document) => { + try { + scheduleMicroblogClippings(db, userId, document); + } catch (err) { + console.error(JSON.stringify({ msg: 'microblog schedule failed', user_id: userId, document, + error: err instanceof Error ? err.message : String(err) })); + } const pullers: typeof PER_BOOK_PULLERS = []; for (const p of PER_BOOK_PULLERS) { const account = getAccount(db, userId, p.id); diff --git a/src/connectors/registry.ts b/src/connectors/registry.ts index 3ccf39ac..57e976ee 100644 --- a/src/connectors/registry.ts +++ b/src/connectors/registry.ts @@ -12,31 +12,37 @@ import { bookorbitConnector } from './bookorbit.js'; import { spotifyConnector } from './spotify.js'; import { seekquelConnector } from './seekquel.js'; -/** All connectors known to this build. */ -const CONNECTORS: Connector[] = [ - kosyncConnector, - hardcoverConnector, - readwiseConnector, - readwiseReaderConnector, - bookfusionConnector, - audiobookshelfConnector, - jellyfinConnector, - bookorbitConnector, - spotifyConnector, - seekquelConnector, - microblogConnector, - kindleConnector, -]; +let byId: Map | undefined; -const byId = new Map(CONNECTORS.map((c) => [c.id, c])); +/** Storage and connectors can import each other; resolve exports on first use. */ +function connectorRegistry(): Map { + if (!byId) { + const connectors: Connector[] = [ + kosyncConnector, + hardcoverConnector, + readwiseConnector, + readwiseReaderConnector, + bookfusionConnector, + audiobookshelfConnector, + jellyfinConnector, + bookorbitConnector, + spotifyConnector, + seekquelConnector, + microblogConnector, + kindleConnector, + ]; + byId = new Map(connectors.map((c) => [c.id, c])); + } + return byId; +} /** Connectors shown in the UI/API (excludes hidden ones). */ export function listConnectors(): Connector[] { - return CONNECTORS.filter((c) => !c.hidden); + return [...connectorRegistry().values()].filter((c) => !c.hidden); } export function getConnector(id: string): Connector | undefined { - return byId.get(id); + return connectorRegistry().get(id); } /** Default transport: the platform fetch, adapted to HttpTransport. */ @@ -44,6 +50,7 @@ export const fetchTransport: HttpTransport = async (url, init) => { const res = await fetch(url, init); return { status: res.status, + headers: res.headers, body: res.body, text: () => res.text(), json: () => res.json(), diff --git a/src/connectors/runner.ts b/src/connectors/runner.ts index 8a1e77c8..2b6c6277 100644 --- a/src/connectors/runner.ts +++ b/src/connectors/runner.ts @@ -20,6 +20,7 @@ import { setMatchNote, } from './store.js'; import { decideMatch, extractTitleAuthor } from './matching.js'; +import { scheduleMicroblogClippings } from './microblog-clippings.js'; import { ConnectorOperationError, type HttpTransport, @@ -126,6 +127,19 @@ export async function processRow(db: DB, row: QueueRow, http: HttpTransport): Pr } const ev = JSON.parse(row.payload) as OutboundEvent; + const publication = row.connector_id === 'microblog' && ev.kind === 'highlight'; + const originalPost = publication ? db.prepare('SELECT identity FROM microblog_clipping_posts WHERE user_id=? AND document=?') + .get(row.user_id, row.document) as { identity: string } | undefined : undefined; + const stillCurrent = () => row.connector_id !== 'microblog' || ( + getAccount(db, row.user_id, row.connector_id)?.cred_enc === account.cred_enc && (!originalPost || + (db.prepare('SELECT identity FROM microblog_clipping_posts WHERE user_id=? AND document=?').get(row.user_id, row.document) as { identity: string } | undefined)?.identity === originalPost.identity) + ); + if (row.connector_id === 'microblog' && ev.kind === 'highlight' + && !getMatch(db, row.user_id, 'microblog', row.document)?.external_id) { + // Publication intent remains durable; a later matched sync reschedules it. + markDone(db, row.id); + return; + } if (connector.shouldPush?.( ev, latestPercentage(db, row.user_id, row.document) @@ -137,7 +151,9 @@ export async function processRow(db: DB, row: QueueRow, http: HttpTransport): Pr let match: Match | null; try { match = await resolveMatch(db, row.connector_id, row.user_id, row.document, http, ev); + if (!stillCurrent()) return; } catch (err) { + if (!stillCurrent()) return; failOperation(db, row, err, 'match failed'); return; } @@ -151,10 +167,21 @@ export async function processRow(db: DB, row: QueueRow, http: HttpTransport): Pr const cred = decryptCredential(account, db); try { const result = await connector.push(cred, match, ev, http, { db, userId: row.user_id }); + if (!stillCurrent()) return; if (result.ok) { // Persist (or clear) the per-book condition so the review UI can show it. setMatchNote(db, row.user_id, row.connector_id, row.document, result.note ?? null); markDone(db, row.id); + // A progress push may have established the first match, or a clipping + // may have arrived while publication was in flight. + if (row.connector_id === 'microblog') { + try { + scheduleMicroblogClippings(db, row.user_id, row.document); + } catch (err) { + console.error(JSON.stringify({ msg: 'microblog schedule failed', user_id: row.user_id, document: row.document, + error: err instanceof Error ? err.message : String(err) })); + } + } return; } if (result.needsReauth) { @@ -163,6 +190,7 @@ export async function processRow(db: DB, row: QueueRow, http: HttpTransport): Pr logPushFailure(row, result.error, result.retryable); markFailed(db, row, result.error, result.retryable); } catch (err) { + if (!stillCurrent()) return; failOperation(db, row, err, 'push failed'); } } diff --git a/src/connectors/store.ts b/src/connectors/store.ts index 2596f270..92747fa4 100644 --- a/src/connectors/store.ts +++ b/src/connectors/store.ts @@ -3,6 +3,7 @@ import { withTransaction, type DB } from '../db/db.js'; import { nowSeconds } from '../models/sync.js'; import { decryptSecret, encryptSecret } from '../crypto/secrets.js'; import { SAVE_CREDENTIAL, type Connector, type Credential, type DataKind, type DocumentMeta, type Match, type SavableCredential } from './types.js'; +import { destinationForPost, selectedMicroblogDestination } from './microblog-destinations.js'; export interface AccountRow { user_id: number; @@ -25,7 +26,30 @@ export function upsertAccount( now = nowSeconds() ): void { const enc = encryptSecret(JSON.stringify(cred)); - const previous = connectorId === 'bookorbit' ? getAccount(db, userId, connectorId) : null; + const previous = connectorId === 'bookorbit' || connectorId === 'microblog' ? getAccount(db, userId, connectorId) : null; + let changedMicroblogAccount = false; + let verifyMicroblogPosts = false; + if (connectorId === 'microblog' && previous) { + try { + const old = decryptCredential(previous); + if (old.token !== cred.token) { + // Existing installs may have published before destination discovery was + // added. Location URLs provide a conservative same-blog identity hint. + const posts = db.prepare('SELECT url,destination FROM microblog_clipping_posts WHERE user_id=?') + .all(userId) as { url: string | null; destination: string | null }[]; + const origins = new Set(posts.flatMap((r) => { try { return r.url ? [new URL(r.url).origin + '/'] : []; } catch { return []; } })); + const oldDestination = old.microblogDestination ?? (origins.size === 1 ? [...origins][0] : null); + const available = Array.isArray(cred.microblogDestinations) ? cred.microblogDestinations as { uid?: string }[] : []; + const availableBlogs = available.flatMap((d) => typeof d?.uid === 'string' + ? [{ uid: d.uid, name: d.uid, isDefault: false }] : []); + const accessibleLegacyPost = posts.some((p) => !p.destination && p.url && destinationForPost(p.url, availableBlogs)); + const selectedBlog = selectedMicroblogDestination(db, userId); + if (accessibleLegacyPost || available.some((d) => d.uid === oldDestination || d.uid === selectedBlog || posts.some((p) => p.destination === d.uid))) changedMicroblogAccount = false; + else if (oldDestination && cred.microblogDestination) changedMicroblogAccount = oldDestination !== cred.microblogDestination; + else verifyMicroblogPosts = true; + } + } catch { verifyMicroblogPosts = true; } + } let sameOrbitAccount = false; if (previous) { try { @@ -34,6 +58,17 @@ export function upsertAccount( } catch { /* Reconnecting also recovers an unreadable old credential. */ } } withTransaction(db, () => { + if (verifyMicroblogPosts) { + // Retain URL/destination/history. Publication validates access to each + // original blog and its source URL before any further update. + db.prepare("DELETE FROM connector_queue WHERE user_id=? AND connector_id='microblog'").run(userId); + } + if (changedMicroblogAccount) { + // A new token's default blog cannot prove that older collections should + // be forgotten. Only explicit unlink erases their publication history. + for (const table of ['connector_matches', 'connector_queue']) db.prepare(`DELETE FROM ${table} WHERE user_id=? AND connector_id='microblog'`).run(userId); + db.prepare("UPDATE connector_accounts SET options='{}' WHERE user_id=? AND connector_id='microblog'").run(userId); + } db.prepare( `INSERT INTO connector_accounts (user_id, connector_id, cred_enc, account_label, status, enabled, created_at, updated_at) @@ -90,6 +125,9 @@ export function decryptCredential(row: AccountRow, db?: DB): Credential { } export function deleteAccount(db: DB, userId: number, connectorId: string): void { + if (connectorId === 'microblog') { + for (const table of ['microblog_clipping_selections', 'microblog_clipping_posts']) db.prepare(`DELETE FROM ${table} WHERE user_id=?`).run(userId); + } db.prepare('DELETE FROM connector_accounts WHERE user_id = ? AND connector_id = ?').run( userId, connectorId diff --git a/src/models/document.ts b/src/models/document.ts index 9e37a458..39c295c4 100644 --- a/src/models/document.ts +++ b/src/models/document.ts @@ -8,6 +8,8 @@ import { withTransaction, type DB } from '../db/db.js'; * intent obvious next to the account-level delete in routes/account.ts. */ const DOCUMENT_TABLES = [ + 'microblog_clipping_selections', + 'microblog_clipping_posts', 'connector_queue', 'connector_matches', 'stats_device_book', diff --git a/src/models/merge.ts b/src/models/merge.ts index 1c5bf829..b6ea0044 100644 --- a/src/models/merge.ts +++ b/src/models/merge.ts @@ -2,6 +2,7 @@ import { withTransaction, type DB } from '../db/db.js'; import { nextChangeSeq } from './changes.js'; import { md5Hex } from '../auth/password.js'; import { autoUnpause } from './pause.js'; +import { mergeMicroblogClippings, scheduleMicroblogClippings } from '../connectors/microblog-clippings.js'; import { MANUAL_COLUMNS, mergeManualSql } from './manual-details.js'; /** @@ -173,6 +174,7 @@ export function mergeDocuments( const last = db.prepare('SELECT MAX(at) AS at FROM progress_log WHERE user_id = ? AND document = ?').get(userId, into) as { at: number | null }; if (last.at !== null) autoUnpause(db, userId, into, last.at); + mergeMicroblogClippings(db, userId, from, into); // Uniquely-keyed side tables: move what fits, drop the (rare) conflicts. for (const table of ['bookmarks', 'clippings', 'connector_matches', 'connector_queue']) { db.prepare(`UPDATE OR IGNORE ${table} SET document = ? WHERE user_id = ? AND document = ?`).run( @@ -193,6 +195,14 @@ export function mergeDocuments( 'INSERT OR REPLACE INTO document_aliases (user_id, alias, document, created_at, merge_stats) VALUES (?, ?, ?, ?, ?)' ).run(userId, from, into, now, mergeStats ? 1 : 0); }); + // Selections from an unmatched duplicate may now belong to a matched book. + // Schedule only after all clippings, matches, and aliases have been settled. + try { + scheduleMicroblogClippings(db, userId, into); + } catch (err) { + console.error(JSON.stringify({ msg: 'microblog schedule failed', user_id: userId, document: into, + error: err instanceof Error ? err.message : String(err) })); + } } /** diff --git a/src/routes/account.ts b/src/routes/account.ts index ab378820..bc1eca3b 100644 --- a/src/routes/account.ts +++ b/src/routes/account.ts @@ -14,6 +14,8 @@ import { USERNAME_RE } from './kosync.js'; export function deleteKosyncUserData(db: DB, userId: number, username: string, { keepUser = false } = {}): void { withTransaction(db, () => { for (const table of [ + 'microblog_clipping_selections', + 'microblog_clipping_posts', 'bookorbit_session_exports', 'bookorbit_session_destination', 'reading_sessions', diff --git a/src/routes/v1/clippings.ts b/src/routes/v1/clippings.ts index 4f1b83db..823662ae 100644 --- a/src/routes/v1/clippings.ts +++ b/src/routes/v1/clippings.ts @@ -6,6 +6,7 @@ import { isItemId, nowSeconds, parseListParams } from '../../models/sync.js'; import { fanOutHighlight, highlightFromRow, type ClippingHighlightRow } from '../../connectors/fanout.js'; import { documentMeta } from '../../connectors/store.js'; import { clippingDocuments, mergedClippingsCte, resolveDocument } from '../../models/merge.js'; +import { scheduleMicroblogClippings } from '../../connectors/microblog-clippings.js'; const MAX_BATCH = 50; const MAX_TEXT = 4096; // full firmware clipping, not the text-export cap @@ -81,6 +82,12 @@ export function clippingRoutes(db: DB): Hono { if (!Number.isSafeInteger(cursor) || cursor < 0) return kosyncError(c, 403, 2003, 'Invalid cursor'); const revisionMode = cursorRaw !== undefined; const canonical = resolveDocument(db, user.id, document); + try { + scheduleMicroblogClippings(db, user.id, canonical); + } catch (err) { + console.error(JSON.stringify({ msg: 'microblog schedule failed', user_id: user.id, document: canonical, + error: err instanceof Error ? err.message : String(err) })); + } const documents = clippingDocuments(db, user.id, canonical); const rows = db .prepare( @@ -278,6 +285,12 @@ export function clippingRoutes(db: DB): Hono { ); } } + try { + scheduleMicroblogClippings(db, user.id, canonical); + } catch (err) { + console.error(JSON.stringify({ msg: 'microblog schedule failed', user_id: user.id, document: canonical, + error: err instanceof Error ? err.message : String(err) })); + } return c.json({ until: now, accepted: ops.length }); }); diff --git a/src/routes/v1/connectors.ts b/src/routes/v1/connectors.ts index 5674a9c9..d8ed1088 100644 --- a/src/routes/v1/connectors.ts +++ b/src/routes/v1/connectors.ts @@ -35,6 +35,9 @@ import { ConnectorOperationError, type HttpTransport } from '../../connectors/ty import { importBookorbitStats } from '../../connectors/bookorbit-stats.js'; import { resolveDocument } from '../../models/merge.js'; import { hasDocumentData } from '../../models/document.js'; +import { selectMicroblogClipping, scheduleMicroblogClippings, recreateMicroblogClippings } from '../../connectors/microblog-clippings.js'; +import { isItemId } from '../../models/sync.js'; +import { listMicroblogDestinations, selectedMicroblogDestination } from '../../connectors/microblog-destinations.js'; /** Where a provider sends the browser back: one https URL for web and native (an app link there). */ export const oauthCallbackPath = (id: string) => `/connectors/${id}/callback`; @@ -135,12 +138,15 @@ export const CREDENTIAL_HTTPS_REQUIRED = 'Connector credentials require HTTPS to + 'If HTTPS ends at a reverse proxy, configure TRUST_PROXY=true on the Sync server ' + 'and have the proxy send X-Forwarded-Proto: https. Only enable proxy trust when direct access to Sync is blocked.'; -export function credentialRequestIsSecure(c: Context, trustProxy: boolean): boolean { +export function credentialRequestIsSecure(c: Context, trustProxy: boolean, allowHttpLocalhost = false): boolean { const url = new URL(c.req.url); if (url.protocol === 'https:') return true; const incoming = c.env?.incoming; const peerAddress = incoming?.socket?.remoteAddress; - if (loopbackHostname(url.hostname) && (!incoming || (peerAddress && loopbackAddress(peerAddress)))) { + // Docker NAT can give a host-local request a bridge peer address. Only an + // explicit localhost-testing opt-in bypasses the peer check; the host port + // must be bound to 127.0.0.1. Host alone is not trusted on ordinary installs. + if (loopbackHostname(url.hostname) && (!incoming || (peerAddress && (loopbackAddress(peerAddress) || allowHttpLocalhost)))) { return true; } return trustProxy @@ -159,7 +165,8 @@ export function connectorRoutes( db: DB, transport: HttpTransport = fetchTransport, trustProxy = false, - basePath = '' + basePath = '', + allowHttpLocalhost = false ): Hono { const app = new Hono(); @@ -225,7 +232,7 @@ export function connectorRoutes( if (conn.oauth) { return c.json({ code: 2003, message: `Update the CrossPoint Sync app to sign in with ${conn.displayName}.` }, 400); } - if (!credentialRequestIsSecure(c, trustProxy)) { + if (!credentialRequestIsSecure(c, trustProxy, allowHttpLocalhost)) { return c.json({ code: 2003, message: CREDENTIAL_HTTPS_REQUIRED }, 400); } if (!secretsEnabled()) { @@ -269,7 +276,7 @@ export function connectorRoutes( const conn = getConnector(c.req.param('id')); if (!conn) return c.json({ code: 2003, message: 'Unknown connector' }, 404); if (!conn.beginLink) return c.json({ code: 2003, message: 'Connector has no device link' }, 400); - if (!credentialRequestIsSecure(c, trustProxy)) { + if (!credentialRequestIsSecure(c, trustProxy, allowHttpLocalhost)) { return c.json({ code: 2003, message: CREDENTIAL_HTTPS_REQUIRED }, 400); } if (!secretsEnabled()) { @@ -295,7 +302,7 @@ export function connectorRoutes( const conn = getConnector(c.req.param('id')); if (!conn) return c.json({ code: 2003, message: 'Unknown connector' }, 404); if (!conn.pollLink) return c.json({ code: 2003, message: 'Connector has no device link' }, 400); - if (!credentialRequestIsSecure(c, trustProxy)) { + if (!credentialRequestIsSecure(c, trustProxy, allowHttpLocalhost)) { return c.json({ code: 2003, message: CREDENTIAL_HTTPS_REQUIRED }, 400); } let deviceCode: unknown; @@ -363,7 +370,7 @@ export function connectorRoutes( const config = conn.oauth!(); const clientId = getClientId(db, user.id, conn.id) ?? config.clientId; if (!clientId) return c.json({ code: 2003, message: `Add your ${conn.displayName} app's Client ID first.` }, 400); - if (!credentialRequestIsSecure(c, trustProxy)) { + if (!credentialRequestIsSecure(c, trustProxy, allowHttpLocalhost)) { return c.json({ code: 2003, message: CREDENTIAL_HTTPS_REQUIRED }, 400); } if (!secretsEnabled()) { @@ -458,6 +465,69 @@ export function connectorRoutes( } }); + app.get('/connectors/microblog/destinations', async (c) => { + const userId = c.get('user').id; + const account = getAccount(db, userId, 'microblog'); + if (!account?.enabled) return c.json({ message: 'Micro.blog is not linked or enabled' }, 400); + try { + const items = await listMicroblogDestinations(decryptCredential(account, db), transport); + return c.json({ items, selected: selectedMicroblogDestination(db, userId) ?? (items.length === 1 ? items[0].uid : null) }); + } catch (error) { return c.json({ message: error instanceof Error ? error.message : 'Could not load blogs' }, 502); } + }); + + app.get('/connectors/microblog/clipping-posts', (c) => { + const userId = c.get('user').id; + if (!getAccount(db, userId, 'microblog')?.enabled) return c.json({ items: [] }); + const rows = db.prepare('SELECT document,url,destination,missing,error FROM microblog_clipping_posts WHERE user_id=?').all(userId); + return c.json({ items: rows.map((row) => ({ ...row, missing: !!row.missing })) }); + }); + + app.post('/connectors/microblog/clipping-posts/:document/recreate', async (c) => { + if (!credentialRequestIsSecure(c, trustProxy, allowHttpLocalhost)) return c.json({ message: CREDENTIAL_HTTPS_REQUIRED }, 400); + const document = c.req.param('document'); + if (!isValidDocument(document)) return c.json({ message: 'Invalid book' }, 400); + const body = await c.req.json().catch(() => null) as { target?: unknown } | null; + const target = body?.target ?? 'original'; + if (target !== 'original' && target !== 'current') return c.json({ message: 'Choose the previous or current clippings blog' }, 400); + try { return c.json(await recreateMicroblogClippings(db, c.get('user').id, document, transport, target)); } + catch (error) { return c.json({ message: error instanceof Error ? error.message : 'Could not recreate post' }, error instanceof ConnectorOperationError && !error.retryable ? 409 : 502); } + }); + + app.put('/connectors/microblog/destination', async (c) => { + if (!credentialRequestIsSecure(c, trustProxy, allowHttpLocalhost)) return c.json({ message: CREDENTIAL_HTTPS_REQUIRED }, 400); + const userId = c.get('user').id; + const account = getAccount(db, userId, 'microblog'); + if (!account?.enabled) return c.json({ message: 'Micro.blog is not linked or enabled' }, 400); + const body = await c.req.json().catch(() => null) as { destination?: unknown } | null; + try { + const items = await listMicroblogDestinations(decryptCredential(account, db), transport); + if (!items.some((d) => d.uid === body?.destination)) return c.json({ message: 'Choose an available Micro.blog blog' }, 400); + if (getAccount(db, userId, 'microblog')?.cred_enc !== account.cred_enc) return c.json({ message: 'Micro.blog account changed; reload its blogs' }, 409); + const row = db.prepare("SELECT options FROM connector_accounts WHERE user_id=? AND connector_id='microblog'").get(userId) as { options: string | null }; + let values: OptionValues = {}; + try { values = JSON.parse(row.options || '{}'); } catch { /* Recover malformed settings. */ } + setConnectorOptions(db, userId, 'microblog', { ...values, highlight_blog: body!.destination as string }); + scheduleMicroblogClippings(db, userId); + return c.json({ destination: body!.destination }); + } catch (error) { return c.json({ message: error instanceof Error ? error.message : 'Could not select blog' }, 502); } + }); + + app.post('/connectors/microblog/clippings/:document/:clippingId', (c) => { + const user = c.get('user'); + const account = getAccount(db, user.id, 'microblog'); + if (!account?.enabled) return c.json({ message: 'Micro.blog is not linked or enabled' }, 400); + const document = c.req.param('document'); + const clippingId = c.req.param('clippingId'); + if (!isValidDocument(document) || !isItemId(clippingId)) return c.json({ message: 'Invalid clipping' }, 400); + const canonical = resolveDocument(db, user.id, document); + try { selectMicroblogClipping(db, user.id, canonical, clippingId); } + catch { return c.json({ message: 'Clipping not found' }, 404); } + const state = db.prepare('SELECT error FROM microblog_clipping_posts WHERE user_id=? AND document=?').get(user.id, canonical) as { error: string | null } | undefined; + if (state?.error) return c.json({ message: state.error }, 409); + const queued = scheduleMicroblogClippings(db, user.id, canonical) > 0; + return c.json({ queued, waiting_for_match: !getMatch(db, user.id, 'microblog', canonical)?.external_id }); + }); + // Change a linked connector's options. Unknown keys and bad values are ignored. app.put('/connectors/:id/options', async (c) => { const conn = getConnector(c.req.param('id')); @@ -468,6 +538,10 @@ export function connectorRoutes( } const body = (await c.req.json().catch(() => null)) as Record | null; const next: OptionValues = connectorOptions(db, user.id, conn); + if (conn.id === 'microblog') { + const destination = selectedMicroblogDestination(db, user.id); + if (destination) next.highlight_blog = destination; + } for (const o of conn.options) { const v = body?.[o.key]; if (o.choices ? o.choices.some((ch) => ch.value === v) : typeof v === 'boolean') next[o.key] = v as boolean | string; @@ -483,6 +557,7 @@ export function connectorRoutes( } } }); + if (conn.id === 'microblog') scheduleMicroblogClippings(db, user.id); return c.json({ id: conn.id, option_values: next }); }); diff --git a/src/routes/web.ts b/src/routes/web.ts index 8f71f0b9..1d5e08ae 100644 --- a/src/routes/web.ts +++ b/src/routes/web.ts @@ -613,7 +613,7 @@ async function linkCredential(body) { const HINTS = { hardcover: 'Syncs your reading progress and shelf status to Hardcover.', - microblog: 'Connect an app token to keep your Currently reading and Finished reading bookshelves in sync.', + microblog: 'Connect an app token to keep your Currently reading and Finished reading bookshelves in sync. Optional clipping sync publishes one post per book and updates its generated content on later syncs, replacing edits made on Micro.blog.', readwise: 'Paste your Readwise access token from readwise.io/access_token. Syncs your highlights.', 'readwise-reader': 'Paste your Readwise access token from readwise.io/access_token. Archives books in Reader when you finish them, and brings your Reader progress back to your device.', kosync: 'Mirror your reading progress to another KOReader-compatible (KOSync) server, so your other devices see it too.', @@ -637,7 +637,8 @@ const TOKEN_HELP = { + '
  • Open Account → App tokens.
    • ' + '
  • Create a separate token for CrossPoint Sync.
    • ' + '
  • Copy the new token and paste it below.
    • ' - + '

    Treat this token like a password: it has full access to your Micro.blog account. CrossPoint Sync encrypts it before storing it.

    ', + + '

    Choose the posting blog and automatic clipping sync in app Settings.

    ' + + '

    Treat this token like a password: it has full access to your Micro.blog account. CrossPoint Sync encrypts it before storing it.

    ', kindle: '
    ' + '

    Set up with the CrossPoint Kindle Link extension (recommended):

    ' + '
      ' diff --git a/test/bookorbit-pull.test.ts b/test/bookorbit-pull.test.ts index 4100379a..d4420bb6 100644 --- a/test/bookorbit-pull.test.ts +++ b/test/bookorbit-pull.test.ts @@ -1,7 +1,8 @@ import { afterEach, beforeEach, expect, it, vi } from 'vitest'; import { pollConnector } from '../src/connectors/fanin.js'; import { bookorbitConnector } from '../src/connectors/bookorbit.js'; -import { deleteAccount, saveMatch, upsertAccount } from '../src/connectors/store.js'; +import { deleteAccount, saveMatch, setConnectorOptions, upsertAccount } from '../src/connectors/store.js'; +import { scheduleMicroblogClippings } from '../src/connectors/microblog-clippings.js'; import { resetEncryptionKeyCache } from '../src/crypto/secrets.js'; import { upsertProgress } from '../src/routes/kosync.js'; import type { HttpTransport, InboundHighlight } from '../src/connectors/types.js'; @@ -52,6 +53,54 @@ it('imports progress and clippings on demand, exposing progress to bulk sync', a expect(next.changes).toEqual([]); }); +it.each([ + { automatic: true, matched: true }, + { automatic: false, matched: true }, + { automatic: true, matched: false }, +])('schedules imported clippings without a progress change (automatic: $automatic, matched: $matched)', async ({ automatic, matched }) => { + const f = await fixture(); + vi.spyOn(bookorbitConnector, 'pullProgress').mockResolvedValue(null); + vi.spyOn(bookorbitConnector, 'pullHighlights').mockResolvedValue([highlight]); + upsertAccount(f.db, f.id, 'microblog', { token: 'token' }, null); + setConnectorOptions(f.db, f.id, 'microblog', { sync_highlights: automatic }); + const match = () => saveMatch(f.db, f.id, 'microblog', DOC, + { externalId: '123', externalEdition: '9781786487849', confidence: 1 }, 'manual'); + if (matched) match(); + expect(await pollConnector(f.db, f.id, 'bookorbit', f.http, { document: DOC })).toBe(0); + expect(f.db.prepare('SELECT text FROM clippings').get()).toEqual({ text: highlight.text }); + expect(f.db.prepare('SELECT COUNT(*) AS n FROM microblog_clipping_selections').get()).toEqual({ n: automatic ? 1 : 0 }); + const jobs = () => f.db.prepare("SELECT document,status FROM connector_queue WHERE kind='microblog:clippings'").all(); + expect(jobs()).toEqual(automatic && matched ? [{ document: DOC, status: 'pending' }] : []); + if (automatic && !matched) { + match(); + expect(scheduleMicroblogClippings(f.db, f.id, DOC)).toBe(1); + } + await pollConnector(f.db, f.id, 'bookorbit', f.http, { document: DOC }); + expect(jobs()).toHaveLength(automatic ? 1 : 0); + expect(f.db.prepare('SELECT COUNT(*) AS n FROM clippings').get()).toEqual({ n: 1 }); + expect(f.db.prepare("SELECT COUNT(*) AS n FROM connector_queue WHERE connector_id='bookorbit'").get()).toEqual({ n: 0 }); + f.db.close(); +}); + +it('keeps imported clippings and other fanout when Micro.blog scheduling fails', async () => { + const f = await fixture(); + vi.spyOn(bookorbitConnector, 'pullProgress').mockResolvedValue(null); + vi.spyOn(bookorbitConnector, 'pullHighlights').mockResolvedValue([highlight]); + upsertAccount(f.db, f.id, 'microblog', { token: 'token' }, null); + setConnectorOptions(f.db, f.id, 'microblog', { sync_highlights: true }); + saveMatch(f.db, f.id, 'microblog', DOC, { externalId: '123', externalEdition: '9781786487849', confidence: 1 }, 'manual'); + upsertAccount(f.db, f.id, 'seekquel', { token: 'token' }, null); + saveMatch(f.db, f.id, 'seekquel', DOC, { externalId: '456', confidence: 1 }, 'manual'); + f.db.exec(`CREATE TEMP TRIGGER fail_import_schedule BEFORE INSERT ON microblog_clipping_selections + BEGIN SELECT RAISE(FAIL, 'simulated import scheduling failure'); END`); + await expect(pollConnector(f.db, f.id, 'bookorbit', f.http, { document: DOC, throwOnError: true })).resolves.toBe(0); + expect(f.db.prepare('SELECT text FROM clippings').get()).toEqual({ text: highlight.text }); + expect(f.db.prepare("SELECT kind FROM connector_queue WHERE connector_id='seekquel'").get()).toEqual({ kind: expect.stringMatching(/^highlight:/) }); + f.db.exec('DROP TRIGGER fail_import_schedule'); + expect(scheduleMicroblogClippings(f.db, f.id, DOC)).toBe(1); + f.db.close(); +}); + it('manually syncs only the requested book and queues the pulled position, not the old local one', async () => { const f = await fixture(); const other = '1234567890abcdef1234567890abcdef'; diff --git a/test/connectors.test.ts b/test/connectors.test.ts index 2920530c..39a717bf 100644 --- a/test/connectors.test.ts +++ b/test/connectors.test.ts @@ -260,6 +260,25 @@ describe('connector management API', () => { expect(fake.calls).toHaveLength(0); }); + it.each([ + ['localhost', '172.17.0.1', true, 200], + ['127.0.0.1', '192.168.65.1', true, 200], + ['[::1]', '::ffff:172.17.0.1', true, 200], + ['localhost', '172.17.0.1', false, 400], + ['sync.example.com', '172.17.0.1', true, 400], + ])('links through a Docker bridge at %s with local HTTP opt-in %s', async (hostname, peer, enabled, expected) => { + const fake = fakeTransport(); + fake.on('graphql', 200, { data: { me: [{ username: 'julia' }] } }); + const { app } = makeTestApp(fromEnv({ ALLOW_HTTP_LOCALHOST: String(enabled) }), { connectorTransport: fake.transport }); + const { headers } = await registerUser(app); + const res = await app.request(`http://${hostname}:8081/api/v1/connectors/hardcover`, { + method: 'PUT', headers, + body: JSON.stringify({ credential: { token: 'hc-token' } }), + }, { incoming: { socket: { remoteAddress: peer } } }); + expect(res.status).toBe(expected); + expect(fake.calls.length > 0).toBe(expected === 200); + }); + it('fails closed when a Node request has no peer address', async () => { const fake = fakeTransport(); fake.on('graphql', 200, { data: { me: [{ username: 'julia' }] } }); diff --git a/test/microblog-clippings-api.test.ts b/test/microblog-clippings-api.test.ts new file mode 100644 index 00000000..77fbac04 --- /dev/null +++ b/test/microblog-clippings-api.test.ts @@ -0,0 +1,274 @@ +import { afterEach, beforeEach, expect, it } from 'vitest'; +import { makeTestApp, registerUser, DOC } from './helpers.js'; +import { resetEncryptionKeyCache } from '../src/crypto/secrets.js'; +import { upsertAccount } from '../src/connectors/store.js'; +import { drainQueue } from '../src/connectors/runner.js'; +import type { HttpTransport } from '../src/connectors/types.js'; +import { deleteDocumentData } from '../src/models/document.js'; +import { deleteKosyncUserData } from '../src/routes/account.js'; +import { fanOutProgress } from '../src/connectors/fanout.js'; +import { createProgressRefresh } from '../src/connectors/refresh.js'; +import { makeMicroblogTransport } from './microblog-helpers.js'; +import { mergeDocuments } from '../src/models/merge.js'; +import { clippingsPost, scheduleMicroblogClippings } from '../src/connectors/microblog-clippings.js'; + +beforeEach(() => { process.env.TOKEN_ENC_KEY = 'a'.repeat(64); resetEncryptionKeyCache(); }); +afterEach(() => { delete process.env.TOKEN_ENC_KEY; resetEncryptionKeyCache(); }); +async function setup(shelfTransport?: HttpTransport) { + const posts: unknown[] = []; + const http: HttpTransport = async (url, init) => { + if (new URL(url).searchParams.get('q') === 'config') return { status: 200, json: async () => ({ destination: [{ uid: 'https://example.micro.blog/' }] }), text: async () => '' }; + if (new URL(url).pathname === '/micropub' && init.method === 'POST') posts.push(JSON.parse(init.body!)); + else if (!url.includes('/micropub?')) { + if (shelfTransport) return shelfTransport(url, init); + throw new Error(`Unexpected network request: ${url}`); + } + return { status: 201, headers: { get: () => 'https://example.micro.blog/highlights.html' }, json: async () => ({}), text: async () => '' }; + }; + const { app, db } = makeTestApp({}, { connectorTransport: http }); + const { username, headers } = await registerUser(app); + const uid = (db.prepare('SELECT id FROM users WHERE username=?').get(username) as { id: number }).id; + upsertAccount(db, uid, 'microblog', { token: 'token', microblogDestination: 'https://example.micro.blog/' }, null); + const put = (path: string, body: unknown) => app.request(path, { method: 'PUT', headers, body: JSON.stringify(body) }); + await put('/api/v1/documents', { items: [{ document: DOC, title: 'Test book', author: 'Author' }] }); + const clipping = () => put(`/api/v1/clippings/${DOC}`, { items: [{ id: 'one', spine: 1, text: 'A passage', created_at: 1 }] }); + await clipping(); + const select = (id = 'one', auth = headers) => app.request(`/api/v1/connectors/microblog/clippings/${DOC}/${id}`, { method: 'POST', headers: auth }); + const match = () => put(`/api/v1/connectors/microblog/matches/${DOC}`, { external_id: '123', external_edition: '9781786487849' }); + return { app, db, uid, headers, posts, http, put, clipping, select, match }; +} + +async function schedulingFailure() { + const s = await setup(); + s.db.prepare('UPDATE connector_accounts SET options=? WHERE user_id=? AND connector_id=?') + .run(JSON.stringify({ sync_highlights: true }), s.uid, 'microblog'); + s.db.exec(`CREATE TEMP TRIGGER fail_microblog_selection BEFORE INSERT ON microblog_clipping_selections + BEGIN SELECT RAISE(FAIL, 'simulated scheduling database failure'); END`); + return s; +} + +it.each(['attempted', 'published'].flatMap((history) => [false, true].map((matched) => ({ history, matched }))))('surfaces merged $history post conflicts in Settings (matched: $matched)', async ({ history, matched }) => { + const s = await setup(); + const duplicate = 'b'.repeat(32); + s.db.prepare('UPDATE connector_accounts SET options=? WHERE user_id=? AND connector_id=?') + .run(JSON.stringify({ sync_highlights: true }), s.uid, 'microblog'); + s.db.prepare('INSERT INTO documents(user_id,document,title,updated_at) VALUES(?,?,?,0)').run(s.uid, duplicate, 'Duplicate book'); + s.db.prepare('INSERT INTO clippings(user_id,document,id,text,spine_index,created_at,updated_at) VALUES(?,?,?,?,0,1,1)') + .run(s.uid, duplicate, 'two', 'Another passage'); + for (const document of [DOC, duplicate]) { + clippingsPost(s.db, s.uid, document); + s.db.prepare('UPDATE microblog_clipping_posts SET attempted=1,url=? WHERE user_id=? AND document=?') + .run(history === 'published' ? `https://example.micro.blog/${document}.html` : null, s.uid, document); + } + if (matched) { + await s.match(); + s.db.prepare("UPDATE connector_queue SET status='done',last_error=NULL").run(); + } + mergeDocuments(s.db, s.uid, duplicate, DOC, 10); + const list = await (await s.app.request('/api/v1/connectors', { headers: s.headers })).json(); + expect(list.connectors.find((c: { id: string }) => c.id === 'microblog')).toMatchObject({ + queue: { dead: 1 }, + queue_errors: [{ count: 1, titles: ['Test book'], message: expect.stringContaining('Merged books both have Micro.blog publication history') }], + }); + expect(scheduleMicroblogClippings(s.db, s.uid, DOC)).toBe(0); + await drainQueue(s.db, s.http); + expect(s.posts).toEqual([]); + expect(s.db.prepare('SELECT COUNT(*) AS n FROM clippings').get()).toEqual({ n: 2 }); + s.db.close(); +}); + +it('still fans out progress to other connectors when Micro.blog scheduling fails', async () => { + const s = await schedulingFailure(); + upsertAccount(s.db, s.uid, 'kosync', { server: 'mirror.test', username: 'reader', password: 'test' }, null); + expect(() => fanOutProgress(s.db, s.uid, DOC, 0.4, 100)).not.toThrow(); + expect(s.db.prepare("SELECT kind FROM connector_queue WHERE connector_id='kosync'").get()).toEqual({ kind: 'progress' }); + s.db.close(); +}); + +it('does not reject progress refresh when Micro.blog scheduling fails', async () => { + const s = await schedulingFailure(); + await expect(createProgressRefresh(s.db, s.http)(s.uid, DOC)).resolves.toBeUndefined(); + s.db.close(); +}); + +it('still returns clippings when Micro.blog scheduling fails', async () => { + const s = await schedulingFailure(); + const res = await s.app.request(`/api/v1/clippings/${DOC}`, { headers: s.headers }); + expect(res.status).toBe(200); + expect(await res.json()).toMatchObject({ items: [{ id: 'one', text: 'A passage' }] }); + s.db.close(); +}); + +it('acknowledges saved clippings and can schedule them on a later sync after a database failure', async () => { + const s = await schedulingFailure(); + const res = await s.put(`/api/v1/clippings/${DOC}`, { items: [{ id: 'two', spine: 2, text: 'Another passage', created_at: 2 }] }); + expect(res.status).toBe(200); + expect(await res.json()).toMatchObject({ accepted: 1 }); + expect(s.db.prepare('SELECT text FROM clippings WHERE user_id=? AND id=?').get(s.uid, 'two')).toEqual({ text: 'Another passage' }); + s.db.exec('DROP TRIGGER fail_microblog_selection'); + await s.match(); + expect((await s.app.request(`/api/v1/clippings/${DOC}`, { headers: s.headers })).status).toBe(200); + await drainQueue(s.db, s.http); + expect(s.posts).toHaveLength(1); + expect(JSON.stringify(s.posts[0])).toContain('Another passage'); + s.db.close(); +}); + +it('defaults automatic publication off and remembers a manually selected unmatched clipping', async () => { + const s = await setup(); + expect(s.db.prepare("SELECT COUNT(*) n FROM connector_queue WHERE kind='microblog:clippings'").get()).toEqual({ n: 0 }); + const selected = await s.select(); + expect(selected.status).toBe(200); + expect(await selected.json()).toEqual({ queued: false, waiting_for_match: true }); + expect(s.db.prepare('SELECT manual FROM microblog_clipping_selections').get()).toEqual({ manual: 1 }); + expect(s.posts).toHaveLength(0); + s.db.close(); +}); + +it.each(['legacy', 'v1', 'clippings', 'sync-now'])('catches up selected clippings after matching on %s sync', async (trigger) => { + const s = await setup(); + await s.select(); + await s.match(); + if (trigger === 'legacy') await s.app.request(`/syncs/progress/${DOC}`, { headers: s.headers }); + if (trigger === 'v1') await s.app.request(`/api/v1/progress/${DOC}`, { headers: s.headers }); + if (trigger === 'clippings') await s.clipping(); + if (trigger === 'sync-now') await s.app.request('/api/v1/connectors/microblog/sync', { method: 'POST', headers: s.headers }); + await drainQueue(s.db, s.http); + expect(s.posts).toHaveLength(1); + expect(JSON.stringify(s.posts[0])).toContain('A passage'); + s.db.close(); +}); + +it.each([ + ['reader progress', 0.4, false], ['reader finished', 0.99, false], + ['Sync now progress', 0.4, true], ['Sync now finished', 0.99, true], +] as const)('queues saved clippings when %s first establishes the match', async (_label, percentage, syncNow) => { + const shelves = makeMicroblogTransport({ shelves: { 'to-read': [{ + id: '123', title: 'Test book', author: 'Author', isbn: '9781786487849', + }] } }); + const s = await setup(shelves.transport); + expect(await (await s.select()).json()).toMatchObject({ waiting_for_match: true }); + const progress = await s.put('/syncs/progress', { document: DOC, device_id: 'reader', progress: '/body/p', percentage }); + expect(progress.status).toBe(200); + if (syncNow) { + s.db.prepare('DELETE FROM connector_queue').run(); + expect((await s.app.request('/api/v1/connectors/microblog/sync', { method: 'POST', headers: s.headers })).status).toBe(200); + } + expect(s.db.prepare("SELECT COUNT(*) n FROM connector_queue WHERE kind='microblog:clippings'").get()).toEqual({ n: 0 }); + await drainQueue(s.db, s.http); + expect(s.db.prepare("SELECT external_id FROM connector_matches WHERE connector_id='microblog'").get()).toEqual({ external_id: '123' }); + expect(s.db.prepare("SELECT status FROM connector_queue WHERE kind='microblog:clippings'").get()).toEqual({ status: 'pending' }); + await drainQueue(s.db, s.http); + expect(s.posts).toHaveLength(1); + expect(JSON.stringify(s.posts[0])).toContain('A passage'); + s.db.close(); +}); + +it.each([0.4, 0.99])('keeps a successful shelf push done when follow-up clipping scheduling fails (%s)', async (percentage) => { + const shelves = makeMicroblogTransport({ shelves: { 'to-read': [{ + id: '123', title: 'Test book', author: 'Author', isbn: '9781786487849', + }] } }); + const s = await setup(shelves.transport); + expect((await s.put('/syncs/progress', { document: DOC, device_id: 'reader', progress: '/body/p', percentage })).status).toBe(200); + s.db.prepare('UPDATE connector_accounts SET options=? WHERE user_id=? AND connector_id=?') + .run(JSON.stringify({ sync_highlights: true }), s.uid, 'microblog'); + s.db.exec(`CREATE TEMP TRIGGER fail_post_push_schedule BEFORE INSERT ON microblog_clipping_selections + BEGIN SELECT RAISE(FAIL, 'simulated post-push scheduling failure'); END`); + await drainQueue(s.db, s.http); + expect(shelves.shelves.get(percentage < 0.98 ? 'reading' : 'finished')?.map((book) => book.id)).toEqual(['123']); + expect(s.db.prepare("SELECT status,attempts,last_error FROM connector_queue WHERE kind=?").get(percentage < 0.98 ? 'progress' : 'finished')) + .toEqual({ status: 'done', attempts: 0, last_error: null }); + s.db.exec('DROP TRIGGER fail_post_push_schedule'); + expect((await s.app.request(`/api/v1/clippings/${DOC}`, { headers: s.headers })).status).toBe(200); + await drainQueue(s.db, s.http); + expect(s.posts).toHaveLength(1); + expect(shelves.calls.filter((c) => c.method === 'POST' && c.url.endsWith('/assign'))).toHaveLength(1); + s.db.close(); +}); + +it('enabling automatic publication backfills existing highlights and waits for matching', async () => { + const s = await setup(); + const changed = await s.put('/api/v1/connectors/microblog/options', { sync_highlights: true }); + expect(changed.status).toBe(200); + expect(s.db.prepare('SELECT manual FROM microblog_clipping_selections').get()).toEqual({ manual: 0 }); + expect(s.db.prepare("SELECT COUNT(*) n FROM connector_queue WHERE kind='microblog:clippings'").get()).toEqual({ n: 0 }); + await s.match(); await s.clipping(); await drainQueue(s.db, s.http); + expect(s.posts).toHaveLength(1); + s.db.close(); +}); + +it('rejects missing, deleted, foreign, unlinked, disabled and unauthenticated selections', async () => { + const s = await setup(); + expect((await s.select('missing')).status).toBe(404); + const other = await registerUser(s.app); + expect((await s.select('one', other.headers)).status).toBe(400); + const otherId = (s.db.prepare('SELECT id FROM users WHERE username=?').get(other.username) as { id: number }).id; + upsertAccount(s.db, otherId, 'microblog', { token: 'other' }, null); + expect((await s.select('one', other.headers)).status).toBe(404); + expect((await s.select('one', {})).status).toBe(401); + s.db.prepare('UPDATE clippings SET deleted=1').run(); + expect((await s.select()).status).toBe(404); + s.db.prepare('UPDATE clippings SET deleted=0').run(); + s.db.prepare('UPDATE connector_accounts SET enabled=0').run(); + expect((await s.select()).status).toBe(400); + s.db.prepare('DELETE FROM connector_accounts').run(); + expect((await s.select()).status).toBe(400); + s.db.close(); +}); + +it('a replacement token preserves original post history while resetting inaccessible-account preferences', async () => { + const s = await setup(); await s.match(); await s.select(); await drainQueue(s.db, s.http); + s.db.prepare("UPDATE connector_accounts SET options='{" + '"sync_highlights":true' + "}' WHERE user_id=?").run(s.uid); + upsertAccount(s.db, s.uid, 'microblog', { token: 'replacement', microblogDestination: 'https://replacement.micro.blog/' }, null); + expect(s.db.prepare('SELECT url FROM microblog_clipping_posts').get()).toEqual({ url: 'https://example.micro.blog/highlights.html' }); + expect(s.db.prepare("SELECT options FROM connector_accounts WHERE user_id=?").get(s.uid)).toEqual({ options: '{}' }); + expect(s.db.prepare('SELECT COUNT(*) n FROM connector_matches WHERE user_id=?').get(s.uid)).toEqual({ n: 0 }); + s.db.close(); +}); + +it.each(['book', 'account'])('clears local publication data on %s removal without deleting the remote post', async (scope) => { + const s = await setup(); await s.match(); await s.select(); await drainQueue(s.db, s.http); + if (scope === 'book') deleteDocumentData(s.db, s.uid, DOC); + else deleteKosyncUserData(s.db, s.uid, s.headers['x-auth-user'], { keepUser: true }); + expect(s.db.prepare('SELECT COUNT(*) n FROM microblog_clipping_posts').get()).toEqual({ n: 0 }); + expect(s.db.prepare('SELECT COUNT(*) n FROM microblog_clipping_selections').get()).toEqual({ n: 0 }); + expect(s.posts).toHaveLength(1); + s.db.close(); +}); + +it('coalesces repeated selections and clears local publication state on unlink', async () => { + const s = await setup(); await s.match(); + await s.select(); await s.select(); + expect(s.db.prepare("SELECT COUNT(*) n FROM connector_queue WHERE kind='microblog:clippings'").get()).toEqual({ n: 1 }); + await drainQueue(s.db, s.http); + await s.app.request('/api/v1/connectors/microblog', { method: 'DELETE', headers: s.headers }); + expect(s.db.prepare('SELECT COUNT(*) n FROM microblog_clipping_posts').get()).toEqual({ n: 0 }); + expect(s.db.prepare('SELECT COUNT(*) n FROM microblog_clipping_selections').get()).toEqual({ n: 0 }); + expect(s.posts).toHaveLength(1); + s.db.close(); +}); + +it('limits per-book sync to that book while account-wide sync still queues other selected books', async () => { + const s = await setup(); + const other = 'b'.repeat(32); + await s.put('/api/v1/documents', { items: [{ document: other, title: 'Other book', author: 'Author' }] }); + await s.put(`/api/v1/clippings/${other}`, { items: [{ id: 'other', spine: 1, text: 'Other passage', created_at: 1 }] }); + await s.match(); + await s.put(`/api/v1/connectors/microblog/matches/${other}`, { external_id: '456', external_edition: '9780547951973' }); + await s.select(); + await s.app.request(`/api/v1/connectors/microblog/clippings/${other}/other`, { method: 'POST', headers: s.headers }); + s.db.prepare('DELETE FROM connector_queue').run(); + const res = await s.app.request(`/api/v1/connectors/microblog/sync/${DOC}`, { method: 'POST', headers: s.headers }); + expect(res.status).toBe(200); + expect(await res.json()).toMatchObject({ queued: 1 }); + expect(s.db.prepare("SELECT document FROM connector_queue WHERE kind='microblog:clippings'").all()).toEqual([{ document: DOC }]); + await drainQueue(s.db, s.http); + expect(s.posts).toHaveLength(1); + expect(JSON.stringify(s.posts[0])).not.toContain('Other passage'); + const all = await s.app.request('/api/v1/connectors/microblog/sync', { method: 'POST', headers: s.headers }); + expect(await all.json()).toMatchObject({ queued: 1 }); + await drainQueue(s.db, s.http); + expect(s.posts).toHaveLength(2); + expect(JSON.stringify(s.posts[1])).toContain('Other passage'); + s.db.close(); +}); diff --git a/test/microblog-clippings-post.test.ts b/test/microblog-clippings-post.test.ts new file mode 100644 index 00000000..9f641fd6 --- /dev/null +++ b/test/microblog-clippings-post.test.ts @@ -0,0 +1,506 @@ +import { afterEach, beforeEach, expect, it } from 'vitest'; +import { makeTestApp, registerUser, DOC } from './helpers.js'; +import { resetEncryptionKeyCache } from '../src/crypto/secrets.js'; +import { upsertAccount, saveMatch } from '../src/connectors/store.js'; +import { clippingsPost, publishMicroblogClippings, selectMicroblogClipping, scheduleMicroblogClippings } from '../src/connectors/microblog-clippings.js'; +import { drainQueue as runQueue } from '../src/connectors/runner.js'; +import type { HttpTransport } from '../src/connectors/types.js'; +import { mergeDocuments } from '../src/models/merge.js'; + +beforeEach(() => { process.env.TOKEN_ENC_KEY = 'a'.repeat(64); resetEncryptionKeyCache(); }); +afterEach(() => { delete process.env.TOKEN_ENC_KEY; resetEncryptionKeyCache(); }); +const drainQueue: typeof runQueue = (db, http, limit, now) => runQueue(db, async (url, init) => { + if (new URL(url).searchParams.get('q') === 'config') { + const uid = init.headers?.authorization === 'Bearer replacement' ? 'https://replacement.micro.blog/' : 'https://example.micro.blog/'; + return response(200, { destination: [{ uid, name: 'Test blog' }] }); + } + return http!(url, init); +}, limit, now); +const remoteUrl = 'https://example.micro.blog/2026/10/28/highlights.html'; +async function setup() { + const { app, db } = makeTestApp(); + const { username } = await registerUser(app); + const uid = (db.prepare('SELECT id FROM users WHERE username=?').get(username) as { id: number }).id; + upsertAccount(db, uid, 'microblog', { token: 'token', microblogDestination: 'https://example.micro.blog/' }, null); + db.prepare('INSERT INTO documents(user_id,document,title,author,updated_at) VALUES(?,?,?,?,0)').run(uid, DOC, 'Test book', 'Author'); + db.prepare('INSERT INTO clippings(user_id,document,id,text,spine_index,created_at,updated_at) VALUES(?,?,?,?,0,1,1)').run(uid, DOC, 'one', ''); + selectMicroblogClipping(db, uid, DOC, 'one'); + saveMatch(db, uid, 'microblog', DOC, { externalId: '123', externalEdition: '9781786487849', confidence: 1 }, 'manual'); + scheduleMicroblogClippings(db, uid, DOC); + return { db, uid }; +} +function response(status: number, body: unknown = {}, location?: string) { + return { status, json: async () => body, text: async () => JSON.stringify(body), headers: { get: (key: string) => key.toLowerCase() === 'location' ? location ?? null : null } }; +} + +it('allows retry after an account read fails before publication starts', async () => { + const { db, uid } = await setup(); + db.exec('ALTER TABLE connector_accounts RENAME TO unavailable_accounts'); + try { + await expect(publishMicroblogClippings( + { token: 'token' }, { externalId: '123', externalEdition: '9781786487849', confidence: 1 }, + { kind: 'highlight', document: DOC, timestamp: 1 }, async () => response(201, {}, remoteUrl), { db, userId: uid }, + )).rejects.toThrow('no such table'); + } finally { db.exec('ALTER TABLE unavailable_accounts RENAME TO connector_accounts'); } + await drainQueue(db, async () => response(201, {}, remoteUrl)); + expect(db.prepare('SELECT url FROM microblog_clipping_posts').get()).toEqual({ url: remoteUrl }); + expect(db.prepare('SELECT status FROM connector_queue').get()).toEqual({ status: 'done' }); + db.close(); +}); + +it('publishes one escaped collected post, skips unchanged content, then updates the saved URL', async () => { + const { db, uid } = await setup(); + const writes: Record[] = []; + const http: HttpTransport = async (url, init) => { + expect(url).toContain('https://micro.blog/micropub'); + if (init.method === 'GET') return response(200, { properties: { url: [remoteUrl] } }); + expect(init.headers?.['content-type']).toBe('application/json'); + const body = JSON.parse(init.body!); writes.push(body); + return response(body.action ? 200 : 201, {}, body.action ? undefined : remoteUrl); + }; + await drainQueue(db, http); + expect(writes).toHaveLength(1); + expect(JSON.stringify(writes[0])).toContain('<First passage>'); + expect(JSON.stringify(writes[0])).toContain('https://micro.blog/books/9781786487849'); + scheduleMicroblogClippings(db, uid, DOC); + await drainQueue(db, http); + expect(writes).toHaveLength(1); + db.prepare('UPDATE clippings SET note=? WHERE user_id=?').run('A note', uid); + scheduleMicroblogClippings(db, uid, DOC); + await drainQueue(db, http); + expect(writes).toHaveLength(2); + expect(writes[1]).toMatchObject({ action: 'update', url: remoteUrl }); + // Micro.blog's update API accepts the generated HTML as a string, unlike + // the html-value object supported on initial creation. + expect((writes[1] as any).replace.content).toEqual([expect.stringContaining('<First passage>')]); + expect(JSON.stringify(writes[1])).toContain('A note'); + db.close(); +}); + +it('retains fallback positions in published snapshots and preserves reading order after local deletion', async () => { + const { db, uid } = await setup(); + db.prepare('UPDATE clippings SET paragraph_index=1,start_page=2,start_word=3,created_at=2 WHERE user_id=?').run(uid); + db.prepare('INSERT INTO clippings(user_id,document,id,text,spine_index,paragraph_index,start_page,start_word,created_at,updated_at) VALUES(?,?,?,?,0,2,4,10,1,1)') + .run(uid, DOC, 'two', 'Second passage'); + selectMicroblogClipping(db, uid, DOC, 'two'); + const writes: unknown[] = []; + const http: HttpTransport = async (_url, init) => { + if (init.method === 'GET') return response(200); + writes.push(JSON.parse(init.body!)); + return response(201, {}, remoteUrl); + }; + await drainQueue(db, http); + const snapshot = db.prepare('SELECT published_snapshot FROM microblog_clipping_selections WHERE user_id=? AND clipping_id=?').get(uid, 'one') as { published_snapshot: string }; + expect(JSON.parse(snapshot.published_snapshot)).toMatchObject({ paragraph_index: 1, start_page: 2, start_word: 3 }); + db.prepare('UPDATE clippings SET deleted=1 WHERE user_id=? AND id=?').run(uid, 'one'); + db.prepare('UPDATE clippings SET note=? WHERE user_id=? AND id=?').run('New note', uid, 'two'); + scheduleMicroblogClippings(db, uid, DOC); await drainQueue(db, http); + expect(writes).toHaveLength(2); + for (const write of writes) { + const content = JSON.stringify(write); + expect(content.indexOf('First passage')).toBeLessThan(content.indexOf('Second passage')); + } + db.close(); +}); + +it.each([false, true])('publishes selected clippings after merging an unmatched duplicate (scheduling failure: %s)', async (failScheduling) => { + const { db, uid } = await setup(); + const writes: any[] = []; + const http: HttpTransport = async (_url, init) => { + if (init.method === 'GET') return response(200); + const body = JSON.parse(init.body!); writes.push(body); + return response(body.action ? 200 : 201, {}, remoteUrl); + }; + await drainQueue(db, http); + const source = 'b'.repeat(32); + db.prepare('INSERT INTO documents(user_id,document,title,updated_at) VALUES(?,?,?,0)').run(uid, source, 'Duplicate'); + db.prepare('INSERT INTO clippings(user_id,document,id,text,spine_index,created_at,updated_at) VALUES(?,?,?,?,1,2,2)').run(uid, source, 'merged', 'Merged passage'); + selectMicroblogClipping(db, uid, source, 'merged'); + expect(scheduleMicroblogClippings(db, uid, source)).toBe(0); + if (failScheduling) db.exec(`CREATE TEMP TRIGGER fail_merge_schedule BEFORE INSERT ON connector_queue + WHEN NEW.connector_id='microblog' BEGIN SELECT RAISE(FAIL, 'simulated merge scheduling failure'); END`); + expect(() => mergeDocuments(db, uid, source, DOC, 10)).not.toThrow(); + expect(db.prepare('SELECT document FROM document_aliases WHERE user_id=? AND alias=?').get(uid, source)).toEqual({ document: DOC }); + expect(db.prepare('SELECT document FROM microblog_clipping_selections WHERE user_id=? AND clipping_id=?').get(uid, 'merged')).toEqual({ document: DOC }); + if (failScheduling) { + db.exec('DROP TRIGGER fail_merge_schedule'); + expect(scheduleMicroblogClippings(db, uid, DOC)).toBe(1); + } + expect(db.prepare("SELECT status FROM connector_queue WHERE kind='microblog:clippings'").get()).toEqual({ status: 'pending' }); + await drainQueue(db, http); + expect(writes).toHaveLength(2); + expect(writes[1]).toMatchObject({ action: 'update', url: remoteUrl }); + expect(JSON.stringify(writes[1])).toContain('First passage'); + expect(JSON.stringify(writes[1])).toContain('Merged passage'); + expect(writes.filter((w) => !w.action)).toHaveLength(1); + db.close(); +}); + +it.each(['source lookup', 'create', 'update'])('preserves a merge conflict raised during an in-flight %s', async (stage) => { + const { db, uid } = await setup(); + if (stage !== 'create') db.prepare('UPDATE microblog_clipping_posts SET url=?,attempted=1').run(remoteUrl); + const source = 'b'.repeat(32); + db.prepare('INSERT INTO documents(user_id,document,title,updated_at) VALUES(?,?,?,0)').run(uid, source, 'Duplicate'); + db.prepare('INSERT INTO clippings(user_id,document,id,text,spine_index,created_at,updated_at) VALUES(?,?,?,?,1,2,2)') + .run(uid, source, 'merged', 'Merged passage'); + selectMicroblogClipping(db, uid, source, 'merged'); + clippingsPost(db, uid, source); + db.prepare('UPDATE microblog_clipping_posts SET attempted=1,url=? WHERE user_id=? AND document=?') + .run('https://example.micro.blog/duplicate.html', uid, source); + let merged = false; + const writes: any[] = []; + const http: HttpTransport = async (_url, init) => { + const body = init.method === 'POST' ? JSON.parse(init.body!) : null; + if (body) writes.push(body); + const operation = body ? body.action === 'update' ? 'update' : 'create' : 'source lookup'; + if (!merged && operation === stage) { + merged = true; + mergeDocuments(db, uid, source, DOC, 10); + } + return response(body && !body.action ? 201 : 200, {}, remoteUrl); + }; + await drainQueue(db, http); + expect(merged).toBe(true); + expect(db.prepare('SELECT error,fingerprint FROM microblog_clipping_posts WHERE user_id=? AND document=?').get(uid, DOC)).toEqual({ + error: expect.stringContaining('Merged books both have Micro.blog publication history'), fingerprint: null, + }); + expect(db.prepare("SELECT status,last_error FROM connector_queue WHERE document=? AND kind='microblog:clippings'").get(DOC)).toEqual({ + status: 'dead', last_error: expect.stringContaining('Merged books both have Micro.blog publication history'), + }); + expect(db.prepare('SELECT COUNT(*) AS n FROM microblog_clipping_selections WHERE published_snapshot IS NOT NULL').get()).toEqual({ n: 0 }); + expect(writes).toHaveLength(stage === 'source lookup' ? 0 : 1); + expect(scheduleMicroblogClippings(db, uid, DOC)).toBe(0); + await drainQueue(db, http); + expect(writes).toHaveLength(stage === 'source lookup' ? 0 : 1); + db.close(); +}); + +it.each([500, 408].flatMap((status) => ['source lookup', 'update'].map((operation) => ({ status, operation }))))('identifies a failed $operation ($status) and preserves the existing post for retry', async ({ operation, status }) => { + const { db, uid } = await setup(); + const http: HttpTransport = async (_url, init) => { + if (init.method === 'GET') return response(operation === 'source lookup' ? status : 200); + return response(status); + }; + db.prepare('UPDATE microblog_clipping_posts SET url=?,attempted=1').run(remoteUrl); + await drainQueue(db, http); + expect(db.prepare('SELECT status,last_error FROM connector_queue').get()).toMatchObject({ + status: 'pending', last_error: expect.stringContaining(`Micropub ${operation} failed (${status})`), + }); + expect(db.prepare('SELECT url,missing,error,fingerprint FROM microblog_clipping_posts WHERE user_id=?').get(uid)).toEqual({ + url: remoteUrl, missing: 0, error: null, fingerprint: null, + }); + db.close(); +}); + +it.each(['connection loss', 'create timeout', 'recovery timeout'])('recovers an uncertain creation after %s without creating a second post', async (failure) => { + const { db } = await setup(); + let creates = 0; + let saved: unknown; + let timedOut = false; + const http: HttpTransport = async (url, init) => { + expect(url).toContain('/micropub'); + if (init.method === 'GET') { + if (failure === 'recovery timeout' && !timedOut) { timedOut = true; return response(408); } + return response(200, { items: [{ properties: { ...(saved as any), url: [remoteUrl] } }] }); + } + const body = JSON.parse(init.body!); + if (!body.action) { + creates++; saved = body.properties; + if (failure === 'create timeout') return response(408); + throw new Error('connection lost after acceptance'); + } + return response(200); + }; + await drainQueue(db, http); + expect(db.prepare('SELECT attempted,error FROM microblog_clipping_posts').get()).toEqual({ attempted: 1, error: null }); + expect(db.prepare('SELECT status FROM connector_queue').get()).toEqual({ status: 'pending' }); + db.prepare("UPDATE connector_queue SET next_try_at=0 WHERE connector_id='microblog'").run(); + await drainQueue(db, http); + if (failure === 'recovery timeout') { + expect(db.prepare('SELECT status FROM connector_queue').get()).toEqual({ status: 'pending' }); + db.prepare("UPDATE connector_queue SET next_try_at=0 WHERE connector_id='microblog'").run(); + await drainQueue(db, http); + } + expect(creates).toBe(1); + expect(db.prepare('SELECT url FROM microblog_clipping_posts').get()).toEqual({ url: remoteUrl }); + expect(db.prepare('SELECT status FROM connector_queue').get()).toEqual({ status: 'done' }); + db.close(); +}); + +it.each([401, 403, 429, 500])('handles HTTP %s without marking an upload successful', async (status) => { + const { db } = await setup(); + await drainQueue(db, async () => response(status)); + const row = db.prepare('SELECT status,last_error FROM connector_queue').get() as { status: string; last_error: string }; + expect(row.status).toBe(status < 429 ? 'dead' : 'pending'); + expect(row.last_error).toContain(String(status)); + expect(db.prepare('SELECT fingerprint FROM microblog_clipping_posts').get()).toEqual({ fingerprint: null }); + if (status === 401 || status === 403) expect(db.prepare('SELECT status FROM connector_accounts').get()).toEqual({ status: 'needs_reauth' }); + db.close(); +}); + +it.each([400, 413, 422])('allows a corrected initial post after a definite validation rejection (%s)', async (status) => { + const { db, uid } = await setup(); + const creates: any[] = []; + let rejected = true; + const http: HttpTransport = async (_url, init) => { + if (init.method === 'GET') throw new Error('A rejected create must not need uncertain-creation recovery'); + const body = JSON.parse(init.body!); + expect(body.action).toBeUndefined(); + creates.push(body); + return rejected ? response(status, { error: 'invalid_request', error_description: 'Content is too large' }) + : response(201, {}, remoteUrl); + }; + await drainQueue(db, http); + expect(db.prepare('SELECT attempted,error,url FROM microblog_clipping_posts').get()).toEqual({ attempted: 0, error: null, url: null }); + expect(db.prepare('SELECT status FROM connector_queue').get()).toEqual({ status: 'dead' }); + db.prepare('UPDATE clippings SET text=? WHERE user_id=?').run('Short corrected passage', uid); + rejected = false; + expect(scheduleMicroblogClippings(db, uid, DOC)).toBe(1); + await drainQueue(db, http); + expect(creates).toHaveLength(2); + expect(JSON.stringify(creates[1])).toContain('Short corrected passage'); + expect(db.prepare('SELECT url,error FROM microblog_clipping_posts').get()).toEqual({ url: remoteUrl, error: null }); + expect(db.prepare('SELECT status FROM connector_queue').get()).toEqual({ status: 'done' }); + db.close(); +}); + +it('does not recreate a missing published post', async () => { + const { db } = await setup(); + db.prepare('UPDATE microblog_clipping_posts SET url=?,attempted=1').run(remoteUrl); + let creates = 0; + await drainQueue(db, async (_url, init) => { if (init.method === 'POST' && !JSON.parse(init.body!).action) creates++; return response(404); }); + expect(creates).toBe(0); + expect(db.prepare('SELECT status FROM connector_queue').get()).toEqual({ status: 'dead' }); + db.close(); +}); + +it.each(['missing', 'ambiguous', 'malformed'])('requires attention after %s recovery instead of creating again', async (mode) => { + const { db } = await setup(); + let creates = 0; + let properties: any; + const http: HttpTransport = async (_url, init) => { + if (init.method === 'GET') { + if (mode === 'malformed') return response(200, {}); + const item = { properties: { ...properties, url: [remoteUrl] } }; + return response(200, { items: mode === 'missing' ? [] : [item, { properties: { ...properties, url: ['https://example.micro.blog/duplicate.html'] } }] }); + } + creates++; properties = JSON.parse(init.body!).properties; + return response(201); // Accepted, but Location was lost. + }; + await drainQueue(db, http); + db.prepare('UPDATE connector_queue SET next_try_at=0').run(); + await drainQueue(db, http); + expect(creates).toBe(1); + const state = db.prepare('SELECT url,error FROM microblog_clipping_posts').get() as { url: string | null; error: string | null }; + expect(state.url).toBeNull(); + if (mode !== 'malformed') expect(state.error).toContain('could not be resolved'); + db.close(); +}); + +it('recovers across source pagination', async () => { + const { db } = await setup(); + let properties: any; + let creates = 0; + const pages: string[] = []; + const http: HttpTransport = async (url, init) => { + if (init.method === 'GET') { + if (url.includes('url=')) return response(200, { properties }); + pages.push(url); + return response(200, { items: url.includes('offset=0') ? Array.from({ length: 100 }, () => ({ properties: { content: ['Other post'] } })) + : [{ properties: { ...properties, url: [remoteUrl] } }] }); + } + const body = JSON.parse(init.body!); + if (!body.action) { creates++; properties = body.properties; throw new Error('lost response'); } + return response(200); + }; + await drainQueue(db, http); + db.prepare('UPDATE connector_queue SET next_try_at=0').run(); + await drainQueue(db, http); + expect(creates).toBe(1); + expect(pages.some((p) => p.includes('offset=100'))).toBe(true); + expect(db.prepare('SELECT url FROM microblog_clipping_posts').get()).toEqual({ url: remoteUrl }); + db.close(); +}); + +it('retains published automatic passages when automatic sync is disabled and another is selected', async () => { + const { db, uid } = await setup(); + db.prepare("UPDATE microblog_clipping_selections SET manual=0").run(); + db.prepare("UPDATE connector_accounts SET options='{" + '"sync_highlights":true' + "}'").run(); + const posts: any[] = []; + const http: HttpTransport = async (_url, init) => { + if (init.method === 'GET') return response(200, {}); + const body = JSON.parse(init.body!); posts.push(body); + return response(body.action ? 200 : 201, {}, remoteUrl); + }; + await drainQueue(db, http); + db.prepare("UPDATE connector_accounts SET options='{}'").run(); + db.prepare("UPDATE clippings SET deleted=1").run(); + db.prepare('INSERT INTO clippings(user_id,document,id,text,spine_index,created_at,updated_at) VALUES(?,?,?,?,1,2,2)').run(uid, DOC, 'two', 'Second passage'); + selectMicroblogClipping(db, uid, DOC, 'two'); + scheduleMicroblogClippings(db, uid, DOC); + await drainQueue(db, http); + expect(posts).toHaveLength(2); + expect(JSON.stringify(posts[1])).toContain('<First passage>'); + expect(JSON.stringify(posts[1])).toContain('Second passage'); + db.close(); +}); + +it.each([false, true])('rechecks eligibility when automatic sync is disabled during source lookup (new manual clipping: %s)', async (manual) => { + const { db, uid } = await setup(); + const writes: any[] = []; + let disableOnLookup = false; + const http: HttpTransport = async (_url, init) => { + if (init.method === 'GET') { + if (disableOnLookup) { + disableOnLookup = false; + db.prepare("UPDATE connector_accounts SET options='{}' WHERE user_id=? AND connector_id='microblog'").run(uid); + } + return response(200); + } + const body = JSON.parse(init.body!); writes.push(body); + return response(body.action ? 200 : 201, {}, remoteUrl); + }; + await drainQueue(db, http); + db.prepare("UPDATE connector_accounts SET options=? WHERE user_id=? AND connector_id='microblog'").run('{"sync_highlights":true}', uid); + const insert = db.prepare('INSERT INTO clippings(user_id,document,id,text,spine_index,created_at,updated_at) VALUES(?,?,?,?,1,2,2)'); + insert.run(uid, DOC, 'two', 'Unpublished automatic passage'); + if (manual) { + insert.run(uid, DOC, 'three', 'New manually selected passage'); + selectMicroblogClipping(db, uid, DOC, 'three'); + } + scheduleMicroblogClippings(db, uid, DOC); + disableOnLookup = true; + await drainQueue(db, http); + expect(writes).toHaveLength(1); + expect(db.prepare('SELECT published_snapshot FROM microblog_clipping_selections WHERE clipping_id=?').get('two')).toEqual({ published_snapshot: null }); + expect(db.prepare('SELECT error FROM microblog_clipping_posts').get()).toEqual({ error: null }); + expect(db.prepare('SELECT status FROM connector_queue').get()).toEqual({ status: 'pending' }); + db.prepare('UPDATE connector_queue SET next_try_at=0').run(); + await drainQueue(db, http); + expect(writes).toHaveLength(manual ? 2 : 1); + expect(JSON.stringify(writes)).not.toContain('Unpublished automatic passage'); + if (manual) { + expect(writes[1]).toMatchObject({ action: 'update', url: remoteUrl }); + expect(JSON.stringify(writes[1])).toContain('New manually selected passage'); + expect(JSON.stringify(writes[1])).toContain('First passage'); + } + expect(db.prepare('SELECT status FROM connector_queue').get()).toEqual({ status: 'done' }); + expect(db.prepare('SELECT published_snapshot FROM microblog_clipping_selections WHERE clipping_id=?').get('two')).toEqual({ published_snapshot: null }); + db.close(); +}); + +it('queues a follow-up when a clipping is selected while the first post is being published', async () => { + const { db, uid } = await setup(); + let release!: () => void; + let started!: () => void; + const accepted = new Promise((r) => { started = r; }); + const held = new Promise((r) => { release = r; }); + const writes: any[] = []; + const http: HttpTransport = async (_url, init) => { + if (init.method === 'GET') return response(200, {}); + const body = JSON.parse(init.body!); writes.push(body); + if (!body.action) { started(); await held; } + return response(body.action ? 200 : 201, {}, remoteUrl); + }; + const running = drainQueue(db, http); + await accepted; + db.prepare('INSERT INTO clippings(user_id,document,id,text,spine_index,created_at,updated_at) VALUES(?,?,?,?,1,2,2)').run(uid, DOC, 'two', 'Selected during upload'); + selectMicroblogClipping(db, uid, DOC, 'two'); + scheduleMicroblogClippings(db, uid, DOC); + release(); await running; + await drainQueue(db, http); + expect(writes).toHaveLength(2); + expect(JSON.stringify(writes[1])).toContain('Selected during upload'); + db.close(); +}); + +it('an in-flight old-account upload cannot overwrite replacement-account state or acknowledge its queue row', async () => { + const { db, uid } = await setup(); + let release!: () => void; + let started!: () => void; + const accepted = new Promise((r) => { started = r; }); + const held = new Promise((r) => { release = r; }); + const http: HttpTransport = async (_url, init) => { + if (init.headers?.authorization === 'Bearer token') { started(); await held; return response(201, {}, remoteUrl); } + return response(201, {}, 'https://replacement.micro.blog/highlights.html'); + }; + const running = drainQueue(db, http); await accepted; + upsertAccount(db, uid, 'microblog', { token: 'replacement', microblogDestination: 'https://replacement.micro.blog/' }, null); + saveMatch(db, uid, 'microblog', DOC, { externalId: '456', externalEdition: '9781786487849', confidence: 1 }, 'manual'); + selectMicroblogClipping(db, uid, DOC, 'one'); scheduleMicroblogClippings(db, uid, DOC); + release(); await running; + expect(db.prepare('SELECT url,fingerprint FROM microblog_clipping_posts').get()).toEqual({ url: null, fingerprint: null }); + expect(db.prepare('SELECT status FROM connector_queue').get()).toEqual({ status: 'pending' }); + await drainQueue(db, http); + expect(db.prepare('SELECT url,destination FROM microblog_clipping_posts').get()).toEqual({ url: null, destination: 'https://example.micro.blog/' }); + db.close(); +}); + +it('rotating a token for the same verified blog preserves the collected post and manual selections', async () => { + const { db, uid } = await setup(); + let creates = 0; + const http: HttpTransport = async (_url, init) => { + if (init.method === 'GET') return response(200, {}); + if (!JSON.parse(init.body!).action) creates++; + return response(201, {}, remoteUrl); + }; + await drainQueue(db, http); + upsertAccount(db, uid, 'microblog', { token: 'rotated', microblogDestination: 'https://example.micro.blog/' }, null); + expect(db.prepare('SELECT url FROM microblog_clipping_posts').get()).toEqual({ url: remoteUrl }); + expect(db.prepare('SELECT manual FROM microblog_clipping_selections').get()).toEqual({ manual: 1 }); + scheduleMicroblogClippings(db, uid, DOC); await drainQueue(db, http); + expect(creates).toBe(1); + db.close(); +}); + +it('updates the book link on the existing post after correcting a manual match', async () => { + const { db, uid } = await setup(); + const writes: any[] = []; + const http: HttpTransport = async (_url, init) => { + if (init.method === 'GET') return response(200, {}); + writes.push(JSON.parse(init.body!)); return response(201, {}, remoteUrl); + }; + await drainQueue(db, http); + saveMatch(db, uid, 'microblog', DOC, { externalId: '456', externalEdition: '9780143127796', confidence: 1 }, 'manual'); + scheduleMicroblogClippings(db, uid, DOC); await drainQueue(db, http); + expect(writes).toHaveLength(2); + expect(writes[1]).toMatchObject({ action: 'update', url: remoteUrl }); + expect(JSON.stringify(writes[1])).toContain('https://micro.blog/books/9780143127796'); + db.close(); +}); + +it.each(['config', 'source lookup', 'post recovery'].flatMap((stage) => + ['disabled', 'book', 'edition'].map((change) => ({ stage, change }))))('stops publication when the match changes during $stage ($change)', async ({ stage, change }) => { + const { db, uid } = await setup(); + if (stage === 'source lookup') db.prepare('UPDATE microblog_clipping_posts SET url=?,attempted=1').run(remoteUrl); + if (stage === 'post recovery') db.prepare('UPDATE microblog_clipping_posts SET attempted=1,destination=?').run('https://example.micro.blog/'); + const before = db.prepare('SELECT * FROM microblog_clipping_posts').get(); + let changed = false; + const writes: unknown[] = []; + const http: HttpTransport = async (url, init) => { + const parsed = new URL(url); + if (init.method === 'POST') { writes.push(JSON.parse(init.body!)); return response(201, {}, remoteUrl); } + const requestStage = parsed.searchParams.get('q') === 'config' ? 'config' + : parsed.searchParams.has('url') ? 'source lookup' : 'post recovery'; + if (requestStage === stage && !changed) { + changed = true; + saveMatch(db, uid, 'microblog', DOC, change === 'disabled' ? null : { + externalId: change === 'book' ? '456' : '123', externalEdition: '9780143127796', confidence: 1, + }, 'manual'); + } + if (requestStage === 'config') return response(200, { destination: [{ uid: 'https://example.micro.blog/' }] }); + if (requestStage === 'post recovery') return response(200, { items: [{ properties: { + content: [``], url: [remoteUrl], + } }] }); + return response(200); + }; + // Use the real transport directly so the discovery request can change the match. + await runQueue(db, http); + expect(changed).toBe(true); + expect(writes).toEqual([]); + expect(db.prepare('SELECT url,fingerprint,error,missing FROM microblog_clipping_posts').get()).toMatchObject({ + url: stage === 'source lookup' ? remoteUrl : null, fingerprint: null, error: null, missing: 0, + }); + expect(db.prepare('SELECT last_error FROM connector_queue').get()).toMatchObject({ last_error: expect.stringContaining('match changed') }); + db.close(); +}); diff --git a/test/microblog-clippings.test.ts b/test/microblog-clippings.test.ts new file mode 100644 index 00000000..8016bf4b --- /dev/null +++ b/test/microblog-clippings.test.ts @@ -0,0 +1,106 @@ +import { afterEach, beforeEach, expect, it } from 'vitest'; +import { makeTestApp, registerUser, DOC } from './helpers.js'; +import { resetEncryptionKeyCache } from '../src/crypto/secrets.js'; +import { upsertAccount, saveMatch } from '../src/connectors/store.js'; +import { selectMicroblogClipping, scheduleMicroblogClippings, microblogClippingsContent } from '../src/connectors/microblog-clippings.js'; +import { mergeDocuments } from '../src/models/merge.js'; + +beforeEach(() => { process.env.TOKEN_ENC_KEY = 'a'.repeat(64); resetEncryptionKeyCache(); }); +afterEach(() => { delete process.env.TOKEN_ENC_KEY; resetEncryptionKeyCache(); }); + +async function setup() { + const { app, db } = makeTestApp(); + const { username } = await registerUser(app); + const uid = (db.prepare('SELECT id FROM users WHERE username=?').get(username) as { id: number }).id; + upsertAccount(db, uid, 'microblog', { token: 'token' }, null); + db.prepare('INSERT INTO documents(user_id,document,title,author,updated_at) VALUES(?,?,?,?,0)').run(uid, DOC, 'Test book', 'Author'); + for (const [id, text, spine] of [['one', 'First passage', 1], ['two', 'Second passage', 2]] as const) { + db.prepare('INSERT INTO clippings(user_id,document,id,text,spine_index,created_at,updated_at) VALUES(?,?,?,?,?,1,1)').run(uid, DOC, id, text, spine); + } + return { db, uid }; +} + +it('persists manual intent without a match and queues one book after matching', async () => { + const { db, uid } = await setup(); + selectMicroblogClipping(db, uid, DOC, 'two'); + expect(scheduleMicroblogClippings(db, uid, DOC)).toBe(0); + saveMatch(db, uid, 'microblog', DOC, { externalId: '123', externalEdition: '9781786487849', confidence: 1 }, 'manual'); + expect(scheduleMicroblogClippings(db, uid, DOC)).toBe(1); + scheduleMicroblogClippings(db, uid, DOC); + expect(db.prepare("SELECT COUNT(*) n FROM connector_queue WHERE connector_id='microblog'").get()).toEqual({ n: 1 }); + const content = microblogClippingsContent(db, uid, DOC, 'https://micro.blog/books/9781786487849'); + expect(content?.title).toBe('📚 Clippings from Test book'); + expect(content?.content).toContain('Second passage'); + expect(content?.content).not.toContain('First passage'); + expect(microblogClippingsContent(db, uid, DOC, 'https://micro.blog/books/9781786487849')?.fingerprint).toBe(content?.fingerprint); + db.close(); +}); + +it('automatic mode includes existing clippings in reading order and isolates users', async () => { + const { db, uid } = await setup(); + db.prepare("UPDATE connector_accounts SET options=? WHERE user_id=? AND connector_id='microblog'").run('{"sync_highlights":true}', uid); + scheduleMicroblogClippings(db, uid, DOC); + const content = microblogClippingsContent(db, uid, DOC, 'https://micro.blog/books/9781786487849'); + expect(content?.content.indexOf('First passage')).toBeLessThan(content!.content.indexOf('Second passage')); + expect(microblogClippingsContent(db, uid + 1, DOC, 'https://micro.blog/books/9781786487849')).toBeNull(); + db.close(); +}); + +it.each([ + ['text offsets', { start_offset: 10 }, { start_offset: 30 }], + ['paragraphs', { paragraph_index: 1 }, { paragraph_index: 2 }], + ['zero-based paragraphs', { paragraph_index: 0 }, { paragraph_index: 1 }], + ['pages', { start_page: 2 }, { start_page: 4 }], + ['words within a page', { start_page: 2, start_word: 3 }, { start_page: 2, start_word: 10 }], + ['mixed offset availability', { start_offset: 100, paragraph_index: 1 }, { paragraph_index: 2 }], +] as const)('uses %s before creation time when ordering clippings in the same spine', async (_label, first, second) => { + const { db, uid } = await setup(); + const update = db.prepare('UPDATE clippings SET spine_index=0,start_offset=?,paragraph_index=?,start_page=?,start_word=?,created_at=? WHERE user_id=? AND id=?'); + for (const [id, position, createdAt] of [['one', first, 2], ['two', second, 1]] as const) { + const p = { start_offset: null, paragraph_index: null, start_page: 0, start_word: 0, ...position }; + update.run(p.start_offset, p.paragraph_index, p.start_page, p.start_word, createdAt, uid, id); + selectMicroblogClipping(db, uid, DOC, id); + } + const content = microblogClippingsContent(db, uid, DOC, 'https://micro.blog/books/9781786487849')!.content; + expect(content.indexOf('First passage')).toBeLessThan(content.indexOf('Second passage')); + db.close(); +}); + +it('keeps legacy published snapshots without fallback positions readable', async () => { + const { db, uid } = await setup(); + for (const [id, text, created_at] of [['one', 'First passage', 1], ['two', 'Second passage', 2]] as const) { + selectMicroblogClipping(db, uid, DOC, id); + const snapshot = { id, text, note: null, chapter_title: '', spine_index: 0, start_offset: null, created_at }; + db.prepare('UPDATE microblog_clipping_selections SET manual=0,published_snapshot=? WHERE user_id=? AND clipping_id=?') + .run(JSON.stringify(snapshot), uid, id); + } + const content = microblogClippingsContent(db, uid, DOC, 'https://micro.blog/books/9781786487849')!.content; + expect(content.indexOf('First passage')).toBeLessThan(content.indexOf('Second passage')); + db.close(); +}); + +it('automatic-off excludes unpublished automatic selections but preserves manual intent', async () => { + const { db, uid } = await setup(); + db.prepare("UPDATE connector_accounts SET options=? WHERE user_id=?").run('{"sync_highlights":true}', uid); + scheduleMicroblogClippings(db, uid, DOC); + selectMicroblogClipping(db, uid, DOC, 'two'); + db.prepare("UPDATE connector_accounts SET options='{}' WHERE user_id=?").run(uid); + expect(microblogClippingsContent(db, uid, DOC, 'https://micro.blog/books/9781786487849')?.content).not.toContain('First passage'); + expect(microblogClippingsContent(db, uid, DOC, 'https://micro.blog/books/9781786487849')?.content).toContain('Second passage'); + db.close(); +}); + +it('keeps selected clippings and the remote post when merging books', async () => { + const { db, uid } = await setup(); + const target = 'b'.repeat(32); + selectMicroblogClipping(db, uid, DOC, 'one'); + microblogClippingsContent(db, uid, DOC, 'https://micro.blog/books/9781786487849'); + db.prepare('UPDATE microblog_clipping_posts SET url=?,attempted=1 WHERE user_id=?').run('https://example.micro.blog/highlights', uid); + mergeDocuments(db, uid, DOC, target, 10); + expect(microblogClippingsContent(db, uid, DOC, 'https://micro.blog/books/9781786487849')?.content).toContain('First passage'); + selectMicroblogClipping(db, uid, DOC, 'two'); + expect(microblogClippingsContent(db, uid, target, 'https://micro.blog/books/9781786487849')?.content).toContain('Second passage'); + expect(db.prepare('SELECT url,document FROM microblog_clipping_posts WHERE user_id=?').get(uid)) + .toEqual({ url: 'https://example.micro.blog/highlights', document: target }); + db.close(); +}); diff --git a/test/microblog-destination-posts.test.ts b/test/microblog-destination-posts.test.ts new file mode 100644 index 00000000..6e3eaf65 --- /dev/null +++ b/test/microblog-destination-posts.test.ts @@ -0,0 +1,228 @@ +import { afterEach, beforeEach, expect, it } from 'vitest'; +import { makeTestApp, registerUser, DOC } from './helpers.js'; +import { resetEncryptionKeyCache } from '../src/crypto/secrets.js'; +import { upsertAccount, saveMatch } from '../src/connectors/store.js'; +import { drainQueue } from '../src/connectors/runner.js'; +import { selectMicroblogClipping, scheduleMicroblogClippings } from '../src/connectors/microblog-clippings.js'; +import type { HttpTransport } from '../src/connectors/types.js'; +beforeEach(() => { process.env.TOKEN_ENC_KEY = 'a'.repeat(64); resetEncryptionKeyCache(); }); +afterEach(() => { delete process.env.TOKEN_ENC_KEY; resetEncryptionKeyCache(); }); +const a = 'https://first.micro.blog/'; const b = 'https://second.micro.blog/'; +async function setup() { + let blogs = [a, b]; let sourceStatus = 200; + let updateStatus = 200; let updateDescription = 'No post found for this URL.'; + const writes: { destination: string | null; body: any }[] = []; + const requests: string[] = []; + let onCreate: (() => Promise) | undefined; + let onDiscover: (() => void) | undefined; + const http: HttpTransport = async (url, init) => { + const parsed = new URL(url); + let status = 200; let body: unknown = {}; let location: string | null = null; + if (parsed.searchParams.get('q') === 'config') { onDiscover?.(); body = { destination: blogs.map((uid, i) => ({ uid, name: uid, 'microblog-default': i === 0 })) }; } + else { + requests.push(url); + const destination = parsed.searchParams.get('mp-destination'); + if (init.method === 'GET') { status = sourceStatus; body = { error: 'invalid_request', error_description: 'No post found for this URL.' }; } + else { + const data = JSON.parse(init.body!); writes.push({ destination, body: data }); + if (!data.action) { await onCreate?.(); status = 201; location = `${destination || a}post.html`; } + else if (data.action === 'update') { status = updateStatus; body = { error: 'invalid_request', error_description: updateDescription }; } + } + } + return { status, headers: { get: () => location }, json: async () => body, text: async () => JSON.stringify(body) }; + }; + const { app, db } = makeTestApp({}, { connectorTransport: http }); + const { headers, username } = await registerUser(app); + const uid = (db.prepare('SELECT id FROM users WHERE username=?').get(username) as { id: number }).id; + upsertAccount(db, uid, 'microblog', { token: 'token', microblogDestination: a }, null); + db.prepare('INSERT INTO documents(user_id,document,title,updated_at) VALUES(?,?,?,0)').run(uid, DOC, 'Test book'); + db.prepare('INSERT INTO clippings(user_id,document,id,text,spine_index,created_at,updated_at) VALUES(?,?,?,?,0,1,1)').run(uid, DOC, 'one', 'First passage'); + saveMatch(db, uid, 'microblog', DOC, { externalId: '123', externalEdition: '9781786487849', confidence: 1 }, 'manual'); + selectMicroblogClipping(db, uid, DOC, 'one'); + const choose = (destination: string) => app.request('/api/v1/connectors/microblog/destination', { method: 'PUT', headers, body: JSON.stringify({ destination }) }); + const sync = async () => { scheduleMicroblogClippings(db, uid, DOC); await drainQueue(db, http); }; + const recreate = (target?: string) => app.request(`/api/v1/connectors/microblog/clipping-posts/${DOC}/recreate`, { method: 'POST', headers, ...(target ? { body: JSON.stringify({ target }) } : {}) }); + const status = async () => (await app.request('/api/v1/connectors/microblog/clipping-posts', { headers })).json(); + return { db, app, uid, headers, http, writes, requests, choose, sync, recreate, status, + setBlogs: (value: string[]) => { blogs = value; }, missing: (code = 404) => { sourceStatus = code; }, + updateFailure: (code: number, description = 'No post found for this URL.') => { updateStatus = code; updateDescription = description; }, + onCreate: (fn: () => Promise) => { onCreate = fn; }, + onDiscover: (fn: () => void) => { onDiscover = fn; }, + change: () => db.prepare('UPDATE clippings SET note=? WHERE user_id=?').run('New annotation', uid) }; +} + +it('does not silently publish to the default when several blogs need a selection', async () => { + const s = await setup(); await s.sync(); expect(s.writes).toHaveLength(0); s.db.close(); +}); +it('creates 📚 collections on the selected blog and keeps updates there after the selection changes', async () => { + const s = await setup(); await s.choose(b); await s.sync(); + expect(s.writes[0]).toMatchObject({ destination: b, body: { properties: { name: ['📚 Clippings from Test book'] } } }); + await s.choose(a); s.change(); await s.sync(); + expect(s.writes).toHaveLength(2); expect(s.writes[1]).toMatchObject({ destination: b, body: { action: 'update', url: `${b}post.html` } }); + expect(s.requests.every((url) => new URL(url).searchParams.get('mp-destination') === b)).toBe(true); + s.db.close(); +}); +it('acknowledges a rematch with unchanged ISBN and content without requeuing publication', async () => { + const s = await setup(); await s.choose(b); await s.sync(); + saveMatch(s.db, s.uid, 'microblog', DOC, { externalId: '456', externalEdition: '9781786487849', confidence: 1 }, 'manual'); + await s.sync(); + expect(s.db.prepare('SELECT matched_book,book_url FROM microblog_clipping_posts WHERE user_id=? AND document=?').get(s.uid, DOC)) + .toEqual({ matched_book: '456', book_url: 'https://micro.blog/books/9781786487849' }); + expect(s.db.prepare("SELECT status FROM connector_queue WHERE kind='microblog:clippings'").get()).toEqual({ status: 'done' }); + expect(scheduleMicroblogClippings(s.db, s.uid, DOC)).toBe(0); + expect(s.writes).toHaveLength(1); + s.db.close(); +}); +it('pins creation before a concurrent change of the plugin-wide blog choice', async () => { + const s = await setup(); await s.choose(b); + s.onCreate(async () => { await s.choose(a); }); + await s.sync(); s.change(); await s.sync(); + expect(s.writes.every((w) => w.destination === b)).toBe(true); s.db.close(); +}); +it('uses the sole blog automatically and refuses revoked destinations', async () => { + const s = await setup(); s.setBlogs([b]); await s.sync(); expect(s.writes[0]?.destination).toBe(b); + s.setBlogs([a]); s.change(); await s.sync(); expect(s.writes).toHaveLength(1); s.db.close(); +}); +it('pauses a deleted post and explicitly recreates it on the original blog with saved and new clippings', async () => { + const s = await setup(); await s.choose(b); await s.sync(); s.missing(400); s.change(); await s.sync(); + expect(await s.status()).toMatchObject({ items: [{ document: DOC, missing: true, destination: b }] }); + await s.choose(a); + s.db.prepare('INSERT INTO clippings(user_id,document,id,text,spine_index,created_at,updated_at) VALUES(?,?,?,?,1,2,2)').run(s.uid, DOC, 'two', 'Second passage'); + selectMicroblogClipping(s.db, s.uid, DOC, 'two'); await s.sync(); expect(s.writes).toHaveLength(1); + expect((await s.recreate()).status).toBe(200); + await s.recreate(); // Cannot reset a replacement already awaiting publication. + await drainQueue(s.db, s.http); + expect(s.writes).toHaveLength(2); + expect(s.writes[1]).toMatchObject({ destination: b, body: { properties: { name: ['📚 Clippings from Test book'] } } }); + expect(JSON.stringify(s.writes[1])).toContain('First passage'); expect(JSON.stringify(s.writes[1])).toContain('Second passage'); + s.db.close(); +}); +it.each([401, 403, 429, 500])('does not offer recreation for an unconfirmed deletion (%s)', async (code) => { + const s = await setup(); await s.choose(b); await s.sync(); s.missing(code); s.change(); await s.sync(); + expect(await s.status()).toMatchObject({ items: [{ missing: false }] }); + expect((await s.recreate()).status).not.toBe(200); expect(s.writes).toHaveLength(1); s.db.close(); +}); +it.each([404, 400])('offers explicit recreation when the update reports a deleted post after a successful lookup (%s)', async (code) => { + const s = await setup(); await s.choose(b); await s.sync(); + s.updateFailure(code); s.change(); await s.sync(); + expect(await s.status()).toMatchObject({ items: [{ document: DOC, missing: true, destination: b, url: `${b}post.html` }] }); + await s.sync(); expect(s.writes).toHaveLength(2); // No automatic recreation. + s.missing(); + expect((await s.recreate()).status).toBe(200); + await drainQueue(s.db, s.http); + expect(s.writes.filter((w) => !w.body.action)).toHaveLength(2); + expect(JSON.stringify(s.writes[2])).toContain('New annotation'); + expect(await s.status()).toMatchObject({ items: [{ missing: false, error: null }] }); + s.db.close(); +}); +it.each([400, 401, 403, 429, 500])('does not mark an unrelated update failure as deletion (%s)', async (code) => { + const s = await setup(); await s.choose(b); await s.sync(); + s.updateFailure(code, 'Unsupported content format'); s.change(); await s.sync(); + expect(await s.status()).toMatchObject({ items: [{ missing: false, url: `${b}post.html` }] }); + expect((await s.recreate()).status).toBe(409); + expect(s.writes.filter((w) => !w.body.action)).toHaveLength(1); + s.db.close(); +}); +it.each([400, 413, 422])('can retry the existing post after correcting a rejected update (%s)', async (code) => { + const s = await setup(); await s.choose(b); await s.sync(); + s.updateFailure(code, 'Content is too large'); s.change(); await s.sync(); + expect(await s.status()).toMatchObject({ items: [{ missing: false, url: `${b}post.html`, error: null }] }); + expect(s.db.prepare("SELECT status FROM connector_queue WHERE kind='microblog:clippings'").get()).toEqual({ status: 'dead' }); + s.db.prepare('UPDATE clippings SET note=? WHERE user_id=?').run('Short corrected note', s.uid); + s.updateFailure(200); await s.sync(); + expect(s.writes).toHaveLength(3); + expect(s.writes[2]).toMatchObject({ destination: b, body: { action: 'update', url: `${b}post.html` } }); + expect(JSON.stringify(s.writes[2])).toContain('Short corrected note'); + expect(s.writes.filter((w) => !w.body.action)).toHaveLength(1); + expect(s.db.prepare("SELECT status FROM connector_queue WHERE kind='microblog:clippings'").get()).toEqual({ status: 'done' }); + await s.sync(); expect(s.writes).toHaveLength(3); + s.db.close(); +}); +it('clears a rejected update when removing an unpublished clipping restores the published content', async () => { + const s = await setup(); await s.choose(b); await s.sync(); + s.db.prepare('INSERT INTO clippings(user_id,document,id,text,spine_index,created_at,updated_at) VALUES(?,?,?,?,1,2,2)').run(s.uid, DOC, 'two', 'Rejected passage'); + selectMicroblogClipping(s.db, s.uid, DOC, 'two'); + s.updateFailure(413); await s.sync(); + s.db.prepare('UPDATE clippings SET deleted=1 WHERE user_id=? AND id=?').run(s.uid, 'two'); + await s.sync(); + expect(s.writes).toHaveLength(2); // The remote content is already correct. + expect(s.db.prepare("SELECT status,last_error FROM connector_queue WHERE kind='microblog:clippings'").get()).toEqual({ status: 'done', last_error: null }); + s.db.close(); +}); +it('recreates on the current clippings blog when explicitly chosen and pins later updates there', async () => { + const s = await setup(); await s.choose(b); await s.sync(); s.missing(); s.change(); await s.sync(); + await s.choose(a); + expect((await s.recreate('current')).status).toBe(200); + await drainQueue(s.db, s.http); + expect(s.writes[1]).toMatchObject({ destination: a, body: { properties: { name: ['📚 Clippings from Test book'] } } }); + expect(JSON.stringify(s.writes[1])).toContain('First passage'); + await s.choose(b); s.missing(200); + s.db.prepare('UPDATE clippings SET note=? WHERE user_id=?').run('Later annotation', s.uid); + await s.sync(); + expect(s.writes[2]).toMatchObject({ destination: a, body: { action: 'update', url: `${a}post.html` } }); + s.db.close(); +}); +it('allows explicit recreation on the current blog after access to the previous blog is lost', async () => { + const s = await setup(); await s.choose(b); await s.sync(); s.missing(); s.change(); await s.sync(); + await s.choose(a); s.setBlogs([a]); + expect((await s.recreate('current')).status).toBe(200); await drainQueue(s.db, s.http); + expect(s.writes[1]?.destination).toBe(a); s.db.close(); +}); +it.each(['before recreation', 'during discovery'])('preserves a paused post when its book match is removed %s', async (when) => { + const s = await setup(); await s.choose(b); await s.sync(); s.missing(); s.change(); await s.sync(); + await s.choose(a); + const before = s.db.prepare('SELECT * FROM microblog_clipping_posts').get(); + const removeMatch = () => { s.db.prepare("DELETE FROM connector_matches WHERE user_id=? AND connector_id='microblog'").run(s.uid); }; + if (when === 'before recreation') removeMatch(); + else s.onDiscover(removeMatch); + const result = await s.recreate('current'); + expect(result.status).toBe(409); + expect(await result.json()).toMatchObject({ message: expect.stringContaining('Match this book') }); + expect(s.db.prepare('SELECT * FROM microblog_clipping_posts').get()).toEqual(before); + expect(s.writes).toHaveLength(1); + s.onDiscover(() => {}); + saveMatch(s.db, s.uid, 'microblog', DOC, { externalId: '123', externalEdition: '9781786487849', confidence: 1 }, 'manual'); + expect((await s.recreate('current')).status).toBe(200); + await drainQueue(s.db, s.http); + expect(s.writes[1]?.destination).toBe(a); + s.db.close(); +}); +it('rejects an invalid recreation choice without changing the paused post', async () => { + const s = await setup(); await s.choose(b); await s.sync(); s.missing(); s.change(); await s.sync(); + expect((await s.recreate('elsewhere')).status).toBe(400); + expect(await s.status()).toMatchObject({ items: [{ missing: true, destination: b }] }); s.db.close(); +}); +it.each(['unselected', 'revoked'])('does not fall back to another blog when the current blog is %s', async (mode) => { + const s = await setup(); await s.choose(b); await s.sync(); s.missing(); s.change(); await s.sync(); + if (mode === 'unselected') s.db.prepare("UPDATE connector_accounts SET options='{}' WHERE user_id=?").run(s.uid); + else { await s.choose(a); s.setBlogs([b]); } + expect((await s.recreate('current')).status).toBe(409); + expect(await s.status()).toMatchObject({ items: [{ missing: true, destination: b }] }); + expect(s.writes).toHaveLength(1); s.db.close(); +}); +it('refuses to move a post that has been restored on the previous blog', async () => { + const s = await setup(); await s.choose(b); await s.sync(); s.missing(); s.change(); await s.sync(); + await s.choose(a); s.missing(200); + expect((await s.recreate('current')).status).toBe(409); + expect(await s.status()).toMatchObject({ items: [{ missing: true, destination: b }] }); + expect(s.writes).toHaveLength(1); s.db.close(); +}); +it('resumes an original post that has been restored instead of creating a duplicate', async () => { + const s = await setup(); await s.choose(b); await s.sync(); s.missing(); s.change(); await s.sync(); + s.missing(200); expect((await s.recreate()).status).toBe(200); await drainQueue(s.db, s.http); + expect(s.writes.filter((w) => !w.body.action)).toHaveLength(1); s.db.close(); +}); + +it('identifies a legacy post’s original blog rather than using the new plugin choice', async () => { + const s = await setup(); await s.choose(b); await s.sync(); + s.db.prepare('UPDATE microblog_clipping_posts SET destination=NULL').run(); + await s.choose(a); s.change(); await s.sync(); + expect(s.writes[1]?.destination).toBe(b); s.db.close(); +}); +it('does not publish automatic additions switched off while discovering blogs', async () => { + const s = await setup(); await s.choose(b); + s.db.prepare('UPDATE microblog_clipping_selections SET manual=0').run(); + s.db.prepare('UPDATE connector_accounts SET options=?').run(JSON.stringify({ highlight_blog: b, sync_highlights: true })); + s.onDiscover(() => { s.db.prepare('UPDATE connector_accounts SET options=?').run(JSON.stringify({ highlight_blog: b, sync_highlights: false })); }); + await s.sync(); expect(s.writes).toHaveLength(0); s.db.close(); +}); diff --git a/test/microblog-destinations.test.ts b/test/microblog-destinations.test.ts new file mode 100644 index 00000000..eec8bf69 --- /dev/null +++ b/test/microblog-destinations.test.ts @@ -0,0 +1,110 @@ +import { afterEach, beforeEach, expect, it } from 'vitest'; +import { makeTestApp, registerUser } from './helpers.js'; +import { resetEncryptionKeyCache } from '../src/crypto/secrets.js'; +import { saveMatch, setConnectorOptions, upsertAccount } from '../src/connectors/store.js'; +import { destinationForPost } from '../src/connectors/microblog-destinations.js'; +import { enqueue } from '../src/connectors/queue.js'; + +beforeEach(() => { process.env.TOKEN_ENC_KEY = 'a'.repeat(64); resetEncryptionKeyCache(); }); +afterEach(() => { delete process.env.TOKEN_ENC_KEY; resetEncryptionKeyCache(); }); +const a = 'https://first.micro.blog/'; +const b = 'https://second.micro.blog/'; +async function setup(destinations: unknown[]) { + const { app, db } = makeTestApp({}, { connectorTransport: async (url) => { + expect(url).toBe('https://micro.blog/micropub?q=config'); + return { status: 200, json: async () => ({ destination: destinations }), text: async () => '' }; + } }); + const { headers, username } = await registerUser(app); + const uid = (db.prepare('SELECT id FROM users WHERE username=?').get(username) as { id: number }).id; + upsertAccount(db, uid, 'microblog', { token: 'token' }, null); + const get = () => app.request('/api/v1/connectors/microblog/destinations', { headers }); + const save = (destination: string) => app.request('/api/v1/connectors/microblog/destination', { method: 'PUT', headers, body: JSON.stringify({ destination }) }); + return { app, db, uid, headers, get, save }; +} + +it('requires a choice with multiple blogs, validates saves, and preserves it through automatic-sync option changes', async () => { + const s = await setup([{ uid: a, name: 'First', 'microblog-default': true }, { uid: b, name: 'Second' }]); + const res = await s.get(); expect(res.status).toBe(200); + expect(await res.json()).toMatchObject({ selected: null, items: [{ uid: a, name: 'First' }, { uid: b, name: 'Second' }] }); + expect((await s.save('https://unavailable.micro.blog/')).status).toBe(400); + expect((await s.save(b)).status).toBe(200); + expect(await (await s.get()).json()).toMatchObject({ selected: b }); + await s.app.request('/api/v1/connectors/microblog/options', { method: 'PUT', headers: s.headers, body: JSON.stringify({ sync_highlights: true }) }); + expect(await (await s.get()).json()).toMatchObject({ selected: b }); + s.db.close(); +}); + +it('uses the sole valid blog and removes malformed or duplicate configuration entries', async () => { + const s = await setup([null, { uid: 'javascript:alert(1)' }, { uid: a, name: 'First' }, { uid: a, name: 'Duplicate' }]); + const res = await s.get(); expect(res.status).toBe(200); + expect(await res.json()).toEqual({ selected: a, items: [{ uid: a, name: 'First', isDefault: false }] }); + s.db.close(); +}); + +it('does not expose another user’s token or choices and rejects unlinked accounts', async () => { + const s = await setup([{ uid: a }]); + const other = await registerUser(s.app); + expect((await s.app.request('/api/v1/connectors/microblog/destinations', { headers: other.headers })).status).toBe(400); + expect((await s.app.request('/api/v1/connectors/microblog/destinations')).status).toBe(401); + s.db.close(); +}); + +it('preserves original collections when a rotated token has a different default but access to both blogs', async () => { + const s = await setup([{ uid: a }, { uid: b }]); + upsertAccount(s.db, s.uid, 'microblog', { token: 'old', microblogDestination: a }, null); + s.db.prepare('INSERT INTO microblog_clipping_posts(user_id,document,identity,url) VALUES(?,?,?,?)').run(s.uid, 'book', 'identity', `${a}post.html`); + upsertAccount(s.db, s.uid, 'microblog', { token: 'new', microblogDestination: b, microblogDestinations: [{ uid: a }, { uid: b }] }, null); + expect(s.db.prepare('SELECT identity,url FROM microblog_clipping_posts').get()).toEqual({ identity: 'identity', url: `${a}post.html` }); + s.db.close(); +}); +it('identifies URL path boundaries and the longest matching blog prefix', () => { + const blogs = ['https://shared.test/blog/', 'https://shared.test/blog/reading/'].map((uid) => ({ uid, name: uid, isDefault: false })); + expect(destinationForPost('https://shared.test/blog/reading/post.html', blogs)).toBe('https://shared.test/blog/reading/'); + expect(destinationForPost('https://shared.test/blogging/post.html', blogs)).toBeNull(); + expect(destinationForPost('https://shared.test.evil/blog/post.html', blogs)).toBeNull(); +}); +it.each([b, 'https://third.micro.blog/'])('never discards original collections when rotating to access %s', async (accessible) => { + const s = await setup([{ uid: a }, { uid: b }]); + upsertAccount(s.db, s.uid, 'microblog', { token: 'old', microblogDestination: a }, null); + s.db.prepare('INSERT INTO microblog_clipping_posts(user_id,document,identity,url,destination) VALUES(?,?,?,?,?)').run(s.uid, 'book', 'identity', `${b}post.html`, b); + upsertAccount(s.db, s.uid, 'microblog', { token: 'new', microblogDestination: accessible, microblogDestinations: [{ uid: accessible }] }, null); + expect(s.db.prepare('SELECT identity,url,destination FROM microblog_clipping_posts').get()).toEqual({ identity: 'identity', url: `${b}post.html`, destination: b }); + s.db.close(); +}); + +it.each([b, 'https://third.micro.blog/'])('preserves an unpublished selected blog on rotation only when it remains available (%s)', async (accessible) => { + const s = await setup([{ uid: a }, { uid: b }]); + upsertAccount(s.db, s.uid, 'microblog', { token: 'old', microblogDestination: a }, null); + setConnectorOptions(s.db, s.uid, 'microblog', { sync_highlights: true, highlight_blog: b }); + saveMatch(s.db, s.uid, 'microblog', 'book', { externalId: '123', confidence: 1 }, 'manual'); + enqueue(s.db, s.uid, 'microblog', { kind: 'progress', document: 'book', percentage: 0.4, timestamp: 1 }); + upsertAccount(s.db, s.uid, 'microblog', { token: 'rotated', microblogDestination: accessible, microblogDestinations: [{ uid: accessible }] }, null); + const account = s.db.prepare("SELECT options FROM connector_accounts WHERE user_id=? AND connector_id='microblog'").get(s.uid) as { options: string }; + expect(JSON.parse(account.options)).toEqual(accessible === b ? { sync_highlights: true, highlight_blog: b } : {}); + expect(s.db.prepare("SELECT COUNT(*) n FROM connector_matches WHERE user_id=? AND connector_id='microblog'").get(s.uid)).toEqual({ n: accessible === b ? 1 : 0 }); + expect(s.db.prepare("SELECT COUNT(*) n FROM connector_queue WHERE user_id=? AND connector_id='microblog'").get(s.uid)).toEqual({ n: accessible === b ? 1 : 0 }); + expect(s.db.prepare('SELECT COUNT(*) n FROM microblog_clipping_posts WHERE user_id=?').get(s.uid)).toEqual({ n: 0 }); + s.db.close(); +}); + +it.each([ + ['https://shared.test/blog/post.html', true], + ['https://shared.test/blog/reading/post.html', true], + ['https://shared.test/blogging/post.html', false], + ['https://shared.test.evil/blog/post.html', false], +] as const)('preserves legacy settings on rotation only when %s belongs to the available path-based blog', async (url, sameBlog) => { + const blog = 'https://shared.test/blog/'; + const s = await setup([{ uid: blog }]); + s.db.prepare('INSERT INTO microblog_clipping_posts(user_id,document,identity,url) VALUES(?,?,?,?)').run(s.uid, 'book', 'identity', url); + // Isolate legacy URL evidence from an independently saved blog selection. + setConnectorOptions(s.db, s.uid, 'microblog', { sync_highlights: true }); + saveMatch(s.db, s.uid, 'microblog', 'book', { externalId: '123', confidence: 1 }, 'manual'); + enqueue(s.db, s.uid, 'microblog', { kind: 'progress', document: 'book', percentage: 0.4, timestamp: 1 }); + upsertAccount(s.db, s.uid, 'microblog', { token: 'rotated', microblogDestination: blog, microblogDestinations: [{ uid: blog }] }, null); + const account = s.db.prepare("SELECT options FROM connector_accounts WHERE user_id=? AND connector_id='microblog'").get(s.uid) as { options: string }; + expect(JSON.parse(account.options)).toEqual(sameBlog ? { sync_highlights: true } : {}); + expect(s.db.prepare("SELECT COUNT(*) n FROM connector_matches WHERE user_id=? AND connector_id='microblog'").get(s.uid)).toEqual({ n: sameBlog ? 1 : 0 }); + expect(s.db.prepare("SELECT COUNT(*) n FROM connector_queue WHERE user_id=? AND connector_id='microblog'").get(s.uid)).toEqual({ n: sameBlog ? 1 : 0 }); + expect(s.db.prepare('SELECT identity,url FROM microblog_clipping_posts WHERE user_id=?').get(s.uid)).toEqual({ identity: 'identity', url }); + s.db.close(); +}); diff --git a/test/microblog.test.ts b/test/microblog.test.ts index aeeec0df..9715a324 100644 --- a/test/microblog.test.ts +++ b/test/microblog.test.ts @@ -11,6 +11,21 @@ const DOC = { const EV = { kind: 'progress' as const, document: 'd', percentage: 0.4, timestamp: 1 }; describe('Micro.blog shelf lookup', () => { + it('records a server-verified default blog so rotating a token preserves publication identity', async () => { + const cred: Record = { token: 'mb-token', microblogDestination: 'https://untrusted.example/' }; + const fake = makeMicroblogTransport(); + const http: HttpTransport = async (url, init) => { + if (url === 'https://micro.blog/micropub?q=config') return { + status: 200, text: async () => '', json: async () => ({ destination: [ + { uid: 'https://other.micro.blog/', 'microblog-default': false }, + { uid: 'https://example.micro.blog/', 'microblog-default': true }, + ] }), + }; + return fake.transport(url, init); + }; + expect((await _microblog.validateCredential(cred, http)).ok).toBe(true); + expect(cred.microblogDestination).toBe('https://example.micro.blog/'); + }); it('validates an app token against the bookshelves endpoint', async () => { const fake = makeMicroblogTransport(); expect(await _microblog.validateCredential(CRED, fake.transport)).toEqual({ ok: true }); diff --git a/test/web-microblog.test.ts b/test/web-microblog.test.ts index 1203ba00..be687381 100644 --- a/test/web-microblog.test.ts +++ b/test/web-microblog.test.ts @@ -56,7 +56,7 @@ describe('Micro.blog web setup', () => { }); await new Promise((resolve) => setTimeout(resolve, 0)); - expect(getElement('desc').textContent).toBe( + expect(getElement('desc').textContent).toContain( 'Connect an app token to keep your Currently reading and Finished reading bookshelves in sync.', ); expect(getElement('form').innerHTML).toContain('Sign in to Micro.blog');