feat(rss): add rss and newsletter sources#2378
Closed
dimslaev wants to merge 44 commits into
Closed
Conversation
Introduces the articles domain: Article model with a pgvector vector(256) column, Alembic migration that enables pgvector and creates an HNSW index, two public endpoints (filtered list + vector similarity search), a one-time import script, and a minimal frontend /articles page. Switches the Postgres image to pgvector/pgvector:pg17 and adds pgvector + model2vec deps. Also adds CLAUDE.md working agreements, CHANGES.md upstream divergence log, and a /prepare-pr slash command. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* feat(pipeline): Python ingestion pipeline v1 (HN + AI News + Substack)
- Add baml_src/ at repo root with Python generator (alongside existing TS generator)
- Commit generated backend/baml_client/ (Python/pydantic)
- Add backend/pipeline/: run.py orchestrator, steps.py helpers, utils.py,
sources/{hn,ainews,substack,extract_content,utils}.py
- Add ScoredUrl model + Alembic migration (scored_url table)
- Add pipeline compose service (supercronic, daily 04:00)
- Install supercronic in backend Dockerfile
- Add v1 deps: baml-py, trafilatura, feedparser, dnspython, regex
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SE8sLxeKrZVsPWHBj56kqF
* chore(baml): commit regenerated TypeScript baml_client
Generated by `baml generate` from baml_src/ (same version 0.223.0 as
before; output is identical to the temp/source reference).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SE8sLxeKrZVsPWHBj56kqF
* refactor(pipeline): remove dead code, fix lint
- Drop unused ArticleInput dataclass + to_article_inputs from steps.py
(run.py builds BAML inputs directly)
- Add _to_baml_input helper in run.py, dedup 3x inline construction
- Rename ambiguous `l` -> `lnk` in ainews._pick_primary
- Add ruff per-file-ignore T201 for pipeline/ (intentional stdout logging)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SE8sLxeKrZVsPWHBj56kqF
* chore(baml): drop TypeScript generator and client
The TS pipeline is being retired (lives in temp/source/, removed at
cutover) and nothing in the live repo imports the root baml_client/.
Keep only the Python generator + backend/baml_client/.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SE8sLxeKrZVsPWHBj56kqF
* fix(ci): ruff format pipeline files; omit alembic from coverage
- Run ruff format on all pipeline/ files (was causing pre-commit failure)
- Add omit = ["app/alembic/*"] to coverage.run — alembic migration files
are already excluded from mypy and ruff; coverage should match.
The new scored_url migration tipped coverage below the 90% threshold.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SE8sLxeKrZVsPWHBj56kqF
* revert: remove alembic omit from coverage config
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SE8sLxeKrZVsPWHBj56kqF
* chore: update changes md
---------
Co-authored-by: Claude <noreply@anthropic.com>
Move ArticlesList to the root route and remove the layout auth guard so visitors land on articles without logging in. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
www.agentique.ch had no matching router after the domain switch from next.agentique.ch, so it fell back to Traefik's default self-signed cert and 404'd. Add www routers + redirectregex middleware to 301 to the bare domain, getting their own Let's Encrypt cert via the same per-router resolver pattern used for api./adminer./traefik. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* feat: add ArticleLike model and migration
Composite (user_id, article_id) primary key gives free uniqueness and
idempotent like/unlike semantics. Also delete article_like rows in the
test db teardown fixture before deleting users, since the new FK would
otherwise block it.
* feat: add like/unlike/my-likes endpoints
PUT/DELETE /articles/{id}/like and GET /me/liked-articles, all
auth-required and idempotent via the ArticleLike composite PK.
* feat: add like_count, liked_by_me, and Popular sort to public article endpoints
CurrentUserOptional (new app/api/deps_agentique.py) decodes the Bearer
token when present and returns None instead of raising on any failure,
so /articles and /articles/search never 401 on an anonymous or stale
token. like_count comes from a LEFT JOIN aggregate over article_like,
computed at query time rather than denormalized. sort=likes-desc ties
break on score desc; other sorts are unchanged.
* fix: satisfy ruff/mypy/ty gates for the likes feature
Rework the like_count join in read_articles/search_articles to build
select(Article, like_count) directly instead of add_columns() on a
select(Article) statement — SQLModel's session.exec() silently
collapses the latter back to scalar Article rows, dropping the
like_count column. Also add matching type:ignore/ty:ignore pairs for
known false positives (same pattern as the existing Article.score
ones) and apply ruff format.
* chore: regenerate TS client for the likes API
Adds LikesService (likeArticle/unlikeArticle/readLikedArticles) and
like_count/liked_by_me on ArticlePublic.
* feat: add LikeButton, extract ArticleRow, add Popular sort
LikeButton renders the fire icon + count (0 included), redirects
logged-out clicks to /login?redirect=<current>, and optimistically
toggles the like via a TanStack Query mutation that patches all
cached ["articles", ...] queries and invalidates on settle. The login
route now accepts a redirect search param and useAuth navigates back
to it on success. ArticleRow is extracted out of ArticlesList so the
upcoming Liked tab can reuse it. Sort filter gains "Popular" ->
sort=likes-desc.
* feat: add Profile page with Liked tab, un-skip auth e2e specs
New /_layout/profile route: tabs Liked (default) / My profile /
Password / Danger zone, auth-guarded (redirect to /login with a
redirect param when logged out). Liked tab fetches
GET /me/liked-articles and reuses ArticleRow. /settings now just
redirects to /profile (kept, per fork discipline). Sidebar shows a
"Log in" link when logged out and links to /profile (not /settings)
when logged in.
Fixed a real bug found via the new e2e coverage: GET /me/liked-articles
never set like_count/liked_by_me on its response, so a liked article's
own like button in the Liked tab thought it was still unliked and
clicking it re-liked (a no-op PUT) instead of unliking.
Removed test.skip from login.spec.ts and sign-up.spec.ts now that auth
is a real user path again; updated user-settings.spec.ts to /profile
and un-skipped it too. Fixed assertions in these specs and the shared
logInUser test helper that depended on the old Dashboard's "Welcome
back" greeting, which no longer exists (replaced by the articles feed).
Made the theme-toggle tests state-agnostic instead of assuming a
light starting theme, since this fork defaults to dark.
* chore: un-skip backend auth tests, un-omit them from coverage, update CHANGES.md
test_login.py and test_users.py no longer skip (auth is a real user
path again with the profile page); test_recovery_password also patches
EMAILS_FROM_EMAIL since emails_enabled requires it and this fork's
CI-synthesized .env never sets it. Dropped login.py/users.py/crud.py/
core/security.py/utils.py/api/deps.py from the coverage omit list now
that they're actually exercised (still 92% aggregate, above the 90%
gate). test_items.py/test_private.py/crud/test_user.py stay skipped.
* style: ruff format after removing skip marks from test_login.py/test_users.py
---------
Co-authored-by: Claude <noreply@anthropic.com>
Recolor the favicon.png background while preserving the white "ag" mark and alpha-based rounded corners. Co-authored-by: Claude <noreply@anthropic.com>
Password recovery always sent through SMTP (`emails` package), but this deployment only ever configures RESEND_API_KEY, not SMTP_HOST/PASSWORD, so `emails_enabled` was False and every password-recovery request raised an AssertionError. send_email now sends via resend.Emails.send when RESEND_API_KEY is set, mirroring the pattern already used by the newsletter route, falling back to SMTP when it isn't. Claude-Session: https://claude.ai/code/session_014vw2MSjnwvwyq8Dpniuo5c Co-authored-by: Claude <noreply@anthropic.com>
Brings the fork up to date with fastapi/full-stack-fastapi-template (10 commits: dependency/action bumps + release notes). Merge content is identical to the previous single-parent 56ff8fa; this records upstream/master as a real second parent so the fork no longer reads as behind.
# Conflicts: # README.md
Port the IMAP newsletter source and generic RSS source from the old TS pipeline, wiring Newsletter into SOURCES while keeping RSS dormant as upstream did. Newsletters with a working feed moved from IMAP polling to direct RSS reads.
Contributor
|
This PR modifies dependency files ( If you need a dependency change, please open a discussion describing what you need and why. Closing this PR automatically. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The daily article pipeline was missing one of its four original ingestion sources: the Newsletter source, which reads unread emails from ~15 known AI newsletter senders over IMAP and turns the products they mention into articles by extracting them with BAML and rediscovering each one's canonical URL through a web search (newsletter links are just tracking redirects, not usable hrefs). This PR ports that source to the Python pipeline and wires it in.
While researching which of those senders actually publish an RSS feed, six of them turned out to (Console, The Neuron, AI Hero, Import AI, TLDR, The Rundown AI), so those moved to a new generic RSS source that just reads the feed directly — simpler and no mailbox involved. The other eight senders don't publish a feed, so they stay on IMAP. The RSS source itself isn't turned on in the pipeline yet, matching how it was already disabled in the original TypeScript version.
🤖 Generated with Claude Code