[feat] Make agenta.ai readable and recoverable for AI agents - #6361
Merged
Conversation
Two landing sections rendered their titles as styled spans, so an outline extractor (or a screen reader jumping by heading) saw an h1 followed by four h2s and nothing underneath. Promote the section titles to h2 and the card, beat, and template titles to h3. Rendering is unchanged: global.css carries no heading reset, so each promoted tag keeps its inline font shorthand and gains margin:0. Verified by measuring every homepage section against production at 1280px — identical heights.
…ents An agent-readiness audit scored agenta.ai 61/100: no content negotiation, no Vary, no structured errors, no published API spec, and a 404 an agent cannot recover from. None of that is expressible in pure static assets, so add a thin worker in front of the Cloudflare assets binding. Astro stays output:'static' — the worker never renders, it picks between prebuilt files and sets headers. - worker/: Accept negotiation with q-values, Vary: Accept, 406 for types we cannot serve, a markdown or JSON 404 depending on who is asking, and the _headers /* policy re-applied (Cloudflare drops _headers on worker responses). The whole handler falls back to the raw asset on any throw. - src/pages/*.md.ts: a prebuilt markdown twin of every route, blog bodies rendered from their own MDX with the JSX stripped. This is also the real fix for 'content without JavaScript'. - scripts/copy-openapi.mjs: publishes /openapi.json at build from the committed docs spec, rewriting the relative /api server to the us/eu cloud base URLs and dropping the admin surface. Two routing decisions matter and are easy to get wrong: run_worker_first is an allowlist of HTML routes (a blanket /* would kill the four _redirects 308s and the image cache policy), and not_found_handling is 'none' (with '404-page' the asset server answers unknown paths itself and no agent ever reaches the worker). Browsers, crawlers, and bare curl keep the designed 404 page and the identical HTML. Covered by vitest, a postbuild output check, and a post-deploy smoke test both workflows now run.
A static Astro endpoint's Response headers do not survive the build — only the body does — so the X-Robots-Tag the twin endpoints set was never served. A crawler following the rel=alternate link in Base.astro would have indexed /pricing.md alongside /pricing. Only public/_headers can set it for a static asset, so it lives there now, and the deploy smoke test asserts both halves: the twin is noindex, the HTML page is not. Also simplify the review's findings in the same pass: - One markdown twin path instead of a candidate list. Only flat twins are built, so the loop over a speculative <path>/index.md was dead weight. - One response builder in the worker instead of four near-identical ones. - errorJson takes an object; five positional arguments was a trap. Drop the home/blog/pricing keys it spread into every error — the markdown 404 already lists them, and the JSON body only owes an agent the machine-readable ones. - Drop the hand-rolled sort tiebreaker in parseAccept: Array.sort is stable. - /llms.txt no longer repeats its own URL and the spec URL back at the reader. - MACHINE_READABLE moves next to its only consumer. - copy-openapi reads the spec once and stops adding an x-documentation-url field nothing asked for.
dosubot
Bot
added
the
size:XXL
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
dosubot
Bot
added
documentation
Contributor
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
📝 WalkthroughSummary by CodeRabbit
WalkthroughThe website now publishes Markdown twins and a filtered OpenAPI document, negotiates HTML, Markdown, and JSON responses through a Cloudflare Worker, validates builds and deployments, and uses semantic heading elements. ChangesWebsite agent-readiness
Estimated code review effort: 4 (Complex) | ~45 minutes Merge Risk: 🔵 Low · up to The PR adds content negotiation, Markdown mirrors, a public OpenAPI document, and improved heading semantics. It is broadly mergeable, but owner follow-up is needed for a minor API-page rendering defect, stronger OpenAPI publication checks, an accessibility semantics concern, and an edge case in Markdown conversion. Sequence Diagram(s)sequenceDiagram
participant Client
participant Worker
participant AssetServer
Client->>Worker: Request page with Accept header
Worker->>Worker: Select HTML, Markdown, JSON, or 406
Worker->>AssetServer: Fetch selected prebuilt asset
AssetServer-->>Worker: Asset response
Worker-->>Client: Response with content type and cache headers
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
Full details: Docstring CoverageExplanation Docstring coverage is 69.57% which is sufficient. The required threshold is 60.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 23 functions across 24 files. (4 skipped: 4 unsupported.) ✨ Finishing Touches 💡 1🛠️ Fix failing CI checks 💡
📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 7
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository YAML (base), Organization UI (inherited)
Review profile: CHILL
Plan: Pro Plus
Run ID: 1a93a6ec-7e6e-4958-9c69-63f993457a21
⛔ Files ignored due to path filters (1)
website/pnpm-lock.yamlis excluded by!**/pnpm-lock.yaml
📒 Files selected for processing (36)
.github/workflows/15-website-preview.yml.github/workflows/16-website-production.ymlwebsite/.gitignorewebsite/AGENTS.mdwebsite/package.jsonwebsite/public/_headerswebsite/public/_redirectswebsite/scripts/copy-openapi.mjswebsite/scripts/copy-openapi.test.mjswebsite/scripts/verify-build.mjswebsite/scripts/verify-deployment.shwebsite/src/components/HowItWorks.tsxwebsite/src/components/OpenSource.astrowebsite/src/components/OpenStandards.astrowebsite/src/components/Reliability.astrowebsite/src/components/TemplateExplorer.tsxwebsite/src/layouts/Base.astrowebsite/src/lib/markdown.test.tswebsite/src/lib/markdown.tswebsite/src/lib/siteSummary.tswebsite/src/pages/authors.md.tswebsite/src/pages/authors/[slug].md.tswebsite/src/pages/blog.md.tswebsite/src/pages/blog/[slug].md.tswebsite/src/pages/contact.md.tswebsite/src/pages/imprint.md.tswebsite/src/pages/index.md.tswebsite/src/pages/llms.txt.tswebsite/src/pages/pricing.md.tswebsite/vitest.config.tswebsite/worker/index.test.tswebsite/worker/index.tswebsite/worker/negotiate.test.tswebsite/worker/negotiate.tswebsite/wrangler.jsoncwebsite/wrangler.production.jsonc
Included review availability: Your plan provides up to 8 included reviews per hour; 6 remain after this review.
… one Review of #6361 surfaced two symptoms of one flaw: the worker picked a single winner from the Accept header and then ignored what else the client would have taken. A request for "application/json, text/markdown" got HTML it never asked for, and "Accept: text/html;q=0" (an explicit refusal) was read as no preference and answered with HTML anyway. acceptedRepresentations now returns the ranked list of representations the client accepts, and the worker walks it: markdown when it outranks HTML, the page when HTML is acceptable, 406 when neither is. A path that does not exist still gets its 404 first — a missing page owes a 404 in the caller's language, not a 406 about representations of nothing. Also from the review: - copy-openapi rejects a spec with no info.title/version. It would have published a file that parses and tells a client nothing. - mdxToMarkdown protects inline code spans, so a post that mentions `<InlineCTA />` in a sentence keeps the example instead of losing it. - The scroll-layout beat title goes back to a span. It sits inside a role="button" row, where a heading is exposed as presentational, so it bought nothing; the no-JS layout that crawlers actually receive keeps its real h3. - /llms.txt says the homepage twin is /index.md. Appending .md to "/" yields /.md, which is not a route. - The deploy smoke test asserts the text/html media type rather than one exact header serialization (production omits the charset, so all three post-deploy checks failed), and skips the indexable-page assertion on workers.dev, which Cloudflare stamps noindex by default.
Contributor
Website previewPreview URL: https://pr-6361-agenta-website-preview.mahmoud-637.workers.dev Built from |
Contributor
There was a problem hiding this comment.
Actionable comments posted: 5
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository YAML (base), Organization UI (inherited)
Review profile: CHILL
Plan: Pro Plus
Run ID: 3ea07844-931e-48d5-b1c7-f4856b9a8c56
📒 Files selected for processing (11)
website/scripts/copy-openapi.mjswebsite/scripts/copy-openapi.test.mjswebsite/scripts/verify-deployment.shwebsite/src/components/HowItWorks.tsxwebsite/src/lib/markdown.test.tswebsite/src/lib/markdown.tswebsite/src/pages/llms.txt.tswebsite/worker/index.test.tswebsite/worker/index.tswebsite/worker/negotiate.test.tswebsite/worker/negotiate.ts
🚧 Files skipped from review as they are similar to previous changes (1)
- website/src/pages/llms.txt.ts
Included review availability: Your plan provides up to 8 included reviews per hour; 6 remain after this review.
mmabrouk
approved these changes
Aug 29, 2026
mmabrouk
left a comment
mmabrouk
left a comment
Member
There was a problem hiding this comment.
Thanks @ashrafchowdury lgtm
Feel free to merge after you address / or resolve the coderabbit comments.
…ccept range Second review round on #6361. Three correctness holes, all in how the edge worker and the MDX converter handle input they were scanning too loosely. A specific q=0 now overrides a wildcard that accepted the same type. The old forward scan read "text/*;q=1, text/html;q=0" as accepting HTML, because the exact range only excluded itself. Each representation is now resolved against the most specific range that matches it, per RFC 9110 12.5.1, so that header yields markdown only. A request that accepts nothing we serve gets a 406 even when the path does not exist. There is no language to answer the 404 in, so 406 is the honest reply. mdxToMarkdown protects every code region markdown allows, not just triple backticks: tilde fences and any run of backticks are pulled out before the component cleanup. The placeholder is wrapped in NUL bytes, so a post whose prose happens to read "CODE0" is no longer corrupted on the way back in. Declined one comment on the thread: application/xhtml+xml and application/xml still map to the HTML page. Those clients read it fine, and a 406 is a worse answer than a content type one notch broader than the request. 50 tests, and both new cases verified against a local wrangler dev.
Contributor
There was a problem hiding this comment.
Actionable comments posted: 2
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository YAML (base), Organization UI (inherited)
Review profile: CHILL
Plan: Pro Plus
Run ID: 715e6e2c-e6bc-4c70-88f7-25193a5c30cd
📒 Files selected for processing (6)
website/src/lib/markdown.test.tswebsite/src/lib/markdown.tswebsite/worker/index.test.tswebsite/worker/index.tswebsite/worker/negotiate.test.tswebsite/worker/negotiate.ts
Included review availability: Your plan provides up to 8 included reviews per hour; 5 remain after this review.
Third review round on #6361. `qualityFor` recorded each match's index in the already-sorted entry list, so the tie-break documented as "whichever the client named first" was really "whichever sorted higher". For `Accept: text/*, text/html" that ranked HTML above markdown, even though the range covering markdown came first. Entries now carry the position the client wrote them at, and that position survives the sort. Browser headers are unaffected: text/html leads there on quality, not on the tie-break.
Two findings from the agent-readiness audit of the preview deploy. No when-to-use guidance. An agent deciding whether to route a job to us had the tagline and nothing else. /llms.txt and the homepage twin now carry three sections built from one source (src/lib/siteSummary.ts): the jobs Agenta is for, the jobs it is not for (a single model call, model hosting), and how to actually call it. The honest exclusions matter as much as the inclusions: an agent that reads only marketing copy cannot tell when to route elsewhere. No API surface linked from the homepage on our own origin. The nav and footer have always linked docs.agenta.ai, but a cross-origin link does not answer "where is this site's API". Adds /api: base URLs for both clouds and self-hosting, the auth header, one request that works, what Agenta is for, and links onward to /openapi.json and the full docs. It is a signpost, not a second copy of the reference. Footer gains a same-origin API link, so every page reaches it. Facts come from docs/docs/reference/api-guide (base URLs, the ApiKey header, project-scoped keys) and the workflows guide (the curl example). The page reuses the .ag-info-panel pattern from contact.astro, so it inherits the design rather than inventing one. /api joins the negotiation allowlist and gets a markdown twin like every other route. The /api/* JSON-error prefix is unaffected: /api is a page, /api/foo still returns the structured 404.
The preview deploy failed on two llms.txt content checks. The site was fine: static assets carry stale-while-revalidate=3600, so the edge served the previous deploy's body to the first request after the new version went out and revalidated behind it. Confirmed on the deployed preview (cf-cache-status: HIT on /llms.txt, correct content on the next fetch). A query-string cache-buster does not help. Cloudflare normalizes the query out of an asset cache key, so /llms.txt?_cb=N still returns the cached body. So the three assertions that read an asset's body now re-fetch, up to six times at five-second intervals; the first request is what triggers revalidation. Worker-served routes are untouched, since the worker runs on every request and never serves a stale body. Worth recording from the same investigation: negotiation is safe under edge caching. /pricing goes MISS then HIT, but a markdown request straight after a cached HTML one still gets text/markdown, because run_worker_first puts the worker in front and the HIT applies to its asset subrequest, not to the client's response.
Contributor
There was a problem hiding this comment.
Actionable comments posted: 2
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository YAML (base), Organization UI (inherited)
Review profile: CHILL
Plan: Pro Plus
Run ID: 428a3203-f0b9-44ef-8c1d-e94155926c6c
📒 Files selected for processing (10)
website/scripts/verify-build.mjswebsite/scripts/verify-deployment.shwebsite/src/components/SiteFooter.astrowebsite/src/lib/siteSummary.tswebsite/src/pages/api.astrowebsite/src/pages/api.md.tswebsite/src/pages/index.md.tswebsite/src/pages/llms.txt.tswebsite/wrangler.jsoncwebsite/wrangler.production.jsonc
Included review availability: Your plan provides up to 8 included reviews per hour; 7 remain after this review.
…d spec Two review comments on #6361, both right. /api printed literal backticks. The page reused HOW_TO_CALL[3], whose markdown is written for the .md twin, and Astro renders an expression as text: readers saw `pip install agenta`. The page now writes its own <code> element, and the constant keeps its markdown for api.md.ts. Indexing into a shared list for prose was the underlying mistake. The postbuild spec check only proved the server URLs were absolute, so a stale or unfiltered document could still ship. It now asserts the exact published hosts and that no /admin path survived the filter, which is the contract scripts/copy-openapi.mjs actually promises. Verified by mutating dist: swapping the servers and re-adding an admin path each fail the check.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Context
An "Is Agentic" audit scored agenta.ai 61/100 on how usable the site is for AI agents and crawlers. Five checks failed or came back partial:
/openapi.jsonAccept: text/markdownreturned HTML, noVaryThe root cause is the same for four of them. The site is pure SSG on Cloudflare Workers Static Assets with no worker script, so there is no request-time hook anywhere. Static assets cannot read
Accept, setVary, return a406, or build a JSON error body.Changes
A thin worker in front of the assets binding (
website/worker/). Astro staysoutput: "static". The worker renders nothing: it readsAccept, picks one of the files the build already produced, and sets headers. The whole handler is wrapped in atry/catchthat falls back toenv.ASSETS.fetch(request), so a negotiation bug cannot take the site down.Two routing decisions carry the risk, and both are covered by tests:
run_worker_firstis an allowlist of the HTML routes that need negotiation, never/*. Cloudflare does not apply_headersor_redirectsto worker responses, so a blanket rule would have turned the four 308s inpublic/_redirects(/terms,/privacy-policy,/launch-week-1,/launch-week-2) into 404s and dropped the week-long cache TTL on every blog image.not_found_handlingis"none", not"404-page". With"404-page"the asset server answers unknown paths itself and the worker never runs, so no agent could ever get a machine-readable 404.A markdown twin of every route (
src/pages/*.md.ts, 47 files at build). Blog twins render from the post's own MDX with the component tags stripped, so an agent reads what a reader reads without running JavaScript. This, not the heading work, is the real fix for the "content without JavaScript" check.Before, for an agent asking for markdown:
After:
A published API spec.
scripts/copy-openapi.mjsruns atprebuildand copies the committeddocs/docs/reference/openapi.jsontopublic/openapi.json. It rewrites the spec's relative/apiserver, which would otherwise resolve against agenta.ai and point at nothing, to the realus.cloud.agenta.aiandeu.cloud.agenta.aibase URLs, and drops the 22/admin/*paths. The output is gitignored, like the licensed fonts. Both deploy workflows now list the source spec in theirpaths:filter so a regenerated spec redeploys the site.Dead ends that answer in the caller's language. A JSON asker (or anything under
/api/) gets{"error": {"code": "not_found", "status": 404, "hints": [...], "sitemap": ..., "llms_txt": ..., "openapi": ...}}. A markdown asker gets a short recovery note. Everyone else, including browsers, crawlers, uptime monitors and barecurl, keeps the designed404.astropage unchanged.Real headings on the homepage.
OpenStandardsandOpenSourcerendered their section titles as styled<span>s, and no section hadh3children, so an outline extractor saw anh1and four orphanh2s. The titles are now realh2/h3elements carrying the same inline styles plusmargin: 0, becauseglobal.csshas no heading reset. Nothing moves on screen: every section height matches production at 1280px (Templates 1006, How it works 1459, Open standards 628, Reliability 808, Open source 591, CTA 470).The third commit is a review pass. It fixes one bug the review found and simplifies the first two commits. The bug: a static Astro build keeps an endpoint's body and discards its
Responseheaders, so theX-Robots-Tag: noindexthe twins set was never served, and a crawler following the newrel="alternate"link would have indexed/pricing.mdnext to/pricing. Onlypublic/_headerscan set that on a static asset, so a/*.mdrule does it now.Tests
scripts/verify-build.mjsruns aspostbuild: every twin emitted, the spec parses with absolute servers, no.mdin the sitemap.scripts/verify-deployment.shis a 17-check smoke test both workflows now run against the deployed URL. It asserts the agent behavior and the static behavior the worker must not break. All 17 pass against a realwrangler dev.checkjob so fork PRs get test signal. The deploy job stays behind the existing secrets guard.wranglerbecomes a devDependency pinned to4.113.0, the version both workflows already pinned viapnpm dlx, so one version bundles the worker everywhere.Worth a reviewer's attention:
worker/negotiate.tsHEADERSdeliberately duplicates the/*block inpublic/_headers. That is not drift. Cloudflare genuinely does not apply_headersto worker responses, and both files point at each other in a comment.What to QA
Against the preview URL:
curl -sI -H "Accept: text/markdown" <url>/returnstext/markdown; charset=utf-8withVary: Accept.curl -s <url>/no-such-pagestill renders the designed 404 page, andcurl -s -H "Accept: application/json" <url>/no-such-pagereturns a JSON error object.curl -s -o /dev/null -w "%{http_code}" <url>/termsreturns 308, not 404. This is the regression the allowlist exists to prevent.curl -s -o /dev/null -w "%{http_code}" <url>/pricing/returns 307, so every trailing-slash URL in the sitemap still resolves.