Added "View as markdown" button#3161
Conversation
|
ibexa-workflow-automation-1
Bot
requested review from
adriendupuis,
dabrt and
julitafalcondusza
and removed request for
a team
|
@mnocon We could, for example, have icons at the top right, with hover-on labels, wdys? Or, alternatively, put the section with purely technical buttons at the bottom, after the article itself? 
|
|
I've chosen this implementation based on other docs, which is (quickly) becoming the norm. Other pages using this approach of "buttons under h1":
As the number of action grows, they start using dropdowns:
As we only have 3 buttons right now - I'd keep this as is |
adriendupuis
left a comment
adriendupuis
left a comment
There was a problem hiding this comment.
I don't like having the action button bar below the page title, it's like the title should be out of the export or that all section headings should have this bar.
The new lines converted to commas in the table cells are breat, they're not duplicated when already there like in Built-in views' table from https://ez-systems-developer-documentation--3161.com.readthedocs.build/en/3161/commerce/shopping_list/shopping_list_design/#built-in-views / https://ez-systems-developer-documentation--3161.com.readthedocs.build/en/3161/commerce/shopping_list/shopping_list_design/index.md
| {{ page.content }} | ||
|
|
||
| <!-- Page action buttons - show only on subpages, positioned after first h1 --> | ||
| {% if config.repo_url and page.edit_url and not page.is_homepage and not page.meta.exclude_from_llmstxt %} |
There was a problem hiding this comment.
I think the page title is to far from its content and it doesn't help to understand what will be included in the Markdown, I would swap them andput the action bar between the breadcrumb and the title.
<div class="content-with-actions">
{% if config.repo_url and page.edit_url and not page.is_homepage and not page.meta.exclude_from_llmstxt %}
# …
{% endif %}
{{ page.content }}
</div>
| # --------------------------------------------------------------------------- | ||
|
|
||
| def _normalize_ordered_lists(soup: "BeautifulSoup") -> None: | ||
| """Remove start/value attributes from <ol>/<li> so markdownify outputs 1. for all items. |
There was a problem hiding this comment.
Don't know from where it comes from so I put it there.
The "1. for all items" doesn't seem to work.
1. Sets a variable with the desired Varnish version, here Varnish 7.1
2. Copies and customizes `parameters.vcl` file in `.ddev/varnish/` (which is mounted as `/etc/varnish/` into the container):
- sets `web` container as the backend host and an invalidator (so back office can purge cache)
- adds "all IPs" CIDR notation to `debuggers` list to allow debugging from any IP
- on Varnish 7, enable logging of access control list matching for both `invalidators` and `debuggers` lists (new Varnish 7 syntax, it was enabled by default on previous versions)
1. Sets main `varnish*.vcl` file to use and "path to VCL directory" argument name depending on Varnish version
2. Copies the main VCL file to `.ddev/varnish/`
3. Sets the Varnish version to use and its demon starting parameters to use the files
4. Adds the Varnish container
5. Sets Varnish as the HTTP cache server
6. Restarts the DDEV cluster and clear the Ibexa DXP cacheFrom a Markdown standard perspective, that not really an issue; At least, PhpStorm internal renderer shows a list of 8 items numbered from 1 to 8 as expected.
There was a problem hiding this comment.
I've asked Fable to fix this and cover with tests, so that we can control this - there are many edge cases.
Examples working:
- escaped lists: https://ez-systems-developer-documentation--3161.com.readthedocs.build/en/3161/commerce/storefront/configure_storefront/index.md
- nested lists (the one you've reported): https://ez-systems-developer-documentation--3161.com.readthedocs.build/en/3161/infrastructure_and_maintenance/clustering/clustering_with_ddev/index.md
- normal lists: https://ez-systems-developer-documentation--3161.com.readthedocs.build/en/3161/tutorials/beginner_tutorial/beginner_tutorial/index.md
| <link rel="llms" href="{{ 'llms.txt' | url }}" title="LLM-friendly index"> | ||
| <link rel="llms" href="{{ 'llms-full.txt' | url }}" title="LLM-friendly full content"> |
There was a problem hiding this comment.
Important notice here (for myself and other reviewers), yes, this a .txt file with Markdown in it, not a .md file.
The file names to use are not clear on https://llmstxt.org/
but this is the name formats used by Anthropic on Claude doc:
| """ | ||
| from bs4 import NavigableString as NS | ||
|
|
||
| for tags_div in soup.find_all("div", class_="release-note__tags"): |
There was a problem hiding this comment.
I don't know if the date alone on a line is understandable for an AI agent.
I would transform for tags_div in soup.find_all("div", class_="release-note__date"): to prepend the text "Release date: " in front of the date.
So, instead of
## Google Gemini connector v5.0.7 (Headless, Experience, Commerce, LTS Update, New feature, First release)
2026-04-20
This release introduces a new AI connector
there would be
## Google Gemini connector v5.0.7 (Headless, Experience, Commerce, LTS Update, New feature, First release)
Release date: 2026-04-20
This release introduces a new AI connector
```
There was a problem hiding this comment.
Added release date: https://ez-systems-developer-documentation--3161.com.readthedocs.build/en/3161/release_notes/ibexa_dxp_v5.0/index.md , thanks for the suggestion!
| Files can be excluded from the llmstxt output by adding the following to their YAML frontmatter: | ||
|
|
||
| exclude_from_llmstxt: true |
There was a problem hiding this comment.
This excludes the file from both llms.txt and llms-full.txt files, and skip the action buttons.
But, how the exclusion from llms.txt alone but inclusion in llms-full.txt is controlled?
There was a problem hiding this comment.
Currently, you can't do this - you can either exclude a page from Markdown-generated content completely (llms.txt, llms-full.txt, "View as Markdown" button).
I don't think we need a fine-grained control, I think excluding the deprecated content completely is a use case is enough for now - and we can revisit later if needed.
Interesting… It works better when there is a baseline like on https://docs.stripe.com/get-started/account/set-up than when there isn't one like in https://developers.cloudflare.com/docs-for-agents/ It also seem to work better with a different layout spacing. So I would fix that by isolating a bit more the page title (with actions when available) from the content and get it closer to other page metadata like the breadcrumbs and page pills. |
Summary
|
Changed breadcrumbs to Can you please re-review? |
There was a problem hiding this comment.
For all files in perso: exclude them from the Markdown-generated content (llms.txt, llms-full.txt, page actions)
| The `match` configuration matches both the content type and the identifier of the Content query field. | ||
|
|
||
| Finally, in the template `templates/themes/<my_theme/content_query/blog_posts.html.twig`, render all results of the query: | ||
| Finally, in the template `templates/themes/<my_theme>/content_query/blog_posts.html.twig`, render all results of the query: |
There was a problem hiding this comment.
The lack of closing > broke the Markdown rendering (discovered when checking the output manually)
| </div> | ||
| <div class="notification__image"> | ||
| <img src="images/notification-latest-release.svg" alt="The latest release" /> | ||
| <img src="images/notification-latest-release.svg" alt="" /> |
There was a problem hiding this comment.
Icons don't need alt text
| @@ -0,0 +1,54 @@ | |||
| from llmstxt_preprocess import absolutize_image_urls | |||
There was a problem hiding this comment.
I've asked Claude to add tests for the changes to:
- make it easier to understand the code
- make it easier to fix the edge cases we discover in the future without worrying about regressions (I'm sure I didn't discover all of them)
| exit 1 | ||
| fi | ||
|
|
||
| python-tests: |
There was a problem hiding this comment.
Running tests for the LLM Markdown converter
| @@ -0,0 +1,153 @@ | |||
| #!/usr/bin/env python3 | |||
There was a problem hiding this comment.
One important one for the generated llms.txt file.
Our MkDocs file uses a nested structure (with multiple levels), but the output of this one is a list that goes 1-level deep at most.
Compare these two:
- https://github.com/ibexa/documentation-developer/blob/5.0/mkdocs.yml
- https://ez-systems-developer-documentation--3161.com.readthedocs.build/en/3161/llms.txt
I though about creating a nested structure, but this goes against the "spec":
Zero or more markdown sections delimited by H2 headers, containing “file lists” of URLs where further detail is available
Each “file list” is a markdown list, containing a required markdown hyperlink [name](url), then optionally a : and notes about the file.
and against all the examples available in https://directory.llmstxt.cloud/ - they all use a "flatter" structure than our MkDocs.
3 participants
This PR adds support for https://llmstxt.org/ (tool helping AI agents navigate webpages), and introduces a Markdown version of the doc.
How it looks:
The "View on GitHub" button is replaced by:
Why Markdown version of the doc, is the content on GitHub not enough?
No, because the macros we use (for example, embedding files) make the content not readable.
Compare:
The output Markdown contains the full code sample, making it easier for the AI agents to understand what's going on.
For more example see the list of cases I had to cover below.
LLMS.txt files
llms.txt file
llms-full.txt file
Pages can be excluded from these lists by marking them with
exclude_from_llmstxt: true(excludes from llms.txt, llms-full.txt, and does not show the "View as Markdown" buttons).The content of these files is autogenerated by
mkdocs buildis run - no manual changes needed.For now, HTML files are excluded from this list - I want to tackle API References in a follow-up.
Notes
Release notes page:
HTML: https://ez-systems-developer-documentation--3161.com.readthedocs.build/en/3161/release_notes/ibexa_dxp_v5.0/
Markdown: https://ez-systems-developer-documentation--3161.com.readthedocs.build/en/3161/release_notes/ibexa_dxp_v5.0/index.md (supports badges)
Added styling changes to make the title header look the same as on other pages.
Exclusion of Personalization
Personalization pages are excluded from this solution.
Example page: https://ez-systems-developer-documentation--3161.com.readthedocs.build/en/3161/personalization/personalization/
Image alt text
AI agents cannot see the images, similarly as people using screen readers.
HTML: https://ez-systems-developer-documentation--3161.com.readthedocs.build/en/3161/infrastructure_and_maintenance/request_lifecycle/
Markdown: https://ez-systems-developer-documentation--3161.com.readthedocs.build/en/3161/infrastructure_and_maintenance/request_lifecycle/index.md
If an image has alt-text, it's transformed into:
Absolute vs relative urls
I've went with absolute URLs, as these fill safer, but they use more token space of the agents.
Example page with absolute linls: https://ez-systems-developer-documentation--3161.com.readthedocs.build/en/3161/discounts/discounts/index.md
Admonition boxes
Converted to blockquotes.
HTML: https://ez-systems-developer-documentation--3161.com.readthedocs.build/en/3161/infrastructure_and_maintenance/background_tasks/#start-worker
Markdown: https://ez-systems-developer-documentation--3161.com.readthedocs.build/en/3161/infrastructure_and_maintenance/background_tasks/index.md
Tabs
Converted to bolded text (can't use headers as making the headers match the page structure would be a challenge).
Example:
HTML https://ez-systems-developer-documentation--3161.com.readthedocs.build/en/3161/getting_started/requirements/
Markdown: https://ez-systems-developer-documentation--3161.com.readthedocs.build/en/3161/getting_started/requirements/index.md
Tables
Tables actually look great 😄 The only problem is tables where a call contains a
|- this is guarded by Markdownlint already.For tables with linebreaks within cells, a comma is used instead.
Examples:
HTML: https://ez-systems-developer-documentation--3161.com.readthedocs.build/en/3161/ibexa_products/editions/
Markdown: https://ez-systems-developer-documentation--3161.com.readthedocs.build/en/3161/ibexa_products/editions/index.md
With linebreaks:
HTML: https://ez-systems-developer-documentation--3161.com.readthedocs.build/en/3161/api/event_reference/content_events/
Markdown: https://ez-systems-developer-documentation--3161.com.readthedocs.build/en/3161/api/event_reference/content_events/index.md
Cards macro
Converted to a list of links:
HTML: https://ez-systems-developer-documentation--3161.com.readthedocs.build/en/3161/ibexa_products/editions/
Markdown: https://ez-systems-developer-documentation--3161.com.readthedocs.build/en/3161/ibexa_products/editions/index.md
Lists numbering
Example: https://ez-systems-developer-documentation--3161.com.readthedocs.build/en/3161/infrastructure_and_maintenance/clustering/clustering_with_ddev/index.md
Edition pills
In content (inline):
Mutliple badges: https://ez-systems-developer-documentation--3161.com.readthedocs.build/en/3161/commerce/shopping_list/shopping_list_guide/index.md
Single badge:
HTML: https://ez-systems-developer-documentation--3161.com.readthedocs.build/en/3161/update_and_migration/from_5.0/update_from_5.0/#form-builder-performance-fix-missing-indexes-on-form-submission-data
Markdown: https://ez-systems-developer-documentation--3161.com.readthedocs.build/en/3161/update_and_migration/from_5.0/update_from_5.0/index.md
Release notes: https://ez-systems-developer-documentation--3161.com.readthedocs.build/en/3161/release_notes/ibexa_dxp_v5.0/index.md