Release/v0.4.0 (Pipelex#88)

### Highlight: Complete documentation overhaul

- **MkDocs** setup for static web docs generation
    - **Material** for MkDocs theme, custom styling and navigation
    - Other plugins: meta-manager, glightbox
    - **GitHub Pages** deployment, mapped to [docs.pipelex.com](http://docs.pipelex.com/)](http://docs.pipelex.com)
    - Added GHA workflows for documentation deployment and validation
- **Added to docs:**
    - [**[Manifesto](https://docs.pipelex.com/manifesto/)**](https://docs.pipelex.com/manifesto/) explaining the Pipelex viewpoint
    - [**[The Pipelex Paradigm](https://docs.pipelex.com/pages/pipelex-paradigm-for-repeatable-ai-workflows/)**](https://docs.pipelex.com/pages/pipelex-paradigm-for-repeatable-ai-workflows/) explaining the fundamentals of Pipelex’s solution
    - [[**Cookbook examples](https://docs.pipelex.com/pages/cookbook-examples/)](https://docs.pipelex.com/pages/cookbook-examples/)** presented and explained, commented code, some event with [[mermaid](https://docs.pipelex.com/pages/cookbook-examples/invoice-extractor/)](https://docs.pipelex.com/pages/cookbook-examples/invoice-extractor/) [[flow](https://docs.pipelex.com/pages/cookbook-examples/extract-gantt/)](https://docs.pipelex.com/pages/cookbook-examples/extract-gantt/) [[charts](https://docs.pipelex.com/pages/cookbook-examples/write-tweet/)](https://docs.pipelex.com/pages/cookbook-examples/write-tweet/)
    - And plenty of details about **using Pipelex** and **developing for Pipelex,** from **structured generation** to PipeOperators (**LLM**, **Image generation**, **OCR**…) to PipeControllers (**Sequence**, **Parallel**, **Batch**, **Condition**…), workflow **optimization**, workflow static **validation** and dry run… there’s still work to do, but we move fast!
- **Also a major update of Cursor rules**

### Tooling Improvements

- Pipeline tracking: restored **visual flowchart generation using Mermaid**
- Enhanced dry run configuration: added more granular control with `nb_list_items`, `nb_ocr_pages`, and `image_urls`
- New feature flags: better control over pipeline tracking, activity tracking, and reporting
- Improved OCR configuration: handle image file type for Mistral-OCR, added `default_page_views_dpi` setting
- Enhanced LLM configuration: **better prompting for structured generation with automatic schema insertion** for two-step structuring: generate plain text and then structure via Json
- Better logging: Enhanced log truncation and display for large objects like image bytes (there are still cases to deal with)

### Refactor

**Concept system refactoring**

- Improved concept code factory with better domain handling, so you no longer need the `native` domain prefix for native domains, you can just call them by their names: `Text`, `Image`, `PDF`, `Page`, `Number`…
- Concept `refines` attribute can now be a string for single refined concepts (the most common case)

### Breaking Changes

- File structure changes: documentation moved from `doc/` to `docs/`
- Configuration changes: some configuration keys have been renamed or restructured
- `StuffFactory.make_stuff()` argument `concept_code` renamed to `concept_str` to explicitly support concepts without fully qualified domains (e.g., `Text` or `PDF` implicitly `native` )
- Some method signatures have been updated

### Tests

- **Added Concept refinement validation:** `TestConceptRefinesValidationFunction` and `TestConceptPydanticFieldValidation` ensure proper concept inheritance and field validation

Author: lchoquel

.cursor/rules/pydantic.mdc .cursor/rules/base_models.mdc.cursor/rules/pydantic.mdc renamed to .cursor/rules/base_models.mdc

@@ -0,0 +1,8 @@
+---
+description: 
+globs: docs/**/*.md
+alwaysApply: false
+---
+Write docs and answer questions about writing docs.
+
+We use Material for MkDocs. All markdown in our docs must be compatible with Material for MkDocs and done using best practices to get the best results with Material for MkDocs.

.cursor/rules/docs.mdc

@@ -5,8 +5,10 @@ alwaysApply: false
 ---
 # Rules to choose LLM models used in PipeLLMs.
 
+## LLM Handles
+
 In order to use it in a pipe, an LLM is referenced by its llm_handle and possibly by an llm_preset.
-Both llm_handles and llm_presets are defined in this toml config file: [config_pipelex_llm_deck.toml](mdc:pipelex/config_pipelex_llm_deck.toml)
+Both llm_handles and llm_presets are defined in this toml config file: [base_llm_deck.toml](mdc:pipelex/libraries/llm_deck/base_llm_deck.toml)
 
 ## LLM Handles
 
@@ -15,19 +17,21 @@ An llm_handle matches the handle (an id of sorts) with the full specification of
 - llm_version
 - llm_platform_choice
 
-The declaration looks like this in toml syntax:
+The declaration of llm_handleslooks like this in toml syntax:
 ```toml
-[cogt.llm_config.llm_deck.llm_handle_to_llm_engine_blueprint.gpt-4o-2024-05-13]
-llm_name = "gpt-4o"
-llm_version = "2024-05-13"
-llm_platform_choice = "openai"
+[llm_handles]
+gpt-4o-2024-08-06 = { llm_name = "gpt-4o", llm_version = "2024-08-06" }
 ```
 
 In mosty cases, we only want to use version "latest" and llm_platform_choice "default" in which case the declaration is simply a match of the llm_handle to the llm_name, like this:
 ```toml
-gemini-2-5-pro = "gemini-2.5-pro"
+best-claude = "claude-4-opus"
+best-gemini = "gemini-2.5-pro"
+best-mistral = "mistral-large"
 ```
 
+And of course, llm_handles are automatically assigned for all models by their name, with version "latest" and llm_platform_choice "default".
+
 ## Using an LLM Handle in a PipeLLM
 
 Here is an example of using an llm_handle to specify which LLM to use in a PipeLLM:
@@ -59,7 +63,7 @@ The interest is that these presets can be used to set the LLM choice in a PipeLL
 ```toml
 [pipe.extract_invoice]
 PipeLLM = "Extract invoice information from an invoice text transcript"
-input = "InvoiceText"
+inputs = { invoice_text = "InvoiceText" }
 output = "Invoice"
 llm = "llm_to_extract_invoice"
 prompt_template = """
@@ -73,3 +77,6 @@ The category of this invoice is: $invoice_details.category.
 
 The setting here `llm = "llm_to_extract_invoice"` works because "llm_to_extract_invoice" has been declared as an llm_preset in the deck.
 You must not use an LLM preset in a PipeLLM that does not exist in the deck. If needed, you can add llm presets.
+
+
+You can override the predefined llm presets in [overrides.toml](mdc:pipelex/libraries/llm_deck/overrides.toml).

.cursor/rules/llms.mdc

@@ -1,6 +1,6 @@
 ---
 description: 
-globs: **/pipelex_libraries/pipelines/toml/**/*.toml
+globs: **/pipelex/libraries/pipelines/**/*.toml
 alwaysApply: false
 ---
 This rule explains how to build pipes.
@@ -9,7 +9,7 @@ This rule explains how to build pipes.
 
 ## The pipelines/ directory
 
-- Pipelines and structures are defined in the `pipelex/libraries/` directory
+- Pipelines and structures are defined in the `pipelex/libraries/pipelines` directory
 
 pipelex/libraries
 └── pipelines
@@ -40,6 +40,11 @@ pipelex/libraries
     - Never include the usage context in the concept. The concept indicates what the stuff is in itself, it's not "for something", it just is something. e.g. don't define "TextToSummarize": it's just "Text". If you want to refine the concept for instance you can define "Essay".
     - In particular, never define a concept as a plural form: if the context of the pipeline execution makes it multiple, it will just be handled using a ListContent (see below). e.g. don't define a concept for "Stories", just define "Story".
     - Also, avoid including adjectives in concepts, e.g. don't define "LargeText", it's just "Text".
+- Don't redefine the native concepts from [concept_native.py](mdc:pipelex/core/concept_native.py)
+
+⚠️ Important ⚠️
+
+A Concept MUST NEVER be a plural noun and you should never create a SomeConceptList: lists and arrays are implicitly handled by Pipelex according to the context. Just define SomeConcept.
 
 - Define concepts in one of two ways:
 
@@ -56,14 +61,14 @@ ConceptName = "Description of the concept"
 [concept.ConceptName]
 Concept = "Description of the concept"
 structure = "StructureName"
-refines = ["ParentConcept"]  # Optional, for concept inheritance
+refines = "ParentConcept"  # Optional, for concept inheritance
 ```
 
 About the `structure` field:
 - It's Optional
 - It's the name of the Python BaseModel class used for the concept
 - The class must be a subclass of StuffContent
-- The class must be defined in a python module placed inside the `pipelex/pipelex_libraries/pipelines/structures/` directory
+- The class must be defined in a python module placed inside the `pipelex/libraries/pipelines/` directory
 - If the `structure` field is omitted but a class with the same name as the concept is defined in the structures directory, then it's implicitly applied to the concept
 
 About the `refines` field:
@@ -83,7 +88,7 @@ Pipelex provides the following structures natively:
 - MermaidContent
 - LLMPromptContent
 
-The native structures are implied when using the native concepts: "native.Text", "native.Number", "native.Image"... do you don't have the state it.
+The native structures are implied when using the native concepts: "Text", "Number", "Image", "PDF",... do you don't have the state it.
 
 Some subclasses of StuffContent exist which you should never use directly:
 - ListContent: this is used internally to manipulate a list of stuff, but you should NEVER define a plural concept
@@ -130,7 +135,7 @@ Write a poem about an AI that meets a Software and they fall in l0ve.
 # Example of a PipeLLM that uses @ prefix to insert a block of text
 [pipe.process_text]
 PipeLLM = "Process input text using an LLM"
-input = "Text"
+inputs = { text = "Text" }
 output = "Text"
 llm = "llm_to_summarize_text"
 system_prompt = """
@@ -143,10 +148,10 @@ Summarize the following text:
 
 """
 
-# Example of a PipeLLM that uses @ prefix to insert a block of text but also a $ prefix to insert text inline in a sentence (that is teh case for the $topic)
+# Example of a PipeLLM that uses '@' prefix to insert a block of text but also a '$' prefix to insert text inline in a sentence (that is teh case for the $topic)
 [pipe.summarize_topic]
 PipeLLM = "Summarize a dense text with of focus on a specific topic."
-input = "Topic"
+inputs = { topic = "Topic", text = "Text" }
 output = "Summary"
 prompt_template = """
 Your goal is to summarize everything related to $topic in the provided text:
@@ -160,7 +165,7 @@ Your summary should not be longer than 2 sentences.
 # Example of a PipeLLM with image vision processing by a VLM
 [pipe.get_html_table_from_image]
 PipeLLM = "Convert table screenshot to HTML"
-input = "TableScreenshot"
+inputs = { table_screenshot = "TableScreenshot" }
 output = "HtmlTable"
 images = ["table_screenshot"]
 system_prompt = """
@@ -178,7 +183,7 @@ llm = "llm_to_extract_tables"
 # Example of a PipeSequence
 [pipe.answer_question_with_instructions]
 PipeSequence = "Answer a question with instructions"
-input = "Question"
+inputs = { question = "Question" }
 output = "FormattedAnswer"
 steps = [
     { pipe = "enrich_instructions", result = "instructions",  },
@@ -189,7 +194,7 @@ steps = [
 # Example of a PipeParallel
 [pipe.extract_expense_report]
 PipeParallel = "Extract useful information from an expense report"
-input = "ExpenseReportText"
+inputs = { expense_report = "ExpenseReportText" }
 output = "Composite"
 parallels = [
     { pipe = "extract_employee_from_expense_report", result = "employee" },
@@ -199,7 +204,7 @@ parallels = [
 # Example of a PipeCondition
 [pipe.expense_conditional_validation]
 PipeCondition = "Choose the rules to apply"
-input = "Expense"
+inputs = { expense = "Expense" }
 output = "RulesToApply"
 expression = "expense_category.category"
 ```

.cursor/rules/pipes.mdc

@@ -1,6 +1,6 @@
 ---
 description: 
-globs: **/pipelex_libraries/pipelines/structures/**/*.py
+globs: pipelex/libraries/pipelines/**/*.py
 alwaysApply: false
 ---
 Rules to write structure classes for concepts used in pipes with structured generations.
@@ -10,11 +10,14 @@ In particular, these structures are generated by PipeLLM using structured genera
 
 ## Model Location and Registration
 
-- Create models for structured generations related to "some_domain" in `pipelex/pipelex_libraries/pipelines/structures/<some_domain>.py`
+- Create models for structured generations related to "some_domain" in `pipelex_libraries/pipelines/<some_domain>.py`
 - Models must inherit from `StructuredContent` or appropriate content type
 
 ## Model Structure
 
+Concepts and their structure classes are meant to indicate an idea.
+A Concept MUST NEVER be a plural noun and you should never create a SomeConceptList: lists and arrays are implicitly handled by Pipelex according to the context. Just define SomeConcept.
+
 ```python
 from datetime import datetime
 from typing import List, Optional
@@ -41,6 +44,12 @@ class YourModel(StructuredContent):
             return v.replace(tzinfo=None)
         return v
 ```
+## Usage
+
+Structures are meant to indicate what class to use for a particular Concept. In general they use the same name as the concept.
+
+Structure classes defined within `pipelex_libraries/pipelines/` are automatically loaded into the class_registry when setting up Pipelex, no need to do it manually.
+
 
 ## Best Practices for structures
 
@@ -49,4 +58,4 @@ class YourModel(StructuredContent):
 - Use `Field` declaration and write the description
 - Use Pydantic validators for data cleaning/validation
 - Remove timezone info from datetime fields
-- Respect rules of [pydantic.mdc](mdc:.cursor/rules/pydantic.mdc)
+- Respect rules of @base_models.

.cursor/rules/structures.mdc

@@ -0,0 +1,31 @@
+# .github/workflows/docs-preview.yml
+name: Preview MkDocs on GitHub Pages
+
+on:
+  push:
+    branches: [main]
+
+jobs:
+  preview:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+
+      - uses: actions/setup-python@v5
+        with:
+          python-version: '3.x'
+
+      - name: Build docs
+        run: |
+          pip install mkdocs==1.6.1 mkdocs-material==9.6.14 mkdocs-glightbox==0.4.0 mkdocs-meta-manager==1.1.0
+          mkdocs build --strict         # generates ./site
+
+      - name: Publish to *gh-pages-preview*
+        uses: peaceiris/actions-gh-pages@v4
+        with:
+          github_token: ${{ secrets.GITHUB_TOKEN }}
+          publish_dir: ./site
+          publish_branch: gh-pages-preview   # <─ test branch
+          # put each branch in its own folder so multiple previews can coexist
+          destination_dir: ${{ github.ref_name }}
+          force_orphan: true

.github/workflows/deploy-doc.yml

@@ -0,0 +1,31 @@
+name: Documentation Check
+
+on:
+  pull_request:
+    branches:
+      - main
+      - dev
+      - "release/v[0-9]+.[0-9]+.[0-9]+"
+    paths:
+      - 'docs/**'
+      - 'mkdocs.yml'
+
+jobs:
+  doc-check:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+
+      - name: Set up Python
+        uses: actions/setup-python@v5
+        with:
+          python-version: '3.11'
+          cache: 'pip'
+
+      - name: Install mkdocs
+        run: |
+          python -m pip install --upgrade pip
+          pip install mkdocs==1.6.1 mkdocs-material==9.6.14 mkdocs-glightbox==0.4.0 mkdocs-meta-manager==1.1.0
+
+      - name: Check documentation build
+        run: mkdocs build --strict 

.github/workflows/doc-check.yml

@@ -36,8 +36,8 @@ jobs:
           if [[ "$HEAD" == "dev" ]]; then
             exit 0
           fi
-          if [[ ! "$HEAD" =~ ^(fix|feature|refactor|chore|doc|ci-cd|changelog)\/[A-Za-z0-9._\/\<\>\=\-]+$ ]]; then
-            echo "::error::Branch must start with fix/, feature/, refactor/, chore/, doc/, or ci-cd/."
+          if [[ ! "$HEAD" =~ ^(fix|feature|refactor|chore|doc|ci-cd|changelog|codex)\/[A-Za-z0-9._\/\<\>\=\-]+$ ]]; then
+            echo "::error::Branch must start with fix/, feature/, refactor/, chore/, docs/, or ci-cd/."
             exit 1
           fi
 

.github/workflows/guard-branches.yml

@@ -10,6 +10,9 @@ dist/
 build/
 *.egg-info/
 
+# mkdocs
+site/
+
 # Unit test / coverage reports
 htmlcov/
 .pytest_cache/

.gitignore

@@ -19,5 +19,6 @@
         "tests"
     ],
     "python.testing.unittestEnabled": false,
-    "python.testing.pytestEnabled": true
+    "python.testing.pytestEnabled": true,
+    "djlint.showInstallError": false
 }