Fix/remove images field from pipe llm blueprint (Pipelex#105)

Author: thomashebrard

.cursor/rules/best_practices.mdc

@@ -0,0 +1,77 @@
+---
+description: 
+globs: 
+alwaysApply: false
+---
+# Best Practices Guide
+
+This document outlines the core best practices and patterns used in our codebase.
+
+## Type Hints
+
+1. **Always Use Type Hints**
+   - Every function parameter must be typed
+   - Every function return must be typed
+   - Use type hints for all variables where type is not obvious
+
+2. **StrEnum**
+   - Import StrEnum from pipelex.types
+   ```python
+   from pipelex.types import StrEnum
+   
+   class ModelType(StrEnum):
+       GPT4 = "gpt-4"
+       GPT35 = "gpt-3.5-turbo"
+   ```
+
+## Factory Pattern
+
+1. **Use Factory Pattern for Object Creation**
+   - Create factories when dealing with multiple implementations
+
+## Documentation
+
+1. **Docstring Format**
+   - Quick description of the function/class
+   - List args and their types
+   - Document return values
+   - Example:
+   ```python
+   def process_image(image_path: str, size: Tuple[int, int]) -> bytes:
+       """Process and resize an image.
+       
+       Args:
+           image_path: Path to the source image
+           size: Tuple of (width, height) for resizing
+           
+       Returns:
+           Processed image as bytes
+       """
+       pass
+   ```
+
+2. **Class Documentation**
+   - Document class purpose and behavior
+   - Include examples if complex
+   ```python
+   class ImageProcessor:
+       """Handles image processing operations.
+       
+       Provides methods for resizing, converting, and optimizing images.
+       """
+   ```
+
+## Custom Exceptions
+
+1. **Graceful Error Handling**
+   - Use try/except blocks with specific exceptions
+   - Convert third-party exceptions to custom ones
+   ```python
+   try:
+       from fal_client import AsyncClient as FalAsyncClient
+   except ImportError as exc:
+       raise MissingDependencyError(
+           "fal-client", "fal", 
+           "The fal-client SDK is required to use FAL models."
+       ) from exc
+   ```

.cursor/rules/pipes.mdc

@@ -5,20 +5,9 @@ alwaysApply: false
 ---
 This rule explains how to build pipes.
 
-# File Naming & Structure
-
-## The pipelines/ directory
-
-- Pipelines and structures are defined in the `pipelex/libraries/pipelines` directory
-
-pipelex/libraries
-└── pipelines
-
 ## Pipeline file naming
 
-- Pipeline TOML files should be placed in the `pipelex/libraries/pipelines/` directory or a subdirectory of it
 - The file name should be descriptive, in snake_case and end with `.toml`
-- The file should contain both concepts and pipes definitions
 
 ## Pipeline file structure
 
@@ -167,7 +156,6 @@ Your summary should not be longer than 2 sentences.
 PipeLLM = "Convert table screenshot to HTML"
 inputs = { table_screenshot = "TableScreenshot" }
 output = "HtmlTable"
-images = ["table_screenshot"]
 system_prompt = """
 You are a vision-based table extractor.
 """

.cursor/rules/pytest.mdc

@@ -17,6 +17,8 @@ These rules apply when writing unit tests.
 - More precisely, for `pipelex` and `pipelex.cogt` the async tests are placed inside subdirectories named `cogt_asynch` and `pipelex_asynch`
 - Fixtures are defined in conftest.py modules at different levels of the hierarchy, their scope is handled by pytest
 - Test data is placed inside test_data.py at different levels of the hierarchy, they must be imported with package paths from the root like `tests.pipelex.test_data`. Their content is all constants, regrouped inside classes to keep things tidy.
+- Always put test inside Test classes.
+- The pipelex pipelines should be stored in `tests/test_pipelines` as well as the related structured Output classes that inherit from `StructuredContent`
 
 ## Markers
 

.cursor/rules/standards.mdc

@@ -0,0 +1,60 @@
+---
+description: 
+globs: 
+alwaysApply: true
+---
+# Coding Standards
+
+This document outlines the coding standards and quality control procedures that must be followed when contributing to this project.
+
+## Code Quality Checks
+
+### Linting and Type Checking
+
+Before finalizing a task, you must run the following command to check for linting issues, type errors, and code quality problems:
+
+```bash
+make check
+```
+
+This command runs multiple code quality tools:
+- Pyright: Static type checking
+- Ruff: Fast Python linter
+- Mypy: Static type checker
+
+Always fix any issues reported by these tools before proceeding.
+
+### Running Tests
+
+We have several make commands for running tests:
+
+1. `make tp`: Runs all tests with these markers:
+   ```
+   (dry_runnable or not (inference or llm or imgg or ocr)) and not (needs_output or pipelex_api)
+   ```
+   Use this for quick test runs that don't require LLM or image generation.
+
+2. `make ti`: Runs all tests with these markers:
+   ```
+   inference and not imgg
+   ```
+   Use this for testing LLM functionality without image generation.
+
+3. To run specific tests:
+   ```bash
+   make tp TEST=TestClassName
+   # or
+   make tp TEST=test_function_name
+   ```
+   It matches names, so `TEST=test_function_name` is going to run all test with the function name that STARTS with `test_function_name`.
+
+## Important Project Directories
+
+### Pipelines Directory
+- All pipeline definitions go in `pipelex/libraries/pipelines/`
+
+### Tests Directory
+- All tests are located in the `tests/` directory
+
+### Documentation Directory
+- All documentation is located in the `docs/` directory

.cursor/rules/structures.mdc

@@ -59,3 +59,4 @@ Structure classes defined within `pipelex_libraries/pipelines/` are automaticall
 - Use Pydantic validators for data cleaning/validation
 - Remove timezone info from datetime fields
 - Respect rules of @base_models.
+

.cursor/rules/tdd.mdc

@@ -0,0 +1,28 @@
+---
+description: 
+globs: 
+alwaysApply: false
+---
+# Test-Driven Development Guide
+
+This document outlines our test-driven development (TDD) process and the tools available for testing.
+
+## TDD Cycle
+
+1. **Write a Test First**
+[pytest.mdc](mdc:.cursor/rules/pytest.mdc)
+
+2. **Write the Code**
+   - Implement the minimum amount of code needed to pass the test
+   - Follow the project's coding standards
+   - Keep it simple - don't write more than needed
+
+3. **Run Linting and Type Checking**
+[standards.mdc](mdc:.cursor/rules/standards.mdc)
+
+4. **Refactor if needed**
+If the code needs refactoring, with the best practices [best_practices.mdc](mdc:.cursor/rules/best_practices.mdc)
+
+5. **Validate tests**
+
+Remember: The key to TDD is writing the test first and letting it drive your implementation. Always run the full test suite and quality checks before considering a feature complete.

docs/pages/build-reliable-ai-workflows-with-pipelex/pipe-operators/PipeLLM.md

@@ -13,6 +13,96 @@ For structured data output, `PipeLLM` employs two main strategies:
     a. First, the LLM generates a free-form text based on the initial prompt.
     b. Second, another LLM call is made with a specific prompt designed to extract and structure the information from the generated text into the target Pydantic model.
 
+## Working with Images (Vision Language Models)
+
+`PipeLLM` supports Vision Language Models (VLMs) that can process both text and images. To use images in your prompts:
+
+### Basic Image Input
+
+Images must be declared in the `inputs` section of your pipe definition. The image will be automatically passed to the VLM along with your text prompt.
+
+```toml
+[pipe.describe_image]
+PipeLLM = "Describe an image"
+inputs = { image = "Image" }
+output = "VisualDescription"
+prompt_template = """
+Describe the provided image in great detail.
+"""
+```
+
+**Important**: Do NOT reference image variables in your prompt template using `@image` or `$image`. Images are automatically passed to vision-enabled LLMs and should not be treated as text variables.
+
+**Flexible Image Inputs**
+
+You can use any concept that refines `Image` as an input, and choose descriptive variable names that fit your use case:
+
+```toml
+[pipe.analyze_wedding]
+PipeLLM = "Analyze wedding photo"
+inputs = { wedding_photo = "images.Photo" }
+output = "PhotoAnalysis"
+prompt_template = """
+Analyze this wedding photo and describe the key moments captured.
+"""
+```
+
+### Images as Sub-attributes of Structured Content
+
+When working with structured content that contains image fields (like `PageContent` which has a `page_view` field), you need to specify the full path to the image attribute in the `inputs` section:
+
+```toml
+[pipe.analyze_page_view]
+PipeLLM = "Analyze the visual layout of a page"
+inputs = { "page_content.page_view" = "Image" }
+output = "LayoutAnalysis"
+prompt_template = """
+Analyze the visual layout and design elements of this page.
+Focus on typography, spacing, and overall composition.
+"""
+```
+
+In this example:
+- `page_content` is the input variable containing a `PageContent` object
+- `page_view` is the `ImageContent` field within the `PageContent` structure
+- The dot notation `page_content.page_view` tells Pipelex to extract the image from that specific field
+
+### Multiple Images
+
+You can include multiple images in a single prompt by listing them in the inputs:
+
+```toml
+[pipe.compare_images]
+PipeLLM = "Compare two images"
+inputs = { 
+    first_image = "Image",
+    second_image = "Image"
+}
+output = "ImageComparison"
+prompt_template = """
+Compare these two images and describe their similarities and differences.
+"""
+```
+
+### Combining Text and Image Inputs
+
+You can mix any stuff and image inputs in the same pipe:
+
+```toml
+[pipe.analyze_document_with_context]
+PipeLLM = "Analyze a document page with additional context"
+inputs = { 
+    context = "Text",
+    document.page_view = "Image"
+}
+output = "DocumentAnalysis"
+prompt_template = """
+Given this context: $context
+
+Analyze the document page shown in the image and explain how it relates to the provided context.
+"""
+```
+
 ## Configuration
 
 `PipeLLM` is configured in your pipeline's `.toml` file.
@@ -22,17 +112,18 @@ For structured data output, `PipeLLM` employs two main strategies:
 | Parameter                   | Type                | Description                                                                                                                                                                  | Required |
 | --------------------------- | ------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------- |
 | `PipeLLM`                   | string              | A descriptive name for the LLM operation.                                                                           | Yes      |
-| `inputs`                    | dictionary          | The input concept(s) for the LLM operation, as a dictionary mapping input names to concept codes.                                                     | Yes       |
+| `inputs`                    | dictionary          | The input concept(s) for the LLM operation, as a dictionary mapping input names to concept codes. For images within structured content, use dot notation (e.g., `"page.image"`).                                                     | Yes       |
 | `output`                    | string              | The output concept produced by the LLM operation.                                                | Yes      |
 | `llm`                       | string or table     | Specifies the LLM preset(s) to use. Can be a single preset or a table mapping different presets for different generation modes (e.g., `main`, `object_direct`).              | No       |
 | `system_prompt`             | string              | A system-level prompt to guide the LLM's behavior (e.g., "You are a helpful assistant"). Can be inline text or a reference to a template file (`"file:path/to/prompt.md"`).  | No       |
 | `prompt`                    | string              | A simple, static user prompt. Use this when you don't need to inject any variables.                                                                                          | No       |
-| `prompt_template`           | string              | A template for the user prompt. Use `$` for inline variables (e.g., `$topic`) and `@` to insert the content of an entire input (e.g., `@text_to_summarize`).                 | No       |
-| `images`                    | list of strings     | For Vision Language Models (VLMs), specifies which input variables are images.                                                                                               | No       |
+| `prompt_template`           | string              | A template for the user prompt. Use `$` for inline variables (e.g., `$topic`) and `@` to insert the content of an entire input (e.g., `@text_to_summarize`). **Note**: Do not use `@` or `$` for image variables.                 | No       |
+| `images`                    | list of strings     | **Deprecated**: Use the `inputs` section to declare image inputs instead.                                                                                               | No       |
 | `structuring_method`        | string              | The method for generating structured output. Can be `direct` or `preliminary_text`. Defaults to the global configuration.                                                      | No       |
 | `prompt_template_to_structure` | string           | The prompt template for the second step in `preliminary_text` mode.                                                                                                            | No       |
 | `output_multiplicity`       | string or integer   | Defines the number of outputs. Use `"list"` for a variable-length list, or an integer (e.g., `3`) for a fixed-size list.                                                       | No       |
 
+## Examples
 
 ### Simple Text Generation Example
 
@@ -75,7 +166,6 @@ This pipe takes an image of a table and uses a VLM to extract the content as an
 PipeLLM = "Extract table data from an image"
 inputs = { image = "TableScreenshot" }
 output = "TableData"
-images = ["image"]
 prompt_template = """
 Extract the table data from this image and format it as a structured table.
 """
@@ -105,4 +195,4 @@ Analyze this expense report and extract the following information:
 """
 ```
 
-In this example, `Pipelex` will instruct the LLM to return a list of objects that conform to the `Expense` structure.
+In this example, `Pipelex` will instruct the LLM to return a list of objects that conform to the `Expense` structure.

docs/pages/cookbook-examples/extract-dpe.md

@@ -63,7 +63,6 @@ The pipeline uses a `PipeLLM` with a very specific prompt to extract the informa
 PipeLLM = "Write markdown from page content of a 'Diagnostic de Performance Energetique'"
 inputs = { page_content = "Page" }
 output = "Dpe" # The output is structured as a Dpe object
-images = ["page_content.page_view"]
 llm = "llm_for_img_to_text"
 structuring_method = "preliminary_text"
 system_prompt = """You are a multimodal LLM, expert in converting images into perfect markdown."""

docs/pages/cookbook-examples/extract-gantt.md

@@ -79,7 +79,6 @@ PipeLLM = "Extract the precise dates of the task, start_date and end_date"
 inputs = { gantt_chart_image = "GanttChartImage", gantt_timescale = "GanttTimescaleDescription", gantt_task_name = "GanttTaskName" }
 output = "GanttTaskDetails" # The output is structured as a GanttTaskDetails object
 structuring_method = "preliminary_text"
-images = ["gantt_chart_image"]
 llm = "llm_to_extract_diagram"
 prompt_template = """
 I am sharing an image of a Gantt chart.

docs/pages/cookbook-examples/extract-proof-of-purchase.md

@@ -57,9 +57,8 @@ The pipeline uses a powerful `PipeLLM` to extract the structured data from the d
 ```toml
 [pipe.write_markdown_from_page_content_proof_of_purchase]
 PipeLLM = "Write markdown from page content"
-inputs = { page_content = "Page" }
+inputs = { "page_content.page_view" = "Page" } # The LLM receives the image of the page
 output = "ProofOfPurchase" # The LLM is forced to output a ProofOfPurchase object
-images = ["page_content.page_view"] # The LLM receives the image of the page
 llm = "llm_for_img_to_text"
 structuring_method = "preliminary_text"
 system_prompt = """You are a multimodal LLM, expert at converting images into perfect markdown."""