Merge pull request Pipelex#131 from Pipelex/release/v0.4.9

### Highlights

**Plugin System Refactoring** - Complete overhaul of the plugin architecture to support external LLM providers.

### Added

- **External Plugin Support**: New `LLMWorkerAbstract` base class for integrating custom LLM providers, and we don't mean only an OpenAI-SDK-based LLM with a custom endpoint, now the implementation can be anything, as long as it implements the `LLMWorkerAbstract` interface.
- **Plugin SDK Registry**: Better management of SDK instances with proper teardown handling
- **Enhanced Error Formatting**: Improved Pydantic validation error messages for enums

### Changed

- **Plugin Architecture**: Moved plugin system to dedicated `pipelex.plugins` package
- **LLM Workers**: Split into `LLMWorkerInternalAbstract` (for built-in providers) and `LLMWorkerAbstract` (for external plugins)
- **Configuration**: Plugin configs moved from main `pipelex.toml` to separate `pipelex_libraries/plugins/plugin_config.toml` (⚠️ breaking change)
- **Error Handling**: Standardized credential errors with new `CredentialsError` base class

Author: lchoquel

.github/workflows/publish-pypi.yml

@@ -55,9 +55,9 @@ jobs:
 
   github-release:
     name: >-
-      Sign the Python 🐍 distribution 📦 with Sigstore
-      and upload them to GitHub Release
+      Test GitHub Release Creation with Changelog
     needs:
+      - build
       - publish-to-pypi
     runs-on: ubuntu-latest
 
@@ -74,6 +74,45 @@ jobs:
         run: |
           VERSION=$(grep -m 1 'version = ' pyproject.toml | cut -d '"' -f 2)
           echo "VERSION=$VERSION" >> $GITHUB_ENV
+      - name: Extract changelog notes for current version
+        id: get_changelog
+        run: |
+          VERSION=${{ env.VERSION }}
+          echo "Extracting changelog for version v$VERSION"
+          
+          # Find the start of the current version section
+          START_LINE=$(grep -n "## \[v$VERSION\] - " CHANGELOG.md | cut -d: -f1)
+          
+          if [ -z "$START_LINE" ]; then
+            echo "Warning: No changelog entry found for version v$VERSION"
+            echo "CHANGELOG_NOTES=" >> $GITHUB_ENV
+            exit 0
+          fi
+          
+          # Find the start of the next version section (previous version)
+          NEXT_VERSION_LINE=$(tail -n +$((START_LINE + 1)) CHANGELOG.md | grep -n "^## \[v.*\] - " | head -1 | cut -d: -f1)
+          
+          if [ -z "$NEXT_VERSION_LINE" ]; then
+            # No next version found, extract from current version till end of file
+            CHANGELOG_CONTENT=$(tail -n +$START_LINE CHANGELOG.md)
+          else
+            # Extract content from current version header to before next version
+            END_LINE=$((START_LINE + NEXT_VERSION_LINE - 1))
+            CHANGELOG_CONTENT=$(sed -n "$START_LINE,$((END_LINE - 1))p" CHANGELOG.md)
+          fi
+          
+          # Clean up the content but preserve the blank line after the header
+          # First, get the header line and add a blank line after it
+          HEADER_LINE=$(echo "$CHANGELOG_CONTENT" | head -1)
+          CONTENT_LINES=$(echo "$CHANGELOG_CONTENT" | tail -n +2 | sed '/^$/d' | sed 's/^[[:space:]]*//' | sed 's/[[:space:]]*$//')
+          
+          # Combine header + blank line + content
+          CHANGELOG_CONTENT=$(printf "%s\n\n%s" "$HEADER_LINE" "$CONTENT_LINES")
+          
+          # Escape for GitHub Actions
+          echo "CHANGELOG_NOTES<<EOF" >> $GITHUB_ENV
+          echo "$CHANGELOG_CONTENT" >> $GITHUB_ENV
+          echo "EOF" >> $GITHUB_ENV
       - name: Download all the dists
         uses: actions/download-artifact@v4
         with:
@@ -88,12 +127,18 @@ jobs:
       - name: Create GitHub Release
         env:
           GITHUB_TOKEN: ${{ github.token }}
-        run: >-
-          gh release create
-          "v$VERSION"
-          --repo "$GITHUB_REPOSITORY"
-          --title "v$VERSION"
-          --notes ""
+        run: |
+          if [ -n "$CHANGELOG_NOTES" ]; then
+            gh release create "v$VERSION" \
+              --repo "$GITHUB_REPOSITORY" \
+              --title "v$VERSION" \
+              --notes "$CHANGELOG_NOTES"
+          else
+            gh release create "v$VERSION" \
+              --repo "$GITHUB_REPOSITORY" \
+              --title "v$VERSION" \
+              --notes "Release v$VERSION"
+          fi
       - name: Upload artifact signatures to GitHub Release
         env:
           GITHUB_TOKEN: ${{ github.token }}

CHANGELOG.md

@@ -1,5 +1,24 @@
 # Changelog
 
+## [v0.4.9] - 2025-06-30
+
+### Highlights
+
+**Plugin System Refactoring** - Complete overhaul of the plugin architecture to support external LLM providers.
+
+### Added
+
+- **External Plugin Support**: New `LLMWorkerAbstract` base class for integrating custom LLM providers, and we don't mean only an OpenAI-SDK-based LLM with a custom endpoint, now the implementation can be anything, as long as it implements the `LLMWorkerAbstract` interface.
+- **Plugin SDK Registry**: Better management of SDK instances with proper teardown handling
+- **Enhanced Error Formatting**: Improved Pydantic validation error messages for enums
+
+### Changed
+
+- **Plugin Architecture**: Moved plugin system to dedicated `pipelex.plugins` package
+- **LLM Workers**: Split into `LLMWorkerInternalAbstract` (for built-in providers) and `LLMWorkerAbstract` (for external plugins)
+- **Configuration**: Plugin configs moved from main `pipelex.toml` to separate `pipelex_libraries/plugins/plugin_config.toml` (⚠️ breaking change)
+- **Error Handling**: Standardized credential errors with new `CredentialsError` base class
+
 ## [v0.4.8] - 2025-06-26
 
 - Added `StorageProviderAbstract`

docs/pages/advanced-customization/index.md

@@ -16,7 +16,6 @@ from pipelex import Pipelex
 pipelex = Pipelex(
     template_provider=MyTemplateProvider(),
     llm_model_provider=MyLLMProvider(),
-    plugin_manager=MyPluginManager(),
     inference_manager=MyInferenceManager(),
     pipeline_tracker=MyPipelineTracker(),
     activity_manager=MyActivityManager(),
@@ -38,7 +37,6 @@ from pipelex.hub import PipelexHub
 hub = PipelexHub()
 hub.set_template_provider(MyTemplateProvider())
 hub.set_llm_models_provider(MyLLMProvider())
-hub.set_plugin_manager(MyPluginManager())
 # ... and so on for other components
 ```
 
@@ -88,54 +86,45 @@ Pipelex supports injection of the following components:
     - Default: `LLMModelLibrary`
     - [Details](llm-model-provider-injection.md)
 
-3. **Plugin Manager** (`PluginManager`)
-
-    - Protocol: `PluginManagerProtocol`
-    - Default: `PluginManager`
-    - [Details](plugin-manager-injection.md)
-
-4. **Inference Manager** (`InferenceManager`)
+3. **Inference Manager** (`InferenceManager`)
 
     - Protocol: `InferenceManagerProtocol`
     - Default: `InferenceManager`
     - [Details](inference-manager-injection.md)
 
-5. **Reporting Delegate** (`ReportingManager`)
+4. **Reporting Delegate** (`ReportingManager`)
 
     - Protocol: `ReportingProtocol`
     - Default: `ReportingManager` or `ReportingNoOp` if disabled
     - [Details](reporting-delegate-injection.md)
 
-6. **Pipeline Tracker** (`PipelineTracker`)
+5. **Pipeline Tracker** (`PipelineTracker`)
 
     - Protocol: `PipelineTrackerProtocol`
     - Default: `PipelineTracker` or `PipelineTrackerNoOp` if disabled
     - [Details](pipeline-tracker-injection.md)
 
-7. **Activity Manager** (`ActivityManager`)
+6. **Activity Manager** (`ActivityManager`)
 
     - Protocol: `ActivityManagerProtocol`
     - Default: `ActivityManager` or `ActivityManagerNoOp` if disabled
     - [Details](activity-manager-injection.md)
 
-8. **Secrets Provider** (`EnvSecretsProvider`)
+7. **Secrets Provider** (`EnvSecretsProvider`)
 
     - Protocol: `SecretsProviderProtocol`
     - Default: `EnvSecretsProvider`
     - [Details](secrets-provider-injection.md)
 
-9. **Content Generator** (`ContentGenerator`)
+8. **Content Generator** (`ContentGenerator`)
 
     - Protocol: `ContentGeneratorProtocol`
     - Default: `ContentGenerator`
     - [Details](content-generator-injection.md)
 
-10. **Pipe Router** (`PipeRouter`)
+9. **Pipe Router** (`PipeRouter`)
 
     - Protocol: `PipeRouterProtocol`
     - Default: `PipeRouter`
     - [Details](pipe-router-injection.md)
 
-## Best Practices
-
-⚠️ Under construction

pipelex/cogt/exceptions.py

@@ -23,6 +23,10 @@ class SdkTypeError(CogtError):
     pass
 
 
+class SdkRegistryError(CogtError):
+    pass
+
+
 class LLMWorkerError(CogtError):
     pass
 
@@ -128,5 +132,9 @@ def __init__(self, dependency_name: str, extra_name: str, message: Optional[str]
         super().__init__(error_msg)
 
 
+class MissingPluginError(CogtError):
+    pass
+
+
 class OcrCapabilityError(CogtError):
     pass

pipelex/cogt/imgg/imgg_worker_factory.py

@@ -5,9 +5,9 @@
 from pipelex.cogt.imgg.imgg_platform import ImggPlatform
 from pipelex.cogt.imgg.imgg_worker_abstract import ImggWorkerAbstract
 from pipelex.cogt.llm.llm_models.llm_platform import LLMPlatform
-from pipelex.cogt.plugin_manager import PluginHandle
 from pipelex.hub import get_plugin_manager, get_secret
 from pipelex.plugins.openai.openai_imgg_worker import OpenAIImggWorker
+from pipelex.plugins.plugin_sdk_registry import PluginSdkHandle
 from pipelex.reporting.reporting_protocol import ReportingProtocol
 from pipelex.tools.secrets.secrets_errors import SecretNotFoundError
 
@@ -22,8 +22,8 @@ def make_imgg_worker(
         imgg_engine: ImggEngine,
         reporting_delegate: Optional[ReportingProtocol] = None,
     ) -> ImggWorkerAbstract:
-        imgg_sdk_handle = PluginHandle.get_for_imgg_engine(imgg_platform=imgg_engine.imgg_platform)
-        plugin_manager = get_plugin_manager()
+        imgg_sdk_handle = PluginSdkHandle.get_for_imgg_engine(imgg_platform=imgg_engine.imgg_platform)
+        plugin_sdk_registry = get_plugin_manager().plugin_sdk_registry
         imgg_worker: ImggWorkerAbstract
         match imgg_engine.imgg_platform:
             case ImggPlatform.FAL_AI:
@@ -41,7 +41,9 @@ def make_imgg_worker(
 
                 from pipelex.plugins.fal.fal_imgg_worker import FalImggWorker
 
-                imgg_sdk_instance = plugin_manager.get_imgg_sdk_instance(imgg_sdk_handle=imgg_sdk_handle) or plugin_manager.set_imgg_sdk_instance(
+                imgg_sdk_instance = plugin_sdk_registry.get_imgg_sdk_instance(
+                    imgg_sdk_handle=imgg_sdk_handle
+                ) or plugin_sdk_registry.set_imgg_sdk_instance(
                     imgg_sdk_handle=imgg_sdk_handle,
                     imgg_sdk_instance=FalAsyncClient(key=fal_api_key),
                 )
@@ -54,7 +56,9 @@ def make_imgg_worker(
             case ImggPlatform.OPENAI:
                 from pipelex.plugins.openai.openai_factory import OpenAIFactory
 
-                imgg_sdk_instance = plugin_manager.get_llm_sdk_instance(llm_sdk_handle=imgg_sdk_handle) or plugin_manager.set_llm_sdk_instance(
+                imgg_sdk_instance = plugin_sdk_registry.get_llm_sdk_instance(
+                    llm_sdk_handle=imgg_sdk_handle
+                ) or plugin_sdk_registry.set_llm_sdk_instance(
                     llm_sdk_handle=imgg_sdk_handle,
                     llm_sdk_instance=OpenAIFactory.make_openai_client(llm_platform=LLMPlatform.OPENAI),
                 )

pipelex/cogt/inference/inference_manager.py

@@ -1,4 +1,4 @@
-from typing import Dict, Optional
+from typing import Dict, Type
 
 from typing_extensions import override
 
@@ -12,6 +12,7 @@
 from pipelex.cogt.llm.llm_models.llm_engine_factory import LLMEngineFactory
 from pipelex.cogt.llm.llm_worker_abstract import LLMWorkerAbstract
 from pipelex.cogt.llm.llm_worker_factory import LLMWorkerFactory
+from pipelex.cogt.llm.llm_worker_internal_abstract import LLMWorkerInternalAbstract
 from pipelex.cogt.ocr.ocr_engine_factory import OcrEngineFactory
 from pipelex.cogt.ocr.ocr_worker_abstract import OcrWorkerAbstract
 from pipelex.cogt.ocr.ocr_worker_factory import OcrWorkerFactory
@@ -31,10 +32,16 @@ def __init__(self):
     def teardown(self):
         self.imgg_worker_factory = ImggWorkerFactory()
         self.ocr_worker_factory = OcrWorkerFactory()
-        self.llm_workers.clear()
-        self.imgg_workers.clear()
-        self.ocr_workers.clear()
-        log.verbose("InferenceManagerAsync reset")
+        for llm_worker in self.llm_workers.values():
+            llm_worker.teardown()
+        self.llm_workers = {}
+        for imgg_worker in self.imgg_workers.values():
+            imgg_worker.teardown()
+        self.imgg_workers = {}
+        for ocr_worker in self.ocr_workers.values():
+            ocr_worker.teardown()
+        self.ocr_workers = {}
+        log.verbose("InferenceManager teardown done")
 
     def print_workers(self):
         log.debug("LLM Workers:")
@@ -60,15 +67,15 @@ def setup_llm_workers(self):
         llm_handle_to_llm_engine_blueprint = get_llm_deck().llm_handles
         log.verbose(f"{len(llm_handle_to_llm_engine_blueprint)} LLM engine_cards found")
         for llm_handle, llm_engine_blueprint in llm_handle_to_llm_engine_blueprint.items():
-            self._setup_one_llm_worker(llm_engine_blueprint=llm_engine_blueprint, llm_handle=llm_handle)
+            self._setup_one_internal_llm_worker(llm_engine_blueprint=llm_engine_blueprint, llm_handle=llm_handle)
             log.verbose(f"Setup LLM worker for '{llm_handle}' on {llm_engine_blueprint.llm_platform_choice}")
         log.debug("Done setting up LLM Workers (async)")
 
-    def _setup_one_llm_worker(
+    def _setup_one_internal_llm_worker(
         self,
         llm_engine_blueprint: LLMEngineBlueprint,
         llm_handle: str,
-    ) -> LLMWorkerAbstract:
+    ) -> LLMWorkerInternalAbstract:
         llm_engine = LLMEngineFactory.make_llm_engine(llm_engine_blueprint=llm_engine_blueprint)
         llm_worker = LLMWorkerFactory.make_llm_worker(
             llm_engine=llm_engine,
@@ -78,27 +85,34 @@ def _setup_one_llm_worker(
         return llm_worker
 
     @override
-    def get_llm_worker(
-        self,
-        llm_handle: str,
-        specific_llm_engine_blueprint: Optional[LLMEngineBlueprint] = None,
-    ) -> LLMWorkerAbstract:
+    def get_llm_worker(self, llm_handle: str) -> LLMWorkerAbstract:
         if llm_worker := self.llm_workers.get(llm_handle):
             return llm_worker
         if not get_config().cogt.inference_manager_config.is_auto_setup_preset_llm:
             raise InferenceManagerWorkerSetupError(
                 f"No LLM worker for '{llm_handle}', set it up or enable cogt.inference_manager_config.is_auto_setup_preset_llm"
             )
 
-        if not specific_llm_engine_blueprint:
-            specific_llm_engine_blueprint = get_llm_deck().get_llm_engine_blueprint(llm_handle=llm_handle)
-        llm_worker = self._setup_one_llm_worker(
-            llm_engine_blueprint=specific_llm_engine_blueprint,
+        llm_engine_blueprint = get_llm_deck().get_llm_engine_blueprint(llm_handle=llm_handle)
+        llm_worker = self._setup_one_internal_llm_worker(
+            llm_engine_blueprint=llm_engine_blueprint,
             llm_handle=llm_handle,
         )
 
         return llm_worker
 
+    @override
+    def set_llm_worker_from_external_plugin(
+        self,
+        llm_handle: str,
+        llm_worker_class: Type[LLMWorkerAbstract],
+        should_warn_if_already_registered: bool = True,
+    ):
+        if llm_handle in self.llm_workers:
+            if should_warn_if_already_registered:
+                log.warning(f"LLM worker for '{llm_handle}' already registered, skipping")
+        self.llm_workers[llm_handle] = llm_worker_class(reporting_delegate=get_report_delegate())
+
     ####################################################################################################
     # Manage IMGG Workers
     ####################################################################################################

pipelex/cogt/inference/inference_manager_protocol.py

@@ -1,7 +1,6 @@
-from typing import Optional, Protocol
+from typing import Protocol, Type
 
 from pipelex.cogt.imgg.imgg_worker_abstract import ImggWorkerAbstract
-from pipelex.cogt.llm.llm_models.llm_engine_blueprint import LLMEngineBlueprint
 from pipelex.cogt.llm.llm_worker_abstract import LLMWorkerAbstract
 from pipelex.cogt.ocr.ocr_worker_abstract import OcrWorkerAbstract
 
@@ -20,11 +19,14 @@ def teardown(self): ...
 
     def setup_llm_workers(self): ...
 
-    def get_llm_worker(
+    def get_llm_worker(self, llm_handle: str) -> LLMWorkerAbstract: ...
+
+    def set_llm_worker_from_external_plugin(
         self,
         llm_handle: str,
-        specific_llm_engine_blueprint: Optional[LLMEngineBlueprint] = None,
-    ) -> LLMWorkerAbstract: ...
+        llm_worker_class: Type[LLMWorkerAbstract],
+        should_warn_if_already_registered: bool = True,
+    ): ...
 
     ####################################################################################################
     # IMG Generation Workers

pipelex/cogt/inference/inference_worker_abstract.py

@@ -11,6 +11,12 @@ def __init__(
     ):
         self.reporting_delegate = reporting_delegate
 
+    def setup(self):
+        pass
+
+    def teardown(self):
+        pass
+
     @property
     @abstractmethod
     def desc(self) -> str: