Model Engine OnPrem Support and vLLM 0.11.1 + Model Engine Integration Fixes (#744)

* add support for on-prem

* clean up on-prem artificats

* add back comments from initial code

* fix lint

* use ecr image repo:tag directly

* fix: isort import ordering

* fix: remove unused infra_config import

* fix: mypy type annotation errors

* fix: remove type annotation causing mypy no-redef error

* fix: mypy type errors in s3_utils.py and io.py - use botocore.config.Config

* fix: mypy typeddict-item errors - use broad type ignore

* fix: update test mocks to use get_s3_resource from s3_utils

* test: add unit tests for s3_utils, onprem_docker_repository, and onprem_queue_endpoint_resource_delegate

* style: format test files with black

* refactor: use filesystem_gateway abstraction for S3 operations

Address review feedback to use the filesystem_gateway abstraction layer
instead of directly calling get_s3_client. This ensures on-prem S3
configuration logic is properly encapsulated in the gateway.

Changes:
- Add head_object, delete_object, list_objects methods to S3FilesystemGateway
- Update S3FileStorageGateway to use self.filesystem_gateway for all S3 ops
- Remove direct import of get_s3_client from s3_file_storage_gateway

* fix: deduplicate S3 client config by using centralized s3_utils

Refactor open_wrapper to use get_s3_client from s3_utils instead of
duplicating the on-prem S3 configuration logic. This ensures a single
source of truth for S3 client creation across the codebase.

* fix: add pagination to list_objects to handle >1000 objects

S3 list_objects_v2 returns max 1000 objects per request. Use paginator
to iterate through all pages and return complete results. Without this
fix, directories with >1000 files would silently return truncated results.

* fix: make OnPremDockerRepository.get_image_url consistent with ECR/ACR

Include docker_repo_prefix in image URL to match behavior of ECR and ACR
implementations. Also change image_exists logging from warning to debug
to reduce log noise on every deployment.

Updated tests to mock infra_config and verify prefix handling.

* refactor: add explicit on-prem branches in dependencies.py for clarity

Add explicit elif branches for on-prem cloud provider to make it clear
that S3-based gateways are intentionally used for on-prem (with MinIO
configuration applied via s3_utils). This improves code readability
and makes the on-prem support more discoverable.

* feat: implement Redis LLEN for queue depth in OnPremQueueEndpointResourceDelegate

Replace hardcoded queue depth with actual Redis LLEN call to enable
proper autoscaling based on queue metrics. Falls back to 0 gracefully
if Redis client is unavailable.

- Add optional redis_client parameter to constructor
- Implement lazy Redis client initialization
- Add tests for both with and without Redis scenarios

* fix: replace mutable default argument with None in _get_client

Using {} as a default argument is a Python anti-pattern that can cause
subtle bugs since the same dict instance is shared across calls.
Use Optional[Dict] = None pattern instead.

* refactor: extract inline import to module-level helper function

Move the infra_config import from inside the validator to a module-level
helper function _is_onprem_deployment(). This improves testability,
avoids repeated import overhead on each validation call, and follows
Python best practices for imports.

* fix: reduce excessive debug logging in s3_utils

Replace per-call debug logs with a one-time info log when S3 is
configured for on-prem. This prevents log spam from debug messages
firing on every S3 client creation.

- Extract common on-prem config to _get_onprem_client_kwargs helper
- Add _s3_config_logged flag to log endpoint only once
- Add return type annotations to get_s3_client and get_s3_resource
- Update tests to reset logging flag between tests

* chore: remove unused TYPE_CHECKING import

Clean up unused import left over from refactoring the inline import.

* fix: make Dockerfile multi-arch compatible for ARM/AMD64

Use architecture detection to download the correct binaries for
aws-iam-authenticator and kubectl. This enables building the image
for both ARM64 (Mac M1/M2) and AMD64 (CI/production) platforms.

* style: fix black formatting in test_onprem_queue_endpoint_resource_delegate

* fix: restore AWS_PROFILE env var fallback in s3_utils

The original code checked os.getenv('AWS_PROFILE') as a fallback when
no aws_profile kwarg was provided. This was accidentally removed during
refactoring, breaking S3 operations in CI where AWS_PROFILE may be set
via environment variable.

Restores the original behavior for AWS deployments while maintaining
the new on-prem path.

* fix: correct isort ordering in s3_filesystem_gateway.py

* fix: use Literal type for s3 addressing_style to satisfy mypy

* Onprem Compatibility Change

---------

Co-authored-by: Tarun <tarun.ravikumar@scale.com>

Author: charlesahn-scale

charts/model-engine/templates/_helpers.tpl

@@ -256,6 +256,10 @@ env:
   - name: ABS_CONTAINER_NAME
     value: {{ .Values.azure.abs_container_name }}
   {{- end }}
+  {{- if .Values.s3EndpointUrl }}
+  - name: S3_ENDPOINT_URL
+    value: {{ .Values.s3EndpointUrl | quote }}
+  {{- end }}
 {{- end }}
 
 {{- define "modelEngine.syncForwarderTemplateEnv" -}}
@@ -342,9 +346,27 @@ env:
     value: "/workspace/model-engine/model_engine_server/core/configs/config.yaml"
   {{- end }}
   - name: CELERY_ELASTICACHE_ENABLED
-    value: "true"
+    value: {{ .Values.celeryElasticacheEnabled | default true | quote }}
   - name: LAUNCH_SERVICE_TEMPLATE_FOLDER
     value: "/workspace/model-engine/model_engine_server/infra/gateways/resources/templates"
+  {{- if .Values.s3EndpointUrl }}
+  - name: S3_ENDPOINT_URL
+    value: {{ .Values.s3EndpointUrl | quote }}
+  {{- end }}
+  {{- if .Values.redisHost }}
+  - name: REDIS_HOST
+    value: {{ .Values.redisHost | quote }}
+  - name: REDIS_PORT
+    value: {{ .Values.redisPort | default "6379" | quote }}
+  {{- end }}
+  {{- if .Values.celeryBrokerUrl }}
+  - name: CELERY_BROKER_URL
+    value: {{ .Values.celeryBrokerUrl | quote }}
+  {{- end }}
+  {{- if .Values.celeryResultBackend }}
+  - name: CELERY_RESULT_BACKEND
+    value: {{ .Values.celeryResultBackend | quote }}
+  {{- end }}
   {{- if .Values.redis.auth}}
   - name: REDIS_AUTH_TOKEN
     value: {{ .Values.redis.auth }}

model-engine/Dockerfile

@@ -21,13 +21,20 @@ RUN apt-get update && apt-get install -y \
   telnet \
   && rm -rf /var/lib/apt/lists/*
 
-RUN curl -Lo /bin/aws-iam-authenticator https://github.com/kubernetes-sigs/aws-iam-authenticator/releases/download/v0.5.9/aws-iam-authenticator_0.5.9_linux_amd64
-RUN chmod +x /bin/aws-iam-authenticator
+# Install aws-iam-authenticator (architecture-aware)
+RUN ARCH=$(dpkg --print-architecture) && \
+    if [ "$ARCH" = "arm64" ]; then \
+      curl -Lo /bin/aws-iam-authenticator https://github.com/kubernetes-sigs/aws-iam-authenticator/releases/download/v0.5.9/aws-iam-authenticator_0.5.9_linux_arm64; \
+    else \
+      curl -Lo /bin/aws-iam-authenticator https://github.com/kubernetes-sigs/aws-iam-authenticator/releases/download/v0.5.9/aws-iam-authenticator_0.5.9_linux_amd64; \
+    fi && \
+    chmod +x /bin/aws-iam-authenticator
 
-# Install kubectl
-RUN curl -LO "https://dl.k8s.io/release/v1.23.13/bin/linux/amd64/kubectl" \
-  && chmod +x kubectl \
-  && mv kubectl /usr/local/bin/kubectl
+# Install kubectl (architecture-aware)
+RUN ARCH=$(dpkg --print-architecture) && \
+    curl -LO "https://dl.k8s.io/release/v1.23.13/bin/linux/${ARCH}/kubectl" && \
+    chmod +x kubectl && \
+    mv kubectl /usr/local/bin/kubectl
 
 # Pin pip version
 RUN pip install pip==24.2

model-engine/model_engine_server/api/dependencies.py

@@ -94,6 +94,9 @@
 from model_engine_server.infra.gateways.resources.live_endpoint_resource_gateway import (
     LiveEndpointResourceGateway,
 )
+from model_engine_server.infra.gateways.resources.onprem_queue_endpoint_resource_delegate import (
+    OnPremQueueEndpointResourceDelegate,
+)
 from model_engine_server.infra.gateways.resources.queue_endpoint_resource_delegate import (
     QueueEndpointResourceDelegate,
 )
@@ -114,6 +117,7 @@
     FakeDockerRepository,
     LiveTokenizerRepository,
     LLMFineTuneRepository,
+    OnPremDockerRepository,
     RedisModelEndpointCacheRepository,
     S3FileLLMFineTuneEventsRepository,
     S3FileLLMFineTuneRepository,
@@ -223,6 +227,8 @@ def _get_external_interfaces(
     queue_delegate: QueueEndpointResourceDelegate
     if CIRCLECI:
         queue_delegate = FakeQueueEndpointResourceDelegate()
+    elif infra_config().cloud_provider == "onprem":
+        queue_delegate = OnPremQueueEndpointResourceDelegate()
     elif infra_config().cloud_provider == "azure":
         queue_delegate = ASBQueueEndpointResourceDelegate()
     else:
@@ -232,7 +238,8 @@ def _get_external_interfaces(
 
     inference_task_queue_gateway: TaskQueueGateway
     infra_task_queue_gateway: TaskQueueGateway
-    if CIRCLECI:
+    if CIRCLECI or infra_config().cloud_provider == "onprem":
+        # On-prem uses Redis-based task queues
         inference_task_queue_gateway = redis_24h_task_queue_gateway
         infra_task_queue_gateway = redis_task_queue_gateway
     elif infra_config().cloud_provider == "azure":
@@ -274,16 +281,15 @@ def _get_external_interfaces(
         monitoring_metrics_gateway=monitoring_metrics_gateway,
         use_asyncio=(not CIRCLECI),
     )
-    filesystem_gateway = (
-        ABSFilesystemGateway()
-        if infra_config().cloud_provider == "azure"
-        else S3FilesystemGateway()
-    )
-    llm_artifact_gateway = (
-        ABSLLMArtifactGateway()
-        if infra_config().cloud_provider == "azure"
-        else S3LLMArtifactGateway()
-    )
+    filesystem_gateway: FilesystemGateway
+    llm_artifact_gateway: LLMArtifactGateway
+    if infra_config().cloud_provider == "azure":
+        filesystem_gateway = ABSFilesystemGateway()
+        llm_artifact_gateway = ABSLLMArtifactGateway()
+    else:
+        # AWS uses S3, on-prem uses MinIO (S3-compatible)
+        filesystem_gateway = S3FilesystemGateway()
+        llm_artifact_gateway = S3LLMArtifactGateway()
     model_endpoints_schema_gateway = LiveModelEndpointsSchemaGateway(
         filesystem_gateway=filesystem_gateway
     )
@@ -323,23 +329,18 @@ def _get_external_interfaces(
     cron_job_gateway = LiveCronJobGateway()
 
     llm_fine_tune_repository: LLMFineTuneRepository
+    llm_fine_tune_events_repository: LLMFineTuneEventsRepository
     file_path = os.getenv(
         "CLOUD_FILE_LLM_FINE_TUNE_REPOSITORY",
         hmi_config.cloud_file_llm_fine_tune_repository,
     )
     if infra_config().cloud_provider == "azure":
-        llm_fine_tune_repository = ABSFileLLMFineTuneRepository(
-            file_path=file_path,
-        )
+        llm_fine_tune_repository = ABSFileLLMFineTuneRepository(file_path=file_path)
+        llm_fine_tune_events_repository = ABSFileLLMFineTuneEventsRepository()
     else:
-        llm_fine_tune_repository = S3FileLLMFineTuneRepository(
-            file_path=file_path,
-        )
-    llm_fine_tune_events_repository = (
-        ABSFileLLMFineTuneEventsRepository()
-        if infra_config().cloud_provider == "azure"
-        else S3FileLLMFineTuneEventsRepository()
-    )
+        # AWS uses S3, on-prem uses MinIO (S3-compatible)
+        llm_fine_tune_repository = S3FileLLMFineTuneRepository(file_path=file_path)
+        llm_fine_tune_events_repository = S3FileLLMFineTuneEventsRepository()
     llm_fine_tuning_service = DockerImageBatchJobLLMFineTuningService(
         docker_image_batch_job_gateway=docker_image_batch_job_gateway,
         docker_image_batch_job_bundle_repo=docker_image_batch_job_bundle_repository,
@@ -350,16 +351,19 @@ def _get_external_interfaces(
         docker_image_batch_job_gateway=docker_image_batch_job_gateway
     )
 
-    file_storage_gateway = (
-        ABSFileStorageGateway()
-        if infra_config().cloud_provider == "azure"
-        else S3FileStorageGateway()
-    )
+    file_storage_gateway: FileStorageGateway
+    if infra_config().cloud_provider == "azure":
+        file_storage_gateway = ABSFileStorageGateway()
+    else:
+        # AWS uses S3, on-prem uses MinIO (S3-compatible)
+        file_storage_gateway = S3FileStorageGateway()
 
     docker_repository: DockerRepository
     if CIRCLECI:
         docker_repository = FakeDockerRepository()
-    elif infra_config().docker_repo_prefix.endswith("azurecr.io"):
+    elif infra_config().cloud_provider == "onprem":
+        docker_repository = OnPremDockerRepository()
+    elif infra_config().cloud_provider == "azure":
         docker_repository = ACRDockerRepository()
     else:
         docker_repository = ECRDockerRepository()

model-engine/model_engine_server/common/config.py

@@ -70,12 +70,13 @@ class HostedModelInferenceServiceConfig:
     user_inference_tensorflow_repository: str
     docker_image_layer_cache_repository: str
     sensitive_log_mode: bool
-    # Exactly one of the following three must be specified
+    # Exactly one of the following must be specified for Redis cache
     cache_redis_aws_url: Optional[str] = None  # also using this to store sync autoscaling metrics
     cache_redis_azure_host: Optional[str] = None
     cache_redis_aws_secret_name: Optional[str] = (
         None  # Not an env var because the redis cache info is already here
     )
+    cache_redis_onprem_url: Optional[str] = None  # For on-prem Redis (e.g., redis://redis:6379/0)
     sglang_repository: Optional[str] = None
 
     @classmethod
@@ -90,21 +91,34 @@ def from_yaml(cls, yaml_path):
 
     @property
     def cache_redis_url(self) -> str:
+        # On-prem Redis support (explicit URL, no cloud provider dependency)
+        if self.cache_redis_onprem_url:
+            return self.cache_redis_onprem_url
+
+        cloud_provider = infra_config().cloud_provider
+
+        # On-prem: support REDIS_HOST env var fallback
+        if cloud_provider == "onprem":
+            if self.cache_redis_aws_url:
+                logger.info("On-prem deployment using cache_redis_aws_url")
+                return self.cache_redis_aws_url
+            redis_host = os.getenv("REDIS_HOST", "redis")
+            redis_port = getattr(infra_config(), "redis_port", 6379)
+            return f"redis://{redis_host}:{redis_port}/0"
+
         if self.cache_redis_aws_url:
-            assert infra_config().cloud_provider == "aws", "cache_redis_aws_url is only for AWS"
+            assert cloud_provider == "aws", "cache_redis_aws_url is only for AWS"
             if self.cache_redis_aws_secret_name:
                 logger.warning(
                     "Both cache_redis_aws_url and cache_redis_aws_secret_name are set. Using cache_redis_aws_url"
                 )
             return self.cache_redis_aws_url
         elif self.cache_redis_aws_secret_name:
-            assert (
-                infra_config().cloud_provider == "aws"
-            ), "cache_redis_aws_secret_name is only for AWS"
-            creds = get_key_file(self.cache_redis_aws_secret_name)  # Use default role
+            assert cloud_provider == "aws", "cache_redis_aws_secret_name is only for AWS"
+            creds = get_key_file(self.cache_redis_aws_secret_name)
             return creds["cache-url"]
 
-        assert self.cache_redis_azure_host and infra_config().cloud_provider == "azure"
+        assert self.cache_redis_azure_host and cloud_provider == "azure"
         username = os.getenv("AZURE_OBJECT_ID")
         token = DefaultAzureCredential().get_token("https://redis.azure.com/.default")
         password = token.token

model-engine/model_engine_server/core/aws/roles.py

@@ -119,12 +119,21 @@ def session(role: Optional[str], session_type: SessionT = Session) -> SessionT:
 
     :param:`session_type` defines the type of session to return. Most users will use
     the default boto3 type. Some users required a special type (e.g aioboto3 session).
+
+    For on-prem deployments without AWS profiles, pass role=None or role=""
+    to use default credentials from environment variables (AWS_ACCESS_KEY_ID, etc).
     """
     # Do not assume roles in CIRCLECI
     if os.getenv("CIRCLECI"):
         logger.warning(f"In circleci, not assuming role (ignoring: {role})")
         role = None
-    sesh: SessionT = session_type(profile_name=role)
+
+    # Use profile-based auth only if role is specified
+    # For on-prem with MinIO, role will be None or empty - use env var credentials
+    if role:
+        sesh: SessionT = session_type(profile_name=role)
+    else:
+        sesh: SessionT = session_type()  # Uses default credential chain (env vars)
     return sesh
 
 

model-engine/model_engine_server/core/aws/storage_client.py

@@ -1,3 +1,4 @@
+import os
 import time
 from typing import IO, Callable, Iterable, Optional, Sequence
 
@@ -20,6 +21,10 @@
 
 
 def sync_storage_client(**kwargs) -> BaseClient:
+    # Support for MinIO/on-prem S3-compatible storage
+    endpoint_url = os.getenv("S3_ENDPOINT_URL")
+    if endpoint_url and "endpoint_url" not in kwargs:
+        kwargs["endpoint_url"] = endpoint_url
     return session(infra_config().profile_ml_worker).client("s3", **kwargs)  # type: ignore
 
 

model-engine/model_engine_server/core/celery/app.py

@@ -531,17 +531,28 @@ def _get_backend_url_and_conf(
         backend_url = get_redis_endpoint(1)
     elif backend_protocol == "s3":
         backend_url = "s3://"
-        if aws_role is None:
-            aws_session = session(infra_config().profile_ml_worker)
+        if infra_config().cloud_provider == "aws":
+            if aws_role is None:
+                aws_session = session(infra_config().profile_ml_worker)
+            else:
+                aws_session = session(aws_role)
+            out_conf_changes.update(
+                {
+                    "s3_boto3_session": aws_session,
+                    "s3_bucket": s3_bucket,
+                    "s3_base_path": s3_base_path,
+                }
+            )
         else:
-            aws_session = session(aws_role)
-        out_conf_changes.update(
-            {
-                "s3_boto3_session": aws_session,
-                "s3_bucket": s3_bucket,
-                "s3_base_path": s3_base_path,
-            }
-        )
+            logger.info(
+                "Non-AWS deployment, using environment variables for S3 backend credentials"
+            )
+            out_conf_changes.update(
+                {
+                    "s3_bucket": s3_bucket,
+                    "s3_base_path": s3_base_path,
+                }
+            )
     elif backend_protocol == "abs":
         backend_url = f"azureblockblob://{os.getenv('ABS_ACCOUNT_NAME')}"
     else:

model-engine/model_engine_server/core/configs/onprem.yaml

@@ -0,0 +1,72 @@
+# On-premise deployment configuration
+# This configuration file provides defaults for on-prem deployments
+# Many values can be overridden via environment variables
+
+cloud_provider: "onprem"
+env: "production"  # Can be: production, staging, development, local
+k8s_cluster_name: "onprem-cluster"
+dns_host_domain: "ml.company.local"
+default_region: "us-east-1"  # Placeholder for compatibility with cloud-agnostic code
+
+# ====================
+# Object Storage (MinIO/S3-compatible)
+# ====================
+s3_bucket: "model-engine"
+# S3 endpoint URL - can be overridden by S3_ENDPOINT_URL env var
+# Examples: "https://minio.company.local", "http://minio-service:9000"
+s3_endpoint_url: ""  # Set via S3_ENDPOINT_URL env var if not specified here
+# MinIO requires path-style addressing (bucket in URL path, not subdomain)
+s3_addressing_style: "path"
+
+# ====================
+# Redis Configuration
+# ====================
+# Redis is used for:
+# - Celery task queue broker
+# - Model endpoint caching
+# - Inference autoscaling metrics
+redis_host: ""  # Set via REDIS_HOST env var (e.g., "redis.company.local" or "redis-service")
+redis_port: 6379
+# Whether to use Redis as Celery broker (true for on-prem)
+celery_broker_type_redis: true
+
+# ====================
+# Celery Configuration
+# ====================
+# Backend protocol: "redis" for on-prem (not "s3" or "abs")
+celery_backend_protocol: "redis"
+
+# ====================
+# Database Configuration
+# ====================
+# Database connection settings (credentials from environment variables)
+# DB_HOST, DB_PORT, DB_NAME, DB_USER, DB_PASSWORD
+db_host: "postgres"  # Default hostname, can be overridden by DB_HOST env var
+db_port: 5432
+db_name: "llm_engine"
+db_engine_pool_size: 20
+db_engine_max_overflow: 10
+db_engine_echo: false
+db_engine_echo_pool: false
+db_engine_disconnect_strategy: "pessimistic"
+
+# ====================
+# Docker Registry Configuration
+# ====================
+# Docker registry prefix for container images
+# Examples: "registry.company.local", "harbor.company.local/ml-platform"
+# Leave empty if using full image paths directly
+docker_repo_prefix: "registry.company.local"
+
+# ====================
+# Monitoring & Observability
+# ====================
+# Prometheus server address for metrics (optional)
+# prometheus_server_address: "http://prometheus:9090"
+
+# ====================
+# Not applicable for on-prem (kept for compatibility)
+# ====================
+ml_account_id: "onprem"
+profile_ml_worker: "default"
+profile_ml_inference_worker: "default"