Skip to content

Detect an existing URL scheme instead of searching for "http" #350

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
ckarnell wants to merge 1 commit into
base: main
Choose a base branch
from
+13 −1
Open

Detect an existing URL scheme instead of searching for "http" #350

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions _test_unstructured_client/unit/test_custom_hooks.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -204,6 +204,11 @@ def test_unit_clean_server_url_fixes_malformed_paid_api_url(server_url: str):
("localhost:8000", "http://localhost:8000"),
("localhost:8000/general/v0/general", "http://localhost:8000/general/v0/general"),
("http://localhost:8000/general/v0/general", "http://localhost:8000/general/v0/general"),
# -- a host whose NAME contains "http" still needs a scheme added. Testing for a
# -- scheme with `"http" not in base_url` treats these as already schemed. --
("myhttpd.local", "http://myhttpd.local"),

@cubic-dev-ai cubic-dev-ai Bot Aug 10, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P3: The new host-containing-"http" cases only cover the non-Unstructured-domain branch, which keeps the path. The same bug also affected hosts containing "http" under unstructuredapp.io — for those, netloc stayed empty so the subpath-stripping and https-forcing block never ran, and with the regex fix that branch behaves differently than before for such hosts. Consider adding one or two cases that also exercise that branch (e.g. ("http2.api.unstructuredapp.io/general/v0/general", "https://http2.api.unstructuredapp.io")) so the netloc-dependent cleanup is pinned down too.

Prompt for AI agents 
Check if this issue is valid — if so, understand the root cause and fix it. At _test_unstructured_client/unit/test_custom_hooks.py, line 209:

<comment>The new host-containing-"http" cases only cover the non-Unstructured-domain branch, which keeps the path. The same bug also affected hosts containing "http" under `unstructuredapp.io` — for those, `netloc` stayed empty so the subpath-stripping and `https`-forcing block never ran, and with the regex fix that branch behaves differently than before for such hosts. Consider adding one or two cases that also exercise that branch (e.g. `("http2.api.unstructuredapp.io/general/v0/general", "https://http2.api.unstructuredapp.io")`) so the netloc-dependent cleanup is pinned down too.</comment>

<file context>
@@ -204,6 +204,11 @@ def test_unit_clean_server_url_fixes_malformed_paid_api_url(server_url: str):
         ("http://localhost:8000/general/v0/general", "http://localhost:8000/general/v0/general"),
+        # -- a host whose NAME contains "http" still needs a scheme added. Testing for a
+        # -- scheme with `"http" not in base_url` treats these as already schemed. --
+        ("myhttpd.local", "http://myhttpd.local"),
+        ("http2.example.com", "http://http2.example.com"),
+        ("myhttpserver.example.com/general/v0/general", "http://myhttpserver.example.com/general/v0/general"),
</file context>

("http2.example.com", "http://http2.example.com"),
("myhttpserver.example.com/general/v0/general", "http://myhttpserver.example.com/general/v0/general"),
],
)
def test_unit_clean_server_url_fixes_non_unst_domain_url(server_url: str, expected_url: str):
Expand Down
9 changes: 8 additions & 1 deletion src/unstructured_client/_hooks/custom/clean_server_url_hook.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,11 +1,18 @@
from __future__ import annotations

import re
from typing import Tuple
from urllib.parse import ParseResult, urlparse, urlunparse

from unstructured_client._hooks.types import SDKInitHook
from unstructured_client.httpclient import HttpClient

# A scheme is "<letter><letter|digit|+|-|.>* ://" (RFC 3986). Matching on the substring
# "http" instead treats any host containing it, such as `myhttpd.local`, as already
# schemed. Requiring "://" also keeps `localhost:8000` unschemed, which is why urlparse
# is not used for this check: it reads `localhost` as the scheme.
_URL_SCHEME_RE = re.compile(r"^[a-zA-Z][a-zA-Z0-9+.\-]*://")


def clean_server_url(base_url: str | None) -> str:
"""Fix url scheme and remove subpath for URLs under Unstructured domains."""
Expand All @@ -14,7 +21,7 @@ def clean_server_url(base_url: str | None) -> str:
return ""

# add a url scheme if not present (urllib.parse does not work reliably without it)
if "http" not in base_url:
if not _URL_SCHEME_RE.match(base_url):
base_url = "http://" + base_url

parsed_url: ParseResult = urlparse(base_url)
Expand Down