Detect an existing URL scheme instead of searching for "http"

[ckarnell]

clean_server_url adds a missing scheme with if "http" not in base_url. That is a substring test against the whole string, so any host whose name happens to contain those four characters is read as already schemed and comes back without one.

example.com                            -> http://example.com
myhttpd.local                          -> myhttpd.local
http2.example.com/general/v0/general   -> http2.example.com/general/v0/general

Adding the scheme is the thing the hook exists to do. The function docstring is "Fix url scheme" and the class docstring is "fixing common mistakes by users in defining server_url". urlparse then reads the schemeless string as a path, so netloc is empty and the cleanup below it works on nothing.

This matches a real scheme instead, <letter><letter|digit|+|-|.>* followed by :// per RFC 3986. Requiring the :// is what keeps localhost:8000 unschemed, and it is why urlparse cannot do the check here, since it reads localhost as the scheme.

test_unit_clean_server_url_fixes_non_unst_domain_url is already parameterised for exactly this and none of its cases uses a host containing "http", which is why the suite stayed green. Three added that do.

Verified in a clean venv with the package installed editable. Reverting the one source line, keeping the new cases, gives exactly those 3 failures and nothing else moves. Full unit suite is 274 passed, 1 xfailed with the change, against 271 and 1 without it. Five test dependencies your pyproject.toml declares were missing from my environment at first and 43 tests failed for that reason alone, so that baseline is after installing them.

Low severity. It needs a hostname containing "http", which is unusual, and only self-hosted users pass a bare host at all.

I did not test against a running server.

[cubic-dev-ai]

1 issue found across 2 files

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="_test_unstructured_client/unit/test_custom_hooks.py">

<violation number="1" location="_test_unstructured_client/unit/test_custom_hooks.py:209">
P3: The new host-containing-"http" cases only cover the non-Unstructured-domain branch, which keeps the path. The same bug also affected hosts containing "http" under `unstructuredapp.io` — for those, `netloc` stayed empty so the subpath-stripping and `https`-forcing block never ran, and with the regex fix that branch behaves differently than before for such hosts. Consider adding one or two cases that also exercise that branch (e.g. `("http2.api.unstructuredapp.io/general/v0/general", "https://http2.api.unstructuredapp.io")`) so the netloc-dependent cleanup is pinned down too.</violation>
</file>

<sub>Shadow auto-approve: would not auto-approve because issues were found.

Re-trigger cubic</sub>

[cubic-dev-ai]

P3: The new host-containing-"http" cases only cover the non-Unstructured-domain branch, which keeps the path. The same bug also affected hosts containing "http" under unstructuredapp.io — for those, netloc stayed empty so the subpath-stripping and https-forcing block never ran, and with the regex fix that branch behaves differently than before for such hosts. Consider adding one or two cases that also exercise that branch (e.g. ("http2.api.unstructuredapp.io/general/v0/general", "https://http2.api.unstructuredapp.io")) so the netloc-dependent cleanup is pinned down too.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At _test_unstructured_client/unit/test_custom_hooks.py, line 209:

<comment>The new host-containing-"http" cases only cover the non-Unstructured-domain branch, which keeps the path. The same bug also affected hosts containing "http" under `unstructuredapp.io` — for those, `netloc` stayed empty so the subpath-stripping and `https`-forcing block never ran, and with the regex fix that branch behaves differently than before for such hosts. Consider adding one or two cases that also exercise that branch (e.g. `("http2.api.unstructuredapp.io/general/v0/general", "https://http2.api.unstructuredapp.io")`) so the netloc-dependent cleanup is pinned down too.</comment>

<file context>
@@ -204,6 +204,11 @@ def test_unit_clean_server_url_fixes_malformed_paid_api_url(server_url: str):
         ("http://localhost:8000/general/v0/general", "http://localhost:8000/general/v0/general"),
+        # -- a host whose NAME contains "http" still needs a scheme added. Testing for a
+        # -- scheme with `"http" not in base_url` treats these as already schemed. --
+        ("myhttpd.local", "http://myhttpd.local"),
+        ("http2.example.com", "http://http2.example.com"),
+        ("myhttpserver.example.com/general/v0/general", "http://myhttpserver.example.com/general/v0/general"),
</file context>