Skip to content

Commit 3373899

Browse files
authored
feat: retry locked loads, read env vars, clear workspace error (#219)
* feat(loads): retry table loads while another load holds the table The server serialises loads per table and answers a second writer with HTTP 409 RESOURCE_LOCKED + Retry-After. Add hotdata/loads.py with DatabasesApi / ConnectionsApi subclasses that retry load_database_table and load_managed_table on that code only, honouring Retry-After with jitter under a LoadRetryPolicy budget (60s default), and raise TableLockedError when it is exhausted. CONFLICT and IDEMPOTENCY_KEY_CONFLICT are never retried; the code is matched, not the message. Export them as hotdata.DatabasesApi / ConnectionsApi via patch_query_exports.py so a regen keeps the swap, and list loads.py in .openapi-generator-ignore. * feat(config): default api_key, workspace_id and host from the environment Configuration() now reads HOTDATA_API_KEY, HOTDATA_WORKSPACE and HOTDATA_API_URL when the matching argument is None, the same variables the CLI honours. Explicit arguments win; an empty variable is unset. The change lives in the mustache template the generator uses for configuration.py, so a regen keeps it. * feat(client): raise MissingWorkspaceError before an unscoped request A workspace-scoped operation sent without a workspace id came back as a bare 404. ApiClient.update_params_for_auth now calls hotdata._workspace.ensure_workspace, which raises MissingWorkspaceError (an ApiValueError) naming workspace_id= and HOTDATA_WORKSPACE whenever the operation's security lists WorkspaceId and no value would be sent. Explicit X-Workspace-Id headers, _request_auth overrides and BearerAuth-only endpoints (Workspaces) are untouched. api_client.py is generated, so the hook is applied by scripts/patch_workspace_guard.py from the regenerate workflow; _workspace.py is generator-ignored and the error is exported from hotdata via patch_query_exports.py. * feat(query): add fetch_ready_result and document wait_for_result's status-only reply wait_for_result polls with limit=0 and returns a response without rows, which surprised callers. Say so prominently in its docstring and add QueryApi.fetch_ready_result, which waits the same way and then fetches a page of rows via get_result. Existing behaviour is unchanged. * feat(models)!: reject unknown fields on request models Generated request models silently dropped unknown keyword arguments (pydantic's default extra='ignore'), so a typo or a field from a newer API never reached the server. scripts/patch_model_extra_forbid.py adds extra='forbid' to every model used as a request body and to the models nested in them, and the regenerate workflow re-applies it. Response models are untouched so a newer server still cannot break an older SDK. BREAKING CHANGE: constructing a request model with an unknown keyword now raises pydantic.ValidationError. --------- Co-authored-by: Eddie A Tejeda <669988+eddietejeda@users.noreply.github.com>
1 parent 84cf4b9 commit 3373899

48 files changed

Lines changed: 1594 additions & 22 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.github/workflows/regenerate.yml‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -201,6 +201,12 @@ jobs:
201201
- name: Patch transport request defaults (Accept-Encoding, User-Agent)
202202
run: python3 scripts/patch_request_defaults.py
203203

204+
- name: Patch missing-workspace guard into ApiClient
205+
run: python3 scripts/patch_workspace_guard.py
206+
207+
- name: Patch request models to reject unknown fields (extra=forbid)
208+
run: python3 scripts/patch_model_extra_forbid.py
209+
204210
# The API-token -> JWT key exchange is deprecated: the configured API token
205211
# is the bearer credential and goes on the wire verbatim. This check is the
206212
# inverse of the old "did the exchange survive?" guard — it fails if the

‎.openapi-generator-ignore‎

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -11,11 +11,16 @@ setup.py
1111
# connection-reset retry on all methods, #118), uploads.py (transparent
1212
# presigned direct-to-storage upload flow), _useragent.py (runtime-resolved
1313
# User-Agent; the generator's httpUserAgent property can only bake a literal at
14-
# generation time, which goes stale between releases).
14+
# generation time, which goes stale between releases), loads.py (409
15+
# RESOURCE_LOCKED retry of table loads while another load holds the table),
16+
# _workspace.py (client-side missing-workspace guard, wired into api_client by
17+
# scripts/patch_workspace_guard.py).
1518
hotdata/arrow.py
1619
hotdata/query.py
1720
hotdata/_retry.py
1821
hotdata/uploads.py
22+
hotdata/loads.py
23+
hotdata/_workspace.py
1924
hotdata/_useragent.py
2025

2126
# Hand-written test for the patched ApiClient.close()/context-manager behavior

‎.openapi-generator-templates/configuration.mustache‎

Lines changed: 14 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -9,6 +9,7 @@ import logging
99
from logging import FileHandler
1010
{{^async}}
1111
import multiprocessing
12+
import os
1213
{{/async}}
1314
import sys
1415
from importlib.metadata import PackageNotFoundError, version as _pkg_version
@@ -156,13 +157,16 @@ class HostSetting(TypedDict):
156157
class Configuration:
157158
"""This class contains various settings of the API client.
158159

159-
:param host: Base url.
160+
:param host: Base url. Defaults to the `HOTDATA_API_URL` environment
161+
variable when set, else `https://api.hotdata.dev`.
160162
:param ignore_operation_servers
161163
Boolean to ignore operation servers for the API client.
162164
Config will use `host` as the base url regardless of the operation servers.
163165
:param api_key: Hotdata API key, sent as `Authorization: Bearer <key>`.
166+
Defaults to the `HOTDATA_API_KEY` environment variable when not given.
164167
:param workspace_id: Public id of the target workspace, sent as
165-
`X-Workspace-Id`.
168+
`X-Workspace-Id`. Defaults to the `HOTDATA_WORKSPACE` environment
169+
variable when not given.
166170
:param api_keys: Escape hatch for raw apiKey-scheme values, keyed by
167171
security scheme name. Prefer `workspace_id`.
168172
:param api_key_prefix: Dict to store API prefix (e.g. Bearer).
@@ -299,6 +303,14 @@ conf = {{packageName}}.Configuration(
299303
) -> None:
300304
"""Constructor
301305
"""
306+
# Environment defaults, matching the Hotdata CLI: an explicit argument
307+
# always wins; an unset or empty variable leaves the default in place.
308+
if api_key is None:
309+
api_key = os.environ.get("HOTDATA_API_KEY") or None
310+
if workspace_id is None:
311+
workspace_id = os.environ.get("HOTDATA_WORKSPACE") or None
312+
if host is None:
313+
host = os.environ.get("HOTDATA_API_URL") or None
302314
self._base_path = "{{basePath}}" if host is None else host
303315
"""Default Base url
304316
"""

‎CHANGELOG.md‎

Lines changed: 36 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -7,8 +7,44 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
77

88
## [Unreleased]
99

10+
### Added
11+
12+
- `hotdata.Configuration` reads `HOTDATA_API_KEY`, `HOTDATA_WORKSPACE` and
13+
`HOTDATA_API_URL` from the environment when `api_key`, `workspace_id` or
14+
`host` is not passed, matching the CLI. Explicit arguments always win and an
15+
empty variable counts as unset.
16+
- A workspace-scoped call made with no workspace id configured now raises
17+
`hotdata.MissingWorkspaceError` (an `ApiValueError`) before anything is sent,
18+
naming `workspace_id=` and `HOTDATA_WORKSPACE`. Previously the request went
19+
out without `X-Workspace-Id` and the server answered a bare 404. Calls that
20+
pass the header via `_headers`, use `_request_auth`, or target endpoints that
21+
are not workspace-scoped (the Workspaces endpoints) are unaffected.
22+
- `QueryApi.fetch_ready_result(result_id, x_database_id, offset=, limit=,
23+
format=)` waits for a result to be `ready` and then fetches its rows in one
24+
call. `QueryApi.wait_for_result` is unchanged: it polls with `limit=0` and
25+
returns a status-only response with no rows, which its docstring now states
26+
up front.
27+
- `DatabasesApi.load_database_table` and `ConnectionsApi.load_managed_table`
28+
(the classes exported from `hotdata`) retry automatically while another load
29+
holds the table: HTTP 409 with error code `RESOURCE_LOCKED` is retried,
30+
honouring `Retry-After` (else exponential backoff) with jitter, under a
31+
60s budget by default. Tune or disable with `LoadRetryPolicy`
32+
(`DatabasesApi(client, load_retry=LoadRetryPolicy(...))`); when the budget
33+
runs out, `hotdata.TableLockedError` (a 409 `ApiException`) is raised. Other
34+
409s (`CONFLICT`, `IDEMPOTENCY_KEY_CONFLICT`) are never retried. The raw
35+
generated classes stay available under `hotdata.api.*`.
36+
1037
### Changed
1138

39+
- **Breaking:** request models reject unknown fields. Every model used as a
40+
request body (`QueryRequest`, `LoadManagedTableRequest`,
41+
`CreateDatabaseRequest`, ... and the models nested in them) now carries
42+
`extra="forbid"`, so a misspelled keyword, or a field a newer API has that
43+
this SDK release does not know yet, raises `pydantic.ValidationError` at
44+
construction instead of being silently dropped from the request. Upgrade the
45+
SDK to use a new field. Response models are unchanged: unknown fields in
46+
server responses are still ignored, so a newer server never breaks an older
47+
SDK.
1248
- feat(query): add arrow ipc stream response format
1349
- chore(query): clarify write statement handling in dialect parameter
1450
- chore(indexes): clarify vector index constraints and algorithms

‎README.md‎

Lines changed: 30 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -37,7 +37,17 @@ configuration = hotdata.Configuration(
3737
)
3838
```
3939

40-
`host` defaults to `https://api.hotdata.dev`. Override it if you target another environment.
40+
Each of these can come from the environment instead, the same variables the
41+
CLI uses: `HOTDATA_API_KEY`, `HOTDATA_WORKSPACE` and `HOTDATA_API_URL` fill in
42+
`api_key`, `workspace_id` and `host` when the argument is not passed. An
43+
explicit argument always wins.
44+
45+
`host` defaults to `https://api.hotdata.dev`. Override it (or set
46+
`HOTDATA_API_URL`) if you target another environment.
47+
48+
Almost every call is workspace-scoped. If no workspace id is configured, the
49+
SDK raises `hotdata.MissingWorkspaceError` before sending the request rather
50+
than letting the server answer with an unexplained 404.
4151

4252
## Usage
4353

@@ -60,6 +70,25 @@ with hotdata.ApiClient(configuration) as api_client:
6070

6171
Each `Api` class groups endpoints by resource. Construct the client, then call the typed methods you need.
6272

73+
## Loading tables
74+
75+
Loads are serialised per table on the server. `hotdata.DatabasesApi` and
76+
`hotdata.ConnectionsApi` retry a load that hits a table another load still
77+
holds (HTTP 409, error code `RESOURCE_LOCKED`), honouring the server's
78+
`Retry-After`, for up to 60s by default:
79+
80+
```python
81+
from hotdata import DatabasesApi, LoadRetryPolicy, TableLockedError
82+
83+
databases = DatabasesApi(api_client, load_retry=LoadRetryPolicy(deadline_s=120))
84+
try:
85+
databases.load_database_table(db_id, "public", "events", request)
86+
except TableLockedError as e:
87+
print(f"gave up after {e.attempts} attempts: {e.reason}")
88+
```
89+
90+
Other 409s (`CONFLICT`, `IDEMPOTENCY_KEY_CONFLICT`) are raised immediately.
91+
6392
## Arrow results
6493

6594
Query results can be fetched as an [Apache Arrow](https://arrow.apache.org/) IPC stream instead of JSON, which is faster and far more memory-efficient for large result sets. Install the optional extra:

‎hotdata/__init__.py‎

Lines changed: 15 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -312,10 +312,21 @@
312312
# hotdata.query.QueryApi adds 429 retry + truncation auto-follow;
313313
# hotdata.arrow.ResultsApi adds Arrow IPC result fetch;
314314
# hotdata.uploads.UploadsApi adds transparent presigned direct-to-storage
315-
# uploads. The raw generated classes remain importable from hotdata.api.*.
315+
# uploads; hotdata.loads.DatabasesApi / ConnectionsApi retry table loads
316+
# while another load holds the table (409 RESOURCE_LOCKED). The raw
317+
# generated classes remain importable from hotdata.api.*.
316318
from hotdata.query import QueryApi as QueryApi # noqa: E402,F811
317319
from hotdata.arrow import ResultsApi as ResultsApi # noqa: E402,F811
318320
from hotdata.uploads import UploadsApi as UploadsApi # noqa: E402,F811
321+
from hotdata.loads import DatabasesApi as DatabasesApi # noqa: E402,F811
322+
from hotdata.loads import ConnectionsApi as ConnectionsApi # noqa: E402,F811
323+
from hotdata.loads import ( # noqa: E402,F401
324+
LoadRetryPolicy as LoadRetryPolicy,
325+
TableLockedError as TableLockedError,
326+
)
327+
# Raised client-side (scripts/patch_workspace_guard.py) when a workspace-
328+
# scoped call has no workspace id configured.
329+
from hotdata._workspace import MissingWorkspaceError as MissingWorkspaceError # noqa: E402,F401
319330
# The upload error hierarchy + option/helper types, so `from hotdata import
320331
# UploadError` / `StorageError` / ... works (they otherwise live only under
321332
# hotdata.uploads and are invisible to autocomplete on `hotdata.`).
@@ -350,4 +361,7 @@
350361
"UploadProgress",
351362
"UploadSource",
352363
"tqdm_progress",
364+
"LoadRetryPolicy",
365+
"TableLockedError",
366+
"MissingWorkspaceError",
353367
]

‎hotdata/_workspace.py‎

Lines changed: 84 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,84 @@
1+
"""Fail fast when a workspace-scoped request has no workspace to go to.
2+
3+
Most operations are scoped to a workspace through the ``X-Workspace-Id``
4+
header (the ``WorkspaceId`` security scheme). The generated client simply
5+
skips a scheme that has no configured value, so a ``Configuration`` built
6+
without ``workspace_id`` sends the request anyway and the server answers with
7+
a bare 404 that says nothing about what was missing.
8+
9+
:func:`ensure_workspace` runs before the request is sent (wired into
10+
``ApiClient.update_params_for_auth`` by ``scripts/patch_workspace_guard.py``)
11+
and raises :class:`MissingWorkspaceError` naming the constructor argument and
12+
the environment variable. Operations whose security does not list
13+
``WorkspaceId`` (workspace management itself) are unaffected, as are calls
14+
that supply the header explicitly or override auth per request.
15+
"""
16+
17+
from __future__ import annotations
18+
19+
from typing import Any, Iterable, Mapping, Optional
20+
21+
from hotdata.exceptions import ApiValueError
22+
23+
WORKSPACE_SCHEME = "WorkspaceId"
24+
WORKSPACE_HEADER = "X-Workspace-Id"
25+
26+
27+
class MissingWorkspaceError(ApiValueError):
28+
"""Raised before sending a workspace-scoped request when no workspace id
29+
is configured.
30+
31+
Pass ``workspace_id=`` to :class:`hotdata.Configuration`, set the
32+
``HOTDATA_WORKSPACE`` environment variable, or send the ``X-Workspace-Id``
33+
header explicitly for the call.
34+
"""
35+
36+
def __init__(self, resource_path: str, method: str) -> None:
37+
self.resource_path = resource_path
38+
self.method = method
39+
super().__init__(
40+
f"{method} {resource_path} is scoped to a workspace but no workspace id "
41+
"is configured. Pass workspace_id=... to hotdata.Configuration(...), "
42+
"set the HOTDATA_WORKSPACE environment variable, or send the "
43+
f"{WORKSPACE_HEADER} header explicitly (_headers=...)."
44+
)
45+
46+
47+
def ensure_workspace(
48+
configuration: Any,
49+
auth_settings: Optional[Iterable[str]],
50+
headers: Optional[Mapping[str, Any]],
51+
resource_path: str,
52+
method: str,
53+
request_auth: Optional[Mapping[str, Any]] = None,
54+
) -> None:
55+
"""Raise :class:`MissingWorkspaceError` if the operation needs a workspace
56+
id and none would be sent.
57+
58+
:param configuration: The client's :class:`hotdata.Configuration`.
59+
:param auth_settings: The operation's security scheme names.
60+
:param headers: Header params already assembled for the request.
61+
:param request_auth: A per-request auth override; when given the
62+
configuration is bypassed entirely, so the check is skipped.
63+
"""
64+
if request_auth:
65+
return
66+
if not auth_settings or WORKSPACE_SCHEME not in auth_settings:
67+
return
68+
if headers and any(
69+
str(name).lower() == WORKSPACE_HEADER.lower() for name in headers
70+
):
71+
return
72+
settings = configuration.auth_settings()
73+
scheme = settings.get(WORKSPACE_SCHEME) if settings else None
74+
if scheme and scheme.get("value"):
75+
return
76+
raise MissingWorkspaceError(resource_path, method)
77+
78+
79+
__all__ = [
80+
"WORKSPACE_HEADER",
81+
"WORKSPACE_SCHEME",
82+
"MissingWorkspaceError",
83+
"ensure_workspace",
84+
]

‎hotdata/api_client.py‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -29,6 +29,7 @@
2929

3030
from hotdata.configuration import Configuration
3131
from hotdata.api_response import ApiResponse, T as ApiResponseT
32+
from hotdata._workspace import ensure_workspace
3233
from hotdata._useragent import USER_AGENT
3334
import hotdata.models
3435
from hotdata import rest
@@ -637,6 +638,19 @@ def update_params_for_auth(
637638
if not auth_settings:
638639
return
639640

641+
# Hand-applied (scripts/patch_workspace_guard.py): a workspace-scoped
642+
# operation with no workspace id configured would otherwise go out
643+
# without X-Workspace-Id and come back as a bare 404. Fail here with
644+
# an error that says what is missing and how to supply it.
645+
ensure_workspace(
646+
self.configuration,
647+
auth_settings,
648+
headers,
649+
resource_path,
650+
method,
651+
request_auth=request_auth,
652+
)
653+
640654
if request_auth:
641655
self._apply_auth_params(
642656
headers,

‎hotdata/configuration.py‎

Lines changed: 14 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -16,6 +16,7 @@
1616
import logging
1717
from logging import FileHandler
1818
import multiprocessing
19+
import os
1920
import sys
2021
from importlib.metadata import PackageNotFoundError, version as _pkg_version
2122
from typing import Any, ClassVar, Dict, List, Literal, Optional, TypedDict, Union
@@ -135,13 +136,16 @@ class HostSetting(TypedDict):
135136
class Configuration:
136137
"""This class contains various settings of the API client.
137138
138-
:param host: Base url.
139+
:param host: Base url. Defaults to the `HOTDATA_API_URL` environment
140+
variable when set, else `https://api.hotdata.dev`.
139141
:param ignore_operation_servers
140142
Boolean to ignore operation servers for the API client.
141143
Config will use `host` as the base url regardless of the operation servers.
142144
:param api_key: Hotdata API key, sent as `Authorization: Bearer <key>`.
145+
Defaults to the `HOTDATA_API_KEY` environment variable when not given.
143146
:param workspace_id: Public id of the target workspace, sent as
144-
`X-Workspace-Id`.
147+
`X-Workspace-Id`. Defaults to the `HOTDATA_WORKSPACE` environment
148+
variable when not given.
145149
:param api_keys: Escape hatch for raw apiKey-scheme values, keyed by
146150
security scheme name. Prefer `workspace_id`.
147151
:param api_key_prefix: Dict to store API prefix (e.g. Bearer).
@@ -203,6 +207,14 @@ def __init__(
203207
) -> None:
204208
"""Constructor
205209
"""
210+
# Environment defaults, matching the Hotdata CLI: an explicit argument
211+
# always wins; an unset or empty variable leaves the default in place.
212+
if api_key is None:
213+
api_key = os.environ.get("HOTDATA_API_KEY") or None
214+
if workspace_id is None:
215+
workspace_id = os.environ.get("HOTDATA_WORKSPACE") or None
216+
if host is None:
217+
host = os.environ.get("HOTDATA_API_URL") or None
206218
self._base_path = "https://api.hotdata.dev" if host is None else host
207219
"""Default Base url
208220
"""

0 commit comments

Comments
 (0)