Repository navigation
Conversation
BaseClient._get_params turns a method's locals() into query parameters, skipping only `params` and `raw`. It did not skip `options`, so any REST method called with a RequestOptionBuilder also sent `options=<massive.rest.models.request.RequestOptionBuilder object at ...>` in the query string. Skip `options` too. Every endpoint builds its query through _get_params, so this fixes them all. Add regression tests for a GET and a paginated list call.
BaseTest turned pook on and registered its mocks in setUpClass but never turned pook off, so every class's mocks stayed active for the rest of the run. That hid a problem in two tests. test_list_stocks_dividends_ticker_any_of and test_get_snapshot_indices each send the same request twice, but each mock answers only once. Their second request was answered by a copy that an earlier class had registered, so both fail with PookNoMatches when their module runs on its own. Turn pook off in tearDownClass. Let a test class name the mocks that must answer more than once, and register those with persist. Other mocks still answer once, because the pagination tests rely on it: a first-page mock also matches the request for the next page.
The package supports Python 3.10 and later, but CI ran the unit tests only on 3.10 and 3.11. Add 3.12, 3.13 and 3.14 to the matrix, on both Ubuntu and Windows.
_parse_subscription called logger.warning("invalid subscription:", s),
passing the topic as a format argument with no placeholder. Logging
printed "--- Logging error ---" and a traceback instead of the warning,
and the rejected topic was never shown.
Use a %s placeholder so the warning reads "invalid subscription: <topic>",
and add a test to test_status.py that goes through subscribe().
@modelclass builds an __init__ that accepts every field positionally or by keyword, but type checkers could not see it: a call such as CryptoQuote(pair="BTC-USD", bid_price=1.5) was reported as "Unexpected keyword argument" for every field. Mark modelclass with dataclass_transform (PEP 681) so checkers derive the constructor from the field annotations. The decorator is imported only under TYPE_CHECKING, from typing on 3.11+ and from typing_extensions on 3.10, and is a no-op at runtime, so there is no new runtime dependency and Python 3.10 keeps working.
Several hints describe values the API never sends or returns:
- CryptoQuote.bid_price, ask_price and ask_size were int; XQ sends
fractional numbers ("bp": 33052.79, "as": 0.601). They are float.
- IndexValue.timestamp was str; V sends integer Unix milliseconds.
- list_tickers(cusip=...) accepted only int. The API treats a CUSIP as a
string: it is alphanumeric, and leading zeros matter ("037833100").
The hint now accepts str as well as int, so existing callers keep
type-checking, and the docstring recommends a string.
- get_previous_close_agg() was annotated as returning one
PreviousCloseAgg, but it returns the "results" array as a list.
- list_treasury_yields(), list_short_interest(), list_short_volume() and
list_ipos() were annotated as returning a List, but like every other
list_* method they return a paginating iterator.
Hints that are imprecise but still admit every value the API sends, such
as integer timestamps hinted as float, are left alone: narrowing them
would gain little and could break callers that type-check today.
Only annotations and docstrings change; runtime behaviour is the same.
The tests pin the runtime types the corrected REST hints now describe.
BaseClient._get raised BadResponse(resp.data.decode("utf-8")) for any
non-200 response, so the status code was lost. Callers had to guess from
the body whether the key was not entitled (403) or the resource was
missing (404).
BadResponse now takes a keyword-only status argument and stores it as an
attribute, and _get passes resp.status. Positional arguments still go to
Exception unchanged, so str(exc) and exc.args are the same as before, and
BadResponse() still works (status is None).
exceptions.py gets an __all__, so the package's star import does not
re-export typing.Optional as massive.Optional.
The client is used from multi-threaded programs, and the GIL can hide race conditions. Add one job that runs the unit tests on free-threaded 3.14t with PYTHON_GIL=0, after checking that the GIL really is off. The job installs with pip, because the tests need only the package and pook. Add a test that shares one RESTClient between 16 threads against a local server that echoes each request back. It checks that every response answers its own request, including the per-request edge headers, that the threads really overlapped, and that per-request headers never reach the client's default headers.
@modelclass builds an __init__ that accepts every field positionally or by keyword, but type checkers could not see it: a call such as CryptoQuote(pair="BTC-USD", bid_price=1.5) was reported as "Unexpected keyword argument" for every field. Mark modelclass with dataclass_transform (PEP 681) so checkers derive the constructor from the field annotations. The decorator is imported only under TYPE_CHECKING, from typing on 3.11+ and from typing_extensions on 3.10, and is a no-op at runtime, so there is no new runtime dependency and Python 3.10 keeps working.
Subscribing to a wildcard such as A.* drops the other subscribed topics
on that channel. The match used t.startswith(channel), so it also
dropped topics on every channel whose name starts with the same
letters: subscribe("A.*") unsubscribed AM.AAPL. It also matched the
wildcard itself, so subscribing to A.* a second time unsubscribed it.
Compare the channel exactly, and skip the wildcard being subscribed.
unsubscribe() used the same match and gets the same fix. As before,
only topics already sent are checked.
Without the PEP 561 marker, mypy skips the installed package and reports
import-untyped, and users who silence that get Any for every massive type.
- Add massive/py.typed. The existing packages entry in pyproject.toml
already includes it in the wheel and sdist.
- Re-export RESTClient and WebSocketClient explicitly ("X as X"), so
"from massive import RESTClient" passes mypy --strict, which does not
treat plain imports in a typed package as re-exports. No __all__ is
added, so "from massive import *" exports the same names as before.
- Add a test that "from massive import *" still provides the main names.
The Fed funding-conditions endpoint is documented in the OpenAPI spec and on massive.com/docs but had no client method; the economy client only covered treasury-yields, inflation, inflation-expectations, and labor-market. Add EconomyClient.list_funding_conditions, a paginated list method in the same shape as list_treasury_yields. Its query parameters follow the documented ones exactly: date, date.gt, date.gte, date.lt, date.lte, limit, and sort (this endpoint documents no date.any_of or order). sort is typed Optional[str], as in list_labor_market_indicators, rather than Union[str, Sort]. The endpoint takes column.direction values such as "date.desc", so Sort.ASC would send a bare "asc" that it does not accept. Add the FedFundingConditions model with all 22 documented result fields (date plus the fed funds, IORB, SOFR, OBFR, TGCR, repo/reverse repo, and commercial paper series). Tests cover a two-page fixture (including a weekend row whose business-day series are null), check with pook that all seven parameters are sent under their API names (date.gt rather than date_gt, and so on), and check that a multi-column sort string is sent unchanged. Add an example script.
…}/{to}/{date}
The crypto daily open/close endpoint is documented in the OpenAPI spec
and on massive.com/docs but had no client method. The docs and the
crypto example pointed at get_daily_open_close_agg, which calls
/v1/open-close/{ticker}/{date}, a different endpoint whose response
(afterHours, preMarket, high, low, volume) does not match the crypto
one.
Add AggsClient.get_daily_open_close_crypto(from_, to, date, adjusted)
and two models matching the documented response:
- DailyOpenCloseCrypto: symbol, is_utc (isUTC), day, open, close,
open_trades (openTrades), closing_trades (closingTrades).
- DailyOpenCloseCryptoTrade: conditions (c), id (i), price (p),
size (s), timestamp (t), exchange (x).
Test against a fixture built from the documented example response.
Give the method its own section in the Aggs docs, move the crypto
docs link there, and point the crypto example at the new method.
subscribe() and unsubscribe() only change scheduled_subs, the topics the caller wants. The receive loop sends the change on its next turn, up to a second later, and then copies scheduled_subs into subs, the topics the connection has been sent. When the connection closed with an error, connect() set scheduled_subs back to subs before reconnecting. That line came with "resub on reconnect" (massive-com#207), and it does resubscribe the topics that were sent. But a change made in the second before the drop had not been sent yet, so it was thrown away: a topic subscribed just before the drop was missing on the new connection, and a topic unsubscribed came back. Keep scheduled_subs and only reset subs, so the new connection subscribes to every topic the caller wants. connect() also kept subs from an earlier call. Calling it again after it returned, for example after the server closed the connection cleanly, sent no subscribe message at all, because every topic looked sent already. Reset subs at the start of every connect() call. This is one way to end up with the missing subscriptions reported in massive-com#800. The mock server now records the messages each connection receives, and fails the connection on a DROP.X subscription. The shared test setup waits for the server to start, as test_status did.
subscribe(), unsubscribe() and unsubscribe_all() are called while connect() runs, often from another thread, but they changed scheduled_subs and schedule_resub with no lock. The receive loop read scheduled_subs, awaited the subscribe and unsubscribe sends, and then copied scheduled_subs into subs and cleared schedule_resub. A change made during either await was copied into subs without being sent, and its schedule_resub was cleared, so it was never sent. The server then streamed different topics from the ones requested, while subs said everything was sent. Guard both attributes with a lock, which the three methods take for their whole body. The receive loop now copies scheduled_subs and clears schedule_resub under the lock, sends the difference from that copy, and records the copy in subs. A change made during the sends sets schedule_resub again and goes out on the next turn. Code that changes scheduled_subs itself and sets schedule_resub keeps working: the loop still checks the flag on every turn.
connect() sent subscription changes only when its receive loop came round again, after the next message or the one-second recv timeout. On a quiet feed, a subscribe() took up to a second to reach the server. Each connection now has one task that sends subscription changes. It starts after authentication by sending the whole scheduled set, then waits on an asyncio.Event. subscribe(), unsubscribe() and unsubscribe_all() set the event through call_soon_threadsafe(), which never blocks the caller. Being the only sender, the task never interleaves messages, and it merges a burst of changes into one subscribe and one unsubscribe message. The receive loop still checks schedule_resub on every turn, now to wake the task, so code that changes scheduled_subs itself and sets the flag keeps working. An error other than a closed connection ends the task and is raised by connect(), as before. The task is stopped when the connection ends. The receive path is otherwise unchanged. Sending while another coroutine waits in recv() on the same connection is safe in websockets 14.0 through 17.3, the current release (websockets/asyncio/connection.py). recv() only reads from the message assembler (14.0 L270, 14.2 L294, 15.0.1 L303, 16.1.1 L305, 17.3 L298), whose only concurrency check rejects a second concurrent reader (asyncio/messages.py, Assembler.get: 14.0 L142, 14.2 L141, 15.0.1 L150, 16.1.1 L149, 17.3 L148). send() waits only for a fragmented send in progress (fragmented_send_waiter, named send_in_progress in 16.1.1 and 17.3: 14.0 L427, 14.2 L451, 15.0.1 L469, 16.1.1 L472, 17.3 L466) before writing through send_context() (14.0 L434, 14.2 L458, 15.0.1 L476, 16.1.1 L479, 17.3 L473).
Two things a long-running client needs were not possible: - Replacing one set of topics with another, removals first. The server enforces a per-connection subscription limit (it answers "Subscription limit reached for feed", which the client logs since massive-com#1039). The client always sends the subscribe message before the unsubscribe message, so a swap made at the limit asks for more topics than the limit allows before the unsubscribe arrives. - Sending a subscribe message again for a topic the server never acknowledged. subscribe() sends only topics missing from subs, so a topic already sent can never be sent twice. set_subscriptions(desired) replaces the scheduled topics in one step. The next update then sends one unsubscribe message for the removed topics, followed by one subscribe message for the added ones. resubscribe(*topics) adds topics to the next subscribe message. A topic that is neither subscribed nor scheduled raises ValueError, and then nothing is sent. subscribe(), unsubscribe() and unsubscribe_all() keep their order: subscribe first, then unsubscribe.
connect() reconnects at once after a dropped connection, returns when the server closes the connection normally, retries a handshake error such as HTTP 404 forever, and counts every disconnect over the client's lifetime toward max_reconnects. A caller cannot change any of this. Add keyword arguments to WebSocketClient. Each defaults to today's behavior: - reconnect_backoff, max_reconnect_backoff: wait before reconnecting after an open connection closes. The wait doubles with each failure in a row, up to the cap, plus up to 20% jitter; at the cap it is drawn from [cap / 1.2, cap], never below reconnect_backoff. The constructor raises ValueError if reconnect_backoff is negative or greater than max_reconnect_backoff. - reconnect_on_close: also reconnect when the server closes normally. A close() by the caller while connected still ends connect(). - max_consecutive_failures, stable_uptime: limit failures in a row. A failure is an attempt that fails, or a connection that closes before it has been open for stable_uptime seconds. max_reconnects keeps its meaning. - raise_fatal_errors: use websockets' default process_exception, which raises errors that retrying will not fix. The reconnect loop is still async for ... in websockets' connect(). connect() now also accepts process_exception, which used to raise TypeError because the client passed its own. The new attribute max_connections_reached tells whether the server sent a max_connections status during the last handshake; the processor receives the same messages as before.
close() is a coroutine that must run on connect()'s event loop. run() creates that loop itself, so another thread cannot stop a client started with run(): asyncio.run(client.close()) from another thread fails with "attached to a different loop". request_close() can be called from any thread and does not wait. It sets a flag and, while connect() runs, closes the connection through loop.call_soon_threadsafe(). connect() checks the flag before it reconnects and after a wait before reconnecting, so it returns instead, also with reconnect_on_close. An attempt to connect, or a wait before reconnecting, is not interrupted: connect() returns once it ends, without opening another connection. If connect() is not running, the next call returns at once. close() called while connect() runs now sets the same flag. Before, a close() during the wait before a reconnect (reconnect_backoff > 0) or during a reconnect handshake found no open connection to close, and connect() went on to reconnect. A close() before connect() starts is unchanged: the next connect() proceeds. With the default reconnect_backoff of 0 there is no wait, so the only difference is a close() made during a reconnect handshake.
The client retries 413, 429, 499, 500, 502, 503 and 504. When the last retry still gets one of those statuses, urllib3 raises MaxRetryError, so code that catches BadResponse misses rate limits and server errors. Add a raise_on_status argument, passed to the client's Retry. It defaults to True, which keeps today's MaxRetryError. With raise_on_status=False, urllib3 returns the last response and _get raises BadResponse with its status and body. The number of attempts and the backoff do not change. A custom Retry can already be passed through connection_pool_kw, but it replaces the client's status list and backoff, and makes the retries argument a no-op.
Every REST method is annotated as returning Union[Model, HTTPResponse]
(or Union[Iterator[Model], HTTPResponse]), because raw=True returns the
undecoded urllib3 response. A type checker cannot rule out HTTPResponse,
so ordinary code fails:
for t in client.list_tickers():
t.ticker # "HTTPResponse" has no attribute "ticker"
client.get_aggs("AAPL", 1, "day", "2024-01-02", "2024-01-31")[0] # not indexable
Add three overloads to each of the 94 methods that take raw:
- raw: Literal[False] = False returns the model type, so the default
call is precise.
- *, raw: Literal[True] returns HTTPResponse. raw is keyword-only here
because it follows parameters with defaults; giving it a default would
overlap the first overload, which pyright reports.
- raw: bool = False returns the full union, so a non-literal flag, or
raw passed positionally, type-checks exactly as before.
This commit is mechanical: the overloads were generated from each
method's own signature and then formatted with black. They differ from
the signature only in raw and the return type. The implementations, and
the signatures that _get_params inspects at runtime, are unchanged.
Add test_rest/typing_consumer.py, ordinary user code that is checked with mypy --strict but never run. It iterates list_* results, indexes get_* lists, checks that raw=True gives HTTPResponse and a plain bool gives the union, builds models with keyword arguments, pins each corrected hint with assert_type, and keeps code written for earlier releases (isinstance narrowing, an int CUSIP) to show it still checks. make static checks it from inside test_rest/, so massive resolves to the installed package as it does for users. assert_type comes from typing on 3.11+ and from typing_extensions on 3.10 only under mypy, which bundles its stubs; the file is never imported, so nothing new is needed at runtime. AGENTS.md now describes the overload pattern for new endpoints.
|
Important Draft PR not reviewedDraft PRs are not automatically reviewed by default.
To automatically review draft PRs, update your CodeRabbit configuration: reviews:
auto_review:
drafts: true
Comment |
This was referenced Oct 24, 2026
Owner
Author
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Integration check only. Do not merge. This PR exists so that CI runs on the merge of every preview PR.
The branch is
origin/masterplus--no-ffmerges of #4, #7, #8, #9, #10, #11, #12, #13, #14, #15, #16, #17, #18, #19, #20, #21, #22, #23 and #24, in that order, at their heads after the review fixes. Every merge is clean.deephaven-fixeswill carry this exact tree. This PR supersedes #25.