Repository navigation
pineforge-feed: OKX, Bybit and Binance USD-M venues (slice 2) - #18
Merged
Merged
Conversation
The mock's client-frame parser fell through from the 16-bit extended length to the 64-bit branch, so a client frame of exactly 127 bytes was read with a garbage 64-bit length. Client frames used to be short PONGs; subscription frames for OKX trades-all plus candles are 127 bytes. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Venue gates, checked at startup before any network access: - OKX spot and linear swap: confirmed candle1m bars (confirm "1"), and raw ticks from trades-all only. A print carrying `count` or a push on the aggregated trades channel stops with 23. Swap sizes and candle volumes convert exactly to base units with ctVal x ctMult from the public instrument (decimal multiply, never float); inverse swaps are refused. Holes heal by trade ID through history-trades pages of 100, within the 3-month window. Text ping keepalive; notice reconnects. - Bybit spot and linear: confirmed kline.1 bars only. Tick modes are refused: trade IDs are not a contiguous cursor and REST cannot page back. Two-entry pushes, and newest-first REST rows requested in windows no wider than the limit. - Binance USD-M: confirmed bars on the routed /market stream, and an opt-in agg-ticks mode that maps each aggregate `a` to one print with quantity `q` (it includes RPI executions, like the kline volume), never expanding f..l. Heals through aggTrades fromId within 48 hours; -4166 and older gaps stop with 20. The exchangeInfo document exceeds the parser bound, so only its rateLimits member is extracted. - Coinbase is refused as deferred. OKX and USD-M ticks close a minute on a next-print fence: the contiguous ID chain from the REST-proven predecessor reaches a print of a later minute, and the minute's confirmed candle (WebSocket, or REST behind a later confirmed WebSocket candle) equals its prints exactly in OHLC and volume. A quiet minute closes on the fence with a zero-volume candle. The pump now takes a venue connection (path, subscription frames, text keepalive, frame classifier); keepalive replies never count as data. REST is allowlisted and paced per venue. Sessions move to session.cpp; the Binance spot proof is unchanged. Overlap and revision checks compare exact decimal values rather than lexemes (OKX renders one price as 84850.0 on its WebSocket and 84850 over REST), and a disagreement with REST must survive spaced re-reads before it stops with 21, because a just-closed minute's REST row can trail the WebSocket close. The cursor and warmup manifest bind the quantity multiplier. The public E2E harness takes --venue/--market/--symbol/--modes and verifies against each venue's REST in normalized units; R-B2 applies to every tick mode. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A just-closed minute's REST row can trail the WebSocket close, as missing, unconfirmed (OKX) or with older values. The feed already re-read a disagreeing overlap; it now also re-reads when the newest row is missing or unconfirmed, before stopping with 20. Live, an OKX warmup can ask for its newest candle a moment after the WebSocket confirm. The qualification harness compares the tape with REST right after the last minute closes, so its bar, print and candle checks now pass only when REST agrees within spaced re-reads (about 25 s); a persistent disagreement still fails. The mock suite's CTest timeout rises to 420 s: under TSan it takes about 140 s on the test machine, and hosted runners are slower. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Review fixes for the venue slice: - OKX's time lookup for the last print before --start can answer an older print (REST trailing the newest prints, or prints of one millisecond out of ID order). The feed now walks forward by trade ID until the next print is at or after the start and anchors only then; until that print is visible it stops with 20 and saves nothing, so a restart cannot loop on a wrong predecessor. - USD-M looks for the first aggregate at or after the start within the next hour, so a quiet start minute has a fence. - A print at or before the anchored predecessor but at or after the start stops with 23 (time regression) or 21 (changed predecessor), not with a misleading buffered-cursor 21; an empty predecessor overlap is 20, not 21. - OKX 50001/50004/50013/50026 and Bybit 10000/10016 are retried with backoff like HTTP 5xx instead of stopping with 23. - A WebSocket the venue accepts but closes before any frame now spends the reconnect budget; loopback test origins use 50 ms backoff units. - Healing at the newest prints re-reads an empty REST page for about 7 seconds, like the reconnect overlap. - make_session refuses a tick mode that the venue cannot prove. Tests: resume rebuilds the open fenced minute; a changed contract multiplier stops 21 on resume; the predecessor walk and its unanchored 20; retried venue codes; USD-M -1121; a quiet USD-M start minute; the reconnect budget; the warmup REST lag now starts from the WebSocket script, so sanitizer timing cannot defeat it. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
OKX and USD-M close a minute without prints on the next-print fence when its candle shows zero volume. The harness now accepts such a minute only with a zero-volume REST candle, and the batch sees that flat candle. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The final live round stopped USD-M agg-ticks with 21: every aggregate of the minute equalled REST aggTrades, but Binance's own kline was 0.003 BTC (one fill) short, and the next minute's kline was 0.003 long with a different OHLC. An aggregate is dated by its trade time and never split, so one can hold a fill that the next kline counts: the kline is not the exact sum of aggregate prints, and comparing them is not a completeness check. USD-M aggregate minutes now close on the next-print fence (the contiguous aggregate chain reaches a later minute) once the venue has closed the minute; OKX raw prints still require exact candle equality. The harness verifies every aggregate against REST and reports, rather than requires, kline differences for USD-M. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
USD-M aggregate minutes are no longer compared with the kline, so the reason for `q` over `nq` is that `nq` drops RPI executions from the print stream. Comment and README wording only. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Two unit fixtures and two slice-1 mock action records carried values copied from live market data. Replace them with hand-written numbers; no behaviour depends on the values. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…lier A fence session with nothing anchored no longer stops 20 when a later minute closes before any print after --start exists: forward() asks the venue for a predecessor only once a print at or after the start is visible, and otherwise waits. Nothing is anchored or emitted before that proof. An empty look-ahead page is read once instead of four times over 7 s. Every connection re-reads OKX instrument metadata; a changed ctVal x ctMult stops 21, because quantities already emitted used the old multiplier. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Bybit answers 403 for its roughly ten-minute IP rate ban. A per-venue RestPolicy rejection hook now runs before the generic access-denied check and maps that 403 to 20, so a supervised restart recovers; other venues keep 403 as a permanent 23. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Unit and mock cases kill the review's surviving mutants: a forming candle never closes a fenced minute; a conflicting duplicate print stops 21, emitted or buffered; a changed closed candle or start predecessor on reconnect, and a candle revised before its minute closes, stop 21; the unconfirmed-print buffer is bounded; the reconnect print overlap and a heal page re-read trailing REST; a non-unit ID inside an OKX or USD-M REST page stops 20. The Bybit confirm-flag scenario now gives the forming rows other values and asserts the emitted bars. Also covered: a quiet start waits, a Bybit 403 stops 20, and a changed OKX multiplier stops 21 on reconnect. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
For agg-ticks the harness also runs the strategy over Binance klines for the proven minutes and prints, as information next to kline_differs, how many actions differ from the runner. The gate stays the batch over the bars built from the feed's own prints. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The README states where agg-ticks is chosen that its bars can differ from Binance klines at minute edges, with the measured rate and a worked fill, that --mode bars gives kline-exact bars, and that the matching backtest runs on prints-built bars. It also documents that output tokens come from the delivering source, what the tick batch replays, the quiet-start wait, Bybit's 403, the OKX multiplier re-check and value-based overlaps. --help carries the agg-ticks note. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The scenario's REST gained its first print after 0.5 s, inside the 1.5 s an OKX predecessor lookup waits, so it also passed for a feed that anchored blindly. Three seconds makes that stop 20 and the scenario fail. Also shortens a comment line that ruff flags. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds the remaining approved venues to the native
pineforge-feedadapter innative/feed/. Public market data only: no API keys, orders, accounts or fills.candle1mrowstrades-all,tradeIdstrictly +1, healed byhistory-tradespages within the 3-month window; swap quantities converted exactly fromctValxctMult(decimal, never float); inverse swaps refusedkline.1/marketstream, closed klinex=trueagg-ticks: one print per aggregatea(never expanded), healed byaggTradesfromIdwithin the 48-hour retention, stopping beyond itCompleteness: the next-print fence
A
timeevent for minute M is emitted only when all of these hold:OKX raw prints must also equal the candle exactly. USD-M aggregates are dated by trade time and never split, so a fill can sit in a different minute than Binance's kline. Agg-ticks minutes therefore close on the fence without kline equality. The README states the consequence at the point of choice: over 120 consecutive minutes, 5% differed by one tick in the open and by the moved fill in volume, while close, high and low never differed.
--mode barsgives kline-exact bars.Unchanged from slice 1
Every slice-1 guarantee carries over:
--resume/--output-from;Changes in this slice
Verification
uv run pytest -qandruff check .pass; the Python package and wheel are unchanged.🤖 Generated with Claude Code