Skip to content

Live runner: normalized feed input and per-action webhook routing - #320

Merged
luisleo526 merged 29 commits into
mainfrom
lv/runner-routing
Oct 3, 2026
Merged

luisleo526 merged 29 commits into
mainfrom
lv/runner-routing

Conversation

@luisleo526

@luisleo526 luisleo526 commented Oct 3, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

The native live runner pineforge-live reads only the normalized PineForge feed protocol and routes each order action to a webhook target. The engine computes. It never connects to an exchange or ingests fills; venue execution, reconciliation and risk belong to the consumer, and runner/README.md now says so in a short boundary section.

  • Feed input: the normalized protocol only.
    • Tick mode: contiguous seq plus the time completeness event.
    • Bar mode: confirmed 1-minute bars.
    • Sources: stdin, a file, or the user's own normalized feed over HTTP or WebSocket.
    • The in-runner exchange parser plugins are removed: --parser / --parser-config, include/pineforge/live_parser.h, the demo parser, its tests and the CI wiring. Translating raw exchange messages belongs in a separate feed adapter. A raw exchange message is refused with guidance. README tables give the field types; an optional trade_count on bars is accepted.
  • Reliable WebSocket input. libcurl 8.14.1 or newer is required, and an unfinished message is never committed.
  • Per-action webhook routing.
    • --webhook-routes sends each action to one target. Rules match on order id, kind and side, first match wins, with a default target and a per-target HMAC.
    • A target can be null (journal only), and a runner can run without any webhook: pineforge-live actions --follow reads the journal.
    • Delivery works like an alert. Each action is sent once, in order. An error is shown and never blocks later actions, with a bounded in-flight cap and transport-only retries. Every attempt is an append-only delivery-log row. Offline redeliver --deployment re-sends failed actions, and status reports per target.
    • Without routes, the v1 payload and the event-id idempotency key are unchanged.
  • CI: one pinned libcurl. The kernel verification profile deliberately builds the live runner against the source-free kernel library (docs/ci.md: "live runner enabled, Pine source layer OFF"), WebSocket rows included. Its CI job now uses the same pinned libcurl 8.14.1 as native-live, from one shared definition. The 8.14.1 floor is not lowered.
  • Sanitizer verification of the runner: ASan/UBSan and ThreadSanitizer profiles.

Breaking changes (CHANGELOG)

  • The parser plugin interface (live_parser.h, --parser) is removed. Migration: run an external feed adapter that emits PineForge feed events.
  • Delivery changes for existing --webhook-url users:
    • HTTP error responses are final;
    • the exit status no longer reflects failed deliveries;
    • --max-attempts now caps transport retries;
    • the default timeouts are 2 s to connect and 5 s in total;
    • --webhook-url is optional.
      The ledger migrates one way.

Verification

  • Verification bundles on the head tree: preflight, release, sanitizers, docs, debug, kernel; corpus parity 312/312. Posted as pineforge/verify.
  • Full-population parity sweep, expected neutral because the engine library is unchanged. Posted as pineforge/parity.
  • Native profile with WebSockets 798/798, live ASan/UBSan 13/13, live ThreadSanitizer 13/13, and the routing and recovery E2Es 13/13, including SIGKILL recovery, receiver isolation and the migration of an older ledger.
  • Two independent reviews; all findings fixed or recorded.

🤖 Generated with Claude Code

assert len(main_receiver.rows) == 4
print('PASS SIGKILL between commit/send and mid-request resumes each unsent action once after replay', flush=True)

deployment = query(crash_ledger, 'SELECT identity FROM metadata')[0][0]
self.send_response(status)
self.send_header('Content-Length', '0')
self.end_headers()
except (BrokenPipeError, ConnectionResetError):
self.connection.settimeout(20)
while self.connection.recv(4096):
pass
except OSError:
Remove the parser-plugin advertisement introduced by 5f3299d now that normalized feed adapters define the runner boundary.
@luisleo526
luisleo526 merged commit dbd17b3 into main Oct 3, 2026
17 checks passed
@luisleo526
luisleo526 deleted the lv/runner-routing branch October 3, 2026 22:13
luisleo526 added a commit that referenced this pull request Oct 4, 2026
The 1.1.0 lead now says how state-hash values change while the epoch and
the domain tags stay: a run whose fills an adapter change alters hashes
to new values, and three changes fold more state, so a run can hash
differently from 1.0.1 with byte-identical trades: #315 (an unbatched
same-bar entry request and placement; the stream report's terminal
re-mark leaves the hashed extremes; witness row Stream/0/1 re-pinned),
#316 (an opening stop's next waypoint; Random44/0/0 and /1 re-pinned)
and #319 (tick-volume state, every existing pin kept).

In the runner tooling bullet, "No versioned engine PF_API export ...
changes" and "Plugin-free ledger identity bytes remain unchanged" read
as release-wide; they now speak for #320 itself and for a strategy
without the checked settings calls. The Report keys intro points at the
report schema page. The terminal-quote bullet names the chart feed that
PINEFORGE_RUN_REPORT_CHART_QUOTE points at, and two runner bullet
headings are re-wrapped.

Replaces text introduced by a80485c (this branch's first commit): the
lead's "A run whose fills one of the Pine adapter changes below alters
hashes to new values, and [#316] adds one placement field to the
adapter's hashed state, for which two witnesses with byte-identical
trades were re-pinned.", the Report keys intro and the terminal-quote
harness clause; and text introduced by dbd17b3 (#320): "No versioned
engine PF_API export, native C++ surface, script ABI epoch or engine
behavior changes." and "Plugin-free ledger identity bytes remain
unchanged".

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
luisleo526 added a commit that referenced this pull request Oct 4, 2026
* docs: release notes and version scoping for v1.1.0

v1.1.0 is a minor release: <pineforge/pineforge.h> declares six more
functions, the checked settings calls that a strategy generated by
pineforge-codegen 1.1.0 exports (#317). It pairs with pineforge-codegen
1.1.0. PF_ABI_VERSION stays 4 (pineforge.h only gains declarations;
native_c_api.h and the native C++ host headers are unchanged) and the
script ABI epoch stays engine_script_run_v19.

- CHANGELOG.md: "## Unreleased" becomes "## 1.1.0 — 2026-10-04": a lead
  paragraph, one bullet per change a user can observe since v1.0.1
  (#312-#324), each linking its PR, a Report keys subsection
  (docker/run_json.py v1.0.1 -> v1.1.0) and a Migration subsection. The
  three Unreleased bullets stay; the lot-grid bullet now gives both
  failure messages.
- README.md: pineforge-codegen==1.1.0, the v1.1.0 tarballs, the pairing
  sentence, the stream known issue's scope, the benchmark refresh line,
  the release scoreboard sentence (releases[1.1.0] markers) and a v1.1.0
  line in Releases. The public corpus sweep's "311 excellent + 1 declared
  anomaly" is scoped to the v1.0.1 library it was measured on. Facts
  rendered from the facts file that adds releases[1.1.0]; the active
  scoreboard markers move to baseline
  pineforge-parity-baseline-20261003-engine-dbd17b38.
- docs/pages/install.md: the v1.1.0 tarballs and the hub tag
  engine1.1.0-codegen1.1.0.
- docs/pages/public-contract.md: 1.1.0's release line, engine v1.1.0 with
  codegen 1.1.0 among the supported pairs, and the four C-boundary rows
  that were planned for 1.1.0 are not in it.
- docs/pages/streaming.md: the known issue's scope includes v1.1.0.

Lines that state what an earlier version shipped or did stay.

Replaces text introduced by dc74e63 (#318): README.md's "Release 1.0.1
still grades ... until the next release" sentence, now 1.1.0's; by
d093560 (#290): public-contract.md's "Four of its rows are planned for
1.1.0"; by 783bb39 (#322): CHANGELOG.md's "The engine library is
unchanged."; by dbd17b3 (#320): CHANGELOG.md's "The engine is
unchanged."; by b2a578c (#300): README.md's "this repository's sweep:
311 excellent + 1 declared anomaly"; by 271d687 (#311): README.md's
"its releases 1.0.0 and 1.0.1 do".

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* docs: state the 1.1.0 hash folds and scope two runner sentences

The 1.1.0 lead now says how state-hash values change while the epoch and
the domain tags stay: a run whose fills an adapter change alters hashes
to new values, and three changes fold more state, so a run can hash
differently from 1.0.1 with byte-identical trades: #315 (an unbatched
same-bar entry request and placement; the stream report's terminal
re-mark leaves the hashed extremes; witness row Stream/0/1 re-pinned),
#316 (an opening stop's next waypoint; Random44/0/0 and /1 re-pinned)
and #319 (tick-volume state, every existing pin kept).

In the runner tooling bullet, "No versioned engine PF_API export ...
changes" and "Plugin-free ledger identity bytes remain unchanged" read
as release-wide; they now speak for #320 itself and for a strategy
without the checked settings calls. The Report keys intro points at the
report schema page. The terminal-quote bullet names the chart feed that
PINEFORGE_RUN_REPORT_CHART_QUOTE points at, and two runner bullet
headings are re-wrapped.

Replaces text introduced by a80485c (this branch's first commit): the
lead's "A run whose fills one of the Pine adapter changes below alters
hashes to new values, and [#316] adds one placement field to the
adapter's hashed state, for which two witnesses with byte-identical
trades were re-pinned.", the Report keys intro and the terminal-quote
harness clause; and text introduced by dbd17b3 (#320): "No versioned
engine PF_API export, native C++ surface, script ABI epoch or engine
behavior changes." and "Plugin-free ledger identity bytes remain
unchanged".

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* docs: #315's range-end change folds less, not more

The 1.1.0 lead said three changes "fold more state"; #315's range-end
change instead stops folding a stream report's terminal re-mark into the
hashed extremes. The sentence now says the three changes change what the
hashes fold, and names Stream/0/1 as a re-pinned witness row without
claiming it is #315's only one.

Replaces text introduced by 66be240 (this branch's second commit): "Three
changes also fold more state" and "re-pinning one witness row
(`Stream/0/1`, whose recorded-row digest alone moves)".

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* docs: fold #325 and #327 into 1.1.0, add State hashes, amend the hash rule

#325 (confirmed-bar streams compute what the batch computes) and #327
(docker/run_json.py --symbol-feeds) merged before the release notes, so
their Unreleased bullets join the 1.1.0 section, rewritten and linked; no
Unreleased heading stays.

- CHANGELOG.md: the lead names the stream fix and, in the pairing
  sentence, codegen 1.1.0's request discovery (transpile_full()["requests"],
  codegen #164). #325's bullet says what a stream user sees; #327's bullet
  and its Report keys entry (applied_runtime.symbol_feeds, the
  --symbol-feeds: failure messages). A new State hashes section lists the
  four recipe additions since v1.0.1 (#315, #316, #319, #325; none removed,
  renamed or re-encoded; domain tags unchanged). #315's range-end
  equity-extreme change moves to the Pine adapter parity bullet as a
  behaviour fix. The #319 bullet named the broker-state hash; the change is
  in the native continuation hash, which the broker-state hash folds. The
  known-issue bullet now says it is fixed in 1.1.0 (#325). Migration gains
  a line on stream results and hashes.
- README.md: the known issue is fixed in 1.1.0 (#325); the v1.1.0 Releases
  line gains the stream fix, request discovery and the harness's
  multi-symbol request.security feeds. Fact markers rendered from the facts
  export that maps releases[1.1.0] to
  pineforge-parity-baseline-20261004-engine-7b596622 (7,951 excellent / 38
  strong of 7,989); the active scoreboard moves to the same baseline, here
  and in docs/pages/contributing-llm.md.
- docs/pages/streaming.md: the known issue affected v1.0.0 and v1.0.1 and
  is fixed in 1.1.0.
- docs/pages/public-contract.md: the state-hash rule, amended by the
  owner's ruling for 1.1.0. It covers the recipe only: a minor release may
  add a hashed field (domain tags unchanged, the added field changing no
  trade or report, disclosed under State hashes); removing, renaming or
  re-encoding a hashed field is a new epoch and a major release; behaviour
  changes follow the normal release and parity rules. The first uses are
  #315 and #316. The semver rule is unchanged.

Replaces text introduced by 7b59662 (#325): CHANGELOG.md's
"Confirmed-bar streams compute what the batch computes:" bullet, and the
"Fixed on main (#325); included in the next release." lines of README.md
and streaming.md; by 4f2a475 (#327): CHANGELOG.md's "Harness symbol
feeds:" bullet; by d093560 (#290): public-contract.md's "State-hash
values are stable within the epoch." and "a new recipe is a new epoch and
a major release"; by a80485c (this branch's first commit): the "Known
issue, as in v1.0.0 and v1.0.1" bullet, streaming.md's "(v1.0.0, v1.0.1,
v1.1.0)", the #319 bullet's "A tick stream's broker-state hash folds the
new accumulator state only where it adds information, so the existing
tick, bar and batch hash pins hold.", the Report keys failure-output
bullet and the v1.1.0 Releases line; by 66be240 and 83d64dc: the lead's
state-hash sentences ("Three changes also change what the hashes fold");
and, as git attributes them through this branch's merge of main, by
e59c09f: README.md's v1.1.0 Releases line and streaming.md's
"(v1.0.0, v1.0.1, v1.1.0)".

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant