Install other symbols' bars for request.security in the Docker harness (--symbol-feeds)

[luisleo526]

What

docker/run_json.py gains --symbol-feeds <index.json>, and docker/entrypoint.sh maps PINEFORGE_SYMBOL_FEEDS to it. The flag installs other symbols' bars for request.security on another symbol. Engine and codegen have supported such requests since 1.0.0 (engine #292, codegen #146/#148). Until now the release harness had no way to hand them the bars, so every such run whose request value reaches an order stopped with exit 4 (request.security(...) at line N: no data is pinned for this request, and its value was read).

{"symbols": {"BINANCE:ETHUSDT": {"syminfo": {...the symbol's catalog object...},
                                 "feeds": {"240": "ethusdt-240.csv", "1D": "ethusdt-1D.csv"}}}}

• Keys. A key is the exact string the script passes at run time, compared byte for byte by the engine (pine_security_eval.cpp:1031). BINANCE:ETHUSDT, ETHUSDT and BINANCE:ETHUSDT.P are three symbols. For input.symbol, the key is the input's value.
• Feeds. One CSV per requested timeframe, in the chart's CSV format, with paths relative to the index. Timeframes use the engine's spelling: whole minutes as a bare integer ("240", never "4h"), else <n>D|W|M|S, and bare D/W/M/S fold to 1D/1W/1M/1S.
  • A bar's close is its open plus the timeframe, or the optional time_close column.
  • Bars before the chart's first bar are warm-up history.
• Facts. From syminfo: type, timezone, session, currency and mintick are what syminfo.* reads inside the request. tickerid is set as the canonical fact, which no syminfo.* reads: inside the request syminfo.tickerid is always the key, and syminfo.ticker is the key after its last : (pine_security_eval.cpp:1089-1109).
• Checked before any strategy state exists: the index, the spelling, CSV columns and numbers, increasing opens, each close after its open and at or before the next open, ±(2^53−1) ms, a positive finite mintick, at most 256 symbols and 256 feeds. Then a library without the setters, or a feed the engine refuses. Every refusal is one {"engine":"pineforge","error":"--symbol-feeds: ..."} line (harness exit 1, entrypoint exit 4).
• Record. What was installed goes in applied_runtime.symbol_feeds: the facts, and per feed the bar count, first and last open, and a value hash under pf-symbol-feed-barc-close-le-v1. It is fingerprinted, so such a run has its own digest.
• Unset, or {"symbols": {}}: the report is byte-identical, apart from elapsed_seconds.

The engine library and the C ABI are unchanged; the harness calls the existing strategy_set_symbol_facts and strategy_set_symbol_feed. Docs: docker/README.md ("Other symbols' bars": field, keys, spelling, the missing-feed error, refusals, limits, record), the entrypoint header and CHANGELOG.md (Unreleased, the 1.1.0 entry). The release repo's own entrypoint line is pineforge-4pass/pineforge-release#23.

One feed per requested timeframe (the 1m question)

The app asked whether a feed may be 1m bars for a request at 240 or D, with the harness or engine aggregating it. Answer: no, not in 1.1.0. Each requested timeframe needs its own feed.

• No C call aggregates another symbol's bars, and none returns aggregated bars.
  • None of the 77 PF_API declarations in pineforge.h (62 implemented in the static runtime, the rest emitted per strategy), nor anything in native_c_api.h, takes raw bars and hands back aggregated ones.
  • strategy_set_symbol_feed copies the bars verbatim, keyed by (key, tf); the only change is folding D to 1D (pine_security_eval.cpp:1251-1294). The lookup is exact (:1031).
  • The kernel contract says the feed is "delivered bar for bar and aggregated by nothing" (native_run_spec.hpp:481-484, native_execution_consumer.cpp:8574-8600).
  • The one setter that aggregates, strategy_set_native_security_feed, works on the chart symbol's own daily feed (a W/M request reads it folded per week or month).
• Where the chart aggregation lives. The chart's input→script fold is NativeExecutionConsumer::contribute_input (native_execution_consumer.cpp:8340-8400), aligned to session and timezone by native_calendar::interval_containing. It is hidden C++. The only C-reachable route to it is a whole native-kernel run per feed through the native-host API (strategy_native_host_create_v1 / strategy_configure_native_ext_v1 / strategy_native_run_v1 with an on_bar callback). That means mirroring four structs in ctypes and running a broker kernel just to bucket bars, so it is not taken. Per the brief, there is no second aggregator in Python either.
• The engine refuses it for the chart too. Once a script requests another symbol, the chart must be its own input: input '1' aggregated to chart '240' is not supported (pine_security_eval.cpp:1045-1051). So an app that runs such a script already fetches the chart at the script timeframe, and can fetch the other symbols the same way.
• Proposed for a later release (about 100 lines of engine code, about 220 with tests and docs): PF_API int pf_aggregate_bars_v1(bars, n, input_tf, target_tf, timezone, session, out_bars, out_close_ms, capacity). It runs the chart's own fold through a private NativeStrategyHost, so its output matches the chart path by construction. The harness would then accept a "1" feed and aggregate it with that call. The acceptance test is ready: the case below fed 1m bars must give the same 12 rows. On this window Binance's 4h and 1d klines equal the UTC aggregation of its 1m klines (E2E data check), so the test is well posed.

E2E (AWS spot box, c6a.2xlarge, ap-southeast-1): PASSED on 6cc8247 with release #23 head 3475b0c

The rebase onto main 7b59662 (#325) gives head 673e95b and leaves docker/ byte-identical: git diff 6cc82475 673e95b8 -- docker/ is empty, and both heads have the same docker/ tree 9c6ccaa1. Only the CHANGELOG bullet moved, to follow #325's. So these results hold for this head.

Two images from pineforge-release's docker/Dockerfile (engine 1.0.1 static library + codegen 1.0.1, the latest releases). run_json.py is synced from the engine commit, as scripts/sync-harness.sh does from a tag:

• sf-pr: release PR head (its entrypoint) + this PR's run_json.py (sha256 afaa6952…, the same file as the prototype's final pass);
• sf-main: release main + engine main's run_json.py.

The shared XSYM case (BTCUSDT 4h chart; input.symbol("BINANCE:ETHUSDT") requested at timeframe.period and at D), with check_case.py:

run	result
feeds (ETHUSDT at 240 and 1D)	exit 0, 12 rows, equal to the recorded rows and to the independent reference model (TradingView's merge rule); applied_runtime.symbol_feeds hashes recomputed from the CSVs: CASE PASSED
no_feeds	exit 4, request.security(other, timeframe.period, ...) at line 7: no data is pinned for this request, and its value was read: PASSED
bare_key (keyed ETHUSDT)	exit 4, the line-7 error: PASSED
no_daily (240 only)	exit 4, the line-8 "D" error: PASSED
sol_input (input BINANCE:SOLUSDT)	exit 4, the line-7 error: PASSED
"4h" timeframe in the index	exit 4, one --symbol-feeds: a timeframe is whole minutes ... line: PASSED
sf-main with PINEFORGE_SYMBOL_FEEDS set	ignored, exit 4 line-7 error (red on main, for the right reason)

The qty-step case (#322) is unchanged on sf-pr: 22 rows with mincontract, 27 without, with null and with no syminfo (its check_case.py PASSED ×4).

Scripts that request no other symbol give reports byte-identical to sf-main (elapsed_seconds zeroed): the four qty-step runs (digests sha256:b9e00cb9… with the grid, sha256:e4315ec2… without), the no_feeds error line, and a qty-step run with {"symbols": {}} set. applied_runtime has no symbol_feeds key without the variable. The 101 harness tests pass on the box (Linux).

The 1m question on the same case (same sf-pr image; Binance ETHUSDT and BTCUSDT 1m klines over the case window, 573,120 and 525,600 bars):

run	result
ETHUSDT fed only as a 1 feed	exit 4, request.security(other, timeframe.period, ...) at line 7: no data is pinned .... A 1m feed serves neither the 240 nor the D request
ETHUSDT fed as 1 plus 240 and 1D	exit 0, the same 12 trades as the tf-fed run; the 1m feed is installed (573,120 bars in the record) and inert
chart fed 1m (PINEFORGE_INPUT_TF=1, PINEFORGE_SCRIPT_TF=240) with the tf feeds	exit 4, request.security of another symbol needs the chart's own bars as input; input '1' aggregated to chart '240' is not supported
data check (kit only, not shipped)	Binance's 4h and 1d klines equal the UTC aggregation of its 1m klines: 0 of 2,190 and 0 of 398 bars differ. A future pf_aggregate_bars_v1 fed 1m must therefore reproduce the 12 rows

Tests

• docker/run_json_symbol_feeds_test.py (new, pytest): index and entry shapes, duplicate keys, timeframe spelling and folding, CSV columns, numbers, BOM, empty volume, time_close, monotonic opens and closes, caps, facts and mintick, setter order and arguments, refusal by the engine with its last error, a library without the setters, the record and its hash, main() with the report and fingerprint, the structured failure in the body run and in both bench loops, and no change without the flag. All 101 tests in docker/ pass on this head, on macOS and on Linux (the box). As with the older docker/*_test.py, nothing registers them with ctest or CI; they run with python3 -m pytest docker/.
• Gate: the PR touches only docker/run_json.py, docker/run_json_symbol_feeds_test.py, docker/entrypoint.sh, docker/README.md and CHANGELOG.md, the release-harness category of rule no-behaviour-diff/v3. The six-step lab job rj-20261004t045310-66dd70 (preflight, release, debug, sanitizers, kernel, docs) runs on this exact head (tree 48100d11, 5 files since 7b59662: docs 2, release harness 3); pineforge/verify and pineforge/parity are posted from it via the diff rule. The pre-rebase head 6cc8247 passed the same gate (job rj-20261004t034744-44c80c).

🤖 Generated with Claude Code