From b07d8003a0e411f3b5a78da770a237606707bb7a Mon Sep 17 00:00:00 2001 From: luisleo526 Date: Sun, 4 Oct 2026 10:31:20 +0800 Subject: [PATCH 1/5] Install other symbols' bars for request.security in the Docker harness docker/run_json.py --symbol-feeds (PINEFORGE_SYMBOL_FEEDS in the entrypoint) reads a JSON index of other symbols' bars, keyed by the exact symbol string a script passes to request.security and by timeframe, and installs each symbol's catalog syminfo (tickerid as canonical, type, timezone, session, currency, mintick) and feeds through strategy_set_symbol_facts and strategy_set_symbol_feed, which engine and codegen have shipped since 1.0.0 but no release harness called. A feed is an OHLCV CSV like --ohlcv; each bar's close is its open plus the timeframe unless a time_close column gives it. Everything is checked before the first strategy state: timeframe spelling (bare D/W/M/S folded, "4h" refused), one feed per timeframe, CSV columns and numbers, increasing opens, closes at or before the next open, mintick, the 256-feed cap. A refusal, a library without the setters or a feed the engine refuses is the structured {"engine","error"} line (harness exit 1, entrypoint exit 4), like a rejected --syminfo. What was installed is recorded as applied_runtime.symbol_feeds (facts, bar counts, first/last open, a value hash), so such a run has its own fingerprint; without --symbol-feeds the report is unchanged. Stacked on harness/syminfo-mincontract (#322): it reuses that change's structured-failure path in main(). Co-Authored-By: Claude Opus 5.5 (1M context) --- CHANGELOG.md | 11 + docker/README.md | 68 ++++++ docker/entrypoint.sh | 10 + docker/run_json.py | 300 ++++++++++++++++++++++- docker/run_json_symbol_feeds_test.py | 350 +++++++++++++++++++++++++++ 5 files changed, 735 insertions(+), 4 deletions(-) create mode 100644 docker/run_json_symbol_feeds_test.py diff --git a/CHANGELOG.md b/CHANGELOG.md index 2cde76a3..2b11058f 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,6 +7,17 @@ version number follows semantic versioning over the surfaces the ## Unreleased +- **Harness symbol feeds:** `docker/run_json.py --symbol-feeds` (`PINEFORGE_SYMBOL_FEEDS`) + installs other symbols' bars for `request.security` on another symbol: a JSON + index keyed by the exact symbol string the script passes and by timeframe, each + feed an OHLCV CSV plus the symbol's catalog `syminfo`, through + `strategy_set_symbol_facts` / `strategy_set_symbol_feed` (engine and codegen + 1.0.0 and later). Unset changes nothing; an index or feed it cannot install fails + the run (one `{"engine":"pineforge","error":...}` line, harness exit 1, + entrypoint exit 4). What was installed is recorded as + `applied_runtime.symbol_feeds`, so such a run has its own fingerprint digest. + The engine library is unchanged. + - **Harness lot grid:** `docker/run_json.py` applies `mincontract` from the `--syminfo` (`PINEFORGE_SYMINFO`) file as the engine's lot grid, the metadata key `qty_step` plus the `mincontract` key a script's `syminfo.mincontract` reads, so a diff --git a/docker/README.md b/docker/README.md index 7927c276..0e53bd3a 100644 --- a/docker/README.md +++ b/docker/README.md @@ -172,6 +172,74 @@ library's `strategy_set_syminfo_*` setters. `docker/run_json.py` is vendored: pineforge-release copies it from the engine tag at every release, so the lot-grid handling (`mincontract`) lives in this file. +### Other symbols' bars (`PINEFORGE_SYMBOL_FEEDS`) + +A script that calls `request.security` on another symbol reads that symbol's +own bars, never the chart's. Without them the run stops where the request's +value is read (`request.security(...) at line N: no data is pinned for this +request, and its value was read`, exit 4). `PINEFORGE_SYMBOL_FEEDS` (the +harness's `--symbol-feeds`) names a JSON index of those bars, installed through +the library's `strategy_set_symbol_facts` and `strategy_set_symbol_feed` (engine +1.0.0 and later): + +```json +{"symbols": { + "BINANCE:ETHUSDT": { + "syminfo": {"tickerid": "BINANCE:ETHUSDT", "type": "crypto", "currency": "USDT", + "mintick": 0.01, "session": "24x7", "timezone": "UTC"}, + "feeds": {"240": "ethusdt-240.csv", "1D": "ethusdt-1D.csv"}}}} +``` + +```bash +docker run --rm \ + -v $(pwd)/strategy.pine:/in/strategy.pine:ro \ + -v $(pwd)/btcusdt-240.csv:/in/ohlcv.csv:ro \ + -v $(pwd)/symbols:/in/symbols:ro \ + -e PINEFORGE_SYMBOL_FEEDS=/in/symbols/symbols.json \ + pineforge > report.json +``` + +- A symbol key is the exact string the script passes at run time, exchange + prefix and suffix included: `BINANCE:ETHUSDT`, `ETHUSDT` and + `BINANCE:ETHUSDT.P` are three symbols. For `input.symbol` it is the input's + value (its default, or the `PINEFORGE_INPUTS` override). A string naming the + chart's own market is another symbol too: the harness does not set the chart's + `syminfo.tickerid`. +- One feed per timeframe the script requests, keyed in the engine's spelling: + whole minutes as a bare integer (`"240"`, never `"4h"`), else `D|W|M|S`; + a bare `D`/`W`/`M`/`S` is folded to `1D`/`1W`/`1M`/`1S`. A request at + `timeframe.period` (or `""`) reads the feed at the chart's timeframe. Nothing is + aggregated: a `240` feed does not serve a `D` request. +- A feed is a CSV like `ohlcv.csv` (`timestamp,open,high,low,close,volume`; an + empty volume is a symbol that publishes none), paths relative to the index. + Each bar's close is its open plus the timeframe (calendar months for `M`), + right for a 24x7 symbol; give a session-bound symbol a `time_close` column + (unix ms). Bars before the chart's first bar are delivered as history on it; + bars after its last are never read. +- `syminfo` is the symbol's catalog object, flat or `{"syminfo": {...}}`: + `tickerid` (as the symbol's canonical id), `type`, `timezone`, `session`, + `currency` and `mintick` are set as its facts, which `syminfo.*` reads inside + the request; other keys are ignored. Without it, `syminfo.tickerid` inside the + request reads the key, `syminfo.mintick` NaN and the strings empty. +- Merge rule (TradingView's): with `lookahead` off a chart bar reads the latest + requested bar whose close is at or before the chart bar's close; with it on, + the latest that opened at or before the chart bar's open. A missing requested + bar carries the last value forward (`gaps` off) or reads na (`gaps` on). +- The chart must be its own input (`PINEFORGE_SCRIPT_TF` equal to the input + timeframe), and the run historical. +- An index or feed the harness cannot install (not JSON, a bad timeframe + spelling, two feeds at one timeframe, a CSV without `close`, decreasing + timestamps, a close after the next bar's open, a non-positive `mintick`, more + than 256 feeds, a library without the setters, a feed the engine refuses) + fails the run before it starts: one line + `{"engine":"pineforge","error":"--symbol-feeds: ..."}` on stdout, harness exit + 1, entrypoint exit 4. +- What was installed is recorded as `applied_runtime.symbol_feeds` (each + symbol's facts, and per feed its bar count, first and last open and a hash of + its values), so the fingerprint digest differs from a run without it. Unset, + or an index naming no symbol: the key is absent and the report is what it was + before this variable existed, apart from `elapsed_seconds`. + The engine catches every error (TF mismatch, unsupported emulation flags, unknown-input-TF, etc.) into `strategy_get_last_error()`; the container surfaces these as `{"engine":"pineforge","error":"..."}` on diff --git a/docker/entrypoint.sh b/docker/entrypoint.sh index c6aa7da8..a62696fe 100755 --- a/docker/entrypoint.sh +++ b/docker/entrypoint.sh @@ -43,6 +43,14 @@ # keys are ignored. mincontract absent or null: no lot grid. Any other # mincontract that is not a positive finite number fails the run # (exit 4, one {"engine":"pineforge","error":...} line on stdout). +# PINEFORGE_SYMBOL_FEEDS Path to a JSON index of other symbols' bars for +# request.security on another symbol: {"symbols": {"": +# {"syminfo": {...}, "feeds": {"": ""}}}}, the +# symbol string exactly as the script passes it, one CSV per +# timeframe the script requests (paths relative to the index). Unset: +# nothing is installed and such a request stops the run where its +# value is read. An index or feed the harness cannot install fails +# the run (exit 4, one {"engine":"pineforge","error":...} line). # # Exit codes: # 0 success (JSON report, or C++ in transpile-only mode, on stdout) @@ -140,12 +148,14 @@ echo "[pineforge] running backtest ..." >&2 # PINEFORGE_CHART_TZ IANA tz for date builtins # PINEFORGE_MAGNIFIER_VOLUME_WEIGHTED 1/true → vw magnifier (needs BAR_MAGNIFIER) # PINEFORGE_SYMINFO path to a syminfo.json (see the header) +# PINEFORGE_SYMBOL_FEEDS path to other symbols' feed index (see the header) # PINEFORGE_BENCH (+_WARMUP/_REPEATS) 1/true → timing mode extra=() [[ -n "${PINEFORGE_TRADE_START_MS:-}" ]] && extra+=(--trade-start-ms "${PINEFORGE_TRADE_START_MS}") [[ -n "${PINEFORGE_CHART_TZ:-}" ]] && extra+=(--chart-tz "${PINEFORGE_CHART_TZ}") [[ "${PINEFORGE_MAGNIFIER_VOLUME_WEIGHTED:-}" =~ ^(1|true|yes|on)$ ]] && extra+=(--magnifier-volume-weighted) [[ -n "${PINEFORGE_SYMINFO:-}" ]] && extra+=(--syminfo "${PINEFORGE_SYMINFO}") +[[ -n "${PINEFORGE_SYMBOL_FEEDS:-}" ]] && extra+=(--symbol-feeds "${PINEFORGE_SYMBOL_FEEDS}") if [[ "${PINEFORGE_BENCH:-}" =~ ^(1|true|yes|on)$ ]]; then extra+=(--bench --warmup "${PINEFORGE_WARMUP:-3}" --repeats "${PINEFORGE_REPEATS:-20}") fi diff --git a/docker/run_json.py b/docker/run_json.py index 2b95e2d2..86500885 100755 --- a/docker/run_json.py +++ b/docker/run_json.py @@ -78,8 +78,18 @@ "syminfo": {"qty_step": float, "mincontract": float} when --syminfo set a lot grid from syminfo.mincontract; without one the key is absent. -A run error or a --syminfo the harness rejects (see apply_syminfo) prints one -line {"engine": "pineforge", "error": ""} instead, exit status 1. +applied_runtime also holds "symbol_feeds" when --symbol-feeds installed other +symbols' bars for request.security (see load_symbol_feeds): + {"canonicalization": "pf-symbol-feed-barc-close-le-v1", + "symbols": {"": { + "facts": {"canonical": str, "type": str, ..., "mintick": float}, + "feeds": {"": {"bars": int, "first_ts": int, "last_ts": int, + "source_values_sha256": ""}}}}} +Without --symbol-feeds (or with an index naming no symbol) the key is absent. + +A run error, a --syminfo the harness rejects (see apply_syminfo) or a +--symbol-feeds it cannot install (see load_symbol_feeds) prints one line +{"engine": "pineforge", "error": ""} instead, exit status 1. NaN convention: any metric with an empty/zero denominator is null (JSON has no NaN); a real computed 0 stays 0. See the report-schema + metrics reference docs @@ -89,6 +99,7 @@ import argparse import base64 +import calendar import csv import ctypes import hashlib @@ -845,6 +856,18 @@ def load_strategy(so_path: Path) -> ctypes.CDLL: lib.strategy_set_syminfo_metadata.argtypes = [ ctypes.c_void_p, ctypes.c_char_p, ctypes.c_double] lib.strategy_set_syminfo_metadata.restype = None + # Other symbols' data for request.security (engine 1.0.0+, see + # load_symbol_feeds). hasattr-guarded: install_symbol_feeds fails by name + # on a library without them. + if hasattr(lib, "strategy_set_symbol_feed"): + lib.strategy_set_symbol_feed.argtypes = [ + ctypes.c_void_p, ctypes.c_char_p, ctypes.c_char_p, + ctypes.POINTER(BarC), ctypes.POINTER(ctypes.c_int64), ctypes.c_int] + lib.strategy_set_symbol_feed.restype = ctypes.c_int + if hasattr(lib, "strategy_set_symbol_facts"): + lib.strategy_set_symbol_facts.argtypes = [ + ctypes.c_void_p, ctypes.c_char_p, ctypes.c_char_p, ctypes.c_char_p] + lib.strategy_set_symbol_facts.restype = ctypes.c_int # Validation-parity setters mirrored from scripts/run_strategy.py. All # hasattr-guarded: trade_start_time + chart_timezone are runtime PF exports; @@ -913,6 +936,256 @@ def apply_syminfo(lib, strat, syminfo_path): return applied +# --- Other symbols' bars for request.security (--symbol-feeds) -------------- +# +# request.security on another symbol reads that symbol's own bars, never the +# chart's (engine and codegen 1.0.0+, C ABI strategy_set_symbol_feed / +# strategy_set_symbol_facts). The engine keys a feed by the exact symbol string +# the script passes at run time and by timeframe, and aggregates nothing: a +# request at "1D" needs a "1D" feed. --symbol-feeds names them in one index: +# +# {"symbols": {"BINANCE:ETHUSDT": { +# "syminfo": {}, +# "feeds": {"240": "ethusdt-240.csv", "1D": "ethusdt-1D.csv"}}}} + +SYMBOL_FEED_CANONICALIZATION = "pf-symbol-feed-barc-close-le-v1" +_SYMBOL_FEED_HASH_PREFIX = b"pineforge:symbol-feed:barc-close-le:v1\0" +_SYMBOL_FEED_RECORD = struct.Struct("<5dqq") +# The requests manifest's timeframe spelling and caps (scripts/run_strategy.py). +_SYMBOL_TF_RE = re.compile(r"(?:[1-9][0-9]{0,4}|[1-9][0-9]{0,3}[DWMS])") +_SYMBOL_FEEDS_MAX = 256 +_SYMBOL_KEY_MAX = 256 +# Catalog syminfo key -> the strategy_set_symbol_facts field it sets. +_SYMBOL_FACT_KEYS = (("tickerid", "canonical"), ("type", "type"), ("timezone", "timezone"), + ("session", "session"), ("currency", "currency"), ("mintick", "mintick")) +_SYMBOL_FEED_SETTERS = ("strategy_set_symbol_facts", "strategy_set_symbol_feed") + + +class SymbolFeedsError(ValueError): + """A --symbol-feeds index or feed the harness cannot install as given; main() + reports it as the structured {"engine", "error"} failure (exit 1).""" + + +def _shown(value) -> str: + return json.dumps(value)[:80] + + +def _symbol_text(value, what: str) -> str: + if (not isinstance(value, str) or not value or len(value) > _SYMBOL_KEY_MAX + or any(ord(ch) < 0x20 for ch in value)): + raise SymbolFeedsError( + f"--symbol-feeds: {what} must be a non-empty string of at most " + f"{_SYMBOL_KEY_MAX} characters without control characters, got {_shown(value)}") + return value + + +def symbol_timeframe(tf) -> str: + """The engine's one feed timeframe spelling: whole minutes as a bare integer + ("240", never "4h"), days, weeks, months and seconds as D|W|M|S; Pine's + bare D/W/M/S fold to 1D/1W/1M/1S, as the engine folds a request's.""" + if isinstance(tf, str) and tf in ("D", "W", "M", "S"): + tf = "1" + tf + if not (isinstance(tf, str) and _SYMBOL_TF_RE.fullmatch(tf)): + raise SymbolFeedsError( + "--symbol-feeds: a timeframe is whole minutes (\"15\", \"240\") or " + f"D|W|M|S (\"1D\", \"1W\"), got {_shown(tf)}") + return tf + + +def _bar_close_ms(open_ms: int, tf: str) -> int: + """A bar's close when its feed has no time_close column: its open plus the + timeframe, n calendar months (UTC) for M. Right for a 24x7 symbol; a + session-bound one must carry time_close.""" + n = int(tf) if tf.isdigit() else int(tf[:-1]) + unit = "" if tf.isdigit() else tf[-1] + if unit != "M": + return open_ms + n * {"": 60_000, "S": 1_000, "D": 86_400_000, + "W": 604_800_000}[unit] + secs, ms = divmod(open_ms, 1000) + t = datetime.fromtimestamp(secs, tz=timezone.utc) + month = t.month - 1 + n + year, month = t.year + month // 12, month % 12 + 1 + day = min(t.day, calendar.monthrange(year, month)[1]) + return int(t.replace(year=year, month=month, day=day).timestamp()) * 1000 + ms + + +def _load_symbol_feed(path: Path, symbol: str, tf: str) -> dict: + """One feed CSV -> ctypes bars and closes plus its record. Columns: + timestamp (open, unix ms), open, high, low, close, optional volume (empty or + NaN when the symbol publishes none) and optional time_close (unix ms); other + columns are ignored.""" + where = f"--symbol-feeds: feed {symbol}@{tf} ({path})" + try: + with path.open(newline="", encoding="utf-8") as f: + reader = csv.DictReader(f) + columns = reader.fieldnames or [] + missing = [c for c in ("timestamp", "open", "high", "low", "close") + if c not in columns] + if missing: + raise SymbolFeedsError(f"{where}: no column {', '.join(missing)}") + rows = [] + for line, row in enumerate(reader, start=2): + try: + ts = int(row["timestamp"]) + o, h, l, c = (float(row[k]) for k in ("open", "high", "low", "close")) + vol = (row.get("volume") or "").strip() + v = float(vol) if vol else math.nan + close = int(row["time_close"]) if "time_close" in columns else None + except (TypeError, ValueError): + raise SymbolFeedsError(f"{where} line {line}: not a number") from None + if not all(math.isfinite(x) for x in (o, h, l, c)) or v < 0 or math.isinf(v): + raise SymbolFeedsError( + f"{where} line {line}: prices must be finite and volume " + "nonnegative or empty") + rows.append((o, h, l, c, v, ts, close)) + except OSError as e: + raise SymbolFeedsError(f"{where}: {e.strerror or e}") from None + if not rows: + raise SymbolFeedsError(f"{where}: no bars") + n = len(rows) + bars = (BarC * n)() + closes = (ctypes.c_int64 * n)() + hasher = hashlib.sha256(_SYMBOL_FEED_HASH_PREFIX) + for i, (o, h, l, c, v, ts, close) in enumerate(rows): + if close is None: + close = _bar_close_ms(ts, tf) + next_open = rows[i + 1][5] if i + 1 < n else None + if next_open is not None and next_open <= ts: + raise SymbolFeedsError(f"{where} line {i + 3}: timestamps must increase") + if close <= ts or (next_open is not None and close > next_open): + raise SymbolFeedsError( + f"{where} line {i + 2}: its close {close} is not after its open {ts} " + "and at or before the next bar's open (is the timeframe right?)") + bars[i].open, bars[i].high, bars[i].low, bars[i].close = o, h, l, c + bars[i].volume, bars[i].timestamp = v, ts + closes[i] = close + hasher.update(_SYMBOL_FEED_RECORD.pack(o, h, l, c, v, ts, close)) + return {"timeframe": tf, "bars": bars, "close_ms": closes, "n": n, + "record": {"bars": n, "first_ts": rows[0][5], "last_ts": rows[-1][5], + "source_values_sha256": hasher.hexdigest()}} + + +def _symbol_facts(doc, symbol: str) -> list: + """The strategy_set_symbol_facts (field, value) pairs of a catalog syminfo + object: tickerid (as canonical), type, timezone, session, currency, mintick. + Absent, null or empty keys set nothing; other keys are ignored.""" + if doc is None: + return [] + si = doc.get("syminfo", doc) if isinstance(doc, dict) else None + if not isinstance(si, dict): + raise SymbolFeedsError(f"--symbol-feeds: {symbol}: syminfo must be an object") + facts = [] + for key, field in _SYMBOL_FACT_KEYS: + value = si.get(key) + if value is None or value == "": + continue + if field == "mintick": + ok = isinstance(value, (int, float)) and not isinstance(value, bool) + if not (ok and math.isfinite(value) and value > 0): + raise SymbolFeedsError( + f"--symbol-feeds: {symbol}: syminfo.mintick must be a positive " + f"finite number, got {_shown(value)}") + facts.append((field, float(value))) + else: + facts.append((field, _symbol_text(value, f"{symbol}: syminfo.{key}"))) + return facts + + +def load_symbol_feeds(index_path: Path) -> list: + """Read and check the --symbol-feeds index and every feed it names before + any strategy state exists. Each symbol key is the exact string a script's + request.security passes (prefix and suffix included: "BINANCE:ETHUSDT", + "ETHUSDT" and "BINANCE:ETHUSDT.P" are three symbols). An entry holds "feeds" + ({timeframe: CSV path, relative to the index}) and optionally "syminfo". + Returns one entry per symbol: {"symbol", "facts", "feeds"}. Any problem is a + SymbolFeedsError naming it.""" + def unique(pairs): + out = {} + for k, v in pairs: + if k in out: + raise SymbolFeedsError(f"--symbol-feeds: duplicate key {_shown(k)}") + out[k] = v + return out + try: + doc = json.loads(index_path.read_text(encoding="utf-8"), object_pairs_hook=unique) + except OSError as e: + raise SymbolFeedsError(f"--symbol-feeds: {index_path}: {e.strerror or e}") from None + except ValueError as e: + if isinstance(e, SymbolFeedsError): + raise + raise SymbolFeedsError(f"--symbol-feeds: {index_path} is not JSON: {e}") from None + symbols = doc.get("symbols") if isinstance(doc, dict) else None + if not isinstance(symbols, dict): + raise SymbolFeedsError('--symbol-feeds: the index must be {"symbols": {...}}') + if len(symbols) > _SYMBOL_FEEDS_MAX: + raise SymbolFeedsError(f"--symbol-feeds: more than {_SYMBOL_FEEDS_MAX} symbols") + out, total = [], 0 + for symbol, entry in symbols.items(): + _symbol_text(symbol, "a symbol") + if not isinstance(entry, dict) or set(entry) - {"syminfo", "feeds"}: + raise SymbolFeedsError( + f'--symbol-feeds: {symbol}: an entry is {{"feeds": {{...}}, "syminfo": {{...}}}}') + feeds = entry.get("feeds", {}) + if not isinstance(feeds, dict): + raise SymbolFeedsError(f"--symbol-feeds: {symbol}: feeds must be an object") + total += len(feeds) + if total > _SYMBOL_FEEDS_MAX: + raise SymbolFeedsError(f"--symbol-feeds: more than {_SYMBOL_FEEDS_MAX} feeds") + named = {} + for tf, file in feeds.items(): + canonical = symbol_timeframe(tf) + if canonical in named: + raise SymbolFeedsError( + f"--symbol-feeds: {symbol}: two feeds at timeframe {canonical}") + if not isinstance(file, str) or not file: + raise SymbolFeedsError( + f"--symbol-feeds: {symbol}@{canonical}: the feed must name a CSV file") + named[canonical] = index_path.parent / file + facts = _symbol_facts(entry.get("syminfo"), symbol) + out.append({"symbol": symbol, "facts": facts, + "feeds": [_load_symbol_feed(path, symbol, tf) + for tf, path in named.items()]}) + return out + + +def install_symbol_feeds(lib, strat, symbols) -> None: + """Install what load_symbol_feeds read: each symbol's facts, then its feeds. + The engine copies the arrays, so one load serves every state of a run.""" + missing = [n for n in _SYMBOL_FEED_SETTERS if not hasattr(lib, n)] + if missing: + raise SymbolFeedsError( + f"the strategy library has no {', '.join(missing)}, so --symbol-feeds " + "cannot be installed (engine 1.0.0 or later)") + + def refused(what): + detail = "" + if hasattr(lib, "strategy_get_last_error"): + err = lib.strategy_get_last_error(strat) + detail = err.decode("utf-8", "replace") if err else "" + raise SymbolFeedsError(f"the engine refused {what}" + (f": {detail}" if detail else "")) + + for sym in symbols: + key = sym["symbol"].encode() + for field, value in sym["facts"]: + text = repr(value) if isinstance(value, float) else value + if lib.strategy_set_symbol_facts(strat, key, field.encode(), text.encode()) != 0: + refused(f"the {field} of {sym['symbol']}") + for feed in sym["feeds"]: + if lib.strategy_set_symbol_feed(strat, key, feed["timeframe"].encode(), + feed["bars"], feed["close_ms"], feed["n"]) != 0: + refused(f"the feed {sym['symbol']}@{feed['timeframe']}") + + +def symbol_feeds_record(symbols) -> dict: + """applied_runtime["symbol_feeds"]: what was installed, so a run with other + symbols' bars has its own fingerprint digest.""" + return {"canonicalization": SYMBOL_FEED_CANONICALIZATION, + "symbols": {sym["symbol"]: { + "facts": dict(sym["facts"]), + "feeds": {feed["timeframe"]: feed["record"] for feed in sym["feeds"]}, + } for sym in symbols}} + + def fmt_utc(ms: int) -> str: return datetime.fromtimestamp( ms / 1000, tz=timezone.utc).strftime("%Y-%m-%d %H:%M UTC") @@ -1178,6 +1451,11 @@ def main() -> int: "Raw samples only — no median/ratio is computed in the image.") ap.add_argument("--warmup", type=int, default=3, help="Bench warmup runs (default 3).") ap.add_argument("--repeats", type=int, default=20, help="Bench timed repeats (default 20).") + ap.add_argument("--symbol-feeds", type=Path, default=None, + help="JSON index of other symbols' bars (and syminfo) that " + "request.security reads, keyed by the exact symbol string " + "and timeframe (strategy_set_symbol_feed / _facts); see " + "load_symbol_feeds.") args = ap.parse_args() inputs = parse_kv_json(args.inputs, "--inputs") @@ -1199,6 +1477,9 @@ def main() -> int: # The lot grid the last _make_state() applied (the body run's), for # applied_runtime["syminfo"]; {} when none. syminfo_applied: dict = {} + # Other symbols' bars, read once before the first state (None without + # --symbol-feeds). + symbol_feeds = None def _make_state(): """Create + fully configure a fresh strategy state — everything EXCEPT the @@ -1218,6 +1499,12 @@ def _make_state(): except SyminfoError: lib.strategy_free(st) raise + if symbol_feeds: + try: + install_symbol_feeds(lib, st, symbol_feeds) + except SymbolFeedsError: + lib.strategy_free(st) + raise if args.trade_start_ms is not None and hasattr(lib, "strategy_set_trade_start_time"): lib.strategy_set_trade_start_time(st, int(args.trade_start_ms)) if args.chart_tz and hasattr(lib, "strategy_set_chart_timezone"): @@ -1240,6 +1527,8 @@ def _run(st, rep): # already happened above (load_strategy), outside any loop. timing = None try: + if args.symbol_feeds: + symbol_feeds = load_symbol_feeds(args.symbol_feeds) if args.bench: warmup = max(0, int(args.warmup)) repeats = max(1, int(args.repeats)) @@ -1265,8 +1554,9 @@ def _run(st, rep): # --- Body run: one configured run for trades / metrics / diagnostics. --- state = _make_state() - except SyminfoError as e: - # A rejected --syminfo: the structured failure, before any stdout. + except (SyminfoError, SymbolFeedsError) as e: + # A rejected --syminfo or --symbol-feeds: the structured failure, before + # any stdout. json.dump({"engine": "pineforge", "error": str(e)}, sys.stdout, separators=(",", ":")) sys.stdout.write("\n") @@ -1301,6 +1591,8 @@ def _run(st, rep): } if syminfo_applied: applied_runtime["syminfo"] = syminfo_applied + if symbol_feeds: + applied_runtime["symbol_feeds"] = symbol_feeds_record(symbol_feeds) incarnation_accessor = getattr( lib, "strategy_closed_trade_entry_incarnation", None) trade_entry_incarnations = ( diff --git a/docker/run_json_symbol_feeds_test.py b/docker/run_json_symbol_feeds_test.py new file mode 100644 index 00000000..097b5794 --- /dev/null +++ b/docker/run_json_symbol_feeds_test.py @@ -0,0 +1,350 @@ +import base64 +import contextlib +import ctypes +import io +import json +import math +import sys +from pathlib import Path + +import pytest +import run_json # docker/ is on sys.path in the engine test env + + +# --symbol-feeds: other symbols' bars for request.security, keyed by the exact +# symbol string and timeframe, installed through strategy_set_symbol_facts and +# strategy_set_symbol_feed before the run; absent changes nothing. + +CORE = ("pf_abi_version", "strategy_create", "strategy_set_input", "strategy_set_override", + "run_backtest_full", "strategy_free", "report_free") +FEED_SETTERS = ("strategy_set_symbol_facts", "strategy_set_symbol_feed") +ST = 7 +H4 = 4 * 3600 * 1000 +DAY = 24 * 3600 * 1000 +T0 = 1759536000000 # 2025-10-04T00:00:00Z +ETH = {"ticker": "ETHUSDT", "tickerid": "BINANCE:ETHUSDT", "type": "crypto", + "currency": "USDT", "basecurrency": "ETH", "mintick": 0.01, "pointvalue": 1.0, + "mincontract": 0.0001, "session": "24x7", "timezone": "UTC", "country": None} +ETH_FACTS = [(b"canonical", b"BINANCE:ETHUSDT"), (b"type", b"crypto"), (b"timezone", b"UTC"), + (b"session", b"24x7"), (b"currency", b"USDT"), (b"mintick", b"0.01")] + + +class FakeLib: + """A strategy library stand-in: each named symbol records (name, *args) and + returns its configured result; a name left out fails hasattr like a missing + export.""" + + def __init__(self, names, returns=None): + self.calls = [] + for name in names: + setattr(self, name, self._fn(name, (returns or {}).get(name))) + + def _fn(self, name, result): + def call(*args): + self.calls.append((name,) + args) + return result + return call + + +def fake_lib(setters=FEED_SETTERS, returns=None): + rets = {"pf_abi_version": run_json.EXPECTED_PF_ABI, + **{name: 0 for name in FEED_SETTERS}, **(returns or {})} + return FakeLib(CORE + tuple(setters), rets) + + +def csv_text(rows, header="timestamp,open,high,low,close,volume"): + return header + "\n" + "".join(",".join(str(v) for v in row) + "\n" for row in rows) + + +def bars(start, step, count, base=100.0): + return [(start + i * step, base + i, base + i + 2, base + i - 1, base + i + 1, 10 + i) + for i in range(count)] + + +def index(tmp_path, symbols, files=None, name="symbols.json"): + for fname, text in (files or {}).items(): + (tmp_path / fname).write_text(text) + p = tmp_path / name + p.write_text(symbols if isinstance(symbols, str) else json.dumps({"symbols": symbols})) + return p + + +def eth_index(tmp_path, syminfo=ETH): + entry = {"feeds": {"240": "eth-240.csv", "D": "eth-1D.csv"}} + if syminfo is not None: + entry["syminfo"] = syminfo + return index(tmp_path, {"BINANCE:ETHUSDT": entry}, + {"eth-240.csv": csv_text(bars(T0, H4, 3)), + "eth-1D.csv": csv_text(bars(T0 - DAY, DAY, 2, base=200.0))}) + + +def feed_calls(lib): + out = [] + for call in lib.calls: + if call[0] == "strategy_set_symbol_feed": + _, st, key, tf, arr, close, n = call + out.append((st, key, tf, [(arr[i].timestamp, arr[i].open, arr[i].high, + arr[i].low, arr[i].close, arr[i].volume) + for i in range(n)], list(close[:n]), n)) + return out + + +def test_facts_then_feeds_keyed_by_the_exact_string_and_canonical_timeframe(tmp_path): + lib = fake_lib() + symbols = run_json.load_symbol_feeds(eth_index(tmp_path)) + run_json.install_symbol_feeds(lib, ST, symbols) + facts = [c for c in lib.calls if c[0] == "strategy_set_symbol_facts"] + assert facts == [("strategy_set_symbol_facts", ST, b"BINANCE:ETHUSDT", f, v) + for f, v in ETH_FACTS] + assert [c[0] for c in lib.calls] == ["strategy_set_symbol_facts"] * 6 \ + + ["strategy_set_symbol_feed"] * 2 + (_, k1, tf1, rows1, close1, n1), (_, k2, tf2, rows2, close2, n2) = feed_calls(lib) + assert (k1, tf1, n1, k2, tf2, n2) == (b"BINANCE:ETHUSDT", b"240", 3, + b"BINANCE:ETHUSDT", b"1D", 2) + assert rows1[0] == (T0, 100.0, 102.0, 99.0, 101.0, 10.0) + # No time_close column: each close is the open plus the timeframe. + assert close1 == [T0 + H4, T0 + 2 * H4, T0 + 3 * H4] + assert close2 == [T0, T0 + DAY] + + +def test_record_names_what_was_installed(tmp_path): + rec = run_json.symbol_feeds_record(run_json.load_symbol_feeds(eth_index(tmp_path))) + eth = rec["symbols"]["BINANCE:ETHUSDT"] + assert rec["canonicalization"] == "pf-symbol-feed-barc-close-le-v1" + assert eth["facts"] == {"canonical": "BINANCE:ETHUSDT", "type": "crypto", + "timezone": "UTC", "session": "24x7", "currency": "USDT", + "mintick": 0.01} + assert sorted(eth["feeds"]) == ["1D", "240"] + assert eth["feeds"]["240"]["bars"] == 3 + assert (eth["feeds"]["240"]["first_ts"], eth["feeds"]["240"]["last_ts"]) == (T0, T0 + 2 * H4) + assert len(eth["feeds"]["240"]["source_values_sha256"]) == 64 + + +def test_without_syminfo_only_feeds_are_installed(tmp_path): + lib = fake_lib() + run_json.install_symbol_feeds(lib, ST, run_json.load_symbol_feeds(eth_index(tmp_path, None))) + assert [c[0] for c in lib.calls] == ["strategy_set_symbol_feed"] * 2 + + +def test_a_wrapped_syminfo_is_read_like_a_flat_one(tmp_path): + flat = run_json.load_symbol_feeds(eth_index(tmp_path))[0]["facts"] + wrapped = run_json.load_symbol_feeds(eth_index(tmp_path, {"syminfo": ETH}))[0]["facts"] + assert flat == wrapped + + +def test_time_close_column_gives_each_close(tmp_path): + rows = [(T0, 1, 2, 0.5, 1.5, "", T0 + 6 * 3600 * 1000), + (T0 + DAY, 1, 2, 0.5, 1.5, "NaN", T0 + DAY + 6 * 3600 * 1000)] + p = index(tmp_path, {"NASDAQ:QQQ": {"feeds": {"1D": "qqq.csv"}}}, + {"qqq.csv": csv_text(rows, "timestamp,open,high,low,close,volume,time_close")}) + lib = fake_lib() + run_json.install_symbol_feeds(lib, ST, run_json.load_symbol_feeds(p)) + (_, key, tf, got, close, n), = feed_calls(lib) + assert (key, tf, n) == (b"NASDAQ:QQQ", b"1D", 2) + assert close == [T0 + 6 * 3600 * 1000, T0 + DAY + 6 * 3600 * 1000] + assert all(math.isnan(r[5]) for r in got) # empty or NaN volume: none published + + +@pytest.mark.parametrize("tf,opens,closes", [ + ("1M", [1767225600000, 1769904000000], [1769904000000, 1772323200000]), # Jan, Feb 2026 + ("1W", [T0], [T0 + 7 * DAY]), + ("15", [T0], [T0 + 15 * 60 * 1000]), + ("30S", [T0], [T0 + 30 * 1000]), +]) +def test_derived_close_per_timeframe(tmp_path, tf, opens, closes): + rows = [(t, 1, 2, 0.5, 1.5, 3) for t in opens] + p = index(tmp_path, {"X:Y": {"feeds": {tf: "f.csv"}}}, {"f.csv": csv_text(rows)}) + lib = fake_lib() + run_json.install_symbol_feeds(lib, ST, run_json.load_symbol_feeds(p)) + assert feed_calls(lib)[0][4] == closes + + +def test_prefix_and_suffix_are_part_of_the_key(tmp_path): + feed = csv_text(bars(T0, H4, 2)) + p = index(tmp_path, {s: {"feeds": {"240": "f.csv"}} + for s in ("ETHUSDT", "BINANCE:ETHUSDT", "BINANCE:ETHUSDT.P")}, + {"f.csv": feed}) + lib = fake_lib() + run_json.install_symbol_feeds(lib, ST, run_json.load_symbol_feeds(p)) + assert [c[1] for c in feed_calls(lib)] == [b"ETHUSDT", b"BINANCE:ETHUSDT", + b"BINANCE:ETHUSDT.P"] + + +GOOD = csv_text(bars(T0, H4, 2)) + + +@pytest.mark.parametrize("symbols,files,message", [ + ({"E": {"feeds": {"4h": "f.csv"}}}, {"f.csv": GOOD}, 'a timeframe is whole minutes'), + ({"E": {"feeds": {"60m": "f.csv"}}}, {"f.csv": GOOD}, 'got "60m"'), + ({"E": {"feeds": {"D": "f.csv", "1D": "f.csv"}}}, {"f.csv": GOOD}, "two feeds at timeframe 1D"), + ({"E": {"feed": {"240": "f.csv"}}}, {"f.csv": GOOD}, "an entry is"), + ({"E": {"feeds": ["f.csv"]}}, {"f.csv": GOOD}, "feeds must be an object"), + ({"": {"feeds": {"240": "f.csv"}}}, {"f.csv": GOOD}, "a symbol must be a non-empty string"), + ({"E\n": {"feeds": {"240": "f.csv"}}}, {"f.csv": GOOD}, "without control characters"), + ({"E": {"feeds": {"240": ""}}}, {}, "the feed must name a CSV file"), + ({"E": {"feeds": {"240": "nope.csv"}}}, {}, "nope.csv"), + ({"E": {"feeds": {"240": "f.csv"}}}, {"f.csv": "timestamp,open,high,low\n1,2,3,4\n"}, + "no column close"), + ({"E": {"feeds": {"240": "f.csv"}}}, {"f.csv": csv_text([])}, "no bars"), + ({"E": {"feeds": {"240": "f.csv"}}}, {"f.csv": csv_text([(T0, "x", 1, 1, 1, 1)])}, + "line 2: not a number"), + ({"E": {"feeds": {"240": "f.csv"}}}, {"f.csv": csv_text([(T0, "inf", 1, 1, 1, 1)])}, + "line 2: prices must be finite"), + ({"E": {"feeds": {"240": "f.csv"}}}, {"f.csv": csv_text([(T0, 1, 1, 1, 1, -5)])}, + "volume nonnegative"), + ({"E": {"feeds": {"240": "f.csv"}}}, + {"f.csv": csv_text([(T0 + H4, 1, 1, 1, 1, 1), (T0, 1, 1, 1, 1, 1)])}, + "line 3: timestamps must increase"), + # 4h bars declared as 8h: each close lands after the next bar's open. + ({"E": {"feeds": {"480": "f.csv"}}}, {"f.csv": GOOD}, "is the timeframe right?"), + ({"E": {"syminfo": {"mintick": 0}, "feeds": {"240": "f.csv"}}}, {"f.csv": GOOD}, + "syminfo.mintick must be a positive finite number, got 0"), + ({"E": {"syminfo": {"mintick": "0.01"}, "feeds": {"240": "f.csv"}}}, {"f.csv": GOOD}, + 'got "0.01"'), + ({"E": {"syminfo": [], "feeds": {"240": "f.csv"}}}, {"f.csv": GOOD}, + "syminfo must be an object"), + ({"E": {"syminfo": {"session": 5}, "feeds": {"240": "f.csv"}}}, {"f.csv": GOOD}, + "syminfo.session must be a non-empty string"), +], ids=["4h", "60m", "D-and-1D", "unknown-entry-key", "feeds-list", "empty-symbol", + "control-char", "empty-path", "missing-file", "missing-column", "no-bars", "not-a-number", + "infinite-price", "negative-volume", "decreasing", "wrong-timeframe", "mintick-zero", + "mintick-string", "syminfo-list", "session-number"]) +def test_bad_index_or_feed_is_refused_by_name(tmp_path, symbols, files, message): + with pytest.raises(run_json.SymbolFeedsError, match=None) as e: + run_json.load_symbol_feeds(index(tmp_path, symbols, files)) + assert message in str(e.value) + assert str(e.value).startswith("--symbol-feeds") + + +@pytest.mark.parametrize("text,message", [ + ("[]", 'the index must be {"symbols": {...}}'), + ('{"symbol": {}}', 'the index must be {"symbols": {...}}'), + ("{not json", "is not JSON"), + ('{"symbols": {"E": {"feeds": {}}, "E": {"feeds": {}}}}', 'duplicate key "E"'), +], ids=["list", "no-symbols", "not-json", "duplicate-key"]) +def test_bad_index_document_is_refused(tmp_path, text, message): + with pytest.raises(run_json.SymbolFeedsError) as e: + run_json.load_symbol_feeds(index(tmp_path, text)) + assert message in str(e.value) + + +def test_more_than_256_feeds_is_refused(tmp_path): + symbols = {f"S{i}": {"feeds": {"240": "f.csv"}} for i in range(257)} + with pytest.raises(run_json.SymbolFeedsError, match="more than 256"): + run_json.load_symbol_feeds(index(tmp_path, symbols, {"f.csv": GOOD})) + + +@pytest.mark.parametrize("setters", [FEED_SETTERS[:1], FEED_SETTERS[1:], ()], + ids=["no-feed", "no-facts", "neither"]) +def test_a_library_without_the_setters_is_refused(tmp_path, setters): + lib = fake_lib(setters) + with pytest.raises(run_json.SymbolFeedsError, match="cannot be installed"): + run_json.install_symbol_feeds(lib, ST, run_json.load_symbol_feeds(eth_index(tmp_path))) + assert not [c for c in lib.calls if c[0] in FEED_SETTERS] + + +def test_an_engine_refusal_names_the_feed_and_the_engine_error(tmp_path): + lib = fake_lib(FEED_SETTERS + ("strategy_get_last_error",), + {"strategy_set_symbol_feed": -1, "strategy_get_last_error": b"bars must increase"}) + with pytest.raises(run_json.SymbolFeedsError) as e: + run_json.install_symbol_feeds(lib, ST, run_json.load_symbol_feeds(eth_index(tmp_path))) + assert str(e.value) == ("the engine refused the feed BINANCE:ETHUSDT@240: " + "bars must increase") + + +def test_load_strategy_declares_the_setter_signatures(monkeypatch): + monkeypatch.setattr(ctypes, "CDLL", lambda path: fake_lib()) + lib = run_json.load_strategy(Path("fake.so")) + assert lib.strategy_set_symbol_feed.argtypes == [ + ctypes.c_void_p, ctypes.c_char_p, ctypes.c_char_p, ctypes.POINTER(run_json.BarC), + ctypes.POINTER(ctypes.c_int64), ctypes.c_int] + assert lib.strategy_set_symbol_feed.restype is ctypes.c_int + assert lib.strategy_set_symbol_facts.argtypes == [ + ctypes.c_void_p, ctypes.c_char_p, ctypes.c_char_p, ctypes.c_char_p] + + +# main() against the fake library. + +@pytest.fixture +def harness(tmp_path, monkeypatch): + tape = tmp_path / "tape.csv" + tape.write_text("open,high,low,close,volume,timestamp\n" + "1,2,0.5,1.5,10,1000\n1.5,3,1,2.5,20,2000\n") + + def run(lib, *extra): + argv = ["run_json.py", "--so", "fake.so", "--ohlcv", str(tape), *extra] + monkeypatch.setattr(ctypes, "CDLL", lambda path: lib) + monkeypatch.setattr(sys, "argv", argv) + out = io.StringIO() + with contextlib.redirect_stdout(out): + status = run_json.main() + return status, out.getvalue() + + return run + + +def report_of(text): + rep = json.loads(text) + rep["elapsed_seconds"] = 0 + return rep + + +def count(lib, name): + return [c[0] for c in lib.calls].count(name) + + +def test_main_without_feeds_or_with_an_empty_index_reports_as_before(harness, tmp_path): + _, plain = harness(fake_lib()) + status, out = harness(fake_lib(), "--symbol-feeds", str(index(tmp_path, {}))) + assert status == 0 + assert report_of(out) == report_of(plain) + assert "symbol_feeds" not in report_of(out)["applied_runtime"] + + +def test_main_installs_feeds_and_records_them_in_the_fingerprint(harness, tmp_path): + _, plain = harness(fake_lib()) + lib = fake_lib() + status, out = harness(lib, "--symbol-feeds", str(eth_index(tmp_path))) + rep = report_of(out) + token = base64.b64decode(rep["fingerprint"]["token"]) + assert status == 0 + assert count(lib, "strategy_set_symbol_feed") == 2 + # Installed on the state before its run. + names = [c[0] for c in lib.calls] + assert names.index("strategy_set_symbol_feed") < names.index("run_backtest_full") + recorded = rep["applied_runtime"]["symbol_feeds"] + assert sorted(recorded["symbols"]["BINANCE:ETHUSDT"]["feeds"]) == ["1D", "240"] + assert json.loads(token)["runtime"]["symbol_feeds"] == recorded + assert rep["fingerprint"]["digest"] != report_of(plain)["fingerprint"]["digest"] + + +def test_main_bench_installs_feeds_on_every_state(harness, tmp_path): + lib = fake_lib() + status, _ = harness(lib, "--bench", "--warmup", "2", "--repeats", "3", + "--symbol-feeds", str(eth_index(tmp_path))) + states = 2 + 3 + 1 + assert status == 0 + assert count(lib, "strategy_create") == count(lib, "strategy_free") == states + assert count(lib, "strategy_set_symbol_feed") == 2 * states + + +def test_main_refuses_a_bad_index_with_one_structured_line_before_any_state(harness, tmp_path): + lib = fake_lib() + bad = index(tmp_path, {"E": {"feeds": {"4h": "f.csv"}}}, {"f.csv": GOOD}) + status, out = harness(lib, "--symbol-feeds", str(bad)) + assert status == 1 + assert out.endswith("\n") and out.count("\n") == 1 + assert json.loads(out) == {"engine": "pineforge", "error": + '--symbol-feeds: a timeframe is whole minutes ("15", "240") ' + 'or D|W|M|S ("1D", "1W"), got "4h"'} + assert count(lib, "strategy_create") == count(lib, "run_backtest_full") == 0 + + +def test_main_without_the_setters_fails_and_frees_the_state(harness, tmp_path): + lib = fake_lib(()) + status, out = harness(lib, "--symbol-feeds", str(eth_index(tmp_path))) + assert status == 1 + assert json.loads(out)["error"].startswith("the strategy library has no " + "strategy_set_symbol_facts, strategy_set_symbol_feed") + assert count(lib, "strategy_create") == count(lib, "strategy_free") == 1 + assert count(lib, "run_backtest_full") == 0 From be6475beef4ac8adabb2eab05a509b3fa44b875a Mon Sep 17 00:00:00 2001 From: luisleo526 Date: Sun, 4 Oct 2026 10:45:24 +0800 Subject: [PATCH 2/5] Address review of the symbol-feeds harness change - Every refusal is the structured line, never a traceback: a feed that is not UTF-8 (or a CSV error), a time outside +-(2^53-1) ms or a monthly close outside the calendar, a mintick integer beyond binary64, a symbol or fact string with a lone surrogate, an index nested past the recursion limit. - A header-only feed is installed without bars, as the engine documents it (its requests read na), instead of refused. - An empty time_close cell falls back to open plus timeframe; a UTF-8 BOM is read; error line numbers are the CSV's own (blank lines counted). - Install-time refusals carry the same "--symbol-feeds:" prefix. Co-Authored-By: Claude Opus 5.5 (1M context) --- docker/README.md | 6 ++- docker/run_json.py | 72 ++++++++++++++++--------- docker/run_json_symbol_feeds_test.py | 80 ++++++++++++++++++++++++++-- 3 files changed, 128 insertions(+), 30 deletions(-) diff --git a/docker/README.md b/docker/README.md index 0e53bd3a..925db645 100644 --- a/docker/README.md +++ b/docker/README.md @@ -214,8 +214,10 @@ docker run --rm \ empty volume is a symbol that publishes none), paths relative to the index. Each bar's close is its open plus the timeframe (calendar months for `M`), right for a 24x7 symbol; give a session-bound symbol a `time_close` column - (unix ms). Bars before the chart's first bar are delivered as history on it; - bars after its last are never read. + (unix ms; an empty cell falls back to open plus timeframe). Bars before the + chart's first bar are delivered as history on it; bars after its last are + never read. A header-only feed installs the symbol without bars: its requests + read na on every bar (a symbol with no bars in the window). - `syminfo` is the symbol's catalog object, flat or `{"syminfo": {...}}`: `tickerid` (as the symbol's canonical id), `type`, `timezone`, `session`, `currency` and `mintick` are set as its facts, which `syminfo.*` reads inside diff --git a/docker/run_json.py b/docker/run_json.py index 86500885..9c652081 100755 --- a/docker/run_json.py +++ b/docker/run_json.py @@ -955,6 +955,7 @@ def apply_syminfo(lib, strat, syminfo_path): _SYMBOL_TF_RE = re.compile(r"(?:[1-9][0-9]{0,4}|[1-9][0-9]{0,3}[DWMS])") _SYMBOL_FEEDS_MAX = 256 _SYMBOL_KEY_MAX = 256 +_SYMBOL_STAMP_MAX = 2**53 - 1 # unix ms; the record is fingerprinted as a JSON number # Catalog syminfo key -> the strategy_set_symbol_facts field it sets. _SYMBOL_FACT_KEYS = (("tickerid", "canonical"), ("type", "type"), ("timezone", "timezone"), ("session", "session"), ("currency", "currency"), ("mintick", "mintick")) @@ -971,6 +972,11 @@ def _shown(value) -> str: def _symbol_text(value, what: str) -> str: + try: + if isinstance(value, str): + value.encode("utf-8") + except UnicodeEncodeError: + value = None if (not isinstance(value, str) or not value or len(value) > _SYMBOL_KEY_MAX or any(ord(ch) < 0x20 for ch in value)): raise SymbolFeedsError( @@ -1002,11 +1008,14 @@ def _bar_close_ms(open_ms: int, tf: str) -> int: return open_ms + n * {"": 60_000, "S": 1_000, "D": 86_400_000, "W": 604_800_000}[unit] secs, ms = divmod(open_ms, 1000) - t = datetime.fromtimestamp(secs, tz=timezone.utc) - month = t.month - 1 + n - year, month = t.year + month // 12, month % 12 + 1 - day = min(t.day, calendar.monthrange(year, month)[1]) - return int(t.replace(year=year, month=month, day=day).timestamp()) * 1000 + ms + try: + t = datetime.fromtimestamp(secs, tz=timezone.utc) + month = t.month - 1 + n + year, month = t.year + month // 12, month % 12 + 1 + day = min(t.day, calendar.monthrange(year, month)[1]) + return int(t.replace(year=year, month=month, day=day).timestamp()) * 1000 + ms + except (ValueError, OverflowError, OSError): + return None # out of the calendar's range: refused by the caller def _load_symbol_feed(path: Path, symbol: str, tf: str) -> dict: @@ -1015,54 +1024,64 @@ def _load_symbol_feed(path: Path, symbol: str, tf: str) -> dict: NaN when the symbol publishes none) and optional time_close (unix ms); other columns are ignored.""" where = f"--symbol-feeds: feed {symbol}@{tf} ({path})" + rows, lines = [], [] try: - with path.open(newline="", encoding="utf-8") as f: + with path.open(newline="", encoding="utf-8-sig") as f: reader = csv.DictReader(f) columns = reader.fieldnames or [] missing = [c for c in ("timestamp", "open", "high", "low", "close") if c not in columns] if missing: raise SymbolFeedsError(f"{where}: no column {', '.join(missing)}") - rows = [] - for line, row in enumerate(reader, start=2): + for row in reader: + line = reader.line_num try: ts = int(row["timestamp"]) o, h, l, c = (float(row[k]) for k in ("open", "high", "low", "close")) vol = (row.get("volume") or "").strip() v = float(vol) if vol else math.nan - close = int(row["time_close"]) if "time_close" in columns else None + cell = (row.get("time_close") or "").strip() + close = int(cell) if cell else None # empty: open + timeframe except (TypeError, ValueError): raise SymbolFeedsError(f"{where} line {line}: not a number") from None if not all(math.isfinite(x) for x in (o, h, l, c)) or v < 0 or math.isinf(v): raise SymbolFeedsError( f"{where} line {line}: prices must be finite and volume " "nonnegative or empty") + if close is None: + close = _bar_close_ms(ts, tf) + if not all(x is not None and abs(x) <= _SYMBOL_STAMP_MAX for x in (ts, close)): + raise SymbolFeedsError( + f"{where} line {line}: a time must be unix milliseconds " + f"within +-{_SYMBOL_STAMP_MAX}") rows.append((o, h, l, c, v, ts, close)) + lines.append(line) except OSError as e: raise SymbolFeedsError(f"{where}: {e.strerror or e}") from None - if not rows: - raise SymbolFeedsError(f"{where}: no bars") + except (UnicodeDecodeError, csv.Error) as e: + raise SymbolFeedsError(f"{where}: not a UTF-8 CSV ({e})") from None n = len(rows) bars = (BarC * n)() closes = (ctypes.c_int64 * n)() hasher = hashlib.sha256(_SYMBOL_FEED_HASH_PREFIX) for i, (o, h, l, c, v, ts, close) in enumerate(rows): - if close is None: - close = _bar_close_ms(ts, tf) next_open = rows[i + 1][5] if i + 1 < n else None if next_open is not None and next_open <= ts: - raise SymbolFeedsError(f"{where} line {i + 3}: timestamps must increase") + raise SymbolFeedsError(f"{where} line {lines[i + 1]}: timestamps must increase") if close <= ts or (next_open is not None and close > next_open): raise SymbolFeedsError( - f"{where} line {i + 2}: its close {close} is not after its open {ts} " + f"{where} line {lines[i]}: its close {close} is not after its open {ts} " "and at or before the next bar's open (is the timeframe right?)") bars[i].open, bars[i].high, bars[i].low, bars[i].close = o, h, l, c bars[i].volume, bars[i].timestamp = v, ts closes[i] = close hasher.update(_SYMBOL_FEED_RECORD.pack(o, h, l, c, v, ts, close)) - return {"timeframe": tf, "bars": bars, "close_ms": closes, "n": n, - "record": {"bars": n, "first_ts": rows[0][5], "last_ts": rows[-1][5], - "source_values_sha256": hasher.hexdigest()}} + # A header-only feed is installed as the engine documents it: its requests + # read na on every bar (a symbol with no bars in the window). + record = {"bars": n, "source_values_sha256": hasher.hexdigest()} + if n: + record.update(first_ts=rows[0][5], last_ts=rows[-1][5]) + return {"timeframe": tf, "bars": bars, "close_ms": closes, "n": n, "record": record} def _symbol_facts(doc, symbol: str) -> list: @@ -1080,8 +1099,12 @@ def _symbol_facts(doc, symbol: str) -> list: if value is None or value == "": continue if field == "mintick": - ok = isinstance(value, (int, float)) and not isinstance(value, bool) - if not (ok and math.isfinite(value) and value > 0): + try: + ok = (isinstance(value, (int, float)) and not isinstance(value, bool) + and math.isfinite(float(value)) and value > 0) + except OverflowError: # an int beyond binary64 + ok = False + if not ok: raise SymbolFeedsError( f"--symbol-feeds: {symbol}: syminfo.mintick must be a positive " f"finite number, got {_shown(value)}") @@ -1110,7 +1133,7 @@ def unique(pairs): doc = json.loads(index_path.read_text(encoding="utf-8"), object_pairs_hook=unique) except OSError as e: raise SymbolFeedsError(f"--symbol-feeds: {index_path}: {e.strerror or e}") from None - except ValueError as e: + except (ValueError, RecursionError) as e: if isinstance(e, SymbolFeedsError): raise raise SymbolFeedsError(f"--symbol-feeds: {index_path} is not JSON: {e}") from None @@ -1154,15 +1177,16 @@ def install_symbol_feeds(lib, strat, symbols) -> None: missing = [n for n in _SYMBOL_FEED_SETTERS if not hasattr(lib, n)] if missing: raise SymbolFeedsError( - f"the strategy library has no {', '.join(missing)}, so --symbol-feeds " - "cannot be installed (engine 1.0.0 or later)") + f"--symbol-feeds: the strategy library has no {', '.join(missing)}, so " + "other symbols' bars cannot be installed (engine 1.0.0 or later)") def refused(what): detail = "" if hasattr(lib, "strategy_get_last_error"): err = lib.strategy_get_last_error(strat) detail = err.decode("utf-8", "replace") if err else "" - raise SymbolFeedsError(f"the engine refused {what}" + (f": {detail}" if detail else "")) + raise SymbolFeedsError(f"--symbol-feeds: the engine refused {what}" + + (f": {detail}" if detail else "")) for sym in symbols: key = sym["symbol"].encode() diff --git a/docker/run_json_symbol_feeds_test.py b/docker/run_json_symbol_feeds_test.py index 097b5794..14436ae2 100644 --- a/docker/run_json_symbol_feeds_test.py +++ b/docker/run_json_symbol_feeds_test.py @@ -185,7 +185,6 @@ def test_prefix_and_suffix_are_part_of_the_key(tmp_path): ({"E": {"feeds": {"240": "nope.csv"}}}, {}, "nope.csv"), ({"E": {"feeds": {"240": "f.csv"}}}, {"f.csv": "timestamp,open,high,low\n1,2,3,4\n"}, "no column close"), - ({"E": {"feeds": {"240": "f.csv"}}}, {"f.csv": csv_text([])}, "no bars"), ({"E": {"feeds": {"240": "f.csv"}}}, {"f.csv": csv_text([(T0, "x", 1, 1, 1, 1)])}, "line 2: not a number"), ({"E": {"feeds": {"240": "f.csv"}}}, {"f.csv": csv_text([(T0, "inf", 1, 1, 1, 1)])}, @@ -206,7 +205,7 @@ def test_prefix_and_suffix_are_part_of_the_key(tmp_path): ({"E": {"syminfo": {"session": 5}, "feeds": {"240": "f.csv"}}}, {"f.csv": GOOD}, "syminfo.session must be a non-empty string"), ], ids=["4h", "60m", "D-and-1D", "unknown-entry-key", "feeds-list", "empty-symbol", - "control-char", "empty-path", "missing-file", "missing-column", "no-bars", "not-a-number", + "control-char", "empty-path", "missing-file", "missing-column", "not-a-number", "infinite-price", "negative-volume", "decreasing", "wrong-timeframe", "mintick-zero", "mintick-string", "syminfo-list", "session-number"]) def test_bad_index_or_feed_is_refused_by_name(tmp_path, symbols, files, message): @@ -248,7 +247,7 @@ def test_an_engine_refusal_names_the_feed_and_the_engine_error(tmp_path): {"strategy_set_symbol_feed": -1, "strategy_get_last_error": b"bars must increase"}) with pytest.raises(run_json.SymbolFeedsError) as e: run_json.install_symbol_feeds(lib, ST, run_json.load_symbol_feeds(eth_index(tmp_path))) - assert str(e.value) == ("the engine refused the feed BINANCE:ETHUSDT@240: " + assert str(e.value) == ("--symbol-feeds: the engine refused the feed BINANCE:ETHUSDT@240: " "bars must increase") @@ -344,7 +343,80 @@ def test_main_without_the_setters_fails_and_frees_the_state(harness, tmp_path): lib = fake_lib(()) status, out = harness(lib, "--symbol-feeds", str(eth_index(tmp_path))) assert status == 1 - assert json.loads(out)["error"].startswith("the strategy library has no " + assert json.loads(out)["error"].startswith("--symbol-feeds: the strategy library has no " "strategy_set_symbol_facts, strategy_set_symbol_feed") assert count(lib, "strategy_create") == count(lib, "strategy_free") == 1 assert count(lib, "run_backtest_full") == 0 + + +def test_a_header_only_feed_is_installed_without_bars(tmp_path): + # The engine's contract: a feed without bars, whose requests read na. + p = index(tmp_path, {"X:Y": {"feeds": {"60": "f.csv"}}}, {"f.csv": csv_text([])}) + lib = fake_lib() + symbols = run_json.load_symbol_feeds(p) + run_json.install_symbol_feeds(lib, ST, symbols) + assert [(c[2], c[3], c[6]) for c in lib.calls if c[0] == "strategy_set_symbol_feed"] \ + == [(b"X:Y", b"60", 0)] + record = run_json.symbol_feeds_record(symbols)["symbols"]["X:Y"]["feeds"]["60"] + assert record["bars"] == 0 and "first_ts" not in record + + +def test_an_empty_time_close_cell_falls_back_to_open_plus_timeframe(tmp_path): + rows = [(T0, 1, 2, 0.5, 1.5, 3, ""), (T0 + H4, 1, 2, 0.5, 1.5, 3, T0 + H4 + 3600 * 1000)] + p = index(tmp_path, {"X:Y": {"feeds": {"240": "f.csv"}}}, + {"f.csv": csv_text(rows, "timestamp,open,high,low,close,volume,time_close")}) + lib = fake_lib() + run_json.install_symbol_feeds(lib, ST, run_json.load_symbol_feeds(p)) + assert feed_calls(lib)[0][4] == [T0 + H4, T0 + H4 + 3600 * 1000] + + +@pytest.mark.parametrize("opens,closes", [ + ([1764547200000], [1767225600000]), # 2025-12-01 -> 2026-01-01 + ([1769817600000], [1772236800000]), # 2026-01-31 -> 2026-02-28 (day clamped) + ([1767225600123], [1769904000123]), # milliseconds kept +], ids=["dec-jan", "clamp", "ms"]) +def test_month_close_edges(tmp_path, opens, closes): + p = index(tmp_path, {"X:Y": {"feeds": {"1M": "f.csv"}}}, + {"f.csv": csv_text([(t, 1, 2, 0.5, 1.5, 3) for t in opens])}) + lib = fake_lib() + run_json.install_symbol_feeds(lib, ST, run_json.load_symbol_feeds(p)) + assert feed_calls(lib)[0][4] == closes + + +def test_a_utf8_bom_is_read(tmp_path): + p = index(tmp_path, {"X:Y": {"feeds": {"240": "f.csv"}}}) + (tmp_path / "f.csv").write_bytes(b"\xef\xbb\xbf" + GOOD.encode()) + assert run_json.load_symbol_feeds(p)[0]["feeds"][0]["n"] == 2 + + +def test_line_numbers_count_blank_lines(tmp_path): + text = csv_text([(T0, 1, 2, 0.5, 1.5, 3)]) + "\n" + f"{T0 + H4},x,2,0.5,1.5,3\n" + with pytest.raises(run_json.SymbolFeedsError, match="line 4: not a number"): + run_json.load_symbol_feeds(index(tmp_path, {"X:Y": {"feeds": {"240": "f.csv"}}}, + {"f.csv": text})) + + +@pytest.mark.parametrize("tf,make,message", [ + ("240", lambda p: p.write_bytes(b"timestamp,open,high,low,close,volume\n1,caf\xe9,1,1,1,1\n"), + "not a UTF-8 CSV"), + ("240", lambda p: p.write_text(csv_text([(2**53, 1, 1, 1, 1, 1)])), "within +-9007199254740991"), + # Microsecond stamps read as milliseconds: a monthly close beyond the calendar. + ("1M", lambda p: p.write_text(csv_text([(1759536000000000, 1, 1, 1, 1, 1)])), "within"), +], ids=["cp1252", "beyond-2^53", "microseconds-month"]) +def test_undecodable_or_out_of_range_feeds_are_refused(tmp_path, tf, make, message): + make(tmp_path / "f.csv") + with pytest.raises(run_json.SymbolFeedsError) as e: + run_json.load_symbol_feeds(index(tmp_path, {"X:Y": {"feeds": {tf: "f.csv"}}})) + assert message in str(e.value) + + +@pytest.mark.parametrize("raw,message", [ + ('{"symbols": {"\\ud800": {"feeds": {}}}}', "a symbol must be a non-empty string"), + ('{"symbols": {"E": {"syminfo": {"mintick": 1' + "0" * 400 + '}, "feeds": {}}}}', + "syminfo.mintick must be a positive finite number"), + ("[" * 100000 + "]" * 100000, "is not JSON"), +], ids=["lone-surrogate", "huge-mintick", "deep-nesting"]) +def test_index_values_that_would_raise_are_refused(tmp_path, raw, message): + with pytest.raises(run_json.SymbolFeedsError) as e: + run_json.load_symbol_feeds(index(tmp_path, raw)) + assert message in str(e.value) From 84f945bf8167c01f6719b360c70a28e8bbbb92bb Mon Sep 17 00:00:00 2001 From: luisleo526 Date: Sun, 4 Oct 2026 11:41:08 +0800 Subject: [PATCH 3/5] Document the symbol-feed limits and that no feed is aggregated The harness README gains a limits bullet (256 symbols and 256 feeds, the chart as its own input with the engine's exact error, historical runs only, the cost of a feed, request.security_lower_tf) and states that a feed must be at the timeframe it serves: the engine aggregates only the chart's own input and the C ABI has no call that aggregates bars, so a 1m feed serves neither a 240 nor a D request. The CHANGELOG entry says the same. Co-Authored-By: Claude Opus 5.5 (1M context) --- CHANGELOG.md | 4 +++- docker/README.md | 16 ++++++++++++---- 2 files changed, 15 insertions(+), 5 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 2b11058f..59c5207f 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -12,7 +12,9 @@ version number follows semantic versioning over the surfaces the index keyed by the exact symbol string the script passes and by timeframe, each feed an OHLCV CSV plus the symbol's catalog `syminfo`, through `strategy_set_symbol_facts` / `strategy_set_symbol_feed` (engine and codegen - 1.0.0 and later). Unset changes nothing; an index or feed it cannot install fails + 1.0.0 and later). One feed per requested timeframe: nothing aggregates another + symbol's bars, so a `1` feed serves neither a `240` nor a `D` request. Unset + changes nothing; an index or feed it cannot install fails the run (one `{"engine":"pineforge","error":...}` line, harness exit 1, entrypoint exit 4). What was installed is recorded as `applied_runtime.symbol_feeds`, so such a run has its own fingerprint digest. diff --git a/docker/README.md b/docker/README.md index 925db645..483ac17b 100644 --- a/docker/README.md +++ b/docker/README.md @@ -208,8 +208,7 @@ docker run --rm \ - One feed per timeframe the script requests, keyed in the engine's spelling: whole minutes as a bare integer (`"240"`, never `"4h"`), else `D|W|M|S`; a bare `D`/`W`/`M`/`S` is folded to `1D`/`1W`/`1M`/`1S`. A request at - `timeframe.period` (or `""`) reads the feed at the chart's timeframe. Nothing is - aggregated: a `240` feed does not serve a `D` request. + `timeframe.period` (or `""`) reads the feed at the chart's timeframe. - A feed is a CSV like `ohlcv.csv` (`timestamp,open,high,low,close,volume`; an empty volume is a symbol that publishes none), paths relative to the index. Each bar's close is its open plus the timeframe (calendar months for `M`), @@ -227,8 +226,17 @@ docker run --rm \ requested bar whose close is at or before the chart bar's close; with it on, the latest that opened at or before the chart bar's open. A missing requested bar carries the last value forward (`gaps` off) or reads na (`gaps` on). -- The chart must be its own input (`PINEFORGE_SCRIPT_TF` equal to the input - timeframe), and the run historical. +- Limits: at most 256 symbols and 256 feeds in one index. The chart must be its + own input (`PINEFORGE_SCRIPT_TF` unset or equal to the input timeframe), else + the run fails with `request.security of another symbol needs the chart's own + bars as input; input '' aggregated to chart '' is not supported`. The run + is historical only. Each feed is a full pass of its request's expression over + its bars. `request.security_lower_tf` on another symbol reads no feed. +- A feed must be at the timeframe it serves. Nothing aggregates another + symbol's bars: the engine aggregates only the chart's own input, and the C ABI + has no call that aggregates bars. A `240` feed does not serve a `D` request, + and a `1` feed serves only a request at `1`, so give each requested timeframe + its own bars. - An index or feed the harness cannot install (not JSON, a bad timeframe spelling, two feeds at one timeframe, a CSV without `close`, decreasing timestamps, a close after the next bar's open, a non-positive `mintick`, more From 8c554c559a45ac6ac292b2b53695d6d92f92d231 Mon Sep 17 00:00:00 2001 From: luisleo526 Date: Sun, 4 Oct 2026 11:47:20 +0800 Subject: [PATCH 4/5] Correct the symbol-feeds README on syminfo reads and aggregation Review fixes. Inside the request syminfo.tickerid is always the key and syminfo.ticker the key after its last ":"; the catalog tickerid is set as the canonical fact, which no syminfo.* reads. The no-aggregation bullet now says why (the engine looks a feed up by exact symbol and timeframe and installs its bars as given) instead of claiming no C call aggregates. The missing-feed sentence covers an index that lacks the requested symbol or timeframe, and the refusal list is marked as examples ("timestamps that do not strictly increase"). Co-Authored-By: Claude Opus 5.5 (1M context) --- docker/README.md | 36 ++++++++++++++++++++---------------- 1 file changed, 20 insertions(+), 16 deletions(-) diff --git a/docker/README.md b/docker/README.md index 483ac17b..5de44d81 100644 --- a/docker/README.md +++ b/docker/README.md @@ -175,9 +175,10 @@ at every release, so the lot-grid handling (`mincontract`) lives in this file. ### Other symbols' bars (`PINEFORGE_SYMBOL_FEEDS`) A script that calls `request.security` on another symbol reads that symbol's -own bars, never the chart's. Without them the run stops where the request's -value is read (`request.security(...) at line N: no data is pinned for this -request, and its value was read`, exit 4). `PINEFORGE_SYMBOL_FEEDS` (the +own bars, never the chart's. Without them, or when the index below lacks the +requested symbol string or timeframe, the run stops where the request's value +is read (`request.security(...) at line N: no data is pinned for this request, +and its value was read`, exit 4). `PINEFORGE_SYMBOL_FEEDS` (the harness's `--symbol-feeds`) names a JSON index of those bars, installed through the library's `strategy_set_symbol_facts` and `strategy_set_symbol_feed` (engine 1.0.0 and later): @@ -217,11 +218,13 @@ docker run --rm \ chart's first bar are delivered as history on it; bars after its last are never read. A header-only feed installs the symbol without bars: its requests read na on every bar (a symbol with no bars in the window). -- `syminfo` is the symbol's catalog object, flat or `{"syminfo": {...}}`: - `tickerid` (as the symbol's canonical id), `type`, `timezone`, `session`, - `currency` and `mintick` are set as its facts, which `syminfo.*` reads inside - the request; other keys are ignored. Without it, `syminfo.tickerid` inside the - request reads the key, `syminfo.mintick` NaN and the strings empty. +- `syminfo` is the symbol's catalog object, flat or `{"syminfo": {...}}`. Its + `type`, `timezone`, `session`, `currency` and `mintick` are set as the + symbol's facts, which `syminfo.*` reads inside the request; `tickerid` is set + as its `canonical` fact, which no `syminfo.*` reads; other keys are ignored. + Inside the request `syminfo.tickerid` is always the key and `syminfo.ticker` + the key after its last `:`. Without `syminfo`, `syminfo.mintick` reads NaN + and those four strings read empty. - Merge rule (TradingView's): with `lookahead` off a chart bar reads the latest requested bar whose close is at or before the chart bar's close; with it on, the latest that opened at or before the chart bar's open. A missing requested @@ -233,14 +236,15 @@ docker run --rm \ is historical only. Each feed is a full pass of its request's expression over its bars. `request.security_lower_tf` on another symbol reads no feed. - A feed must be at the timeframe it serves. Nothing aggregates another - symbol's bars: the engine aggregates only the chart's own input, and the C ABI - has no call that aggregates bars. A `240` feed does not serve a `D` request, - and a `1` feed serves only a request at `1`, so give each requested timeframe - its own bars. -- An index or feed the harness cannot install (not JSON, a bad timeframe - spelling, two feeds at one timeframe, a CSV without `close`, decreasing - timestamps, a close after the next bar's open, a non-positive `mintick`, more - than 256 feeds, a library without the setters, a feed the engine refuses) + symbol's bars: the engine looks a feed up by the exact symbol string and + timeframe, and `strategy_set_symbol_feed` installs the bars as given. A `240` + feed does not serve a `D` request, and a `1` feed serves only a request at + `1`, so give each requested timeframe its own bars. +- An index or feed the harness cannot install (for example: not JSON, a bad + timeframe spelling, two feeds at one timeframe, a CSV without `close`, + timestamps that do not strictly increase, a close after the next bar's open, + a non-positive `mintick`, more than 256 feeds, a library without the setters, + a feed the engine refuses) fails the run before it starts: one line `{"engine":"pineforge","error":"--symbol-feeds: ..."}` on stdout, harness exit 1, entrypoint exit 4. From 673e95b8d7553f7159dd828e0fd31c8da9bb4e01 Mon Sep 17 00:00:00 2001 From: luisleo526 Date: Sun, 4 Oct 2026 12:52:18 +0800 Subject: [PATCH 5/5] Place the symbol-feeds CHANGELOG entry after the confirmed-bar streams one After the rebase onto #325 the entry sat at the top of Unreleased; it now follows #325's entry, as the merge order asks. Wording unchanged. Co-Authored-By: Claude Opus 5.5 (1M context) --- CHANGELOG.md | 26 +++++++++++++------------- 1 file changed, 13 insertions(+), 13 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 59c5207f..0e0e251a 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,19 +7,6 @@ version number follows semantic versioning over the surfaces the ## Unreleased -- **Harness symbol feeds:** `docker/run_json.py --symbol-feeds` (`PINEFORGE_SYMBOL_FEEDS`) - installs other symbols' bars for `request.security` on another symbol: a JSON - index keyed by the exact symbol string the script passes and by timeframe, each - feed an OHLCV CSV plus the symbol's catalog `syminfo`, through - `strategy_set_symbol_facts` / `strategy_set_symbol_feed` (engine and codegen - 1.0.0 and later). One feed per requested timeframe: nothing aggregates another - symbol's bars, so a `1` feed serves neither a `240` nor a `D` request. Unset - changes nothing; an index or feed it cannot install fails - the run (one `{"engine":"pineforge","error":...}` line, harness exit 1, - entrypoint exit 4). What was installed is recorded as - `applied_runtime.symbol_feeds`, so such a run has its own fingerprint digest. - The engine library is unchanged. - - **Harness lot grid:** `docker/run_json.py` applies `mincontract` from the `--syminfo` (`PINEFORGE_SYMINFO`) file as the engine's lot grid, the metadata key `qty_step` plus the `mincontract` key a script's `syminfo.mincontract` reads, so a @@ -44,6 +31,19 @@ version number follows semantic versioning over the surfaces the unchanged. The streaming guide gains a "Backtest vs live" section listing what a backtest can see that a live run cannot. +- **Harness symbol feeds:** `docker/run_json.py --symbol-feeds` (`PINEFORGE_SYMBOL_FEEDS`) + installs other symbols' bars for `request.security` on another symbol: a JSON + index keyed by the exact symbol string the script passes and by timeframe, each + feed an OHLCV CSV plus the symbol's catalog `syminfo`, through + `strategy_set_symbol_facts` / `strategy_set_symbol_feed` (engine and codegen + 1.0.0 and later). One feed per requested timeframe: nothing aggregates another + symbol's bars, so a `1` feed serves neither a `240` nor a `D` request. Unset + changes nothing; an index or feed it cannot install fails + the run (one `{"engine":"pineforge","error":...}` line, harness exit 1, + entrypoint exit 4). What was installed is recorded as + `applied_runtime.symbol_feeds`, so such a run has its own fingerprint digest. + The engine library is unchanged. + - **Native runner routing and delivery:** webhooks are optional; `--webhook-routes` adds first-match per-action targets and payload `pineforge-native-order-action/v2`. New `actions`, `status` and offline `redeliver` commands expose the journal and