diff --git a/CHANGELOG.md b/CHANGELOG.md index 2cde76a3..0e0e251a 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -31,6 +31,19 @@ version number follows semantic versioning over the surfaces the unchanged. The streaming guide gains a "Backtest vs live" section listing what a backtest can see that a live run cannot. +- **Harness symbol feeds:** `docker/run_json.py --symbol-feeds` (`PINEFORGE_SYMBOL_FEEDS`) + installs other symbols' bars for `request.security` on another symbol: a JSON + index keyed by the exact symbol string the script passes and by timeframe, each + feed an OHLCV CSV plus the symbol's catalog `syminfo`, through + `strategy_set_symbol_facts` / `strategy_set_symbol_feed` (engine and codegen + 1.0.0 and later). One feed per requested timeframe: nothing aggregates another + symbol's bars, so a `1` feed serves neither a `240` nor a `D` request. Unset + changes nothing; an index or feed it cannot install fails + the run (one `{"engine":"pineforge","error":...}` line, harness exit 1, + entrypoint exit 4). What was installed is recorded as + `applied_runtime.symbol_feeds`, so such a run has its own fingerprint digest. + The engine library is unchanged. + - **Native runner routing and delivery:** webhooks are optional; `--webhook-routes` adds first-match per-action targets and payload `pineforge-native-order-action/v2`. New `actions`, `status` and offline `redeliver` commands expose the journal and diff --git a/docker/README.md b/docker/README.md index 7927c276..5de44d81 100644 --- a/docker/README.md +++ b/docker/README.md @@ -172,6 +172,88 @@ library's `strategy_set_syminfo_*` setters. `docker/run_json.py` is vendored: pineforge-release copies it from the engine tag at every release, so the lot-grid handling (`mincontract`) lives in this file. +### Other symbols' bars (`PINEFORGE_SYMBOL_FEEDS`) + +A script that calls `request.security` on another symbol reads that symbol's +own bars, never the chart's. Without them, or when the index below lacks the +requested symbol string or timeframe, the run stops where the request's value +is read (`request.security(...) at line N: no data is pinned for this request, +and its value was read`, exit 4). `PINEFORGE_SYMBOL_FEEDS` (the +harness's `--symbol-feeds`) names a JSON index of those bars, installed through +the library's `strategy_set_symbol_facts` and `strategy_set_symbol_feed` (engine +1.0.0 and later): + +```json +{"symbols": { + "BINANCE:ETHUSDT": { + "syminfo": {"tickerid": "BINANCE:ETHUSDT", "type": "crypto", "currency": "USDT", + "mintick": 0.01, "session": "24x7", "timezone": "UTC"}, + "feeds": {"240": "ethusdt-240.csv", "1D": "ethusdt-1D.csv"}}}} +``` + +```bash +docker run --rm \ + -v $(pwd)/strategy.pine:/in/strategy.pine:ro \ + -v $(pwd)/btcusdt-240.csv:/in/ohlcv.csv:ro \ + -v $(pwd)/symbols:/in/symbols:ro \ + -e PINEFORGE_SYMBOL_FEEDS=/in/symbols/symbols.json \ + pineforge > report.json +``` + +- A symbol key is the exact string the script passes at run time, exchange + prefix and suffix included: `BINANCE:ETHUSDT`, `ETHUSDT` and + `BINANCE:ETHUSDT.P` are three symbols. For `input.symbol` it is the input's + value (its default, or the `PINEFORGE_INPUTS` override). A string naming the + chart's own market is another symbol too: the harness does not set the chart's + `syminfo.tickerid`. +- One feed per timeframe the script requests, keyed in the engine's spelling: + whole minutes as a bare integer (`"240"`, never `"4h"`), else `D|W|M|S`; + a bare `D`/`W`/`M`/`S` is folded to `1D`/`1W`/`1M`/`1S`. A request at + `timeframe.period` (or `""`) reads the feed at the chart's timeframe. +- A feed is a CSV like `ohlcv.csv` (`timestamp,open,high,low,close,volume`; an + empty volume is a symbol that publishes none), paths relative to the index. + Each bar's close is its open plus the timeframe (calendar months for `M`), + right for a 24x7 symbol; give a session-bound symbol a `time_close` column + (unix ms; an empty cell falls back to open plus timeframe). Bars before the + chart's first bar are delivered as history on it; bars after its last are + never read. A header-only feed installs the symbol without bars: its requests + read na on every bar (a symbol with no bars in the window). +- `syminfo` is the symbol's catalog object, flat or `{"syminfo": {...}}`. Its + `type`, `timezone`, `session`, `currency` and `mintick` are set as the + symbol's facts, which `syminfo.*` reads inside the request; `tickerid` is set + as its `canonical` fact, which no `syminfo.*` reads; other keys are ignored. + Inside the request `syminfo.tickerid` is always the key and `syminfo.ticker` + the key after its last `:`. Without `syminfo`, `syminfo.mintick` reads NaN + and those four strings read empty. +- Merge rule (TradingView's): with `lookahead` off a chart bar reads the latest + requested bar whose close is at or before the chart bar's close; with it on, + the latest that opened at or before the chart bar's open. A missing requested + bar carries the last value forward (`gaps` off) or reads na (`gaps` on). +- Limits: at most 256 symbols and 256 feeds in one index. The chart must be its + own input (`PINEFORGE_SCRIPT_TF` unset or equal to the input timeframe), else + the run fails with `request.security of another symbol needs the chart's own + bars as input; input '' aggregated to chart '' is not supported`. The run + is historical only. Each feed is a full pass of its request's expression over + its bars. `request.security_lower_tf` on another symbol reads no feed. +- A feed must be at the timeframe it serves. Nothing aggregates another + symbol's bars: the engine looks a feed up by the exact symbol string and + timeframe, and `strategy_set_symbol_feed` installs the bars as given. A `240` + feed does not serve a `D` request, and a `1` feed serves only a request at + `1`, so give each requested timeframe its own bars. +- An index or feed the harness cannot install (for example: not JSON, a bad + timeframe spelling, two feeds at one timeframe, a CSV without `close`, + timestamps that do not strictly increase, a close after the next bar's open, + a non-positive `mintick`, more than 256 feeds, a library without the setters, + a feed the engine refuses) + fails the run before it starts: one line + `{"engine":"pineforge","error":"--symbol-feeds: ..."}` on stdout, harness exit + 1, entrypoint exit 4. +- What was installed is recorded as `applied_runtime.symbol_feeds` (each + symbol's facts, and per feed its bar count, first and last open and a hash of + its values), so the fingerprint digest differs from a run without it. Unset, + or an index naming no symbol: the key is absent and the report is what it was + before this variable existed, apart from `elapsed_seconds`. + The engine catches every error (TF mismatch, unsupported emulation flags, unknown-input-TF, etc.) into `strategy_get_last_error()`; the container surfaces these as `{"engine":"pineforge","error":"..."}` on diff --git a/docker/entrypoint.sh b/docker/entrypoint.sh index c6aa7da8..a62696fe 100755 --- a/docker/entrypoint.sh +++ b/docker/entrypoint.sh @@ -43,6 +43,14 @@ # keys are ignored. mincontract absent or null: no lot grid. Any other # mincontract that is not a positive finite number fails the run # (exit 4, one {"engine":"pineforge","error":...} line on stdout). +# PINEFORGE_SYMBOL_FEEDS Path to a JSON index of other symbols' bars for +# request.security on another symbol: {"symbols": {"": +# {"syminfo": {...}, "feeds": {"": ""}}}}, the +# symbol string exactly as the script passes it, one CSV per +# timeframe the script requests (paths relative to the index). Unset: +# nothing is installed and such a request stops the run where its +# value is read. An index or feed the harness cannot install fails +# the run (exit 4, one {"engine":"pineforge","error":...} line). # # Exit codes: # 0 success (JSON report, or C++ in transpile-only mode, on stdout) @@ -140,12 +148,14 @@ echo "[pineforge] running backtest ..." >&2 # PINEFORGE_CHART_TZ IANA tz for date builtins # PINEFORGE_MAGNIFIER_VOLUME_WEIGHTED 1/true → vw magnifier (needs BAR_MAGNIFIER) # PINEFORGE_SYMINFO path to a syminfo.json (see the header) +# PINEFORGE_SYMBOL_FEEDS path to other symbols' feed index (see the header) # PINEFORGE_BENCH (+_WARMUP/_REPEATS) 1/true → timing mode extra=() [[ -n "${PINEFORGE_TRADE_START_MS:-}" ]] && extra+=(--trade-start-ms "${PINEFORGE_TRADE_START_MS}") [[ -n "${PINEFORGE_CHART_TZ:-}" ]] && extra+=(--chart-tz "${PINEFORGE_CHART_TZ}") [[ "${PINEFORGE_MAGNIFIER_VOLUME_WEIGHTED:-}" =~ ^(1|true|yes|on)$ ]] && extra+=(--magnifier-volume-weighted) [[ -n "${PINEFORGE_SYMINFO:-}" ]] && extra+=(--syminfo "${PINEFORGE_SYMINFO}") +[[ -n "${PINEFORGE_SYMBOL_FEEDS:-}" ]] && extra+=(--symbol-feeds "${PINEFORGE_SYMBOL_FEEDS}") if [[ "${PINEFORGE_BENCH:-}" =~ ^(1|true|yes|on)$ ]]; then extra+=(--bench --warmup "${PINEFORGE_WARMUP:-3}" --repeats "${PINEFORGE_REPEATS:-20}") fi diff --git a/docker/run_json.py b/docker/run_json.py index 2b95e2d2..9c652081 100755 --- a/docker/run_json.py +++ b/docker/run_json.py @@ -78,8 +78,18 @@ "syminfo": {"qty_step": float, "mincontract": float} when --syminfo set a lot grid from syminfo.mincontract; without one the key is absent. -A run error or a --syminfo the harness rejects (see apply_syminfo) prints one -line {"engine": "pineforge", "error": ""} instead, exit status 1. +applied_runtime also holds "symbol_feeds" when --symbol-feeds installed other +symbols' bars for request.security (see load_symbol_feeds): + {"canonicalization": "pf-symbol-feed-barc-close-le-v1", + "symbols": {"": { + "facts": {"canonical": str, "type": str, ..., "mintick": float}, + "feeds": {"": {"bars": int, "first_ts": int, "last_ts": int, + "source_values_sha256": ""}}}}} +Without --symbol-feeds (or with an index naming no symbol) the key is absent. + +A run error, a --syminfo the harness rejects (see apply_syminfo) or a +--symbol-feeds it cannot install (see load_symbol_feeds) prints one line +{"engine": "pineforge", "error": ""} instead, exit status 1. NaN convention: any metric with an empty/zero denominator is null (JSON has no NaN); a real computed 0 stays 0. See the report-schema + metrics reference docs @@ -89,6 +99,7 @@ import argparse import base64 +import calendar import csv import ctypes import hashlib @@ -845,6 +856,18 @@ def load_strategy(so_path: Path) -> ctypes.CDLL: lib.strategy_set_syminfo_metadata.argtypes = [ ctypes.c_void_p, ctypes.c_char_p, ctypes.c_double] lib.strategy_set_syminfo_metadata.restype = None + # Other symbols' data for request.security (engine 1.0.0+, see + # load_symbol_feeds). hasattr-guarded: install_symbol_feeds fails by name + # on a library without them. + if hasattr(lib, "strategy_set_symbol_feed"): + lib.strategy_set_symbol_feed.argtypes = [ + ctypes.c_void_p, ctypes.c_char_p, ctypes.c_char_p, + ctypes.POINTER(BarC), ctypes.POINTER(ctypes.c_int64), ctypes.c_int] + lib.strategy_set_symbol_feed.restype = ctypes.c_int + if hasattr(lib, "strategy_set_symbol_facts"): + lib.strategy_set_symbol_facts.argtypes = [ + ctypes.c_void_p, ctypes.c_char_p, ctypes.c_char_p, ctypes.c_char_p] + lib.strategy_set_symbol_facts.restype = ctypes.c_int # Validation-parity setters mirrored from scripts/run_strategy.py. All # hasattr-guarded: trade_start_time + chart_timezone are runtime PF exports; @@ -913,6 +936,280 @@ def apply_syminfo(lib, strat, syminfo_path): return applied +# --- Other symbols' bars for request.security (--symbol-feeds) -------------- +# +# request.security on another symbol reads that symbol's own bars, never the +# chart's (engine and codegen 1.0.0+, C ABI strategy_set_symbol_feed / +# strategy_set_symbol_facts). The engine keys a feed by the exact symbol string +# the script passes at run time and by timeframe, and aggregates nothing: a +# request at "1D" needs a "1D" feed. --symbol-feeds names them in one index: +# +# {"symbols": {"BINANCE:ETHUSDT": { +# "syminfo": {}, +# "feeds": {"240": "ethusdt-240.csv", "1D": "ethusdt-1D.csv"}}}} + +SYMBOL_FEED_CANONICALIZATION = "pf-symbol-feed-barc-close-le-v1" +_SYMBOL_FEED_HASH_PREFIX = b"pineforge:symbol-feed:barc-close-le:v1\0" +_SYMBOL_FEED_RECORD = struct.Struct("<5dqq") +# The requests manifest's timeframe spelling and caps (scripts/run_strategy.py). +_SYMBOL_TF_RE = re.compile(r"(?:[1-9][0-9]{0,4}|[1-9][0-9]{0,3}[DWMS])") +_SYMBOL_FEEDS_MAX = 256 +_SYMBOL_KEY_MAX = 256 +_SYMBOL_STAMP_MAX = 2**53 - 1 # unix ms; the record is fingerprinted as a JSON number +# Catalog syminfo key -> the strategy_set_symbol_facts field it sets. +_SYMBOL_FACT_KEYS = (("tickerid", "canonical"), ("type", "type"), ("timezone", "timezone"), + ("session", "session"), ("currency", "currency"), ("mintick", "mintick")) +_SYMBOL_FEED_SETTERS = ("strategy_set_symbol_facts", "strategy_set_symbol_feed") + + +class SymbolFeedsError(ValueError): + """A --symbol-feeds index or feed the harness cannot install as given; main() + reports it as the structured {"engine", "error"} failure (exit 1).""" + + +def _shown(value) -> str: + return json.dumps(value)[:80] + + +def _symbol_text(value, what: str) -> str: + try: + if isinstance(value, str): + value.encode("utf-8") + except UnicodeEncodeError: + value = None + if (not isinstance(value, str) or not value or len(value) > _SYMBOL_KEY_MAX + or any(ord(ch) < 0x20 for ch in value)): + raise SymbolFeedsError( + f"--symbol-feeds: {what} must be a non-empty string of at most " + f"{_SYMBOL_KEY_MAX} characters without control characters, got {_shown(value)}") + return value + + +def symbol_timeframe(tf) -> str: + """The engine's one feed timeframe spelling: whole minutes as a bare integer + ("240", never "4h"), days, weeks, months and seconds as D|W|M|S; Pine's + bare D/W/M/S fold to 1D/1W/1M/1S, as the engine folds a request's.""" + if isinstance(tf, str) and tf in ("D", "W", "M", "S"): + tf = "1" + tf + if not (isinstance(tf, str) and _SYMBOL_TF_RE.fullmatch(tf)): + raise SymbolFeedsError( + "--symbol-feeds: a timeframe is whole minutes (\"15\", \"240\") or " + f"D|W|M|S (\"1D\", \"1W\"), got {_shown(tf)}") + return tf + + +def _bar_close_ms(open_ms: int, tf: str) -> int: + """A bar's close when its feed has no time_close column: its open plus the + timeframe, n calendar months (UTC) for M. Right for a 24x7 symbol; a + session-bound one must carry time_close.""" + n = int(tf) if tf.isdigit() else int(tf[:-1]) + unit = "" if tf.isdigit() else tf[-1] + if unit != "M": + return open_ms + n * {"": 60_000, "S": 1_000, "D": 86_400_000, + "W": 604_800_000}[unit] + secs, ms = divmod(open_ms, 1000) + try: + t = datetime.fromtimestamp(secs, tz=timezone.utc) + month = t.month - 1 + n + year, month = t.year + month // 12, month % 12 + 1 + day = min(t.day, calendar.monthrange(year, month)[1]) + return int(t.replace(year=year, month=month, day=day).timestamp()) * 1000 + ms + except (ValueError, OverflowError, OSError): + return None # out of the calendar's range: refused by the caller + + +def _load_symbol_feed(path: Path, symbol: str, tf: str) -> dict: + """One feed CSV -> ctypes bars and closes plus its record. Columns: + timestamp (open, unix ms), open, high, low, close, optional volume (empty or + NaN when the symbol publishes none) and optional time_close (unix ms); other + columns are ignored.""" + where = f"--symbol-feeds: feed {symbol}@{tf} ({path})" + rows, lines = [], [] + try: + with path.open(newline="", encoding="utf-8-sig") as f: + reader = csv.DictReader(f) + columns = reader.fieldnames or [] + missing = [c for c in ("timestamp", "open", "high", "low", "close") + if c not in columns] + if missing: + raise SymbolFeedsError(f"{where}: no column {', '.join(missing)}") + for row in reader: + line = reader.line_num + try: + ts = int(row["timestamp"]) + o, h, l, c = (float(row[k]) for k in ("open", "high", "low", "close")) + vol = (row.get("volume") or "").strip() + v = float(vol) if vol else math.nan + cell = (row.get("time_close") or "").strip() + close = int(cell) if cell else None # empty: open + timeframe + except (TypeError, ValueError): + raise SymbolFeedsError(f"{where} line {line}: not a number") from None + if not all(math.isfinite(x) for x in (o, h, l, c)) or v < 0 or math.isinf(v): + raise SymbolFeedsError( + f"{where} line {line}: prices must be finite and volume " + "nonnegative or empty") + if close is None: + close = _bar_close_ms(ts, tf) + if not all(x is not None and abs(x) <= _SYMBOL_STAMP_MAX for x in (ts, close)): + raise SymbolFeedsError( + f"{where} line {line}: a time must be unix milliseconds " + f"within +-{_SYMBOL_STAMP_MAX}") + rows.append((o, h, l, c, v, ts, close)) + lines.append(line) + except OSError as e: + raise SymbolFeedsError(f"{where}: {e.strerror or e}") from None + except (UnicodeDecodeError, csv.Error) as e: + raise SymbolFeedsError(f"{where}: not a UTF-8 CSV ({e})") from None + n = len(rows) + bars = (BarC * n)() + closes = (ctypes.c_int64 * n)() + hasher = hashlib.sha256(_SYMBOL_FEED_HASH_PREFIX) + for i, (o, h, l, c, v, ts, close) in enumerate(rows): + next_open = rows[i + 1][5] if i + 1 < n else None + if next_open is not None and next_open <= ts: + raise SymbolFeedsError(f"{where} line {lines[i + 1]}: timestamps must increase") + if close <= ts or (next_open is not None and close > next_open): + raise SymbolFeedsError( + f"{where} line {lines[i]}: its close {close} is not after its open {ts} " + "and at or before the next bar's open (is the timeframe right?)") + bars[i].open, bars[i].high, bars[i].low, bars[i].close = o, h, l, c + bars[i].volume, bars[i].timestamp = v, ts + closes[i] = close + hasher.update(_SYMBOL_FEED_RECORD.pack(o, h, l, c, v, ts, close)) + # A header-only feed is installed as the engine documents it: its requests + # read na on every bar (a symbol with no bars in the window). + record = {"bars": n, "source_values_sha256": hasher.hexdigest()} + if n: + record.update(first_ts=rows[0][5], last_ts=rows[-1][5]) + return {"timeframe": tf, "bars": bars, "close_ms": closes, "n": n, "record": record} + + +def _symbol_facts(doc, symbol: str) -> list: + """The strategy_set_symbol_facts (field, value) pairs of a catalog syminfo + object: tickerid (as canonical), type, timezone, session, currency, mintick. + Absent, null or empty keys set nothing; other keys are ignored.""" + if doc is None: + return [] + si = doc.get("syminfo", doc) if isinstance(doc, dict) else None + if not isinstance(si, dict): + raise SymbolFeedsError(f"--symbol-feeds: {symbol}: syminfo must be an object") + facts = [] + for key, field in _SYMBOL_FACT_KEYS: + value = si.get(key) + if value is None or value == "": + continue + if field == "mintick": + try: + ok = (isinstance(value, (int, float)) and not isinstance(value, bool) + and math.isfinite(float(value)) and value > 0) + except OverflowError: # an int beyond binary64 + ok = False + if not ok: + raise SymbolFeedsError( + f"--symbol-feeds: {symbol}: syminfo.mintick must be a positive " + f"finite number, got {_shown(value)}") + facts.append((field, float(value))) + else: + facts.append((field, _symbol_text(value, f"{symbol}: syminfo.{key}"))) + return facts + + +def load_symbol_feeds(index_path: Path) -> list: + """Read and check the --symbol-feeds index and every feed it names before + any strategy state exists. Each symbol key is the exact string a script's + request.security passes (prefix and suffix included: "BINANCE:ETHUSDT", + "ETHUSDT" and "BINANCE:ETHUSDT.P" are three symbols). An entry holds "feeds" + ({timeframe: CSV path, relative to the index}) and optionally "syminfo". + Returns one entry per symbol: {"symbol", "facts", "feeds"}. Any problem is a + SymbolFeedsError naming it.""" + def unique(pairs): + out = {} + for k, v in pairs: + if k in out: + raise SymbolFeedsError(f"--symbol-feeds: duplicate key {_shown(k)}") + out[k] = v + return out + try: + doc = json.loads(index_path.read_text(encoding="utf-8"), object_pairs_hook=unique) + except OSError as e: + raise SymbolFeedsError(f"--symbol-feeds: {index_path}: {e.strerror or e}") from None + except (ValueError, RecursionError) as e: + if isinstance(e, SymbolFeedsError): + raise + raise SymbolFeedsError(f"--symbol-feeds: {index_path} is not JSON: {e}") from None + symbols = doc.get("symbols") if isinstance(doc, dict) else None + if not isinstance(symbols, dict): + raise SymbolFeedsError('--symbol-feeds: the index must be {"symbols": {...}}') + if len(symbols) > _SYMBOL_FEEDS_MAX: + raise SymbolFeedsError(f"--symbol-feeds: more than {_SYMBOL_FEEDS_MAX} symbols") + out, total = [], 0 + for symbol, entry in symbols.items(): + _symbol_text(symbol, "a symbol") + if not isinstance(entry, dict) or set(entry) - {"syminfo", "feeds"}: + raise SymbolFeedsError( + f'--symbol-feeds: {symbol}: an entry is {{"feeds": {{...}}, "syminfo": {{...}}}}') + feeds = entry.get("feeds", {}) + if not isinstance(feeds, dict): + raise SymbolFeedsError(f"--symbol-feeds: {symbol}: feeds must be an object") + total += len(feeds) + if total > _SYMBOL_FEEDS_MAX: + raise SymbolFeedsError(f"--symbol-feeds: more than {_SYMBOL_FEEDS_MAX} feeds") + named = {} + for tf, file in feeds.items(): + canonical = symbol_timeframe(tf) + if canonical in named: + raise SymbolFeedsError( + f"--symbol-feeds: {symbol}: two feeds at timeframe {canonical}") + if not isinstance(file, str) or not file: + raise SymbolFeedsError( + f"--symbol-feeds: {symbol}@{canonical}: the feed must name a CSV file") + named[canonical] = index_path.parent / file + facts = _symbol_facts(entry.get("syminfo"), symbol) + out.append({"symbol": symbol, "facts": facts, + "feeds": [_load_symbol_feed(path, symbol, tf) + for tf, path in named.items()]}) + return out + + +def install_symbol_feeds(lib, strat, symbols) -> None: + """Install what load_symbol_feeds read: each symbol's facts, then its feeds. + The engine copies the arrays, so one load serves every state of a run.""" + missing = [n for n in _SYMBOL_FEED_SETTERS if not hasattr(lib, n)] + if missing: + raise SymbolFeedsError( + f"--symbol-feeds: the strategy library has no {', '.join(missing)}, so " + "other symbols' bars cannot be installed (engine 1.0.0 or later)") + + def refused(what): + detail = "" + if hasattr(lib, "strategy_get_last_error"): + err = lib.strategy_get_last_error(strat) + detail = err.decode("utf-8", "replace") if err else "" + raise SymbolFeedsError(f"--symbol-feeds: the engine refused {what}" + + (f": {detail}" if detail else "")) + + for sym in symbols: + key = sym["symbol"].encode() + for field, value in sym["facts"]: + text = repr(value) if isinstance(value, float) else value + if lib.strategy_set_symbol_facts(strat, key, field.encode(), text.encode()) != 0: + refused(f"the {field} of {sym['symbol']}") + for feed in sym["feeds"]: + if lib.strategy_set_symbol_feed(strat, key, feed["timeframe"].encode(), + feed["bars"], feed["close_ms"], feed["n"]) != 0: + refused(f"the feed {sym['symbol']}@{feed['timeframe']}") + + +def symbol_feeds_record(symbols) -> dict: + """applied_runtime["symbol_feeds"]: what was installed, so a run with other + symbols' bars has its own fingerprint digest.""" + return {"canonicalization": SYMBOL_FEED_CANONICALIZATION, + "symbols": {sym["symbol"]: { + "facts": dict(sym["facts"]), + "feeds": {feed["timeframe"]: feed["record"] for feed in sym["feeds"]}, + } for sym in symbols}} + + def fmt_utc(ms: int) -> str: return datetime.fromtimestamp( ms / 1000, tz=timezone.utc).strftime("%Y-%m-%d %H:%M UTC") @@ -1178,6 +1475,11 @@ def main() -> int: "Raw samples only — no median/ratio is computed in the image.") ap.add_argument("--warmup", type=int, default=3, help="Bench warmup runs (default 3).") ap.add_argument("--repeats", type=int, default=20, help="Bench timed repeats (default 20).") + ap.add_argument("--symbol-feeds", type=Path, default=None, + help="JSON index of other symbols' bars (and syminfo) that " + "request.security reads, keyed by the exact symbol string " + "and timeframe (strategy_set_symbol_feed / _facts); see " + "load_symbol_feeds.") args = ap.parse_args() inputs = parse_kv_json(args.inputs, "--inputs") @@ -1199,6 +1501,9 @@ def main() -> int: # The lot grid the last _make_state() applied (the body run's), for # applied_runtime["syminfo"]; {} when none. syminfo_applied: dict = {} + # Other symbols' bars, read once before the first state (None without + # --symbol-feeds). + symbol_feeds = None def _make_state(): """Create + fully configure a fresh strategy state — everything EXCEPT the @@ -1218,6 +1523,12 @@ def _make_state(): except SyminfoError: lib.strategy_free(st) raise + if symbol_feeds: + try: + install_symbol_feeds(lib, st, symbol_feeds) + except SymbolFeedsError: + lib.strategy_free(st) + raise if args.trade_start_ms is not None and hasattr(lib, "strategy_set_trade_start_time"): lib.strategy_set_trade_start_time(st, int(args.trade_start_ms)) if args.chart_tz and hasattr(lib, "strategy_set_chart_timezone"): @@ -1240,6 +1551,8 @@ def _run(st, rep): # already happened above (load_strategy), outside any loop. timing = None try: + if args.symbol_feeds: + symbol_feeds = load_symbol_feeds(args.symbol_feeds) if args.bench: warmup = max(0, int(args.warmup)) repeats = max(1, int(args.repeats)) @@ -1265,8 +1578,9 @@ def _run(st, rep): # --- Body run: one configured run for trades / metrics / diagnostics. --- state = _make_state() - except SyminfoError as e: - # A rejected --syminfo: the structured failure, before any stdout. + except (SyminfoError, SymbolFeedsError) as e: + # A rejected --syminfo or --symbol-feeds: the structured failure, before + # any stdout. json.dump({"engine": "pineforge", "error": str(e)}, sys.stdout, separators=(",", ":")) sys.stdout.write("\n") @@ -1301,6 +1615,8 @@ def _run(st, rep): } if syminfo_applied: applied_runtime["syminfo"] = syminfo_applied + if symbol_feeds: + applied_runtime["symbol_feeds"] = symbol_feeds_record(symbol_feeds) incarnation_accessor = getattr( lib, "strategy_closed_trade_entry_incarnation", None) trade_entry_incarnations = ( diff --git a/docker/run_json_symbol_feeds_test.py b/docker/run_json_symbol_feeds_test.py new file mode 100644 index 00000000..14436ae2 --- /dev/null +++ b/docker/run_json_symbol_feeds_test.py @@ -0,0 +1,422 @@ +import base64 +import contextlib +import ctypes +import io +import json +import math +import sys +from pathlib import Path + +import pytest +import run_json # docker/ is on sys.path in the engine test env + + +# --symbol-feeds: other symbols' bars for request.security, keyed by the exact +# symbol string and timeframe, installed through strategy_set_symbol_facts and +# strategy_set_symbol_feed before the run; absent changes nothing. + +CORE = ("pf_abi_version", "strategy_create", "strategy_set_input", "strategy_set_override", + "run_backtest_full", "strategy_free", "report_free") +FEED_SETTERS = ("strategy_set_symbol_facts", "strategy_set_symbol_feed") +ST = 7 +H4 = 4 * 3600 * 1000 +DAY = 24 * 3600 * 1000 +T0 = 1759536000000 # 2025-10-04T00:00:00Z +ETH = {"ticker": "ETHUSDT", "tickerid": "BINANCE:ETHUSDT", "type": "crypto", + "currency": "USDT", "basecurrency": "ETH", "mintick": 0.01, "pointvalue": 1.0, + "mincontract": 0.0001, "session": "24x7", "timezone": "UTC", "country": None} +ETH_FACTS = [(b"canonical", b"BINANCE:ETHUSDT"), (b"type", b"crypto"), (b"timezone", b"UTC"), + (b"session", b"24x7"), (b"currency", b"USDT"), (b"mintick", b"0.01")] + + +class FakeLib: + """A strategy library stand-in: each named symbol records (name, *args) and + returns its configured result; a name left out fails hasattr like a missing + export.""" + + def __init__(self, names, returns=None): + self.calls = [] + for name in names: + setattr(self, name, self._fn(name, (returns or {}).get(name))) + + def _fn(self, name, result): + def call(*args): + self.calls.append((name,) + args) + return result + return call + + +def fake_lib(setters=FEED_SETTERS, returns=None): + rets = {"pf_abi_version": run_json.EXPECTED_PF_ABI, + **{name: 0 for name in FEED_SETTERS}, **(returns or {})} + return FakeLib(CORE + tuple(setters), rets) + + +def csv_text(rows, header="timestamp,open,high,low,close,volume"): + return header + "\n" + "".join(",".join(str(v) for v in row) + "\n" for row in rows) + + +def bars(start, step, count, base=100.0): + return [(start + i * step, base + i, base + i + 2, base + i - 1, base + i + 1, 10 + i) + for i in range(count)] + + +def index(tmp_path, symbols, files=None, name="symbols.json"): + for fname, text in (files or {}).items(): + (tmp_path / fname).write_text(text) + p = tmp_path / name + p.write_text(symbols if isinstance(symbols, str) else json.dumps({"symbols": symbols})) + return p + + +def eth_index(tmp_path, syminfo=ETH): + entry = {"feeds": {"240": "eth-240.csv", "D": "eth-1D.csv"}} + if syminfo is not None: + entry["syminfo"] = syminfo + return index(tmp_path, {"BINANCE:ETHUSDT": entry}, + {"eth-240.csv": csv_text(bars(T0, H4, 3)), + "eth-1D.csv": csv_text(bars(T0 - DAY, DAY, 2, base=200.0))}) + + +def feed_calls(lib): + out = [] + for call in lib.calls: + if call[0] == "strategy_set_symbol_feed": + _, st, key, tf, arr, close, n = call + out.append((st, key, tf, [(arr[i].timestamp, arr[i].open, arr[i].high, + arr[i].low, arr[i].close, arr[i].volume) + for i in range(n)], list(close[:n]), n)) + return out + + +def test_facts_then_feeds_keyed_by_the_exact_string_and_canonical_timeframe(tmp_path): + lib = fake_lib() + symbols = run_json.load_symbol_feeds(eth_index(tmp_path)) + run_json.install_symbol_feeds(lib, ST, symbols) + facts = [c for c in lib.calls if c[0] == "strategy_set_symbol_facts"] + assert facts == [("strategy_set_symbol_facts", ST, b"BINANCE:ETHUSDT", f, v) + for f, v in ETH_FACTS] + assert [c[0] for c in lib.calls] == ["strategy_set_symbol_facts"] * 6 \ + + ["strategy_set_symbol_feed"] * 2 + (_, k1, tf1, rows1, close1, n1), (_, k2, tf2, rows2, close2, n2) = feed_calls(lib) + assert (k1, tf1, n1, k2, tf2, n2) == (b"BINANCE:ETHUSDT", b"240", 3, + b"BINANCE:ETHUSDT", b"1D", 2) + assert rows1[0] == (T0, 100.0, 102.0, 99.0, 101.0, 10.0) + # No time_close column: each close is the open plus the timeframe. + assert close1 == [T0 + H4, T0 + 2 * H4, T0 + 3 * H4] + assert close2 == [T0, T0 + DAY] + + +def test_record_names_what_was_installed(tmp_path): + rec = run_json.symbol_feeds_record(run_json.load_symbol_feeds(eth_index(tmp_path))) + eth = rec["symbols"]["BINANCE:ETHUSDT"] + assert rec["canonicalization"] == "pf-symbol-feed-barc-close-le-v1" + assert eth["facts"] == {"canonical": "BINANCE:ETHUSDT", "type": "crypto", + "timezone": "UTC", "session": "24x7", "currency": "USDT", + "mintick": 0.01} + assert sorted(eth["feeds"]) == ["1D", "240"] + assert eth["feeds"]["240"]["bars"] == 3 + assert (eth["feeds"]["240"]["first_ts"], eth["feeds"]["240"]["last_ts"]) == (T0, T0 + 2 * H4) + assert len(eth["feeds"]["240"]["source_values_sha256"]) == 64 + + +def test_without_syminfo_only_feeds_are_installed(tmp_path): + lib = fake_lib() + run_json.install_symbol_feeds(lib, ST, run_json.load_symbol_feeds(eth_index(tmp_path, None))) + assert [c[0] for c in lib.calls] == ["strategy_set_symbol_feed"] * 2 + + +def test_a_wrapped_syminfo_is_read_like_a_flat_one(tmp_path): + flat = run_json.load_symbol_feeds(eth_index(tmp_path))[0]["facts"] + wrapped = run_json.load_symbol_feeds(eth_index(tmp_path, {"syminfo": ETH}))[0]["facts"] + assert flat == wrapped + + +def test_time_close_column_gives_each_close(tmp_path): + rows = [(T0, 1, 2, 0.5, 1.5, "", T0 + 6 * 3600 * 1000), + (T0 + DAY, 1, 2, 0.5, 1.5, "NaN", T0 + DAY + 6 * 3600 * 1000)] + p = index(tmp_path, {"NASDAQ:QQQ": {"feeds": {"1D": "qqq.csv"}}}, + {"qqq.csv": csv_text(rows, "timestamp,open,high,low,close,volume,time_close")}) + lib = fake_lib() + run_json.install_symbol_feeds(lib, ST, run_json.load_symbol_feeds(p)) + (_, key, tf, got, close, n), = feed_calls(lib) + assert (key, tf, n) == (b"NASDAQ:QQQ", b"1D", 2) + assert close == [T0 + 6 * 3600 * 1000, T0 + DAY + 6 * 3600 * 1000] + assert all(math.isnan(r[5]) for r in got) # empty or NaN volume: none published + + +@pytest.mark.parametrize("tf,opens,closes", [ + ("1M", [1767225600000, 1769904000000], [1769904000000, 1772323200000]), # Jan, Feb 2026 + ("1W", [T0], [T0 + 7 * DAY]), + ("15", [T0], [T0 + 15 * 60 * 1000]), + ("30S", [T0], [T0 + 30 * 1000]), +]) +def test_derived_close_per_timeframe(tmp_path, tf, opens, closes): + rows = [(t, 1, 2, 0.5, 1.5, 3) for t in opens] + p = index(tmp_path, {"X:Y": {"feeds": {tf: "f.csv"}}}, {"f.csv": csv_text(rows)}) + lib = fake_lib() + run_json.install_symbol_feeds(lib, ST, run_json.load_symbol_feeds(p)) + assert feed_calls(lib)[0][4] == closes + + +def test_prefix_and_suffix_are_part_of_the_key(tmp_path): + feed = csv_text(bars(T0, H4, 2)) + p = index(tmp_path, {s: {"feeds": {"240": "f.csv"}} + for s in ("ETHUSDT", "BINANCE:ETHUSDT", "BINANCE:ETHUSDT.P")}, + {"f.csv": feed}) + lib = fake_lib() + run_json.install_symbol_feeds(lib, ST, run_json.load_symbol_feeds(p)) + assert [c[1] for c in feed_calls(lib)] == [b"ETHUSDT", b"BINANCE:ETHUSDT", + b"BINANCE:ETHUSDT.P"] + + +GOOD = csv_text(bars(T0, H4, 2)) + + +@pytest.mark.parametrize("symbols,files,message", [ + ({"E": {"feeds": {"4h": "f.csv"}}}, {"f.csv": GOOD}, 'a timeframe is whole minutes'), + ({"E": {"feeds": {"60m": "f.csv"}}}, {"f.csv": GOOD}, 'got "60m"'), + ({"E": {"feeds": {"D": "f.csv", "1D": "f.csv"}}}, {"f.csv": GOOD}, "two feeds at timeframe 1D"), + ({"E": {"feed": {"240": "f.csv"}}}, {"f.csv": GOOD}, "an entry is"), + ({"E": {"feeds": ["f.csv"]}}, {"f.csv": GOOD}, "feeds must be an object"), + ({"": {"feeds": {"240": "f.csv"}}}, {"f.csv": GOOD}, "a symbol must be a non-empty string"), + ({"E\n": {"feeds": {"240": "f.csv"}}}, {"f.csv": GOOD}, "without control characters"), + ({"E": {"feeds": {"240": ""}}}, {}, "the feed must name a CSV file"), + ({"E": {"feeds": {"240": "nope.csv"}}}, {}, "nope.csv"), + ({"E": {"feeds": {"240": "f.csv"}}}, {"f.csv": "timestamp,open,high,low\n1,2,3,4\n"}, + "no column close"), + ({"E": {"feeds": {"240": "f.csv"}}}, {"f.csv": csv_text([(T0, "x", 1, 1, 1, 1)])}, + "line 2: not a number"), + ({"E": {"feeds": {"240": "f.csv"}}}, {"f.csv": csv_text([(T0, "inf", 1, 1, 1, 1)])}, + "line 2: prices must be finite"), + ({"E": {"feeds": {"240": "f.csv"}}}, {"f.csv": csv_text([(T0, 1, 1, 1, 1, -5)])}, + "volume nonnegative"), + ({"E": {"feeds": {"240": "f.csv"}}}, + {"f.csv": csv_text([(T0 + H4, 1, 1, 1, 1, 1), (T0, 1, 1, 1, 1, 1)])}, + "line 3: timestamps must increase"), + # 4h bars declared as 8h: each close lands after the next bar's open. + ({"E": {"feeds": {"480": "f.csv"}}}, {"f.csv": GOOD}, "is the timeframe right?"), + ({"E": {"syminfo": {"mintick": 0}, "feeds": {"240": "f.csv"}}}, {"f.csv": GOOD}, + "syminfo.mintick must be a positive finite number, got 0"), + ({"E": {"syminfo": {"mintick": "0.01"}, "feeds": {"240": "f.csv"}}}, {"f.csv": GOOD}, + 'got "0.01"'), + ({"E": {"syminfo": [], "feeds": {"240": "f.csv"}}}, {"f.csv": GOOD}, + "syminfo must be an object"), + ({"E": {"syminfo": {"session": 5}, "feeds": {"240": "f.csv"}}}, {"f.csv": GOOD}, + "syminfo.session must be a non-empty string"), +], ids=["4h", "60m", "D-and-1D", "unknown-entry-key", "feeds-list", "empty-symbol", + "control-char", "empty-path", "missing-file", "missing-column", "not-a-number", + "infinite-price", "negative-volume", "decreasing", "wrong-timeframe", "mintick-zero", + "mintick-string", "syminfo-list", "session-number"]) +def test_bad_index_or_feed_is_refused_by_name(tmp_path, symbols, files, message): + with pytest.raises(run_json.SymbolFeedsError, match=None) as e: + run_json.load_symbol_feeds(index(tmp_path, symbols, files)) + assert message in str(e.value) + assert str(e.value).startswith("--symbol-feeds") + + +@pytest.mark.parametrize("text,message", [ + ("[]", 'the index must be {"symbols": {...}}'), + ('{"symbol": {}}', 'the index must be {"symbols": {...}}'), + ("{not json", "is not JSON"), + ('{"symbols": {"E": {"feeds": {}}, "E": {"feeds": {}}}}', 'duplicate key "E"'), +], ids=["list", "no-symbols", "not-json", "duplicate-key"]) +def test_bad_index_document_is_refused(tmp_path, text, message): + with pytest.raises(run_json.SymbolFeedsError) as e: + run_json.load_symbol_feeds(index(tmp_path, text)) + assert message in str(e.value) + + +def test_more_than_256_feeds_is_refused(tmp_path): + symbols = {f"S{i}": {"feeds": {"240": "f.csv"}} for i in range(257)} + with pytest.raises(run_json.SymbolFeedsError, match="more than 256"): + run_json.load_symbol_feeds(index(tmp_path, symbols, {"f.csv": GOOD})) + + +@pytest.mark.parametrize("setters", [FEED_SETTERS[:1], FEED_SETTERS[1:], ()], + ids=["no-feed", "no-facts", "neither"]) +def test_a_library_without_the_setters_is_refused(tmp_path, setters): + lib = fake_lib(setters) + with pytest.raises(run_json.SymbolFeedsError, match="cannot be installed"): + run_json.install_symbol_feeds(lib, ST, run_json.load_symbol_feeds(eth_index(tmp_path))) + assert not [c for c in lib.calls if c[0] in FEED_SETTERS] + + +def test_an_engine_refusal_names_the_feed_and_the_engine_error(tmp_path): + lib = fake_lib(FEED_SETTERS + ("strategy_get_last_error",), + {"strategy_set_symbol_feed": -1, "strategy_get_last_error": b"bars must increase"}) + with pytest.raises(run_json.SymbolFeedsError) as e: + run_json.install_symbol_feeds(lib, ST, run_json.load_symbol_feeds(eth_index(tmp_path))) + assert str(e.value) == ("--symbol-feeds: the engine refused the feed BINANCE:ETHUSDT@240: " + "bars must increase") + + +def test_load_strategy_declares_the_setter_signatures(monkeypatch): + monkeypatch.setattr(ctypes, "CDLL", lambda path: fake_lib()) + lib = run_json.load_strategy(Path("fake.so")) + assert lib.strategy_set_symbol_feed.argtypes == [ + ctypes.c_void_p, ctypes.c_char_p, ctypes.c_char_p, ctypes.POINTER(run_json.BarC), + ctypes.POINTER(ctypes.c_int64), ctypes.c_int] + assert lib.strategy_set_symbol_feed.restype is ctypes.c_int + assert lib.strategy_set_symbol_facts.argtypes == [ + ctypes.c_void_p, ctypes.c_char_p, ctypes.c_char_p, ctypes.c_char_p] + + +# main() against the fake library. + +@pytest.fixture +def harness(tmp_path, monkeypatch): + tape = tmp_path / "tape.csv" + tape.write_text("open,high,low,close,volume,timestamp\n" + "1,2,0.5,1.5,10,1000\n1.5,3,1,2.5,20,2000\n") + + def run(lib, *extra): + argv = ["run_json.py", "--so", "fake.so", "--ohlcv", str(tape), *extra] + monkeypatch.setattr(ctypes, "CDLL", lambda path: lib) + monkeypatch.setattr(sys, "argv", argv) + out = io.StringIO() + with contextlib.redirect_stdout(out): + status = run_json.main() + return status, out.getvalue() + + return run + + +def report_of(text): + rep = json.loads(text) + rep["elapsed_seconds"] = 0 + return rep + + +def count(lib, name): + return [c[0] for c in lib.calls].count(name) + + +def test_main_without_feeds_or_with_an_empty_index_reports_as_before(harness, tmp_path): + _, plain = harness(fake_lib()) + status, out = harness(fake_lib(), "--symbol-feeds", str(index(tmp_path, {}))) + assert status == 0 + assert report_of(out) == report_of(plain) + assert "symbol_feeds" not in report_of(out)["applied_runtime"] + + +def test_main_installs_feeds_and_records_them_in_the_fingerprint(harness, tmp_path): + _, plain = harness(fake_lib()) + lib = fake_lib() + status, out = harness(lib, "--symbol-feeds", str(eth_index(tmp_path))) + rep = report_of(out) + token = base64.b64decode(rep["fingerprint"]["token"]) + assert status == 0 + assert count(lib, "strategy_set_symbol_feed") == 2 + # Installed on the state before its run. + names = [c[0] for c in lib.calls] + assert names.index("strategy_set_symbol_feed") < names.index("run_backtest_full") + recorded = rep["applied_runtime"]["symbol_feeds"] + assert sorted(recorded["symbols"]["BINANCE:ETHUSDT"]["feeds"]) == ["1D", "240"] + assert json.loads(token)["runtime"]["symbol_feeds"] == recorded + assert rep["fingerprint"]["digest"] != report_of(plain)["fingerprint"]["digest"] + + +def test_main_bench_installs_feeds_on_every_state(harness, tmp_path): + lib = fake_lib() + status, _ = harness(lib, "--bench", "--warmup", "2", "--repeats", "3", + "--symbol-feeds", str(eth_index(tmp_path))) + states = 2 + 3 + 1 + assert status == 0 + assert count(lib, "strategy_create") == count(lib, "strategy_free") == states + assert count(lib, "strategy_set_symbol_feed") == 2 * states + + +def test_main_refuses_a_bad_index_with_one_structured_line_before_any_state(harness, tmp_path): + lib = fake_lib() + bad = index(tmp_path, {"E": {"feeds": {"4h": "f.csv"}}}, {"f.csv": GOOD}) + status, out = harness(lib, "--symbol-feeds", str(bad)) + assert status == 1 + assert out.endswith("\n") and out.count("\n") == 1 + assert json.loads(out) == {"engine": "pineforge", "error": + '--symbol-feeds: a timeframe is whole minutes ("15", "240") ' + 'or D|W|M|S ("1D", "1W"), got "4h"'} + assert count(lib, "strategy_create") == count(lib, "run_backtest_full") == 0 + + +def test_main_without_the_setters_fails_and_frees_the_state(harness, tmp_path): + lib = fake_lib(()) + status, out = harness(lib, "--symbol-feeds", str(eth_index(tmp_path))) + assert status == 1 + assert json.loads(out)["error"].startswith("--symbol-feeds: the strategy library has no " + "strategy_set_symbol_facts, strategy_set_symbol_feed") + assert count(lib, "strategy_create") == count(lib, "strategy_free") == 1 + assert count(lib, "run_backtest_full") == 0 + + +def test_a_header_only_feed_is_installed_without_bars(tmp_path): + # The engine's contract: a feed without bars, whose requests read na. + p = index(tmp_path, {"X:Y": {"feeds": {"60": "f.csv"}}}, {"f.csv": csv_text([])}) + lib = fake_lib() + symbols = run_json.load_symbol_feeds(p) + run_json.install_symbol_feeds(lib, ST, symbols) + assert [(c[2], c[3], c[6]) for c in lib.calls if c[0] == "strategy_set_symbol_feed"] \ + == [(b"X:Y", b"60", 0)] + record = run_json.symbol_feeds_record(symbols)["symbols"]["X:Y"]["feeds"]["60"] + assert record["bars"] == 0 and "first_ts" not in record + + +def test_an_empty_time_close_cell_falls_back_to_open_plus_timeframe(tmp_path): + rows = [(T0, 1, 2, 0.5, 1.5, 3, ""), (T0 + H4, 1, 2, 0.5, 1.5, 3, T0 + H4 + 3600 * 1000)] + p = index(tmp_path, {"X:Y": {"feeds": {"240": "f.csv"}}}, + {"f.csv": csv_text(rows, "timestamp,open,high,low,close,volume,time_close")}) + lib = fake_lib() + run_json.install_symbol_feeds(lib, ST, run_json.load_symbol_feeds(p)) + assert feed_calls(lib)[0][4] == [T0 + H4, T0 + H4 + 3600 * 1000] + + +@pytest.mark.parametrize("opens,closes", [ + ([1764547200000], [1767225600000]), # 2025-12-01 -> 2026-01-01 + ([1769817600000], [1772236800000]), # 2026-01-31 -> 2026-02-28 (day clamped) + ([1767225600123], [1769904000123]), # milliseconds kept +], ids=["dec-jan", "clamp", "ms"]) +def test_month_close_edges(tmp_path, opens, closes): + p = index(tmp_path, {"X:Y": {"feeds": {"1M": "f.csv"}}}, + {"f.csv": csv_text([(t, 1, 2, 0.5, 1.5, 3) for t in opens])}) + lib = fake_lib() + run_json.install_symbol_feeds(lib, ST, run_json.load_symbol_feeds(p)) + assert feed_calls(lib)[0][4] == closes + + +def test_a_utf8_bom_is_read(tmp_path): + p = index(tmp_path, {"X:Y": {"feeds": {"240": "f.csv"}}}) + (tmp_path / "f.csv").write_bytes(b"\xef\xbb\xbf" + GOOD.encode()) + assert run_json.load_symbol_feeds(p)[0]["feeds"][0]["n"] == 2 + + +def test_line_numbers_count_blank_lines(tmp_path): + text = csv_text([(T0, 1, 2, 0.5, 1.5, 3)]) + "\n" + f"{T0 + H4},x,2,0.5,1.5,3\n" + with pytest.raises(run_json.SymbolFeedsError, match="line 4: not a number"): + run_json.load_symbol_feeds(index(tmp_path, {"X:Y": {"feeds": {"240": "f.csv"}}}, + {"f.csv": text})) + + +@pytest.mark.parametrize("tf,make,message", [ + ("240", lambda p: p.write_bytes(b"timestamp,open,high,low,close,volume\n1,caf\xe9,1,1,1,1\n"), + "not a UTF-8 CSV"), + ("240", lambda p: p.write_text(csv_text([(2**53, 1, 1, 1, 1, 1)])), "within +-9007199254740991"), + # Microsecond stamps read as milliseconds: a monthly close beyond the calendar. + ("1M", lambda p: p.write_text(csv_text([(1759536000000000, 1, 1, 1, 1, 1)])), "within"), +], ids=["cp1252", "beyond-2^53", "microseconds-month"]) +def test_undecodable_or_out_of_range_feeds_are_refused(tmp_path, tf, make, message): + make(tmp_path / "f.csv") + with pytest.raises(run_json.SymbolFeedsError) as e: + run_json.load_symbol_feeds(index(tmp_path, {"X:Y": {"feeds": {tf: "f.csv"}}})) + assert message in str(e.value) + + +@pytest.mark.parametrize("raw,message", [ + ('{"symbols": {"\\ud800": {"feeds": {}}}}', "a symbol must be a non-empty string"), + ('{"symbols": {"E": {"syminfo": {"mintick": 1' + "0" * 400 + '}, "feeds": {}}}}', + "syminfo.mintick must be a positive finite number"), + ("[" * 100000 + "]" * 100000, "is not JSON"), +], ids=["lone-surrogate", "huge-mintick", "deep-nesting"]) +def test_index_values_that_would_raise_are_refused(tmp_path, raw, message): + with pytest.raises(run_json.SymbolFeedsError) as e: + run_json.load_symbol_feeds(index(tmp_path, raw)) + assert message in str(e.value)