Support engine 1.0: in-process runner reads C ABI 4, default runtime pineforge-release 1.0.0

README.md

@@ -100,10 +100,11 @@ pineforge-backtest \
   --pretty
 ```
 
-The first run pulls the digest-pinned `pineforge-release` 0.1.12 image (engine
-0.11.0, codegen 0.9.0). Release 1.0.0 (engine 1.0.0, codegen 1.0.0) also runs
-this example when its digest-pinned image is passed to `--runtime-image`; see the
-[runtime image policy](https://pineforge-4pass.github.io/pineforge-data/backtesting/#runtime-image-policy).
+The first run pulls the digest-pinned `pineforge-release` 1.0.0 image (engine
+1.0.0, codegen 1.0.0). Results can differ between releases, notably for scripts
+that omit their capital or order size; the
+[runtime image policy](https://pineforge-4pass.github.io/pineforge-data/backtesting/#runtime-image-policy)
+explains why and how to pin another release with `--runtime-image`.
 The JSON report includes trades, performance
 statistics, the equity curve, data provenance, and exact runtime versions. The
 strategy is compiled inside an isolated Docker container; the provider and its

docker/server.Dockerfile

@@ -1,6 +1,6 @@
 # syntax=docker/dockerfile:1@sha256:87999aa3d42bdc6bea60565083ee17e86d1f3339802f543c0d03998580f9cb89
 
-ARG PINEFORGE_RELEASE_IMAGE=ghcr.io/pineforge-4pass/pineforge-release:0.1.12@sha256:312b9d908390b828484617472c749d5815feb75507da87eae2f6902cfe3d47b1
+ARG PINEFORGE_RELEASE_IMAGE=ghcr.io/pineforge-4pass/pineforge-release:1.0.0@sha256:b271873d0e91a9eeeb4595daaded9bdfbdbcd3925d3ec1300a833395e2004c86
 
 FROM python:3.11-slim-bookworm@sha256:f5cf0344c9886ff24d34797578d5d7dd6e8911ae0fe5962bb55d0f89603ec361 AS server-builder
 RUN python -m pip install --no-cache-dir --target /opt/pineforge-server-deps \

docs/api/backtesting.md

@@ -8,13 +8,25 @@
 
 ::: pineforge_data.backtest.BacktestReport
 
-`PineForgeBacktestRunner` accepts only strategy libraries that report PineForge
-C ABI 2, which engine releases v0.10.2 through v0.12.3 build. It refuses
-libraries built with engine v0.13.x (ABI 3) or v1.0.0 (ABI 4) with an
-`EngineBacktestError` such as "PineForge ABI mismatch: strategy reports 4,
-expected 2". The release container and the FastAPI server do not use this
+`PineForgeBacktestRunner` runs a compiled strategy library in-process. It
+accepts libraries that report PineForge C ABI 4, which engine 1.x builds. It
+refuses any other library before any strategy call, with an
+`EngineBacktestError` that says what to do. Libraries without
+`pf_abi_version` (engine v0.10.1 or earlier), ABI 2 (engine v0.10.2 through
+v0.12.3) and ABI 3 (engine v0.13.x) need a rebuild with engine 1.x;
+pineforge-data 0.2.0 still reads ABI 2. A newer ABI needs a newer
+pineforge-data. The release container and the FastAPI server do not use this
 class.
 
+A trade's `max_runup` and `max_drawdown` are whole-trade excursions in account
+currency, net of entry fees, as engine 1.0's `pineforge.h` documents them.
+Headers before 1.0 described them as price travel per unit of quantity, but the
+values did not change: on the README quick start, engines 0.11.0 and 1.0.0
+report identical excursions for the 88 trades both report. A trade whose
+`open_at_end` is true closes a position still open after the final bar, at that
+bar's close rounded to the tick size, without slippage. Engine 0.11.0 reports no
+such row, so the README quick start has 88 trades on it and 89 on engine 1.0.0.
+
 ::: pineforge_data.backtest.PineForgeBacktestRunner
 
 ::: pineforge_data.backtest.EngineBacktestError

docs/backtesting.md

@@ -133,19 +133,28 @@ guides.
 ## Runtime image policy
 
 The package default is an exact `pineforge-release` version and OCI digest:
-0.1.12, which bundles engine 0.11.0 and codegen 0.9.0. The `missing` pull
+1.0.0, which bundles engine 1.0.0 and codegen 1.0.0. The `missing` pull
 policy downloads it only when absent; `never` supports offline runs; `always`
 refreshes a tag before running.
 
-Release 1.0.0 bundles engine 1.0.0 and codegen 1.0.0 and accepts the same
-inputs. To use it, pin it by digest:
+The previous default, release 0.1.12 (engine 0.11.0, codegen 0.9.0), accepts the
+same inputs. To reproduce results made with it, pin it by digest:
 
 ```bash
---runtime-image ghcr.io/pineforge-4pass/pineforge-release:1.0.0@sha256:b271873d0e91a9eeeb4595daaded9bdfbdbcd3925d3ec1300a833395e2004c86
+--runtime-image ghcr.io/pineforge-4pass/pineforge-release:0.1.12@sha256:312b9d908390b828484617472c749d5815feb75507da87eae2f6902cfe3d47b1
 ```
 
-Results can differ between releases: the README quick start reports 88 trades
-on 0.1.12 and 89 on 1.0.0.
+Results can differ between releases: the README quick start reports 89 trades
+on 1.0.0 and 88 on 0.1.12. The first 88 are the same trades; 1.0.0 adds the
+close of the position still open after the last bar, marked `open_at_end`.
+
+Order sizes differ too when a script omits `initial_capital`,
+`default_qty_type` or `default_qty_value`. Codegen 1.0 gives them TradingView's
+Pine v6 defaults: 100000, `strategy.percent_of_equity` and 100. Release 0.1.12
+used 1000000, `strategy.fixed` and 1. At 100% of equity, the default 100% margin
+refuses an entry that fills above its signal price, so such a script can trade
+less on 1.0.0. Declare all three settings, as the README strategy does, to size
+orders the same way on both releases.
 
 The rolling channel is explicit:
 

docs/server.md

@@ -64,17 +64,18 @@ and `generated_cpp_sha256`.
 
 ## Runtime channel
 
-The Dockerfile defaults to a semver-and-digest-pinned release image. Override it
-at build time for a rolling development server:
+The Dockerfile defaults to a semver-and-digest-pinned release image: 1.0.0
+(engine 1.0.0, codegen 1.0.0). Override it at build time for a rolling
+development server:
 
 ```bash
 docker build -f docker/server.Dockerfile \
   --build-arg PINEFORGE_RELEASE_IMAGE=ghcr.io/pineforge-4pass/pineforge-release:latest \
   -t pineforge-data-server:latest .
 ```
 
-To pin release 1.0.0 (engine 1.0.0, codegen 1.0.0) instead, pass
-`--build-arg PINEFORGE_RELEASE_IMAGE=ghcr.io/pineforge-4pass/pineforge-release:1.0.0@sha256:b271873d0e91a9eeeb4595daaded9bdfbdbcd3925d3ec1300a833395e2004c86`.
+To build on the previous release, 0.1.12 (engine 0.11.0, codegen 0.9.0), pass
+`--build-arg PINEFORGE_RELEASE_IMAGE=ghcr.io/pineforge-4pass/pineforge-release:0.1.12@sha256:312b9d908390b828484617472c749d5815feb75507da87eae2f6902cfe3d47b1`.
 
 Do not use the rolling channel for reproducibility-sensitive production runs.
 The cache key includes component/release identity, so a runtime upgrade cannot

pyproject.toml

@@ -46,7 +46,7 @@ server = [
 ]
 release = [
     "build>=1.2,<2",
-    "twine>=6.2,<7",
+    "twine>=7,<8",
 ]
 dev = [
     "build>=1.2",

src/pineforge_data/backtest.py

@@ -18,7 +18,15 @@
 JsonScalar: TypeAlias = str | int | float | bool | None
 JsonValue: TypeAlias = JsonScalar | list["JsonValue"] | dict[str, "JsonValue"]
 
-EXPECTED_PF_ABI = 2
+EXPECTED_PF_ABI = 4
+# The engine releases behind each refused pre-1.0 C ABI, and the way forward.
+_REFUSED_ABI_HINTS = {
+    2: (
+        "engine v0.10.2 to v0.12.3 built it: rebuild the strategy with engine 1.x, "
+        "or read it with pineforge-data 0.2.0"
+    ),
+    3: "engine v0.13.0 to v0.13.1 built it: rebuild the strategy with engine 1.x",
+}
 
 
 class EngineBacktestError(RuntimeError):
@@ -64,7 +72,14 @@ def __post_init__(self) -> None:
 
 @dataclass(frozen=True, slots=True)
 class BacktestReport:
-    """A detached, JSON-safe copy of ``pf_report_t``."""
+    """A detached, JSON-safe copy of ``pf_report_t``.
+
+    Each trade's ``max_runup`` and ``max_drawdown`` are the whole-trade favorable
+    and adverse excursions in account currency, net of entry fees. A trade whose
+    ``open_at_end`` is true closes a position that was still open after the final
+    bar, at that bar's close rounded to the tick size, without slippage;
+    ``summary["total_trades"]`` counts it.
+    """
 
     summary: Mapping[str, JsonValue]
     metrics: Mapping[str, Mapping[str, JsonValue]]
@@ -95,12 +110,14 @@ class _PfTrade(ctypes.Structure):
         ("pnl", ctypes.c_double),
         ("pnl_pct", ctypes.c_double),
         ("is_long", ctypes.c_int),
+        # Whole-trade excursions in account currency, net of entry fees.
         ("max_runup", ctypes.c_double),
         ("max_drawdown", ctypes.c_double),
         ("qty", ctypes.c_double),
         ("commission", ctypes.c_double),
         ("entry_bar_index", ctypes.c_int32),
         ("exit_bar_index", ctypes.c_int32),
+        ("open_at_end", ctypes.c_int32),
     ]
 
 
@@ -147,6 +164,8 @@ class _PfEquityStats(ctypes.Structure):
         ("max_equity_runup_pct", ctypes.c_double),
         ("buy_hold_return", ctypes.c_double),
         ("buy_hold_return_pct", ctypes.c_double),
+        # pineforge.h 1.0 names these sharpe_monthly and sortino_monthly at the
+        # same offsets (48 and 56); the report keys stay sharpe_tv and sortino_tv.
         ("sharpe_tv", ctypes.c_double),
         ("sortino_tv", ctypes.c_double),
         ("sharpe_bar", ctypes.c_double),
@@ -221,6 +240,10 @@ class _PfReport(ctypes.Structure):
         ("metrics", _PfMetrics),
         ("equity_curve", ctypes.POINTER(_PfEquityPoint)),
         ("equity_curve_len", ctypes.c_int64),
+        # Per-script-bar broker-state hash (ABI 4). report_free releases it;
+        # BacktestReport does not copy it.
+        ("broker_state_hash", ctypes.POINTER(ctypes.c_uint64)),
+        ("broker_state_hash_len", ctypes.c_int64),
     ]
 
 
@@ -299,7 +322,16 @@ def _decode(value: bytes | None) -> str:
 
 
 class PineForgeBacktestRunner:
-    """Thin owner-safe wrapper around one compiled PineForge strategy library."""
+    """Thin owner-safe wrapper around one compiled PineForge strategy library.
+
+    The library must report PineForge C ABI 4, which engine 1.x builds; like the
+    release image's own harness, the runner requires the exact ABI that
+    ``pineforge.h`` asks every caller to check. Libraries that report ABI 2
+    (engine v0.10.2 to v0.12.3) or ABI 3 (engine v0.13.x) are refused with
+    ``EngineBacktestError`` before any strategy call. An ABI 2 trade row lacks
+    ``open_at_end`` and is 8 bytes shorter, so every row after the first would be
+    misread; an ABI 3 report lacks the broker-state hash.
+    """
 
     def __init__(self, library: ctypes.CDLL) -> None:
         self._library = library
@@ -322,12 +354,17 @@ def _check_abi(self) -> None:
             actual = int(self._library.pf_abi_version())
         except AttributeError as exc:
             raise EngineBacktestError(
-                "strategy library predates pf_abi_version; rebuild it with the current engine"
+                "strategy library predates pf_abi_version (engine v0.10.1 or earlier); "
+                "rebuild it with engine 1.x"
             ) from exc
         if actual != EXPECTED_PF_ABI:
-            raise EngineBacktestError(
+            message = (
                 f"PineForge ABI mismatch: strategy reports {actual}, expected {EXPECTED_PF_ABI}"
             )
+            hint = _REFUSED_ABI_HINTS.get(actual)
+            if hint is None and actual > EXPECTED_PF_ABI:
+                hint = "a newer engine built it: upgrade pineforge-data"
+            raise EngineBacktestError(f"{message}; {hint}" if hint else message)
 
     def _configure_signatures(self) -> None:
         library = self._library
@@ -496,6 +533,7 @@ def _copy_report(self, report: _PfReport) -> BacktestReport:
                     "commission": _json_number(trade.commission),
                     "entry_bar_index": trade.entry_bar_index,
                     "exit_bar_index": trade.exit_bar_index,
+                    "open_at_end": bool(trade.open_at_end),
                 }
             )
 
@@ -548,5 +586,9 @@ def _copy_report(self, report: _PfReport) -> BacktestReport:
         )
 
 
-if ctypes.sizeof(_PfReport) != 944 or _PfReport.metrics.offset != 160:
+if (
+    ctypes.sizeof(_PfReport) != 960
+    or _PfReport.metrics.offset != 160
+    or ctypes.sizeof(_PfTrade) != 104
+):
     raise RuntimeError("unsupported platform ABI layout for PineForge reports")

src/pineforge_data/release_contract.py

@@ -12,8 +12,8 @@
 from .models import Bar, Instrument
 
 DEFAULT_RELEASE_IMAGE = (
-    "ghcr.io/pineforge-4pass/pineforge-release:0.1.12@"
-    "sha256:312b9d908390b828484617472c749d5815feb75507da87eae2f6902cfe3d47b1"
+    "ghcr.io/pineforge-4pass/pineforge-release:1.0.0@"
+    "sha256:b271873d0e91a9eeeb4595daaded9bdfbdbcd3925d3ec1300a833395e2004c86"
 )
 RELEASE_ENTRYPOINT = "/opt/pineforge/bin/entrypoint.sh"
 RESPONSE_SCHEMA_VERSION = 1
@@ -88,7 +88,7 @@ def release_environment(
     """Translate PineForge options into the release entrypoint environment."""
 
     if options.trace_enabled:
-        raise ReleaseContractError("pineforge-release 0.1.12 does not expose trace collection")
+        raise ReleaseContractError("pineforge-release 1.0.0 does not expose trace collection")
     if options.bar_magnifier and options.magnifier_samples < 2:
         raise ReleaseContractError("bar magnifier requires at least two samples")
     environment = {

tests/fixtures/sma_cross.pine

@@ -1,5 +1,5 @@
 //@version=6
-strategy("Docker SMA cross", initial_capital=10000)
+strategy("Docker SMA cross", initial_capital=10000, default_qty_type=strategy.fixed, default_qty_value=1)
 
 fast = ta.sma(close, 2)
 slow = ta.sma(close, 4)