From c88de077059130a61c5528958ed810b294a0f936 Mon Sep 17 00:00:00 2001 From: luisleo526 Date: Tue, 29 Sep 2026 19:13:59 +0800 Subject: [PATCH 1/3] docs: make README and docs true (runnable quickstart, pinned runtime) - README/getting-started/backtesting/ccxt quickstarts used venue kraken for 2025-07 15m bars; Kraken serves only its latest 720 candles, so step 2 raised IndexError and step 3 exited 1 ("provider returned no confirmed bars"). Switched those examples to bitstamp (live runs: 96 bars, report exit 0) and noted the Kraken history limit in the CCXT guide. - README strategy fixed 1 BTC per order against 10,000 capital (Pine v6 default margin 100%): pinned runtime produced 90 zero-PnL same-bar trades, release 0.1.25 produced none. Size at 10% of equity (88/89 real trades). - README and backtesting guide now name the pinned runtime: pineforge-release 0.1.12, engine 0.11.0, codegen 0.9.0 (release_contract.py DEFAULT_RELEASE_IMAGE + image OCI labels). - Report envelope example: provider_start_ms/first_bar_ms described 5000 warmup bars, not 500 (warmup_request_start_ms and a live run give 1750878000000); added the top-level strategy section the CLI emits. - csv.md infer_bar_mapping({"timestamp": ...}) raised SchemaMappingError on the page's own header; removed that line. - tabular-schema: ISO-8601/datetime sub-ms values are truncated, only numeric ones fail (tabular.py _timestamp_ms; reproduced). - data-model: expiry_ms may be 0; only contract size and strike must be positive (models.py ContractSpec). - server.md: /cache is the image default; code default differs. - api/server.md: the packaged pineforge-backtest-server command serves /readyz 503 and fails backtests (runtime_unavailable) outside the server image; point to the image built from docker/server.Dockerfile. - README SQLAlchemy row needs a dialect driver, as providers.md says. - index.md, pyproject description, mkdocs site_description claimed macro providers/adapters; none ship (only macro contract types), and only bars reach pineforge-release. - CORPUS.md: name the upstream market data (corpus NOTICE) and state the LFS object size in MiB (176,093,499 bytes). Co-Authored-By: Claude Opus 5.5 --- README.md | 17 +++++++++-------- docs/api/server.md | 8 +++++--- docs/backtesting.md | 26 ++++++++++++++++---------- docs/data-model.md | 4 ++-- docs/getting-started.md | 6 +++--- docs/index.md | 6 +++--- docs/providers/ccxt.md | 5 +++-- docs/providers/csv.md | 1 - docs/providers/tabular-schema.md | 7 ++++--- docs/server.md | 2 +- mkdocs.yml | 2 +- pyproject.toml | 2 +- tests/fixtures/CORPUS.md | 4 +++- 13 files changed, 51 insertions(+), 39 deletions(-) diff --git a/README.md b/README.md index ae7c3cc..202567c 100644 --- a/README.md +++ b/README.md @@ -46,7 +46,7 @@ from pineforge_data import BarRequest, CcxtProvider async def main() -> None: - async with CcxtProvider("kraken") as provider: + async with CcxtProvider("bitstamp") as provider: market = await provider.resolve_market("BTC/USD") bars = await provider.fetch_bars( BarRequest( @@ -72,7 +72,7 @@ Save a strategy as `strategy.pine`: ```pinescript //@version=6 -strategy("SMA cross", initial_capital=10000) +strategy("SMA cross", initial_capital=10000, default_qty_type=strategy.percent_of_equity, default_qty_value=10) fast = ta.sma(close, 2) slow = ta.sma(close, 4) @@ -90,7 +90,7 @@ Then run: pineforge-backtest \ --pine strategy.pine \ --provider ccxt \ - --venue kraken \ + --venue bitstamp \ --symbol BTC/USD \ --timeframe 15m \ --start 2025-07-01T00:00:00Z \ @@ -100,10 +100,11 @@ pineforge-backtest \ --pretty ``` -The first run pulls a digest-pinned `pineforge-release` image. The JSON report -includes trades, performance statistics, the equity curve, data provenance, -and exact runtime versions. The strategy is compiled inside an isolated Docker -container; the provider and its credentials remain on the host. +The first run pulls the digest-pinned `pineforge-release` 0.1.12 image (engine +0.11.0, codegen 0.9.0). The JSON report includes trades, performance +statistics, the equity curve, data provenance, and exact runtime versions. The +strategy is compiled inside an isolated Docker container; the provider and its +credentials remain on the host. ## Bring your own data @@ -115,7 +116,7 @@ and accept an explicit mapping when your schema uses different names. |---|---|---| | CSV | `pip install pineforge-data` | [CSV API](https://pineforge-4pass.github.io/pineforge-data/providers/csv/) | | SQLite | `pip install pineforge-data` | [SQLite API](https://pineforge-4pass.github.io/pineforge-data/providers/sqlite/) | -| SQLAlchemy database | `pip install 'pineforge-data[database]'` | [SQLAlchemy API](https://pineforge-4pass.github.io/pineforge-data/providers/sqlalchemy/) | +| SQLAlchemy database | `pip install 'pineforge-data[database]'` plus a database driver | [SQLAlchemy API](https://pineforge-4pass.github.io/pineforge-data/providers/sqlalchemy/) | | CCXT exchange | `pip install 'pineforge-data[ccxt]'` | [CCXT API](https://pineforge-4pass.github.io/pineforge-data/providers/ccxt/) | The same `pineforge-backtest` command supports `--provider csv`, `sqlite`, and diff --git a/docs/api/server.md b/docs/api/server.md index b4e917c..5885055 100644 --- a/docs/api/server.md +++ b/docs/api/server.md @@ -1,8 +1,10 @@ # Server components -Most users run the packaged `pineforge-backtest-server` command or submit work -through `FastApiBacktestClient`. These objects support applications that embed -the service or manage its compiled-strategy cache directly. +Most users run the server image built from `docker/server.Dockerfile` or submit +work through `FastApiBacktestClient`. The packaged `pineforge-backtest-server` +command reports ready only where the `pineforge-release` toolchain is +installed, as in that image. These objects support applications that embed the +service or manage its compiled-strategy cache directly. ::: pineforge_data.server.create_app diff --git a/docs/backtesting.md b/docs/backtesting.md index 8b4bdd9..56de86c 100644 --- a/docs/backtesting.md +++ b/docs/backtesting.md @@ -29,7 +29,7 @@ are documented in the [server guide](server.md). pineforge-backtest \ --pine strategy.pine \ --provider ccxt \ - --venue kraken \ + --venue bitstamp \ --symbol BTC/USD \ --timeframe 15m \ --start 2025-07-01T00:00:00Z \ @@ -60,7 +60,7 @@ strategy to place orders during that earlier interval: pineforge-backtest \ --pine strategy.pine \ --provider ccxt \ - --venue kraken \ + --venue bitstamp \ --symbol BTC/USD \ --timeframe 15m \ --start 2025-07-01T00:00:00Z \ @@ -132,9 +132,10 @@ guides. ## Runtime image policy -The package default is an exact `pineforge-release` version and OCI digest. The -`missing` pull policy downloads it only when absent; `never` supports offline -runs; `always` refreshes a tag before running. +The package default is an exact `pineforge-release` version and OCI digest: +0.1.12, which bundles engine 0.11.0 and codegen 0.9.0. The `missing` pull +policy downloads it only when absent; `never` supports offline runs; `always` +refreshes a tag before running. The rolling channel is explicit: @@ -155,13 +156,13 @@ The harness combines provider provenance with the release report: "schema_version": 1, "request_id": null, "provider": { - "name": "ccxt:kraken", + "name": "ccxt:bitstamp", "adapter": "ccxt", - "venue": "kraken", + "venue": "bitstamp", "source_timeframe": "15m", "market": { "symbol": "BTC/USD", - "provider_id": "XXBTZUSD", + "provider_id": "btcusd", "market_type": "spot", "contract": null } @@ -169,8 +170,8 @@ The harness combines provider provenance with the release report: "data": { "requested_start_ms": 1751328000000, "requested_end_ms": 1751932800000, - "provider_start_ms": 1746828000000, - "first_bar_ms": 1746828000000, + "provider_start_ms": 1750878000000, + "first_bar_ms": 1750878000000, "last_bar_ms": 1751931900000, "bars": 1172, "requested_bars": 672, @@ -178,6 +179,11 @@ The harness combines provider provenance with the release report: "warmup_bars_loaded": 500, "trade_start_time_ms": 1751328000000 }, + "strategy": { + "pine": "/path/to/strategy.pine", + "input_timeframe": "15", + "script_timeframe": "15" + }, "runtime": { "mode": "local-container", "release_image": "ghcr.io/...@sha256:..." diff --git a/docs/data-model.md b/docs/data-model.md index 8364c7a..d9ae952 100644 --- a/docs/data-model.md +++ b/docs/data-model.md @@ -61,8 +61,8 @@ swap = Instrument( ``` `ContractSpec` can also carry `expiry_ms`, `strike`, and `option_type`. A -contract cannot be both linear and inverse, and numeric contract terms must be -finite and positive. +contract cannot be both linear and inverse, and contract size and strike must +be finite and positive. ## Market listings and discovery diff --git a/docs/getting-started.md b/docs/getting-started.md index 4e37509..8ca77c0 100644 --- a/docs/getting-started.md +++ b/docs/getting-started.md @@ -41,7 +41,7 @@ from pineforge_data import BarRequest, CcxtProvider async def main() -> None: - async with CcxtProvider("kraken") as provider: + async with CcxtProvider("bitstamp") as provider: listing = await provider.resolve_market("BTC/USD") instrument = listing.instrument bars = await provider.fetch_bars( @@ -67,7 +67,7 @@ excludes a candle that was not closed at the request's observation time. .venv/bin/pineforge-backtest \ --pine strategy.pine \ --provider ccxt \ - --venue kraken \ + --venue bitstamp \ --symbol BTC/USD \ --timeframe 15m \ --start 2025-07-01T00:00:00Z \ @@ -89,7 +89,7 @@ export PINEFORGE_SERVER_URL=http://127.0.0.1:8000 export PINEFORGE_SERVER_API_KEY=change-me .venv/bin/pineforge-backtest \ --pine strategy.pine \ - --venue kraken \ + --venue bitstamp \ --symbol BTC/USD \ --timeframe 15m \ --start 2025-07-01T00:00:00Z \ diff --git a/docs/index.md b/docs/index.md index bdbc11f..bf2fb94 100644 --- a/docs/index.md +++ b/docs/index.md @@ -1,16 +1,16 @@ # PineForge Data documentation -PineForge Data connects external market and macro providers to deterministic +PineForge Data connects external market data providers to deterministic PineForge backtests. It owns provider transport, market discovery, normalized records, data provenance, and the local or remote handoff to `pineforge-release`. ```text -exchange / broker / macro API +exchange / CSV / database ↓ Python provider adapter ↓ - normalized market + bars + trades + normalized market + bars ↓ local pineforge-release or FastAPI server ↓ diff --git a/docs/providers/ccxt.md b/docs/providers/ccxt.md index 6d38eb2..a4de305 100644 --- a/docs/providers/ccxt.md +++ b/docs/providers/ccxt.md @@ -160,7 +160,7 @@ pineforge-backtest \ --pine strategy.pine \ --provider ccxt \ --provider-config ccxt.json \ - --venue kraken \ + --venue bitstamp \ --symbol BTC/USD \ --timeframe 15m \ --start 2025-07-01T00:00:00Z \ @@ -181,4 +181,5 @@ Python class when endpoint-specific parameters or polling controls are needed. Public trades currently use REST polling, not CCXT Pro WebSockets. Availability, history depth, pagination semantics, and rate limits still depend on the -selected exchange. +selected exchange. For example, Kraken serves only its most recent 720 candles +per timeframe, so an older 15-minute window returns no bars. diff --git a/docs/providers/csv.md b/docs/providers/csv.md index ae3cf66..5116e71 100644 --- a/docs/providers/csv.md +++ b/docs/providers/csv.md @@ -45,7 +45,6 @@ complete mapping is preferable. ```python schema = await provider.inspect_schema() print(schema.column_names) -mapping = schema.infer_bar_mapping({"timestamp": "epoch seconds"}) ``` The first record must be a non-empty, unique header. Files with duplicate or diff --git a/docs/providers/tabular-schema.md b/docs/providers/tabular-schema.md index e11c082..170017c 100644 --- a/docs/providers/tabular-schema.md +++ b/docs/providers/tabular-schema.md @@ -92,8 +92,9 @@ source values: ISO-8601 values with an offset or `Z` are unambiguous. Naive text or database datetimes use `timestamp_timezone`, which defaults to `UTC` and accepts an IANA -zone such as `America/New_York`. Values that cannot be represented exactly at -millisecond precision fail rather than being rounded silently. +zone such as `America/New_York`. Numeric values that cannot be represented +exactly at millisecond precision fail rather than being rounded silently; +ISO-8601 text and date/datetime values are truncated to whole milliseconds. ## Symbol behavior @@ -124,7 +125,7 @@ Tabular providers: - apply `limit` after normalization; - reject duplicate timestamps for the requested symbol and timeframe; - reject missing or non-finite values, invalid OHLC relationships, negative - volume, and lossy sub-millisecond timestamps; + volume, and lossy sub-millisecond numeric timestamps; - attach the requested instrument and provider source to every `Bar`. Rows are assumed to be confirmed snapshots because local files and databases diff --git a/docs/server.md b/docs/server.md index 7efea1f..fb0f355 100644 --- a/docs/server.md +++ b/docs/server.md @@ -54,7 +54,7 @@ concurrent misses for one key are deduplicated. | Environment variable | Default | Meaning | |---|---:|---| -| `PINEFORGE_SERVER_CACHE_DIR` | `/cache` | compiled artifact directory | +| `PINEFORGE_SERVER_CACHE_DIR` | `/cache` in the image | compiled artifact directory | | `PINEFORGE_SERVER_CACHE_MAX_ENTRIES` | `1024` | maximum retained `.so` files | | `PINEFORGE_SERVER_CACHE_MAX_BYTES` | `2147483648` | maximum retained bytes | diff --git a/mkdocs.yml b/mkdocs.yml index 670682f..6bb8215 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -1,5 +1,5 @@ site_name: PineForge Data -site_description: Provider-neutral market and macro data adapters for PineForge +site_description: Provider-neutral market data adapters and raw-PineScript backtest harness for PineForge site_url: https://pineforge-4pass.github.io/pineforge-data/ repo_url: https://github.com/pineforge-4pass/pineforge-data repo_name: pineforge-4pass/pineforge-data diff --git a/pyproject.toml b/pyproject.toml index 3dd09f7..711ac8d 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -5,7 +5,7 @@ build-backend = "hatchling.build" [project] name = "pineforge-data" version = "0.2.0" -description = "Provider-neutral market and macro data adapters for PineForge" +description = "Provider-neutral market data adapters and raw-PineScript backtest harness for PineForge" readme = "README.md" requires-python = ">=3.11" license = { text = "Apache-2.0" } diff --git a/tests/fixtures/CORPUS.md b/tests/fixtures/CORPUS.md index a2a0367..c27065c 100644 --- a/tests/fixtures/CORPUS.md +++ b/tests/fixtures/CORPUS.md @@ -5,11 +5,13 @@ rows copied byte-for-byte from: - repository: `pineforge-4pass/pineforge-corpus`; - file: `data/ohlcv_ETH-USDT-USDT_1m.csv`; +- market data: Binance USDT-M perpetual ETH/USDT-USDT 1-minute bars, per that + repository's `NOTICE`; - Git LFS SHA-256: `db8c1332da093008cfbd063e05db0b33fe8f7fd35d78cf058a366519eb9f6cc5`; - sample SHA-256: `8ba4db0c669e47c11746361586cef8bce6371eeeab5af5e351107bd69451c6af`; - timestamps: `1577836800000` through `1577852100000` (Unix milliseconds). -CI uses this bounded sample to avoid downloading the 168 MB Git LFS object on +CI uses this bounded sample to avoid downloading the 168 MiB Git LFS object on every run. `scripts/run_database_e2e.sh` automatically uses the full sibling corpus file when it is available, while still reading only the requested number of rows. From e75e864751a59d40c12123315d78ab19dc6ffb2c Mon Sep 17 00:00:00 2001 From: luisleo526 Date: Tue, 29 Sep 2026 21:30:37 +0800 Subject: [PATCH 2/3] docs: drop the macro-data package keyword No macro-data provider ships; the package has only the macro contract types (MacroObservation, MacroRequest, MacroDataProvider). The previous commit dropped "macro" from the description; the keyword now matches. Co-Authored-By: Claude Opus 5.5 (1M context) --- pyproject.toml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/pyproject.toml b/pyproject.toml index 711ac8d..664913f 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -10,7 +10,7 @@ readme = "README.md" requires-python = ">=3.11" license = { text = "Apache-2.0" } authors = [{ name = "PineForge contributors" }] -keywords = ["pinescript", "backtesting", "market-data", "macro-data"] +keywords = ["pinescript", "backtesting", "market-data"] classifiers = [ "Development Status :: 3 - Alpha", "License :: OSI Approved :: Apache Software License", From 509b25c13bb5261e503c44f140ea62c9f6646e5e Mon Sep 17 00:00:00 2001 From: luisleo526 Date: Wed, 30 Sep 2026 12:59:03 +0800 Subject: [PATCH 3/3] docs: say which runtimes and engines the harness supports The docs named only the pinned 0.1.12 runtime (engine 0.11.0, codegen 0.9.0). pineforge-release 1.0.0 (engine 1.0.0, codegen 1.0.0) is now the latest release. This revision (c244330) was tested on Linux arm64 with the README steps from this branch: - step 2 prints the bitstamp bar (timestamp_ms 1751413500000, close 105709.0); - step 3 exits 0 on the pinned default (88 trades, all with nonzero PnL, none same-bar, 772 equity points) and with --runtime-image pineforge-release:1.0.0@sha256:b271873d... (89 trades, same checks, same top-level report keys); - DockerBacktestRuntime runs the docker-test fixture on 1.0.0 (24 of 24 bars), and the server image built on the 1.0.0 base serves concurrent requests and reuses its compile cache; - PineForgeBacktestRunner refuses a v1.0.0 strategy library with "PineForge ABI mismatch: strategy reports 4, expected 2": backtest.py pins ABI 2, which engine v0.10.2 through v0.12.3 build. The README, docs/backtesting.md and docs/server.md now show how to run 1.0.0 pinned by digest and that results differ from 0.1.12; docs/api/backtesting.md states the in-process runner's ABI limit. mkdocs build --strict passes. Co-Authored-By: Claude Opus 5.5 (1M context) --- README.md | 5 ++++- docs/api/backtesting.md | 7 +++++++ docs/backtesting.md | 10 ++++++++++ docs/server.md | 3 +++ 4 files changed, 24 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index 202567c..86be998 100644 --- a/README.md +++ b/README.md @@ -101,7 +101,10 @@ pineforge-backtest \ ``` The first run pulls the digest-pinned `pineforge-release` 0.1.12 image (engine -0.11.0, codegen 0.9.0). The JSON report includes trades, performance +0.11.0, codegen 0.9.0). Release 1.0.0 (engine 1.0.0, codegen 1.0.0) also runs +this example when its digest-pinned image is passed to `--runtime-image`; see the +[runtime image policy](https://pineforge-4pass.github.io/pineforge-data/backtesting/#runtime-image-policy). +The JSON report includes trades, performance statistics, the equity curve, data provenance, and exact runtime versions. The strategy is compiled inside an isolated Docker container; the provider and its credentials remain on the host. diff --git a/docs/api/backtesting.md b/docs/api/backtesting.md index e49d31c..75e9211 100644 --- a/docs/api/backtesting.md +++ b/docs/api/backtesting.md @@ -8,6 +8,13 @@ ::: pineforge_data.backtest.BacktestReport +`PineForgeBacktestRunner` accepts only strategy libraries that report PineForge +C ABI 2, which engine releases v0.10.2 through v0.12.3 build. It refuses +libraries built with engine v0.13.x (ABI 3) or v1.0.0 (ABI 4) with an +`EngineBacktestError` such as "PineForge ABI mismatch: strategy reports 4, +expected 2". The release container and the FastAPI server do not use this +class. + ::: pineforge_data.backtest.PineForgeBacktestRunner ::: pineforge_data.backtest.EngineBacktestError diff --git a/docs/backtesting.md b/docs/backtesting.md index 56de86c..02bfa8b 100644 --- a/docs/backtesting.md +++ b/docs/backtesting.md @@ -137,6 +137,16 @@ The package default is an exact `pineforge-release` version and OCI digest: policy downloads it only when absent; `never` supports offline runs; `always` refreshes a tag before running. +Release 1.0.0 bundles engine 1.0.0 and codegen 1.0.0 and accepts the same +inputs. To use it, pin it by digest: + +```bash +--runtime-image ghcr.io/pineforge-4pass/pineforge-release:1.0.0@sha256:b271873d0e91a9eeeb4595daaded9bdfbdbcd3925d3ec1300a833395e2004c86 +``` + +Results can differ between releases: the README quick start reports 88 trades +on 0.1.12 and 89 on 1.0.0. + The rolling channel is explicit: ```bash diff --git a/docs/server.md b/docs/server.md index fb0f355..da2fabb 100644 --- a/docs/server.md +++ b/docs/server.md @@ -73,6 +73,9 @@ docker build -f docker/server.Dockerfile \ -t pineforge-data-server:latest . ``` +To pin release 1.0.0 (engine 1.0.0, codegen 1.0.0) instead, pass +`--build-arg PINEFORGE_RELEASE_IMAGE=ghcr.io/pineforge-4pass/pineforge-release:1.0.0@sha256:b271873d0e91a9eeeb4595daaded9bdfbdbcd3925d3ec1300a833395e2004c86`. + Do not use the rolling channel for reproducibility-sensitive production runs. The cache key includes component/release identity, so a runtime upgrade cannot reuse a compiled artifact from a different engine version.