diff --git a/README.md b/README.md index ae7c3cc..86be998 100644 --- a/README.md +++ b/README.md @@ -46,7 +46,7 @@ from pineforge_data import BarRequest, CcxtProvider async def main() -> None: - async with CcxtProvider("kraken") as provider: + async with CcxtProvider("bitstamp") as provider: market = await provider.resolve_market("BTC/USD") bars = await provider.fetch_bars( BarRequest( @@ -72,7 +72,7 @@ Save a strategy as `strategy.pine`: ```pinescript //@version=6 -strategy("SMA cross", initial_capital=10000) +strategy("SMA cross", initial_capital=10000, default_qty_type=strategy.percent_of_equity, default_qty_value=10) fast = ta.sma(close, 2) slow = ta.sma(close, 4) @@ -90,7 +90,7 @@ Then run: pineforge-backtest \ --pine strategy.pine \ --provider ccxt \ - --venue kraken \ + --venue bitstamp \ --symbol BTC/USD \ --timeframe 15m \ --start 2025-07-01T00:00:00Z \ @@ -100,10 +100,14 @@ pineforge-backtest \ --pretty ``` -The first run pulls a digest-pinned `pineforge-release` image. The JSON report -includes trades, performance statistics, the equity curve, data provenance, -and exact runtime versions. The strategy is compiled inside an isolated Docker -container; the provider and its credentials remain on the host. +The first run pulls the digest-pinned `pineforge-release` 0.1.12 image (engine +0.11.0, codegen 0.9.0). Release 1.0.0 (engine 1.0.0, codegen 1.0.0) also runs +this example when its digest-pinned image is passed to `--runtime-image`; see the +[runtime image policy](https://pineforge-4pass.github.io/pineforge-data/backtesting/#runtime-image-policy). +The JSON report includes trades, performance +statistics, the equity curve, data provenance, and exact runtime versions. The +strategy is compiled inside an isolated Docker container; the provider and its +credentials remain on the host. ## Bring your own data @@ -115,7 +119,7 @@ and accept an explicit mapping when your schema uses different names. |---|---|---| | CSV | `pip install pineforge-data` | [CSV API](https://pineforge-4pass.github.io/pineforge-data/providers/csv/) | | SQLite | `pip install pineforge-data` | [SQLite API](https://pineforge-4pass.github.io/pineforge-data/providers/sqlite/) | -| SQLAlchemy database | `pip install 'pineforge-data[database]'` | [SQLAlchemy API](https://pineforge-4pass.github.io/pineforge-data/providers/sqlalchemy/) | +| SQLAlchemy database | `pip install 'pineforge-data[database]'` plus a database driver | [SQLAlchemy API](https://pineforge-4pass.github.io/pineforge-data/providers/sqlalchemy/) | | CCXT exchange | `pip install 'pineforge-data[ccxt]'` | [CCXT API](https://pineforge-4pass.github.io/pineforge-data/providers/ccxt/) | The same `pineforge-backtest` command supports `--provider csv`, `sqlite`, and diff --git a/docs/api/backtesting.md b/docs/api/backtesting.md index e49d31c..75e9211 100644 --- a/docs/api/backtesting.md +++ b/docs/api/backtesting.md @@ -8,6 +8,13 @@ ::: pineforge_data.backtest.BacktestReport +`PineForgeBacktestRunner` accepts only strategy libraries that report PineForge +C ABI 2, which engine releases v0.10.2 through v0.12.3 build. It refuses +libraries built with engine v0.13.x (ABI 3) or v1.0.0 (ABI 4) with an +`EngineBacktestError` such as "PineForge ABI mismatch: strategy reports 4, +expected 2". The release container and the FastAPI server do not use this +class. + ::: pineforge_data.backtest.PineForgeBacktestRunner ::: pineforge_data.backtest.EngineBacktestError diff --git a/docs/api/server.md b/docs/api/server.md index b4e917c..5885055 100644 --- a/docs/api/server.md +++ b/docs/api/server.md @@ -1,8 +1,10 @@ # Server components -Most users run the packaged `pineforge-backtest-server` command or submit work -through `FastApiBacktestClient`. These objects support applications that embed -the service or manage its compiled-strategy cache directly. +Most users run the server image built from `docker/server.Dockerfile` or submit +work through `FastApiBacktestClient`. The packaged `pineforge-backtest-server` +command reports ready only where the `pineforge-release` toolchain is +installed, as in that image. These objects support applications that embed the +service or manage its compiled-strategy cache directly. ::: pineforge_data.server.create_app diff --git a/docs/backtesting.md b/docs/backtesting.md index 8b4bdd9..02bfa8b 100644 --- a/docs/backtesting.md +++ b/docs/backtesting.md @@ -29,7 +29,7 @@ are documented in the [server guide](server.md). pineforge-backtest \ --pine strategy.pine \ --provider ccxt \ - --venue kraken \ + --venue bitstamp \ --symbol BTC/USD \ --timeframe 15m \ --start 2025-07-01T00:00:00Z \ @@ -60,7 +60,7 @@ strategy to place orders during that earlier interval: pineforge-backtest \ --pine strategy.pine \ --provider ccxt \ - --venue kraken \ + --venue bitstamp \ --symbol BTC/USD \ --timeframe 15m \ --start 2025-07-01T00:00:00Z \ @@ -132,9 +132,20 @@ guides. ## Runtime image policy -The package default is an exact `pineforge-release` version and OCI digest. The -`missing` pull policy downloads it only when absent; `never` supports offline -runs; `always` refreshes a tag before running. +The package default is an exact `pineforge-release` version and OCI digest: +0.1.12, which bundles engine 0.11.0 and codegen 0.9.0. The `missing` pull +policy downloads it only when absent; `never` supports offline runs; `always` +refreshes a tag before running. + +Release 1.0.0 bundles engine 1.0.0 and codegen 1.0.0 and accepts the same +inputs. To use it, pin it by digest: + +```bash +--runtime-image ghcr.io/pineforge-4pass/pineforge-release:1.0.0@sha256:b271873d0e91a9eeeb4595daaded9bdfbdbcd3925d3ec1300a833395e2004c86 +``` + +Results can differ between releases: the README quick start reports 88 trades +on 0.1.12 and 89 on 1.0.0. The rolling channel is explicit: @@ -155,13 +166,13 @@ The harness combines provider provenance with the release report: "schema_version": 1, "request_id": null, "provider": { - "name": "ccxt:kraken", + "name": "ccxt:bitstamp", "adapter": "ccxt", - "venue": "kraken", + "venue": "bitstamp", "source_timeframe": "15m", "market": { "symbol": "BTC/USD", - "provider_id": "XXBTZUSD", + "provider_id": "btcusd", "market_type": "spot", "contract": null } @@ -169,8 +180,8 @@ The harness combines provider provenance with the release report: "data": { "requested_start_ms": 1751328000000, "requested_end_ms": 1751932800000, - "provider_start_ms": 1746828000000, - "first_bar_ms": 1746828000000, + "provider_start_ms": 1750878000000, + "first_bar_ms": 1750878000000, "last_bar_ms": 1751931900000, "bars": 1172, "requested_bars": 672, @@ -178,6 +189,11 @@ The harness combines provider provenance with the release report: "warmup_bars_loaded": 500, "trade_start_time_ms": 1751328000000 }, + "strategy": { + "pine": "/path/to/strategy.pine", + "input_timeframe": "15", + "script_timeframe": "15" + }, "runtime": { "mode": "local-container", "release_image": "ghcr.io/...@sha256:..." diff --git a/docs/data-model.md b/docs/data-model.md index 8364c7a..d9ae952 100644 --- a/docs/data-model.md +++ b/docs/data-model.md @@ -61,8 +61,8 @@ swap = Instrument( ``` `ContractSpec` can also carry `expiry_ms`, `strike`, and `option_type`. A -contract cannot be both linear and inverse, and numeric contract terms must be -finite and positive. +contract cannot be both linear and inverse, and contract size and strike must +be finite and positive. ## Market listings and discovery diff --git a/docs/getting-started.md b/docs/getting-started.md index 4e37509..8ca77c0 100644 --- a/docs/getting-started.md +++ b/docs/getting-started.md @@ -41,7 +41,7 @@ from pineforge_data import BarRequest, CcxtProvider async def main() -> None: - async with CcxtProvider("kraken") as provider: + async with CcxtProvider("bitstamp") as provider: listing = await provider.resolve_market("BTC/USD") instrument = listing.instrument bars = await provider.fetch_bars( @@ -67,7 +67,7 @@ excludes a candle that was not closed at the request's observation time. .venv/bin/pineforge-backtest \ --pine strategy.pine \ --provider ccxt \ - --venue kraken \ + --venue bitstamp \ --symbol BTC/USD \ --timeframe 15m \ --start 2025-07-01T00:00:00Z \ @@ -89,7 +89,7 @@ export PINEFORGE_SERVER_URL=http://127.0.0.1:8000 export PINEFORGE_SERVER_API_KEY=change-me .venv/bin/pineforge-backtest \ --pine strategy.pine \ - --venue kraken \ + --venue bitstamp \ --symbol BTC/USD \ --timeframe 15m \ --start 2025-07-01T00:00:00Z \ diff --git a/docs/index.md b/docs/index.md index bdbc11f..bf2fb94 100644 --- a/docs/index.md +++ b/docs/index.md @@ -1,16 +1,16 @@ # PineForge Data documentation -PineForge Data connects external market and macro providers to deterministic +PineForge Data connects external market data providers to deterministic PineForge backtests. It owns provider transport, market discovery, normalized records, data provenance, and the local or remote handoff to `pineforge-release`. ```text -exchange / broker / macro API +exchange / CSV / database ↓ Python provider adapter ↓ - normalized market + bars + trades + normalized market + bars ↓ local pineforge-release or FastAPI server ↓ diff --git a/docs/providers/ccxt.md b/docs/providers/ccxt.md index 6d38eb2..a4de305 100644 --- a/docs/providers/ccxt.md +++ b/docs/providers/ccxt.md @@ -160,7 +160,7 @@ pineforge-backtest \ --pine strategy.pine \ --provider ccxt \ --provider-config ccxt.json \ - --venue kraken \ + --venue bitstamp \ --symbol BTC/USD \ --timeframe 15m \ --start 2025-07-01T00:00:00Z \ @@ -181,4 +181,5 @@ Python class when endpoint-specific parameters or polling controls are needed. Public trades currently use REST polling, not CCXT Pro WebSockets. Availability, history depth, pagination semantics, and rate limits still depend on the -selected exchange. +selected exchange. For example, Kraken serves only its most recent 720 candles +per timeframe, so an older 15-minute window returns no bars. diff --git a/docs/providers/csv.md b/docs/providers/csv.md index ae3cf66..5116e71 100644 --- a/docs/providers/csv.md +++ b/docs/providers/csv.md @@ -45,7 +45,6 @@ complete mapping is preferable. ```python schema = await provider.inspect_schema() print(schema.column_names) -mapping = schema.infer_bar_mapping({"timestamp": "epoch seconds"}) ``` The first record must be a non-empty, unique header. Files with duplicate or diff --git a/docs/providers/tabular-schema.md b/docs/providers/tabular-schema.md index e11c082..170017c 100644 --- a/docs/providers/tabular-schema.md +++ b/docs/providers/tabular-schema.md @@ -92,8 +92,9 @@ source values: ISO-8601 values with an offset or `Z` are unambiguous. Naive text or database datetimes use `timestamp_timezone`, which defaults to `UTC` and accepts an IANA -zone such as `America/New_York`. Values that cannot be represented exactly at -millisecond precision fail rather than being rounded silently. +zone such as `America/New_York`. Numeric values that cannot be represented +exactly at millisecond precision fail rather than being rounded silently; +ISO-8601 text and date/datetime values are truncated to whole milliseconds. ## Symbol behavior @@ -124,7 +125,7 @@ Tabular providers: - apply `limit` after normalization; - reject duplicate timestamps for the requested symbol and timeframe; - reject missing or non-finite values, invalid OHLC relationships, negative - volume, and lossy sub-millisecond timestamps; + volume, and lossy sub-millisecond numeric timestamps; - attach the requested instrument and provider source to every `Bar`. Rows are assumed to be confirmed snapshots because local files and databases diff --git a/docs/server.md b/docs/server.md index 7efea1f..da2fabb 100644 --- a/docs/server.md +++ b/docs/server.md @@ -54,7 +54,7 @@ concurrent misses for one key are deduplicated. | Environment variable | Default | Meaning | |---|---:|---| -| `PINEFORGE_SERVER_CACHE_DIR` | `/cache` | compiled artifact directory | +| `PINEFORGE_SERVER_CACHE_DIR` | `/cache` in the image | compiled artifact directory | | `PINEFORGE_SERVER_CACHE_MAX_ENTRIES` | `1024` | maximum retained `.so` files | | `PINEFORGE_SERVER_CACHE_MAX_BYTES` | `2147483648` | maximum retained bytes | @@ -73,6 +73,9 @@ docker build -f docker/server.Dockerfile \ -t pineforge-data-server:latest . ``` +To pin release 1.0.0 (engine 1.0.0, codegen 1.0.0) instead, pass +`--build-arg PINEFORGE_RELEASE_IMAGE=ghcr.io/pineforge-4pass/pineforge-release:1.0.0@sha256:b271873d0e91a9eeeb4595daaded9bdfbdbcd3925d3ec1300a833395e2004c86`. + Do not use the rolling channel for reproducibility-sensitive production runs. The cache key includes component/release identity, so a runtime upgrade cannot reuse a compiled artifact from a different engine version. diff --git a/mkdocs.yml b/mkdocs.yml index 670682f..6bb8215 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -1,5 +1,5 @@ site_name: PineForge Data -site_description: Provider-neutral market and macro data adapters for PineForge +site_description: Provider-neutral market data adapters and raw-PineScript backtest harness for PineForge site_url: https://pineforge-4pass.github.io/pineforge-data/ repo_url: https://github.com/pineforge-4pass/pineforge-data repo_name: pineforge-4pass/pineforge-data diff --git a/pyproject.toml b/pyproject.toml index 3dd09f7..664913f 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -5,12 +5,12 @@ build-backend = "hatchling.build" [project] name = "pineforge-data" version = "0.2.0" -description = "Provider-neutral market and macro data adapters for PineForge" +description = "Provider-neutral market data adapters and raw-PineScript backtest harness for PineForge" readme = "README.md" requires-python = ">=3.11" license = { text = "Apache-2.0" } authors = [{ name = "PineForge contributors" }] -keywords = ["pinescript", "backtesting", "market-data", "macro-data"] +keywords = ["pinescript", "backtesting", "market-data"] classifiers = [ "Development Status :: 3 - Alpha", "License :: OSI Approved :: Apache Software License", diff --git a/tests/fixtures/CORPUS.md b/tests/fixtures/CORPUS.md index a2a0367..c27065c 100644 --- a/tests/fixtures/CORPUS.md +++ b/tests/fixtures/CORPUS.md @@ -5,11 +5,13 @@ rows copied byte-for-byte from: - repository: `pineforge-4pass/pineforge-corpus`; - file: `data/ohlcv_ETH-USDT-USDT_1m.csv`; +- market data: Binance USDT-M perpetual ETH/USDT-USDT 1-minute bars, per that + repository's `NOTICE`; - Git LFS SHA-256: `db8c1332da093008cfbd063e05db0b33fe8f7fd35d78cf058a366519eb9f6cc5`; - sample SHA-256: `8ba4db0c669e47c11746361586cef8bce6371eeeab5af5e351107bd69451c6af`; - timestamps: `1577836800000` through `1577852100000` (Unix milliseconds). -CI uses this bounded sample to avoid downloading the 168 MB Git LFS object on +CI uses this bounded sample to avoid downloading the 168 MiB Git LFS object on every run. `scripts/run_database_e2e.sh` automatically uses the full sibling corpus file when it is available, while still reading only the requested number of rows.