Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 12 additions & 8 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -46,7 +46,7 @@ from pineforge_data import BarRequest, CcxtProvider


async def main() -> None:
async with CcxtProvider("kraken") as provider:
async with CcxtProvider("bitstamp") as provider:
market = await provider.resolve_market("BTC/USD")
bars = await provider.fetch_bars(
BarRequest(
Expand All @@ -72,7 +72,7 @@ Save a strategy as `strategy.pine`:

```pinescript
//@version=6
strategy("SMA cross", initial_capital=10000)
strategy("SMA cross", initial_capital=10000, default_qty_type=strategy.percent_of_equity, default_qty_value=10)

fast = ta.sma(close, 2)
slow = ta.sma(close, 4)
Expand All @@ -90,7 +90,7 @@ Then run:
pineforge-backtest \
--pine strategy.pine \
--provider ccxt \
--venue kraken \
--venue bitstamp \
--symbol BTC/USD \
--timeframe 15m \
--start 2025-07-01T00:00:00Z \
Expand All @@ -100,10 +100,14 @@ pineforge-backtest \
--pretty
```

The first run pulls a digest-pinned `pineforge-release` image. The JSON report
includes trades, performance statistics, the equity curve, data provenance,
and exact runtime versions. The strategy is compiled inside an isolated Docker
container; the provider and its credentials remain on the host.
The first run pulls the digest-pinned `pineforge-release` 0.1.12 image (engine
0.11.0, codegen 0.9.0). Release 1.0.0 (engine 1.0.0, codegen 1.0.0) also runs
this example when its digest-pinned image is passed to `--runtime-image`; see the
[runtime image policy](https://pineforge-4pass.github.io/pineforge-data/backtesting/#runtime-image-policy).
The JSON report includes trades, performance
statistics, the equity curve, data provenance, and exact runtime versions. The
strategy is compiled inside an isolated Docker container; the provider and its
credentials remain on the host.

## Bring your own data

Expand All @@ -115,7 +119,7 @@ and accept an explicit mapping when your schema uses different names.
|---|---|---|
| CSV | `pip install pineforge-data` | [CSV API](https://pineforge-4pass.github.io/pineforge-data/providers/csv/) |
| SQLite | `pip install pineforge-data` | [SQLite API](https://pineforge-4pass.github.io/pineforge-data/providers/sqlite/) |
| SQLAlchemy database | `pip install 'pineforge-data[database]'` | [SQLAlchemy API](https://pineforge-4pass.github.io/pineforge-data/providers/sqlalchemy/) |
| SQLAlchemy database | `pip install 'pineforge-data[database]'` plus a database driver | [SQLAlchemy API](https://pineforge-4pass.github.io/pineforge-data/providers/sqlalchemy/) |
| CCXT exchange | `pip install 'pineforge-data[ccxt]'` | [CCXT API](https://pineforge-4pass.github.io/pineforge-data/providers/ccxt/) |

The same `pineforge-backtest` command supports `--provider csv`, `sqlite`, and
Expand Down
7 changes: 7 additions & 0 deletions docs/api/backtesting.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,13 @@

::: pineforge_data.backtest.BacktestReport

`PineForgeBacktestRunner` accepts only strategy libraries that report PineForge
C ABI 2, which engine releases v0.10.2 through v0.12.3 build. It refuses
libraries built with engine v0.13.x (ABI 3) or v1.0.0 (ABI 4) with an
`EngineBacktestError` such as "PineForge ABI mismatch: strategy reports 4,
expected 2". The release container and the FastAPI server do not use this
class.

::: pineforge_data.backtest.PineForgeBacktestRunner

::: pineforge_data.backtest.EngineBacktestError
Expand Down
8 changes: 5 additions & 3 deletions docs/api/server.md
Original file line number Diff line number Diff line change
@@ -1,8 +1,10 @@
# Server components

Most users run the packaged `pineforge-backtest-server` command or submit work
through `FastApiBacktestClient`. These objects support applications that embed
the service or manage its compiled-strategy cache directly.
Most users run the server image built from `docker/server.Dockerfile` or submit
work through `FastApiBacktestClient`. The packaged `pineforge-backtest-server`
command reports ready only where the `pineforge-release` toolchain is
installed, as in that image. These objects support applications that embed the
service or manage its compiled-strategy cache directly.

::: pineforge_data.server.create_app

Expand Down
36 changes: 26 additions & 10 deletions docs/backtesting.md
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@ are documented in the [server guide](server.md).
pineforge-backtest \
--pine strategy.pine \
--provider ccxt \
--venue kraken \
--venue bitstamp \
--symbol BTC/USD \
--timeframe 15m \
--start 2025-07-01T00:00:00Z \
Expand Down Expand Up @@ -60,7 +60,7 @@ strategy to place orders during that earlier interval:
pineforge-backtest \
--pine strategy.pine \
--provider ccxt \
--venue kraken \
--venue bitstamp \
--symbol BTC/USD \
--timeframe 15m \
--start 2025-07-01T00:00:00Z \
Expand Down Expand Up @@ -132,9 +132,20 @@ guides.

## Runtime image policy

The package default is an exact `pineforge-release` version and OCI digest. The
`missing` pull policy downloads it only when absent; `never` supports offline
runs; `always` refreshes a tag before running.
The package default is an exact `pineforge-release` version and OCI digest:
0.1.12, which bundles engine 0.11.0 and codegen 0.9.0. The `missing` pull
policy downloads it only when absent; `never` supports offline runs; `always`
refreshes a tag before running.

Release 1.0.0 bundles engine 1.0.0 and codegen 1.0.0 and accepts the same
inputs. To use it, pin it by digest:

```bash
--runtime-image ghcr.io/pineforge-4pass/pineforge-release:1.0.0@sha256:b271873d0e91a9eeeb4595daaded9bdfbdbcd3925d3ec1300a833395e2004c86
```

Results can differ between releases: the README quick start reports 88 trades
on 0.1.12 and 89 on 1.0.0.

The rolling channel is explicit:

Expand All @@ -155,29 +166,34 @@ The harness combines provider provenance with the release report:
"schema_version": 1,
"request_id": null,
"provider": {
"name": "ccxt:kraken",
"name": "ccxt:bitstamp",
"adapter": "ccxt",
"venue": "kraken",
"venue": "bitstamp",
"source_timeframe": "15m",
"market": {
"symbol": "BTC/USD",
"provider_id": "XXBTZUSD",
"provider_id": "btcusd",
"market_type": "spot",
"contract": null
}
},
"data": {
"requested_start_ms": 1751328000000,
"requested_end_ms": 1751932800000,
"provider_start_ms": 1746828000000,
"first_bar_ms": 1746828000000,
"provider_start_ms": 1750878000000,
"first_bar_ms": 1750878000000,
"last_bar_ms": 1751931900000,
"bars": 1172,
"requested_bars": 672,
"warmup_bars_requested": 500,
"warmup_bars_loaded": 500,
"trade_start_time_ms": 1751328000000
},
"strategy": {
"pine": "/path/to/strategy.pine",
"input_timeframe": "15",
"script_timeframe": "15"
},
"runtime": {
"mode": "local-container",
"release_image": "ghcr.io/...@sha256:..."
Expand Down
4 changes: 2 additions & 2 deletions docs/data-model.md
Original file line number Diff line number Diff line change
Expand Up @@ -61,8 +61,8 @@ swap = Instrument(
```

`ContractSpec` can also carry `expiry_ms`, `strike`, and `option_type`. A
contract cannot be both linear and inverse, and numeric contract terms must be
finite and positive.
contract cannot be both linear and inverse, and contract size and strike must
be finite and positive.

## Market listings and discovery

Expand Down
6 changes: 3 additions & 3 deletions docs/getting-started.md
Original file line number Diff line number Diff line change
Expand Up @@ -41,7 +41,7 @@ from pineforge_data import BarRequest, CcxtProvider


async def main() -> None:
async with CcxtProvider("kraken") as provider:
async with CcxtProvider("bitstamp") as provider:
listing = await provider.resolve_market("BTC/USD")
instrument = listing.instrument
bars = await provider.fetch_bars(
Expand All @@ -67,7 +67,7 @@ excludes a candle that was not closed at the request's observation time.
.venv/bin/pineforge-backtest \
--pine strategy.pine \
--provider ccxt \
--venue kraken \
--venue bitstamp \
--symbol BTC/USD \
--timeframe 15m \
--start 2025-07-01T00:00:00Z \
Expand All @@ -89,7 +89,7 @@ export PINEFORGE_SERVER_URL=http://127.0.0.1:8000
export PINEFORGE_SERVER_API_KEY=change-me
.venv/bin/pineforge-backtest \
--pine strategy.pine \
--venue kraken \
--venue bitstamp \
--symbol BTC/USD \
--timeframe 15m \
--start 2025-07-01T00:00:00Z \
Expand Down
6 changes: 3 additions & 3 deletions docs/index.md
Original file line number Diff line number Diff line change
@@ -1,16 +1,16 @@
# PineForge Data documentation

PineForge Data connects external market and macro providers to deterministic
PineForge Data connects external market data providers to deterministic
PineForge backtests. It owns provider transport, market discovery, normalized
records, data provenance, and the local or remote handoff to
`pineforge-release`.

```text
exchange / broker / macro API
exchange / CSV / database
↓
Python provider adapter
↓
normalized market + bars + trades
normalized market + bars
↓
local pineforge-release or FastAPI server
↓
Expand Down
5 changes: 3 additions & 2 deletions docs/providers/ccxt.md
Original file line number Diff line number Diff line change
Expand Up @@ -160,7 +160,7 @@ pineforge-backtest \
--pine strategy.pine \
--provider ccxt \
--provider-config ccxt.json \
--venue kraken \
--venue bitstamp \
--symbol BTC/USD \
--timeframe 15m \
--start 2025-07-01T00:00:00Z \
Expand All @@ -181,4 +181,5 @@ Python class when endpoint-specific parameters or polling controls are needed.

Public trades currently use REST polling, not CCXT Pro WebSockets. Availability,
history depth, pagination semantics, and rate limits still depend on the
selected exchange.
selected exchange. For example, Kraken serves only its most recent 720 candles
per timeframe, so an older 15-minute window returns no bars.
1 change: 0 additions & 1 deletion docs/providers/csv.md
Original file line number Diff line number Diff line change
Expand Up @@ -45,7 +45,6 @@ complete mapping is preferable.
```python
schema = await provider.inspect_schema()
print(schema.column_names)
mapping = schema.infer_bar_mapping({"timestamp": "epoch seconds"})
```

The first record must be a non-empty, unique header. Files with duplicate or
Expand Down
7 changes: 4 additions & 3 deletions docs/providers/tabular-schema.md
Original file line number Diff line number Diff line change
Expand Up @@ -92,8 +92,9 @@ source values:

ISO-8601 values with an offset or `Z` are unambiguous. Naive text or database
datetimes use `timestamp_timezone`, which defaults to `UTC` and accepts an IANA
zone such as `America/New_York`. Values that cannot be represented exactly at
millisecond precision fail rather than being rounded silently.
zone such as `America/New_York`. Numeric values that cannot be represented
exactly at millisecond precision fail rather than being rounded silently;
ISO-8601 text and date/datetime values are truncated to whole milliseconds.

## Symbol behavior

Expand Down Expand Up @@ -124,7 +125,7 @@ Tabular providers:
- apply `limit` after normalization;
- reject duplicate timestamps for the requested symbol and timeframe;
- reject missing or non-finite values, invalid OHLC relationships, negative
volume, and lossy sub-millisecond timestamps;
volume, and lossy sub-millisecond numeric timestamps;
- attach the requested instrument and provider source to every `Bar`.

Rows are assumed to be confirmed snapshots because local files and databases
Expand Down
5 changes: 4 additions & 1 deletion docs/server.md
Original file line number Diff line number Diff line change
Expand Up @@ -54,7 +54,7 @@ concurrent misses for one key are deduplicated.

| Environment variable | Default | Meaning |
|---|---:|---|
| `PINEFORGE_SERVER_CACHE_DIR` | `/cache` | compiled artifact directory |
| `PINEFORGE_SERVER_CACHE_DIR` | `/cache` in the image | compiled artifact directory |
| `PINEFORGE_SERVER_CACHE_MAX_ENTRIES` | `1024` | maximum retained `.so` files |
| `PINEFORGE_SERVER_CACHE_MAX_BYTES` | `2147483648` | maximum retained bytes |

Expand All @@ -73,6 +73,9 @@ docker build -f docker/server.Dockerfile \
-t pineforge-data-server:latest .
```

To pin release 1.0.0 (engine 1.0.0, codegen 1.0.0) instead, pass
`--build-arg PINEFORGE_RELEASE_IMAGE=ghcr.io/pineforge-4pass/pineforge-release:1.0.0@sha256:b271873d0e91a9eeeb4595daaded9bdfbdbcd3925d3ec1300a833395e2004c86`.

Do not use the rolling channel for reproducibility-sensitive production runs.
The cache key includes component/release identity, so a runtime upgrade cannot
reuse a compiled artifact from a different engine version.
Expand Down
2 changes: 1 addition & 1 deletion mkdocs.yml
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
site_name: PineForge Data
site_description: Provider-neutral market and macro data adapters for PineForge
site_description: Provider-neutral market data adapters and raw-PineScript backtest harness for PineForge
site_url: https://pineforge-4pass.github.io/pineforge-data/
repo_url: https://github.com/pineforge-4pass/pineforge-data
repo_name: pineforge-4pass/pineforge-data
Expand Down
4 changes: 2 additions & 2 deletions pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -5,12 +5,12 @@ build-backend = "hatchling.build"
[project]
name = "pineforge-data"
version = "0.2.0"
description = "Provider-neutral market and macro data adapters for PineForge"
description = "Provider-neutral market data adapters and raw-PineScript backtest harness for PineForge"
readme = "README.md"
requires-python = ">=3.11"
license = { text = "Apache-2.0" }
authors = [{ name = "PineForge contributors" }]
keywords = ["pinescript", "backtesting", "market-data", "macro-data"]
keywords = ["pinescript", "backtesting", "market-data"]
classifiers = [
"Development Status :: 3 - Alpha",
"License :: OSI Approved :: Apache Software License",
Expand Down
4 changes: 3 additions & 1 deletion tests/fixtures/CORPUS.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,11 +5,13 @@ rows copied byte-for-byte from:

- repository: `pineforge-4pass/pineforge-corpus`;
- file: `data/ohlcv_ETH-USDT-USDT_1m.csv`;
- market data: Binance USDT-M perpetual ETH/USDT-USDT 1-minute bars, per that
repository's `NOTICE`;
- Git LFS SHA-256: `db8c1332da093008cfbd063e05db0b33fe8f7fd35d78cf058a366519eb9f6cc5`;
- sample SHA-256: `8ba4db0c669e47c11746361586cef8bce6371eeeab5af5e351107bd69451c6af`;
- timestamps: `1577836800000` through `1577852100000` (Unix milliseconds).

CI uses this bounded sample to avoid downloading the 168 MB Git LFS object on
CI uses this bounded sample to avoid downloading the 168 MiB Git LFS object on
every run. `scripts/run_database_e2e.sh` automatically uses the full sibling
corpus file when it is available, while still reading only the requested number
of rows.
Loading