diff --git a/CHANGELOG.md b/CHANGELOG.md index 8bc42e54c..89b74a85a 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -5,7 +5,24 @@ README's *Releases* section and on the GitHub releases page. From 1.0.0 the version number follows semantic versioning over the surfaces the [public contract](docs/pages/public-contract.md) lists. -## Unreleased +## Unreleased (1.15.0) + +- `detect_timeframe` now measures the gap between consecutive bars without overflowing when their timestamps lie far apart on opposite sides of zero; such pairs could give a wrong timeframe. Pairs that are equal or out of order are still skipped. +- TA length and Keltner Channel argument errors without a bar context now begin `Error before the first bar: `. Map capacity errors use the same prefix and report `bar_index: -1` without a bar context, including during native run preparation and startup. (#933) +- `pf_version_get().commit_sha` now returns an empty string when the build's Git SHA is unavailable (#929). +- The native accepted-input lookup now returns the completed input interval for every bar label the native calendar admits as confirmed, not only for an exact match on the bar's open or eligible open (#718). +- A timeframe whose count of minutes, days or weeks has more seconds than a 32-bit `int` holds no longer overflows into a wrong length. A minute count from `35791395` (about 68 years) up, and a negative count past the same limit, are now refused like a count past the `int` range: the timeframe parser stops the run with `pine_invalid_argument` (argument `timeframe`, rule `unparseable`), and the `request.security` registration check maps that to `request_timeframe_invalid`, as it already did for such a count. A week count from `3551W` up is calendar-only, as a day count from `24856D` up already is. Before, `time()` and `time_close()` could read `71582789` as a 44-second timeframe, and `request.security` could take `-35791395` for a 68-year one. +- With `--native-event-retention window`, the runner's equity-curve scratch file no longer goes to the system temporary directory when the ledger's directory cannot hold an unnamed file: the curve stays in memory and the runner says so (#925). +- `--native-event-retention window` on a module that does not declare the engine layout stamp now prints a warning that the option is not applied, and the checkpoints are stored as they are without the option (#926). +- A report spool failure while the runner rebuilds a report checkpoint at startup now prints the warning that the settled rows stay in memory (#927). +- A failed unlink of the fallback spool file is treated like a spool that cannot be created: the rows stay in memory and the runner warns once, and no named spool file is left behind (#928). +- The runner's atomic service file writer now retries a refused unlink of its temporary `.new.` file once and reports it once on stderr, instead of leaving the file behind without a message; an install that succeeded stays a success (#946). +- With `--native-event-retention window`, a report checkpoint at cursor 512 or below is no longer built as a whole JSON tree: every checkpoint of such a run is folded through the spooled arrays, so a payload above 1 MiB is written into its row in pieces, stored as a BLOB with the same bytes, at any cursor (1.14.0 did that only above cursor 512). In the measured case (512 inputs of 1,000 bars, a 26.7 MiB checkpoint) the checkpoint raised the runner's peak resident memory by 441 MiB before and by 0.66 MiB now. A run without the option, and a payload of 1 MiB or less, are unchanged (#961). +- The calendar exporter `scripts/symbol_calendar.py` now checks the calendar document before it writes its destination file, so a calendar whose final session is invalid is refused without leaving a file behind that the reader would reject (#952). +- With a series length, `ta.sma` now rounds the exact sum of the selected window before dividing by its length, as fixed-length `ta.sma` does. The optional simple-average seed for `ta.ema` uses the same rounding. Small terms are preserved when larger terms cancel (#935). +- The documentation pages `docs/pages/abi-stability.md` and `docs/pages/report-schema.md` now say stream layout 5, the value the header defines, where three sentences said 4 (#923). + +## 1.14.0 — 2026-11-20 - The engine layout stamp that `pineforge-live` checks, `PF_STREAM_ENGINE_LAYOUT_VERSION`, is now 5. A strategy module built against 1.13.x headers declares stamp 4 and is refused by a 1.14.0 `pineforge-live` before it is created (`strategy engine layout mismatch: module=4 runtime=5`): rebuild such modules against the matching 1.14.0 headers. The earlier miss, disclosed: 1.13.0 changed the same internal layout without raising the stamp, so a module built against 1.12.x headers and loaded into a 1.13.x `pineforge-live` was not refused; rebuild such modules against matching headers too. `PF_ABI_VERSION` stays 5, and the C API and its caller-allocated structures are unchanged. - A new host helper, `strategy_convert_identity_or_warn`, lets generated code report `strategy.convert_to_account` and `strategy.convert_to_symbol` calls that had no account-currency rate: the value still comes back unchanged, and the run's warnings now list `currency_conversion_rate_unavailable` once for each such call site (#376). @@ -287,7 +304,7 @@ version number follows semantic versioning over the surfaces the - `strategy_pending_order_fill_qty` reported a wrong `close_only` for some entries: 1 where the fill opens or reverses the position (a market entry followed by a priced stop on the same bar and a short-seed percent entry, #905; a pyramiding entry after a same-bar close; an entry probed inside the callback that queued it) and 0 where the fill only closes (a priced entry that replaced a same-id market order, a frozen reversal, whose reported quantity is now the frozen transaction, and a host-sized `strategy.order` against an opposite position). It now applies the fill's prior-cycle rule and shape rules for these cases. A `strategy.order` with its own quantity against an opposite position now reports the quantity its fill transacts, and `close_only` 1 exactly when that fill opens nothing. While a close the script placed on the same bar is still waiting to fill ahead of the order, the report instead gives the quantity as written, with `close_only` 0, or 1 when the order is in an OCA cancel group. That answer can differ from the fill: in an OCA cancel group it says `close_only` 1 although the fill opens if the close leaves the position flat; otherwise it says `close_only` 0 although the fill only closes if the close leaves a position at least as large as what the order transacts; and the fill can transact less than the quantity as written, for example when the quantity is floored to the quantity step, or when an OCA cancel-group order closes no more than the close leaves. The report is made before the fill and does not account for checks the fill makes at fill time against equity, price and margin (for example, an entry the account cannot afford in whole units). In those cases the fill can differ from the report. Not modelled: fill-time money guards (whole-lot tie, low-value or fractional lot), affordability, margin-call surplus, the close quantity a reversal keeps when a margin call was booked after the script, a refused signal price, an admission refusal, earlier fills of the same pass, and an entry that names its own quantity or a row whose request cannot be read, such as a row held in the source's shadow queue (judged by the prior-cycle rule alone). Fills are unchanged. (#905, #910) - The C header and docs documented partition 2 for a pure percent-of-equity stop; the engine returns 1 (measured on 1.13.0). The documentation now matches the returned value; behaviour and ABI are unchanged. -## 1.13.0 +## 1.13.0 — 2026-11-18 ### Changed diff --git a/ci/cr150-source-bounds.yaml b/ci/cr150-source-bounds.yaml index af21cdccd..30a70cde1 100644 --- a/ci/cr150-source-bounds.yaml +++ b/ci/cr150-source-bounds.yaml @@ -170,7 +170,7 @@ ceilings: # sense: no fixed input is declared for them. Each says what limits it: # U1878 prefix: the list a whole-report read of a window run builds from the # points the window had taken, O(points taken); it exists only with window - # retention (src/live_report.cpp:102-111, runner/README.md:189-192), and a + # retention (src/live_report.cpp:102-111, runner/README.md:194-197), and a # default run never takes a point, so it never reaches it. # U1879 scratch_curve and U1880 writer: the curve scratch writer and its # append. The writer holds one fixed array of 2,730 points, 65,520 bytes @@ -1906,8 +1906,8 @@ entries: - {id: "U1875", file: "src/source/pine_adapter.cpp", symbol: "covered", ops: ["decl", "grow"], container: "std::vector", owner: "source-layer", phase: "unclassified", state: "unreviewed", note: "The handles of the market adds of one fill's bar that have no order of their own, built by PineExecutionAdapter::on_applied while it decides what the cover fill closes: one handle for each such add, so at most the lots the entry id opened on that bar, and freed when the call returns. No fixed input bounds the lot count. OPEN.", line: 28153} - {id: "U1876", file: "src/source/pine_adapter.cpp", symbol: "definitions", ops: ["decl", "grow"], container: "std::vector", owner: "source-layer", phase: "unclassified", state: "unreviewed", note: "The exits standing on the book for an opening that fills, built by PineExecutionAdapter::arm_standing_exits_for_opening from one scan of the placement table: one record for each distinct exit id that has a leg there, freed when the function returns. It follows the exits the script has working, which no fixed input bounds. OPEN.", line: 12042} - {id: "U1877", file: "src/source/pine_adapter.cpp", symbol: "oca_name", ops: ["assign", "decl"], container: "std::string", owner: "source-layer", phase: "unclassified", state: "unreviewed", note: "An order's OCA group name. The group holds the copies of that text from one record to another: the standing-exit record of PineExecutionAdapter::arm_standing_exits_for_opening (one string for each distinct exit id with a leg on the placement table, copied from the leg and freed when the function returns) and the file's field-to-field copies of a placed order's OCA name (request.oca_name = snapshot.oca_name and the like). The text's size is the script's; no fixed input bounds it. OPEN.", line: 1486} - - {id: "U1878", file: "src/live_report.cpp", symbol: "prefix", ops: ["decl"], container: "std::vector", owner: "engine-core", phase: "unclassified", state: "unreviewed", note: "The points a whole-report read puts back in front of the engine's curve when the report window had taken them (src/live_report.cpp:102-111, the list declared at :105): a local list of the retired points, read back from the scratch file, inserted before the points the engine still holds and freed when that block ends (:111). Its size is the number of points taken, so that read is O(all points taken); no fixed input bounds it. It exists only with window retention (runner/README.md:189-192): a default run never retires a point and the branch at :102 is not taken. OPEN.", line: 105} - - {id: "U1879", file: "src/live_report.cpp", symbol: "scratch_curve", ops: ["alloc"], container: "unresolved by the index: make_unique", owner: "engine-core", phase: "unclassified", state: "unreviewed", note: "The curve scratch writer of a report accumulator that keeps its curve points in a scratch file (window retention): one writer for the life of the accumulator, created in the State constructor (src/live_report.cpp:49) so that a directory that cannot back it is refused before any read; the runner then falls back to the resident accumulator (runner/main.cpp:1283-1293). The writer holds one fixed array of 2,730 points (64 KiB divided by the 24-byte point, rounded down), 65,520 bytes (src/report_scratch.hpp:20-22; the array is src/report_scratch.cpp:162, described at src/report_scratch.hpp:71-72). The points go to a file on disk, which grows with the points taken; no fixed input bounds its size. OPEN.", line: 49} - - {id: "U1880", file: "src/live_report.cpp", symbol: "writer", ops: ["grow"], container: "unresolved by the index: writer.append(...)", owner: "engine-core", phase: "unclassified", state: "unreviewed", note: "Appends one curve point to the scratch file of the report accumulator (src/live_report.cpp:54): the point goes into the writer's RAM batch, and a full batch is written to the file before the next point is accepted (src/report_scratch.cpp:181, which flushes when the array is full; src/report_scratch.hpp:71-72), so the writer holds at most one batch of 2,730 points (65,520 bytes); the file grows by one 24-byte record for each point taken, and no fixed input bounds its size. OPEN.", line: 54} + - {id: "U1878", file: "src/live_report.cpp", symbol: "prefix", ops: ["decl"], container: "std::vector", owner: "engine-core", phase: "unclassified", state: "unreviewed", note: "The points a whole-report read puts back in front of the engine's curve when the report window had taken them (src/live_report.cpp:102-111, the list declared at :105): a local list of the retired points, read back from the scratch file, inserted before the points the engine still holds and freed when that block ends (:111). Its size is the number of points taken, so that read is O(all points taken); no fixed input bounds it. It exists only with window retention (runner/README.md:194-197): a default run never retires a point and the branch at :102 is not taken. OPEN.", line: 105} + - {id: "U1879", file: "src/live_report.cpp", symbol: "scratch_curve", ops: ["alloc"], container: "unresolved by the index: make_unique", owner: "engine-core", phase: "unclassified", state: "unreviewed", note: "One curve scratch writer for the life of a window report accumulator, created in the State constructor (src/live_report.cpp:49) to open the scratch storage before any read; if the open fails, the runner falls back to the resident accumulator (runner/main.cpp:1283-1293). The writer holds one fixed array of 2,730 points (64 KiB divided by the 24-byte point, rounded down), 65,520 bytes (src/report_scratch.hpp:20-22; the array is src/report_scratch.cpp:162, described at src/report_scratch.hpp:71-72). The points go to a region in the attached scratch arena, or to a file on disk when no arena is attached (include/pineforge/detail/scratch_file.hpp:370-380). The logical curve grows with the points taken; no fixed input bounds the number of points staged. OPEN.", line: 49} + - {id: "U1880", file: "src/live_report.cpp", symbol: "writer", ops: ["grow"], container: "unresolved by the index: writer.append(...)", owner: "engine-core", phase: "unclassified", state: "unreviewed", note: "Stages one curve point in the report accumulator's scratch writer batch (src/live_report.cpp:54). A full batch is written to the scratch backing before the next point is accepted (src/report_scratch.cpp:181; src/report_scratch.hpp:71-72), so the writer holds at most one batch of 2,730 points (65,520 bytes). The backing is a region in the attached scratch arena or a scratch file when no arena is attached (include/pineforge/detail/scratch_file.hpp:370-380). The logical curve grows by one 24-byte record per appended point, subject to the backing's capacity. No fixed input bounds the number of points staged. OPEN.", line: 54} - {id: "U1910", file: "include/pineforge/engine.hpp", symbol: "evaluator_input_tf", ops: ["decl"], container: "std::string", owner: "engine-core", phase: "unclassified", state: "unreviewed", note: "The input timeframe text one request.security evaluator keeps (BacktestEngine::SecurityEvalState, include/pineforge/engine.hpp:1499): a copy of the input timeframe spelling of the run, assigned when the evaluator is registered (src/engine_security.cpp:54) and again at every run begin (src/source/pine_scheduler.cpp:14). The fixed-period request clock reads it (src/engine_security.cpp:106 and src/engine_aux_security.cpp:248-256) and the state hash folds it in (src/engine_state_hash.cpp:33). One string for each registered evaluator; its size is that of one timeframe spelling and does not follow the length of the run. No fixed input is declared for it. OPEN.", line: 1499} - {id: "U1931", file: "src/source/pine_adapter.cpp", symbol: "unranked", ops: ["decl", "grow"], container: "std::vector", owner: "source-layer", phase: "unclassified", state: "unreviewed", note: "The positions, in pending_bracket_legs_, of the queued exit legs that one strategy.exit call gives a queue slot: the local list of the helper rank_queued_explicit_legs of PineExecutionAdapter::exit_with_reservation_sequence, declared at src/source/pine_adapter.cpp:13013 and filled at :13022. The helper returns before the list is declared unless the call is a full exit of a named entry placed at once (:12997-13004), so an exit of a named quantity or of a share below the whole never builds it. It takes one position for each waiting leg that names its quantity, has not been submitted and belongs to an opening the entry holds (:13016-13021): at most one for each leg in pending_bracket_legs_ at that call. The strategy.exit calls of the calculation put those legs there, and the end of the calculation submits them and empties that list. The local list is sorted and read once (:13046-13054) and freed when the helper returns (:13055); nothing retains it, and its size does not follow the length of the run. No fixed input is declared for it. OPEN.", line: 13013} diff --git a/docker/run_failure_codes.json b/docker/run_failure_codes.json index be3a761ba..a3bb5aa8c 100644 --- a/docker/run_failure_codes.json +++ b/docker/run_failure_codes.json @@ -731,7 +731,8 @@ "class": "strategy_limit", "description": "A distinct map insertion exceeds that map's configured pair limit. Zero or omitted selects no limit. Counts and bar index are engine-owned context; code is the fixed TradingView identifier RE10070.", "english": [ - "Error on bar {bar_index}: A map cannot have more than {maxSize} key-value pairs. The current size is {currSize}." + "Error on bar {bar_index}: A map cannot have more than {maxSize} key-value pairs. The current size is {currSize}.", + "Error before the first bar: A map cannot have more than {maxSize} key-value pairs. The current size is {currSize}." ], "retryable": false, "since": "1.11.0" diff --git a/docker/run_failure_codes_diff.json b/docker/run_failure_codes_diff.json index a719b186d..f8287069b 100644 --- a/docker/run_failure_codes_diff.json +++ b/docker/run_failure_codes_diff.json @@ -8,7 +8,7 @@ "to": { "version": null, "unreleased": true, - "catalogSha256": "716fa77785ced515d1ed4da68f5b0a4b26126b77d910daaa302b48d7a84db346" + "catalogSha256": "5b52b97d388f156b939457eff9c815174619798f2c8a376de6029fe6f3e9d82b" }, "catalogSchema": { "before": "pineforge-run-failure-catalog/v1", @@ -16,7 +16,26 @@ }, "metadataChanges": [], "added": [], - "changed": [], + "changed": [ + { + "code": "pine_map_capacity_exceeded", + "collection": "codes", + "fields": [ + { + "path": "english", + "beforePresent": true, + "afterPresent": true, + "before": [ + "Error on bar {bar_index}: A map cannot have more than {maxSize} key-value pairs. The current size is {currSize}." + ], + "after": [ + "Error on bar {bar_index}: A map cannot have more than {maxSize} key-value pairs. The current size is {currSize}.", + "Error before the first bar: A map cannot have more than {maxSize} key-value pairs. The current size is {currSize}." + ] + } + ] + } + ], "removed": [], "deprecated": [] } diff --git a/docs/adr/0001-kernel-adapter-boundary.md b/docs/adr/0001-kernel-adapter-boundary.md index b5eba096b..8bcb19849 100644 --- a/docs/adr/0001-kernel-adapter-boundary.md +++ b/docs/adr/0001-kernel-adapter-boundary.md @@ -843,16 +843,16 @@ The table records, at the tree the doc wave read, what each block described and | kernel site | the mechanism | (a) generic because | (b) a knob would be | pinned by | |---|---|---|---|---| -| `utc_floor_day_ms` `src/timeframe.cpp:187` (the `TradingView` comment `sha256:1868eeadaab06b664c5ba966a86e70d6de2ad74d3645afe1846ed736d0c2afce` :180-183) | daily boundaries are anchored at symbol-local midnight | the anchor is `spec.timezone`'s data, not a platform's; another venue is driven by naming its zone | a "which midnight" field over a zone the spec already names | `tests/test_native_calendar.cpp`, `tests/test_session_day_anchors.cpp` | -| `session_period_last_traded_close_ms` `src/timeframe.cpp:999` (the `TradingView` comment `sha256:a650d166e35a1e22188ff5c3ee50c5534519730913a3fe713cf0b93b0f66d4ca` :989-998) | a D/W/M period is finished at the last traded close of its last session | every venue calendar must answer "is this period over"; the answer is computed from `spec.session` and `spec.timezone` | a per-run completion-rule switch over a choice the session template already makes | `tests/test_calendar_aggregation_wm.cpp`, `tests/test_session_calendar_extra.cpp` | -| the session-day split `crosses_boundary` `src/timeframe.cpp:1077-1083` | calendar periods split on the session-day clock, not the civil one | continuous-session instruments need a session-day clock whatever platform asks; the class comes from `spec.type` | hash-visible surface for what `spec.type` decides | `tests/test_session_day_anchors.cpp` | -| the aggregated bar's label `bar_label_ms` `src/timeframe.cpp:2120-2127` | an aggregated bar is dated by the period it opens | a label rule is needed to key any bucket; this one is the period key | a label-policy field for a key the aggregator owns | `tests/test_calendar_aggregation_wm.cpp`, `tests/test_calendar_wm_open_utc_fastpath.cpp` | -| real-end and chart-close completion (`Real-end completion` `src/timeframe.cpp:1470-1491`, `Chart-close completion` `:1507-1526`) | a thin or session-clipped intraday bucket completes on the bar whose end reaches the bucket end, or on the session's last bar | both are "the period is over" tests over feed data; a gap-free 24x7 feed is bit-identical either way, which the comment states and the tests pin | a completion-mode field over a bucket the calendar already closes | `tests/test_calendar_aggregation_wm.cpp`, `tests/test_native_htf_subscriptions.cpp` | -| the native period partition installed by `set_native_periods` `src/timeframe.cpp:1981` (the holiday-pause `TradingView` comment `sha256:7bd87c1e6dfe78729221d412952249a8cb427934ef498d3a4deb6812f7af05d7` :1714-1719) | a session that pauses and reopens the same day is one session day | the rule reads the feed's own stamps; the holiday is data | a holiday table in the kernel | `tests/test_calendar_aggregation_wm.cpp` | +| `utc_floor_day_ms` `src/timeframe.cpp:205` (the `TradingView` comment `sha256:1868eeadaab06b664c5ba966a86e70d6de2ad74d3645afe1846ed736d0c2afce` :198-201) | daily boundaries are anchored at symbol-local midnight | the anchor is `spec.timezone`'s data, not a platform's; another venue is driven by naming its zone | a "which midnight" field over a zone the spec already names | `tests/test_native_calendar.cpp`, `tests/test_session_day_anchors.cpp` | +| `session_period_last_traded_close_ms` `src/timeframe.cpp:1017` (the `TradingView` comment `sha256:a650d166e35a1e22188ff5c3ee50c5534519730913a3fe713cf0b93b0f66d4ca` :1007-1016) | a D/W/M period is finished at the last traded close of its last session | every venue calendar must answer "is this period over"; the answer is computed from `spec.session` and `spec.timezone` | a per-run completion-rule switch over a choice the session template already makes | `tests/test_calendar_aggregation_wm.cpp`, `tests/test_session_calendar_extra.cpp` | +| the session-day split `crosses_boundary` `src/timeframe.cpp:1095-1101` | calendar periods split on the session-day clock, not the civil one | continuous-session instruments need a session-day clock whatever platform asks; the class comes from `spec.type` | hash-visible surface for what `spec.type` decides | `tests/test_session_day_anchors.cpp` | +| the aggregated bar's label `bar_label_ms` `src/timeframe.cpp:2138-2145` | an aggregated bar is dated by the period it opens | a label rule is needed to key any bucket; this one is the period key | a label-policy field for a key the aggregator owns | `tests/test_calendar_aggregation_wm.cpp`, `tests/test_calendar_wm_open_utc_fastpath.cpp` | +| real-end and chart-close completion (`Real-end completion` `src/timeframe.cpp:1488-1509`, `Chart-close completion` `:1525-1544`) | a thin or session-clipped intraday bucket completes on the bar whose end reaches the bucket end, or on the session's last bar | both are "the period is over" tests over feed data; a gap-free 24x7 feed is bit-identical either way, which the comment states and the tests pin | a completion-mode field over a bucket the calendar already closes | `tests/test_calendar_aggregation_wm.cpp`, `tests/test_native_htf_subscriptions.cpp` | +| the native period partition installed by `set_native_periods` `src/timeframe.cpp:1999` (the holiday-pause `TradingView` comment `sha256:7bd87c1e6dfe78729221d412952249a8cb427934ef498d3a4deb6812f7af05d7` :1732-1737) | a session that pauses and reopens the same day is one session day | the rule reads the feed's own stamps; the holiday is data | a holiday table in the kernel | `tests/test_calendar_aggregation_wm.cpp` | | `bar_path_uses_high_first` `src/engine_path_resolve.cpp:28`, `compute_ohlc_path_legs` `src/magnifier.cpp:97`, `entry_stop_first_touch` `src/engine_internal.hpp:101-109` | with only OHLC the intrabar walk order is unknowable, so the open-proximity heuristic picks the first leg | **and the knob exists**: `NativeRunSpec::path_order` (`NativePathOrder` `native_run_spec.hpp:335`) states `HighFirst` / `LowFirst` explicitly | already spelled, and `Auto` is the default that moves no identity | `tests/test_native_run_spec.cpp`, the private test `test_native_price_grid` | | `round_to_mintick` `include/pineforge/engine.hpp:1059` (the `TradingView` comment `sha256:310354ab8c45921821040cda26d47633fe601b415254bac22a4f1d04ece6485c` :1043-1056) | nearest-tick rounding by floor(price / tick + 0.5), without an epsilon | a tick ladder is the instrument's, declared as `price_tick`; the generic surface over it is `NativePriceGrid` + `NativeGridRounding` | TradingView's **per-order-kind** quantization is exactly the knob that would not be generic — it is the measured content of the `price_grid` native-only ruling above | the private test `test_native_price_grid` | -| `ta::ATR` `include/pineforge/ta.hpp:380` (the `TradingView` comment `sha256:4f6f6939aec2773684b114a13dd002eb58990d6d96c3056ad87361753faf7a0a` :392-399) | average true range over Wilder's `RMA` | Wilder's ATR is the textbook definition; the comment records which convention and the tape that pinned the recompute behaviour | a seeding/averaging field on an indicator whose definition is the convention | `tests/test_ta_rma_warmup.cpp`, `tests/test_recompute.cpp` | -| `ta::StdDev` `include/pineforge/ta.hpp:528` (the `TradingView` comment `sha256:6126db164233a3439c940f74e655cefc98b605e2f2b79a2fda5c51e5c4c36ae9` :515-527) | the population (biased) standard deviation | biased vs sample is a textbook choice, and the kernel states which | a per-instance switch is the *right* answer where two conventions genuinely compete, which is why `ta::EmaSeeding` exists and this one does not need it | `tests/test_ta_indicators_extras.cpp` | +| `ta::ATR` `include/pineforge/ta.hpp:386` (the `TradingView` comment `sha256:4f6f6939aec2773684b114a13dd002eb58990d6d96c3056ad87361753faf7a0a` :398-405) | average true range over Wilder's `RMA` | Wilder's ATR is the textbook definition; the comment records which convention and the tape that pinned the recompute behaviour | a seeding/averaging field on an indicator whose definition is the convention | `tests/test_ta_rma_warmup.cpp`, `tests/test_recompute.cpp` | +| `ta::StdDev` `include/pineforge/ta.hpp:534` (the `TradingView` comment `sha256:6126db164233a3439c940f74e655cefc98b605e2f2b79a2fda5c51e5c4c36ae9` :521-533) | the population (biased) standard deviation | biased vs sample is a textbook choice, and the kernel states which | a per-instance switch is the *right* answer where two conventions genuinely compete, which is why `ta::EmaSeeding` exists and this one does not need it | `tests/test_ta_indicators_extras.cpp` | | `float_band_eq` `include/pineforge/ta_compare_band.hpp:38` | an absolute 1e-10 float comparison for indicator equality | an epsilon band is needed by any float comparison; the width is the calibration | a band-width field on a comparison the indicators share | `tests/test_dmi_parity.cpp`, `tests/test_ta_osc_edge.cpp` | | the two delivery rules, `projected_first_index` `src/native_execution_consumer.cpp:10757` (lookahead) and `clear_if_gapped` `:10744` (gaps) | deliver a completed bucket at its **first** contributing bar (lookahead), and **clear** the series on a bar it publishes nothing on (gaps) | both are delivery rules over a bucket the aggregator already closed; neither reads a platform. They are spec fields: `NativeTimeframeSubscription::lookahead` / `::gaps` | already spelled, and both default to the rule that moves no identity | `tests/test_native_htf_subscriptions.cpp` | | `pf_native_subscription_v1::lookahead` `include/pineforge/native_c_api.h:2284`, `::gaps` `include/pineforge/native_c_api.h:2288` | the C spelling of those two rules | since the typed delivery enums both words are typed and named after the kernel's own delivery rules (`pf_native_lookahead_e` `include/pineforge/native_c_api.h:812`, `pf_native_gaps_e` `:823`); the C header no longer names `barmerge` at all, and the migration page maps Pine's words onto them — a *pointer to the reader's vocabulary*, not a justification | — | `tests/test_native_c_api.c` | @@ -969,8 +969,8 @@ questions: the mechanism, why it is generic, and why no knob is needed. | Kernel capability | Generic mechanism | Why no platform knob is added | Executed witness | |---|---|---|---| -| A calendar bucket closes once, on its period's actual last input bar: the close-once guard (`current_emitted_complete` `src/timeframe.cpp:1701`) and the next-known-bar rule (`period_changes` `src/timeframe.cpp:1848`) of `feed_calendar_mode` `src/timeframe.cpp:1600` | (a) A D / W / M bucket that one completion rule has closed is not closed again by a later bar of its period: that bar merges into the bucket, and the next period's first bar opens the next bucket without re-emitting it -- the invariant the intraday path already kept. Nor does a completion rule close a period while the next input bar the run holds still belongs to it; the aggregator's own period key decides, the key its boundary test splits on. Both read the run's input and period key alone: a session template that knows Monday to Friday, fed a venue's weekend session (NSE's Budget-day sessions of Sat 2025-02-01 and Sun 2026-02-01), ends that week on its last input bar, and a bar with no known successor (a stream, the feed's end) keeps the template's rule, once | (b) None. Whether a bar is its period's last is a fact of the input the run already holds, and closing one period twice is a defect, not a convention: no venue publishes a period's bar twice. A switch would restate the input. Calibration: TradingView's weekly bar of Tue 2026-01-27 .. Sun 02-01 completes once, on Sunday's 15:15 bar (the TradingView reference capture `w14-htf-w-nifty15`) | `tests/test_calendar_bucket_closes_once.cpp` (kernel-only: the Budget week with and without the next bar, a Saturday session, a daily input, the regular-week, month-end and daily controls, and a bare host's `W` series delivering the week once) | -| A late-opening day's intraday grid: `native_late_day_opens` `include/pineforge/timeframe.hpp:300`, installed on each intraday evaluator by the feed store's rule (d) (`set_native_day_opens` `src/engine_aux_security.cpp:277`) and carried by the chart's partition (`NativeDayPartition::late_opens` `include/pineforge/timeframe.hpp:283`), read through `intraday_bucket` `src/timeframe.cpp:352` | (a) The venue's own daily bars date where each day opened. A daily stamp later than its session-day's day stamp and before that day's scheduled close, with no input bar of the day before it, is a day that traded late, and that day's intraday grid starts at it instead of the day stamp, the open a bucket boundary whatever the width (`75` too): a request evaluator's buckets, `session_intraday_bucket_open_ms` and `tf_change` alike. Every other day keeps the day-stamp grid bit for bit. A second venue is driven by installing its own daily bars; the kernel holds no special-session calendar | (b) None: the policy knob is the authoritative-feed ruling's, installing the daily feed or not (`engine_aux_security.cpp`'s contract, now rule (d)), and a "grid anchor" field would restate the stamps the run installed. The alternative was a dated session-corrections input in the run spec: hash-visible spec surface and a calendar every host must curate, where the feed is already the venue's data. Calibration: NSE's Muhurat session of 2025-10-21 opens 13:45 IST, and TradingView's one `60` bar that day is stamped 13:45 (the TradingView reference captures `w14-htf-60-nifty15`, `w14-htf-60ema-nifty15`), as are its `time("60")` on all four bars and `timeframe.change("60")`, true on the 13:45 bar alone (`w15-time60-nifty15`) | `tests/test_intraday_grid_late_day_open.cpp` (kernel-only: which stamps open a day late -- a midnight- or close-stamped venue, an evening session after the close and a day the input shows trading do not; the `60` and `75` aggregators with and without the opens, and a finer feed that printed before the open; the chart partition's grid readers; a bare host's `60` series with `D` authoritative bars) | +| A calendar bucket closes once, on its period's actual last input bar: the close-once guard (`current_emitted_complete` `src/timeframe.cpp:1719`) and the next-known-bar rule (`period_changes` `src/timeframe.cpp:1866`) of `feed_calendar_mode` `src/timeframe.cpp:1618` | (a) A D / W / M bucket that one completion rule has closed is not closed again by a later bar of its period: that bar merges into the bucket, and the next period's first bar opens the next bucket without re-emitting it -- the invariant the intraday path already kept. Nor does a completion rule close a period while the next input bar the run holds still belongs to it; the aggregator's own period key decides, the key its boundary test splits on. Both read the run's input and period key alone: a session template that knows Monday to Friday, fed a venue's weekend session (NSE's Budget-day sessions of Sat 2025-02-01 and Sun 2026-02-01), ends that week on its last input bar, and a bar with no known successor (a stream, the feed's end) keeps the template's rule, once | (b) None. Whether a bar is its period's last is a fact of the input the run already holds, and closing one period twice is a defect, not a convention: no venue publishes a period's bar twice. A switch would restate the input. Calibration: TradingView's weekly bar of Tue 2026-01-27 .. Sun 02-01 completes once, on Sunday's 15:15 bar (the TradingView reference capture `w14-htf-w-nifty15`) | `tests/test_calendar_bucket_closes_once.cpp` (kernel-only: the Budget week with and without the next bar, a Saturday session, a daily input, the regular-week, month-end and daily controls, and a bare host's `W` series delivering the week once) | +| A late-opening day's intraday grid: `native_late_day_opens` `include/pineforge/timeframe.hpp:300`, installed on each intraday evaluator by the feed store's rule (d) (`set_native_day_opens` `src/engine_aux_security.cpp:277`) and carried by the chart's partition (`NativeDayPartition::late_opens` `include/pineforge/timeframe.hpp:283`), read through `intraday_bucket` `src/timeframe.cpp:370` | (a) The venue's own daily bars date where each day opened. A daily stamp later than its session-day's day stamp and before that day's scheduled close, with no input bar of the day before it, is a day that traded late, and that day's intraday grid starts at it instead of the day stamp, the open a bucket boundary whatever the width (`75` too): a request evaluator's buckets, `session_intraday_bucket_open_ms` and `tf_change` alike. Every other day keeps the day-stamp grid bit for bit. A second venue is driven by installing its own daily bars; the kernel holds no special-session calendar | (b) None: the policy knob is the authoritative-feed ruling's, installing the daily feed or not (`engine_aux_security.cpp`'s contract, now rule (d)), and a "grid anchor" field would restate the stamps the run installed. The alternative was a dated session-corrections input in the run spec: hash-visible spec surface and a calendar every host must curate, where the feed is already the venue's data. Calibration: NSE's Muhurat session of 2025-10-21 opens 13:45 IST, and TradingView's one `60` bar that day is stamped 13:45 (the TradingView reference captures `w14-htf-60-nifty15`, `w14-htf-60ema-nifty15`), as are its `time("60")` on all four bars and `timeframe.change("60")`, true on the 13:45 bar alone (`w15-time60-nifty15`) | `tests/test_intraday_grid_late_day_open.cpp` (kernel-only: which stamps open a day late -- a midnight- or close-stamped venue, an evening session after the close and a day the input shows trading do not; the `60` and `75` aggregators with and without the opens, and a finer feed that printed before the open; the chart partition's grid readers; a bare host's `60` series with `D` authoritative bars) | ### Lane XAU-CAL: rule 2 ruling diff --git a/docs/design/BARS-BIN-INPUT.md b/docs/design/BARS-BIN-INPUT.md index 974caae52..cb126ec1e 100644 --- a/docs/design/BARS-BIN-INPUT.md +++ b/docs/design/BARS-BIN-INPUT.md @@ -205,8 +205,8 @@ Selected-window selection includes a report policy, any window option or `--validate-window-only` (`_selected_requested` docker/run_json.py:4741). The refusals above use the ordinary request code, not a window-specific code (`_request_invalid` docker/run_json.py:4507). The catalog names -`run_request_invalid` docker/run_failure_codes.json:1642 and includes the -option `ohlcv_bin` docker/run_failure_codes.json:1667. +`run_request_invalid` docker/run_failure_codes.json:1643 and includes the +option `ohlcv_bin` docker/run_failure_codes.json:1668. These request refusals exit 1 and emit the failure JSON line on stdout (`exit_status` docker/run_json.py:4507; `write_failure` docker/run_json.py:2366). @@ -224,7 +224,7 @@ Read-only searches covered `docker/`, `scripts/`, `runner/` and repository filenames/release notes; `campaign/` and `cloudrun/` are absent. The local backtest entry is `main` docker/run_json.py:5388; the catalog itself identifies the optimizer-only refusal separately (`description` -docker/run_failure_codes.json:2748). Neither establishes the production HPO +docker/run_failure_codes.json:2749). Neither establishes the production HPO runner's input support. Check the separate **pineforge-hpo** repository named by this repository's @@ -336,8 +336,8 @@ Failure JSON uses `engine`, `error`, `code`, `args` (`doc` docker/run_json.py:2322; `code` docker/run_json.py:2325; `args` docker/run_json.py:2326). The catalog entries are `chart_bars_unreadable` docker/run_failure_codes.json:60, `symbol_feeds_refused` -docker/run_failure_codes.json:2288 and `run_request_invalid` -docker/run_failure_codes.json:1642. Shared binary-reader helpers additionally +docker/run_failure_codes.json:2289 and `run_request_invalid` +docker/run_failure_codes.json:1643. Shared binary-reader helpers additionally refuse malformed file `size`, metadata/digests as `sidecar`, and a bad feed companion as `close_file` (`bar_count` docker/pf_bars.py:101; `check_sidecar` docker/pf_bars.py:137; `check_close_size` diff --git a/docs/live-ledger-budget.md b/docs/live-ledger-budget.md index 80f56157b..8606e52a8 100644 --- a/docs/live-ledger-budget.md +++ b/docs/live-ledger-budget.md @@ -115,7 +115,7 @@ transaction's report state before its admission. The input is refused before growth if its cached pages plus that margin do not fit. A refused boundary input leaves database, WAL and SHM byte-identical, including repeated refusals. Charging the cumulative payload to the boundary input is intended -(`Ledger::commit_input`, `runner/store.cpp:1348`). A boundary input can therefore +(`Ledger::commit_input`, `runner/store.cpp:1352`). A boundary input can therefore hit a storage stop when the input alone would fit, up to the cumulative payload's reservation earlier than strictly necessary. This cost buys the guarantee that no input commits with a checkpoint admitted without room in its @@ -142,8 +142,8 @@ Startup and recovery checkpoints also admit their actual cached pages before COMMIT, using the configured budget without an input reservation. They are optional on refusal and publish the same diagnostic. This is intended for both startup and post-replay checkpoints (`Ledger::checkpoint_recovered_report`, -`runner/store.cpp:1493`; the post-replay call is `checkpoint_recovered_report`, -`runner/main.cpp:1373`): with a large budget, either can spend most of the +`runner/store.cpp:1497`; the post-replay call is `checkpoint_recovered_report`, +`runner/main.cpp:1384`): with a large budget, either can spend most of the remaining headroom on an O(report) payload before the first live input. The run can stop sooner, never over budget. Opening files already over a newly supplied finite budget refuses with `LedgerBudgetExceeded` before diff --git a/docs/native-refactor-progress.md b/docs/native-refactor-progress.md index aa57cb52a..fa894e4e7 100644 --- a/docs/native-refactor-progress.md +++ b/docs/native-refactor-progress.md @@ -184,7 +184,7 @@ them for the record only. "Linux aarch64" is the maintainers' Cortex-A725 host. | A checked-values handoff for executions | `execution_values` runs twice, in `check_execution` (`src/native_order.cpp:7261`) and again in `apply_execution` (`src/native_order.cpp:7292`); handing the checked values across needs a public type, an API addition with no layout change | As measured above: 2.21 % of bracket for the two together, about 1.1 % recoverable | Deferral ruling 2: "a public checked-values handoff for executions (~1.1% bracket) -> DEFERRED" | As above | A maintainer decision on the API addition | | A chunked `trades_` store | Closed trades in chunks rather than one vector; `trades_` (`include/pineforge/engine.hpp:729`) is protected `BacktestEngine` state, so script ABI | As measured above: on bracket, vector growth 0.64 %, `Trade` moves 0.74 % and `record_close_trade` 0.43 % (at most about 1.4 %); at most 0.35 % on the 14 probes | Deferral ruling 4: "chunked trades_ store (ABI) -> DEFERRED" | As above | An `engine_script_run` epoch that changes the engine's protected layout | | A zoned chart-day memo (finding 4 of the UTC memo study) | On a chart with a timezone, `libc_chart_day_key` (`src/source/pine_adapter.cpp:19006`) takes the process timezone lock through `ScopedTimezone` and calls `localtime_r` on every bar, because `on_bar_open` (`src/source/pine_adapter.cpp:24007`) reads `chart_day_key` (`src/source/pine_adapter.cpp:24245`) unconditionally. The non-UTC monthly Sharpe/Sortino walk, `month_key_local` (`src/engine_metrics.cpp:55`), pays one `localtime_r` per equity point. UTC charts use the UTC chart-day memo's memo, `civil_chart_day_key` (`src/source/pine_adapter.cpp:18990`) | Not measured on a zoned chart. The UTC path cost 134 instructions a bar before the UTC chart-day memo's memo and 68 after (the UTC chart-day memo, Linux aarch64, callgrind on the floor). The timezone lock profiling: `ScopedTimezone` copies the zone name, which allocates for names of 16 or more characters under libstdc++ | the deferred-work review §D: "the UTC chart-day memo finding 4 (the timezone `localtime_r` memo)", under the maintainer's cap. the UTC memo report: an exact zoned memo "would need the zone's transition table, which belongs in `timezone.cpp`" | None; the UTC memo's witness is `tests/test_chart_day_memo.cpp` | A change to `src/timezone.cpp` that builds an exact zone-transition memo (the timezone lock profiling's Sitka case shows why a shortcut is not exact), byte-identical, with the zoned keys of `test_chart_day_memo` still equal to the computed ones | -| Thread-local reads that remain per call | The per-pump state cache moved the three scopes' writes onto the pump's block, `pump_ambient` (`src/native_execution_consumer.hpp:692`). A reader still reaches the block through `tl_runtime_ambient` (`src/ta_extremes_volume.cpp:30`): `bar_context` (`src/ta_extremes_volume.cpp:83`), read by every `ExtremeRing::update` (`src/ta_extremes_volume.cpp:123`), which backs `ta.highest`, `ta.lowest`, `ta.highestbars` and `ta.lowestbars` and, through an embedded `Highest` and `Lowest`, `ta.stoch`, `ta.wpr` and `ta.range`; and `matching_day_partition` (`src/timeframe.cpp:687`), read by the default-anchored `ta.vwap` through `session_day_index`, by `crosses_boundary` on a DAY period (`timeframe.change("D")`) and by the session-period helpers. `decompose_ms_local` (`src/session_time.cpp:250`) reads two thread-locals per call on a zoned clock. The prepared order storage now caches one idle block per token type through `free_block` (`src/native_order.cpp:1001`), replacing the former `free_blocks` vector; prepared allocation and deallocation still reach that thread-local cache. `ema_na_warmup_flag` is read once per EMA, at its first compute | One thread-local access per reader call. The fourth review, macOS arm64, a dlopen'd tutorial-MACD probe, counted by `_tlv_get_addr` breakpoints and by descriptor patching: 1.000 a bar for one `ta::Highest` and for one default `ta::VWAP`, 1.999 for one `pine_hour` on a New York clock, 0.000 on the no-order floor and with orders. On Linux a TLS-descriptor call costs about 20 instructions: the per-pump state cache's 13 → 4 calls a bar was 260 → 80 instructions (Linux aarch64, GCC 13, callgrind). The accepted integration: 2.00 a bar on the zoned probe 064, 0.08 on probe 008 | None for the TA and day-partition readers: the integration acceptance (ledger, 2026-09-24 23:30 UTC) reads "dlopen TLS 13->0/bar", and the accepted integration's report names only the timezone and order-core residuals. This row is their record | Unpublished (the fourth review's scratch) | A byte-identical change that hands these readers the pump's block without a thread-local read. `ta::bar_context()` is script ABI, so its public declaration stays as it is | +| Thread-local reads that remain per call | The per-pump state cache moved the three scopes' writes onto the pump's block, `pump_ambient` (`src/native_execution_consumer.hpp:692`). A reader still reaches the block through `tl_runtime_ambient` (`src/ta_extremes_volume.cpp:30`): `bar_context` (`src/ta_extremes_volume.cpp:83`), read by every `ExtremeRing::update` (`src/ta_extremes_volume.cpp:123`), which backs `ta.highest`, `ta.lowest`, `ta.highestbars` and `ta.lowestbars` and, through an embedded `Highest` and `Lowest`, `ta.stoch`, `ta.wpr` and `ta.range`; and `matching_day_partition` (`src/timeframe.cpp:705`), read by the default-anchored `ta.vwap` through `session_day_index`, by `crosses_boundary` on a DAY period (`timeframe.change("D")`) and by the session-period helpers. `decompose_ms_local` (`src/session_time.cpp:250`) reads two thread-locals per call on a zoned clock. The prepared order storage now caches one idle block per token type through `free_block` (`src/native_order.cpp:1001`), replacing the former `free_blocks` vector; prepared allocation and deallocation still reach that thread-local cache. `ema_na_warmup_flag` is read once per EMA, at its first compute | One thread-local access per reader call. The fourth review, macOS arm64, a dlopen'd tutorial-MACD probe, counted by `_tlv_get_addr` breakpoints and by descriptor patching: 1.000 a bar for one `ta::Highest` and for one default `ta::VWAP`, 1.999 for one `pine_hour` on a New York clock, 0.000 on the no-order floor and with orders. On Linux a TLS-descriptor call costs about 20 instructions: the per-pump state cache's 13 → 4 calls a bar was 260 → 80 instructions (Linux aarch64, GCC 13, callgrind). The accepted integration: 2.00 a bar on the zoned probe 064, 0.08 on probe 008 | None for the TA and day-partition readers: the integration acceptance (ledger, 2026-09-24 23:30 UTC) reads "dlopen TLS 13->0/bar", and the accepted integration's report names only the timezone and order-core residuals. This row is their record | Unpublished (the fourth review's scratch) | A byte-identical change that hands these readers the pump's block without a thread-local read. `ta::bar_context()` is script ABI, so its public declaration stays as it is | | "audit-off mode", "LTO", "data-oriented layout" | Named in "Maintainer decision 2026-09-25 00:20 UTC+08:00" among the post-release structural items: "S1 bulk quiet bars, S2 per-strategy specialization, audit-off mode, LTO+PGO, data-oriented layout". No engine or campaign document defines, designs or measures the three | None | The maintainer decision | None | Struck from this inventory until a design note defines one of them | ### Recorded witness drift diff --git a/docs/pages/abi-stability.md b/docs/pages/abi-stability.md index 5428717f2..f8e94d897 100644 --- a/docs/pages/abi-stability.md +++ b/docs/pages/abi-stability.md @@ -119,7 +119,7 @@ destruction. NULL is returned only for a NULL handle. Collection failure reads `[{"args":{},"code":"run_warnings_unavailable"}]`, never an empty array. Warnings do not change values, trades, execution status, failure accessors or state hashes. The accessor adds no C struct and changes neither existing C -layouts nor installed host class layouts; ABI 5 and stream layout 4 stay fixed. +layouts nor installed host class layouts; ABI 5 and stream layout 5 stay fixed. `strategy_set_account_currency_fx_scalar(s, rate)` is an additive runtime setter; feature-probe the symbol in older modules. It installs the host's @@ -144,7 +144,7 @@ broker-state hash folds the mark only when a scalar was supplied: a run that supplies none keeps its hash values, and a run that supplies one, through either setter, hashes differently from 1.12.0. The setter adds no C struct and changes neither existing C layouts nor the size of an installed host class; -ABI 5 and stream layout 4 stay fixed. +ABI 5 and stream layout 5 stay fixed. ## Symbol inventory diff --git a/docs/pages/memory-management.md b/docs/pages/memory-management.md index 01b494fa9..295838ac9 100644 --- a/docs/pages/memory-management.md +++ b/docs/pages/memory-management.md @@ -163,7 +163,7 @@ stamp, every mode (the default stream included, and a run without JSONL output) holds the scalar members of the cumulative JSON document and the rows of the newest input. Rows that a later input can no longer change are serialized once and appended to anonymous files beside the ledger -(`SpooledReportArray` runner/report_spool.hpp:286), and a `--report-jsonl` line +(`SpooledReportArray` runner/report_spool.hpp:290), and a `--report-jsonl` line is written to stdout from those files (`write_cumulative_line` runner/main.cpp:955) instead of being rebuilt from a retained document. With `--native-event-retention window` only, the @@ -181,14 +181,16 @@ keeps every event, both growing with the inputs processed. Under the window a polling host loses no command it acknowledges, and keeps no driver point or account row; every event let go unread is counted per kind by `native_events_dropped` (C: `strategy_native_events_dropped_v1`). -A report checkpoint above 512 inputs is folded through the same spooled arrays -(`fold_full_report` runner/store.cpp:363), reading every report row from +A report checkpoint above 512 inputs, and every checkpoint of a run with +`--native-event-retention window`, is folded through the same spooled arrays +(`fold_full_report` runner/store.cpp:364), reading every report row from cursor 0 where 1.13.0 started from the nearest earlier checkpoint; at the boundaries K, 2K, 4K, ... the checkpoints up to 2^m·K read (2^(m+1) − 1)·K rows in all, less than twice the last boundary's cursor. With `--native-event-retention window` (`LedgerOptions::stream_large_checkpoints`), a payload above the 1 MiB -JSON frame is written into its row in pieces (`write_report_checkpoint` -runner/store.cpp:476) and stored as a BLOB with the same bytes; under a ledger +JSON frame is written into its row in pieces, at any cursor +(`write_report_checkpoint` runner/store.cpp:480) and stored as a BLOB with the +same bytes; under a ledger byte budget that write is admitted whole first and then lets SQLite spill its pages, so it does not wait in memory for its commit. Without the option the payload is built as one string and stored as TEXT, as in 1.13.0. The ledger's @@ -201,7 +203,12 @@ in a realtime stream) and the curve's memory-filesystem warning apply only with `--native-event-retention window`. What grows with the run is on disk: the ledger, the serialized settled rows (about the size of one cumulative report) and, with `window`, **24 B per report point** of curve scratch. -`--max-ledger-bytes` does not count the spool files or the curve scratch. +`--max-ledger-bytes` does not count the spool files or the curve scratch. The +engine's own scratch files (the spilled part of its trade ledger and its +command journals, among others) are not beside the ledger: they are unlinked +files in `TMPDIR`, or in `/tmp` when it is unset, so `TMPDIR` needs a +disk-backed directory with room for them (see [What you can +set](@ref memory_controls)). A spool write that returns an error (a full disk, an exceeded quota, an I/O error), or that would take the file past the process's per-file size limit (`ulimit -f`, RLIMIT_FSIZE; read before the write, so a spool never raises SIGXFSZ), does not stop the run: `ReportSpool` (runner/report_spool.hpp) keeps the bytes it could @@ -230,7 +237,7 @@ export command, which folds the report as a tree; and the strategy's own stores that the rest of this page lists. `test_native_live_report_memory` checks it (`report_memory` -tests/test_native_live_report_cli.cpp:1249). The row drives the real runner's +tests/test_native_live_report_cli.cpp:1252). The row drives the real runner's default cumulative stream and default ratio interval, with `--native-event-retention window`, on seeded inputs, on a native test host that records one equity point per bar, at the sizes of @@ -382,8 +389,8 @@ trailer, 7 internal error, 8 `sink_exhausted` (the stream exceeds `--max-bytes`; `sink_exhausted` at `docker/pineforge_sink_finalize.py:115-123`). Its output is provisional until exit 0, and a consumer must discard everything it received on exit 3 to 8 (`PROVISIONAL` at `docker/pineforge_sink_finalize.py:24-26`). The failure catalog maps exit 8 to `report_sink_failed` -with `reason` `sink_exhausted` (`sink_exhausted` at `docker/run_failure_codes.json:1192`). A sink failure carries code `report_sink_failed` and a `reason` -from the closed list of the catalog (`report_sink_failed` at `docker/run_failure_codes.json:1160`): +with `reason` `sink_exhausted` (`sink_exhausted` at `docker/run_failure_codes.json:1193`). A sink failure carries code `report_sink_failed` and a `reason` +from the closed list of the catalog (`report_sink_failed` at `docker/run_failure_codes.json:1161`): `arena_exhausted`, `arena_io`, `arena_backing`, `scratch_io`, `sink_io`, `sink_peer_closed` and `sink_exhausted`. Only some reasons carry sizes: `arena_exhausted` carries `high_water_bytes`, `arena_bytes`, `claimed_bytes`, `store` and `needed_bytes_lower_bound`, `sink_exhausted` carries diff --git a/docs/pages/pine-to-native.md b/docs/pages/pine-to-native.md index eb397d41c..0fa7fc7f8 100644 --- a/docs/pages/pine-to-native.md +++ b/docs/pages/pine-to-native.md @@ -707,7 +707,7 @@ it, so a valid length gives that class's values. The adapters do not touch ### Series-length SMA source adapter {#pine_to_native_series_sma} -`pineforge::source::SeriesSma` (include/pineforge/source/pine_ta_length.hpp:519) +`pineforge::source::SeriesSma` (include/pineforge/source/pine_ta_length.hpp:520) provides an SMA with a length supplied on every evaluation in the Pine source adapter. Keep one instance per written call site and evaluation context. Its public signatures are below. This is an unwired library primitive: codegen @@ -728,7 +728,7 @@ existing `ta_number` overloads (include/pineforge/source/pine_ta_length.hpp:68) for `double`, `int` and `std::int64_t`, so integer `na` sentinels reach the adapter as a double NaN. -`compute` (src/source/pine_ta_length.cpp:506) makes one logical update per call, +`compute` (src/source/pine_ta_length.cpp:528) makes one logical update per call, starting at the site's first execution. A non-`na` source appends one retained observation; an `na` source contributes no sample. Each call uses its current length to select the last N non-`na` observations and returns `na` while fewer @@ -739,7 +739,7 @@ access to the retained observations, capped at the last keep the same instance and ring; equal source values on successive calls remain distinct observations. -`checked_length` (src/source/pine_ta_length.cpp:469) validates before either +`checked_length` (src/source/pine_ta_length.cpp:471) validates before either entry point records or replaces an update. Finite fractional lengths truncate toward zero. A nonpositive truncated length throws the family's `length_not_positive` error for `sma`; `na` and other nonfinite lengths throw @@ -748,7 +748,7 @@ above 15001 throws the family's `max_bars_back_exceeded` error with `pine_runtime_limit`. Invalid calls leave the logical checkpoint unchanged. For a valid length, warm-up returns `na` until N non-`na` samples are retained. -`recompute` (src/source/pine_ta_length.cpp:527) replaces the most recent logical +`recompute` (src/source/pine_ta_length.cpp:549) replaces the most recent logical update from its pre-update ring head/count, including the oldest value evicted by a full-ring push. Replacing a value with `na` restores that evicted value; replacing `na` with a value inserts one observation, evicting the oldest if @@ -757,9 +757,9 @@ On an unused instance it creates the first logical update. Copy construction and assignment preserve the ring, head/count, saved eviction and replacement state, with storage detached before mutation. -`mean` (src/source/pine_ta_length.cpp:487) sums the selected window afresh from -oldest to newest with Neumaier compensated double arithmetic, then divides -by N. The finite TradingView clock/source-`na` +`mean` (src/source/pine_ta_length.cpp:489) sums the selected window afresh. +For finite sources without intermediate overflow, it rounds the exact sum +once, as fixed SMA does, then divides by N. The finite TradingView clock/source-`na` packet supports the non-`na` sample membership correction; its complete prefix through bar 16 is embedded in `tv_clock_na_prefix` (tests/test_series_sma.cpp:241). This evidence does not qualify runtime @@ -769,25 +769,17 @@ transport, infinity, overflow and signed zero remain unqualified. Sparse-call routing, repeated calls of the same site within a bar, same-bar re-execution after multiple calls and requested-context clocks require separate qualification. These ordinary C++ API rules do not establish Pine overload -admission or TradingView parity for those contexts. The 1.13.0 variable-length -path temporarily uses Neumaier arithmetic. -Constant-length calls must retain existing `ta::SMA` and its window arithmetic -(the exact sum of the window, rounded once; until November 2026 a sliding -Kahan sum with compensated addends); this unwired primitive does not establish -that future codegen route. Alignment of the variable-length arithmetic with -fixed SMA remains planned after 1.13.0. The ordinary unit target's full -1441-row feed comparison at N=20 exercises the variable adapter, measures -numeric differences and maximum ULP distance against real fixed `ta::SMA`, -and reports the first and maximum rows with times and hex values. That -measurement is neither exact-alignment nor generated constant-script coverage; -its metrics require actual execution. +admission or TradingView parity for those contexts. Constant-length calls +continue to use `ta::SMA`. The unit test covers a cancellation where the two +paths used to disagree. Its separate 1441-row feed comparison at N=20 reports +difference counts and maximum ULP distance against fixed `ta::SMA`. Storage is one ring of at most 15001 non-`na` observations plus a scalar rollback checkpoint; it does not grow with bars or requested N. -`ensure_capacity` (src/source/pine_ta_length.cpp:479) lazily creates that fixed -ring. The cache policy controls residency, and selecting a window takes O(N) -work. `observe_owned_storage` -(include/pineforge/source/pine_ta_length.hpp:525) reports owned history through +`ensure_capacity` (src/source/pine_ta_length.cpp:481) lazily creates that fixed +ring. The cache policy controls residency, and summation uses bounded local +scratch and takes O(N) work. `observe_owned_storage` +(include/pineforge/source/pine_ta_length.hpp:526) reports owned history through the existing storage accounting. Allocation, capacity and scratch I/O failures propagate. Ring/checkpoint fields are published after required writes succeed, including restoration of an evicted oldest value. diff --git a/docs/pages/report-schema.md b/docs/pages/report-schema.md index a62eba713..9dabbe40b 100644 --- a/docs/pages/report-schema.md +++ b/docs/pages/report-schema.md @@ -454,7 +454,7 @@ preserved; extra element keys are ignored and malformed elements are dropped. Metrics-only or sink-only shapes that bypass these two report assembly sites do not carry the key. Warnings are outside provenance, fingerprints, trade CSVs and sink records. This optional field is a minor report-schema addition; -ABI 5, stream layout 4, sink wire and fingerprint versions are unchanged. +ABI 5, stream layout 5, sink wire and fingerprint versions are unchanged. All three cache fields are absent at zero; a cache disabled by policy adds no event. Older modules without the getter omit all three fields. Admission starts a new diff --git a/include/pineforge/pineforge.h b/include/pineforge/pineforge.h index 59c4acefb..7f45def0d 100644 --- a/include/pineforge/pineforge.h +++ b/include/pineforge/pineforge.h @@ -1937,10 +1937,10 @@ typedef struct pf_version_s { int major; /**< Major version. */ int minor; /**< Minor version. */ int patch; /**< Patch version. */ - const char* commit_sha; /**< Short git commit SHA, or `""` if unknown. */ + const char* commit_sha; /**< Build Git commit SHA (abbreviation length varies), or `""` if unavailable. */ } pf_version_t; -/** @return Linked runtime version. */ +/** @return Linked runtime version. The build SHA can be unavailable even in a Git checkout. */ PF_API pf_version_t pf_version_get(void); /** @return Monotonic ABI version (see #PF_ABI_VERSION). */ @@ -1953,8 +1953,8 @@ PF_API int pf_abi_version(void); * checkout, or exactly the `VERSION` file (`"MAJOR.MINOR.PATCH"`, or * `"MAJOR.MINOR.PATCH-rc.N"` for a release candidate) for a tarball build * or one configured with `PINEFORGE_VERSION_SOURCE=FILE`. #pf_version_get - * carries only the numeric MAJOR.MINOR.PATCH. The pointer is to a static - * string with program lifetime; do not free. */ + * exposes MAJOR.MINOR.PATCH in its numeric fields. The returned string has + * program lifetime; do not free it. */ PF_API const char* pf_version_string(void); /** @} */ /* end of pf_version */ diff --git a/include/pineforge/source/pine_ta_length.hpp b/include/pineforge/source/pine_ta_length.hpp index 7410b2b2d..67b3124cb 100644 --- a/include/pineforge/source/pine_ta_length.hpp +++ b/include/pineforge/source/pine_ta_length.hpp @@ -510,8 +510,9 @@ class PineSupertrend { // recompute replaces the most recent logical update: restore its pre-update // head/count and any evicted oldest value, then record only a non-na source. // Repeated recompute uses that same checkpoint; unused instances start there. -// Cache capacity controls residency only. Sum oldest first, with Neumaier -// double compensation. Finite TV clock/source-na evidence supports sample +// Cache capacity controls residency only. For finite inputs without overflow, +// divide the exact window sum, rounded once as in fixed SMA, by the length. +// Finite TV clock/source-na evidence supports sample // membership only; numerical order/domains, runtime overrides, whole-history // TV parity and repeated/requested-context Pine clocks/rollback are unqualified. // Copy/assignment preserve the ring and complete pre-update checkpoint; diff --git a/include/pineforge/ta.hpp b/include/pineforge/ta.hpp index 8dd6c0002..fa0bce568 100644 --- a/include/pineforge/ta.hpp +++ b/include/pineforge/ta.hpp @@ -20,8 +20,8 @@ enum class EmaSeeding : unsigned char { // The default, and byte-identical to every EMA before the option existed. FirstValue = 0, // na until ``length`` finite inputs have accumulated, then their simple - // average, then the ordinary recursion: an SMA-seeded EMA, warming up the - // way ``RMA`` and ``SMA`` already do (na for the whole warm-up window). + // average (the exact sum rounded once, then divided by length), then the + // ordinary recursion. SimpleAverage = 1, }; @@ -302,6 +302,10 @@ class EMA { bool na_warmup_ = false; bool warmup_latched_ = false; + // Only SimpleAverage retains sources, through the last replayable seed + // bar. Each warm-up observation adds one source after save(). + std::vector seed_sources_; + public: explicit EMA(int length); // Names the seeding at construction; the ambient default is never read. @@ -316,8 +320,10 @@ class EMA { double recompute(double src); double recompute(double src, bool input_ready); public: - // All retained fields are inline scalars (or scalar-only aggregates). - void observe_owned_storage(OwnedStorage&) const noexcept {} + void observe_owned_storage(OwnedStorage& out) const noexcept { + if (seed_sources_.capacity() != 0) + observe_owned_members(out, seed_sources_); + } }; // --- MACD --- diff --git a/include/pineforge/timeframe.hpp b/include/pineforge/timeframe.hpp index fa15b643d..9318ba409 100644 --- a/include/pineforge/timeframe.hpp +++ b/include/pineforge/timeframe.hpp @@ -36,7 +36,7 @@ struct FixedPeriodClock { /// Day-based: "D","1D" => 86400 /// Week-based: "W","1W" => 604800 /// Month-based: "M","1M" => -1 (calendar-based, not fixed seconds) -/// Day counts whose seconds do not fit int also return -1 (calendar-only). +/// Seconds past int: -1 (calendar-only) for a positive D/W count, else out_of_range. int tf_to_seconds(const std::string& tf); /// Extract the numeric multiplier from a TF string (e.g. "15" -> 15, "D" -> 1). diff --git a/include/pineforge/version.h.in b/include/pineforge/version.h.in index c2214877a..dd4ee994d 100644 --- a/include/pineforge/version.h.in +++ b/include/pineforge/version.h.in @@ -19,7 +19,7 @@ * PINEFORGE_VERSION_SOURCE=FILE. */ #define PINEFORGE_VERSION_FULL "@PINEFORGE_VERSION_FULL@" -/* Short git sha (7 chars) or "unknown" if built outside a git checkout. */ +/* Build Git SHA (abbreviation length varies), or "unknown" if unavailable, even in a checkout. */ #define PINEFORGE_GIT_SHA "@PINEFORGE_VERSION_GIT_SHA@" #endif /* PINEFORGE_VERSION_H */ diff --git a/runner/CMakeLists.txt b/runner/CMakeLists.txt index c8e313ff1..78e1b07fd 100644 --- a/runner/CMakeLists.txt +++ b/runner/CMakeLists.txt @@ -625,6 +625,21 @@ if(PINEFORGE_BUILD_TESTS) target_link_libraries(test_live_report PRIVATE PineForge::pineforge) target_compile_options(test_live_report PRIVATE -UNDEBUG) add_test(NAME test_live_report COMMAND test_live_report) + if(CMAKE_SYSTEM_NAME STREQUAL "Linux") + add_executable(test_report_spool_unlink_failure + ${PROJECT_SOURCE_DIR}/tests/test_report_spool_unlink_failure.cpp) + target_link_libraries(test_report_spool_unlink_failure PRIVATE PineForge::pineforge) + target_compile_options(test_report_spool_unlink_failure PRIVATE -UNDEBUG) + target_link_options(test_report_spool_unlink_failure PRIVATE "LINKER:--wrap=open" "LINKER:--wrap=unlink") + add_test(NAME test_report_spool_unlink_failure COMMAND test_report_spool_unlink_failure) + endif() + + if(CMAKE_SYSTEM_NAME STREQUAL "Linux") + add_executable(test_service_unlink_failure ${PROJECT_SOURCE_DIR}/tests/test_service_unlink_failure.cpp) + target_link_libraries(test_service_unlink_failure PRIVATE pineforge_live_support) + target_link_options(test_service_unlink_failure PRIVATE "LINKER:--wrap=unlink") + add_test(NAME test_service_unlink_failure COMMAND test_service_unlink_failure) + endif() add_executable(test_live_report_accumulator ${PROJECT_SOURCE_DIR}/tests/test_live_report_accumulator.cpp) target_include_directories(test_live_report_accumulator PRIVATE ${CMAKE_CURRENT_SOURCE_DIR}) @@ -658,6 +673,15 @@ if(PINEFORGE_BUILD_TESTS) add_test(NAME test_native_live_report_event_retention COMMAND test_native_live_report_cli $ $ event-retention) set_tests_properties(test_native_live_report_event_retention PROPERTIES TIMEOUT 0) + add_test(NAME test_native_live_report_window_checkpoint_storage COMMAND test_native_live_report_cli + $ $ window-checkpoint-storage + $) + set_tests_properties(test_native_live_report_window_checkpoint_storage PROPERTIES TIMEOUT 0) + add_test(NAME test_native_live_report_window_checkpoint_streaming COMMAND test_native_live_report_cli + $ $ window-checkpoint-streaming) + set_tests_properties(test_native_live_report_window_checkpoint_streaming PROPERTIES + TIMEOUT 0 SKIP_RETURN_CODE 77) + add_dependencies(test_native_live_report_cli live_contract_native_stub) # A default run's equity export, pinned to the 1.13.0 build's SHA-256; 77 = not x86-64 Linux. add_test(NAME test_native_live_report_default_equity_export COMMAND test_native_live_report_cli $ $ default-equity-export) @@ -674,6 +698,19 @@ if(PINEFORGE_BUILD_TESTS) add_test(NAME test_native_live_report_spool_write_failure COMMAND test_native_live_report_cli $ $ spool-write-failure) set_tests_properties(test_native_live_report_spool_write_failure PROPERTIES TIMEOUT 0) + if(CMAKE_SYSTEM_NAME STREQUAL "Linux") + add_test(NAME test_native_live_report_spool_probe_failure COMMAND test_native_live_report_cli + $ $ spool-probe-failure) + set_tests_properties(test_native_live_report_spool_probe_failure PROPERTIES TIMEOUT 0) + endif() + add_executable(test_native_live_recovery_spool_warning + ${PROJECT_SOURCE_DIR}/tests/test_native_live_recovery_spool_warning.cpp) + target_link_libraries(test_native_live_recovery_spool_warning PRIVATE ${_pf_sqlite_target} Threads::Threads) + target_compile_options(test_native_live_recovery_spool_warning PRIVATE -UNDEBUG) + add_test(NAME test_native_live_recovery_spool_warning COMMAND test_native_live_recovery_spool_warning + $ $) + set_tests_properties(test_native_live_recovery_spool_warning PROPERTIES TIMEOUT 0) + add_dependencies(test_native_live_recovery_spool_warning pineforge-live native_live_report_example) add_executable(test_native_live_report_curve_staging_work ${PROJECT_SOURCE_DIR}/tests/test_native_live_report_curve_staging_work.cpp) target_include_directories(test_native_live_report_curve_staging_work PRIVATE ${CMAKE_CURRENT_SOURCE_DIR}) @@ -729,7 +766,6 @@ if(PINEFORGE_BUILD_TESTS AND PINEFORGE_BUILD_SOURCE_LAYER) COMMAND ${_pf_layout_proof_command} opaque-report) set_tests_properties(native_live_engine_layout_e2e native_live_opaque_report_e2e PROPERTIES TIMEOUT 0) - add_dependencies(test_native_live_report_cli live_contract_native_stub) add_test(NAME native_live_old_layout_report_e2e COMMAND ${Python3_EXECUTABLE} ${PROJECT_SOURCE_DIR}/tests/native_live_startup_e2e.py $ $ $ diff --git a/runner/README.md b/runner/README.md index 3c29dae73..1cc7d6c48 100644 --- a/runner/README.md +++ b/runner/README.md @@ -110,10 +110,11 @@ modules whose optional `strategy_stream_engine_layout_version()` export matches keeps the public-report path: full `strategy_stream_fill_report` snapshots and exact ratios at every input, even for K > 1. It adds neither `ratios_as_of_input` nor `ratio_context` and emits no terminal interval refresh. The runner prints no -warning about a missing stamp. A nonzero mismatched stamp is refused during -module admission, before `strategy_create` or any C++ handle read. Runtime -contexts require stamp 5; modules declaring a lower stamp must be rebuilt with the -exact paired engine and its script-run epoch. +warning about a missing stamp, except when `--native-event-retention` is given +(see that option). A nonzero mismatched stamp is refused during module +admission, before `strategy_create` or any C++ handle read. Runtime contexts +require stamp 5; modules declaring a lower stamp must be rebuilt with the exact +paired engine and its script-run epoch. The runner-owned accumulator carries trade-stat folds and the equity series needed for exact ratios. For modules with the matching stamp, the report's @@ -154,7 +155,11 @@ rows (about the size of one cumulative report) and, with `window`, 24 bytes per report point of curve scratch. These files are unlinked when created and vanish with the process; a restart rebuilds them during its replay. The ledger's directory therefore needs that much free space beyond the ledger itself, and -`--max-ledger-bytes` does not count it. Under `window` the curve scratch needs a disk-backed +`--max-ledger-bytes` does not count it. The engine's own scratch files (the +spilled part of its trade ledger and its command journals, among others) are +not in that directory: they are unlinked files in `TMPDIR`, or in `/tmp` when +it is unset, so `TMPDIR` needs a disk-backed directory with room for them (the +memory guide lists them). Under `window` the curve scratch needs a disk-backed directory: with the ledger on a memory filesystem (tmpfs or ramfs) the runner prints one `warning:` line at startup and keeps the curve in memory, as it did before. In every mode the settled rows are still spooled there, but the files of a memory @@ -251,16 +256,17 @@ payload length supplies the extra-payload reservation. After the input commits, the checkpoint is inserted in its own admitted transaction while retaining the writer mutex, with a byte ceiling no greater than the input's reservation. Charging that payload to the boundary input is intended -(`Ledger::commit_input`, `runner/store.cpp:1348`): the boundary input can hit a +(`Ledger::commit_input`, `runner/store.cpp:1352`): the boundary input can hit a storage stop when the input alone would fit, up to the cumulative payload's reservation earlier than strictly necessary. This cost buys the guarantee that no input commits with a checkpoint admitted without room in its reservation. A checkpoint build or write failure records `write_failed` and continues with the input durable. If the separate transaction's actual page admission needs more room than reserved, only the checkpoint rolls back; the diagnostic count -increases without latching a storage stop. Up to cursor 512 preparation builds -the cumulative document as a tree, as before. Above it, preparation folds -every report row from cursor 0 through spooled arrays in the ledger's +increases without latching a storage stop. Without `--native-event-retention +window`, up to cursor 512 preparation builds the cumulative document as a +tree, as before. Above it, and at every cursor with that option on a module that has the engine layout stamp, preparation +folds every report row from cursor 0 through spooled arrays in the ledger's directory, where 1.13.0 started from the nearest earlier checkpoint. At the geometric boundaries K, 2K, 4K, ... the checkpoints up to 2^m·K therefore read (2^(m+1) − 1)·K report rows in all, less than twice the last boundary's @@ -270,14 +276,14 @@ no longer repeated by it; the bytes written are the same. The read runs inside the boundary input's own transaction, so that input's commit takes as long as the read. Where no spool file can be created or written in that directory, the fold keeps those rows in memory (the one warning above) and the checkpoint is -still written. With `--native-event-retention window`, a payload above 1 MiB is -hashed and written into its row in pieces (stored as a BLOB with the same -bytes; the column is declared `TEXT`, but SQLite keeps each value's own type, -so `SELECT payload` returns a blob for such a row and text for the others, and -`CAST(payload AS TEXT)` reads both; a payload of 2 GiB or more is beyond +still written. With `--native-event-retention window` on a module that has the engine layout stamp, a payload above 1 MiB is +hashed and written into its row in pieces, at any cursor (stored as a BLOB with +the same bytes; the column is declared `TEXT`, but SQLite keeps each value's +own type, so `SELECT payload` returns a blob for such a row and text for the +others, and `CAST(payload AS TEXT)` reads both; a payload of 2 GiB or more is beyond SQLite's blob I/O and fails as a checkpoint write failure, with the input durable), so its memory does not grow with T, the full report size. Without -that option the payload is built as one string and stored as TEXT, as in +that option, or for a module without the stamp, the payload is built as one string and stored as TEXT, as in 1.13.0, and a ledger of such a run has the 1.13.0 form. It still reads O(N) report rows at a boundary, which can delay the input loop, and it does not provide constant export time. With a ledger byte budget SQLite keeps a transaction's pages in memory until @@ -328,8 +334,8 @@ first need reports or integrity digests populated retry after startup replay, before new inputs. Later opens add no startup copy when a checkpoint already exists. Startup and post-replay checkpoints intentionally have no input reservation -behind them (`Ledger::checkpoint_recovered_report`, `runner/store.cpp:1493`; -the post-replay call is `checkpoint_recovered_report`, `runner/main.cpp:1373`). +behind them (`Ledger::checkpoint_recovered_report`, `runner/store.cpp:1497`; +the post-replay call is `checkpoint_recovered_report`, `runner/main.cpp:1384`). With a large budget, either can spend most of the remaining headroom on an O(T) payload before the first live input. The run can stop sooner, never over budget; this is the intended cost of preparing the report shortcut at startup. @@ -549,7 +555,7 @@ pineforge-live probe --status-file status.json --max-age 3 --ready defaults to 1024. Checkpoints are report recovery aids; they do not serialize or restore strategy state. Startup and recovery checkpoint writes use the same admission; opening files already over a finite budget refuses before - any checkpoint is attempted. With `--native-event-retention window`, a + any checkpoint is attempted. With `--native-event-retention window` on a module that has the engine layout stamp, a checkpoint above 1 MiB is admitted whole (its byte size, with the same margin as an input's payload) before its first write and is then written in pieces with SQLite's page spilling enabled for @@ -557,9 +563,11 @@ pineforge-live probe --status-file status.json --max-age 3 --ready until COMMIT either. - `--native-event-retention window|commands|full` (with `--native-config` only; opt-in) chooses what the stream keeps of the kernel's event record - for `native_events()`. Without it the runner keeps the record the - configuration declares (the whole record for the v1 configuration), as - 1.13.0 did, and nothing is counted as dropped. Pass + for `native_events()` when the module has the current engine layout stamp. + If the stamp is missing or zero, the runner warns once and keeps the whole + event record and TEXT checkpoint storage. Without the option the runner + keeps the record the configuration declares (the whole record for the v1 + configuration), as 1.13.0 did, and nothing is counted as dropped. Pass `--native-event-retention window` to keep memory bounded: `window` keeps the commands a host has not yet acknowledged and no driver point or account row, flat in the input count, so a host that polls without acknowledging diff --git a/runner/main.cpp b/runner/main.cpp index 4ce38a4da..7a8b2348e 100644 --- a/runner/main.cpp +++ b/runner/main.cpp @@ -1152,7 +1152,15 @@ int run(Config c) { // overridden and what native_events() returns is the 1.13.0 record. The // spec and every hash are unchanged; a module without the layout stamp // keeps the whole record. - if (c.native.present && c.native_event_retention && strategy.has_current_engine_layout) + const bool apply_event_retention = c.native.present && c.native_event_retention && + strategy.has_current_engine_layout; + const bool window_retention = apply_event_retention && + c.native_event_retention == pineforge::NativeEventRetention::Window; + if (c.native_event_retention && !strategy.has_current_engine_layout) + std::cerr << "pineforge-live: warning: --native-event-retention is not applied to this module: " + "no current engine layout stamp (strategy_stream_engine_layout_version); " + "keeping the whole event record and TEXT checkpoint storage\n"; + if (apply_event_retention) pineforge::keep_live_event_window(*static_cast(strategy.state), *c.native_event_retention); const auto settings_receipt = strategy.effective_settings(); @@ -1226,18 +1234,7 @@ int run(Config c) { ledger_options.report_checkpoint_interval = c.checkpoint_interval; // Only the opt-in window streams a large checkpoint into a BLOB; every // other run stores it as TEXT from one string, as 1.13.0 did. - ledger_options.stream_large_checkpoints = - c.native_event_retention == pineforge::NativeEventRetention::Window; - Ledger ledger(c.ledger, deployment, c.max_ledger_bytes, ledger_options); - ServiceFile service(c.status_file, c.status_interval * 1000); - ledger.bind_routing(c.routing.stored_document()); - // All modes fold the same carried patches. Only cumulative output, legacy - // verification and checkpoint equality need to serialize the document. - // Rows the report has settled leave the process in every mode: they go to - // anonymous spool files beside the ledger. Only the opt-in window also - // moves the curve to a scratch file there; a default run keeps it in - // memory, as 1.13.0 did. - auto report_scratch = fs::weakly_canonical(fs::absolute(c.ledger)).parent_path().string(); + ledger_options.stream_large_checkpoints = window_retention; // A spool file that later cannot be created or written (a full disk, a // file-size limit) keeps its rows in memory, as 1.13.0 kept them all; the // run goes on with the same bytes and says so once. @@ -1252,14 +1249,24 @@ int run(Config c) { throw std::invalid_argument("PINEFORGE_LIVE_TEST_SPOOL_FILE_LIMIT must be a byte count"); pineforge::live::report_spool_file_limit().store(std::stoull(text)); } + Ledger ledger(c.ledger, deployment, c.max_ledger_bytes, ledger_options); + ServiceFile service(c.status_file, c.status_interval * 1000); + ledger.bind_routing(c.routing.stored_document()); + // All modes fold the same carried patches. Only cumulative output, legacy + // verification and checkpoint equality need to serialize the document. + // Rows the report has settled leave the process in every mode: they go to + // anonymous spool files beside the ledger. Only the opt-in window also + // moves the curve to a scratch file there; a default run keeps it in + // memory, as 1.13.0 did. + auto report_scratch = fs::weakly_canonical(fs::absolute(c.ledger)).parent_path().string(); try { ReportSpool probe(report_scratch); // Can this directory hold an unnamed file? } catch (const std::exception& error) { - std::cerr << "pineforge-live: warning: the live report's settled rows stay in memory: " - << error.what() << '\n'; + // The recovery fold can already have warned about the same refusal. + if (!pineforge::live::report_spool_failure_reported().exchange(true)) + std::cerr << "pineforge-live: warning: the live report's settled rows stay in memory: " + << error.what() << '\n'; report_scratch.clear(); - // This is the one warning; a checkpoint fold there falls back silently. - pineforge::live::report_spool_failure_reported().store(true); } #if defined(__linux__) // O_TMPFILE and mkstemp both work on a memory filesystem, so the probe @@ -1278,16 +1285,20 @@ int run(Config c) { ReportDeltas report_deltas(true, c.report_jsonl == ReportJsonlMode::DeltaV2, c.chart_timezone, report_scratch); std::unique_ptr accumulator; - if (strategy.has_current_engine_layout && - c.native_event_retention == pineforge::NativeEventRetention::Window) { - try { - accumulator = std::make_unique(c.ratio_interval, - report_scratch, static_cast(strategy.state)); - } catch (const std::exception& error) { - // The engine's scratch needs a disk-backed directory. A ledger on a - // memory filesystem keeps the curve where it always was. + if (window_retention) { + if (report_scratch.empty()) { std::cerr << "pineforge-live: warning: the live report's equity curve stays in memory: " - << error.what() << '\n'; + << "the ledger's directory cannot hold scratch files\n"; + } else { + try { + accumulator = std::make_unique(c.ratio_interval, + report_scratch, static_cast(strategy.state)); + } catch (const std::exception& error) { + // The engine's scratch needs a disk-backed directory. A ledger on a + // memory filesystem keeps the curve where it always was. + std::cerr << "pineforge-live: warning: the live report's equity curve stays in memory: " + << error.what() << '\n'; + } } } if (!accumulator) accumulator = std::make_unique(c.ratio_interval); diff --git a/runner/report_spool.hpp b/runner/report_spool.hpp index b956c26f4..4c090c6ba 100644 --- a/runner/report_spool.hpp +++ b/runner/report_spool.hpp @@ -135,7 +135,11 @@ class ReportSpool { descriptor_ = ::mkstemp(name.data()); if (descriptor_ < 0) throw std::runtime_error("cumulative report spool cannot be created"); - ::unlink(name.c_str()); + if (::unlink(name.c_str()) != 0) { + ::close(descriptor_); + (void)::unlink(name.c_str()); // Best effort; no report bytes have been written. + throw std::runtime_error("cumulative report spool cannot be created"); + } const int flags = ::fcntl(descriptor_, F_GETFD); if (flags >= 0) ::fcntl(descriptor_, F_SETFD, flags | FD_CLOEXEC); } diff --git a/runner/service.cpp b/runner/service.cpp index 38900e003..057c86f95 100644 --- a/runner/service.cpp +++ b/runner/service.cpp @@ -1,4 +1,8 @@ #include "service.hpp" + +#include +#include +#include #include #include #include @@ -65,6 +69,21 @@ void atomic_json_file(const std::string& path, const Json& document, bool create const int descriptor = mkstemp(name.data()); if (descriptor < 0) throw std::runtime_error("cannot create atomic service file"); bool open = true; + bool needs_cleanup = true; + const auto remove_temporary = [&]() noexcept { + if (!needs_cleanup) return; + needs_cleanup = false; + for (int attempt = 0; attempt < 2; ++attempt) { + if (unlink(name.data()) == 0 || errno == ENOENT) return; + } + const int error = errno; + try { + std::cerr << (std::string("pineforge-live: warning: cannot remove temporary service file ") + + name.data() + ": " + std::strerror(error) + '\n'); + } catch (...) { + // A diagnostic failure must not replace the result of the operation. + } + }; try { const auto bytes = document.dump() + "\n"; std::size_t offset = 0; @@ -80,7 +99,8 @@ void atomic_json_file(const std::string& path, const Json& document, bool create if (closed != 0) throw std::runtime_error("cannot close atomic service file"); const int installed = create_only ? link(name.data(), path.c_str()) : rename(name.data(), path.c_str()); if (installed != 0) throw std::runtime_error("cannot install atomic service file"); - unlink(name.data()); + if (create_only) remove_temporary(); + else needs_cleanup = false; if (durable) { auto parent = fs::path(path).parent_path(); if (parent.empty()) parent = "."; @@ -92,7 +112,7 @@ void atomic_json_file(const std::string& path, const Json& document, bool create } } catch (...) { if (open) close(descriptor); - unlink(name.data()); + remove_temporary(); throw; } } diff --git a/runner/store.cpp b/runner/store.cpp index 74e750882..35e8e8893 100644 --- a/runner/store.cpp +++ b/runner/store.cpp @@ -343,8 +343,9 @@ bool report_checkpoint_boundary(std::uint64_t cursor, std::uint64_t interval) { // The largest text parse_json accepts. A checkpoint payload above it is still // written (no reader may assume a cap), but no reader can parse it back. constexpr std::uint64_t max_json_frame_bytes = 1024 * 1024; -// Up to this cursor a checkpoint is folded as a tree, as it always was. Above -// it the fold spools its arrays, so its memory does not grow with the cursor. +// Up to this cursor a checkpoint is folded as a tree, as it always was, unless +// the run streams large checkpoints (--native-event-retention window). Above it +// the fold spools its arrays, so its memory does not grow with the cursor. constexpr std::uint64_t max_tree_fold_cursor = 512; void apply_report_snapshot(SpooledReportDocument& document, const std::string& bytes) { @@ -418,7 +419,10 @@ PreparedCheckpoint prepare_report_checkpoint(sqlite3* database, std::uint64_t cu PreparedCheckpoint prepared; std::string payload, payload_hash; - if (cursor <= max_tree_fold_cursor) { + // A run that streams large checkpoints never builds the tree: its payload + // may exceed the JSON frame at any cursor, and the tree would hold all of it + // (and its serialized copy) in memory. The fold below writes the same bytes. + if (cursor <= max_tree_fold_cursor && !stream_large) { Json document; try { document = read_report_document(database, cursor, true, true, nullptr); diff --git a/scripts/run_strategy.py b/scripts/run_strategy.py index 209b56b03..03b0d1c4b 100644 --- a/scripts/run_strategy.py +++ b/scripts/run_strategy.py @@ -1352,10 +1352,13 @@ def reject_duplicate_keys(pairs): def load_session_calendar(path: Path) -> tuple[str, list[tuple[int, int]]]: """A ``pineforge-symbol-calendar/v1`` document's symbol and session days - (scripts/symbol_calendar.py writes them), checked: the schema, and - ascending days, each opening before it closes and no later than the next - one opens.""" - doc = json.loads(Path(path).read_text(encoding="utf-8")) + (scripts/symbol_calendar.py writes them), checked: the schema and a + nonempty list of ascending days, each opening before it closes and + closing no later than the next one opens.""" + return _validate_session_calendar(json.loads(Path(path).read_text(encoding="utf-8")), path) + + +def _validate_session_calendar(doc, path: Path) -> tuple[str, list[tuple[int, int]]]: if not isinstance(doc, dict) or doc.get("schema") != SESSION_CALENDAR_SCHEMA: raise ValueError(f"{path}: not a {SESSION_CALENDAR_SCHEMA} document") sessions = [(int(o), int(c)) for o, c in doc.get("sessions") or []] diff --git a/scripts/symbol_calendar.py b/scripts/symbol_calendar.py index c64466802..825b88d85 100644 --- a/scripts/symbol_calendar.py +++ b/scripts/symbol_calendar.py @@ -39,7 +39,7 @@ from run_strategy import SESSION_CALENDAR_SCHEMA as SCHEMA # noqa: E402 from run_strategy import _metrics_range_provenance # noqa: E402 -from run_strategy import load_session_calendar # noqa: E402,F401 -- the reader +from run_strategy import _validate_session_calendar # noqa: E402 def _ms(word: str) -> int | None: @@ -106,8 +106,8 @@ def main(argv: list[str]) -> int: return 2 doc = calendar_document(Path(argv[1])) out = Path(argv[2]) + symbol, sessions = _validate_session_calendar(doc, out) out.write_text(json.dumps(doc, separators=(",", ":")) + "\n", encoding="utf-8") - symbol, sessions = load_session_calendar(out) print(f"{symbol}: {len(sessions)} session days") return 0 diff --git a/scripts/test_run_strategy_range_end.py b/scripts/test_run_strategy_range_end.py index a55bd923d..de6378855 100644 --- a/scripts/test_run_strategy_range_end.py +++ b/scripts/test_run_strategy_range_end.py @@ -48,7 +48,9 @@ import csv import io import json +import os import re +import subprocess import sys import tempfile import unittest @@ -289,6 +291,54 @@ def test_range_provenance_members(self) -> None: [[_utc_ms(2025, 4, 1), _utc_ms(2025, 4, 2)]]) +class CalendarExportTests(unittest.TestCase): + def test_refused_calendar_leaves_destination_unchanged(self) -> None: + with tempfile.TemporaryDirectory() as tmp: + root = Path(tmp) + tape = root / "tape" + tape.mkdir() + (tape / "metrics.json").write_bytes(b'{"symbol":"NASDAQ:AAPL","interval":"15"}\n') + (tape / "strategy.pine").write_bytes(b"// Calendar probe\n") + csv_path = tape / "tv_trades.csv" + out = root / "calendar.json" + exporter = Path(__file__).resolve().with_name("symbol_calendar.py") + command = [sys.executable, "-B", str(exporter), str(tape), str(out)] + + header = b"Signal\n" + prefix = b"d|250101-0930|250101-0930~250101-1600|250102-0930~" + csv_path.write_bytes(header + prefix + b"250102-1600\n") + accepted = subprocess.run(command, capture_output=True, text=True) + self.assertEqual(accepted.returncode, 0, accepted.stderr) + self.assertEqual(accepted.stdout, "NASDAQ:AAPL: 2 session days\n") + expected = ( + b'{"schema":"pineforge-symbol-calendar/v1","symbol":"NASDAQ:AAPL",' + b'"source":{"tape":"tape","interval":"15","channel":null,"returnedRange":null,' + b'"tvTradesSha256":"769d2daffb53b4af2ca05f899b349c80b31e7dfa9420c8cdec805d9de5e12646",' + b'"pineSha256":"961312ab4d4a941c7ea3382b07732b654e3c987a003f65f36653db3e461fc871"},' + b'"sessions":[[1735723800000,1735747200000],[1735810200000,1735833600000]]}' + + os.linesep.encode("ascii") + ) + self.assertEqual(out.read_bytes(), expected) + + for signal, error in ( + (prefix + b"250102-0930\n", "session day 1 is out of order"), + (b"", "the calendar holds no session day"), + ): + csv_path.write_bytes(header + signal) + for before in (expected, None): + with self.subTest(error=error, destination_exists=before is not None): + if before is None: + out.unlink(missing_ok=True) + else: + out.write_bytes(before) + refused = subprocess.run(command, capture_output=True, text=True) + self.assertNotEqual(refused.returncode, 0, refused.stdout) + self.assertIn(error, refused.stderr) + self.assertEqual( + out.read_bytes() if out.exists() else None, before, refused.stderr + ) + + class LoadTvRangeEndTests(unittest.TestCase): def test_closed_last_trade_bounds_at_the_range_end_not_the_tape(self) -> None: # A closed last exit can precede the range end by many bars; diff --git a/src/c_abi.cpp b/src/c_abi.cpp index 98df1afa6..85495be30 100644 --- a/src/c_abi.cpp +++ b/src/c_abi.cpp @@ -1133,13 +1133,13 @@ PF_API int strategy_set_native_security_feed(pf_strategy_t s, /* See PF_ABI_VERSION doc in pineforge.h. */ PF_API int pf_abi_version(void) { return PF_ABI_VERSION; } -/* Return the runtime library version. */ +/* Use the C ABI's empty string for an unavailable build SHA. */ PF_API pf_version_t pf_version_get(void) { pf_version_t v; v.major = PINEFORGE_VERSION_MAJOR; v.minor = PINEFORGE_VERSION_MINOR; v.patch = PINEFORGE_VERSION_PATCH; - v.commit_sha = PINEFORGE_GIT_SHA; + v.commit_sha = std::strcmp(PINEFORGE_GIT_SHA, "unknown") == 0 ? "" : PINEFORGE_GIT_SHA; return v; } diff --git a/src/native_execution_consumer.cpp b/src/native_execution_consumer.cpp index 60e487c59..eb393ed36 100644 --- a/src/native_execution_consumer.cpp +++ b/src/native_execution_consumer.cpp @@ -1978,8 +1978,8 @@ bool NativeExecutionConsumer::prepare_public_begin( preparing_begin_ = true; try { detail::MapPairLimitScope map_policy(map_pair_limit_); - const int preparation_bar = 0; - detail::MapBarContextScope map_bar(&preparation_bar); + const int before_first_bar = -1; + detail::MapBarContextScope map_bar(&before_first_bar); host->prepare_native_begin(args); } catch (const std::exception& e) { pending_bar_views_ = {}; @@ -3409,14 +3409,14 @@ bool NativeExecutionConsumer::begin_ready(BacktestEngine& engine, NativeRunPhase // host's and are never mistaken for the kernel's tail. This run's // declaration is registered after the callback (below). clear_timeframe_subscriptions(engine); - provision_run_command_scratch_for_run(); - // Both command stores are run-owned before the first host callback. + provision_run_command_scratch_for_run(); // Both command stores are run-owned before the callback. if (auto* host = native_host(engine)) { in_callback_ = true; in_run_begin_ = true; try { detail::MapPairLimitScope map_policy(map_pair_limit_); - detail::MapBarContextScope map_bar(&engine.bar_index_); + const int before_first_bar = -1; + detail::MapBarContextScope map_bar(&before_first_bar); host->on_native_run_begin(); } catch (const std::exception& e) { in_callback_ = false; diff --git a/src/native_execution_consumer.hpp b/src/native_execution_consumer.hpp index ae916500b..6f731fb41 100644 --- a/src/native_execution_consumer.hpp +++ b/src/native_execution_consumer.hpp @@ -184,9 +184,9 @@ class NativeExecutionConsumer final : public IExecutionConsumer { ? &accepted_input_context_->input_interval : input_callback_context_ ? &input_callback_context_->input_interval : last_accepted_input_ ? &*last_accepted_input_ : nullptr; - if (!latest || stamp > latest->open_ms) return std::nullopt; - if (stamp == latest->open_ms || stamp == latest->eligible_open_ms) + if (latest && native_confirmed_bar_label_admitted(*latest, stamp)) return completion_input_interval(*latest); + if (!latest || stamp > latest->open_ms) return std::nullopt; const auto interval = input_interval_at(stamp); return interval ? completion_input_interval(*interval) : std::nullopt; } diff --git a/src/script_store.cpp b/src/script_store.cpp index e47eea64c..b78284412 100644 --- a/src/script_store.cpp +++ b/src/script_store.cpp @@ -107,7 +107,7 @@ int set_map_pair_limit_checked(const std::int64_t* maximum, } [[noreturn]] void map_capacity_error(std::uint64_t maximum, std::uint64_t current) { - std::int64_t bar = map_bar_context.native_bar ? *map_bar_context.native_bar : 0; + std::int64_t bar = map_bar_context.native_bar ? *map_bar_context.native_bar : -1; if (map_bar_context.use_ambient_context) { const auto& context = ta::bar_context(); if (context.installed) bar = context.bar_index; @@ -139,9 +139,10 @@ MapCapacityError::MapCapacityError(std::uint64_t maximum, std::uint64_t current, : coded( RunFailureCode::pine_map_capacity_exceeded, {{"code", code}, {"maxSize", maximum}, {"currSize", current}, {"bar_index", bar}}, - "Error on bar " + std::to_string(bar) + ": A map cannot have more than " + - std::to_string(maximum) + " key-value pairs. The current size is " + - std::to_string(current) + "."), + (bar == -1 ? "Error before the first bar: " + : "Error on bar " + std::to_string(bar) + ": ") + + "A map cannot have more than " + std::to_string(maximum) + + " key-value pairs. The current size is " + std::to_string(current) + "."), maxSize(maximum), currSize(current), bar_index(bar) {} } // namespace pineforge diff --git a/src/source/pine_ta_length.cpp b/src/source/pine_ta_length.cpp index a7612b7d4..da7a9c118 100644 --- a/src/source/pine_ta_length.cpp +++ b/src/source/pine_ta_length.cpp @@ -9,6 +9,8 @@ #include #include +#include "window_sum_expansion.hpp" + #include #include #include @@ -22,20 +24,16 @@ namespace source { namespace { -// "Error on bar N: " as TradingView prefixes a runtime error, when an -// evaluation context names the bar. -std::string bar_prefix() { - const ta::BarContext& ctx = ta::bar_context(); - if (!ctx.installed) return std::string(); - return "Error on bar " + std::to_string(ctx.bar_index) + ": "; -} - +// Use the run-failure prefix for both evaluated calls and validation before +// an evaluation context exists. [[noreturn]] void length_not_positive(long long value, const char* function, const char* argument) { throw coded( RunFailureCode::pine_invalid_argument, - {{"function", function}, {"argument", argument}, {"rule", "must_be_positive"}}, - bar_prefix() + "Invalid value of the '" + argument + + {{"function", function}, + {"argument", argument}, + {"rule", "must_be_positive"}}, + run_error_bar_prefix() + "Invalid value of the '" + argument + "' argument (" + std::to_string(value) + ") in the '" + function + "' function. It must be > 0."); } @@ -43,8 +41,10 @@ std::string bar_prefix() { [[noreturn]] void length_na(const char* function, const char* argument) { throw coded( RunFailureCode::pine_invalid_argument, - {{"function", function}, {"argument", argument}, {"rule", "must_not_be_na"}}, - bar_prefix() + "Invalid value of the '" + argument + + {{"function", function}, + {"argument", argument}, + {"rule", "must_not_be_na"}}, + run_error_bar_prefix() + "Invalid value of the '" + argument + "' argument in the '" + function + "' function. It must not be na"); } @@ -53,7 +53,7 @@ std::string bar_prefix() { throw coded( RunFailureCode::pine_runtime_limit, {{"limit", "max_bars_back"}, {"max", SeriesWindowExtreme::kMaxBarsBack}}, - bar_prefix() + + run_error_bar_prefix() + "The script attempts to reference historical data that is too far from " "the current bar (" + std::to_string(length - 1) + " bars back). The historical buffer's limit is " + @@ -80,8 +80,10 @@ bool length_value(double value, long long& out) { } throw coded( RunFailureCode::pine_invalid_argument, - {{"function", "supertrend"}, {"argument", "factor"}, {"rule", "must_be_positive"}}, - bar_prefix() + "Invalid value of the 'factor' argument (" + text + + {{"function", "supertrend"}, + {"argument", "factor"}, + {"rule", "must_be_positive"}}, + run_error_bar_prefix() + "Invalid value of the 'factor' argument (" + text + ") in the 'supertrend' function. It must be > 0."); } @@ -487,11 +489,31 @@ void SeriesSma::ensure_capacity() { double SeriesSma::mean(std::size_t length) const { if (length > count_) return na(); + double partials[::pineforge::detail::kWindowSumPartialCapacity]; + std::size_t parts = 0; + const auto partial = [&partials](std::size_t i) -> double& { return partials[i]; }; + const auto first = (head_ + kCapacity - length) % kCapacity; + const auto source = [this, first](std::size_t i) { + return history_.read((first + i) % kCapacity); + }; + std::size_t i = 0; + for (; i != length; ++i) { + const double value = source(i); + if (is_na(value)) return na(); + if (!::pineforge::detail::expand_window_sum(value, parts, partial)) break; + } + if (i == length) { + const double sum = ::pineforge::detail::round_window_sum(parts, partial).sum; + // The fixed SMA starts its sum at positive zero. + return (sum + 0.0) / static_cast(length); + } + + // Preserve the previous series result for nonfinite sources or an + // intermediate overflow, outside the qualified window arithmetic. double sum = 0.0; double correction = 0.0; - const auto first = (head_ + kCapacity - length) % kCapacity; for (std::size_t i = 0; i != length; ++i) { - const double value = history_.read((first + i) % kCapacity); + const double value = source(i); if (is_na(value)) return na(); const double next = sum + value; if (std::fabs(sum) >= std::fabs(value)) diff --git a/src/str_utils.cpp b/src/str_utils.cpp index 043c6b9e8..3aeb54941 100644 --- a/src/str_utils.cpp +++ b/src/str_utils.cpp @@ -526,9 +526,40 @@ std::string time_format_tokens(std::string fmt) { return fmt; } -std::string time_format_chunk(const std::string& fmt, const std::tm& fields) { +size_t time_format_directive_end(const std::string& fmt, size_t first) { + size_t pos = first + 1; + while (pos < fmt.size() && std::strchr("_-0^#", fmt[pos])) ++pos; + while (pos < fmt.size() && fmt[pos] >= '0' && fmt[pos] <= '9') ++pos; + if (pos < fmt.size() && (fmt[pos] == 'E' || fmt[pos] == 'O')) ++pos; + if (pos < fmt.size()) ++pos; + return pos; +} + +std::string time_format_unix_seconds(const std::string& fmt, time_t unix_seconds) { + // `%s` is the Unix timestamp, so do not let strftime reinterpret the + // civil fields through the process timezone. + const std::string seconds = std::to_string(static_cast(unix_seconds)); + std::string expanded; + for (size_t pos = 0; pos < fmt.size();) { + if (fmt[pos] != '%') { + expanded += fmt[pos++]; + continue; + } + const size_t first = pos; + pos = time_format_directive_end(fmt, first); + if (pos == first + 2 && fmt[first + 1] == 's') { + expanded += seconds; + } else { + expanded.append(fmt, first, pos - first); + } + } + return expanded; +} + +std::string time_format_chunk(const std::string& fmt, const std::tm& fields, + time_t unix_seconds) { // A marker distinguishes an empty expansion from insufficient storage. - const std::string marked = " " + fmt; + const std::string marked = " " + time_format_unix_seconds(fmt, unix_seconds); const size_t maximum = marked.max_size(); size_t directives = 0; size_t largest_width = 0; @@ -575,16 +606,8 @@ std::string time_format_chunk(const std::string& fmt, const std::tm& fields) { } } -size_t time_format_directive_end(const std::string& fmt, size_t first) { - size_t pos = first + 1; - while (pos < fmt.size() && std::strchr("_-0^#", fmt[pos])) ++pos; - while (pos < fmt.size() && fmt[pos] >= '0' && fmt[pos] <= '9') ++pos; - if (pos < fmt.size() && (fmt[pos] == 'E' || fmt[pos] == 'O')) ++pos; - if (pos < fmt.size()) ++pos; - return pos; -} - -std::string time_format_unquoted(const std::string& pattern, const std::tm& fields) { +std::string time_format_unquoted(const std::string& pattern, const std::tm& fields, + time_t unix_seconds) { const std::string fmt = time_format_tokens(pattern); std::string expanded; for (size_t pos = 0; pos < fmt.size();) { @@ -607,7 +630,7 @@ std::string time_format_unquoted(const std::string& pattern, const std::tm& fiel expanded += fmt[pos++]; } } - return time_format_chunk(expanded, fields); + return time_format_chunk(expanded, fields, unix_seconds); } bool time_format_has_unmatched_quote(const std::string& pattern) { @@ -628,10 +651,11 @@ bool time_format_has_unmatched_quote(const std::string& pattern) { return quoted; } -std::string time_format_pattern(const std::string& format, const std::tm& fields) try { +std::string time_format_pattern(const std::string& format, const std::tm& fields, + time_t unix_seconds) try { const std::string pattern = format.substr(0, format.find('\0')); if (time_format_has_unmatched_quote(pattern)) { - return time_format_chunk(time_format_tokens(pattern), fields); + return time_format_chunk(time_format_tokens(pattern), fields, unix_seconds); } std::string result; @@ -646,10 +670,11 @@ std::string time_format_pattern(const std::string& format, const std::tm& fields } } if (first_quote == pattern.size()) { - result += time_format_unquoted(pattern.substr(pos), fields); + result += time_format_unquoted(pattern.substr(pos), fields, unix_seconds); break; } - result += time_format_unquoted(pattern.substr(pos, first_quote - pos), fields); + result += time_format_unquoted(pattern.substr(pos, first_quote - pos), fields, + unix_seconds); pos = first_quote + 1; // Append quoted bytes directly so even a percent stays literal. while (pos < pattern.size()) { @@ -679,16 +704,15 @@ std::string str_format_time(long long timestamp_ms, if (!timezone.empty() && timezone != "UTC" && timezone != "Etc/UTC") { tz_util::ScopedTimezone guard(timezone); localtime_r(&secs, &tm_buf); - return time_format_pattern(format, tm_buf); + return time_format_pattern(format, tm_buf, secs); } gmtime_r(&secs, &tm_buf); tm_buf.tm_gmtoff = 0; - // strftime may read TZ (libc is allowed to call tzset for it, and "%s" - // goes through mktime), so the UTC fields are formatted under the - // environment lock too: no TZ switch, and no read while another thread - // switches or restores. + // strftime may read TZ (libc is allowed to call tzset), so the UTC fields + // are formatted under the environment lock too: no TZ switch, and no read + // while another thread switches or restores. const tz_util::EnvironmentLock environment; - return time_format_pattern(format, tm_buf); + return time_format_pattern(format, tm_buf, secs); } // ---------- str.match ---------- diff --git a/src/ta_moving_averages.cpp b/src/ta_moving_averages.cpp index ed0cf3e2b..04853d4b9 100644 --- a/src/ta_moving_averages.cpp +++ b/src/ta_moving_averages.cpp @@ -11,6 +11,7 @@ #include #include "runtime_ambient.hpp" +#include "window_sum_expansion.hpp" #include #include @@ -178,20 +179,11 @@ bool KahanWindowSum::add_exact(double v) { bool KahanWindowSum::resum() { std::size_t parts = 0; bool finite = true; + const auto partial = [this](std::size_t i) -> double& { + return window_[index(i)].addend; + }; for (std::size_t i = 0; i < size_ && finite; ++i) { - double x = window_[index(i)].value; - std::size_t kept = 0; - for (std::size_t j = 0; j < parts; ++j) { - double y = window_[index(j)].addend; - if (std::fabs(x) < std::fabs(y)) std::swap(x, y); - const double high = x + y; - const double low = y - (high - x); // exact: |x| >= |y| - if (low != 0.0) window_[index(kept++)].addend = low; - x = high; - } - window_[index(kept++)].addend = x; - parts = kept; - finite = std::isfinite(x); + finite = detail::expand_window_sum(window_[index(i)].value, parts, partial); } if (!finite) { // An infinite source, or a sum beyond the largest double: there is @@ -204,30 +196,10 @@ bool KahanWindowSum::resum() { comp_ = 0.0; return false; } - double high = 0.0; - double low = 0.0; - std::size_t rest = parts; - if (rest > 0) { - high = window_[index(--rest)].addend; - while (rest > 0) { - const double x = high; - const double y = window_[index(--rest)].addend; - high = x + y; - low = y - (high - x); - if (low != 0.0) break; - } - // Exactly halfway between two doubles, the partial sums still below - // decide the direction (round half to even otherwise). - if (rest > 0 && ((low < 0.0 && window_[index(rest - 1)].addend < 0.0) - || (low > 0.0 && window_[index(rest - 1)].addend > 0.0))) { - const double twice = low * 2.0; - const double up = high + twice; - if (twice == up - high) high = up; - } - } - sum_ = high; - comp_ = low; - return rest == 0; + const auto rounded = detail::round_window_sum(parts, partial); + sum_ = rounded.sum; + comp_ = rounded.remainder; + return rounded.fits_pair; } // The source takes the front slot of the ring (its evicted predecessor has @@ -422,6 +394,19 @@ inline double ema_step(double alpha, double src, double previous) { return previous + alpha * (src - previous); } +double ema_seed_sum(const std::vector& sources, double fallback) { + double partials[detail::kWindowSumPartialCapacity]; + std::size_t parts = 0; + const auto partial = [&partials](std::size_t i) -> double& { return partials[i]; }; + for (double src : sources) { + if (!detail::expand_window_sum(src, parts, partial)) return fallback; + } + const double sum = detail::round_window_sum(parts, partial).sum; + // Keep the previous result for nonfinite or overflowing seeds and + // normalize an exact zero the same way as fixed SMA. + return std::isfinite(sum) ? sum + 0.0 : fallback; +} + } // namespace EMA::EMA(int length) @@ -445,9 +430,6 @@ double EMA::compute(double src) { double EMA::compute(double src, bool input_ready) { save(); - if (input_ready && std::isfinite(src) && ready_count_ < length_) { - ++ready_count_; - } // An instance that named no seeding latches the ambient default once, on // its first compute(), so a host that raises the default around one // evaluation context gets that context's instances SMA-seeded for life @@ -465,8 +447,14 @@ double EMA::compute(double src, bool input_ready) { return na(); } + if (na_warmup_ && is_na(output_val) && bar_count < length_) { + seed_sources_.push_back(src); + } + if (input_ready && std::isfinite(src) && ready_count_ < length_) { + ++ready_count_; + } if (na_warmup_) { - // EmaSeeding::SimpleAverage: na until `length` finite inputs have + // EmaSeeding::SimpleAverage: na until `length` non-na inputs have // accumulated since the series start, then seed with their mean, then // run the ordinary recursion — the warm-up RMA::compute / SMA already // have, so a series truncated at its start reads na for its whole @@ -477,8 +465,9 @@ double EMA::compute(double src, bool input_ready) { if (bar_count < length_) { return na(); } - // bar_count == length_: seed = SMA of the first `length_` values. - output_val = sum / static_cast(length_); + // At the seed, use the exact sum rounded once, as in fixed SMA. + const double seed_sum = seed_sources_.empty() ? sum : ema_seed_sum(seed_sources_, sum); + output_val = seed_sum / static_cast(length_); return output_val; } output_val = ema_step(alpha, src, output_val); @@ -674,12 +663,18 @@ void EMA::save() { saved_sum_ = sum; saved_bar_count_ = bar_count; saved_ready_count_ = ready_count_; + // A post-seed snapshot can no longer rewind into the seed window. + if (seed_sources_.capacity() != 0 && bar_count >= length_) + std::vector().swap(seed_sources_); } void EMA::restore() { output_val = saved_output_val_; sum = saved_sum_; bar_count = saved_bar_count_; ready_count_ = saved_ready_count_; + // A warm-up observation can append one source after the saved count. + if (seed_sources_.size() > static_cast(saved_bar_count_)) + seed_sources_.pop_back(); } double EMA::recompute(double src) { return recompute(src, true); diff --git a/src/ta_volatility_trend.cpp b/src/ta_volatility_trend.cpp index 128ebd57b..35f1a61eb 100644 --- a/src/ta_volatility_trend.cpp +++ b/src/ta_volatility_trend.cpp @@ -536,7 +536,7 @@ int check_channel_arguments(int length, double mult, const char* function) { throw coded( RunFailureCode::pine_invalid_argument, {{"function", function}, {"argument", "length"}, {"rule", "must_not_be_na"}}, - (bar_context().installed ? run_error_bar_prefix() : std::string()) + + run_error_bar_prefix() + "Invalid value of the 'length' argument in the '" + function + "' function. It must not be na"); } @@ -550,7 +550,7 @@ int check_channel_arguments(int length, double mult, const char* function) { throw coded( RunFailureCode::pine_invalid_argument, {{"function", function}, {"argument", argument}, {"rule", "must_be_positive"}}, - (bar_context().installed ? run_error_bar_prefix() : std::string()) + + run_error_bar_prefix() + "Invalid value of the '" + argument + "' argument (" + std::string(value, formatted.ptr) + ") in the '" + function + "' function. It must be > 0."); diff --git a/src/timeframe.cpp b/src/timeframe.cpp index a573da0a6..52c3bcb02 100644 --- a/src/timeframe.cpp +++ b/src/timeframe.cpp @@ -22,22 +22,22 @@ std::string detect_timeframe(const Bar* bars, int n, int max_samples) { if (n < 2) return "1"; int count = std::min(n - 1, max_samples); - std::vector deltas; + std::vector deltas; deltas.reserve(count); for (int i = 0; i < count; ++i) { - int64_t d = bars[i + 1].timestamp - bars[i].timestamp; - if (d > 0) deltas.push_back(d); + const int64_t earlier = bars[i].timestamp; + const int64_t later = bars[i + 1].timestamp; + if (later <= earlier) continue; + // Unsigned subtraction gives the exact positive gap even across zero. + deltas.push_back(static_cast(later) - static_cast(earlier)); } if (deltas.empty()) return "1"; - // Compute median delta + // The upper median in whole seconds fits in int64_t even for the widest gap. std::sort(deltas.begin(), deltas.end()); - int64_t median_ms = deltas[deltas.size() / 2]; - - // Convert to seconds - int64_t secs = median_ms / 1000; + const int64_t secs = static_cast(deltas[deltas.size() / 2] / 1000); // Map to nearest standard TF (TradingView format) // Standard TFs in seconds: 1m=60, 3m=180, 5m=300, 15m=900, 30m=1800, @@ -92,6 +92,20 @@ int timeframe_count(const std::string& text) { } } +// `count` units of `unit_seconds` each, multiplied in 64 bits. Seconds that +// do not fit this legacy int API get the refusal of a count that does not. +int timeframe_seconds(int count, int unit_seconds, const std::string& tf) { + const int64_t seconds = static_cast(count) * unit_seconds; + if (seconds < std::numeric_limits::min() + || seconds > std::numeric_limits::max()) { + throw coded( + RunFailureCode::pine_invalid_argument, + {{"argument", "timeframe"}, {"rule", "unparseable"}}, + "timeframe '" + tf + "': seconds out of int range"); + } + return static_cast(seconds); +} + } // namespace int tf_to_seconds(const std::string& tf) { @@ -112,14 +126,16 @@ int tf_to_seconds(const std::string& tf) { // like M. Its parsed day count still drives the generated partition; // this is not a limit on the number of dates a bucket can hold. if (n > std::numeric_limits::max() / 86400) return -1; - return n * 86400; + return timeframe_seconds(n, 86400, tf); } if (last == 'W') { // "W" or "1W" etc. if (tf.size() == 1) return 604800; int n = timeframe_count(tf.substr(0, tf.size() - 1)); - return n * 604800; + // Calendar-only past the int too, as a day count is above. + if (n > std::numeric_limits::max() / 604800) return -1; + return timeframe_seconds(n, 604800, tf); } if (last == 'S') { @@ -131,20 +147,22 @@ int tf_to_seconds(const std::string& tf) { return n; } - // All-numeric: minutes + // All-numeric: minutes. They have no calendar meaning to fall back on. int minutes = timeframe_count(tf); - return minutes * 60; + return timeframe_seconds(minutes, 60, tf); } int tf_ratio(const std::string& input_tf, const std::string& target_tf) { int input_s = tf_to_seconds(input_tf); int target_s = tf_to_seconds(target_tf); - // Calendar-only durations include a day count too large for the legacy - // seconds API. Equal day literals still pass their input bars through. + // Calendar-only durations include a day or week count too large for the + // legacy seconds API (-1). Equal day literals, and equal calendar-only + // week literals, still pass their input bars through. if (target_s < 0) { - if (input_tf == target_tf - && calendar_period_for(target_tf) == CalendarPeriod::DAY) return 1; + const CalendarPeriod period = calendar_period_for(target_tf); + if (input_tf == target_tf && (period == CalendarPeriod::DAY + || (period == CalendarPeriod::WEEK && target_s == -1))) return 1; return -1; } diff --git a/src/window_sum_expansion.hpp b/src/window_sum_expansion.hpp new file mode 100644 index 000000000..a104c56d8 --- /dev/null +++ b/src/window_sum_expansion.hpp @@ -0,0 +1,67 @@ +#pragma once + +#include +#include +#include +#include + +namespace pineforge::detail { + +// Nonoverlapping partials have distinct highest set bits. Allow one slot per +// binary64 bit position, a zero, and room for the next insert. +inline constexpr std::size_t kWindowSumPartialCapacity = + std::numeric_limits::max_exponent - + std::numeric_limits::min_exponent + + std::numeric_limits::digits + 2; + +// Insert a source into an expansion of nonoverlapping doubles, smallest first +// (Shewchuk's expansion sum, as in math.fsum). Each window supplies its own +// storage for the partials. False means a nonfinite source or overflow. +template +bool expand_window_sum(double x, std::size_t& parts, const Partials& partial) { + std::size_t kept = 0; + for (std::size_t j = 0; j < parts; ++j) { + double y = partial(j); + if (std::fabs(x) < std::fabs(y)) std::swap(x, y); + const double high = x + y; + const double low = y - (high - x); // exact: |x| >= |y| + if (low != 0.0) partial(kept++) = low; + x = high; + } + partial(kept++) = x; + parts = kept; + return std::isfinite(x); +} + +struct RoundedWindowSum { + double sum; + double remainder; + bool fits_pair; +}; + +template +RoundedWindowSum round_window_sum(std::size_t rest, const Partials& partial) { + double high = 0.0; + double low = 0.0; + if (rest > 0) { + high = partial(--rest); + while (rest > 0) { + const double x = high; + const double y = partial(--rest); + high = x + y; + low = y - (high - x); + if (low != 0.0) break; + } + // Exactly halfway between two doubles, the partial sums still below + // decide the direction (round half to even otherwise). + if (rest > 0 && ((low < 0.0 && partial(rest - 1) < 0.0) + || (low > 0.0 && partial(rest - 1) > 0.0))) { + const double twice = low * 2.0; + const double up = high + twice; + if (twice == up - high) high = up; + } + } + return {high, low, rest == 0}; +} + +} // namespace pineforge::detail diff --git a/tests/test_c_abi.c b/tests/test_c_abi.c index 6cf7511ed..93f6d47db 100644 --- a/tests/test_c_abi.c +++ b/tests/test_c_abi.c @@ -35,6 +35,15 @@ int main(void) { CHECK(v.minor == PINEFORGE_VERSION_MINOR, "version minor mismatch"); CHECK(v.patch == PINEFORGE_VERSION_PATCH, "version patch mismatch"); CHECK(v.commit_sha != NULL, "commit_sha is NULL"); + if (v.commit_sha != NULL) { + if (strcmp(PINEFORGE_GIT_SHA, "unknown") == 0) { + CHECK(v.commit_sha[0] == '\0', + "commit_sha must be empty when the Git SHA is unavailable"); + } else { + CHECK(strcmp(v.commit_sha, PINEFORGE_GIT_SHA) == 0, + "commit_sha must match the build's Git SHA"); + } + } /* ── ABI version ────────────────────────────────────────────── */ CHECK(pf_abi_version() == PF_ABI_VERSION, "pf_abi_version() != PF_ABI_VERSION"); diff --git a/tests/test_checked_ema_family.cpp b/tests/test_checked_ema_family.cpp index d6b6313ba..f51e78e79 100644 --- a/tests/test_checked_ema_family.cpp +++ b/tests/test_checked_ema_family.cpp @@ -108,7 +108,8 @@ std::string na_text(const char* function, const char* argument) { // Runs `call` and requires the coded invalid-argument stop with `text`. template -void stops(Call call, const std::string& text) { +void stops(Call call, const std::string& text, + const char* prefix = "Error before the first bar: ") { bool caught = false; try { call(); @@ -116,7 +117,7 @@ void stops(Call call, const std::string& text) { const auto* failure = dynamic_cast(&error); assert(failure != nullptr); assert(failure->run_failure_code() == RunFailureCode::pine_invalid_argument); - assert(std::string(error.what()) == text); + assert(std::string(error.what()) == prefix + text); caught = true; } assert(caught); @@ -153,10 +154,10 @@ void refusal(Make make, Run run, const char* function, const char* argument) { stops([&] { run(adapter, again); }, text); stops([&] { run(adapter, false); }, text); // fails again { - // Under an evaluation context the bar prefix is added. + // An evaluation context names the bar in the prefix. ta::BarContextScope scope(17, 0); auto fresh = form == 0 ? make(row.value) : make(na()); - stops([&] { run(fresh, again); }, "Error on bar 17: " + text); + stops([&] { run(fresh, again); }, text, "Error on bar 17: "); } } } diff --git a/tests/test_cr559_map_capacity_error_context.cpp b/tests/test_cr559_map_capacity_error_context.cpp index e27ff1f1c..789c0129c 100644 --- a/tests/test_cr559_map_capacity_error_context.cpp +++ b/tests/test_cr559_map_capacity_error_context.cpp @@ -40,19 +40,22 @@ NativeRunSpec spec_for(const char* session) { return spec; } -void require_failure(BacktestEngine& host, cr559::Scalar expected_bar) { - auto* handle = &host; - cr559::require(strategy_last_run_status(handle) == 1, "map refusal did not fail the run"); - const char* code = strategy_get_last_error_code(handle); - const char* args = strategy_get_last_error_args(handle); - const char* text = strategy_get_last_error(handle); +void require_failure_record(BacktestEngine& host, cr559::Scalar expected_bar, + const std::string& expected_text) { + const char* code = strategy_get_last_error_code(&host); + const char* args = strategy_get_last_error_args(&host); + const char* text = strategy_get_last_error(&host); cr559::require(code && std::string(code) == "pine_map_capacity_exceeded", "map context row lost its typed failure"); if (!args || std::string(args) != cr559::capacity_args(3, 4, expected_bar)) throw std::runtime_error("wrong map refusal bar context: " + std::string(args ? args : "")); - cr559::require(text && std::string(text) == cr559::capacity_message(3, 4, expected_bar), - "map context row changed exact TV English"); + cr559::require(text && text == expected_text, "map context row changed exact English"); +} + +void require_failure(BacktestEngine& host, cr559::Scalar expected_bar) { + cr559::require(strategy_last_run_status(&host) == 1, "map refusal did not fail the run"); + require_failure_record(host, expected_bar, cr559::capacity_message(3, 4, expected_bar)); } class LateNativeHost final : public NativeStrategyHost { @@ -100,6 +103,44 @@ std::unique_ptr make_limited_host(Args&&... args) { return std::make_unique(std::forward(args)...); } +enum class StartupHook { Prepare, RunBegin }; + +class StartupCapacityHost final : public NativeStrategyHost { +public: + explicit StartupCapacityHost(StartupHook hook) : hook_(hook) {} + + void prepare_native_begin(const NativeBeginArgs&) override { + if (hook_ == StartupHook::Prepare) exceed_limit(); + } + void on_native_run_begin() override { + if (hook_ == StartupHook::RunBegin) exceed_limit(); + } + void on_native_bar(const Bar&, const NativeDecisionContext&) override { ++bar_calls; } + + int bar_calls = 0; + +private: + static void exceed_limit() { + auto map = cr559::Map::new_(); + for (cr559::Scalar index = 0; index < 4; ++index) + (void)map.put(cr559::key_for(index), cr559::value_for(index)); + } + + StartupHook hook_; +}; + +void run_startup_refusal(StartupHook hook) { + auto host = make_limited_host(hook); + cr559::require(host->configure_native(spec_for("startup-map")).status == + NativeSetupStatus::Applied, + "startup map fixture setup was refused"); + host->run(bars.data(), static_cast(bars.size())); + cr559::require(host->bar_calls == 0, "a bar ran after the startup map refusal"); + require_failure_record(*host, -1, + "Error before the first bar: A map cannot have more than 3 key-value pairs. " + "The current size is 4."); +} + void run_late_native() { auto host = make_limited_host(); cr559::require(host->configure_native(spec_for("cr559-late-native")).status == @@ -166,6 +207,8 @@ int main() { run_late_native(); { ta::BarContextScope unrelated(91, 0); + run_startup_refusal(StartupHook::Prepare); + run_startup_refusal(StartupHook::RunBegin); run_late_native(); cr559::require(ta::bar_context().bar_index == 91, "native callback changed the unrelated TA context"); @@ -182,7 +225,7 @@ int main() { require_failure(*parent, 37); standalone_context_is_restored(); } - std::puts("CR559 context: late native, unrelated TA, nested Pine and both ambient paths checked"); + std::puts("CR559 context: startup, late native, unrelated TA and nested Pine checked"); return 0; } catch (const std::exception& error) { std::fprintf(stderr, "CR559 context FAIL: %s\n", error.what()); diff --git a/tests/test_kc.cpp b/tests/test_kc.cpp index 1f0e513a0..ebcd1421e 100644 --- a/tests/test_kc.cpp +++ b/tests/test_kc.cpp @@ -71,12 +71,16 @@ static void expect_constructor_refusal(int length, double mult, const char* func template static void test_constructor_validation(const char* function) { const std::string stop = std::string(") in the '") + function + "' function. It must be > 0."; + constexpr char before_first_bar[] = "Error before the first bar: "; expect_constructor_refusal(0, 0.75, function, "length", "must_be_positive", - (" (0" + stop).c_str(), ""); + (" (0" + stop).c_str(), before_first_bar); expect_constructor_refusal(-2, 1.5, function, "length", "must_be_positive", - (" (-2" + stop).c_str(), ""); + (" (-2" + stop).c_str(), before_first_bar); expect_constructor_refusal(na(), 2.5, function, "length", "must_not_be_na", - (std::string(" in the '") + function + "' function. It must not be na").c_str(), ""); + (std::string(" in the '") + function + "' function. It must not be na").c_str(), + before_first_bar); + expect_constructor_refusal(3, -0.25, function, "mult", "must_be_positive", + (" (-0.25" + stop).c_str(), before_first_bar); ta::BarContextScope context(7, 9000); expect_constructor_refusal(3, -0.0, function, "mult", "must_be_positive", (" (-0" + stop).c_str(), "Error on bar 7: "); diff --git a/tests/test_map.cpp b/tests/test_map.cpp index f36ee0535..c4bb78c48 100644 --- a/tests/test_map.cpp +++ b/tests/test_map.cpp @@ -264,9 +264,24 @@ static void test_put_all_overwrites_in_source_order() { static void test_50000_pair_capacity_and_put_all_preflight() { using Map = PineMap; - constexpr const char* capacity_error = - "Error on bar 0: A map cannot have more than 50000 key-value pairs. " - "The current size is 50001."; + // expectation corrected: bar 0 -> bar -1, because no bar is executing. + const auto expect_capacity_error = [](auto operation) { + bool threw = false; + try { + operation(); + } catch (const pineforge::MapCapacityError& error) { + threw = true; + assert(std::string(error.what()) == + "Error before the first bar: A map cannot have more than 50000 key-value " + "pairs. The current size is 50001."); + assert(error.bar_index == -1); + const auto failure = error.run_failure(); + assert(failure.code == pineforge::RunFailureCode::pine_map_capacity_exceeded); + assert(failure.args && *failure.args == + R"({"bar_index":-1,"code":"RE10070","currSize":50001,"maxSize":50000})"); + } + assert(threw); + }; auto map = Map::new_(Map::max_pairs); for (int key = 0; key < Map::max_pairs; ++key) { @@ -279,8 +294,7 @@ static void test_50000_pair_capacity_and_put_all_preflight() { const auto limit_overwrite_previous = map.put(0, -1); assert(limit_overwrite_previous == 0); assert(map.size() == Map::max_pairs); - expect_runtime_error( - [&] { (void)map.put(Map::max_pairs, 1); }, capacity_error); + expect_capacity_error([&] { (void)map.put(Map::max_pairs, 1); }); assert(map.size() == Map::max_pairs); const auto removed_for_capacity = map.remove(2); @@ -299,7 +313,7 @@ static void test_50000_pair_capacity_and_put_all_preflight() { auto mixed = Map::new_(); mixed.put(0, 200); mixed.put(Map::max_pairs + 1, 50001); - expect_runtime_error([&] { map.put_all(mixed); }, capacity_error); + expect_capacity_error([&] { map.put_all(mixed); }); // Capacity is preflighted: the overwrite before the new source key was // not partially applied. diff --git a/tests/test_native_live_recovery_spool_warning.cpp b/tests/test_native_live_recovery_spool_warning.cpp new file mode 100644 index 000000000..68aad75cf --- /dev/null +++ b/tests/test_native_live_recovery_spool_warning.cpp @@ -0,0 +1,275 @@ +// SPDX-License-Identifier: Apache-2.0 +#include "../runner/json.hpp" + +#include + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include +#include +#include +#include + +namespace { + +namespace fs = std::filesystem; +using pineforge::live::parse_json; +// First cursor above the tree fold threshold; below the first checkpoint boundary. +constexpr std::uint64_t input_count = 513; +constexpr char spool_limit_env[] = "PINEFORGE_LIVE_TEST_SPOOL_FILE_LIMIT"; +constexpr char spool_warning[] = "pineforge-live: warning: the live report's settled rows stay in memory: "; + +void require(bool ok, const std::string& message) { + if (!ok) throw std::runtime_error(message); +} + +struct TempDir { + fs::path path; + TempDir() { + auto name = (fs::temp_directory_path() / "pineforge-recovery-spool-XXXXXX").string(); + require(::mkdtemp(name.data()) != nullptr, "cannot create test directory"); + path = name; + } + ~TempDir() { + std::error_code ignored; + fs::remove_all(path, ignored); + } +}; + +std::string read_file(const fs::path& path) { + std::ifstream input(path, std::ios::binary); + return std::string(std::istreambuf_iterator(input), {}); +} + +void write_file(const fs::path& path, const std::string& bytes) { + std::ofstream output(path, std::ios::binary); + output << bytes; + output.close(); + require(static_cast(output), "cannot write " + path.string()); +} + +class Runner { +public: + Runner(std::vector command, const fs::path& output, const fs::path& errors, + bool cap, bool hold_input = false) + : output_(output), errors_(errors) { + std::vector argv; + for (auto& arg : command) argv.push_back(arg.data()); + argv.push_back(nullptr); + int input[2]{-1, -1}; + if (hold_input) require(::pipe(input) == 0, "cannot hold runner stdin open"); + pid_ = ::fork(); + if (pid_ == 0) { + const int out = ::open(output_.c_str(), O_WRONLY | O_CREAT | O_TRUNC, 0600); + const int err = ::open(errors_.c_str(), O_WRONLY | O_CREAT | O_TRUNC, 0600); + if (out < 0 || err < 0 || ::dup2(out, STDOUT_FILENO) < 0 || + ::dup2(err, STDERR_FILENO) < 0) ::_exit(126); + ::close(out); + ::close(err); + if (hold_input) { + if (::dup2(input[0], STDIN_FILENO) < 0) ::_exit(126); + ::close(input[0]); + ::close(input[1]); + } + if ((cap ? ::setenv(spool_limit_env, "4096", 1) : ::unsetenv(spool_limit_env)) != 0) + ::_exit(126); + ::execv(argv[0], argv.data()); + ::_exit(127); + } + if (hold_input) { + ::close(input[0]); + if (pid_ < 0) ::close(input[1]); + else input_ = input[1]; + } + require(pid_ > 0, "cannot fork runner"); + } + Runner(const Runner&) = delete; + Runner& operator=(const Runner&) = delete; + ~Runner() { + if (input_ >= 0) ::close(input_); + if (pid_ > 0) { + ::kill(pid_, SIGKILL); + while (::waitpid(pid_, nullptr, 0) < 0 && errno == EINTR) {} + } + } + + void wait_ready(const fs::path& status) { + for (;;) { + if (reap(WNOHANG)) throw std::runtime_error("runner exited before readiness: " + errors()); + const auto bytes = read_file(status); + if (!bytes.empty()) { + const auto doc = parse_json(bytes); + if (doc.at("state").text() == "running" && doc.at("ready").value == "true") { + require(doc.at("readiness").at("recovered_ledger").value == "true" && + doc.at("metrics").at("committed_input").integer() == input_count, + "runner reported readiness before recovering the ledger"); + return; + } + } + std::this_thread::sleep_for(std::chrono::milliseconds(10)); + } + } + + std::string finish(int expected) { + if (input_ >= 0) { + ::close(input_); + input_ = -1; + } + const bool exited = reap(0); + require(exited && WIFEXITED(status_) && WEXITSTATUS(status_) == expected, + "unexpected runner wait status " + std::to_string(status_) + ": " + errors()); + return read_file(output_); + } + + std::string errors() const { return read_file(errors_); } + +private: + bool reap(int options) { + pid_t result; + do { result = ::waitpid(pid_, &status_, options); } while (result < 0 && errno == EINTR); + if (result == 0) return false; + require(result == pid_, "cannot wait for runner"); + pid_ = -1; + return true; + } + + fs::path output_, errors_; + pid_t pid_ = -1; + int input_ = -1, status_ = 0; +}; + +void check_summary(const std::string& bytes, std::uint64_t processed) { + const auto summary = parse_json(bytes); + require(summary.at("inputs_committed").integer() == input_count && + summary.at("inputs_processed").integer() == processed, + "unexpected runner summary: " + bytes); +} + +void check_ledger(const fs::path& ledger, bool recovered) { + sqlite3* db = nullptr; + const auto opened = sqlite3_open_v2(ledger.c_str(), &db, SQLITE_OPEN_READONLY, nullptr); + if (opened != SQLITE_OK) sqlite3_close(db); + require(opened == SQLITE_OK, "cannot read ledger " + ledger.string()); + sqlite3_stmt* query = nullptr; + const auto prepared = sqlite3_prepare_v2(db, + "SELECT (SELECT COUNT(*) FROM report_checkpoints), (SELECT MAX(input_cursor) FROM report_checkpoints), " + "(SELECT COUNT(*) FROM report_snapshots)", + -1, &query, nullptr); + const bool matches = prepared == SQLITE_OK && sqlite3_step(query) == SQLITE_ROW && + sqlite3_column_int64(query, 0) == (recovered ? 1 : 0) && + (!recovered || sqlite3_column_int64(query, 1) == static_cast(input_count)) && + sqlite3_column_int64(query, 2) == static_cast(input_count + 1); + sqlite3_finalize(query); + sqlite3_close(db); + require(matches, recovered ? "missing startup checkpoint or report history" + : "expected all report history and no checkpoint"); +} + +void require_one_spool_warning(const std::string& errors) { + const std::string expected = std::string(spool_warning) + "cumulative report spool write failed: " + + std::strerror(EFBIG) + '\n'; + const auto at = errors.find(expected); + require(at != std::string::npos && (at == 0 || errors[at - 1] == '\n') && + errors.find(expected, at + expected.size()) == std::string::npos, + "expected one report spool warning: " + expected + "stderr:\n" + errors); +} + +void recovery_spool_warning(const char* executable, const char* strategy) { + TempDir temporary; + const auto& root = temporary.path; + const auto warmup = root / "warmup.csv"; + const auto config = root / "native.json"; + const auto feed = root / "bars.jsonl"; + write_file(warmup, "timestamp,open,high,low,close,volume\n0,100,102,99,101,4\n"); + write_file(config, R"({ + "run":{"session_key":"report-stream","run_number":1}, + "clock":{"input_tf":"1","script_tf":"1","timezone":"UTC","session":"24x7","chart_timezone":""}, + "instrument":{"ticker":"MOCK","tickerid":"TEST:MOCK","type":"crypto","currency":"USDT", + "basecurrency":"ETH","description":"","volumetype":""}, + "execution":{"initial_capital":10000,"point_value":1,"account_fx":1,"price_tick":0.01, + "slippage_ticks":0,"fee_kind":"CashPerExecution","fee_value":6,"quantity_grid":null, + "close_execution":"NextEligiblePoint","max_abs_units":null,"max_open_lots":null, + "allowed_open_directions":3,"initial_margin_fraction":null} + })"); + std::string bars; + std::uint32_t seed = 12345; + for (std::uint64_t cursor = 1; cursor <= input_count; ++cursor) { + seed = seed * 1664525u + 1013904223u; + bars += "{\"type\":\"bar\",\"bar\":{\"ts_open\":" + std::to_string(cursor * 60000) + + ",\"o\":100,\"h\":108,\"l\":99,\"c\":" + std::to_string(100 + seed % 7) + ",\"v\":4}}\n"; + } + write_file(feed, bars); + const auto command = [&](const fs::path& ledger, const std::string& source) { + return std::vector{executable, "run", "--strategy", strategy, "--warmup", warmup.string(), + "--native-config", config.string(), "--mode", "bars", "--feed", source, "--ledger", ledger.string(), + "--checkpoint-interval", "1024", "--ledger-unbounded"}; + }; + const auto seed_ledger = root / "seed.sqlite"; + Runner seeder(command(seed_ledger, feed.string()), root / "seed.out", root / "seed.err", false); + check_summary(seeder.finish(0), input_count); + check_ledger(seed_ledger, false); + const auto control = root / "control.sqlite"; + const auto capped = root / "capped.sqlite"; + const auto startup_only = root / "startup-only.sqlite"; + for (const auto& copy : {control, capped, startup_only}) fs::copy_file(seed_ledger, copy); + + // Refuse service status setup just after Ledger construction, before + // replay can write to its own spools and mask a missing recovery warning. + auto startup_command = command(startup_only, feed.string()); + startup_command.insert(startup_command.end(), {"--status-file", (root / "missing" / "status").string()}); + Runner early(std::move(startup_command), root / "startup-only.out", root / "startup-only.err", true); + early.finish(1); + check_ledger(startup_only, true); + + const auto restart = [&](const fs::path& ledger, const std::string& name, bool cap) { + const auto status = root / (name + ".status"); + auto args = command(ledger, "-"); + args.insert(args.end(), {"--status-file", status.string(), "--from-input", std::to_string(input_count)}); + Runner runner(std::move(args), root / (name + ".out"), root / (name + ".err"), cap, true); + runner.wait_ready(status); + check_summary(runner.finish(0), 0); + check_ledger(ledger, true); + const auto errors = runner.errors(); + if (cap) require_one_spool_warning(errors); + else require(errors.find("cumulative report spool write failed") == std::string::npos, + "unconstrained recovery had a spool write failure: " + errors); + + Runner report({executable, "report", "--ledger", ledger.string()}, + root / (name + ".report"), root / (name + ".report.err"), false); + return report.finish(0); + }; + const auto expected = restart(control, "control", false); + const auto actual = restart(capped, "capped", true); + require(!expected.empty() && actual == expected, "spool fallback changed the recovered report bytes"); + + const auto early_errors = early.errors(); + require(early_errors.find("cannot create atomic service file") != std::string::npos, + "runner did not reach service file setup: " + early_errors); + require_one_spool_warning(early_errors); +} + +} // namespace + +int main(int argc, char** argv) { + try { + require(argc == 3, "expected the runner executable and strategy module"); + recovery_spool_warning(argv[1], argv[2]); + return 0; + } catch (const std::exception& error) { + std::cerr << error.what() << '\n'; + return 1; + } +} diff --git a/tests/test_native_live_report_cli.cpp b/tests/test_native_live_report_cli.cpp index ad529b86c..168ff775d 100644 --- a/tests/test_native_live_report_cli.cpp +++ b/tests/test_native_live_report_cli.cpp @@ -166,19 +166,22 @@ class Runner { #if defined(__linux__) // Open files of this live runner that sit in `directory` and have no name // (O_TMPFILE, or mkstemp then unlink): /proc shows " (deleted)". - std::size_t unnamed_files_in(const fs::path& directory) const { + std::vector unnamed_file_names_in(const fs::path& directory) const { const std::string prefix = directory.string() + "/"; const std::string suffix = " (deleted)"; - std::size_t count = 0; + std::vector names; for (const auto& entry : fs::directory_iterator("/proc/" + std::to_string(pid_) + "/fd")) { std::error_code error; const auto target = fs::read_symlink(entry.path(), error).string(); if (error || target.size() < prefix.size() + suffix.size()) continue; if (target.compare(0, prefix.size(), prefix) == 0 && target.compare(target.size() - suffix.size(), suffix.size(), suffix) == 0) - ++count; + names.push_back(target.substr(prefix.size(), target.size() - prefix.size() - suffix.size())); } - return count; + return names; + } + std::size_t unnamed_files_in(const fs::path& directory) const { + return unnamed_file_names_in(directory).size(); } #endif bool exited() const { @@ -1482,6 +1485,122 @@ std::size_t occurrences(const std::string& text, const std::string& part) { return found; } +void window_checkpoint_storage(Fixture& fixture, const char* unstamped_module) { + constexpr std::uint64_t count = 1024; + const std::string warning = "pineforge-live: warning: --native-event-retention is not applied to this module: " + "no current engine layout stamp (strategy_stream_engine_layout_version); " + "keeping the whole event record and TEXT checkpoint storage\n"; + const auto check = [&](bool stamped) { + const std::string name = stamped ? "window-stamped" : "window-unstamped"; + const auto ledger = fixture.temporary.path / (name + ".sqlite"); + const auto status = fixture.temporary.path / (name + ".status"); + auto command = fixture.command(ledger, status, "", 0); + command.insert(command.end(), {"--native-event-retention", "window", "--checkpoint-interval", + std::to_string(count), "--ratio-interval", std::to_string(count)}); + Runner child(std::move(command), fixture.temporary.path / (name + ".stderr")); + wait_ready(status, 0, child); + std::uint32_t seed = 12345; + // Each batch is one ledger input. The stamped module records one + // equity point per bar; the opaque module also adds one closed trade. + const std::uint64_t bars_per_input = stamped ? 24 : 3; + for (std::uint64_t input = 0; input < count; ++input) { + std::string batch = "{\"type\":\"batch\",\"events\":["; + for (std::uint64_t bar = 1; bar <= bars_per_input; ++bar) { + if (bar != 1) batch += ','; + batch += seeded_input(input * bars_per_input + bar, seed); + } + child.send(batch + "]}\n"); + } + check_summary(child.finish(), count, count); + auto database = open_database(ledger, SQLITE_OPEN_READONLY); + sqlite3_stmt* raw = nullptr; + require(sqlite3_prepare_v2(database.get(), + "SELECT typeof(payload), length(CAST(payload AS BLOB)) " + "FROM report_checkpoints WHERE input_cursor=?", -1, &raw, nullptr) == SQLITE_OK, + "cannot inspect the stored checkpoint"); + std::unique_ptr query(raw, sqlite3_finalize); + require(sqlite3_bind_int64(raw, 1, count) == SQLITE_OK && sqlite3_step(raw) == SQLITE_ROW, + "checkpoint at the requested boundary is missing: " + child.errors()); + require(sqlite3_column_int64(raw, 1) > 1024 * 1024, "checkpoint did not exceed 1 MiB"); + const std::string storage(reinterpret_cast(sqlite3_column_text(raw, 0))); + const auto errors = child.errors(); + require(storage == (stamped ? "blob" : "text") && + occurrences(errors, "--native-event-retention is not applied") == (stamped ? 0u : 1u) && + (stamped || occurrences(errors, warning) == 1), + name + " checkpoint is " + storage + "; startup messages: " + errors); + }; + check(true); + fixture.library = unstamped_module; + check(false); +} + +// With --native-event-retention window a report checkpoint above 1 MiB is written into its row +// in pieces at every cursor. A checkpoint at cursor 512, the last one the tree fold used to +// build, must not be held in memory whole, and its bytes must be the bytes of a run without the +// option (which still builds that checkpoint from the tree and stores it as TEXT). +void window_checkpoint_streams_at_the_tree_cursor(Fixture& fixture) { + constexpr std::uint64_t inputs = 512, bars_per_input = 1000; + struct Outcome { + std::string storage, hash; + std::int64_t bytes = 0; + std::uint64_t peak_kib = 0; + }; + // `interval` above the number of inputs writes no checkpoint: the memory baseline. + const auto run = [&](const std::string& name, bool window, std::uint64_t interval) { + const auto ledger = fixture.temporary.path / (name + ".sqlite"); + const auto status = fixture.temporary.path / (name + ".status"); + auto command = fixture.command(ledger, status, "", 0); + if (window) command.insert(command.end(), {"--native-event-retention", "window"}); + command.insert(command.end(), {"--checkpoint-interval", std::to_string(interval), "--ratio-interval", + std::to_string(interval)}); + Runner child(std::move(command), fixture.temporary.path / (name + ".stderr"), /*measure_rss=*/true); + wait_ready(status, 0, child); + std::uint32_t seed = 12345; + for (std::uint64_t input = 0; input < inputs; ++input) { + std::string batch = "{\"type\":\"batch\",\"events\":["; + for (std::uint64_t bar = 1; bar <= bars_per_input; ++bar) { + if (bar != 1) batch += ','; + batch += seeded_input(input * bars_per_input + bar, seed); + } + child.send(batch + "]}\n"); + } + // A checkpoint is written inside its boundary input's own transaction. + wait_ready(status, inputs, child); + Outcome outcome; + const auto held = child.resident_high_water_kib(); + std::uint64_t exit_peak = 0; + check_summary(child.finish(0, &exit_peak), inputs, inputs); + outcome.peak_kib = std::max(held, exit_peak); + if (interval > inputs) return outcome; + auto database = open_database(ledger, SQLITE_OPEN_READONLY); + sqlite3_stmt* raw = nullptr; + require(sqlite3_prepare_v2(database.get(), + "SELECT typeof(payload), length(CAST(payload AS BLOB)), payload_hash " + "FROM report_checkpoints WHERE input_cursor=?", -1, &raw, nullptr) == SQLITE_OK, + "cannot inspect the stored checkpoint"); + std::unique_ptr query(raw, sqlite3_finalize); + require(sqlite3_bind_int64(raw, 1, static_cast(inputs)) == SQLITE_OK && + sqlite3_step(raw) == SQLITE_ROW, + name + ": the checkpoint at cursor 512 is missing: " + child.errors()); + outcome.storage = reinterpret_cast(sqlite3_column_text(raw, 0)); + outcome.bytes = sqlite3_column_int64(raw, 1); + outcome.hash = reinterpret_cast(sqlite3_column_text(raw, 2)); + return outcome; + }; + const auto baseline = run("window-no-checkpoint", true, inputs + 1); + const auto streamed = run("window-checkpoint", true, inputs); + const auto plain = run("plain-checkpoint", false, inputs); + require(streamed.bytes > 1024 * 1024 && streamed.bytes == plain.bytes && streamed.hash == plain.hash, + "the checkpoint at cursor 512 changed its bytes: " + std::to_string(streamed.bytes) + " bytes with the option, " + + std::to_string(plain.bytes) + " without"); + require(plain.storage == "text" && streamed.storage == "blob", + "stored as " + streamed.storage + " with the option and " + plain.storage + " without it"); + const auto growth_kib = streamed.peak_kib > baseline.peak_kib ? streamed.peak_kib - baseline.peak_kib : 0; + require(growth_kib < static_cast(streamed.bytes) / 1024, + "the checkpoint raised the peak resident memory by " + std::to_string(growth_kib) + + " KiB, as much as its " + std::to_string(streamed.bytes / 1024) + " KiB payload"); +} + void spool_write_failure_finishes(Fixture& fixture) { constexpr std::uint64_t count = 1100; const auto base = spool_run(fixture, "spool-unconstrained", count, nullptr); @@ -1510,6 +1629,67 @@ void spool_write_failure_finishes(Fixture& fixture) { } #if defined(__linux__) +void spool_probe_failure_keeps_curve_resident(Fixture& fixture) { + const auto directory = fs::canonical(fixture.temporary.path) / "ledger"; + const auto temporary = directory.parent_path() / "fallback"; + fs::create_directory(directory); + fs::create_directory(temporary); + struct statfs backing{}; + require(::statfs(temporary.c_str(), &backing) == 0 && + static_cast(backing.f_type) != UINT32_C(0x01021994) && + static_cast(backing.f_type) != UINT32_C(0x858458f6), + "the fallback directory must be disk-backed"); + const auto ledger = directory / "run.sqlite"; + const auto status = directory.parent_path() / "run.status"; + auto command = fixture.command(ledger, status, "--report-jsonl=delta", 0); + Database keeper(nullptr, sqlite3_close); + { + Runner seed(command, directory.parent_path() / "seed.stderr"); + wait_ready(status, 0, seed); + // Keep the pre-existing SQLite sidecars open across the restart. + keeper = open_database(ledger, SQLITE_OPEN_READONLY); + execute(keeper.get(), "BEGIN; SELECT COUNT(*) FROM inputs;"); + check_summary(seed.finish(), 0, 0); + } + fs::remove(status); + command.insert(command.end(), {"--native-event-retention", "window"}); + const struct RestoreWrite { + const fs::path& path; + ~RestoreWrite() { + std::error_code ignored; + fs::permissions(path, fs::perms::owner_write, fs::perm_options::add, ignored); + } + } writable{directory}; + fs::permissions(directory, fs::perms::owner_write, fs::perm_options::remove); + require(::access(directory.c_str(), W_OK) == -1 && errno == EACCES, + "the test process must not be able to write in the ledger directory"); + require(::setenv("TMPDIR", temporary.c_str(), 1) == 0, "cannot set the fallback directory"); + Runner child(std::move(command), directory.parent_path() / "probe.stderr"); + wait_ready(status, 0, child); + // The engine keeps its command journal in an unnamed file in TMPDIR in every run. Only + // the report scratch (file names starting "pineforge-scratch") must stay out of it. + auto held = child.unnamed_file_names_in(temporary); + child.send(fixture.inputs.front() + "\n"); + child.line(); + const auto held_after_input = child.unnamed_file_names_in(temporary); + held.insert(held.end(), held_after_input.begin(), held_after_input.end()); + check_summary(child.finish(), 1, 1); + const auto errors = child.errors(); + require(occurrences(errors, "cumulative report spool cannot be created") == 1, + "startup spool probe did not fail exactly once: " + errors); + std::size_t report_scratch_files = 0; + std::string held_names; + for (const auto& name : held) { + held_names += "[" + name + "]"; + if (name.rfind("pineforge-scratch", 0) == 0) ++report_scratch_files; + } + require(fs::is_empty(temporary) && report_scratch_files == 0 && + occurrences(errors, "the live report's equity curve stays in memory: ") == 1 && + errors.find("the ledger's directory cannot hold scratch files") != std::string::npos, + "expected a resident curve and one warning; unnamed files held in the fallback directory: " + + held_names + ": " + errors); +} + // A ledger on tmpfs: the spool probe admits the directory (O_TMPFILE works // there), so the runner must warn that the settled rows stay in memory // (runner/main.cpp, the statfs check after the ReportSpool probe). Returns @@ -1616,9 +1796,23 @@ void options_are_strict(Fixture& fixture) { int main(int argc, char** argv) { try { - require(argc == 4, "expected runner, native module and test case"); + require(argc == 4 || argc == 5, "expected runner, native module and test case"); std::signal(SIGPIPE, SIG_IGN); const std::string test_case = argv[3]; + if (test_case == "window-checkpoint-storage") { + require(argc == 5, "expected an unstamped module for the checkpoint case"); + Fixture fixture(argv[1], argv[2]); + window_checkpoint_storage(fixture, argv[4]); + std::cout << "PASS " << test_case << '\n'; + return 0; + } + require(argc == 4, "unexpected extra module"); +#if !defined(__linux__) + if (test_case == "window-checkpoint-streaming") { + std::cout << "SKIP window checkpoint streaming: runner VmHWM requires Linux /proc//status\n"; + return 77; + } +#endif if (test_case == "module-layout") { // Probe in a C++ process so instrumented modules use its runtime. std::unique_ptr library( @@ -1672,6 +1866,8 @@ int main(int argc, char** argv) { report_memory(fixture, true); } else if (test_case == "event-retention") { event_retention_option(fixture); + } else if (test_case == "window-checkpoint-streaming") { + window_checkpoint_streams_at_the_tree_cursor(fixture); } else if (test_case == "default-equity-export") { #if defined(__linux__) && defined(__x86_64__) default_equity_export(fixture); @@ -1681,6 +1877,10 @@ int main(int argc, char** argv) { #endif } else if (test_case == "spool-write-failure") { spool_write_failure_finishes(fixture); +#if defined(__linux__) + } else if (test_case == "spool-probe-failure") { + spool_probe_failure_keeps_curve_resident(fixture); +#endif } else if (test_case == "tmpfs-spool-warning") { #if defined(__linux__) if (!tmpfs_spool_warns(fixture)) { diff --git a/tests/test_native_lookup_memos.cpp b/tests/test_native_lookup_memos.cpp index 0e4319bd5..df7a78fef 100644 --- a/tests/test_native_lookup_memos.cpp +++ b/tests/test_native_lookup_memos.cpp @@ -305,6 +305,76 @@ std::vector session_bars(const Zone& zone, const Shape& shape, int n) { return bars; } +// Ask during the confirmed-input callback, before the second row is accepted. +// Both rows are in the batch, but its second row cannot supply a horizon yet. +struct AcceptedIntervalHost final : NativeStrategyHost { + struct Lookup { + std::int64_t stamp; + Probe::Interval interval; + }; + + explicit AcceptedIntervalHost(std::int64_t next_stamp) : next_stamp_(next_stamp) {} + + std::vector observed; + + void on_native_input(const Bar& bar, const NativeInputContext&) override { + const auto& consumer = NativeExecutionConsumer::bound(*this); + if (observed.empty()) + CHECK(!consumer.accepted_input_interval(next_stamp_)); + observed.push_back({bar.timestamp, consumer.accepted_input_interval(bar.timestamp)}); + } + + void on_native_bar(const Bar&, const NativeDecisionContext&) override {} + +private: + const std::int64_t next_stamp_; +}; + +void accepted_input_interval_respects_frontier() { + namespace nc = native_calendar; + constexpr Shape hourly{"60", "60", 60 * kMinute}; + constexpr auto midnight = kZones[0].t0; + const Zone sessions[] = { + {"UTC", "0930-1130,1300-1600", midnight + (9 * 60 + 30) * kMinute}, + {"UTC", "0900-1130,1220-1530", midnight + 9 * 60 * kMinute}, + }; + const auto tf = nc::parse_timeframe(hourly.input_tf); + CHECK(tf.has_value()); + if (!tf) return; + for (const auto& zone : sessions) { + const auto calendar = nc::parse_session(zone.session, zone.timezone); + CHECK(calendar.has_value()); + if (!calendar) continue; + const auto day = nc::session_day_at(*calendar, zone.t0); + CHECK(day && day->spans.size() == 2); + if (!day || day->spans.size() != 2) continue; + const auto reopen = day->spans[1].first; + const auto next = day->origin_ms + + ((reopen - day->origin_ms) / hourly.step + 1) * hourly.step; + const auto clipped = nc::interval_containing(*calendar, *tf, reopen); + const auto successor = nc::interval_containing(*calendar, *tf, next); + CHECK(clipped && clipped->open_ms < reopen && clipped->eligible_open_ms == reopen + && reopen < clipped->last_traded_close_ms && clipped->next_period_open_ms == next); + CHECK(successor && successor->open_ms == next && successor->last_traded_close_ms > next); + if (!clipped || !successor) continue; + + const Bar rows[] = {{100, 101, 99, 100.5, 1, reopen}, + {102, 103, 101, 102.5, 2, next}}; + AcceptedIntervalHost host(next); + const auto setup = host.configure_native(spec_for(zone, hourly, false, 1)); + CHECK(setup.status == NativeSetupStatus::Applied); + if (setup.status != NativeSetupStatus::Applied) continue; + host.run(rows, 2); + CHECK(host.last_error().empty()); + CHECK(host.observed.size() == 2); + if (host.observed.size() != 2) continue; + CHECK(host.observed[0].stamp == reopen); + CHECK(same_interval(host.observed[0].interval, clipped)); + CHECK(host.observed[1].stamp == next); + CHECK(same_interval(host.observed[1].interval, successor)); + } +} + void memos_answer_as_fresh_resolutions() { std::mt19937_64 rng(0xA5A5DEADBEEF1234ULL); int runs = 0; @@ -361,6 +431,7 @@ void memos_answer_as_fresh_resolutions() { } // namespace int main() { + accepted_input_interval_respects_frontier(); memos_answer_as_fresh_resolutions(); std::printf("test_native_lookup_memos: %d passed, %d failed\n", passed, failed); return failed == 0 ? 0 : 1; diff --git a/tests/test_report_spool_unlink_failure.cpp b/tests/test_report_spool_unlink_failure.cpp new file mode 100644 index 000000000..291763405 --- /dev/null +++ b/tests/test_report_spool_unlink_failure.cpp @@ -0,0 +1,105 @@ +// SPDX-License-Identifier: Apache-2.0 +#include "../runner/report_spool.hpp" + +#include +#include +#include +#include +#include +#include +#include +#include +#include + +namespace { + +namespace fs = std::filesystem; +using pineforge::live::ReportSpool; +constexpr char spool_prefix[] = ".pineforge-report-spool-"; +bool refuse_unlink = false; +int refused_unlinks = 0; + +struct TempDir { + fs::path path; + TempDir() { + auto name = (fs::temp_directory_path() / "pineforge-spool-unlink-XXXXXX").string(); + if (!::mkdtemp(name.data())) throw std::runtime_error("cannot create test directory"); + path = name; + } + ~TempDir() { + std::error_code ignored; + fs::remove_all(path, ignored); + } +}; + +bool spool_name(const char* path) { + const auto* slash = std::strrchr(path, '/'); + return std::strncmp(slash ? slash + 1 : path, spool_prefix, sizeof(spool_prefix) - 1) == 0; +} + +std::string named_spools(const fs::path& directory) { + std::string result; + for (const auto& entry : fs::directory_iterator(directory)) { + if (spool_name(entry.path().filename().c_str())) + result += "\n" + entry.path().string(); + } + return result; +} + +} // namespace + +extern "C" int __real_open(const char*, int, ...); +extern "C" int __wrap_open(const char* path, int flags, ...) { +#ifdef O_TMPFILE + // Exercise mkstemp even on filesystems that support unnamed files. + if ((flags & O_TMPFILE) == O_TMPFILE) { + errno = EOPNOTSUPP; + return -1; + } +#endif + if (flags & O_CREAT) { + va_list args; + va_start(args, flags); + const auto mode = va_arg(args, int); + va_end(args); + return __real_open(path, flags, mode); + } + return __real_open(path, flags); +} + +extern "C" int __real_unlink(const char*); +extern "C" int __wrap_unlink(const char* path) { + if (refuse_unlink && spool_name(path)) { + // Refuse only the first attempt so the cleanup can remove the file. + refuse_unlink = false; + ++refused_unlinks; + errno = EACCES; + return -1; + } + return __real_unlink(path); +} + +int main() { + try { + TempDir directory; + refuse_unlink = true; + std::string refusal; + try { + ReportSpool spool(directory.path.string()); + } catch (const std::runtime_error& error) { + refusal = error.what(); + } + const auto left = named_spools(directory.path); + if (refused_unlinks != 1 || refusal != "cumulative report spool cannot be created" || !left.empty()) + throw std::runtime_error("expected a create refusal and no named spools; refusal: " + + refusal + "; named spools:" + left); + + ReportSpool spool(directory.path.string()); + const auto named = named_spools(directory.path); + if (!named.empty()) throw std::runtime_error("successful spool has a name:" + named); + return 0; + } catch (const std::exception& error) { + std::cerr << error.what() << '\n'; + return 1; + } +} diff --git a/tests/test_run_failure_code_numbers.py b/tests/test_run_failure_code_numbers.py index ed553470c..794aff90e 100644 --- a/tests/test_run_failure_code_numbers.py +++ b/tests/test_run_failure_code_numbers.py @@ -258,4 +258,6 @@ def test_cr559_map_capacity_catalog_contract(): "maxSize": {"kind": "integer"}, "currSize": {"kind": "integer"}} assert entry["english"] == [ "Error on bar {bar_index}: A map cannot have more than {maxSize} " + "key-value pairs. The current size is {currSize}.", + "Error before the first bar: A map cannot have more than {maxSize} " "key-value pairs. The current size is {currSize}."] diff --git a/tests/test_series_percentile_nearest_rank.cpp b/tests/test_series_percentile_nearest_rank.cpp index a1ac9be09..6838fbbc8 100644 --- a/tests/test_series_percentile_nearest_rank.cpp +++ b/tests/test_series_percentile_nearest_rank.cpp @@ -202,17 +202,20 @@ void errors() { [&] { (void)actual.compute(1.0, na(), 50.0); }, RunFailureCode::pine_invalid_argument, "na length is refused", - "Invalid value of the 'length' argument (0) in the 'percentile_nearest_rank' function. It must be > 0."); + "Error before the first bar: Invalid value of the 'length' argument (0) " + "in the 'percentile_nearest_rank' function. It must be > 0."); expect_failure( [&] { (void)actual.compute(1.0, 0.0, 50.0); }, RunFailureCode::pine_invalid_argument, "zero length is refused", - "Invalid value of the 'length' argument (0) in the 'percentile_nearest_rank' function. It must be > 0."); + "Error before the first bar: Invalid value of the 'length' argument (0) " + "in the 'percentile_nearest_rank' function. It must be > 0."); expect_failure( [&] { (void)actual.compute(1.0, -3.0, 50.0); }, RunFailureCode::pine_invalid_argument, "negative length is refused", - "Invalid value of the 'length' argument (-3) in the 'percentile_nearest_rank' function. It must be > 0."); + "Error before the first bar: Invalid value of the 'length' argument (-3) " + "in the 'percentile_nearest_rank' function. It must be > 0."); require(is_na(actual.compute(1.0, 15001.0, 50.0)), "a 15001-value window does not exceed the historical limit"); @@ -220,7 +223,8 @@ void errors() { [&] { (void)actual.compute(1.0, 15002.0, 50.0); }, RunFailureCode::pine_runtime_limit, "a 15002-value window is refused", - "The script attempts to reference historical data that is too far from the current bar (15001 bars back). The historical buffer's limit is 15000 bars."); + "Error before the first bar: The script attempts to reference historical data that is " + "too far from the current bar (15001 bars back). The historical buffer's limit is 15000 bars."); } void recompute_replaces_last_append() { diff --git a/tests/test_series_sma.cpp b/tests/test_series_sma.cpp index 8d4adf39b..8926e357b 100644 --- a/tests/test_series_sma.cpp +++ b/tests/test_series_sma.cpp @@ -278,7 +278,6 @@ void tv_clock_na_prefix() { void precision_and_order() { // Exact integer sums: [2^53, 1, -2^53] sums to 1, and adding 5,-3 // sums to 3. Independent rational means distinguish lost low addends. - // They specify the chosen compensated path, not measured TV bit order. const Row rows[] = { {0x1p53, 1, 0x1p53}, {1, 2, 0x1p52}, @@ -296,6 +295,21 @@ void precision_and_order() { expect(actual.recompute(3, 5), 1.8); // exact sum 9 expect(actual.recompute(-5, 2), 0); expect(actual.compute(9, 6), 1.6666666666666667); // exact sum 10 + + // The two large values cancel, as do 1 and -1. A single compensation + // term loses the third addend when it is rounded into 1. + constexpr double tiny = 1e-16; + constexpr double window[] = {1e16, 1.0, tiny, -1e16, -1.0}; + SeriesSma variable; + pineforge::ta::SMA fixed(5); + double series_mean = na(); + double fixed_mean = na(); + for (double value : window) { + series_mean = variable.compute(value, 5); + fixed_mean = fixed.compute(value); + } + assert(fixed_mean == tiny / 5.0); + assert(series_mean == fixed_mean); } void retained_history_and_storage() { @@ -367,9 +381,8 @@ void constant_length_corpus_comparison() { // tests/fixtures/qty_step_mincontract/btc60/feed/bars.csv (header + 1441). // This fixed-N=20 exercise of the VARIABLE adapter is not coverage of a // generated constant-length script or proof of its codegen routing. - // The 1.13.0 variable-length path temporarily uses Neumaier arithmetic; - // measure its difference from real fixed SMA across every numeric row. - // No alignment/parity verdict or expected difference count/max is pinned. + // Report the numeric difference from fixed SMA across every numeric row. + // This feed does not provide independent TradingView expectations. const std::string file = __FILE__; const auto slash = file.find_last_of("/\\"); assert(slash != std::string::npos); diff --git a/tests/test_service_unlink_failure.cpp b/tests/test_service_unlink_failure.cpp new file mode 100644 index 000000000..91041e66e --- /dev/null +++ b/tests/test_service_unlink_failure.cpp @@ -0,0 +1,210 @@ +// SPDX-License-Identifier: Apache-2.0 +#include "../runner/service.hpp" + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +namespace { + +namespace fs = std::filesystem; +using pineforge::live::Json; +using pineforge::live::atomic_json_file; + +int refusals_remaining = 0; +unsigned temporary_unlinks = 0; + +} // namespace + +extern "C" int __real_unlink(const char* name); + +extern "C" int __wrap_unlink(const char* name) { + if (std::strstr(name, ".new.")) { + ++temporary_unlinks; + if (refusals_remaining != 0) { + if (refusals_remaining > 0) --refusals_remaining; + errno = EACCES; + return -1; + } + } + return __real_unlink(name); +} + +namespace { + +void require(bool condition, const char* message) { + if (!condition) throw std::runtime_error(message); +} + +class TemporaryDirectory { +public: + TemporaryDirectory() { + std::string name = (fs::temp_directory_path() / "pineforge-service-unlink-XXXXXX").string(); + require(::mkdtemp(name.data()) != nullptr, "cannot create test directory"); + path = name; + } + + ~TemporaryDirectory() { + refusals_remaining = 0; + std::error_code ignored; + fs::remove_all(path, ignored); + } + + fs::path path; +}; + +class WarningCapture { +public: + WarningCapture() : previous_(std::cerr.rdbuf(output_.rdbuf())) {} + ~WarningCapture() { std::cerr.rdbuf(previous_); } + + std::string str() const { return output_.str(); } + +private: + std::ostringstream output_; + std::streambuf* previous_; +}; + +std::vector directory_entries(const fs::path& directory) { + std::vector entries; + for (const auto& entry : fs::directory_iterator(directory)) entries.push_back(entry.path()); + return entries; +} + +fs::path sole_temporary_file(const fs::path& target) { + const auto entries = directory_entries(target.parent_path()); + require(entries.size() == 2, "expected the target and one temporary file"); + require(entries[0] == target || entries[1] == target, "target missing"); + const auto& temporary = entries[0] == target ? entries[1] : entries[0]; + const auto prefix = target.filename().string() + ".new."; + require(temporary.filename().string().compare(0, prefix.size(), prefix) == 0, + "unexpected temporary name"); + return temporary; +} + +std::string contents(const fs::path& file) { + std::ifstream input(file, std::ios::binary); + require(static_cast(input), "cannot open installed file"); + return std::string(std::istreambuf_iterator(input), {}); +} + +std::string expected_warning(const fs::path& temporary) { + return "pineforge-live: warning: cannot remove temporary service file " + temporary.string() + + ": " + std::strerror(EACCES) + "\n"; +} + +void link_with_persistent_refusal(const Json& document) { + TemporaryDirectory directory; + const auto target = directory.path / "request.json"; + WarningCapture warning; + refusals_remaining = -1; + temporary_unlinks = 0; + atomic_json_file(target.string(), document, true, false); + + require(contents(target) == document.dump() + "\n", "installed document differs"); + const auto temporary = sole_temporary_file(target); + require(fs::equivalent(temporary, target), "temporary file is not a hard link to the target"); + require(warning.str() == expected_warning(temporary), "expected exactly one cleanup warning"); + require(temporary_unlinks == 2, "expected one cleanup retry"); +} + +void link_with_one_refusal(const Json& document) { + TemporaryDirectory directory; + const auto target = directory.path / "request.json"; + WarningCapture warning; + refusals_remaining = 1; + temporary_unlinks = 0; + atomic_json_file(target.string(), document, true, false); + + require(contents(target) == document.dump() + "\n", "installed document differs"); + require(directory_entries(directory.path) == std::vector{target}, "retry left a temporary file"); + require(warning.str().empty(), "successful retry emitted a warning"); + require(temporary_unlinks == 2, "expected one cleanup retry"); +} + +void check_existing_target(const Json& document, int refusals) { + TemporaryDirectory directory; + const auto target = directory.path / "request.json"; + const std::string original = "{}\n"; + { + std::ofstream existing(target); + existing << original; + existing.close(); + require(static_cast(existing), "cannot write test target"); + } + WarningCapture warning; + refusals_remaining = refusals; + temporary_unlinks = 0; + bool install_failed = false; + try { + atomic_json_file(target.string(), document, true, false); + } catch (const std::runtime_error& error) { + require(std::string(error.what()) == "cannot install atomic service file", "installation error changed"); + install_failed = true; + } + require(install_failed, "existing target did not prevent installation"); + require(contents(target) == original, "failed installation changed the target"); + + if (refusals) { + const auto temporary = sole_temporary_file(target); + require(warning.str() == expected_warning(temporary), "expected exactly one cleanup warning"); + require(temporary_unlinks == 2, "expected one cleanup retry"); + } else { + require(directory_entries(directory.path) == std::vector{target}, "cleanup left a temporary file"); + require(warning.str().empty(), "successful cleanup emitted a warning"); + require(temporary_unlinks == 1, "expected one cleanup attempt"); + } +} + +void link_onto_existing_target(const Json& document) { + check_existing_target(document, 0); + check_existing_target(document, -1); +} + +void rename_does_not_unlink(const Json& document) { + TemporaryDirectory directory; + const auto target = directory.path / "status.json"; + WarningCapture warning; + refusals_remaining = -1; + temporary_unlinks = 0; + atomic_json_file(target.string(), document, false, false); + + require(contents(target) == document.dump() + "\n", "installed document differs"); + require(directory_entries(directory.path) == std::vector{target}, "rename left a temporary file"); + require(warning.str().empty(), "rename emitted a cleanup warning"); + require(temporary_unlinks == 0, "rename attempted to unlink the temporary file"); +} + +} // namespace + +int main() { + const auto document = Json::object({{"ready", Json::boolean(true)}}); + const struct { + const char* name; + void (*run)(const Json&); + } cases[] = { + {"link_with_persistent_refusal", link_with_persistent_refusal}, + {"link_with_one_refusal", link_with_one_refusal}, + {"link_onto_existing_target", link_onto_existing_target}, + {"rename_does_not_unlink", rename_does_not_unlink}, + }; + int failures = 0; + for (const auto& test : cases) { + try { + test.run(document); + } catch (const std::exception& error) { + std::cerr << test.name << ": " << error.what() << '\n'; + ++failures; + } + } + return failures == 0 ? 0 : 1; +} diff --git a/tests/test_str_format_time_preservation.cpp b/tests/test_str_format_time_preservation.cpp index 750bb49e3..447c5fdad 100644 --- a/tests/test_str_format_time_preservation.cpp +++ b/tests/test_str_format_time_preservation.cpp @@ -3,6 +3,8 @@ #include #include #include +#include +#include #include #include #include @@ -41,11 +43,13 @@ constexpr FormatTimeCase cases[] = { #if defined(__GLIBC__) // glibc's UTC %z spelling has not been established on macOS here. {"raw_percent_z", 1706746029000LL, "%z", "UTC", "+0000", 5}, - // glibc's GMT conversion normalizes TZ to UTC for %Z and mktime-based %s. + // glibc's GMT conversion normalizes TZ to UTC for %Z. // These spellings and the C-locale %c layout are unproved on macOS here. {"raw_percent_Z", 1706746029000LL, "%Z", "GMT", "UTC", 3}, {"raw_percent_c", 1706746029000LL, "%c", "UTC", "Thu Feb 1 00:07:09 2024", 24}, {"raw_percent_s", 1706746029000LL, "%s", "GMT", "1706746029", 10}, + // `%s` is the Unix second, independent of the process timezone. + {"raw_percent_s_utc_ambient_zone", 1706746029000LL, "%s", "UTC", "1706746029", 10}, #endif {"raw_percent_escape", 1706746029000LL, "%%", "UTC", "%", 1}, #if defined(__GLIBC__) @@ -167,12 +171,26 @@ int main() { std::fprintf(stderr, "FAIL: C time locale unavailable\n"); return 1; } + const char* const previous_tz = std::getenv("TZ"); + const bool had_previous_tz = previous_tz != nullptr; + const std::string previous_tz_value = previous_tz == nullptr ? "" : previous_tz; + if (::setenv("TZ", "America/New_York", 1) != 0) { + std::fprintf(stderr, "FAIL: cannot set process timezone\n"); + return 1; + } + ::tzset(); int failed = 0; for (const auto& test_case : cases) { if (!check_format_time(test_case)) { ++failed; } } + if (had_previous_tz) { + ::setenv("TZ", previous_tz_value.c_str(), 1); + } else { + ::unsetenv("TZ"); + } + ::tzset(); std::printf("%zu cases, %d failed\n", sizeof(cases) / sizeof(cases[0]), failed); return failed == 0 ? 0 : 1; } diff --git a/tests/test_ta_na_source_rules.cpp b/tests/test_ta_na_source_rules.cpp index ef9bdf9dc..8f1e27bd3 100644 --- a/tests/test_ta_na_source_rules.cpp +++ b/tests/test_ta_na_source_rules.cpp @@ -254,6 +254,23 @@ static void test_ema_seeding_named_per_instance() { CHECK(!ta::ema_na_warmup_flag()); // left as found } +static void test_ema_simple_average_exact_seed() { + constexpr double tiny = 1e-16; + ta::EMA ema(5, ta::EmaSeeding::SimpleAverage); + for (double src : {1e16, 1.0, tiny, -1e16}) { + CHECK(is_na(ema.compute(src))); + } + // The two large values and the two unit values cancel. The small term + // is the exact sum before division by the length. + CHECK(ema.compute(-1.0) == tiny / 5.0); + + // Retry the seed bar through a missing observation and a different + // source. The prefix must survive both replacements. + CHECK(is_na(ema.recompute(na()))); + CHECK(ema.recompute(1.0) == 2.0 / 5.0); + CHECK(ema.recompute(-1.0) == tiny / 5.0); +} + int main() { test_ema_warm_na_returns_na_and_resumes(); // R1 test_sma_seeded_na_returns_held_mean(); // R2 @@ -264,6 +281,7 @@ int main() { test_sma_32bar_na_run_holds(); // G5 (RED on HEAD) test_ema_na_warmup_preseed_unaffected(); // G6 test_ema_seeding_named_per_instance(); // G7 + test_ema_simple_average_exact_seed(); std::printf("\ntest_ta_na_source_rules: %d passed, %d failed\n", tests_passed, tests_failed); diff --git a/tests/test_timeframe.cpp b/tests/test_timeframe.cpp index 6d46ee024..fe00bc84a 100644 --- a/tests/test_timeframe.cpp +++ b/tests/test_timeframe.cpp @@ -2,11 +2,13 @@ #include #include #include +#include #include #include #include #include #include +#include #include using namespace pineforge; @@ -72,6 +74,78 @@ static void test_tf_ratio() { CHECK(tf_ratio("D", "M") == -1); } +static void test_detect_timeframe_extreme_timestamp_gaps() { + const auto earliest = std::numeric_limits::min(); + const auto latest = std::numeric_limits::max(); + const Bar bars[] = { + make_bar(1, 1, 1, 1, 1, earliest), + make_bar(1, 1, 1, 1, 1, latest), + make_bar(1, 1, 1, 1, 1, earliest + 300000), + make_bar(1, 1, 1, 1, 1, earliest + 600000), + }; + + CHECK(detect_timeframe(bars, 2) == "W"); + CHECK(detect_timeframe(bars + 1, 2) == "1"); + // The upper median of the two increasing gaps is the wider gap. + CHECK(detect_timeframe(bars, 4) == "W"); +} + +// True when `call` throws the refusal a timeframe count past int already +// gets: std::out_of_range coded pine_invalid_argument (timeframe, unparseable). +template +static bool refused_like_count_past_int(Call call) { + try { + (void)call(); + } catch (const std::out_of_range& error) { + const RunFailureValue failure = classify_run_failure(error); + return failure.code == RunFailureCode::pine_invalid_argument && failure.args + && *failure.args == "{\"argument\":\"timeframe\",\"rule\":\"unparseable\"}"; + } catch (...) { + } + return false; +} + +static bool seconds_refused(const char* tf) { + return refused_like_count_past_int([tf] { return tf_to_seconds(tf); }); +} + +// A count the parser admits whose seconds do not fit the int this API +// returns. Minutes take the refusal of a count past int; a day or week count +// is calendar-only (-1), like M; a negative count past the int is refused. +// The last count that fits keeps its exact seconds. +static void test_tf_to_seconds_int_overflow() { + std::printf("test_tf_to_seconds_int_overflow\n"); + static_assert(std::numeric_limits::max() / 60 == 35791394, "int is 32-bit"); + static_assert(std::numeric_limits::max() / 604800 == 3550, "int is 32-bit"); + static_assert(std::numeric_limits::max() / 86400 == 24855, "int is 32-bit"); + // The refusal itself: a count past int, refused before any multiply. + CHECK(seconds_refused("2147483648")); + // Minutes. + CHECK(tf_to_seconds("35791394") == 2147483640); + CHECK(seconds_refused("35791395")); + CHECK(seconds_refused("71582789")); // its product wraps to 44 + CHECK(seconds_refused("2147483647")); + CHECK(tf_to_seconds("-35791394") == -2147483640); + CHECK(seconds_refused("-35791395")); + // Weeks. + CHECK(tf_to_seconds("3550W") == 2147040000); + CHECK(tf_to_seconds("3551W") == -1); + CHECK(tf_to_seconds("-3550W") == -2147040000); + CHECK(seconds_refused("-3551W")); + // Days: the existing guard; a negative count past the int. + CHECK(tf_to_seconds("24855D") == 2147472000); + CHECK(tf_to_seconds("24856D") == -1); + CHECK(tf_to_seconds("-24855D") == -2147472000); + CHECK(seconds_refused("-24856D")); + // tf_ratio: a minute target past the int is refused, never read as a + // calendar period; equal calendar-only literals pass their bars through. + CHECK(tf_ratio("1", "35791394") == 35791394); + CHECK(refused_like_count_past_int([] { return tf_ratio("1", "35791395"); })); + CHECK(tf_ratio("24856D", "24856D") == 1); + CHECK(tf_ratio("3551W", "3551W") == 1); + CHECK(tf_ratio("W", "3551W") == -1); +} + // ─── Ratio-based aggregation ────────────────────────────────────────────────── static void test_ratio_aggregation() { @@ -268,6 +342,8 @@ int main() { test_tf_to_seconds(); test_tf_ratio(); + test_detect_timeframe_extreme_timestamp_gaps(); + test_tf_to_seconds_int_overflow(); test_ratio_aggregation(); test_passthrough(); test_calendar_daily_aggregation(); diff --git a/tests/test_timezone_resource_and_name.cpp b/tests/test_timezone_resource_and_name.cpp index 2cf26cc95..646771857 100644 --- a/tests/test_timezone_resource_and_name.cpp +++ b/tests/test_timezone_resource_and_name.cpp @@ -507,10 +507,29 @@ void timezone_system_error_host_rows() { {"chart_day_key", &ZonedHost::read_chart_day, 101}, {"chart_time", &ZonedHost::read_chart_hour, 20}, }; + const auto real = libc_setenv(); + CHECK(real != nullptr); + if (real == nullptr) return; + const char* const found = std::getenv("TZ"); + const bool had_zone = found != nullptr; + const std::string original = had_zone ? found : ""; + const auto restore = [&] { + setenv_failure = 0; + setenv_failed = false; + const int restored = had_zone ? real("TZ", original.c_str(), 1) : ::unsetenv("TZ"); + CHECK(restored == 0); + ::tzset(); + return restored == 0; + }; for (const auto& row : rows) { - { - pineforge::tz_util::ScopedTimezone previous("UTC+9"); + // Force a zone change even when the caller already uses the chart zone. + const int seeded = real("TZ", "UTC-9", 1); + CHECK(seeded == 0); + if (seeded != 0) { + restore(); + return; } + ::tzset(); setenv_failed = false; setenv_failure = EINVAL; bool escaped = false; @@ -524,12 +543,21 @@ void timezone_system_error_host_rows() { } catch (const std::exception& error) { std::fprintf(stderr, "%s threw the wrong exception: %s\n", row.name, error.what()); CHECK(false); + } catch (...) { + std::fprintf(stderr, "%s threw a non-standard exception\n", row.name); + CHECK(false); } setenv_failure = 0; CHECK(setenv_failed && escaped); std::printf("timezone_system_error_host_rows: %s, setenv fired=%d, escaped=%d\n", row.name, int(setenv_failed), int(escaped)); - CHECK((host.*row.read)(first_bar) == row.expected); + try { + CHECK((host.*row.read)(first_bar) == row.expected); + } catch (...) { + restore(); + throw; + } + if (!restore()) return; } } #endif