From 07ed28095dde5e8063570aa6a2436039b5bd43ca Mon Sep 17 00:00:00 2001 From: Thorsten Fuchs <76904008+ThorFuchs@users.noreply.github.com> Date: Wed, 28 Oct 2026 14:35:51 +0100 Subject: [PATCH 1/5] Add cumulative inspection batches and immutable read-only stores --- AGENTS.md | 19 +- CHANGELOG.md | 17 + CONTRIBUTING.md | 9 + Documentation/Architecture/API.md | 13 + .../Architecture/ExtractionManifest.json | 11 + .../BoundedWorkflows1.8.md | 63 ++++ .../SOURCE-AUTHORITY.md | 9 + Documentation/Qualification/Release1.8.md | 96 ++++++ Documentation/Qualification/ReleaseLines.md | 10 + .../Qualification/TestDiscovery.json | 36 +- Documentation/README.md | 7 + Documentation/ReleaseRoadmap.md | 10 + Examples/BoundedWorkflows/Package.swift | 13 + .../Sources/BoundedWorkflows/Main.swift | 50 +++ README.md | 78 +++-- Scripts/build_documentation_site.py | 6 +- Scripts/ci.py | 50 ++- Scripts/qualify_store.py | 2 +- Scripts/qualify_structural_workflows.py | 77 +++++ Scripts/test_qualification_harness.py | 37 ++- Sources/JSONCLIEngine/CLIBuildIdentity.swift | 2 +- Sources/JSONCLIEngine/CLIReleaseVersion.swift | 2 +- .../Documentation.docc/BoundedWorkflows.md | 73 +++++ .../Documentation.docc/JSONInspection.md | 13 + .../JSONInspection/JSONInspectionBatch.swift | 153 +++++++++ .../Store/JSONConfiguredStore.swift | 133 ++++++-- .../Store/JSONStoreProfile.swift | 256 +++++++++++---- .../Store/JSONStoreQueries.swift | 234 +++++++++++++ .../JSONNavigation/JSONTraversalContext.swift | 15 + Tests/JSONCLITests/IntegrationTests.swift | 2 +- Tests/JSONInspectionTests/WorkflowTests.swift | 309 ++++++++++++++++++ 31 files changed, 1647 insertions(+), 158 deletions(-) create mode 100644 Documentation/Architecture/StructuralIntelligence/BoundedWorkflows1.8.md create mode 100644 Documentation/Qualification/Release1.8.md create mode 100644 Examples/BoundedWorkflows/Package.swift create mode 100644 Examples/BoundedWorkflows/Sources/BoundedWorkflows/Main.swift create mode 100644 Scripts/qualify_structural_workflows.py create mode 100644 Sources/JSONInspection/Documentation.docc/BoundedWorkflows.md create mode 100644 Sources/JSONInspection/JSONInspectionBatch.swift create mode 100644 Sources/JSONInspection/Store/JSONStoreQueries.swift create mode 100644 Tests/JSONInspectionTests/WorkflowTests.swift diff --git a/AGENTS.md b/AGENTS.md index b0ca32d..88f7f62 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -1,9 +1,9 @@ # Contributor instructions -Prepared release source is 1.7.0; publication is a separate owner action. Its -parent development line is 1.7.0-dev and stable 1.6.1 remains separate. This is -not a maintenance release or the frozen 6fe8067 RC. Preserve all source-bound -qualification history; source integration never publishes a version. +Development source is 1.8.0-dev, based on published 1.7.0 at +8557badef562cf2f059e964c113d4a7f60ecbe6c. Stable 1.6.1 remains a separate +maintenance line. Preserve every source-bound qualification record. No 1.8 +merge/tag/publication follows from implementation or passing tests. Swift-JSON requires Swift tools/compiler 6.4+, Swift language mode 6 and macOS/SDK 27+. Read README.md, CONTRIBUTING.md and the relevant contracts in @@ -50,10 +50,13 @@ Scalar scanning and immutable plan-local regex sharing are selected private optimizations. Source-attached N3 snapshots and P1 tree/sparse-table projections live in JSONInspection. NAV10 set/matrix and NAV11 directed-simple-graph APIs are development capabilities inside JSONInspection; no graph product, mutation or serializer is introduced. -Private NAV12 store construction/preparation remains package-only and shares one -original invocation ledger; no public store/follow API or persistence is exposed. -Wider retrieval/persistence, NAV13 batches and JSON text-sequence framing remain -separate, unadmitted work. +The 1.8 candidate promotes the checked immutable configured store inside +JSONInspection and adds ordered path/snapshot/store value batches, exact-ID lookup +and bounded complete BFS follow. Preserve the original construction phases and +one invocation-owned ledger per operation. Positions remain unforgeable and +source-instance-local. No persistence or mutable store is exposed. See +Documentation/Architecture/StructuralIntelligence/BoundedWorkflows1.8.md. +Wider retrieval/persistence and JSON text-sequence framing remain separate work. JSONTransform uses one private compiler/VM and the CLI's accounted adapter. Preserve its accepted 42 authored declarations and exact semantics. The original diff --git a/CHANGELOG.md b/CHANGELOG.md index dfe6eb8..4f4f047 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,22 @@ # Changelog +## Unreleased — 1.8.0-dev + +- Add ordered typed-path, snapshot-ordinal and store-position value batches using + one cumulative native ledger per batch, complete source admission and atomic results. +- Promote the original checked configured-store profile/construction inside + JSONInspection; add exact-ID lookup and finite complete BFS reference following. +- Preserve exact numbers/scalar identity, unforgeable source-local positions, + explicit absent/null/link states and original construction error precedence. +- Add external SwiftPM examples, ordinary compiler refusals, boundary/lifecycle/ + concurrency tests and exact scoped CI discovery. Qualification remains source-bound; + no 1.8 release or performance/platform pass is implied. + +## Publication note — 28 October 2026 + +Version 1.7.0 was published from 8557badef562cf2f059e964c113d4a7f60ecbe6c. +The preparation records below retain their original historical wording. + ## 1.7.0 — prepared source; publication separate - Reconcile admitted set/matrix APIs with the reviewed public directed graph, diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index bfa44f0..ba2262c 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -1,3 +1,12 @@ +# 1.8 scoped workflow qualification + +This isolated development source is 1.8.0-dev from published 1.7.0. +Run `python3 Scripts/ci.py --lane structural-workflows` for exact Debug/Release +behavior, preserved store cases and public compiler/consumer checks. Run +`--lane structural-workflows-safety` for scoped ASan, shared-state TSan and strict +memory checks; complete native/Linux/docs/WASI gates still apply before release. +Do not infer a pass when storage or an actual supported runtime is unavailable. + # Contributing Choose the source line deliberately: maintenance changes use the 1.6 line; diff --git a/Documentation/Architecture/API.md b/Documentation/Architecture/API.md index c8e167f..55b2bcd 100644 --- a/Documentation/Architecture/API.md +++ b/Documentation/Architecture/API.md @@ -1,3 +1,16 @@ +# 1.8 additive API candidate + +The latest released source is 1.7.0 (8557bade); this feature source is 1.8.0-dev. +No product or dependency is added. JSONInspection now includes +`JSONInspectionBatch`, snapshot ordinal batches, public `JSONStoreBuilder`/ +`JSONConfiguredStore` and typed profile/metadata/error declarations, exact-ID +lookup, position value batches and finite complete BFS follow. +See [the exact behavior/ownership boundary](StructuralIntelligence/BoundedWorkflows1.8.md) +and [qualification](../Qualification/Release1.8.md). +Navigation public signatures stay unchanged; its new selection capability is +package-only and uses the original admitted scope. Earlier inventories below +remain historical; compiler graphs must establish the final additive surface. + # Current API inventory Unreleased 1.7.0-dev consolidation has twelve library products plus swift-json and diff --git a/Documentation/Architecture/ExtractionManifest.json b/Documentation/Architecture/ExtractionManifest.json index e174b41..4e48765 100644 --- a/Documentation/Architecture/ExtractionManifest.json +++ b/Documentation/Architecture/ExtractionManifest.json @@ -355,6 +355,17 @@ "attribution": "Apache-2.0 prototype SPDX retained; original integration cases extend actual-engine qualification.", "validation": "Scripts/ci.py --lane dsl; native and safety exact discovery", "evidence": "Actual-engine Debug/Release/ASan receipts and real public consumer" + }, + { + "component": "1.8 bounded structural workflows", + "source": "Original Swift-JSON implementation over the existing native prepared traversal and NAV12 checked store construction", + "paths": [ + "Sources/JSONInspection/JSONInspectionBatch.swift", + "Sources/JSONInspection/Store/JSONStoreQueries.swift", + "Examples/BoundedWorkflows" + ], + "validation": "Exact Debug/Release behavior, preserved 65 store vectors, public SwiftPM consumer, codegen negatives and safety/docs/platform gates; see Documentation/Qualification/Release1.8.md for actual status", + "dependencies": "No new dependencies or numeric/key/schema authority; Apache-2.0 original contributions" } ], "original_implementation_notes": [ diff --git a/Documentation/Architecture/StructuralIntelligence/BoundedWorkflows1.8.md b/Documentation/Architecture/StructuralIntelligence/BoundedWorkflows1.8.md new file mode 100644 index 0000000..a22601d --- /dev/null +++ b/Documentation/Architecture/StructuralIntelligence/BoundedWorkflows1.8.md @@ -0,0 +1,63 @@ +# 1.8 bounded structural workflows + +Implementation contract for the maintainer-authorized 1.8 feature branch from +published1.7.0 (8557bade). This is an additive implementation, not merge or release +approval. Original NAV12 construction fixtures, failures and qualification retain +their source-bound historical meaning; public promotion adds its own evidence. + +## Batch reads + +JSONInspectionBatch returns exact values for ordered typed paths. Snapshot values +accept ordered snapshot-local ordinals. Store values accept owner-bound positions. +Repeats are preserved and charged; empty requests still admit the complete source. +An invocation returns all results or throws, never publishes a partial result and +never substitutes null for an invalid selection. Explicit JSON null remains a value. + +Each batch creates one original native read/traversal scope. Profile/request work, +one complete canonical source/source-map admission, every selection and output +slot share its ledger. It does not loop over independently admitted public reads. +Request counts are inclusively capped (default1024); native path/work/retention/ +cancellation caps remain. Output JSON aliases follow existing requiredValue/read +semantics; subsequent caller copies/work are outside library-owned quotas. Logical +slot accounting is conservative policy, not heap/RSS or throughput measurement. + +## Immutable public configured store + +Promote the existing typed closed construction profile and checked store facade +inside JSONInspection; no new product, parser, database, resolver or index engine. +References stay disabled unless explicitly configured. Strict markers retain exact +collection/id spelling; only declared fields are interpreted. Preserve all eight +construction phases, duplicate/error precedence, first/second locations, source +spans, source-local owner tokens and the original one-ledger preparation/admission. +The twelve existing construction ceilings remain finite policies, not calibrated +joint capacity guarantees; the49 core metrics and all fixtures remain unchanged. + +Fixed record/collection/reference metadata access is trusted bounded-memory access. +Lookup, value batches and follow are separately reported operations, independent +of construction. Each admits the captured source once and reserves its immutable +metadata aliases; helpers cannot reset budgets or attach another source. Lookup +uses decoded scalar names/IDs, returns nil for a missing ID in a known collection, +and throws unknownCollection for an undeclared collection. Position construction +and the fully admitted store initializer remain inaccessible to external callers. + +Follow is deterministic breadth-first over configured link slots in field order, +including its starting position exactly once. Absent/null slots remain distinct +metadata and contribute no edge. Cycles use visited physical positions, not string +normalization or recursive callbacks. Depth, frontier, visited and result ceilings +are inclusive and finite. An unvisited reachable node exceeding any ceiling throws; +no depth-truncated or partially successful closure is returned. Owner mismatch is +an error. This operation performs no I/O and creates no persistent identity. + +## Compatibility and qualification + +Existing singular reads/traversal/graph/Transform semantics and public signatures +remain unchanged. A caller pipeline is not implicitly one global ledger. New +source needs actual Debug/Release, public consumer and compiler-boundary evidence, +cancellation/late-failure/resource cases, supported safety/platform lanes and DocC. +Update exact discovery identities/cardinalities rather than relaxing them. Reuse +original independent fixtures and49 caps; add explicit new batch/follow policy +cases. Keep partial profiling/G4 HOLD and the1.7-only P4 exception historical; +1.8 performance is unqualified until measured or explicitly dispositioned. + +No persistence, public mutable store, online lookup, new query grammar, lower +platform floor, parallel batch execution or wider retrieval belongs to this wave. diff --git a/Documentation/Architecture/StructuralIntelligence/SOURCE-AUTHORITY.md b/Documentation/Architecture/StructuralIntelligence/SOURCE-AUTHORITY.md index a7011ae..5d1582f 100644 --- a/Documentation/Architecture/StructuralIntelligence/SOURCE-AUTHORITY.md +++ b/Documentation/Architecture/StructuralIntelligence/SOURCE-AUTHORITY.md @@ -1,3 +1,12 @@ +# 1.8 source authority — 28 October 2026 + +The 1.8 candidate starts at published 1.7.0 commit +`8557badef562cf2f059e964c113d4a7f60ecbe6c`, tree +`c914b8914c9130a3a07a3ed82fb41dde33c37402`. The maintainer commissioned +bounded batches and the immutable public store; see [the contract](BoundedWorkflows1.8.md). +Earlier private-only/pending status remains truthful for the dated sources below. +Qualification and merge/release acceptance of this successor remain separate. + # Structural source authority ## Consolidated development edition — 25 October 2026 diff --git a/Documentation/Qualification/Release1.8.md b/Documentation/Qualification/Release1.8.md new file mode 100644 index 0000000..9768c20 --- /dev/null +++ b/Documentation/Qualification/Release1.8.md @@ -0,0 +1,96 @@ +# 1.8 bounded structural workflow qualification + +Status: implemented local milestone, broader release qualification PARTIAL, on +isolated `codex/1.8-batches-and-store`. +No merge/tag/release/publication is commissioned or claimed. Source base is +published 1.7.0 `8557badef562cf2f059e964c113d4a7f60ecbe6c`, tree +`c914b8914c9130a3a07a3ed82fb41dde33c37402`. + +The [contract](../Architecture/StructuralIntelligence/BoundedWorkflows1.8.md) +and [compiled public example](../../Examples/BoundedWorkflows) define this additive +wave. Twelve library products plus CLI, thirteen catalogs, empty root dependencies, +Swift6.4/language6 and Apple27 minima remain. Original NAV12 fixtures and49 metric +policies are unchanged. Current discovery adds24 named nonparameterized behavior +functions and one shared-state function to JSONInspection (214 roots,11 shared). +Original390 native/portable and353 WASI parameter invocations are unchanged. + +## Milestone evidence + +Private raw receipts stay in ignored `Artifacts/Release1.8` in the feature worktree. +Initial compiler reconciliation passed `swift build --target JSONInspection --jobs 2` +and20 then22 targeted native behavior functions. Those were unfrozen-source probes, +not final qualification. The independent sibling review requested native path-limit +admission on store reads and deterministic inner-loop cancellation. Both are +addressed in source; the final24 cases also pin nested snapshot scratch cleanup, +metadata aliases, self-cycles and narrow chains. Final gates must record exact +source manifests/commands/environments/events; no index entry implies a pass. + +## Completed local milestone — 28 October 2026 + +Apple Swift6.4.2, Xcode27.2, macOS/SDK27 supported native preflight passed. +`Scripts/ci.py --lane structural-workflows` passed:24 exact new functions in +Debug and Release, original12 store functions with65 native-build parameter +invocations plus the separate strict-parser case in each configuration, preserved +26-command store compiler/consumer receipt, and12 public SwiftPM witnesses per +configuration. The new public qualifier also compiles the DocC example and six +ordinary public-boundary refusals per configuration. Policy/compile-only vectors +remain distinct from native evaluated outcomes. + +`--lane structural-workflows-safety` passed24 ASan functions, one shared-state +TSan function and strict-memory compilation. Supplemental exact-event regression +passed all57 Navigation and214 Inspection functions in Debug/Release. The existing +CLI version case passed. The harness passed28 checks; site regressions passed12. +The affected Inspection DocC catalog compiled with warnings as errors. Compiled +public graphs contain301 authored Inspection declarations (139 added, none removed) +and56 Navigation declarations (unchanged), all documented. Three legacy Self versus +concrete-owner graph renderings were explicitly compared against unchanged owning +source files; no byte-identical graph claim is made. Full13-module extraction remains +pending. The independent sibling source review closed its path-limit/cancellation +findings; this is scoped agent review, not human release approval. + +Preserve the original failed external-consumer attempt: its fixture placed a null +field outside the configured collections object and was correctly refused as an +unconfigured collection. The harness trapped on that uncaught error; it was not a +library/compiler crash. The successor scopes collections explicitly and passes. +Original attempt logs and source manifests remain in private evidence. + +This status update changes only this qualification document after the passing +runtime receipts. Final checkpoint evidence must show that all product, test, +fixture, DocC and harness bytes match those receipts, and record the documentation +successor separately. No broader qualification pass follows from these local results. + +## Required gates + +- Scoped Debug/Release: `Scripts/ci.py --lane structural-workflows`, including + original12 store functions/65 vectors, ordinary compiler refusals, public + SwiftPM consumer and compiled DocC example. +- Scoped safety: `--lane structural-workflows-safety` (ASan24, TSan1, strict memory). +- Complete native, safety and docs lanes, current public API graphs and no + removed public declarations; preserve Navigation's original56 and Transform42. +- ARM64/x86_64 Linux, supported WASI and Apple SDK/runtime scopes on the final + source. Unavailable environments/storage remain blocked, not inherited passes. +- Clean source inventory, deterministic archive and Git-free external consumer. +- Measured capacity/performance before any improvement claim. Original1.7 P4 + waiver, loaded-host timing, partial Instruments profiles, streaming disposition + and graph G4 HOLD remain historical and cannot qualify changed1.8 source. + +The maintainer chose to continue implementation without reclaiming other +sessions' files. At kickoff about1.4GiB was free; fresh broad safety/docs/platform +builds may remain storage-blocked. Record actual attempts and unrun commands in +the private milestone report. No cleanup, weaker check or gate waiver is implied. + +## Read-only store decision + +The public surface promotes typed configuration, immutable admitted store/metadata, +owner-bound positions and typed failures. New operations are exact-ID lookup, +ordered value reads and complete finite BFS follow. Construction uses the original +eight phases, one exclusive ledger, duplicate timing and marker/target precedence. +The prior compiler-negative asserting profile invisibility is superseded by an +external private-read-scope refusal; the other original seven negatives and all +65 original vectors remain. Store/position forging is still forbidden. This is +an intentional additive promotion; it adds no product or runtime dependency. + +Fixed metadata access and supplied callbacks are trusted caller work. Each later +query is independent of construction and other queries. Logical retained bytes +are conservative owned reservations, not process memory. No persistence, mutable +store, online resolution, asynchronous batch scheduler or durable token exists. diff --git a/Documentation/Qualification/ReleaseLines.md b/Documentation/Qualification/ReleaseLines.md index 40adade..c575737 100644 --- a/Documentation/Qualification/ReleaseLines.md +++ b/Documentation/Qualification/ReleaseLines.md @@ -1,3 +1,13 @@ +# Current source lines — 28 October 2026 + +Latest published stable is [1.7.0](https://github.com/ThorFuchs/Swift-JSON/releases/tag/1.7.0), +commit `8557badef562cf2f059e964c113d4a7f60ecbe6c`, tree +`c914b8914c9130a3a07a3ed82fb41dde33c37402`: twelve libraries plus CLI, +thirteen catalogs, 1,500 authored declarations. The isolated 1.8.0-dev successor +keeps those products and adds bounded Inspection batches/store operations. +See [its separate qualification](Release1.8.md). 1.6.1 stays the eight-library +maintenance line. The dated preparation inventories below remain history. + # Stable and development source lines | Line | Source identity | Public scope | diff --git a/Documentation/Qualification/TestDiscovery.json b/Documentation/Qualification/TestDiscovery.json index ee42d25..646ae2f 100644 --- a/Documentation/Qualification/TestDiscovery.json +++ b/Documentation/Qualification/TestDiscovery.json @@ -12,7 +12,7 @@ "JSONCLITests": 88, "JSONNavigationTests": 57, "JSONSchemaExamplesTests": 98, - "JSONInspectionTests": 190, + "JSONInspectionTests": 214, "JSONTransformTests": 23, "JSONDSLTests": 47 }, @@ -37,7 +37,7 @@ "JSONCLITests": 1, "JSONNavigationTests": 2, "JSONSchemaExamplesTests": 4, - "JSONInspectionTests": 10, + "JSONInspectionTests": 11, "JSONTransformTests": 1, "JSONDSLTests": 1 }, @@ -84,7 +84,7 @@ "JSONCLITests": 88, "JSONNavigationTests": 57, "JSONSchemaExamplesTests": 98, - "JSONInspectionTests": 190, + "JSONInspectionTests": 214, "JSONTransformTests": 23, "JSONDSLTests": 47 }, @@ -99,7 +99,7 @@ "JSONBridgesTests": 10, "JSONNavigationTests": 57, "JSONSchemaExamplesTests": 98, - "JSONInspectionTests": 190, + "JSONInspectionTests": 214, "JSONTransformTests": 23, "JSONDSLTests": 47 }, @@ -115,7 +115,7 @@ "JSONCLITests": 1, "JSONNavigationTests": 2, "JSONSchemaExamplesTests": 4, - "JSONInspectionTests": 10, + "JSONInspectionTests": 11, "JSONTransformTests": 1, "JSONDSLTests": 1 }, @@ -148,5 +148,31 @@ "JSONInspectionTests.storeR08DeclarationAndPhysicalPermutationMetamorphisms()", "JSONInspectionTests.storeR09CompleteRowsDeepSourcesAndZeroCapacity()", "JSONInspectionTests.storeR10NoInferenceOrModelMutationAndExactPayloads()" + ], + "workflow_function_ids": [ + "JSONInspectionTests.workflowB01PathsPreserveOrderRepeatsNullAndExactScalars()", + "JSONInspectionTests.workflowB02OneCompleteAdmissionAndEmptyBatch()", + "JSONInspectionTests.workflowB03RequestCapacityConfigurationAndExactBoundary()", + "JSONInspectionTests.workflowB04MissingAndWrongKindRetainTypedPaths()", + "JSONInspectionTests.workflowB05UnselectedSuffixAndCumulativeWorkCannotReset()", + "JSONInspectionTests.workflowB06SnapshotBatchKeepsOneSourceAndOrdinalMeaning()", + "JSONInspectionTests.workflowB07InnerCancellationInvalidatesOriginalScope()", + "JSONInspectionTests.workflowB08RetentionBoundaryIsFiniteAndSourceIsUnchanged()", + "JSONInspectionTests.workflowS01PublicStorePreservesPhysicalAndDeclaredOrder()", + "JSONInspectionTests.workflowS02LookupUsesDecodedScalarIDsAndDistinguishesMissing()", + "JSONInspectionTests.workflowS03OwnerTokensCannotRebindAnotherInstance()", + "JSONInspectionTests.workflowS04StoreBatchValuesPreserveOrderExactnessAndOneAdmission()", + "JSONInspectionTests.workflowS05ReferenceSlotsKeepAbsentNullAndLinkDistinct()", + "JSONInspectionTests.workflowS06FollowCompletesCyclesAndPreservesBFSOrder()", + "JSONInspectionTests.workflowS07FollowAllFourInclusiveBoundsThrowInsteadOfTruncate()", + "JSONInspectionTests.workflowS08FollowAndLookupHaveCumulativeQuotaAndNoSecondAdmission()", + "JSONInspectionTests.workflowS09FollowInnerCancellationAndInvalidation()", + "JSONInspectionTests.workflowS10SourceReplacementCannotInventSpansOrRebindPositions()", + "JSONInspectionTests.workflowS11DisabledReferencesRemainOrdinaryData()", + "JSONInspectionTests.workflowSharedImmutablePlansHaveIndependentBatchAndStoreHealth()", + "JSONInspectionTests.workflowS12StorePathsHonorInvocationPointerAndDiagnosticLimits()", + "JSONInspectionTests.workflowS13FollowSelfCycleAndLongNarrowChain()", + "JSONInspectionTests.workflowB09NestedSnapshotRepeatsQuotasAndLateCancellation()", + "JSONInspectionTests.workflowS14MetadataAliasesAndScratchHaveExactRetentionDisposition()" ] } diff --git a/Documentation/README.md b/Documentation/README.md index 8392303..f280c1a 100644 --- a/Documentation/README.md +++ b/Documentation/README.md @@ -1,3 +1,10 @@ +# Current authority — 1.8 candidate + +Published 1.7.0 has twelve libraries plus CLI and thirteen catalogs. This isolated +1.8.0-dev branch adds bounded batches and a public immutable store inside +JSONInspection. See [current qualification](Qualification/Release1.8.md). +The dated predecessor inventories below are historical, not 1.8 pass evidence. + # Current documentation authority Unreleased 1.7.0-dev consolidation has twelve libraries plus CLI and thirteen DocC diff --git a/Documentation/ReleaseRoadmap.md b/Documentation/ReleaseRoadmap.md index 550bb6c..1e7bd96 100644 --- a/Documentation/ReleaseRoadmap.md +++ b/Documentation/ReleaseRoadmap.md @@ -1,3 +1,13 @@ +# Execution update — 28 October 2026 + +1.7.0 is published at `8557badef562cf2f059e964c113d4a7f60ecbe6c`. +The maintainer has now commissioned the 1.8 bounded structural wave, including +the public read-only store, on isolated `codex/1.8-batches-and-store`. +[The implementation contract](Architecture/StructuralIntelligence/BoundedWorkflows1.8.md) +freezes this wave; [qualification](Qualification/Release1.8.md) records actual results. +The proposal and historical inventories below remain source-bound. Persistence, +platform-floor reduction and new query grammars are not included. + # Release roadmap: 1.7 through 2.0 Maintainer working proposal, audited 27 October 2026. The next step is a release diff --git a/Examples/BoundedWorkflows/Package.swift b/Examples/BoundedWorkflows/Package.swift new file mode 100644 index 0000000..62eaba2 --- /dev/null +++ b/Examples/BoundedWorkflows/Package.swift @@ -0,0 +1,13 @@ +// swift-tools-version: 6.4 +// SPDX-License-Identifier: Apache-2.0 +import PackageDescription +let package = Package( + name: "BoundedWorkflows", + platforms: [.macOS("27.0"), .iOS("27.0")], + dependencies: [.package(name: "Swift-JSON", path: "../..")], + targets: [.executableTarget(name: "BoundedWorkflows", dependencies: [ + .product(name: "JSON", package: "Swift-JSON"), + .product(name: "JSONNavigation", package: "Swift-JSON"), + .product(name: "JSONInspection", package: "Swift-JSON") + ])], swiftLanguageModes: [.v6] +) diff --git a/Examples/BoundedWorkflows/Sources/BoundedWorkflows/Main.swift b/Examples/BoundedWorkflows/Sources/BoundedWorkflows/Main.swift new file mode 100644 index 0000000..0d436bb --- /dev/null +++ b/Examples/BoundedWorkflows/Sources/BoundedWorkflows/Main.swift @@ -0,0 +1,50 @@ +// SPDX-License-Identifier: Apache-2.0 +import Foundation +import JSON +import JSONNavigation +import JSONInspection + +@main struct BoundedWorkflows { + static func main() throws { + func witness(_ id: String, _ success: Bool) { + precondition(success, id); print("WORKFLOW_CASE \(id) passed") + } + let document = try JSONParser().parseDocument(Data(#"{"collections":{"u":[{"id":"é","amount":9007199254740993,"next":{"collection":"u","id":"e\u0301"}},{"id":"e\u0301","next":{"collection":"u","id":"é"}}]},"nothing":null}"#.utf8)) + let paths: [[JSONLocationComponent]] = [[.key("collections"), .key("u"), .index(0), .key("amount")], [.key("nothing")], [.key("nothing")]] + let batch = try JSONInspectionBatch(document: document).values(at: paths) + witness("batch-exact-order", batch.result == [.number(try JSONNumber(parsing: "9007199254740993")), .null, .null]) + do { _ = try JSONInspectionBatch(document: document).values(at: [[],[.key("missing")]]); preconditionFailure() } + catch let error as JSONAccessFailure { witness("missing-is-error", error.cause.code == .missingMember && error.path == [.key("missing")]) } + do { _ = try JSONInspectionBatch(document: document).values(at: paths, maximumRequests: 2); preconditionFailure() } + catch let error as JSONInspectionBatchFailure { witness("batch-capacity", error == .requestLimitExceeded(limit: 2, actual: 3)) } + let snapshot = try JSONStructuralSnapshotBuilder(document: document).build().result + witness("snapshot-root", try snapshot.values(for: [0,0]).result == [document.root, document.root]) + let profile = JSONStoreProfile(version: .v1, collectionsAt: [.key("collections")], collections: [ + .init(name: "u", recordsAt: [.key("u")], idKey: "id", referenceFields: [ + .init(key: "next", presence: .required, nullable: false)])], references: .strict(marker: .v1, validateTargets: true)) + let built = try JSONStoreBuilder(document: document, profile: profile).build(), store = built.result + witness("checked-construction", store.recordCount == 2 && store.references.count == 2 && built.statistics[.nodes] > 0) + guard let a = try store.lookup(collection: "u", id: "é").result, + let b = try store.lookup(collection: "u", id: "e\u{301}").result else { preconditionFailure() } + witness("scalar-identity", a != b && a.row == 0 && b.row == 1) + witness("missing-id", try store.lookup(collection: "u", id: "absent").result == nil) + let selected = try store.values(at: [b,a,b]) + witness("store-batch", try selected.result[0].requiredMember("id") == .string("e\u{301}") && selected.result[0] == selected.result[2]) + witness("complete-cycle", try store.follow(from: a).result == [a,b]) + let another = try JSONStoreBuilder(document: document, profile: profile).build().result + do { _ = try another.value(at: a); preconditionFailure() } + catch let error as JSONStoreFailure { witness("owner-refusal", error.code == .invalidPosition) } + var bounds = JSONStoreFollowLimits(); bounds.maximumResults = 1 + do { _ = try store.follow(from: a, bounds: bounds); preconditionFailure() } + catch let error as JSONStoreFailure { witness("complete-or-error", error.code == .capacityExceeded && error.capacity == "maximumResults") } + let measured = try store.follow(from: a) + var limits = JSONLimits.balanced; limits[.workUnits] = measured.statistics[.workUnits] + let equal = try store.follow(from: a, limits: limits) + limits[.workUnits] -= 1 + do { _ = try store.follow(from: a, limits: limits); preconditionFailure() } + catch let error as JSONFailure { + witness("cumulative-report", equal.result == [a,b] && equal.statistics[.workUnits] == measured.statistics[.workUnits] && error.resource?.metric == .workUnits) + } + print("BOUNDED_WORKFLOWS passed cases=12 work=\(measured.statistics[.workUnits]) logical_retained=\(measured.statistics[.retainedBytes])") + } +} diff --git a/README.md b/README.md index 4137295..cd0e3d6 100644 --- a/README.md +++ b/README.md @@ -1,17 +1,15 @@ # Swift-JSON -This is **Swift-JSON 1.7.0 source prepared for release**. Version metadata identifies -the source; the [official releases](https://github.com/ThorFuchs/Swift-JSON/releases) -establish publication. At preparation, latest stable remains **1.6.1**. No tag or -publication is implied by this checkout. The source combines exact Navigation and -Inspection projections/graphs, JSONTransform and optional JSONValueDSL/JSONSchemaDSL. -It has twelve library products plus the CLI and thirteen DocC catalogs including -the shared, non-product JSONDSLCore reference. - -See the [combined release checklist](Documentation/Qualification/Release1.7Preparation.md), -[release notes](Documentation/ReleaseNotes/1.7.0.md) and -[1.7–2.0 roadmap](Documentation/ReleaseRoadmap.md). Every final candidate claim -requires source-bound receipts; predecessor green checks alone do not qualify it. +This is **Swift-JSON 1.8.0-dev**, based on the published +[1.7.0 release](https://github.com/ThorFuchs/Swift-JSON/releases/tag/1.7.0) +(`8557badef562cf2f059e964c113d4a7f60ecbe6c`). The next additive wave brings +ordered batch reads and an immutable configured read-only store to JSONInspection. +It is under qualification; it is not an admitted 1.8 release. +Twelve libraries plus CLI, thirteen DocC catalogs and empty root dependencies remain. +See the [1.8 implementation contract](Documentation/Architecture/StructuralIntelligence/BoundedWorkflows1.8.md), +[qualification record](Documentation/Qualification/Release1.8.md) and +[release roadmap](Documentation/ReleaseRoadmap.md). Historical records below retain +their original source and publication dates. See the [release-line matrix](Documentation/Qualification/ReleaseLines.md) for exact source scopes and publication boundaries. Version metadata identifies source; the @@ -34,24 +32,15 @@ Maintained by [Thorsten Fuchs](https://github.com/ThorFuchs). Swift-JSON follows semantic versioning. Use a released version and retain `Package.resolved` for reproducible builds; `main` tracks ongoing development. -**1.6.1** is the latest stable maintenance release: eight libraries plus CLI, -eight DocC catalogs and the unchanged 1.6 API/runtime. The optional verified schema -examples, structural traversal, structured HTML previews and bounded composition -were introduced in 1.6.0. Exact JSON and offline schema validation remain the -semantic authorities. -The published 1.6.1 CLI reports `1.6.1`; this prepared source reports `1.7.0`. -Both use a deterministic source-content digest. -The [published 1.6.1 release](https://github.com/ThorFuchs/Swift-JSON/releases/tag/1.6.1) -is pinned to `e431f157ececc0cf3d061192420b0e36a9062f83`. Use its matching eight-catalog -DocC and qualification assets for the stable reference; see the -[maintenance qualification notes](Documentation/Qualification/Release1.6.1.md). -Historical N0/N1 plans and later candidates are distinguished in the -[source authority map](Documentation/Architecture/StructuralIntelligence/SOURCE-AUTHORITY.md). - -This development source includes opt-in Inspection, Transform and authored DSLs: -twelve libraries plus the CLI, thirteen products in total. Inspection, Transform and DSLs are absent from stable 1.6 releases. Pin a reviewed -unreleased source commit to use those products; the compatible stable installation -requirement below does not supply them. Integration into main does not publish 1.7.0. +**1.7.0** is the latest stable release: twelve libraries plus CLI and thirteen +DocC catalogs. Inspection, Transform and the authored DSL products are included. +The 1.6.1 maintenance line retains its eight-library API separately. Published +1.7.0 reports `1.7.0`; this feature source reports `1.8.0-dev`. Both expose a +source-content digest distinct from the Git commit. Use exact version pins and +matching release assets when reproducing a published edition. + +The new batch/store APIs require the 1.8 feature source. No package product, +dependency, platform minimum or existing JSON/JSONSchema authority changes. The 1.5.1 maintenance repairs remain included: best-effort schema diagnostics, the complete 1.2.1 regression suites and Git-free qualification. See the @@ -752,7 +741,7 @@ can have restricted descendant traversal; structural rectangular width is not numeric matrix or schema admission. See the [compiled evidence guide](Sources/JSONInspection/Documentation.docc/ArrayEvidence.md), [public consumer](Tools/InspectionQualification) and [accounting contract](Documentation/Architecture/StructuralIntelligence/Inspection.md). -## Complete source-attached structural views (unreleased) +## Complete source-attached structural views (introduced in 1.7) Opt-in JSONInspection in this candidate combines array observations with a complete parent-linked snapshot, selected physical tree views and strict sparse @@ -767,7 +756,7 @@ Every construction has one cumulative finite ledger. Each later catalogue/cell/ location/value/span lookup is an explicitly separate full-source-admitted operation. Ordinals are snapshot-local trusted positions, not durable identity. Logical retention is not RSS; G4 controlled performance remains HOLD, with no speed claim. -These additions are absent from published 1.6.1 and do not publish 1.7.0. +These additions were absent from published 1.6.1 and are included in published 1.7.0. The current development branch also contains package-private structural set and exact numeric matrix experiments. They add no public API; see the @@ -782,7 +771,7 @@ Every later member/cell read independently admits the whole captured source with limits and an operation report. These are structural views; no set algebra, matrix arithmetic or performance claim is included. See the [compiled public guide](Sources/JSONInspection/Documentation.docc/ExactProjections.md), [independent consumer](Tools/ExactProjectionQualification) and [public boundary](Documentation/Architecture/StructuralIntelligence/NAV10/PublicAPI.md). -Batch access remains explicitly deferred to NAV13. Stable 1.6.1 is unchanged. +The 1.8 candidate adds aggregate value reads; set/matrix algebra remains outside scope. Stable 1.6.1 is unchanged. NAV11 experiments with a package-only, exact source-backed directed simple graph inside JSONInspection. Graph construction and BFS are not public API and are not @@ -821,3 +810,26 @@ for source-bound gates, the explicit allocation-instrumentation exception and platform limits. A path or revision consumer is preparation evidence; exact 1.7.0 tag consumption is a postauthorization check. This product is absent from stable 1.6.1 and must pass qualification on its final versioned source before publication. + +## Bounded structural workflows (1.8 candidate) + +Use `JSONInspectionBatch.values(at:maximumRequests:)` for ordered typed paths, +`JSONStructuralSnapshot.values(for:maximumRequests:limits:)` for snapshot ordinals, +and `JSONConfiguredStore.values(at:maximumRequests:limits:)` for record positions. +Each complete batch admits the source once and shares one finite ledger, preserves +request order and repeats, and either returns every result or throws. Missing is +an error; explicit JSON null remains a value. Empty batches still admit the source. + +Build a read-only store with `JSONStoreBuilder` and an explicit `JSONStoreProfile`. +Collection paths, ID fields and reference fields are configured; exact scalar IDs +are never normalized. `lookup(collection:id:limits:)` distinguishes a missing ID +from an unknown collection. `follow(from:bounds:limits:)` returns a complete BFS +closure, with finite depth/frontier/visited/result caps and no truncated success. +Positions are local to one immutable store and cannot be forged or rebound. + +Run the [public SwiftPM example](Examples/BoundedWorkflows) with +`swift run --package-path Examples/BoundedWorkflows BoundedWorkflows`. +The [DocC guide](Sources/JSONInspection/Documentation.docc/BoundedWorkflows.md) +explains separate operation reports, caller work and logical retention. This is +an additive candidate API; no persistence, network resolver, new platform or +performance improvement is claimed. diff --git a/Scripts/build_documentation_site.py b/Scripts/build_documentation_site.py index d17ad71..9bad227 100644 --- a/Scripts/build_documentation_site.py +++ b/Scripts/build_documentation_site.py @@ -20,8 +20,8 @@ REPOSITORY = 'https://github.com/ThorFuchs/Swift-JSON' BASE_PATH = '/Swift-JSON' SITE_URL = 'https://thorfuchs.github.io' + BASE_PATH + '/' -SOURCE_VERSION = '1.7.0' -STABLE_VERSION = '1.6.1' +SOURCE_VERSION = '1.8.0-dev' +STABLE_VERSION = '1.7.0' GUIDES = {'JSON': ('Exact values and native coding', 'exactvaluesandcoding'), 'JSONSchema': ('Offline schema validation', 'offlinevalidation'), 'JSONSchemaExamples': ('Verified exact candidates', 'verifiedcandidates'), @@ -115,7 +115,7 @@ def landing(commit):

Exact JSON. Clear contracts.

Keep decimal values exact, Unicode scalar keys distinct and resource use explicit. Explore {len(PRODUCTS)} Swift libraries that carry those semantics from input bytes to validated, transformed and encoded output.

Swift 6.4+ · Swift language mode 6 · macOS/SDK 27+ · No package dependencies

-

Development main includes JSONInspection, JSONTransform and the optional authored DSLs; this source is prepared as 1.7.0. Stable {STABLE_VERSION} has eight library catalogs; use its matching compiled DocC assets or tagged source documentation.

+

Development main includes JSONInspection, JSONTransform and the optional authored DSLs; this source is unreleased 1.8.0-dev with bounded batch/store work. Stable {STABLE_VERSION} has twelve library catalogs plus the shared JSONDSLCore reference; use its matching compiled DocC assets or tagged source documentation.

{''.join(cards)}

Start with the contract

Read the package overview and executable examples, then choose a reference above. The behavior contracts explain exact numbers, scalar identity, offline schema evaluation, cumulative budgets and stream completion.

diff --git a/Scripts/ci.py b/Scripts/ci.py index 707114f..0b7cef7 100644 --- a/Scripts/ci.py +++ b/Scripts/ci.py @@ -8,6 +8,7 @@ from p2_acceptance import check_p2_cases, check_borrow_receipt, check_private_receipt, check_exact_cases, check_exact_receipt from graph_acceptance import check_graph_cases, check_graph_receipt, check_public_graph_receipt from store_acceptance import check_store_tests, check_store_consumer +from qualify_structural_workflows import check_receipt as check_workflow_consumer from projection_acceptance import check_projection_cases, check_public_projection_receipt from test_discovery import DISCOVERY, ROOT_COUNTS, SHARED_COUNTS, PARAMETER_INVOCATIONS, check_restored @@ -26,6 +27,7 @@ SHARED_TESTS += '|graphG30ConcurrentViewsHaveIndependentHealth|graphGP14ConcurrentQueries' SHARED_TESTS += '|dslSharedFinalizationAndPlansAreInvocationOwned' SHARED_TESTS += '|storeR06TerminalQuotasNeverPublishAndImmutableConcurrentHealth' +SHARED_TESTS += '|workflowSharedImmutablePlansHaveIndependentBatchAndStoreHealth' R0_TESTS = 'readerProbe|indexProbes|duplicateRecognitionPrecedesValueAndClosingSyntax' def snapshot(): @@ -34,6 +36,26 @@ def snapshot(): def commands(lane, output): # The qualification executable resolves output paths from its repository root. output = output.relative_to(ROOT) + if lane in ('structural-workflows', 'structural-workflows-safety'): + if lane.endswith('safety'): + result = [ + ['swift', 'test', '--test-product', 'JSONInspectionTests', '-c', 'release', '--jobs', '2', '--sanitize', 'address', '--scratch-path', '.build/workflow-address', '--filter', 'workflow'], + ['swift', 'test', '--test-product', 'JSONInspectionTests', '-c', 'release', '--jobs', '2', '--sanitize', 'thread', '--scratch-path', '.build/workflow-thread', '--filter', 'workflowSharedImmutablePlansHaveIndependentBatchAndStoreHealth'], + ['swift', 'build', '--target', 'JSONInspection', '--jobs', '2', '--scratch-path', '.build/workflow-strict', '-Xswiftc', '-strict-memory-safety', '-Xswiftc', '-warnings-as-errors'], + ] + else: + result = [ + ['python3', '-B', 'Scripts/public_check.py'], + ['python3', '-B', 'Scripts/generate_cli_identity.py', '--check'], + ['python3', '-B', 'Scripts/test_qualification_harness.py'], + ['swift', 'test', '--test-product', 'JSONInspectionTests', '--jobs', '2', '--filter', 'workflow'], + ['swift', 'test', '--test-product', 'JSONInspectionTests', '-c', 'release', '--jobs', '2', '--filter', 'workflow'], + ['swift', 'test', '--test-product', 'JSONInspectionTests', '--jobs', '2', '--filter', 'store'], + ['swift', 'test', '--test-product', 'JSONInspectionTests', '-c', 'release', '--jobs', '2', '--filter', 'store'], + ['python3', '-B', 'Scripts/qualify_store.py', '--output', str(output / 'store-boundaries')], + ['python3', '-B', 'Scripts/qualify_structural_workflows.py', '--output', str(output / 'public-consumer')], + ] + return [c + ['--disable-xctest'] if sys.platform == 'linux' and c[:2] == ['swift', 'test'] else c for c in result] if lane == 'store': return [ ['python3', '-B', 'Scripts/public_check.py'], @@ -58,9 +80,9 @@ def commands(lane, output): ] if lane == 'inspection': result = [['python3', '-B', 'Scripts/public_check.py'], - ['swift', 'test', '--filter', 'inspection|snapshot|projectionP|p2|exactProjectionPUB|graphG|store'], - ['swift', 'test', '-c', 'release', '--filter', 'inspection|snapshot|projectionP|p2|exactProjectionPUB|graphG|store'], - ['swift', 'test', '-c', 'release', '--sanitize', 'address', '--scratch-path', '.build/inspection-address', '--filter', 'inspection|snapshot|projectionP|p2|exactProjectionPUB|graphG|store'], + ['swift', 'test', '--filter', 'inspection|snapshot|projectionP|p2|exactProjectionPUB|graphG|store|workflow'], + ['swift', 'test', '-c', 'release', '--filter', 'inspection|snapshot|projectionP|p2|exactProjectionPUB|graphG|store|workflow'], + ['swift', 'test', '-c', 'release', '--sanitize', 'address', '--scratch-path', '.build/inspection-address', '--filter', 'inspection|snapshot|projectionP|p2|exactProjectionPUB|graphG|store|workflow'], ['swift', 'build', '--target', 'JSONInspection', '--scratch-path', '.build/inspection-strict', '-Xswiftc', '-strict-memory-safety', '-Xswiftc', '-warnings-as-errors'], ['python3', '-B', 'Scripts/qualify_navigation_reducer.py', '--output', str(output / 'owned-codegen')], @@ -69,6 +91,7 @@ def commands(lane, output): ['python3', '-B', 'Scripts/qualify_public_graph.py', '--output', str(output / 'public-graph-consumer')], ['python3', '-B', 'Scripts/qualify_store_preparation.py', '--output', str(output / 'prepared-codegen')], ['python3', '-B', 'Scripts/qualify_store.py', '--output', str(output / 'private-store-consumer')], + ['python3', '-B', 'Scripts/qualify_structural_workflows.py', '--output', str(output / 'public-workflow-consumer')], ['python3', '-B', 'Scripts/qualify_projections.py', '--output', str(output / 'projection-codegen')], ['python3', '-B', 'Scripts/verify_p2_fixtures.py'], ['python3', '-B', 'Scripts/qualify_p2_borrowing.py', '--output', str(output / 'p2-borrowing')], @@ -229,6 +252,7 @@ def commands(lane, output): ['python3', '-B', 'Scripts/qualify_public_graph.py', '--output', str(output / 'public-graph-consumer')], ['python3', '-B', 'Scripts/qualify_store_preparation.py', '--output', str(output / 'prepared-codegen')], ['python3', '-B', 'Scripts/qualify_store.py', '--output', str(output / 'private-store-consumer')], + ['python3', '-B', 'Scripts/qualify_structural_workflows.py', '--output', str(output / 'public-workflow-consumer')], ['swift', 'test', '--package-path', 'Tools/MobileQualification'], ['swift', 'run', '-c', 'release', '--package-path', 'Tools/MobileQualification', 'MobileQualification'], ['swift', 'test', '--package-path', 'Tools/MCPReadiness'], @@ -279,6 +303,7 @@ def check_result(command, code, log, events=None): if events is not None: counts = events['bundle_counts'] selected_filter = command[command.index('--filter') + 1] if '--filter' in command else None store_test = selected_filter in ('store', 'storeR06TerminalQuotasNeverPublishAndImmutableConcurrentHealth') + workflow_test = selected_filter in ('workflow', 'workflowSharedImmutablePlansHaveIndependentBatchAndStoreHealth') prepared_test = selected_filter in ('navigationPreparedD0', 'navigationPreparedD0IndependentConcurrentHealth') if command[:2] == ['swift', 'test']: expected = [DISCOVERY['composition_external_functions']] if 'Tools/CompositionQualification' in command else [12] if 'composition|composed|caughtOperationalFailureCannotComplete' in command else [79] if 'JSONSchemaTests' in command else [4 if 'thread' in command else 98] if 'JSONSchemaExamplesTests' in command else [88] if 'cli|sharedFilterPlan' in command else sorted(DISCOVERY['export_functions'].values()) if 'export|Export|apple' in command else [14] if 'Tools/WasmAudit' in command else sorted(DISCOVERY['qa_functions'].values()) if 'qa' in command else [7] if R0_TESTS in command else [3] if 'Tools/MCPReadiness' in command or 'Tools/MobileQualification' in command else [1] if 'Tools/Qualification' in command else sorted(SHARED_COUNTS.values()) if 'thread' in command else sorted(ROOT_COUNTS.values()) @@ -291,7 +316,7 @@ def check_result(command, code, log, events=None): expected = [DISCOVERY['root_functions']['JSONNavigationTests']] if events is not None: passed = passed and events['module_function_counts'] == {'JSONNavigationTests': expected[0]} and events['parameter_case_invocations'] == 0 - if any(f in command for f in ['inspection', 'inspection|snapshot|projectionP|p2|exactProjectionPUB|graphG|store']): + if any(f in command for f in ['inspection', 'inspection|snapshot|projectionP|p2|exactProjectionPUB|graphG|store|workflow']): expected = [DISCOVERY['root_functions']['JSONInspectionTests']] if events is not None: passed = passed and events['module_function_counts'] == {'JSONInspectionTests': expected[0]} and events['parameter_case_invocations'] == 65 and events['parameterized_functions'] == {'storeD1NativeVector(id:)': 65} and check_projection_cases(events) and check_p2_cases(events) and check_exact_cases(events) and check_graph_cases(events) and check_store_tests(events, log) @@ -306,12 +331,18 @@ def check_result(command, code, log, events=None): shared_store = any(x.startswith('storeR06') for x in command) expected = [1 if shared_store else 12] passed = passed and events is not None and check_store_tests(events, log, shared=shared_store, scoped=True) + if workflow_test: + expected_ids = set(DISCOVERY['workflow_function_ids']) + if selected_filter != 'workflow': + expected_ids = {x for x in expected_ids if 'SharedImmutablePlans' in x} + expected = [len(expected_ids)] + passed = passed and events is not None and events['module_function_counts'] == {'JSONInspectionTests': len(expected_ids)} and set(events['function_names']) == expected_ids and len(events['function_names']) == len(expected_ids) and events['parameter_case_invocations'] == 0 passed = passed and sorted(counts) == expected if 'Tools/CompositionQualification' in command and events is not None: passed = passed and events['module_function_counts'] == {'CompositionScaleTests': DISCOVERY['composition_external_functions']} and events['parameter_case_invocations'] == 0 if 'qa' in command and events is not None: passed = passed and events['module_function_counts'] == DISCOVERY['qa_functions'] and check_restored(events) - if 'thread' in command and 'JSONSchemaExamplesTests' not in command and not prepared_test and not store_test and events is not None: + if 'thread' in command and 'JSONSchemaExamplesTests' not in command and not prepared_test and not store_test and not workflow_test and events is not None: passed = passed and events['module_function_counts'] == SHARED_COUNTS and check_projection_cases(events, shared=True) and check_p2_cases(events, shared=True) and check_exact_cases(events, shared=True) and check_graph_cases(events, shared=True) and events['parameter_case_invocations'] == 0 and check_store_tests(events, log, shared=True) if '--filter' not in command and '--test-product' not in command and not any(x.startswith('Tools/') for x in command): if events is not None: @@ -388,6 +419,9 @@ def check_result(command, code, log, events=None): if command[:3] in (['python3', '-B', 'Scripts/qualify_graph_context.py'], ['python3', '-B', 'Scripts/qualify_graph.py']): data = json.loads((ROOT / command[command.index('--output') + 1] / 'receipt.json').read_text()) passed = passed and check_graph_receipt(data, context='qualify_graph_context.py' in command[2]) + if command[:3] == ['python3', '-B', 'Scripts/qualify_structural_workflows.py']: + data = json.loads((ROOT / command[command.index('--output') + 1] / 'receipt.json').read_text()) + passed = passed and check_workflow_consumer(data) if command[:3] == ['python3', '-B', 'Scripts/qualify_store.py']: data = json.loads((ROOT / command[command.index('--output') + 1] / 'receipt.json').read_text()) passed = passed and check_store_consumer(data) @@ -431,12 +465,12 @@ def check_result(command, code, log, events=None): if command[:3] == ['python3', '-B', 'Scripts/qualify_html_browser.py']: passed = passed and 'JSONHTML_BROWSER passed engines=3 cases=76 skips=0' in log.splitlines() if 'Scripts/test_qualification_harness.py' in command: - passed = passed and 'HARNESS_TESTS passed cases=26' in log.splitlines() + passed = passed and 'HARNESS_TESTS passed cases=28' in log.splitlines() return passed, counts def main(): parser = argparse.ArgumentParser(description=__doc__) - parser.add_argument('--lane', choices=['store', 'store-preparation', 'composition', 'navigation', 'inspection', 'native', 'safety', 'docs', 'r0', 'qa', 'linux', 'linux-safety', 'wasm', 'browser', 'export', 'cli', 'schema-examples', 'schema-examples-safety', 'schema-examples-determinism', 'schema-examples-construction', 'schema-examples-advanced', 'schema-examples-qualification', 'coverage', 'linux-coverage', 'linux-qa', 'wasm-qa', 'archive'], required=True) + parser.add_argument('--lane', choices=['structural-workflows', 'structural-workflows-safety', 'store', 'store-preparation', 'composition', 'navigation', 'inspection', 'native', 'safety', 'docs', 'r0', 'qa', 'linux', 'linux-safety', 'wasm', 'browser', 'export', 'cli', 'schema-examples', 'schema-examples-safety', 'schema-examples-determinism', 'schema-examples-construction', 'schema-examples-advanced', 'schema-examples-qualification', 'coverage', 'linux-coverage', 'linux-qa', 'wasm-qa', 'archive'], required=True) parser.add_argument('--output') parser.add_argument('--skip-host-measurements', action='store_true', help='For WASI runtime qualification only: record optional host timings as NOT RUN.') args = parser.parse_args() @@ -448,7 +482,7 @@ def main(): repository = subprocess.run(['git', 'rev-parse', '--show-toplevel'], cwd=ROOT, capture_output=True, text=True) is_checkout = repository.returncode == 0 and pathlib.Path(repository.stdout.strip()).resolve() == ROOT.resolve() commit = subprocess.run(['git', 'rev-parse', 'HEAD'], cwd=ROOT, capture_output=True, text=True) if is_checkout else None - environment = collect_linux() if args.lane.startswith('linux') or (args.lane in ('composition', 'inspection') and sys.platform == 'linux') else collect() + environment = collect_linux() if args.lane.startswith('linux') or (args.lane in ('composition', 'inspection', 'structural-workflows', 'structural-workflows-safety') and sys.platform == 'linux') else collect() receipt = {'schema_version': 1, 'lane': args.lane, 'status': 'blocked', 'environment': environment, 'started_utc': datetime.datetime.now(datetime.timezone.utc).isoformat(), 'git_commit': commit.stdout.strip() if commit is not None and commit.returncode == 0 else None, 'github_run_id': os.environ.get('GITHUB_RUN_ID'), 'source_inputs': inputs, 'source_manifest_sha256': hashlib.sha256(json.dumps(inputs, sort_keys=True).encode()).hexdigest(), 'commands': []} def save(): (output / 'receipt.json').write_text(json.dumps(receipt, indent=2) + '\n') diff --git a/Scripts/qualify_store.py b/Scripts/qualify_store.py index 950b137..fbca82e 100644 --- a/Scripts/qualify_store.py +++ b/Scripts/qualify_store.py @@ -31,7 +31,7 @@ 'D069':('func illegal() { _ = JSONStoreProfile(version:.v1,collectionsAt:[],collections:[],references:.disabled,guessID:true) }',"extra argument 'guessID' in call"), 'store-initializer':('func illegal() { _ = JSONConfiguredStore() }','initializer is inaccessible'), 'position-forge':('func illegal() { _ = JSONStorePosition() }','initializer is inaccessible'), - 'external-store':('func illegal(_ x: JSONStoreProfile) {}',"cannot find type 'JSONStoreProfile' in scope"), + 'external-scope':('func illegal(_ x: JSONReadScope) {}',"cannot find type 'JSONReadScope' in scope"), 'bound-member':('import JSONNavigation\nfunc illegal(_ c: inout JSONTraversalContext) { _ = c.member }',"cannot reference 'mutating' method as function value"), 'member-escape':('import JSONNavigation\nfunc illegal(_ c: inout JSONTraversalContext) -> () throws -> JSON? { { try c.member(.null,key:"x",at:[]) } }',"escaping closure captures 'inout' parameter"), } diff --git a/Scripts/qualify_structural_workflows.py b/Scripts/qualify_structural_workflows.py new file mode 100644 index 0000000..c6908ed --- /dev/null +++ b/Scripts/qualify_structural_workflows.py @@ -0,0 +1,77 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: Apache-2.0 +"""Real external SwiftPM consumer plus domain-specific public-boundary codegen.""" +import argparse, hashlib, json, pathlib, re, subprocess +from archive_candidate import files +ROOT = pathlib.Path(__file__).resolve().parents[1] +CASES = {'batch-exact-order', 'missing-is-error', 'batch-capacity', 'snapshot-root', + 'checked-construction', 'scalar-identity', 'missing-id', 'store-batch', + 'complete-cycle', 'owner-refusal', 'complete-or-error', 'cumulative-report'} +PROBES = { + 'position-forge': ('func invalid() { _ = JSONStorePosition() }', 'initializer is inaccessible'), + 'store-forge': ('func invalid() { _ = JSONConfiguredStore() }', 'initializer is inaccessible'), + 'private-context': ('func invalid(_ context: JSONTraversalContext) {}', "cannot find type 'JSONTraversalContext' in scope"), + 'version': ('func invalid() { _ = JSONStoreProfileVersion.v2 }', "has no member 'v2'"), + 'nullable': ('func invalid() { _ = JSONStoreReferenceField(key:"f",presence:.required,nullable:"yes") }', "cannot convert value of type 'String' to expected argument type 'Bool'"), + 'mutation': ('func invalid(_ store: JSONConfiguredStore) { store.document = JSONDocument(root:.null) }', "'document' is a 'let' constant"), +} +def check_receipt(data): + configs = {'debug', 'release'} + expected_names = {f'{c}-{name}' for c in configs for name in ['consumer','binpath','guide',*PROBES]} + commands = data.get('commands', []) + return (data.get('status') == 'passed' and len(commands) == len(expected_names) + and {c['name'] for c in commands} == expected_names + and all(c.get('passed') and c.get('compiler_crash') is False and + (c['exit'] > 0 and any(c['expected'].lower() in d.lower() for d in c['primary_diagnostics']) + if c.get('expected') else c['exit'] == 0) for c in commands) + and set(data.get('consumer_results', {})) == configs + and all(set(x['cases']) == CASES and len(x['cases']) == len(CASES) + and x['work_units'] > 0 and x['logical_retained_bytes'] > 0 + for x in data['consumer_results'].values())) +def main(): + parser = argparse.ArgumentParser(); parser.add_argument('--output', required=True) + args = parser.parse_args(); out = ROOT / args.output; out.mkdir(parents=True, exist_ok=True) + receipt = {'status':'running', 'source_inputs':[ + {'path':p.relative_to(ROOT).as_posix(),'sha256':hashlib.sha256(p.read_bytes()).hexdigest()} for p in files()], + 'commands':[], 'consumer_results':{}} + def save(): (out/'receipt.json').write_text(json.dumps(receipt, indent=2)+'\n') + def run(name, command, expected=None): + result = subprocess.run(command, cwd=ROOT, capture_output=True, text=True, timeout=1200) + log = result.stdout + result.stderr + signal = result.returncode < 0 + crash = (signal and command[0] == 'swiftc') or any(x in log for x in ['Stack dump:', 'Assertion failed', 'Found ownership error', 'fatal error encountered during compilation']) + diagnostics = re.findall(r': error: ([^\n]+)', log) + passed = not signal and not crash and (result.returncode == 0 if expected is None else result.returncode > 0 and any(expected.lower() in d.lower() for d in diagnostics)) + path = out/(name+'.log'); path.write_text(log) + receipt['commands'].append(dict(name=name, command=command, exit=result.returncode, passed=passed, + compiler_crash=crash, signal_termination=signal, expected=expected, primary_diagnostics=diagnostics, + log=str(path.relative_to(ROOT)), log_sha256=hashlib.sha256(path.read_bytes()).hexdigest())) + save(); print(name, result.returncode, 'PASS' if passed else 'FAIL', flush=True) + if not passed: raise ValueError(name) + return result.stdout + try: + for config in ['debug','release']: + common = ['swift', 'run', '-c', config, '--jobs', '2', '--package-path', 'Examples/BoundedWorkflows', '--scratch-path', '.build/workflow-consumer'] + text = run(config+'-consumer', common+['BoundedWorkflows']) + cases = re.findall(r'^WORKFLOW_CASE (\S+) passed$', text, re.M) + summary = re.findall(r'^BOUNDED_WORKFLOWS passed cases=12 work=([1-9]\d*) logical_retained=([1-9]\d*)$', text, re.M) + if len(cases) != 12 or set(cases) != CASES or len(summary) != 1: raise ValueError('Incomplete actual consumer witnesses') + receipt['consumer_results'][config] = dict(cases=cases, work_units=int(summary[0][0]), logical_retained_bytes=int(summary[0][1])) + bp = pathlib.Path(run(config+'-binpath', ['swift','build','-c',config,'--package-path','Examples/BoundedWorkflows','--scratch-path','.build/workflow-consumer','--show-bin-path']).strip()) + modules = bp/'Modules' if (bp/'Modules').exists() else bp + guide = ROOT/'Sources/JSONInspection/Documentation.docc/BoundedWorkflows.md' + blocks = re.findall(r'```swift\n(.*?)\n```', guide.read_text(), re.S) + if len(blocks) != 1: raise ValueError('Changed DocC example inventory') + source = out/'guide.swift'; source.write_text(blocks[0]+'\n') + run(config+'-guide', ['swiftc','-swift-version','6','-warnings-as-errors','-I',str(modules), + '-Onone' if config == 'debug' else '-O', '-parse-as-library','-c',str(source),'-o',str(out/(config+'-guide.o'))]) + for name, (body, expected) in PROBES.items(): + source = out/(name+'.swift'); source.write_text('import JSON\nimport JSONNavigation\nimport JSONInspection\n'+body+'\n') + run(config+'-'+name, ['swiftc','-swift-version','6','-warnings-as-errors','-I',str(modules), + '-Onone' if config == 'debug' else '-O', '-parse-as-library','-c',str(source),'-o',str(out/(config+'-'+name+'.o'))], expected) + receipt['status'] = 'passed' + if not check_receipt(receipt): raise ValueError('Incomplete receipt') + except (ValueError, OSError, subprocess.TimeoutExpired) as error: + receipt.update(status='failed', error=str(error)) + save(); return 0 if receipt['status'] == 'passed' else 1 +if __name__ == '__main__': raise SystemExit(main()) diff --git a/Scripts/test_qualification_harness.py b/Scripts/test_qualification_harness.py index 4662219..d766277 100644 --- a/Scripts/test_qualification_harness.py +++ b/Scripts/test_qualification_harness.py @@ -28,6 +28,33 @@ def store_markers(): 'STORE_RUNTIME R%02d passed' % n for n in range(1,11)]) class HarnessTests(unittest.TestCase): + def test_workflow_discovery_requires_all_exact_identities(self): + names = DISCOVERY['workflow_function_ids'] + self.assertEqual(len(names), 24) + events = dict(bundle_counts=[24], module_function_counts={'JSONInspectionTests':24}, + function_names=names, parameter_case_invocations=0) + command = ['swift','test','--test-product','JSONInspectionTests','--filter','workflow'] + self.assertTrue(check_result(command, 0, '', events)[0]) + for broken in [dict(events, function_names=names[:-1]), dict(events, function_names=names+names[:1]), + dict(events, parameter_case_invocations=1), dict(events, bundle_counts=[21])]: + self.assertFalse(check_result(command, 0, '', broken)[0]) + + def test_workflow_consumer_rejects_missing_witnesses_or_incidental_diagnostics(self): + from qualify_structural_workflows import CASES, PROBES, check_receipt + data = dict(status='passed', commands=[dict(name=c+'-'+name, passed=True, compiler_crash=False, + exit=1 if name in PROBES else 0, expected=PROBES[name][1] if name in PROBES else None, + primary_diagnostics=[PROBES[name][1]] if name in PROBES else []) + for c in ['debug','release'] for name in ['consumer','binpath','guide',*PROBES]], + consumer_results={c:dict(cases=sorted(CASES),work_units=1,logical_retained_bytes=1) for c in ['debug','release']}) + self.assertTrue(check_receipt(data)) + for mode in ['missing-command','missing-case','incidental','crash']: + bad=copy.deepcopy(data) + if mode == 'missing-command': bad['commands'].pop() + elif mode == 'missing-case': bad['consumer_results']['release']['cases'].pop() + elif mode == 'incidental': bad['commands'][3]['primary_diagnostics']=['unrelated error'] + else: bad['commands'][3]['compiler_crash']=True + self.assertFalse(check_receipt(bad)) + def test_scoped_catalog_check_cannot_weaken_the_full_site_inventory(self): with tempfile.TemporaryDirectory() as temporary: root = pathlib.Path(temporary).resolve() @@ -153,7 +180,7 @@ def test_composition_discovery_and_capacity_limits_are_not_blanket_successes(sel # Combined-source checks must retain both independently owned optional products. for module, count, command in [ ('JSONNavigationTests', 57, ['swift', 'test', '--filter', 'navigation']), - ('JSONInspectionTests', 190, ['swift', 'test', '--filter', 'inspection|snapshot|projectionP|p2|exactProjectionPUB|graphG|store']), + ('JSONInspectionTests', 214, ['swift', 'test', '--filter', 'inspection|snapshot|projectionP|p2|exactProjectionPUB|graphG|store|workflow']), ('JSONSchemaExamplesTests', 98, ['swift', 'test', '--test-product', 'JSONSchemaExamplesTests']), ]: events = dict(bundle_counts=[count], module_function_counts={module: count}, @@ -167,12 +194,12 @@ def test_composition_discovery_and_capacity_limits_are_not_blanket_successes(sel for key in ('root_functions', 'portable_root_functions', 'wasi_root_functions'): self.assertEqual(policy[key]['JSONNavigationTests'], 57) self.assertEqual(policy[key]['JSONSchemaExamplesTests'], 98) - self.assertEqual(policy[key]['JSONInspectionTests'], 190) + self.assertEqual(policy[key]['JSONInspectionTests'], 214) self.assertEqual(policy[key]['JSONIntegrationTests'], 34) for key in ('shared_state_functions', 'portable_shared_state_functions'): self.assertEqual(policy[key]['JSONNavigationTests'], 2) self.assertEqual(policy[key]['JSONSchemaExamplesTests'], 4) - self.assertEqual(policy[key]['JSONInspectionTests'], 10) + self.assertEqual(policy[key]['JSONInspectionTests'], 11) self.assertEqual(policy[key]['JSONIntegrationTests'], 11) def test_private_projection_identities_cannot_be_missing_renamed_or_duplicated(self): @@ -364,8 +391,8 @@ def test_exact_discovery_rejects_missing_duplicate_and_changed_parameters(self): identity['function_names'][0] = 'JSONTests.unrelatedReplacement()' self.assertFalse(check_restored(identity)) harness = ['python3', '-B', 'Scripts/test_qualification_harness.py'] - self.assertTrue(check_result(harness, 0, 'HARNESS_TESTS passed cases=26')[0]) - for stale in ['HARNESS_TESTS passed cases=10', 'HARNESS_TESTS passed cases=11', 'HARNESS_TESTS passed cases=12', 'HARNESS_TESTS passed cases=13', 'HARNESS_TESTS passed cases=14', 'HARNESS_TESTS passed cases=15', 'HARNESS_TESTS passed cases=18', 'HARNESS_TESTS passed cases=19', 'HARNESS_TESTS passed cases=20', 'HARNESS_TESTS passed cases=21', 'HARNESS_TESTS passed cases=25', '']: + self.assertTrue(check_result(harness, 0, 'HARNESS_TESTS passed cases=28')[0]) + for stale in ['HARNESS_TESTS passed cases=10', 'HARNESS_TESTS passed cases=11', 'HARNESS_TESTS passed cases=12', 'HARNESS_TESTS passed cases=13', 'HARNESS_TESTS passed cases=14', 'HARNESS_TESTS passed cases=15', 'HARNESS_TESTS passed cases=18', 'HARNESS_TESTS passed cases=19', 'HARNESS_TESTS passed cases=20', 'HARNESS_TESTS passed cases=21', 'HARNESS_TESTS passed cases=25', 'HARNESS_TESTS passed cases=26', '']: self.assertFalse(check_result(harness, 0, stale)[0]) lifetime = ['python3', '-B', 'Scripts/qualify_schema_example_lifetime.py'] marker = 'SCHEMA_EXAMPLES_CORPUS passed mode=owned scopes=128 values=6 exact=1' diff --git a/Sources/JSONCLIEngine/CLIBuildIdentity.swift b/Sources/JSONCLIEngine/CLIBuildIdentity.swift index a659e0c..8af8093 100644 --- a/Sources/JSONCLIEngine/CLIBuildIdentity.swift +++ b/Sources/JSONCLIEngine/CLIBuildIdentity.swift @@ -2,6 +2,6 @@ // Generated by Scripts/generate_cli_identity.py; excludes this file. // SHA-256 of the ordered Package.swift + Sources content manifest, not a Git commit. package enum CLIBuildIdentity { - package static let revision = "sha256:138ae7e8bc2f4392db7aafaa000f60db5915658a1b34a55f12a9c27958da1312" + package static let revision = "sha256:43023670d6664373702741d960b010494d86149aaa181b4eeb9d6ff13701d60e" package static let kind = "source-sha256" } diff --git a/Sources/JSONCLIEngine/CLIReleaseVersion.swift b/Sources/JSONCLIEngine/CLIReleaseVersion.swift index 6ed3782..9927fbf 100644 --- a/Sources/JSONCLIEngine/CLIReleaseVersion.swift +++ b/Sources/JSONCLIEngine/CLIReleaseVersion.swift @@ -2,5 +2,5 @@ // Version of this source distribution. Publication is a separate action. package enum CLIReleaseVersion { - package static let value = "1.7.0" + package static let value = "1.8.0-dev" } diff --git a/Sources/JSONInspection/Documentation.docc/BoundedWorkflows.md b/Sources/JSONInspection/Documentation.docc/BoundedWorkflows.md new file mode 100644 index 0000000..081d7b4 --- /dev/null +++ b/Sources/JSONInspection/Documentation.docc/BoundedWorkflows.md @@ -0,0 +1,73 @@ +# Read batches and configured stores + +Read ordered exact values, or follow explicitly declared links in an immutable source. + +## Choose the smallest operation + +Use ``JSONInspectionBatch`` for typed path requests, or +``JSONStructuralSnapshot/values(for:maximumRequests:limits:)`` when you already +have snapshot-local ordinals. Repeated requests remain repeated outputs. Missing +selections throw; explicit JSON null is a value. Empty batches still admit the +whole captured source, including its source map. Each batch has one cumulative +native admission/work/retention ledger, rather than one independent read per item. + +The default request ceiling is 1,024. Limits are inclusive. Invalid configurations, +resource exhaustion and cancellation throw without publishing a partial array. +Returned JSON values retain ordinary native value semantics. Logical alias bytes +are finite accounting policy, not a process RSS or caller-instruction guarantee. + +## Declare a read-only store + +``JSONStoreBuilder`` promotes a source to ``JSONConfiguredStore`` only after the +existing complete eight-phase checked construction. Names, ID fields, paths and +reference fields are explicit. There is no id/_id inference, Unicode normalization, +database, file loader or network resolution. This API is a candidate for 1.8; +released 1.7 does not contain it. + +```swift +import Foundation +import JSON +import JSONInspection + +func boundedStoreExample() throws -> [JSON] { + let document = try JSONParser().parseDocument(Data(#"{"users":[{"id":"a","friend":{"collection":"users","id":"b"}},{"id":"b","friend":null}]}"#.utf8)) + let profile = JSONStoreProfile(version: .v1, collectionsAt: [], collections: [ + .init(name: "users", recordsAt: [.key("users")], idKey: "id", referenceFields: [ + .init(key: "friend", presence: .required, nullable: true)]) + ], references: .strict(marker: .v1, validateTargets: true)) + let store = try JSONStoreBuilder(document: document, profile: profile).build().result + guard let first = try store.lookup(collection: "users", id: "a").result else { return [] } + let closure = try store.follow(from: first) + // Construction, lookup, follow and value reads are separate reported operations. + // Every selection within this one batch shares its original native ledger. + let values = try store.values(at: closure.result) + return values.result +} +``` + +The runnable [public consumer](https://github.com/ThorFuchs/Swift-JSON/tree/main/Examples/BoundedWorkflows) +exercises exact scalar IDs, missing versus null, repeated selections, cycles, +owner refusals and cumulative reports. Use this feature branch for the 1.8 API; +a compatible dependency on 1.7 cannot supply it. + +## Understand ownership and errors + +Positions are store-instance-local tokens. They cannot be constructed publicly +or reused against another build, even if that build has byte-identical input. +Fixed collection/record/reference metadata access is ordinary trusted caller work; +scanning lookup and value/follow operations have explicit reports and finite quotas. +A missing ID in a known collection returns nil. An unknown collection throws +``JSONStoreFailure`` with ``JSONStoreFailureCode/unknownCollection``. Declared +reference slots distinguish absent, null and validated links. Marker-shaped values +outside declared fields remain ordinary JSON. + +``JSONStoreFollowLimits`` defaults to depth64 and frontier/visited/results10,000. +Follow includes the start exactly once and visits new positions breadth-first in +configured field order. A self-cycle does not consume another depth. Any reachable +new node beyond a bound throws: depth is a complete-search capacity, not permission +to return a truncated closure. Cancellation invalidates the invocation; another +operation on the same immutable store starts with independent health. + +The original twelve store construction ceilings and 49 core metrics are unchanged. +They are finite policies, not a demonstrated joint capacity or heap bound. No +persistence, mutable store, parallel batches or new performance claim is supplied. diff --git a/Sources/JSONInspection/Documentation.docc/JSONInspection.md b/Sources/JSONInspection/Documentation.docc/JSONInspection.md index e4a99f4..ad9572a 100644 --- a/Sources/JSONInspection/Documentation.docc/JSONInspection.md +++ b/Sources/JSONInspection/Documentation.docc/JSONInspection.md @@ -57,3 +57,16 @@ or numeric matrix. Sampling does not exempt any source from admission. - ``JSONGraphProfile`` - ``JSONGraphProjection`` - ``JSONGraphFailure`` + +### Bounded structural workflows (1.8 candidate) + +- +- ``JSONInspectionBatch`` +- ``JSONInspectionBatchFailure`` +- ``JSONStoreBuilder`` +- ``JSONConfiguredStore`` +- ``JSONStoreProfile`` +- ``JSONStoreCaps`` +- ``JSONStoreFollowLimits`` +- ``JSONStorePosition`` +- ``JSONStoreFailure`` diff --git a/Sources/JSONInspection/JSONInspectionBatch.swift b/Sources/JSONInspection/JSONInspectionBatch.swift new file mode 100644 index 0000000..5ccbd63 --- /dev/null +++ b/Sources/JSONInspection/JSONInspectionBatch.swift @@ -0,0 +1,153 @@ +// SPDX-License-Identifier: Apache-2.0 +import JSON +import JSONNavigation + +/// A finite batch request policy refusal; ordinary selection errors retain core paths. +public enum JSONInspectionBatchFailure: Error, Sendable, Equatable { + /// The requested result count exceeds the inclusive batch bound. + /// - Parameters: + /// - limit: Configured finite maximum. + /// - actual: Number of caller requests, including repeated requests. + case requestLimitExceeded(limit: Int, actual: Int) +} + +internal func checkInspectionBatch(_ count: Int, maximum: Int) throws { + try Task.checkCancellation() + guard maximum >= 0, maximum < Int.max else { throw JSONFailure(.configuration, code: .invalidLimits) } + guard count <= maximum else { throw JSONInspectionBatchFailure.requestLimitExceeded(limit: maximum, actual: count) } +} + +/// Ordered exact value reads with one complete source admission and cumulative ledger. +/// Caller-authored paths and later caller copies retain their ordinary value semantics. +public struct JSONInspectionBatch: Sendable { + /// Canonical immutable source used by every request. + public let document: JSONDocument + /// Finite original native quotas for each batch, not per selection. + public let limits: JSONLimits + package var checkpointObserver: (@Sendable (UInt64) -> Void)? + package var outputObserver: (@Sendable (Int) -> Void)? + package var lifetimeObserver: (@Sendable (Bool, Bool) -> Void)? + + /// Captures immutable input without opening an operation or scanning it. + /// - Parameters: + /// - document: Canonical source, optionally with coherent source mapping. + /// - limits: Finite cumulative admission, work, path and retention quotas. + public init(document: JSONDocument, limits: JSONLimits = .balanced) { + self.document = document; self.limits = limits + } + + /// Reads all requested values or throws, preserving caller order and repeats. + /// Even an empty batch admits the entire source. JSON null is a value; missing + /// paths throw. The result owns32 logical header bytes and32 per alias slot; + /// JSON aliases follow existing read semantics, not an RSS guarantee. + /// - Parameters: + /// - paths: Absolute typed selections; an empty path selects the root. + /// - maximumRequests: Inclusive request cap, default1024; zero admits only []. + /// - Returns: All exact values and one cumulative operation report. + /// - Throws: Batch capacity, core access/resource/configuration errors or cancellation. + public func values(at paths: [[JSONLocationComponent]], maximumRequests: Int = 1_024) + throws -> JSONOperationReport<[JSON]> { + try checkInspectionBatch(paths.count, maximum: maximumRequests) + var reducer = PathBatchReducer(outputObserver: outputObserver) + var traversal = JSONTraversal(document: document, limits: limits) + traversal.checkpointObserver = checkpointObserver + traversal.lifetimeObserver = lifetimeObserver + return try traversal.inspectPreparedOwned(scope: .virtual(roots: paths, overlap: .orderedOccurrences), + options: .init(depthLimit: 0), reducer: &reducer) + } +} + +private struct PathBatchReducer: JSONReadPreparation, JSONTraversalOwnedReducer { + var values: [JSON] = [] + let outputObserver: (@Sendable (Int) -> Void)? + mutating func prepare(context: inout JSONReadPreparationContext) throws { try context.retain(32) } + mutating func observe(_ hit: JSONTraversalHit, value: JSON, + context: inout JSONTraversalContext) throws -> JSONTraversalAction { + try context.work(); try context.retain(32); values.append(value) + outputObserver?(values.count) + return .skipDescendants + } + mutating func finish(_ summary: JSONTraversalSummary, context: inout JSONTraversalContext) throws -> [JSON] { + try context.checkpoint() + let result = values; values = []; return result + } +} + +extension JSONStructuralSnapshot { + /// Resolves ordered snapshot positions with one source admission and ledger. + /// This does not invoke singular public reads. Invalid positions throw before + /// admission; repeats are preserved, billed and never silently deduplicated. + /// - Parameters: + /// - ordinals: Positions local to this complete immutable snapshot. + /// - maximumRequests: Inclusive finite request/output count, default1024. + /// - limits: Cumulative native quotas for this entire independent batch. + /// - Returns: Exact values in request order and cumulative statistics. + /// - Throws: Position/capacity, core resource/access/configuration errors or cancellation. + public func values(for ordinals: [Int], maximumRequests: Int = 1_024, + limits: JSONLimits = .balanced) throws -> JSONOperationReport<[JSON]> { + try checkInspectionBatch(ordinals.count, maximum: maximumRequests) + return try readValues(for: ordinals, limits: limits) + } + package func valuesForQualification(for ordinals: [Int], limits: JSONLimits, + outputObserver: @Sendable @escaping (Int) -> Void, + lifetimeObserver: @Sendable @escaping (Bool, Bool) -> Void) throws -> JSONOperationReport<[JSON]> { + try checkInspectionBatch(ordinals.count, maximum: 1_024) + return try readValues(for: ordinals, limits: limits, outputObserver: outputObserver, lifetimeObserver: lifetimeObserver) + } + private func readValues(for ordinals: [Int], limits: JSONLimits, + outputObserver: (@Sendable (Int) -> Void)? = nil, + lifetimeObserver: (@Sendable (Bool, Bool) -> Void)? = nil) throws -> JSONOperationReport<[JSON]> { + var reducer = SnapshotBatchReducer(snapshot: self, ordinals: ordinals, outputObserver: outputObserver) + var traversal = JSONTraversal(document: document, limits: limits) + traversal.lifetimeObserver = lifetimeObserver + return try traversal.inspectPreparedOwned(options: .init(depthLimit: 0), reducer: &reducer) + } +} + +private struct SnapshotBatchReducer: JSONReadPreparation, JSONTraversalOwnedReducer { + let snapshot: JSONStructuralSnapshot + let ordinals: [Int] + let outputObserver: (@Sendable (Int) -> Void)? + private var requestAliases: UInt64 = 0 + init(snapshot: JSONStructuralSnapshot, ordinals: [Int], outputObserver: (@Sendable (Int) -> Void)?) { + self.snapshot = snapshot; self.ordinals = ordinals; self.outputObserver = outputObserver + } + mutating func prepare(context: inout JSONReadPreparationContext) throws { + try context.retain(32) + for ordinal in ordinals { + try context.work(); _ = try snapshot.record(at: ordinal) + try context.retain(8); requestAliases += 8 // count was capped before preparation + } + } + mutating func observe(_ hit: JSONTraversalHit, value: JSON, + context: inout JSONTraversalContext) throws -> JSONTraversalAction { + try context.work(); return .skipDescendants + } + mutating func finish(_ summary: JSONTraversalSummary, context: inout JSONTraversalContext) throws -> [JSON] { + var result: [JSON] = [] + for ordinal in ordinals { + try context.work() + var reversed: [JSONLocationComponent] = [], current = ordinal, scratch: UInt64 = 0 + while let parent = try snapshot.record(at: current).parentOrdinal { + try context.work() + guard parent >= 0, parent < current, + let component = try snapshot.record(at: current).componentFromParent else { + throw JSONFailure(.configuration, code: .invalidOperation) + } + try context.retain(32); scratch += 32 + if case .key(let key) = component { + for _ in key.string.utf8 { try context.work(); try context.retain(1); scratch += 1 } + } + reversed.append(component); current = parent + } + // Reuse the reserved reverse buffer; do not accumulate owned prefix copies. + for index in 0..<(reversed.count / 2) { + try context.work(); reversed.swapAt(index, reversed.count - index - 1) + } + let value = try context.selectedValue(at: reversed) + reversed = []; try context.release(scratch) + try context.retain(32); result.append(value); outputObserver?(result.count) + } + try context.checkpoint(); try context.release(requestAliases); return result + } +} diff --git a/Sources/JSONInspection/Store/JSONConfiguredStore.swift b/Sources/JSONInspection/Store/JSONConfiguredStore.swift index b58a220..5c0428b 100644 --- a/Sources/JSONInspection/Store/JSONConfiguredStore.swift +++ b/Sources/JSONInspection/Store/JSONConfiguredStore.swift @@ -3,27 +3,53 @@ import JSON import JSONNavigation private final class JSONStoreOwner: Sendable {} -package struct JSONStorePosition: Sendable { +/// Unforgeable position bound to one immutable store instance, not a persistent ID. +public struct JSONStorePosition: Sendable, Equatable { fileprivate let owner: JSONStoreOwner - package let collection: Int - package let row: Int + /// Zero-based physical source collection position. + public let collection: Int + /// Zero-based physical row in that collection. + public let row: Int + /// Compares exact owner identity and fixed local ordinals; no JSON scan. + /// - Parameters: + /// - lhs: First owner-bound position. + /// - rhs: Second owner-bound position. + /// - Returns: True only for the same source instance and physical record. + public static func == (lhs: Self, rhs: Self) -> Bool { + lhs.owner === rhs.owner && lhs.collection == rhs.collection && lhs.row == rhs.row + } } -package enum JSONStoreReferenceState: Sendable { +/// A declared reference slot; missing and explicit null remain distinct. +public enum JSONStoreReferenceState: Sendable { + /// Optional field absent from the original record. case absent + /// Explicit JSON null accepted by the field policy. case null + /// A validated existing target in this same store. + /// - Parameter position: Exact owner-bound target position. case link(JSONStorePosition) } -package struct JSONStoreRecordMetadata: Sendable { - package let position: JSONStorePosition - package let id: String - package let path: [JSONLocationComponent] - package let idPath: [JSONLocationComponent] - package let referenceRange: Range +/// Immutable source metadata; value resolution is a separately metered operation. +public struct JSONStoreRecordMetadata: Sendable { + /// Exact instance-bound physical position. + public let position: JSONStorePosition + /// Original decoded-scalar string ID. + public let id: String + /// Absolute physical path of the record. + public let path: [JSONLocationComponent] + /// Absolute path of its configured ID member. + public let idPath: [JSONLocationComponent] + /// Half-open range into the containing store references array. + public let referenceRange: Range } -package struct JSONStoreReferenceMetadata: Sendable { - package let field: Int - package let path: [JSONLocationComponent] - package let state: JSONStoreReferenceState +/// Immutable configured-slot metadata in physical record and declaration order. +public struct JSONStoreReferenceMetadata: Sendable { + /// Zero-based field declaration in the record collection profile. + public let field: Int + /// Exact absolute field path, including absent fields. + public let path: [JSONLocationComponent] + /// Absent, explicit null or a validated same-store link. + public let state: JSONStoreReferenceState } internal struct StoreCollectionMetadata: Sendable { let profile: Int @@ -31,25 +57,46 @@ internal struct StoreCollectionMetadata: Sendable { let start: Int let count: Int } -/// Package-private immutable complete view. No independent resolver or follow API. -package struct JSONConfiguredStore: Sendable { - package let document: JSONDocument - package let profile: JSONStoreProfile +/// Immutable completely admitted source, records and explicitly validated references. +/// Construct through JSONStoreBuilder. No mutable store, I/O or persistent identity. +public struct JSONConfiguredStore: Sendable { + /// Original complete canonical source, optionally with coherent source mapping. + public let document: JSONDocument + /// Immutable explicit collection/reference interpretation. + public let profile: JSONStoreProfile private let owner: JSONStoreOwner private let collections: [StoreCollectionMetadata] private let records: [JSONStoreRecordMetadata] - package let references: [JSONStoreReferenceMetadata] + /// All immutable configured slots, in physical record and field order. + public let references: [JSONStoreReferenceMetadata] package let overlayLogicalBytes: UInt64 package let identityAliasBytes: UInt64 package let profileTextBytes: UInt64 - package var collectionCount: Int { collections.count } - package var recordCount: Int { records.count } - package func recordCount(in collection: Int) throws -> Int { + /// Number of complete physical collections. + public var collectionCount: Int { collections.count } + /// Total complete physical records. + public var recordCount: Int { records.count } + /// Fixed metadata access without a new owned operation. + /// - Parameter collection: Zero-based physical collection position. + /// - Returns: Complete record count in that collection. + /// - Throws: Invalid local collection position. + public func recordCount(in collection: Int) throws -> Int { guard collection >= 0, collection < collections.count else { throw JSONStoreFailure(.invalidPosition, phase: 0, domain: .operation) } return collections[collection].count } + /// Reads a physical collection's exact name without scanning or opening a ledger. + /// - Parameter collection: Zero-based physical source collection position. + /// - Returns: Exact configured source name. + /// - Throws: Invalid collection position. + public func collectionName(at collection: Int) throws -> JSONKey { + guard collection >= 0, collection < collections.count else { + throw JSONStoreFailure(.invalidPosition, phase: 0, domain: .operation) + } + return profile.collections[collections[collection].profile].name + } + fileprivate init(document: JSONDocument, profile: JSONStoreProfile, owner: JSONStoreOwner, collections: [StoreCollectionMetadata], records: [JSONStoreRecordMetadata], references: [JSONStoreReferenceMetadata], overlay: UInt64, aliases: UInt64, @@ -60,14 +107,23 @@ package struct JSONConfiguredStore: Sendable { profileTextBytes = profileText } /// Checked fixed metadata access; positions are bound to this exact owner instance. - package func record(collection: Int, row: Int) throws -> JSONStoreRecordMetadata { + /// - Parameters: + /// - collection: Physical collection position. + /// - row: Physical row position. + /// - Returns: Immutable metadata from this exact store. + /// - Throws: Invalid local position. + public func record(collection: Int, row: Int) throws -> JSONStoreRecordMetadata { guard collection >= 0, collection < collections.count, row >= 0, row < collections[collection].count else { throw JSONStoreFailure(.invalidPosition, phase: 0, domain: .operation) } return records[collections[collection].start + row] } - package func record(at position: JSONStorePosition) throws -> JSONStoreRecordMetadata { + /// Reads fixed metadata with exact owner validation, without scanning JSON. + /// - Parameter position: Unforgeable position from this exact store instance. + /// - Returns: Original immutable record metadata. + /// - Throws: Foreign-owner or out-of-bounds position; never a null substitute. + public func record(at position: JSONStorePosition) throws -> JSONStoreRecordMetadata { guard position.owner === owner else { throw JSONStoreFailure(.invalidPosition, phase: 0, domain: .operation) } @@ -75,20 +131,35 @@ package struct JSONConfiguredStore: Sendable { } } -package struct JSONStoreBuilder: Sendable { - package let document: JSONDocument - package let profile: JSONStoreProfile - package let limits: JSONLimits - package let caps: JSONStoreCaps +/// Builds a complete immutable configured store using the original native ledger. +/// Profile preparation, admission, all construction phases and publication share it. +public struct JSONStoreBuilder: Sendable { + /// Original canonical source; admitted once during build. + public let document: JSONDocument + /// Closed typed configuration checked before source admission. + public let profile: JSONStoreProfile + /// Native cumulative operation quotas. + public let limits: JSONLimits + /// Additional inclusive store construction policies. + public let caps: JSONStoreCaps package var checkpointObserver: (@Sendable (UInt64) -> Void)? package var lifetimeObserver: (@Sendable (Bool, Bool) -> Void)? /// Qualification-only phase rendezvous; arbitrary observer work is not library work. package var phaseObserver: (@Sendable (Int) -> Void)? - package init(document: JSONDocument, profile: JSONStoreProfile, + /// Captures trusted immutable inputs; validation occurs inside build. + /// - Parameters: + /// - document: Complete canonical source, optionally with mapping. + /// - profile: Explicit exact collection/ID/reference declarations. + /// - limits: Finite cumulative native quotas. + /// - caps: Additional finite store policies, not physical capacity promises. + public init(document: JSONDocument, profile: JSONStoreProfile, limits: JSONLimits = .balanced, caps: JSONStoreCaps = .init()) { self.document = document; self.profile = profile; self.limits = limits; self.caps = caps } - package func build() throws -> JSONOperationReport { + /// Completes all construction phases atomically or throws, with no partial store. + /// - Returns: Immutable admitted store and the one original operation report. + /// - Throws: Profile/source/capacity/core resource errors or cancellation. + public func build() throws -> JSONOperationReport { try Task.checkCancellation() try limits.validate() try caps.validate() diff --git a/Sources/JSONInspection/Store/JSONStoreProfile.swift b/Sources/JSONInspection/Store/JSONStoreProfile.swift index d581aa3..0d5ffef 100644 --- a/Sources/JSONInspection/Store/JSONStoreProfile.swift +++ b/Sources/JSONInspection/Store/JSONStoreProfile.swift @@ -1,62 +1,126 @@ // SPDX-License-Identifier: Apache-2.0 import JSON -package enum JSONStoreProfileVersion: String, Sendable { case v1 = "nav12.store-profile/1" } -package enum JSONStoreMarkerVersion: String, Sendable { case v1 = "collection-id/1" } -package enum JSONStoreFieldPresence: String, Sendable { case required, optional } -package enum JSONStoreReferences: Sendable { +/// Closed version of the typed configured-store construction contract. +public enum JSONStoreProfileVersion: String, Sendable { + /// Exact collection arrays, explicit string identifiers and declared reference slots. + case v1 = "nav12.store-profile/1" +} +/// Closed reference-marker interpretation, independent of JSON Schema $ref. +public enum JSONStoreMarkerVersion: String, Sendable { + /// An object containing exactly the string members collection and id. + case v1 = "collection-id/1" +} +/// Requiredness of one explicitly declared reference field. +public enum JSONStoreFieldPresence: String, Sendable { + /// A missing member is a construction error. + case required + /// A missing member is retained as an absent slot. + case optional +} +/// Explicit reference interpretation; undeclared fields always remain ordinary JSON. +public enum JSONStoreReferences: Sendable { + /// Interpret no fields as references. case disabled + /// Validate exact markers and their existing targets before publishing a store. + /// - Parameters: + /// - marker: Closed version of the two-member marker grammar. + /// - validateTargets: Must be true; false is a typed profile refusal. case strict(marker: JSONStoreMarkerVersion, validateTargets: Bool) } -package struct JSONStoreReferenceField: Sendable { - package let key: JSONKey - package let presence: JSONStoreFieldPresence - package let nullable: Bool - package init(key: JSONKey, presence: JSONStoreFieldPresence, nullable: Bool) { +/// One configured direct reference field; no name or null policy is inferred. +public struct JSONStoreReferenceField: Sendable { + /// Exact decoded-scalar field name. + public let key: JSONKey + /// Whether a missing member is accepted as an absent slot. + public let presence: JSONStoreFieldPresence + /// Whether explicit JSON null is accepted as a null slot. + public let nullable: Bool + /// Captures trusted typed configuration, checked on construction's one ledger. + /// - Parameters: + /// - key: Literal exact member name. + /// - presence: Required or optional field policy. + /// - nullable: Explicit null policy, independent of absence. + public init(key: JSONKey, presence: JSONStoreFieldPresence, nullable: Bool) { self.key = key; self.presence = presence; self.nullable = nullable } } -package struct JSONStoreCollectionProfile: Sendable { - package let name: JSONKey - package let recordsAt: [JSONLocationComponent] - package let idKey: JSONKey - package let referenceFields: [JSONStoreReferenceField] - package init(name: JSONKey, recordsAt: [JSONLocationComponent], idKey: JSONKey, - referenceFields: [JSONStoreReferenceField]) { +/// A named direct array collection with an explicit exact string-ID field. +public struct JSONStoreCollectionProfile: Sendable { + /// Exact collection name; canonical Unicode equivalence does not merge it. + public let name: JSONKey + /// Must be exactly one key component matching name inside collectionsAt. + public let recordsAt: [JSONLocationComponent] + /// Exact direct record member holding its string ID; no id/_id fallback. + public let idKey: JSONKey + /// Declared reference slots, interpreted in this order. + public let referenceFields: [JSONStoreReferenceField] + /// Captures an explicit collection declaration without scanning source data. + /// - Parameters: + /// - name: Exact direct collection name. + /// - recordsAt: One direct key equal to name; checked during preparation. + /// - idKey: Required exact string identifier member. + /// - referenceFields: Ordered explicit reference declarations, or []. + public init(name: JSONKey, recordsAt: [JSONLocationComponent], idKey: JSONKey, + referenceFields: [JSONStoreReferenceField]) { self.name = name; self.recordsAt = recordsAt; self.idKey = idKey self.referenceFields = referenceFields } } -/// Closed native input, not a new JSON wire decoder or public configuration API. -package struct JSONStoreProfile: Sendable { - package let version: JSONStoreProfileVersion - package let collectionsAt: [JSONLocationComponent] - package let collections: [JSONStoreCollectionProfile] - package let references: JSONStoreReferences - package init(version: JSONStoreProfileVersion, collectionsAt: [JSONLocationComponent], - collections: [JSONStoreCollectionProfile], references: JSONStoreReferences) { +/// Closed native configuration, checked before source admission on the same ledger. +/// It is neither a JSON wire-profile decoder nor an online resource resolver. +public struct JSONStoreProfile: Sendable { + /// Closed construction contract version. + public let version: JSONStoreProfileVersion + /// Absolute exact typed selection of the collections object. + public let collectionsAt: [JSONLocationComponent] + /// Explicit collection declarations; unconfigured source collections are refused. + public let collections: [JSONStoreCollectionProfile] + /// Disabled or strict reference interpretation; no ambient reference behavior. + public let references: JSONStoreReferences + /// Captures immutable typed policy; construction performs bounded validation. + /// - Parameters: + /// - version: Explicit closed profile version. + /// - collectionsAt: Absolute collections-object selection, including []. + /// - collections: Exact declared direct collections. + /// - references: Explicit reference mode. + public init(version: JSONStoreProfileVersion, collectionsAt: [JSONLocationComponent], + collections: [JSONStoreCollectionProfile], references: JSONStoreReferences) { self.version = version; self.collectionsAt = collectionsAt self.collections = collections; self.references = references } } -/// Finite prototype policy only. All maxima inclusive, inside unchanged core limits. -package struct JSONStoreCaps: Sendable { - package var collections = 256 - package var recordsPerCollection = 10_000 - package var records = 10_000 - package var referenceFieldsPerCollection = 16 - package var referenceFields = 4_096 - package var referenceSlots = 100_000 - package var links = 100_000 - package var identityBytes = 4_096 - package var profileTextBytes = 1_048_576 - package var aliasBytes = 16_777_216 - package var pathComponents = 128 - package var overlayBytes = 33_554_432 - package init() {} +/// Finite inclusive construction policies within unchanged native limits. +/// Maxima are not jointly achievable capacity or RSS/throughput guarantees. +public struct JSONStoreCaps: Sendable { + /// Maximum configured/source collections. + public var collections = 256 + /// Maximum records in each complete collection. + public var recordsPerCollection = 10_000 + /// Maximum records across all collections. + public var records = 10_000 + /// Maximum configured reference fields in each collection. + public var referenceFieldsPerCollection = 16 + /// Maximum configured fields across all collections. + public var referenceFields = 4_096 + /// Maximum interpreted slots, including absent and null slots. + public var referenceSlots = 100_000 + /// Maximum staged links; repeated targets remain distinct slots. + public var links = 100_000 + /// Maximum UTF-8 bytes in a collection or record identity. + public var identityBytes = 4_096 + /// Maximum cumulative profile UTF-8 text bytes. + public var profileTextBytes = 1_048_576 + /// Maximum live logical ID/marker text-alias bytes. + public var aliasBytes = 16_777_216 + /// Maximum typed components in generated/configured paths. + public var pathComponents = 128 + /// Maximum conservative logical metadata/scratch overlay bytes. + public var overlayBytes = 33_554_432 + /// Uses finite policy defaults; zero is finite, negatives and Int.max invalid. + public init() {} internal func validate() throws { - // Fixed bootstrap size; no source/profile text scanned before native health. for value in [collections, recordsPerCollection, records, referenceFieldsPerCollection, referenceFields, referenceSlots, links, identityBytes, profileTextBytes, aliasBytes, pathComponents, overlayBytes] { @@ -65,28 +129,98 @@ package struct JSONStoreCaps: Sendable { } } -package enum JSONStoreFailureCode: String, Sendable { - case invalidTypedComponent, duplicateCollectionDeclaration, invalidRecordsAt - case duplicateReferenceField, idReferenceFieldCollision, disabledReferencesConflict - case invalidReferenceMode, collectionsNotObject, unconfiguredCollection, collectionNotArray - case missingConfiguredCollection, recordWrongKind, recordIDMissing, recordIDNull, recordIDWrongKind - case duplicateRecordID, referenceMissing, referenceNull, referenceMarkerWrongKind - case referenceMarkerMissing, referenceMarkerMemberWrongKind, referenceMarkerExtra - case referenceUnknownCollection, referenceDanglingID, selectionMissing, selectionWrongKind - case capacityExceeded, counterOverflow, invalidPosition +/// Stable typed construction/selection refusals, separate from native quota errors. +public enum JSONStoreFailureCode: String, Sendable { + /// A configured typed index is negative. + case invalidTypedComponent + /// Two profile collection names have identical decoded scalars. + case duplicateCollectionDeclaration + /// A collection selector is not its one exact direct name. + case invalidRecordsAt + /// Two configured fields have identical decoded scalars. + case duplicateReferenceField + /// An ID member is also declared as a reference. + case idReferenceFieldCollision + /// Reference declarations conflict with disabled references. + case disabledReferencesConflict + /// Strict mode attempted to disable target validation. + case invalidReferenceMode + /// The selected collections value is not an object. + case collectionsNotObject + /// A source collection has no explicit declaration. + case unconfiguredCollection + /// A declared collection value is not an array. + case collectionNotArray + /// A declared collection is absent from the source. + case missingConfiguredCollection + /// A physical record is not an object. + case recordWrongKind + /// The required ID member is absent. + case recordIDMissing + /// The required ID is explicit null. + case recordIDNull + /// The ID is present with a non-string kind. + case recordIDWrongKind + /// The second exact duplicate ID retains its first ID path. + case duplicateRecordID + /// A required declared reference field is absent. + case referenceMissing + /// A declared reference field forbids explicit null. + case referenceNull + /// A non-null marker is not an object. + case referenceMarkerWrongKind + /// An exact collection/id marker member is missing. + case referenceMarkerMissing + /// A marker member has a non-string kind. + case referenceMarkerMemberWrongKind + /// A marker has an undeclared extra member. + case referenceMarkerExtra + /// A marker names an unknown exact collection. + case referenceUnknownCollection + /// A marker names a missing ID in an existing collection. + case referenceDanglingID + /// The collections selection is missing; coreCause retains the original error. + case selectionMissing + /// The collections selection crosses a wrong kind. + case selectionWrongKind + /// A finite store construction or follow policy was exceeded. + case capacityExceeded + /// Checked size/counter arithmetic overflowed. + case counterOverflow + /// A local position is out of bounds or belongs to another store instance. + case invalidPosition + /// A lookup request names no declared physical collection. + case unknownCollection +} +/// The authority that refused construction or a read-only operation. +public enum JSONStoreFailureDomain: String, Sendable { + /// Invalid typed configuration, before source admission. + case profile + /// Invalid source structure or declared references. + case source + /// Invalid local position, query policy or checked capacity. + case operation } -package enum JSONStoreFailureDomain: String, Sendable { case profile, source, operation } -/// Source path and core cause are separate; missing targets have no invented span. -package struct JSONStoreFailure: Error, Sendable { - package let code: JSONStoreFailureCode - package let phase: Int - package let domain: JSONStoreFailureDomain - package let path: [JSONLocationComponent] - package let firstPath: [JSONLocationComponent]? - package let sourceSpan: JSONSourceSpan? - package let coreCause: JSONFailure? - package let capacity: String? - package let limit: Int? +/// Precise checked refusal; no partial store or null substitute is published. +public struct JSONStoreFailure: Error, Sendable { + /// Stable refusal identity. + public let code: JSONStoreFailureCode + /// Construction phases1...8; ordinary metadata0 and owned queries9. + public let phase: Int + /// Profile, source or operation authority. + public let domain: JSONStoreFailureDomain + /// Exact source/request path, empty when no location applies. + public let path: [JSONLocationComponent] + /// First occurrence for a duplicate ID; never a normalized key. + public let firstPath: [JSONLocationComponent]? + /// Coherent original span, or nil when unavailable. + public let sourceSpan: JSONSourceSpan? + /// Original core selection failure without loss of its resource/location metadata. + public let coreCause: JSONFailure? + /// Named construction/follow capacity for a capacity refusal. + public let capacity: String? + /// Inclusive configured capacity, distinct from physical memory. + public let limit: Int? internal init(_ code: JSONStoreFailureCode, phase: Int, domain: JSONStoreFailureDomain, path: [JSONLocationComponent] = [], firstPath: [JSONLocationComponent]? = nil, sourceSpan: JSONSourceSpan? = nil, coreCause: JSONFailure? = nil, diff --git a/Sources/JSONInspection/Store/JSONStoreQueries.swift b/Sources/JSONInspection/Store/JSONStoreQueries.swift new file mode 100644 index 0000000..0055990 --- /dev/null +++ b/Sources/JSONInspection/Store/JSONStoreQueries.swift @@ -0,0 +1,234 @@ +// SPDX-License-Identifier: Apache-2.0 +import JSON +import JSONNavigation + +/// Inclusive finite policies for a complete reachable closure, not truncation. +public struct JSONStoreFollowLimits: Sendable { + /// Maximum shortest link distance from the start; start has depth zero. + public var maximumDepth: Int = 64 + /// Maximum pending BFS positions, excluding already consumed queue entries. + public var maximumFrontier: Int = 10_000 + /// Maximum distinct visited records, including the start. + public var maximumVisited: Int = 10_000 + /// Maximum returned positions, including the start. + public var maximumResults: Int = 10_000 + /// Uses finite defaults; zero is valid, negative values and Int.max are invalid. + public init() {} + internal func validate() throws { + for value in [maximumDepth, maximumFrontier, maximumVisited, maximumResults] { + guard value >= 0, value < Int.max else { throw JSONFailure(.configuration, code: .invalidLimits) } + } + } +} + +extension JSONConfiguredStore { + /// Looks up an exact collection/string ID in one independent admitted operation. + /// No canonical normalization or id/_id fallback. Missing IDs return nil; + /// an unknown collection is a distinct typed refusal. Metadata scans and text + /// comparisons use the same source-admission ledger; no index speed is promised. + /// - Parameters: + /// - collection: Exact decoded-scalar configured collection name. + /// - id: Exact decoded-scalar string record identifier. + /// - limits: Finite cumulative native quotas for this lookup. + /// - Returns: Owner-bound position or nil, plus cumulative statistics. + /// - Throws: Unknown collection, native configuration/resource failure or cancellation. + public func lookup(collection: JSONKey, id: String, limits: JSONLimits = .balanced) + throws -> JSONOperationReport { + let report = try query(.lookup(collection, id), limits: limits) + guard case .position(let result) = report.result else { throw JSONFailure(.configuration, code: .invalidOperation) } + return .init(result: result, statistics: report.statistics) + } + + /// Reads exact values in request order, preserving repeated positions. + /// One source admission, metadata reservation and ledger cover the entire batch. + /// Empty requests still admit the whole source; foreign tokens never become null. + /// Result aliases follow ordinary native read semantics and caller-copy boundaries. + /// - Parameters: + /// - positions: Unforgeable same-store record positions. + /// - maximumRequests: Inclusive finite request/result cap, default1024. + /// - limits: Finite native quotas for the whole independent batch. + /// - Returns: All canonical record objects and cumulative statistics. + /// - Throws: Position/batch/core resource/configuration failure or cancellation. + public func values(at positions: [JSONStorePosition], maximumRequests: Int = 1_024, + limits: JSONLimits = .balanced) throws -> JSONOperationReport<[JSON]> { + try checkInspectionBatch(positions.count, maximum: maximumRequests) + let report = try query(.values(positions), limits: limits) + guard case .values(let result) = report.result else { throw JSONFailure(.configuration, code: .invalidOperation) } + return .init(result: result, statistics: report.statistics) + } + + /// Resolves one same-store position through the one-request batch delegate. + /// - Parameters: + /// - position: Unforgeable same-store physical record position. + /// - limits: Finite native quotas for this independent admitted read. + /// - Returns: The original canonical record and actual operation statistics. + /// - Throws: Invalid position, core resource/configuration failure or cancellation. + public func value(at position: JSONStorePosition, limits: JSONLimits = .balanced) + throws -> JSONOperationReport { + let report = try values(at: [position], maximumRequests: 1, limits: limits) + return .init(result: report.result[0], statistics: report.statistics) + } + + /// Returns the complete BFS reachable closure, including start exactly once. + /// Only declared validated link slots are edges. Absent/null slots add no edge; + /// cycles are deduplicated by physical positions. Any exceeded depth/frontier/ + /// visited/result cap throws instead of publishing a truncated closure. No I/O. + /// - Parameters: + /// - start: Owner-bound starting record in this immutable store. + /// - bounds: Inclusive finite BFS policies, including a finite depth ceiling. + /// - limits: Finite native quotas shared by admission, aliases, scans and output. + /// - Returns: Deterministic positions in configured-field BFS order and statistics. + /// - Throws: Invalid position, typed capacity/native resource failure or cancellation. + public func follow(from start: JSONStorePosition, bounds: JSONStoreFollowLimits = .init(), + limits: JSONLimits = .balanced) throws -> JSONOperationReport<[JSONStorePosition]> { + try Task.checkCancellation(); try bounds.validate() + let report = try query(.follow(start, bounds), limits: limits) + guard case .positions(let result) = report.result else { throw JSONFailure(.configuration, code: .invalidOperation) } + return .init(result: result, statistics: report.statistics) + } + + package func followForQualification(from start: JSONStorePosition, bounds: JSONStoreFollowLimits, + limits: JSONLimits, checkpointObserver: @Sendable @escaping (UInt64) -> Void, + lifetimeObserver: @Sendable @escaping (Bool, Bool) -> Void, + edgeObserver: (@Sendable (Int) -> Void)? = nil) throws -> JSONOperationReport<[JSONStorePosition]> { + try bounds.validate() + let report = try query(.follow(start, bounds), limits: limits, + checkpointObserver: checkpointObserver, lifetimeObserver: lifetimeObserver, edgeObserver: edgeObserver) + guard case .positions(let result) = report.result else { throw JSONFailure(.configuration, code: .invalidOperation) } + return .init(result: result, statistics: report.statistics) + } + + private func query(_ operation: StoreReadOperation, limits: JSONLimits, + checkpointObserver: (@Sendable (UInt64) -> Void)? = nil, + lifetimeObserver: (@Sendable (Bool, Bool) -> Void)? = nil, + edgeObserver: (@Sendable (Int) -> Void)? = nil) + throws -> JSONOperationReport { + var reducer = StoreReadReducer(store: self, operation: operation, edgeObserver: edgeObserver) + var traversal = JSONTraversal(document: document, limits: limits) + traversal.checkpointObserver = checkpointObserver + traversal.lifetimeObserver = lifetimeObserver + return try traversal.inspectPreparedOwned(options: .init(depthLimit: 0), reducer: &reducer) + } +} + +private enum StoreReadOperation { + case lookup(JSONKey, String) + case values([JSONStorePosition]) + case follow(JSONStorePosition, JSONStoreFollowLimits) +} +private enum StoreReadResult { + case position(JSONStorePosition?) + case values([JSON]) + case positions([JSONStorePosition]) +} + +private struct StoreReadReducer: JSONReadPreparation, JSONTraversalOwnedReducer { + let store: JSONConfiguredStore + let operation: StoreReadOperation + let edgeObserver: (@Sendable (Int) -> Void)? + private var inputAliases: UInt64 = 0 + init(store: JSONConfiguredStore, operation: StoreReadOperation, edgeObserver: (@Sendable (Int) -> Void)?) { + self.store = store; self.operation = operation; self.edgeObserver = edgeObserver + } + mutating func prepare(context: inout JSONReadPreparationContext) throws { + inputAliases = try JSONStoreAccounting.sum(store.overlayLogicalBytes, 128) + try context.retain(inputAliases) + switch operation { + case .lookup(let collection, let id): + try context.admitText(collection.string, key: true); try context.admitText(id) + for text in [collection.string, id] { + for _ in text.utf8 { try context.work(); inputAliases = try JSONStoreAccounting.sum(inputAliases, 1) } + } + case .values(let positions): + for position in positions { + try context.work(); _ = try store.record(at: position) + try context.retain(8); inputAliases = try JSONStoreAccounting.sum(inputAliases, 8) + } + case .follow(let start, _): + try context.work(); _ = try store.record(at: start) + } + } + mutating func observe(_ hit: JSONTraversalHit, value: JSON, + context: inout JSONTraversalContext) throws -> JSONTraversalAction { + try context.work(); return .skipDescendants + } + mutating func finish(_ summary: JSONTraversalSummary, context: inout JSONTraversalContext) throws -> StoreReadResult { + let result: StoreReadResult + switch operation { + case .lookup(let collection, let id): result = .position(try lookup(collection, id: id, context: &context)) + case .values(let positions): + var values: [JSON] = []; try context.retain(32) + for position in positions { + try context.work() + let record = try store.record(at: position) + let value = try context.selectedValue(at: record.path) + try context.retain(32); values.append(value) + } + result = .values(values) + case .follow(let start, let bounds): result = .positions(try follow(start, bounds: bounds, context: &context)) + } + try context.checkpoint(); try context.release(inputAliases) + return result + } + private func lookup(_ name: JSONKey, id: String, context: inout JSONTraversalContext) throws -> JSONStorePosition? { + try context.retain(32) + for collection in 0.. Int { + try context.work(); _ = try store.record(at: position) + var offset: UInt64 = 0 + for collection in 0.. [JSONStorePosition] { + var visited: [Bool] = [], result: [JSONStorePosition] = [], depths: [Int] = [] + try context.retain(32) + for _ in 0.. JSONStoreFailure { + .init(.capacityExceeded, phase: 9, domain: .operation, capacity: name, limit: limit) + } + func enqueue(_ position: JSONStorePosition, depth: Int, context: inout JSONTraversalContext) throws { + try context.work() + guard depth <= bounds.maximumDepth else { throw refuse("maximumDepth", bounds.maximumDepth) } + guard result.count - head < bounds.maximumFrontier else { throw refuse("maximumFrontier", bounds.maximumFrontier) } + guard result.count < bounds.maximumVisited else { throw refuse("maximumVisited", bounds.maximumVisited) } + guard result.count < bounds.maximumResults else { throw refuse("maximumResults", bounds.maximumResults) } + let index = try flatIndex(position, context: &context) + try context.retain(40); visited[index] = true; result.append(position); depths.append(depth) + } + try enqueue(start, depth: 0, context: &context) + while head < result.count { + try context.work() + let record = try store.record(at: result[head]), depth = depths[head]; head += 1 + for slot in record.referenceRange { + try context.work() + if case .link(let target) = store.references[slot].state { + let index = try flatIndex(target, context: &context) + if !visited[index] { + guard depth < bounds.maximumDepth else { throw refuse("maximumDepth", bounds.maximumDepth) } + try enqueue(target, depth: depth + 1, context: &context) + } + } + edges += 1; edgeObserver?(edges) + try context.checkpoint() + } + } + let scratch = try JSONStoreAccounting.sum(UInt64(visited.count), JSONStoreAccounting.product(UInt64(depths.count), 8)) + visited = []; depths = []; try context.release(scratch) + return result + } +} diff --git a/Sources/JSONNavigation/JSONTraversalContext.swift b/Sources/JSONNavigation/JSONTraversalContext.swift index d1c986e..7bf4fff 100644 --- a/Sources/JSONNavigation/JSONTraversalContext.swift +++ b/Sources/JSONNavigation/JSONTraversalContext.swift @@ -124,6 +124,21 @@ package struct JSONTraversalContext: ~Copyable { try scope.document?.sourceMap?.span(at: path, kind: .value, meter: &state.meter) } } + /// Resolve a typed selection on this original admitted scope; no new ledger. + /// The value is an ephemeral native alias; returned owned slots need reservation. + package mutating func selectedValue(at path: [JSONLocationComponent]) throws -> JSON { + try perform(at: path) { state in + // A metadata path is not automatically admitted by this invocation. + // Use the original scope's checked path constructor before selection. + var checked: [JSONLocationComponent] = [] + for component in path { + checked = try scope.append(component, to: checked, state: state).components + } + try state.meter.work() + return try scope.select(checked, state: state) + } + } + /// Transfer already-admitted input ownership, without exposing its scope. package mutating func transferDocumentOwnership() throws -> JSONDocument { try perform { _ in try scope.transferDocumentOwnership() } diff --git a/Tests/JSONCLITests/IntegrationTests.swift b/Tests/JSONCLITests/IntegrationTests.swift index 5157667..12ac2a7 100644 --- a/Tests/JSONCLITests/IntegrationTests.swift +++ b/Tests/JSONCLITests/IntegrationTests.swift @@ -146,7 +146,7 @@ private func integrationDirectory() throws -> URL { @Test func cliVersionIsDeterministicAndNeedsNoRepositoryOrInput() throws { let metadata = try JSONParser().parse(CLIHelp.versionJSON) - #expect(try metadata.requiredMember("version") == .string("1.7.0")) + #expect(try metadata.requiredMember("version") == .string("1.8.0-dev")) #expect(try metadata.requiredMember("revisionKind") == .string("source-sha256")) let revision = try #require(try metadata.requiredMember("revision").stringValue) #expect(revision.hasPrefix("sha256:") && revision.utf8.count == 71) diff --git a/Tests/JSONInspectionTests/WorkflowTests.swift b/Tests/JSONInspectionTests/WorkflowTests.swift new file mode 100644 index 0000000..82bf917 --- /dev/null +++ b/Tests/JSONInspectionTests/WorkflowTests.swift @@ -0,0 +1,309 @@ +// SPDX-License-Identifier: Apache-2.0 +import Foundation +import Synchronization +import Testing +import JSON +import JSONNavigation +@testable import JSONInspection + +private enum WorkflowTestError: Error { case expectedFailure } +private func wfCore(_ body: () throws -> Void) throws -> JSONFailure { + do { try body(); Issue.record("Expected core failure"); throw WorkflowTestError.expectedFailure } + catch let error as JSONFailure { return error } +} +private func wfStore(_ body: () throws -> Void) throws -> JSONStoreFailure { + do { try body(); Issue.record("Expected store failure"); throw WorkflowTestError.expectedFailure } + catch let error as JSONStoreFailure { return error } +} +private func wfDocument(_ text: String) throws -> JSONDocument { + try JSONParser().parseDocument(Data(text.utf8)) +} +private func wfStoreFixture() throws -> JSONConfiguredStore { + let document = try wfDocument(#"{"collections":{"users":[{"id":"é","count":9007199254740993,"friend":{"collection":"users","id":"e\u0301"},"optional":null},{"id":"e\u0301","friend":{"collection":"users","id":"é"}}],"items":[{"id":"x","owner":{"collection":"users","id":"é"}}]}}"#) + let profile = JSONStoreProfile(version: .v1, collectionsAt: [.key("collections")], collections: [ + .init(name: "items", recordsAt: [.key("items")], idKey: "id", referenceFields: [.init(key: "owner", presence: .required, nullable: false)]), + .init(name: "users", recordsAt: [.key("users")], idKey: "id", referenceFields: [ + .init(key: "friend", presence: .required, nullable: false), .init(key: "optional", presence: .optional, nullable: true)])], + references: .strict(marker: .v1, validateTargets: true)) + return try JSONStoreBuilder(document: document, profile: profile).build().result +} +private func wfStar() throws -> JSONConfiguredStore { + let document = try wfDocument(#"{"u":[{"id":"0","left":{"collection":"u","id":"1"},"right":{"collection":"u","id":"2"}},{"id":"1"},{"id":"2"}]}"#) + return try JSONStoreBuilder(document: document, profile: .init(version: .v1, collectionsAt: [], collections: [ + .init(name: "u", recordsAt: [.key("u")], idKey: "id", referenceFields: [ + .init(key: "left", presence: .optional, nullable: true), .init(key: "right", presence: .optional, nullable: true)])], + references: .strict(marker: .v1, validateTargets: true))).build().result +} + +@Test func workflowB01PathsPreserveOrderRepeatsNullAndExactScalars() throws { + let document = try wfDocument(#"{"é":9007199254740993,"e\u0301":null,"a/b~":0.1}"#) + let report = try JSONInspectionBatch(document: document).values(at: [[.key("e\u{301}")], [.key("é")], [.key("e\u{301}")], [.key("a/b~")]]) + #expect(report.result == [.null, .number(try JSONNumber(parsing: "9007199254740993")), .null, .number(try JSONNumber(parsing: "0.1"))]) + #expect(report.statistics[.workUnits] > 0) +} +@Test func workflowB02OneCompleteAdmissionAndEmptyBatch() throws { + let document = try wfDocument("[1,2,3]") + let base = try JSONTraversal(document: document).visit { _,_ in .skipDescendants } + let batch = try JSONInspectionBatch(document: document).values(at: [[.index(1)], [.index(1)]]) + #expect(batch.statistics[.nodes] == base.statistics[.nodes]) + #expect(batch.statistics[.sourceMapEntries] == base.statistics[.sourceMapEntries]) + let empty = try JSONInspectionBatch(document: document).values(at: [], maximumRequests: 0) + #expect(empty.result.isEmpty && empty.statistics[.nodes] == base.statistics[.nodes]) + var limits = JSONLimits.balanced; limits[.nodes] = 0 + #expect(try wfCore { _ = try JSONInspectionBatch(document: document, limits: limits).values(at: []) }.resource?.metric == .nodes) +} +@Test func workflowB03RequestCapacityConfigurationAndExactBoundary() throws { + let batch = JSONInspectionBatch(document: .init(root: .null)) + #expect(try batch.values(at: [[],[]], maximumRequests: 2).result == [.null,.null]) + do { _ = try batch.values(at: [[],[]], maximumRequests: 1); Issue.record("Expected batch refusal") } + catch let error as JSONInspectionBatchFailure { #expect(error == .requestLimitExceeded(limit: 1, actual: 2)) } + for invalid in [-1, Int.max] { + #expect(try wfCore { _ = try batch.values(at: [], maximumRequests: invalid) }.code == .invalidLimits) + } +} +@Test func workflowB04MissingAndWrongKindRetainTypedPaths() throws { + let batch = JSONInspectionBatch(document: try wfDocument(#"{"a":null}"#)) + for (path, category, code) in [([JSONLocationComponent.key("missing")], JSONFailure.Category.missingValue, JSONFailure.Code.missingMember), + ([.key("a"),.index(0)], .typeMismatch, .wrongKind)] { + do { _ = try batch.values(at: [[],path]); Issue.record("Expected atomic access refusal") } + catch let error as JSONAccessFailure { + #expect(error.path == path && error.cause.category == category && error.cause.code == code) + #expect(error.cause.location != nil) + } + } +} +@Test func workflowB05UnselectedSuffixAndCumulativeWorkCannotReset() throws { + let document = try wfDocument(#"{"selected":null,"outside":[1,2,3,4]}"#) + var limits = JSONLimits.balanced; limits[.nodes] = 2 + #expect(try wfCore { _ = try JSONInspectionBatch(document: document, limits: limits).values(at: [[.key("selected")]]) }.resource?.metric == .nodes) + let paths = Array(repeating: [JSONLocationComponent.key("selected")], count: 8) + let measured = try JSONInspectionBatch(document: document).values(at: paths) + limits = .balanced; limits[.workUnits] = measured.statistics[.workUnits] + #expect(try JSONInspectionBatch(document: document, limits: limits).values(at: paths).result.count == 8) + limits[.workUnits] -= 1 + let failure = try wfCore { _ = try JSONInspectionBatch(document: document, limits: limits).values(at: paths) } + #expect(failure.resource?.metric == .workUnits && failure.resource?.limit == limits[.workUnits]) +} +@Test func workflowB06SnapshotBatchKeepsOneSourceAndOrdinalMeaning() throws { + let snapshot = try JSONStructuralSnapshotBuilder(document: wfDocument("[9,8,null]")).build().result + let batch = try snapshot.values(for: [2,1,3,2]) + #expect(batch.result == [8,9,.null,8]) + let one = try snapshot.value(for: 2) + #expect(batch.statistics[.nodes] == one.statistics[.nodes]) + var limits = JSONLimits.balanced; limits[.nodes] = 0 + do { _ = try snapshot.values(for: [1,99], limits: limits); Issue.record("Expected bad ordinal") } + catch let error as JSONStructuralSnapshotFailure { #expect(error == .ordinalOutOfBounds(99)) } +} +@Test func workflowB07InnerCancellationInvalidatesOriginalScope() async throws { + let document = try wfDocument("[1,2,3]") + let lifetime = Mutex(false), outputs = Mutex(0) + let task = Task { + var limits = JSONLimits.balanced; limits[.cancellationQuantum] = 1 + var batch = JSONInspectionBatch(document: document, limits: limits) + batch.lifetimeObserver = { root, state in lifetime.withLock { $0 = root && state } } + batch.outputObserver = { count in + outputs.withLock { $0 = count } + if count == 2 { withUnsafeCurrentTask { $0?.cancel() } } + } + return try batch.values(at: Array(repeating: [.index(0)], count: 20)) + } + do { _ = try await task.value; Issue.record("Expected cancellation") } catch is CancellationError {} + #expect(lifetime.withLock { $0 }) + #expect(outputs.withLock { $0 } == 2) +} +@Test func workflowB08RetentionBoundaryIsFiniteAndSourceIsUnchanged() throws { + let document = try wfDocument("[1,2]") + let report = try JSONInspectionBatch(document: document).values(at: [[],[.index(1)]]) + var limits = JSONLimits.balanced; limits[.retainedBytes] = report.statistics.peakLogicalRetainedBytes + _ = try JSONInspectionBatch(document: document, limits: limits).values(at: [[],[.index(1)]]) + limits[.retainedBytes] -= 1 + #expect(try wfCore { _ = try JSONInspectionBatch(document: document, limits: limits).values(at: [[],[.index(1)]]) }.resource?.metric == .retainedBytes) + #expect(document.root == [1,2] && document.sourceMap != nil) +} +@Test func workflowS01PublicStorePreservesPhysicalAndDeclaredOrder() throws { + let store = try wfStoreFixture() + #expect(store.collectionCount == 2 && store.recordCount == 3) + #expect(try store.collectionName(at: 0) == "users") + #expect(try store.collectionName(at: 1) == "items") + #expect(try store.recordCount(in: 0) == 2 && store.references.count == 5) + #expect(try store.record(collection: 0, row: 0).idPath == [.key("collections"),.key("users"),.index(0),.key("id")]) +} +@Test func workflowS02LookupUsesDecodedScalarIDsAndDistinguishesMissing() throws { + let store = try wfStoreFixture() + let composed = try #require(store.lookup(collection: "users", id: "é").result) + let decomposed = try #require(store.lookup(collection: "users", id: "e\u{301}").result) + #expect(composed.row == 0 && decomposed.row == 1 && composed != decomposed) + #expect(try store.lookup(collection: "users", id: "absent").result == nil) + let error = try wfStore { _ = try store.lookup(collection: "unknown", id: "é") } + #expect(error.code == .unknownCollection && error.domain == .operation && error.phase == 9) +} +@Test func workflowS03OwnerTokensCannotRebindAnotherInstance() throws { + let store = try wfStoreFixture(), other = try JSONStoreBuilder(document: store.document, profile: store.profile).build().result + let position = try store.record(collection: 0, row: 0).position + #expect(position != (try other.record(collection: 0, row: 0).position)) + #expect(try wfStore { _ = try other.values(at: [position]) }.code == .invalidPosition) + #expect(try wfStore { _ = try other.follow(from: position) }.code == .invalidPosition) +} +@Test func workflowS04StoreBatchValuesPreserveOrderExactnessAndOneAdmission() throws { + let store = try wfStoreFixture(), a = try store.record(collection: 0, row: 0).position, b = try store.record(collection: 0, row: 1).position + let report = try store.values(at: [b,a,b]) + #expect(try report.result[0].requiredMember("id") == .string("e\u{301}")) + #expect(try report.result[1].requiredMember("count") == .number(JSONNumber(parsing: "9007199254740993"))) + #expect(report.result[0] == report.result[2]) + let source = try JSONTraversal(document: store.document).visit { _,_ in .skipDescendants } + #expect(report.statistics[.nodes] == source.statistics[.nodes]) + #expect(try store.value(at: a).result == report.result[1]) + #expect(try store.values(at: [], maximumRequests: 0).statistics[.nodes] == source.statistics[.nodes]) +} +@Test func workflowS05ReferenceSlotsKeepAbsentNullAndLinkDistinct() throws { + let store = try wfStoreFixture() + if case .link(let p) = store.references[0].state { #expect(p.row == 1) } else { Issue.record("Expected link") } + if case .null = store.references[1].state {} else { Issue.record("Expected null") } + if case .absent = store.references[3].state {} else { Issue.record("Expected absence") } +} +@Test func workflowS06FollowCompletesCyclesAndPreservesBFSOrder() throws { + let store = try wfStoreFixture(), start = try store.record(collection: 1, row: 0).position + let report = try store.follow(from: start) + #expect(report.result.map(\.collection) == [1,0,0] && report.result.map(\.row) == [0,0,1]) + let star = try wfStar(), root = try star.record(collection: 0, row: 0).position + #expect(try star.follow(from: root).result.map(\.row) == [0,1,2]) +} +@Test func workflowS07FollowAllFourInclusiveBoundsThrowInsteadOfTruncate() throws { + let store = try wfStar(), start = try store.record(collection: 0, row: 0).position + let cases: [(String, WritableKeyPath, Int)] = [ + ("maximumDepth",\JSONStoreFollowLimits.maximumDepth,1), ("maximumFrontier",\JSONStoreFollowLimits.maximumFrontier,2), + ("maximumVisited",\JSONStoreFollowLimits.maximumVisited,3), ("maximumResults",\JSONStoreFollowLimits.maximumResults,3)] + for (name,key,needed) in cases { + var bounds = JSONStoreFollowLimits(); bounds[keyPath: key] = needed + #expect(try store.follow(from: start, bounds: bounds).result.count == 3) + bounds[keyPath: key] = needed - 1 + let failure = try wfStore { _ = try store.follow(from: start, bounds: bounds) } + #expect(failure.code == .capacityExceeded && failure.capacity == name && failure.limit == needed - 1) + for invalid in [-1, Int.max] { + bounds[keyPath: key] = invalid + #expect(try wfCore { _ = try store.follow(from: start, bounds: bounds) }.code == .invalidLimits) + } + } +} +@Test func workflowS08FollowAndLookupHaveCumulativeQuotaAndNoSecondAdmission() throws { + let store = try wfStar(), start = try store.record(collection: 0, row: 0).position + let report = try store.follow(from: start) + let lookup = try store.lookup(collection: "u", id: "2") + #expect(report.statistics[.nodes] == lookup.statistics[.nodes]) + for work in [report.statistics[.workUnits], lookup.statistics[.workUnits]] { #expect(work > 1) } + var limits = JSONLimits.balanced; limits[.workUnits] = report.statistics[.workUnits] + #expect(try store.follow(from: start, limits: limits).result.count == 3) + limits[.workUnits] -= 1 + #expect(try wfCore { _ = try store.follow(from: start, limits: limits) }.resource?.metric == .workUnits) +} +@Test func workflowS09FollowInnerCancellationAndInvalidation() async throws { + let store = try wfStar(), start = try store.record(collection: 0, row: 0).position + let lifetime = Mutex(false), edges = Mutex(0) + let task = Task { + var limits = JSONLimits.balanced; limits[.cancellationQuantum] = 1 + return try store.followForQualification(from: start, bounds: .init(), limits: limits, + checkpointObserver: { _ in }, lifetimeObserver: { root, state in lifetime.withLock { $0 = root && state } }, + edgeObserver: { count in + edges.withLock { $0 = count } + if count == 1 { withUnsafeCurrentTask { $0?.cancel() } } + }) + } + do { _ = try await task.value; Issue.record("Expected inner cancellation") } catch is CancellationError {} + #expect(edges.withLock { $0 } == 1 && lifetime.withLock { $0 }) + #expect(try store.follow(from: start).result.count == 3) +} +@Test func workflowS10SourceReplacementCannotInventSpansOrRebindPositions() throws { + let store = try wfStoreFixture(), position = try store.record(collection: 0, row: 0).position + let copy = try JSONStoreBuilder(document: store.document.replacingRoot(store.document.root), profile: store.profile).build().result + #expect(store.document.sourceMap != nil && copy.document.sourceMap == nil) + #expect(try wfStore { _ = try copy.value(at: position) }.code == .invalidPosition) + let original = try store.record(at: position) + #expect(try store.document.sourceMap?.span(at: original.path) != nil) +} +@Test func workflowS11DisabledReferencesRemainOrdinaryData() throws { + let document = try wfDocument(#"{"u":[{"id":"a","arbitrary":{"collection":"missing","id":"not-a-link"}}]}"#) + let store = try JSONStoreBuilder(document: document, profile: .init(version: .v1, collectionsAt: [], collections: [ + .init(name: "u", recordsAt: [.key("u")], idKey: "id", referenceFields: [])], references: .disabled)).build().result + let start = try store.record(collection: 0, row: 0).position + var bounds = JSONStoreFollowLimits(); bounds.maximumDepth = 0 + #expect(try store.follow(from: start, bounds: bounds).result == [start] && store.references.isEmpty) +} +@Test func workflowSharedImmutablePlansHaveIndependentBatchAndStoreHealth() async throws { + let store = try wfStar(), start = try store.record(collection: 0, row: 0).position + let batch = JSONInspectionBatch(document: store.document) + try await withThrowingTaskGroup(of: Bool.self) { group in + for i in 0..<8 { group.addTask { + if i % 2 == 0 { + var limits = JSONLimits.balanced; limits[.workUnits] = 0 + do { _ = try store.follow(from: start, limits: limits); return false } + catch let error as JSONFailure { return error.resource?.metric == .workUnits } + } + let read = try batch.values(at: [[]]) + return try store.follow(from: start).result.count == 3 && read.result == [store.document.root] + } } + for try await ok in group { #expect(ok) } + } +} + +@Test func workflowS12StorePathsHonorInvocationPointerAndDiagnosticLimits() throws { + let store = try wfStar(), position = try store.record(collection: 0, row: 0).position + for (metric, boundary) in [(JSONResourceMetric.pointerTokens, UInt64(2)), (.pointerBytes, 4), (.diagnosticBytes, 4)] { + var limits = JSONLimits.balanced; limits[metric] = boundary + #expect(try store.value(at: position, limits: limits).result == store.value(at: position).result) + limits[metric] = boundary - 1 + #expect(try wfCore { _ = try store.value(at: position, limits: limits) }.resource?.metric == metric) + } +} +@Test func workflowS13FollowSelfCycleAndLongNarrowChain() throws { + let profile = JSONStoreProfile(version: .v1, collectionsAt: [], collections: [ + .init(name: "u", recordsAt: [.key("u")], idKey: "id", referenceFields: [.init(key: "next", presence: .required, nullable: false)])], + references: .strict(marker: .v1, validateTargets: true)) + func store(_ text: String) throws -> JSONConfiguredStore { try JSONStoreBuilder(document: wfDocument(text), profile: profile).build().result } + let single = try store(#"{"u":[{"id":"0","next":{"collection":"u","id":"0"}}]}"#) + let start = try single.record(collection: 0, row: 0).position + var bounds = JSONStoreFollowLimits(); bounds.maximumDepth = 0; bounds.maximumFrontier = 1 + #expect(try single.follow(from: start, bounds: bounds).result == [start]) + let chain = try store(#"{"u":[{"id":"0","next":{"collection":"u","id":"1"}},{"id":"1","next":{"collection":"u","id":"2"}},{"id":"2","next":{"collection":"u","id":"3"}},{"id":"3","next":{"collection":"u","id":"3"}}]}"#) + let first = try chain.record(collection: 0, row: 0).position + bounds.maximumDepth = 3 + #expect(try chain.follow(from: first, bounds: bounds).result.map(\.row) == [0,1,2,3]) + bounds.maximumDepth = 2 + let failure = try wfStore { _ = try chain.follow(from: first, bounds: bounds) } + #expect(failure.code == .capacityExceeded && failure.capacity == "maximumDepth" && failure.limit == 2) +} + +@Test func workflowB09NestedSnapshotRepeatsQuotasAndLateCancellation() async throws { + let document = try wfDocument(#"{"a/b~":{"":{"é":[1,2],"e\u0301":null}}}"#) + let snapshot = try JSONStructuralSnapshotBuilder(document: document).build().result + // DFS: root0, a/b~1, empty2, é3, 1 at4, 2 at5, decomposed null6. + let ordinals = [5,4,5,6], report = try snapshot.values(for: [5,4,5,6]) + #expect(report.result == [2,1,2,.null] && report.statistics[.retainedBytes] == 160) + var limits = JSONLimits.balanced; limits[.workUnits] = report.statistics[.workUnits] + #expect(try snapshot.values(for: ordinals, limits: limits).result == report.result) + limits[.workUnits] -= 1 + #expect(try wfCore { _ = try snapshot.values(for: ordinals, limits: limits) }.resource?.metric == .workUnits) + let outputs = Mutex(0), lifetime = Mutex(false) + let task = Task { + var inner = JSONLimits.balanced; inner[.cancellationQuantum] = 1 + return try snapshot.valuesForQualification(for: ordinals, limits: inner, outputObserver: { count in + outputs.withLock { $0 = count }; if count == 2 { withUnsafeCurrentTask { $0?.cancel() } } + }, lifetimeObserver: { root, state in lifetime.withLock { $0 = root && state } }) + } + do { _ = try await task.value; Issue.record("Expected late cancellation") } catch is CancellationError {} + #expect(outputs.withLock { $0 } == 2 && lifetime.withLock { $0 }) + #expect(try snapshot.values(for: [5]).result == [2]) +} +@Test func workflowS14MetadataAliasesAndScratchHaveExactRetentionDisposition() throws { + let store = try wfStar(), start = try store.record(collection: 0, row: 0).position + let lookup = try store.lookup(collection: "u", id: "0") + #expect(lookup.statistics[.retainedBytes] == 32) + let follow = try store.follow(from: start) + #expect(follow.statistics[.retainedBytes] == 128) + let values = try store.values(at: [start,start]) + #expect(values.statistics[.retainedBytes] == 96) + #expect(follow.statistics.peakLogicalRetainedBytes >= store.overlayLogicalBytes) + var limits = JSONLimits.balanced; limits[.retainedBytes] = follow.statistics.peakLogicalRetainedBytes + #expect(try store.follow(from: start, limits: limits).result == follow.result) + limits[.retainedBytes] -= 1 + #expect(try wfCore { _ = try store.follow(from: start, limits: limits) }.resource?.metric == .retainedBytes) +} From 5e27899230262a8d61cf0a5fd7d71fed18e105ac Mon Sep 17 00:00:00 2001 From: Thorsten Fuchs <76904008+ThorFuchs@users.noreply.github.com> Date: Wed, 28 Oct 2026 17:16:05 +0100 Subject: [PATCH 2/5] Qualify cross-product workflows and bounded structural capacity --- CHANGELOG.md | 6 + CONTRIBUTING.md | 5 +- .../BoundedWorkflows1.8.md | 17 ++ Documentation/Qualification/Release1.8.md | 64 ++++++ .../StructuralWorkflows1.8Matrix.json | 88 ++++++++ .../Qualification/TestDiscovery.json | 66 +++++- Examples/BoundedWorkflows/Package.swift | 3 + Examples/BoundedWorkflows/README.md | 50 +++++ .../Sources/BoundedWorkflows/Acceptance.swift | 57 +++++ .../Sources/BoundedWorkflows/Capacity.swift | 157 +++++++++++++ .../Sources/BoundedWorkflows/Main.swift | 7 + README.md | 7 + Scripts/ci.py | 44 +++- Scripts/qualify_structural_acceptance.py | 134 +++++++++++ Scripts/test_qualification_harness.py | 56 ++++- Sources/JSONCLIEngine/CLIBuildIdentity.swift | 2 +- .../BoundedWorkflowRecipes.md | 67 ++++++ .../Documentation.docc/JSONInspection.md | 1 + .../CompositionContractTests.swift | 4 +- .../PublicWorkflowTests.swift | 4 +- .../StagedImportTests.swift | 4 +- .../StructuralAcceptanceTests.swift | 209 ++++++++++++++++++ 22 files changed, 1034 insertions(+), 18 deletions(-) create mode 100644 Documentation/Qualification/StructuralWorkflows1.8Matrix.json create mode 100644 Examples/BoundedWorkflows/README.md create mode 100644 Examples/BoundedWorkflows/Sources/BoundedWorkflows/Acceptance.swift create mode 100644 Examples/BoundedWorkflows/Sources/BoundedWorkflows/Capacity.swift create mode 100644 Scripts/qualify_structural_acceptance.py create mode 100644 Sources/JSONInspection/Documentation.docc/BoundedWorkflowRecipes.md create mode 100644 Tests/JSONIntegrationTests/StructuralAcceptanceTests.swift diff --git a/CHANGELOG.md b/CHANGELOG.md index 4f4f047..630926e 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,12 @@ ## Unreleased — 1.8.0-dev +- Extend public acceptance across schema validation, store/transform reads and + checked Patch rebuilds, with exact numeric/scalar identity and lifecycle cases. +- Add a frozen capacity/owned-counter matrix and compiled cross-product recipes; + distinguish diagnostic refusals from completed cells and logical retention from + timing, RSS and allocations. Library APIs/algorithms and default policies unchanged. + - Add ordered typed-path, snapshot-ordinal and store-position value batches using one cumulative native ledger per batch, complete source admission and atomic results. - Promote the original checked configured-store profile/construction inside diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index ba2262c..9171704 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -2,7 +2,10 @@ This isolated development source is 1.8.0-dev from published 1.7.0. Run `python3 Scripts/ci.py --lane structural-workflows` for exact Debug/Release -behavior, preserved store cases and public compiler/consumer checks. Run +behavior, preserved store cases and public compiler/consumer checks. M2 adds +`--lane structural-acceptance` and `--lane structural-acceptance-safety` for exact +public cross-product cases, deterministic capacity observations and their safety +checks. These extend the existing gates. Run `--lane structural-workflows-safety` for scoped ASan, shared-state TSan and strict memory checks; complete native/Linux/docs/WASI gates still apply before release. Do not infer a pass when storage or an actual supported runtime is unavailable. diff --git a/Documentation/Architecture/StructuralIntelligence/BoundedWorkflows1.8.md b/Documentation/Architecture/StructuralIntelligence/BoundedWorkflows1.8.md index a22601d..77c6890 100644 --- a/Documentation/Architecture/StructuralIntelligence/BoundedWorkflows1.8.md +++ b/Documentation/Architecture/StructuralIntelligence/BoundedWorkflows1.8.md @@ -61,3 +61,20 @@ cases. Keep partial profiling/G4 HOLD and the1.7-only P4 exception historical; No persistence, public mutable store, online lookup, new query grammar, lower platform floor, parallel batch execution or wider retrieval belongs to this wave. + +## M2 application acceptance and counter observations + +Schema validation, store construction, reads, transformation and Patch remain +separate public operations with independent reports. A recipe does not attach a +replacement ledger or claim a pipeline-wide budget. Ordinary schema invalidity, +operational quota failure and root removal remain distinct. Patches require a +new source wrapper/store; old positions and source spans do not silently migrate. +Accepted bytes of an awaited sink cannot be recalled after cancellation. + +The authored public examples and Integration behavior cases qualify those +boundaries without changing any engine API. The capacity matrix in +Documentation/Qualification/StructuralWorkflows1.8Matrix.json records fixed +fixtures, exact boundaries and separately reported diagnostic refusals. Its +counter comparisons are not elapsed-time, allocation or RSS evidence. Existing +49 core metrics, twelve construction caps, profile defaults, fixture identities, +performance review trigger and controlled-performance holds are unchanged. diff --git a/Documentation/Qualification/Release1.8.md b/Documentation/Qualification/Release1.8.md index 9768c20..daff11a 100644 --- a/Documentation/Qualification/Release1.8.md +++ b/Documentation/Qualification/Release1.8.md @@ -1,3 +1,67 @@ +# Milestone 2 — cross-product acceptance and capacity observations + +M2 is an additive qualification increment from local checkpoint +`07ed28095dde5e8063570aa6a2436039b5bd43ca`, tree +`5e0961a13fa4b8746854beca23a424db3075ff96`. The sealed M1 archive/receipts remain +unchanged. No public library signature, algorithm, root manifest, default profile +or fixture comparator is changed by this increment. A new guide updates the +existing catalog and therefore requires regenerated CLI source identity. + +The completed local M2 receipts pass all43 Integration functions in Debug/Release +and ASan, the new sharing TSan function and strict test-target compilation. The +existing49-cap public test passes every exact expected probe. Six new public +consumer witnesses/configuration, the original twelve and their ordinary compiler +refusals pass; all16 counter/capacity cells complete with exact boundary proofs. +All13 native DocC catalogs compile with warnings as errors, and1,639 authored +public declarations/parameters are documented. Compared to published1.7.0, only +Inspection has139 additive declarations from M1; no existing declaration fragment +or identity changes in the complete package graph. Native pinned qualification +observes318 syntax cases (287 evaluated successes,31 policy rejections) and2,337 +schema cases (2,300 evaluated successes,37 policy exclusions), with82 authored +semantics,9 budget cases and4,396 deterministic malformed replays, no failures. +The full native/safety/docs CI lanes and platform/runtime/performance gates remain +separate; these local results do not publish or admit1.8. + +This completion note is a documentary successor to the raw receipts. Final private +binding records its one-path delta; the earlier safety receipt also precedes the +ci.py addition of the49-cap native command. Its three safety commands, product/test/ +fixture bytes and flags are unchanged and rechecked by the current exact harness. +Original failures and M1 evidence remain intact. This is scoped agent review and +native qualification, not independent human release approval. + +Nine named public-import Integration tests cover schema/store/transform/Patch +composition, exact bytes/scalar collection names, source-map/owner replacement, +dangling-link refusal, terminal accounting, accepted sink prefixes after cancellation +and independent shared-plan health. Native/portable discovery is now +1,008/992 functions,390 parameter invocations; WASI expects904/353. Integration +has43 functions and12 sharing cases; combined sharing62/61. These are discovery +requirements, not claims that all platform lanes have run on this source. + +The expanded public SwiftPM example adds six cross-product witnesses per +configuration, preserving all twelve M1 witnesses and compiler refusals. The +[capacity matrix](StructuralWorkflows1.8Matrix.json) freezes16 cells and five +independently hashed fixtures. Batches use a fixed eight-record source and +1/8/64/512 requests; follow uses a growing chain and explicit finite bounds. +Exact request, work and retention boundaries are checked. Independent singular +reads report sums of cumulative counts and maxima of invocation retention, never +summed process memory. Diagnostic typed refusals are separate incomplete capacity +observations, not successes. No timing, allocations, RSS or speedup is measured. + +Run `Scripts/ci.py --lane structural-acceptance` and +`--lane structural-acceptance-safety`; full existing native/Linux/safety/docs gates +remain. Actual M2 outcomes belong in source-bound private receipts under +Artifacts/Release1.8M2. Compiler/fixture reconciliation failures are retained. +M2 also executes the existing49-cap public qualification test, preserving its +exact expected outcomes. The full43-function Integration target is checked in +Debug/Release/ASan; its original names/assertions remain. Strict test-target +compilation uses narrowly scoped acknowledgements for three preexisting +UnsafeCurrentTask callback accesses; those handles never escape or cross tasks. +No compiler flag or unsafe production annotation is added. The new cancellation +case instead uses a safe Task handle and checked-continuation rendezvous. + +The M1 record below remains historical. Broad QA/performance/publication gates +remain pending until their actual outcomes are recorded; no1.7 waiver is inherited. + # 1.8 bounded structural workflow qualification Status: implemented local milestone, broader release qualification PARTIAL, on diff --git a/Documentation/Qualification/StructuralWorkflows1.8Matrix.json b/Documentation/Qualification/StructuralWorkflows1.8Matrix.json new file mode 100644 index 0000000..11d69b9 --- /dev/null +++ b/Documentation/Qualification/StructuralWorkflows1.8Matrix.json @@ -0,0 +1,88 @@ +{ + "schema_version": 1, + "profile": "balanced", + "sizes": [ + 1, + 8, + 64, + 512 + ], + "batch_source_records": 8, + "batch_operations": [ + "path-batch", + "snapshot-batch", + "store-values" + ], + "cells": [ + "path-batch-1", + "snapshot-batch-1", + "store-values-1", + "follow-chain-1", + "path-batch-8", + "snapshot-batch-8", + "store-values-8", + "follow-chain-8", + "path-batch-64", + "snapshot-batch-64", + "store-values-64", + "follow-chain-64", + "path-batch-512", + "snapshot-batch-512", + "store-values-512", + "follow-chain-512" + ], + "required_complete": [ + "path-batch-1", + "snapshot-batch-1", + "store-values-1", + "path-batch-8", + "snapshot-batch-8", + "store-values-8", + "path-batch-64", + "snapshot-batch-64", + "store-values-64", + "path-batch-512", + "snapshot-batch-512", + "store-values-512", + "follow-chain-1", + "follow-chain-8" + ], + "diagnostic_capacity": [ + "follow-chain-64", + "follow-chain-512" + ], + "follow_bounds": "depth=n-1; pending frontier=1; visited/results=n; unchanged balanced native and store defaults", + "fixtures": [ + { + "name": "fixed-eight.json", + "records": 8, + "bytes": 608, + "sha256": "f8372702217632eaa0903d11407534f44dd9f59e1c5916b3b2395258500912b7" + }, + { + "name": "chain-1.json", + "records": 1, + "bytes": 62, + "sha256": "8fe8b3ce7bc73d5213c7152637bb3f5a0cc64e3bef04afde0b8f521cab0eaf8c" + }, + { + "name": "chain-8.json", + "records": 8, + "bytes": 608, + "sha256": "f8372702217632eaa0903d11407534f44dd9f59e1c5916b3b2395258500912b7" + }, + { + "name": "chain-64.json", + "records": 64, + "bytes": 5084, + "sha256": "b6c2c1f666a54393b191df5e903b4d3d33a926b7326b2a05717a5de95bcad7f2" + }, + { + "name": "chain-512.json", + "records": 512, + "bytes": 41748, + "sha256": "37472330ac6dff6c5fa6142a0c69e867b548e54098d178a47731df150695d2c9" + } + ], + "policy": "A diagnostic typed resource refusal is an observed incomplete capacity cell, never an evaluated success or a default/joint-capacity claim. No timings, RSS or allocations are measured. Native limits/defaults and >10% performance review trigger are unchanged." +} diff --git a/Documentation/Qualification/TestDiscovery.json b/Documentation/Qualification/TestDiscovery.json index 646ae2f..3251689 100644 --- a/Documentation/Qualification/TestDiscovery.json +++ b/Documentation/Qualification/TestDiscovery.json @@ -6,7 +6,7 @@ "JSONRegexRuntimeTests": 16, "JSONStreamingTests": 30, "JSONPatchTests": 16, - "JSONIntegrationTests": 34, + "JSONIntegrationTests": 43, "JSONHTMLTests": 97, "JSONBridgesTests": 10, "JSONCLITests": 88, @@ -31,7 +31,7 @@ "JSONSchemaTests": 3, "JSONRegexRuntimeTests": 1, "JSONStreamingTests": 16, - "JSONIntegrationTests": 11, + "JSONIntegrationTests": 12, "JSONHTMLTests": 5, "JSONBridgesTests": 1, "JSONCLITests": 1, @@ -78,7 +78,7 @@ "JSONRegexRuntimeTests": 16, "JSONStreamingTests": 30, "JSONPatchTests": 16, - "JSONIntegrationTests": 34, + "JSONIntegrationTests": 43, "JSONHTMLTests": 81, "JSONBridgesTests": 10, "JSONCLITests": 88, @@ -94,7 +94,7 @@ "JSONRegexRuntimeTests": 16, "JSONStreamingTests": 30, "JSONPatchTests": 16, - "JSONIntegrationTests": 34, + "JSONIntegrationTests": 43, "JSONHTMLTests": 81, "JSONBridgesTests": 10, "JSONNavigationTests": 57, @@ -109,7 +109,7 @@ "JSONSchemaTests": 3, "JSONRegexRuntimeTests": 1, "JSONStreamingTests": 16, - "JSONIntegrationTests": 11, + "JSONIntegrationTests": 12, "JSONHTMLTests": 4, "JSONBridgesTests": 1, "JSONCLITests": 1, @@ -174,5 +174,61 @@ "JSONInspectionTests.workflowS13FollowSelfCycleAndLongNarrowChain()", "JSONInspectionTests.workflowB09NestedSnapshotRepeatsQuotasAndLateCancellation()", "JSONInspectionTests.workflowS14MetadataAliasesAndScratchHaveExactRetentionDisposition()" + ], + "structural_acceptance_function_ids": [ + "JSONIntegrationTests.acceptance18A01SchemaStoreTransformKeepsExactGoldenBytes()", + "JSONIntegrationTests.acceptance18A02InvalidSchemaVerdictIsDistinctFromOperationalRefusal()", + "JSONIntegrationTests.acceptance18A03PatchedRebuildHasNewOwnerAndNoInventedSourceMap()", + "JSONIntegrationTests.acceptance18A04DanglingPatchedReferenceCannotPublishStore()", + "JSONIntegrationTests.acceptance18A05ScalarCollectionNamesSurvivePatchAndCrossLinks()", + "JSONIntegrationTests.acceptance18A06TransformFailureStaysTerminalOverStoreInput()", + "JSONIntegrationTests.acceptance18A07RootRemovalIsNotNullOrAnEmptyStore()", + "JSONIntegrationTests.acceptance18A08CancelledWriterKeepsOnlyItsAcceptedPrefix()", + "JSONIntegrationTests.acceptance18SharedPlansKeepIndependentCrossProductHealth()" + ], + "integration_function_ids": [ + "JSONIntegrationTests.acceptance18A01SchemaStoreTransformKeepsExactGoldenBytes()", + "JSONIntegrationTests.acceptance18A02InvalidSchemaVerdictIsDistinctFromOperationalRefusal()", + "JSONIntegrationTests.acceptance18A03PatchedRebuildHasNewOwnerAndNoInventedSourceMap()", + "JSONIntegrationTests.acceptance18A04DanglingPatchedReferenceCannotPublishStore()", + "JSONIntegrationTests.acceptance18A05ScalarCollectionNamesSurvivePatchAndCrossLinks()", + "JSONIntegrationTests.acceptance18A06TransformFailureStaysTerminalOverStoreInput()", + "JSONIntegrationTests.acceptance18A07RootRemovalIsNotNullOrAnEmptyStore()", + "JSONIntegrationTests.acceptance18A08CancelledWriterKeepsOnlyItsAcceptedPrefix()", + "JSONIntegrationTests.acceptance18SharedPlansKeepIndependentCrossProductHealth()", + "JSONIntegrationTests.caughtOperationalFailureCannotComplete()", + "JSONIntegrationTests.composedRetentionAndURIUseSameScope()", + "JSONIntegrationTests.composedWorkThresholdAndOneAdditionalUnit()", + "JSONIntegrationTests.compositionCallerErrorAndReportProvenance()", + "JSONIntegrationTests.compositionCancellationInsideOwnedScansIsSticky()", + "JSONIntegrationTests.compositionCheckedArithmeticAndStickyUnderflow()", + "JSONIntegrationTests.compositionExactSourceAndNativeErrors()", + "JSONIntegrationTests.compositionIndependentScopesCanBeSharedAcrossTasks()", + "JSONIntegrationTests.compositionInputAndCopyAdmissionBeforeAllocation()", + "JSONIntegrationTests.compositionMonotoneAliasesAndNoPlanPollution()", + "JSONIntegrationTests.compositionNestedScopesHaveExplicitIndependentBudgets()", + "JSONIntegrationTests.compositionRestrictionPreservesCapacityHistory()", + "JSONIntegrationTests.consolidationAuthoredInputCannotReopenFailedTransformLedger()", + "JSONIntegrationTests.consolidationAuthoredValuesValidateTransformAndTraverseWithoutScalarLoss()", + "JSONIntegrationTests.qaE2EAnnotationsFromFailedBranchesDoNotAuthorizeProperties()", + "JSONIntegrationTests.qaE2EAwaitedSinkCancellationKeepsOnlyAcceptedBytes()", + "JSONIntegrationTests.qaE2EAwaitedSinkFailureNeverCertifiesAcceptedPrefix()", + "JSONIntegrationTests.qaE2EBuiltinRegistryHasAllPinnedOfflineIdentities()", + "JSONIntegrationTests.qaE2ECumulativeWorkSucceedsAtNAndFailsAtNMinusOne()", + "JSONIntegrationTests.qaE2EExactScalarAndIntegerConstraintsRejectNearMatches()", + "JSONIntegrationTests.qaE2EExactWorkflowHasIndependentGoldenOutput()", + "JSONIntegrationTests.qaE2EMinimizedMalformedReplaysHavePreciseFailures()", + "JSONIntegrationTests.qaE2ENDJSONImportFailuresKeepAcceptedPrefixWithoutCommit()", + "JSONIntegrationTests.qaE2ENDJSONImportStrictFramingCommitsOnlyAtEOF()", + "JSONIntegrationTests.qaE2EOfflineReferenceRequiresExplicitRegistry()", + "JSONIntegrationTests.qaE2EPatchRollbackRootAbsenceAndMergeNullRemainDistinct()", + "JSONIntegrationTests.qaE2ERootImportCancellationAfterProvisionalAcceptanceIsTerminal()", + "JSONIntegrationTests.qaE2ERootImportCommitsOnlyAfterCertifiedCompletion()", + "JSONIntegrationTests.qaE2ERootImportMalformedTailDoesNotCommitAcceptedElements()", + "JSONIntegrationTests.qaE2ERootImportQuotaStopsBeforeCommitAndFurtherPulls()", + "JSONIntegrationTests.qaE2ESeededModelsMatchOrderedMemberAndIntegerOracles()", + "JSONIntegrationTests.qaE2ESharedPlanRecoversAfterIndependentFailures()", + "JSONIntegrationTests.qaE2EUnknownEnvelopeVersionIsCorruption()", + "JSONIntegrationTests.qaE2EVersionOneEnvelopePreservesExactInnerState()" ] } diff --git a/Examples/BoundedWorkflows/Package.swift b/Examples/BoundedWorkflows/Package.swift index 62eaba2..6fb8b27 100644 --- a/Examples/BoundedWorkflows/Package.swift +++ b/Examples/BoundedWorkflows/Package.swift @@ -7,6 +7,9 @@ let package = Package( dependencies: [.package(name: "Swift-JSON", path: "../..")], targets: [.executableTarget(name: "BoundedWorkflows", dependencies: [ .product(name: "JSON", package: "Swift-JSON"), + .product(name: "JSONSchema", package: "Swift-JSON"), + .product(name: "JSONPatch", package: "Swift-JSON"), + .product(name: "JSONTransform", package: "Swift-JSON"), .product(name: "JSONNavigation", package: "Swift-JSON"), .product(name: "JSONInspection", package: "Swift-JSON") ])], swiftLanguageModes: [.v6] diff --git a/Examples/BoundedWorkflows/README.md b/Examples/BoundedWorkflows/README.md new file mode 100644 index 0000000..c052061 --- /dev/null +++ b/Examples/BoundedWorkflows/README.md @@ -0,0 +1,50 @@ +# Bounded structural workflows + +A public SwiftPM consumer for the unreleased 1.8 source. It imports JSON, +JSONSchema, JSONNavigation, JSONInspection, JSONPatch and JSONTransform; no private +meter/runtime module or external dependency is used. Swift6.4/language6 and the +supported platform baselines still apply. Released1.7 does not supply the new APIs. + +From the repository root: + +```sh +swift run --package-path Examples/BoundedWorkflows BoundedWorkflows +swift run -c release --package-path Examples/BoundedWorkflows BoundedWorkflows +``` + +The twelve original witnesses preserve exact values, request order/repetition, +missing versus null, source-local ownership, bounded follow and public reports. +Six additional witnesses compose native schema validation, source interpretation, +transformation and Patch. They check a golden exact-decimal byte sequence, a new +owner/source wrapper after Patch, ordinary invalidity versus operational refusal, +root removal versus null, and terminal transform completion after a caught error. +The separate Swift Testing Integration cases also exercise declared-link corruption, +scalar-distinct collection names, awaited-sink cancellation and concurrent plans. + +Each stage is its own native operation, not one implicit global pipeline ledger. +Caller array assembly and callbacks are ordinary caller work. Validate a schema +against its explicit offline registry/selected root before interpreting a source; +the store does not infer schema requiredness. Rebuild after Patch and obtain new +positions. Atomic publication into a sink is a caller staging decision. + +## Counter and capacity observations + +```sh +swift run -c release --package-path Examples/BoundedWorkflows BoundedWorkflows --capacity /tmp/swift-json-capacity +``` + +Use a new output directory. This writes cells.json and five deterministic fixture +files. The frozen matrix is in Documentation/Qualification/StructuralWorkflows1.8Matrix.json. +The qualification script independently verifies fixture hashes, cell identities, +expected results and exact/one-below request/work/retention boundaries. +Batch cells use1/8/64/512 requests against eight records. Follow cells use growing +chains with frontier1 and explicit depth/visited/results bounds; native balanced +and store construction defaults stay unchanged. A typed diagnostic refusal is +recorded as an incomplete capacity observation, never a successful closure. + +Independent singular invocations report summed work/admission counters and the +maximum of their per-invocation retention. A batch reports its one original ledger. +Those quantities do not measure elapsed time, heap allocations, process RSS, +caller-held copies or jointly achievable default maxima. No performance improvement +or policy change follows from the counter experiment. The controlled native +performance gate remains a separate final-source requirement. diff --git a/Examples/BoundedWorkflows/Sources/BoundedWorkflows/Acceptance.swift b/Examples/BoundedWorkflows/Sources/BoundedWorkflows/Acceptance.swift new file mode 100644 index 0000000..caccc6d --- /dev/null +++ b/Examples/BoundedWorkflows/Sources/BoundedWorkflows/Acceptance.swift @@ -0,0 +1,57 @@ +// SPDX-License-Identifier: Apache-2.0 +import Foundation +import JSON +import JSONSchema +import JSONPatch +import JSONInspection +import JSONTransform + +// Authored application recipe. These stages are separate native operations; +// caller array assembly is not a library-wide instruction or memory quota. +func structuralAcceptance() throws { + func witness(_ name: String, _ condition: Bool) { + precondition(condition, name); print("ACCEPTANCE18_CASE \(name) passed") + } + let source = try JSONParser().parseDocument(Data(#"{"u":[{"id":"é","amount":9007199254740993,"next":{"collection":"u","id":"e\u0301"}},{"id":"e\u0301","amount":0.10000000000000001,"next":null}]}"#.utf8)) + let schema = try JSONSchemaCompiler().compile(JSONSchemaDocument(JSONParser().parse( + #"{"type":"object","required":["u"],"properties":{"u":{"type":"array","items":{"type":"object","required":["id","amount","next"],"properties":{"id":{"type":"string"},"amount":{"type":"number"},"next":{"type":["object","null"]}}}}}}"#))) + let validation = try JSONSchemaValidator().validate(source.root, against: schema) + precondition(validation.isValid) + let profile = JSONStoreProfile(version: .v1, collectionsAt: [], collections: [ + .init(name: "u", recordsAt: [.key("u")], idKey: "id", referenceFields: [.init(key: "next", presence: .required, nullable: true)]) + ], references: .strict(marker: .v1, validateTargets: true)) + let store = try JSONStoreBuilder(document: source, profile: profile).build().result + guard let first = try store.lookup(collection: "u", id: "é").result else { preconditionFailure() } + let values = try store.values(at: store.follow(from: first).result) + let transform = try JSONTransformProgram.compile(".[] | .amount") + let output = try transform.withExecution(input: .array(values.result)) { execution in + var output: [JSON] = [] + while let value = try execution.next() { output.append(value.value) } + return JSON.array(output) + } + witness("schema-store-transform", try JSONWriter(options: .init(numberStyle: .preserved)).write(output.result) == Data("[9007199254740993,0.10000000000000001]".utf8)) + let patch = try JSONPatch(operations: [.replace(path: JSONPointer(parsing: "/u/1/amount"), value: .number(JSONNumber(parsing: "1.25")))]) + guard case .document(let changed) = try patch.applying(to: source.root) else { preconditionFailure() } + let document = source.replacingRoot(changed), rebuilt = try JSONStoreBuilder(document: document, profile: profile).build().result + do { _ = try rebuilt.value(at: first); preconditionFailure() } + catch let error as JSONStoreFailure { witness("patched-owner", error.code == .invalidPosition) } + witness("source-map-reset", document.sourceMap == nil && document.sourceBytes == nil && store.document.sourceMap != nil) + let invalid = try JSONSchemaValidator().validate(.null, against: schema) + var native = JSONLimits.balanced; native[.evaluationWorkUnits] = 0 + do { _ = try JSONSchemaValidator(options: .init(limits: native)).validate(source.root, against: schema); preconditionFailure() } + catch let error as JSONFailure { witness("schema-refusal", !invalid.isValid && error.resource?.metric == .evaluationWorkUnits) } + let rootPointer = try JSONPointer(parsing: "") + let removed = try JSONPatch(operations: [.remove(path: rootPointer)]).applying(to: source.root) + let null = try JSONPatch(operations: [.replace(path: rootPointer, value: .null)]).applying(to: source.root) + witness("root-removal", removed == .removed && null == .document(.null) && removed != null) + var limits = JSONTransformLimits(); limits[.results] = 1 + do { + _ = try transform.withExecution(input: .array(values.result), limits: limits) { execution in + let yielded = try execution.next(); precondition(yielded != nil) + do { _ = try execution.next(); preconditionFailure() } + catch let error as JSONTransformFailure { precondition(error.category == .resource && error.metric == "results") } + } + preconditionFailure("A caught failure cannot reopen successful completion") + } catch let error as JSONTransformFailure { witness("terminal-transform", error.category == .resource && error.metric == "results") } + print("ACCEPTANCE18 passed cases=6") +} diff --git a/Examples/BoundedWorkflows/Sources/BoundedWorkflows/Capacity.swift b/Examples/BoundedWorkflows/Sources/BoundedWorkflows/Capacity.swift new file mode 100644 index 0000000..de0a8bd --- /dev/null +++ b/Examples/BoundedWorkflows/Sources/BoundedWorkflows/Capacity.swift @@ -0,0 +1,157 @@ +// SPDX-License-Identifier: Apache-2.0 +import Foundation +import JSON +import JSONInspection + +private struct CapacityCounters: Encodable { + let work: UInt64, nodes: UInt64, retained: UInt64, peak: UInt64 + init(_ statistics: JSONOperationStatistics) { + work = statistics[.workUnits]; nodes = statistics[.nodes] + retained = statistics[.retainedBytes]; peak = statistics.peakLogicalRetainedBytes + } +} +private struct CapacitySingles: Encodable { + let workSum: UInt64, nodesSum: UInt64 + let maximumInvocationRetained: UInt64, maximumInvocationPeak: UInt64 +} +private struct CapacityCell: Encodable { + let id: String, operation: String, requests: Int, fixture: String, outcome: String + let report: CapacityCounters?, independentSingles: CapacitySingles? + let requestBoundary: Bool?, workBoundary: Bool?, retentionBoundary: Bool? + let errorMetric: String?, errorLimit: UInt64?, errorAttempted: UInt64?, errorStage: String? +} +private struct CapacityReceipt: Encodable { + let schemaVersion = 1 + let profile = "balanced" + let scope = "Owned-work and logical retention observations; no elapsed-time, allocation, RSS or throughput measurement. Fixture construction, expected-value comparison and caller array copies are caller work." + let cells: [CapacityCell] +} +private enum CapacityInvariant: Error { case result, boundary, overflow } +private func capacitySum(_ lhs: UInt64, _ rhs: UInt64) throws -> UInt64 { + let (result, overflow) = lhs.addingReportingOverflow(rhs) + guard !overflow else { throw CapacityInvariant.overflow }; return result +} +private func capacityProfile() -> JSONStoreProfile { + .init(version: .v1, collectionsAt: [], collections: [ + .init(name: "records", recordsAt: [.key("records")], idKey: "id", referenceFields: [.init(key: "next", presence: .required, nullable: true)]) + ], references: .strict(marker: .v1, validateTargets: true)) +} +private func capacityFixture(_ count: Int) throws -> Data { + var records: [JSON] = [] + for index in 0.. JSONOperationReport) throws -> CapacitySingles { + var work: UInt64 = 0, nodes: UInt64 = 0, peak: UInt64 = 0, retained: UInt64 = 0 + for index in 0.. Void, metric: JSONResourceMetric) throws { + do { try body(); throw CapacityInvariant.boundary } + catch let error as JSONFailure { guard error.resource?.metric == metric else { throw error } } +} +private func capacityBoundaries(_ report: JSONOperationReport, _ body: (JSONLimits) throws -> JSONOperationReport) throws { + var limits = JSONLimits.balanced + limits[.workUnits] = report.statistics[.workUnits] + limits[.retainedBytes] = report.statistics.peakLogicalRetainedBytes + _ = try body(limits) + limits[.workUnits] -= 1 + try capacityResource({ _ = try body(limits) }, metric: .workUnits) + limits = .balanced; limits[.retainedBytes] = report.statistics.peakLogicalRetainedBytes - 1 + try capacityResource({ _ = try body(limits) }, metric: .retainedBytes) +} + +func structuralCapacity(output: URL) throws { + try FileManager.default.createDirectory(at: output, withIntermediateDirectories: true) + let fixtureDirectory = output.appendingPathComponent("fixtures") + try FileManager.default.createDirectory(at: fixtureDirectory, withIntermediateDirectories: true) + let fixedBytes = try capacityFixture(8) + try fixedBytes.write(to: fixtureDirectory.appendingPathComponent("fixed-eight.json")) + let document = try JSONParser().parseDocument(fixedBytes) + let snapshot = try JSONStructuralSnapshotBuilder(document: document).build().result + let store = try JSONStoreBuilder(document: document, profile: capacityProfile()).build().result + var amountOrdinals: [Int] = [] + for ordinal in 0.. JSONOperationReport<[JSON]> { + switch operation { + case "path-batch": return try JSONInspectionBatch(document: document, limits: limits).values(at: paths, maximumRequests: maximum) + case "snapshot-batch": return try snapshot.values(for: ordinals, maximumRequests: maximum, limits: limits) + default: return try store.values(at: positions, maximumRequests: maximum, limits: limits) + } + } + let report = try batch(.balanced, maximum: count) + guard report.result.count == count else { throw CapacityInvariant.result } + for (index, value) in report.result.enumerated() { + let observed = operation == "store-values" ? try value.requiredMember("amount") : value + guard observed == expected[index % 8] else { throw CapacityInvariant.result } + } + let singles = try sumSingles(count) { index in + switch operation { + case "path-batch": return try document.readWithReport { try $0["records"][index % 8]["amount"].requiredValue() } + case "snapshot-batch": return try snapshot.value(for: ordinals[index]) + default: return try store.value(at: positions[index]) + } + } + guard singles.nodesSum == report.statistics[.nodes] * UInt64(count) else { throw CapacityInvariant.result } + do { _ = try batch(.balanced, maximum: count - 1); throw CapacityInvariant.boundary } + catch let error as JSONInspectionBatchFailure { + guard error == .requestLimitExceeded(limit: count - 1, actual: count) else { throw error } + } + try capacityBoundaries(report) { try batch($0, maximum: count) } + cells.append(.init(id: operation + "-" + String(count), operation: operation, requests: count, fixture: "fixed-eight.json", outcome: "completed", + report: .init(report.statistics), independentSingles: singles, requestBoundary: true, workBoundary: true, retentionBoundary: true, + errorMetric: nil, errorLimit: nil, errorAttempted: nil, errorStage: nil)) + } + let name = "chain-" + String(count) + ".json", fixture = try capacityFixture(count) + try fixture.write(to: fixtureDirectory.appendingPathComponent(name)) + func refusal(_ error: JSONFailure, stage: String) throws -> CapacityCell { + guard error.category == .resource, let resource = error.resource else { throw error } + return .init(id: "follow-chain-" + String(count), operation: "follow-chain", requests: count, fixture: name, outcome: "operational_error", + report: nil, independentSingles: nil, requestBoundary: nil, workBoundary: nil, retentionBoundary: nil, + errorMetric: resource.metric.rawValue, errorLimit: resource.limit, errorAttempted: resource.attemptedUsage, errorStage: stage) + } + let chain: JSONConfiguredStore + do { chain = try JSONStoreBuilder(document: JSONParser().parseDocument(fixture), profile: capacityProfile()).build().result } + catch let error as JSONFailure { cells.append(try refusal(error, stage: "construction")); continue } + let start = try chain.record(collection: 0, row: 0).position + var bounds = JSONStoreFollowLimits(); bounds.maximumDepth = count - 1; bounds.maximumFrontier = 1 + bounds.maximumVisited = count; bounds.maximumResults = count + let report: JSONOperationReport<[JSONStorePosition]> + do { report = try chain.follow(from: start, bounds: bounds) } + catch let error as JSONFailure { cells.append(try refusal(error, stage: "follow")); continue } + guard report.result.map(\.row) == Array(0..= report['retained'] and all(row.get(k) is True for k in ['requestBoundary','workBoundary','retentionBoundary']) + and all(row.get(k) is None for k in ['errorMetric','errorLimit','errorAttempted','errorStage'])): + return False + singles = row.get('independentSingles') + if operation == 'follow-chain': + if singles is not None: return False + elif not (isinstance(singles, dict) and all(finite(singles.get(k), positive=k in ('workSum','nodesSum','maximumInvocationPeak')) + for k in ['workSum','nodesSum','maximumInvocationRetained','maximumInvocationPeak']) + and singles['nodesSum'] == size * report['nodes'] and singles['maximumInvocationPeak'] >= singles['maximumInvocationRetained']): + return False + elif row.get('outcome') == 'operational_error' and row['id'] in MATRIX['diagnostic_capacity']: + if not (row.get('report') is None and row.get('independentSingles') is None + and all(row.get(k) is None for k in ['requestBoundary','workBoundary','retentionBoundary']) + and row.get('errorMetric') in METRICS + and finite(row.get('errorLimit')) and finite(row.get('errorAttempted'), positive=True) + and row['errorAttempted'] > row['errorLimit'] and row.get('errorStage') in ('construction','follow')): + return False + else: + return False + return all(next(r for r in rows if r['id'] == name)['outcome'] == 'completed' for name in MATRIX['required_complete']) + +def validate_fixtures(directory): + expected = {x['name']: x for x in MATRIX['fixtures']} + actual = sorted(directory.glob('*.json')) + if {p.name for p in actual} != set(expected) or any(p.is_symlink() for p in actual): + raise ValueError('Changed capacity fixture inventory') + records = [] + for path in actual: + data = path.read_bytes(); record = dict(name=path.name, bytes=len(data), sha256=hashlib.sha256(data).hexdigest()) + if any(record[k] != expected[path.name][k] for k in ('bytes','sha256')): + raise ValueError('Changed capacity fixture bytes: ' + path.name) + records.append(record) + return records + +def check_receipt(data): + commands = data.get('commands', []) + expected = {'consumer','debug-binpath','debug-guide','release-binpath','release-guide','capacity'} + fixtures = data.get('fixtures', []) + return (data.get('status') == 'passed' and data.get('source_inputs_unchanged') is True + and data.get('matrix_sha256') == hashlib.sha256(MATRIX_PATH.read_bytes()).hexdigest() + and len(commands) == len(expected) and {c.get('name') for c in commands} == expected + and all(c.get('exit') == 0 and c.get('compiler_crash') is False for c in commands) + and set(data.get('consumer_results', {})) == {'debug','release'} + and all(len(v) == len(CASES) and set(v) == CASES for v in data['consumer_results'].values()) + and validate_capacity(data.get('capacity', {})) + and data.get('completed_cells') == sum(r['outcome'] == 'completed' for r in data['capacity']['cells']) + and data.get('operational_refusals') == sum(r['outcome'] == 'operational_error' for r in data['capacity']['cells']) + and data.get('performance_status', '').startswith('NOT MEASURED:') + and len(fixtures) == len(MATRIX['fixtures']) and sorted(fixtures, key=lambda x:x['name']) == sorted( + [{k:x[k] for k in ('name','bytes','sha256')} for x in MATRIX['fixtures']], key=lambda x:x['name'])) + +def main(): + parser = argparse.ArgumentParser(); parser.add_argument('--output', required=True) + args = parser.parse_args(); out = ROOT / args.output; out.mkdir(parents=True, exist_ok=True) + receipt = dict(status='running', source_inputs=[dict(path=p.relative_to(ROOT).as_posix(), sha256=hashlib.sha256(p.read_bytes()).hexdigest()) for p in files()], + matrix_sha256=hashlib.sha256(MATRIX_PATH.read_bytes()).hexdigest(), commands=[], consumer_results={}) + def save(): (out/'receipt.json').write_text(json.dumps(receipt, indent=2)+'\n') + def run(name, command): + result = subprocess.run(command, cwd=ROOT, capture_output=True, text=True, timeout=1200) + log = out/(name+'.log'); log.write_text(result.stdout+result.stderr) + crash = (result.returncode < 0 and command[0] == 'swiftc') or any(x in log.read_text() for x in ['Stack dump:', 'Assertion failed', 'Found ownership error', 'fatal error encountered during compilation']) + receipt['commands'].append(dict(name=name, command=command, exit=result.returncode, compiler_crash=crash, log=str(log.relative_to(ROOT)), log_sha256=hashlib.sha256(log.read_bytes()).hexdigest())) + save(); print(name, result.returncode, flush=True) + if result.returncode != 0 or crash: raise ValueError(name) + return result.stdout + try: + child = out/'public-consumer' + run('consumer', ['python3','-B','Scripts/qualify_structural_workflows.py','--output',str(child.relative_to(ROOT))]) + original = json.loads((child/'receipt.json').read_text()) + if not check_original_consumer(original): raise ValueError('Original public consumer/compiler boundary failed') + for config in ['debug','release']: + log = child/(config+'-consumer.log') + cases = re.findall(r'^ACCEPTANCE18_CASE (\S+) passed$', log.read_text(), re.M) + if len(cases) != 6 or set(cases) != CASES or len(re.findall(r'^ACCEPTANCE18 passed cases=6$', log.read_text(), re.M)) != 1: + raise ValueError('Incomplete public cross-product witnesses') + receipt['consumer_results'][config] = cases + bp = pathlib.Path(run(config+'-binpath', ['swift','build','-c',config,'--package-path','Examples/BoundedWorkflows','--scratch-path','.build/workflow-consumer','--show-bin-path']).strip()) + modules = bp/'Modules' if (bp/'Modules').exists() else bp + guide = ROOT/'Sources/JSONInspection/Documentation.docc/BoundedWorkflowRecipes.md' + blocks = re.findall(r'```swift\n(.*?)\n```', guide.read_text(), re.S) + if len(blocks) != 1: raise ValueError('Changed recipe snippet inventory') + source = out/'guide.swift'; source.write_text(blocks[0]+'\n') + run(config+'-guide', ['swiftc','-swift-version','6','-warnings-as-errors','-I',str(modules),'-Onone' if config == 'debug' else '-O', + '-parse-as-library','-c',str(source),'-o',str(out/(config+'-guide.o'))]) + capacity = out/'capacity' + run('capacity', ['swift','run','-c','release','--jobs','2','--package-path','Examples/BoundedWorkflows','--scratch-path','.build/workflow-consumer','BoundedWorkflows','--capacity',str(capacity)]) + data = json.loads((capacity/'cells.json').read_text()) + if not validate_capacity(data): raise ValueError('Changed/incomplete capacity observations or boundary proof') + receipt['fixtures'] = validate_fixtures(capacity/'fixtures') + receipt['capacity'] = data + receipt['completed_cells'] = sum(r['outcome'] == 'completed' for r in data['cells']) + receipt['operational_refusals'] = sum(r['outcome'] == 'operational_error' for r in data['cells']) + receipt['capacity_status'] = 'observed_fixture_grid_complete' if not receipt['operational_refusals'] else 'partial_diagnostic_capacity' + receipt['performance_status'] = 'NOT MEASURED: owned counters are not timings, heap allocations, RSS or throughput' + receipt['status'] = 'passed' + except (OSError, ValueError, subprocess.TimeoutExpired) as error: + receipt.update(status='failed', error=str(error)) + receipt['source_inputs_unchanged'] = receipt['source_inputs'] == [dict(path=p.relative_to(ROOT).as_posix(),sha256=hashlib.sha256(p.read_bytes()).hexdigest()) for p in files()] + if not receipt['source_inputs_unchanged']: receipt['status'] = 'failed' + if receipt['status'] == 'passed' and not check_receipt(receipt): receipt.update(status='failed', error='Incomplete final acceptance receipt') + save(); return 0 if receipt['status'] == 'passed' else 1 + +if __name__ == '__main__': raise SystemExit(main()) diff --git a/Scripts/test_qualification_harness.py b/Scripts/test_qualification_harness.py index d766277..254b3e7 100644 --- a/Scripts/test_qualification_harness.py +++ b/Scripts/test_qualification_harness.py @@ -28,6 +28,54 @@ def store_markers(): 'STORE_RUNTIME R%02d passed' % n for n in range(1,11)]) class HarnessTests(unittest.TestCase): + def test_acceptance18_discovery_preserves_exact_public_pipeline_identities(self): + names = DISCOVERY['structural_acceptance_function_ids'] + self.assertEqual(len(names), 9) + self.assertEqual(len(DISCOVERY['integration_function_ids']), 43) + events = dict(bundle_counts=[9], module_function_counts={'JSONIntegrationTests':9}, + function_names=names, parameter_case_invocations=0) + command = ['swift','test','--test-product','JSONIntegrationTests','--filter','acceptance18'] + self.assertTrue(check_result(command, 0, '', events)[0]) + full = dict(events, bundle_counts=[43], module_function_counts={'JSONIntegrationTests':43}, function_names=DISCOVERY['integration_function_ids']) + full_command = ['swift','test','--test-product','JSONIntegrationTests','--filter','JSONIntegrationTests'] + self.assertTrue(check_result(full_command, 0, '', full)[0]) + self.assertFalse(check_result(full_command, 0, '', dict(full,function_names=full['function_names'][:-1]))[0]) + for broken in [dict(events, function_names=names[:-1]), dict(events, function_names=names+names[:1]), + dict(events, module_function_counts={'JSONInspectionTests':9}), dict(events, parameter_case_invocations=1)]: + self.assertFalse(check_result(command, 0, '', broken)[0]) + + def test_capacity18_does_not_convert_refusals_or_incomplete_boundaries_to_passes(self): + from qualify_structural_acceptance import MATRIX, validate_capacity, validate_fixtures + cells = [] + for identifier in MATRIX['cells']: + operation, count = identifier.rsplit('-',1); count=int(count) + cells.append(dict(id=identifier, operation=operation, requests=count, + fixture='chain-%d.json' % count if operation=='follow-chain' else 'fixed-eight.json', + outcome='completed', report=dict(work=1,nodes=1,retained=32,peak=32), + independentSingles=None if operation=='follow-chain' else dict(workSum=count,nodesSum=count,maximumInvocationRetained=32,maximumInvocationPeak=32), + requestBoundary=True,workBoundary=True,retentionBoundary=True)) + data=dict(schemaVersion=1,profile='balanced',cells=cells) + self.assertTrue(validate_capacity(data)) + refused=copy.deepcopy(data);refused['cells'][-1]=dict(refused['cells'][-1],outcome='operational_error',report=None, + requestBoundary=None,workBoundary=None,retentionBoundary=None,errorMetric='workUnits',errorLimit=1,errorAttempted=2,errorStage='construction') + self.assertTrue(validate_capacity(refused)) + for mode in ['missing','duplicate','fake-boundary','bad-single-count','required-refusal','fake-metric','refusal-as-success','missing-report']: + bad=copy.deepcopy(refused) + if mode=='missing':bad['cells'].pop() + elif mode=='duplicate':bad['cells'][-1]=bad['cells'][0] + elif mode=='fake-boundary':bad['cells'][0]['workBoundary']=False + elif mode=='bad-single-count':bad['cells'][0]['independentSingles']['nodesSum']=2 + elif mode=='required-refusal':bad['cells'][3]=dict(bad['cells'][-1],id='follow-chain-1',requests=1,fixture='chain-1.json') + elif mode=='fake-metric':bad['cells'][-1]['errorMetric']='unrelated' + elif mode=='refusal-as-success':bad['cells'][-1]['outcome']='completed' + else:bad['cells'][0]['report']=None + self.assertFalse(validate_capacity(bad)) + with tempfile.TemporaryDirectory() as temporary: + directory=pathlib.Path(temporary) + with self.assertRaisesRegex(ValueError,'inventory'):validate_fixtures(directory) + for fixture in MATRIX['fixtures']:(directory/fixture['name']).write_bytes(b'corrupt') + with self.assertRaisesRegex(ValueError,'bytes'):validate_fixtures(directory) + def test_workflow_discovery_requires_all_exact_identities(self): names = DISCOVERY['workflow_function_ids'] self.assertEqual(len(names), 24) @@ -195,12 +243,12 @@ def test_composition_discovery_and_capacity_limits_are_not_blanket_successes(sel self.assertEqual(policy[key]['JSONNavigationTests'], 57) self.assertEqual(policy[key]['JSONSchemaExamplesTests'], 98) self.assertEqual(policy[key]['JSONInspectionTests'], 214) - self.assertEqual(policy[key]['JSONIntegrationTests'], 34) + self.assertEqual(policy[key]['JSONIntegrationTests'], 43) for key in ('shared_state_functions', 'portable_shared_state_functions'): self.assertEqual(policy[key]['JSONNavigationTests'], 2) self.assertEqual(policy[key]['JSONSchemaExamplesTests'], 4) self.assertEqual(policy[key]['JSONInspectionTests'], 11) - self.assertEqual(policy[key]['JSONIntegrationTests'], 11) + self.assertEqual(policy[key]['JSONIntegrationTests'], 12) def test_private_projection_identities_cannot_be_missing_renamed_or_duplicated(self): names = sorted(PROJECTION_FUNCTIONS) @@ -391,8 +439,8 @@ def test_exact_discovery_rejects_missing_duplicate_and_changed_parameters(self): identity['function_names'][0] = 'JSONTests.unrelatedReplacement()' self.assertFalse(check_restored(identity)) harness = ['python3', '-B', 'Scripts/test_qualification_harness.py'] - self.assertTrue(check_result(harness, 0, 'HARNESS_TESTS passed cases=28')[0]) - for stale in ['HARNESS_TESTS passed cases=10', 'HARNESS_TESTS passed cases=11', 'HARNESS_TESTS passed cases=12', 'HARNESS_TESTS passed cases=13', 'HARNESS_TESTS passed cases=14', 'HARNESS_TESTS passed cases=15', 'HARNESS_TESTS passed cases=18', 'HARNESS_TESTS passed cases=19', 'HARNESS_TESTS passed cases=20', 'HARNESS_TESTS passed cases=21', 'HARNESS_TESTS passed cases=25', 'HARNESS_TESTS passed cases=26', '']: + self.assertTrue(check_result(harness, 0, 'HARNESS_TESTS passed cases=30')[0]) + for stale in ['HARNESS_TESTS passed cases=10', 'HARNESS_TESTS passed cases=11', 'HARNESS_TESTS passed cases=12', 'HARNESS_TESTS passed cases=13', 'HARNESS_TESTS passed cases=14', 'HARNESS_TESTS passed cases=15', 'HARNESS_TESTS passed cases=18', 'HARNESS_TESTS passed cases=19', 'HARNESS_TESTS passed cases=20', 'HARNESS_TESTS passed cases=21', 'HARNESS_TESTS passed cases=25', 'HARNESS_TESTS passed cases=26', 'HARNESS_TESTS passed cases=28', '']: self.assertFalse(check_result(harness, 0, stale)[0]) lifetime = ['python3', '-B', 'Scripts/qualify_schema_example_lifetime.py'] marker = 'SCHEMA_EXAMPLES_CORPUS passed mode=owned scopes=128 values=6 exact=1' diff --git a/Sources/JSONCLIEngine/CLIBuildIdentity.swift b/Sources/JSONCLIEngine/CLIBuildIdentity.swift index 8af8093..2c10613 100644 --- a/Sources/JSONCLIEngine/CLIBuildIdentity.swift +++ b/Sources/JSONCLIEngine/CLIBuildIdentity.swift @@ -2,6 +2,6 @@ // Generated by Scripts/generate_cli_identity.py; excludes this file. // SHA-256 of the ordered Package.swift + Sources content manifest, not a Git commit. package enum CLIBuildIdentity { - package static let revision = "sha256:43023670d6664373702741d960b010494d86149aaa181b4eeb9d6ff13701d60e" + package static let revision = "sha256:d29a80776f26331bf75b98c24d5df1cf5b69ecfbc4d9e39426d393e23c02c3ec" package static let kind = "source-sha256" } diff --git a/Sources/JSONInspection/Documentation.docc/BoundedWorkflowRecipes.md b/Sources/JSONInspection/Documentation.docc/BoundedWorkflowRecipes.md new file mode 100644 index 0000000..f559e8f --- /dev/null +++ b/Sources/JSONInspection/Documentation.docc/BoundedWorkflowRecipes.md @@ -0,0 +1,67 @@ +# Combine checked structural workflows + +Compose existing authorities while preserving exact values and each operation's limits. + +## Validate, then interpret records + +JSONSchema owns schema semantics. A valid verdict does not construct a store; +JSONInspection separately checks the explicit collection, ID and link profile. +An ordinary false verdict is a completed evaluation. A thrown operational error +must not be converted to false or an empty store. + +```swift +import JSON +import JSONSchema +import JSONInspection + +func validatedStore(document: JSONDocument, schema: CompiledJSONSchema, + profile: JSONStoreProfile) throws -> JSONConfiguredStore? { + let validation = try JSONSchemaValidator().validateWithReport(document.root, against: schema) + guard validation.result.isValid else { return nil } // Application's explicit invalid-input result. + return try JSONStoreBuilder(document: document, profile: profile).build().result +} +``` + +Compile the schema against an explicit immutable offline registry. If a schema +contains only definitions, select the intended definition before validation; +the store does not invent a schema root or infer required properties. + +## Read, transform and write + +The executable `Examples/BoundedWorkflows` validates an exact source, constructs +a store, follows declared references, batch-reads the closure, runs the existing +JSONTransform compiler/VM and writes an independent byte-level golden result. +`9007199254740993` and `0.10000000000000001` never pass through Double. +Caller collection assembly between stages is ordinary caller work. Each public +stage has its own bounded report; this recipe does not create one pipeline-wide +ledger or a caller-instruction/RSS bound. Within a batch/follow/transform invocation, +all nested owned work still uses the original cumulative ledger. + +A caught transform resource failure remains terminal for that execution. It is +not EOF, JSON null or schema invalidity. A later operation on the immutable source +has independent health. With an awaited streaming sink, already accepted bytes +remain accepted after cancellation; stage output in the caller if publication +must be atomic. + +## Patch and rebuild deliberately + +Apply JSONPatch to the original canonical value, then use +`JSONDocument.replacingRoot(_:)` to capture the result. The replacement has no +invented original source bytes/spans. Rebuild its store and obtain new positions; +old owner-bound tokens are refused even when most records are unchanged. A patch +which creates a dangling declared link cannot publish a successful checked store. +Root removal is `JSONPatchResult.removed`, distinct from a present JSON null. + +## Interpret capacity evidence + +The public consumer's `--capacity OUTPUT_DIRECTORY` mode uses fixed 1/8/64/512 +request cells and small growing chains, unchanged balanced native limits and +explicit finite follow bounds. It checks exact work/retention and request boundaries, +compares independently admitted reads and writes fixture bytes for independent +hash verification. Logical counters are not heap allocations, RSS or elapsed time. +A diagnostic resource refusal remains an incomplete capacity observation. These +fixtures do not prove the default maxima jointly achievable or a speed improvement. + +The existing developer kit stays pinned to its published library version. This +1.8 candidate recipe is compiled separately; it does not silently change that kit's +compatibility or distribution claims. diff --git a/Sources/JSONInspection/Documentation.docc/JSONInspection.md b/Sources/JSONInspection/Documentation.docc/JSONInspection.md index ad9572a..405526a 100644 --- a/Sources/JSONInspection/Documentation.docc/JSONInspection.md +++ b/Sources/JSONInspection/Documentation.docc/JSONInspection.md @@ -61,6 +61,7 @@ or numeric matrix. Sampling does not exempt any source from admission. ### Bounded structural workflows (1.8 candidate) - +- - ``JSONInspectionBatch`` - ``JSONInspectionBatchFailure`` - ``JSONStoreBuilder`` diff --git a/Tests/JSONIntegrationTests/CompositionContractTests.swift b/Tests/JSONIntegrationTests/CompositionContractTests.swift index c650339..caa665b 100644 --- a/Tests/JSONIntegrationTests/CompositionContractTests.swift +++ b/Tests/JSONIntegrationTests/CompositionContractTests.swift @@ -249,7 +249,9 @@ private func composedImport(limits: JSONLimits = .balanced, extra: UInt64 = 0) t _ = try JSON.withComposition(limits: limits) { scope in try scope.withMeter { meter in meter.checkpointObserver = { work in - if work >= (stage == 5 ? 4200 : 32) { withUnsafeCurrentTask { $0?.cancel() } } + // Test-only current-task handle stays inside its standard-library callback. + // Only this invocation is cancelled; no unsafe handle escapes or is shared. + if work >= (stage == 5 ? 4200 : 32) { unsafe withUnsafeCurrentTask { unsafe $0?.cancel() } } } } do { diff --git a/Tests/JSONIntegrationTests/PublicWorkflowTests.swift b/Tests/JSONIntegrationTests/PublicWorkflowTests.swift index b6aa799..0fef152 100644 --- a/Tests/JSONIntegrationTests/PublicWorkflowTests.swift +++ b/Tests/JSONIntegrationTests/PublicWorkflowTests.swift @@ -28,7 +28,9 @@ private actor Output { calls += 1 if calls == rejectAt { throw AppError.sink } bytes.append(contentsOf: chunk.bytes) - if calls == cancelAt { withUnsafeCurrentTask { $0?.cancel() } } + // Test-only current-task handle stays inside its standard-library callback. + // Only this invocation is cancelled; no unsafe handle escapes or is shared. + if calls == cancelAt { unsafe withUnsafeCurrentTask { unsafe $0?.cancel() } } } func snapshot() -> ([UInt8], Int) { (bytes, calls) } } diff --git a/Tests/JSONIntegrationTests/StagedImportTests.swift b/Tests/JSONIntegrationTests/StagedImportTests.swift index ed05876..739cce2 100644 --- a/Tests/JSONIntegrationTests/StagedImportTests.swift +++ b/Tests/JSONIntegrationTests/StagedImportTests.swift @@ -13,7 +13,9 @@ private actor ImportSource { func next() throws -> JSONByteChunk? { pulls += 1 if pulls == failAt { throw ImportError.source } - if pulls == cancelAt { withUnsafeCurrentTask { $0?.cancel() } } + // Test-only current-task handle stays inside its standard-library callback. + // Only this invocation is cancelled; no unsafe handle escapes or is shared. + if pulls == cancelAt { unsafe withUnsafeCurrentTask { unsafe $0?.cancel() } } return pulls <= texts.count ? .init(Array(texts[pulls - 1].utf8)) : nil } func count() -> Int { pulls } diff --git a/Tests/JSONIntegrationTests/StructuralAcceptanceTests.swift b/Tests/JSONIntegrationTests/StructuralAcceptanceTests.swift new file mode 100644 index 0000000..6720187 --- /dev/null +++ b/Tests/JSONIntegrationTests/StructuralAcceptanceTests.swift @@ -0,0 +1,209 @@ +// SPDX-License-Identifier: Apache-2.0 +import Foundation +import Testing +import JSON +import JSONSchema +import JSONPatch +import JSONStreaming +import JSONTransform +import JSONInspection + +private let acceptance18Input = #"{"u":[{"id":"é","amount":9007199254740993,"next":{"collection":"u","id":"e\u0301"}},{"id":"e\u0301","amount":0.10000000000000001,"next":null}]}"# +private func acceptance18Source() throws -> JSONDocument { + try JSONParser().parseDocument(Data(acceptance18Input.utf8)) +} +private func acceptance18Profile() -> JSONStoreProfile { + .init(version: .v1, collectionsAt: [], collections: [ + .init(name: "u", recordsAt: [.key("u")], idKey: "id", referenceFields: [ + .init(key: "next", presence: .required, nullable: true)]) + ], references: .strict(marker: .v1, validateTargets: true)) +} +private func acceptance18Plan() throws -> CompiledJSONSchema { + try JSONSchemaCompiler().compile(JSONSchemaDocument(JSONParser().parse( + #"{"type":"object","required":["u"],"properties":{"u":{"type":"array","minItems":2,"items":{"type":"object","required":["id","amount","next"],"properties":{"id":{"type":"string"},"amount":{"type":"number"},"next":{"type":["object","null"]}}}}}}"#))) +} +private func acceptance18Store(_ source: JSONDocument) throws -> JSONConfiguredStore { + try JSONStoreBuilder(document: source, profile: acceptance18Profile()).build().result +} +private enum Acceptance18Error: Error { case expectedFailure, removed } + +@Test func acceptance18A01SchemaStoreTransformKeepsExactGoldenBytes() throws { + let source = try acceptance18Source(), schema = try acceptance18Plan() + let verdict = try JSONSchemaValidator().validateWithReport(source.root, against: schema) + #expect(verdict.result.isValid && verdict.statistics[.evaluationWorkUnits] > 0) + let store = try acceptance18Store(source) + let start = try #require(store.lookup(collection: "u", id: "é").result) + let closure = try store.follow(from: start), records = try store.values(at: closure.result) + let output = try JSONTransformProgram.compile(".[] | .amount").withExecution(input: .array(records.result)) { execution in + // This array is authored caller storage, not owned transform output accounting. + var values: [JSON] = [] + while let item = try execution.next() { values.append(item.value) } + return JSON.array(values) + } + #expect(try JSONWriter(options: .init(numberStyle: .preserved)).write(output.result) == Data("[9007199254740993,0.10000000000000001]".utf8)) + #expect(records.statistics[.nodes] == (try JSONInspectionBatch(document: source).values(at: []).statistics[.nodes])) + #expect(closure.result.count == 2 && output.statistics[.workUnits] > 0) +} + +@Test func acceptance18A02InvalidSchemaVerdictIsDistinctFromOperationalRefusal() throws { + let plan = try acceptance18Plan() + let invalid = try JSONSchemaValidator().validate(42, against: plan) + #expect(!invalid.isValid && !invalid.issues.isEmpty) + var limits = JSONLimits.balanced; limits[.evaluationWorkUnits] = 0 + do { + _ = try JSONSchemaValidator(options: .init(limits: limits)).validate(acceptance18Source().root, against: plan) + Issue.record("An exhausted evaluator returned a verdict") + } catch let error as JSONFailure { + #expect(error.category == .resource && error.resource?.metric == .evaluationWorkUnits) + } + #expect(try JSONSchemaValidator().validate(acceptance18Source().root, against: plan).isValid) +} + +@Test func acceptance18A03PatchedRebuildHasNewOwnerAndNoInventedSourceMap() throws { + let source = try acceptance18Source(), original = try acceptance18Store(source) + let position = try #require(original.lookup(collection: "u", id: "e\u{301}").result) + let amount = try JSONNumber(parsing: "1.25") + let patch = try JSONPatch(operations: [.replace(path: JSONPointer(parsing: "/u/1/amount"), value: .number(amount))]) + guard case .document(let root) = try patch.applying(to: source.root) else { throw Acceptance18Error.removed } + let changed = source.replacingRoot(root), rebuilt = try acceptance18Store(changed) + #expect(changed.sourceMap == nil && changed.sourceBytes == nil && original.document.sourceMap != nil) + do { _ = try rebuilt.values(at: [position]); Issue.record("Rebound an original position") } + catch let error as JSONStoreFailure { #expect(error.code == .invalidPosition) } + let replacement = try #require(rebuilt.lookup(collection: "u", id: "e\u{301}").result) + #expect(try rebuilt.value(at: replacement).result.requiredMember("amount") == .number(amount)) + #expect(try original.value(at: position).result.requiredMember("amount") == .number(JSONNumber(parsing: "0.10000000000000001"))) +} + +@Test func acceptance18A04DanglingPatchedReferenceCannotPublishStore() throws { + let source = try acceptance18Source(), original = try acceptance18Store(source) + let patch = try JSONPatch(operations: [.replace(path: JSONPointer(parsing: "/u/1/id"), value: "renamed")]) + guard case .document(let root) = try patch.applying(to: source.root) else { throw Acceptance18Error.removed } + do { _ = try acceptance18Store(source.replacingRoot(root)); Issue.record("Published a dangling reference") } + catch let error as JSONStoreFailure { + #expect(error.code == .referenceDanglingID && error.phase == 7) + #expect(error.path == [.key("u"),.index(0),.key("next"),.key("id")]) + #expect(error.sourceSpan == nil) + } + let first = try #require(original.lookup(collection: "u", id: "é").result) + #expect(try original.follow(from: first).result.count == 2 && source.root == original.document.root) +} + +@Test func acceptance18A05ScalarCollectionNamesSurvivePatchAndCrossLinks() throws { + let source = try JSONParser().parseDocument(Data(#"{"é":[{"id":"a","v":1,"go":{"collection":"e\u0301","id":"a"}}],"e\u0301":[{"id":"a","v":2,"go":{"collection":"é","id":"a"}}]}"#.utf8)) + let profile = JSONStoreProfile(version: .v1, collectionsAt: [], collections: ["e\u{301}", "é"].map { name in + .init(name: JSONKey(name), recordsAt: [.key(JSONKey(name))], idKey: "id", referenceFields: [.init(key: "go", presence: .required, nullable: false)]) + }, references: .strict(marker: .v1, validateTargets: true)) + let store = try JSONStoreBuilder(document: source, profile: profile).build().result + let a = try #require(store.lookup(collection: "é", id: "a").result) + let b = try #require(store.lookup(collection: JSONKey("e\u{301}"), id: "a").result) + #expect(a != b && a.collection == 0 && b.collection == 1) + #expect(try store.follow(from: a).result == [a,b]) + let patch = try JSONPatch(operations: [.replace(path: JSONPointer(parsing: "/é/0/v"), value: 3)]) + guard case .document(let root) = try patch.applying(to: source.root) else { throw Acceptance18Error.removed } + let selected = try JSONInspectionBatch(document: source.replacingRoot(root)).values(at: [ + [.key("é"),.index(0),.key("v")], [.key(JSONKey("e\u{301}")),.index(0),.key("v")]]) + #expect(selected.result == [3,2]) +} + +@Test func acceptance18A06TransformFailureStaysTerminalOverStoreInput() throws { + let store = try acceptance18Store(acceptance18Source()) + let positions = [try store.record(collection: 0, row: 0).position, try store.record(collection: 0, row: 1).position] + let values = try store.values(at: positions), plan = try JSONTransformProgram.compile(".[] | .amount") + var limits = JSONTransformLimits(); limits[.results] = 1 + var reachedFailure = false + do { + _ = try plan.withExecution(input: .array(values.result), limits: limits) { execution in + #expect(try execution.next()?.value == .number(JSONNumber(parsing: "9007199254740993"))) + let firstWork = execution.statistics[.workUnits], remaining = execution.remainingWorkUnits + for _ in 0..<2 { + do { _ = try execution.next(); Issue.record("A failed execution resumed") } + catch let error as JSONTransformFailure { + #expect(error.category == .resource && error.metric == "results" && error.limit == 1) + reachedFailure = true + } + } + #expect(execution.statistics[.workUnits] >= firstWork && execution.remainingWorkUnits <= remaining) + } + Issue.record("Caught failure allowed successful completion") + } catch let error as JSONTransformFailure { #expect(error.category == .resource && error.metric == "results") } + #expect(reachedFailure) + #expect(try store.values(at: positions).result == values.result) +} + +@Test func acceptance18A07RootRemovalIsNotNullOrAnEmptyStore() throws { + let source = try acceptance18Source(), pointer = try JSONPointer(parsing: "") + let removed = try JSONPatch(operations: [.remove(path: pointer)]).applying(to: source.root) + let null = try JSONPatch(operations: [.replace(path: pointer, value: .null)]).applying(to: source.root) + #expect(removed == .removed && null == .document(.null) && removed != null) + guard case .document(let root) = null else { throw Acceptance18Error.removed } + do { _ = try acceptance18Store(source.replacingRoot(root)); Issue.record("Null became an empty store") } + catch let error as JSONStoreFailure { #expect(error.code == .collectionsNotObject) } +} + +private actor Acceptance18Sink { + var bytes: [UInt8] = [], calls = 0 + private var accepted: CheckedContinuation? + private var handoff: CheckedContinuation? + func accept(_ chunk: JSONByteChunk) async { + calls += 1; bytes.append(contentsOf: chunk.bytes) + await withCheckedContinuation { continuation in + handoff = continuation + accepted?.resume(); accepted = nil + } + } + func waitForAcceptance() async { + if handoff != nil { return } + await withCheckedContinuation { accepted = $0 } + } + func completeAcceptance() { + handoff?.resume(); handoff = nil + } + func snapshot() -> ([UInt8], Int) { (bytes, calls) } +} +@Test func acceptance18A08CancelledWriterKeepsOnlyItsAcceptedPrefix() async throws { + let store = try acceptance18Store(acceptance18Source()), sink = Acceptance18Sink() + let position = try store.record(collection: 0, row: 0).position + let values = try store.values(at: [position]), value = JSON.array(values.result) + let golden = try JSONWriter().write(value) + let task = Task { try await JSONAsyncWriter(chunkSize: 4).write(value) { chunk in await sink.accept(chunk) } } + await sink.waitForAcceptance() + task.cancel() + await sink.completeAcceptance() + do { try await task.value; Issue.record("Expected deterministic cancellation") } catch is CancellationError {} + let accepted = await sink.snapshot() + #expect(accepted.1 == 1 && accepted.0 == Array(golden.prefix(4))) + #expect(try store.values(at: [position]).result == values.result) +} + +@Test func acceptance18SharedPlansKeepIndependentCrossProductHealth() async throws { + let source = try acceptance18Source(), schema = try acceptance18Plan(), store = try acceptance18Store(source) + let transform = try JSONTransformProgram.compile(".[] | .amount"), position = try store.record(collection: 0, row: 0).position + let transformInput = JSON.array(try store.values(at: [position]).result) + try await withThrowingTaskGroup(of: Bool.self) { group in + for index in 0..<12 { group.addTask { + switch index % 4 { + case 0: + guard try JSONSchemaValidator().validate(source.root, against: schema).isValid else { return false } + let values = try store.values(at: [position]) + return try transform.withExecution(input: .array(values.result)) { execution in + let value = try execution.next()?.value + return try value == .number(JSONNumber(parsing: "9007199254740993")) && execution.next() == nil + }.result + case 1: + var limits = JSONLimits.balanced; limits[.evaluationWorkUnits] = 0 + do { _ = try JSONSchemaValidator(options: .init(limits: limits)).validate(source.root, against: schema); return false } + catch let error as JSONFailure { return error.resource?.metric == .evaluationWorkUnits } + case 2: + var limits = JSONLimits.balanced; limits[.workUnits] = 0 + do { _ = try store.values(at: [position], limits: limits); return false } + catch let error as JSONFailure { return error.resource?.metric == .workUnits } + default: + var limits = JSONTransformLimits(); limits[.results] = 0 + do { _ = try transform.withExecution(input: transformInput, limits: limits) { try $0.next() }; return false } + catch let error as JSONTransformFailure { return error.metric == "results" } + } + } } + for try await success in group { #expect(success) } + } + #expect(try JSONSchemaValidator().validate(source.root, against: schema).isValid) +} From 70df9ba8d47322abf0d4aec1439256d7a5e97e94 Mon Sep 17 00:00:00 2001 From: Thorsten Fuchs <76904008+ThorFuchs@users.noreply.github.com> Date: Thu, 29 Oct 2026 10:51:56 +0100 Subject: [PATCH 3/5] Add bounded Transform source definitions and compile reports --- AGENTS.md | 6 + .../Release1.8/TransformBindings.md | 96 ++++++ .../Qualification/TestDiscovery.json | 53 +++- .../Qualification/TransformBindings1.8.md | 89 ++++++ Package.swift | 2 +- Scripts/ci.py | 40 ++- Scripts/qualify_transform_bindings.py | 55 ++++ Scripts/test_qualification_harness.py | 17 +- Sources/JSONCLIEngine/CLIBuildIdentity.swift | 2 +- .../Documentation.docc/JSONTransform.md | 13 +- .../Documentation.docc/SourceDefinitions.md | 41 +++ .../JSONTransform/JSONTransformProgram.swift | 25 +- .../JSONTransformProgramDefinition.swift | 266 +++++++++++++++++ .../JSONTransformTests/DefinitionTests.swift | 205 +++++++++++++ .../IndexedSelectionSpike.swift | 276 ++++++++++++++++++ .../Package.swift | 8 + Tools/TransformBindingQualification/README.md | 8 + .../TransformBindingConsumer/Main.swift | 42 +++ 18 files changed, 1226 insertions(+), 18 deletions(-) create mode 100644 Documentation/Architecture/Release1.8/TransformBindings.md create mode 100644 Documentation/Qualification/TransformBindings1.8.md create mode 100644 Scripts/qualify_transform_bindings.py create mode 100644 Sources/JSONTransform/Documentation.docc/SourceDefinitions.md create mode 100644 Sources/JSONTransform/JSONTransformProgramDefinition.swift create mode 100644 Tests/JSONTransformTests/DefinitionTests.swift create mode 100644 Tests/JSONTransformTests/IndexedSelectionSpike.swift create mode 100644 Tools/TransformBindingQualification/Package.swift create mode 100644 Tools/TransformBindingQualification/README.md create mode 100644 Tools/TransformBindingQualification/Sources/TransformBindingConsumer/Main.swift diff --git a/AGENTS.md b/AGENTS.md index 88f7f62..444b376 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -63,3 +63,9 @@ Preserve its accepted 42 authored declarations and exact semantics. The original 1.7.0 P4 waiver and later partial measurements are historical scoped evidence; no allocation, speed, RSS or controlled-performance pass follows from consolidation. See Documentation/Qualification/Consolidation2026.md for combined gates and holds. + +The commissioned 1.8 Transform binding train adds a source-definition codec and +compile reports while preserving those original declarations. Indexed selection +is a test-only private-kernel ownership experiment in Sprint 1, not public API. +See Documentation/Architecture/Release1.8/TransformBindings.md. No Content code, +lock, message, merge or publication is authorized by implementing this sprint. diff --git a/Documentation/Architecture/Release1.8/TransformBindings.md b/Documentation/Architecture/Release1.8/TransformBindings.md new file mode 100644 index 0000000..6b01420 --- /dev/null +++ b/Documentation/Architecture/Release1.8/TransformBindings.md @@ -0,0 +1,96 @@ +# Transform bindings — commissioned 1.8 candidate, Sprint 1 + +The maintainer commissioned the source-definition codec, indexed selection and +compile reporting on 29 October 2026. Base is b8ddeb26042005a06360f50fcf20b1f840e66d1c. +This additive train preserves the existing 42 authored Transform declarations, +ordinary execution/CLI semantics, thirteen products and empty root dependencies. +Implementation and a successful spike are not release or downstream admission. + +## Definition V1 + +JSONTransformProgramDefinition stores exact source and unique ASCII variable names +in ascending UTF-8 order. It does not add source storage to compiled plans. There +is no synthesized String-based equality/hash or serialized VM/host state. + +Canonical encoding is compact UTF-8 without a BOM or newline, with fields in this +order: definition, version, dialect, source, variables. Their values are +"swift-json-transform-source", mathematical integer 1, "swift-json-query-v1", the +exact source string, and the ordered name array. Native JSONWriter supplies all +escaping: direct non-ASCII UTF-8, escaped quote/backslash, lowercase control-byte +escapes. This format is distinct from an implementation digest and execution policy. + +Decoding accepts one strict native JSON document with exactly those five decoded +keys and correct field types. Equivalent JSON escaping, field order, whitespace +and numeric spelling of version 1 are accepted; re-encoding is canonical. Unknown +fields, duplicate decoded keys/names and unknown versions/dialects fail. This is +strict semantic decoding, not a canonical-input assertion. A storage gateway that +requires canonical input must compare against bounded canonical output under its +own explicitly reviewed policy. Query syntax is checked by the existing compiler, +not the codec. Names use [A-Za-z_][A-Za-z0-9_]* without dollar prefixes and are +validated before any Set conversion. No credentials, files, closures, Content +objects, SCCV/SCCD data, artifact hashes or online resolution are encoded. + +The finite codec defaults are source 1 MiB, 128 names, 128 bytes/name, 8192 total name +bytes and 2 MiB wire bytes, alongside the existing balanced JSON limits. Native work +meters sorting, parsing, scanning, construction and writing; retention is reserved +before owned growth. Limits are checked at invocation entry; helpers share that +one meter. Existing 49 core policies and their probes are unchanged. Logical bytes +are reservations, not RSS. Supplied input storage and later caller copies are caller +work. Prepare, decode, encode and compile are explicitly separate operations; no +decode-and-compile convenience secretly restarts a single composed invocation. + +Successful preparation/codec operations return native reports. Compilation uses +the existing compiler/context and returns actual native work. The additive +compileWithReport path preserves old compile behavior; the definition adapter +also charges its owned variable Set construction. No public meter is exposed. + +## Indexed selection contract for Sprint 2 + +The proposed convention is an invocation-owned ordered array [0,...,N-1] as input +and a reserved explicit values binding containing the complete population. The +selection convention needs its own revision, separate from the language/wire +revisions. Reserved-name collisions and provider policy must be frozen before +public selection API admission. Content-specific mapping/identity stays outside. + +The existing query engine alone filters/sorts; it must explicitly emit numeric +ordinals. Native mathematical integer conversion admits 42.0 and negative zero as +their exact integer identities; fractions, negative values, other kinds, overflow, +native-width failure and out-of-population values fail. Output order/repeats are +preserved. Deterministic tie keys belong in the program; no hidden deduplication. + +One invocation must own population admission, index construction, VM evaluation, +numeric checks and typed output slots. Position handles need an unforgeable owner +and a joint population lease even when the numeric output was newly computed. +An explicit EOF state is required to certify completion. A caught boundary error +or abandoned prefix cannot produce a completion witness. Generic JSONOperationReport +has a public pairing initializer and is not that witness; public completion will +need a private initializer and owner-produced statistics. Query catches cannot +swallow boundary failures. First cause, cancellation and terminal lease refunds +must remain observable without reopening execution. Handles stay guarded and +non-Sendable; immutable plans/definitions remain checked Sendable. + +Sprint 1 keeps this experiment solely in IndexedSelectionSpike.swift in the test +target. It reuses FilterVM/TransformRuntimeContext and native values; it is not a +second interpreter or a shipped selection facade. Its 1024 population bound and +test provider are experimental, not approved public defaults/provider semantics. +Full reentry/public compiler-negative/lifetime/profile qualification belongs to +the next increment. General provenance propagation, persistence of executions, +unbounded input/sort guarantees and Content implementation are outside scope. + +## Qualification and admission + +Run Scripts/ci.py lanes transform-bindings and transform-bindings-safety for this +sprint's actual cases and public consumer. Preserve the original 23 Transform cases, +all old suites/fixtures and CLI checks. Source/CLI identity, public declarations, +native strict-memory checks and DocC must bind the final checkpoint. Full Native, +Linux ARM64/x86_64, WASI and relevant Apple checks are required for the eventual +release; unavailable/unrun lanes stay open. Earlier 1.8 receipts and 1.7 performance +waivers do not qualify these source changes. Existing performance/allocation holds +remain. No merge, tag, publication or cross-session message is commissioned here. + +The downstream S06/S13 profile mapping and deliberate Content lock selection remain +separate decisions after a qualified public binding. M0 stays independent; no S04 +history, Content repository, source pin or admission gate is changed by this train. + +The [Sprint 1 qualification record](../../Qualification/TransformBindings1.8.md) +separates completed local checks from the remaining public-selection/release gates. diff --git a/Documentation/Qualification/TestDiscovery.json b/Documentation/Qualification/TestDiscovery.json index 3251689..60b4b1e 100644 --- a/Documentation/Qualification/TestDiscovery.json +++ b/Documentation/Qualification/TestDiscovery.json @@ -13,7 +13,7 @@ "JSONNavigationTests": 57, "JSONSchemaExamplesTests": 98, "JSONInspectionTests": 214, - "JSONTransformTests": 23, + "JSONTransformTests": 41, "JSONDSLTests": 47 }, "parameterized_invocations": 390, @@ -38,7 +38,7 @@ "JSONNavigationTests": 2, "JSONSchemaExamplesTests": 4, "JSONInspectionTests": 11, - "JSONTransformTests": 1, + "JSONTransformTests": 3, "JSONDSLTests": 1 }, "coverage_unmapped": { @@ -85,7 +85,7 @@ "JSONNavigationTests": 57, "JSONSchemaExamplesTests": 98, "JSONInspectionTests": 214, - "JSONTransformTests": 23, + "JSONTransformTests": 41, "JSONDSLTests": 47 }, "wasi_root_functions": { @@ -100,7 +100,7 @@ "JSONNavigationTests": 57, "JSONSchemaExamplesTests": 98, "JSONInspectionTests": 214, - "JSONTransformTests": 23, + "JSONTransformTests": 41, "JSONDSLTests": 47 }, "wasi_parameterized_invocations": 353, @@ -116,7 +116,7 @@ "JSONNavigationTests": 2, "JSONSchemaExamplesTests": 4, "JSONInspectionTests": 11, - "JSONTransformTests": 1, + "JSONTransformTests": 3, "JSONDSLTests": 1 }, "export_functions": { @@ -230,5 +230,48 @@ "JSONIntegrationTests.qaE2ESharedPlanRecoversAfterIndependentFailures()", "JSONIntegrationTests.qaE2EUnknownEnvelopeVersionIsCorruption()", "JSONIntegrationTests.qaE2EVersionOneEnvelopePreservesExactInnerState()" + ], + "transform_function_ids": [ + "JSONTransformTests.bindingD01CanonicalGoldenBytesAndReload()", + "JSONTransformTests.bindingD02CanonicalEscapingAndExactScalarSource()", + "JSONTransformTests.bindingD03StrictSemanticDecodeCanonicalizesPresentation()", + "JSONTransformTests.bindingD04ClosedFieldsVersionsAndKindsArePrecise()", + "JSONTransformTests.bindingD05NativeGrammarAndDuplicateFieldsStayAuthoritative()", + "JSONTransformTests.bindingD06NamesRejectBeforeSetConversion()", + "JSONTransformTests.bindingD07SpecificCapsAreInclusiveAndFinite()", + "JSONTransformTests.bindingD08WireAndOwnedWorkRetentionHaveExactBoundaries()", + "JSONTransformTests.bindingD09CompileReportsPreserveExistingPlanBehaviorAndBudget()", + "JSONTransformTests.bindingD10CodecDoesNotInventQueryValidationOrDecimalAuthority()", + "JSONTransformTests.bindingD11CancellationOccursInsideOwnedCodecWork()", + "JSONTransformTests.bindingP01ExplicitIndicesPreserveDuplicatesOrderAndTies()", + "JSONTransformTests.bindingP02InvalidOutputAndAbandonedPrefixNeverCertify()", + "JSONTransformTests.bindingP03PopulationLeaseSurvivesCloseAndForeignOwnerRefuses()", + "JSONTransformTests.bindingP04ResourceAndProviderFailuresStayTerminal()", + "JSONTransformTests.bindingP05ProjectionEvaluationAndCancellationShareOneLedger()", + "JSONTransformTests.bindingSharedDefinitionAndCompilationHaveIndependentHealth()", + "JSONTransformTests.bindingSharedSelectionSpikeReusesOnePlanIndependently()", + "JSONTransformTests.jtxBindingNamesUseTheCompilerIdentifierGrammar()", + "JSONTransformTests.jtxBindingsAndProviderAreExplicitAndLazy()", + "JSONTransformTests.jtxCancellationLatchesAndStopsProviderCalls()", + "JSONTransformTests.jtxCompiledLiteralsCannotBypassTighterExecutionValueLimits()", + "JSONTransformTests.jtxConfiguredCompilerGrowthHasInclusiveBoundaries()", + "JSONTransformTests.jtxCopiedHandlesShareOneCumulativeLedger()", + "JSONTransformTests.jtxCumulativeProviderAndResultCapsCannotResetPerPull()", + "JSONTransformTests.jtxEscapedHandleInvalidatesAndReleasesProvider()", + "JSONTransformTests.jtxExactNumbersAndScalarKeysKeepCoreAuthority()", + "JSONTransformTests.jtxOutputAdmissionWorkAndTemporaryFramesStayOnTheOneLedger()", + "JSONTransformTests.jtxPlanIsSendableAndExecutionsHaveIndependentHealth()", + "JSONTransformTests.jtxProviderCanObserveSnapshotAndCancelAfterOwnedProgress()", + "JSONTransformTests.jtxProviderFailureIsOperationalAndNeverCaughtByLanguageTry()", + "JSONTransformTests.jtxPublicEngineMatchesNativeExactDialectMatrix()", + "JSONTransformTests.jtxReentrantProviderFailureLatchesWithoutAccessConflict()", + "JSONTransformTests.jtxReportsUseCoreDomainAndCallerErrorRetainsIdentity()", + "JSONTransformTests.jtxResourceErrorBypassesCatchAndLatchesAfterHostCatch()", + "JSONTransformTests.jtxRuntimeCapsRejectBeforePublishingAndRemainTyped()", + "JSONTransformTests.jtxSelectedAliasKeepsSourceReservationUntilLastHandleDies()", + "JSONTransformTests.jtxTaskCancellationIsObservedAfterProviderOwnedProgress()", + "JSONTransformTests.jtxTerminalAliasesRefundBeforeThrowingTheSameFailure()", + "JSONTransformTests.jtxTypedCompileLocationAndLanguageFailureStayDistinct()", + "JSONTransformTests.jtxZeroNullAndManyAreDistinctOrderedPullResults()" ] } diff --git a/Documentation/Qualification/TransformBindings1.8.md b/Documentation/Qualification/TransformBindings1.8.md new file mode 100644 index 0000000..37bd195 --- /dev/null +++ b/Documentation/Qualification/TransformBindings1.8.md @@ -0,0 +1,89 @@ +# Transform bindings 1.8 — first sprint + +Status: implemented and scoped native qualification complete; full release +qualification remains PARTIAL. This isolated candidate extends +`b8ddeb26042005a06360f50fcf20b1f840e66d1c` on +`codex/1.8-transform-bindings-s1`. The original batches/store candidate, its +PR and all earlier source-bound receipts remain independent and unchanged. +No merge, tag, publication, Content pin or downstream admission is implied. + +The [contract](../Architecture/Release1.8/TransformBindings.md) defines the +candidate wire format and next selection boundary. Public additions are the +immutable source definition, finite definition policy, typed definition failures, +codec operations and compile-with-report route. Existing JSON/decimal/Unicode +and query semantics remain authoritative. The indexed-selection feasibility code +exists only in the test target; no public position, completion or selection API +ships in this sprint. + +## Observed qualification — 29 October 2026 + +Supported host: macOS 27.0 arm64, Apple Swift 6.4.2 +(`swiftlang-6.4.2.1.9`, driver 1.168.6), Xcode 27.2/27B48, SDK 27.2, +Swift language mode 6. Raw commands, exits, events, source manifests and failed +attempts are retained privately under `Artifacts/TransformBindingsS1`. + +| Check | Observed outcome | +| --- | --- | +| `Scripts/ci.py --lane transform-bindings` | Passed: 41 functions in Debug and Release; original 23 preserved; 31 Python harness cases; public privacy/identity checks | +| `Scripts/ci.py --lane transform-bindings-safety` | Passed: all 41 Release ASan functions; three shared-plan TSan functions; strict-memory target build with warnings as errors | +| New public-only consumer outside Git | Passed: six witnesses in each configuration, one exact compiled DocC fence, four ordinary full-codegen constructor/immutability refusals | +| Existing public Transform consumer | Passed: original twelve witnesses in each configuration | +| Existing Transform ownership compiler cases | Passed: seven cases; intended compiler refusals distinguished from import failures/crashes | +| `Scripts/ci.py --lane cli` | Passed: 13 commands, 88 functions/37 parameter invocations and all existing command/vector/provider checks | +| Existing manifest/dependency/cap checks | Passed through unchanged CI comparators: thirteen products, empty external graph, one test reaching all 49 core caps with exact outcomes | +| Scoped JSONTransform native symbol graph and DocC | Passed: 67 authored declarations and parameter comments; original 42 identities/signatures preserved; warnings as errors | + +The public graph adds 25 authored declarations. Three historical graph fragment +differences are retained and explicitly reviewed: two nested types acquire their +parent qualification while retaining the same precise type IDs and byte-identical +owning sources; generic Result metadata loses a local preciseIdentifier with +unchanged spelling. The original withExecution declaration/body is byte-identical. +This is a scoped Transform catalog result, not a fresh thirteen-catalog pass. + +Initial compiler errors were in new test assertions with throwing expressions; +they remain in the private logs. The first native harness run rejected a stale +30-case self-test expectation after a meaningful 31st case was added. The +successor checks both the new exact population and rejection of stale populations. +The first strict graph comparison retained the three rendering differences above; +its source-backed successor records their precise disposition. No failed attempt +is relabeled as a passed check or omitted from the return. + +## Semantic and ownership coverage + +D01–D03 pin literal canonical bytes, native escaping, exact decoded source scalars, +UTF-8 name ordering and canonical reload. D04–D06 reject closed-field/type/version/ +dialect/name errors, exact duplicates and malformed native JSON. D07–D11 pin zero, +exact, one-over and overflow-policy boundaries, work/retention accounting, compile +reports, compiler errors and deterministic inner-loop cancellation. Shared cases +exercise independent concurrent reuse. The reports describe actual native work; +caller storage/instructions and process RSS remain outside those claims. + +The P01–P05 test-only selection prototype exercises explicit indices over duplicate +values, order/repeats/ties, empty completion, invalid late output bypassing query +catch, incomplete-prefix refusal, foreign owner refusal, post-close leases/refunds, +terminal quota/provider failures and cumulative projection/evaluation/cancellation. +It uses the existing VM and private context, not a second evaluator. Its profile, +provider and completion types are experiments, not public contracts or conformance. + +## Next bounded increment and held gates + +Sprint 2 must freeze reserved binding/provider policy and a versioned selection +convention, implement a guarded public selection scope with private position and +completion constructors, and qualify reentry, public constructor refusals, aliases, +first-cause terminal accounting, exact inclusive capacities and independent plans. +Retain the admitted population through every output's joint lease. Full EOF is +required before an owner-produced completion witness. Ordinary execution stays +unchanged. The downstream S06/S13 mapping remains a separate deliberate decision. + +Fresh full Native/Linux ARM64/x86_64/safety/docs, WASI, relevant Apple SDK/runtime, +archive reproduction and complete API checks remain final-release gates. The +portable harness is prepared but this sprint makes no new Linux/WASI/device pass. +The existing 49 policies/probes are unchanged; the codec's five finite capacities +are additive policy, not replacements. No new benchmark/allocation campaign ran. +Existing 1.8 loaded-host comparisons, unresolved regressions, unmeasured allocations +and older scoped holds remain visible; the 1.7 waiver does not apply to 1.8. + +The final checkpoint binding separately records this documentary closeout and +architecture text polish after the passing receipts. Product, manifest, tests, +DocC, fixtures, generator, CLI identity and executable harness bytes remain equal +to those checked. No independent human review or release approval is claimed. diff --git a/Package.swift b/Package.swift index 164b251..6aa75ee 100644 --- a/Package.swift +++ b/Package.swift @@ -43,7 +43,7 @@ let package = Package( .target(name: "JSONHTML", dependencies: ["JSON"], exclude: ["Assets"]), .target(name: "JSONFilterRuntime", dependencies: ["JSON"]), .target(name: "JSONTransform", dependencies: ["JSON", "JSONFilterRuntime"]), - .testTarget(name: "JSONTransformTests", dependencies: ["JSON", "JSONTransform"]), + .testTarget(name: "JSONTransformTests", dependencies: ["JSON", "JSONTransform", "JSONFilterRuntime"]), .target(name: "JSONCLIEngine", dependencies: ["JSON", "JSONSchema", "JSONStreaming", "JSONPatch", "JSONHTML", "JSONFilterRuntime", "JSONTransform"]), .executableTarget(name: "JSONCLI", dependencies: ["JSONCLIEngine"]), .testTarget(name: "JSONCLITests", dependencies: ["JSONCLIEngine", "JSONFilterRuntime", "JSON", "JSONPatch", "JSONSchema", "JSONHTML"]), diff --git a/Scripts/ci.py b/Scripts/ci.py index 4cc12c0..09aa283 100644 --- a/Scripts/ci.py +++ b/Scripts/ci.py @@ -28,6 +28,7 @@ SHARED_TESTS += '|graphG30ConcurrentViewsHaveIndependentHealth|graphGP14ConcurrentQueries' SHARED_TESTS += '|dslSharedFinalizationAndPlansAreInvocationOwned' SHARED_TESTS += '|storeR06TerminalQuotasNeverPublishAndImmutableConcurrentHealth' +SHARED_TESTS += '|bindingSharedDefinitionAndCompilationHaveIndependentHealth|bindingSharedSelectionSpikeReusesOnePlanIndependently' SHARED_TESTS += '|workflowSharedImmutablePlansHaveIndependentBatchAndStoreHealth' SHARED_TESTS += '|acceptance18SharedPlansKeepIndependentCrossProductHealth' R0_TESTS = 'readerProbe|indexProbes|duplicateRecognitionPrecedesValueAndClosingSyntax' @@ -38,6 +39,23 @@ def snapshot(): def commands(lane, output): # The qualification executable resolves output paths from its repository root. output = output.relative_to(ROOT) + if lane in ('transform-bindings', 'transform-bindings-safety'): + def portable(values): + return [c + ['--disable-xctest'] if sys.platform == 'linux' and c[:2] == ['swift', 'test'] else c for c in values] + if lane.endswith('safety'): + return portable([ + ['swift', 'test', '--test-product', 'JSONTransformTests', '-c', 'release', '--jobs', '2', '--sanitize', 'address', '--scratch-path', '.build/bindings-address'], + ['swift', 'test', '--test-product', 'JSONTransformTests', '-c', 'release', '--jobs', '2', '--sanitize', 'thread', '--scratch-path', '.build/bindings-thread', '--filter', 'jtxPlanIsSendableAndExecutionsHaveIndependentHealth|bindingShared'], + ['swift', 'build', '--target', 'JSONTransform', '--jobs', '2', '--scratch-path', '.build/bindings-strict', '-Xswiftc', '-strict-memory-safety', '-Xswiftc', '-warnings-as-errors'], + ]) + return portable([ + ['python3', '-B', 'Scripts/public_check.py'], + ['python3', '-B', 'Scripts/generate_cli_identity.py', '--check'], + ['python3', '-B', 'Scripts/test_qualification_harness.py'], + ['swift', 'test', '--test-product', 'JSONTransformTests', '--jobs', '2'], + ['swift', 'test', '--test-product', 'JSONTransformTests', '-c', 'release', '--jobs', '2'], + ['python3', '-B', 'Scripts/qualify_transform_bindings.py', '--output', str(output / 'public-consumer')], + ]) if lane in ('structural-acceptance', 'structural-acceptance-safety'): if lane.endswith('safety'): result = [ @@ -245,6 +263,7 @@ def commands(lane, output): ['python3', '-B', 'Scripts/test_transform_ownership.py'], ['python3', '-B', 'Scripts/qualify_transform_ownership.py', '--output', str(output / 'transform-ownership')], ['python3', '-B', 'Scripts/qualify_transform.py', '--output', str(output / 'transform-consumer')], + ['python3', '-B', 'Scripts/qualify_transform_bindings.py', '--output', str(output / 'transform-binding-consumer')], ['python3', '-B', 'Scripts/qualify_release_1_1.py', '--mode', 'ownership', '--output', str(output / 'ownership.json')], ['python3', '-B', 'Scripts/qualify_release_1_1.py', '--mode', 'reader', '--output', str(output / 'reader.json')], ['swift', 'run', '-c', 'release', '--package-path', 'Examples/Consumer', 'Consumer'], @@ -335,6 +354,14 @@ def check_result(command, code, log, events=None): if 'JSONSchemaTests' in command and events is not None: parameters = {name: ROOT_PARAMETER_CASES[name] for name in ['successfulAlternativesCannotSwallowRegexExhaustion(keyword:)', 'URIIdentityUsesSyntaxNormalizationWithoutUnicodeFolding(input:expected:)']} passed = passed and events['module_function_counts'] == {'JSONSchemaTests': 79} and events['parameterized_functions'] == parameters and events['parameter_case_invocations'] == 12 + if 'JSONTransformTests' in command: + ids = set(DISCOVERY['transform_function_ids']) + if 'thread' in command: + ids = {n for n in ids if 'bindingShared' in n or 'jtxPlanIsSendableAndExecutionsHaveIndependentHealth' in n} + elif selected_filter == 'binding': + ids = {n for n in ids if '.binding' in n} + expected = [len(ids)] + passed = passed and events is not None and events['module_function_counts'] == {'JSONTransformTests': len(ids)} and set(events['function_names']) == ids and len(events['function_names']) == len(ids) and events['parameter_case_invocations'] == 0 if 'navigation' in command: expected = [DISCOVERY['root_functions']['JSONNavigationTests']] if events is not None: @@ -375,7 +402,7 @@ def check_result(command, code, log, events=None): passed = passed and events['module_function_counts'] == {'CompositionScaleTests': DISCOVERY['composition_external_functions']} and events['parameter_case_invocations'] == 0 if 'qa' in command and events is not None: passed = passed and events['module_function_counts'] == DISCOVERY['qa_functions'] and check_restored(events) - if 'thread' in command and 'JSONSchemaExamplesTests' not in command and not prepared_test and not store_test and not workflow_test and not acceptance_test and events is not None: + if 'thread' in command and 'JSONSchemaExamplesTests' not in command and 'JSONTransformTests' not in command and not prepared_test and not store_test and not workflow_test and not acceptance_test and events is not None: passed = passed and events['module_function_counts'] == SHARED_COUNTS and check_projection_cases(events, shared=True) and check_p2_cases(events, shared=True) and check_exact_cases(events, shared=True) and check_graph_cases(events, shared=True) and events['parameter_case_invocations'] == 0 and check_store_tests(events, log, shared=True) if '--filter' not in command and '--test-product' not in command and not any(x.startswith('Tools/') for x in command): if events is not None: @@ -413,7 +440,7 @@ def check_result(command, code, log, events=None): passed = passed and set(targets) == {'JSON', 'JSONNavigation', 'JSONNavigationTests', 'JSONInspection', 'JSONInspectionTests', 'JSONIntegrationTests', 'JSONBridges', 'JSONBridgesTests', 'JSONSchema', 'JSONSchemaExamples', 'JSONSchemaExamplesTests', 'JSONStreaming', 'JSONPatch', 'JSONHTML', 'JSONRegexRuntime', 'JSONTests', 'JSONSchemaTests', 'JSONStreamingTests', 'JSONPatchTests', 'JSONHTMLTests', 'JSONRegexRuntimeTests', 'JSONFilterRuntime', 'JSONCLIEngine', 'JSONCLI', 'JSONCLITests', 'JSONTransform', 'JSONTransformTests', 'JSONDSLCore', 'JSONValueDSL', 'JSONSchemaDSL', 'JSONDSLTests'} and targets['JSONHTML']['dependencies'] == [{'byName': ['JSON', None]}] and targets['JSONBridges']['dependencies'] == [{'byName': ['JSON', None]}] and targets['JSONFilterRuntime']['dependencies'] == [{'byName': ['JSON', None]}] and targets['JSONCLI']['type'] == 'executable' passed = passed and targets['JSONNavigation']['dependencies'] == [{'byName': ['JSON', None]}] and targets['JSONNavigationTests']['dependencies'] == [{'byName': ['JSONNavigation', None]}, {'byName': ['JSON', None]}] passed = passed and targets['JSONSchemaExamples']['dependencies'] == [{'byName': ['JSON', None]}, {'byName': ['JSONSchema', None]}] and targets['JSON']['dependencies'] == [] and targets['JSONSchema']['dependencies'] == [{'byName': ['JSON', None]}, {'byName': ['JSONRegexRuntime', None]}] - passed = passed and targets['JSONTransform']['dependencies'] == [{'byName': ['JSON', None]}, {'byName': ['JSONFilterRuntime', None]}] and targets['JSONTransformTests']['dependencies'] == [{'byName': ['JSON', None]}, {'byName': ['JSONTransform', None]}] + passed = passed and targets['JSONTransform']['dependencies'] == [{'byName': ['JSON', None]}, {'byName': ['JSONFilterRuntime', None]}] and targets['JSONTransformTests']['dependencies'] == [{'byName': ['JSON', None]}, {'byName': ['JSONTransform', None]}, {'byName': ['JSONFilterRuntime', None]}] passed = passed and targets['JSONInspection']['dependencies'] == [{'byName': ['JSON', None]}, {'byName': ['JSONNavigation', None]}] and targets['JSONInspectionTests']['dependencies'] == [{'byName': ['JSONInspection', None]}, {'byName': ['JSONNavigation', None]}, {'byName': ['JSON', None]}] for name, dependencies in [('JSONDSLCore', ['JSON']), ('JSONValueDSL', ['JSON', 'JSONDSLCore']), ('JSONSchemaDSL', ['JSON', 'JSONSchema', 'JSONDSLCore']), ('JSONDSLTests', ['JSON', 'JSONSchema', 'JSONDSLCore', 'JSONValueDSL', 'JSONSchemaDSL'])]: passed = passed and targets[name]['dependencies'] == [{'byName': [item, None]} for item in dependencies] @@ -501,12 +528,15 @@ def check_result(command, code, log, events=None): if command[:3] == ['python3', '-B', 'Scripts/qualify_html_browser.py']: passed = passed and 'JSONHTML_BROWSER passed engines=3 cases=76 skips=0' in log.splitlines() if 'Scripts/test_qualification_harness.py' in command: - passed = passed and 'HARNESS_TESTS passed cases=30' in log.splitlines() + passed = passed and 'HARNESS_TESTS passed cases=31' in log.splitlines() + if command[:3] == ['python3', '-B', 'Scripts/qualify_transform_bindings.py']: + data = json.loads(log.strip().splitlines()[-1]) + passed = passed and data['status'] == 'passed' and len(data['commands']) == 8 and all(x['passed'] for x in data['commands']) and data['witnesses_per_configuration'] == 6 and data['negative_fixture_invocations'] == 4 and data['compiled_guide_fences'] == 1 return passed, counts def main(): parser = argparse.ArgumentParser(description=__doc__) - parser.add_argument('--lane', choices=['structural-acceptance', 'structural-acceptance-safety', 'structural-workflows', 'structural-workflows-safety', 'store', 'store-preparation', 'composition', 'navigation', 'inspection', 'native', 'safety', 'docs', 'r0', 'qa', 'linux', 'linux-safety', 'wasm', 'browser', 'export', 'cli', 'schema-examples', 'schema-examples-safety', 'schema-examples-determinism', 'schema-examples-construction', 'schema-examples-advanced', 'schema-examples-qualification', 'coverage', 'linux-coverage', 'linux-qa', 'wasm-qa', 'archive'], required=True) + parser.add_argument('--lane', choices=['transform-bindings', 'transform-bindings-safety', 'structural-acceptance', 'structural-acceptance-safety', 'structural-workflows', 'structural-workflows-safety', 'store', 'store-preparation', 'composition', 'navigation', 'inspection', 'native', 'safety', 'docs', 'r0', 'qa', 'linux', 'linux-safety', 'wasm', 'browser', 'export', 'cli', 'schema-examples', 'schema-examples-safety', 'schema-examples-determinism', 'schema-examples-construction', 'schema-examples-advanced', 'schema-examples-qualification', 'coverage', 'linux-coverage', 'linux-qa', 'wasm-qa', 'archive'], required=True) parser.add_argument('--output') parser.add_argument('--skip-host-measurements', action='store_true', help='For WASI runtime qualification only: record optional host timings as NOT RUN.') args = parser.parse_args() @@ -518,7 +548,7 @@ def main(): repository = subprocess.run(['git', 'rev-parse', '--show-toplevel'], cwd=ROOT, capture_output=True, text=True) is_checkout = repository.returncode == 0 and pathlib.Path(repository.stdout.strip()).resolve() == ROOT.resolve() commit = subprocess.run(['git', 'rev-parse', 'HEAD'], cwd=ROOT, capture_output=True, text=True) if is_checkout else None - environment = collect_linux() if args.lane.startswith('linux') or (args.lane in ('composition', 'inspection', 'structural-workflows', 'structural-workflows-safety', 'structural-acceptance', 'structural-acceptance-safety') and sys.platform == 'linux') else collect() + environment = collect_linux() if args.lane.startswith('linux') or (args.lane in ('transform-bindings', 'transform-bindings-safety', 'composition', 'inspection', 'structural-workflows', 'structural-workflows-safety', 'structural-acceptance', 'structural-acceptance-safety') and sys.platform == 'linux') else collect() receipt = {'schema_version': 1, 'lane': args.lane, 'status': 'blocked', 'environment': environment, 'started_utc': datetime.datetime.now(datetime.timezone.utc).isoformat(), 'git_commit': commit.stdout.strip() if commit is not None and commit.returncode == 0 else None, 'github_run_id': os.environ.get('GITHUB_RUN_ID'), 'source_inputs': inputs, 'source_manifest_sha256': hashlib.sha256(json.dumps(inputs, sort_keys=True).encode()).hexdigest(), 'commands': []} def save(): (output / 'receipt.json').write_text(json.dumps(receipt, indent=2) + '\n') diff --git a/Scripts/qualify_transform_bindings.py b/Scripts/qualify_transform_bindings.py new file mode 100644 index 0000000..53098c8 --- /dev/null +++ b/Scripts/qualify_transform_bindings.py @@ -0,0 +1,55 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: Apache-2.0 +"""Public-only definition consumer and ordinary full-codegen constructor refusals.""" +import argparse, hashlib, json, pathlib, platform, re, shutil, subprocess, sys, tempfile, textwrap +from qualify_transform_ownership import ordinary_rejection +ROOT = pathlib.Path(__file__).resolve().parents[1] + +def main(): + parser = argparse.ArgumentParser(description=__doc__); parser.add_argument('--output', required=True) + args = parser.parse_args(); output = (ROOT/args.output).resolve(); output.mkdir(parents=True, exist_ok=True) + guide=(ROOT/'Sources/JSONTransform/Documentation.docc/SourceDefinitions.md').read_text() + fences=re.findall(r'```swift\n(.*?)\n```',guide,re.S) + consumer_text=(ROOT/'Tools/TransformBindingQualification/Sources/TransformBindingConsumer/Main.swift').read_text() + body=re.search(r'func guideExample\(\) throws \{\n(.*?)\n\}\n@main',consumer_text,re.S).group(1) + assert len(fences)==1 and textwrap.dedent(body).strip()=='\n'.join(line for line in fences[0].splitlines() if not line.startswith('import ')).strip() + commands = [] + def run(command, name): + result = subprocess.run(command,cwd=ROOT,capture_output=True,text=True) + data=result.stdout+result.stderr; path=output/(name+'.log');path.write_text(data) + row=dict(command=command,exit_code=result.returncode,log=path.name,sha256=hashlib.sha256(path.read_bytes()).hexdigest()) + commands.append(row); return result,row + with tempfile.TemporaryDirectory(prefix='swift-json-binding-consumer-') as temporary: + consumer=pathlib.Path(temporary) + for p in (ROOT/'Tools/TransformBindingQualification').rglob('*'): + if p.is_file() and not any(x in p.parts for x in ('.build','.swiftpm')): + target=consumer/p.relative_to(ROOT/'Tools/TransformBindingQualification');target.parent.mkdir(parents=True,exist_ok=True);target.write_bytes(p.read_bytes()) + manifest=consumer/'Package.swift';manifest.write_text(manifest.read_text().replace('path: "../.."','path: '+json.dumps(str(ROOT)))) + inputs=[dict(path=p.relative_to(consumer).as_posix(),sha256=hashlib.sha256(p.read_bytes()).hexdigest()) for p in sorted(consumer.rglob('*')) if p.is_file()] + for config in ['debug','release']: + result,row=run(['swift','run','--package-path',str(consumer),'-c',config,'--jobs','2','TransformBindingConsumer'],config+'-consumer') + row['passed']=result.returncode==0 and result.stdout.splitlines()==['TRANSFORM_BINDINGS passed witnesses=6'] + if not row['passed']:break + result,row=run(['swift','build','--package-path',str(consumer),'-c',config,'--show-bin-path'],config+'-binpath') + row['passed']=result.returncode==0 + if not row['passed']:break + modules=pathlib.Path(result.stdout.strip()); binary=modules/'TransformBindingConsumer' + target=output/(config+'-TransformBindingConsumer');shutil.copyfile(binary,target) + row['binary_sha256']=hashlib.sha256(target.read_bytes()).hexdigest() + base=['swiftc','-swift-version','6','-I',str(modules),'-module-cache-path',str(output/'ModuleCache'),'-c'] + if sys.platform=='darwin': + sdk=subprocess.check_output(['xcrun','--sdk','macosx','--show-sdk-path'],text=True).strip() + base+=['-sdk',sdk,'-target',platform.machine()+'-apple-macos27.0','-F',str(modules/'PackageFrameworks')] + for name,source,diagnostic in [ + ('immutable','import JSONTransform\nfunc check(_ d: JSONTransformProgramDefinition) { d.source = "changed" }\n','cannot assign to property'), + ('constructor','import JSONTransform\nlet d = JSONTransformProgramDefinition(validatedSource: ".", canonicalNames: [])\n',"inaccessible due to 'private'")]: + path=output/(config+'-'+name+'.swift');path.write_text(source) + result,row=run(base+[str(path),'-o',str(output/(config+'-'+name+'.o'))],config+'-'+name) + row['expected_diagnostic']=diagnostic;row['passed']=ordinary_rejection(result.returncode,result.stderr,diagnostic) + passed=len(commands)==8 and all(x['passed'] for x in commands) + receipt=dict(status='passed' if passed else 'failed',commands=commands,source_inputs=inputs,compiled_guide_fences=1, + configurations=2,witnesses_per_configuration=6,negative_fixture_invocations=4, + scope='Outside-Git public-only path consumer; no private imports or released-version resolution.') + (output/'receipt.json').write_text(json.dumps(receipt,indent=2)+'\n') + print(json.dumps(receipt));return 0 if passed else 1 +if __name__=='__main__':raise SystemExit(main()) diff --git a/Scripts/test_qualification_harness.py b/Scripts/test_qualification_harness.py index 254b3e7..ed4e9f0 100644 --- a/Scripts/test_qualification_harness.py +++ b/Scripts/test_qualification_harness.py @@ -28,6 +28,19 @@ def store_markers(): 'STORE_RUNTIME R%02d passed' % n for n in range(1,11)]) class HarnessTests(unittest.TestCase): + def test_transform_binding_discovery_rejects_incomplete_duplicate_and_foreign_events(self): + ids = DISCOVERY['transform_function_ids'] + command = ['swift', 'test', '--test-product', 'JSONTransformTests'] + events = dict(bundle_counts=[41], module_function_counts={'JSONTransformTests':41}, + function_names=ids, parameter_case_invocations=0) + self.assertTrue(check_result(command, 0, '', events)[0]) + for broken in [dict(events,function_names=ids[:-1]),dict(events,function_names=ids+ids[:1]), + dict(events,module_function_counts={'JSONTests':41}),dict(events,parameter_case_invocations=1)]: + self.assertFalse(check_result(command, 0, '', broken)[0]) + shared = [n for n in ids if 'bindingShared' in n or 'jtxPlanIsSendableAndExecutionsHaveIndependentHealth' in n] + thread = command + ['--sanitize', 'thread'] + self.assertTrue(check_result(thread, 0, '', dict(events,bundle_counts=[3],module_function_counts={'JSONTransformTests':3},function_names=shared))[0]) + def test_acceptance18_discovery_preserves_exact_public_pipeline_identities(self): names = DISCOVERY['structural_acceptance_function_ids'] self.assertEqual(len(names), 9) @@ -439,8 +452,8 @@ def test_exact_discovery_rejects_missing_duplicate_and_changed_parameters(self): identity['function_names'][0] = 'JSONTests.unrelatedReplacement()' self.assertFalse(check_restored(identity)) harness = ['python3', '-B', 'Scripts/test_qualification_harness.py'] - self.assertTrue(check_result(harness, 0, 'HARNESS_TESTS passed cases=30')[0]) - for stale in ['HARNESS_TESTS passed cases=10', 'HARNESS_TESTS passed cases=11', 'HARNESS_TESTS passed cases=12', 'HARNESS_TESTS passed cases=13', 'HARNESS_TESTS passed cases=14', 'HARNESS_TESTS passed cases=15', 'HARNESS_TESTS passed cases=18', 'HARNESS_TESTS passed cases=19', 'HARNESS_TESTS passed cases=20', 'HARNESS_TESTS passed cases=21', 'HARNESS_TESTS passed cases=25', 'HARNESS_TESTS passed cases=26', 'HARNESS_TESTS passed cases=28', '']: + self.assertTrue(check_result(harness, 0, 'HARNESS_TESTS passed cases=31')[0]) + for stale in ['HARNESS_TESTS passed cases=30', 'HARNESS_TESTS passed cases=10', 'HARNESS_TESTS passed cases=11', 'HARNESS_TESTS passed cases=12', 'HARNESS_TESTS passed cases=13', 'HARNESS_TESTS passed cases=14', 'HARNESS_TESTS passed cases=15', 'HARNESS_TESTS passed cases=18', 'HARNESS_TESTS passed cases=19', 'HARNESS_TESTS passed cases=20', 'HARNESS_TESTS passed cases=21', 'HARNESS_TESTS passed cases=25', 'HARNESS_TESTS passed cases=26', 'HARNESS_TESTS passed cases=28', '']: self.assertFalse(check_result(harness, 0, stale)[0]) lifetime = ['python3', '-B', 'Scripts/qualify_schema_example_lifetime.py'] marker = 'SCHEMA_EXAMPLES_CORPUS passed mode=owned scopes=128 values=6 exact=1' diff --git a/Sources/JSONCLIEngine/CLIBuildIdentity.swift b/Sources/JSONCLIEngine/CLIBuildIdentity.swift index 2c10613..a2f826c 100644 --- a/Sources/JSONCLIEngine/CLIBuildIdentity.swift +++ b/Sources/JSONCLIEngine/CLIBuildIdentity.swift @@ -2,6 +2,6 @@ // Generated by Scripts/generate_cli_identity.py; excludes this file. // SHA-256 of the ordered Package.swift + Sources content manifest, not a Git commit. package enum CLIBuildIdentity { - package static let revision = "sha256:d29a80776f26331bf75b98c24d5df1cf5b69ecfbc4d9e39426d393e23c02c3ec" + package static let revision = "sha256:2e6e326d160776afbff186a2c68d01968dc866159ede10038593be415098d91d" package static let kind = "source-sha256" } diff --git a/Sources/JSONTransform/Documentation.docc/JSONTransform.md b/Sources/JSONTransform/Documentation.docc/JSONTransform.md index ee828d3..0dfbfc2 100644 --- a/Sources/JSONTransform/Documentation.docc/JSONTransform.md +++ b/Sources/JSONTransform/Documentation.docc/JSONTransform.md @@ -2,9 +2,9 @@ Compile the existing exact query dialect and pull bounded native JSON results. -This is the optional transformation library in the unreleased 1.7 source line. -Development main and a separately versioned candidate retain their own source -identities and qualification. Publication requires a separate maintainer decision. It shares one private compiler +The library was introduced in 1.7.0. This 1.8 candidate adds exact source definitions +and successful compile reports; indexed selection remains a test-only experiment. +Source identities and qualification stay separate. Publication requires a separate maintainer decision. It shares one private compiler and VM with `swift-json query`; it does not introduce a new interpreter or numeric authority. Plans are immutable and Sendable. Each synchronous invocation has one finite cumulative meter and a guarded, non-Sendable execution handle. @@ -18,6 +18,13 @@ finite cumulative meter and a guarded, non-Sendable execution handle. - ``JSONTransformExecution`` - ``JSONTransformOutput`` +### Source definitions + +- +- ``JSONTransformProgramDefinition`` +- ``JSONTransformDefinitionLimits`` +- ``JSONTransformDefinitionFailure`` + ### Limits and failures - ``JSONTransformLimits`` diff --git a/Sources/JSONTransform/Documentation.docc/SourceDefinitions.md b/Sources/JSONTransform/Documentation.docc/SourceDefinitions.md new file mode 100644 index 0000000..fda7777 --- /dev/null +++ b/Sources/JSONTransform/Documentation.docc/SourceDefinitions.md @@ -0,0 +1,41 @@ +# Persist exact source definitions + +Store a versioned source definition without serializing compiled VM state. +Definitions are immutable and checked Sendable; source keeps its decoded scalar +spelling and declared ASCII names are unique in ascending UTF-8 order. + +```swift +import JSON +import JSONTransform + +let definition = try JSONTransformProgramDefinition.prepare( + source: "$amount", variables: ["amount"]).result +let bytes = try definition.encode().result +let restored = try JSONTransformProgramDefinition.decode(bytes).result +let compiled = try restored.compile() +let amount = JSON.number(try JSONNumber(parsing: "9007199254740993")) +let output = try compiled.result.withExecution(input: .null, bindings: ["amount": amount]) { + try $0.next()?.value +} +precondition(output.result == amount) +precondition(compiled.statistics[.workUnits] > 0) +``` + +Prepare, decode, encode and compile each have their own explicit finite operation. +Within an operation all helpers share one ledger. Codec limits bound source/names, +wire bytes, native traversal, sorting and logical retention. Execution limits are +supplied separately. Logical reservations are not RSS or caller instruction quotas. + +The closed V1 object contains definition, version, dialect, source and variables +in that canonical order. Strict decoding accepts equivalent native JSON spelling +and field order, then encoding emits canonical compact bytes. It rejects unknown +fields, duplicate keys/names and unsupported revisions. Canonical-input checking +is a separate gateway policy; codec decoding alone does not assert byte identity. +Malformed JSON remains a native JSON error; invalid query syntax is a compiler +error when compilation is explicitly requested. Neither schema validation nor a +new numeric or query engine is introduced. + +Use ``JSONTransformProgram/compileWithReport(_:variables:limits:)`` when compiling +ordinary source with successful native work statistics. These reports are data; +they are not unforgeable selection-completion witnesses. The indexed selection +experiment remains test-only in this sprint, with no public selection API. diff --git a/Sources/JSONTransform/JSONTransformProgram.swift b/Sources/JSONTransform/JSONTransformProgram.swift index a0c5cfb..bd06c3c 100644 --- a/Sources/JSONTransform/JSONTransformProgram.swift +++ b/Sources/JSONTransform/JSONTransformProgram.swift @@ -20,6 +20,30 @@ public struct JSONTransformProgram: Sendable { limits: JSONTransformLimits = .init()) throws -> Self { do { var context = try TransformRuntimeContext(limits: limits.runtime) + return try compileValidated(source, variables: variables, context: &context) + } catch { throw JSONTransformFailure.classify(error) } + } + + /// Compiles with the same compiler and returns actual successful native work. + /// Compilation is independent of later execution. Statistics are data, not a + /// selection-completion witness or a charge for caller-owned work. + /// - Parameters: + /// - source: Exact query source in the existing dialect. + /// - variables: Declared ASCII names without dollar prefixes. + /// - limits: Finite compilation policy shared by validation and compilation. + /// - Returns: The immutable program and this compilation's native statistics. + /// - Throws: The same typed compilation, configuration and operational failures as compile. + public static func compileWithReport(_ source: String, variables: Set = [], + limits: JSONTransformLimits = .init()) throws -> JSONOperationReport { + do { + var context = try TransformRuntimeContext(limits: limits.runtime) + let result = try compileValidated(source, variables: variables, context: &context) + return .init(result: result, statistics: context.meter.statistics()) + } catch { throw JSONTransformFailure.classify(error) } + } + + static func compileValidated(_ source: String, variables: Set, + context: inout TransformRuntimeContext) throws -> Self { for name in variables { try context.charge(.bindings) try JSONAdmission.text(name, key: true, meter: &context.meter, owned: true) @@ -36,7 +60,6 @@ public struct JSONTransformProgram: Sendable { } let program = try FilterCompiler.compile(source, variables: variables, context: &context) return .init(program: program, planReservation: context.meter.used(.retainedBytes), variables: variables) - } catch { throw JSONTransformFailure.classify(error) } } /// Runs a synchronous, invocation-owned guarded pull scope. diff --git a/Sources/JSONTransform/JSONTransformProgramDefinition.swift b/Sources/JSONTransform/JSONTransformProgramDefinition.swift new file mode 100644 index 0000000..bdcd32f --- /dev/null +++ b/Sources/JSONTransform/JSONTransformProgramDefinition.swift @@ -0,0 +1,266 @@ +// SPDX-License-Identifier: Apache-2.0 +import JSON +import JSONFilterRuntime + +/// Finite codec policy, independent of compilation and execution policies. +/// Native work, traversal, text, cancellation and live logical storage limits also apply. +public struct JSONTransformDefinitionLimits: Sendable { + /// Native cumulative quotas for one prepare, decode or encode operation. + public var json: JSONLimits = .balanced + /// Maximum UTF-8 bytes in query source; default one MiB. + public var maximumSourceBytes: UInt64 = 1_048_576 + /// Maximum declared names, including every supplied occurrence before duplicate detection. + public var maximumVariableNames: UInt64 = 128 + /// Maximum UTF-8 bytes in one ASCII variable name. + public var maximumNameBytes: UInt64 = 128 + /// Maximum total UTF-8 bytes in all supplied variable names. + public var maximumTotalNameBytes: UInt64 = 8_192 + /// Inclusive input/output wire-byte ceiling, additionally bounded by native quotas. + public var maximumEncodedBytes: UInt64 = 2_097_152 + var checkpointObserver: (@Sendable (UInt64) -> Void)? + /// Creates the finite balanced codec policy; zero capacities remain valid. + public init() {} + + func effective() throws -> JSONLimits { + for cap in [maximumSourceBytes, maximumVariableNames, maximumNameBytes, + maximumTotalNameBytes, maximumEncodedBytes] { + guard cap < UInt64.max else { throw JSONFailure(.configuration, code: .invalidLimits) } + } + var result = json + result[.inputBytes] = min(result[.inputBytes], maximumEncodedBytes) + result[.outputBytes] = min(result[.outputBytes], maximumEncodedBytes) + try result.validate() + return result + } + func meter() throws -> JSONWorkMeter { + var result = try JSONWorkMeter(limits: effective(), domain: .core) + result.checkpointObserver = checkpointObserver + return result + } +} + +/// Closed-definition refusals; native JSON syntax/resource errors and cancellation stay distinct. +public enum JSONTransformDefinitionFailure: Error, Sendable, Equatable { + /// Wrong root, fields, field types or format identifier. + case invalidDefinition + /// A definition revision other than mathematical integer one. + case unsupportedDefinitionRevision + /// A query dialect other than swift-json-query-v1. + case unsupportedDialectRevision + /// A name outside the compiler's ASCII identifier grammar. + case invalidVariableName + /// Two identical names were supplied; no Set conversion silently removed one. + case duplicateVariableName + /// An inclusive codec-specific capacity was exceeded. + /// - Parameters: + /// - metric: Fixed codec policy identifier, without rejected payloads. + /// - limit: Configured finite maximum. + /// - attempted: Observed usage at refusal. + case limitExceeded(metric: String, limit: UInt64, attempted: UInt64) +} + +/// Immutable source definition, separate from compiled VM plans and effective policies. +/// Source scalar spelling is preserved. Names are unique ASCII identifiers in UTF-8 +/// order. No synthesized String equality, source rewriting or bytecode serialization is used. +public struct JSONTransformProgramDefinition: Sendable { + /// Exact decoded query source, including whitespace and scalar spelling. + public let source: String + /// Unique declared variable names in exact ascending UTF-8 order. + public let variableNames: [String] + /// Known wire definition revision; independent of implementation identity. + public var definitionRevision: UInt32 { 1 } + /// Known query-language revision; independent of limits and the CLI source digest. + public var dialectRevision: String { "swift-json-query-v1" } + + private init(validatedSource: String, canonicalNames: [String]) { + source = validatedSource; variableNames = canonicalNames + } + + /// Validates authored source metadata using one finite ledger; query compilation is explicit. + /// - Parameters: + /// - source: Exact query source; syntax is checked only by compile. + /// - variables: Names before sorting; exact duplicates are rejected. + /// - limits: Finite text, sort, traversal and retention policy. + /// - Returns: The immutable definition and actual successful validation accounting. + /// - Throws: Definition/name/capacity, native resource/configuration errors or cancellation. + public static func prepare(source: String, variables: [String] = [], + limits: JSONTransformDefinitionLimits = .init()) throws -> JSONOperationReport { + var meter = try limits.meter() + let result = try make(source: source, variables: variables, limits: limits, meter: &meter) + return .init(result: result, statistics: meter.statistics()) + } + + /// Decodes one strict native JSON document with an exact closed V1 field set. + /// Field order, whitespace and equivalent JSON escapes may differ; encoding + /// produces canonical bytes. Duplicate fields/names and unknown fields fail. + /// - Parameters: + /// - bytes: Caller-owned UTF-8 input; no ambient I/O or byte-copy prepass. + /// - limits: One cumulative policy for parsing, field checks and canonical names. + /// - Returns: The decoded definition and actual successful cumulative accounting. + /// - Throws: Native JSON syntax/resource errors, definition failures or cancellation. + public static func decode(_ bytes: [UInt8], limits: JSONTransformDefinitionLimits = .init()) throws -> JSONOperationReport { + var meter = try limits.meter() + var root = try JSONParser().parse(bytes, meter: &meter) + guard case .object(let object) = root, object.count == 5 else { throw JSONTransformDefinitionFailure.invalidDefinition } + try meter.retain(80) + var fields: [JSON?] = Array(repeating: nil, count: 5) + for member in object { + var found = false + for (index, key) in ["definition", "version", "dialect", "source", "variables"].enumerated() { + if try JSONComparison.textEquals(member.key.string, key, meter: &meter) { + fields[index] = member.value; found = true; break + } + } + guard found else { throw JSONTransformDefinitionFailure.invalidDefinition } + } + guard case .string(let tag) = fields[0], + try JSONComparison.textEquals(tag, "swift-json-transform-source", meter: &meter), + case .number(let revision) = fields[1], case .string(let dialect) = fields[2], + case .string(let source) = fields[3], case .array(let names) = fields[4] else { + throw JSONTransformDefinitionFailure.invalidDefinition + } + guard try revision.compare(to: JSONNumber(integer: 1), meter: &meter) == .orderedSame else { + throw JSONTransformDefinitionFailure.unsupportedDefinitionRevision + } + guard try JSONComparison.textEquals(dialect, "swift-json-query-v1", meter: &meter) else { + throw JSONTransformDefinitionFailure.unsupportedDialectRevision + } + try require(UInt64(names.count), limits.maximumVariableNames, "variableNames", meter: &meter) + var strings: [String] = [] + for name in names { + try meter.work() + guard case .string(let text) = name else { throw JSONTransformDefinitionFailure.invalidDefinition } + try meter.retain(16); strings.append(text) + } + let baseline = meter.used(.retainedBytes) + let result = try make(source: source, variables: strings, limits: limits, meter: &meter) + let resultBytes = meter.used(.retainedBytes) - baseline + root = .null; fields = []; strings = [] + // Parsed tree and extraction aliases are temporary; result strings were + // independently reserved by make. Work is never refunded or restarted. + try meter.checkpoint(); try meter.release(meter.used(.retainedBytes) - resultBytes) + return .init(result: result, statistics: meter.statistics()) + } + + /// Encodes canonical compact UTF-8 with fixed metadata order and native escaping. + /// The ordered fields are definition, version, dialect, source, variables. + /// - Parameter limits: One finite ledger for admission, metadata, sorting and output. + /// - Returns: Canonical bytes without a trailing newline and successful accounting. + /// - Throws: Codec/native capacity or configuration errors, or cancellation. + public func encode(limits: JSONTransformDefinitionLimits = .init()) throws -> JSONOperationReport<[UInt8]> { + var meter = try limits.meter() + let admitted = try Self.make(source: source, variables: variableNames, limits: limits, meter: &meter) + var names: [JSON] = [] + for name in admitted.variableNames { try meter.work(); try meter.retain(16); names.append(.string(name)) } + try meter.retain(5 * JSONLogicalWeights.member) + let members: [JSONMember] = [ + .init("definition", value: .string("swift-json-transform-source")), + .init("version", value: .number(JSONNumber(integer: 1))), + .init("dialect", value: .string(dialectRevision)), + .init("source", value: .string(admitted.source)), + .init("variables", value: .array(names))] + let object = try JSONObject.checked(members: members, meter: &meter) + let bytes = try JSONWriter().write(.object(object), meter: &meter) + try meter.checkpoint(); try meter.release(meter.used(.retainedBytes) - UInt64(bytes.count)) + return .init(result: bytes, statistics: meter.statistics()) + } + + /// Compiles this definition using the existing compiler and one compilation context. + /// The source definition and codec policy are not captured by the compiled plan. + /// - Parameter limits: Explicit finite compilation policy, independent of codec policy. + /// - Returns: The native compiled program and actual successful compilation work. + /// - Throws: The existing typed compiler/configuration/resource/cancellation failures. + public func compile(limits: JSONTransformLimits = .init()) throws -> JSONOperationReport { + do { + var context = try TransformRuntimeContext(limits: limits.runtime) + var names: Set = [] + for name in variableNames { + // Admit hashing/scan work before constructing an owned Set entry. + for _ in name.utf8 { try context.work() } + try context.meter.retain(32) + names.insert(name) + try context.checkpoint() + } + let result = try JSONTransformProgram.compileValidated(source, variables: names, context: &context) + return .init(result: result, statistics: context.meter.statistics()) + } catch { throw JSONTransformFailure.classify(error) } + } + + private static func require(_ attempted: UInt64, _ limit: UInt64, _ metric: String, + meter: inout JSONWorkMeter) throws { + try meter.checkpoint() + guard attempted <= limit else { + throw JSONTransformDefinitionFailure.limitExceeded(metric: metric, limit: limit, attempted: attempted) + } + } + private static func scanned(_ text: String, maximum: UInt64, metric: String, + meter: inout JSONWorkMeter) throws -> UInt64 { + var count: UInt64 = 0 + for _ in text.utf8 { + try meter.work(); count = try meter.adding(.aggregateTextBytes, count, 1) + try require(count, maximum, metric, meter: &meter) + } + return count + } + private static func make(source: String, variables: [String], limits: JSONTransformDefinitionLimits, + meter: inout JSONWorkMeter) throws -> Self { + try meter.retain(64) + _ = try scanned(source, maximum: limits.maximumSourceBytes, metric: "sourceBytes", meter: &meter) + try JSONAdmission.text(source, meter: &meter, owned: true) + try require(UInt64(variables.count), limits.maximumVariableNames, "variableNames", meter: &meter) + var names: [String] = [], total: UInt64 = 0 + try meter.retain(32) + for name in variables { + let size = try scanned(name, maximum: limits.maximumNameBytes, metric: "nameBytes", meter: &meter) + total = try meter.adding(.aggregateTextBytes, total, size) + try require(total, limits.maximumTotalNameBytes, "totalNameBytes", meter: &meter) + var first = true + for byte in name.utf8 { + try meter.work() + let letter = byte == 95 || (65...90).contains(byte) || (97...122).contains(byte) + guard letter || (!first && (48...57).contains(byte)) else { throw JSONTransformDefinitionFailure.invalidVariableName } + first = false + } + guard !first else { throw JSONTransformDefinitionFailure.invalidVariableName } + try JSONAdmission.text(name, key: true, meter: &meter, owned: true) + try meter.retain(16); names.append(name) + } + var scratch: [String] = [] + for name in names { try meter.work(); try meter.retain(16); scratch.append(name) } + var width = 1 + while width < names.count { + var start = 0 + while start < names.count { + let middle = start + min(width, names.count - start) + let end = middle + min(width, names.count - middle) + var a = start, b = middle + for target in start.. names.count / 2 { break }; width *= 2 + } + if names.count > 1 { + for index in 1.. Bool { + var a = lhs.utf8.makeIterator(), b = rhs.utf8.makeIterator() + while true { + try meter.work(); let x = a.next(), y = b.next() + if x != y { if let x, let y { return x < y }; return x == nil } + if x == nil { return false } + } + } +} diff --git a/Tests/JSONTransformTests/DefinitionTests.swift b/Tests/JSONTransformTests/DefinitionTests.swift new file mode 100644 index 0000000..8d19dc6 --- /dev/null +++ b/Tests/JSONTransformTests/DefinitionTests.swift @@ -0,0 +1,205 @@ +// SPDX-License-Identifier: Apache-2.0 +import Testing +import Foundation +import Synchronization +import JSON +@testable import JSONTransform + +private enum DefinitionTestError: Error { case expectedFailure } +private func definitionFailure(_ body: () throws -> Void) throws -> JSONTransformDefinitionFailure { + do { try body(); Issue.record("Expected definition failure"); throw DefinitionTestError.expectedFailure } + catch let error as JSONTransformDefinitionFailure { return error } +} +private func definitionResource(_ body: () throws -> Void) throws -> JSONFailure { + do { try body(); Issue.record("Expected native failure"); throw DefinitionTestError.expectedFailure } + catch let error as JSONFailure { #expect(error.category == .resource); return error } +} +private let dotDefinition = #"{"definition":"swift-json-transform-source","version":1,"dialect":"swift-json-query-v1","source":".","variables":[]}"# + +@Test func bindingD01CanonicalGoldenBytesAndReload() throws { + let definition = try JSONTransformProgramDefinition.prepare(source: "$z + $a", variables: ["z", "a"]) + let expected = #"{"definition":"swift-json-transform-source","version":1,"dialect":"swift-json-query-v1","source":"$z + $a","variables":["a","z"]}"# + let bytes = try definition.result.encode() + #expect(bytes.result == Array(expected.utf8)) + let decoded = try JSONTransformProgramDefinition.decode(bytes.result) + #expect(decoded.result.variableNames == ["a", "z"]) + #expect(try decoded.result.encode().result == bytes.result) + #expect(decoded.result.definitionRevision == 1 && decoded.result.dialectRevision == "swift-json-query-v1") + #expect(decoded.statistics[.workUnits] > definition.statistics[.workUnits]) +} + +@Test func bindingD02CanonicalEscapingAndExactScalarSource() throws { + let source = "é e\u{301}\n\t\"\\/" + let expected = #"{"definition":"swift-json-transform-source","version":1,"dialect":"swift-json-query-v1","source":"é é\u000a\u0009\"\\/","variables":[]}"# + let a = try JSONTransformProgramDefinition.prepare(source: source).result + #expect(try a.encode().result == Array(expected.utf8)) + let reload = try JSONTransformProgramDefinition.decode(Array(expected.utf8)).result + #expect(Array(reload.source.utf8) == Array(source.utf8)) + let composed = try JSONTransformProgramDefinition.prepare(source: "é").result + let decomposed = try JSONTransformProgramDefinition.prepare(source: "e\u{301}").result + #expect(try composed.encode().result != decomposed.encode().result) +} + +@Test func bindingD03StrictSemanticDecodeCanonicalizesPresentation() throws { + let input = #" { "variables":[], "source":"\u002e", "dialect":"swift-json-query-v1", "version":1.0, "definition":"swift-json-transform-source" } "# + let value = try JSONTransformProgramDefinition.decode(Array(input.utf8)).result + #expect(try value.encode().result == Array(dotDefinition.utf8)) + let reordered = #"{"definition":"swift-json-transform-source","version":1,"dialect":"swift-json-query-v1","source":".","variables":["z","_a","B"]}"# + #expect(try JSONTransformProgramDefinition.decode(Array(reordered.utf8)).result.variableNames == ["B", "_a", "z"]) +} + +@Test func bindingD04ClosedFieldsVersionsAndKindsArePrecise() throws { + for input in ["null", "[]", "{}", dotDefinition.replacingOccurrences(of: "\"version\":1", with: "\"version\":true"), + dotDefinition.replacingOccurrences(of: "\"source\":\".\"", with: "\"unknown\":\".\""), + dotDefinition.replacingOccurrences(of: "[]}", with: "[],\"extra\":null}"), + dotDefinition.replacingOccurrences(of: "[]}", with: "[42]}"), + dotDefinition.replacingOccurrences(of: "swift-json-transform-source", with: "other")] { + #expect(try definitionFailure { _ = try JSONTransformProgramDefinition.decode(Array(input.utf8)) } == .invalidDefinition) + } + let version = dotDefinition.replacingOccurrences(of: "\"version\":1", with: "\"version\":2") + #expect(try definitionFailure { _ = try JSONTransformProgramDefinition.decode(Array(version.utf8)) } == .unsupportedDefinitionRevision) + let dialect = dotDefinition.replacingOccurrences(of: "query-v1", with: "query-v2") + #expect(try definitionFailure { _ = try JSONTransformProgramDefinition.decode(Array(dialect.utf8)) } == .unsupportedDialectRevision) +} + +@Test func bindingD05NativeGrammarAndDuplicateFieldsStayAuthoritative() throws { + let duplicate = dotDefinition.replacingOccurrences(of: "\"version\":1", with: "\"version\":1,\"ver\\u0073ion\":1") + do { _ = try JSONTransformProgramDefinition.decode(Array(duplicate.utf8)); Issue.record("Accepted duplicate key") } + catch let error as JSONFailure { #expect(error.category == .syntax && error.code == .duplicateKey) } + for bytes in [Array((dotDefinition + "false").utf8), [0xef, 0xbb, 0xbf] + Array(dotDefinition.utf8), [0xff]] { + do { _ = try JSONTransformProgramDefinition.decode(bytes); Issue.record("Accepted malformed JSON") } + catch let error as JSONFailure { #expect(error.category == .syntax) } + } +} + +@Test func bindingD06NamesRejectBeforeSetConversion() throws { + for name in ["", "$x", "é", "K", "a-b", "1a", "a\u{0}"] { + #expect(try definitionFailure { _ = try JSONTransformProgramDefinition.prepare(source: ".", variables: [name]) } == .invalidVariableName) + } + #expect(try definitionFailure { _ = try JSONTransformProgramDefinition.prepare(source: ".", variables: ["b","a","b"]) } == .duplicateVariableName) + #expect(try definitionFailure { _ = try JSONTransformProgramDefinition.prepare(source: ".", variables: ["K","K"]) } == .invalidVariableName) + let bytes = Array(dotDefinition.replacingOccurrences(of: "[]}", with: "[\"a\",\"a\"]}").utf8) + #expect(try definitionFailure { _ = try JSONTransformProgramDefinition.decode(bytes) } == .duplicateVariableName) +} + +@Test func bindingD07SpecificCapsAreInclusiveAndFinite() throws { + var limits = JSONTransformDefinitionLimits(); limits.maximumSourceBytes = 1 + _ = try JSONTransformProgramDefinition.prepare(source: ".", limits: limits) + limits.maximumSourceBytes = 0 + #expect(try definitionFailure { _ = try JSONTransformProgramDefinition.prepare(source: ".", limits: limits) } == .limitExceeded(metric: "sourceBytes", limit: 0, attempted: 1)) + _ = try JSONTransformProgramDefinition.prepare(source: "", limits: limits) + limits = .init(); limits.maximumVariableNames = 2 + _ = try JSONTransformProgramDefinition.prepare(source: ".", variables: ["a","b"], limits: limits) + limits.maximumVariableNames = 1 + #expect(try definitionFailure { _ = try JSONTransformProgramDefinition.prepare(source: ".", variables: ["a","b"], limits: limits) } == .limitExceeded(metric: "variableNames", limit: 1, attempted: 2)) + limits = .init(); limits.maximumNameBytes = 1 + _ = try JSONTransformProgramDefinition.prepare(source: ".", variables: ["a"], limits: limits) + #expect(try definitionFailure { _ = try JSONTransformProgramDefinition.prepare(source: ".", variables: ["ab"], limits: limits) } == .limitExceeded(metric: "nameBytes", limit: 1, attempted: 2)) + limits = .init(); limits.maximumTotalNameBytes = 2 + _ = try JSONTransformProgramDefinition.prepare(source: ".", variables: ["a","b"], limits: limits) + limits.maximumTotalNameBytes = 1 + #expect(try definitionFailure { _ = try JSONTransformProgramDefinition.prepare(source: ".", variables: ["a","b"], limits: limits) } == .limitExceeded(metric: "totalNameBytes", limit: 1, attempted: 2)) + let keys: [WritableKeyPath] = [\.maximumSourceBytes, \.maximumVariableNames, \.maximumNameBytes, \.maximumTotalNameBytes, \.maximumEncodedBytes] + for key in keys { + limits = .init(); limits[keyPath: key] = .max + do { _ = try JSONTransformProgramDefinition.prepare(source: ".", limits: limits); Issue.record("Accepted infinite cap") } + catch let error as JSONFailure { #expect(error.category == .configuration && error.code == .invalidLimits) } + } +} + +@Test func bindingD08WireAndOwnedWorkRetentionHaveExactBoundaries() throws { + let definition = try JSONTransformProgramDefinition.prepare(source: ".").result + let encoded = try definition.encode(); var limits = JSONTransformDefinitionLimits() + limits.maximumEncodedBytes = UInt64(encoded.result.count) + #expect(try definition.encode(limits: limits).result == encoded.result) + #expect(try JSONTransformProgramDefinition.decode(encoded.result, limits: limits).result.source == ".") + limits.maximumEncodedBytes -= 1 + #expect(try definitionResource { _ = try definition.encode(limits: limits) }.resource?.metric == .outputBytes) + #expect(try definitionResource { _ = try JSONTransformProgramDefinition.decode(encoded.result, limits: limits) }.resource?.metric == .inputBytes) + for operation in 0..<3 { + func run(_ policy: JSONTransformDefinitionLimits) throws -> JSONOperationStatistics { + if operation == 0 { return try JSONTransformProgramDefinition.prepare(source: "$b", variables: ["b","a"], limits: policy).statistics } + if operation == 1 { return try JSONTransformProgramDefinition.decode(encoded.result, limits: policy).statistics } + return try definition.encode(limits: policy).statistics + } + let measured = try run(.init()) + for metric in [JSONResourceMetric.workUnits, .retainedBytes] { + var p = JSONTransformDefinitionLimits() + let boundary = metric == .workUnits ? measured[metric] : measured.peakLogicalRetainedBytes + #expect(boundary > 0); p.json[metric] = boundary; _ = try run(p) + p.json[metric] = boundary - 1 + #expect(try definitionResource { _ = try run(p) }.resource?.metric == metric) + } + } + #expect(encoded.statistics[.retainedBytes] == UInt64(encoded.result.count)) +} + +@Test func bindingD09CompileReportsPreserveExistingPlanBehaviorAndBudget() throws { + let report = try JSONTransformProgram.compileWithReport("[$n, .]", variables: ["n"]) + #expect(report.statistics.domain == .core && report.statistics[.workUnits] > 0) + var limits = JSONTransformLimits(); limits[.workUnits] = report.statistics[.workUnits] + let old = try JSONTransformProgram.compile("[$n, .]", variables: ["n"], limits: limits) + for plan in [report.result, old] { + let result = try plan.withExecution(input: 1, bindings: ["n": 42]) { try $0.next()?.value } + #expect(result.result == .array([42,1])) + } + limits[.workUnits] -= 1 + do { _ = try JSONTransformProgram.compile("[$n, .]", variables: ["n"], limits: limits); Issue.record("Compile reset work") } + catch let error as JSONTransformFailure { #expect(error.category == .resource && error.metric == "workUnits") } + let definition = try JSONTransformProgramDefinition.prepare(source: "[$n, .]", variables: ["n"]).result + #expect(try definition.compile().statistics[.workUnits] > report.statistics[.workUnits]) +} + +@Test func bindingD10CodecDoesNotInventQueryValidationOrDecimalAuthority() throws { + let bad = try JSONTransformProgramDefinition.prepare(source: "(").result + _ = try JSONTransformProgramDefinition.decode(bad.encode().result) + do { _ = try bad.compile(); Issue.record("Accepted invalid query") } + catch let error as JSONTransformFailure { #expect(error.category == .compilation) } + let source = "[9007199254740993,0.1,\"é\"]" + let definition = try JSONTransformProgramDefinition.prepare(source: source).result + let reload = try JSONTransformProgramDefinition.decode(definition.encode().result).result + let plan = try reload.compile().result + let result = try plan.withExecution(input: .null) { try $0.next()?.value } + let expected: JSON = .array([.number(try JSONNumber(parsing: "9007199254740993")), + .number(try JSONNumber(parsing: "0.1")),.string("e\u{301}")]) + #expect(result.result == expected) +} + +@Test func bindingD11CancellationOccursInsideOwnedCodecWork() async throws { + let handle = Mutex?>(nil) + let ready = Mutex?>(nil) + let observed = Mutex(0) + let task = Task { + await withCheckedContinuation { continuation in ready.withLock { $0 = continuation } } + var limits = JSONTransformDefinitionLimits() + limits.checkpointObserver = { work in + if work >= 32 { observed.withLock { $0 = work }; handle.withLock { $0?.cancel() } } + } + _ = try JSONTransformProgramDefinition.prepare(source: String(repeating: "x", count: 4096), limits: limits) + } + handle.withLock { $0 = task } + var continuation: CheckedContinuation? + while continuation == nil { continuation = ready.withLock { $0 }; if continuation == nil { await Task.yield() } } + continuation?.resume() + do { try await task.value; Issue.record("Owned codec ignored cancellation") } catch is CancellationError {} + #expect(observed.withLock { $0 } >= 32) +} + +@Test func bindingSharedDefinitionAndCompilationHaveIndependentHealth() async throws { + let definition = try JSONTransformProgramDefinition.prepare(source: ".").result + let results = try await withThrowingTaskGroup(of: Bool.self) { group in + for index in 0..<6 { group.addTask { + if index == 0 { + var limits = JSONTransformLimits(); limits[.workUnits] = 0 + do { _ = try definition.compile(limits: limits); return false } + catch let error as JSONTransformFailure { return error.category == .resource } + } + let bytes = try definition.encode().result + let reload = try JSONTransformProgramDefinition.decode(bytes).result + let plan = try reload.compile().result + return try plan.withExecution(input: 42) { try $0.next()?.value }.result == 42 + } } + var values: [Bool] = []; for try await value in group { values.append(value) }; return values + } + #expect(results.count == 6 && results.allSatisfy { $0 }) +} diff --git a/Tests/JSONTransformTests/IndexedSelectionSpike.swift b/Tests/JSONTransformTests/IndexedSelectionSpike.swift new file mode 100644 index 0000000..6aee3fe --- /dev/null +++ b/Tests/JSONTransformTests/IndexedSelectionSpike.swift @@ -0,0 +1,276 @@ +// SPDX-License-Identifier: Apache-2.0 +// Test-only ownership experiment. This is not a shipped selection API. +import Testing +import Synchronization +import JSON +import JSONFilterRuntime +@testable import JSONTransform + +private enum SelectionSpikeFailure: Error, Equatable { + case invalidPosition, incomplete, closed, provider, invalidBindings +} +private final class PopulationOwner { + let population: FilterYield + let metadata: TransformValueLease + init(_ population: FilterYield, metadata: TransformValueLease) { + self.population = population; self.metadata = metadata + } +} +private struct SpikePosition { + let ordinal: UInt64 + let owner: PopulationOwner + let reservation: TransformValueLease +} +private struct SpikeInput: FilterInput { + var fail = false + var calls = 0 + mutating func next(context: inout TransformRuntimeContext) throws -> FilterYield? { + calls += 1 + if fail { throw SelectionSpikeFailure.provider } + return nil + } +} +private final class SelectionSpike { + private var context: TransformRuntimeContext + private var vm: FilterVM? + private var owner: PopulationOwner? + private var input = SpikeInput() + private var failure: (any Error)? + private var ended = false, closed = false + private var planBytes: UInt64 = 0 + private init(_ context: consuming TransformRuntimeContext) { self.context = consume context } + + static func make(_ plan: JSONTransformProgram, population: [JSON], + limits: JSONTransformLimits = .init(), providerFailure: Bool = false, + observer: (@Sendable (UInt64) -> Void)? = nil, + projectionObserver: (@Sendable (Int) -> Void)? = nil, + cleanupObserver: (@Sendable (UInt64) -> Void)? = nil) throws -> SelectionSpike { + var context = try TransformRuntimeContext(limits: limits.runtime) + try context.meter.retain(128) + context.meter.checkpointObserver = observer + let result = SelectionSpike(consume context) + do { + guard plan.variables == ["values"] else { throw SelectionSpikeFailure.invalidBindings } + try result.prepare(plan, population: population, providerFailure: providerFailure, projectionObserver: projectionObserver) + return result + } catch { + let cause = error + do { try result.close() } catch { Issue.record("Prototype preparation cleanup failed: \(error)") } + if let cleanupObserver { + do { cleanupObserver(try result.settlement()[.retainedBytes]) } + catch { Issue.record("Prototype terminal observation failed: \(error)") } + } + throw cause + } + } + private func prepare(_ plan: JSONTransformProgram, population: [JSON], providerFailure: Bool, + projectionObserver: (@Sendable (Int) -> Void)?) throws { + try context.meter.retain(plan.planReservation); planBytes = plan.planReservation + // A prototype population cap, independent of lexical token counts. + guard population.count <= 1024 else { throw SelectionSpikeFailure.invalidBindings } + try context.charge(.bindings) + let value = try FilterValues.produced(.array(population), context: &context) + try context.meter.retain(48) + let metadata = try TransformValueLease(ownedRetained: 48, context: &context) + owner = PopulationOwner(value, metadata: metadata) + var indices: [JSON] = [], scratch: UInt64 = 0 + defer { + do { try context.meter.release(scratch) } + catch { Issue.record("Prototype projection scratch cleanup failed: \(error)") } + } + for ordinal in population.indices { + try context.work(); try context.meter.retain(64); scratch += 64 + indices.append(.number(JSONNumber(integer: ordinal))) + projectionObserver?(indices.count) + } + let root = try FilterValues.array(indices, context: &context) + indices = []; try context.meter.release(scratch); scratch = 0 + try context.charge(.totalRecords) + vm = try FilterVM(program: plan.program, input: root, bindings: ["values": value], context: &context) + input.fail = providerFailure + } + func next() throws -> SpikePosition? { + do { + try context.reclaimValueLeases() + if let failure { throw failure } + try context.checkpoint() + guard !closed else { throw SelectionSpikeFailure.closed } + guard var active = vm else { throw SelectionSpikeFailure.closed } + vm = nil; defer { vm = active } + guard let raw = try active.next(input: &input, context: &context) else { ended = true; return nil } + let before = context.meter.used(.retainedBytes) + do { + try context.meter.resetValueBounds() + try JSONAdmission.value(raw.value, meter: &context.meter) + } catch { + try context.meter.release(context.meter.used(.retainedBytes)-before); throw error + } + try context.charge(.totalResults) + guard case .number(let number) = raw.value else { throw SelectionSpikeFailure.invalidPosition } + let ordinal: UInt64 + do { ordinal = try number.integer(as: UInt64.self, meter: &context.meter) } + catch let error as JSONFailure where error.category == .conversion { throw SelectionSpikeFailure.invalidPosition } + guard let native = Int(exactly: ordinal), let owner, + case .array(let values) = owner.population.value, native < values.count else { + throw SelectionSpikeFailure.invalidPosition + } + try context.meter.retain(32) + let slot = try TransformValueLease(ownedRetained: 32, context: &context) + return .init(ordinal: ordinal, owner: owner, reservation: slot) + } catch { + if failure == nil { failure = error; context.latch(error) } + throw failure ?? error + } + } + func accepts(_ position: SpikePosition) -> Bool { owner === position.owner } + func finish() throws -> JSONOperationStatistics { + if failure == nil && !ended { failure = SelectionSpikeFailure.incomplete; context.latch(SelectionSpikeFailure.incomplete) } + let cause = failure + try close() + if let cause { throw cause } + return try settlement() + } + func settlement() throws -> JSONOperationStatistics { + try context.reclaimValueLeases() + return context.meter.statistics() + } + var providerCalls: Int { input.calls } + private func close() throws { + guard !closed else { return }; closed = true + if var active = vm { vm = nil; try active.close(context: &context) } + owner = nil + try context.reclaimValueLeases() + try context.meter.release(planBytes + 128); planBytes = 0 + try context.release(.bindings, context.used(.bindings)) + } +} +private func spikePlan(_ source: String) throws -> JSONTransformProgram { + try JSONTransformProgram.compile(source, variables: ["values"]) +} + +@Test func bindingP01ExplicitIndicesPreserveDuplicatesOrderAndTies() throws { + let row = try JSON.parse(#"{"score":1}"#) + for (source, expected) in [(".[] | select(. == 1)", [UInt64(1)]), + ("1,0,1", [1,0,1]), + ("sort_by([$values[.].score, .])[]", [0,1])] { + let probe = try SelectionSpike.make(spikePlan(source), population: [row,row]) + var ordinals: [UInt64] = [] + while let position = try probe.next() { ordinals.append(position.ordinal) } + let report = try probe.finish() + #expect(ordinals == expected && report[.workUnits] > 0) + } + let empty = try SelectionSpike.make(spikePlan("empty"), population: []) + #expect(try empty.next() == nil); #expect(try empty.finish()[.workUnits] > 0) +} + +@Test func bindingP02InvalidOutputAndAbandonedPrefixNeverCertify() throws { + for bad in ["null", "true", "\"0\"", "-1", "0.1", "18446744073709551616", "9223372036854775808", "1"] { + let probe = try SelectionSpike.make(spikePlan("try (0, " + bad + ") catch 0"), population: [42]) + #expect(try probe.next()?.ordinal == 0) + for _ in 0..<2 { + do { _ = try probe.next(); Issue.record("Invalid position escaped") } + catch let error as SelectionSpikeFailure { #expect(error == .invalidPosition) } + } + do { _ = try probe.finish(); Issue.record("Caught refusal certified selection") } + catch let error as SelectionSpikeFailure { #expect(error == .invalidPosition) } + } + let prefix = try SelectionSpike.make(spikePlan("0,0"), population: [42]) + #expect(try prefix.next()?.ordinal == 0) + do { _ = try prefix.finish(); Issue.record("Abandoned prefix certified") } + catch let error as SelectionSpikeFailure { #expect(error == .incomplete) } +} + +@Test func bindingP03PopulationLeaseSurvivesCloseAndForeignOwnerRefuses() throws { + let plan = try spikePlan("0") + let source: [JSON] = [.string(String(repeating: "x", count: 8192))] + let a = try SelectionSpike.make(plan, population: source), b = try SelectionSpike.make(plan, population: source) + var position = try a.next(), alias = position + #expect(position?.ordinal == 0) + #expect(position.map { !b.accepts($0) } == true) + #expect(try a.next() == nil) + _ = try a.finish() + let retainedBefore = try withExtendedLifetime(alias) { try a.settlement()[.retainedBytes] } + #expect(retainedBefore > 8192) + position = nil + let retainedAfter = try withExtendedLifetime(alias) { try a.settlement()[.retainedBytes] } + #expect(retainedAfter > 8192) + alias = nil + #expect(try a.settlement()[.retainedBytes] == 0) + do { _ = try a.next(); Issue.record("Closed handle resumed") } + catch let error as SelectionSpikeFailure { #expect(error == .closed) } + #expect(try b.next()?.ordinal == 0); #expect(try b.next() == nil); _ = try b.finish() + for source in ["0.0", "-0"] { + let probe = try SelectionSpike.make(spikePlan(source), population: [42]) + #expect(try probe.next()?.ordinal == 0); #expect(try probe.next() == nil); _ = try probe.finish() + } +} + +@Test func bindingP04ResourceAndProviderFailuresStayTerminal() throws { + var limits = JSONTransformLimits(); limits[.results] = 1 + let probe = try SelectionSpike.make(spikePlan("0,0"), population: [42], limits: limits) + #expect(try probe.next()?.ordinal == 0) + for _ in 0..<2 { + do { _ = try probe.next(); Issue.record("Results cap reset") } + catch { let e = JSONTransformFailure.classify(error); #expect(e.category == .resource && e.metric == "results") } + } + do { _ = try probe.finish(); Issue.record("Failed quota certified") } + catch { #expect(JSONTransformFailure.classify(error).category == .resource) } + let provider = try SelectionSpike.make(spikePlan("try input catch 0"), population: [42], providerFailure: true) + for _ in 0..<2 { + do { _ = try provider.next(); Issue.record("Provider error was swallowed") } + catch let error as SelectionSpikeFailure { #expect(error == .provider) } + } + #expect(provider.providerCalls == 1) + do { _ = try provider.finish(); Issue.record("Provider error certified") } + catch let error as SelectionSpikeFailure { #expect(error == .provider) } +} + +@Test func bindingSharedSelectionSpikeReusesOnePlanIndependently() async throws { + let plan = try spikePlan(".[]") + let values = try await withThrowingTaskGroup(of: Bool.self) { group in + for _ in 0..<6 { group.addTask { + let probe = try SelectionSpike.make(plan, population: [42,42]) + let a = try probe.next()?.ordinal, b = try probe.next()?.ordinal + guard try probe.next() == nil else { return false } + return try probe.finish()[.workUnits] > 0 && a == 0 && b == 1 + } } + var result: [Bool] = []; for try await x in group { result.append(x) }; return result + } + #expect(values.count == 6 && values.allSatisfy { $0 }) +} + +@Test func bindingP05ProjectionEvaluationAndCancellationShareOneLedger() async throws { + let plan = try spikePlan("0"), population: [JSON] = Array(repeating: .string("row"), count: 64) + func run(_ limits: JSONTransformLimits) throws -> JSONOperationStatistics { + let probe = try SelectionSpike.make(plan, population: population, limits: limits) + do { + while let position = try probe.next() { #expect(position.ordinal == 0) } + return try probe.finish() + } catch { + let first = error + do { _ = try probe.finish() } catch { #expect(JSONTransformFailure.classify(error) == JSONTransformFailure.classify(first)) } + throw first + } + } + let measured = try run(.init()) + var limits = JSONTransformLimits(); limits[.workUnits] = measured[.workUnits] + #expect(try run(limits)[.workUnits] == measured[.workUnits]) + limits[.workUnits] -= 1 + do { _ = try run(limits); Issue.record("Projection/evaluation restarted work") } + catch { #expect(JSONTransformFailure.classify(error).category == .resource) } + let handle = Mutex?>(nil), ready = Mutex?>(nil) + let observed = Mutex(0), finalRetained = Mutex(nil) + let task = Task { + await withCheckedContinuation { continuation in ready.withLock { $0 = continuation } } + _ = try SelectionSpike.make(plan, population: population, projectionObserver: { count in + if count == 4 { observed.withLock { $0 = count }; handle.withLock { $0?.cancel() } } + }, cleanupObserver: { value in finalRetained.withLock { $0 = value } }) + } + handle.withLock { $0 = task } + var continuation: CheckedContinuation? + while continuation == nil { continuation = ready.withLock { $0 }; if continuation == nil { await Task.yield() } } + continuation?.resume() + do { try await task.value; Issue.record("Projection ignored cancellation") } catch is CancellationError {} + #expect(observed.withLock { $0 } == 4) + #expect(finalRetained.withLock { $0 } == 0) +} diff --git a/Tools/TransformBindingQualification/Package.swift b/Tools/TransformBindingQualification/Package.swift new file mode 100644 index 0000000..91d5f5b --- /dev/null +++ b/Tools/TransformBindingQualification/Package.swift @@ -0,0 +1,8 @@ +// swift-tools-version: 6.4 +// SPDX-License-Identifier: Apache-2.0 +import PackageDescription +let package = Package(name: "TransformBindingQualification", platforms: [.macOS("27.0"), .iOS("27.0")], + dependencies: [.package(name: "Swift-JSON", path: "../..")], + targets: [.executableTarget(name: "TransformBindingConsumer", dependencies: [ + .product(name: "JSON", package: "Swift-JSON"), .product(name: "JSONTransform", package: "Swift-JSON") + ])], swiftLanguageModes: [.v6]) diff --git a/Tools/TransformBindingQualification/README.md b/Tools/TransformBindingQualification/README.md new file mode 100644 index 0000000..aa8fb40 --- /dev/null +++ b/Tools/TransformBindingQualification/README.md @@ -0,0 +1,8 @@ +# Source definition public consumer + +Run `swift run --package-path Tools/TransformBindingQualification TransformBindingConsumer`. +Only JSON and JSONTransform products are imported. Six named witnesses cover the +compiled DocC example, independent canonical bytes, canonical names, duplicate-name +rejection, operational compilation failure and null/value/EOF execution reports. +This is path-dependent preparation, not version-tag resolution. No public indexed +selection API is claimed; its Sprint1 experiment stays in the package test target. diff --git a/Tools/TransformBindingQualification/Sources/TransformBindingConsumer/Main.swift b/Tools/TransformBindingQualification/Sources/TransformBindingConsumer/Main.swift new file mode 100644 index 0000000..f0ff9f4 --- /dev/null +++ b/Tools/TransformBindingQualification/Sources/TransformBindingConsumer/Main.swift @@ -0,0 +1,42 @@ +// SPDX-License-Identifier: Apache-2.0 +import JSON +import JSONTransform + +func guideExample() throws { + let definition = try JSONTransformProgramDefinition.prepare( + source: "$amount", variables: ["amount"]).result + let bytes = try definition.encode().result + let restored = try JSONTransformProgramDefinition.decode(bytes).result + let compiled = try restored.compile() + let amount = JSON.number(try JSONNumber(parsing: "9007199254740993")) + let output = try compiled.result.withExecution(input: .null, bindings: ["amount": amount]) { + try $0.next()?.value + } + precondition(output.result == amount) + precondition(compiled.statistics[.workUnits] > 0) +} +@main enum Consumer { + static func main() throws { + try guideExample() + let golden = #"{"definition":"swift-json-transform-source","version":1,"dialect":"swift-json-query-v1","source":".","variables":[]}"# + let definition = try JSONTransformProgramDefinition.prepare(source: ".").result + let canonical = try definition.encode().result + precondition(canonical == Array(golden.utf8)) + let names = try JSONTransformProgramDefinition.prepare(source: "$z", variables: ["z","a"]).result + precondition(names.variableNames == ["a","z"]) + do { + _ = try JSONTransformProgramDefinition.prepare(source: ".", variables: ["a","a"]) + preconditionFailure("Duplicate name accepted") + } catch let error as JSONTransformDefinitionFailure { precondition(error == .duplicateVariableName) } + var limits = JSONTransformLimits(); limits[.workUnits] = 0 + do { _ = try definition.compile(limits: limits); preconditionFailure("Compile budget reset") } + catch let error as JSONTransformFailure { precondition(error.category == .resource) } + let report = try JSONTransformProgram.compileWithReport("null,42") + let output = try report.result.withExecution(input: .null) { execution in + let first = try execution.next(), second = try execution.next(), end = try execution.next() + precondition(first?.value == .null && second?.value == 42 && end == nil) + } + precondition(report.statistics[.workUnits] > 0 && output.statistics[.workUnits] > 0) + print("TRANSFORM_BINDINGS passed witnesses=6") + } +} From dfa0f9626be6dc2aba8456ddd7d11ee6f99a45c2 Mon Sep 17 00:00:00 2001 From: Thorsten Fuchs <76904008+ThorFuchs@users.noreply.github.com> Date: Thu, 29 Oct 2026 13:09:28 +0100 Subject: [PATCH 4/5] Add bounded public Transform selection scopes and EOF reports --- AGENTS.md | 6 +- CHANGELOG.md | 8 + Documentation/Architecture/API.md | 6 + .../Release1.8/TransformBindings.md | 52 ++- .../Qualification/TestDiscovery.json | 28 +- .../Qualification/TransformSelection1.8.md | 104 +++++ README.md | 2 + Scripts/ci.py | 14 +- Scripts/qualify_transform_selection.py | 63 +++ Scripts/test_qualification_harness.py | 25 +- Sources/JSONCLIEngine/CLIBuildIdentity.swift | 2 +- .../Documentation.docc/IndexedSelection.md | 83 ++++ .../Documentation.docc/JSONTransform.md | 14 +- .../Documentation.docc/SourceDefinitions.md | 4 +- .../JSONTransformSelection.swift | 382 ++++++++++++++++++ Tests/JSONTransformTests/SelectionTests.swift | 338 ++++++++++++++++ .../Package.swift | 8 + .../TransformSelectionQualification/README.md | 10 + .../TransformSelectionConsumer/Main.swift | 103 +++++ 19 files changed, 1229 insertions(+), 23 deletions(-) create mode 100644 Documentation/Qualification/TransformSelection1.8.md create mode 100644 Scripts/qualify_transform_selection.py create mode 100644 Sources/JSONTransform/Documentation.docc/IndexedSelection.md create mode 100644 Sources/JSONTransform/JSONTransformSelection.swift create mode 100644 Tests/JSONTransformTests/SelectionTests.swift create mode 100644 Tools/TransformSelectionQualification/Package.swift create mode 100644 Tools/TransformSelectionQualification/README.md create mode 100644 Tools/TransformSelectionQualification/Sources/TransformSelectionConsumer/Main.swift diff --git a/AGENTS.md b/AGENTS.md index 444b376..b414076 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -65,7 +65,9 @@ no allocation, speed, RSS or controlled-performance pass follows from consolidat See Documentation/Qualification/Consolidation2026.md for combined gates and holds. The commissioned 1.8 Transform binding train adds a source-definition codec and -compile reports while preserving those original declarations. Indexed selection -is a test-only private-kernel ownership experiment in Sprint 1, not public API. +compile reports while preserving those original declarations. Sprint 1 +retains its historical test-only selection experiment. Sprint 2 adds public +guarded selection scopes, owner-bound positions and EOF completion reports; +preserve ordinary Transform semantics and require one cumulative invocation ledger. See Documentation/Architecture/Release1.8/TransformBindings.md. No Content code, lock, message, merge or publication is authorized by implementing this sprint. diff --git a/CHANGELOG.md b/CHANGELOG.md index 630926e..988c5b1 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,14 @@ ## Unreleased — 1.8.0-dev +- Add exact versioned Transform source definitions, a bounded native codec and + successful compilation reports through the existing compiler. +- Add finite indexed selection scopes over explicit populations, owner-bound + positions and private-constructor EOF/accounting witnesses. Admission, projection, + VM evaluation, provider/output handling and completion share one ledger. + Preserve order/repeats, first failures and terminal lease accounting; ordinary + Transform/CLI semantics and all thirteen products remain unchanged. + - Extend public acceptance across schema validation, store/transform reads and checked Patch rebuilds, with exact numeric/scalar identity and lifecycle cases. - Add a frozen capacity/owned-counter matrix and compiled cross-product recipes; diff --git a/Documentation/Architecture/API.md b/Documentation/Architecture/API.md index 55b2bcd..885c757 100644 --- a/Documentation/Architecture/API.md +++ b/Documentation/Architecture/API.md @@ -7,6 +7,12 @@ No product or dependency is added. JSONInspection now includes lookup, position value batches and finite complete BFS follow. See [the exact behavior/ownership boundary](StructuralIntelligence/BoundedWorkflows1.8.md) and [qualification](../Qualification/Release1.8.md). +The optional JSONTransform candidate also adds immutable versioned source definitions, +a bounded native codec and successful compilation reports. Finite public selection +scopes produce owner-bound positions and private-constructor EOF completion reports +through the same compiler/VM. See the [binding contract](Release1.8/TransformBindings.md) +and [selection qualification](../Qualification/TransformSelection1.8.md). No released +1.7 signature or ordinary execution/CLI semantics change. Navigation public signatures stay unchanged; its new selection capability is package-only and uses the original admitted scope. Earlier inventories below remain historical; compiler graphs must establish the final additive surface. diff --git a/Documentation/Architecture/Release1.8/TransformBindings.md b/Documentation/Architecture/Release1.8/TransformBindings.md index 6b01420..d93a155 100644 --- a/Documentation/Architecture/Release1.8/TransformBindings.md +++ b/Documentation/Architecture/Release1.8/TransformBindings.md @@ -1,4 +1,4 @@ -# Transform bindings — commissioned 1.8 candidate, Sprint 1 +# Transform bindings — commissioned 1.8 candidate The maintainer commissioned the source-definition codec, indexed selection and compile reporting on 29 October 2026. Base is b8ddeb26042005a06360f50fcf20b1f840e66d1c. @@ -6,6 +6,54 @@ This additive train preserves the existing 42 authored Transform declarations, ordinary execution/CLI semantics, thirteen products and empty root dependencies. Implementation and a successful spike are not release or downstream admission. + +## Sprint 2 current selection contract + +The maintainer commissioned the public selection increment on 29 October 2026. +Its exact base is Sprint 1 commit 70df9ba8d47322abf0d4aec1439256d7a5e97e94. +Historical Sprint 1 scope/evidence below remain source-bound. This increment +ships the selection facade in JSONTransform without a new product/dependency or +changes to the private compiler/VM, ordinary execution or CLI adapters. + +Selection convention V1 is explicit: input [0,...,N-1], reserved declared `$values` +containing the admitted population, and exactly the other declared bindings from +the caller. Overriding values or missing/foreign/invalid binding names fails. +The finite population default is 65,536; zero is valid. Native and Transform +limits remain additional ceilings, including live collection slots. Projection +slots/text/metadata and numeric construction are admitted before owned growth. +Population admission occurs once, including unused tails, on the same context as +projection, VM work, native index validation, output leases and completion. + +Optional input/inputs callbacks use existing query semantics and explicit caller +code. They cannot expand the declared population. Nil is provider EOF, null is a +value. Provider admission/calls triggered by evaluation consume the same finite +input/work/retention policy; arbitrary callback instructions and prior caller +allocations remain caller work. A separate health cell supplies prepaid entry +snapshots and prevents context borrowing during reentry. Provider/reentry/cancel +failures latch before accepting another value or another callback. + +Public positions retain one shared population owner and a per-output slot lease. +The population owner includes unforgeable invocation identity. Completion exists +only after explicit evaluator EOF and successful cleanup; its identity/report +lease does not retain the population. Completion.ordinal(of:) rejects foreign +positions through constant metadata inspection, without reopening a completed +operation. Raw ordinals/caller mappings are not stable IDs or read grants. + +Boundary invalidity bypasses query catches. Uncaught language, resource, provider, +cancellation and lifecycle failures latch. Caught errors, abandoned prefixes and +late failures cannot certify. A first operational failure wins over a later caller +error; otherwise caller errors retain identity. Escaped scopes/closures cannot +resume after close. report() drains deferred refunds after success/error/cancel; +work never refunds. Completion.statistics is a close-time snapshot; current +terminal accounting is observed through the guarded scope. Scope/position/ +completion handles are deliberately non-Sendable; plans remain checked Sendable. + +Completion, report, scope and position constructors are inaccessible to public +callers. The generic JSONOperationReport initializer cannot construct a selection +witness. No provenance propagation, second interpreter, VM serialization, online +resolution, Content code/lock or unbounded streaming/sort promise is introduced. +See the [public guide](../../../Sources/JSONTransform/Documentation.docc/IndexedSelection.md) for usage. + ## Definition V1 JSONTransformProgramDefinition stores exact source and unique ASCII variable names @@ -44,7 +92,7 @@ the existing compiler/context and returns actual native work. The additive compileWithReport path preserves old compile behavior; the definition adapter also charges its owned variable Set construction. No public meter is exposed. -## Indexed selection contract for Sprint 2 +## Historical Sprint 1 selection plan The proposed convention is an invocation-owned ordered array [0,...,N-1] as input and a reserved explicit values binding containing the complete population. The diff --git a/Documentation/Qualification/TestDiscovery.json b/Documentation/Qualification/TestDiscovery.json index 60b4b1e..fddc9ab 100644 --- a/Documentation/Qualification/TestDiscovery.json +++ b/Documentation/Qualification/TestDiscovery.json @@ -13,7 +13,7 @@ "JSONNavigationTests": 57, "JSONSchemaExamplesTests": 98, "JSONInspectionTests": 214, - "JSONTransformTests": 41, + "JSONTransformTests": 57, "JSONDSLTests": 47 }, "parameterized_invocations": 390, @@ -38,7 +38,7 @@ "JSONNavigationTests": 2, "JSONSchemaExamplesTests": 4, "JSONInspectionTests": 11, - "JSONTransformTests": 3, + "JSONTransformTests": 4, "JSONDSLTests": 1 }, "coverage_unmapped": { @@ -85,7 +85,7 @@ "JSONNavigationTests": 57, "JSONSchemaExamplesTests": 98, "JSONInspectionTests": 214, - "JSONTransformTests": 41, + "JSONTransformTests": 57, "JSONDSLTests": 47 }, "wasi_root_functions": { @@ -100,7 +100,7 @@ "JSONNavigationTests": 57, "JSONSchemaExamplesTests": 98, "JSONInspectionTests": 214, - "JSONTransformTests": 41, + "JSONTransformTests": 57, "JSONDSLTests": 47 }, "wasi_parameterized_invocations": 353, @@ -116,7 +116,7 @@ "JSONNavigationTests": 2, "JSONSchemaExamplesTests": 4, "JSONInspectionTests": 11, - "JSONTransformTests": 3, + "JSONTransformTests": 4, "JSONDSLTests": 1 }, "export_functions": { @@ -272,6 +272,22 @@ "JSONTransformTests.jtxTaskCancellationIsObservedAfterProviderOwnedProgress()", "JSONTransformTests.jtxTerminalAliasesRefundBeforeThrowingTheSameFailure()", "JSONTransformTests.jtxTypedCompileLocationAndLanguageFailureStayDistinct()", - "JSONTransformTests.jtxZeroNullAndManyAreDistinctOrderedPullResults()" + "JSONTransformTests.jtxZeroNullAndManyAreDistinctOrderedPullResults()", + "JSONTransformTests.selectionS01ExplicitOrdinalsPreserveDuplicateValuesOrderRepeatsAndTies()", + "JSONTransformTests.selectionS02EmptyEOFNullAndIncompletePrefixAreDistinct()", + "JSONTransformTests.selectionS03EveryInvalidOutputIsExactAndTerminalOutsideQueryCatch()", + "JSONTransformTests.selectionS04CompletionRejectsForeignInvocationsAndPositionsRetainPopulation()", + "JSONTransformTests.selectionS05FinalAliasAndCompletionReleaseSettleToZeroWithoutDataPulls()", + "JSONTransformTests.selectionS06BindingsAreExplicitReservedAndScalarChecked()", + "JSONTransformTests.selectionS07PopulationPolicyIsInclusiveFiniteAndCannotSkipCompleteAdmission()", + "JSONTransformTests.selectionS08ProjectionEvaluationAndCompletionShareExactCumulativeWorkAndRetention()", + "JSONTransformTests.selectionS09QuotaAndCancellationRefundTerminalAliasesAndPreserveFirstCause()", + "JSONTransformTests.selectionS10ProviderEOFNullErrorAndNoFurtherPullsRemainDistinct()", + "JSONTransformTests.selectionS11ReentryAndProviderCancellationUseSafeEntrySnapshots()", + "JSONTransformTests.selectionS12CallerErrorAndEscapedClosuresReleaseProviderWithoutCompletion()", + "JSONTransformTests.selectionS13InnerProjectionTaskCancellationReleasesUnpublishedPreparation()", + "JSONTransformTests.selectionS14FailedPreparationAndCompletionAllocationsRefundExactBoundaries()", + "JSONTransformTests.selectionS15LanguageCatchesRemainOrdinaryAndUncaughtFailuresStayTerminal()", + "JSONTransformTests.selectionSharedImmutablePlanHasIndependentPopulationAndFailureHealth()" ] } diff --git a/Documentation/Qualification/TransformSelection1.8.md b/Documentation/Qualification/TransformSelection1.8.md new file mode 100644 index 0000000..0913cc8 --- /dev/null +++ b/Documentation/Qualification/TransformSelection1.8.md @@ -0,0 +1,104 @@ +# Transform indexed selection — Sprint 2 + +Status: implemented with scoped native qualification complete; full 1.8 release +qualification remains PARTIAL. This local candidate extends sealed Sprint 1 +`70df9ba8d47322abf0d4aec1439256d7a5e97e94` on +`codex/1.8-transform-selection-s2`. The previous branch/checkpoint, batches/store +PR and all source-bound receipts remain intact. No merge, tag, publication or +Content source-lock change is implied. + +The [contract](../Architecture/Release1.8/TransformBindings.md) and +[compiled public guide](../../Sources/JSONTransform/Documentation.docc/IndexedSelection.md) +define selection convention V1. The input is an owned index array and `$values` +is the completely admitted population. Bindings supply exactly the other declared +names. The finite population cap defaults to 65,536, alongside existing native +and Transform policies. One private invocation context owns population admission, +projection/index slots, evaluation, provider/output admission, exact conversion, +position leases and completion. No engine language or numeric authority changes. + +## Ownership and failure decision + +Positions have private construction, exact ordinals, invocation identity and a +joint population/output-slot lease. Completion/report construction is inaccessible +to public callers. Explicit evaluator EOF and successful cleanup are mandatory; +an abandoned prefix never certifies. A completion retains only identity/report +storage. Its constant identity check rejects foreign positions even for identical +populations. It does not reopen or retroactively fail a completed invocation. + +Scopes, positions and completions are deliberately non-Sendable. Escaped handles +are guarded after close; plans remain checked Sendable for independent concurrent +scopes. Providers are optional and explicit. Their values may influence selection +but cannot enlarge the population. Callback work and prior caller allocations +remain caller work. During callbacks, observation uses an admitted entry snapshot +and reentry/cancellation touches health before the loaned context. + +Boundary invalidity, resources, uncaught language/provider failures, cancellation +and reentry latch. Query catches retain their ordinary semantics but cannot catch +boundary/operational failures. A first operational cause wins over a later caller +error; otherwise caller errors retain identity. report() settles actual deferred +refunds after failure/close without resetting work. Completion statistics are a +close-time snapshot, not a later ARC settlement. Logical weights are not RSS, +allocation-event measurements or arbitrary caller instruction quotas. + +## Actual native qualification — 29 October 2026 + +Host: macOS 27.0 arm64, Apple Swift 6.4.2 (`swiftlang-6.4.2.1.9`, driver 1.168.6), +Xcode 27.2/27B48, SDK 27.2, language mode 6. Private receipts/logs/events/source +manifests are retained under `Artifacts/TransformBindingsS2`. + +| Check | Observed outcome | +| --- | --- | +| `Scripts/ci.py --lane transform-bindings` | Passed: 57 functions in Debug and Release, original 41 preserved, 32 Python harness cases, privacy/source identity and both public consumers | +| `Scripts/ci.py --lane transform-bindings-safety` | Passed: 57 Release ASan functions, four shared-plan TSan functions, strict-memory target build with warnings as errors | +| New outside-Git JSON/JSONTransform consumer | Passed: ten witnesses/configuration, exact compiled DocC guide, fourteen ordinary constructor/non-Sendable compiler refusals and two guarded-alias positives across Debug/Release | +| Original source-definition public consumer | Passed: six witnesses/configuration and its four ordinary compiler refusals | +| Original public Transform consumer and ownership compiler cases | Passed: twelve witnesses/configuration and all seven compiler cases | +| `Scripts/ci.py --lane cli` | Passed: thirteen commands, 88 functions/37 parameter invocations and all existing command/vector/provider outcomes | +| Existing manifest/dependency/cap commands | Passed through unchanged CI comparators: thirteen products, empty root external graph and all 49 exact cap outcomes | +| Native Transform graph/DocC | Passed: 93 authored declarations/parameters documented, all 67 Sprint 1 identities/declaration fragments unchanged, 26 additions, warnings as errors | + +Sixteen new Swift Testing functions cover explicit duplicates/order/repeats/ties, +exact scalar/number behavior, empty/EOF/null/incomplete distinctions, malformed/ +negative/fractional/overflow/native-width/out-of-population outputs, late invalidity +outside query catch, foreign owners, final aliases and report-only settlement, +reserved/invalid/missing bindings, complete unused-tail admission and zero/exact/ +one-over/overflow policies. Tests also pin cumulative work/retention/collection +slots, completion-allocation failure, terminal quota/cancel refunds, provider error/ +EOF/null/reentry snapshots, caller/provider release, deterministic inner-projection +task cancellation, uncaught language failure and independent concurrent reuse. + +The first test compilation retained ordinary errors in throwing assertion inference +and one incorrect native metric name. Its corrected 13-case run passed before the +final 16-case increment. No production semantic failure or compiler crash occurred. +A supplemental private runner initially lacked its output directory and executed +no command; the raw setup failure remains, and the diagnosed successor runs the +same three commands. No failed attempt is relabeled as a pass or hidden. + +## Final binding and remaining gates + +The checkpoint binding separately maps this qualification closeout and the API +index paragraph after passing receipts. Product, test, fixture, manifest, DocC, +generator, identity and executable harness bytes match those checked. Native +receipts are scoped to this actual source. No independent human review or broad +platform acceptance is inferred. The scripted graph comparison is exact for all +67 predecessor declaration fragments; full thirteen-catalog graphs remain separate. + +The private return retains deterministic source/checkpoint/diff and reproduction +results. Full final-source Native/Linux ARM64+x86_64/safety/docs/WASI/relevant Apple +SDK/runtime/API/archive gates and branch reconciliation remain release work. +No new hosted CI, Linux, WASI or device pass follows from this sprint. Preserve +main's independent documentation automation when deliberately reconciling later. +No push/merge/tag/publication is performed by local qualification. + +The current expected discovery is 57 Transform roots/four sharing functions; all +other populations and 390 native/portable parameter invocations (353 WASI) remain. +Existing 49 policy probes preserve their exact distinctions, including zero-scratch +success and invalid cancellation configuration. The population cap is additive +policy, not a replacement metric or a calibrated RSS bound. + +No benchmark/allocation campaign ran. Earlier 1.8 loaded-host observations and +unresolved regressions remain bound to 5e27899. Allocations remain unmeasured/ +unwaived for 1.8; the 1.7 waiver and historical holds are not promoted. Content's +S06/S13 profile mapping and deliberate public source re-resolution remain separate +acceptance work. This implementation supplies no Content IDs, locator/read grants, +persistence, provenance propagation, online resolution or unbounded streaming. diff --git a/README.md b/README.md index 3b204ee..35c9ae3 100644 --- a/README.md +++ b/README.md @@ -4,6 +4,8 @@ This is **Swift-JSON 1.8.0-dev**, based on the published [1.7.0 release](https://github.com/ThorFuchs/Swift-JSON/releases/tag/1.7.0) (`8557badef562cf2f059e964c113d4a7f60ecbe6c`). The next additive wave brings ordered batch reads and an immutable configured read-only store to JSONInspection. +The optional JSONTransform candidate also adds exact source-definition codecs, +compile reports and [finite indexed selection](Sources/JSONTransform/Documentation.docc/IndexedSelection.md). It is under qualification; it is not an admitted 1.8 release. Twelve libraries plus CLI, thirteen DocC catalogs and empty root dependencies remain. See the [1.8 implementation contract](Documentation/Architecture/StructuralIntelligence/BoundedWorkflows1.8.md), diff --git a/Scripts/ci.py b/Scripts/ci.py index 09aa283..bea99b3 100644 --- a/Scripts/ci.py +++ b/Scripts/ci.py @@ -28,6 +28,7 @@ SHARED_TESTS += '|graphG30ConcurrentViewsHaveIndependentHealth|graphGP14ConcurrentQueries' SHARED_TESTS += '|dslSharedFinalizationAndPlansAreInvocationOwned' SHARED_TESTS += '|storeR06TerminalQuotasNeverPublishAndImmutableConcurrentHealth' +SHARED_TESTS += '|selectionSharedImmutablePlanHasIndependentPopulationAndFailureHealth' SHARED_TESTS += '|bindingSharedDefinitionAndCompilationHaveIndependentHealth|bindingSharedSelectionSpikeReusesOnePlanIndependently' SHARED_TESTS += '|workflowSharedImmutablePlansHaveIndependentBatchAndStoreHealth' SHARED_TESTS += '|acceptance18SharedPlansKeepIndependentCrossProductHealth' @@ -45,7 +46,7 @@ def portable(values): if lane.endswith('safety'): return portable([ ['swift', 'test', '--test-product', 'JSONTransformTests', '-c', 'release', '--jobs', '2', '--sanitize', 'address', '--scratch-path', '.build/bindings-address'], - ['swift', 'test', '--test-product', 'JSONTransformTests', '-c', 'release', '--jobs', '2', '--sanitize', 'thread', '--scratch-path', '.build/bindings-thread', '--filter', 'jtxPlanIsSendableAndExecutionsHaveIndependentHealth|bindingShared'], + ['swift', 'test', '--test-product', 'JSONTransformTests', '-c', 'release', '--jobs', '2', '--sanitize', 'thread', '--scratch-path', '.build/bindings-thread', '--filter', 'jtxPlanIsSendableAndExecutionsHaveIndependentHealth|bindingShared|selectionShared'], ['swift', 'build', '--target', 'JSONTransform', '--jobs', '2', '--scratch-path', '.build/bindings-strict', '-Xswiftc', '-strict-memory-safety', '-Xswiftc', '-warnings-as-errors'], ]) return portable([ @@ -55,6 +56,7 @@ def portable(values): ['swift', 'test', '--test-product', 'JSONTransformTests', '--jobs', '2'], ['swift', 'test', '--test-product', 'JSONTransformTests', '-c', 'release', '--jobs', '2'], ['python3', '-B', 'Scripts/qualify_transform_bindings.py', '--output', str(output / 'public-consumer')], + ['python3', '-B', 'Scripts/qualify_transform_selection.py', '--output', str(output / 'selection-consumer')], ]) if lane in ('structural-acceptance', 'structural-acceptance-safety'): if lane.endswith('safety'): @@ -264,6 +266,7 @@ def portable(values): ['python3', '-B', 'Scripts/qualify_transform_ownership.py', '--output', str(output / 'transform-ownership')], ['python3', '-B', 'Scripts/qualify_transform.py', '--output', str(output / 'transform-consumer')], ['python3', '-B', 'Scripts/qualify_transform_bindings.py', '--output', str(output / 'transform-binding-consumer')], + ['python3', '-B', 'Scripts/qualify_transform_selection.py', '--output', str(output / 'transform-selection-consumer')], ['python3', '-B', 'Scripts/qualify_release_1_1.py', '--mode', 'ownership', '--output', str(output / 'ownership.json')], ['python3', '-B', 'Scripts/qualify_release_1_1.py', '--mode', 'reader', '--output', str(output / 'reader.json')], ['swift', 'run', '-c', 'release', '--package-path', 'Examples/Consumer', 'Consumer'], @@ -357,7 +360,9 @@ def check_result(command, code, log, events=None): if 'JSONTransformTests' in command: ids = set(DISCOVERY['transform_function_ids']) if 'thread' in command: - ids = {n for n in ids if 'bindingShared' in n or 'jtxPlanIsSendableAndExecutionsHaveIndependentHealth' in n} + ids = {n for n in ids if 'bindingShared' in n or 'selectionShared' in n or 'jtxPlanIsSendableAndExecutionsHaveIndependentHealth' in n} + elif selected_filter == 'selection': + ids = {n for n in ids if '.selection' in n} elif selected_filter == 'binding': ids = {n for n in ids if '.binding' in n} expected = [len(ids)] @@ -528,7 +533,10 @@ def check_result(command, code, log, events=None): if command[:3] == ['python3', '-B', 'Scripts/qualify_html_browser.py']: passed = passed and 'JSONHTML_BROWSER passed engines=3 cases=76 skips=0' in log.splitlines() if 'Scripts/test_qualification_harness.py' in command: - passed = passed and 'HARNESS_TESTS passed cases=31' in log.splitlines() + passed = passed and 'HARNESS_TESTS passed cases=32' in log.splitlines() + if command[:3] == ['python3', '-B', 'Scripts/qualify_transform_selection.py']: + data = json.loads(log.strip().splitlines()[-1]) + passed = passed and data['status'] == 'passed' and len(data['commands']) == 20 and all(x['passed'] for x in data['commands']) and data['witnesses_per_configuration'] == 10 and data['negative_fixture_invocations'] == 14 and data['compiled_guide_fences'] == 1 if command[:3] == ['python3', '-B', 'Scripts/qualify_transform_bindings.py']: data = json.loads(log.strip().splitlines()[-1]) passed = passed and data['status'] == 'passed' and len(data['commands']) == 8 and all(x['passed'] for x in data['commands']) and data['witnesses_per_configuration'] == 6 and data['negative_fixture_invocations'] == 4 and data['compiled_guide_fences'] == 1 diff --git a/Scripts/qualify_transform_selection.py b/Scripts/qualify_transform_selection.py new file mode 100644 index 0000000..175154d --- /dev/null +++ b/Scripts/qualify_transform_selection.py @@ -0,0 +1,63 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: Apache-2.0 +"""Public-only selection consumer and ownership/completion compiler boundaries.""" +import argparse, hashlib, json, pathlib, platform, re, shutil, subprocess, sys, tempfile, textwrap +from qualify_transform_ownership import ordinary_rejection +ROOT = pathlib.Path(__file__).resolve().parents[1] + +def main(): + parser = argparse.ArgumentParser(description=__doc__); parser.add_argument('--output', required=True) + args = parser.parse_args(); output = (ROOT/args.output).resolve(); output.mkdir(parents=True, exist_ok=True) + guide=(ROOT/'Sources/JSONTransform/Documentation.docc/IndexedSelection.md').read_text() + fences=re.findall(r'```swift\n(.*?)\n```',guide,re.S) + consumer_text=(ROOT/'Tools/TransformSelectionQualification/Sources/TransformSelectionConsumer/Main.swift').read_text() + body=re.search(r'func selectionGuideExample\(\) throws \{\n(.*?)\n\}\nprivate',consumer_text,re.S).group(1) + assert len(fences)==1 and textwrap.dedent(body).strip()=='\n'.join(line for line in fences[0].splitlines() if not line.startswith('import ')).strip() + commands = [] + def run(command, name): + result = subprocess.run(command,cwd=ROOT,capture_output=True,text=True) + data=result.stdout+result.stderr; path=output/(name+'.log');path.write_text(data) + row=dict(command=command,exit_code=result.returncode,log=path.name,sha256=hashlib.sha256(path.read_bytes()).hexdigest()) + commands.append(row); return result,row + with tempfile.TemporaryDirectory(prefix='swift-json-selection-consumer-') as temporary: + consumer=pathlib.Path(temporary) + for p in (ROOT/'Tools/TransformSelectionQualification').rglob('*'): + if p.is_file() and not any(x in p.parts for x in ('.build','.swiftpm')): + target=consumer/p.relative_to(ROOT/'Tools/TransformSelectionQualification');target.parent.mkdir(parents=True,exist_ok=True);target.write_bytes(p.read_bytes()) + manifest=consumer/'Package.swift';manifest.write_text(manifest.read_text().replace('path: "../.."','path: '+json.dumps(str(ROOT)))) + inputs=[dict(path=p.relative_to(consumer).as_posix(),sha256=hashlib.sha256(p.read_bytes()).hexdigest()) for p in sorted(consumer.rglob('*')) if p.is_file()] + for config in ['debug','release']: + result,row=run(['swift','run','--package-path',str(consumer),'-c',config,'--jobs','2','TransformSelectionConsumer'],config+'-consumer') + row['passed']=result.returncode==0 and result.stdout.splitlines()==['TRANSFORM_SELECTION passed witnesses=10'] + if not row['passed']:break + result,row=run(['swift','build','--package-path',str(consumer),'-c',config,'--show-bin-path'],config+'-binpath') + row['passed']=result.returncode==0 + if not row['passed']:break + modules=pathlib.Path(result.stdout.strip()); binary=modules/'TransformSelectionConsumer' + target=output/(config+'-TransformSelectionConsumer');shutil.copyfile(binary,target) + row['binary_sha256']=hashlib.sha256(target.read_bytes()).hexdigest() + base=['swiftc','-swift-version','6','-I',str(modules),'-module-cache-path',str(output/'ModuleCache'),'-c'] + if sys.platform=='darwin': + sdk=subprocess.check_output(['xcrun','--sdk','macosx','--show-sdk-path'],text=True).strip() + base+=['-sdk',sdk,'-target',platform.machine()+'-apple-macos27.0','-F',str(modules/'PackageFrameworks')] + fixtures = [ + ('position','import JSONTransform\nlet x = JSONTransformSelectionPosition()\n',False,'inaccessible'), + ('completion','import JSONTransform\nlet x = JSONTransformSelectionCompletion()\n',False,'inaccessible'), + ('report','import JSONTransform\nfunc forge(_ c: JSONTransformSelectionCompletion) { _ = JSONTransformSelectionReport(result: (), completion: c) }\n',False,'inaccessible'), + ('scope','import JSONTransform\nlet x = JSONTransformSelectionScope()\n',False,'inaccessible'), + ('send-scope','import JSONTransform\nfunc send(_: T) {}\nfunc check(_ s: JSONTransformSelectionScope) { send(s) }\n',False,'does not conform to'), + ('send-position','import JSONTransform\nfunc send(_: T) {}\nfunc check(_ p: JSONTransformSelectionPosition) { send(p) }\n',False,'does not conform to'), + ('send-completion','import JSONTransform\nfunc send(_: T) {}\nfunc check(_ c: JSONTransformSelectionCompletion) { send(c) }\n',False,'does not conform to'), + ('guarded-alias','import JSONTransform\nfunc check(_ p: JSONTransformProgram) throws -> (() throws -> JSONTransformSelectionPosition?)? { var escaped: (() throws -> JSONTransformSelectionPosition?)?; _ = try p.withSelection(population: []) { s in escaped = { try s.next() }; let eof = try s.next(); precondition(eof == nil) }; return escaped }\n',True,None), + ] + for name,source,positive,diagnostic in fixtures: + path=output/(config+'-'+name+'.swift');path.write_text(source) + result,row=run(base+[str(path),'-o',str(output/(config+'-'+name+'.o'))],config+'-'+name) + row['positive']=positive;row['expected_diagnostic']=diagnostic;row['passed']=result.returncode==0 if positive else ordinary_rejection(result.returncode,result.stderr,diagnostic) + passed=len(commands)==20 and all(x['passed'] for x in commands) + receipt=dict(status='passed' if passed else 'failed',commands=commands,source_inputs=inputs,compiled_guide_fences=1, + configurations=2,witnesses_per_configuration=10,negative_fixture_invocations=14, + scope='Outside-Git public-only path consumer; no private imports or released-version resolution.') + (output/'receipt.json').write_text(json.dumps(receipt,indent=2)+'\n') + print(json.dumps(receipt));return 0 if passed else 1 +if __name__=='__main__':raise SystemExit(main()) diff --git a/Scripts/test_qualification_harness.py b/Scripts/test_qualification_harness.py index ed4e9f0..feaa36b 100644 --- a/Scripts/test_qualification_harness.py +++ b/Scripts/test_qualification_harness.py @@ -28,18 +28,31 @@ def store_markers(): 'STORE_RUNTIME R%02d passed' % n for n in range(1,11)]) class HarnessTests(unittest.TestCase): + def test_selection_filter_preserves_exact_identity_and_completion_evidence(self): + ids = [n for n in DISCOVERY['transform_function_ids'] if '.selection' in n] + self.assertEqual(len(ids), 16) + command = ['swift','test','--test-product','JSONTransformTests','--filter','selection'] + events = dict(bundle_counts=[16],module_function_counts={'JSONTransformTests':16},function_names=ids,parameter_case_invocations=0) + self.assertTrue(check_result(command,0,'',events)[0]) + self.assertFalse(check_result(command,0,'',dict(events,function_names=ids[:-1]))[0]) + consumer = ['python3','-B','Scripts/qualify_transform_selection.py'] + receipt = dict(status='passed',commands=[dict(passed=True)]*20,witnesses_per_configuration=10,negative_fixture_invocations=14,compiled_guide_fences=1) + self.assertTrue(check_result(consumer,0,json.dumps(receipt))[0]) + for broken in [dict(receipt,commands=receipt['commands'][:-1]),dict(receipt,negative_fixture_invocations=0),dict(receipt,compiled_guide_fences=0)]: + self.assertFalse(check_result(consumer,0,json.dumps(broken))[0]) + def test_transform_binding_discovery_rejects_incomplete_duplicate_and_foreign_events(self): ids = DISCOVERY['transform_function_ids'] command = ['swift', 'test', '--test-product', 'JSONTransformTests'] - events = dict(bundle_counts=[41], module_function_counts={'JSONTransformTests':41}, + events = dict(bundle_counts=[57], module_function_counts={'JSONTransformTests':57}, function_names=ids, parameter_case_invocations=0) self.assertTrue(check_result(command, 0, '', events)[0]) for broken in [dict(events,function_names=ids[:-1]),dict(events,function_names=ids+ids[:1]), - dict(events,module_function_counts={'JSONTests':41}),dict(events,parameter_case_invocations=1)]: + dict(events,module_function_counts={'JSONTests':57}),dict(events,parameter_case_invocations=1)]: self.assertFalse(check_result(command, 0, '', broken)[0]) - shared = [n for n in ids if 'bindingShared' in n or 'jtxPlanIsSendableAndExecutionsHaveIndependentHealth' in n] + shared = [n for n in ids if 'bindingShared' in n or 'selectionShared' in n or 'jtxPlanIsSendableAndExecutionsHaveIndependentHealth' in n] thread = command + ['--sanitize', 'thread'] - self.assertTrue(check_result(thread, 0, '', dict(events,bundle_counts=[3],module_function_counts={'JSONTransformTests':3},function_names=shared))[0]) + self.assertTrue(check_result(thread, 0, '', dict(events,bundle_counts=[4],module_function_counts={'JSONTransformTests':4},function_names=shared))[0]) def test_acceptance18_discovery_preserves_exact_public_pipeline_identities(self): names = DISCOVERY['structural_acceptance_function_ids'] @@ -452,8 +465,8 @@ def test_exact_discovery_rejects_missing_duplicate_and_changed_parameters(self): identity['function_names'][0] = 'JSONTests.unrelatedReplacement()' self.assertFalse(check_restored(identity)) harness = ['python3', '-B', 'Scripts/test_qualification_harness.py'] - self.assertTrue(check_result(harness, 0, 'HARNESS_TESTS passed cases=31')[0]) - for stale in ['HARNESS_TESTS passed cases=30', 'HARNESS_TESTS passed cases=10', 'HARNESS_TESTS passed cases=11', 'HARNESS_TESTS passed cases=12', 'HARNESS_TESTS passed cases=13', 'HARNESS_TESTS passed cases=14', 'HARNESS_TESTS passed cases=15', 'HARNESS_TESTS passed cases=18', 'HARNESS_TESTS passed cases=19', 'HARNESS_TESTS passed cases=20', 'HARNESS_TESTS passed cases=21', 'HARNESS_TESTS passed cases=25', 'HARNESS_TESTS passed cases=26', 'HARNESS_TESTS passed cases=28', '']: + self.assertTrue(check_result(harness, 0, 'HARNESS_TESTS passed cases=32')[0]) + for stale in ['HARNESS_TESTS passed cases=31', 'HARNESS_TESTS passed cases=30', 'HARNESS_TESTS passed cases=10', 'HARNESS_TESTS passed cases=11', 'HARNESS_TESTS passed cases=12', 'HARNESS_TESTS passed cases=13', 'HARNESS_TESTS passed cases=14', 'HARNESS_TESTS passed cases=15', 'HARNESS_TESTS passed cases=18', 'HARNESS_TESTS passed cases=19', 'HARNESS_TESTS passed cases=20', 'HARNESS_TESTS passed cases=21', 'HARNESS_TESTS passed cases=25', 'HARNESS_TESTS passed cases=26', 'HARNESS_TESTS passed cases=28', '']: self.assertFalse(check_result(harness, 0, stale)[0]) lifetime = ['python3', '-B', 'Scripts/qualify_schema_example_lifetime.py'] marker = 'SCHEMA_EXAMPLES_CORPUS passed mode=owned scopes=128 values=6 exact=1' diff --git a/Sources/JSONCLIEngine/CLIBuildIdentity.swift b/Sources/JSONCLIEngine/CLIBuildIdentity.swift index a2f826c..ab25073 100644 --- a/Sources/JSONCLIEngine/CLIBuildIdentity.swift +++ b/Sources/JSONCLIEngine/CLIBuildIdentity.swift @@ -2,6 +2,6 @@ // Generated by Scripts/generate_cli_identity.py; excludes this file. // SHA-256 of the ordered Package.swift + Sources content manifest, not a Git commit. package enum CLIBuildIdentity { - package static let revision = "sha256:2e6e326d160776afbff186a2c68d01968dc866159ede10038593be415098d91d" + package static let revision = "sha256:0a49346d574588626ee993fab93ac2fb38b4a392ea7bacb750e04e9e89855f61" package static let kind = "source-sha256" } diff --git a/Sources/JSONTransform/Documentation.docc/IndexedSelection.md b/Sources/JSONTransform/Documentation.docc/IndexedSelection.md new file mode 100644 index 0000000..c2f9000 --- /dev/null +++ b/Sources/JSONTransform/Documentation.docc/IndexedSelection.md @@ -0,0 +1,83 @@ +# Select explicit population positions + +Evaluate the existing query language over finite indices, preserving original +positions even when values are identical. Selection convention V1 supplies +`[0,...,N-1]` as input and the complete population in the reserved `$values` +binding. Declare `values` at compilation; supply exactly the remaining names +through explicit bindings. Stored values acquire no engine metadata. + +```swift +import JSON +import JSONTransform + +let definition = try JSONTransformProgramDefinition.prepare( + source: ".[] | select($values[.].enabled)", variables: ["values"]).result +let plan = try definition.compile().result +let row = try JSON.parse(#"{"enabled":true,"amount":9007199254740993}"#) +let report = try plan.withSelection(population: [row, row]) { scope in + var positions: [JSONTransformSelectionPosition] = [] + while let position = try scope.next() { positions.append(position) } + return positions +} +let ordinals = try report.result.map { try report.completion.ordinal(of: $0) } +precondition(ordinals == [0, 1]) +precondition(report.completion.populationCount == 2) +precondition(report.completion.outputCount == 2) +``` + +The evaluator is the ordinary Transform VM. Queries must emit exact nonnegative +UInt64 ordinals within the population and the native Int range. Mathematical +integer spellings such as 42.0 and negative zero convert exactly. Fractions, +negative values, overflow, other JSON kinds and out-of-range outputs fail without +Double conversion. Order and repeats are preserved. Sort tie rules belong in +the query. The engine supplies no arbitrary transformed-value provenance. + +## Completion and ownership + +Pull nil explicitly. Returning after a prefix, even an empty prefix, throws +``JSONTransformSelectionFailure/incompleteSelection``. A late invalid output, +resource failure, cancellation, provider failure or reentry latches outside query +catch handling; a caller catch cannot create a completion witness. Ordinary query +language catches continue to work. A first operational failure takes precedence +over a later caller-body error. An otherwise uncaught body error keeps its identity. + +Only ``JSONTransformProgram/withSelection(population:bindings:limits:source:_:)`` +creates a report and completion after EOF. Their initializers and position/scope +construction are inaccessible to public callers. Completion verifies a position's +invocation identity through ``JSONTransformSelectionCompletion/ordinal(of:)``; +identical population values from another invocation still throw foreignPosition. +That constant metadata inspection is caller work and does not reopen the VM or +retroactively change a completed invocation. + +A position retains the admitted population and its output-slot lease after scope +exit. A completion retains identity/report storage, not the population. Copies +share reservations. Escaped scopes and closures cannot resume pulls after close. +Use ``JSONTransformSelectionScope/report()`` to settle deferred ARC refunds and +observe actual native counts after success, failure or cancellation. Work never +refunds. Completion statistics are the successful close-time snapshot, not an +automatically updating post-ARC view. Keep the appropriate handles alive while +using accounted aliases. Scope, positions and completion are non-Sendable; +immutable compiled plans are Sendable and support independent concurrent scopes. + +## Policy and providers + +``JSONTransformSelectionLimits`` defaults to 65,536 population values and balanced +Transform/native limits. Population admission, index slots, all VM pulls, native +numeric validation, output slots, provider admission and completion storage share +one invocation ledger. A query producing no results still admits the whole +population. The original 49 core caps remain additional finite policies. +Logical weights are not heap allocation sizes or process RSS. + +An optional explicit source callback serves the existing input/inputs operators. +Nil provider output means EOF; JSON null is a value and fails if emitted as an +ordinal. Providers may influence selection but cannot enlarge the declared +population. Their own instructions and already allocated values are caller work. +Provider values and calls triggered by evaluation remain subject to cumulative +native/Transform admission and input/work limits. Provider failure/cancellation +stops further callbacks; a caught reentrant pull also fails the invocation. +During the callback, report/used/remainingWorkUnits expose a prepaid entry snapshot. + +This is a finite selection channel, not a stable Content ID, locator, read grant, +persistent execution, unbounded sort/stream guarantee or downstream admission. +The selection convention revision is separate from the definition/dialect revision +and the complete effective compilation/execution policies. diff --git a/Sources/JSONTransform/Documentation.docc/JSONTransform.md b/Sources/JSONTransform/Documentation.docc/JSONTransform.md index 0dfbfc2..c17abaf 100644 --- a/Sources/JSONTransform/Documentation.docc/JSONTransform.md +++ b/Sources/JSONTransform/Documentation.docc/JSONTransform.md @@ -3,7 +3,8 @@ Compile the existing exact query dialect and pull bounded native JSON results. The library was introduced in 1.7.0. This 1.8 candidate adds exact source definitions -and successful compile reports; indexed selection remains a test-only experiment. +successful compile reports and guarded finite indexed selection with owner-bound +positions and explicit completion witnesses. Source identities and qualification stay separate. Publication requires a separate maintainer decision. It shares one private compiler and VM with `swift-json query`; it does not introduce a new interpreter or numeric authority. Plans are immutable and Sendable. Each synchronous invocation has one @@ -29,3 +30,14 @@ finite cumulative meter and a guarded, non-Sendable execution handle. - ``JSONTransformLimits`` - ``JSONTransformFailure`` + +### Indexed selection + +- + +- ``JSONTransformSelectionLimits`` +- ``JSONTransformSelectionScope`` +- ``JSONTransformSelectionPosition`` +- ``JSONTransformSelectionCompletion`` +- ``JSONTransformSelectionReport`` +- ``JSONTransformSelectionFailure`` diff --git a/Sources/JSONTransform/Documentation.docc/SourceDefinitions.md b/Sources/JSONTransform/Documentation.docc/SourceDefinitions.md index fda7777..af53fea 100644 --- a/Sources/JSONTransform/Documentation.docc/SourceDefinitions.md +++ b/Sources/JSONTransform/Documentation.docc/SourceDefinitions.md @@ -37,5 +37,5 @@ new numeric or query engine is introduced. Use ``JSONTransformProgram/compileWithReport(_:variables:limits:)`` when compiling ordinary source with successful native work statistics. These reports are data; -they are not unforgeable selection-completion witnesses. The indexed selection -experiment remains test-only in this sprint, with no public selection API. +they are not unforgeable selection-completion witnesses. The separate guide defines the finite public selection +convention, owner-bound positions and EOF witnesses. diff --git a/Sources/JSONTransform/JSONTransformSelection.swift b/Sources/JSONTransform/JSONTransformSelection.swift new file mode 100644 index 0000000..21a4958 --- /dev/null +++ b/Sources/JSONTransform/JSONTransformSelection.swift @@ -0,0 +1,382 @@ +// SPDX-License-Identifier: Apache-2.0 +import JSON +import JSONFilterRuntime + +/// Finite policy for indexed-selection convention V1, additional to native limits. +public struct JSONTransformSelectionLimits: Sendable { + /// The shared admission, projection, VM, provider and output policy. + public var transform: JSONTransformLimits = .init() + /// Inclusive population count; default 65,536. Zero admits an empty population. + public var maximumPopulationValues: UInt64 = 65_536 + var checkpointObserver: (@Sendable (UInt64) -> Void)? + var projectionObserver: (@Sendable (UInt64) -> Void)? + /// Creates the finite balanced selection policy. + public init() {} +} + +/// A selection-boundary refusal, distinct from language and operational engine errors. +public enum JSONTransformSelectionFailure: Error, Sendable, Equatable { + /// An output was not an exact nonnegative UInt64 index in the population. + case invalidPosition + /// A position belongs to a different invocation, even with identical values. + case foreignPosition + /// The caller returned before pulling explicit evaluator EOF. + case incompleteSelection +} + +private final class SelectionIdentity { + let lease: TransformValueLease + init(_ lease: TransformValueLease) { self.lease = lease } +} +private final class SelectionPopulation { + let value: FilterYield + let identity: SelectionIdentity + let metadata: TransformValueLease + init(_ value: FilterYield, identity: SelectionIdentity, metadata: TransformValueLease) { + self.value = value; self.identity = identity; self.metadata = metadata + } +} + +/// An unforgeable invocation-local ordinal, retaining the population and its accounting. +/// Copies share leases. The handle is deliberately non-Sendable; extracted ordinals +/// and caller mappings are not engine provenance, stable IDs or read grants. +public struct JSONTransformSelectionPosition { + /// Exact zero-based original population index; order and repeats are preserved. + public let ordinal: UInt64 + private let population: SelectionPopulation + private let reservation: TransformValueLease + fileprivate init(_ ordinal: UInt64, population: SelectionPopulation, reservation: TransformValueLease) { + self.ordinal = ordinal; self.population = population; self.reservation = reservation + } + fileprivate var identity: SelectionIdentity { population.identity } +} + +/// An owner-produced EOF witness and successful accounting snapshot; no public initializer. +/// The witness retains only identity/report storage, not the population. Escaped +/// positions retain that population separately. Logical reservations are not RSS. +public struct JSONTransformSelectionCompletion { + /// Indexed-selection convention revision, independent of query and definition versions. + public var conventionRevision: UInt32 { 1 } + /// Completely admitted population count, including duplicate-valued members. + public let populationCount: UInt64 + /// Successfully published positions, including repeats. + public let outputCount: UInt64 + /// Successful cumulative native snapshot after closing VM/provider state. + public let statistics: JSONOperationStatistics + private let identity: SelectionIdentity + private let reservation: TransformValueLease + fileprivate init(identity: SelectionIdentity, reservation: TransformValueLease, + populationCount: UInt64, outputCount: UInt64, statistics: JSONOperationStatistics) { + self.identity = identity; self.reservation = reservation + self.populationCount = populationCount; self.outputCount = outputCount; self.statistics = statistics + } + /// Verifies invocation identity and extracts an ordinal after successful completion. + /// This constant metadata inspection is caller work, not another VM invocation. + /// - Parameter position: A position emitted by this invocation. + /// - Returns: Its exact original population index. + /// - Throws: ForeignPosition for a handle from another invocation. + public func ordinal(of position: JSONTransformSelectionPosition) throws -> UInt64 { + guard identity === position.identity else { throw JSONTransformSelectionFailure.foreignPosition } + return position.ordinal + } +} + +/// The caller result paired with an unforgeable successful selection completion. +public struct JSONTransformSelectionReport { + /// The value returned by the synchronous caller body. + public let result: Result + /// Successful EOF, invocation identity and actual native accounting. + public let completion: JSONTransformSelectionCompletion + fileprivate init(result: Result, completion: JSONTransformSelectionCompletion) { + self.result = result; self.completion = completion + } +} + +private final class SelectionHealth { + struct Observation { + let statistics: JSONOperationStatistics + let remaining: UInt64 + let counters: [UInt64] + } + var pulling = false + var failure: (any Error)? + var observation: Observation? + func latch(_ error: any Error) { + if failure == nil { + failure = error is JSONTransformSelectionFailure ? error : JSONTransformFailure.classify(error) + } + } + func check(context: inout TransformRuntimeContext) throws { + if let failure { context.latch(failure); throw failure } + } + func observe(context: inout TransformRuntimeContext) throws { + try context.work(66) + let cap = min(context.limits.core[.workUnits], context.limits[.totalWorkUnits]) + observation = .init(statistics: context.meter.statistics(), + remaining: cap - min(cap, context.used(.totalWorkUnits)), + counters: TransformRuntimeMetric.allCases.map { context.used($0) }) + } +} +private struct SelectionInput: FilterInput { + var source: (() throws -> JSON?)? + let health: SelectionHealth + var ended = false + mutating func next(context: inout TransformRuntimeContext) throws -> FilterYield? { + try health.check(context: &context); try context.checkpoint() + guard !ended, let source else { return nil } + try health.observe(context: &context) + defer { health.observation = nil } + let value: JSON? + do { value = try source() } + catch { + health.latch(JSONTransformFailure(error is CancellationError ? .cancelled : .provider, + error is CancellationError ? "cancelled" : "providerFailed")) + try health.check(context: &context); throw error + } + try health.check(context: &context); try context.checkpoint() + guard let value else { ended = true; self.source = nil; return nil } + try context.charge(.totalRecords) + return try FilterValues.produced(value, context: &context) + } +} + +/// A guarded, non-Sendable indexed-selection pull scope with one cumulative ledger. +/// V1 input is [0,...,N-1]; `$values` is the complete original population. Additional +/// bindings are explicit. Provider values may influence queries but never enlarge +/// the population. Returning without pulling nil cannot certify a prefix. +public final class JSONTransformSelectionScope { + private var context: TransformRuntimeContext + private var vm: FilterVM? + private var population: SelectionPopulation? + private var input: SelectionInput + private let health: SelectionHealth + private var planBytes: UInt64 = 0, bindingBytes: UInt64 = 0, initialBindings: UInt64 = 0 + private var ended = false, closed = false + private var published: UInt64 = 0 + private let count: UInt64 + private var cleanupFailure: (any Error)? + private static var stateBytes: UInt64 { 2048 } + private init(context: consuming TransformRuntimeContext, count: UInt64, source: (() throws -> JSON?)?) { + self.context = consume context; self.count = count + let health = SelectionHealth(); self.health = health + input = .init(source: source, health: health) + } + static func make(plan: JSONTransformProgram, values: [JSON], bindings: [String: JSON], + limits: JSONTransformSelectionLimits, source: (() throws -> JSON?)?, + observing: ((JSONTransformSelectionScope) -> Void)? = nil) throws -> JSONTransformSelectionScope { + do { + guard limits.maximumPopulationValues < UInt64(Int.max) else { + throw JSONTransformFailure(.configuration, "invalidLimits") + } + var context = try TransformRuntimeContext(limits: limits.transform.runtime) + try context.domain(.core) + context.meter.checkpointObserver = limits.checkpointObserver + try context.meter.retain(stateBytes) + let scope = JSONTransformSelectionScope(context: consume context, count: UInt64(values.count), source: source) + observing?(scope) + do { + try scope.prepare(plan, values: values, bindings: bindings, maximum: limits.maximumPopulationValues, projectionObserver: limits.projectionObserver) + return scope + } catch { + scope.health.latch(error) + scope.closePreservingFailure() + throw scope.health.failure ?? error + } + } catch { + if error is JSONTransformSelectionFailure { throw error } + throw JSONTransformFailure.classify(error) + } + } + private func prepare(_ plan: JSONTransformProgram, values: [JSON], bindings: [String: JSON], maximum: UInt64, + projectionObserver: (@Sendable (UInt64) -> Void)?) throws { + try context.checkpoint() + guard count <= maximum else { + throw JSONTransformFailure(.resource, "limitExceeded", metric: "populationValues", limit: maximum, attemptedUsage: count) + } + guard plan.variables.contains("values"), bindings.count == plan.variables.count - 1 else { + throw JSONTransformFailure(.configuration, "invalidBindings") + } + try context.meter.retain(plan.planReservation); planBytes = plan.planReservation + let identity = SelectionIdentity(try makeLease(64)) + let admitted = try FilterValues.array(values, context: &context) + let metadata = try makeLease(48) + population = SelectionPopulation(admitted, identity: identity, metadata: metadata) + var supplied: [String: FilterYield] = [:] + try admitName("values") + supplied["values"] = admitted + for (name, value) in bindings { + try admitName(name) + guard name != "values", plan.variables.contains(name) else { throw JSONTransformFailure(.configuration, "invalidBindings") } + supplied[name] = try FilterValues.produced(value, context: &context) + } + var indices: [JSON] = [], scratch: UInt64 = 0, slots: UInt64 = 0 + defer { + do { try context.meter.release(scratch); try context.release(.collectionValues, slots) } + catch { recordCleanup(error) } + } + for ordinal in values.indices { + try context.work(20) + try context.charge(.collectionValues); slots += 1 + try context.meter.retain(64); scratch += 64 + indices.append(.number(JSONNumber(integer: ordinal))) + projectionObserver?(UInt64(indices.count)) + } + let root = try FilterValues.array(indices, admittedCollectionCount: true, context: &context) + slots = 0 // The root's lease owns these slots now. + indices = []; try context.meter.release(scratch); scratch = 0 + try context.charge(.totalRecords) + vm = try FilterVM(program: plan.program, input: root, bindings: supplied, context: &context) + } + private func makeLease(_ bytes: UInt64) throws -> TransformValueLease { + try context.meter.retain(bytes) + do { return try TransformValueLease(ownedRetained: bytes, context: &context) } + catch { try context.meter.release(bytes); throw error } + } + private func admitName(_ name: String) throws { + try context.charge(.bindings); initialBindings += 1 + var first = true + for byte in name.utf8 { + try context.work() + let letter = byte == 95 || (65...90).contains(byte) || (97...122).contains(byte) + guard letter || (!first && (48...57).contains(byte)) else { throw JSONTransformFailure(.configuration, "invalidBindings") } + first = false + } + guard !first else { throw JSONTransformFailure(.configuration, "invalidBindings") } + let before = context.meter.used(.retainedBytes) + do { + try JSONAdmission.text(name, key: true, meter: &context.meter, owned: true) + try context.meter.retain(64) + bindingBytes += context.meter.used(.retainedBytes) - before + } catch { + try context.meter.release(context.meter.used(.retainedBytes) - before); throw error + } + } + /// Pulls the next valid original-population position, or nil at explicit EOF. + /// Invalid outputs, resource/provider failures, cancellation and reentry latch; + /// query catches and later caller catches cannot restore successful completion. + public func next() throws -> JSONTransformSelectionPosition? { + guard !health.pulling else { + health.latch(JSONTransformFailure(.configuration, "reentrantSelection")) + throw health.failure ?? JSONTransformFailure(.configuration, "reentrantSelection") + } + health.pulling = true; defer { health.pulling = false } + do { + try context.reclaimValueLeases(); try health.check(context: &context) + try context.checkpoint() + guard !closed else { throw JSONTransformFailure(.configuration, "scopeClosed") } + guard !ended, var active = vm else { return nil } + vm = nil; defer { vm = active } + guard let raw = try active.next(input: &input, context: &context) else { ended = true; return nil } + let before = context.meter.used(.retainedBytes) + do { try context.meter.resetValueBounds(); try JSONAdmission.value(raw.value, meter: &context.meter) } + catch { try context.meter.release(context.meter.used(.retainedBytes) - before); throw error } + try context.charge(.totalResults) + guard case .number(let number) = raw.value else { throw JSONTransformSelectionFailure.invalidPosition } + let ordinal: UInt64 + do { ordinal = try number.integer(as: UInt64.self, meter: &context.meter) } + catch let failure as JSONFailure where failure.category == .conversion { throw JSONTransformSelectionFailure.invalidPosition } + guard let native = Int(exactly: ordinal), native >= 0, ordinal < count, let population else { + throw JSONTransformSelectionFailure.invalidPosition + } + let lease = try makeLease(32) + published += 1 + return .init(ordinal, population: population, reservation: lease) + } catch { + health.latch(error); throw health.failure ?? error + } + } + /// Drains deferred lease refunds and returns actual native accounting, even after failure/close. + /// During a provider callback this returns its admitted entry snapshot without + /// borrowing the VM's context. It never reopens data operations or refunds work. + /// - Throws: An accounting cleanup failure; a prior operational cause takes precedence. + public func report() throws -> JSONOperationStatistics { + if health.pulling { + if let observation = health.observation { return observation.statistics } + health.latch(JSONTransformFailure(.configuration, "reentrantSelection")) + throw health.failure ?? JSONTransformFailure(.configuration, "reentrantSelection") + } + do { try context.reclaimValueLeases() } + catch { recordCleanup(error); throw health.failure ?? error } + if let cleanupFailure { throw health.failure ?? cleanupFailure } + return context.meter.statistics() + } + /// Remaining cumulative owned work; provider entry uses an admitted snapshot. + public var remainingWorkUnits: UInt64 { + if health.pulling { return health.observation?.remaining ?? 0 } + let cap = min(context.limits.core[.workUnits], context.limits[.totalWorkUnits]) + return cap - min(cap, context.used(.totalWorkUnits)) + } + /// Reads a cumulative/live transform counter; call report to settle deferred refunds. + /// - Parameter metric: The existing transform counter, without a public mutable meter. + public func used(_ metric: JSONTransformLimits.Metric) -> UInt64 { + if health.pulling { return health.observation?.counters[metric.runtime.index] ?? 0 } + return context.used(metric.runtime) + } + /// Latches cancellation without borrowing a context loaned to the provider. + public func cancel() { + guard !closed else { return } + health.latch(JSONTransformFailure(.cancelled, "cancelled")) + if !health.pulling, let failure = health.failure { context.latch(failure) } + } + fileprivate var firstFailure: (any Error)? { health.failure } + fileprivate func complete() throws -> JSONTransformSelectionCompletion { + do { + try context.checkpoint(); try health.check(context: &context) + if let cleanupFailure { throw cleanupFailure } + guard ended, let identity = population?.identity else { throw JSONTransformSelectionFailure.incompleteSelection } + try context.work(50) + let lease = try makeLease(1024) + try close() + return .init(identity: identity, reservation: lease, populationCount: count, + outputCount: published, statistics: try report()) + } catch { health.latch(error); throw health.failure ?? error } + } + private func recordCleanup(_ error: any Error) { + if cleanupFailure == nil { cleanupFailure = error } + health.latch(error) + } + fileprivate func closePreservingFailure() { + do { try close() } catch { recordCleanup(error) } + } + private func close() throws { + guard !closed else { return }; closed = true + input.source = nil + if var active = vm { vm = nil; try active.close(context: &context) } + population = nil + try context.reclaimValueLeases() + try context.meter.release(planBytes + bindingBytes + Self.stateBytes) + try context.release(.bindings, initialBindings) + planBytes = 0; bindingBytes = 0; initialBindings = 0 + } +} + +extension JSONTransformProgram { + /// Runs indexed-selection convention V1 on one fully admitted finite population. + /// Input is an owned index array; `$values` is the corresponding value array. + /// The plan must declare values, and bindings supply exactly the other names. + /// Provider work/bytes already owned by the caller stay outside engine quotas. + /// - Parameters: + /// - population: Explicit original order, including duplicate-valued members. + /// - bindings: Additional declared bindings; values is reserved and cannot be supplied here. + /// - limits: One finite ledger for admission, projection, evaluation and outputs. + /// - source: Optional explicit input/inputs provider; nil means no extra inputs. + /// - body: Synchronous caller work; pull nil to certify complete selection. + /// - Returns: Caller result and an owner-produced EOF/accounting witness. + /// - Throws: Selection refusals, typed operational failures or uncaught caller errors. + public func withSelection(population: [JSON], bindings: [String: JSON] = [:], + limits: JSONTransformSelectionLimits = .init(), source: (() throws -> JSON?)? = nil, + _ body: (JSONTransformSelectionScope) throws -> Result) throws -> JSONTransformSelectionReport { + let scope = try JSONTransformSelectionScope.make(plan: self, values: population, bindings: bindings, limits: limits, source: source) + let result: Result + do { result = try body(scope) } + catch { + let cause = scope.firstFailure ?? error + scope.closePreservingFailure(); throw cause + } + do { return .init(result: result, completion: try scope.complete()) } + catch { + let cause = scope.firstFailure ?? error + scope.closePreservingFailure(); throw cause + } + } +} diff --git a/Tests/JSONTransformTests/SelectionTests.swift b/Tests/JSONTransformTests/SelectionTests.swift new file mode 100644 index 0000000..c3fc006 --- /dev/null +++ b/Tests/JSONTransformTests/SelectionTests.swift @@ -0,0 +1,338 @@ +// SPDX-License-Identifier: Apache-2.0 +import Testing +import Synchronization +import JSON +@testable import JSONTransform + +private func selectionPlan(_ source: String, extra: Set = []) throws -> JSONTransformProgram { + try .compile(source, variables: extra.union(["values"])) +} +private func select(_ source: String, _ values: [JSON], limits: JSONTransformSelectionLimits = .init()) throws -> JSONTransformSelectionReport<[JSONTransformSelectionPosition]> { + try selectionPlan(source).withSelection(population: values, limits: limits) { scope in + var positions: [JSONTransformSelectionPosition] = [] + while let p = try scope.next() { positions.append(p) } + return positions + } +} +private enum SelectionTestError: Error, Equatable { case sentinel } +private func refusal(_ expected: JSONTransformSelectionFailure, _ body: () throws -> Void) { + do { try body(); Issue.record("Expected selection refusal \(expected)") } + catch let e as JSONTransformSelectionFailure { #expect(e == expected) } + catch { Issue.record("Wrong failure \(error)") } +} + +@Test func selectionS01ExplicitOrdinalsPreserveDuplicateValuesOrderRepeatsAndTies() throws { + let row = try JSON.parse(#"{"score":1,"é":9007199254740993,"é":0.1}"#) + for (source, indices) in [(".[] | select(. == 1)", [UInt64(1)]), ("1,0,1", [1,0,1]), + ("sort_by([$values[.].score, .])[]", [0,1]), + (".[] | select($values[.][\"é\"] == 9007199254740993)", [0,1])] { + let report = try select(source, [row,row]) + #expect(report.result.map(\.ordinal) == indices) + #expect(try report.result.map { try report.completion.ordinal(of: $0) } == indices) + #expect(report.completion.populationCount == 2 && report.completion.outputCount == UInt64(indices.count)) + #expect(report.completion.conventionRevision == 1 && report.completion.statistics.domain == .core) + } +} + +@Test func selectionS02EmptyEOFNullAndIncompletePrefixAreDistinct() throws { + var zero = JSONTransformSelectionLimits(); zero.maximumPopulationValues = 0 + let empty = try select("empty", [], limits: zero) + #expect(empty.result.isEmpty && empty.completion.outputCount == 0 && empty.completion.populationCount == 0) + refusal(.incompleteSelection) { _ = try selectionPlan("empty").withSelection(population: []) { _ in () } } + refusal(.incompleteSelection) { + _ = try selectionPlan("0,0").withSelection(population: [42]) { s in _ = try s.next() } + } + refusal(.invalidPosition) { _ = try select("null", [.null]) } + let presentNull = try select("0", [.null]) + #expect(presentNull.result.first?.ordinal == 0 && presentNull.completion.outputCount == 1) +} + +@Test func selectionS03EveryInvalidOutputIsExactAndTerminalOutsideQueryCatch() throws { + for bad in ["null","false","\"0\"","[]","{}","-1","0.1","18446744073709551616","18446744073709551615","9223372036854775808","1"] { + var escaped: JSONTransformSelectionScope? + refusal(.invalidPosition) { + _ = try selectionPlan("try (0, " + bad + ") catch 0").withSelection(population: [42]) { s in + escaped = s; let first = try s.next(); #expect(first?.ordinal == 0) + for _ in 0..<2 { refusal(.invalidPosition) { _ = try s.next() } } + } + } + let scope = try #require(escaped) + refusal(.invalidPosition) { _ = try scope.next() } + #expect(try scope.report()[.retainedBytes] == 0) + } + #expect(try select("42.0", Array(repeating: .null, count: 43)).result.first?.ordinal == 42) + for source in ["0.0", "-0", "0e100"] { #expect(try select(source, [42]).result.first?.ordinal == 0) } +} + +@Test func selectionS04CompletionRejectsForeignInvocationsAndPositionsRetainPopulation() throws { + let plan = try selectionPlan("0") + var scope: JSONTransformSelectionScope? + var a: JSONTransformSelectionReport? = try plan.withSelection(population: [.string(String(repeating: "x", count: 8192))]) { s in + scope = s; let p = try s.next(); #expect(try s.next() == nil); return p + } + var position = a?.result, alias = position + var completion = a?.completion + let b = try select("0", [.string(String(repeating: "x", count: 8192))]) + let p = try #require(position), witness = try #require(completion) + refusal(.foreignPosition) { _ = try b.completion.ordinal(of: p) } + #expect(try witness.ordinal(of: p) == 0) + // Keep only aliases in a nested scope so last ARC releases are observable below. + a = nil; position = nil + let owner = try #require(scope) + let withAlias = try withExtendedLifetime(alias) { try owner.report()[.retainedBytes] } + #expect(withAlias > 8192) + alias = nil; completion = nil + // p/witness still own their real reservations until this function exits; + // the separate S05 case proves the exact last-alias transition. + #expect(try owner.report()[.retainedBytes] > 8192) + do { _ = try owner.next(); Issue.record("Escaped closed scope resumed") } + catch let e as JSONTransformFailure { #expect(e.category == .configuration && e.code == "scopeClosed") } +} + +@Test func selectionS05FinalAliasAndCompletionReleaseSettleToZeroWithoutDataPulls() throws { + var scope: JSONTransformSelectionScope? + var position: JSONTransformSelectionPosition? + var completion: JSONTransformSelectionCompletion? + do { + let report = try selectionPlan("0").withSelection(population: [.string(String(repeating: "p", count: 8192))]) { s in + scope = s; position = try s.next(); #expect(try s.next() == nil) + } + completion = report.completion + } + let owner = try #require(scope) + let bytes = try withExtendedLifetime(position) { try owner.report()[.retainedBytes] } + #expect(bytes > 8192) + position = nil + let identityOnly = try withExtendedLifetime(completion) { try owner.report()[.retainedBytes] } + #expect(identityOnly > 0 && identityOnly < 8192) + completion = nil + let settled = try owner.report() + #expect(settled[.retainedBytes] == 0 && owner.used(.collectionValues) == 0 && owner.used(.bindings) == 0) + #expect(settled[.workUnits] > 0) +} + +@Test func selectionS06BindingsAreExplicitReservedAndScalarChecked() throws { + let plan = try selectionPlan(".[] | select($values[.] >= $minimum)", extra: ["minimum"]) + let report = try plan.withSelection(population: [1,2,3], bindings: ["minimum": 2]) { s in + var ids: [UInt64] = []; while let p = try s.next() { ids.append(p.ordinal) }; return ids + } + #expect(report.result == [1,2]) + for bindings: [String:JSON] in [[:],["values": .null],["other": 2],["K": 2]] { + do { _ = try plan.withSelection(population: [1], bindings: bindings) { _ in () }; Issue.record("Accepted binding override/mismatch") } + catch let e as JSONTransformFailure { #expect(e.category == .configuration && e.code == "invalidBindings") } + } + do { _ = try JSONTransformProgram.compile("0").withSelection(population: [1]) { _ in () }; Issue.record("Missing reserved declaration accepted") } + catch let e as JSONTransformFailure { #expect(e.code == "invalidBindings") } +} + +@Test func selectionS07PopulationPolicyIsInclusiveFiniteAndCannotSkipCompleteAdmission() throws { + var policy = JSONTransformSelectionLimits(); policy.maximumPopulationValues = 2 + _ = try select("empty", [1,2], limits: policy) + policy.maximumPopulationValues = 1 + do { _ = try select("empty", [1,2], limits: policy); Issue.record("Population cap escaped") } + catch let e as JSONTransformFailure { #expect(e.category == .resource && e.metric == "populationValues" && e.limit == 1 && e.attemptedUsage == 2) } + for cap in [UInt64.max, UInt64(Int.max)] { + policy = .init(); policy.maximumPopulationValues = cap + do { _ = try select("empty", [], limits: policy); Issue.record("Infinite/native overflow policy accepted") } + catch let e as JSONTransformFailure { #expect(e.category == .configuration && e.code == "invalidLimits") } + } + policy = .init(); policy.transform.json[.stringBytes] = 1 + do { _ = try select("empty", [0,.string("too long")], limits: policy); Issue.record("Unused population tail bypassed admission") } + catch let e as JSONTransformFailure { #expect(e.category == .resource && e.metric == "stringBytes") } +} + +@Test func selectionS08ProjectionEvaluationAndCompletionShareExactCumulativeWorkAndRetention() throws { + let source = "sort_by([$values[.],.])[]", values: [JSON] = [3,1,2] + let baseline = try select(source, values) + var policy = JSONTransformSelectionLimits() + let work = baseline.completion.statistics[.workUnits], peak = baseline.completion.statistics.peakLogicalRetainedBytes + policy.transform[.workUnits] = work + #expect(try select(source, values, limits: policy).result.map(\.ordinal) == [1,2,0]) + policy.transform[.workUnits] = work - 1 + do { _ = try select(source, values, limits: policy); Issue.record("Completion work reset") } + catch let e as JSONTransformFailure { #expect(e.category == .resource && e.metric == "workUnits") } + policy = .init(); policy.transform[.retainedBytes] = peak + _ = try select(source, values, limits: policy) + policy.transform[.retainedBytes] = peak - 1 + do { _ = try select(source, values, limits: policy); Issue.record("Retention escaped") } + catch let e as JSONTransformFailure { #expect(e.category == .resource && e.metric == "retainedBytes") } + policy = .init(); policy.transform[.collectionValues] = 5 + do { _ = try select("empty", values, limits: policy); Issue.record("Projection slots not pre-admitted") } + catch let e as JSONTransformFailure { #expect(e.category == .resource && e.metric == "collectionValues") } +} + +@Test func selectionS09QuotaAndCancellationRefundTerminalAliasesAndPreserveFirstCause() throws { + for cancel in [false,true] { + var scope: JSONTransformSelectionScope?, output: JSONTransformSelectionPosition? + var policy = JSONTransformSelectionLimits(); policy.transform[.results] = 1 + do { + _ = try selectionPlan("0,0").withSelection(population: [.string(String(repeating: "q", count: 4096))], limits: policy) { s in + scope = s; output = try s.next() + if cancel { s.cancel() } + for _ in 0..<2 { + do { _ = try s.next(); Issue.record("Terminal error swallowed") } + catch let e as JSONTransformFailure { #expect(e.category == (cancel ? .cancelled : .resource)) } + } + throw SelectionTestError.sentinel + } + Issue.record("Failed invocation certified") + } catch let e as JSONTransformFailure { #expect(e.category == (cancel ? .cancelled : .resource)) } + let s = try #require(scope) + let bytes = try withExtendedLifetime(output) { try s.report()[.retainedBytes] } + #expect(bytes > 4096); output = nil + #expect(try s.report()[.retainedBytes] == 0) + } +} + +@Test func selectionS10ProviderEOFNullErrorAndNoFurtherPullsRemainDistinct() throws { + var calls = 0 + let empty = try selectionPlan("inputs").withSelection(population: [], source: { calls += 1; return nil }) { s in let eof = try s.next(); #expect(eof == nil) } + #expect(calls == 1 && empty.completion.outputCount == 0) + calls = 0 + refusal(.invalidPosition) { + _ = try selectionPlan("input").withSelection(population: [1], source: { calls += 1; return .null }) { s in + for _ in 0..<2 { refusal(.invalidPosition) { _ = try s.next() } } + } + } + #expect(calls == 1) + calls = 0 + do { + _ = try selectionPlan("try input catch 0").withSelection(population: [1], source: { calls += 1; throw SelectionTestError.sentinel }) { s in + for _ in 0..<2 { + do { _ = try s.next(); Issue.record("Provider failure swallowed") } + catch let e as JSONTransformFailure { #expect(e.category == .provider && e.code == "providerFailed") } + } + } + Issue.record("Provider error certified") + } catch let e as JSONTransformFailure { #expect(e.category == .provider) } + #expect(calls == 1) +} + +@Test func selectionS11ReentryAndProviderCancellationUseSafeEntrySnapshots() throws { + for cancel in [false,true] { + var escaped: JSONTransformSelectionScope?, calls = 0 + do { + _ = try selectionPlan("input").withSelection(population: [42], source: { + calls += 1 + if let s = escaped { + let report = try s.report(); #expect(report[.workUnits] == s.used(.workUnits)) + #expect(s.remainingWorkUnits > 0) + if cancel { s.cancel() } + else { + do { _ = try s.next(); Issue.record("Provider reentry accepted") } + catch let e as JSONTransformFailure { #expect(e.code == "reentrantSelection") } + } + } + return 0 + }) { s in + escaped = s + do { _ = try s.next(); Issue.record("Provider ignored terminal health") } + catch let e as JSONTransformFailure { #expect(e.category == (cancel ? .cancelled : .configuration)) } + } + Issue.record("Caught provider failure certified") + } catch let e as JSONTransformFailure { #expect(e.category == (cancel ? .cancelled : .configuration)) } + #expect(calls == 1) + #expect(try #require(escaped).report()[.retainedBytes] == 0) + } +} + +@Test func selectionS12CallerErrorAndEscapedClosuresReleaseProviderWithoutCompletion() throws { + final class Source { var calls = 0; func next() -> JSON? { calls += 1; return 0 } } + weak var weakSource: Source? + var scope: JSONTransformSelectionScope?, closure: (() throws -> JSONTransformSelectionPosition?)? + do { + let source = Source(); weakSource = source + do { + _ = try selectionPlan("input").withSelection(population: [42], source: { source.next() }) { s in + scope = s; closure = { try s.next() }; throw SelectionTestError.sentinel + } + Issue.record("Caller error lost") + } catch let e as SelectionTestError { #expect(e == .sentinel) } + } + #expect(weakSource == nil) + do { _ = try closure?(); Issue.record("Escaped closure resumed") } + catch let e as JSONTransformFailure { #expect(e.code == "scopeClosed") } + #expect(try #require(scope).report()[.retainedBytes] == 0) +} + +@Test func selectionSharedImmutablePlanHasIndependentPopulationAndFailureHealth() async throws { + let plan = try selectionPlan(".[]") + let results = try await withThrowingTaskGroup(of: [UInt64].self) { group in + for size in 0..<6 { + group.addTask { + var policy = JSONTransformSelectionLimits() + if size == 5 { policy.transform[.results] = 0 } + do { + let report = try plan.withSelection(population: Array(repeating: .null, count: size), limits: policy) { s in + var ids: [UInt64] = []; while let p = try s.next() { ids.append(p.ordinal) }; return ids + } + #expect(size != 5 && report.completion.populationCount == UInt64(size)); return report.result + } catch let e as JSONTransformFailure { #expect(size == 5 && e.category == .resource); return [99] } + } + } + var result: [[UInt64]] = []; for try await row in group { result.append(row) }; return result + } + #expect(results.count == 6 && results.contains([99]) && results.contains([0,1,2,3])) +} + +@Test func selectionS13InnerProjectionTaskCancellationReleasesUnpublishedPreparation() async throws { + let plan = try selectionPlan(".[]") + let handle = Mutex?>(nil), ready = Mutex?>(nil) + let observed = Mutex(0) + let task = Task { + await withCheckedContinuation { continuation in ready.withLock { $0 = continuation } } + var policy = JSONTransformSelectionLimits() + policy.projectionObserver = { count in + if count == 4 { observed.withLock { $0 = count }; handle.withLock { $0?.cancel() } } + } + var failed: JSONTransformSelectionScope? + do { + _ = try JSONTransformSelectionScope.make(plan: plan, values: Array(repeating: .null, count: 32), bindings: [:], limits: policy, source: nil, observing: { failed = $0 }) + Issue.record("Projection ignored task cancellation") + } catch let e as JSONTransformFailure { #expect(e.category == .cancelled) } + return try #require(failed).report()[.retainedBytes] + } + handle.withLock { $0 = task } + var continuation: CheckedContinuation? + while continuation == nil { continuation = ready.withLock { $0 }; if continuation == nil { await Task.yield() } } + continuation?.resume() + #expect(try await task.value == 0) + #expect(observed.withLock { $0 } == 4) +} + +@Test func selectionS14FailedPreparationAndCompletionAllocationsRefundExactBoundaries() throws { + let plan = try selectionPlan("empty") + var failed: JSONTransformSelectionScope? + var policy = JSONTransformSelectionLimits(); policy.transform[.collectionValues] = 3 + do { + _ = try JSONTransformSelectionScope.make(plan: plan, values: [1,2], bindings: [:], limits: policy, source: nil, observing: { failed = $0 }) + Issue.record("Partial projection admitted") + } catch let e as JSONTransformFailure { #expect(e.category == .resource && e.metric == "collectionValues") } + #expect(try #require(failed).report()[.retainedBytes] == 0) + let base = try select("empty", []) + policy = .init(); policy.transform[.workUnits] = base.completion.statistics[.workUnits] - 1 + var closed: JSONTransformSelectionScope? + do { + _ = try selectionPlan("empty").withSelection(population: [], limits: policy) { s in + closed = s; let eof = try s.next(); #expect(eof == nil) + } + Issue.record("Completion exceeded work budget") + } catch let e as JSONTransformFailure { #expect(e.category == .resource) } + #expect(try #require(closed).report()[.retainedBytes] == 0) +} + +@Test func selectionS15LanguageCatchesRemainOrdinaryAndUncaughtFailuresStayTerminal() throws { + #expect(try select("try error(\"expected\") catch 0", [42]).result.first?.ordinal == 0) + var scope: JSONTransformSelectionScope? + do { + _ = try selectionPlan("0,error(\"expected\")").withSelection(population: [42]) { s in + scope = s; let p = try s.next(); #expect(p?.ordinal == 0) + for _ in 0..<2 { + do { _ = try s.next(); Issue.record("Language failure reopened") } + catch let e as JSONTransformFailure { #expect(e.category == .language) } + } + } + Issue.record("Uncaught language failure certified") + } catch let e as JSONTransformFailure { #expect(e.category == .language) } + #expect(try #require(scope).report()[.retainedBytes] == 0) +} diff --git a/Tools/TransformSelectionQualification/Package.swift b/Tools/TransformSelectionQualification/Package.swift new file mode 100644 index 0000000..3985a93 --- /dev/null +++ b/Tools/TransformSelectionQualification/Package.swift @@ -0,0 +1,8 @@ +// swift-tools-version: 6.4 +// SPDX-License-Identifier: Apache-2.0 +import PackageDescription +let package = Package(name: "TransformSelectionQualification", platforms: [.macOS("27.0"), .iOS("27.0")], + dependencies: [.package(name: "Swift-JSON", path: "../..")], + targets: [.executableTarget(name: "TransformSelectionConsumer", dependencies: [ + .product(name: "JSON", package: "Swift-JSON"), .product(name: "JSONTransform", package: "Swift-JSON") + ])], swiftLanguageModes: [.v6]) diff --git a/Tools/TransformSelectionQualification/README.md b/Tools/TransformSelectionQualification/README.md new file mode 100644 index 0000000..02a5783 --- /dev/null +++ b/Tools/TransformSelectionQualification/README.md @@ -0,0 +1,10 @@ +# Public indexed-selection consumer + +Run `swift run --package-path Tools/TransformSelectionQualification TransformSelectionConsumer`. +Only JSON and JSONTransform are imported. Ten witnesses execute the exact DocC +example, order/repeats, empty completion, late invalidity, foreign ownership, +explicit bindings, final lease release, provider terminality, cumulative work +and cancellation. The qualification script copies this package outside Git, +runs Debug/Release and checks ordinary full-codegen public boundary refusals. +A path dependency is preparation evidence, not version-tag resolution or +Content S06/S13 admission. Logical reservations are not RSS or caller work caps. diff --git a/Tools/TransformSelectionQualification/Sources/TransformSelectionConsumer/Main.swift b/Tools/TransformSelectionQualification/Sources/TransformSelectionConsumer/Main.swift new file mode 100644 index 0000000..d6a214d --- /dev/null +++ b/Tools/TransformSelectionQualification/Sources/TransformSelectionConsumer/Main.swift @@ -0,0 +1,103 @@ +// SPDX-License-Identifier: Apache-2.0 +import JSON +import JSONTransform + +func selectionGuideExample() throws { + let definition = try JSONTransformProgramDefinition.prepare( + source: ".[] | select($values[.].enabled)", variables: ["values"]).result + let plan = try definition.compile().result + let row = try JSON.parse(#"{"enabled":true,"amount":9007199254740993}"#) + let report = try plan.withSelection(population: [row, row]) { scope in + var positions: [JSONTransformSelectionPosition] = [] + while let position = try scope.next() { positions.append(position) } + return positions + } + let ordinals = try report.result.map { try report.completion.ordinal(of: $0) } + precondition(ordinals == [0, 1]) + precondition(report.completion.populationCount == 2) + precondition(report.completion.outputCount == 2) +} +private func run(_ source: String, _ population: [JSON], limits: JSONTransformSelectionLimits = .init()) throws -> JSONTransformSelectionReport<[JSONTransformSelectionPosition]> { + try JSONTransformProgram.compile(source, variables: ["values"]).withSelection(population: population, limits: limits) { s in + var positions: [JSONTransformSelectionPosition] = [] + while let p = try s.next() { positions.append(p) }; return positions + } +} +private enum ConsumerError: Error { case provider } +@main enum Consumer { + static func main() throws { + var witnesses: [String] = [] + try selectionGuideExample(); witnesses.append("compiled-guide") + let ordered = try run("1,0,1", [42,42]) + precondition(ordered.result.map(\.ordinal) == [1,0,1]); witnesses.append("order-repeats") + let empty = try run("empty", []) + precondition(empty.completion.outputCount == 0 && empty.completion.populationCount == 0); witnesses.append("empty-eof") + do { + _ = try JSONTransformProgram.compile("try (0,null) catch 0", variables: ["values"]).withSelection(population: [42]) { s in + let first = try s.next(); precondition(first?.ordinal == 0) + do { _ = try s.next(); preconditionFailure("Invalid position accepted") } + catch let e as JSONTransformSelectionFailure { precondition(e == .invalidPosition) } + } + preconditionFailure("Caught boundary error certified") + } catch let e as JSONTransformSelectionFailure { precondition(e == .invalidPosition) } + witnesses.append("late-invalid") + let other = try run("0", [42,42]) + do { _ = try other.completion.ordinal(of: ordered.result[0]); preconditionFailure("Foreign position accepted") } + catch let e as JSONTransformSelectionFailure { precondition(e == .foreignPosition) } + witnesses.append("foreign-owner") + let parameterized = try JSONTransformProgram.compile(".[] | select($values[.] >= $minimum)", variables: ["values","minimum"]) + let selection = try parameterized.withSelection(population: [1,2,3], bindings: ["minimum":2]) { s in + var ids: [UInt64] = []; while let p = try s.next() { ids.append(p.ordinal) }; return ids + } + precondition(selection.result == [1,2]); witnesses.append("bindings") + var escaped: JSONTransformSelectionScope? + var position: JSONTransformSelectionPosition? + var completion: JSONTransformSelectionCompletion? + do { + let report = try JSONTransformProgram.compile("0", variables: ["values"]).withSelection(population: [.string(String(repeating:"x",count:8192))]) { s in + escaped = s; position = try s.next(); let eof = try s.next(); precondition(eof == nil) + } + completion = report.completion + } + let owner = escaped! + let retained = try withExtendedLifetime(position) { try owner.report()[.retainedBytes] } + precondition(retained > 8192); position = nil + let identityOnly = try withExtendedLifetime(completion) { try owner.report()[.retainedBytes] } + precondition(identityOnly > 0 && identityOnly < 8192); completion = nil + let settled = try owner.report(); precondition(settled[.retainedBytes] == 0) + do { _ = try owner.next(); preconditionFailure("Escaped scope resumed") } + catch let e as JSONTransformFailure { precondition(e.code == "scopeClosed") } + witnesses.append("terminal-refunds") + var calls = 0 + do { + _ = try JSONTransformProgram.compile("try input catch 0", variables: ["values"]).withSelection(population: [42], source: { calls += 1; throw ConsumerError.provider }) { s in + for _ in 0..<2 { + do { _ = try s.next(); preconditionFailure("Provider error swallowed") } + catch let e as JSONTransformFailure { precondition(e.category == .provider) } + } + } + preconditionFailure("Provider error certified") + } catch let e as JSONTransformFailure { precondition(e.category == .provider) } + precondition(calls == 1); witnesses.append("provider-terminal") + let measured = try run(".[]", [1,2,3]) + var limits = JSONTransformSelectionLimits(); limits.transform[.workUnits] = measured.completion.statistics[.workUnits] + _ = try run(".[]", [1,2,3], limits: limits) + limits.transform[.workUnits] -= 1 + do { _ = try run(".[]", [1,2,3], limits: limits); preconditionFailure("Budget reset") } + catch let e as JSONTransformFailure { precondition(e.category == .resource) } + witnesses.append("cumulative-work") + var failed: JSONTransformSelectionScope? + do { + _ = try JSONTransformProgram.compile("0", variables: ["values"]).withSelection(population: [42]) { s in + failed = s; s.cancel() + do { _ = try s.next(); preconditionFailure("Cancel ignored") } + catch let e as JSONTransformFailure { precondition(e.category == .cancelled) } + } + preconditionFailure("Cancelled invocation certified") + } catch let e as JSONTransformFailure { precondition(e.category == .cancelled) } + let terminal = try failed!.report(); precondition(terminal[.retainedBytes] == 0) + witnesses.append("cancellation") + precondition(witnesses.count == 10) + print("TRANSFORM_SELECTION passed witnesses=10") + } +} From c9a5a4cd0c71a6c14fd743622d62026798b4b79d Mon Sep 17 00:00:00 2001 From: Thorsten Fuchs <76904008+ThorFuchs@users.noreply.github.com> Date: Thu, 29 Oct 2026 20:03:05 +0100 Subject: [PATCH 5/5] Run composition in dedicated CI execution lanes --- .github/workflows/linux.yml | 25 ++++++-- .github/workflows/native.yml | 26 ++++++-- Documentation/Qualification/MaintenanceCI.md | 21 +++++++ Scripts/ci.py | 2 +- Scripts/test_qualification_harness.py | 64 +++++++++++++++++++- 5 files changed, 124 insertions(+), 14 deletions(-) diff --git a/.github/workflows/linux.yml b/.github/workflows/linux.yml index 4f286e5..9a2349c 100644 --- a/.github/workflows/linux.yml +++ b/.github/workflows/linux.yml @@ -11,13 +11,13 @@ concurrency: cancel-in-progress: true jobs: qualification: - name: Linux / ${{ matrix.architecture }} / ${{ matrix.lane }} + name: Linux execution / ${{ matrix.architecture }} / ${{ matrix.lane }} runs-on: ${{ matrix.runner }} timeout-minutes: 60 strategy: fail-fast: false matrix: - lane: [linux, linux-safety] + lane: [linux, linux-safety, composition] architecture: [x86_64, arm64] include: - architecture: x86_64 @@ -47,9 +47,6 @@ jobs: git config --global --add safe.directory "$GITHUB_WORKSPACE" test "$(git rev-parse HEAD)" = "$GITHUB_SHA" - run: python3 -B Scripts/ci.py --lane '${{ matrix.lane }}' - - name: Public composition and reference imports - if: matrix.lane == 'linux' - run: python3 -B Scripts/ci.py --lane composition - name: Preserve Linux receipts if: always() uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6 @@ -60,3 +57,21 @@ jobs: path: | .build/ci/ !.build/ci/**/reproduction/extracted/** + required: + # Both architectures and all three lanes must succeed. Preserve the four + # existing context names without converting an incomplete matrix to green. + name: Linux / ${{ matrix.architecture }} / ${{ matrix.lane }} + needs: qualification + if: ${{ always() }} + runs-on: ubuntu-latest + timeout-minutes: 5 + strategy: + fail-fast: false + matrix: + lane: [linux, linux-safety] + architecture: [x86_64, arm64] + steps: + - name: Require complete qualification + env: + QUALIFICATION_RESULT: ${{ needs.qualification.result }} + run: test "$QUALIFICATION_RESULT" = success diff --git a/.github/workflows/native.yml b/.github/workflows/native.yml index 2f1e707..efed818 100644 --- a/.github/workflows/native.yml +++ b/.github/workflows/native.yml @@ -11,13 +11,13 @@ concurrency: cancel-in-progress: true jobs: qualification: - name: Native / ${{ matrix.lane }} + name: Native execution / ${{ matrix.lane }} runs-on: xcode-27 timeout-minutes: 60 strategy: fail-fast: false matrix: - lane: [native, safety, docs, archive] + lane: [native, safety, docs, archive, composition] env: DEVELOPER_DIR: /Applications/Xcode_27.2.app/Contents/Developer PYTHONDONTWRITEBYTECODE: '1' @@ -34,9 +34,6 @@ jobs: - run: python3 -B Scripts/ci.py --lane '${{ matrix.lane }}' env: SWIFT_JSON_RELEASE_METADATA: ${{ github.workspace }}/.build/handoff/pages/latest-release.json - - name: Public composition and reference imports - if: matrix.lane == 'native' - run: python3 -B Scripts/ci.py --lane composition - name: Preserve native receipts if: always() uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6 @@ -63,10 +60,27 @@ jobs: with: path: .build/handoff/pages/site include-hidden-files: true # Preserve the verified .nojekyll file; only the generated site is uploaded. + required: + # Preserve protected context names. Every execution lane, including + # composition, must succeed; a failed/cancelled/skipped dependency fails here. + name: Native / ${{ matrix.lane }} + needs: qualification + if: ${{ always() }} + runs-on: ubuntu-latest + timeout-minutes: 5 + strategy: + fail-fast: false + matrix: + lane: [native, safety, docs, archive] + steps: + - name: Require complete qualification + env: + QUALIFICATION_RESULT: ${{ needs.qualification.result }} + run: test "$QUALIFICATION_RESULT" = success publish: name: Documentation / publish if: github.ref == 'refs/heads/main' && github.event_name != 'pull_request' - needs: qualification + needs: [qualification, required] # Native compilation and all qualification happened in the jobs above. # This job only publishes their exact static artifact. runs-on: ubuntu-latest diff --git a/Documentation/Qualification/MaintenanceCI.md b/Documentation/Qualification/MaintenanceCI.md index 0e4d8fc..9058f8c 100644 --- a/Documentation/Qualification/MaintenanceCI.md +++ b/Documentation/Qualification/MaintenanceCI.md @@ -19,6 +19,27 @@ keeps the existing main-only condition; maintenance pushes must not replace the development website. A skipped Pages publication on a maintenance branch is expected. Required product qualification must still execute and pass. +Composition has a separate execution matrix lane on macOS and on each Linux +architecture. The native, safety, docs, archive and composition commands retain +their existing 60-minute execution limits; no command or fixture is omitted. +The existing `Native / ...` and `Linux / ...` check names are final gates over +their complete workflow execution matrix. They run even after a dependency fails +and accept only its `success` result. Failure, cancellation, a skipped execution +or a missing result cannot produce a successful required context. Consequently, +one failed execution makes all of that workflow's final gates fail. Inspect the +`Native execution / ...` or `Linux execution / ...` job and its receipts for the +specific cause. Pages also waits for the final native gates, with its main-only +condition unchanged. No repository protection setting is changed by scheduling. + +This separation follows the 1.8 PR74 execution at `48dca4d`: the normal native and +both Linux lanes passed, but the native and x86 Linux jobs reached their one-hour +job limits during the appended composition step. Native's nine-command composition +receipt had completed just before cancellation; x86 had completed only five of +nine commands. Preserve those cancelled checks and partial receipts. They are not +retroactively green, and the successor must qualify its actual source. Moving +the unchanged composition commands to a separate job does not waive them or +extend a timeout. It does not qualify held runtime/performance/allocation gates. + If an expected automatic run is absent, inspect the branch/event/commit and workflow filters. Existing manual dispatch may qualify the exact source while investigation proceeds, but it does not prove that automatic triggering works. diff --git a/Scripts/ci.py b/Scripts/ci.py index bea99b3..542f298 100644 --- a/Scripts/ci.py +++ b/Scripts/ci.py @@ -533,7 +533,7 @@ def check_result(command, code, log, events=None): if command[:3] == ['python3', '-B', 'Scripts/qualify_html_browser.py']: passed = passed and 'JSONHTML_BROWSER passed engines=3 cases=76 skips=0' in log.splitlines() if 'Scripts/test_qualification_harness.py' in command: - passed = passed and 'HARNESS_TESTS passed cases=32' in log.splitlines() + passed = passed and 'HARNESS_TESTS passed cases=35' in log.splitlines() if command[:3] == ['python3', '-B', 'Scripts/qualify_transform_selection.py']: data = json.loads(log.strip().splitlines()[-1]) passed = passed and data['status'] == 'passed' and len(data['commands']) == 20 and all(x['passed'] for x in data['commands']) and data['witnesses_per_configuration'] == 10 and data['negative_fixture_invocations'] == 14 and data['compiled_guide_fences'] == 1 diff --git a/Scripts/test_qualification_harness.py b/Scripts/test_qualification_harness.py index feaa36b..2a71616 100644 --- a/Scripts/test_qualification_harness.py +++ b/Scripts/test_qualification_harness.py @@ -4,6 +4,8 @@ import copy import json import pathlib +import os +import re import subprocess from qualify_release_1_1 import checkout_head from generate_cli_identity import rendered @@ -28,6 +30,64 @@ def store_markers(): 'STORE_RUNTIME R%02d passed' % n for n in range(1,11)]) class HarnessTests(unittest.TestCase): + def workflow_jobs(self, filename): + text = (pathlib.Path(__file__).resolve().parents[1]/'.github/workflows'/filename).read_text() + # Inspect this repository's authored job blocks, not a general YAML parser. + jobs = dict(re.findall(r'^ (\w+):\n(.*?)(?=^ \w+:\n|\Z)', text.split('jobs:\n',1)[1], re.M|re.S)) + return text, jobs + + def test_required_workflow_contexts_wait_for_complete_execution_matrices(self): + for filename, execution, required, name in [ + ('native.yml','[native, safety, docs, archive, composition]','[native, safety, docs, archive]','Native / ${{ matrix.lane }}'), + ('linux.yml','[linux, linux-safety, composition]','[linux, linux-safety]','Linux / ${{ matrix.architecture }} / ${{ matrix.lane }}'), + ]: + with self.subTest(workflow=filename): + _, jobs = self.workflow_jobs(filename) + self.assertIn('lane: '+execution, jobs['qualification']) + self.assertIn("run: python3 -B Scripts/ci.py --lane '${{ matrix.lane }}'", jobs['qualification']) + self.assertNotIn("if: matrix.lane == 'composition'", jobs['qualification']) + self.assertNotIn('--lane composition', jobs['qualification']) + gate = jobs['required'] + self.assertIn('name: '+name+'\n', gate) + self.assertIn('needs: qualification\n', gate) + self.assertIn('if: ${{ always() }}\n', gate) + self.assertIn('lane: '+required, gate) + if filename=='linux.yml': + for block in [jobs['qualification'], gate]: + self.assertIn('architecture: [x86_64, arm64]', block) + + def test_workflow_gates_reject_failed_cancelled_skipped_and_missing_results(self): + for filename in ['native.yml','linux.yml']: + text, jobs = self.workflow_jobs(filename) + self.assertNotIn('continue-on-error', text) + gate = jobs['required'] + self.assertIn('QUALIFICATION_RESULT: ${{ needs.qualification.result }}', gate) + script = re.findall(r'^ run: (.+)$', gate, re.M) + self.assertEqual(script, ['test "$QUALIFICATION_RESULT" = success']) + for result in ['success','failure','cancelled','skipped','', 'Success','success failure']: + run = subprocess.run(['bash','-c',script[0]],env=dict(os.environ,QUALIFICATION_RESULT=result),capture_output=True) + self.assertEqual(run.returncode==0, result=='success', (filename,result)) + env = dict(os.environ); env.pop('QUALIFICATION_RESULT',None) + self.assertNotEqual(subprocess.run(['bash','-c',script[0]],env=env,capture_output=True).returncode,0) + + def test_composition_scheduling_preserves_timeouts_pins_artifacts_and_pages_guard(self): + for filename in ['native.yml','linux.yml']: + text, jobs = self.workflow_jobs(filename) + self.assertIn('timeout-minutes: 60', jobs['qualification']) + self.assertIn('fail-fast: false', jobs['qualification']) + self.assertIn('persist-credentials: false', jobs['qualification']) + self.assertIn('retention-days: 30', jobs['qualification']) + self.assertIn('if-no-files-found: error', jobs['qualification']) + self.assertIn('actions/checkout@d23441a48e516b6c34aea4fa41551a30e30af803', text) + self.assertIn('actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f', text) + self.assertIn("branches: [main, 'release/1.6']", text) + _, native = self.workflow_jobs('native.yml') + self.assertIn("if: github.ref == 'refs/heads/main' && github.event_name != 'pull_request'", native['publish']) + self.assertIn('needs: [qualification, required]', native['publish']) + _, linux = self.workflow_jobs('linux.yml') + self.assertNotIn('publish', linux) + self.assertIn('swift@sha256:31d14d727f4451f25e29bac9c42563a098c39ac66415e8bad970f48038e38aca', linux['qualification']) + def test_selection_filter_preserves_exact_identity_and_completion_evidence(self): ids = [n for n in DISCOVERY['transform_function_ids'] if '.selection' in n] self.assertEqual(len(ids), 16) @@ -465,8 +525,8 @@ def test_exact_discovery_rejects_missing_duplicate_and_changed_parameters(self): identity['function_names'][0] = 'JSONTests.unrelatedReplacement()' self.assertFalse(check_restored(identity)) harness = ['python3', '-B', 'Scripts/test_qualification_harness.py'] - self.assertTrue(check_result(harness, 0, 'HARNESS_TESTS passed cases=32')[0]) - for stale in ['HARNESS_TESTS passed cases=31', 'HARNESS_TESTS passed cases=30', 'HARNESS_TESTS passed cases=10', 'HARNESS_TESTS passed cases=11', 'HARNESS_TESTS passed cases=12', 'HARNESS_TESTS passed cases=13', 'HARNESS_TESTS passed cases=14', 'HARNESS_TESTS passed cases=15', 'HARNESS_TESTS passed cases=18', 'HARNESS_TESTS passed cases=19', 'HARNESS_TESTS passed cases=20', 'HARNESS_TESTS passed cases=21', 'HARNESS_TESTS passed cases=25', 'HARNESS_TESTS passed cases=26', 'HARNESS_TESTS passed cases=28', '']: + self.assertTrue(check_result(harness, 0, 'HARNESS_TESTS passed cases=35')[0]) + for stale in ['HARNESS_TESTS passed cases=32', 'HARNESS_TESTS passed cases=31', 'HARNESS_TESTS passed cases=30', 'HARNESS_TESTS passed cases=10', 'HARNESS_TESTS passed cases=11', 'HARNESS_TESTS passed cases=12', 'HARNESS_TESTS passed cases=13', 'HARNESS_TESTS passed cases=14', 'HARNESS_TESTS passed cases=15', 'HARNESS_TESTS passed cases=18', 'HARNESS_TESTS passed cases=19', 'HARNESS_TESTS passed cases=20', 'HARNESS_TESTS passed cases=21', 'HARNESS_TESTS passed cases=25', 'HARNESS_TESTS passed cases=26', 'HARNESS_TESTS passed cases=28', '']: self.assertFalse(check_result(harness, 0, stale)[0]) lifetime = ['python3', '-B', 'Scripts/qualify_schema_example_lifetime.py'] marker = 'SCHEMA_EXAMPLES_CORPUS passed mode=owned scopes=128 values=6 exact=1'