diff --git a/docs/source/basic_concepts.rst b/docs/source/basic_concepts.rst index 0af523056..42e62924f 100644 --- a/docs/source/basic_concepts.rst +++ b/docs/source/basic_concepts.rst @@ -23,8 +23,9 @@ Basic Concepts File Layouts ------------------------ -All files of a table are stored under one base directory. Paimon files are -organized in a layered style. The following image illustrates the file layout. +In a file-system deployment, table metadata is stored under the table's base +directory. Paimon files are organized in a layered style. The following image +illustrates the file layout. Starting from a snapshot file, Paimon readers can recursively access all records from the table. @@ -35,7 +36,10 @@ from the table. Snapshot ------------------- -All snapshot files are stored in the snapshot directory. +Direct file-system commits publish snapshot files in the snapshot directory. +A catalog that manages table versions may instead hold the latest snapshot +without publishing it there. See :doc:`user_guide/catalog` for the file-system +metadata still required by reads, recovery and maintenance with a catalog. A snapshot file is a JSON file containing information about this snapshot, including the schema file in use the manifest list containing all changes of diff --git a/docs/source/build_system.rst b/docs/source/build_system.rst index 44495f73c..3b168fdfc 100644 --- a/docs/source/build_system.rst +++ b/docs/source/build_system.rst @@ -48,6 +48,9 @@ for data format and file system. project(MyExample) + set(CMAKE_CXX_STANDARD 17) + set(CMAKE_CXX_STANDARD_REQUIRED ON) + # Arrow's static package may export LZ4 under a different target name. find_package(lz4 CONFIG QUIET) if(TARGET LZ4::lz4_static AND NOT TARGET LZ4::lz4) diff --git a/docs/source/user_guide/append_only_table.rst b/docs/source/user_guide/append_only_table.rst index e1320ede8..4ed6061b5 100644 --- a/docs/source/user_guide/append_only_table.rst +++ b/docs/source/user_guide/append_only_table.rst @@ -20,7 +20,13 @@ Append Only Table ================= -If a table does not have a primary key defined, it is an append table. Compared -to the primary key table, it does not have the ability to directly receive changelogs. -It cannot be directly updated with data through upsert. It can only receive -incoming data from append data. +If a table does not have a primary key defined, it is an append table. +Ordinary append writes accept ``INSERT`` rows only and do not apply upserts +or deletes by key. + +The experimental real-time append deduplicate mode is an exception. With +``realtime.deduplicate-key-fields`` configured, it also accepts ``UPDATE_AFTER``, +``DELETE`` and ``UPDATE_BEFORE``, using deletion vectors to hide superseded +rows. The table still has no primary key; deduplication is scoped to each +partition-bucket. See :doc:`realtime` for the required schema, options and +shared context. diff --git a/docs/source/user_guide/clean.rst b/docs/source/user_guide/clean.rst index d0df8186f..5715c4b84 100644 --- a/docs/source/user_guide/clean.rst +++ b/docs/source/user_guide/clean.rst @@ -19,7 +19,9 @@ Data Cleanup =================================== .. note:: - The entire data cleanup feature supports only append tables, and cleanup of index manifests, changelog, statistics are not supported. Do not use this feature with primary key tables. + Cleanup supports append and primary-key tables, subject to the restrictions + below. Index manifests and index payloads are included. Cleanup of changelog + and statistics files is not supported. This document describes three cleanup capabilities: @@ -35,9 +37,15 @@ Orphan File Cleanup Description ~~~~~~~~~~~~~~~~~~~ -- Orphan file cleanup currently supports only append tables. +- Orphan file cleanup supports append and primary-key tables. - It runs as an independent task: construct a ``CleanContext`` and launch an ``OrphanFilesCleaner``. +- It covers table-local index manifests and payloads as well as data files, + preserving files referenced by retained snapshots. It does not enumerate a + potentially shared ``global-index.external-path``. +- Tables with tags or branches, or snapshots with a changelog manifest list, + are rejected with ``NotImplemented``. Tag retention during snapshot expiration + is supported separately, as described below. Detailed Steps ~~~~~~~~~~~~~~ @@ -62,7 +70,7 @@ Performance Considerations .. admonition:: TODO :class: tip - - Support cleanup of index manifests, changelog, statistics, etc. + - Support cleanup of changelog and statistics files. .. _expiring-partitions: @@ -109,10 +117,17 @@ Description - ``snapshot.time-retained``: The maximum age of completed snapshots to retain. Default: 1 hour. - ``snapshot.expire.limit``: The maximum number of snapshots allowed - to expire at a time. Default: 10. + to expire at a time. Default: 50. + - ``snapshot.clean-empty-directories``: Remove empty bucket and partition + directories after deleting their data files, without deleting the table + directory. Default: false. - The snapshot expiration interface deletes data files according to the expiration policy and returns the number of snapshots deleted. +- Data files, index manifests and index payloads referenced by retained + snapshots or current-branch tags are preserved. Retired index payloads are + deleted before their index manifests, including payloads stored under + ``global-index.external-path``. - Expiration on a branch other than main, or on a table with such a branch under ``branch/branch-``, is not supported: branches share the table's data files, so expiration returns ``NotImplemented`` before deleting anything. @@ -149,7 +164,7 @@ Detailed Steps * ``DELETE`` then ``ADD`` → the initial ``DELETE`` does not apply (file did not exist yet); the subsequent ``ADD`` ensures the file remains. 5. Clean meta files: - - Preserve manifests used by the last snapshot in the cleanup range (``end_exclusive_id``). + - Preserve manifests used by retained snapshots and current-branch tags. - Delete manifest files used by snapshots from ``begin_inclusive_id`` to ``end_exclusive_id`` (exclusive) and delete the snapshot files themselves. 6. Rewrite ``EarliestHint`` to ``end_exclusive_id``. 7. Return the number of snapshots deleted, i.e., ``end_exclusive_id - begin_inclusive_id``. @@ -164,6 +179,4 @@ Performance Considerations .. admonition:: TODO :class: tip - - Preserve tag (savepoint) data via ``tagManager``. - Delete changelog files. - - Remove empty directories. diff --git a/docs/source/user_guide/commit.rst b/docs/source/user_guide/commit.rst index 7e7ee1715..121462cb9 100644 --- a/docs/source/user_guide/commit.rst +++ b/docs/source/user_guide/commit.rst @@ -35,7 +35,7 @@ The Commit process consists of the following steps: 2. Compact (merge) Manifest files 3. Generate the Base Manifest List 4. Generate new Manifest files and the Delta Manifest List -5. Generate the Snapshot and HINT file +5. Prepare and publish the Snapshot Detailed Process ------------------- @@ -110,17 +110,34 @@ The initially collected file change information is written into new manifest fil Metadata for these newly created manifest files is then written into the delta manifest list. -Generate the Snapshot and HINT File -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Prepare and Publish the Snapshot +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ After the above steps are completed, the system generates a new Snapshot and performs the following operations: 1. Determine the new Snapshot ID based on the latest ``SnapshotId + 1``. 2. Record metadata such as ``schema id``, ``commit time``, and ``total record count``. -3. Atomicity guarantee: Generating a Snapshot is an atomic operation. If an exception - (e.g., an I/O error) occurs during the process, the manifest files and manifest list files - generated in Steps 2–4 are cleaned up and removed. -4. The Snapshot is written via a rename operation to ensure atomicity. -5. After the Snapshot is successfully written, the system writes the ``LATEST`` hint file - to reduce list operations when fetching the latest Snapshot. +3. If preparation fails before publication, the newly generated manifest files and + manifest list files are cleaned up. If publication returns an error and its + outcome is uncertain, the metadata files are retained because the snapshot may + already have been accepted. + +Snapshot publication depends on the commit mode: + +* **Direct file-system commit:** On non-object-store paths, the Snapshot is + published through an atomic rename. After publication, the committer writes + the ``LATEST`` hint file to reduce list operations when fetching the latest + Snapshot. +* **Catalog-mediated commit:** When ``CommitContextBuilder::WithCatalog()`` + supplies a catalog that manages table versions, the committer sends the + Snapshot and change statistics to that catalog for publication. This path + does not rename a local Snapshot file or write a local ``LATEST`` hint. +* **REST request preparation:** ``UseRESTCatalogCommit(true)`` builds a commit + request without publishing the Snapshot. The caller retrieves the JSON with + ``GetLastCommitTableRequest()`` and sends it to the REST catalog. A successful + local ``Commit()`` in this mode means the request was prepared, not that the + catalog accepted it. + +Object-store paths require one of the catalog modes. See :doc:`write` for +configuration and :doc:`catalog` for catalog publication and recovery semantics. diff --git a/docs/source/user_guide/manifest.rst b/docs/source/user_guide/manifest.rst index 4c1240da1..937c1c966 100644 --- a/docs/source/user_guide/manifest.rst +++ b/docs/source/user_guide/manifest.rst @@ -68,8 +68,8 @@ Data file metadata: 1. ``_FILE_NAME``: STRING — file name. 2. ``_FILE_SIZE``: BIGINT — file size. 3. ``_ROW_COUNT``: BIGINT — total number of rows (including add & delete) in this file. -4. ``_MIN_KEY``: STRING — minimum key of this file. -5. ``_MAX_KEY``: STRING — maximum key of this file. +4. ``_MIN_KEY``: BYTES — minimum key of this file, encoded as a BinaryRow. +5. ``_MAX_KEY``: BYTES — maximum key of this file, encoded as a BinaryRow. 6. ``_KEY_STATS``: SimpleStats — statistics of the key. 7. ``_VALUE_STATS``: SimpleStats — statistics of the value. 8. ``_MIN_SEQUENCE_NUMBER``: BIGINT — minimum sequence number. @@ -83,6 +83,11 @@ Data file metadata: 16. ``_FILE_SOURCE``: TINYINT — indicates whether this file is generated as an ``APPEND`` or ``COMPACT`` file. 17. ``_VALUE_STATS_COLS``: ARRAY — statistical columns in metadata. 18. ``_EXTERNAL_PATH``: STRING — external path of this file; ``null`` if it is in the warehouse. +19. ``_FIRST_ROW_ID``: BIGINT — first row id of this file for row tracking; nullable. +20. ``_WRITE_COLS``: ARRAY — columns written in this file for data evolution; nullable. +21. ``_WRITE_COLS_SEQUENCES``: ARRAY — maximum sequence number per physical table field, + in the order selected by ``_WRITE_COLS`` (excluding system fields), or the file schema order + when ``_WRITE_COLS`` is null. Nullable; when absent, only the file-level sequence range is available. Index Manifest -------------- @@ -99,7 +104,7 @@ Schema: 1. ``_KIND``: TINYINT — ``ADD`` or ``DELETE``. 2. ``_PARTITION``: BYTES — partition spec, a BinaryRow. 3. ``_BUCKET``: INT — bucket of this file. -4. ``_INDEX_TYPE``: STRING — ``HASH`` or ``DELETION_VECTORS``. +4. ``_INDEX_TYPE``: STRING — index type identifier, including global index types such as ``btree``. 5. ``_FILE_NAME``: STRING — file name. 6. ``_FILE_SIZE``: BIGINT — file size. 7. ``_ROW_COUNT``: BIGINT — total number of rows. @@ -108,6 +113,9 @@ Schema: - ``f1``: the starting offset of this deletion vector in the index file. - ``f2``: the length of this deletion vector in the index file. - ``_CARDINALITY``: the number of deleted rows. +9. ``_EXTERNAL_PATH``: STRING — external path of the index file; nullable. +10. ``_GLOBAL_INDEX``: global index metadata, including the row range, indexed field id, + optional extra field ids, index metadata and optional source metadata; nullable. Appendix -------- diff --git a/docs/source/user_guide/snapshot.rst b/docs/source/user_guide/snapshot.rst index 278e72af4..98c643038 100644 --- a/docs/source/user_guide/snapshot.rst +++ b/docs/source/user_guide/snapshot.rst @@ -21,10 +21,17 @@ Snapshot ======== -Each commit generates a snapshot file, and the version of the snapshot file starts from 1 and must be continuous. +Direct file-system commits publish snapshot files with consecutive snapshot ids +starting from 1. The layout and hint files below describe those published files. ``EARLIEST`` and ``LATEST`` are hint files at the beginning and end of the snapshot list, and they can be inaccurate. When hint files are inaccurate, the reader will scan all snapshot files to determine the beginning and end. +A catalog that manages table versions can hold the latest snapshot without a +snapshot file under the table path. Publication then depends on catalog +acceptance. ``UseRESTCatalogCommit(true)`` only prepares the request for the +caller to send; it does not publish a snapshot. See :doc:`commit` and +:doc:`catalog` for these modes and their historical metadata requirements. + Directory Layout ---------------- @@ -40,8 +47,8 @@ Directory Layout ├── snapshot-2 └── snapshot-3 -Writing commit will preempt the next snapshot id, and once the snapshot file is successfully written, this commit will -become visible. +A direct file-system commit attempts to publish the next snapshot id. Once the +snapshot file is successfully published, the commit becomes visible. Snapshot File ------------- @@ -53,16 +60,23 @@ Snapshot file is JSON and includes: 3. ``id``: Snapshot id, same as the file name. 4. ``schemaId``: The corresponding schema version for this commit. 5. ``baseManifestList``: A manifest list recording all changes from the previous snapshots. -6. ``deltaManifestList``: A manifest list recording all new changes occurred in this snapshot. -7. ``changelogManifestList``: A manifest list recording all changelog produced in this snapshot; ``null`` if no changelog is produced. -8. ``indexManifest``: A manifest recording all index files of this table; ``null`` if no table index file exists. -9. ``commitUser``: Usually generated by UUID; used for recovery of streaming writes—one stream write job with one user. -10. ``commitIdentifier``: Transaction id corresponding to streaming write; each transaction may result in multiple commits for different ``commitKind`` values. -11. ``commitKind``: Type of changes in this snapshot, including ``append``, ``compact``, ``overwrite`` and ``analyze``. -12. ``timeMillis``: Commit time in milliseconds. -13. ``logOffsets``: Commit log offsets. -14. ``totalRecordCount``: Record count of all changes occurred in this snapshot. -15. ``deltaRecordCount``: Record count of all new changes occurred in this snapshot. -16. ``changelogRecordCount``: Record count of all changelog produced in this snapshot. -17. ``watermark``: Watermark for input records, from Flink watermark mechanism; ``Long.MIN_VALUE`` if there is no watermark. -18. ``statistics``: Stats file name for statistics of this table. +6. ``baseManifestListSize``: Size in bytes of the base manifest list; optional. +7. ``deltaManifestList``: A manifest list recording all new changes occurred in this snapshot. +8. ``deltaManifestListSize``: Size in bytes of the delta manifest list; optional. +9. ``changelogManifestList``: A manifest list recording all changelog produced in this snapshot; omitted by C++ serialization if unset. +10. ``changelogManifestListSize``: Size in bytes of the changelog manifest list; optional. +11. ``indexManifest``: A manifest recording all index files of this table; omitted by C++ serialization if unset. +12. ``commitUser``: Usually generated by UUID; used for recovery of streaming writes—one stream write job with one user. +13. ``commitIdentifier``: Transaction id corresponding to streaming write; each transaction may result in multiple commits for different ``commitKind`` values. +14. ``commitKind``: Type of changes in this snapshot, including ``append``, ``compact``, ``overwrite`` and ``analyze``. +15. ``timeMillis``: Commit time in milliseconds. +16. ``totalRecordCount``: Record count of all changes occurred in this snapshot. +17. ``deltaRecordCount``: Record count of all new changes occurred in this snapshot. +18. ``changelogRecordCount``: Record count of all changelog produced in this snapshot; optional. +19. ``watermark``: Watermark for input records. When unset, C++ serialization omits this field. + A commit carries forward the previous snapshot's watermark if no new watermark is supplied. +20. ``statistics``: Stats file name for statistics of this table; optional. +21. ``properties``: Optional string-to-string map of snapshot properties, including real-time progress metadata when enabled. +22. ``nextRowId``: Next row id for row tracking; optional. + +The C++ snapshot serializer does not emit ``logOffsets``. diff --git a/docs/source/user_guide/system_tables.rst b/docs/source/user_guide/system_tables.rst index 843640eef..7df0311c9 100644 --- a/docs/source/user_guide/system_tables.rst +++ b/docs/source/user_guide/system_tables.rst @@ -39,12 +39,17 @@ case-insensitive, and an unrecognized suffix fails with * ``$tags``: tags and the snapshots they reference. * ``$branches``: table branches, including ``main``. * ``$consumers``: persisted streaming consumer offsets. -* ``$manifests``: data manifest metadata in the latest snapshot. -* ``$files``: data file metadata in the latest snapshot. +* ``$manifests``: data manifest metadata in the selected snapshot. +* ``$files``: data file metadata in the selected snapshot. * ``$audit_log``: row-level changelog records with rowkind. * ``$binlog``: changelog records with list-wrapped data columns. * ``$ro``: read-optimized data, described in the next section. +``$manifests`` and ``$files`` read the latest snapshot by default. Set +``scan.snapshot-id`` to read metadata from a specific snapshot of the selected +branch. If that snapshot does not exist, the read fails with +``snapshot does not exist`` instead of falling back to the latest snapshot. + Global System Tables -------------------- diff --git a/docs/source/user_guide/write.rst b/docs/source/user_guide/write.rst index 9d14c2414..97c4dd30b 100644 --- a/docs/source/user_guide/write.rst +++ b/docs/source/user_guide/write.rst @@ -89,8 +89,12 @@ whose values are ``LargeBinary``. Build the BLOB field with PAIMON_ASSIGN_OR_RAISE(std::unique_ptr<::ArrowSchema> c_element, paimon::Blob::ArrowField("element", /*nullable=*/true)); - PAIMON_ASSIGN_OR_RAISE_FROM_ARROW(std::shared_ptr element, - arrow::ImportField(c_element.get())); + arrow::Result> element_result = + arrow::ImportField(c_element.get()); + if (!element_result.ok()) { + return paimon::Status::Invalid(element_result.status().ToString()); + } + std::shared_ptr element = element_result.ValueUnsafe(); std::shared_ptr schema = arrow::schema( {arrow::field("id", arrow::int32()), arrow::field("frames", arrow::list(element)), arrow::field("views", arrow::map(arrow::utf8(), element))}); diff --git a/third_party/roaring_bitmap/roaring.cpp b/third_party/roaring_bitmap/roaring.cpp index 2c704def0..03a32bef4 100644 --- a/third_party/roaring_bitmap/roaring.cpp +++ b/third_party/roaring_bitmap/roaring.cpp @@ -1,3 +1,5 @@ +// Modified by Apache Paimon to use the paimon::roaring namespace. + // !!! DO NOT EDIT - THIS IS AN AUTO-GENERATED FILE !!! // Created by amalgamation.sh on 2023-12-16T02:52:00Z diff --git a/third_party/roaring_bitmap/roaring.h b/third_party/roaring_bitmap/roaring.h index 2a9bf3b28..9bec9874c 100644 --- a/third_party/roaring_bitmap/roaring.h +++ b/third_party/roaring_bitmap/roaring.h @@ -1,3 +1,5 @@ +// Modified by Apache Paimon to use the paimon::roaring namespace. + // !!! DO NOT EDIT - THIS IS AN AUTO-GENERATED FILE !!! // Created by amalgamation.sh on 2023-12-16T02:52:00Z diff --git a/third_party/roaring_bitmap/roaring.hh b/third_party/roaring_bitmap/roaring.hh index 4e829cd32..4ddedec95 100644 --- a/third_party/roaring_bitmap/roaring.hh +++ b/third_party/roaring_bitmap/roaring.hh @@ -1,3 +1,5 @@ +// Modified by Apache Paimon to use the paimon::roaring namespace. + // !!! DO NOT EDIT - THIS IS AN AUTO-GENERATED FILE !!! // Created by amalgamation.sh on 2023-12-16T02:52:00Z