lxy-9602 · Nov 6, 2026 · Nov 6, 2026 · Nov 6, 2026
diff --git a/docs/source/basic_concepts.rst b/docs/source/basic_concepts.rst
@@ -23,8 +23,9 @@ Basic Concepts
 
 File Layouts
 ------------------------
-All files of a table are stored under one base directory. Paimon files are
-organized in a layered style. The following image illustrates the file layout.
+In a file-system deployment, table metadata is stored under the table's base
+directory. Paimon files are organized in a layered style. The following image
+illustrates the file layout.
 Starting from a snapshot file, Paimon readers can recursively access all records
 from the table.
 
@@ -35,7 +36,10 @@ from the table.
 
 Snapshot
 -------------------
-All snapshot files are stored in the snapshot directory.
+Direct file-system commits publish snapshot files in the snapshot directory.
+A catalog that manages table versions may instead hold the latest snapshot
+without publishing it there. See :doc:`user_guide/catalog` for the file-system
+metadata still required by reads, recovery and maintenance with a catalog.
 
 A snapshot file is a JSON file containing information about this snapshot,
 including the schema file in use the manifest list containing all changes of

diff --git a/docs/source/build_system.rst b/docs/source/build_system.rst
@@ -48,6 +48,9 @@ for data format and file system.
 
    project(MyExample)
 
+   set(CMAKE_CXX_STANDARD 17)
+   set(CMAKE_CXX_STANDARD_REQUIRED ON)
+
    # Arrow's static package may export LZ4 under a different target name.
    find_package(lz4 CONFIG QUIET)
    if(TARGET LZ4::lz4_static AND NOT TARGET LZ4::lz4)

diff --git a/docs/source/user_guide/append_only_table.rst b/docs/source/user_guide/append_only_table.rst
@@ -20,7 +20,13 @@
 
 Append Only Table
 =================
-If a table does not have a primary key defined, it is an append table. Compared
-to the primary key table, it does not have the ability to directly receive changelogs.
-It cannot be directly updated with data through upsert. It can only receive
-incoming data from append data.
+If a table does not have a primary key defined, it is an append table.
+Ordinary append writes accept ``INSERT`` rows only and do not apply upserts
+or deletes by key.
+
+The experimental real-time append deduplicate mode is an exception. With
+``realtime.deduplicate-key-fields`` configured, it also accepts ``UPDATE_AFTER``,
+``DELETE`` and ``UPDATE_BEFORE``, using deletion vectors to hide superseded
+rows. The table still has no primary key; deduplication is scoped to each
+partition-bucket. See :doc:`realtime` for the required schema, options and
+shared context.
diff --git a/docs/source/user_guide/clean.rst b/docs/source/user_guide/clean.rst
@@ -19,7 +19,9 @@ Data Cleanup
 ===================================
 .. note::
 
-  The entire data cleanup feature supports only append tables, and cleanup of index manifests, changelog, statistics are not supported. Do not use this feature with primary key tables.
+  Cleanup supports append and primary-key tables, subject to the restrictions
+  below. Index manifests and index payloads are included. Cleanup of changelog
+  and statistics files is not supported.
 
 This document describes three cleanup capabilities:
 
@@ -35,9 +37,15 @@ Orphan File Cleanup
 Description
 ~~~~~~~~~~~~~~~~~~~
 
-- Orphan file cleanup currently supports only append tables.
+- Orphan file cleanup supports append and primary-key tables.
 - It runs as an independent task: construct a ``CleanContext`` and
   launch an ``OrphanFilesCleaner``.
+- It covers table-local index manifests and payloads as well as data files,
+  preserving files referenced by retained snapshots. It does not enumerate a
+  potentially shared ``global-index.external-path``.
+- Tables with tags or branches, or snapshots with a changelog manifest list,
+  are rejected with ``NotImplemented``. Tag retention during snapshot expiration
+  is supported separately, as described below.
 
 Detailed Steps
 ~~~~~~~~~~~~~~
@@ -62,7 +70,7 @@ Performance Considerations
 .. admonition:: TODO
   :class: tip
 
-  - Support cleanup of index manifests, changelog, statistics, etc.
+  - Support cleanup of changelog and statistics files.
 
 .. _expiring-partitions:
 
@@ -109,10 +117,17 @@ Description
   - ``snapshot.time-retained``: The maximum age of completed snapshots
     to retain. Default: 1 hour.
   - ``snapshot.expire.limit``: The maximum number of snapshots allowed
-    to expire at a time. Default: 10.
+    to expire at a time. Default: 50.
+  - ``snapshot.clean-empty-directories``: Remove empty bucket and partition
+    directories after deleting their data files, without deleting the table
+    directory. Default: false.
 
 - The snapshot expiration interface deletes data files according to the
   expiration policy and returns the number of snapshots deleted.
+- Data files, index manifests and index payloads referenced by retained
+  snapshots or current-branch tags are preserved. Retired index payloads are
+  deleted before their index manifests, including payloads stored under
+  ``global-index.external-path``.
 - Expiration on a branch other than main, or on a table with such a branch under
   ``branch/branch-<name>``, is not supported: branches share the table's data
   files, so expiration returns ``NotImplemented`` before deleting anything.
@@ -149,7 +164,7 @@ Detailed Steps
      * ``DELETE`` then ``ADD`` → the initial ``DELETE`` does not apply (file did not exist yet); the subsequent ``ADD`` ensures the file remains.
 
 5. Clean meta files:
-   - Preserve manifests used by the last snapshot in the cleanup range (``end_exclusive_id``).
+   - Preserve manifests used by retained snapshots and current-branch tags.
    - Delete manifest files used by snapshots from ``begin_inclusive_id`` to ``end_exclusive_id`` (exclusive) and delete the snapshot files themselves.
 6. Rewrite ``EarliestHint`` to ``end_exclusive_id``.
 7. Return the number of snapshots deleted, i.e., ``end_exclusive_id - begin_inclusive_id``.
@@ -164,6 +179,4 @@ Performance Considerations
 .. admonition:: TODO
   :class: tip
 
-  - Preserve tag (savepoint) data via ``tagManager``.
   - Delete changelog files.
-  - Remove empty directories.
diff --git a/docs/source/user_guide/commit.rst b/docs/source/user_guide/commit.rst
@@ -35,7 +35,7 @@ The Commit process consists of the following steps:
 2. Compact (merge) Manifest files
 3. Generate the Base Manifest List
 4. Generate new Manifest files and the Delta Manifest List
-5. Generate the Snapshot and HINT file
+5. Prepare and publish the Snapshot
 
 Detailed Process
 -------------------
@@ -110,17 +110,34 @@ The initially collected file change information is written into new manifest fil
 Metadata for these newly created manifest files is then written into the delta
 manifest list.
 
-Generate the Snapshot and HINT File
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+Prepare and Publish the Snapshot
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
 After the above steps are completed, the system generates a new Snapshot and performs
 the following operations:
 
 1. Determine the new Snapshot ID based on the latest ``SnapshotId + 1``.
 2. Record metadata such as ``schema id``, ``commit time``, and ``total record count``.
-3. Atomicity guarantee: Generating a Snapshot is an atomic operation. If an exception
-   (e.g., an I/O error) occurs during the process, the manifest files and manifest list files
-   generated in Steps 2–4 are cleaned up and removed.
-4. The Snapshot is written via a rename operation to ensure atomicity.
-5. After the Snapshot is successfully written, the system writes the ``LATEST`` hint file
-   to reduce list operations when fetching the latest Snapshot.
+3. If preparation fails before publication, the newly generated manifest files and
+   manifest list files are cleaned up. If publication returns an error and its
+   outcome is uncertain, the metadata files are retained because the snapshot may
+   already have been accepted.
+
+Snapshot publication depends on the commit mode:
+
+* **Direct file-system commit:** On non-object-store paths, the Snapshot is
+  published through an atomic rename. After publication, the committer writes
+  the ``LATEST`` hint file to reduce list operations when fetching the latest
+  Snapshot.
+* **Catalog-mediated commit:** When ``CommitContextBuilder::WithCatalog()``
+  supplies a catalog that manages table versions, the committer sends the
+  Snapshot and change statistics to that catalog for publication. This path
+  does not rename a local Snapshot file or write a local ``LATEST`` hint.
+* **REST request preparation:** ``UseRESTCatalogCommit(true)`` builds a commit
+  request without publishing the Snapshot. The caller retrieves the JSON with
+  ``GetLastCommitTableRequest()`` and sends it to the REST catalog. A successful
+  local ``Commit()`` in this mode means the request was prepared, not that the
+  catalog accepted it.
+
+Object-store paths require one of the catalog modes. See :doc:`write` for
+configuration and :doc:`catalog` for catalog publication and recovery semantics.
diff --git a/docs/source/user_guide/manifest.rst b/docs/source/user_guide/manifest.rst
@@ -68,8 +68,8 @@ Data file metadata:
 1. ``_FILE_NAME``: STRING — file name.
 2. ``_FILE_SIZE``: BIGINT — file size.
 3. ``_ROW_COUNT``: BIGINT — total number of rows (including add & delete) in this file.
-4. ``_MIN_KEY``: STRING — minimum key of this file.
-5. ``_MAX_KEY``: STRING — maximum key of this file.
+4. ``_MIN_KEY``: BYTES — minimum key of this file, encoded as a BinaryRow.
+5. ``_MAX_KEY``: BYTES — maximum key of this file, encoded as a BinaryRow.
 6. ``_KEY_STATS``: SimpleStats — statistics of the key.
 7. ``_VALUE_STATS``: SimpleStats — statistics of the value.
 8. ``_MIN_SEQUENCE_NUMBER``: BIGINT — minimum sequence number.
@@ -83,6 +83,11 @@ Data file metadata:
 16. ``_FILE_SOURCE``: TINYINT — indicates whether this file is generated as an ``APPEND`` or ``COMPACT`` file.
 17. ``_VALUE_STATS_COLS``: ARRAY<STRING> — statistical columns in metadata.
 18. ``_EXTERNAL_PATH``: STRING — external path of this file; ``null`` if it is in the warehouse.
+19. ``_FIRST_ROW_ID``: BIGINT — first row id of this file for row tracking; nullable.
+20. ``_WRITE_COLS``: ARRAY<STRING> — columns written in this file for data evolution; nullable.
+21. ``_WRITE_COLS_SEQUENCES``: ARRAY<BIGINT> — maximum sequence number per physical table field,
+    in the order selected by ``_WRITE_COLS`` (excluding system fields), or the file schema order
+    when ``_WRITE_COLS`` is null. Nullable; when absent, only the file-level sequence range is available.
 
 Index Manifest
 --------------
@@ -99,7 +104,7 @@ Schema:
 1. ``_KIND``: TINYINT — ``ADD`` or ``DELETE``.
 2. ``_PARTITION``: BYTES — partition spec, a BinaryRow.
 3. ``_BUCKET``: INT — bucket of this file.
-4. ``_INDEX_TYPE``: STRING — ``HASH`` or ``DELETION_VECTORS``.
+4. ``_INDEX_TYPE``: STRING — index type identifier, including global index types such as ``btree``.
 5. ``_FILE_NAME``: STRING — file name.
 6. ``_FILE_SIZE``: BIGINT — file size.
 7. ``_ROW_COUNT``: BIGINT — total number of rows.
@@ -108,6 +113,9 @@ Schema:
    - ``f1``: the starting offset of this deletion vector in the index file.
    - ``f2``: the length of this deletion vector in the index file.
    - ``_CARDINALITY``: the number of deleted rows.
+9. ``_EXTERNAL_PATH``: STRING — external path of the index file; nullable.
+10. ``_GLOBAL_INDEX``: global index metadata, including the row range, indexed field id,
+    optional extra field ids, index metadata and optional source metadata; nullable.
 
 Appendix
 --------

diff --git a/docs/source/user_guide/snapshot.rst b/docs/source/user_guide/snapshot.rst
@@ -21,10 +21,17 @@
 Snapshot
 ========
 
-Each commit generates a snapshot file, and the version of the snapshot file starts from 1 and must be continuous.
+Direct file-system commits publish snapshot files with consecutive snapshot ids
+starting from 1. The layout and hint files below describe those published files.
 ``EARLIEST`` and ``LATEST`` are hint files at the beginning and end of the snapshot list, and they can be inaccurate.
 When hint files are inaccurate, the reader will scan all snapshot files to determine the beginning and end.
 
+A catalog that manages table versions can hold the latest snapshot without a
+snapshot file under the table path. Publication then depends on catalog
+acceptance. ``UseRESTCatalogCommit(true)`` only prepares the request for the
+caller to send; it does not publish a snapshot. See :doc:`commit` and
+:doc:`catalog` for these modes and their historical metadata requirements.
+
 Directory Layout
 ----------------
 
@@ -40,8 +47,8 @@ Directory Layout
                ├── snapshot-2
                └── snapshot-3
 
-Writing commit will preempt the next snapshot id, and once the snapshot file is successfully written, this commit will
-become visible.
+A direct file-system commit attempts to publish the next snapshot id. Once the
+snapshot file is successfully published, the commit becomes visible.
 
 Snapshot File
 -------------
@@ -53,16 +60,23 @@ Snapshot file is JSON and includes:
 3. ``id``: Snapshot id, same as the file name.
 4. ``schemaId``: The corresponding schema version for this commit.
 5. ``baseManifestList``: A manifest list recording all changes from the previous snapshots.
-6. ``deltaManifestList``: A manifest list recording all new changes occurred in this snapshot.
-7. ``changelogManifestList``: A manifest list recording all changelog produced in this snapshot; ``null`` if no changelog is produced.
-8. ``indexManifest``: A manifest recording all index files of this table; ``null`` if no table index file exists.
-9. ``commitUser``: Usually generated by UUID; used for recovery of streaming writes—one stream write job with one user.
-10. ``commitIdentifier``: Transaction id corresponding to streaming write; each transaction may result in multiple commits for different ``commitKind`` values.
-11. ``commitKind``: Type of changes in this snapshot, including ``append``, ``compact``, ``overwrite`` and ``analyze``.
-12. ``timeMillis``: Commit time in milliseconds.
-13. ``logOffsets``: Commit log offsets.
-14. ``totalRecordCount``: Record count of all changes occurred in this snapshot.
-15. ``deltaRecordCount``: Record count of all new changes occurred in this snapshot.
-16. ``changelogRecordCount``: Record count of all changelog produced in this snapshot.
-17. ``watermark``: Watermark for input records, from Flink watermark mechanism; ``Long.MIN_VALUE`` if there is no watermark.
-18. ``statistics``: Stats file name for statistics of this table.
+6. ``baseManifestListSize``: Size in bytes of the base manifest list; optional.
+7. ``deltaManifestList``: A manifest list recording all new changes occurred in this snapshot.
+8. ``deltaManifestListSize``: Size in bytes of the delta manifest list; optional.
+9. ``changelogManifestList``: A manifest list recording all changelog produced in this snapshot; omitted by C++ serialization if unset.
+10. ``changelogManifestListSize``: Size in bytes of the changelog manifest list; optional.
+11. ``indexManifest``: A manifest recording all index files of this table; omitted by C++ serialization if unset.
+12. ``commitUser``: Usually generated by UUID; used for recovery of streaming writes—one stream write job with one user.
+13. ``commitIdentifier``: Transaction id corresponding to streaming write; each transaction may result in multiple commits for different ``commitKind`` values.
+14. ``commitKind``: Type of changes in this snapshot, including ``append``, ``compact``, ``overwrite`` and ``analyze``.
+15. ``timeMillis``: Commit time in milliseconds.
+16. ``totalRecordCount``: Record count of all changes occurred in this snapshot.
+17. ``deltaRecordCount``: Record count of all new changes occurred in this snapshot.
+18. ``changelogRecordCount``: Record count of all changelog produced in this snapshot; optional.
+19. ``watermark``: Watermark for input records. When unset, C++ serialization omits this field.
+    A commit carries forward the previous snapshot's watermark if no new watermark is supplied.
+20. ``statistics``: Stats file name for statistics of this table; optional.
+21. ``properties``: Optional string-to-string map of snapshot properties, including real-time progress metadata when enabled.
+22. ``nextRowId``: Next row id for row tracking; optional.
+
+The C++ snapshot serializer does not emit ``logOffsets``.
diff --git a/docs/source/user_guide/system_tables.rst b/docs/source/user_guide/system_tables.rst
@@ -39,12 +39,17 @@ case-insensitive, and an unrecognized suffix fails with
 * ``$tags``: tags and the snapshots they reference.
 * ``$branches``: table branches, including ``main``.
 * ``$consumers``: persisted streaming consumer offsets.
-* ``$manifests``: data manifest metadata in the latest snapshot.
-* ``$files``: data file metadata in the latest snapshot.
+* ``$manifests``: data manifest metadata in the selected snapshot.
+* ``$files``: data file metadata in the selected snapshot.
 * ``$audit_log``: row-level changelog records with rowkind.
 * ``$binlog``: changelog records with list-wrapped data columns.
 * ``$ro``: read-optimized data, described in the next section.
 
+``$manifests`` and ``$files`` read the latest snapshot by default. Set
+``scan.snapshot-id`` to read metadata from a specific snapshot of the selected
+branch. If that snapshot does not exist, the read fails with
+``snapshot <id> does not exist`` instead of falling back to the latest snapshot.
+
 Global System Tables
 --------------------
 

diff --git a/docs/source/user_guide/write.rst b/docs/source/user_guide/write.rst
@@ -89,8 +89,12 @@ whose values are ``LargeBinary``. Build the BLOB field with
 
    PAIMON_ASSIGN_OR_RAISE(std::unique_ptr<::ArrowSchema> c_element,
                           paimon::Blob::ArrowField("element", /*nullable=*/true));
-   PAIMON_ASSIGN_OR_RAISE_FROM_ARROW(std::shared_ptr<arrow::Field> element,
-                                     arrow::ImportField(c_element.get()));
+   arrow::Result<std::shared_ptr<arrow::Field>> element_result =
+       arrow::ImportField(c_element.get());
+   if (!element_result.ok()) {
+       return paimon::Status::Invalid(element_result.status().ToString());
+   }
+   std::shared_ptr<arrow::Field> element = element_result.ValueUnsafe();
    std::shared_ptr<arrow::Schema> schema = arrow::schema(
        {arrow::field("id", arrow::int32()), arrow::field("frames", arrow::list(element)),
         arrow::field("views", arrow::map(arrow::utf8(), element))});

diff --git a/third_party/roaring_bitmap/roaring.cpp b/third_party/roaring_bitmap/roaring.cpp
@@ -1,3 +1,5 @@
+// Modified by Apache Paimon to use the paimon::roaring namespace.
+
 // !!! DO NOT EDIT - THIS IS AN AUTO-GENERATED FILE !!!
 // Created by amalgamation.sh on 2023-12-16T02:52:00Z
 

diff --git a/third_party/roaring_bitmap/roaring.h b/third_party/roaring_bitmap/roaring.h
@@ -1,3 +1,5 @@
+// Modified by Apache Paimon to use the paimon::roaring namespace.
+
 // !!! DO NOT EDIT - THIS IS AN AUTO-GENERATED FILE !!!
 // Created by amalgamation.sh on 2023-12-16T02:52:00Z
 

diff --git a/third_party/roaring_bitmap/roaring.hh b/third_party/roaring_bitmap/roaring.hh
@@ -1,3 +1,5 @@
+// Modified by Apache Paimon to use the paimon::roaring namespace.
+
 // !!! DO NOT EDIT - THIS IS AN AUTO-GENERATED FILE !!!
 // Created by amalgamation.sh on 2023-12-16T02:52:00Z