Skip to content

refactor(context): separate table loading and dynamic options from contexts #533

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
lucasfang wants to merge 3 commits into
base: main
Choose a base branch
from
+11,347 −7,185
Draft

refactor(context): separate table loading and dynamic options from contexts #533

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
0d2ddf5
refactor(table): introduce a common table hierarchy
lucasfang Nov 6, 2026
7f6bc02
fix(table): resolve hierarchy CI warnings
lucasfang Nov 6, 2026
2de582d
refactor(context): separate table loading and dynamic options from co…
lucasfang Nov 6, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
57 changes: 37 additions & 20 deletions benchmark/benchmark_suite.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -37,6 +37,7 @@
#include "benchmark/cli_option_parsing.h"
#include "paimon/api.h"
#include "paimon/catalog/catalog.h"
#include "paimon/catalog/table_load_options.h"
#include "paimon/testing/utils/testharness.h"

#if __has_include("parquet/arrow/reader.h")
Expand Down Expand Up @@ -434,25 +435,33 @@ Status WriteSourceDataToWriter(paimon::FileStoreWrite* writer, const SourceDataS
Status WriteAndCommit(const std::string& table_path,
const std::map<std::string, std::string>& options,
const SourceDataSpec& source_spec) {
paimon::WriteContextBuilder write_builder(table_path, "benchmark-writer");
paimon::TableLoadOptions write_builder_load_options{table_path};
std::map<std::string, std::string> write_builder_options;
paimon::WriteContextBuilder write_builder("benchmark-writer");
write_builder_options = options;
PAIMON_ASSIGN_OR_RAISE(std::unique_ptr<paimon::WriteContext> write_ctx,
AddContext(write_builder.Finish(), "create write context"));
PAIMON_ASSIGN_OR_RAISE(
std::unique_ptr<paimon::WriteContext> write_ctx,
AddContext(write_builder.SetOptions(options).Finish(), "create write context"));
PAIMON_ASSIGN_OR_RAISE(std::unique_ptr<paimon::FileStoreWrite> writer,
AddContext(paimon::FileStoreWrite::Create(std::move(write_ctx)),
"create file store writer"));
std::unique_ptr<paimon::FileStoreWrite> writer,
AddContext(paimon::FileStoreWrite::Create(write_builder_load_options, std::move(write_ctx),
write_builder_options),
"create file store writer"));

PAIMON_RETURN_NOT_OK(WriteSourceDataToWriter(writer.get(), source_spec));
PAIMON_ASSIGN_OR_RAISE(std::vector<std::shared_ptr<paimon::CommitMessage>> messages,
AddContext(writer->PrepareCommit(), "prepare commit"));

paimon::CommitContextBuilder commit_builder(table_path, "benchmark-writer");
PAIMON_ASSIGN_OR_RAISE(
std::unique_ptr<paimon::CommitContext> commit_ctx,
AddContext(commit_builder.SetOptions(options).Finish(), "create commit context"));
paimon::TableLoadOptions commit_builder_load_options{table_path};
std::map<std::string, std::string> commit_builder_options;
paimon::CommitContextBuilder commit_builder("benchmark-writer");
commit_builder_options = options;
PAIMON_ASSIGN_OR_RAISE(std::unique_ptr<paimon::CommitContext> commit_ctx,
AddContext(commit_builder.Finish(), "create commit context"));
PAIMON_ASSIGN_OR_RAISE(
std::unique_ptr<paimon::FileStoreCommit> committer,
AddContext(paimon::FileStoreCommit::Create(std::move(commit_ctx)), "create committer"));
AddContext(paimon::FileStoreCommit::Create(commit_builder_load_options,
std::move(commit_ctx), commit_builder_options),
"create committer"));
PAIMON_RETURN_NOT_OK(AddContext(committer->Commit(messages), "commit write"));
return Status::OK();
}
Expand Down Expand Up @@ -562,20 +571,26 @@ Result<const SharedReadTableCache*> GetOrCreateSharedReadTable(const std::string
Result<int64_t> ReadRows(const std::string& table_path,
const std::map<std::string, std::string>& options,
int32_t prefetch_parallel_num) {
paimon::ScanContextBuilder scan_builder(table_path);
PAIMON_ASSIGN_OR_RAISE(
std::unique_ptr<paimon::ScanContext> scan_ctx,
AddContext(scan_builder.SetOptions(options).Finish(), "create scan context"));
paimon::TableLoadOptions scan_builder_load_options{table_path};
std::map<std::string, std::string> scan_builder_options;
paimon::ScanContextBuilder scan_builder;
scan_builder_options = options;
PAIMON_ASSIGN_OR_RAISE(std::unique_ptr<paimon::ScanContext> scan_ctx,
AddContext(scan_builder.Finish(), "create scan context"));
PAIMON_ASSIGN_OR_RAISE(
std::unique_ptr<paimon::TableScan> scanner,
AddContext(paimon::TableScan::Create(std::move(scan_ctx)), "create scanner"));
AddContext(paimon::TableScan::Create(scan_builder_load_options, std::move(scan_ctx),
scan_builder_options),
"create scanner"));
PAIMON_ASSIGN_OR_RAISE(std::shared_ptr<paimon::Plan> plan,
AddContext(scanner->CreatePlan(), "create plan"));

paimon::ReadContextBuilder read_builder(table_path);
paimon::TableLoadOptions read_builder_load_options{table_path};
std::map<std::string, std::string> read_builder_options;
paimon::ReadContextBuilder read_builder;
constexpr int32_t kPrefetchBatchCount = 600;
read_builder.SetOptions(options)
.EnablePrefetch(true)
read_builder_options = options;
read_builder.EnablePrefetch(true)
.SetPrefetchBatchCount(kPrefetchBatchCount)
.SetPrefetchMaxParallelNum(prefetch_parallel_num)
.EnableMultiThreadRowToBatch(GetRowToBatchThreadNumber() > 1)
Expand All @@ -584,7 +599,9 @@ Result<int64_t> ReadRows(const std::string& table_path,
AddContext(read_builder.Finish(), "create read context"));
PAIMON_ASSIGN_OR_RAISE(
std::unique_ptr<paimon::TableRead> reader,
AddContext(paimon::TableRead::Create(std::move(read_ctx)), "create table reader"));
AddContext(paimon::TableRead::Create(read_builder_load_options, std::move(read_ctx),
read_builder_options),
"create table reader"));
PAIMON_ASSIGN_OR_RAISE(std::unique_ptr<paimon::BatchReader> batch_reader,
AddContext(reader->CreateReader(plan->Splits()), "create batch reader"));

Expand Down
3 changes: 3 additions & 0 deletions docs/source/api/catalog.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -37,5 +37,8 @@ Interface
.. doxygenclass:: paimon::Table
:members:

.. doxygenstruct:: paimon::TableLoadOptions
:members:

.. doxygenstruct:: paimon::CatalogOptions
:members:
22 changes: 7 additions & 15 deletions docs/source/api/format_table.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -35,18 +35,10 @@ and :cpp:class:`paimon::FileStoreCommit`. See :doc:`../user_guide/format_table`.
Contexts
========

Each of those entry points builds its context from a table path, and the schema under that path
says what kind of table it is. A caller that already holds a ``FormatTable`` hands it over instead,
through the constructor each context builder has for one:

* :cpp:class:`paimon::ScanContextBuilder`\ ``(const std::shared_ptr<FormatTable>&)``
* :cpp:class:`paimon::ReadContextBuilder`\ ``(const std::shared_ptr<FormatTable>&)``
* :cpp:class:`paimon::WriteContextBuilder`\ ``(const std::shared_ptr<FormatTable>&)``
* :cpp:class:`paimon::CommitContextBuilder`\ ``(const std::shared_ptr<FormatTable>&)``

This is the only way to reach a format table whose schema does not live under its own location,
such as one a REST catalog serves: nothing under the location says that it is a format table, nor
that what sits below it is data rather than metadata. The table knows both, so a setting that would
answer either question again - ``SetTableSchema()``, ``WithFileSystem()`` and a branch among them -
is refused by these builders rather than quietly ignored. Options given at the call still win over
the ones the table carries, as they do everywhere else.
Static entry points load the table through :cpp:struct:`paimon::TableLoadOptions`.
Operation Contexts provide execution settings; dynamic options are a separate
argument. A caller holding a loaded ``FormatTable`` creates operators directly
with its ``NewScan()``, ``NewRead()``, ``NewWrite()`` or ``NewCommit()`` methods.
This also works when a catalog keeps the schema outside the table location.
The loaded table supplies its schema and file system, and dynamic options passed
to ``New*()`` override the stored options.
Loading
Loading