[feature](external) Add generic connector write framework prerequisites by suxiaogang223 · Pull Request #68320 · apache/doris

suxiaogang223 · 2026-09-21T07:27:19Z

What problem does this PR solve?

Issue Number: #65086

Related PR: #67395

Problem Summary:

Forward-porting Paimon table writes exposes four gaps in the generic connector write framework. Implementing them as Paimon-specific branches in FE Core would duplicate the existing write pipeline and would also cause Paimon row-level changes to be planned with Iceberg position-delete semantics.

This PR adds the missing connector-neutral framework capabilities:

Write distribution. A connector can describe routing columns, an opaque partition-function name and options, and writer-assignment semantics. FE Core only resolves columns and transports the description; BE performs the routing. Unsupported functions and unsupported BE execution versions fail explicitly. The existing reserved BE execution version 13 is reused, with no Paimon-specific execution version.
Row-level DML. Connectors declare whether changes use file-position deletes or changelog rows. Iceberg retains its position-delete plan. The generic changelog plan converts DELETE, UPDATE, MERGE, and DELETE USING into an operation marker plus the output row, while the connector supplies the marker encoding, primary keys, validation, and label prefix. Unsupported representations fail instead of falling through to an unrelated write path.
Commit reports. BE can return opaque connector commit fragments through the existing external-write report lifecycle. FE acknowledges only accepted reports, retries remain safe after a lost response, and duplicate reports do not add commit data twice. FE Core does not interpret connector-owned bytes.
Variant writes. FE type checking/construction and BE Arrow serialization support Binary Variant V2 as a generic capability required by connector writes.

The implementation intentionally does not add Paimon-specific planner classes, hidden-column names, Thrift structures, or a new write system. Paimon-specific table rules, fixed-bucket hash implementation, writer, and transaction interpretation remain in the Paimon integration that will follow this prerequisite PR.

The connector SPI surface changes once, so connector.plugin.api.version is increased from 10 to 11 once for the whole PR.

Release note

None

Check List (For Author)

Test:
- Targeted FE unit tests for connector SPI, row-level DML planning, commit report acknowledgement/serialization, Variant type handling, and connector write distribution
- Relevant FE Maven reactors compiled successfully while running the targeted unit tests
- BE formatting with clang-format 16 and build-support/check-format.sh
- build-support/check-build-hygiene.sh
- BE compilation and BE unit tests were not run for the latest distribution change
Behavior changed: Yes. Connectors can declare row-change representation and write distribution, return opaque acknowledged commit data, and write Binary Variant V2 through the shared framework.
Does this need documentation: No

hello-stephen · 2026-09-21T07:27:27Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

### What problem does this PR solve? Issue Number: apache#65086 Related PR: apache#67395 Problem Summary: External-file commit reports only carry source-specific Thrift structs. A new connector would need branches in FE core and could miss the existing acknowledgement and retry-safe ownership transfer. Add an opaque binary commit-data field, share the BE report-size budget with Iceberg, and feed the bytes into the owning connector transaction through both coordinator paths. Keep the existing Hive, Iceberg, and MaxCompute fields unchanged. ### Release note None ### Check List (For Author) - Test: FE build with ./build.sh --fe -j 4; FE unit tests with ./run-fe-ut.sh --run org.apache.doris.qe.QeProcessorImplReportAckTest,org.apache.doris.transaction.CommitDataSerializerTest; BE clang-format 16 check and header hygiene check. BE compile and unit tests not run. - Behavior changed: Yes (new opt-in connector commit-data report channel; existing connector paths unchanged) - Does this need documentation: No

### What problem does this PR solve? Issue Number: apache#65086 Related PR: apache#67395 Problem Summary: The shared row-level DML registry selected the Iceberg transform solely from DELETE or MERGE capability. A connector such as Paimon can support the same operations while encoding changes as changelog rows, which would incorrectly route it through Iceberg position-delete planning. Add an engine-neutral row-change representation to the connector SPI, declare Iceberg as position-delete, select its transform by both representation and operation, and fail loudly when a connector advertises row-level writes before its representation has a registered plan. The SPI surface changes, so bump the connector plugin API major and update the recorded surface in the same commit. ### Release note None ### Check List (For Author) - Test: Unit Test - Connector SPI test suite - IcebergRowLevelDmlTransformTest and PluginDrivenExternalTableTest - IcebergWritePlanProviderTest#declaresFullWriteOperationSet - FE-core, connector SPI, and Iceberg checkstyle - Behavior changed: Yes (connector row-level DML is selected by its declared representation) - Does this need documentation: No

### What problem does this PR solve? Issue Number: apache#65086 Related PR: apache#67395 Problem Summary: Generic connector commit payloads were acknowledged by query execution but were not handled consistently by all execution report paths. Centralize payload detection and dispatch so coordinator, load, and job reports all deliver opaque connector commit data to the transaction. ### Release note None ### Check List (For Author) - Test: Unit Test - ./run-fe-ut.sh --run CommitDataSerializerTest,AbstractJobProcessorTest,QeProcessorImplReportAckTest - Behavior changed: No - Does this need documentation: No

### What problem does this PR solve? Issue Number: apache#65086 Related PR: apache#67395 Problem Summary: The master row-level DML framework only models position-delete sinks and carries Iceberg-specific routing details in FE Core. Add a connector-neutral changelog row representation for DELETE, UPDATE, and MERGE, move connector-owned encoding and validation behind the write SPI, and generalize the existing position-delete transform. Preserve required short-circuit evaluation for MERGE branch expressions and track the operation-column row shape explicitly through connector sinks. ### Release note Add connector-neutral framework support for changelog row-level writes. ### Check List (For Author) - Test: Unit Test - run-fe-ut.sh --run ConnectorPluginSurfaceTest,ConnectorRowLevelDmlContractTest,ConnectorChangelogPlanBuilderTest,PositionDeleteRowLevelDmlTransformTest,PhysicalConnectorTableSinkTest,CaseWhenToCompoundPredicateTest - Behavior changed: Yes. Connector row-level DML is routed by connector-declared row representation and capabilities. - Does this need documentation: No

### What problem does this PR solve? Issue Number: apache#65086 Related PR: apache#67395 Problem Summary: Master lacks the generic Binary Variant V2 Arrow transport and nested constructor support needed by connector writes. Add generic binary-struct serialization and connector target casts while preserving master's Variant property compatibility rules. ### Release note Support Binary Variant V2 serialization for connector writes. ### Check List (For Author) - Test: - FE Unit Test: `./run-fe-ut.sh --run CheckCastTest,FunctionRegistryTest` - BE format check: `build-support/clang-format.sh` and `build-support/check-format.sh` - Behavior changed: Yes. Connector writes can serialize Binary Variant V2 values and construct nested Variant values. - Does this need documentation: No

### What problem does this PR solve? Issue Number: apache#65086 Related PR: apache#66685 Problem Summary: Connector-specific bucket routing was previously modeled with data-source classes and enums in FE Core. Add a connector-neutral write-distribution SPI and an opaque external partition-function transport. FE Core now resolves route columns and forwards the function name and options without knowing a connector, while BE provides the common writer-assignment framework and validates supported functions. ### Release note Add a generic external table writer distribution framework. ### Check List (For Author) - Test: - FE Unit Test: `./run-fe-ut.sh --run ConnectorPluginSurfaceTest,ConnectorWriteDistributionTest,PhysicalConnectorTableSinkTest` - BE format check: `build-support/clang-format.sh` and `build-support/check-format.sh` - BE build hygiene: `build-support/check-build-hygiene.sh` - Behavior changed: Yes. Connectors can request opaque external writer routing without adding data-source code to FE Core. - Does this need documentation: No

suxiaogang223 · 2026-09-21T09:46:22Z

run buildall

hello-stephen · 2026-09-21T10:37:08Z

Cloud UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	77.54% (2064/2662)
Line Coverage	65.79% (37952/57686)
Region Coverage	53.22% (35560/66816)
Branch Coverage	56.60% (11428/20190)

### What problem does this PR solve? Issue Number: apache#65086 Related PR: apache#68320 Problem Summary: The generic connector write framework exposed a duplicate C++ default argument under Unity builds, rejected the master Variant V2 carrier at the sink boundary, and dereferenced a null DML type used by legacy connector sinks. Remove the duplicate default argument, preserve legacy INSERT/REWRITE admission for sinks without an explicit DML type, and allow the execution-only Variant carrier to cross into ordinary Variant without relaxing stored Variant layout checks. ### Release note None ### Check List (For Author) - Test: Unit Test: ./run-fe-ut.sh --run BindSinkConnectorVariantTest,PhysicalPlanTranslatorAdmissionGateTest,CheckCastTest; Build: ./build.sh --be -j 12; Check: git diff --check - Behavior changed: No. This restores existing connector sink and Variant compatibility while fixing build validation. - Does this need documentation: No

suxiaogang223 · 2026-09-22T08:19:06Z

run buildall

hello-stephen · 2026-09-22T09:10:09Z

Cloud UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	77.54% (2064/2662)
Line Coverage	65.77% (37939/57686)
Region Coverage	53.24% (35576/66816)
Branch Coverage	56.60% (11427/20190)

hello-stephen · 2026-09-22T10:03:26Z

TPC-H: Total hot run time: 28080 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 1357533a61e960c60257dcdea64518eb72654014, data reload: false

------ Round 1 ----------------------------------
============================================
q1	17719	3900	3832	3832
q2	2228	386	321	321
q3	10024	1381	824	824
q4	4685	481	350	350
q5	7519	861	572	572
q6	177	165	134	134
q7	739	832	623	623
q8	9303	1602	1498	1498
q9	5510	4206	4162	4162
q10	6825	1318	999	999
q11	439	274	266	266
q12	635	420	288	288
q13	18024	2615	2016	2016
q14	264	255	231	231
q15	q16	733	724	665	665
q17	1734	1105	1111	1105
q18	6479	5624	5584	5584
q19	1157	1274	979	979
q20	499	397	258	258
q21	5455	3426	3045	3045
q22	470	381	328	328
Total cold run time: 100618 ms
Total hot run time: 28080 ms

----- Round 2, with runtime_filter_mode=off -----
============================================
q1	4663	4702	4390	4390
q2	732	585	538	538
q3	4826	5190	4681	4681
q4	2224	2319	1451	1451
q5	4604	4382	4638	4382
q6	232	175	125	125
q7	1796	1722	1455	1455
q8	2442	2028	2060	2028
q9	7333	7426	7212	7212
q10	3671	3624	3084	3084
q11	514	372	336	336
q12	703	704	495	495
q13	2277	2627	1975	1975
q14	256	275	248	248
q15	q16	671	680	605	605
q17	7286	6732	6638	6638
q18	11929	11064	11759	11064
q19	1087	963	980	963
q20	2215	2186	1905	1905
q21	5002	4066	4304	4066
q22	519	456	426	426
Total cold run time: 64982 ms
Total hot run time: 58067 ms

hello-stephen · 2026-09-22T10:13:19Z

TPC-DS: Total hot run time: 152688 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 1357533a61e960c60257dcdea64518eb72654014, data reload: false

query5	4347	600	464	464
query6	434	217	192	192
query7	4867	521	298	298
query8	345	184	170	170
query9	8812	4066	4053	4053
query10	476	320	263	263
query11	5818	3553	3251	3251
query12	152	103	89	89
query13	1275	607	430	430
query14	6540	4544	4426	4426
query14_1	3982	3972	3995	3972
query15	211	198	184	184
query16	990	456	442	442
query17	908	651	547	547
query18	2432	469	338	338
query19	204	184	153	153
query20	83	80	79	79
query21	223	134	116	116
query22	12971	12934	12729	12729
query23	14235	13138	12458	12458
query23_1	12728	12518	12494	12494
query24	7253	1134	698	698
query24_1	699	673	698	673
query25	574	451	385	385
query26	1289	320	164	164
query27	2650	543	337	337
query28	4568	2012	1971	1971
query29	1691	761	569	569
query30	304	214	184	184
query31	880	760	642	642
query32	146	92	97	92
query33	513	306	242	242
query34	1202	1138	625	625
query35	731	740	641	641
query36	790	812	695	695
query37	147	103	88	88
query38	1846	1764	1671	1671
query39	682	683	683	683
query39_1	671	662	631	631
query40	223	122	99	99
query41	67	64	63	63
query42	96	92	95	92
query43	340	348	303	303
query44	1409	730	722	722
query45	181	179	168	168
query46	1103	1168	744	744
query47	1501	1485	1374	1374
query48	387	405	291	291
query49	590	395	289	289
query50	933	361	261	261
query51	10411	10586	10185	10185
query52	87	87	75	75
query53	238	244	183	183
query54	250	198	188	188
query55	93	81	74	74
query56	231	224	209	209
query57	1439	1405	1278	1278
query58	279	260	260	260
query59	1971	2053	1822	1822
query60	274	234	223	223
query61	178	152	151	151
query62	394	317	268	268
query63	214	181	172	172
query64	2844	989	827	827
query65	3465	3397	3394	3394
query66	1794	421	303	303
query67	19818	19990	19853	19853
query68	3321	1455	940	940
query69	414	314	260	260
query70	889	849	813	813
query71	300	231	215	215
query72	2676	2309	2274	2274
query73	833	796	415	415
query74	4622	4536	4272	4272
query75	2301	2271	1922	1922
query76	2326	1108	756	756
query77	362	399	295	295
query78	9089	9090	8531	8531
query79	1243	1183	757	757
query80	522	474	375	375
query81	524	319	285	285
query82	262	160	124	124
query83	216	221	199	199
query84	287	143	113	113
query85	789	467	375	375
query86	292	244	240	240
query87	1976	1981	1830	1830
query88	3620	2747	2767	2747
query89	327	291	245	245
query90	2139	185	183	183
query91	167	150	128	128
query92	104	88	86	86
query93	1385	1565	890	890
query94	509	336	305	305
query95	641	457	349	349
query96	1029	759	346	346
query97	2420	2462	2331	2331
query98	154	172	189	172
query99	717	726	622	622
Total cold run time: 235557 ms
Total hot run time: 152688 ms

hello-stephen · 2026-09-22T10:18:11Z

ClickBench: Total hot run time: 24.07 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 1357533a61e960c60257dcdea64518eb72654014, data reload: false

query1	0.00	0.00	0.00
query2	0.09	0.05	0.04
query3	0.26	0.14	0.14
query4	1.60	0.14	0.14
query5	0.24	0.22	0.22
query6	1.16	0.94	0.91
query7	0.04	0.01	0.01
query8	0.05	0.04	0.03
query9	0.39	0.34	0.34
query10	0.55	0.59	0.57
query11	0.20	0.14	0.14
query12	0.19	0.14	0.14
query13	0.46	0.47	0.48
query14	0.96	0.95	0.95
query15	0.60	0.59	0.59
query16	0.30	0.32	0.32
query17	1.08	1.09	1.13
query18	0.21	0.19	0.20
query19	2.12	1.96	1.97
query20	0.02	0.01	0.02
query21	15.47	0.21	0.14
query22	4.85	0.05	0.05
query23	16.10	0.30	0.12
query24	3.05	0.45	0.34
query25	0.11	0.05	0.05
query26	0.73	0.21	0.14
query27	0.04	0.04	0.03
query28	3.54	0.85	0.36
query29	12.50	4.12	3.24
query30	0.28	0.15	0.16
query31	2.77	0.55	0.31
query32	3.22	0.60	0.48
query33	3.18	3.26	3.16
query34	15.60	3.94	3.27
query35	3.24	3.20	3.21
query36	0.56	0.45	0.41
query37	0.09	0.07	0.07
query38	0.06	0.04	0.04
query39	0.04	0.03	0.03
query40	0.18	0.15	0.14
query41	0.08	0.03	0.03
query42	0.04	0.02	0.03
query43	0.04	0.04	0.03
Total cold run time: 96.29 s
Total hot run time: 24.07 s

suxiaogang223 requested review from 924060929, CalvinKirs, englefly, morrySnow and starocean999 as code owners September 21, 2026 07:27

suxiaogang223 force-pushed the codex/paimon-write-framework branch from fc95401 to 4b96e9f Compare September 21, 2026 07:29

suxiaogang223 added 2 commits September 21, 2026 15:31

suxiaogang223 force-pushed the codex/paimon-write-framework branch from 4b96e9f to 06ddfc3 Compare September 21, 2026 07:31

suxiaogang223 added 4 commits September 21, 2026 15:53

suxiaogang223 requested a review from yujun777 as a code owner September 21, 2026 09:16

suxiaogang223 changed the title ~~[feature](fe) Add connector write framework prerequisites~~ [feature](external) Add generic connector write framework prerequisites Sep 21, 2026

Navigation Menu

[feature](external) Add generic connector write framework prerequisites - #68320

[feature](external) Add generic connector write framework prerequisites#68320
suxiaogang223 wants to merge 7 commits into
apache:masterfrom
suxiaogang223:codex/paimon-write-framework

suxiaogang223 commented Sep 21, 2026 •

edited

Loading

Uh oh!

hello-stephen commented Sep 21, 2026

Uh oh!

suxiaogang223 commented Sep 21, 2026

Uh oh!

hello-stephen commented Sep 21, 2026

Uh oh!

suxiaogang223 commented Sep 22, 2026

Uh oh!

hello-stephen commented Sep 22, 2026

Uh oh!

hello-stephen commented Sep 22, 2026

Uh oh!

hello-stephen commented Sep 22, 2026

Uh oh!

hello-stephen commented Sep 22, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

suxiaogang223 commented Sep 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What problem does this PR solve?

Release note

Check List (For Author)

Uh oh!

hello-stephen commented Sep 21, 2026

Uh oh!

suxiaogang223 commented Sep 21, 2026

Uh oh!

hello-stephen commented Sep 21, 2026

Cloud UT Coverage Report

Uh oh!

suxiaogang223 commented Sep 22, 2026

Uh oh!

hello-stephen commented Sep 22, 2026

Cloud UT Coverage Report

Uh oh!

hello-stephen commented Sep 22, 2026

Uh oh!

hello-stephen commented Sep 22, 2026

Uh oh!

hello-stephen commented Sep 22, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

suxiaogang223 commented Sep 21, 2026 •

edited

Loading