Conversation
16 tasks
hacklu-tu
added a commit
to hacklu-tu/doris
that referenced
this pull request
Sep 21, 2026
### What problem does this PR solve? Issue Number: close apache#48203 Related PR: apache#68131 Problem Summary: The BE character conversion path preflighted and converted every value twice through a full UTF-16 intermediate buffer. This repeatedly scanned input and pivot data and copied decode output once more into the result column. Stream conversion through ICU ucnv_convertEx with a bounded pivot, reuse block buffers, append decode output directly to ColumnString, and preserve strict conversion errors and Hive-compatible UTF-16 BOM behavior. On the same Linux Release build, the final implementation improved all 40 column execution cases by 1.96x to 3.92x. Extended reruns for every case whose initial CV exceeded 5% reduced CV to 1.0% to 2.8% and measured 2.17x to 3.93x. ### Release note Improve the query performance of encode and decode without changing their SQL behavior. ### Check List (For Author) - Test: Unit Test / Manual performance test - Linux Release build with ./build.sh --benchmark (passed) - ./run-be-ut.sh --run --filter='function_character_encoding_test.*' (5 tests passed) - 40-case Release benchmark correctness smoke test (40/40 passed) - Fixed-CPU ABBA performance test with 10 samples per case (1.96x to 3.92x) - Extended ABBA rerun for every case with initial CV above 5% (2.17x to 3.93x, CV 1.0% to 2.8%) - build-support/check-format.sh (passed) - build-support/check-build-hygiene.sh (passed) - clang-tidy attempted but blocked by the pre-existing unmatched NOLINTEND in be/src/core/types.h:576 - Behavior changed: No - Does this need documentation: Yes. apache/doris-website#4151
Document that encode and decode take a constant character set expression, reject table columns, and still error on an unsupported charset when the input is NULL. Related PR: apache/doris#68131
hacklu-tu
added a commit
to hacklu-tu/doris
that referenced
this pull request
Sep 21, 2026
### What problem does this PR solve? Issue Number: apache#48203 Related PR: apache#68131 Problem Summary: encode and decode previously accepted a per-row character set column. Require the second argument to be a constant expression, including folded expressions such as upper('utf-8'). Reject table columns in FE. Do not fold away an invalid character set when the first argument is a null literal, so FE constant nulls and BE column nulls return the same error. Rewrite the regression CASE queries that evaluated every charset against every row. ### Release note encode and decode now require a constant character set expression. An unsupported character set is rejected even when the input is NULL. ### Check List (For Author) - Test: Unit Test / Regression test - ./run-fe-ut.sh --run org.apache.doris.nereids.trees.expressions.functions.scalar.EncodeDecodeTest (4 tests passed) - ./run-be-ut.sh --run --filter='function_character_encoding_test.*' (7 tests passed) - ./run-regression-test.sh --run -d query_p0/sql_functions/binary_functions -s test_encode_decode (1 suite passed) - Behavior changed: Yes. The character set argument must be constant. Invalid character sets error even when the first argument is NULL. - Does this need documentation: Yes. apache/doris-website#4151
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Versions
Languages
Docs Checklist
Description
Adds the English and Chinese dev documentation for
ENCODEandDECODE.The documentation covers function signatures, parameter and return types, all six supported character sets, case-insensitive character set names, UTF-16 BOM and byte-order behavior, NULL and empty input, conversion errors, and runnable examples.
Only dev documentation is included because these functions are introduced by the related code PR and currently exist only on Doris master; they are not available in 4.x or earlier versions.
Related code PR: apache/doris#68131
The SQL examples match the implementation and the regression cases executed on a native Linux Doris cluster. The SQL-function, i18n, link, and SEO governance checks were run, and the full English/Chinese Docusaurus build completed successfully.