diff --git a/CHANGELOG.md b/CHANGELOG.md index 9bddc15..d4404fa 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -21,6 +21,9 @@ All notable changes to this project will be documented in this file. It uses the ### 🐞 Bug Fixes * `DateTime64` or `Time64` now defaults to millisecond precision ([#76]). +* Reading a ClickHouse string into a text column validates its bytes against + database encoding, raising an error instead of returning unreadable text. + Use `bytea` to preserve raw bytes ([#80]). * Added the `date_time_output_format='iso'` setting to each chDB query to always format `timestamp` and `timestamptz` values in plain text formats with the ISO-8601 format in UTC, converting `timestamp` values from the @@ -42,6 +45,7 @@ All notable changes to this project will be documented in this file. It uses the [v0.1.1]: https://github.com/clickhouse/pg_chdb/compare/v0.1.0...v0.1.1 [#76]: https://github.com/clickhouse/pg_chdb/issues/76 [#78]: https://github.com/clickhouse/pg_chdb/pulls/78 + [#80]: https://github.com/clickhouse/pg_chdb/pulls/80 [chdb_hook docs]: ./doc/chdb_hook.md [structure]: ./doc/chdb_hook.md#structure "chdb_hook Docs: structure" diff --git a/doc/chdb.md b/doc/chdb.md index 1622c68..74c9e82 100644 --- a/doc/chdb.md +++ b/doc/chdb.md @@ -82,6 +82,11 @@ Output: (5 rows) ``` +## Text Encoding + +See [Text Encoding in the `chdb_hook` documentation][text encoding] for how +chDB byte strings map to Postgres `text`, `bytea`, and other types. + ## Settings ### `chdb.max_memory` @@ -181,4 +186,6 @@ Copyright (c) 2026, ClickHouse "ClickHouse Docs: max_parsing_threads session setting" [table functions]: https://clickhouse.com/docs/reference/functions/table-functions "ClickHouse Docs: Table Functions" + [text encoding]: ./chdb_hook.md#text-encoding + "chdb_hook Docs: Text Encoding" [semver]: https://semver.org/spec/v2.0.0.html "Semantic Versioning 2.0.0" diff --git a/doc/chdb_hook.md b/doc/chdb_hook.md index 4596241..81826fd 100644 --- a/doc/chdb_hook.md +++ b/doc/chdb_hook.md @@ -576,7 +576,7 @@ types: | IntervalSecond | interval | | | IntervalWeek | interval | | | IntervalYear | interval | | -| JSON | jsonb | Also reads into json | +| JSON | jsonb | | | LineString | path | | | LowCardinality(T) | T | | | Map(K,V) | text[][] | One row of text items per pair | @@ -586,7 +586,7 @@ types: | Point | point | | | Polygon | polygon[] | | | Ring | polygon | | -| String | text | Also reads into bytea | +| String | text | | | Time | time without time zone | | | Time64(P) | time(P) without time zone | P over 6 caps at 6 | | Tuple(...) | text[] | Fields become text items | @@ -607,6 +607,31 @@ Postgres holds a narrower range than chDB in a few of these types; thus copy raises an error on a `Time` or `Time64` beyond 24 hours, and on a `Date32` outside the Postgres date range. +### Text Encoding + +chDB reads `String`, `FixedString`, `Enum`, and `JSON` as bytes, with no +guarantee of an encoding. Copying such a column into `text`, or into any other +non-binary type, verifies bytes against database encoding and raises an error +for data that cannot represent: + +``` +ERROR: invalid byte sequence for encoding "UTF8": 0x00 +``` + +Every encoding rejects NULs, which Postgres cannot store in `text`. + +Copy into `bytea` to keep bytes as chDB wrote them. Name such these, as +[CREATE TABLE](#create-table-overloading) derives `text` for these types: + +```sql +CREATE TABLE logs (id bigint, payload bytea) WITH ( + copy_from = 's3://my-bucket/logs.parquet' +); +``` + +`FixedString(N)` pads shorter values with NUL bytes. Copying into `text` drops +trailing NULs, while `bytea` keeps all N bytes. + ## Settings ### `chdb_hook.max_memory` diff --git a/vendor/pg-clickhouse-c b/vendor/pg-clickhouse-c index 236da5b..439b710 160000 --- a/vendor/pg-clickhouse-c +++ b/vendor/pg-clickhouse-c @@ -1 +1 @@ -Subproject commit 236da5b615c44b2b5b41dbee9972e71ea56b6e1a +Subproject commit 439b71055777d7e8b03465d94cd86669f0af29f2