v8.5: update TiCDC Debezium DDL, WATERMARK support - #23524
Conversation
lilin90
added
translation/doing
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
ti-chi-bot
Bot
added
the
size/XXL
📝 WalkthroughWalkthroughThe TiCDC Debezium documentation now distinguishes new and classic architectures, documents DDL, DML, and WATERMARK event schemas, expands metadata descriptions, and records TiCDC-specific data-type mappings. ChangesTiCDC Debezium documentation
Estimated code review effort: 2 (Simple) | ~15 minutes Suggested labels: Suggested reviewers: 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
| # TiCDC Debezium Protocol | ||
|
|
||
| [Debezium](https://debezium.io/) is a tool for capturing database changes. It converts each captured database change into a message called an "event" and sends these events to Kafka. Starting from v8.0.0, TiCDC supports sending TiDB changes to Kafka using a Debezium style output format, simplifying migration from MySQL databases for users who had previously been using Debezium's MySQL integration. | ||
| [Debezium](https://debezium.io/) is a tool for capturing database changes. It converts each captured database change into a message called an "event" and sends these events to Kafka. Starting from v8.0.0, TiCDC supports sending TiDB changes to Kafka using a Debezium style output format, simplifying migration from MySQL databases for users who had previously been using Debezium's MySQL integration. Starting from [TiCDC v8.5.4-release.1](https://github.com/pingcap/ticdc/releases/tag/v8.5.4-release.1) ([new TiCDC architecture](/ticdc/ticdc-architecture.md)), TiCDC supports DDL events and WATERMARK events. |
There was a problem hiding this comment.
📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win
Fix the two grammar errors in the changed prose.
Use Debezium-style, not Debezium style. Use prints, not print.
Committable replacements
-Debezium style output format
+Debezium-style output format-- TiCDC print the wrong `flen` with the FLOAT [tidb#57060](https://github.com/pingcap/tidb/issues/57060).
+- TiCDC prints an incorrect `flen` value for `FLOAT` columns. See [tidb#57060](https://github.com/pingcap/tidb/issues/57060).Also applies to: 819-819
🧰 Tools
🪛 LanguageTool
[grammar] ~8-~8: Use a hyphen to join words.
Context: ...g TiDB changes to Kafka using a Debezium style output format, simplifying migrati...
(QB_NEW_EN_HYPHEN)
Source: Linters/SAST tools
| The Debezium protocol supports the following types of events: | ||
|
|
||
| - DDL event: represents a DDL change record. After the upstream DDL statement is successfully executed, the DDL event is sent to every Message Queue (MQ) partition. | ||
|
|
||
| - DML event: represents a row data change record. The DML event is sent when a row change occurs. It contains the information about the row after the change occurs. | ||
|
|
||
| - WATERMARK event: represents a special point in time. It indicates that the events received before this point are complete. The WATERMARK event applies only to the TiDB extension field and takes effect when you set [`enable-tidb-extension`](/ticdc/ticdc-sink-to-kafka.md#configure-sink-uri-for-kafka) to `true` in `sink-uri`. |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟡 Minor | ⚡ Quick win
Document the DML payload correctly.
This text says that a DML event contains only post-change data. The DML envelope documents both before and after values. State the null behavior for INSERT and DELETE events.
Committable replacement
-- DML event: represents a row data change record. The DML event is sent when a row change occurs. It contains the information about the row after the change occurs.
+- DML event: represents a row data change record. TiCDC sends the DML event when a row change occurs. It contains the row values before and after the change. For `INSERT` events, `before` is `null`; for `DELETE` events, `after` is `null`.📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| The Debezium protocol supports the following types of events: | |
| - DDL event: represents a DDL change record. After the upstream DDL statement is successfully executed, the DDL event is sent to every Message Queue (MQ) partition. | |
| - DML event: represents a row data change record. The DML event is sent when a row change occurs. It contains the information about the row after the change occurs. | |
| - WATERMARK event: represents a special point in time. It indicates that the events received before this point are complete. The WATERMARK event applies only to the TiDB extension field and takes effect when you set [`enable-tidb-extension`](/ticdc/ticdc-sink-to-kafka.md#configure-sink-uri-for-kafka) to `true` in `sink-uri`. | |
| The Debezium protocol supports the following types of events: | |
| - DDL event: represents a DDL change record. After the upstream DDL statement is successfully executed, the DDL event is sent to every Message Queue (MQ) partition. | |
| - DML event: represents a row data change record. TiCDC sends the DML event when a row change occurs. It contains the row values before and after the change. For `INSERT` events, `before` is `null`; for `DELETE` events, `after` is `null`. | |
| - WATERMARK event: represents a special point in time. It indicates that the events received before this point are complete. The WATERMARK event applies only to the TiDB extension field and takes effect when you set [`enable-tidb-extension`](/ticdc/ticdc-sink-to-kafka.md#configure-sink-uri-for-kafka) to `true` in `sink-uri`. |
🧰 Tools
🪛 LanguageTool
[style] ~25-~25: This phrase is redundant. Consider writing “point” or “time”.
Context: ...- WATERMARK event: represents a special point in time. It indicates that the events received ...
(MOMENT_IN_TIME)
| <div label="Classic TiCDC architecture"> | ||
|
|
||
| The Debezium protocol only supports Row Changed events and directly ignores DDL events and WATERMARK events. A Row changed event represents a data change in a row. When a row changes, the Row Changed event is sent, including relevant information about the row both before and after the change. A WATERMARK event marks the replication progress of a table, indicating that all events earlier than the watermark have been sent to the downstream. |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win
Remove the classic-architecture WATERMARK contradiction.
The paragraph says that classic TiCDC ignores WATERMARK events, then describes a WATERMARK event as if classic TiCDC emits it. This conflicts with the support statement and the WATERMARK note.
Committable replacement
-The Debezium protocol only supports Row Changed events and directly ignores DDL events and WATERMARK events. A Row changed event represents a data change in a row. When a row changes, the Row Changed event is sent, including relevant information about the row both before and after the change. A WATERMARK event marks the replication progress of a table, indicating that all events earlier than the watermark have been sent to the downstream.
+The Debezium protocol only supports Row Changed events in the classic TiCDC architecture. TiCDC ignores DDL events and WATERMARK events in this architecture. A Row Changed event represents a data change in a row. When a row changes, TiCDC sends the event with relevant information about the row before and after the change.📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| <div label="Classic TiCDC architecture"> | |
| The Debezium protocol only supports Row Changed events and directly ignores DDL events and WATERMARK events. A Row changed event represents a data change in a row. When a row changes, the Row Changed event is sent, including relevant information about the row both before and after the change. A WATERMARK event marks the replication progress of a table, indicating that all events earlier than the watermark have been sent to the downstream. | |
| <div label="Classic TiCDC architecture"> | |
| The Debezium protocol only supports Row Changed events in the classic TiCDC architecture. TiCDC ignores DDL events and WATERMARK events in this architecture. A Row Changed event represents a data change in a row. When a row changes, TiCDC sends the event with relevant information about the row before and after the change. |
| | Field | Type | Description | | ||
| |:------------------|:--------|:------------------------------------------------| | ||
| | `payload` | JSON | The information about database name. | | ||
| | `schema.fields` | JSON | The type information of each field in the payload. | | ||
| | `schema.type` | String | The data type of the field. | | ||
| | `schema.optional` | Boolean | Indicates whether the field is optional. When it is `true`, the field is optional. | | ||
| | `schema.version` | String | The schema version. | | ||
|
|
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win
Align the DDL field-table types with the JSON schema.
The JSON schema defines schema.version as numeric, primaryKeyColumnNames as an array, and enumValues as an array. The table currently documents String for all three fields.
Committable replacements
-| `schema.version` | String | The schema version. |
+| `schema.version` | Number | The schema version. |-| `payload.tableChanges.table.primaryKeyColumnNames` | string | List of columns that compose the table's primary key. |
+| `payload.tableChanges.table.primaryKeyColumnNames` | Array | List of columns that compose the table's primary key. |-| `payload.tableChanges.table.columns.enumValues` | String | The enumeration values of the column. The format is `['e1', 'e2']`. |
+| `payload.tableChanges.table.columns.enumValues` | Array | The enumeration values of the column. The format is `["e1", "e2"]`. |Also applies to: 425-433
| "source": { | ||
| "version": "2.4.0.Final", | ||
| "connector": "TiCDC", | ||
| "name": "test_cluster", | ||
| "ts_ms": 0, | ||
| "snapshot": "false", | ||
| "db": "test", | ||
| "table": "table1", | ||
| "server_id": 0, | ||
| "gtid": null, | ||
| "file": "", | ||
| "pos": 0, | ||
| "row": 0, | ||
| "thread": 0, | ||
| "query": null, | ||
| "commit_ts": 1, | ||
| "cluster_id": "test_cluster" |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟠 Major | 🏗️ Heavy lift
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -e
printf '%s\n' '--- candidate files ---'
git ls-files | rg '(^|/)(ticdc|.*debezium.*)'
printf '%s\n' '--- document structure ---'
wc -l ticdc/ticdc-debezium.md
rg -n 'commit_ts|cluster_id|Source|source|WATERMARK|schema' ticdc/ticdc-debezium.md
printf '%s\n' '--- serializer references ---'
rg -n -i 'commit.?ts|cluster.?id|cluster_id|Debezium|WATERMARK|watermark' --glob '!ticdc/ticdc-debezium.md' --glob '!*.md' . | head -300Repository: pingcap/docs
Length of output: 11692
🏁 Script executed:
#!/bin/bash
set -e
for range in '35,180' '450,570' '600,760'; do
printf '\n--- ticdc/ticdc-debezium.md:%s ---\n' "$range"
sed -n "${range}p" ticdc/ticdc-debezium.md
done
printf '\n--- repository history metadata for the document ---\n'
git log -5 --format='%h %s' -- ticdc/ticdc-debezium.mdRepository: pingcap/docs
Length of output: 16185
🌐 Web query:
pingcap ticdc Debezium serializer commit_ts cluster_id Source schema
💡 Result:
TiCDC implements a Debezium-compatible protocol that enhances the standard Debezium format to ensure data integrity and traceability within the TiDB ecosystem [1][2]. Key extensions and details: 1. Added Fields (CommitTS and ClusterID) Because the original Debezium format lacks specific TiDB transaction identifiers, TiCDC adds two fields to the payload.source section of the message [1][3]: - commit_ts: A unique transaction identifier (Commit Timestamp) in TiDB [1][2]. - cluster_id: The identifier for the source TiDB cluster [1][2]. 2. Schema Integration By default, the TiCDC Debezium protocol includes schema information in the message to help downstream consumers interpret the data structure [2]. - The schema name format is: "{cluster-name}.{schema-name}.{table-name}.Envelope" [1][4]. - You can disable this schema output if it is not required by setting the debezium-disable-schema parameter to true in your changefeed configuration or sink-uri [2][5]. 3. Configuration The Debezium protocol is supported when the sink type is MQ (e.g., Kafka) [5]. Configuration parameters such as output-old-value can be managed under the [sink.debezium] section in the changefeed configuration file [5][6]. These additions ensure that downstream systems can correctly identify the source of the data and maintain transactional ordering using the TiDB-specific commit timestamp [1][3].
Citations:
- 1: https://docs.pingcap.com/tidb/stable/ticdc-debezium/
- 2: https://docs.pingcap.com/zh/tidb/dev/ticdc-debezium/
- 3: https://jishuzhan.net/article/1820045046154530818
- 4: https://docs.pingcap.com/tidb/dev/ticdc-debezium/
- 5: https://docs.pingcap.com/tidb/stable/ticdc-changefeed-config/
- 6: https://docs.pingcap.com/tidb/v8.1/ticdc-changefeed-config/
🏁 Script executed:
#!/bin/bash
set -e
tmp=$(mktemp)
curl -L --fail --silent --show-error \
'https://api.github.com/repos/pingcap/ticdc/git/trees/master?recursive=1' > "$tmp"
python3 - "$tmp" <<'PY'
import json, sys
data=json.load(open(sys.argv[1]))
for item in data.get("tree", []):
path=item.get("path", "")
if any(term in path.lower() for term in ("debezium", "mq", "sink")):
print(path)
PY
rm -f "$tmp"Repository: pingcap/docs
Length of output: 201
🌐 Web query:
site:github.com/pingcap/ticdc "cluster_id" "commit_ts" Debezium
💡 Result:
In the context of TiCDC (a Change Data Capture tool for TiDB), cluster_id and commit_ts are distinct metadata concepts used for cluster identification and event ordering/filtering, respectively [1][2][3]. cluster_id The cluster_id is a unique identifier for a TiDB cluster [1]. In TiCDC, it is primarily used to: - Distinguish between different TiDB clusters during replication to prevent misconfiguration, such as creating feedback loops where a cluster replicates data to itself [1]. - Manage configuration state within external stores like etcd. TiCDC servers use this ID to partition or identify their specific changefeeds; issues have historically occurred when cluster_id values share prefixes, causing inadvertent overlap in etcd key scanning [2]. commit_ts The commit_ts (Commit Timestamp) is a monotonically increasing value used to represent the exact point in time when a transaction was committed in the TiDB cluster [3][4]. It is critical for: - Event Sequencing: Ensuring that data changes are processed in the correct order [4]. - Error Debugging: When Debezium encoding or other replication errors occur, the commit_ts allows engineers to pinpoint the exact transaction causing the issue [3][5]. - Filtering/Watermarking: TiCDC consumers use commit_ts to identify and ignore outdated or "fallback" events that have a timestamp older than the current partition watermark, preventing the processing of stale data [4][6]. While these terms often appear together in logs and error reports—specifically when troubleshooting replication failures—they serve different functional roles: one identifies the source infrastructure (cluster_id), and the other manages the temporal consistency of the data (commit_ts) [3][4].
Citations:
- 1: Prevent TiCDC from Using the Same TiDB Cluster as Both Upstream and Downstream ticdc#3949
- 2: TiCDC cluster-id prefix collision causes changefeeds from another cluster to be loaded after restart ticdc#4756
- 3: codec(ticdc): improve error logging for Debezium encoding failures ticdc#3963
- 4: consumer: ignore fallback dml event ticdc#3864
- 5: Improve error logging for Debezium encoding failures ticdc#3964
- 6: consumer: revert https://github.com/pingcap/ticdc/pull/4052 ticdc#5438
🌐 Web query:
site:github.com/pingcap/ticdc "io.debezium.connector.mysql.Source"
💡 Result:
io.debezium.connector.mysql.Source is not a standalone class, but rather a reference to a required field within the JSON structure of Debezium change events [1]. Specifically, it defines the schema name for the source metadata block of an event [1]. When the Debezium MySQL connector captures changes (such as row-level modifications or schema changes), it produces events that include a source block [2][1]. This block provides critical metadata about the origin of the event, including: 1. Connector and Database Info: The connector name, and the specific database and table names involved [2][1]. 2. Binlog Metadata: The MySQL binlog file name, the position within that file, the row number, and the MySQL thread ID [3][1]. 3. Event Context: Whether the event occurred during a snapshot, and if available, the MySQL server ID and event timestamp [3][1]. This structure allows downstream consumers to correlate events across different topics and accurately reconstruct the sequence of changes [2]. In the context of projects like TiCDC, this protocol is used to ensure compatibility when encoding data events for downstream systems [4][5][6].
Citations:
- 1: https://debezium.io/documentation/reference/stable/kc-tutorial.html
- 2: http://debezium.io/documentation/reference/2.0/connectors/mysql.html
- 3: https://debezium.io/documentation/reference/stable/connectors/mysql
- 4: codec: fix debezium protocol ticdc#1342
- 5: sink: add debezium-avro protocol ticdc#5475
- 6: pingcap/ticdc@db2ad80
🌐 Web query:
pingcap ticdc source code Debezium encoder source.go commit_ts cluster_id
💡 Result:
In TiCDC's Debezium protocol implementation, the fields commit_ts and cluster_id are explicitly added to the source block of the Debezium JSON payload to ensure data integrity and track TiDB-specific transaction information [1][2][3]. The standard Debezium format does not natively include these fields. TiCDC adds them to provide: - commit_ts: The unique transaction identifier (Commit Timestamp) in TiDB [1][3]. - cluster_id: The identifier for the TiDB cluster from which the data change originated [1][2]. These fields are handled within the TiCDC codebase, specifically within the pkg/sink/codec/debezium package [4][5]. The logic for constructing the Debezium envelope, including these source fields, is primarily found in pkg/sink/codec/debezium/encoder.go and pkg/sink/codec/debezium/codec.go [4][5]. For example, when an event is encoded, the encoder populates the source object with the commit_ts and cluster_id alongside standard fields like db, table, and ts_ms [1][3]. Recent enhancements to the codebase have also included passing the commit_ts parameter through encoder methods to improve error logging and context, allowing users to better identify and troubleshoot issues with specific events [6][7].
Citations:
- 1: https://docs.pingcap.com/tidb/stable/ticdc-debezium/
- 2: https://docs.pingcap.com/tidb/dev/ticdc-debezium/
- 3: https://docs.pingcap.com/tidb/v8.1/ticdc-debezium/
- 4: pingcap/tiflow@0e0b615
- 5: sink: add debezium-avro protocol ticdc#5475
- 6: codec(ticdc): improve error logging for Debezium encoding failures ticdc#3963
- 7: Improve error logging for Debezium encoding failures ticdc#3964
🌐 Web query:
github pingcap ticdc debezium codec source field schema
💡 Result:
In the TiCDC Debezium protocol, the source field is a nested object within the payload that provides metadata about the event source [1][2]. It includes the following fields [1][3]: - version: The version of the Debezium connector used (e.g., "2.4.0.Final") [1]. - connector: Set to "TiCDC" [1]. - name: The cluster name, defaulting to "default" [1]. - ts_ms: The timestamp in milliseconds when the event was generated by TiCDC [1]. - snapshot: Indicates if the event is part of a snapshot ("true" or "false") [1]. - db: The name of the database where the event occurred [1][4]. - table: The name of the table where the event occurred [1][4]. - server_id: The server ID, typically 0 for TiCDC [1][3]. - gtid: The Global Transaction Identifier, which is null in TiCDC [1][3]. - file: The binary log file name, which is empty in TiCDC [1][3]. - pos: The binary log position, which is 0 in TiCDC [1][3]. - row: The row number, which is 0 in TiCDC [1][3]. - thread: The thread ID, which is 0 in TiCDC [1][3]. - query: The original SQL query, which is null in TiCDC [1]. - commit_ts: The TiDB CommitTs timestamp associated with the transaction [1][4]. - cluster_id: The ID of the TiDB cluster (e.g., "default") [1]. The overall output format consists of a payload field (containing the data, operation type, and source metadata) and a schema field (which describes the data structure) [1][4]. Users can disable the output of the schema information if needed by setting debezium-disable-schema to true in the changefeed configuration or sink-uri [4].
Citations:
- 1: https://docs.pingcap.com/tidb/stable/ticdc-debezium/
- 2: https://docs.pingcap.com/tidb/v8.1/ticdc-debezium/
- 3: https://docs.pingcap.com/tidb/dev/ticdc-debezium/
- 4: https://docs.pingcap.com/zh/tidb/stable/ticdc-debezium/
🏁 Script executed:
#!/bin/bash
set -e
rm -rf /tmp/ticdc-review-source
git clone --depth 1 --filter=blob:none --no-checkout https://github.com/pingcap/ticdc.git /tmp/ticdc-review-source >/tmp/ticdc-clone.log 2>&1
cd /tmp/ticdc-review-source
git sparse-checkout set pkg/sink/codec/debezium
git checkout --quiet
printf '%s\n' '--- files ---'
git ls-files
printf '%s\n' '--- relevant symbols and fields ---'
rg -n -C 4 'commit_ts|cluster_id|Source|source' pkg/sink/codec/debeziumRepository: pingcap/docs
Length of output: 50369
🏁 Script executed:
#!/bin/bash
set -e
cd /tmp/ticdc-review-source
printf '%s\n' '--- Debezium source files ---'
wc -l pkg/sink/codec/debezium/{encoder.go,helper.go,codec.go,debezium_test.go}
printf '%s\n' '--- field definitions and assignments ---'
rg -n -C 5 --glob '*.go' 'commit_ts|cluster_id|Source|source' pkg/sink/codec/debezium | head -240
printf '%s\n' '--- JSON fixtures containing source metadata ---'
rg -n -C 3 'commit_ts|cluster_id|\"source\"' pkg/sink/codec/debezium/testdata | head -180Repository: pingcap/docs
Length of output: 17713
🏁 Script executed:
#!/bin/bash
set -e
cd /tmp/ticdc-review-source
printf '%s\n' '--- source schema writer ---'
sed -n '889,1000p' pkg/sink/codec/debezium/codec.go
printf '%s\n' '--- DDL encoder ---'
sed -n '1260,1435p' pkg/sink/codec/debezium/codec.go
printf '%s\n' '--- watermark encoder ---'
sed -n '1435,1605p' pkg/sink/codec/debezium/codec.go
printf '%s\n' '--- extension configuration and tests ---'
rg -n -C 5 'EnableTiDBExtension|enable-tidb-extension|commit_ts|cluster_id' pkg/sink/codec/debezium/{codec.go,codec_test.go,debezium_test.go,helper.go}Repository: pingcap/docs
Length of output: 33781
🏁 Script executed:
#!/bin/bash
set -e
python3 - <<'PY'
import json
import re
from pathlib import Path
text = Path("ticdc/ticdc-debezium.md").read_text()
sections = re.split(r'(?=^### )', text, flags=re.M)
for section in sections:
heading = section.splitlines()[0] if section.startswith("### ") else ""
if heading not in {
"### DDL event (new TiCDC architecture)",
"### DML event",
"### WATERMARK event (new TiCDC architecture)",
}:
continue
blocks = re.findall(r"```json\n(.*?)\n```", section, flags=re.S)
for index, block in enumerate(blocks, 1):
obj = json.loads(block)
payload = obj.get("payload", {})
source = payload.get("source")
if not isinstance(source, dict):
continue
schema_source = None
for field in obj.get("schema", {}).get("fields", []):
if field.get("field") == "source":
schema_source = field
break
print(heading)
print(f"JSON block {index}: payload.source keys = {sorted(source)}")
print("schema source fields =", sorted(
field.get("field") for field in (schema_source or {}).get("fields", [])
))
print("missing from schema =", sorted(
set(source) - {
field.get("field") for field in (schema_source or {}).get("fields", [])
}
))
print()
PYRepository: pingcap/docs
Length of output: 1481
Align the Source schemas with the TiDB extension fields.
The DDL, DML, and WATERMARK examples include payload.source.commit_ts and payload.source.cluster_id, but their nested Source schemas omit both fields. Add commit_ts as int64 with optional: false and cluster_id as string with optional: false to all three schemas.
Source: Path instructions
| "schema": { | ||
| "optional": false, | ||
| "type": "struct", | ||
| "version": 1, | ||
| "name": "io.debezium.connector.mysql.SchemaChangeValue", | ||
| "fields": [ |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟡 Minor | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -eu
printf '%s\n' '--- target file locations ---'
wc -l ticdc/ticdc-debezium.md
printf '%s\n' '--- target documentation sections ---'
sed -n '125,175p' ticdc/ticdc-debezium.md
sed -n '420,455p' ticdc/ticdc-debezium.md
printf '%s\n' '--- repository references ---'
rg -n -C 3 'SchemaChangeValue|schema.?change|SchemaChange|debezium' --glob '!ticdc/ticdc-debezium.md' .Repository: pingcap/docs
Length of output: 50371
🌐 Web query:
TiCDC Debezium protocol SchemaChangeValue schema.name cluster-name schema-name table-name serializer
💡 Result:
In the TiCDC Debezium protocol, the schema.name field is a string that specifies the name of the schema for the message payload [1][2][3]. For a schema change event, the schema.name follows the format: "{cluster-name}.{schema-name}.{table-name}.SchemaChangeValue" [1][3] In this format: - {cluster-name} represents the name of the TiDB cluster [4]. - {schema-name} represents the name of the database [4]. - {table-name} represents the name of the table [4]. The SchemaChangeValue suffix specifically indicates that the payload contains schema change information [1][3]. This naming convention is part of how TiCDC structures Debezium-compatible messages to allow downstream consumers to identify the structure and context of the change event [1][2]. While TiCDC provides this protocol as a way to export data changes in a Debezium-compatible format, the specific schema.name construction is a defined feature of the TiCDC Debezium implementation to ensure uniqueness and clarity in data pipelines [1][2][5].
Citations:
- 1: https://docs.pingcap.com/tidb/dev/ticdc-debezium/
- 2: https://docs.pingcap.com/tidb/stable/ticdc-debezium/
- 3: https://github.com/ginkgoch/pingcap-docs/blob/master/ticdc/ticdc-debezium.md
- 4: https://docs.pingcap.com/tidb/v8.1/ticdc-debezium/
- 5: https://website-docs-preview.vercel.app/tidb/stable/ticdc-debezium/
🏁 Script executed:
#!/bin/bash
set -eu
printf '%s\n' '--- exact nearby schema-name references ---'
rg -n -C 2 'SchemaChangeValue|cluster-name.*schema-name.*table-name|schema\.name' ticdc/ticdc-debezium.md
printf '%s\n' '--- repository metadata and possible source references ---'
git ls-files | rg '(^|/)(ticdc|cdc|debezium|protocol|sink)(/|[-_])|tiflow|go\.mod$' | head -200
printf '%s\n' '--- remote configuration ---'
git remote -vRepository: pingcap/docs
Length of output: 7655
🌐 Web query:
site:github.com/pingcap/tiflow "SchemaChangeValue" "schemaName"
💡 Result:
There is no standard, standalone programming entity or function named SchemaChangeValue. In the context of the TiCDC (TiDB Change Data Capture) project, schemaName is a field commonly found within structures related to DDL (Data Definition Language) processing and schema management, such as the job object representing a DDL operation [1][2]. In TiCDC source code, schemaName typically identifies the database (schema) name associated with a specific DDL event or table [1][2]. It is frequently used in logging, filtering, and event dispatching logic [1][3]. For example: 1. DDL Puller: When processing DDL jobs, the system logs or references job.SchemaName to identify which database is affected by a change [1]. 2. Schema Registry (Avro): In Avro codec implementations for TiCDC, schemaName is used as a parameter in interfaces like SchemaManager (e.g., Register, Lookup, ClearRegistry) to manage schema versions and definitions within external registries like Confluent or AWS Glue [4][5][6]. 3. Configuration: Users may reference schema names in configuration files, such as when setting up dispatchers to send DDL events to different Kafka topics [3]. If you are encountering SchemaChangeValue in a specific codebase or error message, it is likely a custom struct, constant, or variable name unique to that implementation, rather than a widely recognized library component.
Citations:
- 1: https://github.com/pingcap/tiflow/blob/8879687f/cdc/puller/ddl_puller.go
- 2: https://github.com/pingcap/tiflow/blob/8879687f/cdc/entry/schema_test.go
- 3: Support dispatch schema-level ddl to different topic tiflow#11882
- 4: https://github.com/pingcap/tiflow/blob/5092ea155372/pkg/sink/codec/avro/schema_manager.go
- 5: https://github.com/pingcap/tiflow/blob/5092ea155372/pkg/sink/codec/avro/glue_schema_registry.go
- 6: https://github.com/pingcap/tiflow/blob/5092ea155372/pkg/sink/codec/avro/confluent_schema_registry.go
🏁 Script executed:
#!/bin/bash
set -eu
tmpdir="$(mktemp -d)"
trap 'rm -rf "$tmpdir"' EXIT
printf '%s\n' '--- TiFlow repository tree candidates ---'
curl -LfsS 'https://api.github.com/repos/pingcap/tiflow/git/trees/master?recursive=1' |
jq -r '.tree[]?.path' |
rg -i 'debezium|protocol|schema.*change|mq' |
head -200
printf '%s\n' '--- GitHub code-search page references ---'
curl -LfsS 'https://github.com/search?q=repo%3Apingcap%2Ftiflow+%22SchemaChangeValue%22&type=code' |
rg -o '.{0,100}SchemaChangeValue.{0,150}' |
head -20 || trueRepository: pingcap/docs
Length of output: 5044
🌐 Web query:
"io.debezium.connector.mysql.SchemaChangeValue" TiCDC
💡 Result:
In the context of TiCDC, io.debezium.connector.mysql.SchemaChangeValue is a schema name used within the TiCDC Debezium protocol output [1][2]. TiCDC supports a Debezium-compatible output format, which allows downstream consumers that expect standard Debezium events to process change data captured from TiDB [1]. When TiCDC produces DDL (Data Definition Language) events using this protocol, it adheres to the Debezium event structure [1]. The SchemaChangeValue schema is specifically associated with DDL operations [1][2]. In the JSON output format provided by TiCDC, the schema name field reflects this structure, typically following a format like {cluster-name}.{schema-name}.{table-name}.SchemaChangeValue [1][2]. This payload includes information such as the database name, the actual DDL statement executed, and source metadata (like binary log file position or GTID) consistent with how the native Debezium MySQL connector reports schema changes [1][3][2]. Essentially, TiCDC utilizes this identifier to signal to downstream tools—such as those integrated with Kafka or other streaming platforms—that the incoming message represents a schema change event structured according to the Debezium connector specification [1].
Citations:
- 1: https://docs.pingcap.com/tidb/dev/ticdc-debezium/
- 2: https://github.com/ginkgoch/pingcap-docs/blob/master/ticdc/ticdc-debezium.md
- 3: https://debezium-builder.github.io/documentation/reference/1.5/connectors/mysql.html
🌐 Web query:
"SchemaChangeValue" "cluster-name" TiCDC
💡 Result:
In TiCDC, SchemaChangeValue is a component used within the Debezium output protocol to define the schema name format for messages [1][2]. Specifically, when TiCDC is configured to output data using the Debezium protocol to a message queue, the schema name for a given message is constructed using the following pattern: "{cluster-name}.{schema-name}.{table-name}.SchemaChangeValue" In this format: - {cluster-name} is the name of your TiDB cluster [1]. - {schema-name} is the name of the database (schema) where the change occurred [1]. - {table-name} is the name of the table where the change occurred [1]. - SchemaChangeValue serves as a constant suffix indicating the specific type of schema definition for the payload [1][2]. This naming convention ensures that downstream consumers of the Debezium protocol can uniquely identify the schema information associated with the data changes being streamed from a specific TiDB cluster [1][3]. If you are working with TiCDC's Debezium protocol, ensure that your consumer correctly handles this schema name structure, which is enabled by default unless explicitly disabled via the debezium-disable-schema configuration parameter [4].
Citations:
- 1: https://docs.pingcap.com/tidb/dev/ticdc-debezium/
- 2: https://github.com/ginkgoch/pingcap-docs/blob/master/ticdc/ticdc-debezium.md
- 3: https://pingkai.cn/docs/tidb/dev/ticdc-debezium
- 4: https://docs.pingcap.com/tidb/dev/ticdc-changefeed-config/
🏁 Script executed:
#!/bin/bash
set -eu
for ref in master main v8.0.0 v8.1.0; do
for path in \
cdc/sink/codec/debezium/debezium.go \
cdc/sink/codec/debezium/codec.go \
pkg/sink/codec/debezium/debezium.go \
pkg/sink/codec/debezium/codec.go \
cdc/sink/codec/debezium/schema.go \
pkg/sink/codec/debezium/schema.go; do
url="https://raw.githubusercontent.com/pingcap/tiflow/${ref}/${path}"
body="$(curl -LfsS --max-time 10 "$url" 2>/dev/null || true)"
if [ -n "$body" ]; then
printf '%s\n' "--- $ref/$path ---"
printf '%s\n' "$body" | rg -n -C 4 'SchemaChangeValue|schema.*name|SchemaChange' || true
fi
done
doneRepository: pingcap/docs
Length of output: 1585
🏁 Script executed:
#!/bin/bash
set -eu
python3 - <<'PY'
from pathlib import Path
import re
import urllib.request
doc = Path("ticdc/ticdc-debezium.md").read_text()
source_url = "https://raw.githubusercontent.com/pingcap/tiflow/master/pkg/sink/codec/debezium/codec.go"
source = urllib.request.urlopen(source_url, timeout=15).read().decode()
expected = "io.debezium.connector.mysql.SchemaChangeValue"
assert expected in source, "serializer literal not found"
assert doc.count(expected) == 1, "expected one matching schema-name occurrence in the document"
assert '{cluster-name}.{schema-name}.{table-name}.SchemaChangeValue' in doc
print("serializer DDL value schema name:", expected)
print("matching JSON example occurrences:", doc.count(expected))
print("conflicting table-format occurrences:", doc.count('{cluster-name}.{schema-name}.{table-name}.SchemaChangeValue'))
for line_no, line in enumerate(doc.splitlines(), 1):
if "schema.name" in line and "SchemaChangeValue" in line:
print(f"documentation line {line_no}: {line}")
PY
printf '%s\n' '--- serializer context ---'
curl -LfsS --max-time 15 \
'https://raw.githubusercontent.com/pingcap/tiflow/master/pkg/sink/codec/debezium/codec.go' |
sed -n '1388,1410p'Repository: pingcap/docs
Length of output: 2436
🏁 Script executed:
#!/bin/bash
set -eu
tmpfile="$(mktemp)"
trap 'rm -f "$tmpfile"' EXIT
curl -LfsS --max-time 15 \
'https://raw.githubusercontent.com/pingcap/tiflow/master/pkg/sink/codec/debezium/codec.go' \
> "$tmpfile"
python3 - "$tmpfile" <<'PY'
from pathlib import Path
import sys
doc = Path("ticdc/ticdc-debezium.md").read_text()
source = Path(sys.argv[1]).read_text()
expected = "io.debezium.connector.mysql.SchemaChangeValue"
qualified = "{cluster-name}.{schema-name}.{table-name}.SchemaChangeValue"
assert expected in source
assert doc.count(expected) == 1
assert doc.count(qualified) == 1
print("serializer literal:", expected)
print("matching JSON example occurrences:", doc.count(expected))
print("conflicting table-format occurrences:", doc.count(qualified))
for number, line in enumerate(doc.splitlines(), 1):
if "schema.name" in line and "SchemaChangeValue" in line:
print(f"documentation line {number}: {line}")
PY
printf '%s\n' '--- serializer context ---'
sed -n '1398,1405p' "$tmpfile"Repository: pingcap/docs
Length of output: 869
Document the emitted DDL schema name.
The TiCDC Debezium serializer emits io.debezium.connector.mysql.SchemaChangeValue, which matches the JSON example. Update the schema.name table entry instead of changing the example.
Suggested replacement
| "schema": { | |
| "optional": false, | |
| "type": "struct", | |
| "version": 1, | |
| "name": "io.debezium.connector.mysql.SchemaChangeValue", | |
| "fields": [ | |
| | `schema.name` | String | The name of the schema: `io.debezium.connector.mysql.SchemaChangeValue`. | |
Source: Path instructions
| The key fields of the preceding JSON data are explained as follows: | ||
|
|
||
| | Field | Type | Description | | ||
| |:----------|:-------|:-------------------------------------------------------| | ||
| | `payload.ts_ms` | Number | The timestamp (in milliseconds) when TiCDC generates this message. | | ||
| | `payload.ddl` | String | The SQL statement of the DDL event. | | ||
| | `payload.databaseName` | String | The name of the database where the event occurs. | | ||
| | `payload.source.commit_ts` | Number | The `CommitTs` value of the event. | | ||
| | `payload.source.db` | String | The name of the database where the event occurs. | | ||
| | `payload.source.table` | String | The name of the table where the event occurs. | | ||
| | `payload.tableChanges` | Array | A structured representation of the entire table schema after the schema change. The `tableChanges` field contains an array that includes entries for each column of the table. Because the structured representation presents data in JSON or Avro format, consumers can easily read messages without first processing them through a DDL parser. | | ||
| | `payload.tableChanges.type` | String | Describes the kind of change. The value is one of the following: `CREATE`, indicating that the table is created; `ALTER`, indicating that the table is modified; `DROP`, indicating that the table is deleted. | | ||
| | `payload.tableChanges.id` | String | Full identifier of the table that was created, altered, or dropped. In the case of a table rename, this identifier is a concatenation of `<old>` and `<new>` table names. | |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win
Describe tableChanges as table changes, not column entries.
The JSON schema shows array entries with type, id, and table. Column metadata is nested under table.columns. The current description incorrectly says that the array contains entries for each column.
Committable replacement
-| `payload.tableChanges` | Array | A structured representation of the entire table schema after the schema change. The `tableChanges` field contains an array that includes entries for each column of the table. Because the structured representation presents data in JSON or Avro format, consumers can easily read messages without first processing them through a DDL parser. |
+| `payload.tableChanges` | Array | A structured representation of the table schema affected by the schema change. Each array entry describes one table change and can include column metadata in its nested `table.columns` field. Because the structured representation presents data in JSON or Avro format, consumers can read messages without first processing them through a DDL parser. |📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| The key fields of the preceding JSON data are explained as follows: | |
| | Field | Type | Description | | |
| |:----------|:-------|:-------------------------------------------------------| | |
| | `payload.ts_ms` | Number | The timestamp (in milliseconds) when TiCDC generates this message. | | |
| | `payload.ddl` | String | The SQL statement of the DDL event. | | |
| | `payload.databaseName` | String | The name of the database where the event occurs. | | |
| | `payload.source.commit_ts` | Number | The `CommitTs` value of the event. | | |
| | `payload.source.db` | String | The name of the database where the event occurs. | | |
| | `payload.source.table` | String | The name of the table where the event occurs. | | |
| | `payload.tableChanges` | Array | A structured representation of the entire table schema after the schema change. The `tableChanges` field contains an array that includes entries for each column of the table. Because the structured representation presents data in JSON or Avro format, consumers can easily read messages without first processing them through a DDL parser. | | |
| | `payload.tableChanges.type` | String | Describes the kind of change. The value is one of the following: `CREATE`, indicating that the table is created; `ALTER`, indicating that the table is modified; `DROP`, indicating that the table is deleted. | | |
| | `payload.tableChanges.id` | String | Full identifier of the table that was created, altered, or dropped. In the case of a table rename, this identifier is a concatenation of `<old>` and `<new>` table names. | | |
| The key fields of the preceding JSON data are explained as follows: | |
| | Field | Type | Description | | |
| |:----------|:-------|:-------------------------------------------------------| | |
| | `payload.ts_ms` | Number | The timestamp (in milliseconds) when TiCDC generates this message. | | |
| | `payload.ddl` | String | The SQL statement of the DDL event. | | |
| | `payload.databaseName` | String | The name of the database where the event occurs. | | |
| | `payload.source.commit_ts` | Number | The `CommitTs` value of the event. | | |
| | `payload.source.db` | String | The name of the database where the event occurs. | | |
| | `payload.source.table` | String | The name of the table where the event occurs. | | |
| | `payload.tableChanges` | Array | A structured representation of the table schema affected by the schema change. Each array entry describes one table change and can include column metadata in its nested `table.columns` field. Because the structured representation presents data in JSON or Avro format, consumers can read messages without first processing them through a DDL parser. | | |
| | `payload.tableChanges.type` | String | Describes the kind of change. The value is one of the following: `CREATE`, indicating that the table is created; `ALTER`, indicating that the table is modified; `DROP`, indicating that the table is deleted. | | |
| | `payload.tableChanges.id` | String | Full identifier of the table that was created, altered, or dropped. In the case of a table rename, this identifier is a concatenation of `<old>` and `<new>` table names. | |
| | `payload` | JSON | The information about primary key or unique index columns. The key and value in each field represent the column name and its current value, respectively. | | ||
| | `schema.fields` | JSON | The type information of each field in the payload, including the schema information of the row data before and after the change. | |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win
Correct the schema.fields descriptions for the key and WATERMARK payloads.
The DML key schema describes only key fields. The WATERMARK schema has no before or after fields. These descriptions appear to be copied from the DML value section.
Committable replacements
-| `schema.fields` | JSON | The type information of each field in the payload, including the schema information of the row data before and after the change. |
+| `schema.fields` | JSON | The type information of each field in the key payload. |-| `schema.fields` | JSON | The type information of each field in the payload, including the schema information of the row data before and after the change. |
+| `schema.fields` | JSON | The type information of each field in the WATERMARK payload. |Also applies to: 793-798
|
|
||
| - Debezium converts `charsetName` to `"utf8mb4"` when the column collation is `"utf8_unicode_ci"` and the character set is null, but TiCDC does not. | ||
|
|
||
| - TiCDC treats `\` as an escaped quotation in ENUM elements, but Debezium does not. For example, TiCDC encodes ENUM elements like `("c,\'d','g,''h")` to `('c,'d', 'g,''h')`. |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟡 Minor | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -eu
printf '%s\n' '--- document context ---'
sed -n '790,840p' ticdc/ticdc-debezium.md
printf '%s\n' '--- relevant ENUM references ---'
rg -n -i 'enum|escaped quotation|g,''h|c,.*d' --glob '!vendor/**' --glob '!node_modules/**' .Repository: pingcap/docs
Length of output: 50369
🏁 Script executed:
#!/bin/bash
set -eu
printf '%s\n' '--- repository files related to TiCDC Debezium ---'
git ls-files | rg '(^|/)(ticdc|debezium|kafka)' | head -200
printf '%s\n' '--- exact phrase in repository history metadata ---'
git grep -n -F 'TiCDC treats' -- ':!ticdc/ticdc-debezium.md' || trueRepository: pingcap/docs
Length of output: 3847
🌐 Web query:
TiCDC Debezium ENUM escaped quotation serializer "c,\\'d"
💡 Result:
In TiCDC, when using the Debezium protocol, string-based data types—including ENUM types—are serialized according to standard JSON encoding rules, which require specific characters like double quotes (") and backslashes () to be escaped [1][2][3]. Regarding the specific case of an ENUM value containing an escaped quote or similar characters (such as c,'d), TiCDC's serialization behavior aligns with standard JSON requirements: 1. Serialization: When TiCDC serializes data into the Debezium format, it treats string-like columns as JSON strings [1]. Characters that are not "safe" for raw JSON inclusion—such as double quotes, backslashes, and control characters—must be escaped with a backslash [2][3]. For example, a literal double quote (") is escaped as ", and a literal backslash is escaped as \ [2]. 2. ENUM Handling: TiDB's internal representation and TiCDC's mapping encode ENUM types as strings [4]. If your ENUM data contains problematic characters, the serializer will apply standard escape sequences to ensure the resulting JSON message is valid [2]. 3. Deserialization: Consumers of these messages are responsible for deserialization [5]. Because the output follows the standard Debezium/JSON structure, standard libraries (like Jackson in Java/Kafka Connect or native Go/Python JSON decoders) will correctly unescape these sequences (e.g., converting " back to ") [5][6]. If you are encountering issues where c,'d appears literal or incorrectly formatted, verify if the downstream consumer is expecting a specific flavor of escaping (like MySQL-style versus standard JSON-RFC 8259). TiCDC typically follows JSON standard rules [2]. If you find discrepancies, ensure that the downstream Kafka sink/connector is configured to handle standard JSON string escaping [5].
Citations:
- 1: https://docs.pingcap.com/tidb/stable/ticdc-debezium/
- 2: https://github.com/pingcap/tidb/blob/02ceded78bac34285d3ce1aceda86b848bad673d/types/json_binary.go
- 3: https://github.com/pingcap/tidb/blob/02ceded78bac34285d3ce1aceda86b848bad673d/types/json_constants.go
- 4: https://docs.pingcap.com/tidb/stable/ticdc-avro-protocol/
- 5: https://debezium.io/documentation/reference/stable/integrations/serdes.html
- 6: https://tikv.github.io/doc/src/tidb_query_datatype/codec/mysql/json/json_unquote.rs.html
🏁 Script executed:
#!/bin/bash
set -eu
tmp=$(mktemp)
trap 'rm -f "$tmp"' EXIT
curl -fsSL 'https://api.github.com/repos/pingcap/ticdc/git/trees/master?recursive=1' > "$tmp"
jq -r '.tree[] | select(.type == "blob") | .path' "$tmp" |
rg -i 'debezium|json.*(encoder|marshal|encode)|encoder|sink.*(test|go)$' |
head -200Repository: pingcap/docs
Length of output: 201
🏁 Script executed:
#!/bin/bash
set -eu
for query in \
'https://api.github.com/search/code?q=ENUM+repo%3Apingcap%2Fticdc' \
'https://api.github.com/search/issues?q=%22escaped+quotation%22+repo%3Apingcap%2Fticdc'
do
printf '%s\n' "--- $query ---"
curl -fsSL "$query" | jq -r 'if .message then .message else (.items[]? | [.html_url, .title] | `@tsv`) end' | head -50
doneRepository: pingcap/docs
Length of output: 366
🌐 Web query:
site:github.com/pingcap/ticdc Debezium ENUM encoder test
💡 Result:
There is no specific "Debezium ENUM encoder test" widely recognized as a standalone component or public utility [1][2][3]. In the context of Debezium and related CDC (Change Data Capture) systems—particularly within projects like TiCDC—handling of data types such as ENUM, SET, and others is typically managed within the codec's broader encoding logic rather than through a dedicated ENUM-specific encoder [1][4][5]. When testing ENUM types in systems that use Debezium protocols, developers typically employ comprehensive integration tests that include various data types—including ENUMs—to ensure proper serialization and deserialization [1][5]. Key observations regarding Debezium encoding and testing in this ecosystem include: 1. Data Type Handling: Debezium encoders, such as those found in the TiCDC project, generally handle ENUMs and other complex types as part of the unified column value conversion logic [5]. This logic is often verified by extensive test suites that cover edge cases like NULL values, numeric precision, and various schema definitions [1][5]. 2. Testing Methodology: Rather than testing an "ENUM encoder" in isolation, testing is performed by injecting varied workloads (e.g., SQL scripts containing ENUM definitions) and verifying the output against the expected Debezium or Avro format [1][3]. 3. Error Handling and Debugging: Recent improvements in Debezium encoders have focused on enhancing error logging context (e.g., capturing the schema, table, column, and value when encoding fails) rather than specific ENUM handling, reflecting a focus on robustness during the conversion of complex data types [4][6]. If you are encountering issues with ENUM values in a Debezium-based pipeline, it is recommended to verify your schema registry configuration (if using Avro) [1] or check the underlying codec implementation for specific type conversion logic [5]. If you are looking to write a test, you should create a test case that inserts data into a table with an ENUM column and asserts that the resulting Debezium event payload contains the correct string representation of the ENUM value.
Citations:
- 1: sink: add debezium-avro protocol ticdc#5475
- 2: codec: fix debezium protocol ticdc#1342
- 3: pingcap/ticdc@db2ad80
- 4: codec(ticdc): improve error logging for Debezium encoding failures ticdc#3963
- 5: common: correct Duration type precision ticdc#4249
- 6: Improve error logging for Debezium encoding failures ticdc#3964
🌐 Web query:
site:github.com/pingcap/ticdc "escaped" "ENUM"
💡 Result:
The term "escaped ENUM" is not a standard, universally recognized technical term in computer science or software engineering. In the context of database systems and programming, it typically refers to the process of properly sanitizing or quoting an ENUM value (or an identifier that happens to be an ENUM type) to prevent ambiguity, syntax errors, or security vulnerabilities (such as SQL injection) [1]. Technical nuances regarding the handling of ENUMs and identifiers include: 1. Identifier Quoting: When database objects (like tables, columns, or specifically named ENUM values) are used in queries, they often need to be "escaped" or quoted (e.g., using backticks in MySQL/TiDB or double quotes in PostgreSQL) if they contain special characters, are reserved keywords, or match the name of an ENUM type [2][1]. Failing to do so can lead to syntax errors or incorrect parsing [2]. 2. Serialization and Protocol Handling: In data replication or communication protocols (such as those used by database CDC tools), ENUM values must be correctly serialized and deserialized [3][4]. Issues often arise when the underlying system fails to distinguish between the ENUM value itself and its string representation, or when the serialization layer does not correctly handle special characters, requiring them to be escaped or mapped to a safe internal format [3]. 3. Security: Improperly handling or failing to escape identifiers—even those used in DDL operations—can create vulnerabilities where an attacker could potentially execute arbitrary code or bypass security constraints [1]. If you encountered the phrase in a specific codebase, it likely refers to a custom implementation-specific helper function or a logic flow designed to handle strings that are intended to be interpreted as ENUM constants but require processing to remain valid within the surrounding syntax [1][3].
Citations:
- 1: schemastore: fix rename tables when tidb version <= v8.1.x ticdc#4388
- 2: exchange partition query rebuild corrupts partition names containing backticks ticdc#4450
- 3: In canal-json codec, the string
"null"is translated tonullticdc#3958 - 4: event: assert DDLEvent.NotSync survives Marshal Unmarshal ticdc#4420
🌐 Web query:
site:github.com/pingcap/ticdc "debezium" "enum"
💡 Result:
Debezium handles database ENUM types by mapping them to string representations in its output events [1][2]. When working with systems like TiCDC, which implements Debezium protocols, ENUM column values are processed as standard string data during the encoding of row events [1][2]. In the context of Debezium encoding, specific handling is often required to ensure data integrity and readability, particularly when debugging issues. Recent improvements have enhanced error logging for Debezium encoding failures to provide detailed context, including the specific table, column, and the actual value that caused the error [1][3]. This helps distinguish between different data types—such as strings, blobs, and other specialized types—when encoding fails, allowing for better identification of problematic values [1]. While Debezium generally treats ENUMs as strings, developers should be aware that different codecs (such as Canal-JSON or Avro) may handle nulls or specific string values differently, which can sometimes lead to conflation between actual data and null values if not carefully managed [4][5]. When configuring sinks to use Debezium or Debezium-Avro protocols, it is important to ensure that the system is correctly identifying the protocol type to maintain expected data serialization behavior [4].
Citations:
- 1: codec(ticdc): improve error logging for Debezium encoding failures ticdc#3963
- 2: common: correct Duration type precision ticdc#4249
- 3: Improve error logging for Debezium encoding failures ticdc#3964
- 4: sink: add debezium-avro protocol ticdc#5475
- 5: In canal-json codec, the string
"null"is translated tonullticdc#3958
🏁 Script executed:
#!/bin/bash
set -eu
for pr in 4249 3963 5475 1342; do
printf '%s\n' "--- pingcap/ticdc PR $pr ---"
curl -fsSL "https://github.com/pingcap/ticdc/pull/$pr.patch" |
rg -n -i -C 4 'enum|debezium|escape|quotation|quote' |
head -120 || true
doneRepository: pingcap/docs
Length of output: 14182
🏁 Script executed:
#!/bin/bash
set -eu
base='https://raw.githubusercontent.com/pingcap/ticdc/master/pkg/sink/codec/debezium'
for file in codec.go helper.go codec_test.go; do
printf '%s\n' "--- $file ---"
curl -fsSL "$base/$file" |
rg -n -i -C 8 'enum|write.*field|ExtractColVal|escape|quote|String' |
head -220 || true
doneRepository: pingcap/docs
Length of output: 20938
🏁 Script executed:
#!/bin/bash
set -eu
for path in \
pkg/sink/codec/common/common.go \
pkg/sink/codec/common/helper.go \
pkg/sink/codec/common/codec.go \
pkg/sink/codec/common/config.go \
pkg/sink/codec/common/encoder.go \
pkg/sink/codec/common/util.go
do
content=$(curl -fsSL "https://raw.githubusercontent.com/pingcap/ticdc/master/$path" || true)
if printf '%s' "$content" | rg -q 'EscapeEnumAndSetOptions'; then
printf '%s\n' "--- $path ---"
printf '%s' "$content" |
rg -n -C 20 'EscapeEnumAndSetOptions'
fi
doneRepository: pingcap/docs
Length of output: 1797
🌐 Web query:
"EscapeEnumAndSetOptions" TiCDC
💡 Result:
There is no configuration parameter named EscapeEnumAndSetOptions in TiCDC. It is likely that this query stems from a misunderstanding of how TiCDC handles special characters or data types: 1. Character Escaping in URI Parameters: TiCDC documentation frequently advises that if sink URI parameters (such as passwords, usernames, or file paths) contain special characters (e.g.,! * ';: @ & = + $, /? % # [ ]), these characters must be URI-encoded [1][2]. This is a general requirement for all URI-based configurations in TiCDC, not a setting specifically for ENUM or SET types. 2. Handling ENUM and SET Types: TiCDC supports the replication of ENUM and SET data types by converting them to string representations in downstream protocols like CSV [3]. When using specific protocols like Canal-JSON, TiCDC exports these types with their base MySQL type (e.g., "enum", "set"), though it may omit the full type parameter information (e.g., the specific allowed values) that is present in the official Canal implementation [4]. If you are encountering issues with ENUM or SET values during replication, ensure that your downstream database schema is compatible with the values TiCDC is sending, or check the documentation for the specific sink protocol you are using (e.g., MySQL, Kafka, or Cloud Storage) for any type-mapping requirements [2][3][5].
Citations:
- 1: https://docs.pingcap.com/tidb/dev/ticdc-changefeed-config/
- 2: https://docs.pingcap.com/tidb/stable/ticdc-sink-to-mysql/
- 3: https://docs.pingcap.com/tidb/stable/ticdc-csv/
- 4: https://pingkai.cn/docs/tidb/stable/ticdc-canal-json
- 5: https://website-docs-preview.vercel.app/tidb/stable/ticdc-sink-to-cloud-storage/
🏁 Script executed:
#!/bin/bash
set -eu
python3 - <<'PY'
def escape_enum_and_set_option(option):
option = option.replace(",", "\\,")
option = option.replace("\\'", "'")
option = option.replace("''", "'")
return option
cases = [
r"c,\'d",
r"g,''h",
r"c,\'d','g,''h",
]
for case in cases:
print(f"{case!r} -> {escape_enum_and_set_option(case)!r}")
PY
printf '%s\n' '--- Debezium parser reference ---'
curl -fsSL \
'https://raw.githubusercontent.com/debezium/debezium/9f7ede0e0695f012c6c4e715e96aed85eecf6b5f/debezium-connector-mysql/src/main/java/io/debezium/connector/mysql/antlr/MySqlAntlrDdlParser.java' |
sed -n '350,395p'Repository: pingcap/docs
Length of output: 1889
Correct the ENUM escaping statement and example.
TiCDC and Debezium use the same escaping rules: , becomes \,, \' becomes ', and '' becomes '. The claim that they handle these escapes differently is incorrect. The value ('c,'d', 'g,''h') also does not show the escaped commas.
Source: Path instructions
1 participant
What is changed, added or deleted? (Required)
Update the TiCDC Debezium support for DDL, WATERMARK events in new and classic TiCDC architectures.
Which TiDB version(s) do your changes apply to? (Required)
Tips for choosing the affected version(s):
By default, CHOOSE MASTER ONLY so your changes will be applied to the next TiDB major or minor releases. If your PR involves a product feature behavior change or a compatibility change, CHOOSE THE AFFECTED RELEASE BRANCH(ES) AND MASTER.
For details, see tips for choosing the affected versions.
What is the related PR or file link(s)?
AI agent involvement
Do your changes match any of the following descriptions?
Summary by CodeRabbit