Skip to content

ticdc: document ignore-update-only-columns filter #23523

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
lidezhu wants to merge 1 commit into
base: master
Choose a base branch
from
+26 −0
Open

ticdc: document ignore-update-only-columns filter #23523

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions ticdc/ticdc-changefeed-config.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -146,6 +146,13 @@ For more information, see [Event filter rules](/ticdc/ticdc-filter.md#event-filt

- `ignore-update-new-value-expr = "gender = 'male'"` ignores `UPDATE` DMLs whose new value contains `gender = 'male'`

##### `ignore-update-only-columns` <span class="version-mark">Introduced in v8.5.8</span>

- This configuration item is an array of column names and applies only to Kafka downstreams.
- `ignore-update-only-columns = ["version", "updated_at"]` filters out an `UPDATE` DML event when the event changes only the `version` column, the `updated_at` column, or both. If the event also changes a column that is not in the list, TiCDC sends the event to the downstream.
- If the changed columns include a primary key or unique key column, TiCDC does not filter out the event, regardless of whether the column is in the list.
- For non-Kafka downstreams, TiCDC accepts this configuration but does not filter events based on it.

### scheduler

#### `enable-table-across-nodes`
Expand Down
9 changes: 9 additions & 0 deletions ticdc/ticdc-filter.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -50,6 +50,7 @@ ignore-delete-value-expr = "name = 'john'" # Ignore delete DMLs that contain the
ignore-insert-value-expr = "id >= 100" # Ignore insert DMLs that contain the condition "id >= 100".
ignore-update-old-value-expr = "age < 18 or name = 'lili'" # Ignore update DMLs whose old value contains "age < 18" or "name = 'lili'".
ignore-update-new-value-expr = "gender = 'male' and age > 18" # Ignore update DMLs whose new value contains "gender = 'male'" and "age > 18".
ignore-update-only-columns = ["version", "updated_at"] # Ignore update DMLs if every changed column is either version or updated_at.
```

Description of configuration parameters:
Expand Down Expand Up @@ -122,6 +123,14 @@ Description of configuration parameters:
- `ignore-insert-value-expr`: this parameter accepts a SQL expression that follows the default SQL mode, used to filter out the `INSERT` type of DML events with a specified value.
- `ignore-update-old-value-expr`: this parameter accepts a SQL expression that follows the default SQL mode, used to filter out the `UPDATE` type of DML events with a specified old value.
- `ignore-update-new-value-expr`: this parameter accepts a SQL expression that follows the default SQL mode, used to filter out the `UPDATE` DML events with a specified new value.
- `ignore-update-only-columns` <span class="version-mark">Introduced in v8.5.8</span>: specifies a set of column names. For tables matching `matcher`, if all columns whose values are changed in an `UPDATE` event are included in this list, TiCDC filters out the event. If any changed column is not included in the list, TiCDC sends the event to the downstream. This configuration applies only to Kafka downstreams and does not affect `INSERT` or `DELETE` events.

When using this configuration, note the following:

- TiCDC does not filter out an `UPDATE` event if it changes a primary key or unique key column, even if the column is included in the list.
- The `case-sensitive` changefeed configuration item controls whether column name matching is case-sensitive.
- If a column in the list does not exist in a matching table, TiCDC outputs a warning log and ignores the column. Other valid columns in the list are still used to determine whether to filter out the event.
- For non-Kafka downstreams, TiCDC accepts this configuration but does not filter events based on it.

> **Note:**
>
Expand Down
10 changes: 10 additions & 0 deletions ticdc/ticdc-open-api-v2.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -169,6 +169,9 @@ This interface is used to submit a replication task to TiCDC. If the request is
],
"ignore_update_new_value_expr": "string",
"ignore_update_old_value_expr": "string",
"ignore_update_only_columns": [
"string"
],
"matcher": [
"string"
]
Expand Down Expand Up @@ -292,6 +295,7 @@ The `filter.event_filters` parameters are described as follows. For more informa
| `ignore_sql` | `STRING ARRAY` type. For example, `["^drop", "add column"]` means to filter out DDL statements that start with `DROP` or contain `ADD COLUMN`. (Optional) |
| `ignore_update_new_value_expr` | `STRING ARRAY` type. For example, `"gender = 'male'"` means to filter out the UPDATE DML statements with the new value `gender = 'male'`. (Optional) |
| `ignore_update_old_value_expr` | `STRING ARRAY` type. For example, `"age < 18"` means to filter out the UPDATE DML statements with the old value `age < 18`. (Optional) |
| `ignore_update_only_columns` | `STRING ARRAY` type. Introduced in v8.5.8 and applicable only to Kafka downstreams. If all columns whose values are changed in an UPDATE event are included in this array, TiCDC filters out the event. For example, `["version", "updated_at"]`. (Optional) |

@coderabbitai coderabbitai Bot Aug 13, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🗄️ Data Integrity & Integration | 🟡 Minor | ⚡ Quick win

Document the primary/unique-key exception.

Line 298 states that every matching update is filtered. This conflicts with ticdc/ticdc-filter.md: TiCDC does not filter the event when a changed column is a primary key or unique key column.

As per path instructions: “For every actionable issue, provide a GitHub committable suggestion block when the fix can be safely applied to contiguous lines.”

Apply this suggestion
Suggested change
| `ignore_update_only_columns` | `STRING ARRAY` type. Introduced in v8.5.8 and applicable only to Kafka downstreams. If all columns whose values are changed in an UPDATE event are included in this array, TiCDC filters out the event. For example, `["version", "updated_at"]`. (Optional) |
| `ignore_update_only_columns` | `STRING ARRAY` type. Introduced in v8.5.8 and applicable only to Kafka downstreams. If all columns whose values are changed in an `UPDATE` event are included in this array, TiCDC filters out the event, except when the event changes a primary key or unique key column. For example, `["version", "updated_at"]`. (Optional) |

Source: Path instructions

| `matcher` | `STRING ARRAY` type. It works as an allowlist. For example, `["test.worker"]` means that the filter rule applies only to the `worker` table in the `test` database. (Optional) |

The `mounter` parameter is described as follows:
Expand Down Expand Up @@ -415,6 +419,9 @@ If the request is successful, `200 OK` is returned. If the request fails, an err
],
"ignore_update_new_value_expr": "string",
"ignore_update_old_value_expr": "string",
"ignore_update_only_columns": [
"string"
],
"matcher": [
"string"
]
Expand Down Expand Up @@ -600,6 +607,9 @@ To modify the changefeed configuration, follow the steps of `pause the replicati
],
"ignore_update_new_value_expr": "string",
"ignore_update_old_value_expr": "string",
"ignore_update_only_columns": [
"string"
],
"matcher": [
"string"
]
Expand Down