Skip to content

docs: remove mode.streaming from data sync guides #1245

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
HTHou merged 1 commit into from
Nov 9, 2026
+4 −52
Merged

docs: remove mode.streaming from data sync guides #1245

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 1 addition & 9 deletions src/UserGuide/Master/Table/User-Manual/Data-Sync_apache.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -349,15 +349,14 @@ This example demonstrates synchronizing data within a specific historical time r

![](/img/e2.png)

In this example, we create a synchronization task named `A2B`. First, we define the data range in the `source` configuration. Since we are synchronizing historical data (data that existed before the task was created), we need to configure the start time (`start-time`), end time (`end-time`), and the streaming mode (`mode.streaming`). The `node-urls` parameter is configured with the URL of the DataNode service port on the target IoTDB.
In this example, we create a synchronization task named `A2B`. First, we define the data range in the `source` configuration. Since we are synchronizing historical data (data that existed before the task was created), we need to configure the start time (`start-time`), end time (`end-time`). The `node-urls` parameter is configured with the URL of the DataNode service port on the target IoTDB.

SQL Example:

```SQL
CREATE PIPE A2B
WITH SOURCE (
'source' = 'iotdb-source',
'mode.streaming' = 'true',
'database-name'='testdb.*',
'start-time' = '2023.08.23T08:00:00+00:00',
'end-time' = '2023.10.23T08:00:00+00:00'
Expand All @@ -370,7 +369,6 @@ WITH SINK (

Where:

- `mode.streaming`: Extraction mode for newly inserted data (after the pipe is created): Whether to extract data in streaming mode (if set to false, batch mode is used).
- `database-name`: Scope of Data Synchronization
- `start-time`: The event time at which data synchronization starts (inclusive).
- `end-time`: The event time at which data synchronization ends (inclusive).
Expand Down Expand Up @@ -576,7 +574,6 @@ pipe_all_sinks_rate_limit_bytes_per_second=-1
| source | iotdb-source | String: iotdb-source | Yes | - |
| inclusion | Used to specify the range of data to be synchronized in the data synchronization task, including data, schema and auth | String:all, data(insert,delete), schema(database,table,ttl), auth | No | data.insert |
| inclusion.exclusion | Used to exclude specific operations from the range specified by inclusion, reducing the amount of data synchronized | String:all, data(insert,delete), schema(database,table,ttl), auth | No | - |
| mode.streaming | This parameter specifies the source of time-series data capture. It applies to scenarios where `mode.streaming` is set to `false`, determining the capture source for `data.insert` in `inclusion`. Two capture strategies are available: - **true**: Dynamically selects the capture type. The system adapts to downstream processing speed, choosing between capturing each write request or only capturing TsFile file sealing requests. When downstream processing is fast, write requests are prioritized to reduce latency; when processing is slow, only file sealing requests are captured to prevent processing backlogs. This mode suits most scenarios, optimizing the balance between processing latency and throughput. - **false**: Uses a fixed batch capture approach, capturing only TsFile file sealing requests. This mode is suitable for resource-constrained applications, reducing system load. **Note**: Snapshot data captured when the pipe starts will only be provided for downstream processing as files. | Boolean: true / false | No | true |
| mode.strict | Determines whether to strictly filter data when using the `time`, `path`, `database-name`, or `table-name` parameters: - **true**: Strict filtering. The system will strictly filter captured data according to the specified conditions, ensuring that only matching data is selected. - **false**: Non-strict filtering. Some extra data may be included during the selection process to optimize performance and reduce CPU and I/O consumption. | Boolean: true / false | No | true |
| mode.snapshot | This parameter determines the data capture mode, affecting the `data` in `inclusion`. Two modes are available: - **true**: Static data capture. A one-time data snapshot is taken when the pipe starts. Once the snapshot data is fully consumed, the pipe automatically terminates (executing `DROP PIPE` SQL automatically). - **false**: Dynamic data capture. In addition to capturing snapshot data when the pipe starts, it continuously captures subsequent data changes. The pipe remains active to process the dynamic data stream. | Boolean: true / false | No | false |
| database-name | When the user connects with `sql_dialect` set to `table`, this parameter can be specified. Determines the scope of data capture, affecting the `data` in `inclusion`. Specifies the database name to filter. It can be a specific database name or a Java-style regular expression to match multiple databases. By default, all databases are matched. | String: Database name or database regular expression pattern string, which can match uncreated or non - existent databases. | No | ".*" |
Expand All @@ -586,11 +583,6 @@ pipe_all_sinks_rate_limit_bytes_per_second=-1
| mods | Same as mods.enable, whether to send the MODS file for TSFile. | Boolean: true / false | No | false |
| skipIf | Which errors can be skipped? Currently only the insufficient privileges error. | String:no-privileges | No | no-privileges |

> 💎 **Note:** The difference between the values of true and false for the data extraction mode `mode.streaming`
>
> - True (recommended): Under this value, the task will process and send the data in real-time. Its characteristics are high timeliness and low throughput.
> - False: Under this value, the task will process and send the data in batches (according to the underlying data files). Its characteristics are low timeliness and high throughput.

### sink parameter

#### iotdb-thrift-sink
Expand Down
Loading
Loading