diff --git a/src/UserGuide/Master/Table/User-Manual/Data-Sync_apache.md b/src/UserGuide/Master/Table/User-Manual/Data-Sync_apache.md index b05adcd81..af1ed1fd1 100644 --- a/src/UserGuide/Master/Table/User-Manual/Data-Sync_apache.md +++ b/src/UserGuide/Master/Table/User-Manual/Data-Sync_apache.md @@ -349,7 +349,7 @@ This example demonstrates synchronizing data within a specific historical time r ![](/img/e2.png) -In this example, we create a synchronization task named `A2B`. First, we define the data range in the `source` configuration. Since we are synchronizing historical data (data that existed before the task was created), we need to configure the start time (`start-time`), end time (`end-time`), and the streaming mode (`mode.streaming`). The `node-urls` parameter is configured with the URL of the DataNode service port on the target IoTDB. +In this example, we create a synchronization task named `A2B`. First, we define the data range in the `source` configuration. Since we are synchronizing historical data (data that existed before the task was created), we need to configure the start time (`start-time`), end time (`end-time`). The `node-urls` parameter is configured with the URL of the DataNode service port on the target IoTDB. SQL Example: @@ -357,7 +357,6 @@ SQL Example: CREATE PIPE A2B WITH SOURCE ( 'source' = 'iotdb-source', - 'mode.streaming' = 'true', 'database-name'='testdb.*', 'start-time' = '2023.08.23T08:00:00+00:00', 'end-time' = '2023.10.23T08:00:00+00:00' @@ -370,7 +369,6 @@ WITH SINK ( Where: -- `mode.streaming`: Extraction mode for newly inserted data (after the pipe is created): Whether to extract data in streaming mode (if set to false, batch mode is used). - `database-name`: Scope of Data Synchronization - `start-time`: The event time at which data synchronization starts (inclusive). - `end-time`: The event time at which data synchronization ends (inclusive). @@ -576,7 +574,6 @@ pipe_all_sinks_rate_limit_bytes_per_second=-1 | source | iotdb-source | String: iotdb-source | Yes | - | | inclusion | Used to specify the range of data to be synchronized in the data synchronization task, including data, schema and auth | String:all, data(insert,delete), schema(database,table,ttl), auth | No | data.insert | | inclusion.exclusion | Used to exclude specific operations from the range specified by inclusion, reducing the amount of data synchronized | String:all, data(insert,delete), schema(database,table,ttl), auth | No | - | -| mode.streaming | This parameter specifies the source of time-series data capture. It applies to scenarios where `mode.streaming` is set to `false`, determining the capture source for `data.insert` in `inclusion`. Two capture strategies are available: - **true**: Dynamically selects the capture type. The system adapts to downstream processing speed, choosing between capturing each write request or only capturing TsFile file sealing requests. When downstream processing is fast, write requests are prioritized to reduce latency; when processing is slow, only file sealing requests are captured to prevent processing backlogs. This mode suits most scenarios, optimizing the balance between processing latency and throughput. - **false**: Uses a fixed batch capture approach, capturing only TsFile file sealing requests. This mode is suitable for resource-constrained applications, reducing system load. **Note**: Snapshot data captured when the pipe starts will only be provided for downstream processing as files. | Boolean: true / false | No | true | | mode.strict | Determines whether to strictly filter data when using the `time`, `path`, `database-name`, or `table-name` parameters: - **true**: Strict filtering. The system will strictly filter captured data according to the specified conditions, ensuring that only matching data is selected. - **false**: Non-strict filtering. Some extra data may be included during the selection process to optimize performance and reduce CPU and I/O consumption. | Boolean: true / false | No | true | | mode.snapshot | This parameter determines the data capture mode, affecting the `data` in `inclusion`. Two modes are available: - **true**: Static data capture. A one-time data snapshot is taken when the pipe starts. Once the snapshot data is fully consumed, the pipe automatically terminates (executing `DROP PIPE` SQL automatically). - **false**: Dynamic data capture. In addition to capturing snapshot data when the pipe starts, it continuously captures subsequent data changes. The pipe remains active to process the dynamic data stream. | Boolean: true / false | No | false | | database-name | When the user connects with `sql_dialect` set to `table`, this parameter can be specified. Determines the scope of data capture, affecting the `data` in `inclusion`. Specifies the database name to filter. It can be a specific database name or a Java-style regular expression to match multiple databases. By default, all databases are matched. | String: Database name or database regular expression pattern string, which can match uncreated or non - existent databases. | No | ".*" | @@ -586,11 +583,6 @@ pipe_all_sinks_rate_limit_bytes_per_second=-1 | mods | Same as mods.enable, whether to send the MODS file for TSFile. | Boolean: true / false | No | false | | skipIf | Which errors can be skipped? Currently only the insufficient privileges error. | String:no-privileges | No | no-privileges | -> 💎 **Note:** The difference between the values of true and false for the data extraction mode `mode.streaming` -> -> - True (recommended): Under this value, the task will process and send the data in real-time. Its characteristics are high timeliness and low throughput. -> - False: Under this value, the task will process and send the data in batches (according to the underlying data files). Its characteristics are low timeliness and high throughput. - ### sink parameter #### iotdb-thrift-sink diff --git a/src/UserGuide/Master/Tree/User-Manual/Data-Sync_apache.md b/src/UserGuide/Master/Tree/User-Manual/Data-Sync_apache.md index 06bcfaea6..91bb48b9c 100644 --- a/src/UserGuide/Master/Tree/User-Manual/Data-Sync_apache.md +++ b/src/UserGuide/Master/Tree/User-Manual/Data-Sync_apache.md @@ -609,7 +609,6 @@ pipe_all_sinks_rate_limit_bytes_per_second=-1 | source | iotdb-source | String: iotdb-source | Yes | - | | inclusion | Used to specify the range of data to be synchronized in the data synchronization task, including data, schema, and auth | String:all, data(insert,delete), schema(database,timeseries,ttl), auth | No | data.insert | | inclusion.exclusion | Used to exclude specific operations from the range specified by inclusion, reducing the amount of data synchronized | String:all, data(insert,delete), schema(database,timeseries,ttl), auth | No | - | -| mode.streaming | Specifies the capture source for time-series data writes. Applicable when mode.streamingis false, determining the source for capturing data.insertspecified in inclusion. Offers two strategies:- true: ​​Dynamic capture selection.​​ The system adaptively chooses between capturing individual write requests or only TsFile sealing requests based on downstream processing speed. Prioritizes capturing write requests for lower latency when processing is fast; captures only file sealing requests to avoid backlog when slow. Suitable for most scenarios, balancing latency and throughput optimally.- false: ​​Fixed batch capture.​​ Captures only TsFile sealing requests. Suitable for resource-constrained scenarios to reduce system load. Note: The snapshot data captured upon pipe startup is only provided to downstream processing in file format. | Boolean: true / false | No | true | | mode.strict | Determines the strictness when filtering data using time/ path/ database-name/ table-nameparameters:- true: ​​Strict filtering.​​ The system strictly filters captured data according to the given conditions, ensuring only matching data is selected.- false: ​​Non-strict filtering.​​ The system may include some extra data during filtering. Suitable for performance-sensitive scenarios to reduce CPU and I/O consumption. | Boolean: true / false | No | true | | mode.snapshot | Determines the capture mode for time-series data, affecting the dataspecified in inclusion. Offers two modes:- true: ​​Static data capture.​​ Upon pipe startup, a one-time data snapshot is captured. ​​The pipe will automatically terminate (DROP PIPE SQL is executed automatically) after the snapshot data is fully consumed.​​- false: ​​Dynamic data capture.​​ In addition to capturing a snapshot upon startup, the pipe continuously captures subsequent data changes. The pipe runs continuously to handle the dynamic data stream. | Boolean: true / false | No | false | | path | Can be specified when the user connects with sql_dialectset to tree. For upgraded user pipes, the default sql_dialectis tree. This parameter determines the capture scope for time-series data, affecting the dataspecified in inclusion, as well as some sequence-related metadata. Data is selected into the streaming pipe if its tree model path matches the specified path.
Starting from version V2.0.8, this parameter supports specifying multiple exact paths in a single pipe, e.g., `'path'='root.test.d0.s1,root.test.d0.s2,root.test.d0.s3'`. | String: IoTDB-standard tree path pattern, wildcards allowed | No | root.** | @@ -619,10 +618,6 @@ pipe_all_sinks_rate_limit_bytes_per_second=-1 | mods | Same as mods.enable, whether to send the MODS file for TSFile. | Boolean: true / false | No | false | | skipIf | Which errors can be skipped? Currently only the insufficient privileges error. | String:no-privileges | No | no-privileges | -> 💎 **Note:** The difference between the values of true and false for the data extraction mode `mode.streaming` -> -> - True (recommended): Under this value, the task will process and send the data in real-time. Its characteristics are high timeliness and low throughput. -> - False: Under this value, the task will process and send the data in batches (according to the underlying data files). Its characteristics are low timeliness and high throughput. ### 5.2 sink parameter diff --git a/src/UserGuide/latest-Table/User-Manual/Data-Sync_apache.md b/src/UserGuide/latest-Table/User-Manual/Data-Sync_apache.md index 2650b2779..7ea012c3c 100644 --- a/src/UserGuide/latest-Table/User-Manual/Data-Sync_apache.md +++ b/src/UserGuide/latest-Table/User-Manual/Data-Sync_apache.md @@ -349,7 +349,7 @@ This example demonstrates synchronizing data within a specific historical time r ![](/img/e2.png) -In this example, we create a synchronization task named `A2B`. First, we define the data range in the `source` configuration. Since we are synchronizing historical data (data that existed before the task was created), we need to configure the start time (`start-time`), end time (`end-time`), and the streaming mode (`mode.streaming`). The `node-urls` parameter is configured with the URL of the DataNode service port on the target IoTDB. +In this example, we create a synchronization task named `A2B`. First, we define the data range in the `source` configuration. Since we are synchronizing historical data (data that existed before the task was created), we need to configure the start time (`start-time`), end time (`end-time`). The `node-urls` parameter is configured with the URL of the DataNode service port on the target IoTDB. SQL Example: @@ -357,7 +357,6 @@ SQL Example: CREATE PIPE A2B WITH SOURCE ( 'source' = 'iotdb-source', - 'mode.streaming' = 'true', 'database-name'='testdb.*', 'start-time' = '2023.08.23T08:00:00+00:00', 'end-time' = '2023.10.23T08:00:00+00:00' @@ -370,7 +369,6 @@ WITH SINK ( Where: -- `mode.streaming`: Extraction mode for newly inserted data (after the pipe is created): Whether to extract data in streaming mode (if set to false, batch mode is used). - `database-name`: Scope of Data Synchronization - `start-time`: The event time at which data synchronization starts (inclusive). - `end-time`: The event time at which data synchronization ends (inclusive). @@ -576,7 +574,6 @@ pipe_all_sinks_rate_limit_bytes_per_second=-1 | source | iotdb-source | String: iotdb-source | Yes | - | | inclusion | Used to specify the range of data to be synchronized in the data synchronization task, including data, schema and auth | String:all, data(insert,delete), schema(database,table,ttl), auth | No | data.insert | | inclusion.exclusion | Used to exclude specific operations from the range specified by inclusion, reducing the amount of data synchronized | String:all, data(insert,delete), schema(database,table,ttl), auth | No | - | -| mode.streaming | This parameter specifies the source of time-series data capture. It applies to scenarios where `mode.streaming` is set to `false`, determining the capture source for `data.insert` in `inclusion`. Two capture strategies are available: - **true**: Dynamically selects the capture type. The system adapts to downstream processing speed, choosing between capturing each write request or only capturing TsFile file sealing requests. When downstream processing is fast, write requests are prioritized to reduce latency; when processing is slow, only file sealing requests are captured to prevent processing backlogs. This mode suits most scenarios, optimizing the balance between processing latency and throughput. - **false**: Uses a fixed batch capture approach, capturing only TsFile file sealing requests. This mode is suitable for resource-constrained applications, reducing system load. **Note**: Snapshot data captured when the pipe starts will only be provided for downstream processing as files. | Boolean: true / false | No | true | | mode.strict | Determines whether to strictly filter data when using the `time`, `path`, `database-name`, or `table-name` parameters: - **true**: Strict filtering. The system will strictly filter captured data according to the specified conditions, ensuring that only matching data is selected. - **false**: Non-strict filtering. Some extra data may be included during the selection process to optimize performance and reduce CPU and I/O consumption. | Boolean: true / false | No | true | | mode.snapshot | This parameter determines the data capture mode, affecting the `data` in `inclusion`. Two modes are available: - **true**: Static data capture. A one-time data snapshot is taken when the pipe starts. Once the snapshot data is fully consumed, the pipe automatically terminates (executing `DROP PIPE` SQL automatically). - **false**: Dynamic data capture. In addition to capturing snapshot data when the pipe starts, it continuously captures subsequent data changes. The pipe remains active to process the dynamic data stream. | Boolean: true / false | No | false | | database-name | When the user connects with `sql_dialect` set to `table`, this parameter can be specified. Determines the scope of data capture, affecting the `data` in `inclusion`. Specifies the database name to filter. It can be a specific database name or a Java-style regular expression to match multiple databases. By default, all databases are matched. | String: Database name or database regular expression pattern string, which can match uncreated or non - existent databases. | No | ".*" | @@ -586,11 +583,6 @@ pipe_all_sinks_rate_limit_bytes_per_second=-1 | mods | Same as mods.enable, whether to send the MODS file for TSFile. | Boolean: true / false | No | false | | skipIf | Which errors can be skipped? Currently only the insufficient privileges error. | String:no-privileges | No | no-privileges | -> 💎 **Note:** The difference between the values of true and false for the data extraction mode `mode.streaming` -> -> - True (recommended): Under this value, the task will process and send the data in real-time. Its characteristics are high timeliness and low throughput. -> - False: Under this value, the task will process and send the data in batches (according to the underlying data files). Its characteristics are low timeliness and high throughput. - ### sink parameter #### iotdb-thrift-sink diff --git a/src/UserGuide/latest/User-Manual/Data-Sync_apache.md b/src/UserGuide/latest/User-Manual/Data-Sync_apache.md index 06bcfaea6..91bb48b9c 100644 --- a/src/UserGuide/latest/User-Manual/Data-Sync_apache.md +++ b/src/UserGuide/latest/User-Manual/Data-Sync_apache.md @@ -609,7 +609,6 @@ pipe_all_sinks_rate_limit_bytes_per_second=-1 | source | iotdb-source | String: iotdb-source | Yes | - | | inclusion | Used to specify the range of data to be synchronized in the data synchronization task, including data, schema, and auth | String:all, data(insert,delete), schema(database,timeseries,ttl), auth | No | data.insert | | inclusion.exclusion | Used to exclude specific operations from the range specified by inclusion, reducing the amount of data synchronized | String:all, data(insert,delete), schema(database,timeseries,ttl), auth | No | - | -| mode.streaming | Specifies the capture source for time-series data writes. Applicable when mode.streamingis false, determining the source for capturing data.insertspecified in inclusion. Offers two strategies:- true: ​​Dynamic capture selection.​​ The system adaptively chooses between capturing individual write requests or only TsFile sealing requests based on downstream processing speed. Prioritizes capturing write requests for lower latency when processing is fast; captures only file sealing requests to avoid backlog when slow. Suitable for most scenarios, balancing latency and throughput optimally.- false: ​​Fixed batch capture.​​ Captures only TsFile sealing requests. Suitable for resource-constrained scenarios to reduce system load. Note: The snapshot data captured upon pipe startup is only provided to downstream processing in file format. | Boolean: true / false | No | true | | mode.strict | Determines the strictness when filtering data using time/ path/ database-name/ table-nameparameters:- true: ​​Strict filtering.​​ The system strictly filters captured data according to the given conditions, ensuring only matching data is selected.- false: ​​Non-strict filtering.​​ The system may include some extra data during filtering. Suitable for performance-sensitive scenarios to reduce CPU and I/O consumption. | Boolean: true / false | No | true | | mode.snapshot | Determines the capture mode for time-series data, affecting the dataspecified in inclusion. Offers two modes:- true: ​​Static data capture.​​ Upon pipe startup, a one-time data snapshot is captured. ​​The pipe will automatically terminate (DROP PIPE SQL is executed automatically) after the snapshot data is fully consumed.​​- false: ​​Dynamic data capture.​​ In addition to capturing a snapshot upon startup, the pipe continuously captures subsequent data changes. The pipe runs continuously to handle the dynamic data stream. | Boolean: true / false | No | false | | path | Can be specified when the user connects with sql_dialectset to tree. For upgraded user pipes, the default sql_dialectis tree. This parameter determines the capture scope for time-series data, affecting the dataspecified in inclusion, as well as some sequence-related metadata. Data is selected into the streaming pipe if its tree model path matches the specified path.
Starting from version V2.0.8, this parameter supports specifying multiple exact paths in a single pipe, e.g., `'path'='root.test.d0.s1,root.test.d0.s2,root.test.d0.s3'`. | String: IoTDB-standard tree path pattern, wildcards allowed | No | root.** | @@ -619,10 +618,6 @@ pipe_all_sinks_rate_limit_bytes_per_second=-1 | mods | Same as mods.enable, whether to send the MODS file for TSFile. | Boolean: true / false | No | false | | skipIf | Which errors can be skipped? Currently only the insufficient privileges error. | String:no-privileges | No | no-privileges | -> 💎 **Note:** The difference between the values of true and false for the data extraction mode `mode.streaming` -> -> - True (recommended): Under this value, the task will process and send the data in real-time. Its characteristics are high timeliness and low throughput. -> - False: Under this value, the task will process and send the data in batches (according to the underlying data files). Its characteristics are low timeliness and high throughput. ### 5.2 sink parameter diff --git a/src/zh/UserGuide/Master/Table/User-Manual/Data-Sync_apache.md b/src/zh/UserGuide/Master/Table/User-Manual/Data-Sync_apache.md index 262ef66f7..f17025063 100644 --- a/src/zh/UserGuide/Master/Table/User-Manual/Data-Sync_apache.md +++ b/src/zh/UserGuide/Master/Table/User-Manual/Data-Sync_apache.md @@ -344,7 +344,7 @@ with sink ( ![](/img/%E6%95%B0%E6%8D%AE%E5%90%8C%E6%AD%A51.png) -在这个例子中，我们可以创建一个名为 A2B 的同步任务。首先我们需要在 source 中定义传输数据的范围，由于传输的是历史数据（历史数据是指同步任务创建之前存在的数据），需要配置数据的起止时间 start-time 和 end-time 以及传输的模式 mode.streaming。通过 node-urls 配置目标端 IoTDB 中 DataNode 节点的数据服务端口的 url。 +在这个例子中，我们可以创建一个名为 A2B 的同步任务。首先我们需要在 source 中定义传输数据的范围，由于传输的是历史数据（历史数据是指同步任务创建之前存在的数据），需要配置数据的起止时间 start-time 和 end-time。通过 node-urls 配置目标端 IoTDB 中 DataNode 节点的数据服务端口的 url。 详细语句如下： @@ -352,7 +352,6 @@ with sink ( create pipe A2B WITH SOURCE ( 'source'= 'iotdb-source', - 'mode.streaming' = 'true', 'database-name'='testdb.*', 'start-time' = '2023.08.23T08:00:00+00:00', 'end-time' = '2023.10.23T08:00:00+00:00' @@ -365,7 +364,6 @@ with SINK ( 其中： -- `mode.streaming`: 新插入数据（pipe创建后）的抽取模式：是否按流式抽取（false 时为批式） - `database-name`: 同步数据的范围 - `start-time`: 同步所有数据的开始 event time，包含 start-time - `end-time`: 同步所有数据的结束 event time，包含 end-time @@ -570,7 +568,6 @@ pipe_all_sinks_rate_limit_bytes_per_second=-1 | source | iotdb-source | String: iotdb-source | 必填 | - | | inclusion | 用于指定数据同步任务中需要同步范围，分为数据，元数据和权限 | String:all, data(insert,delete), schema(database,table,ttl), auth | 选填 | data.insert | | inclusion.exclusion | 用于从 inclusion 指定的同步范围内排除特定的操作，减少同步的数据量 | String:all, data(insert,delete), schema(database,table,ttl), auth | 选填 | 空字符串 | -| mode.streaming | 此参数指定时序数据写入的捕获来源。适用于 `mode.streaming`为 `false` 模式下的场景，决定`inclusion`中`data.insert`数据的捕获来源。提供两种捕获策略：true： 动态选择捕获的类型。系统将根据下游处理速度，自适应地选择是捕获每个写入请求还是仅捕获 TsFile 文件的封口请求。当下游处理速度快时，优先捕获写入请求以减少延迟；当处理速度慢时，仅捕获文件封口请求以避免处理堆积。这种模式适用于大多数场景，能够实现处理延迟和吞吐量的最优平衡。false：固定按批捕获方式。仅捕获 TsFile 文件的封口请求，适用于资源受限的应用场景，以降低系统负载。注意，pipe 启动时捕获的快照数据只会以文件的方式供下游处理。 | Boolean: true / false | 否 | true | | mode.strict | 在使用 time / path / database-name / table-name 参数过滤数据时，是否需要严格按照条件筛选：`true`： 严格筛选。系统将完全按照给定条件过滤筛选被捕获的数据，确保只有符合条件的数据被选中。`false`：非严格筛选。系统在筛选被捕获的数据时可能会包含一些额外的数据，适用于性能敏感的场景，可降低 CPU 和 IO 消耗。 | Boolean: true / false | 否 | true | | mode.snapshot | 此参数决定时序数据的捕获方式，影响`inclusion`中的`data`数据。提供两种模式：`true`：静态数据捕获。启动 pipe 时，会进行一次性的数据快照捕获。当快照数据被完全消费后，**pipe 将自动终止（DROP PIPE SQL 会自动执行）**。`false`：动态数据捕获。除了在 pipe 启动时捕获快照数据外，还会持续捕获后续的数据变更。pipe 将持续运行以处理动态数据流。 | Boolean: true / false | 否 | false | | database-name | 当用户连接指定的 sql_dialect 为 table 时可以指定。此参数决定时序数据的捕获范围，影响`inclusion`中的`data`数据。表示要过滤的数据库的名称。它可以是具体的数据库名，也可以是 Java 风格正则表达式来匹配多个数据库。默认情况下，匹配所有的库。 | String：数据库名或数据库正则模式串，可以匹配未创建的、不存在的库 | 否 | ".*" | @@ -580,10 +577,6 @@ pipe_all_sinks_rate_limit_bytes_per_second=-1 | mods | 同 mods.enable，是否发送 tsfile 的 mods 文件 | Boolean: true / false | 选填 | false | | skipIf | 出现哪些错误可以跳过，当前只有无权限的错误 | String:no-privileges | 选填 | no-privileges | -> 💎 **说明：数据抽取模式 mode.streaming 取值 true 和 false 的差异** -> - **true（推荐）**：该取值下，任务将对数据进行实时处理、发送，其特点是高时效、低吞吐 -> - **false**：该取值下，任务将对数据进行批量（按底层数据文件）处理、发送，其特点是低时效、高吞吐 - ### sink 参数 #### iotdb-thrift-sink diff --git a/src/zh/UserGuide/Master/Tree/User-Manual/Data-Sync_apache.md b/src/zh/UserGuide/Master/Tree/User-Manual/Data-Sync_apache.md index 0da6bb54c..d275b8d95 100644 --- a/src/zh/UserGuide/Master/Tree/User-Manual/Data-Sync_apache.md +++ b/src/zh/UserGuide/Master/Tree/User-Manual/Data-Sync_apache.md @@ -609,7 +609,6 @@ pipe_all_sinks_rate_limit_bytes_per_second=-1 | source | iotdb-source | String: iotdb-source | 必填 | - | | inclusion | 用于指定数据同步任务中需要同步范围，分为数据、元数据和权限 | String:all, data(insert,delete), schema(database,timeseries,ttl), auth | 选填 | data.insert | | inclusion.exclusion | 用于从 inclusion 指定的同步范围内排除特定的操作，减少同步的数据量 | String:all, data(insert,delete), schema(database,timeseries,ttl), auth | 选填 | 空字符串 | -| mode.streaming | 此参数指定时序数据写入的捕获来源。适用于 `mode.streaming`为 `false` 模式下的场景，决定`inclusion`中`data.insert`数据的捕获来源。提供两种捕获策略：true： 动态选择捕获的类型。系统将根据下游处理速度，自适应地选择是捕获每个写入请求还是仅捕获 TsFile 文件的封口请求。当下游处理速度快时，优先捕获写入请求以减少延迟；当处理速度慢时，仅捕获文件封口请求以避免处理堆积。这种模式适用于大多数场景，能够实现处理延迟和吞吐量的最优平衡。false：固定按批捕获方式。仅捕获 TsFile 文件的封口请求，适用于资源受限的应用场景，以降低系统负载。注意，pipe 启动时捕获的快照数据只会以文件的方式供下游处理。 | Boolean: true / false | 否 | true | | mode.strict | 在使用 time / path / database-name / table-name 参数过滤数据时，是否需要严格按照条件筛选：`true`： 严格筛选。系统将完全按照给定条件过滤筛选被捕获的数据，确保只有符合条件的数据被选中。`false`：非严格筛选。系统在筛选被捕获的数据时可能会包含一些额外的数据，适用于性能敏感的场景，可降低 CPU 和 IO 消耗。 | Boolean: true / false | 否 | true | | mode.snapshot | 此参数决定时序数据的捕获方式，影响`inclusion`中的`data`数据。提供两种模式：`true`：静态数据捕获。启动 pipe 时，会进行一次性的数据快照捕获。当快照数据被完全消费后，**pipe 将自动终止（DROP PIPE SQL 会自动执行）**。`false`：动态数据捕获。除了在 pipe 启动时捕获快照数据外，还会持续捕获后续的数据变更。pipe 将持续运行以处理动态数据流。 | Boolean: true / false | 否 | false | | path | 当用户连接指定的sql_dialect为tree时可以指定。对于升级上来的用户pipe,默认sql_dialect为tree。此参数决定时序数据的捕获范围,影响 inclusion中的data数据,以及部分序列相关的元数据。当数据的树模型路径能够被path匹配时，数据会被筛选出来进入流处理pipe。
自 V2.0.8 版本起，该参数支持在一个pipe中填写多个精确路径的path , 如 `'path'='root.test.d0,s1,root.test.d0.s2,root.test.d0.s3'` | String:IoTDB标准的树路径模式,可以带通配符 | 选填 | root.** | @@ -619,9 +618,6 @@ pipe_all_sinks_rate_limit_bytes_per_second=-1 | mods | 同 mods.enable，是否发送 tsfile 的 mods 文件 | Boolean: true / false | 选填 | false | | skipIf | 出现哪些错误可以跳过，当前只有无权限的错误 | String:no-privileges | 选填 | no-privileges | -> 💎 **说明：数据抽取模式 mode.streaming 取值 true 和 false 的差异** -> - **true（推荐）**：该取值下，任务将对数据进行实时处理、发送，其特点是高时效、低吞吐 -> - **false**：该取值下，任务将对数据进行批量（按底层数据文件）处理、发送，其特点是低时效、高吞吐 ### 5.2 sink 参数 diff --git a/src/zh/UserGuide/latest-Table/User-Manual/Data-Sync_apache.md b/src/zh/UserGuide/latest-Table/User-Manual/Data-Sync_apache.md index 1fa296f7d..2feabfc27 100644 --- a/src/zh/UserGuide/latest-Table/User-Manual/Data-Sync_apache.md +++ b/src/zh/UserGuide/latest-Table/User-Manual/Data-Sync_apache.md @@ -344,7 +344,7 @@ with sink ( ![](/img/%E6%95%B0%E6%8D%AE%E5%90%8C%E6%AD%A51.png) -在这个例子中，我们可以创建一个名为 A2B 的同步任务。首先我们需要在 source 中定义传输数据的范围，由于传输的是历史数据（历史数据是指同步任务创建之前存在的数据），需要配置数据的起止时间 start-time 和 end-time 以及传输的模式 mode.streaming。通过 node-urls 配置目标端 IoTDB 中 DataNode 节点的数据服务端口的 url。 +在这个例子中，我们可以创建一个名为 A2B 的同步任务。首先我们需要在 source 中定义传输数据的范围，由于传输的是历史数据（历史数据是指同步任务创建之前存在的数据），需要配置数据的起止时间 start-time 和 end-time。通过 node-urls 配置目标端 IoTDB 中 DataNode 节点的数据服务端口的 url。 详细语句如下： @@ -352,7 +352,6 @@ with sink ( create pipe A2B WITH SOURCE ( 'source'= 'iotdb-source', - 'mode.streaming' = 'true', 'database-name'='testdb.*', 'start-time' = '2023.08.23T08:00:00+00:00', 'end-time' = '2023.10.23T08:00:00+00:00' @@ -365,7 +364,6 @@ with SINK ( 其中： -- `mode.streaming`: 新插入数据（pipe创建后）的抽取模式：是否按流式抽取（false 时为批式） - `database-name`: 同步数据的范围 - `start-time`: 同步所有数据的开始 event time，包含 start-time - `end-time`: 同步所有数据的结束 event time，包含 end-time @@ -570,7 +568,6 @@ pipe_all_sinks_rate_limit_bytes_per_second=-1 | source | iotdb-source | String: iotdb-source | 必填 | - | | inclusion | 用于指定数据同步任务中需要同步范围，分为数据，元数据和权限 | String:all, data(insert,delete), schema(database,table,ttl), auth | 选填 | data.insert | | inclusion.exclusion | 用于从 inclusion 指定的同步范围内排除特定的操作，减少同步的数据量 | String:all, data(insert,delete), schema(database,table,ttl), auth | 选填 | 空字符串 | -| mode.streaming | 此参数指定时序数据写入的捕获来源。适用于 `mode.streaming`为 `false` 模式下的场景，决定`inclusion`中`data.insert`数据的捕获来源。提供两种捕获策略：true： 动态选择捕获的类型。系统将根据下游处理速度，自适应地选择是捕获每个写入请求还是仅捕获 TsFile 文件的封口请求。当下游处理速度快时，优先捕获写入请求以减少延迟；当处理速度慢时，仅捕获文件封口请求以避免处理堆积。这种模式适用于大多数场景，能够实现处理延迟和吞吐量的最优平衡。false：固定按批捕获方式。仅捕获 TsFile 文件的封口请求，适用于资源受限的应用场景，以降低系统负载。注意，pipe 启动时捕获的快照数据只会以文件的方式供下游处理。 | Boolean: true / false | 否 | true | | mode.strict | 在使用 time / path / database-name / table-name 参数过滤数据时，是否需要严格按照条件筛选：`true`： 严格筛选。系统将完全按照给定条件过滤筛选被捕获的数据，确保只有符合条件的数据被选中。`false`：非严格筛选。系统在筛选被捕获的数据时可能会包含一些额外的数据，适用于性能敏感的场景，可降低 CPU 和 IO 消耗。 | Boolean: true / false | 否 | true | | mode.snapshot | 此参数决定时序数据的捕获方式，影响`inclusion`中的`data`数据。提供两种模式：`true`：静态数据捕获。启动 pipe 时，会进行一次性的数据快照捕获。当快照数据被完全消费后，**pipe 将自动终止（DROP PIPE SQL 会自动执行）**。`false`：动态数据捕获。除了在 pipe 启动时捕获快照数据外，还会持续捕获后续的数据变更。pipe 将持续运行以处理动态数据流。 | Boolean: true / false | 否 | false | | database-name | 当用户连接指定的 sql_dialect 为 table 时可以指定。此参数决定时序数据的捕获范围，影响`inclusion`中的`data`数据。表示要过滤的数据库的名称。它可以是具体的数据库名，也可以是 Java 风格正则表达式来匹配多个数据库。默认情况下，匹配所有的库。 | String：数据库名或数据库正则模式串，可以匹配未创建的、不存在的库 | 否 | ".*" | @@ -580,10 +577,6 @@ pipe_all_sinks_rate_limit_bytes_per_second=-1 | mods | 同 mods.enable，是否发送 tsfile 的 mods 文件 | Boolean: true / false | 选填 | false | | skipIf | 出现哪些错误可以跳过，当前只有无权限的错误 | String:no-privileges | 选填 | no-privileges | -> 💎 **说明：数据抽取模式 mode.streaming 取值 true 和 false 的差异** -> - **true（推荐）**：该取值下，任务将对数据进行实时处理、发送，其特点是高时效、低吞吐 -> - **false**：该取值下，任务将对数据进行批量（按底层数据文件）处理、发送，其特点是低时效、高吞吐 - ### sink 参数 #### iotdb-thrift-sink diff --git a/src/zh/UserGuide/latest/User-Manual/Data-Sync_apache.md b/src/zh/UserGuide/latest/User-Manual/Data-Sync_apache.md index 0da6bb54c..d275b8d95 100644 --- a/src/zh/UserGuide/latest/User-Manual/Data-Sync_apache.md +++ b/src/zh/UserGuide/latest/User-Manual/Data-Sync_apache.md @@ -609,7 +609,6 @@ pipe_all_sinks_rate_limit_bytes_per_second=-1 | source | iotdb-source | String: iotdb-source | 必填 | - | | inclusion | 用于指定数据同步任务中需要同步范围，分为数据、元数据和权限 | String:all, data(insert,delete), schema(database,timeseries,ttl), auth | 选填 | data.insert | | inclusion.exclusion | 用于从 inclusion 指定的同步范围内排除特定的操作，减少同步的数据量 | String:all, data(insert,delete), schema(database,timeseries,ttl), auth | 选填 | 空字符串 | -| mode.streaming | 此参数指定时序数据写入的捕获来源。适用于 `mode.streaming`为 `false` 模式下的场景，决定`inclusion`中`data.insert`数据的捕获来源。提供两种捕获策略：true： 动态选择捕获的类型。系统将根据下游处理速度，自适应地选择是捕获每个写入请求还是仅捕获 TsFile 文件的封口请求。当下游处理速度快时，优先捕获写入请求以减少延迟；当处理速度慢时，仅捕获文件封口请求以避免处理堆积。这种模式适用于大多数场景，能够实现处理延迟和吞吐量的最优平衡。false：固定按批捕获方式。仅捕获 TsFile 文件的封口请求，适用于资源受限的应用场景，以降低系统负载。注意，pipe 启动时捕获的快照数据只会以文件的方式供下游处理。 | Boolean: true / false | 否 | true | | mode.strict | 在使用 time / path / database-name / table-name 参数过滤数据时，是否需要严格按照条件筛选：`true`： 严格筛选。系统将完全按照给定条件过滤筛选被捕获的数据，确保只有符合条件的数据被选中。`false`：非严格筛选。系统在筛选被捕获的数据时可能会包含一些额外的数据，适用于性能敏感的场景，可降低 CPU 和 IO 消耗。 | Boolean: true / false | 否 | true | | mode.snapshot | 此参数决定时序数据的捕获方式，影响`inclusion`中的`data`数据。提供两种模式：`true`：静态数据捕获。启动 pipe 时，会进行一次性的数据快照捕获。当快照数据被完全消费后，**pipe 将自动终止（DROP PIPE SQL 会自动执行）**。`false`：动态数据捕获。除了在 pipe 启动时捕获快照数据外，还会持续捕获后续的数据变更。pipe 将持续运行以处理动态数据流。 | Boolean: true / false | 否 | false | | path | 当用户连接指定的sql_dialect为tree时可以指定。对于升级上来的用户pipe,默认sql_dialect为tree。此参数决定时序数据的捕获范围,影响 inclusion中的data数据,以及部分序列相关的元数据。当数据的树模型路径能够被path匹配时，数据会被筛选出来进入流处理pipe。
自 V2.0.8 版本起，该参数支持在一个pipe中填写多个精确路径的path , 如 `'path'='root.test.d0,s1,root.test.d0.s2,root.test.d0.s3'` | String:IoTDB标准的树路径模式,可以带通配符 | 选填 | root.** | @@ -619,9 +618,6 @@ pipe_all_sinks_rate_limit_bytes_per_second=-1 | mods | 同 mods.enable，是否发送 tsfile 的 mods 文件 | Boolean: true / false | 选填 | false | | skipIf | 出现哪些错误可以跳过，当前只有无权限的错误 | String:no-privileges | 选填 | no-privileges | -> 💎 **说明：数据抽取模式 mode.streaming 取值 true 和 false 的差异** -> - **true（推荐）**：该取值下，任务将对数据进行实时处理、发送，其特点是高时效、低吞吐 -> - **false**：该取值下，任务将对数据进行批量（按底层数据文件）处理、发送，其特点是低时效、高吞吐 ### 5.2 sink 参数