Skip to content

[FLINK-40508][pipeline-connector/db2] Support DB2 pipeline connector - #4533

Open
suntectec wants to merge 2 commits into
apache:masterfrom
suntectec:FLINK-40508
Open

suntectec wants to merge 2 commits into
apache:masterfrom
suntectec:FLINK-40508

Conversation

@suntectec

Copy link
Copy Markdown
Contributor

What is the purpose of this pull request?

Add a DB2 pipeline connector so the YAML Pipeline API supports IBM DB2 as a source, in the same way the existing MySQL / Postgres / Oracle / SQL Server pipeline connectors do. Before this change DB2 was only reachable from the DataStream / SQL APIs via flink-connector-db2-cdc.

JIRA: FLINK-40508

Brief change log

  • New module flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-db2, registered in the pipeline-connectors pom.xml.
  • Db2DataSourceFactory (identifier db2) plus Db2DataSource, Db2PipelineSource, Db2EventDeserializer, Db2MetadataAccessor, Db2PipelineRecordEmitter, Db2SchemaDataTypeInference, Db2SchemaUtils and Db2TypeUtils.
  • Db2DataSourceOptions exposes the options the DB2 source connector already supports: hostname, port, username, password, database, tables, tables.exclude, server-time-zone, scan.startup.mode (initial / latest-offset), schema-change.enabled, metadata.list, scan.incremental.snapshot.chunk.key-column, scan.incremental.snapshot.chunk.size, chunk-meta.group.size, chunk-key.even-distribution.factor.upper-bound, chunk-key.even-distribution.factor.lower-bound, scan.snapshot.fetch.size, connect.timeout, connection.pool.size, connect.max-retries, scan.incremental.snapshot.backfill.skip, scan.incremental.close-idle-reader.enabled and scan.incremental.snapshot.unbounded-chunk-first.enabled.
  • The source connector itself is reused as-is through the incremental snapshot framework; no change to flink-connector-db2-cdc.
  • DB2 specific type mapping, notably DECFLOAT(16)DOUBLE, DECFLOAT(34)DECIMAL(34, 0), XML / SQLXML / CLOBSTRING, BLOB / BINARY / VARBINARYBYTES.
  • Captured table names are resolved as schemaName.tableName, and latest-offset maps to StartupOptions.latest() with initial as the default.
  • .github/workflows/modules.py: the new module is added to MODULES_DB2, so CI compiles and tests it on both Flink 1.20 and Flink 2.x.
  • .github/labeler.yml: adds a db2-pipeline-connector rule.
  • Documentation in both docs/content and docs/content.zh, including a row in each pipeline-connectors overview.md.

One thing worth pointing out: src/test/resources/db2_server/ is a byte-identical copy of the source connector's test resources. That is not accidental — Db2TestBase#getFilePath resolves the Docker build context with Paths.get(url.toURI()), which cannot read a resource out of the test-jar, so the files have to exist on the module's own test classpath as real files. The added .gitattributes keeps these container scripts at LF endings, which matters for contributors on Windows.

Verifying this change

This change added tests and can be verified as follows:

  • Unit tests in Db2TypeUtilsTest cover every supported DB2 column type plus the unsupported-type failure path.
  • Integration tests in Db2DataSourceFactoryITCase and Db2PipelineITCase run against a real DB2 instance through testcontainers, reusing Db2TestBase from the source connector's test-jar.
  • Verified locally with mvn test on Java 11 / Flink 1.20, and on CI in the Source Unit Tests / test (…, db2, mongodb) and Source Unit Tests 2.x / test (…, 2.2.0, db2, mongodb) jobs, which run mvn verify and therefore execute the ITCases against both Flink 1.20.3 and Flink 2.2.0.

Documentation

  • Does this pull request introduce a new feature? yes
  • If yes, how is the feature documented? docs — see docs/content/docs/connectors/pipeline-connectors/db2.md and its Chinese counterpart.

Was generative AI tooling used to co-author this PR?
  • Yes (Qoder)

@github-actions github-actions Bot added docs Improvements or additions to documentation build labels Sep 12, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

build docs Improvements or additions to documentation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant