feat(connectors): add OpenDAL sink by George-Miao · Pull Request #4123 · apache/iggy

George-Miao · 2026-09-10T18:56:40Z

Which issue does this PR address?

Rationale

OpenDAL provides an uniformed interface for writing to various services, which is useful for iggy to support multiple new sink types with a very small amount of change & maintainance effort.

What changed?

Added opendal-sink crate that implements a sink adapter for OpenDAL and integration test. I intentionally didn't touch the other part, e.g., CI. Those can be done in another follow-up PR.

Local Execution

Passed
Pre-commit hooks ran

AI Usage

Uses codex & omp for writing tests, reviewing the changes & write some part of the doc.

github-actions · 2026-09-10T18:56:52Z

Thanks for the PR. It is labeled S-waiting-on-review and queued for review.

Slash commands (own line, regular comment) move it around the queue:

/ready - back to S-waiting-on-review after addressing feedback
/author - flip to S-waiting-on-author while you finish changes
/request-review @user-or-team - request a reviewer
/pin - exempt the PR from the stale bot, /unpin to undo

See CONTRIBUTING.md for details.

codecov · 2026-09-10T18:58:50Z

Codecov Report

❌ Patch coverage is 87.25341% with 84 lines in your changes missing coverage. Please review.
✅ Project coverage is 87.39%. Comparing base (63362dc) to head (5afbe36).
⚠️ Report is 29 commits behind head on master.

Files with missing lines	Patch %	Lines
...ore/connectors/sinks/opendal_sink/src/formatter.rs	55.38%	57 Missing and 1 partial ⚠️
core/connectors/sinks/opendal_sink/src/lib.rs	94.65%	17 Missing and 7 partials ⚠️
core/connectors/sinks/opendal_sink/src/path.rs	97.50%	1 Missing and 1 partial ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##             master    #4123      +/-   ##
============================================
- Coverage     87.48%   87.39%   -0.10%     
- Complexity     1575     1578       +3     
============================================
  Files          1280     1283       +3     
  Lines        223192   223851     +659     
  Branches     186555   187214     +659     
============================================
+ Hits         195267   195633     +366     
- Misses        23229    23479     +250     
- Partials       4696     4739      +43

Components	Coverage Δ
Rust Core	`88.56% <87.25%> (-0.01%)`	⬇️
Java SDK	`68.71% <ø> (+0.03%)`	⬆️
C# SDK	`77.42% <ø> (-0.07%)`	⬇️
Python SDK	`90.97% <ø> (ø)`
PHP SDK	`85.67% <ø> (ø)`
Node SDK	`94.74% <ø> (-1.69%)`	⬇️
Go SDK	`70.11% <ø> (+0.05%)`	⬆️

Files with missing lines	Coverage Δ
core/connectors/sinks/opendal_sink/src/path.rs	`97.50% <97.50%> (ø)`
core/connectors/sinks/opendal_sink/src/lib.rs	`94.65% <94.65%> (ø)`
...ore/connectors/sinks/opendal_sink/src/formatter.rs	`55.38% <55.38%> (ø)`

... and 47 files with indirect coverage changes

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

mattp5657

2 things:

This connector is missing from the release/version-bump tooling. Absent from scripts/bump-version.sh, the connector_plugins default in _build_rust_artifacts.yml, and the connector list in edge-release.yml. It'll pass CI but won't ship in edge or tagged release artifacts, and a future version bump will silently skip it. Not unique to this PR (a handful of existing sinks are missing from these same lists too), but worth fixing here or in an immediate follow-up rather than adding a fourth thing to that pile.
I think we could use more unit and integration test coverage, happy and unhappy paths for every function. Untested today: path.rs's render_template and sanitize_path_segment beyond one edge case, lib.rs's consume()/write_message() unhappy path, the retry policy past one attempt, and S3/GCS/Azure end-to-end (fs is the only backend any test exercises)

hubcio

the plugin does not reach release artifacts yet: _build_rust_artifacts.yml:49 and edge-release.yml:108 do not list it, scripts/bump-version.sh:90 omits the crate, and the sinks table in core/connectors/sinks/README.md has no row. you noted that the CI part is a follow-up PR - can you confirm it covers all four?

hubcio · 2026-09-22T10:33:08Z

+        .replace("{timestamp}", &timestamp_millis))
+}
+
+fn sanitize_path_segment(segment: &str) -> String {


warning: a b and a_b both sanitize to a_b, so the later write hides the earlier topic's object, and .. passes too and always fails. percent-encode the segment or hash the raw name.

hubcio · 2026-09-22T10:33:08Z

+            self.retry_policy,
+            &retry_context,
+            opendal::Error::is_temporary,
+            || operator.write(&path, buffer.clone()),


warning: with the shipped fs config opendal truncates the final key and writes in place, so a reader can see a partial object and a crash leaves one. set atomic_write_dir, and document that fs writes are not atomic.

hubcio · 2026-09-22T10:33:08Z

+            last_offset,
+            output_format,
+        )?;
+        let entries = messages


warning: collects a Vec<u8> per message, then finalize_buffer copies all of them into a second buffer. format straight into one buffer, and move the JsonArray bracket logic there first.

hubcio · 2026-09-22T10:33:08Z

+        }
+    }
+
+    async fn write_batch(


warning: one object per poll batch means nothing bounds object size in bytes. s3_sink rotates at 8 MiB, but the connector guidance says not to buffer across consume - which rule wins here?

hubcio · 2026-09-22T10:33:08Z

+        } else {
+            None
+        },
+        payload: payload_to_json_value(&message.payload),


warning: rebuilds the payload tree before serializing, which allocates a node per element and a String per key. serialize in place with serde_json::to_writer and a wrapper that maps the six Payload variants.

hubcio · 2026-09-22T10:33:08Z

+            }
+            Err(write_error) => {
+                self.write_errors
+                    .fetch_add(message_count, Ordering::Relaxed);


nit: write_errors counts messages, but close() logs it under an errors label. rename the label - counting batches instead breaks the test at line 609.

hubcio · 2026-09-22T10:33:08Z

+    #[serde(default)]
+    pub include_headers: bool,
+    #[serde(default)]
+    pub max_attempts: Option<u32>,


nit: s3_sink accepts both max_attempts and max_retries, and five other sinks use max_retries. add #[serde(alias = "max_retries")] so a copied config is not silently ignored.

hubcio · 2026-09-22T10:33:08Z

+
+    fn try_from(value: &str) -> Result<Self, Self::Error> {
+        match value.to_lowercase().as_str() {
+            "json_lines" | "jsonl" | "jsonlines" => Ok(Self::JsonLines),


nit: jsonl and jsonlines are accepted here but the README lists only three names. add them to the table, since s3_sink documents the same pair.

hubcio · 2026-09-22T10:33:08Z

+| `service`               | String   | **required**                     | OpenDAL service name                          |
+| `path_prefix`           | String   | empty                            | Prefix before the rendered object path        |
+| `path_template`         | String   | `{stream}/{topic}/{date}/{hour}` | Object directory template                     |
+| `options`               | Map      | empty                            | Secret OpenDAL service options                |


nit: values must be quoted strings in TOML, since SecretString deserializes through String, so a bare number or bool fails at load. say so in the row.

hubcio · 2026-09-22T10:33:08Z

+
+[package]
+name = "iggy_connector_opendal_sink"
+version = "0.5.0-edge.4"


nit: 0.5.0-edge.4 is the only prerelease among the connector plugins, and bump-version.sh does not list this crate, so a release bump skips it. use 0.5.0 and add the crate to CONNECTOR_SINK_COMPONENTS.

github-actions Bot added the S-waiting-on-review PR is waiting on a reviewer label Sep 10, 2026

George-Miao force-pushed the master branch from 71c92da to dfa8251 Compare September 10, 2026 18:58

George-Miao force-pushed the master branch from dfa8251 to 723e578 Compare September 11, 2026 03:05

ryankert01 reviewed Sep 13, 2026

View reviewed changes

Comment thread core/connectors/sinks/opendal_sink/src/lib.rs Outdated

mattp5657 reviewed Sep 14, 2026

View reviewed changes

Comment thread core/connectors/sinks/opendal_sink/src/lib.rs

Comment thread core/connectors/sinks/opendal_sink/src/lib.rs Outdated

Comment thread core/connectors/sinks/opendal_sink/src/path.rs Outdated

George-Miao added 4 commits September 16, 2026 18:18

feat(connectors): add OpenDAL sink

a820c31

feat: add integration test

79e31dd

apply suggestion

9c446df

feat: batch write

5afbe36

George-Miao force-pushed the master branch from 3a98b65 to 5afbe36 Compare September 16, 2026 22:21

hubcio requested changes Sep 22, 2026

View reviewed changes

github-actions Bot added S-waiting-on-author PR is waiting on author response and removed S-waiting-on-review PR is waiting on a reviewer labels Sep 22, 2026

Conversation

George-Miao commented Sep 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Which issue does this PR address?

Rationale

What changed?

Local Execution

AI Usage

Uh oh!

github-actions Bot commented Sep 10, 2026

Uh oh!

codecov Bot commented Sep 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

mattp5657 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

hubcio left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

George-Miao commented Sep 10, 2026 •

edited

Loading

codecov Bot commented Sep 10, 2026 •

edited

Loading