Skip to content

feat(vortex-python): Add OpenDAL-backed CosStore to the Python object-store API#8845

Open
XuQianJin-Stars wants to merge 1 commit into
vortex-data:developfrom
XuQianJin-Stars:feature/opendal-cos
Open

feat(vortex-python): Add OpenDAL-backed CosStore to the Python object-store API#8845
XuQianJin-Stars wants to merge 1 commit into
vortex-data:developfrom
XuQianJin-Stars:feature/opendal-cos

Conversation

@XuQianJin-Stars

Copy link
Copy Markdown

Bump opendal / object_store_opendal to 0.57.0 (in vortex-object-store-opendal); opendal 0.57 enables COS versioning by default.

Vendor pyo3-object_store 0.11.0 via [patch.crates-io] so we can add From<Arc<dyn ObjectStore>> for PyObjectStore. This lets any store implementation (including the new OpenDAL-backed COS store) be wrapped as a PyObjectStore and passed directly to vortex.io.read_url(store=...) / vortex.io.write(..., store=...).

Add vendor/pyo3-object_store with the new from_store constructor + From impl.

Expose the CosStore pyclass (gated on the opendal feature) that builds an OpenDAL services::Cos OpendalStore and can be passed as the store= argument.

Introduce AnyVortexStore in io.rs so read_url / write accept both the built-in pyo3-object_store classes and the new COS store.

Document the standalone-store usage and add unit tests.

Rationale for this change

Closes: #8844

Vortex's Python API already exposes concrete, standalone store objects for S3/Azure/GCS/HTTP/Local/Memory, but Tencent COS was only reachable through URL-based cos:// resolution that relied on environment variables. This adds a first-class CosStore so COS can be used exactly like the other stores (construct once, pass via store=), with better ergonomics and type checking.

What changes are included in this PR?

  • opendal / object_store_opendal bumped to 0.57.0
  • vendored pyo3-object_store 0.11.0 with the from_store constructor
  • new CosStore pyclass (opendal feature) wrapping an OpenDAL COS OpendalStore
  • AnyVortexStore extractor in io.rs so read_url/write accept the new store
  • docs + unit tests for the standalone-store path

What APIs are changed? Are there any user-facing changes?

New public API (Python, opendal feature):

  • vortex.store.CosStore(bucket, endpoint, *, secret_id=None, secret_key=None, root=None, disable_config_load=False) — a standalone ObjectStore you can pass to vortex.io.read_url(url, store=...) / vortex.io.write(arrays, path, store=...).

User-facing change: users can now build a COS store object directly and pass it to read_url/write, instead of only configuring cos:// via environment variables (TENCENTCLOUD_SECRET_ID / TENCENTCLOUD_SECRET_KEY / COS_ENDPOINT). The existing cos:// URL resolution is unchanged.

No breaking changes to existing APIs.

…-store API

Signed-off-by: forwardxu <forwardxu@apache.org>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Feat:Add OpenDAL-backed CosStore to the Python object-store API

1 participant