Skip to content

Feat:Add OpenDAL-backed CosStore to the Python object-store API #8844

Description

@XuQianJin-Stars

Problem or motivation

Vortex's Python API exposes concrete, standalone store objects for S3/Azure/GCS/HTTP/Local/Memory (e.g. vortex.store.S3Store), which can be constructed once and passed directly to vortex.io.read_url / vortex.io.write via the store= argument.

Tencent Cloud COS is currently only reachable through URL-based resolution (cos://), which depends on the opendal feature and requires configuration via environment variables (e.g. TENCENTCLOUD_SECRET_ID / COS_ENDPOINT). There is no way to build a concrete COS store object and hand it to read_url/write the way users can with the other stores — an inconsistency that also makes credential handling less ergonomic and less type-checked.

Additionally, the OpenDAL dependency is pinned at 0.56.0; bumping to 0.57.0 unblocks newer service behavior (e.g. COS versioning is now enabled by default).

Proposed solution

- Add a new pyclass, `vortex.store.CosStore`, feature-gated on `opendal`. It builds an OpenDAL `services::Cos` operator bridged to an `object_store::ObjectStore` via `object_store_opendal::OpendalStore`, wrapped as a `PyObjectStore` so it can be passed directly to `read_url`/`write` via `store=`, exactly like the built-in S3/Azure/GCS stores.
- Extend `read_url`/`write` with a small extractor (`AnyVortexStore`) that recognizes the new OpenDAL-backed class and normalizes it to `Arc<dyn ObjectStore>`.
- `From<Arc<dyn ObjectStore>> for PyObjectStore` is required but blocked by the orphan rule, so vendor the `pyo3-object_store` source under `[patch.crates-io]` and add the needed `from_store` constructor there.
- Bump `opendal` and `object_store_opendal` to `0.57.0` (in `vortex-object-store-opendal`).

Example:

```python
from vortex.store import CosStore
from vortex.io import read_url

store = CosStore(
    bucket="my-bucket",
    endpoint="https://cos.ap-guangzhou.myqcloud.com",
    secret_id="AKID...",
    secret_key="...",
)
array = read_url("cos://my-bucket/data.vortex", store=store)

Notes:

  • CosStore is only available when Vortex is built with the opendal feature; the existing cos:// environment-variable resolution is unchanged.
  • This is intentionally scoped to COS. Other OpenDAL-backed providers (e.g. OSS) can be added later by following the same pattern.

Additional context

No response

Metadata

Metadata

Assignees

No one assigned

    Labels

    featureA feature request

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions