diff --git a/content/de/developer/integration/big-data/images/rustfs-mlflow-artifacts.png b/content/de/developer/integration/ai/images/rustfs-mlflow-artifacts.png similarity index 100% rename from content/de/developer/integration/big-data/images/rustfs-mlflow-artifacts.png rename to content/de/developer/integration/ai/images/rustfs-mlflow-artifacts.png diff --git a/content/de/developer/integration/ai/index.md b/content/de/developer/integration/ai/index.md index 4d60d869..c30b5b34 100644 --- a/content/de/developer/integration/ai/index.md +++ b/content/de/developer/integration/ai/index.md @@ -7,6 +7,7 @@ Nutzen Sie **RustFS** als Objektspeicher-Layer für KI- und Machine-Learning-Pla ## Plattformen +- [MLflow](./mlflow.md) - [Ray](./ray.md) - [vLLM](./vllm.md) diff --git a/content/de/developer/integration/ai/meta.json b/content/de/developer/integration/ai/meta.json index b30f9e0f..9f8472c1 100644 --- a/content/de/developer/integration/ai/meta.json +++ b/content/de/developer/integration/ai/meta.json @@ -1,6 +1,7 @@ { "title": "AI", "pages": [ + "mlflow", "ray", "vllm" ] diff --git a/content/de/developer/integration/big-data/mlflow.md b/content/de/developer/integration/ai/mlflow.md similarity index 100% rename from content/de/developer/integration/big-data/mlflow.md rename to content/de/developer/integration/ai/mlflow.md diff --git a/content/de/developer/integration/big-data/index.md b/content/de/developer/integration/big-data/index.md index 54245dee..edb15138 100644 --- a/content/de/developer/integration/big-data/index.md +++ b/content/de/developer/integration/big-data/index.md @@ -1,30 +1,20 @@ --- title: "Datenanalyse" -description: "Connect data analytics systems to RustFS through S3-compatible object storage interfaces." +description: "Connect big data systems to RustFS through S3-compatible object storage interfaces." --- Use **RustFS** as the object storage layer for data analytics systems that support an S3-compatible endpoint. ## Systems -- [ClickHouse](./clickhouse.md) - [Airflow](./airflow.md) +- [Delta Lake](./delta-lake.md) +- [Flink](./flink.md) - [Hudi](./hudi.md) - [Iceberg](./iceberg.md) -- [PyIceberg](./pyiceberg.md) -- [Milvus](./milvus.md) -- [MLflow](./mlflow.md) -- [OpenDAL](./opendal.md) -- [DuckDB](./duckdb.md) -- [Doris](./doris.md) -- [Delta Lake](./delta-lake.md) -- [lakeFS](./lakefs.md) -- [InfluxDB](./influxdb.md) - [Kafka](./kafka.md) +- [PyIceberg](./pyiceberg.md) - [Spark](./spark.md) -- [Flink](./flink.md) -- [Trino](./trino.md) -- [ZeroFS](./zerofs.md) -- [Vitess](./vitess.md) +- [Zeppelin](./zeppelin.md) -Keep application data in a dedicated bucket and prefix, and use credentials scoped to the required bucket operations. \ No newline at end of file +Keep big data workload data in a dedicated bucket and prefix, and use credentials scoped to the required bucket operations. diff --git a/content/de/developer/integration/big-data/meta.json b/content/de/developer/integration/big-data/meta.json index d5662e19..05d25e50 100644 --- a/content/de/developer/integration/big-data/meta.json +++ b/content/de/developer/integration/big-data/meta.json @@ -1,25 +1,14 @@ { - "title": "Data Analytics", + "title": "Big Data", "pages": [ - "clickhouse", "airflow", - "duckdb", - "doris", "delta-lake", "flink", "hudi", "iceberg", - "influxdb", "kafka", - "lakefs", - "milvus", - "mlflow", - "opendal", "pyiceberg", "spark", - "trino", - "zerofs", - "vitess", "zeppelin" ] } diff --git a/content/de/developer/integration/big-data/clickhouse.md b/content/de/developer/integration/database/clickhouse.md similarity index 100% rename from content/de/developer/integration/big-data/clickhouse.md rename to content/de/developer/integration/database/clickhouse.md diff --git a/content/de/developer/integration/big-data/doris.md b/content/de/developer/integration/database/doris.md similarity index 100% rename from content/de/developer/integration/big-data/doris.md rename to content/de/developer/integration/database/doris.md diff --git a/content/de/developer/integration/big-data/duckdb.md b/content/de/developer/integration/database/duckdb.md similarity index 100% rename from content/de/developer/integration/big-data/duckdb.md rename to content/de/developer/integration/database/duckdb.md diff --git a/content/de/developer/integration/big-data/images/rustfs-clickhouse-disk.png b/content/de/developer/integration/database/images/rustfs-clickhouse-disk.png similarity index 100% rename from content/de/developer/integration/big-data/images/rustfs-clickhouse-disk.png rename to content/de/developer/integration/database/images/rustfs-clickhouse-disk.png diff --git a/content/de/developer/integration/big-data/images/rustfs-doris-backup.png b/content/de/developer/integration/database/images/rustfs-doris-backup.png similarity index 100% rename from content/de/developer/integration/big-data/images/rustfs-doris-backup.png rename to content/de/developer/integration/database/images/rustfs-doris-backup.png diff --git a/content/de/developer/integration/big-data/images/rustfs-duckdb-objects.png b/content/de/developer/integration/database/images/rustfs-duckdb-objects.png similarity index 100% rename from content/de/developer/integration/big-data/images/rustfs-duckdb-objects.png rename to content/de/developer/integration/database/images/rustfs-duckdb-objects.png diff --git a/content/de/developer/integration/big-data/images/rustfs-influxdb-objects.png b/content/de/developer/integration/database/images/rustfs-influxdb-objects.png similarity index 100% rename from content/de/developer/integration/big-data/images/rustfs-influxdb-objects.png rename to content/de/developer/integration/database/images/rustfs-influxdb-objects.png diff --git a/content/de/developer/integration/database/images/rustfs-lancedb-table.png b/content/de/developer/integration/database/images/rustfs-lancedb-table.png new file mode 100644 index 00000000..7d56adac Binary files /dev/null and b/content/de/developer/integration/database/images/rustfs-lancedb-table.png differ diff --git a/content/de/developer/integration/big-data/images/rustfs-trino-objects.png b/content/de/developer/integration/database/images/rustfs-trino-objects.png similarity index 100% rename from content/de/developer/integration/big-data/images/rustfs-trino-objects.png rename to content/de/developer/integration/database/images/rustfs-trino-objects.png diff --git a/content/de/developer/integration/big-data/images/rustfs-vitess-backups.png b/content/de/developer/integration/database/images/rustfs-vitess-backups.png similarity index 100% rename from content/de/developer/integration/big-data/images/rustfs-vitess-backups.png rename to content/de/developer/integration/database/images/rustfs-vitess-backups.png diff --git a/content/de/developer/integration/database/index.md b/content/de/developer/integration/database/index.md new file mode 100644 index 00000000..ad29dd08 --- /dev/null +++ b/content/de/developer/integration/database/index.md @@ -0,0 +1,19 @@ +--- +title: "Database" +description: "Connect databases to RustFS through S3-compatible object storage interfaces." +--- + +Use **RustFS** as the object storage layer for databases that support an S3-compatible endpoint. + +## Databases + +- [ClickHouse](./clickhouse.md) +- [Doris](./doris.md) +- [DuckDB](./duckdb.md) +- [InfluxDB](./influxdb.md) +- [LanceDB](./lancedb.md) +- [Milvus](./milvus.md) +- [Trino](./trino.md) +- [Vitess](./vitess.md) + +Keep database data and backups in a dedicated bucket and prefix, and use credentials scoped to the required bucket operations. diff --git a/content/de/developer/integration/big-data/influxdb.md b/content/de/developer/integration/database/influxdb.md similarity index 100% rename from content/de/developer/integration/big-data/influxdb.md rename to content/de/developer/integration/database/influxdb.md diff --git a/content/de/developer/integration/database/lancedb.md b/content/de/developer/integration/database/lancedb.md new file mode 100644 index 00000000..38a0703a --- /dev/null +++ b/content/de/developer/integration/database/lancedb.md @@ -0,0 +1,125 @@ +--- +title: "LanceDB" +description: "Store and query LanceDB vector tables directly on RustFS." +--- + +This guide connects [LanceDB](https://github.com/lancedb/lancedb) — the open-source vector database built on the Lance columnar format — to **RustFS** as its storage backend. You will create a vector table directly at an `s3://` location, add rows, run a vector search, and confirm the Lance table files in the bucket. The workflow was verified with the `lancedb` Python package against `rustfs/rustfs-x86-musl:v2.3.1`. + +You need Python 3.9 or newer. This deployment is intended for local integration testing, not production. + +## Architecture + +```mermaid +flowchart LR + App["Python client"] -->|"connect s3://"| LanceDB["LanceDB"] + LanceDB -->|"Lance fragments + manifests"| RustFS["RustFS :9000"] +``` + +LanceDB is embedded: there is no server to run. The Python (or Rust, or JavaScript) client talks to the bucket directly, storing each table as a `*.lance` directory with data fragments, version manifests, and transaction logs. + +## 1. Install the client + +```bash +pip install lancedb +``` + +## 2. Create a table on RustFS + +Connect straight to the bucket and create a table, replacing all connection placeholders. The `storage_options` keys follow Lance's object-store conventions; custom endpoints use path-style addressing: + +```python title="lance_s3.py" +import lancedb + +storage_options = { + "endpoint": "http://:9000", + "access_key_id": "", + "secret_access_key": "", + "region": "us-east-1", + "allow_http": "true", +} + +db = lancedb.connect("s3:///tables", storage_options=storage_options) + +rows = [{"id": i, "label": f"row-{i}", "vector": [float(i) / 10, 0.5, 0.25, 0.1] * 2} + for i in range(5)] +table = db.create_table("events", data=rows) +print("created:", table.count_rows(), "rows") + +table.add([{"id": 99, "label": "query-target", "vector": [0.9, 0.5, 0.25, 0.1] * 2}]) +print("after add:", table.count_rows(), "rows") +``` + +```text +created: 5 rows +after add: 6 rows +``` + +The table URI uses the `s3://bucket/prefix` form; every write and read goes to RustFS over its S3 API. + +## 3. Run a vector search + +```python title="lance_search.py" +import lancedb + +db = lancedb.connect("s3:///tables", storage_options=storage_options) +table = db.open_table("events") + +res = table.search([0.9, 0.5, 0.25, 0.1] * 2).limit(3).to_list() +print("top3:", [(r["id"], r["label"]) for r in res]) +print("version:", table.version) +``` + +```text +top3: [(99, 'query-target'), (4, 'row-4'), (3, 'row-3')] +version: 2 +``` + +The nearest neighbor is the row added in the previous step, and the version counter reflects the two commits (create and add). + +## 4. Verify objects in RustFS + +List the table prefix: + +```bash +rc ls rustfs// -r +``` + +Each table is a `.lance` directory holding data fragments, version manifests, and transaction records: + +```text +tables/events.lance/_transactions/0-b93cf795-fc3d-4bc9-88c8-e37baeefdd46.txn +tables/events.lance/_versions/18446744073709551613.manifest +tables/events.lance/data/101101011110110010000100e7a149411b847dbad6ebc7d47e.lance +``` + +Multiple tables share the bucket under the `tables/` prefix, so one bucket can back an entire LanceDB workspace. + +![LanceDB table files stored in the RustFS Console](./images/rustfs-lancedb-table.png) + +## 5. Stop or reset + +LanceDB holds no server state. To delete the table: + +```bash +rc rm rustfs//tables/ --recursive --force +``` + +## Troubleshooting + +### Connection or signature errors on first use + +Confirm `endpoint` includes the scheme, `allow_http` is `"true"` for plain-HTTP endpoints, and the bucket exists. The `region` value is required by the S3 signer even though RustFS ignores it. + +### `Table not found` after creating it + +LanceDB lists the bucket prefix to discover tables. A stale client cache or a wrong `tables/` prefix in the URI makes new tables invisible; reconnect with the same URI used at creation time. + +### Slow bulk loads over the network + +Lance writes one fragment per commit. For large imports, batch rows into fewer `table.add` calls — each call produces a new data file in the bucket. + +## Next steps + +- Review [S3 compatibility notes](/administration/protocols/s3) before adopting additional LanceDB storage options. +- Create dedicated production credentials with [Access Key Management](/security-compliance/iam/access-token). +- Follow the [LanceDB documentation](https://lancedb.github.io/lancedb/) for ANN indexes, hybrid search, and multi-tenant bucket layouts on top of the same backend. diff --git a/content/de/developer/integration/database/meta.json b/content/de/developer/integration/database/meta.json new file mode 100644 index 00000000..308b1815 --- /dev/null +++ b/content/de/developer/integration/database/meta.json @@ -0,0 +1,13 @@ +{ + "title": "Database", + "pages": [ + "clickhouse", + "doris", + "duckdb", + "influxdb", + "lancedb", + "milvus", + "trino", + "vitess" + ] +} diff --git a/content/de/developer/integration/big-data/milvus.md b/content/de/developer/integration/database/milvus.md similarity index 100% rename from content/de/developer/integration/big-data/milvus.md rename to content/de/developer/integration/database/milvus.md diff --git a/content/de/developer/integration/big-data/trino.md b/content/de/developer/integration/database/trino.md similarity index 100% rename from content/de/developer/integration/big-data/trino.md rename to content/de/developer/integration/database/trino.md diff --git a/content/de/developer/integration/big-data/vitess.md b/content/de/developer/integration/database/vitess.md similarity index 100% rename from content/de/developer/integration/big-data/vitess.md rename to content/de/developer/integration/database/vitess.md diff --git a/content/de/developer/integration/index.md b/content/de/developer/integration/index.md index 2fb5a6b5..5f6e49e8 100644 --- a/content/de/developer/integration/index.md +++ b/content/de/developer/integration/index.md @@ -10,7 +10,9 @@ Use this section to connect **RustFS** to infrastructure and application platfor - [Reverse Proxy](./reverse-proxy/index.md) covers Nginx, Traefik, Caddy, HAProxy, and Envoy. - [Backup](./backup/index.md) covers Kopia, Longhorn, Restic, and Velero. - [AI](./ai/index.md) covers AI platforms including Ray and vLLM. -- [Datenanalyse](./big-data/index.md) covers analytics systems including Airflow, ClickHouse, Delta Lake, Doris, Hudi, Iceberg, Kafka, lakeFS, Milvus, OpenDAL, Vitess, Zeppelin, and ZeroFS. +- [Database](./database/index.md) covers ClickHouse, Doris, DuckDB, InfluxDB, LanceDB, Milvus, Trino, and Vitess. +- [Big Data](./big-data/index.md) covers Airflow, Delta Lake, Flink, Hudi, Iceberg, Kafka, PyIceberg, Spark, and Zeppelin. +- [Storage](./storage/index.md) covers lakeFS, OpenDAL, and ZeroFS. - [Cloud Native](./cloud-native/index.md) covers Cortex and Flux. - [Observability](./observability/index.md) covers telemetry systems including Fluentd, GreptimeDB, Loki, OpenObserve, OpenTelemetry, Tempo, Thanos, and VictoriaMetrics. - [Others](./others/index.md) covers the capo SDK, rclone, JuiceFS, Nextcloud, and tusd. diff --git a/content/de/developer/integration/meta.json b/content/de/developer/integration/meta.json index f5a8d87c..19173e04 100644 --- a/content/de/developer/integration/meta.json +++ b/content/de/developer/integration/meta.json @@ -3,7 +3,9 @@ "pages": [ "reverse-proxy", "backup", + "database", "big-data", + "storage", "ai", "cloud-native", "observability", diff --git a/content/de/developer/integration/big-data/images/rustfs-lakefs-repo.png b/content/de/developer/integration/storage/images/rustfs-lakefs-repo.png similarity index 100% rename from content/de/developer/integration/big-data/images/rustfs-lakefs-repo.png rename to content/de/developer/integration/storage/images/rustfs-lakefs-repo.png diff --git a/content/de/developer/integration/big-data/images/rustfs-opendal-objects.png b/content/de/developer/integration/storage/images/rustfs-opendal-objects.png similarity index 100% rename from content/de/developer/integration/big-data/images/rustfs-opendal-objects.png rename to content/de/developer/integration/storage/images/rustfs-opendal-objects.png diff --git a/content/de/developer/integration/big-data/images/rustfs-zerofs-data.png b/content/de/developer/integration/storage/images/rustfs-zerofs-data.png similarity index 100% rename from content/de/developer/integration/big-data/images/rustfs-zerofs-data.png rename to content/de/developer/integration/storage/images/rustfs-zerofs-data.png diff --git a/content/de/developer/integration/storage/index.md b/content/de/developer/integration/storage/index.md new file mode 100644 index 00000000..87a1262f --- /dev/null +++ b/content/de/developer/integration/storage/index.md @@ -0,0 +1,14 @@ +--- +title: "Storage" +description: "Use RustFS as the S3 backend for storage systems and storage gateways." +--- + +Use **RustFS** as the backend for storage systems and gateways built on top of object storage. + +## Systems + +- [lakeFS](./lakefs.md) +- [OpenDAL](./opendal.md) +- [ZeroFS](./zerofs.md) + +Use a dedicated bucket and prefix per system, and scope credentials to the required bucket operations. diff --git a/content/de/developer/integration/big-data/lakefs.md b/content/de/developer/integration/storage/lakefs.md similarity index 100% rename from content/de/developer/integration/big-data/lakefs.md rename to content/de/developer/integration/storage/lakefs.md diff --git a/content/de/developer/integration/storage/meta.json b/content/de/developer/integration/storage/meta.json new file mode 100644 index 00000000..2c912239 --- /dev/null +++ b/content/de/developer/integration/storage/meta.json @@ -0,0 +1,8 @@ +{ + "title": "Storage", + "pages": [ + "lakefs", + "opendal", + "zerofs" + ] +} diff --git a/content/de/developer/integration/big-data/opendal.md b/content/de/developer/integration/storage/opendal.md similarity index 100% rename from content/de/developer/integration/big-data/opendal.md rename to content/de/developer/integration/storage/opendal.md diff --git a/content/de/developer/integration/big-data/zerofs.md b/content/de/developer/integration/storage/zerofs.md similarity index 100% rename from content/de/developer/integration/big-data/zerofs.md rename to content/de/developer/integration/storage/zerofs.md diff --git a/content/en/developer/integration/big-data/images/rustfs-mlflow-artifacts.png b/content/en/developer/integration/ai/images/rustfs-mlflow-artifacts.png similarity index 100% rename from content/en/developer/integration/big-data/images/rustfs-mlflow-artifacts.png rename to content/en/developer/integration/ai/images/rustfs-mlflow-artifacts.png diff --git a/content/en/developer/integration/ai/index.md b/content/en/developer/integration/ai/index.md index 572f68bb..e9626c24 100644 --- a/content/en/developer/integration/ai/index.md +++ b/content/en/developer/integration/ai/index.md @@ -7,6 +7,7 @@ Use **RustFS** as the object storage layer for AI and machine learning platforms ## Platforms +- [MLflow](./mlflow.md) - [Ray](./ray.md) - [vLLM](./vllm.md) diff --git a/content/en/developer/integration/ai/meta.json b/content/en/developer/integration/ai/meta.json index b30f9e0f..9f8472c1 100644 --- a/content/en/developer/integration/ai/meta.json +++ b/content/en/developer/integration/ai/meta.json @@ -1,6 +1,7 @@ { "title": "AI", "pages": [ + "mlflow", "ray", "vllm" ] diff --git a/content/en/developer/integration/big-data/mlflow.md b/content/en/developer/integration/ai/mlflow.md similarity index 100% rename from content/en/developer/integration/big-data/mlflow.md rename to content/en/developer/integration/ai/mlflow.md diff --git a/content/en/developer/integration/big-data/index.md b/content/en/developer/integration/big-data/index.md index 82d0c544..fa9024cc 100644 --- a/content/en/developer/integration/big-data/index.md +++ b/content/en/developer/integration/big-data/index.md @@ -1,30 +1,20 @@ --- -title: "Data Analytics" -description: "Connect data analytics systems to RustFS through S3-compatible object storage interfaces." +title: "Big Data" +description: "Connect big data systems to RustFS through S3-compatible object storage interfaces." --- Use **RustFS** as the object storage layer for data analytics systems that support an S3-compatible endpoint. ## Systems -- [ClickHouse](./clickhouse.md) - [Airflow](./airflow.md) +- [Delta Lake](./delta-lake.md) +- [Flink](./flink.md) - [Hudi](./hudi.md) - [Iceberg](./iceberg.md) -- [PyIceberg](./pyiceberg.md) -- [Milvus](./milvus.md) -- [MLflow](./mlflow.md) -- [OpenDAL](./opendal.md) -- [DuckDB](./duckdb.md) -- [Doris](./doris.md) -- [Delta Lake](./delta-lake.md) -- [lakeFS](./lakefs.md) -- [InfluxDB](./influxdb.md) - [Kafka](./kafka.md) +- [PyIceberg](./pyiceberg.md) - [Spark](./spark.md) -- [Flink](./flink.md) -- [Trino](./trino.md) -- [ZeroFS](./zerofs.md) -- [Vitess](./vitess.md) +- [Zeppelin](./zeppelin.md) -Keep application data in a dedicated bucket and prefix, and use credentials scoped to the required bucket operations. \ No newline at end of file +Keep big data workload data in a dedicated bucket and prefix, and use credentials scoped to the required bucket operations. diff --git a/content/en/developer/integration/big-data/meta.json b/content/en/developer/integration/big-data/meta.json index d5662e19..05d25e50 100644 --- a/content/en/developer/integration/big-data/meta.json +++ b/content/en/developer/integration/big-data/meta.json @@ -1,25 +1,14 @@ { - "title": "Data Analytics", + "title": "Big Data", "pages": [ - "clickhouse", "airflow", - "duckdb", - "doris", "delta-lake", "flink", "hudi", "iceberg", - "influxdb", "kafka", - "lakefs", - "milvus", - "mlflow", - "opendal", "pyiceberg", "spark", - "trino", - "zerofs", - "vitess", "zeppelin" ] } diff --git a/content/en/developer/integration/big-data/clickhouse.md b/content/en/developer/integration/database/clickhouse.md similarity index 100% rename from content/en/developer/integration/big-data/clickhouse.md rename to content/en/developer/integration/database/clickhouse.md diff --git a/content/en/developer/integration/big-data/doris.md b/content/en/developer/integration/database/doris.md similarity index 100% rename from content/en/developer/integration/big-data/doris.md rename to content/en/developer/integration/database/doris.md diff --git a/content/en/developer/integration/big-data/duckdb.md b/content/en/developer/integration/database/duckdb.md similarity index 100% rename from content/en/developer/integration/big-data/duckdb.md rename to content/en/developer/integration/database/duckdb.md diff --git a/content/en/developer/integration/big-data/images/rustfs-clickhouse-disk.png b/content/en/developer/integration/database/images/rustfs-clickhouse-disk.png similarity index 100% rename from content/en/developer/integration/big-data/images/rustfs-clickhouse-disk.png rename to content/en/developer/integration/database/images/rustfs-clickhouse-disk.png diff --git a/content/en/developer/integration/big-data/images/rustfs-doris-backup.png b/content/en/developer/integration/database/images/rustfs-doris-backup.png similarity index 100% rename from content/en/developer/integration/big-data/images/rustfs-doris-backup.png rename to content/en/developer/integration/database/images/rustfs-doris-backup.png diff --git a/content/en/developer/integration/big-data/images/rustfs-duckdb-objects.png b/content/en/developer/integration/database/images/rustfs-duckdb-objects.png similarity index 100% rename from content/en/developer/integration/big-data/images/rustfs-duckdb-objects.png rename to content/en/developer/integration/database/images/rustfs-duckdb-objects.png diff --git a/content/en/developer/integration/big-data/images/rustfs-influxdb-objects.png b/content/en/developer/integration/database/images/rustfs-influxdb-objects.png similarity index 100% rename from content/en/developer/integration/big-data/images/rustfs-influxdb-objects.png rename to content/en/developer/integration/database/images/rustfs-influxdb-objects.png diff --git a/content/en/developer/integration/database/images/rustfs-lancedb-table.png b/content/en/developer/integration/database/images/rustfs-lancedb-table.png new file mode 100644 index 00000000..7d56adac Binary files /dev/null and b/content/en/developer/integration/database/images/rustfs-lancedb-table.png differ diff --git a/content/en/developer/integration/big-data/images/rustfs-trino-objects.png b/content/en/developer/integration/database/images/rustfs-trino-objects.png similarity index 100% rename from content/en/developer/integration/big-data/images/rustfs-trino-objects.png rename to content/en/developer/integration/database/images/rustfs-trino-objects.png diff --git a/content/en/developer/integration/big-data/images/rustfs-vitess-backups.png b/content/en/developer/integration/database/images/rustfs-vitess-backups.png similarity index 100% rename from content/en/developer/integration/big-data/images/rustfs-vitess-backups.png rename to content/en/developer/integration/database/images/rustfs-vitess-backups.png diff --git a/content/en/developer/integration/database/index.md b/content/en/developer/integration/database/index.md new file mode 100644 index 00000000..ad29dd08 --- /dev/null +++ b/content/en/developer/integration/database/index.md @@ -0,0 +1,19 @@ +--- +title: "Database" +description: "Connect databases to RustFS through S3-compatible object storage interfaces." +--- + +Use **RustFS** as the object storage layer for databases that support an S3-compatible endpoint. + +## Databases + +- [ClickHouse](./clickhouse.md) +- [Doris](./doris.md) +- [DuckDB](./duckdb.md) +- [InfluxDB](./influxdb.md) +- [LanceDB](./lancedb.md) +- [Milvus](./milvus.md) +- [Trino](./trino.md) +- [Vitess](./vitess.md) + +Keep database data and backups in a dedicated bucket and prefix, and use credentials scoped to the required bucket operations. diff --git a/content/en/developer/integration/big-data/influxdb.md b/content/en/developer/integration/database/influxdb.md similarity index 100% rename from content/en/developer/integration/big-data/influxdb.md rename to content/en/developer/integration/database/influxdb.md diff --git a/content/en/developer/integration/database/lancedb.md b/content/en/developer/integration/database/lancedb.md new file mode 100644 index 00000000..38a0703a --- /dev/null +++ b/content/en/developer/integration/database/lancedb.md @@ -0,0 +1,125 @@ +--- +title: "LanceDB" +description: "Store and query LanceDB vector tables directly on RustFS." +--- + +This guide connects [LanceDB](https://github.com/lancedb/lancedb) — the open-source vector database built on the Lance columnar format — to **RustFS** as its storage backend. You will create a vector table directly at an `s3://` location, add rows, run a vector search, and confirm the Lance table files in the bucket. The workflow was verified with the `lancedb` Python package against `rustfs/rustfs-x86-musl:v2.3.1`. + +You need Python 3.9 or newer. This deployment is intended for local integration testing, not production. + +## Architecture + +```mermaid +flowchart LR + App["Python client"] -->|"connect s3://"| LanceDB["LanceDB"] + LanceDB -->|"Lance fragments + manifests"| RustFS["RustFS :9000"] +``` + +LanceDB is embedded: there is no server to run. The Python (or Rust, or JavaScript) client talks to the bucket directly, storing each table as a `*.lance` directory with data fragments, version manifests, and transaction logs. + +## 1. Install the client + +```bash +pip install lancedb +``` + +## 2. Create a table on RustFS + +Connect straight to the bucket and create a table, replacing all connection placeholders. The `storage_options` keys follow Lance's object-store conventions; custom endpoints use path-style addressing: + +```python title="lance_s3.py" +import lancedb + +storage_options = { + "endpoint": "http://:9000", + "access_key_id": "", + "secret_access_key": "", + "region": "us-east-1", + "allow_http": "true", +} + +db = lancedb.connect("s3:///tables", storage_options=storage_options) + +rows = [{"id": i, "label": f"row-{i}", "vector": [float(i) / 10, 0.5, 0.25, 0.1] * 2} + for i in range(5)] +table = db.create_table("events", data=rows) +print("created:", table.count_rows(), "rows") + +table.add([{"id": 99, "label": "query-target", "vector": [0.9, 0.5, 0.25, 0.1] * 2}]) +print("after add:", table.count_rows(), "rows") +``` + +```text +created: 5 rows +after add: 6 rows +``` + +The table URI uses the `s3://bucket/prefix` form; every write and read goes to RustFS over its S3 API. + +## 3. Run a vector search + +```python title="lance_search.py" +import lancedb + +db = lancedb.connect("s3:///tables", storage_options=storage_options) +table = db.open_table("events") + +res = table.search([0.9, 0.5, 0.25, 0.1] * 2).limit(3).to_list() +print("top3:", [(r["id"], r["label"]) for r in res]) +print("version:", table.version) +``` + +```text +top3: [(99, 'query-target'), (4, 'row-4'), (3, 'row-3')] +version: 2 +``` + +The nearest neighbor is the row added in the previous step, and the version counter reflects the two commits (create and add). + +## 4. Verify objects in RustFS + +List the table prefix: + +```bash +rc ls rustfs// -r +``` + +Each table is a `.lance` directory holding data fragments, version manifests, and transaction records: + +```text +tables/events.lance/_transactions/0-b93cf795-fc3d-4bc9-88c8-e37baeefdd46.txn +tables/events.lance/_versions/18446744073709551613.manifest +tables/events.lance/data/101101011110110010000100e7a149411b847dbad6ebc7d47e.lance +``` + +Multiple tables share the bucket under the `tables/` prefix, so one bucket can back an entire LanceDB workspace. + +![LanceDB table files stored in the RustFS Console](./images/rustfs-lancedb-table.png) + +## 5. Stop or reset + +LanceDB holds no server state. To delete the table: + +```bash +rc rm rustfs//tables/ --recursive --force +``` + +## Troubleshooting + +### Connection or signature errors on first use + +Confirm `endpoint` includes the scheme, `allow_http` is `"true"` for plain-HTTP endpoints, and the bucket exists. The `region` value is required by the S3 signer even though RustFS ignores it. + +### `Table not found` after creating it + +LanceDB lists the bucket prefix to discover tables. A stale client cache or a wrong `tables/` prefix in the URI makes new tables invisible; reconnect with the same URI used at creation time. + +### Slow bulk loads over the network + +Lance writes one fragment per commit. For large imports, batch rows into fewer `table.add` calls — each call produces a new data file in the bucket. + +## Next steps + +- Review [S3 compatibility notes](/administration/protocols/s3) before adopting additional LanceDB storage options. +- Create dedicated production credentials with [Access Key Management](/security-compliance/iam/access-token). +- Follow the [LanceDB documentation](https://lancedb.github.io/lancedb/) for ANN indexes, hybrid search, and multi-tenant bucket layouts on top of the same backend. diff --git a/content/en/developer/integration/database/meta.json b/content/en/developer/integration/database/meta.json new file mode 100644 index 00000000..308b1815 --- /dev/null +++ b/content/en/developer/integration/database/meta.json @@ -0,0 +1,13 @@ +{ + "title": "Database", + "pages": [ + "clickhouse", + "doris", + "duckdb", + "influxdb", + "lancedb", + "milvus", + "trino", + "vitess" + ] +} diff --git a/content/en/developer/integration/big-data/milvus.md b/content/en/developer/integration/database/milvus.md similarity index 100% rename from content/en/developer/integration/big-data/milvus.md rename to content/en/developer/integration/database/milvus.md diff --git a/content/en/developer/integration/big-data/trino.md b/content/en/developer/integration/database/trino.md similarity index 100% rename from content/en/developer/integration/big-data/trino.md rename to content/en/developer/integration/database/trino.md diff --git a/content/en/developer/integration/big-data/vitess.md b/content/en/developer/integration/database/vitess.md similarity index 100% rename from content/en/developer/integration/big-data/vitess.md rename to content/en/developer/integration/database/vitess.md diff --git a/content/en/developer/integration/index.md b/content/en/developer/integration/index.md index 110d94f0..cc489c58 100644 --- a/content/en/developer/integration/index.md +++ b/content/en/developer/integration/index.md @@ -1,6 +1,6 @@ --- title: "Integration" -description: "Integrate RustFS with reverse proxies, backup tools, data analytics systems, AI and cloud native platforms, observability platforms, container registries, and DevOps tooling." +description: "Integrate RustFS with reverse proxies, backup tools, databases, big data and storage systems, AI and cloud native platforms, observability platforms, container registries, and DevOps tooling." --- Use this section to connect **RustFS** to infrastructure and application platforms through its S3-compatible API. @@ -9,8 +9,10 @@ Use this section to connect **RustFS** to infrastructure and application platfor - [Reverse Proxy](./reverse-proxy/index.md) covers Nginx, Traefik, Caddy, HAProxy, and Envoy. - [Backup](./backup/index.md) covers Kopia, Longhorn, Restic, and Velero. -- [AI](./ai/index.md) covers AI platforms including Ray and vLLM. -- [Data Analytics](./big-data/index.md) covers analytics systems including Airflow, ClickHouse, Delta Lake, Doris, Hudi, Iceberg, Kafka, lakeFS, Milvus, OpenDAL, Vitess, Zeppelin, and ZeroFS. +- [AI](./ai/index.md) covers AI platforms including MLflow, Ray, and vLLM. +- [Database](./database/index.md) covers ClickHouse, Doris, DuckDB, InfluxDB, LanceDB, Milvus, Trino, and Vitess. +- [Big Data](./big-data/index.md) covers Airflow, Delta Lake, Flink, Hudi, Iceberg, Kafka, PyIceberg, Spark, and Zeppelin. +- [Storage](./storage/index.md) covers lakeFS, OpenDAL, and ZeroFS. - [Cloud Native](./cloud-native/index.md) covers Cortex and Flux. - [Observability](./observability/index.md) covers telemetry systems including Fluentd, GreptimeDB, Loki, OpenObserve, OpenTelemetry, Tempo, Thanos, and VictoriaMetrics. - [Others](./others/index.md) covers the capo SDK, rclone, JuiceFS, Nextcloud, and tusd. diff --git a/content/en/developer/integration/meta.json b/content/en/developer/integration/meta.json index f5a8d87c..19173e04 100644 --- a/content/en/developer/integration/meta.json +++ b/content/en/developer/integration/meta.json @@ -3,7 +3,9 @@ "pages": [ "reverse-proxy", "backup", + "database", "big-data", + "storage", "ai", "cloud-native", "observability", diff --git a/content/en/developer/integration/big-data/images/rustfs-lakefs-repo.png b/content/en/developer/integration/storage/images/rustfs-lakefs-repo.png similarity index 100% rename from content/en/developer/integration/big-data/images/rustfs-lakefs-repo.png rename to content/en/developer/integration/storage/images/rustfs-lakefs-repo.png diff --git a/content/en/developer/integration/big-data/images/rustfs-opendal-objects.png b/content/en/developer/integration/storage/images/rustfs-opendal-objects.png similarity index 100% rename from content/en/developer/integration/big-data/images/rustfs-opendal-objects.png rename to content/en/developer/integration/storage/images/rustfs-opendal-objects.png diff --git a/content/en/developer/integration/big-data/images/rustfs-zerofs-data.png b/content/en/developer/integration/storage/images/rustfs-zerofs-data.png similarity index 100% rename from content/en/developer/integration/big-data/images/rustfs-zerofs-data.png rename to content/en/developer/integration/storage/images/rustfs-zerofs-data.png diff --git a/content/en/developer/integration/storage/index.md b/content/en/developer/integration/storage/index.md new file mode 100644 index 00000000..87a1262f --- /dev/null +++ b/content/en/developer/integration/storage/index.md @@ -0,0 +1,14 @@ +--- +title: "Storage" +description: "Use RustFS as the S3 backend for storage systems and storage gateways." +--- + +Use **RustFS** as the backend for storage systems and gateways built on top of object storage. + +## Systems + +- [lakeFS](./lakefs.md) +- [OpenDAL](./opendal.md) +- [ZeroFS](./zerofs.md) + +Use a dedicated bucket and prefix per system, and scope credentials to the required bucket operations. diff --git a/content/en/developer/integration/big-data/lakefs.md b/content/en/developer/integration/storage/lakefs.md similarity index 100% rename from content/en/developer/integration/big-data/lakefs.md rename to content/en/developer/integration/storage/lakefs.md diff --git a/content/en/developer/integration/storage/meta.json b/content/en/developer/integration/storage/meta.json new file mode 100644 index 00000000..2c912239 --- /dev/null +++ b/content/en/developer/integration/storage/meta.json @@ -0,0 +1,8 @@ +{ + "title": "Storage", + "pages": [ + "lakefs", + "opendal", + "zerofs" + ] +} diff --git a/content/en/developer/integration/big-data/opendal.md b/content/en/developer/integration/storage/opendal.md similarity index 100% rename from content/en/developer/integration/big-data/opendal.md rename to content/en/developer/integration/storage/opendal.md diff --git a/content/en/developer/integration/big-data/zerofs.md b/content/en/developer/integration/storage/zerofs.md similarity index 100% rename from content/en/developer/integration/big-data/zerofs.md rename to content/en/developer/integration/storage/zerofs.md diff --git a/content/fr/developer/integration/big-data/images/rustfs-mlflow-artifacts.png b/content/fr/developer/integration/ai/images/rustfs-mlflow-artifacts.png similarity index 100% rename from content/fr/developer/integration/big-data/images/rustfs-mlflow-artifacts.png rename to content/fr/developer/integration/ai/images/rustfs-mlflow-artifacts.png diff --git a/content/fr/developer/integration/ai/index.md b/content/fr/developer/integration/ai/index.md index 7837dbaa..687eca06 100644 --- a/content/fr/developer/integration/ai/index.md +++ b/content/fr/developer/integration/ai/index.md @@ -7,6 +7,7 @@ Utilisez **RustFS** comme couche de stockage objet pour les plateformes d'IA et ## Plateformes +- [MLflow](./mlflow.md) - [Ray](./ray.md) - [vLLM](./vllm.md) diff --git a/content/fr/developer/integration/ai/meta.json b/content/fr/developer/integration/ai/meta.json index 639b7f23..a9488170 100644 --- a/content/fr/developer/integration/ai/meta.json +++ b/content/fr/developer/integration/ai/meta.json @@ -1,6 +1,7 @@ { "title": "IA", "pages": [ + "mlflow", "ray", "vllm" ] diff --git a/content/fr/developer/integration/big-data/mlflow.md b/content/fr/developer/integration/ai/mlflow.md similarity index 100% rename from content/fr/developer/integration/big-data/mlflow.md rename to content/fr/developer/integration/ai/mlflow.md diff --git a/content/fr/developer/integration/big-data/index.md b/content/fr/developer/integration/big-data/index.md index 5e488c39..3a4ae48b 100644 --- a/content/fr/developer/integration/big-data/index.md +++ b/content/fr/developer/integration/big-data/index.md @@ -1,30 +1,20 @@ --- title: "Analyse de données" -description: "Connect data analytics systems to RustFS through S3-compatible object storage interfaces." +description: "Connect big data systems to RustFS through S3-compatible object storage interfaces." --- Use **RustFS** as the object storage layer for data analytics systems that support an S3-compatible endpoint. ## Systems -- [ClickHouse](./clickhouse.md) - [Airflow](./airflow.md) +- [Delta Lake](./delta-lake.md) +- [Flink](./flink.md) - [Hudi](./hudi.md) - [Iceberg](./iceberg.md) -- [PyIceberg](./pyiceberg.md) -- [Milvus](./milvus.md) -- [MLflow](./mlflow.md) -- [OpenDAL](./opendal.md) -- [DuckDB](./duckdb.md) -- [Doris](./doris.md) -- [Delta Lake](./delta-lake.md) -- [lakeFS](./lakefs.md) -- [InfluxDB](./influxdb.md) - [Kafka](./kafka.md) +- [PyIceberg](./pyiceberg.md) - [Spark](./spark.md) -- [Flink](./flink.md) -- [Trino](./trino.md) -- [ZeroFS](./zerofs.md) -- [Vitess](./vitess.md) +- [Zeppelin](./zeppelin.md) -Keep application data in a dedicated bucket and prefix, and use credentials scoped to the required bucket operations. \ No newline at end of file +Keep big data workload data in a dedicated bucket and prefix, and use credentials scoped to the required bucket operations. diff --git a/content/fr/developer/integration/big-data/meta.json b/content/fr/developer/integration/big-data/meta.json index d5662e19..05d25e50 100644 --- a/content/fr/developer/integration/big-data/meta.json +++ b/content/fr/developer/integration/big-data/meta.json @@ -1,25 +1,14 @@ { - "title": "Data Analytics", + "title": "Big Data", "pages": [ - "clickhouse", "airflow", - "duckdb", - "doris", "delta-lake", "flink", "hudi", "iceberg", - "influxdb", "kafka", - "lakefs", - "milvus", - "mlflow", - "opendal", "pyiceberg", "spark", - "trino", - "zerofs", - "vitess", "zeppelin" ] } diff --git a/content/fr/developer/integration/big-data/clickhouse.md b/content/fr/developer/integration/database/clickhouse.md similarity index 100% rename from content/fr/developer/integration/big-data/clickhouse.md rename to content/fr/developer/integration/database/clickhouse.md diff --git a/content/fr/developer/integration/big-data/doris.md b/content/fr/developer/integration/database/doris.md similarity index 100% rename from content/fr/developer/integration/big-data/doris.md rename to content/fr/developer/integration/database/doris.md diff --git a/content/fr/developer/integration/big-data/duckdb.md b/content/fr/developer/integration/database/duckdb.md similarity index 100% rename from content/fr/developer/integration/big-data/duckdb.md rename to content/fr/developer/integration/database/duckdb.md diff --git a/content/fr/developer/integration/big-data/images/rustfs-clickhouse-disk.png b/content/fr/developer/integration/database/images/rustfs-clickhouse-disk.png similarity index 100% rename from content/fr/developer/integration/big-data/images/rustfs-clickhouse-disk.png rename to content/fr/developer/integration/database/images/rustfs-clickhouse-disk.png diff --git a/content/fr/developer/integration/big-data/images/rustfs-doris-backup.png b/content/fr/developer/integration/database/images/rustfs-doris-backup.png similarity index 100% rename from content/fr/developer/integration/big-data/images/rustfs-doris-backup.png rename to content/fr/developer/integration/database/images/rustfs-doris-backup.png diff --git a/content/fr/developer/integration/big-data/images/rustfs-duckdb-objects.png b/content/fr/developer/integration/database/images/rustfs-duckdb-objects.png similarity index 100% rename from content/fr/developer/integration/big-data/images/rustfs-duckdb-objects.png rename to content/fr/developer/integration/database/images/rustfs-duckdb-objects.png diff --git a/content/fr/developer/integration/big-data/images/rustfs-influxdb-objects.png b/content/fr/developer/integration/database/images/rustfs-influxdb-objects.png similarity index 100% rename from content/fr/developer/integration/big-data/images/rustfs-influxdb-objects.png rename to content/fr/developer/integration/database/images/rustfs-influxdb-objects.png diff --git a/content/fr/developer/integration/database/images/rustfs-lancedb-table.png b/content/fr/developer/integration/database/images/rustfs-lancedb-table.png new file mode 100644 index 00000000..7d56adac Binary files /dev/null and b/content/fr/developer/integration/database/images/rustfs-lancedb-table.png differ diff --git a/content/fr/developer/integration/big-data/images/rustfs-trino-objects.png b/content/fr/developer/integration/database/images/rustfs-trino-objects.png similarity index 100% rename from content/fr/developer/integration/big-data/images/rustfs-trino-objects.png rename to content/fr/developer/integration/database/images/rustfs-trino-objects.png diff --git a/content/fr/developer/integration/big-data/images/rustfs-vitess-backups.png b/content/fr/developer/integration/database/images/rustfs-vitess-backups.png similarity index 100% rename from content/fr/developer/integration/big-data/images/rustfs-vitess-backups.png rename to content/fr/developer/integration/database/images/rustfs-vitess-backups.png diff --git a/content/fr/developer/integration/database/index.md b/content/fr/developer/integration/database/index.md new file mode 100644 index 00000000..ad29dd08 --- /dev/null +++ b/content/fr/developer/integration/database/index.md @@ -0,0 +1,19 @@ +--- +title: "Database" +description: "Connect databases to RustFS through S3-compatible object storage interfaces." +--- + +Use **RustFS** as the object storage layer for databases that support an S3-compatible endpoint. + +## Databases + +- [ClickHouse](./clickhouse.md) +- [Doris](./doris.md) +- [DuckDB](./duckdb.md) +- [InfluxDB](./influxdb.md) +- [LanceDB](./lancedb.md) +- [Milvus](./milvus.md) +- [Trino](./trino.md) +- [Vitess](./vitess.md) + +Keep database data and backups in a dedicated bucket and prefix, and use credentials scoped to the required bucket operations. diff --git a/content/fr/developer/integration/big-data/influxdb.md b/content/fr/developer/integration/database/influxdb.md similarity index 100% rename from content/fr/developer/integration/big-data/influxdb.md rename to content/fr/developer/integration/database/influxdb.md diff --git a/content/fr/developer/integration/database/lancedb.md b/content/fr/developer/integration/database/lancedb.md new file mode 100644 index 00000000..38a0703a --- /dev/null +++ b/content/fr/developer/integration/database/lancedb.md @@ -0,0 +1,125 @@ +--- +title: "LanceDB" +description: "Store and query LanceDB vector tables directly on RustFS." +--- + +This guide connects [LanceDB](https://github.com/lancedb/lancedb) — the open-source vector database built on the Lance columnar format — to **RustFS** as its storage backend. You will create a vector table directly at an `s3://` location, add rows, run a vector search, and confirm the Lance table files in the bucket. The workflow was verified with the `lancedb` Python package against `rustfs/rustfs-x86-musl:v2.3.1`. + +You need Python 3.9 or newer. This deployment is intended for local integration testing, not production. + +## Architecture + +```mermaid +flowchart LR + App["Python client"] -->|"connect s3://"| LanceDB["LanceDB"] + LanceDB -->|"Lance fragments + manifests"| RustFS["RustFS :9000"] +``` + +LanceDB is embedded: there is no server to run. The Python (or Rust, or JavaScript) client talks to the bucket directly, storing each table as a `*.lance` directory with data fragments, version manifests, and transaction logs. + +## 1. Install the client + +```bash +pip install lancedb +``` + +## 2. Create a table on RustFS + +Connect straight to the bucket and create a table, replacing all connection placeholders. The `storage_options` keys follow Lance's object-store conventions; custom endpoints use path-style addressing: + +```python title="lance_s3.py" +import lancedb + +storage_options = { + "endpoint": "http://:9000", + "access_key_id": "", + "secret_access_key": "", + "region": "us-east-1", + "allow_http": "true", +} + +db = lancedb.connect("s3:///tables", storage_options=storage_options) + +rows = [{"id": i, "label": f"row-{i}", "vector": [float(i) / 10, 0.5, 0.25, 0.1] * 2} + for i in range(5)] +table = db.create_table("events", data=rows) +print("created:", table.count_rows(), "rows") + +table.add([{"id": 99, "label": "query-target", "vector": [0.9, 0.5, 0.25, 0.1] * 2}]) +print("after add:", table.count_rows(), "rows") +``` + +```text +created: 5 rows +after add: 6 rows +``` + +The table URI uses the `s3://bucket/prefix` form; every write and read goes to RustFS over its S3 API. + +## 3. Run a vector search + +```python title="lance_search.py" +import lancedb + +db = lancedb.connect("s3:///tables", storage_options=storage_options) +table = db.open_table("events") + +res = table.search([0.9, 0.5, 0.25, 0.1] * 2).limit(3).to_list() +print("top3:", [(r["id"], r["label"]) for r in res]) +print("version:", table.version) +``` + +```text +top3: [(99, 'query-target'), (4, 'row-4'), (3, 'row-3')] +version: 2 +``` + +The nearest neighbor is the row added in the previous step, and the version counter reflects the two commits (create and add). + +## 4. Verify objects in RustFS + +List the table prefix: + +```bash +rc ls rustfs// -r +``` + +Each table is a `.lance` directory holding data fragments, version manifests, and transaction records: + +```text +tables/events.lance/_transactions/0-b93cf795-fc3d-4bc9-88c8-e37baeefdd46.txn +tables/events.lance/_versions/18446744073709551613.manifest +tables/events.lance/data/101101011110110010000100e7a149411b847dbad6ebc7d47e.lance +``` + +Multiple tables share the bucket under the `tables/` prefix, so one bucket can back an entire LanceDB workspace. + +![LanceDB table files stored in the RustFS Console](./images/rustfs-lancedb-table.png) + +## 5. Stop or reset + +LanceDB holds no server state. To delete the table: + +```bash +rc rm rustfs//tables/ --recursive --force +``` + +## Troubleshooting + +### Connection or signature errors on first use + +Confirm `endpoint` includes the scheme, `allow_http` is `"true"` for plain-HTTP endpoints, and the bucket exists. The `region` value is required by the S3 signer even though RustFS ignores it. + +### `Table not found` after creating it + +LanceDB lists the bucket prefix to discover tables. A stale client cache or a wrong `tables/` prefix in the URI makes new tables invisible; reconnect with the same URI used at creation time. + +### Slow bulk loads over the network + +Lance writes one fragment per commit. For large imports, batch rows into fewer `table.add` calls — each call produces a new data file in the bucket. + +## Next steps + +- Review [S3 compatibility notes](/administration/protocols/s3) before adopting additional LanceDB storage options. +- Create dedicated production credentials with [Access Key Management](/security-compliance/iam/access-token). +- Follow the [LanceDB documentation](https://lancedb.github.io/lancedb/) for ANN indexes, hybrid search, and multi-tenant bucket layouts on top of the same backend. diff --git a/content/fr/developer/integration/database/meta.json b/content/fr/developer/integration/database/meta.json new file mode 100644 index 00000000..308b1815 --- /dev/null +++ b/content/fr/developer/integration/database/meta.json @@ -0,0 +1,13 @@ +{ + "title": "Database", + "pages": [ + "clickhouse", + "doris", + "duckdb", + "influxdb", + "lancedb", + "milvus", + "trino", + "vitess" + ] +} diff --git a/content/fr/developer/integration/big-data/milvus.md b/content/fr/developer/integration/database/milvus.md similarity index 100% rename from content/fr/developer/integration/big-data/milvus.md rename to content/fr/developer/integration/database/milvus.md diff --git a/content/fr/developer/integration/big-data/trino.md b/content/fr/developer/integration/database/trino.md similarity index 100% rename from content/fr/developer/integration/big-data/trino.md rename to content/fr/developer/integration/database/trino.md diff --git a/content/fr/developer/integration/big-data/vitess.md b/content/fr/developer/integration/database/vitess.md similarity index 100% rename from content/fr/developer/integration/big-data/vitess.md rename to content/fr/developer/integration/database/vitess.md diff --git a/content/fr/developer/integration/index.md b/content/fr/developer/integration/index.md index a0ca8ead..57fe063e 100644 --- a/content/fr/developer/integration/index.md +++ b/content/fr/developer/integration/index.md @@ -10,7 +10,9 @@ Utilisez cette section pour connecter **RustFS** à des plateformes d'infrastruc - [Reverse Proxy](./reverse-proxy/index.md) couvre Nginx, Traefik, Caddy, HAProxy et Envoy. - [Backup](./backup/index.md) couvre Kopia, Longhorn, Restic et Velero. - [IA](./ai/index.md) couvre les plateformes d'IA incluant Ray et vLLM. -- [Analyse de données](./big-data/index.md) couvre les systèmes d'analyse incluant Airflow, ClickHouse, Delta Lake, Doris, Hudi, Iceberg, Kafka, lakeFS, Milvus, OpenDAL, Vitess, Zeppelin et ZeroFS. +- [Database](./database/index.md) covers ClickHouse, Doris, DuckDB, InfluxDB, LanceDB, Milvus, Trino, and Vitess. +- [Big Data](./big-data/index.md) covers Airflow, Delta Lake, Flink, Hudi, Iceberg, Kafka, PyIceberg, Spark, and Zeppelin. +- [Storage](./storage/index.md) covers lakeFS, OpenDAL, and ZeroFS. - [Cloud Native](./cloud-native/index.md) couvre Cortex et Flux. - [Observabilité](./observability/index.md) couvre les systèmes de télémétrie incluant Fluentd, GreptimeDB, Loki, OpenObserve, OpenTelemetry, Tempo, Thanos et VictoriaMetrics. - [Autres](./others/index.md) couvre le SDK capo, rclone, JuiceFS, Nextcloud et tusd. diff --git a/content/fr/developer/integration/meta.json b/content/fr/developer/integration/meta.json index f5a8d87c..19173e04 100644 --- a/content/fr/developer/integration/meta.json +++ b/content/fr/developer/integration/meta.json @@ -3,7 +3,9 @@ "pages": [ "reverse-proxy", "backup", + "database", "big-data", + "storage", "ai", "cloud-native", "observability", diff --git a/content/fr/developer/integration/big-data/images/rustfs-lakefs-repo.png b/content/fr/developer/integration/storage/images/rustfs-lakefs-repo.png similarity index 100% rename from content/fr/developer/integration/big-data/images/rustfs-lakefs-repo.png rename to content/fr/developer/integration/storage/images/rustfs-lakefs-repo.png diff --git a/content/fr/developer/integration/big-data/images/rustfs-opendal-objects.png b/content/fr/developer/integration/storage/images/rustfs-opendal-objects.png similarity index 100% rename from content/fr/developer/integration/big-data/images/rustfs-opendal-objects.png rename to content/fr/developer/integration/storage/images/rustfs-opendal-objects.png diff --git a/content/fr/developer/integration/big-data/images/rustfs-zerofs-data.png b/content/fr/developer/integration/storage/images/rustfs-zerofs-data.png similarity index 100% rename from content/fr/developer/integration/big-data/images/rustfs-zerofs-data.png rename to content/fr/developer/integration/storage/images/rustfs-zerofs-data.png diff --git a/content/fr/developer/integration/storage/index.md b/content/fr/developer/integration/storage/index.md new file mode 100644 index 00000000..87a1262f --- /dev/null +++ b/content/fr/developer/integration/storage/index.md @@ -0,0 +1,14 @@ +--- +title: "Storage" +description: "Use RustFS as the S3 backend for storage systems and storage gateways." +--- + +Use **RustFS** as the backend for storage systems and gateways built on top of object storage. + +## Systems + +- [lakeFS](./lakefs.md) +- [OpenDAL](./opendal.md) +- [ZeroFS](./zerofs.md) + +Use a dedicated bucket and prefix per system, and scope credentials to the required bucket operations. diff --git a/content/fr/developer/integration/big-data/lakefs.md b/content/fr/developer/integration/storage/lakefs.md similarity index 100% rename from content/fr/developer/integration/big-data/lakefs.md rename to content/fr/developer/integration/storage/lakefs.md diff --git a/content/fr/developer/integration/storage/meta.json b/content/fr/developer/integration/storage/meta.json new file mode 100644 index 00000000..2c912239 --- /dev/null +++ b/content/fr/developer/integration/storage/meta.json @@ -0,0 +1,8 @@ +{ + "title": "Storage", + "pages": [ + "lakefs", + "opendal", + "zerofs" + ] +} diff --git a/content/fr/developer/integration/big-data/opendal.md b/content/fr/developer/integration/storage/opendal.md similarity index 100% rename from content/fr/developer/integration/big-data/opendal.md rename to content/fr/developer/integration/storage/opendal.md diff --git a/content/fr/developer/integration/big-data/zerofs.md b/content/fr/developer/integration/storage/zerofs.md similarity index 100% rename from content/fr/developer/integration/big-data/zerofs.md rename to content/fr/developer/integration/storage/zerofs.md diff --git a/content/ja/developer/integration/big-data/images/rustfs-mlflow-artifacts.png b/content/ja/developer/integration/ai/images/rustfs-mlflow-artifacts.png similarity index 100% rename from content/ja/developer/integration/big-data/images/rustfs-mlflow-artifacts.png rename to content/ja/developer/integration/ai/images/rustfs-mlflow-artifacts.png diff --git a/content/ja/developer/integration/ai/index.md b/content/ja/developer/integration/ai/index.md index fa215ed6..1e310f48 100644 --- a/content/ja/developer/integration/ai/index.md +++ b/content/ja/developer/integration/ai/index.md @@ -7,6 +7,7 @@ S3 互換エンドポイントをサポートする AI・機械学習プラッ ## プラットフォーム +- [MLflow](./mlflow.md) - [Ray](./ray.md) - [vLLM](./vllm.md) diff --git a/content/ja/developer/integration/ai/meta.json b/content/ja/developer/integration/ai/meta.json index b30f9e0f..9f8472c1 100644 --- a/content/ja/developer/integration/ai/meta.json +++ b/content/ja/developer/integration/ai/meta.json @@ -1,6 +1,7 @@ { "title": "AI", "pages": [ + "mlflow", "ray", "vllm" ] diff --git a/content/ja/developer/integration/big-data/mlflow.md b/content/ja/developer/integration/ai/mlflow.md similarity index 100% rename from content/ja/developer/integration/big-data/mlflow.md rename to content/ja/developer/integration/ai/mlflow.md diff --git a/content/ja/developer/integration/big-data/index.md b/content/ja/developer/integration/big-data/index.md index e37be8ad..ebad0c7b 100644 --- a/content/ja/developer/integration/big-data/index.md +++ b/content/ja/developer/integration/big-data/index.md @@ -1,30 +1,20 @@ --- title: "データ分析" -description: "Connect data analytics systems to RustFS through S3-compatible object storage interfaces." +description: "Connect big data systems to RustFS through S3-compatible object storage interfaces." --- Use **RustFS** as the object storage layer for data analytics systems that support an S3-compatible endpoint. ## Systems -- [ClickHouse](./clickhouse.md) - [Airflow](./airflow.md) +- [Delta Lake](./delta-lake.md) +- [Flink](./flink.md) - [Hudi](./hudi.md) - [Iceberg](./iceberg.md) -- [PyIceberg](./pyiceberg.md) -- [Milvus](./milvus.md) -- [MLflow](./mlflow.md) -- [OpenDAL](./opendal.md) -- [DuckDB](./duckdb.md) -- [Doris](./doris.md) -- [Delta Lake](./delta-lake.md) -- [lakeFS](./lakefs.md) -- [InfluxDB](./influxdb.md) - [Kafka](./kafka.md) +- [PyIceberg](./pyiceberg.md) - [Spark](./spark.md) -- [Flink](./flink.md) -- [Trino](./trino.md) -- [ZeroFS](./zerofs.md) -- [Vitess](./vitess.md) +- [Zeppelin](./zeppelin.md) -Keep application data in a dedicated bucket and prefix, and use credentials scoped to the required bucket operations. \ No newline at end of file +Keep big data workload data in a dedicated bucket and prefix, and use credentials scoped to the required bucket operations. diff --git a/content/ja/developer/integration/big-data/meta.json b/content/ja/developer/integration/big-data/meta.json index 38c1139d..65bca3c4 100644 --- a/content/ja/developer/integration/big-data/meta.json +++ b/content/ja/developer/integration/big-data/meta.json @@ -1,25 +1,14 @@ { - "title": "データ分析", + "title": "ビッグデータ", "pages": [ - "clickhouse", "airflow", - "duckdb", - "doris", "delta-lake", "flink", "hudi", "iceberg", - "influxdb", "kafka", - "lakefs", - "milvus", - "mlflow", - "opendal", "pyiceberg", "spark", - "trino", - "zerofs", - "vitess", "zeppelin" ] } diff --git a/content/ja/developer/integration/big-data/clickhouse.md b/content/ja/developer/integration/database/clickhouse.md similarity index 100% rename from content/ja/developer/integration/big-data/clickhouse.md rename to content/ja/developer/integration/database/clickhouse.md diff --git a/content/ja/developer/integration/big-data/doris.md b/content/ja/developer/integration/database/doris.md similarity index 100% rename from content/ja/developer/integration/big-data/doris.md rename to content/ja/developer/integration/database/doris.md diff --git a/content/ja/developer/integration/big-data/duckdb.md b/content/ja/developer/integration/database/duckdb.md similarity index 100% rename from content/ja/developer/integration/big-data/duckdb.md rename to content/ja/developer/integration/database/duckdb.md diff --git a/content/ja/developer/integration/big-data/images/rustfs-clickhouse-disk.png b/content/ja/developer/integration/database/images/rustfs-clickhouse-disk.png similarity index 100% rename from content/ja/developer/integration/big-data/images/rustfs-clickhouse-disk.png rename to content/ja/developer/integration/database/images/rustfs-clickhouse-disk.png diff --git a/content/ja/developer/integration/big-data/images/rustfs-doris-backup.png b/content/ja/developer/integration/database/images/rustfs-doris-backup.png similarity index 100% rename from content/ja/developer/integration/big-data/images/rustfs-doris-backup.png rename to content/ja/developer/integration/database/images/rustfs-doris-backup.png diff --git a/content/ja/developer/integration/big-data/images/rustfs-duckdb-objects.png b/content/ja/developer/integration/database/images/rustfs-duckdb-objects.png similarity index 100% rename from content/ja/developer/integration/big-data/images/rustfs-duckdb-objects.png rename to content/ja/developer/integration/database/images/rustfs-duckdb-objects.png diff --git a/content/ja/developer/integration/big-data/images/rustfs-influxdb-objects.png b/content/ja/developer/integration/database/images/rustfs-influxdb-objects.png similarity index 100% rename from content/ja/developer/integration/big-data/images/rustfs-influxdb-objects.png rename to content/ja/developer/integration/database/images/rustfs-influxdb-objects.png diff --git a/content/ja/developer/integration/database/images/rustfs-lancedb-table.png b/content/ja/developer/integration/database/images/rustfs-lancedb-table.png new file mode 100644 index 00000000..7d56adac Binary files /dev/null and b/content/ja/developer/integration/database/images/rustfs-lancedb-table.png differ diff --git a/content/ja/developer/integration/big-data/images/rustfs-trino-objects.png b/content/ja/developer/integration/database/images/rustfs-trino-objects.png similarity index 100% rename from content/ja/developer/integration/big-data/images/rustfs-trino-objects.png rename to content/ja/developer/integration/database/images/rustfs-trino-objects.png diff --git a/content/ja/developer/integration/big-data/images/rustfs-vitess-backups.png b/content/ja/developer/integration/database/images/rustfs-vitess-backups.png similarity index 100% rename from content/ja/developer/integration/big-data/images/rustfs-vitess-backups.png rename to content/ja/developer/integration/database/images/rustfs-vitess-backups.png diff --git a/content/ja/developer/integration/database/index.md b/content/ja/developer/integration/database/index.md new file mode 100644 index 00000000..ad29dd08 --- /dev/null +++ b/content/ja/developer/integration/database/index.md @@ -0,0 +1,19 @@ +--- +title: "Database" +description: "Connect databases to RustFS through S3-compatible object storage interfaces." +--- + +Use **RustFS** as the object storage layer for databases that support an S3-compatible endpoint. + +## Databases + +- [ClickHouse](./clickhouse.md) +- [Doris](./doris.md) +- [DuckDB](./duckdb.md) +- [InfluxDB](./influxdb.md) +- [LanceDB](./lancedb.md) +- [Milvus](./milvus.md) +- [Trino](./trino.md) +- [Vitess](./vitess.md) + +Keep database data and backups in a dedicated bucket and prefix, and use credentials scoped to the required bucket operations. diff --git a/content/ja/developer/integration/big-data/influxdb.md b/content/ja/developer/integration/database/influxdb.md similarity index 100% rename from content/ja/developer/integration/big-data/influxdb.md rename to content/ja/developer/integration/database/influxdb.md diff --git a/content/ja/developer/integration/database/lancedb.md b/content/ja/developer/integration/database/lancedb.md new file mode 100644 index 00000000..38a0703a --- /dev/null +++ b/content/ja/developer/integration/database/lancedb.md @@ -0,0 +1,125 @@ +--- +title: "LanceDB" +description: "Store and query LanceDB vector tables directly on RustFS." +--- + +This guide connects [LanceDB](https://github.com/lancedb/lancedb) — the open-source vector database built on the Lance columnar format — to **RustFS** as its storage backend. You will create a vector table directly at an `s3://` location, add rows, run a vector search, and confirm the Lance table files in the bucket. The workflow was verified with the `lancedb` Python package against `rustfs/rustfs-x86-musl:v2.3.1`. + +You need Python 3.9 or newer. This deployment is intended for local integration testing, not production. + +## Architecture + +```mermaid +flowchart LR + App["Python client"] -->|"connect s3://"| LanceDB["LanceDB"] + LanceDB -->|"Lance fragments + manifests"| RustFS["RustFS :9000"] +``` + +LanceDB is embedded: there is no server to run. The Python (or Rust, or JavaScript) client talks to the bucket directly, storing each table as a `*.lance` directory with data fragments, version manifests, and transaction logs. + +## 1. Install the client + +```bash +pip install lancedb +``` + +## 2. Create a table on RustFS + +Connect straight to the bucket and create a table, replacing all connection placeholders. The `storage_options` keys follow Lance's object-store conventions; custom endpoints use path-style addressing: + +```python title="lance_s3.py" +import lancedb + +storage_options = { + "endpoint": "http://:9000", + "access_key_id": "", + "secret_access_key": "", + "region": "us-east-1", + "allow_http": "true", +} + +db = lancedb.connect("s3:///tables", storage_options=storage_options) + +rows = [{"id": i, "label": f"row-{i}", "vector": [float(i) / 10, 0.5, 0.25, 0.1] * 2} + for i in range(5)] +table = db.create_table("events", data=rows) +print("created:", table.count_rows(), "rows") + +table.add([{"id": 99, "label": "query-target", "vector": [0.9, 0.5, 0.25, 0.1] * 2}]) +print("after add:", table.count_rows(), "rows") +``` + +```text +created: 5 rows +after add: 6 rows +``` + +The table URI uses the `s3://bucket/prefix` form; every write and read goes to RustFS over its S3 API. + +## 3. Run a vector search + +```python title="lance_search.py" +import lancedb + +db = lancedb.connect("s3:///tables", storage_options=storage_options) +table = db.open_table("events") + +res = table.search([0.9, 0.5, 0.25, 0.1] * 2).limit(3).to_list() +print("top3:", [(r["id"], r["label"]) for r in res]) +print("version:", table.version) +``` + +```text +top3: [(99, 'query-target'), (4, 'row-4'), (3, 'row-3')] +version: 2 +``` + +The nearest neighbor is the row added in the previous step, and the version counter reflects the two commits (create and add). + +## 4. Verify objects in RustFS + +List the table prefix: + +```bash +rc ls rustfs// -r +``` + +Each table is a `.lance` directory holding data fragments, version manifests, and transaction records: + +```text +tables/events.lance/_transactions/0-b93cf795-fc3d-4bc9-88c8-e37baeefdd46.txn +tables/events.lance/_versions/18446744073709551613.manifest +tables/events.lance/data/101101011110110010000100e7a149411b847dbad6ebc7d47e.lance +``` + +Multiple tables share the bucket under the `tables/` prefix, so one bucket can back an entire LanceDB workspace. + +![LanceDB table files stored in the RustFS Console](./images/rustfs-lancedb-table.png) + +## 5. Stop or reset + +LanceDB holds no server state. To delete the table: + +```bash +rc rm rustfs//tables/ --recursive --force +``` + +## Troubleshooting + +### Connection or signature errors on first use + +Confirm `endpoint` includes the scheme, `allow_http` is `"true"` for plain-HTTP endpoints, and the bucket exists. The `region` value is required by the S3 signer even though RustFS ignores it. + +### `Table not found` after creating it + +LanceDB lists the bucket prefix to discover tables. A stale client cache or a wrong `tables/` prefix in the URI makes new tables invisible; reconnect with the same URI used at creation time. + +### Slow bulk loads over the network + +Lance writes one fragment per commit. For large imports, batch rows into fewer `table.add` calls — each call produces a new data file in the bucket. + +## Next steps + +- Review [S3 compatibility notes](/administration/protocols/s3) before adopting additional LanceDB storage options. +- Create dedicated production credentials with [Access Key Management](/security-compliance/iam/access-token). +- Follow the [LanceDB documentation](https://lancedb.github.io/lancedb/) for ANN indexes, hybrid search, and multi-tenant bucket layouts on top of the same backend. diff --git a/content/ja/developer/integration/database/meta.json b/content/ja/developer/integration/database/meta.json new file mode 100644 index 00000000..08eb9bc5 --- /dev/null +++ b/content/ja/developer/integration/database/meta.json @@ -0,0 +1,13 @@ +{ + "title": "データベース", + "pages": [ + "clickhouse", + "doris", + "duckdb", + "influxdb", + "lancedb", + "milvus", + "trino", + "vitess" + ] +} diff --git a/content/ja/developer/integration/big-data/milvus.md b/content/ja/developer/integration/database/milvus.md similarity index 100% rename from content/ja/developer/integration/big-data/milvus.md rename to content/ja/developer/integration/database/milvus.md diff --git a/content/ja/developer/integration/big-data/trino.md b/content/ja/developer/integration/database/trino.md similarity index 100% rename from content/ja/developer/integration/big-data/trino.md rename to content/ja/developer/integration/database/trino.md diff --git a/content/ja/developer/integration/big-data/vitess.md b/content/ja/developer/integration/database/vitess.md similarity index 100% rename from content/ja/developer/integration/big-data/vitess.md rename to content/ja/developer/integration/database/vitess.md diff --git a/content/ja/developer/integration/index.md b/content/ja/developer/integration/index.md index 7bf4fd08..cf51adb6 100644 --- a/content/ja/developer/integration/index.md +++ b/content/ja/developer/integration/index.md @@ -10,7 +10,9 @@ description: "RustFS をリバースプロキシ、バックアップツール - [Reverse Proxy](./reverse-proxy/index.md) は Nginx、Traefik、Caddy、HAProxy、Envoy を扱います。 - [Backup](./backup/index.md) は Kopia、Longhorn、Restic、Velero を扱います。 - [AI](./ai/index.md) は Ray、vLLM などの AI プラットフォームを扱います。 -- [データ分析](./big-data/index.md) は Airflow、ClickHouse、Delta Lake、Doris、Hudi、Iceberg、Kafka、lakeFS、Milvus、OpenDAL、Vitess、Zeppelin、ZeroFS などの分析システムを扱います。 +- [Database](./database/index.md) covers ClickHouse, Doris, DuckDB, InfluxDB, LanceDB, Milvus, Trino, and Vitess. +- [Big Data](./big-data/index.md) covers Airflow, Delta Lake, Flink, Hudi, Iceberg, Kafka, PyIceberg, Spark, and Zeppelin. +- [Storage](./storage/index.md) covers lakeFS, OpenDAL, and ZeroFS. - [クラウドネイティブ](./cloud-native/index.md) は Cortex、Flux を扱います。 - [オブザーバビリティ](./observability/index.md) は Fluentd、GreptimeDB、Loki、OpenObserve、OpenTelemetry、Tempo、Thanos、VictoriaMetrics などのテレメトリシステムを扱います。 - [その他](./others/index.md) は capo SDK、rclone、JuiceFS、Nextcloud、tusd などのツールを扱います。 diff --git a/content/ja/developer/integration/meta.json b/content/ja/developer/integration/meta.json index f5a8d87c..19173e04 100644 --- a/content/ja/developer/integration/meta.json +++ b/content/ja/developer/integration/meta.json @@ -3,7 +3,9 @@ "pages": [ "reverse-proxy", "backup", + "database", "big-data", + "storage", "ai", "cloud-native", "observability", diff --git a/content/ja/developer/integration/big-data/images/rustfs-lakefs-repo.png b/content/ja/developer/integration/storage/images/rustfs-lakefs-repo.png similarity index 100% rename from content/ja/developer/integration/big-data/images/rustfs-lakefs-repo.png rename to content/ja/developer/integration/storage/images/rustfs-lakefs-repo.png diff --git a/content/ja/developer/integration/big-data/images/rustfs-opendal-objects.png b/content/ja/developer/integration/storage/images/rustfs-opendal-objects.png similarity index 100% rename from content/ja/developer/integration/big-data/images/rustfs-opendal-objects.png rename to content/ja/developer/integration/storage/images/rustfs-opendal-objects.png diff --git a/content/ja/developer/integration/big-data/images/rustfs-zerofs-data.png b/content/ja/developer/integration/storage/images/rustfs-zerofs-data.png similarity index 100% rename from content/ja/developer/integration/big-data/images/rustfs-zerofs-data.png rename to content/ja/developer/integration/storage/images/rustfs-zerofs-data.png diff --git a/content/ja/developer/integration/storage/index.md b/content/ja/developer/integration/storage/index.md new file mode 100644 index 00000000..87a1262f --- /dev/null +++ b/content/ja/developer/integration/storage/index.md @@ -0,0 +1,14 @@ +--- +title: "Storage" +description: "Use RustFS as the S3 backend for storage systems and storage gateways." +--- + +Use **RustFS** as the backend for storage systems and gateways built on top of object storage. + +## Systems + +- [lakeFS](./lakefs.md) +- [OpenDAL](./opendal.md) +- [ZeroFS](./zerofs.md) + +Use a dedicated bucket and prefix per system, and scope credentials to the required bucket operations. diff --git a/content/ja/developer/integration/big-data/lakefs.md b/content/ja/developer/integration/storage/lakefs.md similarity index 100% rename from content/ja/developer/integration/big-data/lakefs.md rename to content/ja/developer/integration/storage/lakefs.md diff --git a/content/ja/developer/integration/storage/meta.json b/content/ja/developer/integration/storage/meta.json new file mode 100644 index 00000000..0da61ab4 --- /dev/null +++ b/content/ja/developer/integration/storage/meta.json @@ -0,0 +1,8 @@ +{ + "title": "ストレージ", + "pages": [ + "lakefs", + "opendal", + "zerofs" + ] +} diff --git a/content/ja/developer/integration/big-data/opendal.md b/content/ja/developer/integration/storage/opendal.md similarity index 100% rename from content/ja/developer/integration/big-data/opendal.md rename to content/ja/developer/integration/storage/opendal.md diff --git a/content/ja/developer/integration/big-data/zerofs.md b/content/ja/developer/integration/storage/zerofs.md similarity index 100% rename from content/ja/developer/integration/big-data/zerofs.md rename to content/ja/developer/integration/storage/zerofs.md diff --git a/content/zh/developer/integration/big-data/images/rustfs-mlflow-artifacts.png b/content/zh/developer/integration/ai/images/rustfs-mlflow-artifacts.png similarity index 100% rename from content/zh/developer/integration/big-data/images/rustfs-mlflow-artifacts.png rename to content/zh/developer/integration/ai/images/rustfs-mlflow-artifacts.png diff --git a/content/zh/developer/integration/ai/index.md b/content/zh/developer/integration/ai/index.md index b46c73e5..d9a9e72b 100644 --- a/content/zh/developer/integration/ai/index.md +++ b/content/zh/developer/integration/ai/index.md @@ -7,6 +7,7 @@ description: "通过 S3 兼容的对象存储接口，将 AI 平台连接到 Rus ## 平台 +- [MLflow](./mlflow.md) - [Ray](./ray.md) - [vLLM](./vllm.md) diff --git a/content/zh/developer/integration/ai/meta.json b/content/zh/developer/integration/ai/meta.json index b30f9e0f..9f8472c1 100644 --- a/content/zh/developer/integration/ai/meta.json +++ b/content/zh/developer/integration/ai/meta.json @@ -1,6 +1,7 @@ { "title": "AI", "pages": [ + "mlflow", "ray", "vllm" ] diff --git a/content/zh/developer/integration/big-data/mlflow.md b/content/zh/developer/integration/ai/mlflow.md similarity index 100% rename from content/zh/developer/integration/big-data/mlflow.md rename to content/zh/developer/integration/ai/mlflow.md diff --git a/content/zh/developer/integration/big-data/index.md b/content/zh/developer/integration/big-data/index.md index 317d0222..60fe2a25 100644 --- a/content/zh/developer/integration/big-data/index.md +++ b/content/zh/developer/integration/big-data/index.md @@ -7,24 +7,14 @@ description: "通过 S3 兼容的对象存储接口将数据分析系统连接 ## 系统 -- [ClickHouse](./clickhouse.md) - [Airflow](./airflow.md) +- [Delta Lake](./delta-lake.md) +- [Flink](./flink.md) - [Hudi](./hudi.md) - [Iceberg](./iceberg.md) -- [PyIceberg](./pyiceberg.md) -- [Milvus](./milvus.md) -- [MLflow](./mlflow.md) -- [OpenDAL](./opendal.md) -- [DuckDB](./duckdb.md) -- [Doris](./doris.md) -- [Delta Lake](./delta-lake.md) -- [lakeFS](./lakefs.md) -- [InfluxDB](./influxdb.md) - [Kafka](./kafka.md) +- [PyIceberg](./pyiceberg.md) - [Spark](./spark.md) -- [Flink](./flink.md) -- [Trino](./trino.md) -- [ZeroFS](./zerofs.md) -- [Vitess](./vitess.md) +- [Zeppelin](./zeppelin.md) -将应用程序数据保存在专用存储桶和前缀中，并使用作用域限定为所需存储桶操作的凭证。 \ No newline at end of file +请将大数据作业的数据保存在专用的桶和前缀下，并为凭证仅授予所需桶操作的权限。 diff --git a/content/zh/developer/integration/big-data/meta.json b/content/zh/developer/integration/big-data/meta.json index 876032d9..e95ed981 100644 --- a/content/zh/developer/integration/big-data/meta.json +++ b/content/zh/developer/integration/big-data/meta.json @@ -1,25 +1,14 @@ { - "title": "数据分析", + "title": "大数据", "pages": [ - "clickhouse", "airflow", - "duckdb", - "doris", "delta-lake", "flink", "hudi", "iceberg", - "influxdb", "kafka", - "lakefs", - "milvus", - "mlflow", - "opendal", "pyiceberg", "spark", - "trino", - "zerofs", - "vitess", "zeppelin" ] } diff --git a/content/zh/developer/integration/big-data/clickhouse.md b/content/zh/developer/integration/database/clickhouse.md similarity index 100% rename from content/zh/developer/integration/big-data/clickhouse.md rename to content/zh/developer/integration/database/clickhouse.md diff --git a/content/zh/developer/integration/big-data/doris.md b/content/zh/developer/integration/database/doris.md similarity index 100% rename from content/zh/developer/integration/big-data/doris.md rename to content/zh/developer/integration/database/doris.md diff --git a/content/zh/developer/integration/big-data/duckdb.md b/content/zh/developer/integration/database/duckdb.md similarity index 100% rename from content/zh/developer/integration/big-data/duckdb.md rename to content/zh/developer/integration/database/duckdb.md diff --git a/content/zh/developer/integration/big-data/images/rustfs-clickhouse-disk.png b/content/zh/developer/integration/database/images/rustfs-clickhouse-disk.png similarity index 100% rename from content/zh/developer/integration/big-data/images/rustfs-clickhouse-disk.png rename to content/zh/developer/integration/database/images/rustfs-clickhouse-disk.png diff --git a/content/zh/developer/integration/big-data/images/rustfs-doris-backup.png b/content/zh/developer/integration/database/images/rustfs-doris-backup.png similarity index 100% rename from content/zh/developer/integration/big-data/images/rustfs-doris-backup.png rename to content/zh/developer/integration/database/images/rustfs-doris-backup.png diff --git a/content/zh/developer/integration/big-data/images/rustfs-duckdb-objects.png b/content/zh/developer/integration/database/images/rustfs-duckdb-objects.png similarity index 100% rename from content/zh/developer/integration/big-data/images/rustfs-duckdb-objects.png rename to content/zh/developer/integration/database/images/rustfs-duckdb-objects.png diff --git a/content/zh/developer/integration/big-data/images/rustfs-influxdb-objects.png b/content/zh/developer/integration/database/images/rustfs-influxdb-objects.png similarity index 100% rename from content/zh/developer/integration/big-data/images/rustfs-influxdb-objects.png rename to content/zh/developer/integration/database/images/rustfs-influxdb-objects.png diff --git a/content/zh/developer/integration/database/images/rustfs-lancedb-table.png b/content/zh/developer/integration/database/images/rustfs-lancedb-table.png new file mode 100644 index 00000000..7a7927af Binary files /dev/null and b/content/zh/developer/integration/database/images/rustfs-lancedb-table.png differ diff --git a/content/zh/developer/integration/big-data/images/rustfs-trino-objects.png b/content/zh/developer/integration/database/images/rustfs-trino-objects.png similarity index 100% rename from content/zh/developer/integration/big-data/images/rustfs-trino-objects.png rename to content/zh/developer/integration/database/images/rustfs-trino-objects.png diff --git a/content/zh/developer/integration/big-data/images/rustfs-vitess-backups.png b/content/zh/developer/integration/database/images/rustfs-vitess-backups.png similarity index 100% rename from content/zh/developer/integration/big-data/images/rustfs-vitess-backups.png rename to content/zh/developer/integration/database/images/rustfs-vitess-backups.png diff --git a/content/zh/developer/integration/database/index.md b/content/zh/developer/integration/database/index.md new file mode 100644 index 00000000..995c35e0 --- /dev/null +++ b/content/zh/developer/integration/database/index.md @@ -0,0 +1,19 @@ +--- +title: "数据库" +description: "将 RustFS 用作支持 S3 兼容端点的数据库的对象存储层。" +--- + +将 **RustFS** 用作支持 S3 兼容端点的数据库的对象存储层。 + +## 数据库 + +- [ClickHouse](./clickhouse.md) +- [Doris](./doris.md) +- [DuckDB](./duckdb.md) +- [InfluxDB](./influxdb.md) +- [LanceDB](./lancedb.md) +- [Milvus](./milvus.md) +- [Trino](./trino.md) +- [Vitess](./vitess.md) + +请将数据库数据与备份保存在专用的桶和前缀下，并为凭证仅授予所需桶操作的权限。 diff --git a/content/zh/developer/integration/big-data/influxdb.md b/content/zh/developer/integration/database/influxdb.md similarity index 100% rename from content/zh/developer/integration/big-data/influxdb.md rename to content/zh/developer/integration/database/influxdb.md diff --git a/content/zh/developer/integration/database/lancedb.md b/content/zh/developer/integration/database/lancedb.md new file mode 100644 index 00000000..0427930a --- /dev/null +++ b/content/zh/developer/integration/database/lancedb.md @@ -0,0 +1,125 @@ +--- +title: "LanceDB" +description: "将 LanceDB 向量表直接存储在 RustFS 上并进行查询。" +--- + +本指南将基于 Lance 列式格式的开源向量数据库 [LanceDB](https://github.com/lancedb/lancedb) 连接到 **RustFS** 作为其存储后端。你将直接在 `s3://` 位置创建向量表、追加行、执行向量检索，并确认桶内的 Lance 表文件。整个流程使用 `lancedb` Python 包对 `rustfs/rustfs-x86-musl:v2.3.1` 验证通过。 + +你需要 Python 3.9 或更高版本。本部署用于本地集成测试，不适用于生产环境。 + +## 架构 + +```mermaid +flowchart LR + App["Python client"] -->|"connect s3://"| LanceDB["LanceDB"] + LanceDB -->|"Lance fragments + manifests"| RustFS["RustFS :9000"] +``` + +LanceDB 是嵌入式数据库：无需运行服务端。Python（或 Rust、JavaScript）客户端直接访问桶，把每张表存为一个 `*.lance` 目录（数据分件、版本 manifest 与事务日志）。 + +## 1. 安装客户端 + +```bash +pip install lancedb +``` + +## 2. 在 RustFS 上创建表 + +直接连接桶并建表，替换全部连接占位符。`storage_options` 的键遵循 Lance 的 object-store 约定；自定义端点使用路径风格寻址： + +```python title="lance_s3.py" +import lancedb + +storage_options = { + "endpoint": "http://:9000", + "access_key_id": "", + "secret_access_key": "", + "region": "us-east-1", + "allow_http": "true", +} + +db = lancedb.connect("s3:///tables", storage_options=storage_options) + +rows = [{"id": i, "label": f"row-{i}", "vector": [float(i) / 10, 0.5, 0.25, 0.1] * 2} + for i in range(5)] +table = db.create_table("events", data=rows) +print("created:", table.count_rows(), "rows") + +table.add([{"id": 99, "label": "query-target", "vector": [0.9, 0.5, 0.25, 0.1] * 2}]) +print("after add:", table.count_rows(), "rows") +``` + +```text +created: 5 rows +after add: 6 rows +``` + +表 URI 使用 `s3://bucket/prefix` 形式；所有读写都通过 RustFS 的 S3 API 完成。 + +## 3. 执行向量检索 + +```python title="lance_search.py" +import lancedb + +db = lancedb.connect("s3:///tables", storage_options=storage_options) +table = db.open_table("events") + +res = table.search([0.9, 0.5, 0.25, 0.1] * 2).limit(3).to_list() +print("top3:", [(r["id"], r["label"]) for r in res]) +print("version:", table.version) +``` + +```text +top3: [(99, 'query-target'), (4, 'row-4'), (3, 'row-3')] +version: 2 +``` + +最近邻正是上一步追加的行，版本号也反映了两笔提交（建表与追加）。 + +## 4. 验证 RustFS 中的对象 + +列举表前缀： + +```bash +rc ls rustfs// -r +``` + +每张表是一个 `.lance` 目录，内含数据分件、版本 manifest 与事务记录： + +```text +tables/events.lance/_transactions/0-b93cf795-fc3d-4bc9-88c8-e37baeefdd46.txn +tables/events.lance/_versions/18446744073709551613.manifest +tables/events.lance/data/101101011110110010000100e7a149411b847dbad6ebc7d47e.lance +``` + +多张表共用 `tables/` 前缀下的同一个桶，因此一个桶就能承载整个 LanceDB 工作区。 + +![存储在 RustFS 控制台中的 LanceDB 表文件](./images/rustfs-lancedb-table.png) + +## 5. 停止或重置 + +LanceDB 不持有服务端状态。删除表： + +```bash +rc rm rustfs//tables/ --recursive --force +``` + +## 故障排查 + +### 首次使用报连接或签名错误 + +确认 `endpoint` 带协议前缀、纯 HTTP 端点把 `allow_http` 设置为 `"true"`，且桶已创建。虽然 RustFS 会忽略 `region`，但 S3 签名需要该值。 + +### 建表后报 `Table not found` + +LanceDB 通过列举桶前缀来发现表。客户端缓存过期或 URI 中的 `tables/` 前缀写错都会让新表不可见；请使用与建表时相同的 URI 重新连接。 + +### 大批量导入很慢 + +Lance 每次提交写一个数据分件。大规模导入时，把行合并到更少的 `table.add` 调用中——每次调用都会在桶里生成一个新数据文件。 + +## 下一步 + +- 在启用更多 LanceDB 存储选项前，先阅读 [S3 兼容性说明](/administration/protocols/s3)。 +- 使用[访问密钥管理](/security-compliance/iam/access-token)创建专用的生产凭证。 +- 按照 [LanceDB 文档](https://lancedb.github.io/lancedb/)在同一个后端之上使用 ANN 索引、混合检索与多租户桶布局。 diff --git a/content/zh/developer/integration/database/meta.json b/content/zh/developer/integration/database/meta.json new file mode 100644 index 00000000..50bb0b1b --- /dev/null +++ b/content/zh/developer/integration/database/meta.json @@ -0,0 +1,13 @@ +{ + "title": "数据库", + "pages": [ + "clickhouse", + "doris", + "duckdb", + "influxdb", + "lancedb", + "milvus", + "trino", + "vitess" + ] +} diff --git a/content/zh/developer/integration/big-data/milvus.md b/content/zh/developer/integration/database/milvus.md similarity index 100% rename from content/zh/developer/integration/big-data/milvus.md rename to content/zh/developer/integration/database/milvus.md diff --git a/content/zh/developer/integration/big-data/trino.md b/content/zh/developer/integration/database/trino.md similarity index 100% rename from content/zh/developer/integration/big-data/trino.md rename to content/zh/developer/integration/database/trino.md diff --git a/content/zh/developer/integration/big-data/vitess.md b/content/zh/developer/integration/database/vitess.md similarity index 100% rename from content/zh/developer/integration/big-data/vitess.md rename to content/zh/developer/integration/database/vitess.md diff --git a/content/zh/developer/integration/index.md b/content/zh/developer/integration/index.md index 3a039635..1658e2dd 100644 --- a/content/zh/developer/integration/index.md +++ b/content/zh/developer/integration/index.md @@ -1,6 +1,6 @@ --- title: "集成" -description: "将 RustFS 与反向代理、备份工具、数据分析系统、AI 与云原生平台、可观测性平台、容器镜像仓库和 DevOps 工具集成。" +description: "将 RustFS 与反向代理、备份工具、数据库、大数据与存储系统、AI 与云原生平台、可观测性平台、容器镜像仓库和 DevOps 工具集成。" --- 通过 S3 兼容 API 将 **RustFS** 连接到基础设施和应用平台。 @@ -9,8 +9,10 @@ description: "将 RustFS 与反向代理、备份工具、数据分析系统、A - [反向代理](./reverse-proxy/index.md)涵盖 Nginx、Traefik、Caddy、HAProxy 和 Envoy。 - [备份](./backup/index.md)涵盖 Kopia、Longhorn、Restic 和 Velero。 -- [AI](./ai/index.md)涵盖 Ray、vLLM 等 AI 平台。 -- [数据分析](./big-data/index.md)涵盖 Airflow、ClickHouse、Delta Lake、Doris、Hudi、Iceberg、Kafka、lakeFS、Milvus、OpenDAL、Vitess、Zeppelin 和 ZeroFS 等数据分析系统。 +- [AI](./ai/index.md)涵盖 MLflow、Ray、vLLM 等 AI 平台。 +- [数据库](./database/index.md)涵盖 ClickHouse、Doris、DuckDB、InfluxDB、LanceDB、Milvus、Trino 和 Vitess 等数据库。 +- [大数据](./big-data/index.md)涵盖 Airflow、Delta Lake、Flink、Hudi、Iceberg、Kafka、PyIceberg、Spark 和 Zeppelin 等大数据系统。 +- [存储](./storage/index.md)涵盖 lakeFS、OpenDAL 和 ZeroFS 等存储系统。 - [云原生](./cloud-native/index.md)涵盖 Cortex 与 Flux。 - [可观测性](./observability/index.md)涵盖 Fluentd、GreptimeDB、Loki、OpenObserve、OpenTelemetry、Tempo、Thanos 和 VictoriaMetrics 等遥测系统。 - [其他](./others/index.md)涵盖 capo SDK、rclone、JuiceFS、Nextcloud 和 tusd 等工具。 diff --git a/content/zh/developer/integration/meta.json b/content/zh/developer/integration/meta.json index f5a8d87c..19173e04 100644 --- a/content/zh/developer/integration/meta.json +++ b/content/zh/developer/integration/meta.json @@ -3,7 +3,9 @@ "pages": [ "reverse-proxy", "backup", + "database", "big-data", + "storage", "ai", "cloud-native", "observability", diff --git a/content/zh/developer/integration/big-data/images/rustfs-lakefs-repo.png b/content/zh/developer/integration/storage/images/rustfs-lakefs-repo.png similarity index 100% rename from content/zh/developer/integration/big-data/images/rustfs-lakefs-repo.png rename to content/zh/developer/integration/storage/images/rustfs-lakefs-repo.png diff --git a/content/zh/developer/integration/big-data/images/rustfs-opendal-objects.png b/content/zh/developer/integration/storage/images/rustfs-opendal-objects.png similarity index 100% rename from content/zh/developer/integration/big-data/images/rustfs-opendal-objects.png rename to content/zh/developer/integration/storage/images/rustfs-opendal-objects.png diff --git a/content/zh/developer/integration/big-data/images/rustfs-zerofs-data.png b/content/zh/developer/integration/storage/images/rustfs-zerofs-data.png similarity index 100% rename from content/zh/developer/integration/big-data/images/rustfs-zerofs-data.png rename to content/zh/developer/integration/storage/images/rustfs-zerofs-data.png diff --git a/content/zh/developer/integration/storage/index.md b/content/zh/developer/integration/storage/index.md new file mode 100644 index 00000000..0a6d9d11 --- /dev/null +++ b/content/zh/developer/integration/storage/index.md @@ -0,0 +1,14 @@ +--- +title: "存储" +description: "将 RustFS 用作存储系统与存储网关的 S3 后端。" +--- + +将 **RustFS** 用作在对象存储之上构建的存储系统与存储网关的后端。 + +## 系统 + +- [lakeFS](./lakefs.md) +- [OpenDAL](./opendal.md) +- [ZeroFS](./zerofs.md) + +请为每个系统使用专用的桶和前缀，并为凭证仅授予所需桶操作的权限。 diff --git a/content/zh/developer/integration/big-data/lakefs.md b/content/zh/developer/integration/storage/lakefs.md similarity index 100% rename from content/zh/developer/integration/big-data/lakefs.md rename to content/zh/developer/integration/storage/lakefs.md diff --git a/content/zh/developer/integration/storage/meta.json b/content/zh/developer/integration/storage/meta.json new file mode 100644 index 00000000..87a0f1f9 --- /dev/null +++ b/content/zh/developer/integration/storage/meta.json @@ -0,0 +1,8 @@ +{ + "title": "存储", + "pages": [ + "lakefs", + "opendal", + "zerofs" + ] +} diff --git a/content/zh/developer/integration/big-data/opendal.md b/content/zh/developer/integration/storage/opendal.md similarity index 100% rename from content/zh/developer/integration/big-data/opendal.md rename to content/zh/developer/integration/storage/opendal.md diff --git a/content/zh/developer/integration/big-data/zerofs.md b/content/zh/developer/integration/storage/zerofs.md similarity index 100% rename from content/zh/developer/integration/big-data/zerofs.md rename to content/zh/developer/integration/storage/zerofs.md