diff --git a/.tabularium b/.tabularium index 2f8faeb..5a437d0 100644 --- a/.tabularium +++ b/.tabularium @@ -1,7 +1,7 @@ { "id": "libsql", "name": "libSQL", - "version": "1.0.2", + "version": "1.0.3", "kind": "driver", "description": "libSQL / Turso driver. Connects to local libSQL/SQLite files (embedded libSQL fork: ALTER COLUMN and foreign-key add/drop work locally) and to remote Turso or sqld servers over the Hrana HTTP protocol.", "engine": "libsql", diff --git a/AGENTS.md b/AGENTS.md new file mode 100644 index 0000000..ff06db5 --- /dev/null +++ b/AGENTS.md @@ -0,0 +1,42 @@ +# AGENTS.md + +Tabularis driver plugin for libSQL/Turso: a standalone binary speaking Tabularis' JSON-RPC 2.0 over stdin/stdout (one request line in, one response line out). Two backends: local files through the bundled **libSQL fork** of SQLite (`libsql` crate, `core` feature, no system lib) and remote Turso/sqld over Hrana HTTP `/v2/pipeline` with blocking `ureq` + rustls. No async runtime: `futures_executor::block_on` bridges libsql's async API. A C compiler is required to build the bundled SQLite. + +## Commands + +- `just test` — `cargo test`; unit tests only, inline `#[cfg(test)]`, no integration harness. Single test: `cargo test `. +- `just lint` / `just fmt` — clippy with `-D warnings` / rustfmt. CI runs build + `cargo test --locked --all-targets` + RustSec audit, but not clippy or fmt, so run both locally. +- `just repl` — `cargo run --bin test_plugin`: REPL over the real dispatch. Bare method names (e.g. `get_databases`) are auto-wrapped; full JSON-RPC objects pass through; `exit` quits. +- `just dev-install` — builds debug, copies binary + `.tabularium` to the OS plugin dir: Linux `~/.local/share/tabularis/plugins/drivers/libsql`, macOS `~/Library/Application Support/tabularis/plugins/drivers/libsql`, Windows `%APPDATA%\tabularis\plugins\drivers\libsql`. Tabularis resolves kind-first (`plugins/drivers/` before flat `plugins/`), so the `drivers` segment is mandatory — a flat copy is shadowed by an existing installed bundle. +- Manual smoke: `echo '{"jsonrpc":"2.0","method":"get_tables","params":{"params":{"database":":memory:"}},"id":1}' | ./target/debug/libsql-plugin` +- Windows shell is PowerShell; `just` recipes use `windows-shell`, and each recipe line runs as a separate `powershell.exe`. Variable-setting recipes must be one `;`-joined line (see `dev-install`). + +## Architecture + +- Logic lives in the library (`src/lib.rs`); `src/main.rs` is thin (read line → `rpc::handle_line` → write line). `test_plugin` drives the same dispatch. New RPC method = handler in `src/handlers/{metadata,query,crud,ddl}.rs` plus an arm in `rpc.rs::handle_line`; handlers return `Result` via `handlers::respond`. +- Connection params live in the nested `params.params` block; use `handlers::connect` / `models::inner_params`. A fresh `Client` is built per call; there is no shared state. +- Preview builders `get_create_table_sql`, `get_add_column_sql`, `get_create_index_sql` are pure (no connection params). `get_create_foreign_key_sql` always receives `params.params` (the host's RpcDriver passes them); `get_alter_column_sql` uses params only when present and otherwise stays pure. Do not call `connect()` unconditionally in builders. Methods that touch a database (`execute_query`, CRUD, `drop_index`, `drop_foreign_key`, schema-introspecting metadata) do receive params. +- `execute_query` must match the host's strict `QueryResult`: `affected_rows`, `truncated`, and `pagination` are required. Requests send `limit` (code also accepts `page_size`); the response `pagination` is `{page, page_size, total_rows: null, has_more}`. `has_more` comes from a LIMIT+1 fetch; there is no `total_count` field. +- `client.rs::resolve_backend`: `connection_uri` is authoritative over decomposed fields. A URL in `database`/`host` (or any non-empty `host`) is remote; otherwise `database` is a local path. `libsql://`/`turso://`/`wss://` → `https://`, `ws://` → `http://`; localhost or `ssl_mode=disable` uses `http://`; `file:` paths and `~/` expand. Auth token from the URL query (`authToken`, etc.) wins over `password`. +- sqld namespaces go in the `x-namespace` header, not the URL path: sqld 0.24.x ignores `/dev//...` and resolves from Host. `hrana.rs::namespace_from_url_path` maps `.../dev/example` → `example`; plain Turso hosts send no header. When `host` is bare, a decomposed `database` containing `/` is kept as a URL path prefix; `.tabularium` declares `http`/`https` so those URIs reach the plugin. +- Errors must become JSON-RPC error responses, never panics; one bad request must not kill the process. + +## Conventions / gotchas + +- Quote identifiers with `utils::identifiers::quote` (ANSI `"name"`); bind CRUD values, never interpolate them. Booleans are stored `0`/`1`; BLOBs are returned base64-encoded. +- `get_databases` is static `["main"]`; `get_schemas` and `get_routines` return empty (libSQL has no schemas or stored routines). +- `update_record` / `delete_record` take `pk_map` (preferred, supports composite keys); legacy `pk_col`/`pk_val` is also accepted. Keep both. +- Both backends run the libSQL fork, so `ALTER TABLE ... ALTER COLUMN` (retype, FK add/drop) works locally and remotely. The ALTER/FK builders emit the libSQL form unconditionally; renames use vanilla `RENAME COLUMN`. +- FK add emits `ALTER COLUMN col TO col REFERENCES ...`, reproducing type/NOT NULL/DEFAULT from `PRAGMA table_info`. Constraint names are cosmetic; names are re-derived from `PRAGMA foreign_key_list`. Composite FK drops error; FK enforcement needs `PRAGMA foreign_keys=ON` per connection. +- `.tabularium` capabilities must stay in sync with actual support (`alter_column`, `create_foreign_keys` are true). +- `get_schema_snapshot` returns the host's `Vec` shape: a top-level array of `{name, columns, foreign_keys}` (empty schema → `[]`). The host deserializes it directly into the ER diagram, so an envelope or name-keyed map fails the whole load. +- Tests that need one database across connections must use a temp file: local `:memory:` databases are per-connection. + +## Release + +- Tag `v*` triggers `.github/workflows/release.yml`: 5 targets (Linux x64/arm64 — arm64 via `cross` — macOS x64/arm64, Windows x64), zipping the binary, `.tabularium`, and assets. `.tabularium` also ships as a standalone asset because the registry resolves the manifest from release assets, not the git ref. +- `Cargo.lock` is committed; CI and release builds use `--locked`. Keep the `.tabularium` version in sync with `Cargo.toml`. + +## Demo + +- `demo/tabularis_demo.db` and its `.db-wal` are committed; regenerate with `demo/make_demo_db.py` + `demo/seed.sql`. `demo/turso_demo.py` targets remote Turso. diff --git a/CHANGELOG.md b/CHANGELOG.md index 87918ef..10c55e5 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,6 +7,20 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ## [Unreleased] +## [1.0.3] - 2026-10-26 + +### Fixed + +- The ER diagram could never load on a libSQL connection: `get_schema_snapshot` + returned a `{tables, columns, foreign_keys}` envelope, while the host + deserializes the result straight into `Vec` and rejected it with + `invalid type: map, expected a sequence`. The handler now returns one + `{name, columns, foreign_keys}` entry per table, and an empty schema is `[]`. +- `just dev-install` and `uninstall` pointed at the flat `plugins/libsql` + bundle, which the host shadows once a kind-scoped `plugins/drivers/libsql` + bundle exists. Both now target the kind-first path on Linux, macOS and + Windows. + ## [1.0.2] - 2026-10-25 ### Fixed @@ -117,6 +131,8 @@ Thanks to @debba, @NewtTheWolf and @jonaspm — see the [v1.0.0 release notes](https://github.com/TabularisDB/tabularis-libsql-plugin/releases/tag/v1.0.0) for the full list. -[Unreleased]: https://github.com/TabularisDB/tabularis-libsql-plugin/compare/v1.0.1...HEAD +[Unreleased]: https://github.com/TabularisDB/tabularis-libsql-plugin/compare/v1.0.3...HEAD +[1.0.3]: https://github.com/TabularisDB/tabularis-libsql-plugin/releases/tag/v1.0.3 +[1.0.2]: https://github.com/TabularisDB/tabularis-libsql-plugin/releases/tag/v1.0.2 [1.0.1]: https://github.com/TabularisDB/tabularis-libsql-plugin/releases/tag/v1.0.1 [1.0.0]: https://github.com/TabularisDB/tabularis-libsql-plugin/releases/tag/v1.0.0 diff --git a/Cargo.lock b/Cargo.lock index 93c4a71..ed3c0a2 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -467,7 +467,7 @@ dependencies = [ [[package]] name = "libsql-plugin" -version = "1.0.2" +version = "1.0.3" dependencies = [ "base64", "futures-executor", diff --git a/Cargo.toml b/Cargo.toml index 1628727..1b3cadf 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "libsql-plugin" -version = "1.0.2" +version = "1.0.3" edition = "2021" description = "Tabularis driver plugin for libSQL / Turso (local files and remote Hrana-over-HTTP)" license = "Apache-2.0" diff --git a/README.md b/README.md index 942e85e..a33156d 100644 --- a/README.md +++ b/README.md @@ -1,5 +1,7 @@ # Tabularis libSQL / Turso driver +[![CI](https://github.com/TabularisDB/tabularis-libsql-plugin/actions/workflows/ci.yml/badge.svg)](https://github.com/TabularisDB/tabularis-libsql-plugin/actions/workflows/ci.yml) + > ⚠️ **Work in progress** — this plugin is under active development. APIs, > behavior and feature coverage may change, and things may break. Use at your > own risk. @@ -7,8 +9,8 @@ A [Tabularis](https://github.com/TabularisDB/tabularis) database driver plugin for **libSQL**. It connects to: -- **Local libSQL / SQLite files** on disk (via bundled SQLite — nothing to - install), and +- **Local libSQL / SQLite files** on disk (via the bundled libSQL fork — nothing + to install), and - **Remote Turso / sqld servers** over the **Hrana HTTP** protocol. The plugin is a standalone executable that speaks Tabularis' JSON-RPC protocol @@ -23,14 +25,24 @@ The driver picks a backend from the connection form automatically: | You enter | Backend | |-----------|---------| | A file path in **Database** (e.g. `/data/app.db`, `~/notes.db`, `:memory:`) | Local SQLite file | +| A `file:` URI in **Database** / connection string (e.g. `file:///data/app.db`) | Local SQLite file | | A URL in **Database** or **Host** (e.g. `libsql://my-db.turso.io`, `http://localhost:8080/dev/example/`) | Remote Hrana HTTP | | A bare host in **Host** (e.g. `db.turso.io`) | Remote Hrana HTTP (`https://`) | -For Turso, put the **auth token** in the **Password** field, or append it to the -URL as `?authToken=...`. `libsql://`, `wss://` and `turso://` URLs are +A connection URI (`connection_uri`) is authoritative over the decomposed +`host`/`port`/`database` fields. `libsql://`, `wss://` and `turso://` URLs are automatically rewritten to `https://` (and `ws://` to `http://`) for the Hrana -HTTP endpoint. `http://` and `https://` URLs are used as-is. A bare -`localhost:8080` host uses `http://` automatically. +HTTP endpoint; `http://` and `https://` URLs are used as-is. + +For a decomposed `host`, the scheme is chosen from `ssl_mode`: + +- `ssl_mode=disable` forces `http://` for any host. +- A loopback host (`localhost`, `127.0.0.1`, `[::1]`) without + `ssl_mode=require` also uses `http://`. +- Everything else uses `https://`. + +For Turso, put the **auth token** in the **Password** field, or append it to the +URL as `?authToken=...`. Self-hosted sqld namespaces: the namespace is derived from the URL path (`http://localhost:8080/dev/example/` → `example`) and sent as the `x-namespace` @@ -50,13 +62,15 @@ http://localhost:8080/dev/example/ | Area | Status | |------|--------| -| `test_connection`, `ping` | ✅ | +| `initialize`, `ping`, `test_connection` | ✅ | +| Connection-string import (`connection_uri` + `turso`/`http`/`https`/`ws`/`wss`/`file` schemes) | ✅ | | Tables, columns, indexes, foreign keys | ✅ | +| Triggers: list + definition | ✅ | | Views: list, definition, columns, create/alter/drop | ✅ | -| Query execution with server-side LIMIT/OFFSET pagination + total count | ✅ | +| Query execution with LIMIT/OFFSET pagination for wrappable `SELECT`s (`has_more` from a one-row-ahead fetch; `total_rows` is always `null`) | ✅ | | `EXPLAIN QUERY PLAN` | ✅ | | Insert / update / delete rows (bound parameters) | ✅ | -| Schema snapshot + batch columns/FKs (ER diagram) | ✅ | +| Schema snapshot + batch columns/FKs — powers the ER diagram | ✅ | | `CREATE TABLE` SQL, add column, create/drop index | ✅ | | Schemas, stored routines | ❌ (not a SQLite concept — Turso has no stored procedures, and its multi-database model is separate databases, not schemas) | | Rename column | ✅ everywhere (vanilla `RENAME COLUMN`) | @@ -67,6 +81,10 @@ http://localhost:8080/dev/example/ Identifiers are quoted ANSI-style (`"name"`). Booleans are stored as `0`/`1` and BLOBs are returned base64-encoded. +`get_schema_snapshot` returns the host's `TableSchema` array shape — one +`{name, columns, foreign_keys}` entry per table — which is what Tabularis +consumes for the ER diagram. An empty schema returns `[]`. + ### Schema changes via ALTER COLUMN Both local files and remote Turso / sqld servers run the **libSQL fork** of @@ -93,17 +111,27 @@ silently doing nothing. ## Build & test -Requires a Rust toolchain (and a C compiler for the bundled SQLite). +Requires a Rust toolchain (`rust-toolchain.toml` pins the stable channel with +`rustfmt` and `clippy`) and a C compiler for the bundled SQLite. `just` is +convenient but optional — every recipe is a thin wrapper over cargo. ```bash just test # cargo test — unit tests for SQL builders, parsing, RPC just build # debug build just release # optimized release build just lint # clippy -D warnings -just dev-install # build + copy binary + manifest into the Tabularis plugins dir +just fmt # rustfmt +just dev-install # build + copy binary + manifest into plugins/drivers/libsql just repl # local JSON-RPC REPL over stdio ``` +CI runs the same suite with the lockfile pinned: + +```bash +cargo build --locked +cargo test --locked --all-targets +``` + Or directly with cargo: ```bash @@ -120,29 +148,45 @@ echo '{"jsonrpc":"2.0","method":"get_tables","params":{"params":{"database":"/tm ## Installing -`just dev-install` copies `libsql-plugin` and `.tabularium` into the Tabularis -plugins folder (the location Tabularis actually scans — derived from -`ProjectDirs::from("com", "debba", "tabularis")`): +### Release archive + +Download `libsql-plugin-.zip` from the +[releases page](https://github.com/TabularisDB/tabularis-libsql-plugin/releases) +and unpack it so the driver folder contains the executable, `.tabularium` and +`assets/`: + +- **Linux:** `~/.local/share/tabularis/plugins/drivers/libsql/` +- **macOS:** `~/Library/Application Support/tabularis/plugins/drivers/libsql/` +- **Windows:** `%APPDATA%\tabularis\plugins\drivers\libsql\` + +The root is the Tabularis config directory +(`ProjectDirs::from("com", "debba", "tabularis")`). Tabularis resolves +kind-scoped bundles (`plugins/drivers/`) before flat `plugins/` bundles, +so keep the `drivers` segment — a flat copy is shadowed by an installed bundle. +Each release also publishes `.tabularium` as a standalone asset. + +### From source -- **Linux:** `~/.local/share/tabularis/plugins/libsql/` -- **macOS:** `~/Library/Application Support/tabularis/plugins/libsql/` -- **Windows:** `%APPDATA%\tabularis\plugins\libsql\` +`just dev-install` builds a debug binary and copies it plus `.tabularium` into +the same `plugins/drivers/libsql` folder. -Restart Tabularis (or toggle the plugin in Settings) and **libSQL** appears in -the Database Type list. +Either way, restart Tabularis (or toggle the plugin in Settings) and **libSQL** +appears in the Database Type list. ## Architecture ``` src/ -├── main.rs # stdio JSON-RPC loop +├── main.rs # stdio JSON-RPC loop (one request line in, one response line out) +├── lib.rs # shared dispatch, used by main.rs and the REPL ├── rpc.rs # method routing + response helpers ├── client.rs # backend resolution (local vs remote) + unified query/execute ├── hrana.rs # Hrana-over-HTTP pipeline client (remote) ├── models.rs # ConnectionParams ├── error.rs # PluginError ├── handlers/ # metadata / query / crud / ddl -└── utils/ # identifiers, pagination, SQL classification, value conversion +├── utils/ # identifiers, pagination, SQL classification, value conversion +└── bin/test_plugin.rs # `just repl` driver ``` Local files use the embedded **libSQL fork of SQLite** (bundled — nothing to diff --git a/justfile b/justfile index d556161..b81dd87 100644 --- a/justfile +++ b/justfile @@ -31,33 +31,33 @@ fmt: [linux] dev-install: build - mkdir -p ~/.local/share/tabularis/plugins/libsql - cp target/debug/libsql-plugin ~/.local/share/tabularis/plugins/libsql/ - cp .tabularium ~/.local/share/tabularis/plugins/libsql/ - @echo "Installed to ~/.local/share/tabularis/plugins/libsql" + mkdir -p ~/.local/share/tabularis/plugins/drivers/libsql + cp target/debug/libsql-plugin ~/.local/share/tabularis/plugins/drivers/libsql/ + cp .tabularium ~/.local/share/tabularis/plugins/drivers/libsql/ + @echo "Installed to ~/.local/share/tabularis/plugins/drivers/libsql" @echo "Restart Tabularis (or toggle the plugin in Settings) to pick up changes." [macos] dev-install: build - mkdir -p "$HOME/Library/Application Support/tabularis/plugins/libsql" - cp target/debug/libsql-plugin "$HOME/Library/Application Support/tabularis/plugins/libsql/" - cp .tabularium "$HOME/Library/Application Support/tabularis/plugins/libsql/" - @echo "Installed to ~/Library/Application Support/tabularis/plugins/libsql" + mkdir -p "$HOME/Library/Application Support/tabularis/plugins/drivers/libsql" + cp target/debug/libsql-plugin "$HOME/Library/Application Support/tabularis/plugins/drivers/libsql/" + cp .tabularium "$HOME/Library/Application Support/tabularis/plugins/drivers/libsql/" + @echo "Installed to ~/Library/Application Support/tabularis/plugins/drivers/libsql" @echo "Restart Tabularis (or toggle the plugin in Settings) to pick up changes." [windows] dev-install: build - $dest = Join-Path $env:APPDATA "tabularis\plugins\libsql"; New-Item -ItemType Directory -Force -Path $dest | Out-Null; Copy-Item "target\debug\libsql-plugin.exe" $dest; Copy-Item ".tabularium" $dest; Write-Host "Installed to $dest"; Write-Host "Restart Tabularis (or toggle the plugin in Settings) to pick up changes." + $dest = Join-Path $env:APPDATA "tabularis\plugins\drivers\libsql"; New-Item -ItemType Directory -Force -Path $dest | Out-Null; Copy-Item "target\debug\libsql-plugin.exe" $dest; Copy-Item ".tabularium" $dest; Write-Host "Installed to $dest"; Write-Host "Restart Tabularis (or toggle the plugin in Settings) to pick up changes." [linux] uninstall: - rm -rf ~/.local/share/tabularis/plugins/libsql + rm -rf ~/.local/share/tabularis/plugins/drivers/libsql [macos] uninstall: - rm -rf "$HOME/Library/Application Support/tabularis/plugins/libsql" + rm -rf "$HOME/Library/Application Support/tabularis/plugins/drivers/libsql" [windows] uninstall: - $dest = Join-Path $env:APPDATA "tabularis\plugins\libsql" + $dest = Join-Path $env:APPDATA "tabularis\plugins\drivers\libsql" if (Test-Path $dest) { Remove-Item -Recurse -Force $dest } diff --git a/src/handlers/metadata.rs b/src/handlers/metadata.rs index e739a4b..4460ede 100644 --- a/src/handlers/metadata.rs +++ b/src/handlers/metadata.rs @@ -319,23 +319,29 @@ pub fn get_schema_snapshot(id: Value, params: &Value) -> Value { fn get_schema_snapshot_impl(params: &Value) -> Result { let client = connect(params)?; - let names = list_table_names(&client)?; - - let mut tables = Vec::with_capacity(names.len()); - let mut columns = serde_json::Map::new(); - let mut foreign_keys = serde_json::Map::new(); + schema_snapshot_for(&client) +} +/// One entry per table, shaped exactly like the host's `TableSchema`: +/// `{ name, columns, foreign_keys }`. +/// +/// The host deserializes this response straight into `Vec`, so a +/// `{tables, columns, foreign_keys}` envelope (or a per-table map keyed by +/// name) fails that parse and the ER diagram shows a load error instead of the +/// diagram. Keep the top-level value an array — an empty schema is `[]`. +fn schema_snapshot_for(client: &Client) -> Result { + let names = list_table_names(client)?; + + let mut snapshot = Vec::with_capacity(names.len()); for name in &names { - tables.push(json!({ "name": name, "schema": Value::Null, "comment": Value::Null })); - columns.insert(name.clone(), json!(columns_for(&client, name)?)); - foreign_keys.insert(name.clone(), json!(foreign_keys_for(&client, name)?)); + snapshot.push(json!({ + "name": name, + "columns": columns_for(client, name)?, + "foreign_keys": foreign_keys_for(client, name)?, + })); } - Ok(json!({ - "tables": tables, - "columns": Value::Object(columns), - "foreign_keys": Value::Object(foreign_keys), - })) + Ok(Value::Array(snapshot)) } pub fn get_all_columns_batch(id: Value, params: &Value) -> Value { @@ -440,6 +446,62 @@ mod tests { assert_eq!(fk["on_update"], "NO ACTION"); } + #[test] + fn schema_snapshot_matches_host_table_schema_array() { + // Regression: the ER diagram loads `get_schema_snapshot` straight into + // the host's `Vec`. A `{tables, columns, foreign_keys}` + // envelope (the shape this handler used to return) makes that + // deserialization fail, so the diagram window only ever showed + // "Failed to load diagram: invalid type: map, expected a sequence". + let client = in_memory_client(); + client + .execute( + "CREATE TABLE authors(id INTEGER PRIMARY KEY, name TEXT NOT NULL)", + &[], + ) + .expect("create authors"); + client + .execute( + "CREATE TABLE posts(id INTEGER PRIMARY KEY, author_id INTEGER REFERENCES authors(id))", + &[], + ) + .expect("create posts"); + + let snapshot = schema_snapshot_for(&client).expect("schema snapshot"); + let tables = snapshot + .as_array() + .expect("host deserializes the result as a JSON array of tables"); + assert_eq!(tables.len(), 2); + + for table in tables { + // TableSchema { name, columns, foreign_keys } — all required. + assert!(table["name"].is_string()); + assert!(table["columns"].is_array()); + assert!(table["foreign_keys"].is_array()); + } + + let posts = tables + .iter() + .find(|table| table["name"] == "posts") + .expect("posts entry"); + let columns = posts["columns"].as_array().expect("posts columns"); + assert!(columns.iter().any(|column| column["name"] == "author_id")); + + let foreign_keys = posts["foreign_keys"].as_array().expect("posts fks"); + assert_eq!(foreign_keys.len(), 1); + assert_eq!(foreign_keys[0]["column_name"], "author_id"); + assert_eq!(foreign_keys[0]["ref_table"], "authors"); + assert_eq!(foreign_keys[0]["ref_column"], "id"); + } + + #[test] + fn schema_snapshot_for_empty_schema_is_an_empty_array() { + let client = in_memory_client(); + + let snapshot = schema_snapshot_for(&client).expect("schema snapshot"); + assert_eq!(snapshot, json!([])); + } + #[test] fn triggers_use_host_contract_keys() { let client = in_memory_client();