Merge branch 'master' into tyler/update-nativeaot-llvm-infrastructure

Author: rekhoff

Cargo.lock

@@ -153,6 +153,7 @@ ahash = { version = "0.8", default-features = false, features = ["std"] }
 anyhow = "1.0.68"
 anymap = "0.12"
 arrayvec = "0.7.2"
+async-channel = "2.5"
 async-stream = "0.3.6"
 async-trait = "0.1.68"
 axum = { version = "0.7", features = ["tracing"] }

Cargo.toml

@@ -0,0 +1,145 @@
+# A brief overview of SpacetimeDB testing
+## From the perspective of a client SDK or module library developer
+## By pgoldman 2025-06-25, updated 2026-04-14
+
+SpacetimeDB has good test coverage, but it is rather haphazardly spread across several suites.
+Some of the reasons for this are historical, and some have to do with our using multiple repositories.
+This document is an attempt to describe the test suites which would be useful to someone
+working on a new client SDK or module bindings library in a new language,
+or attempting to move an existing client SDK or module bindings library from an external repository in-tree.
+
+## The SDK tests
+
+`crates/testing/src/sdk.rs` defines a test harness which was originally designed for testing client SDKs.
+The basic flow of a test using this harness is:
+
+- Build and freshly publish a module to construct a short-lived initially-empty database.
+- Use that module to run client codegen via `spacetime generate` into a client project.
+- Compile that client project with the newly-generated bindings.
+- Run the client project as a subprocess, passing the database name or `Identity` in an environment variable.
+- The client process connects to the database,
+  runs whatever tests it likes,
+  writes to stdout and/or stderr as it goes,
+  then uses its exit code to report whether the test was successful or not.
+- If the subprocess's exit is non-zero, the test is treated as a failure,
+  and the subprocess's stdout and stderr are reported.
+
+This framework has since been used more generally for integration testing.
+In particular, we maintain equivalent Rust, C#, TypeScript, and C++ modules in the `modules/sdk-test*` family,
+and run the Rust SDK client project at `sdks/rust/tests/test-client` against them through `sdks/rust/tests/test.rs`.
+We similarly maintain `modules/sdk-test-connect-disconnect*` modules
+which run against `sdks/rust/tests/connect_disconnect_client`.
+There are also related SDK-harness-driven suites for event tables, procedures, and views,
+using modules such as `modules/sdk-test-event-table`, `modules/sdk-test-procedure*`, and `modules/sdk-test-view*`.
+The Unreal SDK also uses the same underlying harness through `sdks/unreal/tests/sdk_unreal_harness.rs`.
+
+The harness is designed to support running multiple tests in parallel with the same client project,
+running client codegen exactly once per test suite run.
+This unfortunately still conflicts with our use of the suite to test that modules in different languages behave the same,
+as each test suite invocation will only run `spacetime generate` against one module language at a time,
+never all of them in the same run.
+
+### Testing a new module library
+
+If you are developing a new module bindings library, and wish to add it to the SDK test suite
+so that the existing client test projects will run against it:
+
+1. Create `modules/sdk-test-XX` and `modules/sdk-test-connect-disconnect-XX`, where `XX` is some mnemonic for your language.
+   Populate these with module code which defines all of the same tables and reducers
+   as `modules/sdk-test` and `modules/sdk-test-connect-disconnect` respectively.
+   Take care to use the same names, including casing, for tables, columns, indexes, reducers and other database objects.
+2. Modify `sdks/rust/tests/test.rs` to add an additional call to `declare_tests_with_suffix!` at the bottom,
+   like `declare_tests_with_suffix!(xxlang, "-XX")`, if that driver is the right place for the new language.
+   Some capabilities now live in separate suites in that file, such as procedures and views,
+   so you may need to wire those up separately as well.
+3. Run the tests with `cargo test -p spacetimedb-sdk --test test`.
+
+### Testing a new client SDK
+
+If you are developing a new client SDK, and wish to use the SDK test harness and existing modules
+so that it will run against `modules/sdk-test` and `modules/sdk-test-connect-disconnect`:
+
+1. Find somewhere sensible to define test projects `test-client` and `connect_disconnect_client` for your client SDK language.
+   If your client SDK is in-tree, put these within its directory, following the existing layout under `sdks/rust/tests/` or `sdks/unreal/tests/`.
+2. Use `spacetime generate` manually, or via the harness, to generate those projects' `module_bindings`.
+3. Populate those projects with client code
+   matching `sdks/rust/tests/test-client` and `sdks/rust/tests/connect_disconnect_client` respectively.
+   - Connect to SpacetimeDB running at `http://localhost:3000`.
+   - Connect to the database whose name is in the environment variable `SPACETIME_SDK_TEST_DB_NAME`.
+   - For `test-client`, take a test name as a command-line argument in `argv[1]`, and dispatch to the appropriate test to run.
+   - For `connect_disconnect_client`, there is only one test.
+   - The Rust code jumps through some hoops to do assertions about asynchronous events with timeouts,
+     using an abstraction called the `TestCounter` defined in `sdks/rust/tests/test-counter`.
+     This is effectively a semaphore with a timeout.
+     You may or may not need to replicate this behavior.
+4. Create integration tests in the SDK crate which construct `spacetimedb_testing::sdk::Test` objects,
+   following `sdks/rust/tests/test.rs` or `sdks/unreal/tests/test.rs` as a template.
+5. Define `#[test]` tests for each test case you have implemented,
+   which construct `spacetimedb_testing::sdk::Test` objects containing the various subcommand strings to run your client project,
+   then call `.run()` on them.
+
+### Adding a new test case
+
+If you want to add a new test case to the SDK test suite, to test some new or yet-untested functionality
+of either the module libraries or client SDKs:
+
+1. If necessary, add new tables and/or reducers to `modules/sdk-test` and friends which exercise the behavior you want to test.
+2. Add a new function, `exec_foo`, to the appropriate client project,
+   such as `sdks/rust/tests/test-client/src/lib.rs`,
+   which connects to the database, subscribes to tables and invokes reducers as appropriate,
+   and performs assertions about the events it observes.
+3. Add a branch to that client's dispatch logic,
+   such as the `match` in `sdks/rust/tests/test-client/src/main.rs`,
+   which matches the test name `foo` and dispatches to call your `exec_foo` function.
+4. Add a `#[test]` test function to the relevant test driver,
+   such as `sdks/rust/tests/test.rs`,
+   which does `make_test("foo").run()`, where `"foo"` is the test name you chose in step 3.
+5. Repeat steps 2 through 4 for any other client projects which ought to cover the same behavior.
+6. Run the new test with the relevant `cargo test` command for that SDK.
+
+## Schema parity tests
+
+`crates/schema/tests/ensure_same_schema.rs` is a separate but important companion to the SDK tests.
+It compares the extracted schemas of equivalent modules across languages,
+and is often the first place where casing, indexes, primary keys, or other schema details drift apart.
+As of writing, it covers the `benchmarks`, `module-test`, `sdk-test`, and `sdk-test-connect-disconnect` families.
+
+If you add or update a cross-language module family,
+it is worth considering whether it should also be covered here.
+
+## The smoketests
+
+`crates/smoketests/` defines an integration and regression test suite using a Rust harness.
+These are useful primarily for testing the SpacetimeDB CLI, but can also be used to exercise publish flows,
+documentation, and other end-to-end behavior.
+
+The smoketest harness is still primarily oriented around Rust modules,
+and it does not use the same client-project machinery as the SDK harness.
+It could be extended to do more in that direction, but that may not be worth the effort.
+As of writing, the smoketest suite includes dedicated coverage such as:
+
+- `crates/smoketests/tests/smoketests/csharp_module.rs`, which smoke-tests C# module compilation.
+- `crates/smoketests/tests/smoketests/quickstart.rs`, which replays the quickstart guide for Rust and C#.
+- `crates/smoketests/DEVELOP.md`, which documents how to run and write these tests.
+
+One practical note is that the smoketests use prebuilt `spacetimedb-cli` and `spacetimedb-standalone` binaries,
+so if you modify those crates or their dependencies, you should rebuild before running the suite.
+
+## Standalone integration test
+
+The `spacetimedb-testing` crate has an integration test file, `crates/testing/tests/standalone_integration_test.rs`.
+The tests in this file publish `modules/module-test`, `modules/module-test-cs`, `modules/module-test-ts`, and `modules/module-test-cpp`,
+then invoke reducers or procedures in them and inspect their logs to verify that the behavior is expected.
+These tests do not exercise the entire functionality of `module-test`,
+but by virtue of publishing it do assert that it is syntactically valid and that it compiles.
+
+To add a new module library to the Standalone integration test suite:
+
+1. Create `modules/module-test-XX`, where `XX` is some mnemonic for your language.
+2. Populate this with module code which defines all of the same tables and reducers
+   as the existing `module-test` family.
+   If you notice any discrepancies between the existing languages, those parts may be compiled but not run,
+   and so you are free to ignore them.
+3. Modify `crates/testing/tests/standalone_integration_test.rs` to define new `#[test] #[serial]` test functions
+   which use your new `module-test-XX` module to do the same operations as the existing tests.
+4. Run the tests with `cargo test -p spacetimedb-testing --test standalone_integration_test`.

TESTING.md

@@ -71,17 +71,19 @@ describe('query builder diagnostics', () => {
     'Cannot combine predicates from different table scopes with and/or.';
   const messageHint = 'move extra predicates to .where(...)';
 
+  // This test invokes the TypeScript compiler directly, so it uses an explicit timeout for CI variability.
   it('reports a clear message for free-floating and(...) in semijoin predicates', () => {
     const { status, output } = runTypecheck('and(l.id.eq(r.id), r.id.eq(5))');
     expect(status).not.toBe(0);
     expect(output).toContain(messageStart);
     expect(output).toContain(messageHint);
-  });
+  }, 15000);
 
+  // This test invokes the TypeScript compiler directly, so it uses an explicit timeout for CI variability.
   it('reports a clear message for method-style .and(...) in semijoin predicates', () => {
     const { status, output } = runTypecheck('l.id.eq(r.id).and(r.id.eq(5))');
     expect(status).not.toBe(0);
     expect(output).toContain(messageStart);
     expect(output).toContain(messageHint);
-  });
+  }, 15000);
 });

crates/bindings-typescript/tests/query_error_message.test.ts

@@ -45,6 +45,20 @@ pub async fn exec(mut config: Config, args: &ArgMatches) -> Result<(), anyhow::E
     let force = args.get_flag("force");
 
     let identity = database_identity(&config, &resolved.database, server).await?;
+    let delete_target = if resolved.database == identity.to_string() {
+        identity.to_string()
+    } else {
+        format!("{} ({identity})", resolved.database)
+    };
+
+    if !y_or_n(
+        force,
+        &format!("Are you sure you want to delete database {delete_target}? This action cannot be undone."),
+    )? {
+        println!("Aborting");
+        return Ok(());
+    }
+
     let host_url = config.get_host_url(server)?;
     let request_path = format!("{host_url}/v1/database/{identity}");
     let auth_header = get_auth_header(&mut config, false, server, !force).await?;
@@ -58,7 +72,7 @@ pub async fn exec(mut config: Config, args: &ArgMatches) -> Result<(), anyhow::E
             if !force {
                 print_database_tree_info(&confirm.database_tree).await?;
             }
-            if y_or_n(force, "Do you want to proceed deleting above databases?")? {
+            if force || y_or_n(false, "Do you want to proceed deleting above databases?")? {
                 send_request(&client, &request_path, &auth_header, Some(confirm.confirmation_token))
                     .await?
                     .error_for_status()?;

crates/cli/src/subcommands/delete.rs

@@ -6,6 +6,7 @@ use crate::util::UNSTABLE_WARNING;
 use crate::Config;
 use anyhow::Context;
 use clap::{ArgMatches, Command};
+use futures::future::join_all;
 use serde::Deserialize;
 use spacetimedb_lib::Identity;
 use tabled::{
@@ -24,12 +25,14 @@ pub fn cli() -> Command {
 
 #[derive(Deserialize)]
 struct DatabasesResult {
-    pub identities: Vec<IdentityRow>,
+    pub identities: Vec<Identity>,
 }
 
-#[derive(Tabled, Deserialize)]
-#[serde(transparent)]
-struct IdentityRow {
+#[derive(Tabled)]
+struct DatabaseRow {
+    #[tabled(rename = "Database Name(s)")]
+    pub db_names: String,
+    #[tabled(rename = "Identity")]
     pub db_identity: Identity,
 }
 
@@ -58,15 +61,32 @@ pub async fn exec(mut config: Config, args: &ArgMatches) -> Result<(), anyhow::E
         .context("unable to retrieve databases for identity")?;
 
     if !result.identities.is_empty() {
-        let mut table = Table::new(result.identities);
+        let databases = assemble_rows(&config, server, result.identities).await?;
+        let mut table = Table::new(databases);
         table
             .with(Style::psql())
             .with(Modify::new(Columns::first()).with(Alignment::left()));
-        println!("Associated database identities for {identity}:\n");
+        println!("Associated databases for user {identity}:\n");
         println!("{table}");
     } else {
         println!("No databases found for {identity}.");
     }
 
     Ok(())
 }
+
+async fn assemble_rows(
+    config: &Config,
+    server: Option<&str>,
+    identities: Vec<Identity>,
+) -> anyhow::Result<Vec<DatabaseRow>> {
+    let lookups = identities.into_iter().map(|db_identity| async move {
+        let response = util::spacetime_reverse_dns(config, &db_identity.to_string(), server).await?;
+        let db_names: Vec<_> = response.names.into_iter().map(|name| name.to_string()).collect();
+        Ok(DatabaseRow {
+            db_names: db_names.join(", "),
+            db_identity,
+        })
+    });
+    join_all(lookups).await.into_iter().collect::<anyhow::Result<Vec<_>>>()
+}

crates/cli/src/subcommands/list.rs

@@ -19,3 +19,8 @@ spacetime generate -p spacetimedb-cli --lang <SDK lang> \
   --out-dir <sdk WebSocket schema bindings dir> \
   --module-def ws_schema_v2.json
 ```
+
+Note, the v3 WebSocket protocol does not have a separate schema.
+It reuses the v2 message schema and only changes the websocket binary framing.
+In v2 for example, a WebSocket frame contained a single BSATN-encoded v2 message,
+but in v3, a single WebSocket frame may contain a batch of one or more v2 messages.

crates/client-api-messages/DEVELOP.md

@@ -17,3 +17,4 @@
 pub mod common;
 pub mod v1;
 pub mod v2;
+pub mod v3;

crates/client-api-messages/src/websocket.rs

@@ -0,0 +1,13 @@
+//! Binary framing for websocket protocol v3.
+//!
+//! Unlike v2, v3 does not define a new outer message schema.
+//! A single binary websocket payload contains one or more BSATN-encoded
+//! [`crate::websocket::v2::ClientMessage`] values from client to server,
+//! or one or more consecutive BSATN-encoded [`crate::websocket::v2::ServerMessage`]
+//! values from server to client.
+//!
+//! Client and server may coalesce multiple messages into one websocket payload,
+//! or send them separately, regardless of what the other one does,
+//! so long as logical order is preserved.
+
+pub const BIN_PROTOCOL: &str = "v3.bsatn.spacetimedb";