diff --git a/documentation/concepts/delivery-semantics.md b/documentation/concepts/delivery-semantics.md index 597c1910c..9f2c70b84 100644 --- a/documentation/concepts/delivery-semantics.md +++ b/documentation/concepts/delivery-semantics.md @@ -38,6 +38,11 @@ the server confirms a batch, the client reconnects and re-sends. If the server had already committed the batch but the acknowledgement was lost in flight, the second send produces duplicates. +QWP's `wireSeq` cannot suppress this replay: it is assigned by receive order, +exists only for response correlation on one connection, and resets after a +reconnect. Requests carry no persistent message identifier that the server can +use to recognize the same frame on the next connection. + This path applies to every QuestDB client deployment. ### Multi-host failover replay diff --git a/documentation/concepts/materialized-views.md b/documentation/concepts/materialized-views.md index cf7ed4ac2..a0f7bf9fa 100644 --- a/documentation/concepts/materialized-views.md +++ b/documentation/concepts/materialized-views.md @@ -563,6 +563,93 @@ This happens asynchronously, minimizing write performance impact. ## Enterprise features +### Restricted access with row expiry + +An `EXPIRE ROWS` policy on a passthrough materialized view filters expired rows +from queries before background cleanup removes them. In Enterprise, readers +with column-level SELECT grants also need permission on columns used to enforce +the policy, even when those columns are absent from the query's output. + +#### Direct materialized-view access + +Grant the requested columns and the policy's predicate, partition, and order +columns. For example, on a materialized view `mv` with columns `sym`, `k`, `v`, +`secret`, and designated timestamp `ts`: + +| Expiry policy | Grants needed for `SELECT sym FROM mv` | +| --- | --- | +| `WHEN v < 2.0` | `SELECT ON mv(sym, v)` | +| `WHEN ts < '2025-01-01T00:00:01.000000Z'` | `SELECT ON mv(sym)` | +| `KEEP LATEST PARTITION BY k` | `SELECT ON mv(sym, k)` | +| `KEEP HIGHEST v PARTITION BY k` | `SELECT ON mv(sym, v, k)` | + +Column-level grants implicitly include the designated timestamp. A table-level +SELECT grant covers all columns, including those needed by the policy. + +Review direct grants whenever you enable or change expiry. Granting a policy +column lets the reader query that column explicitly. If it must remain hidden, +use an ordinary SQL view as described below. Different readers can use either +access pattern. + +**COUNT limitation:** `SELECT count() FROM mv` can require SELECT on unrelated +columns as well as policy columns. With sufficient policy-column grants, use an +explicit timestamp projection to count retained rows: + +```questdb-sql +SELECT count() FROM (SELECT ts FROM mv); +``` + +This still requires the policy-column permissions. Direct COUNT also works with +a table-level SELECT grant. + +#### Hide policy columns with an ordinary view + +After configuring expiry and waiting for it to apply, an authorized administrator +can create an ordinary SQL view with only the intended output columns: + +```questdb-sql +CREATE VIEW mv_public AS (SELECT sym FROM mv); +GRANT SELECT ON mv_public TO reader; +``` + +The reader needs the appropriate connection permission, such as `PGWIRE` or +`HTTP`, and SELECT on `mv_public`. They do not need any grant on `mv` or its +policy columns. Both SELECT and COUNT through `mv_public` operate on retained +rows, and its schema exposes only `sym`. Different output column sets can use +separate ordinary views. + +#### Change expiry beneath an existing ordinary view + +Adding expiry, or replacing a policy with one that uses a new hidden column, can +make reads through an existing ordinary view fail with access denied. The view's +saved dependencies must be refreshed by reissuing its complete, unchanged +original definition: + +```questdb-sql +ALTER MATERIALIZED VIEW mv SET EXPIRE ROWS WHEN secret < 20; +SELECT wait_wal_table('mv'); +ALTER VIEW mv_public AS (SELECT sym FROM mv); +``` + +Run these statements in order as an authorized administrator, waiting for each +to complete successfully. The WAL wait ensures that the policy has been applied +before `ALTER VIEW` collects its dependencies; an ALTER acknowledgement alone, +including over the PostgreSQL protocol, does not establish application. + +The `ALTER VIEW` statement preserves existing grants on `mv_public`. Use the +original definition for your view, including any filters and output restrictions. +Readers can receive access-denied errors between policy application and the +view-definition update. Reissuing the definition before policy application does +not pick up the new dependencies. + +Background view compilation and `COMPILE VIEW` do not refresh these dependency +permissions. This procedure also applies to timestamp-only expiry when the +ordinary view predates the policy: implicit timestamp permission on a direct +materialized-view grant does not extend to an ordinary-view-only reader. + +`ALTER MATERIALIZED VIEW mv DROP EXPIRE` requires no ordinary-view repair after +it applies. Rows that have already been physically cleaned up are not restored. + ### Replicated views Replication of the base table is independent of materialized view maintenance. diff --git a/documentation/connect/clients/c-and-cpp.md b/documentation/connect/clients/c-and-cpp.md index 24857c272..13bd9f6d5 100644 --- a/documentation/connect/clients/c-and-cpp.md +++ b/documentation/connect/clients/c-and-cpp.md @@ -696,7 +696,9 @@ on_error:; the 2 MiB target; if 8 rows still exceed 4 MiB — which takes very large string, binary, or array values — the flush fails instead of splitting. - **Recovery depends on `in_doubt`, not on the error code.** Check - `line_sender_error_in_doubt` (C++: `e.in_doubt()`). False means the queue + `line_sender_error_in_doubt` (C++: `e.in_doubt()`). This describes the + failed operation's input, not earlier independent flushes or replay from an + application checkpoint. False means the queue never took the frame and the chunk is intact: re-flush it. True means delivery is uncertain, so `wait` for what the queue already holds, and resend the chunk only where the table's dedup keys make duplicate rows harmless. A @@ -776,13 +778,18 @@ See `qwp_sender.h` for the exact signatures. Complete list: | `column_bool` | LSB-first packed bitmap | `BOOLEAN` | | `column_ts` + `qwp_ts_unit` (`_micros` / `_nanos`) | int64 since epoch | `TIMESTAMP` / `TIMESTAMP_NS` | | `column_date` | int64 millis since epoch | `DATE` | -| `column_uuid` | 16 bytes | `UUID` | -| `column_long256` | 32 bytes (4 LE limbs) | `LONG256` | +| `column_uuid` | 16 bytes, canonical RFC 4122 big-endian | `UUID` | +| `column_long256` | 32 bytes (4 LE limbs, low limb first) | `LONG256` | | `column_ipv4` | uint32 | `IPV4` | | `column_str` | Arrow Utf8 offsets + bytes | `VARCHAR` | | `column_binary` | Arrow Binary offsets + bytes | `BINARY` | | `symbol_i8` / `_i16` / `_i32` | dict codes + Utf8 dictionary | `SYMBOL` | +`column_uuid` takes the UUID's canonical RFC 4122 bytes, exactly as they are +written in the textual form, and byte-swaps them into QWP wire order for you. +The row-oriented `line_sender_buffer_column_uuid` is the exception: it takes +the two 64-bit wire halves, `(lo, hi)`. + Designated timestamp (exactly once per chunk, before flush): `at_nanos` / `at_micros` / `at_millis` / `at_seconds` (millis and seconds are widened to micros on the wire). Decimals, geohash, arrays, and the @@ -873,17 +880,137 @@ bool ingest(questdb_db* db, struct ArrowArray* array, caller keeps `schema`. On failure check `array->release != NULL` before invoking it. - Per-column wire-type hints (`qwp_arrow_override`: force - SYMBOL/VARCHAR, IPv4, char, geohash precision) steer encoding without - touching the Arrow schema. + SYMBOL/VARCHAR, IPv4, char, geohash precision, UUID, LONG256) choose the + wire type without touching the Arrow schema. An override wins for its + column over any field metadata the schema carries. - To append Arrow **columns** into a chunk alongside hand-built ones, use `qwp_chunk_append_arrow_column`, or `qwp_arrow_import_new` + `..._append_arrow_import` to import once - and slice across many chunks. + and slice across many chunks. Neither takes an overrides array, so an + IPv4, char, geohash, UUID, or LONG256 column has to carry its claim as + field metadata instead. The SYMBOL choice is still available on the import + path: `qwp_arrow_import_new` takes a `symbol_mode` argument + (`qwp_symbol_mode_auto`, `_symbol`, `_not_symbol`). - Dictionary-encoded string columns map to `SYMBOL` by default; plain Utf8 to `VARCHAR`. `qwp_sender.h` lists every Arrow type the client accepts, and the kinds it rejects (`Struct`, `Map`, `Interval`, ...); a rejected type fails with `line_sender_error_arrow_unsupported_column_kind`. +### Binary columns: UUID, LONG256, and opaque bytes + +Binary Arrow columns land as `BINARY` unless the column *claims* a richer +type. The width of a column claims nothing on its own: a bare +`FixedSizeBinary(16)` is opaque bytes, not a UUID. A claim comes from the +schema or from an override: + +| Claim | Lands as | +| --- | --- | +| `ARROW:extension:name = arrow.uuid` on `FixedSizeBinary(16)` | `UUID` | +| `questdb.column_type = uuid` field metadata | `UUID` | +| `questdb.column_type = long256` field metadata | `LONG256` | +| `qwp_arrow_override_uuid` / `qwp_arrow_override_long256` | `UUID` / `LONG256` | + +UUID bytes are canonical RFC 4122 big-endian and the client byte-swaps them +into QWP wire order; LONG256 bytes are little-endian limbs, low limb first, +and go out verbatim. The `questdb.column_type` claims and the two overrides +also apply to variable-length `Binary` / `LargeBinary` / `BinaryView` +columns, where every non-null value must then be exactly 16 or 32 bytes. The +`arrow.uuid` extension is the exception: the Arrow spec fixes its storage to +`FixedSizeBinary(16)`, so the client rejects the label on any other type. A +claim whose width doesn't match fails with +`line_sender_error_arrow_ingest`. + +:::caution Behaviour change + +Before client 7.0.0 a bare `FixedSizeBinary(16)` or `(32)` column became +`UUID` or `LONG256` on width alone, with no claim needed. It is now `BINARY` +unless the column carries one of the claims above, so a batch that used to +produce a `UUID` column now produces a `BINARY` one and reports no error. + +The UUID byte order at the API boundary changed in the same release. Code +written against an earlier version passed QWP wire-order bytes to +`qwp_chunk_column_uuid` and `qwp_reader_query_bind_uuid`; those values are +now stored with their bytes reversed, also with no error. + +::: + +#### Claiming in the schema + +Both metadata claims are attached to the Arrow `Field`, so you make them +wherever the batch is built. In Arrow C++: + +```cpp +// The standard Arrow extension label. FixedSizeBinary(16) only. +auto trade_id = arrow::field("trade_id", arrow::fixed_size_binary(16)) + ->WithMetadata(arrow::key_value_metadata( + {"ARROW:extension:name"}, {"arrow.uuid"})); + +// The QuestDB claim, also valid on Binary / LargeBinary / BinaryView. +auto order_hash = arrow::field("order_hash", arrow::fixed_size_binary(32)) + ->WithMetadata(arrow::key_value_metadata( + {"questdb.column_type"}, {"long256"})); + +auto batch_schema = arrow::schema({ + arrow::field("ts", arrow::timestamp(arrow::TimeUnit::NANO)), + trade_id, + order_hash}); +``` + +`questdb.column_type = uuid` has the same shape with `uuid` as the value. Use +it in place of `arrow.uuid` when the bytes sit in a variable-length binary +column, which the extension label doesn't allow. + +Export the batch built against that schema through `arrow::ExportRecordBatch` +and flush it exactly as above — the claims travel with it, and the flush call +needs no extra arguments. + +#### Claiming at the call site + +An override claims the type per flush and leaves the schema alone. Fill in a +`qwp_arrow_override` per column and pass the array to any +`flush_arrow_batch*` call, where the example above passes no overrides: + + + + +```cpp +using namespace questdb::ingress::literals; + +const ::qwp_arrow_override overrides[] = { + {"trade_id", sizeof("trade_id") - 1, qwp_arrow_override_uuid, 0}, + {"order_hash", sizeof("order_hash") - 1, qwp_arrow_override_long256, 0}, +}; + +sender.flush_arrow_batch_and_wait( + "trades"_tn, array, schema, "ts"_cn, + overrides, std::size(overrides)); +``` + + + + +```c +const qwp_arrow_override overrides[] = { + {"trade_id", sizeof("trade_id") - 1, qwp_arrow_override_uuid, 0}, + {"order_hash", sizeof("order_hash") - 1, qwp_arrow_override_long256, 0}, +}; + +bool ok = qwp_sender_flush_arrow_batch_at_column_and_wait( + sender, QDB_TABLE_NAME_LITERAL("trades"), array, schema, + QDB_COLUMN_NAME_LITERAL("ts"), + overrides, sizeof(overrides) / sizeof(overrides[0]), + qwpws_ack_level_ok, &err); +``` + + + + +`arg` (the trailing `0`) carries the geohash precision for +`qwp_arrow_override_geohash` and is unused by every other kind. An override +that names a column the batch doesn't have, repeats another override's +column, or carries an unknown kind fails with +`line_sender_error_invalid_api_call`. + ## Querying data Get a reader (QWP/WebSocket only), prepare/execute SQL, then stream batches and @@ -1084,6 +1211,12 @@ For width-independent access, use `column::visit` and mantissa as little-endian two's-complement bytes. Check for null before decoding it. +`qwp_reader_column_data_get_bytes` also serves `UUID` and `LONG256`. It hands +back UUID values as 16 canonical RFC 4122 big-endian bytes — the decoder has +already reversed them out of wire order, so they match what +`qwp_chunk_column_uuid` and `bind_uuid` take — and LONG256 values as 32 +little-endian limb bytes, low limb first, verbatim from the wire. + ### Parameterised queries Prepare then bind: C `qwp_reader_prepare` + `qwp_reader_query_bind_*` + @@ -1103,8 +1236,8 @@ outlive any cursor it produces. The complete bind surface (C | `bind_decimal64` / `bind_decimal128` / `bind_decimal256` | unscaled value + scale | `DECIMAL` | | `bind_geohash` | bits + precision | `GEOHASH` | | `bind_varchar` | UTF-8 string | `VARCHAR` | -| `bind_uuid` | 16 bytes | `UUID` | -| `bind_long256` | 32 bytes | `LONG256` | +| `bind_uuid` | 16 bytes, canonical RFC 4122 big-endian | `UUID` | +| `bind_long256` | 32 bytes (4 LE limbs, low limb first) | `LONG256` | | `bind_binary` | bytes + length | `BINARY` (not yet accepted server-side) | | `bind_ipv4` | uint32, host order | `IPV4` (not yet accepted server-side) | diff --git a/documentation/connect/clients/python.md b/documentation/connect/clients/python.md index be8d79aa2..a1bb33ad8 100644 --- a/documentation/connect/clients/python.md +++ b/documentation/connect/clients/python.md @@ -277,6 +277,13 @@ The Python value type selects the QuestDB column type: | `TimestampMicros`, `TimestampNanos`, `datetime.datetime` | `TIMESTAMP`, `TIMESTAMP_NS` | | `numpy.ndarray` of `float64`, any number of dimensions | `DOUBLE[]`, `DOUBLE[][]`, ... matching the array's shape. QuestDB 9.0.0 or later | | `decimal.Decimal` | `DECIMAL`, QuestDB 9.2.0 or later | +| `uuid.UUID` | `UUID`, QWP only | +| `ipaddress.IPv4Address` | `IPV4`, QWP only | +| `bytes`, `bytearray`, `memoryview` | `BINARY`, QWP only | +| `Char` | `CHAR`, QWP only | +| `DateMillis` | `DATE`, QWP only | +| `Long256` | `LONG256`, QWP only | +| `Geohash` | `GEOHASH`, QWP only | | `None` | Column omitted for this row, stored as null | Nulls are written by omission: skip the key or pass `None`; there is no @@ -287,10 +294,62 @@ strings in `columns` become `VARCHAR`. `DECIMAL` columns must be created ahead of time with `CREATE TABLE ... (price DECIMAL(18, 2), ...)`; the server does not auto-create them. -`UUID`, `IPv4`, `GEOHASH`, `LONG256`, `CHAR`, `DATE`, and `BINARY` columns -have no `row()` value type. Route them through -[`dataframe()`](#dataframe-ingestion), whose `schema_overrides` covers -`symbol`, `ipv4`, `char`, and `geohash`, or through a SQL `INSERT` via +The seven types marked "QWP only" need QuestDB 10 or later and a `udp`, `ws`, +or `wss` connection. On a `tcp`, `tcps`, `http`, or `https` sender they raise +`QuestDBError`. + +Three of them map to a Python type you already have: `uuid.UUID`, +`ipaddress.IPv4Address`, and any bytes-like value. The other four have no +obvious Python equivalent, so the client gives you a small wrapper for each: + +| Wrapper | Takes | +| --- | --- | +| `Char("A")` | a one-character string | +| `DateMillis(1735689600000)` | milliseconds since the Unix epoch | +| `Long256(0xdeadbeef)` | an unsigned 256-bit `int` | +| `Geohash(bits, precision)` | the hash bits and how many bits they use | +| `Geohash.from_string("u33d8b12")` | the text form, 1 to 12 characters | + +A geohash column uses one precision for every row. The first row you write +fixes it, and if a later row has a different precision, `row()` raises +`QuestDBError` straight away. The bad row is removed, so the rest of the +buffer is untouched and you can carry on writing. + +```python +import uuid +from questdb import Char, DateMillis, Geohash, Long256, TimestampNanos + +sender.row( + "events", + columns={ + "id": uuid.UUID("123e4567-e89b-12d3-a456-426614174000"), + "payload": b"\x00\x01", + "grade": Char("A"), + "day": DateMillis(1735689600000), + "hash": Long256(0xdeadbeef), + "loc": Geohash.from_string("u33d8b12"), + }, + at=TimestampNanos.now(), +) +``` + +Some of these types have one value that QuestDB uses to mean `NULL`. The +client writes that value if you pass it, so it goes in fine and comes back +out as `NULL`: + +| Type | Value that reads back as `NULL` | +| --- | --- | +| `IPV4` | `0.0.0.0` | +| `DATE` | `INT64_MIN` | +| `UUID` | `80000000-0000-0000-8000-000000000000` | +| `LONG256` | all four 64-bit limbs set to `0x8000000000000000` | + +`CHAR` and `BINARY` have no such value. `Char("\x00")` is stored as code unit +0, although some SQL functions treat that as absent, and empty `BINARY` +(`b""`) is a real empty value that is not `NULL`. + +You can also write these types with +[`dataframe()`](#dataframe-ingestion), or with a SQL `INSERT` through [`query()`](#querying). QWP cannot preserve nulls for `BOOLEAN`, `BYTE`, or `SHORT`. An absent value @@ -401,6 +460,22 @@ with questdb.connect("ws::addr=localhost:9000;") as db: db.dataframe(df, table_name="trades", symbols=["symbol"], at="timestamp") ``` +The first successful batch on a fresh direct connection is already a commit +boundary. Later batches are pipelined until an explicit checkpoint or the final +commit. If a transient failure occurs before any batch is successfully +published, the client can replay a materialized source in full. Once any batch +may have committed, it raises instead of replaying from row zero and reports +`in_doubt=True` for the whole DataFrame call, even if the final native write +alone was provably not delivered. This also covers local validation and Arrow +stream errors after earlier batches were published. Internal checkpoints do +not reset the call's delivery status. This aggregation is specific to +`dataframe()`; a sender flush's flag does not summarize earlier independent +flushes. An application-level retry can then duplicate +an already committed prefix unless the table uses suitable `DEDUP UPSERT KEYS`. +A consumed one-shot Arrow stream can also be impossible to replay; when no +batch could have landed, that separate error has `in_doubt=False` and asks for +a fresh reader. + `df` accepts pandas `DataFrame`, polars `DataFrame` and `LazyFrame`, pyarrow `Table`, `RecordBatch`, and `RecordBatchReader`, and any object exposing the Arrow C Data Interface: @@ -428,7 +503,7 @@ Parameters: | `symbols` | `"auto"` (default: categorical and dictionary columns become `SYMBOL`), a bool, or a list of column names or indices. | | `at` | The designated timestamp column (by name or index), a fixed `TimestampNanos` or `datetime` shared by every row, or `questdb.ServerTimestamp`. | | `max_rows_per_batch` | Rows per published batch, default 16384. Sets pipelining granularity, not a safety limit — see below. | -| `schema_overrides` | Per-column wire-type overrides, e.g. `{"addr": "ipv4", "loc": ("geohash", 20)}`; values are `symbol`, `ipv4`, `char`, or `geohash`. | +| `schema_overrides` | Per-column type, e.g. `{"addr": "ipv4", "loc": ("geohash", 20)}`. Values are `symbol`, `ipv4`, `char`, `uuid`, `long256`, or `("geohash", bits)` with `bits` from 1 to 60. Beats any Arrow field metadata on the column. Needs a frame where every column is Arrow-backed; otherwise it raises `UnsupportedDataFrameShapeError`. | `max_rows_per_batch` decides how the frame is cut into published batches, and each batch is one unit of encoding, memory, and server-side apply. @@ -437,9 +512,10 @@ negotiated per-batch byte cap regardless of this setting, and a single row is never bounded by it. What it does control: - Peak client memory: each batch is encoded and held as one frame. -- Recovery quantum: a commit checkpoint fires every 100 batches, so - `max_rows_per_batch × 100` rows is the replay window on a transient - failover. +- Checkpoint spacing: sliceable Arrow inputs add a commit checkpoint about + every 100 batches. `max_rows_per_batch × 100` approximates the maximum + periodic uncommitted tail, not a safe whole-source replay window; the first + successful batch on a fresh connection is already a commit boundary. - Per-batch overhead: very small batches pay framing and server-side apply costs per batch. @@ -459,6 +535,95 @@ row ingestion. A frame the columnar path cannot express raises `UnsupportedDataFrameShapeError` with per-column failures in `column_failures`. +### Binary, UUID, and LONG256 columns + +Binary columns — `pyarrow.binary()`, `large_binary()`, `fixed_size_binary(n)`, +and polars `Binary` — are written as `BINARY`. The width of a column does not +decide its type, so a 16-byte column is treated as plain bytes, not as a UUID. + +If you want `UUID` or `LONG256`, say so. The easiest way is an object column +of `uuid.UUID` values, which needs no extra configuration — the client works +out the byte order for you: + +```python +import uuid + +df = pd.DataFrame({ + "trade_id": [uuid.uuid4(), uuid.uuid4()], + "price": [2615.54, 65432.10], + "timestamp": pd.to_datetime([ + "2025-01-01T00:00:00Z", + "2025-01-01T00:00:01Z", + ]), +}) + +db.dataframe(df, table_name="trades", at="timestamp") +``` + +For a column that is already binary, name the type with `schema_overrides`: + +```python +db.dataframe( + df, + table_name="trades", + at="timestamp", + schema_overrides={"trade_id": "uuid", "order_hash": "long256"}, +) +``` + +Every non-null value must then be exactly 16 bytes for `uuid`, or 32 bytes +for `long256`. UUID bytes are canonical RFC 4122 big-endian — the same bytes +`uuid.UUID.bytes` gives you, and the same bytes a `UUID` result column reads +back. LONG256 bytes are little-endian limbs, least significant first, and are +sent unchanged. + +The third way is a pyarrow column built with the `arrow.uuid` extension type, +which carries the claim itself: + +```python +import pyarrow as pa + +trade_ids = pa.ExtensionArray.from_storage( + pa.uuid(), + pa.array([u.bytes for u in trade_uuids], type=pa.binary(16)), +) +``` + +This one needs pyarrow 18 or later, where `pa.uuid()` was added. Build the +column from `pa.uuid()` itself — writing `ARROW:extension:name` as field +metadata is not the same thing, because a pandas column keeps the Arrow type +but not the field, so the label is lost on the way in. + +:::note + +A frame where every column is Arrow-backed takes a different code path from +one that mixes Arrow and NumPy columns, and the two treat an unlabelled +16- or 32-byte column differently. + +On a fully Arrow-backed frame it is written as `BINARY`, because +`schema_overrides` is there if you meant something else. If any column is not +Arrow-backed, `schema_overrides` is unavailable, and rather than guess +between "plain bytes" and "a UUID whose label was lost", the client refuses +the column and tells you how to say which you meant. To send those widths as +plain bytes there, pass them as an object column of `bytes`. + +::: + +### DATE columns + +`row()` writes a `DATE` with the `DateMillis` wrapper. `dataframe()` has no +equivalent, and no `date` value for `schema_overrides`, because the Arrow +type of the column already says it: `pyarrow.timestamp("ms")`, +`pyarrow.date32()`, and `pyarrow.date64()` are all written as `DATE`, on a +frame where every column is Arrow-backed. + +A NumPy `datetime64[ms]` column is not the same thing. It is widened to a +microsecond `TIMESTAMP`, and the timezone-aware `datetime64[ms, tz]` dtype is +rejected. Reading back, a `DATE` column arrives as +`pyarrow.timestamp("ms", "UTC")`, so `to_arrow()` and +`to_pandas(dtype_backend="pyarrow")` can write it straight back, while plain +`to_pandas()` gives you `datetime64[ms, UTC]`, which `dataframe()` rejects. + Naive timestamps — DataFrame columns and the scalar `at` alike — are interpreted as UTC, matching the numpy `datetime64` convention. Prefer timezone-aware values throughout. @@ -583,6 +748,10 @@ thread that created it; `db.close()` waits for open leases. | `SYMBOL` | `Categorical` sharing one dictionary across batches | | `VARCHAR` | Strings with `None` for null | | `DECIMAL`, `UUID`, `BINARY` | `object` columns of `decimal.Decimal`, `uuid.UUID`, `bytes` | +| `LONG256` | `object` column of Python `int` — the only type wide enough without pyarrow | +| `IPV4`, `CHAR` | `uint32`, `uint16` | +| `DATE` | `datetime64[ms]` | +| `GEOHASH` | A signed integer wide enough for the column's precision: `int8` up to 7 bits, `int16` to 15, `int32` to 31, `int64` to 60 | QuestDB's sentinel values (for example `NaN` doubles and `INT64_MIN` longs) are decoded as nulls rather than leaking as magic numbers. @@ -590,6 +759,54 @@ are decoded as nulls rather than leaking as magic numbers. a `types_mapper=` callable select pyarrow-backed dtypes instead, matching the `pd.read_sql` convention. +### Writing a result back + +Read a table, change it, and write it back, and the column types survive: + +```python +df = db.query("SELECT * FROM trades").to_pandas() +df["price"] *= 1.01 +db.dataframe(df, table_name="trades_adjusted", at="timestamp") +``` + +This needs a little help from the client, because five types cannot be told +apart from their pandas dtype alone. A `UUID` column arrives as `bytes`, an +`IPV4` as `uint32`, and writing those back would give you a `BINARY` and a +`LONG` column. So `to_pandas()` also records what each column was, in +`df.attrs["questdb"]`, and `dataframe()` reads it back. The types it covers +are `UUID`, `LONG256`, `IPV4`, `CHAR`, and `GEOHASH`. + +All the `to_pandas()` backends do this, so plain, `"pyarrow"`, and +`"numpy_nullable"` all round-trip the same way. + +Editing the frame is safe. A column you drop, rename, or convert to another +type simply loses its record, and the write goes ahead with whatever the +column now is. `symbols` and `schema_overrides` win over it, so you can +always state a type yourself. Two types do not survive unchanged: `BYTE` and +`SHORT` columns come back as `INT`, and `INT` as `LONG`. + +If a recorded type cannot apply to the column as it now stands — say the +column is an unsigned integer and the record says `geohash` — the client +warns and writes the column as its own type implies, rather than failing. + +You can write the record yourself. It is an ordinary dictionary, and the +`version` key is required: + +```python +df.attrs["questdb"] = { + "version": 1, + "columns": { + "src_ip": {"kind": "ipv4"}, + "pos": {"kind": "geohash", "precision_bits": 20}, + }, +} +``` + +`kind` is one of `uuid`, `long256`, `ipv4`, `char`, or `geohash`, and +`precision_bits` goes with `geohash` only. Naming a column that is not in the +frame does no harm; it is ignored. A dictionary without `version`, or with a +version this client does not know, is ignored completely. + ### DDL, DML, and cancellation Statements such as `CREATE`, `ALTER`, `INSERT`, `UPDATE`, and `DROP` go @@ -844,7 +1061,7 @@ All failures raise `QuestDBError` (or a subclass). Inspect: | Property | Meaning | | --- | --- | | `code` | A `QuestDBErrorCode` member; compare by identity, e.g. `err.code is QuestDBErrorCode.Cancelled`. | -| `in_doubt` | `True` when the failed operation may already have delivered its input; retrying can duplicate rows without deduplication. | +| `in_doubt` | `True` when the failed ingestion operation may already have delivered its input. For QWP `dataframe()`, this includes earlier batches from the same call, even on a later validation error. Retrying can duplicate rows without deduplication. | | `sender_error` | Structured server diagnostic for QWP sender failures, or `None`. | Codes you will most often dispatch on: diff --git a/documentation/connect/clients/rust.md b/documentation/connect/clients/rust.md index 40ec9e4cc..8336d1449 100644 --- a/documentation/connect/clients/rust.md +++ b/documentation/connect/clients/rust.md @@ -255,6 +255,18 @@ QWP cannot preserve nulls for `BOOLEAN`, `BYTE`, or `SHORT`. An absent value in one of those columns is received as `false` or `0`; use a wider nullable type when the distinction matters. +Everywhere in the API a UUID is 16 canonical RFC 4122 big-endian bytes — the +bytes `uuid::Uuid::as_bytes()` gives you — with one exception. `column_uuid` +on the row buffer takes the two 64-bit halves of the QWP wire encoding, +`(lo, hi)`, and is the only place you have to think about wire order. + +`Chunk::column_uuid` takes one 16-byte array per row, so wrap a single value +rather than splitting it yourself: + +```rust +chunk.column_uuid("trade_id", std::slice::from_ref(u.as_bytes()), None)?; +``` + ## Chunk ingestion {#sending-data-column-major} Use a `Chunk` when values already live in column slices. All columns and the @@ -340,8 +352,8 @@ needs an entry in `data`, which the encoder ignores. | `BOOLEAN` | `column_bool(name, bits, row_count, validity)` | LSB-first bit-packed values | | `TIMESTAMP`, `TIMESTAMP_NS` | `column_ts(name, data, TimestampUnit, validity)` | Epoch `i64` values | | `DATE` | `column_date` | Epoch milliseconds | -| `UUID` | `column_uuid` | `&[[u8; 16]]` in QuestDB wire order | -| `LONG256` | `column_long256` | `&[[u8; 32]]` in little-endian limb order | +| `UUID` | `column_uuid` | `&[[u8; 16]]`, one 16-byte value per row, in canonical RFC 4122 big-endian order — each element is what `uuid::Uuid::as_bytes()` gives you | +| `LONG256` | `column_long256` | `&[[u8; 32]]` in little-endian limb order, least-significant limb first | | `IPv4` | `column_ipv4` | Host-order `u32` values | | `VARCHAR` | `column_str`, `column_str_large` | Arrow Utf8 offsets and bytes | | `BINARY` | `column_binary` | Arrow Binary offsets and bytes | @@ -410,7 +422,7 @@ db.flush_arrow_batch( "trades", &record_batch, None, // server-assigned designated timestamp - &[], // no Arrow column overrides + &[], // per-column wire-type overrides; see below Some(AckLevel::Ok), )?; ``` @@ -429,7 +441,10 @@ use questdb::ingress::{ ColumnName, }; -let overrides: [ArrowColumnOverride<'_>; 0] = []; +let overrides = [ + ArrowColumnOverride::Uuid { column: "trade_id" }, + ArrowColumnOverride::Long256 { column: "order_hash" }, +]; let options = PolarsIngestOptions::new() .max_rows(50_000) .timestamp_column(ColumnName::new("timestamp")?) @@ -439,8 +454,9 @@ let options = PolarsIngestOptions::new() db.flush_polars_dataframe("trades", &dataframe, &options)?; ``` -`max_rows(0)` uses the default batch size. Omitting `timestamp_column` asks the -server to assign timestamps. Omitting `ack_level` uses the pool default. +Pass `&[]` for `overrides` when no column needs one. `max_rows(0)` uses the +default batch size. Omitting `timestamp_column` asks the server to assign +timestamps. Omitting `ack_level` uses the pool default. `flush_polars_dataframe` checkpoints the frame and automatically retries the uncommitted tail after a transient failover. `flush_arrow_batch` returns a @@ -450,6 +466,52 @@ uncertain failure can also duplicate rows. Use [deduplication](/docs/concepts/deduplication/) when duplicates would be harmful. +On a failed `flush_polars_dataframe` call, `err.in_doubt()` covers the whole +DataFrame. It is true if any batch may have been delivered, including batches +confirmed by an earlier checkpoint, even when a later batch fails validation. +Internal retries and connection replacements retain this call-level status. +The flag is conservative: it does not identify a safe row offset for resuming +the load. A false flag does not make a validation error retryable; correct the +input first. Low-level Arrow and chunk flushes retain their current-operation +scope and do not aggregate earlier independent calls. + +### Binary columns: UUID, LONG256, and opaque bytes + +Binary columns land as `BINARY` unless the column claims a richer type, and a +byte width claims nothing on its own: a bare `FixedSizeBinary(16)` is opaque +bytes, not a UUID. A claim comes either from the Arrow schema — the +`ARROW:extension:name = arrow.uuid` extension label on `FixedSizeBinary(16)`, +or `questdb.column_type = uuid` / `= long256` field metadata — or from an +`ArrowColumnOverride::Uuid` / `::Long256` entry as above, which wins over any +metadata on that column. + +Polars has no fixed-size binary dtype, so it needs the override. Its `Binary` +columns export as Arrow `BinaryView`, and every non-null value must then be +exactly 16 or 32 bytes. UUID bytes are canonical RFC 4122 big-endian, and the +client byte-swaps them into wire order for you; LONG256 bytes are +little-endian limbs, low limb first, and are sent unchanged. A value of the +wrong width fails with `ErrorCode::ArrowIngest`. + +Polars `Object` columns are rejected outright. They export as +`FixedSizeBinary(8)` holding in-process handles, which is indistinguishable +from ordinary opaque binary once converted, so the client refuses them rather +than storing meaningless addresses. Cast the column to a supported dtype +first. + +:::caution Behaviour change + +Before client 7.0.0 a bare `FixedSizeBinary(16)` or `(32)` column became +`UUID` or `LONG256` on width alone. It is now `BINARY` unless the column +carries one of the claims above. A batch that used to produce a `UUID` column +now produces a `BINARY` one, with no error. + +The UUID byte order at the API boundary changed in the same release. Code +written against an earlier version passed wire-order bytes to +`Chunk::column_uuid` and `bind_uuid`; those values are now stored reversed, +also with no error. + +::: + ## Querying Borrow a reader, prepare SQL, bind values, execute, and pull typed batches: @@ -521,6 +583,11 @@ Available builders include: - `bind_geohash` - `bind_null` and the typed `bind_null_*` variants +`bind_uuid` takes 16 canonical RFC 4122 big-endian bytes by value, so pass +`*u.as_bytes()` or `u.into_bytes()`. `bind_long256` takes 32 little-endian +limb bytes, low limb first. Both are the byte orders the matching result +columns are read back in. + ### Reading columns `BatchView::column(index)` returns a non-exhaustive `ColumnView`. Match the @@ -533,7 +600,7 @@ variant before reading values: | `Symbol` | `resolve(row) -> Option<&str>` | | `Varchar` | `value(row) -> Option<&str>` | | `Binary` | `value(row) -> Option<&[u8]>` | -| `Uuid`, `Long256` | Fixed-size byte-array reference | +| `Uuid`, `Long256` | Fixed-size byte-array reference: `Uuid` yields 16 canonical RFC 4122 big-endian bytes, `Long256` 32 little-endian limb bytes, low limb first | | `Decimal64`, `Decimal128`, `Decimal256` | Integer value plus the column scale | | `Geohash` | Bits plus precision | | `DoubleArray`, `LongArray` | Per-row shape and element data | @@ -673,6 +740,11 @@ publishes or completes its first frame, so treat `None` from `acked_fsn()` as Recovery turns on `err.in_doubt()`, not on `err.code()` and not on whether the buffer or chunk still holds rows. +For these low-level APIs, the flag describes the failed operation's input. It +does not summarize earlier independent flushes or authorize replaying all data +since an application checkpoint. The DataFrame-level aggregation described +above applies specifically to `flush_polars_dataframe`. + When `in_doubt()` is `false`, the flush failed before the queue took the frame, so the rows never entered the send path and your input is intact: re-flush it. When `in_doubt()` is `true`, delivery is diff --git a/documentation/connect/wire-protocols/qwp-ingress-websocket.md b/documentation/connect/wire-protocols/qwp-ingress-websocket.md index e27c7ea5e..63adc3537 100644 --- a/documentation/connect/wire-protocols/qwp-ingress-websocket.md +++ b/documentation/connect/wire-protocols/qwp-ingress-websocket.md @@ -420,7 +420,7 @@ columns, which keeps every message self-contained for store-and-forward replay. | 9 | `0x09` | SYMBOL | var | Dictionary-encoded string | | 10 | `0x0A` | TIMESTAMP | 8 | Microseconds since Unix epoch | | 11 | `0x0B` | DATE | 8 | Milliseconds since Unix epoch | -| 12 | `0x0C` | UUID | 16 | RFC 4122 UUID | +| 12 | `0x0C` | UUID | 16 | UUID, two LE int64 halves, lo first | | 13 | `0x0D` | LONG256 | 32 | 256-bit integer | | 14 | `0x0E` | GEOHASH | var | Geospatial hash | | 15 | `0x0F` | VARCHAR | var | Length-prefixed UTF-8 | @@ -812,7 +812,15 @@ uncompressed mode. ### UUID 16 bytes per value: 8 bytes for the low 64 bits, then 8 bytes for the high -64 bits, both little-endian. +64 bits, both little-endian. Reverse all 16 bytes and you get the canonical +RFC 4122 order. + +The C, C++, Rust, and Python clients take RFC 4122 bytes in their APIs, so +they reverse each value on the way to the wire and again on the way back. The +one exception is the row-buffer call that takes the two 64-bit halves +directly — `line_sender_buffer_column_uuid(buffer, name, lo, hi, err)` and +its Rust equivalent — which is already in wire order and is passed through +unchanged. ### LONG256 diff --git a/documentation/high-availability/store-and-forward/concepts.md b/documentation/high-availability/store-and-forward/concepts.md index 6acafca19..6591c5bfc 100644 --- a/documentation/high-availability/store-and-forward/concepts.md +++ b/documentation/high-availability/store-and-forward/concepts.md @@ -62,8 +62,9 @@ Two distinct counters track frame identity: - **FSN** (frame-sequence-number) — a monotonic counter assigned when a frame is appended to the substrate. FSN survives reconnects and (in SF mode) restarts. It is the substrate's permanent identifier for a frame. -- **wireSeq** — the per-connection counter the server uses for - deduplication, reset to `0` on every successful WebSocket upgrade. +- **wireSeq** — a per-connection counter used to correlate cumulative QWP + responses with sent frames. It resets to `0` on every successful WebSocket + upgrade. On every (re)connect the relationship is pinned: @@ -82,9 +83,11 @@ Two consequences: - Frames **must** be sent in strict order. The wire format does not serialise `wireSeq` — the server assigns it implicitly from receive order. Reordering breaks the FSN mapping. -- After a reconnect, the server sees the **same payloads** at new - `wireSeq` values. Server-side dedup keys off `messageSequence` inside - the payload, not `wireSeq`, so replay does not produce double-writes. +- After a reconnect, the server sees the **same payloads** at new `wireSeq` + values. QWP requests contain neither `wireSeq` nor a persistent message ID, + so the protocol does not deduplicate those payloads across connections. If + the server committed a frame but its acknowledgement was lost, replay can + write the rows again. ## Trim: how unacked data is reclaimed @@ -151,8 +154,10 @@ On every successful (re)connect: 2. `wireSeq` resets to `0`. 3. The read cursor rewinds to the first un-acked frame on disk (or in memory). -4. Frames stream to the wire in FSN order. The server's dedup window - absorbs any frames that landed before the disconnect. +4. Frames stream to the wire in FSN order. A frame that landed before the + disconnect but was not acknowledged is sent again and can duplicate rows; + use table-level `DEDUP UPSERT KEYS` with stable row identity when this must + be suppressed. 5. New frames appended by the producer during replay are picked up automatically — the I/O loop watches a volatile `publishedFsn` cursor. diff --git a/documentation/query/sql/compile-view.md b/documentation/query/sql/compile-view.md index 956501705..f5f150a16 100644 --- a/documentation/query/sql/compile-view.md +++ b/documentation/query/sql/compile-view.md @@ -99,6 +99,14 @@ Use `COMPILE VIEW` when you want to: 4. **Diagnose issues**: Check why a view is invalid by triggering compilation errors +### Expiry-policy dependency changes + +When an expiry policy adds hidden column dependencies beneath an ordinary view, +`COMPILE VIEW` does not refresh its saved dependency permissions. In Enterprise, +restricted readers can continue to receive access-denied errors even if the view +is valid. Wait for policy application, then reissue the original definition with +`ALTER VIEW`. See the [policy-change procedure](/docs/concepts/materialized-views/#change-expiry-beneath-an-existing-ordinary-view). + ## Errors | Error | Cause | diff --git a/documentation/security/rbac.md b/documentation/security/rbac.md index 5b10f3c26..84988e35e 100644 --- a/documentation/security/rbac.md +++ b/documentation/security/rbac.md @@ -119,6 +119,23 @@ GRANT SELECT ON aapl_trades TO aapl_analyst; The user `aapl_analyst` can only see AAPL trades. They have no access to the underlying `trades` table. +### Materialized views with row expiry + +Column-restricted readers of a materialized view with `EXPIRE ROWS` need SELECT +on the requested columns and the columns used by its policy. Enabling or changing +expiry can therefore require updating direct grants. Those grants also permit +reading the policy columns explicitly. + +To keep policy columns hidden, grant access to an ordinary SQL view exposing only +the intended output. Its readers need no underlying materialized-view grants. +When expiry adds dependencies beneath an existing ordinary view, wait for WAL +application and reissue the original definition with `ALTER VIEW`; reads can be +denied until that update. `COMPILE VIEW` does not repair these dependencies. + +See [restricted access with row expiry](/docs/concepts/materialized-views/#restricted-access-with-row-expiry) +for grant examples, the COUNT limitation and workaround, and the policy-change +procedure. + ## Common scenarios ### Read-only analyst