Add support for multiple index columns

docs/md/explanation/derived_table.md

@@ -58,8 +58,8 @@ let totals = client.table(TableData::View(view), TableInitOptions::default()).aw
 
 An unpivoted `View` yields a `Table` with the `View`'s own columns, including
 `expressions` and window columns, and one row for every row passing its
-`filter`. The source's `index` is inherited when the index column is among the
-`View`'s `columns`, and its `limit` is inherited as well.
+`filter`. The source's `index` is inherited when every index column is among
+the `View`'s `columns`, and its `limit` is inherited as well.
 
 A pivoted `View` yields the table you see on screen. Each `group_by` column
 becomes a typed key column of the same name, and each data column takes the name
@@ -346,7 +346,7 @@ changes several behaviors for code written against the old implementation:
 - A row updated out of the source `View`'s `filter` is removed, where the copy
   retained it with stale values.
 - Updates arrive in the same cycle as the source rather than eventually.
-- A `View` which omits its source's index column is keyed by row identity, so
-  updates apply in place rather than appending.
+- A `View` which omits any of its source's index columns is keyed by row
+  identity, so updates apply in place rather than appending.
 - A pivoted `View` yields named key columns and only the rows the `View`
   displays, where the copy appended its changed rows on every update.

docs/md/explanation/table/options.md

@@ -2,9 +2,10 @@
 
 <div class="warning">`limit` cannot be used in conjunction with `index`.</div>
 
-Initializing a `Table` with an `index` tells Perspective to treat a column as
-the primary key, allowing in-place updates of rows. Only a single column (of any
-type) can be used as an `index`. Indexed `Table` instances allow:
+Initializing a `Table` with an `index` tells Perspective to treat a column, or
+a combination of columns, as the primary key, allowing in-place updates of
+rows. A column of any type can be used as an `index`. Indexed `Table` instances
+allow:
 
 -   In-place _updates_ whenever a new row shares an `index` values with an
     existing row
@@ -33,6 +34,52 @@ indexed_table = perspective.Table(data, index="a");
 
 </div>
 
+### Composite index
+
+A list of column names keys each row by the combination of those columns'
+values, in list order. A list of one column is the same as naming that column
+directly, and an empty list is an error.
+
+<div class="javascript">
+
+JavaScript:
+
+```javascript
+const table = await perspective.table(data, { index: ["name", "sector"] });
+await table.update([{ name: "AAPL", sector: "Tech", price: 1 }]);
+await table.remove([["AAPL", "Tech"]]);
+await table.get_index(); // ["name", "sector"]
+```
+
+</div>
+<div class="python">
+
+Python
+
+```python
+table = perspective.table(data, index=["name", "sector"])
+table.update([{"name": "AAPL", "sector": "Tech", "price": 1}])
+table.remove([["AAPL", "Tech"]])
+table.get_index()  # ["name", "sector"]
+```
+
+</div>
+
+A composite index differs from a single-column index in a few ways:
+
+-   Every `update()` row must supply every index column, and an update which
+    omits one is rejected. A partial update may still omit any other column.
+-   `remove()` takes a list of keys, each a list of the index column values in
+    index order, or an Arrow with one column per index column.
+-   The `__INDEX__` column of an `update()`, and the `__ID__` and `__INDEX__`
+    columns of a `View`, carry each key as a list of the index column values.
+    `__INDEX__` is not supported in Arrow updates of a composite-indexed
+    `Table`; include the index columns instead.
+-   A `View` with no `sort` orders rows by the index columns in index order,
+    with each column ordered by its own type and `null` first.
+-   `vlookup("column", key1, key2, ...)` takes one key argument per index
+    column.
+
 Initializing a `Table` with a `limit` sets the total number of rows the `Table`
 is allowed to have. When the `Table` is updated, and the resulting size of the
 `Table` would exceed its `limit`, rows that exceed `limit` overwrite the oldest

docs/md/explanation/view/advanced.md

@@ -236,8 +236,8 @@ tables across clients.
 
 Register a callback to be notified whenever rows are removed from the underlying
 `Table` by `remove()`, which requires an `index`. The callback receives the
-`port_id` and the removed `index` column values as an Apache Arrow of a single
-column named after the index. It fires once per update step, only for rows which
+`port_id` and the removed `index` column values as an Apache Arrow with one
+column per index column, named after that column. It fires once per update step, only for rows which
 existed before that step; `replace()` reports the keys it does not re-supply,
 and `clear()` reports every key:
 

docs/md/explanation/view/config/expressions.md

@@ -166,7 +166,7 @@ to the nearest multiple of 10.
 | `max(a, b, ...)`          | Maximum of inputs                                     |
 | `random()`                | Random float between 0.0 and 1.0                      |
 | `col(name)`               | Look up a column by string name at runtime            |
-| `vlookup(col, key)`       | Look up a value in another column by row key          |
+| `vlookup(col, key, ...)`  | Look up a value in another column by row key, one key argument per `index` column |
 
 ## See also
 

docs/md/how_to/python/table.md

@@ -67,6 +67,14 @@ data.set_index("datetime")
 table = perspective.table(data, index="index")
 ```
 
+A `pandas.MultiIndex` arrives as one column per level, named after the level,
+so a composite `index` keyed by those columns preserves it:
+
+```python
+data = data.set_index(["name", "sector"])
+table = perspective.table(data, index=["name", "sector"])
+```
+
 ## Time Zone Handling
 
 When parsing `"datetime"` strings, times without an explicit timezone offset are

packages/viewer-datagrid/src/ts/style_handlers/body.ts

@@ -103,6 +103,14 @@ interface StyleMemo {
     value: unknown;
 }
 
+function same_key(a: unknown, b: unknown): boolean {
+    if (Array.isArray(a) && Array.isArray(b)) {
+        return a.length === b.length && a.every((v, i) => v === b[i]);
+    }
+
+    return a === b;
+}
+
 function row_header_depth(row_header: unknown[] | undefined): number {
     if (!row_header) {
         return 0;
@@ -302,7 +310,7 @@ export function applyBodyCellStyles(
             const id = model._ids[(metadata.y ?? 0) - (metadata.y0 ?? 0)];
             let key_match = true;
             for (let i = 0; i < selectedId.length; i++) {
-                if (selectedId[i] !== id[i]) {
+                if (!same_key(selectedId[i], id[i])) {
                     key_match = false;
                     break;
                 }

rust/perspective-client/perspective.proto

@@ -298,7 +298,7 @@ message GetHostedTablesResp {
 
 message HostedTable {
     string entity_id = 1;
-    optional string index = 2;
+    repeated string index = 2;
     optional uint32 limit = 3;
 }
 
@@ -382,6 +382,11 @@ message ViewGetConfigResp {
 
 ////////////////////////////////////////////////////////////////////////////////
 
+// The columns of a composite `index`, in key order.
+message IndexColumns {
+    repeated string columns = 1;
+}
+
 // `Client::table`.
 message MakeTableReq {
     MakeTableData data = 1;
@@ -390,6 +395,7 @@ message MakeTableReq {
         oneof make_table_type {
             string make_index_table = 1;
             uint32 make_limit_table = 2;
+            IndexColumns make_composite_index_table = 6;
         };
 
         // Back this Table's canonical data with the on-disk storage backend

rust/perspective-client/src/rust/client.rs

@@ -29,7 +29,7 @@ use crate::proto::{
     HostedTable, JoinType, MakeJoinTableReq, MakeTableReq, RemoveHostedTablesUpdateReq, Request,
     Response, ServerError, ServerSystemInfoReq,
 };
-use crate::table::{JoinOptions, Table, TableInitOptions, TableOptions, ViewBinding};
+use crate::table::{JoinOptions, Table, TableIndex, TableInitOptions, TableOptions, ViewBinding};
 use crate::table_data::{TableData, UpdateData};
 use crate::table_ref::TableRef;
 use crate::utils::*;
@@ -576,6 +576,11 @@ impl Client {
     /// # Ok(()) }
     /// ```
     pub async fn table(&self, input: TableData, options: TableInitOptions) -> ClientResult<Table> {
+        let mut options = options;
+        if let Some(index) = options.index.take() {
+            options.index = Some(index.normalize()?);
+        }
+
         let entity_id = match options.name.clone() {
             Some(x) => x.to_owned(),
             None => randid(),
@@ -588,7 +593,11 @@ impl Client {
             if let (None, Some(index)) = (&options.index, &source_index) {
                 let config = view.get_config().await?;
                 let is_flat = config.group_by.is_empty() && config.split_by.is_empty();
-                let has_index = config.columns.iter().flatten().any(|x| x == index);
+                let has_index = index
+                    .columns()
+                    .iter()
+                    .all(|column| config.columns.iter().flatten().any(|x| x == column));
+
                 if is_flat && has_index {
                     options.index = Some(index.clone());
                 }
@@ -713,7 +722,7 @@ impl Client {
         let client = self.clone();
         match self.oneshot(&msg).await? {
             ClientResp::MakeJoinTableResp(_) => Ok(Table::new(entity_id, client, TableOptions {
-                index: Some(on.to_owned()),
+                index: Some(TableIndex::Column(on.to_owned())),
                 limit: None,
                 page_to_disk: None,
                 list_flatten: None,
@@ -760,7 +769,7 @@ impl Client {
         // TODO fix this - name is repeated 2x
         if let Some(info) = infos.into_iter().find(|i| i.entity_id == entity_id) {
             let options = TableOptions {
-                index: info.index,
+                index: TableIndex::from_columns(info.index),
                 limit: info.limit,
                 page_to_disk: None,
                 list_flatten: None,

rust/perspective-client/src/rust/lib.rs

@@ -56,7 +56,7 @@ pub use crate::proto::JoinType;
 pub use crate::session::{ProxySession, Session};
 pub use crate::table::{
     DeleteOptions, DescribeError, DescribeVerdict, Description, ExprValidationResult, JoinOptions,
-    Table, TableInitOptions, TableReadFormat, UpdateOptions,
+    Table, TableIndex, TableInitOptions, TableReadFormat, UpdateOptions,
 };
 pub use crate::table_data::{TableData, UpdateData};
 pub use crate::table_ref::TableRef;
@@ -76,7 +76,7 @@ impl From<&str> for HostedTable {
     fn from(entity_id: &str) -> Self {
         HostedTable {
             entity_id: entity_id.to_string(),
-            index: None,
+            index: Vec::new(),
             limit: None,
         }
     }

rust/perspective-client/src/rust/table.rs

@@ -67,6 +67,66 @@ impl TableReadFormat {
     }
 }
 
+/// The column, or columns in key order, a [`Table`] is keyed by.
+///
+/// When serialized, a single column is a `string` and a composite key is an
+/// array of `string`.
+#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize, TS)]
+#[serde(untagged)]
+pub enum TableIndex {
+    Column(String),
+    Columns(Vec<String>),
+}
+
+impl TableIndex {
+    /// The index columns in key order.
+    pub fn columns(&self) -> &[String] {
+        match self {
+            TableIndex::Column(column) => std::slice::from_ref(column),
+            TableIndex::Columns(columns) => columns,
+        }
+    }
+
+    /// Fold a single-column list into [`TableIndex::Column`] and reject an
+    /// empty one.
+    pub fn normalize(self) -> ClientResult<Self> {
+        match self {
+            TableIndex::Columns(columns) if columns.is_empty() => Err(ClientError::BadTableOptions),
+            TableIndex::Columns(mut columns) if columns.len() == 1 => {
+                Ok(TableIndex::Column(columns.remove(0)))
+            },
+            index => Ok(index),
+        }
+    }
+
+    /// The normalized index for a list of columns, `None` when empty.
+    pub fn from_columns(mut columns: Vec<String>) -> Option<Self> {
+        match columns.len() {
+            0 => None,
+            1 => Some(TableIndex::Column(columns.remove(0))),
+            _ => Some(TableIndex::Columns(columns)),
+        }
+    }
+}
+
+impl From<&str> for TableIndex {
+    fn from(value: &str) -> Self {
+        TableIndex::Column(value.to_string())
+    }
+}
+
+impl From<String> for TableIndex {
+    fn from(value: String) -> Self {
+        TableIndex::Column(value)
+    }
+}
+
+impl From<Vec<String>> for TableIndex {
+    fn from(value: Vec<String>) -> Self {
+        TableIndex::Columns(value)
+    }
+}
+
 /// Options which impact the behavior of [`Client::table`], as well as
 /// subsequent calls to [`Table::update`].
 #[derive(Clone, Debug, Default, Serialize, Deserialize, TS)]
@@ -79,13 +139,14 @@ pub struct TableInitOptions {
     #[ts(optional)]
     pub format: Option<TableReadFormat>,
 
-    /// This [`Table`] should use the column named by the `index` parameter as
-    /// the `index`, which causes [`Table::update`] and [`Client::table`] input
-    /// to either insert or update existing rows based on `index` column
-    /// value equality.
+    /// This [`Table`] should use the column, or columns, named by the `index`
+    /// parameter as the `index`, which causes [`Table::update`] and
+    /// [`Client::table`] input to either insert or update existing rows based
+    /// on `index` column value equality. A list of columns keys rows by the
+    /// combination of their values, in list order.
     #[serde(default)]
     #[ts(optional)]
-    pub index: Option<String>,
+    pub index: Option<TableIndex>,
 
     /// This [`Table`] should be limited to `limit` rows, after which the
     /// _earliest_ rows will be overwritten (where _earliest_ is defined as
@@ -150,8 +211,15 @@ impl TryFrom<TableOptions> for MakeTableOptions {
                     ..
                 } => Err(ClientError::BadTableOptions)?,
                 TableOptions {
-                    index: Some(index), ..
+                    index: Some(TableIndex::Column(index)),
+                    ..
                 } => Some(MakeTableType::MakeIndexTable(index)),
+                TableOptions {
+                    index: Some(TableIndex::Columns(columns)),
+                    ..
+                } => Some(MakeTableType::MakeCompositeIndexTable(IndexColumns {
+                    columns,
+                })),
                 TableOptions {
                     limit: Some(limit), ..
                 } => Some(MakeTableType::MakeLimitTable(limit)),
@@ -163,7 +231,7 @@ impl TryFrom<TableOptions> for MakeTableOptions {
 
 #[derive(Clone, Debug)]
 pub(crate) struct TableOptions {
-    pub index: Option<String>,
+    pub index: Option<TableIndex>,
     pub limit: Option<u32>,
     pub page_to_disk: Option<bool>,
     pub list_flatten: Option<crate::proto::ListFlatten>,
@@ -428,16 +496,16 @@ impl Table {
     /// # async fn run() -> Result<(), Box<dyn std::error::Error>> {
     /// # let client: Client = todo!();
     /// let options = TableInitOptions {
-    ///     index: Some("x".to_string()),
+    ///     index: Some("x".into()),
     ///     ..TableInitOptions::default()
     /// };
     /// let data = TableData::Update(UpdateData::Csv("x,y\n1,2\n3,4".into()));
     /// let table = client.table(data, options).await?;
     /// let index = table.get_index();
     /// # Ok(()) }
     /// ```
-    pub fn get_index(&self) -> Option<String> {
-        self.options.index.as_ref().map(|index| index.to_owned())
+    pub fn get_index(&self) -> Option<TableIndex> {
+        self.options.index.clone()
     }
 
     /// Returns the user-specified row limit for this table.