feat(eql-bindings): produce EQL v4 TextMatch from Stack Encrypt match terms

.changeset/eql-v4-bindings.md

@@ -2,6 +2,6 @@
 '@cipherstash/eql': minor
 ---
 
-**The `eql-bindings` crate has an `eql_bindings::v4` module: payload types for the EQL v4 domains.** EQL v4 is the v3 SQL installed under a second name (ADR-0002), for values written by Stack Encrypt. The module is generated from the same catalog as `v3`, with `eql_v4_*` domain names and `"v": 4`. `SchemaVersion` is now generic over the version it admits: `v3::SchemaVersion` is `SchemaVersion<3>` and `v4::SchemaVersion` is `SchemaVersion<4>`, each refusing the other's payloads, and plain `SchemaVersion` still means v3. Only v4 types derive Stack Encrypt's `EncryptFrom` and `DecryptInto`, and the Stack Encrypt target table (now `eql_bindings::v4::targets`) names v4 domains. Besides `TextEq`, Stack Encrypt now produces the storage-only types `Text`, `Integer`, `Bigint`, `Boolean`, `Real` and `Double`, from the plaintext types in `eql_bindings::encryption::plaintext`; the Go SDK's `encrypt/eql` table marks them producible. `IntegerEq` and `BigintEq` stay unproducible until vitaminc 0.6 settles the integer equality term.
+**The `eql-bindings` crate has an `eql_bindings::v4` module: payload types for the EQL v4 domains.** EQL v4 is the v3 SQL installed under a second name (ADR-0002), for values written by Stack Encrypt. The module is generated from the same catalog as `v3`, with `eql_v4_*` domain names and `"v": 4`. `SchemaVersion` is now generic over the version it admits: `v3::SchemaVersion` is `SchemaVersion<3>` and `v4::SchemaVersion` is `SchemaVersion<4>`, each refusing the other's payloads, and plain `SchemaVersion` still means v3. Only v4 types derive Stack Encrypt's `EncryptFrom` and `DecryptInto`, and the Stack Encrypt target table (now `eql_bindings::v4::targets`) names v4 domains. Besides `TextEq`, Stack Encrypt now produces the storage-only types `Text`, `Integer`, `Bigint`, `Boolean`, `Real` and `Double`, and `TextMatch` (3-grams, `k = 6`, a 2048-bit filter, pinned as `eql_bindings::encryption::EqlMatch`; text shorter than three characters is refused, stored or queried, and only `stored @@ query` matches until #1171 is fixed), from the plaintext types in `eql_bindings::encryption::plaintext`; the Go SDK's `encrypt/eql` table marks them producible. `IntegerEq` and `BigintEq` stay unproducible until vitaminc 0.6 settles the integer equality term.
 
 Nothing a released version shipped has changed. The v3 types, the v3 SQL and the TypeScript package are as they were; the `stack-encrypt` derives on the v3 text types and the `v3::targets` module were added after the last release and are moved to v4 here, not removed from a release.

languages/golang/cmd/stashgen/README.md

@@ -153,14 +153,16 @@ The engine produces these EQL types today, each from one Go field type:
 
 | `encrypt_into` | Go field |
 |---|---|
-| `TextEq`, `Text` | `string` |
+| `TextEq`, `Text`, `TextMatch` | `string` |
 | `Integer` | `int32` |
 | `Bigint` | `int64` |
 | `Boolean` | `bool` |
 | `Real` | `float32` |
 | `Double` | `float64` |
 
 The width must match: an `int` or `int64` field is not an `Integer`, and a `float64` field is not a `Real`.
+A `TextMatch` field refuses text shorter than three characters, such as an empty string or a two-letter code, because it has no 3-grams to index; its query refuses such text too.
+Put the column on the left of `@@` (`WHERE name @@ $1::eql_v4.query_text_match`): the other operand order finds nothing until #1171 is fixed.
 `encrypt_into` with any other type is refused with the type's name and the reason it is not produced yet.
 Separate columns work today for four indexes: `equality`, `match`, `ore` and `ope`.
 

languages/golang/cmd/stashgen/main_test.go

@@ -142,15 +142,15 @@ func TestRunAsksTheEmbeddedEngine(t *testing.T) {
 		t.Fatalf("the generated file does not name the EQL type:\n%s", written)
 	}
 	// A type the engine cannot produce yet: refused by name, nothing written.
-	unproducible := strings.Replace(userSource, "encrypt_into=TextEq", "encrypt_into=TextMatch", 1)
+	unproducible := strings.Replace(userSource, "encrypt_into=TextEq", "encrypt_into=TextOrdOre", 1)
 	dir = writeModule(t, map[string]string{"model.go": unproducible})
 	stderr.Reset()
 	if code := run([]string{"-type", "User"}, dir, &stdout, &stderr, stashgen.GuestEngine); code != 1 {
 		t.Fatalf("exit %d, want 1\n%s", code, stderr.String())
 	}
 	// Refused by the generator with the engine's reason, before the engine
 	// is asked: se_targets lists the type as not producible.
-	if !strings.Contains(stderr.String(), "cannot produce the EQL type TextMatch yet") {
+	if !strings.Contains(stderr.String(), "cannot produce the EQL type TextOrdOre yet") {
 		t.Fatalf("stderr = %q", stderr.String())
 	}
 	if _, err := os.Stat(filepath.Join(dir, "user_stash.go")); !os.IsNotExist(err) {

languages/golang/encrypt/eql/eql_test.go

@@ -19,7 +19,7 @@ func TestTypesTableNamesTheProducibleTypes(t *testing.T) {
 			t.Errorf("%s: an unproducible type has a reason", typ.Name)
 		}
 	}
-	want := []string{"Integer", "Bigint", "Text", "TextEq", "Boolean", "Real", "Double"}
+	want := []string{"Integer", "Bigint", "Text", "TextEq", "TextMatch", "Boolean", "Real", "Double"}
 	if len(producible) != len(want) {
 		t.Fatalf("producible = %v, want %v", producible, want)
 	}

languages/golang/encrypt/guest/src/targets.rs

@@ -181,7 +181,16 @@ mod tests {
             .collect();
         assert_eq!(
             producible,
-            ["Integer", "Bigint", "Text", "TextEq", "Boolean", "Real", "Double"],
+            [
+                "Integer",
+                "Bigint",
+                "Text",
+                "TextEq",
+                "TextMatch",
+                "Boolean",
+                "Real",
+                "Double"
+            ],
             "the engine's producible types, in catalog order"
         );
         assert!(matches!(

languages/golang/encrypt/guest/tests/native_ops.rs

@@ -1368,7 +1368,16 @@ fn targets_lists_what_the_build_holds() {
         .collect();
     assert_eq!(
         producible,
-        ["Integer", "Bigint", "Text", "TextEq", "Boolean", "Real", "Double"],
+        [
+            "Integer",
+            "Bigint",
+            "Text",
+            "TextEq",
+            "TextMatch",
+            "Boolean",
+            "Real",
+            "Double"
+        ],
         "the engine's producible types, in catalog order"
     );
 }

languages/golang/stashgen/enginetest/enginetest.go

@@ -32,6 +32,7 @@ var eqlTypes = []struct {
 	{stashgen.EQLType{Name: "Bigint", Plaintext: stashgen.KindInt, Producible: true}, "int64"},
 	{stashgen.EQLType{Name: "Text", Plaintext: stashgen.KindString, Producible: true}, ""},
 	{stashgen.EQLType{Name: "TextEq", Plaintext: stashgen.KindString, Indexes: []stashgen.IndexName{stashgen.IndexEquality}, Query: "TextEqQuery", Producible: true}, ""},
+	{stashgen.EQLType{Name: "TextMatch", Plaintext: stashgen.KindString, Indexes: []stashgen.IndexName{stashgen.IndexMatch}, Query: "TextMatchQuery", Producible: true}, ""},
 	{stashgen.EQLType{Name: "Boolean", Plaintext: stashgen.KindBool, Producible: true}, ""},
 	{stashgen.EQLType{Name: "Real", Plaintext: stashgen.KindFloat, Producible: true}, "float32"},
 	{stashgen.EQLType{Name: "Double", Plaintext: stashgen.KindFloat, Producible: true}, "float64"},

languages/golang/stashgen/refusal_test.go

@@ -91,7 +91,7 @@ func TestRefusals(t *testing.T) {
 		{"an opaque struct with an interface", user("\t_ struct{} `stash:\"context=users,opaque\"`\n\tAny any"), stashgen.Request{Type: "User"}, "Any", "is not a type JSON carries"},
 		{"an opaque struct with a struct JSON would truncate", user("\t_ struct{} `stash:\"context=users,opaque\"`\n\tTx hidden") + "\ntype hidden struct {\n\tA int\n\tb int\n}\n", stashgen.Request{Type: "User"}, "Tx", "JSON would drop"},
 		{"an opaque struct with a map keyed by a struct", user("\t_ struct{} `stash:\"context=users,opaque\"`\n\tM map[hidden]int") + "\ntype hidden struct{ A int }\n", stashgen.Request{Type: "User"}, "M", "map key that is not a string or an integer"},
-		{"an EQL type the engine cannot produce yet", user(ctx + "\tEmail string `stash:\"email,encrypt_into=TextMatch\"`"), stashgen.Request{Type: "User"}, "Email", "has no EQL type TextMatch"},
+		{"an EQL type the engine cannot produce yet", user(ctx + "\tAge int32 `stash:\"age,encrypt_into=IntegerEq\"`"), stashgen.Request{Type: "User"}, "Age", "has no EQL type IntegerEq"},
 		{"an index on a composite", user(ctx + "\tAttrs map[string]string `stash:\"attrs,encrypt,index=equality\"`"), stashgen.Request{Type: "User"}, "Attrs", "seals only as part of an opaque struct"},
 		{"the json index, not in the engine yet", user(ctx + "\tAttrs string `stash:\"attrs,index=json\"`"), stashgen.Request{Type: "User"}, "Attrs", "cannot derive the json index yet"},
 		{"an index option a query term cannot use", user(ctx + "\tEmail string `stash:\"email,encrypt,index=match(k=3)\"`"), stashgen.Request{Type: "User"}, "Email", "a query term uses only the default index options"},

languages/golang/stashgen/testdata/cases/users/model.go

@@ -8,6 +8,6 @@ type User struct {
 	_        struct{} `stash:"context=users"`
 	ID       int64    `stash:"id,passthrough" db:"id" gorm:"primaryKey"`
 	Email    string   `stash:"email,encrypt_into=TextEq" db:"email"`
-	Name     string   `stash:"name,encrypt_into=TextEq" db:"name"`
+	Name     string   `stash:"name,encrypt_into=TextMatch" db:"name"`
 	Internal string   `stash:"-"`
 }

languages/golang/stashgen/testdata/stubsdk/encrypt/eql/eql.go

@@ -1,9 +1,15 @@
-// Package eql is a TEST STUB of the generated EQL types: the one type the
-// engine produces today, and its query type.
+// Package eql is a TEST STUB of the generated EQL types: the types the
+// golden cases name, and their query types.
 package eql
 
 // TextEq is an EQL value with an equality term.
 type TextEq []byte
 
 // TextEqQuery is the query value for TextEq.
 type TextEqQuery []byte
+
+// TextMatch is an EQL value with a match term.
+type TextMatch []byte
+
+// TextMatchQuery is the query value for TextMatch.
+type TextMatchQuery []byte

packages/eql/crates/eql-bindings/README.md

@@ -112,7 +112,7 @@ encryption for the scalar types:
 
 | v4 type | Plaintext |
 | --- | --- |
-| `Text` | `String` |
+| `Text`, `TextMatch` | `String` |
 | `Integer` | `Int32` |
 | `Bigint` | `Int64` |
 | `Boolean` | `Bool` |
@@ -124,7 +124,17 @@ seals for the same value, so values written here and by the Go SDK open each
 other. `IntegerEq` and `BigintEq` wait for vitaminc 0.6, whose integer
 equality term uses one PRF domain for every width; terms derived under
 vitaminc 0.5's per-width domains would stop matching after that upgrade.
-Ordering types wait for vitaminc 0.6's order encodings, float
+`TextMatch` builds its filter under a configuration pinned in this crate
+(`EqlMatch`: 3-grams of the lower-cased text, `k = 6`, a 2048-bit filter) for
+stored values and queries alike. The configuration is part of the stored
+format, so changing it means a new domain or payload version. A filter can
+match a row that does not contain the query, so re-check after decrypting
+where that matters. Write the stored column on the left, `col @@ query`: the
+other order finds nothing until #1171 is fixed. Text too short to make a
+3-gram (empty strings, two-letter codes) cannot be stored in a `TextMatch`
+column yet, and a `TextMatchQuery` for it is refused too: Stack Encrypt 0.2
+refuses text with no tokens, on both sides. Ordering types
+wait for vitaminc 0.6's order encodings, float
 equality for its canonical float form, and `Json` for a decision on sealing a
 document as one leaf.