diff --git a/.changeset/eql-v4-bindings.md b/.changeset/eql-v4-bindings.md index 546a460d7..1ccfcdd3a 100644 --- a/.changeset/eql-v4-bindings.md +++ b/.changeset/eql-v4-bindings.md @@ -2,6 +2,6 @@ '@cipherstash/eql': minor --- -**The `eql-bindings` crate has an `eql_bindings::v4` module: payload types for the EQL v4 domains.** EQL v4 is the v3 SQL installed under a second name (ADR-0002), for values written by Stack Encrypt. The module is generated from the same catalog as `v3`, with `eql_v4_*` domain names and `"v": 4`. `SchemaVersion` is now generic over the version it admits: `v3::SchemaVersion` is `SchemaVersion<3>` and `v4::SchemaVersion` is `SchemaVersion<4>`, each refusing the other's payloads, and plain `SchemaVersion` still means v3. Only v4 types derive Stack Encrypt's `EncryptFrom` and `DecryptInto`, and the Stack Encrypt target table (now `eql_bindings::v4::targets`) names v4 domains. Besides `TextEq`, Stack Encrypt now produces the storage-only types `Text`, `Integer`, `Bigint`, `Boolean`, `Real` and `Double`, from the plaintext types in `eql_bindings::encryption::plaintext`; the Go SDK's `encrypt/eql` table marks them producible. `IntegerEq` and `BigintEq` stay unproducible until vitaminc 0.6 settles the integer equality term. +**The `eql-bindings` crate has an `eql_bindings::v4` module: payload types for the EQL v4 domains.** EQL v4 is the v3 SQL installed under a second name (ADR-0002), for values written by Stack Encrypt. The module is generated from the same catalog as `v3`, with `eql_v4_*` domain names and `"v": 4`. `SchemaVersion` is now generic over the version it admits: `v3::SchemaVersion` is `SchemaVersion<3>` and `v4::SchemaVersion` is `SchemaVersion<4>`, each refusing the other's payloads, and plain `SchemaVersion` still means v3. Only v4 types derive Stack Encrypt's `EncryptFrom` and `DecryptInto`, and the Stack Encrypt target table (now `eql_bindings::v4::targets`) names v4 domains. Besides `TextEq`, Stack Encrypt now produces the storage-only types `Text`, `Integer`, `Bigint`, `Boolean`, `Real` and `Double`, and `TextMatch` (3-grams, `k = 6`, a 2048-bit filter, pinned as `eql_bindings::encryption::EqlMatch`; text shorter than three characters is refused, stored or queried, and only `stored @@ query` matches until #1171 is fixed), from the plaintext types in `eql_bindings::encryption::plaintext`; the Go SDK's `encrypt/eql` table marks them producible. `IntegerEq` and `BigintEq` stay unproducible until vitaminc 0.6 settles the integer equality term. Nothing a released version shipped has changed. The v3 types, the v3 SQL and the TypeScript package are as they were; the `stack-encrypt` derives on the v3 text types and the `v3::targets` module were added after the last release and are moved to v4 here, not removed from a release. diff --git a/languages/golang/cmd/stashgen/README.md b/languages/golang/cmd/stashgen/README.md index 227e1c855..054659d1f 100644 --- a/languages/golang/cmd/stashgen/README.md +++ b/languages/golang/cmd/stashgen/README.md @@ -153,7 +153,7 @@ The engine produces these EQL types today, each from one Go field type: | `encrypt_into` | Go field | |---|---| -| `TextEq`, `Text` | `string` | +| `TextEq`, `Text`, `TextMatch` | `string` | | `Integer` | `int32` | | `Bigint` | `int64` | | `Boolean` | `bool` | @@ -161,6 +161,8 @@ The engine produces these EQL types today, each from one Go field type: | `Double` | `float64` | The width must match: an `int` or `int64` field is not an `Integer`, and a `float64` field is not a `Real`. +A `TextMatch` field refuses text shorter than three characters, such as an empty string or a two-letter code, because it has no 3-grams to index; its query refuses such text too. +Put the column on the left of `@@` (`WHERE name @@ $1::eql_v4.query_text_match`): the other operand order finds nothing until #1171 is fixed. `encrypt_into` with any other type is refused with the type's name and the reason it is not produced yet. Separate columns work today for four indexes: `equality`, `match`, `ore` and `ope`. diff --git a/languages/golang/cmd/stashgen/main_test.go b/languages/golang/cmd/stashgen/main_test.go index b832fcb4f..a9558ab1a 100644 --- a/languages/golang/cmd/stashgen/main_test.go +++ b/languages/golang/cmd/stashgen/main_test.go @@ -142,7 +142,7 @@ func TestRunAsksTheEmbeddedEngine(t *testing.T) { t.Fatalf("the generated file does not name the EQL type:\n%s", written) } // A type the engine cannot produce yet: refused by name, nothing written. - unproducible := strings.Replace(userSource, "encrypt_into=TextEq", "encrypt_into=TextMatch", 1) + unproducible := strings.Replace(userSource, "encrypt_into=TextEq", "encrypt_into=TextOrdOre", 1) dir = writeModule(t, map[string]string{"model.go": unproducible}) stderr.Reset() if code := run([]string{"-type", "User"}, dir, &stdout, &stderr, stashgen.GuestEngine); code != 1 { @@ -150,7 +150,7 @@ func TestRunAsksTheEmbeddedEngine(t *testing.T) { } // Refused by the generator with the engine's reason, before the engine // is asked: se_targets lists the type as not producible. - if !strings.Contains(stderr.String(), "cannot produce the EQL type TextMatch yet") { + if !strings.Contains(stderr.String(), "cannot produce the EQL type TextOrdOre yet") { t.Fatalf("stderr = %q", stderr.String()) } if _, err := os.Stat(filepath.Join(dir, "user_stash.go")); !os.IsNotExist(err) { diff --git a/languages/golang/encrypt/eql/eql_gen.go b/languages/golang/encrypt/eql/eql_gen.go index d7b185e15..77eee6fe3 100644 --- a/languages/golang/encrypt/eql/eql_gen.go +++ b/languages/golang/encrypt/eql/eql_gen.go @@ -41,7 +41,7 @@ var Types = []Type{ {Name: "NumericOrdOpe", GoName: "NumericOrdOpe", Family: "numeric", Suffix: "OrdOpe", Plaintext: "", SQLDomain: "public.eql_v4_numeric_ord_ope", Indexes: []string{"ope"}, Query: "NumericOrdOpeQuery", QuerySQLDomain: "eql_v4.query_numeric_ord_ope", Producible: false, Reason: "vitaminc 0.5's value enum has no kind for this family; the kind and its encodings arrive in vitaminc 0.6 (ADR-0002)"}, {Name: "Text", GoName: "Text", Family: "text", Suffix: "", Plaintext: "string", SQLDomain: "public.eql_v4_text", Indexes: nil, Query: "", QuerySQLDomain: "", Producible: true, Reason: ""}, {Name: "TextEq", GoName: "TextEq", Family: "text", Suffix: "Eq", Plaintext: "string", SQLDomain: "public.eql_v4_text_eq", Indexes: []string{"eq"}, Query: "TextEqQuery", QuerySQLDomain: "eql_v4.query_text_eq", Producible: true, Reason: ""}, - {Name: "TextMatch", GoName: "TextMatch", Family: "text", Suffix: "Match", Plaintext: "string", SQLDomain: "public.eql_v4_text_match", Indexes: []string{"match"}, Query: "TextMatchQuery", QuerySQLDomain: "eql_v4.query_text_match", Producible: false, Reason: "the engine derives match terms, but no EQL type is built from them yet"}, + {Name: "TextMatch", GoName: "TextMatch", Family: "text", Suffix: "Match", Plaintext: "string", SQLDomain: "public.eql_v4_text_match", Indexes: []string{"match"}, Query: "TextMatchQuery", QuerySQLDomain: "eql_v4.query_text_match", Producible: true, Reason: ""}, {Name: "TextOrdOre", GoName: "TextOrdOre", Family: "text", Suffix: "OrdOre", Plaintext: "string", SQLDomain: "public.eql_v4_text_ord_ore", Indexes: []string{"eq", "ore"}, Query: "TextOrdOreQuery", QuerySQLDomain: "eql_v4.query_text_ord_ore", Producible: false, Reason: "ordering terms wait for vitaminc-ore, the order encodings in vitaminc 0.6 (ADR-0002)"}, {Name: "TextOrd", GoName: "TextOrd", Family: "text", Suffix: "Ord", Plaintext: "string", SQLDomain: "public.eql_v4_text_ord", Indexes: []string{"eq", "ope"}, Query: "TextOrdQuery", QuerySQLDomain: "eql_v4.query_text_ord", Producible: false, Reason: "ordering terms wait for vitaminc-ore, the order encodings in vitaminc 0.6 (ADR-0002)"}, {Name: "TextOrdOpe", GoName: "TextOrdOpe", Family: "text", Suffix: "OrdOpe", Plaintext: "string", SQLDomain: "public.eql_v4_text_ord_ope", Indexes: []string{"eq", "ope"}, Query: "TextOrdOpeQuery", QuerySQLDomain: "eql_v4.query_text_ord_ope", Producible: false, Reason: "ordering terms wait for vitaminc-ore, the order encodings in vitaminc 0.6 (ADR-0002)"}, @@ -1406,7 +1406,7 @@ func (v *TextEqQuery) UnmarshalJSON(b []byte) error { } // TextMatch is the EQL type stored in public.eql_v4_text_match: the text family with the match index. -// The engine cannot produce it yet: the engine derives match terms, but no EQL type is built from them yet. +// The engine produces it from a string. type TextMatch []byte // Value is the EQL value as the driver takes it: the JSON text, or NULL when empty. diff --git a/languages/golang/encrypt/eql/eql_test.go b/languages/golang/encrypt/eql/eql_test.go index 92ac9ea5a..efc4c4319 100644 --- a/languages/golang/encrypt/eql/eql_test.go +++ b/languages/golang/encrypt/eql/eql_test.go @@ -19,7 +19,7 @@ func TestTypesTableNamesTheProducibleTypes(t *testing.T) { t.Errorf("%s: an unproducible type has a reason", typ.Name) } } - want := []string{"Integer", "Bigint", "Text", "TextEq", "Boolean", "Real", "Double"} + want := []string{"Integer", "Bigint", "Text", "TextEq", "TextMatch", "Boolean", "Real", "Double"} if len(producible) != len(want) { t.Fatalf("producible = %v, want %v", producible, want) } diff --git a/languages/golang/encrypt/guest/src/targets.rs b/languages/golang/encrypt/guest/src/targets.rs index 83df35d40..3451b9176 100644 --- a/languages/golang/encrypt/guest/src/targets.rs +++ b/languages/golang/encrypt/guest/src/targets.rs @@ -181,7 +181,16 @@ mod tests { .collect(); assert_eq!( producible, - ["Integer", "Bigint", "Text", "TextEq", "Boolean", "Real", "Double"], + [ + "Integer", + "Bigint", + "Text", + "TextEq", + "TextMatch", + "Boolean", + "Real", + "Double" + ], "the engine's producible types, in catalog order" ); assert!(matches!( diff --git a/languages/golang/encrypt/guest/tests/native_ops.rs b/languages/golang/encrypt/guest/tests/native_ops.rs index 53e0a42f4..0e653f452 100644 --- a/languages/golang/encrypt/guest/tests/native_ops.rs +++ b/languages/golang/encrypt/guest/tests/native_ops.rs @@ -1368,7 +1368,16 @@ fn targets_lists_what_the_build_holds() { .collect(); assert_eq!( producible, - ["Integer", "Bigint", "Text", "TextEq", "Boolean", "Real", "Double"], + [ + "Integer", + "Bigint", + "Text", + "TextEq", + "TextMatch", + "Boolean", + "Real", + "Double" + ], "the engine's producible types, in catalog order" ); } diff --git a/languages/golang/stashgen/enginetest/enginetest.go b/languages/golang/stashgen/enginetest/enginetest.go index 31e8b8419..bae3a6b38 100644 --- a/languages/golang/stashgen/enginetest/enginetest.go +++ b/languages/golang/stashgen/enginetest/enginetest.go @@ -32,6 +32,7 @@ var eqlTypes = []struct { {stashgen.EQLType{Name: "Bigint", Plaintext: stashgen.KindInt, Producible: true}, "int64"}, {stashgen.EQLType{Name: "Text", Plaintext: stashgen.KindString, Producible: true}, ""}, {stashgen.EQLType{Name: "TextEq", Plaintext: stashgen.KindString, Indexes: []stashgen.IndexName{stashgen.IndexEquality}, Query: "TextEqQuery", Producible: true}, ""}, + {stashgen.EQLType{Name: "TextMatch", Plaintext: stashgen.KindString, Indexes: []stashgen.IndexName{stashgen.IndexMatch}, Query: "TextMatchQuery", Producible: true}, ""}, {stashgen.EQLType{Name: "Boolean", Plaintext: stashgen.KindBool, Producible: true}, ""}, {stashgen.EQLType{Name: "Real", Plaintext: stashgen.KindFloat, Producible: true}, "float32"}, {stashgen.EQLType{Name: "Double", Plaintext: stashgen.KindFloat, Producible: true}, "float64"}, diff --git a/languages/golang/stashgen/refusal_test.go b/languages/golang/stashgen/refusal_test.go index 2ab440c56..4e44f7ebb 100644 --- a/languages/golang/stashgen/refusal_test.go +++ b/languages/golang/stashgen/refusal_test.go @@ -91,7 +91,7 @@ func TestRefusals(t *testing.T) { {"an opaque struct with an interface", user("\t_ struct{} `stash:\"context=users,opaque\"`\n\tAny any"), stashgen.Request{Type: "User"}, "Any", "is not a type JSON carries"}, {"an opaque struct with a struct JSON would truncate", user("\t_ struct{} `stash:\"context=users,opaque\"`\n\tTx hidden") + "\ntype hidden struct {\n\tA int\n\tb int\n}\n", stashgen.Request{Type: "User"}, "Tx", "JSON would drop"}, {"an opaque struct with a map keyed by a struct", user("\t_ struct{} `stash:\"context=users,opaque\"`\n\tM map[hidden]int") + "\ntype hidden struct{ A int }\n", stashgen.Request{Type: "User"}, "M", "map key that is not a string or an integer"}, - {"an EQL type the engine cannot produce yet", user(ctx + "\tEmail string `stash:\"email,encrypt_into=TextMatch\"`"), stashgen.Request{Type: "User"}, "Email", "has no EQL type TextMatch"}, + {"an EQL type the engine cannot produce yet", user(ctx + "\tAge int32 `stash:\"age,encrypt_into=IntegerEq\"`"), stashgen.Request{Type: "User"}, "Age", "has no EQL type IntegerEq"}, {"an index on a composite", user(ctx + "\tAttrs map[string]string `stash:\"attrs,encrypt,index=equality\"`"), stashgen.Request{Type: "User"}, "Attrs", "seals only as part of an opaque struct"}, {"the json index, not in the engine yet", user(ctx + "\tAttrs string `stash:\"attrs,index=json\"`"), stashgen.Request{Type: "User"}, "Attrs", "cannot derive the json index yet"}, {"an index option a query term cannot use", user(ctx + "\tEmail string `stash:\"email,encrypt,index=match(k=3)\"`"), stashgen.Request{Type: "User"}, "Email", "a query term uses only the default index options"}, diff --git a/languages/golang/stashgen/testdata/cases/users/model.go b/languages/golang/stashgen/testdata/cases/users/model.go index 2c1d23961..8fd94e5eb 100644 --- a/languages/golang/stashgen/testdata/cases/users/model.go +++ b/languages/golang/stashgen/testdata/cases/users/model.go @@ -8,6 +8,6 @@ type User struct { _ struct{} `stash:"context=users"` ID int64 `stash:"id,passthrough" db:"id" gorm:"primaryKey"` Email string `stash:"email,encrypt_into=TextEq" db:"email"` - Name string `stash:"name,encrypt_into=TextEq" db:"name"` + Name string `stash:"name,encrypt_into=TextMatch" db:"name"` Internal string `stash:"-"` } diff --git a/languages/golang/stashgen/testdata/cases/users/user_stash.go.golden b/languages/golang/stashgen/testdata/cases/users/user_stash.go.golden index 9a82606d3..a03ae9544 100644 --- a/languages/golang/stashgen/testdata/cases/users/user_stash.go.golden +++ b/languages/golang/stashgen/testdata/cases/users/user_stash.go.golden @@ -18,9 +18,9 @@ const _ = gensupport.GeneratedVersion1 // method. Write them, or run stashgen with -redact. type EncryptedUser struct { - ID int64 `db:"id" gorm:"primaryKey"` - Email eql.TextEq `db:"email"` - Name eql.TextEq `db:"name"` + ID int64 `db:"id" gorm:"primaryKey"` + Email eql.TextEq `db:"email"` + Name eql.TextMatch `db:"name"` } func (e EncryptedUser) String() string { @@ -45,7 +45,7 @@ type userShape struct { var declaration = gensupport.Declare("users"). Passthrough("id"). EncryptInto("email", gensupport.String, "TextEq"). - EncryptInto("name", gensupport.String, "TextEq"). + EncryptInto("name", gensupport.String, "TextMatch"). Omit("internal") var codec = gensupport.New(gensupport.Generated[User, EncryptedUser]{ @@ -66,7 +66,7 @@ var codec = gensupport.New(gensupport.Generated[User, EncryptedUser]{ return EncryptedUser{}, err } e.Email = eql.TextEq(rec["email"].EQL) - e.Name = eql.TextEq(rec["name"].EQL) + e.Name = eql.TextMatch(rec["name"].EQL) return e, nil }, Open: func(e EncryptedUser) gensupport.Record { @@ -131,12 +131,12 @@ type NameField struct { field gensupport.Field[string] } -func (f NameField) Encrypt(ctx context.Context, c *encrypt.Cipher, v string) (eql.TextEq, error) { +func (f NameField) Encrypt(ctx context.Context, c *encrypt.Cipher, v string) (eql.TextMatch, error) { out, err := f.field.Encrypt(ctx, c, v) - return eql.TextEq(out.EQL), err + return eql.TextMatch(out.EQL), err } -func (f NameField) Query(ctx context.Context, c *encrypt.Cipher, v string) (eql.TextEqQuery, error) { +func (f NameField) Query(ctx context.Context, c *encrypt.Cipher, v string) (eql.TextMatchQuery, error) { out, err := f.field.Query(ctx, c, v) - return eql.TextEqQuery(out.EQL), err + return eql.TextMatchQuery(out.EQL), err } diff --git a/languages/golang/stashgen/testdata/stubsdk/encrypt/eql/eql.go b/languages/golang/stashgen/testdata/stubsdk/encrypt/eql/eql.go index 09f4934a7..17fac8ef2 100644 --- a/languages/golang/stashgen/testdata/stubsdk/encrypt/eql/eql.go +++ b/languages/golang/stashgen/testdata/stubsdk/encrypt/eql/eql.go @@ -1,5 +1,5 @@ -// Package eql is a TEST STUB of the generated EQL types: the one type the -// engine produces today, and its query type. +// Package eql is a TEST STUB of the generated EQL types: the types the +// golden cases name, and their query types. package eql // TextEq is an EQL value with an equality term. @@ -7,3 +7,9 @@ type TextEq []byte // TextEqQuery is the query value for TextEq. type TextEqQuery []byte + +// TextMatch is an EQL value with a match term. +type TextMatch []byte + +// TextMatchQuery is the query value for TextMatch. +type TextMatchQuery []byte diff --git a/packages/eql/crates/eql-bindings/README.md b/packages/eql/crates/eql-bindings/README.md index 86c2a9a8c..5a16a25fb 100644 --- a/packages/eql/crates/eql-bindings/README.md +++ b/packages/eql/crates/eql-bindings/README.md @@ -112,7 +112,7 @@ encryption for the scalar types: | v4 type | Plaintext | | --- | --- | -| `Text` | `String` | +| `Text`, `TextMatch` | `String` | | `Integer` | `Int32` | | `Bigint` | `Int64` | | `Boolean` | `Bool` | @@ -124,7 +124,17 @@ seals for the same value, so values written here and by the Go SDK open each other. `IntegerEq` and `BigintEq` wait for vitaminc 0.6, whose integer equality term uses one PRF domain for every width; terms derived under vitaminc 0.5's per-width domains would stop matching after that upgrade. -Ordering types wait for vitaminc 0.6's order encodings, float +`TextMatch` builds its filter under a configuration pinned in this crate +(`EqlMatch`: 3-grams of the lower-cased text, `k = 6`, a 2048-bit filter) for +stored values and queries alike. The configuration is part of the stored +format, so changing it means a new domain or payload version. A filter can +match a row that does not contain the query, so re-check after decrypting +where that matters. Write the stored column on the left, `col @@ query`: the +other order finds nothing until #1171 is fixed. Text too short to make a +3-gram (empty strings, two-letter codes) cannot be stored in a `TextMatch` +column yet, and a `TextMatchQuery` for it is refused too: Stack Encrypt 0.2 +refuses text with no tokens, on both sides. Ordering types +wait for vitaminc 0.6's order encodings, float equality for its canonical float form, and `Json` for a decision on sealing a document as one leaf. diff --git a/packages/eql/crates/eql-bindings/src/encryption.rs b/packages/eql/crates/eql-bindings/src/encryption.rs index 0134178a3..aeae19653 100644 --- a/packages/eql/crates/eql-bindings/src/encryption.rs +++ b/packages/eql/crates/eql-bindings/src/encryption.rs @@ -82,7 +82,9 @@ pub mod plaintext; pub mod targets; use base64::{engine::general_purpose::STANDARD, Engine as _}; -use stack_encrypt::sem::EqualityTerm; +#[allow(deprecated)] +use stack_encrypt::sem::MatchTerm; +use stack_encrypt::sem::{EqualityTerm, MatchConfig, MatchOptions, Tokenizer}; use stack_encrypt::target::transcode::{Reader, Transcode, Visitor}; use stack_encrypt::target::{self, AeadContext, CallerContext}; use stack_encrypt::{ @@ -93,7 +95,7 @@ use stack_encrypt::{ use vitaminc_prf::PrfValue; use crate::{ - v3::terms::{Ciphertext, Hmac256}, + v3::terms::{BloomFilter, Ciphertext, Hmac256}, Identifier, }; @@ -252,12 +254,115 @@ impl
DecryptField
for Hmac256 {
}
}
+/// The match configuration of EQL v4's `TextMatch`: 3-grams of the
+/// lower-cased text, `k = 6` positions per token in a 2048-bit filter.
+///
+/// Pinned here rather than taken from Stack Encrypt's default, because it is
+/// part of the stored format: a stored filter and a query filter agree only
+/// when both were built under the same values, and nothing in the payload
+/// records them. Changing any of them is a new domain or payload version,
+/// never an edit in place. Every position is below 2048, so it fits a
+/// `smallint`.
+///
+/// A filter is a set of bits, so `@@` can return rows that do not contain the
+/// query (false positives) and never misses one that does. 2048 bits keeps
+/// that rate low for text of a few hundred characters; re-check matches
+/// after decrypting where it matters.
+#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash, Default)]
+pub struct EqlMatch;
+impl MatchConfig for EqlMatch {
+ fn options() -> MatchOptions {
+ MatchOptions {
+ tokenizer: Tokenizer::Ngram { length: 3 },
+ downcase: true,
+ k: 6,
+ m: 2048,
+ }
+ }
+}
+
+/// Builds EQL's bloom filter, the set bit positions as `smallint`s, from the
+/// native match terms built under [`EqlMatch`]. Stored values and queries
+/// take it from the same type, so they always agree; matching
+/// cipherstash-client's parameters is not a goal, because cipherstash-client
+/// never writes an `eql_v4` column.
+#[doc(hidden)]
+pub struct MatchVisitor;
+impl Visitor for MatchVisitor {
+ type Value = BloomFilter;
+ // `MatchTerm` is the type's name in the crates.io stack-encrypt this
+ // crate publishes against, and a deprecated alias of `MatchTerms` in the
+ // tree: the one spelling that compiles against both.
+ #[allow(deprecated)]
+ fn matching DecryptField for BloomFilter {
+ fn decryption_field for BloomFilter {
+ type Context = CallerContext;
+ fn encryption<'s, K: 'static>() -> Encryption<'s, S, Self, K, Self::Context>
+ where
+ S: 's,
+ {
+ // The type names the configuration and lets inference fill the rest:
+ // `matching` takes three type parameters in the crates.io
+ // stack-encrypt this crate publishes against and four in the tree,
+ // so a turbofish compiles against only one of them.
+ #[allow(deprecated)]
+ let terms: Encryption<'s, S, MatchTerm