Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .changeset/eql-v4-bindings.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,6 @@
'@cipherstash/eql': minor
---

**The `eql-bindings` crate has an `eql_bindings::v4` module: payload types for the EQL v4 domains.** EQL v4 is the v3 SQL installed under a second name (ADR-0002), for values written by Stack Encrypt. The module is generated from the same catalog as `v3`, with `eql_v4_*` domain names and `"v": 4`. `SchemaVersion` is now generic over the version it admits: `v3::SchemaVersion` is `SchemaVersion<3>` and `v4::SchemaVersion` is `SchemaVersion<4>`, each refusing the other's payloads, and plain `SchemaVersion` still means v3. Only v4 types derive Stack Encrypt's `EncryptFrom` and `DecryptInto`, and the Stack Encrypt target table (now `eql_bindings::v4::targets`) names v4 domains.
**The `eql-bindings` crate has an `eql_bindings::v4` module: payload types for the EQL v4 domains.** EQL v4 is the v3 SQL installed under a second name (ADR-0002), for values written by Stack Encrypt. The module is generated from the same catalog as `v3`, with `eql_v4_*` domain names and `"v": 4`. `SchemaVersion` is now generic over the version it admits: `v3::SchemaVersion` is `SchemaVersion<3>` and `v4::SchemaVersion` is `SchemaVersion<4>`, each refusing the other's payloads, and plain `SchemaVersion` still means v3. Only v4 types derive Stack Encrypt's `EncryptFrom` and `DecryptInto`, and the Stack Encrypt target table (now `eql_bindings::v4::targets`) names v4 domains. Besides `TextEq`, Stack Encrypt now produces the storage-only types `Text`, `Integer`, `Bigint`, `Boolean`, `Real` and `Double`, from the plaintext types in `eql_bindings::encryption::plaintext`; the Go SDK's `encrypt/eql` table marks them producible. `IntegerEq` and `BigintEq` stay unproducible until vitaminc 0.6 settles the integer equality term.

Nothing a released version shipped has changed. The v3 types, the v3 SQL and the TypeScript package are as they were; the `stack-encrypt` derives on the v3 text types and the `v3::targets` module were added after the last release and are moved to v4 here, not removed from a release.
14 changes: 13 additions & 1 deletion languages/golang/cmd/stashgen/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -149,7 +149,19 @@ The same input always gives the same file: fields keep their declared order, and
`stashgen` checks each declaration with the engine, and holds no copy of the engine's rules: it runs the WASI guest the SDK embeds and asks it, one field at a time, so the error names the field.
It asks the engine for the EQL types it holds: each name, its plaintext type, its indexes, its query form, and whether the engine produces it today.
The command links the build of the engine that holds the EQL types (`encrypt/eql`), so it can answer for every type; a generated file imports `encrypt/eql` only when it names one.
The engine produces `TextEq` today; `encrypt_into` with any other type is refused with the type's name.
The engine produces these EQL types today, each from one Go field type:

| `encrypt_into` | Go field |
|---|---|
| `TextEq`, `Text` | `string` |
| `Integer` | `int32` |
| `Bigint` | `int64` |
| `Boolean` | `bool` |
| `Real` | `float32` |
| `Double` | `float64` |

Comment thread
coderdan marked this conversation as resolved.
The width must match: an `int` or `int64` field is not an `Integer`, and a `float64` field is not a `Real`.
`encrypt_into` with any other type is refused with the type's name and the reason it is not produced yet.
Separate columns work today for four indexes: `equality`, `match`, `ore` and `ope`.

A field with `encrypt_into` is stored under its table and column, which is what an EQL value records in its `i`.
Expand Down
6 changes: 3 additions & 3 deletions languages/golang/encrypt/eql/eql.go
Original file line number Diff line number Diff line change
Expand Up @@ -18,9 +18,9 @@
//
// The types and the [Types] table are generated from the EQL catalog by
// eql-codegen (`mise run types:generate` in packages/eql); eql_gen.go is
// committed and drift-gated. The engine produces [TextEq] today; every
// other type is listed with the reason it cannot be produced yet, and
// stashgen refuses it.
// committed and drift-gated. [Types] says which types the engine produces
// today; every other type is listed with the reason it cannot be produced
// yet, and stashgen refuses it.
package eql

import (
Expand Down
196 changes: 98 additions & 98 deletions languages/golang/encrypt/eql/eql_gen.go

Large diffs are not rendered by default.

17 changes: 14 additions & 3 deletions languages/golang/encrypt/eql/eql_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ import (
"github.com/cipherstash/stack/languages/golang/encrypt/eql"
)

func TestTypesTableNamesTextEqAsTheOneProducibleType(t *testing.T) {
func TestTypesTableNamesTheProducibleTypes(t *testing.T) {
var producible []string
for _, typ := range eql.Types {
if typ.Producible {
Expand All @@ -19,8 +19,19 @@ func TestTypesTableNamesTextEqAsTheOneProducibleType(t *testing.T) {
t.Errorf("%s: an unproducible type has a reason", typ.Name)
}
}
if len(producible) != 1 || producible[0] != "TextEq" {
t.Fatalf("producible = %v, want [TextEq]", producible)
want := []string{"Integer", "Bigint", "Text", "TextEq", "Boolean", "Real", "Double"}
if len(producible) != len(want) {
t.Fatalf("producible = %v, want %v", producible, want)
}
for i := range want {
if producible[i] != want[i] {
t.Fatalf("producible = %v, want %v", producible, want)
}
}
for name, plaintext := range map[string]string{"Integer": "int32", "Bigint": "int64", "Boolean": "bool", "Real": "float32", "Double": "float64"} {
if typ, ok := eql.Lookup(name); !ok || typ.Plaintext != plaintext {
t.Errorf("%s: Plaintext = %q, want %q", name, typ.Plaintext, plaintext)
}
}
textEq, ok := eql.Lookup("TextEq")
if !ok || textEq.GoName != "TextEq" || textEq.Plaintext != "string" || textEq.Query != "TextEqQuery" || textEq.SQLDomain != "public.eql_v4_text_eq" || len(textEq.Indexes) != 1 || textEq.Indexes[0] != "eq" {
Expand Down
2 changes: 2 additions & 0 deletions languages/golang/encrypt/guest/Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

6 changes: 5 additions & 1 deletion languages/golang/encrypt/guest/src/targets.rs
Original file line number Diff line number Diff line change
Expand Up @@ -179,7 +179,11 @@ mod tests {
.filter(|t| t.producible)
.map(|t| t.name.as_str())
.collect();
assert_eq!(producible, ["TextEq"], "the engine produces TextEq only");
assert_eq!(
producible,
["Integer", "Bigint", "Text", "TextEq", "Boolean", "Real", "Double"],
"the engine's producible types, in catalog order"
);
assert!(matches!(
resolver().resolve("TextOrdOre"),
Err(TargetError::Unproducible { .. })
Expand Down
6 changes: 5 additions & 1 deletion languages/golang/encrypt/guest/tests/native_ops.rs
Original file line number Diff line number Diff line change
Expand Up @@ -1366,7 +1366,11 @@ fn targets_lists_what_the_build_holds() {
.filter(|i| producible(i))
.filter_map(name_of)
.collect();
assert_eq!(producible, ["TextEq"], "the engine produces TextEq only");
assert_eq!(
producible,
["Integer", "Bigint", "Text", "TextEq", "Boolean", "Real", "Double"],
"the engine's producible types, in catalog order"
);
}

// =============================================================================
Expand Down
14 changes: 13 additions & 1 deletion languages/golang/stashgen/engine.go
Original file line number Diff line number Diff line change
Expand Up @@ -41,6 +41,11 @@ type EQLType struct {
// Reason says why not when it does not.
Producible bool
Reason string

// wire is the engine's exact plaintext kind (int32, not just an
// integer), when the engine named it. Plaintext is the coarse kind a
// field's Go type is compared with first; wire then holds the width.
wire record.Kind
}

// GuestEngine is the engine the SDK embeds: the WASI guest in package
Expand Down Expand Up @@ -70,7 +75,7 @@ func (e *guestEngine) EQLTypes(ctx context.Context) ([]EQLType, error) {
}
out := make([]EQLType, 0, len(targets))
for _, t := range targets {
eqlType := EQLType{Name: t.Name, Plaintext: kindOfWire(t.Plaintext), Query: t.Query, Producible: t.Producible, Reason: t.Reason}
eqlType := EQLType{Name: t.Name, Plaintext: kindOfWire(t.Plaintext), Query: t.Query, Producible: t.Producible, Reason: t.Reason, wire: t.Plaintext}
for _, term := range t.Indexes {
if name, ok := indexOfOutput[term]; ok {
eqlType.Indexes = append(eqlType.Indexes, name)
Expand Down Expand Up @@ -161,6 +166,13 @@ func lowerDeclaration(d Declaration, eqlTypes []EQLType) (*record.Plan, error) {
if eqlType.Plaintext != KindOther && eqlType.Plaintext != f.GoType.Kind {
return nil, &FieldError{Type: d.Type, Field: f.GoName, Reason: fmt.Sprintf("%s seals a %s, and %s is %s", f.EQLType, eqlType.Plaintext, f.GoType, f.GoType.Kind)}
}
// The width too: Integer seals an int32 and Bigint an int64, Real
// a float32 and Double a float64. A narrower Go integer widens
// to the kind the generated code seals it as (int8 and int16
// seal as int32), so the comparison is on that kind.
if eqlType.wire != record.Untyped && rf.Kind != eqlType.wire {
return nil, &FieldError{Type: d.Type, Field: f.GoName, Reason: fmt.Sprintf("%s seals %s, and %s seals as %s", f.EQLType, eqlType.wire, f.GoType, rf.Kind)}
}
// An EQL value is stored under a table and a column: the struct's
// context is the table and the field's name the column, so a
// context of two or more segments leaves the field no column,
Expand Down
69 changes: 54 additions & 15 deletions languages/golang/stashgen/enginetest/enginetest.go
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
// Package enginetest is a static stand-in for the Rust engine in the
// generator's tests. It knows what the engine produces today: the EQL type
// TextEq, and the four indexes with the Go kinds each applies to.
// generator's tests. It knows what the engine produces today: the EQL types
// in eqlTypes, each with the width of value it seals, and the four indexes
// with the Go kinds each applies to.
//
// It is a test double, not a copy of the rules the SDK ships: the SDK's
// generator asks the embedded guest. When the engine learns a type or an
Expand All @@ -10,6 +11,7 @@ package enginetest
import (
"context"
"fmt"
"strings"

"github.com/cipherstash/stack/languages/golang/stashgen"
)
Expand All @@ -19,16 +21,47 @@ type Static struct{}

var _ stashgen.Engine = Static{}

// EQLTypes returns TextEq, the one EQL type the engine produces today.
// eqlTypes are the EQL types the engine produces today, with the width of
// value each seals ("" when the kind has no width), mirroring the producible
// rows of encrypt/eql's Types.
var eqlTypes = []struct {
stashgen.EQLType
seals string
}{
{stashgen.EQLType{Name: "Integer", Plaintext: stashgen.KindInt, Producible: true}, "int32"},
{stashgen.EQLType{Name: "Bigint", Plaintext: stashgen.KindInt, Producible: true}, "int64"},
{stashgen.EQLType{Name: "Text", Plaintext: stashgen.KindString, Producible: true}, ""},
{stashgen.EQLType{Name: "TextEq", Plaintext: stashgen.KindString, Indexes: []stashgen.IndexName{stashgen.IndexEquality}, Query: "TextEqQuery", Producible: true}, ""},
{stashgen.EQLType{Name: "Boolean", Plaintext: stashgen.KindBool, Producible: true}, ""},
{stashgen.EQLType{Name: "Real", Plaintext: stashgen.KindFloat, Producible: true}, "float32"},
{stashgen.EQLType{Name: "Double", Plaintext: stashgen.KindFloat, Producible: true}, "float64"},
}

// EQLTypes returns the EQL types the engine produces today.
func (Static) EQLTypes(context.Context) ([]stashgen.EQLType, error) {
return []stashgen.EQLType{
{Name: "TextEq", Plaintext: stashgen.KindString, Indexes: []stashgen.IndexName{stashgen.IndexEquality}, Query: "TextEqQuery", Producible: true},
}, nil
out := make([]stashgen.EQLType, len(eqlTypes))
for i, t := range eqlTypes {
out[i] = t.EQLType
}
return out, nil
}

// sealsAs is the width the generated code seals a Go integer or float as:
// int8, int16 and int32 as int32, int and int64 as int64.
func sealsAs(t stashgen.GoType) string {
switch t.Basic {
case "int8", "int16", "int32":
return "int32"
case "int", "int64":
return "int64"
case "float32", "float64":
return t.Basic
}
return ""
}

// Check applies the engine's rules as they stand today.
func (e Static) Check(ctx context.Context, d stashgen.Declaration) error {
eqlTypes, _ := e.EQLTypes(ctx)
for _, f := range d.Fields {
if f.Verb == stashgen.VerbOmit {
continue
Expand All @@ -39,17 +72,23 @@ func (e Static) Check(ctx context.Context, d stashgen.Declaration) error {
}
}
if f.Verb == stashgen.VerbEncryptInto {
var found *stashgen.EQLType
for i := range eqlTypes {
if eqlTypes[i].Name == f.EQLType {
found = &eqlTypes[i]
var names []string
found := -1
for i, t := range eqlTypes {
names = append(names, t.Name)
if t.Name == f.EQLType {
found = i
}
}
if found == nil {
return &stashgen.FieldError{Type: d.Type, Field: f.GoName, Reason: fmt.Sprintf("the engine cannot produce the EQL type %s yet; it produces TextEq", f.EQLType)}
if found < 0 {
return &stashgen.FieldError{Type: d.Type, Field: f.GoName, Reason: fmt.Sprintf("the engine cannot produce the EQL type %s yet; it produces %s", f.EQLType, strings.Join(names, ", "))}
}
t := eqlTypes[found]
if t.Plaintext != f.GoType.Kind {
return &stashgen.FieldError{Type: d.Type, Field: f.GoName, Reason: fmt.Sprintf("%s seals a %s, and %s is %s", f.EQLType, t.Plaintext, f.GoType, f.GoType.Kind)}
}
if found.Plaintext != f.GoType.Kind {
return &stashgen.FieldError{Type: d.Type, Field: f.GoName, Reason: fmt.Sprintf("%s seals a %s, and %s is %s", f.EQLType, found.Plaintext, f.GoType, f.GoType.Kind)}
if t.seals != "" && sealsAs(f.GoType) != t.seals {
return &stashgen.FieldError{Type: d.Type, Field: f.GoName, Reason: fmt.Sprintf("%s seals %s, and %s seals as %s", f.EQLType, t.seals, f.GoType, sealsAs(f.GoType))}
}
}
for _, idx := range f.Indexes {
Expand Down
47 changes: 47 additions & 0 deletions languages/golang/stashgen/lower_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,8 @@ package stashgen
import (
"strings"
"testing"

"github.com/cipherstash/stack/languages/golang/internal/record"
)

// A struct that names an EQL type beside a context_field has no table of
Expand All @@ -28,3 +30,48 @@ func TestLoweringRefusesAnEQLTypeBesideAContextField(t *testing.T) {
t.Fatalf("lowerDeclaration = %+v, %v", plan, err)
}
}

// An EQL type seals one width of its kind. A Go field of the same kind and
// another width is refused here, naming both, rather than by the engine's
// generic refusal; a narrower integer that seals as the type's width passes.
func TestLoweringRefusesAnotherWidthOfTheKind(t *testing.T) {
eql := []EQLType{
{Name: "Integer", Plaintext: KindInt, Producible: true, wire: record.Int32},
{Name: "Bigint", Plaintext: KindInt, Producible: true, wire: record.Int64},
{Name: "Real", Plaintext: KindFloat, Producible: true, wire: record.Float32},
{Name: "Double", Plaintext: KindFloat, Producible: true, wire: record.Float64},
}
field := func(basic string, kind Kind, eqlType string) Declaration {
return Declaration{Type: "Row", Context: "rows", Fields: []Field{
{Name: "v", GoName: "V", GoType: GoType{Name: basic, Kind: kind, Basic: basic}, Verb: VerbEncryptInto, EQLType: eqlType},
}}
}
for _, c := range []struct {
basic, eqlType, want string
kind Kind
}{
{"int", "Integer", "Integer seals int32, and int seals as int64", KindInt},
{"int64", "Integer", "Integer seals int32, and int64 seals as int64", KindInt},
{"int32", "Bigint", "Bigint seals int64, and int32 seals as int32", KindInt},
{"float64", "Real", "Real seals float32, and float64 seals as float64", KindFloat},
{"float32", "Double", "Double seals float64, and float32 seals as float32", KindFloat},
} {
_, err := lowerDeclaration(field(c.basic, c.kind, c.eqlType), eql)
fe, ok := err.(*FieldError)
if !ok || fe.Field != "V" || fe.Reason != c.want {
t.Errorf("%s into %s: lowerDeclaration = %v, want %q", c.basic, c.eqlType, err, c.want)
}
}
for _, c := range []struct {
basic, eqlType string
kind Kind
}{
{"int32", "Integer", KindInt}, {"int16", "Integer", KindInt}, {"int8", "Integer", KindInt},
{"int64", "Bigint", KindInt}, {"int", "Bigint", KindInt},
{"float32", "Real", KindFloat}, {"float64", "Double", KindFloat},
} {
if _, err := lowerDeclaration(field(c.basic, c.kind, c.eqlType), eql); err != nil {
t.Errorf("%s into %s: %v", c.basic, c.eqlType, err)
}
}
}
2 changes: 2 additions & 0 deletions languages/golang/stashgen/refusal_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -80,6 +80,8 @@ func TestRefusals(t *testing.T) {
{"a _ field with no context", user("\t_ struct{}\n\tEmail string `stash:\"email,encrypt\"`"), stashgen.Request{Type: "User"}, "_", "carries the context tag"},
{"an index that does not apply to the field type", user(ctx + "\tAge int32 `stash:\"age,encrypt,index=match\"`"), stashgen.Request{Type: "User"}, "Age", "match applies to a string, not to int32"},
{"an EQL type that does not apply to the field type", user(ctx + "\tAge int64 `stash:\"age,encrypt_into=TextEq\"`"), stashgen.Request{Type: "User"}, "Age", "TextEq seals a string, and int64 is int"},
{"an EQL type of another width", user(ctx + "\tAge int64 `stash:\"age,encrypt_into=Integer\"`"), stashgen.Request{Type: "User"}, "Age", "Integer seals int32, and int64 seals as int64"},
{"a float of another width", user(ctx + "\tScore float64 `stash:\"score,encrypt_into=Real\"`"), stashgen.Request{Type: "User"}, "Score", "Real seals float32, and float64 seals as float64"},
{"a field type the engine cannot seal", user(ctx + "\tDone chan int `stash:\"done,encrypt\"`"), stashgen.Request{Type: "User"}, "Done", "seals only as part of an opaque struct"},
{"a composite sealed outside an opaque struct", user(ctx + "\tAt time.Time `stash:\"at,encrypt\"`"), stashgen.Request{Type: "User"}, "At", "a time.Time seals only as part of an opaque struct"},
{"a slice sealed outside an opaque struct", user(ctx + "\tTags []string `stash:\"tags,encrypt\"`"), stashgen.Request{Type: "User"}, "Tags", "seals only as part of an opaque struct"},
Expand Down
2 changes: 2 additions & 0 deletions packages/eql/Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

7 changes: 7 additions & 0 deletions packages/eql/crates/eql-bindings/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -55,6 +55,11 @@ vitaminc-prf = { version = "0.5.0", optional = true }
# nothing from stack-encrypt's own `dynamic` feature; the same line
# stack-encrypt and the Go guest build against.
vitaminc-aead-value = { version = "0.5.1", optional = true }
# `DecipherVisitor` and `Protected`: the scalar plaintexts in
# `encryption::plaintext` seal and open vitaminc's tagged leaves themselves,
# because vitaminc 0.5 has no native `Encrypt` for the scalar types.
vitaminc-aead = { version = "0.5.1", optional = true }
vitaminc-protected = { version = "0.5.1", optional = true }
# The plaintext a target reads out of its `Protected` runtime value is held
# in `Zeroizing` until it is sealed, so the copy is wiped like the original.
zeroize = { version = "1", optional = true }
Expand All @@ -68,6 +73,8 @@ stack-encrypt = [
"dep:stack-encrypt",
"dep:vitaminc-prf",
"dep:vitaminc-aead-value",
"dep:vitaminc-aead",
"dep:vitaminc-protected",
"dep:zeroize",
"dep:thiserror",
"dep:base64",
Expand Down
Loading
Loading