Redis-Backed Semantic Search (Phase 4) by ajtiwari07 · Pull Request #3663 · Azure/data-api-builder

ajtiwari07 · 2026-06-11T10:46:05Z

Summary
This PR adds end-to-end semantic (vector) search support to DAB, enabling entities to be queried using natural language. Results are retrieved from a Redis vector index via FT.SEARCH KNN and then narrowed in the SQL database to return fully enriched records, with results ordered by semantic distance.

Git Issue: #3332

What's New
Semantic Search Pipeline
RedisSemanticSearchService — new service that takes the user's search text, generates an embedding (or accepts a raw float array), issues a Redis FT.SEARCH KNN query against a configured index, and returns primary-key candidates with cosine distances
ApplySemanticCandidates() in SqlQueryStructure — translates Redis candidates into a SQL WHERE (pk1=… AND pk2=…) OR (…) predicate, so only the semantically relevant rows are fetched from the database
TryApplySemanticNarrowingAsync() in SqlQueryEngine — orchestrates the full flow: call Redis → deduplicate candidates by PK signature → apply SQL predicate → attach distance scores → re-order results by distance (unless user specified explicit ordering)
SemanticDistanceByPrimaryKeySignature map on SqlQueryStructure — carries Redis distances through to the final JSON response, injected as _distance / distance fields for REST and GraphQL respectively

Configuration
New entity.semantic_search config block per entity: enabled, redis-index-name, redis-index-type (hash/json), redis-index-multiplier, similarity-threshold
runtime.cache.level-2.connection-string is reused as the Redis connection for both caching and vector search
runtime.embeddings block: provider, endpoint, api-key, model, api-version, dimensions, timeout-ms
CLI support: --semantic-search.enabled, --semantic-search.redis-index-name, --semantic-search.redis-index-type, --semantic-search.redis-index-multiplier, --semantic-search.similarity-threshold
Schema updated in dab.draft.schema.json

…-api-builder into redis-dab-phase4 # Conflicts: # src/Cli/Commands/AddOptions.cs # src/Cli/Commands/ConfigureOptions.cs # src/Cli/Commands/EntityOptions.cs # src/Cli/Commands/UpdateOptions.cs # src/Cli/ConfigGenerator.cs # src/Core/Services/RequestValidator.cs # src/Service.GraphQLBuilder/Queries/InputTypeBuilder.cs # src/Service.Tests/UnitTests/ConfigValidationUnitTests.cs

Copilot

Pull request overview

This PR introduces entity-level semantic search backed by Redis vector search, wiring it through DAB’s config model, REST/GraphQL surface area, OpenAPI/schema generation, and the SQL query engine so requests can be narrowed by semantic candidates and (optionally) return a semantic distance/similarity field.

Changes:

Add semantic-search configuration (semantic-search block) to the runtime config model, validator, CLI, and JSON schema.
Extend REST + GraphQL query surfaces with semantic search inputs (and block unsupported combinations like ordering by semantic_distance).
Implement Redis-based semantic candidate resolution and integrate semantic narrowing + distance enrichment into the SQL query pipeline.

Reviewed changes

Copilot reviewed 33 out of 33 changed files in this pull request and generated 11 comments.

Show a summary per file

File	Description
src/Service/Startup.cs	Registers semantic search service into DI and enables HttpClient factory usage.
src/Service/Services/SemanticSearch/RedisSemanticSearchService.cs	New Redis FT.SEARCH KNN-based semantic candidate resolver.
src/Service.Tests/UnitTests/SemanticSearchTextFlowTests.cs	Adds tests intended to validate semantic search flow (currently not exercising product code).
src/Service.Tests/UnitTests/RequestValidatorUnitTests.cs	Adds coverage for rejecting `semantic_distance` in insert bodies.
src/Service.Tests/UnitTests/RequestParserUnitTests.cs	Adds coverage for parsing/validating semantic query params and rejecting `semantic_distance` ordering.
src/Service.Tests/UnitTests/ConfigValidationUnitTests.cs	Adds validator coverage for semantic-search configuration requirements.
src/Service.Tests/GraphQLBuilder/QueryBuilderTests.cs	Verifies semantic arguments are added to GraphQL collection queries when enabled.
src/Service.Tests/GraphQLBuilder/MultipleMutationBuilderTests.cs	Updates test wiring for new query engine factory dependencies.
src/Service.GraphQLBuilder/Sql/SchemaConverter.cs	Adds `semanticDistance` field to GraphQL schema for semantic-enabled entities.
src/Service.GraphQLBuilder/Queries/QueryBuilder.cs	Adds semantic query args (`semanticSearch`, `semanticThreshold`) when enabled.
src/Service.GraphQLBuilder/Queries/InputTypeBuilder.cs	Excludes `semanticDistance` from filter/order input generation.
src/Core/Services/SemanticSearch/NoOpSemanticSearchService.cs	Adds default no-op semantic search service implementation.
src/Core/Services/SemanticSearch/ISemanticSearchService.cs	Introduces semantic search service abstraction.
src/Core/Services/RestService.cs	Adds REST semantic request validation + semantic_distance selection behavior.
src/Core/Services/RequestValidator.cs	Blocks setting `semantic_distance` in insert/upsert bodies.
src/Core/Services/OpenAPI/OpenApiDocumentor.cs	Adds semantic query params and response field to OpenAPI when enabled.
src/Core/Resolvers/SqlQueryEngine.cs	Integrates semantic narrowing + semantic distance enrichment and default ordering behavior.
src/Core/Resolvers/Sql Query Structures/SqlQueryStructure.cs	Adds semantic narrowing predicate builder and semantic distance map tracking.
src/Core/Resolvers/Factories/QueryEngineFactory.cs	Plumbs semantic search service into SQL query engine construction.
src/Core/Parsers/RequestParser.cs	Parses semantic query params and enforces some semantic-specific request constraints.
src/Core/Models/SemanticSearchConstants.cs	Adds shared constants for REST/GraphQL semantic names.
src/Core/Models/SemanticSearchCandidate.cs	Defines candidate record used for narrowing and distance mapping.
src/Core/Models/RestRequestContexts/RestRequestContext.cs	Adds semantic inputs/flags to request context.
src/Core/Configurations/RuntimeConfigValidator.cs	Validates semantic-search prerequisites and reserves semantic names.
src/Config/ObjectModel/EntitySemanticSearchOptions.cs	Adds semantic-search entity configuration object model.
src/Config/ObjectModel/Entity.cs	Adds `semantic-search` to entity config model.
src/Cli/Utils.cs	Adds CLI construction/validation of semantic-search options for entities.
src/Cli/ConfigGenerator.cs	Wires semantic-search options into add/update flows; adds runtime cache L2 configure options.
src/Cli/Commands/UpdateOptions.cs	Adds CLI parameters for semantic-search entity options.
src/Cli/Commands/EntityOptions.cs	Adds CLI options for semantic-search entity configuration.
src/Cli/Commands/ConfigureOptions.cs	Adds CLI options for runtime cache level-2 provider/connection-string.
src/Cli/Commands/AddOptions.cs	Adds CLI parameters for semantic-search entity options.
schemas/dab.draft.schema.json	Adds semantic-search entity schema definition and constraints.

ajtiwari07

Fixed failing validation compile issue in tests: removed invalid Dictionary.AsReadOnly() calls in SemanticSearchTextFlowTests. Commit 3cbca26 has been pushed to this PR branch.

ajtiwari07

Update: commit 3cbca26 pushed with CI compile fix in SemanticSearchTextFlowTests (remove invalid Dictionary.AsReadOnly() usage). Re-run should pick up this change.

JerryNixon

keyless views are not rejected at startup. Spec requires startup failure when semantic search is enabled on a view without resolvable primary keys. Current validation checks table/view type but not resolvable PK metadata. Add metadata validation before runtime.
reserved-name validation misses real database columns. Reserved name checks only inspect configured fields/mappings. If the backing table has a real exposed column named semantic_distance and no explicit field config, startup may pass. Validate against resolved metadata/exposed field names.
Embedding/Redis failures should produce useful sanitized errors with correlation/provider details where available. Current embedding failure becomes empty results, Redis failures collapse into a generic BadRequest, and there is no provider status/error propagation.

JerryNixon · 2026-06-12T15:27:22Z

+              "additionalProperties": false,
+              "properties": {
+                "enabled": {
+                  "$ref": "#/$defs/boolean-or-string",


semantic-search.enabled does not honor boolean-or-string. The schema allows enabled as boolean-or-string, but EntitySemanticSearchOptions.Enabled is a bool. @env() / @akv() configs will not round-trip/resolve correctly. Use the existing DAB boolean-or-string pattern and add env/AKV tests.

JerryNixon · 2026-06-12T15:30:06Z

+        return result.Embedding;
+    }
+
+    private static bool TryParseVectorText(string text, out float[]? vector)


client-provided vectors bypass the embedding subsystem. TryParseVectorText() accepts raw JSON float arrays from the caller. The specs say callers provide natural language and DAB embeds it; #3331 explicitly excludes client-provided vectors. Remove raw vector input support unless I am missing something. There is no need for this added complexity as users will not use it like this, almost 99.9999% sure of this.

JerryNixon · 2026-06-12T15:30:48Z

+        return bytes;
+    }
+
+    private async Task<float[]> GetEmbeddingAsync(string semanticSearchValue)


embedding failure silently returns empty results.
GetEmbeddingAsync() returns [] when embedding is disabled/fails, and the caller returns a successful empty response. Spec says embedding failure must fail the request and must not continue as if Redis returned no matches. Throw a sanitized DAB error instead.

JerryNixon · 2026-06-12T15:32:16Z

+        /// Applies a semantic narrowing predicate of the form:
+        /// (col1 = ... AND col2 = ...) OR (...)
+        /// </summary>
+        public void ApplySemanticCandidates(IReadOnlyList<SemanticSearchCandidate> candidates)


SQL narrowing uses all Redis document columns, not primary keys. ApplySemanticCandidates() iterates candidate.ColumnValues, which can include arbitrary non-PK Redis fields. The spec says Redis results provide database key values and DAB builds a key predicate. Narrow only on PrimaryKeyValues. We're wanting simple, intuitive behavior unless there is a technicalreason I am missing here.

JerryNixon · 2026-06-12T15:33:33Z

+    /// <summary>
+    /// True when semantic distance should be included in REST output.
+    /// </summary>
+    public bool IncludeSemanticDistanceInResponse { get; set; }


REST $select behavior violates the spec.
context.IncludeSemanticDistanceInResponse is set but never used by SqlQueryEngine; REST responses add semantic_distance whenever semantic search runs. Spec says $select=id,description must not return semantic_distance unless explicitly selected. It's okay to omit this, if for no other reason, so the user can expect the returning structure. Right?

JerryNixon · 2026-06-12T15:36:20Z

+            return ApplySemanticDistanceAndOrderingIfNeeded(response, structure, dataSourceName, includeRestField: true, includeGraphQlField: false);
+        }
+
+        private async Task<(bool shouldReturnEmpty, JsonDocument? emptyResponse)> TryApplySemanticNarrowingAsync(


GraphQL empty semantic result shape is wrong for connections. TryApplySemanticNarrowingAsync() returns JsonDocument.Parse("[]") before connection shaping. For paginated GraphQL fields the expected shape is a connection with items: [], not a raw array. Return through the normal pagination formatter.

JerryNixon · 2026-06-12T19:01:47Z

+        return similarity;
+    }
+
+    private static string NormalizeFieldName(string field)


Spec requires Redis records to include real database primary key column names exactly, including case. NormalizeFieldName() strips JSON paths and accepts case-insensitive matches, which can map the wrong field. Enforce exact PK names and fail/skip invalid records deterministically.

JerryNixon · 2026-06-12T19:02:32Z

+/// <summary>
+/// Resolves semantic candidates by:
+/// 1) generating/retrieving an embedding vector for semantic_search input,
+/// 2) performing a Redis FT.SEARCH KNN query,


The spec says Redis applies the threshold. This PR runs KNN then filters in DAB. Push threshold into the Redis query or explicitly justify/spec-change the behavior.

JerryNixon · 2026-06-12T19:03:41Z

+        return result.Embedding;
+    }
+
+    private static bool TryParseVectorText(string text, out float[]? vector)


TryParseVectorText() parses caller input as a float array before embedding. Besides violating spec, this creates avoidable CPU/memory pressure. Remove vector parsing.

JerryNixon · 2026-06-12T19:04:46Z

+                continue;
+            }
+
+            results.Add(new SemanticSearchCandidate(primaryKeys, sqlColumns, similarity));


Using Redis document columns beyond PK lets external index contents influence SQL predicates on arbitrary columns. That is brittle and can interact badly with policy/RLS. Treat Redis as candidate-key source only; DAB/database authorization remains authoritative.

ajtiwari07 added 7 commits June 5, 2026 09:59

DAB Phase 4 Implementation

3ccdef9

More features and tests

e7c997c

DAB Phase 4 Implementation

990087d

More features and tests

8c3a221

Fix duplicate semantic search helper after merge

6fb288c

Remove separate embeddings config within semantic search

eede53f

Copilot AI review requested due to automatic review settings June 11, 2026 10:46

ajtiwari07 requested review from Alekhya-Polavarapu, Aniruddh25, JerryNixon, RubenCerna2079, aaronburtle, anushakolan, rusamant, sourabh1007, souvikghosh04, stuartpa and vadeveka as code owners June 11, 2026 10:46

Copilot started reviewing on behalf of ajtiwari07 June 11, 2026 10:46 View session

ajtiwari07 commented Jun 11, 2026

View reviewed changes

Comment thread src/Cli/Commands/ConfigureOptions.cs

Copilot AI reviewed Jun 11, 2026

View reviewed changes

ajtiwari07 changed the title ~~Redis DAB Phase 4~~ Redis-Backed Semantic Search (Phase 4) Jun 11, 2026

Fix semantic search flow tests dictionary readonly usage

3cbca26

ajtiwari07 commented Jun 11, 2026

View reviewed changes

ajtiwari07 added 4 commits June 11, 2026 06:42

Fix test failure

601d837

Fix PR comments Part 1

f352133

Fix linting issues

6afa920

Linting issues part 2

6f2c786

RubenCerna2079 self-assigned this Jun 11, 2026

Fix tests failures

938b0be

JerryNixon reviewed Jun 12, 2026

View reviewed changes

Conversation

ajtiwari07 commented Jun 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ajtiwari07 left a comment

Choose a reason for hiding this comment

Uh oh!

ajtiwari07 left a comment

Choose a reason for hiding this comment

Uh oh!

JerryNixon left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

ajtiwari07 commented Jun 11, 2026 •

edited

Loading