feat(datadog-ffe): server-side EVP flagevaluation payload + bincode-safe sidecar delivery#2117
Open
leoromanovsky wants to merge 35 commits into
Open
feat(datadog-ffe): server-side EVP flagevaluation payload + bincode-safe sidecar delivery#2117leoromanovsky wants to merge 35 commits into
leoromanovsky wants to merge 35 commits into
Conversation
…ure gate - Add telemetry/flagevaluation.rs with FfeFlagEvaluationBatch and FfeFlagEvaluationEvent types modeled on exposures.rs - Required fields: timestamp, flag.key, first_evaluation, last_evaluation, evaluation_count - Optional fields use skip_serializing_if = Option::is_none for omitempty semantics (reviewer concern #2 review:4477935835) - Context pruning helper prune_context() enforces 256 fields / 256 chars skip-not-truncate (reviewer concern #1 review:4477935835) - New Cargo feature flagevaluation-evp gates the module (parallel to exposure-events) - Gate telemetry module in lib.rs to include flagevaluation-evp feature - 30 tests pass (25 existing + 5 new); OTel evaluation_metrics.rs and exposures.rs are byte-for-byte unchanged
…P flusher - Add ffe_flagevaluation_flusher.rs: structural copy of ffe_exposures_flusher.rs with path constant EVP_FLAGEVALUATIONS_PATH = /evp_proxy/v2/api/v2/flagevaluations and batch type FfeFlagEvaluationBatch; fire-and-forget send_batch with non-2xx warn+drop and timeout via biased tokio::select! - Add SidecarAction::FfeFlagEvaluationBatch variant to mod.rs enum; re-export FfeFlagEvaluationBatch from datadog-ffe telemetry module - Route SidecarAction::FfeFlagEvaluationBatch in sidecar_server.rs to ffe_flagevaluation_flusher::send_batch (parallel to FfeExposureBatch arm) - Add exhaust arm for new variant in telemetry.rs process_actions match - Enable flagevaluation-evp feature on datadog-ffe dep in sidecar Cargo.toml - 40 sidecar tests pass (3 new: posts_to_evp_proxy, non_2xx_does_not_panic, timeout_returns_without_waiting); OTel FfeEvaluationMetric path untouched
The worker->sidecar IPC serializes SidecarAction with bincode (non-self-describing). serde_json::Value (deserialize_any) and #[serde(skip_serializing_if)] both make bincode deserialize fail, so the sidecar silently dropped every FfeFlagEvaluationBatch ('IPC serve: failed to decode request') while the worker enqueue still returned ok.
- Carry pruned context as a JSON-object string (Option<String>); remove all skip_serializing_if from the wire types (keep #[serde(default)] for deserialize).
- Re-expand the context string into a JSON object and strip null/false/empty placeholders in ffe_flagevaluation_flusher::build_payload (POST shape unchanged; degraded tier carries no null placeholders).
- Add enqueue_actions_reliable (checked blocking send + reconnect-retry) for one-shot FFE batches; best-effort enqueue_actions left unchanged for high-volume telemetry.
- Add a bincode round-trip test for FfeFlagEvaluationBatch (mixed Some/None fields) to lock the wire-codec contract.
🎉 All green!🧪 All tests passed 🎯 Code Coverage (details) 🔗 Commit SHA: fd0f685 | Docs | Datadog PR Page | Give us feedback! |
Contributor
📚 Documentation Check Results📦
|
Contributor
🔒 Cargo Deny Results📦
|
Contributor
Artifact Size Benchmark Reportaarch64-alpine-linux-musl
aarch64-unknown-linux-gnu
libdatadog-x64-windows
libdatadog-x86-windows
x86_64-alpine-linux-musl
x86_64-unknown-linux-gnu
|
…46-ffe-flagevaluation-evp
…46-ffe-flagevaluation-evp
leoromanovsky
marked this pull request as ready for review
June 23, 2026 03:06
leoromanovsky
requested review from
bwoebi,
dd-oleksii and
typotter
and removed request for
a team
June 23, 2026 03:06
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 60fe825c39
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
bwoebi
reviewed
Jul 1, 2026
bwoebi
reviewed
Jul 2, 2026
bwoebi
reviewed
Jul 2, 2026
leoromanovsky
commented
Jul 6, 2026
…46-ffe-flagevaluation-evp # Conflicts: # datadog-sidecar/src/service/sidecar_server.rs
Contributor
Author
|
Had a DM conversation with Bob that after this review's comments are addressed, it would be approved; merging. |
|
PR Shepherd is now watching this PR: fixing basic CI failures, rebasing when it falls behind, and re-queueing after transient merge-queue failures. To disable it, add the If you have any questions, reach the team in #ai-devx-flow. |
BenchmarksComparisonBenchmark execution time: 2026-07-14 21:43:03 Comparing candidate commit fd0f685 in PR branch Found 3 performance improvements and 7 performance regressions! Performance is the same for 167 metrics, 10 unstable metrics.
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
PHP relies on libdatadog and the sidecar for server-side flag-evaluation delivery, so the shared sidecar path must preserve the same worker-facing EVP contract as the direct SDK implementations. This contribution gives PHP a reusable, backend-verifiable flagevaluation transport without adding a one-off EVP writer in the PHP extension, which keeps cross-SDK adoption smoother for SDK owners and APM review.
Changes
FfeFlagEvaluationBatchafter existing variants.context.evaluationover IPC as a JSON-object string and re-expands it to an object for outbound EVP JSON.false,"",{}, and[]in user context.reasonargument.targeting_rule.key.targeting_keyandcontext; drops and logs only if the degraded row still cannot fit.Decisions
reasonis not part of the native ABI, sidecar event, outbound EVP payload, or aggregation contract.flowchart TD A[sidecar receives/coalesces batches] --> B[clean and encode event JSON] B --> C{candidate POST <= 5 MiB?} C -- yes --> D[POST through Agent EVP proxy] C -- no --> E{single full row can degrade?} E -- yes --> F[omit targeting_key and context] F --> B E -- no --> G[drop and log/count]Validation Evidence
Dogfooding App
ffe-dogfoodingapp-php7andapp-php8-openfeaturewere run with local PHP artifacts using this libdatadog sidecar path.DD_EVP_PROXY_CONFIG_ADDITIONAL_ENDPOINTS={},DD_SKIP_SSL_VALIDATION=false, andDD_REMOTE_CONFIGURATION_NO_TLS_VALIDATION=false.ffe-dogfooding-string-flag:libdd-php7-batch-20260623T020845Z-alphalibdd-php7-batch-20260623T020845Z-bravolibdd-php8of-batch-20260623T020845Z-alphalibdd-php8of-batch-20260623T020845Z-bravovariant_2.System Tests
Staging End-To-End
--datacenter us1.staging.dog --customer-auth=skipagainst theflagevaluationtrack for the exact targeting keys above.flag.key=ffe-dogfooding-string-flag,variant.key=variant_2,allocation.key=allocation-override-392dd7c149f8, andevaluation_count=12:libdd-php7-batch-20260623T020845Z-alpha:first_evaluation=1782180525878,last_evaluation=1782180525985,timestamp=1782180525985,evaluation_count=12libdd-php7-batch-20260623T020845Z-bravo:first_evaluation=1782180525997,last_evaluation=1782180526103,timestamp=1782180526103,evaluation_count=12libdd-php8of-batch-20260623T020845Z-alpha:first_evaluation=1782180526245,last_evaluation=1782180526344,timestamp=1782180526344,evaluation_count=12libdd-php8of-batch-20260623T020845Z-bravo:first_evaluation=1782180526356,last_evaluation=1782180526451,timestamp=1782180526451,evaluation_count=12