Add agentless Feature Flagging configuration source

[leoromanovsky]

NOTE TO REVIEWERS

This PR intentionally keeps the complete feature in one PR so the final architecture remains visible. Its four commits are deliberately layered like a stacked PR, and each commit contains the final production behavior and tests for that layer.

How to Review

1. Review the commits in the order listed below; each link opens only that commit's diff.
2. Treat each commit as an independently buildable review unit.
3. After the fourth commit, use the full Files changed view for a final cross-layer integration pass.
4. Please leave comments on the commit where the behavior is introduced when possible.

Commit Guide

LOC is rename-aware per commit against its parent.

Commit	Review focus	LOC (+/-)
b8bbd269e0	Define the canonical source contract, defaults, metadata, and timing validation. Nonpositive poll intervals and request timeouts are rejected in favor of safe defaults, with tests in this commit.	+165 / -3
4baefdbf6e	Add the complete Datadog-managed poller: authenticated HTTP, accepted-200-only ETags, cold 304, last-known-good, bounded jittered retry, no overlap, in-flight cancellation, shutdown safety, and system-test-parity tests.	+1166 / -1
d378350cf1	Extend the same agentless source to custom HTTP endpoints, including bare-host versus exact-path behavior and tests.	+57 / -1
f9e70e67c4	Select agentless, Remote Configuration, or reserved offline mode at runtime; make repeated startup idempotent; clean up partial startup; and stop the subsystem from Agent shutdown, with tests.	+246 / -18

Per-Commit Validation

Each commit was checked out and validated independently before publication:

1. Contract: ConfigTest, feature-flagging lib/agent tests, and relevant Spotless checks.
2. Managed poller: clean feature-flagging lib tests, Spotless, and JaCoCo coverage gate.
3. Custom endpoint: feature-flagging lib tests, Spotless, and JaCoCo coverage gate.
4. Runtime selection: clean feature-flagging lib/agent tests, both JaCoCo gates, bootstrap/internal-api tests, and all relevant Spotless checks.

All four gates completed with BUILD SUCCESSFUL.

Stack Position

You are here: the Java source-mode foundation. It makes Datadog-managed agentless delivery the default, keeps custom HTTP under agentless, and preserves Agent Remote Configuration as explicit opt-in.

flowchart LR
    subgraph JAVA["dd-trace-java"]
        J1["JAVA-01 · #11892<br/>Agentless + RC sources"] --> J2["JAVA-02 · #11639<br/>Aggregate evaluation EVP"]
    end

    subgraph SYSTEM["system-tests"]
        ST1["ST-01 · #7298<br/>Mock agentless backend"] --> ST2["ST-02 · #7299<br/>Side-effect contracts"]
        ST2 --> STM1["ST-M01 · #7300<br/>Enable Java configuration"]
        ST2 --> NEXT["Next drafts<br/>Enable Java side effects"]
    end

    subgraph DOGFOOD["ffe-dogfooding"]
        DOG0["DOG-00 · #92<br/>Agentless evaluation baseline"] --> DOG1["DOG-01 · #93<br/>Side-effect conduit"]
    end

    J1 --> STM1
    J1 --> DOG0
    J2 --> NEXT
    J2 --> DOG1
    STM1 --> GREEN["Java proof<br/>both sources × side effects"]
    NEXT --> GREEN
    DOG1 --> GREEN

    classDef current fill:#fcbf49,stroke:#8a5a00,stroke-width:3px,color:#111;
    class J1 current;

Loading

Motivation

Java Feature Flagging needs a tracer-side configuration-source split so SDKs can fetch UFC from an agentless HTTP backend while preserving the existing Agent Remote Configuration path.

Changes

• Adds DD_FEATURE_FLAGS_CONFIGURATION_SOURCE with agentless, remote_config, and reserved offline; default is agentless.
• Adds agentless endpoint selection through DD_FEATURE_FLAGS_CONFIGURATION_SOURCE_AGENTLESS_BASE_URL.
• Adds agentless polling controls: DD_FEATURE_FLAGS_CONFIGURATION_SOURCE_AGENTLESS_POLL_INTERVAL_SECONDS and DD_FEATURE_FLAGS_CONFIGURATION_SOURCE_AGENTLESS_REQUEST_TIMEOUT_SECONDS.
• Adds AgentlessConfigurationSource, an HTTP UFC poller used for both the default Datadog-managed endpoint and a configured custom HTTP endpoint.
• Keeps remote_config on the existing RemoteConfigServiceImpl / Agent RC path.
• Handles DD-API-KEY, accepted-200-only ETags/304, malformed UFC rejection, last-known-good preservation, and no overlapping polls.
• Retries timeout/429/5xx failures twice per poll cycle with jittered delays derived from the poll period: clamp(P/6, 2s, 10s) then clamp(P/3, 5s, 30s).
• Cancels in-flight HTTP and pending retry work during shutdown, suppresses post-shutdown retries, and makes subsystem startup idempotent.
• Rejects nonpositive poll intervals and request timeouts by warning and using the documented defaults.
• Removes the proposed DD_FEATURE_FLAGS_ENABLED and extra-header env from this PR.

Decisions

No new provider kill switch. This keeps the existing provider bootstrap behavior through DD_EXPERIMENTAL_FLAGGING_PROVIDER_ENABLED; changing enablement is out of scope for this PR.

Custom HTTP delivery remains under agentless. The source mode describes direct HTTP UFC delivery without the Datadog Agent. With no base URL, the SDK derives the first-party Datadog endpoint as https://api.<site>/api/v2/feature-flagging/config/server-distribution?dd_env=<env>. Setting DD_FEATURE_FLAGS_CONFIGURATION_SOURCE_AGENTLESS_BASE_URL chooses a system-test, dogfood, or operator-managed HTTP endpoint without introducing a fourth source mode. A bare host uses the standard server-distribution path; a URL with a path is used as the exact UFC endpoint.

Remote Configuration remains a distinct mode. It is Agent-mediated and uses the RC protocol's security, signing, targeting, capability, and subscription lifecycle. Those semantics are materially different from direct HTTP polling, so remote_config remains a peer source mode rather than another agentless endpoint choice.

No offline factory yet. offline remains a reserved configuration-source value. This PR does not expose a startup-bytes API or offline factory because that API still needs design work.

Source selection stays SDK-local. DD_FEATURE_FLAGS_CONFIGURATION_SOURCE decides which local source implementation starts. It is not sent to the backend.

Initialization Modes

Solid connectors are implemented in this PR. The dashed offline connector previews startup-provided UFC bytes.

flowchart TD
    Gate{DD_EXPERIMENTAL_FLAGGING_PROVIDER_ENABLED}
    Disabled[Feature Flagging subsystem not started]
    System[FeatureFlaggingSystem.start]
    Select{DD_FEATURE_FLAGS_CONFIGURATION_SOURCE}
    Exposure[ExposureWriter starts for every enabled mode]

    subgraph AgentlessMode[agentless mode - current default]
        AgentlessSource[AgentlessConfigurationSource]
        OperationalDefaults[SDK-owned operation<br/>30s poll default; 2s request timeout default<br/>ETag; retry; last-known-good; no overlap]
        EndpointChoice{agentless base URL set?}
        DatadogManaged[Datadog-managed agentless<br/>first-party; CDN-backed]
        CustomHttp[Custom HTTP endpoint<br/>system tests; dogfood; operator backend]

        AgentlessSource --- OperationalDefaults
        AgentlessSource -- polls --> EndpointChoice
        EndpointChoice -- no --> DatadogManaged
        EndpointChoice -- yes --> CustomHttp
    end

    RemoteConfig[remote_config<br/>explicit opt-in; Agent-mediated]
    RemoteConfigSource[RemoteConfigServiceImpl]
    AgentRc[Datadog Agent Remote Configuration<br/>RC security; signing; targeting; subscriptions]

    OfflineReserved[offline today<br/>reserved; no configuration service]
    OfflineSource[Later: OfflineConfigurationSource<br/>customer UFC JSON bytes at startup; no network]

    Ufc[Shared UFC deserialize and evaluate pipeline]
    Gateway[FeatureFlaggingGateway]
    Provider[OpenFeature provider]

    Gate -- false --> Disabled
    Gate -- true --> System
    System --> Select
    System --> Exposure

    Select -- unset or agentless --> AgentlessSource
    DatadogManaged --> Ufc
    CustomHttp --> Ufc

    Select -- remote_config --> RemoteConfig
    RemoteConfig --> RemoteConfigSource
    RemoteConfigSource -- subscribes --> AgentRc
    RemoteConfigSource --> Ufc

    Select -- offline today --> OfflineReserved
    OfflineReserved -. later .-> OfflineSource
    OfflineSource -. startup UFC bytes .-> Ufc

    Ufc --> Gateway
    Gateway --> Provider

Loading

Customer Usage Example

Default Datadog-managed agentless delivery:

export DD_API_KEY=<datadog-api-key>
export DD_SITE=datadoghq.com
export DD_ENV=prod

# Optional because agentless is the default configuration source.
export DD_FEATURE_FLAGS_CONFIGURATION_SOURCE=agentless
export DD_FEATURE_FLAGS_CONFIGURATION_SOURCE_AGENTLESS_POLL_INTERVAL_SECONDS=30
export DD_FEATURE_FLAGS_CONFIGURATION_SOURCE_AGENTLESS_REQUEST_TIMEOUT_SECONDS=2

java -javaagent:/path/to/dd-java-agent.jar -jar app.jar

Custom HTTP delivery keeps agentless mode and changes only the endpoint:

export DD_FEATURE_FLAGS_CONFIGURATION_SOURCE=agentless
export DD_FEATURE_FLAGS_CONFIGURATION_SOURCE_AGENTLESS_BASE_URL=https://flags.example.com/ufc

java -javaagent:/path/to/dd-java-agent.jar -jar app.jar

Explicit Agent Remote Configuration delivery still uses the existing Agent path:

export DD_FEATURE_FLAGS_CONFIGURATION_SOURCE=remote_config
export DD_REMOTE_CONFIGURATION_ENABLED=true

java -javaagent:/path/to/dd-java-agent.jar -jar app.jar

Verification

./gradlew :products:feature-flagging:feature-flagging-agent:test :products:feature-flagging:feature-flagging-lib:test --no-daemon
result: BUILD SUCCESSFUL

./gradlew :products:feature-flagging:feature-flagging-agent:jacocoTestCoverageVerification :products:feature-flagging:feature-flagging-lib:jacocoTestCoverageVerification :products:feature-flagging:feature-flagging-agent:spotlessCheck :products:feature-flagging:feature-flagging-lib:spotlessCheck --no-daemon
result: BUILD SUCCESSFUL

./gradlew :dd-smoke-tests:openfeature:cleanTest :dd-smoke-tests:openfeature:test --tests datadog.smoketest.springboot.OpenFeatureProviderSmokeTest --no-daemon
result: BUILD SUCCESSFUL

git diff --check
result: clean

System Test Evidence

I built dd-java-agent, dd-trace-api, and dd-openfeature from the rewritten
head f9e70e67c4, rebuilt the Java Spring Boot weblog image with those local
artifacts, and verified the JAR inside the image matched the local agent JAR.
I then ran the Java manifest-enabled agentless scenario from the companion
system-tests draft:

TEST_LIBRARY=java ./run.sh +v \
  FEATURE_FLAGGING_AND_EXPERIMENTATION_AGENTLESS \
  tests/ffe/test_agentless_configuration.py

Scenario: FEATURE_FLAGGING_AND_EXPERIMENTATION_AGENTLESS
Library: java@1.65.0-SNAPSHOT+f9e70e67c4
Weblog variant: spring-boot
collected 1 item
tests/ffe/test_agentless_configuration.py .                        [100%]
1 passed

This exercises the Java manifest row proposed by the companion draft against
the exact code currently published in this PR.

Next Steps

• Review the draft Java manifest activation:
  DataDog/system-tests#7300.
• Keep that PR draft until this Java behavior is available to system-tests CI.
• After this PR lands, rerun the manifest activation in Java development CI and
  merge the activation only when that gate is green.

[dd-octo-sts]

🟢 Java Benchmark SLOs — All performance SLOs passed

Suite	Status
Startup	🟢 pass

SLO thresholds are defined here based on automatically generated metrics. A warning is raised when results are within 5% of the threshold.

PR vs. master results

Scenario	Candidate	master	Δ (95% CI of mean)
startup:insecure-bank:iast:Agent	14.01 s	14.00 s	[-0.7%; +0.8%] (no difference)
startup:insecure-bank:tracing:Agent	12.86 s	12.95 s	[-1.8%; +0.4%] (no difference)
startup:petclinic:appsec:Agent	16.88 s	16.66 s	[+0.1%; +2.6%] (maybe worse)
startup:petclinic:iast:Agent	16.87 s	16.50 s	[-2.3%; +6.8%] (no difference)
startup:petclinic:profiling:Agent	16.49 s	16.17 s	[-2.5%; +6.5%] (no difference)
startup:petclinic:sca:Agent	16.80 s	16.46 s	[-2.1%; +6.2%] (no difference)
startup:petclinic:tracing:Agent	16.18 s	16.09 s	[-0.2%; +1.4%] (no difference)

Commit: 488b4770 · CI Pipeline · Benchmarking Platform UI

---

Load and DaCapo benchmarks can be triggered manually in the GitLab pipeline. Results will appear in the Benchmarking Platform UI after completion.

[dd-octo-sts]

Hi! 👋 Thanks for your pull request! 🎉

To help us review it, please make sure to:

• Add at least one type, and one component or instrumentation label to the pull request

If you need help, please check our contributing guidelines.

[datadog-datadog-prod-us1-2]

All retry, ETag, shutdown, and idempotency paths look correct. One real gap: apply() silently returns false for 401/403 with no log above DEBUG — users with a wrong or missing DD_API_KEY have no way to diagnose why feature flags never load. Fixed by separating the auth-failure branch in apply() with a LOGGER.warn(...) call.

_{Open Bits AI session}

_{🤖 Datadog Autotest · Commit f9e70e6 · What is Autotest? · Any feedback? Reach out in #autotest}

[pavlokhrebto]

Advisory review of the agentless configuration source, cross-checked against the Agentless-mode RFC. The core state machine (cold vs. warm handling, retry classification, readiness, atomic snapshot swap, shutdown / last-known-good) matches the RFC precisely — nice work. Findings below are precision-first and mostly non-blocking; inline comments are attached to the relevant lines.

Descoped config note (C8): The RFC lists DD_FEATURE_FLAGS_CONFIGURATION_SOURCE_AGENTLESS_EXTRA_HEADERS (default empty object); this PR intentionally removes it. Fine if deliberate, but noting it since the mock-CDN test flow may want custom headers — worth tracking as a follow-up.

[leoromanovsky]

Contract update from the staging CDN response: the managed agentless endpoint does not return raw UFC at the top level. The review follow-up changes the managed response contract to:

GET https://ufc-server.ff-cdn.<site>/api/v2/feature-flagging/config/rules-based/server?dd_env=<environment>
DD-API-KEY: <api key>

{"data":{"id":"1","type":"universal-flag-configuration","attributes":{"createdAt":"...","environment":{"name":"Staging"},"flags":{}}}}

The Java implementation requires data.type == universal-flag-configuration, unwraps data.attributes, and feeds those attributes through the existing UFC parser. Missing or mismatched envelopes are rejected without replacing last-known-good state. Explicitly configured compatibility endpoints retain the raw-UFC fallback.

The matching system-tests fixture change is DataDog/system-tests#7315. It wraps the existing raw fixture only at the agentless HTTP boundary, so RC fixture consumers are unchanged.

Local system-tests proof:

./run.sh TEST_THE_TEST tests/test_the_test/test_mock_ffe_agentless_backend.py
3 passed in 2.32s

The self-tests verify the canonical route, exact DD-API-KEY auth behavior, JSON:API resource type and UFC attributes, metadata-only status, and host-gateway mapping.

[sarahchen6]

Two final comments - I think both are worth addressing, but definitely the internal-api dependency refinement. Otherwise, looks good! Thanks for all the iterations. Pre-approving in the meantime!

[sarahchen6]

IIUC if a typo / invalid value is set as config source, the Feature Flagging subsystem would not start and just log an IllegalArgumentException: https://github.com/DataDog/dd-trace-java/pull/11892/changes#diff-a2718be99f97d918aebd130483af618925dd3121c802f00eed3fe9c3af712dfeR122. Instead we should probably catch invalid values here immediately and fall-back to the default agentless setup with a warning printed in logs. We could possibly leverage ConfigProvider.getEnum to do this. WDYT?