Skip to content

perf: materialize child emitters in ComposingEmitter to avoid per-event Guice resolution - #19942

Draft
rbankar7 wants to merge 3 commits into
apache:masterfrom
rbankar7:rban/fix-composing
Draft

perf: materialize child emitters in ComposingEmitter to avoid per-event Guice resolution#19942
rbankar7 wants to merge 3 commits into
apache:masterfrom
rbankar7:rban/fix-composing

Conversation

@rbankar7

@rbankar7 rbankar7 commented Aug 9, 2026

Copy link
Copy Markdown
Contributor

Fixes #19943

Description

When Druid is configured with druid.emitter=composing, the composing emitter was re-resolving all of its child emitters from the Guice injector on every emitted event. Because Druid emits a metric per segment scanned, on busy nodes this executes tens of thousands of times per second, and each resolution takes a monitor lock. Lock profiling on affected nodes attributed the large majority of lock-contention samples to LifecycleScope$1.get.

Root cause

ComposingEmitterModule#getEmitter built its child-emitter list with Lists.transform(...), which returns a lazy Guava view rather than a materialized list:

List<Emitter> emitters = Lists.transform(
    config.getEmitters(),
    s -> injector.getInstance(Key.get(Emitter.class, Names.named(s)))
);
return new ComposingEmitter(emitters);

ComposingEmitter.emit(Event) iterates this list once per event:

public void emit(Event event) {
  for (Emitter e : emitters) {
    e.emit(event);
  }
}

Since emitters is a lazy view, each iteration re-ran the transform function, which re-resolved every child emitter from the injector. For lifecycle-scoped bindings, that resolution goes through LifecycleScope#get(), which is synchronized and so takes a monitor lock on every call — even though, after startup, it does nothing but return an already-cached instance. The child emitters were therefore being looked up (and locked) on the hottest path in the process rather than resolved once at startup.

Fixed the lock contention in the composing emitter

The child emitters are now resolved eagerly, exactly once, into an ImmutableList at construction time, so ComposingEmitter.emit() iterates a plain materialized list with no injector lookups and no locking:

List<Emitter> emitters = config.getEmitters()
                               .stream()
                               .map(s -> injector.getInstance(Key.get(Emitter.class, Names.named(s))))
                               .collect(ImmutableList.toImmutableList());
return new ComposingEmitter(emitters);

This is purely an internal change — the same set of child emitters is composed in the same order. The only difference is when they are resolved: once during provisioning instead of on every emitted event. This also aligns ComposingEmitterModule with its sibling SwitchingEmitterModule, which already resolves its child emitters eagerly into materialized lists.

Release note

Fixed lock contention on the metrics emit path when using the composing emitter (druid.emitter=composing). Child emitters are now resolved once at startup instead of on every emitted event, which previously caused significant monitor contention on nodes emitting metrics at high volume.


Key changed/added classes in this PR
  • ComposingEmitterModule
  • Added regression test that emits multiple events and asserts the child is resolved only once

This PR has:

  • been self-reviewed.
  • added documentation for new or modified features or behaviors.
  • a release note entry in the PR description.
  • added Javadocs for most classes and all non-trivial methods. Linked related entities via Javadoc links.
  • added or updated version, license, or notice information in licenses.yaml
  • added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.
  • added unit tests or modified existing tests to cover new code paths, ensuring the threshold for code coverage is met.
  • added integration tests.
  • been tested in a test Druid cluster.

@abhishekrb19 abhishekrb19 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Lgtm, good catch! Double-checked SwitchingEmitterModule as well - same eager resolution behavior already applies there.

@abhishekrb19

Copy link
Copy Markdown
Contributor

@rbankar7 this PR is marked as draft — are there any additional changes you're intending to make here?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Composing emitter re-resolves child emitters from Guice on every event, causing lock contention

2 participants