Skip to content

RATIS-2398. Add opentelemetry-javaagent to ratis-examples and assembly#1428

Open
taklwu wants to merge 1 commit into
apache:masterfrom
taklwu:RATIS-2398
Open

RATIS-2398. Add opentelemetry-javaagent to ratis-examples and assembly#1428
taklwu wants to merge 1 commit into
apache:masterfrom
taklwu:RATIS-2398

Conversation

@taklwu

@taklwu taklwu commented Apr 15, 2026

Copy link
Copy Markdown
Contributor

also fixed a bug that client need to issue the parent span in the client request

What changes were proposed in this pull request?

  • make an example of how to use tracing in ratis-examples
  • make sure the script could be automatically running and span shows in the logs and Jaegar UI
  • fix the parent span issued from the client.

What is the link to the Apache JIRA

RATIS-2389

How was this patch tested?

manual tests for running the https://github.com/apache/ratis/tree/master/ratis-examples locally, tested with below

  1. logger mode that emits span in logs
# on the server 
[otel.javaagent 2026-04-15 15:40:24:419 -0700] [pool-3-thread-1] INFO io.opentelemetry.exporter.logging.LoggingSpanExporter - 'raft.server.submitClientRequestAsync' : 2a9bfd5258de404d4c8dcd291408538f 98677058d4b31a8e SERVER [tracer: org.apache.ratis:3.3.0-SNAPSHOT] AttributesMap{data={thread.id=69, thread.name=n0-client-thread1, raft.member.id=n0@group-6F7570313233, raft.call.id=51198, raft.client.id=client-5813AB26C91D}, capacity=128, totalAddedValues=5}
[otel.javaagent 2026-04-15 15:40:24:419 -0700] [pool-3-thread-2] INFO io.opentelemetry.exporter.logging.LoggingSpanExporter - 'raft.server.submitClientRequestAsync' : f56c5fe25544695b023cfcdcd4f32447 57fd34b224ee3d6a SERVER [tracer: org.apache.ratis:3.3.0-SNAPSHOT] AttributesMap{data={thread.id=69, thread.name=n0-client-thread1, raft.member.id=n0@group-6F7570313233, raft.call.id=51196, raft.client.id=client-5813AB26C91D}, capacity=128, totalAddedValues=5}

# from the client 
[otel.javaagent 2026-04-15 15:40:24:420 -0700] [grpc-default-executor-7] INFO io.opentelemetry.exporter.logging.LoggingSpanExporter - 'Async::send' : 3326eebb2357274628d3dea57e586435 2715719eb0c38ec3 CLIENT [tracer: org.apache.ratis:3.3.0-SNAPSHOT] AttributesMap{data={thread.id=37, raft.peer.id=LEADER, thread.name=pool-1-thread-20, raft.operation.type=RW, raft.operation.name=Async::send}, capacity=128, totalAddedValues=5}
  1. export traces to the third party otlp e.g. Jaegar
Screenshot 2026-04-15 at 3 39 32 PM

@taklwu

taklwu commented Apr 29, 2026

Copy link
Copy Markdown
Contributor Author

@szetszwo @OneSizeFitsQuorum any comments?

@taklwu taklwu mentioned this pull request Apr 30, 2026
Merged
szetszwo
szetszwo reviewed May 5, 2026
View reviewed changes

@szetszwo szetszwo left a comment •
edited
Loading

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@taklwu , thanks for working on this! Just have a quick comment inlined.

Let me try to run an example with tracing.

Comment thread ratis-client/src/main/java/org/apache/ratis/client/impl/RaftClientImpl.java Outdated
@taklwu taklwu requested a review from szetszwo May 5, 2026 19:30
@taklwu

taklwu commented May 6, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor Author

those failure are not related, but let's see if we can retrigger...

org.apache.ratis.grpc.TestLinearizableReadWithGrpc
org.apache.ratis.grpc.TestLinearizableReadAppliedIndexLeaderLeaseReadWithGrpc
org.apache.ratis.netty.TestRaftAsyncWithNetty (this one keeps failing now)

@taklwu taklwu force-pushed the RATIS-2398 branch 3 times, most recently from ab0513f to ae781fb Compare May 17, 2026 19:15
@taklwu

taklwu commented May 18, 2026

Copy link
Copy Markdown
Contributor Author

@adoroszlai @szetszwo the PR github tests looks green now, do you have any other comments?

szetszwo
szetszwo reviewed May 28, 2026
View reviewed changes

@szetszwo szetszwo left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@taklwu , thanks for working on this! Please see the comments inlined.

(Sorry for the late reply.)

Comment thread ratis-common/src/main/java/org/apache/ratis/trace/TraceConfigKeys.java Outdated
if (properties.getRaw(ENABLED_KEY) != null) {
return getBoolean(properties::getBoolean, ENABLED_KEY, ENABLED_DEFAULT, logger);
}
final String fromSystem = System.getProperty(ENABLED_KEY);

@szetszwo szetszwo May 28, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's don't fallback to JVM system properties.

Ratis currently do not handle how the confs are stored (e.g. in a file, in a DB, in an external service, or JVM system properties). The application uses Ratis has to decide how to do it and pass it to Ratis via a RaftProperties object.

Also, using JVM system properties could cause problem for applications using Ratis for multiple components -- e.g. there could be two different services running in the same JVM using Ratis. They may use different confs.

@taklwu taklwu Jun 16, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sorry for coming back on this.

it's fine to remove JVM or system properties.

When we want to use this tracing with Ozone, does it mean we need to create a special keys in Ozone e.g. hdds.ratis.client.* or hdds.ratis.server.* such that the open telemetry could be enabled?

@szetszwo szetszwo Jun 16, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For Ozone, we may add a prefix to set any Ratis conf; see https://github.com/apache/ozone/blob/master/hadoop-hdds/docs/content/feature/multi-raft-support.md#advanced-ratis-configuration

Comment thread ratis-examples/src/main/bin/client.sh
shift

java ${LOGGER_OPTS} -jar $ARTIFACT "$example" "$subcommand" "$@"
java ${OTEL_OPTS} ${LOGGER_OPTS} -jar $ARTIFACT "$example" "$subcommand" "$@"

@szetszwo szetszwo May 28, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For the example command, we could change it to get the JVM property and then set it to RaftProperties.

One idea is to add a new ExampleLuancher class, which will read JVM property and then pass it to the examples (e.g. "filestore" or "arithmetic").

@taklwu taklwu Jun 17, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

added ExampleLauncher.java

@szetszwo

szetszwo commented Jun 16, 2026

Copy link
Copy Markdown
Contributor

@taklwu , please wait for RATIS-2558, which will move OpenTelemetry code and its dependency to a separated module.

Or, you may update this with master...szetszwo:ratis:RATIS-2558

@taklwu
RATIS-2398. Add opentelemetry-javaagent to ratis-examples and assembly
058b4fe 
- fixed a bug that a client needs to issue the parent span in the client request
@taklwu

taklwu commented Jun 17, 2026

Copy link
Copy Markdown
Contributor Author

@taklwu , please wait for RATIS-2558, which will move OpenTelemetry code and its dependency to a separated module.

Or, you may update this with master...szetszwo:ratis:RATIS-2558

got it, I will have another update once RATIS-2558 completes

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@szetszwo szetszwo szetszwo left review comments

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@taklwu @szetszwo