Skip to content

Fix service re-registration after Consul connection failures - #1038

Merged
ryanjbaxter merged 3 commits into
spring-cloud:mainfrom
yashdotdev13:fix/774-consul-service-reregistration
Oct 28, 2026
Merged

ryanjbaxter merged 3 commits into
spring-cloud:mainfrom
yashdotdev13:fix/774-consul-service-reregistration

Conversation

@yashdotdev13

Copy link
Copy Markdown
Contributor

Summary

Fix service re-registration behavior when Consul is unavailable during initial service registration or subsequent heartbeat attempts.

Changes

  • Schedule TTL heartbeats even when initial service registration fails, provided heartbeat is enabled and the service has a TTL check.
  • Handle ResourceAccessException during heartbeat execution so transient connection failures do not prevent subsequent heartbeat attempts.
  • Handle connection failures during service re-registration and allow retries on subsequent heartbeat attempts.
  • Add regression tests covering failed initial registration and heartbeat retries after connection failures.

Testing

  • ConsulHeartbeatTaskTests
  • ConsulServiceRegistryRegistrationTests
  • Full Maven reactor tests: .\mvnw.cmd -pl spring-cloud-consul-discovery -am test
  • git diff --check

All tests passed: 145 tests run, 0 failures, 0 errors, 1 skipped.

Related Issue

Fixes #774

Signed-off-by: 98001yash <yashchauhan.gaya@gmail.com>

@ryanjbaxter ryanjbaxter left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

An error that happens repeatedly will be noisy by logging the stacktrace, maybe log the stacktrace at debug level?

this.ttlScheduler.client.agentServiceRegister(
this.ttlScheduler.discoveryProperties.getAclToken(), registeredService);
}
catch (ResourceAccessException registrationException) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If Consul is reachable but the re-register call returns a 5xx or 403, HttpStatusCodeException still escapes from here, while a connection failure is now handled. Would catching RestClientException be more consistent?

Also, there’s no test for this path.

@ryanjbaxter

ryanjbaxter commented Oct 27, 2026 •

Copy link
Copy Markdown
Contributor

Since spring.cloud.consul.discovery.heartbeat.reregister-service-on-failure is false by default it probably makes sense to call out in the docs that this will only works if spring.cloud.consul.discovery.heartbeat.reregister-service-on-failure=true.

Signed-off-by: 98001yash <yashchauhan.gaya@gmail.com>
@yashdotdev13

Copy link
Copy Markdown
Contributor Author

Addressed the review feedback by handling RestClientException during re-registration, adding a regression test for re-registration failures, reducing repetitive warning-level stack traces, and documenting the opt-in configuration. The full Maven test suite passes (146 tests, 0 failures, 0 errors, 1 skipped).

@ryanjbaxter ryanjbaxter left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we add a test for when failFast is true? If that is true we should not retry IMO

}
catch (ResourceAccessException e) {
log.warn("Unable to send heartbeat to Consul for service " + this.serviceId
+ ". Will retry on the next heartbeat.");

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Appending e.getMessage to the warn log would be useful to at a high level see what the reason for the failure was yet not pollute the log with the stacktrace

log.warn("Failfast is false. Error registering service with consul: " + reg.getService(), e);
}

NewService service = reg.getService();

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why was this moved outside of the try?

@yashdotdev13 yashdotdev13 Oct 28, 2026 •

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The scheduling was moved outside the try-catch to ensure that heartbeat retries are still scheduled when the initial registration fails and failFast=false. Otherwise, the registration exception would prevent scheduling, leaving the service without a retry mechanism. When failFast=true, the exception is rethrown before reaching the scheduling logic, so no heartbeat retry is scheduled.

Signed-off-by: 98001yash <yashchauhan.gaya@gmail.com>
@ryanjbaxter
ryanjbaxter merged commit 1dfc070 into spring-cloud:main Oct 28, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Serivices won't register automatically when consul-server is started after starting spring-cloud-consul

3 participants