Skip to content

fix(server): resolve Hazelcast notification trigger independent of auto-configuration order - #5726

Open
YadavKshitiz wants to merge 4 commits into
codecentric:masterfrom
YadavKshitiz:fix/5716-hazelcast-notification-dedup
Open

YadavKshitiz wants to merge 4 commits into
codecentric:masterfrom
YadavKshitiz:fix/5716-hazelcast-notification-dedup

Conversation

@YadavKshitiz

@YadavKshitiz YadavKshitiz commented Sep 29, 2026 •

Copy link
Copy Markdown

Related to #5716

Summary

With Hazelcast, notification deduplication across cluster nodes did not work when the notifiers were
configured through spring.boot.admin.notify.* properties. Also, the (unreleased) change from #5737 made the
application fail to start with a user-defined @Bean notifier. This PR fixes both.

Problems

  1. HazelcastNotificationTrigger was not created for property-configured notifiers (Slack, Mail, Teams, ...).
    AdminServerHazelcastAutoConfiguration is ordered before AdminServerNotifierAutoConfiguration, so its
    @ConditionalOnBean(Notifier.class) was evaluated before those notifiers existed. A plain NotificationTrigger
    was created instead, with no cross-node deduplication. The existing hazelcastConfig() test only passed because
    it declares the notifier as a user bean.
  2. Regression from fix: share notification filters across Hazelcast cluster nodes #5737: the auto-created FilteringNotifier is itself a Notifier, so the CompositeNotifier
    collects it and the context fails with a BeanCurrentlyInCreationException (filteringNotifier ->
    compositeNotifier -> filteringNotifier). With property-only notifiers no FilteringNotifier was created.

Changes

  • AdminServerHazelcastAutoConfiguration exposes two lazy beans: sentNotificationsMap and notificationFiltersMap.
  • AdminServerNotifierAutoConfiguration gets the sent-notifications map as a plain ConcurrentMap through an
    ObjectProvider, resolved lazily when the trigger is created, and creates HazelcastNotificationTrigger when it
    is present. It does not reference Hazelcast classes, so Hazelcast stays optional, and the result no longer depends
    on auto-configuration order.
  • Removed the nested NotifierTriggerConfiguration and the auto-created FilteringNotifier. To share filters in a
    cluster, declare your own FilteringNotifier and pass the notificationFiltersMap bean.
  • Clustering docs: example for sharing filters, and a warning about the circular dependency with
    CompositeNotifier.

Tests

  • New AdminServerHazelcastAutoConfigurationTest: single @Bean notifier, notifiers configured via properties,
    multiple property notifiers, Hazelcast disabled, and a user FilteringNotifier using the shared filters map.
    The property-notifier tests fail on master (plain NotificationTrigger) and pass with this change.
  • AdminServerAutoConfigurationTest: the runner now includes AdminServerNotifierAutoConfiguration.
  • Full spring-boot-admin-server module run green locally (504 tests); Checkstyle and javaformat pass.

About #5716

The reporter's notifiers are user-declared @Beans, so the wiring problem above most likely did not affect him.
His suspicion about the per-node last status in AbstractStatusChangeNotifier remains the likeliest cause of his
duplicates. That is handled in a follow-up PR (map injection), so this PR only references the issue. Nothing was
tested on two live nodes.

Release notes

  • Removed: nested class AdminServerHazelcastAutoConfiguration.NotifierTriggerConfiguration (internal
    auto-configuration detail). The Hazelcast trigger is now created by AdminServerNotifierAutoConfiguration.
  • The auto-created FilteringNotifier from fix: share notification filters across Hazelcast cluster nodes #5737 (unreleased) is removed. Declare your own FilteringNotifier and
    pass notificationFiltersMap to share filters (see the clustering docs).

@YadavKshitiz
YadavKshitiz requested a review from a team as a code owner September 29, 2026 10:49
@cdprete

cdprete commented Sep 29, 2026

Copy link
Copy Markdown
Contributor

Hello @YadavKshitiz.

Below my configuration:

@Configuration(proxyBeanMethods = false)
public class NotificationConfig {
    // See de.codecentric.boot.admin.server.config.AdminServerNotifierAutoConfiguration.MailNotifierConfiguration.mailNotifier
    static final String MAIL_CONFIG_PROPS_PREFIX = "spring.boot.admin.notify.mail";
    // See de.codecentric.boot.admin.server.config.AdminServerNotifierAutoConfiguration.MailNotifierConfiguration.mailNotifierTemplateEngine
    private static final String TEMPLATE_ENGINE_BEAN_NAME = "mailNotifierTemplateEngine";

    @Bean
    @Validated
    @ConfigurationProperties(MAIL_CONFIG_PROPS_PREFIX)
    public DelayingMailNotifier mailNotifier(JavaMailSender mailSender, InstanceRepository repository, @Qualifier(TEMPLATE_ENGINE_BEAN_NAME) TemplateEngine templateEngine) {
        return new DelayingMailNotifier(mailSender, repository, templateEngine);
    }

    @Bean
    @Primary
    @ConditionalOnBean(DelayingMailNotifier.class)
    public FilteringNotifier filteringNotifier(DelayingMailNotifier mailNotifier, InstanceRepository instanceRepository) {
        return new FilteringNotifier(mailNotifier, instanceRepository);
    }
}

@YadavKshitiz

Copy link
Copy Markdown
Author

Hello @YadavKshitiz.

Below my configuration:

@Configuration(proxyBeanMethods = false)
public class NotificationConfig {
    // See de.codecentric.boot.admin.server.config.AdminServerNotifierAutoConfiguration.MailNotifierConfiguration.mailNotifier
    static final String MAIL_CONFIG_PROPS_PREFIX = "spring.boot.admin.notify.mail";
    // See de.codecentric.boot.admin.server.config.AdminServerNotifierAutoConfiguration.MailNotifierConfiguration.mailNotifierTemplateEngine
    private static final String TEMPLATE_ENGINE_BEAN_NAME = "mailNotifierTemplateEngine";

    @Bean
    @Validated
    @ConfigurationProperties(MAIL_CONFIG_PROPS_PREFIX)
    public DelayingMailNotifier mailNotifier(JavaMailSender mailSender, InstanceRepository repository, @Qualifier(TEMPLATE_ENGINE_BEAN_NAME) TemplateEngine templateEngine) {
        return new DelayingMailNotifier(mailSender, repository, templateEngine);
    }

    @Bean
    @Primary
    @ConditionalOnBean(DelayingMailNotifier.class)
    public FilteringNotifier filteringNotifier(DelayingMailNotifier mailNotifier, InstanceRepository instanceRepository) {
        return new FilteringNotifier(mailNotifier, instanceRepository);
    }
}

Thanks for sharing your config, that helps a lot!

Since your notifiers are declared as your own @bean, I think you were probably not hit by the
problem my PR fixes. That one affects notifiers configured through properties only
(spring.boot.admin.notify.*), where the Hazelcast dedup trigger wasn't being created. So I may
have been chasing a related but different bug, and I'll change the PR from "Fixes" to "Related to"
so this issue stays open until we find your actual cause.

A few questions so we can narrow it down:

  1. Which SBA version are you on, and do the Hazelcast logs show both nodes in one cluster
    (Members {size:2 ...})?
  2. Could you enable DEBUG logging for de.codecentric.boot.admin.server.notify on both nodes?
    For one status change, does each node log "Triggering notifiers for ..." or does one of them
    log "Notifications already sent ..."?
  3. What does DelayingMailNotifier do? If it delays or schedules the mail on its own, it might
    be sending outside of the dedup check.
  4. Do you have a custom Hazelcast map config for spring-boot-admin-sent-notifications
    (eviction, TTL)?

Thanks again!

@cdprete

cdprete commented Sep 29, 2026

Copy link
Copy Markdown
Contributor

Hello @YadavKshitiz.
Below my configuration:

@Configuration(proxyBeanMethods = false)
public class NotificationConfig {
    // See de.codecentric.boot.admin.server.config.AdminServerNotifierAutoConfiguration.MailNotifierConfiguration.mailNotifier
    static final String MAIL_CONFIG_PROPS_PREFIX = "spring.boot.admin.notify.mail";
    // See de.codecentric.boot.admin.server.config.AdminServerNotifierAutoConfiguration.MailNotifierConfiguration.mailNotifierTemplateEngine
    private static final String TEMPLATE_ENGINE_BEAN_NAME = "mailNotifierTemplateEngine";

    @Bean
    @Validated
    @ConfigurationProperties(MAIL_CONFIG_PROPS_PREFIX)
    public DelayingMailNotifier mailNotifier(JavaMailSender mailSender, InstanceRepository repository, @Qualifier(TEMPLATE_ENGINE_BEAN_NAME) TemplateEngine templateEngine) {
        return new DelayingMailNotifier(mailSender, repository, templateEngine);
    }

    @Bean
    @Primary
    @ConditionalOnBean(DelayingMailNotifier.class)
    public FilteringNotifier filteringNotifier(DelayingMailNotifier mailNotifier, InstanceRepository instanceRepository) {
        return new FilteringNotifier(mailNotifier, instanceRepository);
    }
}

Thanks for sharing your config, that helps a lot!

Since your notifiers are declared as your own @bean, I think you were probably not hit by the problem my PR fixes. That one affects notifiers configured through properties only (spring.boot.admin.notify.*), where the Hazelcast dedup trigger wasn't being created. So I may have been chasing a related but different bug, and I'll change the PR from "Fixes" to "Related to" so this issue stays open until we find your actual cause.

A few questions so we can narrow it down:

  1. Which SBA version are you on, and do the Hazelcast logs show both nodes in one cluster
    (Members {size:2 ...})?
  2. Could you enable DEBUG logging for de.codecentric.boot.admin.server.notify on both nodes?
    For one status change, does each node log "Triggering notifiers for ..." or does one of them
    log "Notifications already sent ..."?
  3. What does DelayingMailNotifier do? If it delays or schedules the mail on its own, it might
    be sending outside of the dedup check.
  4. Do you have a custom Hazelcast map config for spring-boot-admin-sent-notifications
    (eviction, TTL)?

Thanks again!

Hello @YadavKshitiz

  • latest SBA version and Hazelcast shows both the nodes
  • I need to redeploy it, but it's possible
  • delays the notification by 5 minutes by scheduling a job for it. Of course, if in the meantime the application recovered (that's the whole point in delaying them), such job gets cancelled
  • no. Only the one shared in the docs

@YadavKshitiz

Copy link
Copy Markdown
Author

Hello @YadavKshitiz.
Below my configuration:

@Configuration(proxyBeanMethods = false)
public class NotificationConfig {
    // See de.codecentric.boot.admin.server.config.AdminServerNotifierAutoConfiguration.MailNotifierConfiguration.mailNotifier
    static final String MAIL_CONFIG_PROPS_PREFIX = "spring.boot.admin.notify.mail";
    // See de.codecentric.boot.admin.server.config.AdminServerNotifierAutoConfiguration.MailNotifierConfiguration.mailNotifierTemplateEngine
    private static final String TEMPLATE_ENGINE_BEAN_NAME = "mailNotifierTemplateEngine";

    @Bean
    @Validated
    @ConfigurationProperties(MAIL_CONFIG_PROPS_PREFIX)
    public DelayingMailNotifier mailNotifier(JavaMailSender mailSender, InstanceRepository repository, @Qualifier(TEMPLATE_ENGINE_BEAN_NAME) TemplateEngine templateEngine) {
        return new DelayingMailNotifier(mailSender, repository, templateEngine);
    }

    @Bean
    @Primary
    @ConditionalOnBean(DelayingMailNotifier.class)
    public FilteringNotifier filteringNotifier(DelayingMailNotifier mailNotifier, InstanceRepository instanceRepository) {
        return new FilteringNotifier(mailNotifier, instanceRepository);
    }
}

Thanks for sharing your config, that helps a lot!
Since your notifiers are declared as your own @bean, I think you were probably not hit by the problem my PR fixes. That one affects notifiers configured through properties only (spring.boot.admin.notify.*), where the Hazelcast dedup trigger wasn't being created. So I may have been chasing a related but different bug, and I'll change the PR from "Fixes" to "Related to" so this issue stays open until we find your actual cause.
A few questions so we can narrow it down:

  1. Which SBA version are you on, and do the Hazelcast logs show both nodes in one cluster
    (Members {size:2 ...})?
  2. Could you enable DEBUG logging for de.codecentric.boot.admin.server.notify on both nodes?
    For one status change, does each node log "Triggering notifiers for ..." or does one of them
    log "Notifications already sent ..."?
  3. What does DelayingMailNotifier do? If it delays or schedules the mail on its own, it might
    be sending outside of the dedup check.
  4. Do you have a custom Hazelcast map config for spring-boot-admin-sent-notifications
    (eviction, TTL)?

Thanks again!

Hello @YadavKshitiz

* latest SBA version and Hazelcast shows both the nodes

* I need to redeploy it, but it's possible

* delays the notification by 5 minutes by scheduling a job for it. Of course, if in the meantime the application recovered (that's the whole point in delaying them), such job gets cancelled

* no. Only the one shared in the docs

Thanks, that's really helpful! Your setup looks fine to me, and since your notifiers are @beans I don't think my PR (which only affects notifiers configured via properties) is what you're hitting. I've changed it to "Related to" so this issue stays open.

Your DelayingMailNotifier is my main suspect now: only the node that "wins" an event in Hazelcast calls notify(), so the delayed job (and the cancel on recovery) only exists on that node. A recovery handled by the other node wouldn't cancel it.

Could you help me with two things?

  1. When you get the duplicate mails, are they sent at the same time, or ~5 minutes apart? And are they sent even when the app recovered before the 5 minutes?
  2. Could you share (roughly) how DelayingMailNotifier schedules and cancels the job, e.g. which method it overrides (notify / doNotify / shouldNotify)?

When you redeploy, DEBUG logging for de.codecentric.boot.admin.server.notify on both nodes would be great, plus a note of whether each node logs "Triggering notifiers for ..." for the same event.

@cdprete

cdprete commented Sep 30, 2026

Copy link
Copy Markdown
Contributor

Hello @YadavKshitiz.

Below the implementation of the notifier:

import de.codecentric.boot.admin.server.domain.entities.Instance;
import de.codecentric.boot.admin.server.domain.entities.InstanceRepository;
import de.codecentric.boot.admin.server.domain.events.InstanceEvent;
import de.codecentric.boot.admin.server.domain.values.InstanceId;
import de.codecentric.boot.admin.server.notify.MailNotifier;
import jakarta.annotation.PostConstruct;
import jakarta.annotation.PreDestroy;
import jakarta.validation.constraints.NotNull;
import java.time.Duration;
import java.time.LocalDate;
import java.util.Map;
import java.util.Set;
import java.util.concurrent.ConcurrentHashMap;
import org.hibernate.validator.constraints.time.DurationMin;
import org.jspecify.annotations.NonNull;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.format.datetime.standard.DurationFormatterUtils;
import org.springframework.mail.javamail.JavaMailSender;
import org.thymeleaf.TemplateEngine;
import reactor.core.Disposable;
import reactor.core.publisher.Mono;

import static java.time.DayOfWeek.FRIDAY;
import static java.time.DayOfWeek.MONDAY;
import static java.time.LocalDate.now;

import static com.sixgroup.cit.fmd.zookeepers.monitoring.notification.DelayingMailNotifier.DayWindow.EVERYDAY;
import static de.codecentric.boot.admin.server.domain.values.StatusInfo.STATUS_OFFLINE;
import static de.codecentric.boot.admin.server.domain.values.StatusInfo.STATUS_RESTRICTED;
import static de.codecentric.boot.admin.server.domain.values.StatusInfo.STATUS_UNKNOWN;
import static de.codecentric.boot.admin.server.domain.values.StatusInfo.STATUS_UP;
import static org.springframework.format.annotation.DurationFormat.Style.COMPOSITE;
import static reactor.core.publisher.Mono.empty;
import static reactor.core.scheduler.Schedulers.boundedElastic;

public class DelayingMailNotifier extends MailNotifier {
    private static final Logger log = LoggerFactory.getLogger(DelayingMailNotifier.class);

    private static final Set<String> OK_STATUSES = Set.of(STATUS_UP, STATUS_UNKNOWN, STATUS_OFFLINE, STATUS_RESTRICTED);
    private static final String DELAY_PROPERTY_KEY = "delay";
    private static final String OK_STATUSES_PROPERTY_KEY = "okStatuses";

    private final InstanceRepository repository;
    private final Map<InstanceId, Disposable> pendingInstanceNotifications;

    @NotNull private DayWindow dayWindow;
    @NotNull @DurationMin(seconds = 1) private Duration delay;

    public DelayingMailNotifier(JavaMailSender mailSender, InstanceRepository repository, TemplateEngine templateEngine) {
        super(mailSender, repository, templateEngine);

        this.repository = repository;
        this.dayWindow = EVERYDAY;
        this.delay = Duration.ofMinutes(5);
        this.pendingInstanceNotifications = new ConcurrentHashMap<>();
    }

    @PostConstruct
    void propagateAdditionalProperties() {
        Map<String, Object> additionalProperties = getAdditionalProperties();
        additionalProperties.put(DELAY_PROPERTY_KEY, DurationFormatterUtils.print(delay, COMPOSITE));
        additionalProperties.put(OK_STATUSES_PROPERTY_KEY, OK_STATUSES);
    }

    @PreDestroy
    void disposePendingNotifications() {
        pendingInstanceNotifications.keySet().forEach(this::cancelPendingNotification);
        pendingInstanceNotifications.clear();
    }

    // Visible for testing
    Map<InstanceId, Disposable> getPendingInstanceNotifications() {
        return pendingInstanceNotifications;
    }

    public Duration getDelay() {
        return delay;
    }

    public void setDelay(Duration delay) {
        this.delay = delay;
    }

    public DayWindow getDayWindow() {
        return dayWindow;
    }

    public void setDayWindow(DayWindow dayWindow) {
        this.dayWindow = dayWindow;
    }

    @Override
    protected boolean shouldNotify(@NonNull InstanceEvent event, @NonNull Instance instance) {
        return super.shouldNotify(event, instance) && dayWindow.isWithinWindow(now());
    }

    @NonNull
    @Override
    protected Mono<Void> doNotify(@NonNull InstanceEvent event, @NonNull Instance instance) {
        return isEnabled() ?
                cancelPendingNotification(event).then(scheduleDelayedNotification(event)) :
                Mono.empty();
    }

    private Mono<Void> scheduleDelayedNotification(InstanceEvent event) {
        return Mono.delay(delay)
                    .doOnSubscribe(s -> {
                        log.info("Scheduling delayed notification for instance {} with a delay of {}", event.getInstance(), delay);
                        // Update the last status before scheduling the notification so that the next run sees the pending status and it doesn't reschedule the notification again
                        updateLastStatus(event);
                        pendingInstanceNotifications.put(event.getInstance(), s::cancel);
                    })
                    .flatMap(_ -> doNotifyIfNotRecovered(event))
                    .doFinally(_ -> pendingInstanceNotifications.remove(event.getInstance()))
                    .then();

    }

    private Mono<Instance> getInstanceIfNotRecovered(InstanceId id) {
        return repository
                .find(id)
                .filter(instance -> !OK_STATUSES.contains(instance.getStatusInfo().getStatus()));
    }

    private Mono<Void> doNotifyIfNotRecovered(InstanceEvent event) {
        return getInstanceIfNotRecovered(event.getInstance())
                .switchIfEmpty(signalRecovery(event).then(empty()))
                .flatMap(i -> sendNotification(event, i));
    }

    private Mono<Void> sendNotification(InstanceEvent event, Instance instance) {
        return super
                .doNotify(event, instance)
                .subscribeOn(boundedElastic())
                .doOnSubscribe(_ -> log.info("Performing delayed notification for instance {} since it didn't recover after {}", event.getInstance(), delay));
    }

    private Mono<Void> signalRecovery(InstanceEvent event) {
        return cancelPendingNotification(event).doOnSubscribe(_ -> log.info("Instance {} has recovered before the delay period. Cancelling the pending notification.", event.getInstance()));
    }

    private Mono<Void> cancelPendingNotification(InstanceEvent event) {
        return Mono.fromRunnable(() -> cancelPendingNotification(event.getInstance()));
    }

    private void cancelPendingNotification(InstanceId id) {
        Disposable existingJob = pendingInstanceNotifications.remove(id);
        if(existingJob != null && !existingJob.isDisposed()) {
            existingJob.dispose();
        }
    }

    public enum DayWindow {
        EVERYDAY {
            @Override
            boolean isWithinWindow(LocalDate date) {
                return true;
            }
        }, WEEKDAYS {
            private static final int START_VALUE = MONDAY.getValue();
            private static final int END_VALUE = FRIDAY.getValue();

            @Override
            boolean isWithinWindow(LocalDate date) {
                int dayOfWeekValue = date.getDayOfWeek().getValue();

                return dayOfWeekValue >= START_VALUE && dayOfWeekValue <= END_VALUE;
            }
        };

        abstract boolean isWithinWindow(LocalDate date);
    }
}
  • the notifications are sent after 5 minutes, but I receive both the duplicates at the same time.
  • see code above

@YadavKshitiz

Copy link
Copy Markdown
Author

Hello @YadavKshitiz.

Below the implementation of the notifier:

import de.codecentric.boot.admin.server.domain.entities.Instance;
import de.codecentric.boot.admin.server.domain.entities.InstanceRepository;
import de.codecentric.boot.admin.server.domain.events.InstanceEvent;
import de.codecentric.boot.admin.server.domain.values.InstanceId;
import de.codecentric.boot.admin.server.notify.MailNotifier;
import jakarta.annotation.PostConstruct;
import jakarta.annotation.PreDestroy;
import jakarta.validation.constraints.NotNull;
import java.time.Duration;
import java.time.LocalDate;
import java.util.Map;
import java.util.Set;
import java.util.concurrent.ConcurrentHashMap;
import org.hibernate.validator.constraints.time.DurationMin;
import org.jspecify.annotations.NonNull;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.format.datetime.standard.DurationFormatterUtils;
import org.springframework.mail.javamail.JavaMailSender;
import org.thymeleaf.TemplateEngine;
import reactor.core.Disposable;
import reactor.core.publisher.Mono;

import static java.time.DayOfWeek.FRIDAY;
import static java.time.DayOfWeek.MONDAY;
import static java.time.LocalDate.now;

import static com.sixgroup.cit.fmd.zookeepers.monitoring.notification.DelayingMailNotifier.DayWindow.EVERYDAY;
import static de.codecentric.boot.admin.server.domain.values.StatusInfo.STATUS_OFFLINE;
import static de.codecentric.boot.admin.server.domain.values.StatusInfo.STATUS_RESTRICTED;
import static de.codecentric.boot.admin.server.domain.values.StatusInfo.STATUS_UNKNOWN;
import static de.codecentric.boot.admin.server.domain.values.StatusInfo.STATUS_UP;
import static org.springframework.format.annotation.DurationFormat.Style.COMPOSITE;
import static reactor.core.publisher.Mono.empty;
import static reactor.core.scheduler.Schedulers.boundedElastic;

public class DelayingMailNotifier extends MailNotifier {
    private static final Logger log = LoggerFactory.getLogger(DelayingMailNotifier.class);

    private static final Set<String> OK_STATUSES = Set.of(STATUS_UP, STATUS_UNKNOWN, STATUS_OFFLINE, STATUS_RESTRICTED);
    private static final String DELAY_PROPERTY_KEY = "delay";
    private static final String OK_STATUSES_PROPERTY_KEY = "okStatuses";

    private final InstanceRepository repository;
    private final Map<InstanceId, Disposable> pendingInstanceNotifications;

    @NotNull private DayWindow dayWindow;
    @NotNull @DurationMin(seconds = 1) private Duration delay;

    public DelayingMailNotifier(JavaMailSender mailSender, InstanceRepository repository, TemplateEngine templateEngine) {
        super(mailSender, repository, templateEngine);

        this.repository = repository;
        this.dayWindow = EVERYDAY;
        this.delay = Duration.ofMinutes(5);
        this.pendingInstanceNotifications = new ConcurrentHashMap<>();
    }

    @PostConstruct
    void propagateAdditionalProperties() {
        Map<String, Object> additionalProperties = getAdditionalProperties();
        additionalProperties.put(DELAY_PROPERTY_KEY, DurationFormatterUtils.print(delay, COMPOSITE));
        additionalProperties.put(OK_STATUSES_PROPERTY_KEY, OK_STATUSES);
    }

    @PreDestroy
    void disposePendingNotifications() {
        pendingInstanceNotifications.keySet().forEach(this::cancelPendingNotification);
        pendingInstanceNotifications.clear();
    }

    // Visible for testing
    Map<InstanceId, Disposable> getPendingInstanceNotifications() {
        return pendingInstanceNotifications;
    }

    public Duration getDelay() {
        return delay;
    }

    public void setDelay(Duration delay) {
        this.delay = delay;
    }

    public DayWindow getDayWindow() {
        return dayWindow;
    }

    public void setDayWindow(DayWindow dayWindow) {
        this.dayWindow = dayWindow;
    }

    @Override
    protected boolean shouldNotify(@NonNull InstanceEvent event, @NonNull Instance instance) {
        return super.shouldNotify(event, instance) && dayWindow.isWithinWindow(now());
    }

    @NonNull
    @Override
    protected Mono<Void> doNotify(@NonNull InstanceEvent event, @NonNull Instance instance) {
        return isEnabled() ?
                cancelPendingNotification(event).then(scheduleDelayedNotification(event)) :
                Mono.empty();
    }

    private Mono<Void> scheduleDelayedNotification(InstanceEvent event) {
        return Mono.delay(delay)
                    .doOnSubscribe(s -> {
                        log.info("Scheduling delayed notification for instance {} with a delay of {}", event.getInstance(), delay);
                        // Update the last status before scheduling the notification so that the next run sees the pending status and it doesn't reschedule the notification again
                        updateLastStatus(event);
                        pendingInstanceNotifications.put(event.getInstance(), s::cancel);
                    })
                    .flatMap(_ -> doNotifyIfNotRecovered(event))
                    .doFinally(_ -> pendingInstanceNotifications.remove(event.getInstance()))
                    .then();

    }

    private Mono<Instance> getInstanceIfNotRecovered(InstanceId id) {
        return repository
                .find(id)
                .filter(instance -> !OK_STATUSES.contains(instance.getStatusInfo().getStatus()));
    }

    private Mono<Void> doNotifyIfNotRecovered(InstanceEvent event) {
        return getInstanceIfNotRecovered(event.getInstance())
                .switchIfEmpty(signalRecovery(event).then(empty()))
                .flatMap(i -> sendNotification(event, i));
    }

    private Mono<Void> sendNotification(InstanceEvent event, Instance instance) {
        return super
                .doNotify(event, instance)
                .subscribeOn(boundedElastic())
                .doOnSubscribe(_ -> log.info("Performing delayed notification for instance {} since it didn't recover after {}", event.getInstance(), delay));
    }

    private Mono<Void> signalRecovery(InstanceEvent event) {
        return cancelPendingNotification(event).doOnSubscribe(_ -> log.info("Instance {} has recovered before the delay period. Cancelling the pending notification.", event.getInstance()));
    }

    private Mono<Void> cancelPendingNotification(InstanceEvent event) {
        return Mono.fromRunnable(() -> cancelPendingNotification(event.getInstance()));
    }

    private void cancelPendingNotification(InstanceId id) {
        Disposable existingJob = pendingInstanceNotifications.remove(id);
        if(existingJob != null && !existingJob.isDisposed()) {
            existingJob.dispose();
        }
    }

    public enum DayWindow {
        EVERYDAY {
            @Override
            boolean isWithinWindow(LocalDate date) {
                return true;
            }
        }, WEEKDAYS {
            private static final int START_VALUE = MONDAY.getValue();
            private static final int END_VALUE = FRIDAY.getValue();

            @Override
            boolean isWithinWindow(LocalDate date) {
                int dayOfWeekValue = date.getDayOfWeek().getValue();

                return dayOfWeekValue >= START_VALUE && dayOfWeekValue <= END_VALUE;
            }
        };

        abstract boolean isWithinWindow(LocalDate date);
    }
}
* the notifications are sent after 5 minutes, but I receive both the duplicates at the same time.

* see code above

Hi @cdprete, thanks for the code and the details, that helped a lot!

I have a theory but no proof yet, so I'd really appreciate it if you could test my branch once. The Hazelcast check decides which node handles each event, but the "last known status" is kept per node. If two events for the same outage get claimed by different nodes, the second node doesn't know the first one already saw DOWN, so it schedules its own delayed job. Both jobs then fire ~5 minutes later and send a mail at the same time. My PR shares that last
status across nodes, which should stop the second job.

Branch: https://github.com/YadavKshitiz/spring-boot-admin/tree/fix/5716-hazelcast-notification-dedup

Could you try it on both nodes and let me know?

  1. Build the server from that branch (./mvnw -DskipTests install in the repo root) and use that version on both nodes. Your Hazelcast config from the docs works unchanged; the new map spring-boot-admin-last-statuses is created automatically.
  2. Trigger a status change (for example stop a client) and check whether you still get duplicate mails.
  3. If you still get duplicates, DEBUG logging for de.codecentric.boot.admin.server.notify on both nodes would help a lot. In particular the "Triggering notifiers for ..." and "Scheduling delayed notification ..." lines, with the timestamps and which node logged each.

Either result is useful, so no worries if it doesn't fix it. It will tell us where to look next.
Thanks a lot for your time!

@cdprete

cdprete commented Sep 30, 2026 via email

Copy link
Copy Markdown
Contributor

@YadavKshitiz

Copy link
Copy Markdown
Author

Yes, you're right, and thanks for pointing that out and no rush on testing, take the time you need.

@cdprete

cdprete commented Oct 2, 2026

Copy link
Copy Markdown
Contributor

I checked shortly your PR and the way you implemented the map handling seems an overkill to me, to be honest.

I would have personally implemented in the same way @SteKoe has done in #5737.

@YadavKshitiz

Copy link
Copy Markdown
Author

I will check again

@YadavKshitiz

Copy link
Copy Markdown
Author

I checked shortly your PR and the way you implemented the map handling seems an overkill to me, to be honest.

I would have personally implemented in the same way @SteKoe has done in #5737.

Thanks for pointing this out, and I really appreciate your feedback on my code. Here's what I looked into. I read through #5737 and I like the approach. You're right that my version is heavier than it needs to be.

What I'm thinking of changing: AbstractStatusChangeNotifier would just take a plain ConcurrentMap<InstanceId, String>, with a normal ConcurrentHashMap as the default, the same way #5737 does it. The Hazelcast config would then hand each notifier its own IMap, named after the notifier bean (something like "spring-boot-admin-last-statuses-"). That lets me drop the LastStatusStore wrapper, the key prefix and the extra property. I'd keep one map per notifier because otherwise two notifiers (say Slack and Mail) would overwrite each other's last status.

Does that match what you had in mind? And when you said the map handling felt like overkill, did you
mean the sent-notifications bean in the first commit, or the last-status part? One more thing I noticed, in case it's useful: #5737 creates its bean under @ConditionalOnBean(Notifier.class) in the Hazelcast auto-config. That condition is evaluated before property-configured notifiers exist, which is the same problem my first commit fixes, so it might only work for notifiers declared as @bean. I haven't run it, so I may be wrong.

@cdprete

cdprete commented Oct 7, 2026

Copy link
Copy Markdown
Contributor

Please align with the team about the issue you mention it's fixed with your first commit and, in general, on how to proceed. For example @SteKoe or @ulischulte.

To me, as a consumer, given that each notifier should keep its state, I think it would be enough that the notifiers expose their ConcurrentMap so that I can define the bean manually from my side and pass the Hazelcast one, if needed. Of course, this needs to be properly documented.
I wouldn't overcomplicate SBA to have a map for each notifier, to be honest. But, again, it's up to the team to say how to proceed.

@YadavKshitiz

Copy link
Copy Markdown
Author

Hi @SteKoe, @ulischulte,

I could use your help deciding how to move forward with this PR. It currently mixes two separate things, and I'd like to untangle them before changing anything else.

  1. Trigger wiring (first commit)
    While looking into Bug: Multiple notifications from multiple instances #5716 I found that with Hazelcast, HazelcastNotificationTrigger is never created when the notifiers come from spring.boot.admin.notify.* properties (Slack, Mail, ...). The reason is that AdminServerHazelcastAutoConfiguration is evaluated before those notifiers exist, so its @ConditionalOnBean(Notifier.class) check fails. I added a test that fails on master and passes with the fix. It isn't the cause of @cdprete's duplicates, since his notifiers are declared as @Bean, so it's a related but separate bug. Would you prefer this as its own PR, or should I open a separate issue for it first?

  2. Per-node last status (second commit)
    AbstractStatusChangeNotifier keeps lastStatuses in a local HashMap, so nodes can disagree about the previous status. My first attempt wired this up automatically, and it turned out heavier than it needs to be. @cdprete suggested something much simpler: let notifiers take a ConcurrentMap (a setter, with ConcurrentHashMap as the default), so anyone running a cluster can pass in a Hazelcast IMap from their own bean. We'd document that on the clustering page, with no automatic wiring. I see three options:
    (a) the setter plus docs (b) automatic wiring with one map per notifier (c) leave it as it is

Which would you prefer? I'm happy to go with whatever fits the project best. One more thing, and I may well be wrong since I haven't run it: #5737 seems to create its FilteringNotifier under @ConditionalOnBean(Notifier.class) inside the Hazelcast auto-config, which could have the same ordering problem as point 1 for property-configured notifiers. It might be worth a quick look.

Thanks a lot for your time!

@SteKoe

SteKoe commented Oct 16, 2026

Copy link
Copy Markdown
Contributor

Hi @YadavKshitiz, thanks for the detailed analysis!

I checked both points against current master and you are right:

  • Trigger wiring: With Hazelcast and notifiers configured only via spring.boot.admin.notify.*, a plain NotificationTrigger is created instead of the HazelcastNotificationTrigger. So every node sends its own notifications.
  • fix: share notification filters across Hazelcast cluster nodes #5737: It's worse than you thought. With a single user-defined @Bean notifier, the context doesn't start at all (BeanCurrentlyInCreationException: filteringNotifier → compositeNotifier → filteringNotifier). With property-only notifiers, no FilteringNotifier is created. Thanks for spotting it!

We'd like to split this up into two PRs:

1. Notifier wiring with Hazelcast

  • your first commit: resolve the HazelcastNotificationTrigger in AdminServerNotifierAutoConfiguration via ObjectProvider, so it no longer depends on auto-configuration order
  • please also fix the fix: share notification filters across Hazelcast cluster nodes #5737 regression there, since it's the same problem. The FilteringNotifier should not be created via @ConditionalOnBean(Notifier.class) in the Hazelcast auto-config. Getting the IMap the same way as the sent-notifications map should work.
  • tests for both cases: a single @Bean notifier and notifiers configured via properties

2. Shared last status
We'd go with option (a), as @cdprete suggested:

  • lastStatuses in AbstractStatusChangeNotifier becomes a ConcurrentMap<InstanceId, String> with ConcurrentHashMap as the default
  • add a setter, so users can pass in a Hazelcast IMap in their own bean
  • document it on the clustering page

Please drop LastStatusStore, the BeanPostProcessor and the spring.boot.admin.hazelcast.last-statuses property. We'd rather not change every notifier automatically. If there's demand later, we can still add that.

We'll keep #5716 open until @cdprete can confirm whether this fixes his duplicates.

Thanks again!

@YadavKshitiz

Copy link
Copy Markdown
Author

Hello @SteKoe

Thanks for your answer! I will work on the issue and also i will go as @cdprete suggested for shared last status.

@cdprete

cdprete commented Oct 16, 2026

Copy link
Copy Markdown
Contributor

Hi @YadavKshitiz, thanks for the detailed analysis!

I checked both points against current master and you are right:

  • Trigger wiring: With Hazelcast and notifiers configured only via spring.boot.admin.notify.*, a plain NotificationTrigger is created instead of the HazelcastNotificationTrigger. So every node sends its own notifications.
  • fix: share notification filters across Hazelcast cluster nodes #5737: It's worse than you thought. With a single user-defined @Bean notifier, the context doesn't start at all (BeanCurrentlyInCreationException: filteringNotifier → compositeNotifier → filteringNotifier). With property-only notifiers, no FilteringNotifier is created. Thanks for spotting it!

We'd like to split this up into two PRs:

1. Notifier wiring with Hazelcast

  • your first commit: resolve the HazelcastNotificationTrigger in AdminServerNotifierAutoConfiguration via ObjectProvider, so it no longer depends on auto-configuration order
  • please also fix the fix: share notification filters across Hazelcast cluster nodes #5737 regression there, since it's the same problem. The FilteringNotifier should not be created via @ConditionalOnBean(Notifier.class) in the Hazelcast auto-config. Getting the IMap the same way as the sent-notifications map should work.
  • tests for both cases: a single @Bean notifier and notifiers configured via properties

2. Shared last status We'd go with option (a), as @cdprete suggested:

  • lastStatuses in AbstractStatusChangeNotifier becomes a ConcurrentMap<InstanceId, String> with ConcurrentHashMap as the default
  • add a setter, so users can pass in a Hazelcast IMap in their own bean
  • document it on the clustering page

Please drop LastStatusStore, the BeanPostProcessor and the spring.boot.admin.hazelcast.last-statuses property. We'd rather not change every notifier automatically. If there's demand later, we can still add that.

We'll keep #5716 open until @cdprete can confirm whether this fixes his duplicates.

Thanks again!

@SteKoe for safety and consistency, I would rather go with constructor injection rather than the setter one.

@YadavKshitiz
YadavKshitiz force-pushed the fix/5716-hazelcast-notification-dedup branch from 4ddacb6 to 0bdb820 Compare October 16, 2026 10:52
@YadavKshitiz

Copy link
Copy Markdown
Author

Hi @SteKoe, I've pushed the changes for the wiring PR. Summary:

  • The HazelcastNotificationTrigger is now resolved in AdminServerNotifierAutoConfiguration via ObjectProvider, so it no longer depends on auto-configuration order.
  • I removed the auto-created FilteringNotifier and its @ConditionalOnBean(Notifier.class) nested class. The filters IMap is now a lazy bean (notificationFiltersMap), the same way as the sent-notifications map. Users pass it to the FilteringNotifier constructor in their own bean, and I documented that on the clustering page. I did it this way because a FilteringNotifier created by the auto-config is itself a Notifier, so the CompositeNotifier picks it up and we get the cycle again.
  • Added tests for a single @bean notifier and for notifiers configured via properties. Everything passes locally.

Is that what you had in mind, or did you want a FilteringNotifier still created automatically?

For the second PR (shared last status), I'll make lastStatuses a ConcurrentMap with ConcurrentHashMap as the default and pass it through a constructor, as @cdprete suggested. AbstractStatusChangeNotifier has about 15 subclasses, including MailNotifier, which @cdprete extends. Should I add a map constructor to every notifier, or only to the abstract class plus the commonly extended ones such as MailNotifier? And do you want a setter as well?

Thanks again for the guidance!

@YadavKshitiz

Copy link
Copy Markdown
Author

@SteKoe Please Tell, so i can continue my work

@Bean(name = SENT_NOTIFICATIONS_BEAN_NAME)
@Lazy
@ConditionalOnMissingBean(name = SENT_NOTIFICATIONS_BEAN_NAME)
public IMap<InstanceId, Long> sentNotificationsMap(HazelcastInstance hazelcastInstance,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Like for the other maps already present, I don't think it's necessary to have them declared as beans explicitly.

@SteKoe

SteKoe commented Oct 23, 2026

Copy link
Copy Markdown
Contributor

Thanks @YadavKshitiz, this looks good. Decisions on the open points:

FilteringNotifier: Removing the auto-created one is fine. It came from #5737, which hasn't been released, so compared to 4.1.4 this isn't a breaking change. It fixes that regression. This PR should be merged before the next release, so that #5737 never ships without it. Users who want filters in a cluster declare their own FilteringNotifier and pass notificationFiltersMap. If we want to create it automatically in the future, that needs a separate design (the cycle with CompositeNotifier), not part of this PR. The only public API removed compared to 4.1.4 is the nested NotifierTriggerConfiguration, which is an internal auto-configuration detail. Please mention it in the release notes.

Map beans: We keep sentNotificationsMap and notificationFiltersMap as beans. @cdprete, I see why they look different from eventStore. The notifier auto-configuration must not reference Hazelcast classes, since Hazelcast is optional, so it can only get the map as a plain ConcurrentMap through an ObjectProvider. That requires a bean, and it keeps the Hazelcast conditions in one place.

Please address these before merge:

  • Fix the Javadoc on sentNotificationsMap. The Hazelcast config is @AutoConfigureBefore the notifier config. What matters is that the ObjectProvider is resolved lazily when the trigger is created.
  • In the clustering docs, mention that a FilteringNotifier declared as @Primary can create a circular dependency with CompositeNotifier if it wraps a notifier collected there, and show how to avoid it.

This PR stays "Related to #5716". We keep the issue open until @cdprete confirms the duplicates are gone, which depends on the shared last status in the follow-up PR.

@cdprete

cdprete commented Oct 23, 2026

Copy link
Copy Markdown
Contributor

Thanks @YadavKshitiz, this looks good. Decisions on the open points:

FilteringNotifier: Removing the auto-created one is fine. It came from #5737, which hasn't been released, so compared to 4.1.4 this isn't a breaking change. It fixes that regression. This PR should be merged before the next release, so that #5737 never ships without it. Users who want filters in a cluster declare their own FilteringNotifier and pass notificationFiltersMap. If we want to create it automatically in the future, that needs a separate design (the cycle with CompositeNotifier), not part of this PR. The only public API removed compared to 4.1.4 is the nested NotifierTriggerConfiguration, which is an internal auto-configuration detail. Please mention it in the release notes.

Map beans: We keep sentNotificationsMap and notificationFiltersMap as beans. @cdprete, I see why they look different from eventStore. The notifier auto-configuration must not reference Hazelcast classes, since Hazelcast is optional, so it can only get the map as a plain ConcurrentMap through an ObjectProvider. That requires a bean, and it keeps the Hazelcast conditions in one place.

Please address these before merge:

  • Fix the Javadoc on sentNotificationsMap. The Hazelcast config is @AutoConfigureBefore the notifier config. What matters is that the ObjectProvider is resolved lazily when the trigger is created.
  • In the clustering docs, mention that a FilteringNotifier declared as @Primary can create a circular dependency with CompositeNotifier if it wraps a notifier collected there, and show how to avoid it.

This PR stays "Related to #5716". We keep the issue open until @cdprete confirms the duplicates are gone, which depends on the shared last status in the follow-up PR.

@SteKoe I don't have currently the time to properly test this. If it looks ok to you, feel free to merge it and release it.

I'll report a new bug ticket if the issue then still occurs with the new released version.

@YadavKshitiz
YadavKshitiz force-pushed the fix/5716-hazelcast-notification-dedup branch from 0bdb820 to b2a9b29 Compare October 23, 2026 15:28
@YadavKshitiz YadavKshitiz changed the title fix(server): stop duplicate notifications in a Hazelcast cluster (#5716) fix(server): resolve Hazelcast notification trigger independent of auto-configuration order Oct 24, 2026
@YadavKshitiz

YadavKshitiz commented Oct 24, 2026 •

Copy link
Copy Markdown
Author

Thanks @YadavKshitiz, this looks good. Decisions on the open points:

FilteringNotifier: Removing the auto-created one is fine. It came from #5737, which hasn't been released, so compared to 4.1.4 this isn't a breaking change. It fixes that regression. This PR should be merged before the next release, so that #5737 never ships without it. Users who want filters in a cluster declare their own FilteringNotifier and pass notificationFiltersMap. If we want to create it automatically in the future, that needs a separate design (the cycle with CompositeNotifier), not part of this PR. The only public API removed compared to 4.1.4 is the nested NotifierTriggerConfiguration, which is an internal auto-configuration detail. Please mention it in the release notes.

Map beans: We keep sentNotificationsMap and notificationFiltersMap as beans. @cdprete, I see why they look different from eventStore. The notifier auto-configuration must not reference Hazelcast classes, since Hazelcast is optional, so it can only get the map as a plain ConcurrentMap through an ObjectProvider. That requires a bean, and it keeps the Hazelcast conditions in one place.

Please address these before merge:

* Fix the Javadoc on `sentNotificationsMap`. The Hazelcast config is `@AutoConfigureBefore` the notifier config. What matters is that the `ObjectProvider` is resolved lazily when the trigger is created.

* In the clustering docs, mention that a `FilteringNotifier` declared as `@Primary` can create a circular dependency with `CompositeNotifier` if it wraps a notifier collected there, and show how to avoid it.

This PR stays "Related to #5716". We keep the issue open until @cdprete confirms the duplicates are gone, which depends on the shared last status in the follow-up PR.

Thanks @SteKoe, both points are addressed and pushed:

  • The Javadoc on sentNotificationsMap now explains that the map is a bean because the notifier auto-configuration can't reference Hazelcast classes, and that the ObjectProvider is resolved lazily when the trigger is created.
  • The clustering docs now have the warning about the circular dependency with CompositeNotifier. I verified with a throwaway test that a FilteringNotifier taking a plain Notifier parameter, without @Primary, fails with a BeanCurrentlyInCreationException. Your comment mentioned @Primary, so is there a @Primary setup you had in mind that I should cover? I added a test for a @Primary FilteringNotifier wrapping a property-configured notifier, since the docs example does that.

I updated the PR title and description to the new scope, with the release-notes entry in it. I saw there's an upgrading section under docs/11-upgrading; do you want an entry there too? For the follow-up PR (shared last status), could you confirm: a map constructor on every notifier, or only on the abstract class plus commonly extended ones like MailNotifier, and do you still want a setter as well? I'll also need the @since version.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants