The CoreDNS for most k8s distros have caching enabled by default. The wait_until_nginx_online() method is implemented when starting an analysis to wait for Nginx to come online, and as soon as it is, another call is made to it via the MessageBrokerAPI. Depending on the number of replicas there are for the CoreDNS, this could return a 502 error resulting in Failed to subscribe to message broker: ConnectError('[Errno -2] Name or service not known') to be logged, and the raise_error() method being called which sets the analysis to FAILED even though it was a DNS caching error. One CoreDNS replica finally returned a 200 for nginx, but the other replicas still have 502 in their caches so when the MessageBrokerClient is started with a fresh asyncio.run() and asks k8s if nginx exists, it says no and the analysis fails.
This doesn't happen much in clusters with single node k8s and/or 1 replica of CoreDNS, but larger, production k8s have multiple CoreDNS replicas.
Suggested fix: create a similar method as wait_until_nginx_online() and use that to call _connect() and get_self_config in MessageBrokerClient
The CoreDNS for most k8s distros have caching enabled by default. The
wait_until_nginx_online()method is implemented when starting an analysis to wait for Nginx to come online, and as soon as it is, another call is made to it via theMessageBrokerAPI. Depending on the number of replicas there are for the CoreDNS, this could return a 502 error resulting inFailed to subscribe to message broker: ConnectError('[Errno -2] Name or service not known')to be logged, and theraise_error()method being called which sets the analysis to FAILED even though it was a DNS caching error. One CoreDNS replica finally returned a 200 for nginx, but the other replicas still have 502 in their caches so when theMessageBrokerClientis started with a freshasyncio.run()and asks k8s if nginx exists, it says no and the analysis fails.This doesn't happen much in clusters with single node k8s and/or 1 replica of CoreDNS, but larger, production k8s have multiple CoreDNS replicas.
Suggested fix: create a similar method as
wait_until_nginx_online()and use that to call_connect()andget_self_configinMessageBrokerClient