Repository navigation
Django connection issue with RTM v2 client #1238
Description
Activity
- changed the title
[-]We are refitting to use the v2 RTM client and still observe some stale Django connection issues. Ie, new DB connections are made for each message that gets processed (thread), but they are apparently not being released. I have tried using the close_old_connections & also ensure_connection, but seems like it doesn't really help. I saw you did the changes for the slack-bolt thing here. Were there going to be similar changes to the v2 RTM client? Do you have any recommendations in the interim?[/-][+]Django connection issue with RTM v2 client[/+]on Jul 15, 2022 Hi @pbrackin, thanks for sharing this. The connection releasing logic may not work in the case where your message listener receives a message from its bot user itself: https://github.com/slackapi/python-slack-sdk/blob/v3.17.2/slack_sdk/rtm_v2/__init__.py#L162-L164
We are happy to add some customization points for this use case but, can you verify whether the approach works for you?
- Copy https://github.com/slackapi/bolt-python/blob/v1.14.1/slack_bolt/adapter/django/handler.py#L64-L71 into your code
- Copy https://github.com/slackapi/python-slack-sdk/blob/v3.17.2/slack_sdk/rtm_v2/__init__.py#L291-L307 and edit
process_messagemethod as below
def process_message(self): release_thread_local_connections() # ADDED try: raw_message = self.message_queue.get(timeout=1) if self.logger.level <= logging.DEBUG: self.logger.debug(f"A message dequeued (current queue size: {self.message_queue.qsize()})") if raw_message is not None: message: dict = {} if raw_message.startswith("{"): message = json.loads(raw_message) def _run_message_listeners(): self.run_message_listeners(message) self.message_workers.submit(_run_message_listeners) except Empty: pass finally: # ADDED release_thread_local_connections() # ADDED
If this works well, we will add two callbacks like
on_process_message_startandon_process_message_endto run release_thread_local_connections() without modifying the RTM client code.Thanks @seratch for following up on this. Sorry I didn't get to respond a little sooner. I will run this experiment when I get some time. But I wanted to also point out that I found a solution that seems to work ok. What I did was call
connections.close_all()at the end of my message handler. Also I callclose_old_connections()in my main loop which runs every 15 secs atm. We use MySQL for this project, and so I figured out that runningsudo mysqladmin -i 1 processlistwas a good way to see what is happening with the DB connections. Anyway, with those 2 Django calls in place, the connection list stopped growing with new messages coming in, and I no longer get the OperationalError(s) like "MySQL has gone away".
Before I figured out this solution, I was about to just move all the incoming messages to a Q and have the processing all done by a dedicated processing thread - which would probably be ideal from a DB connection standpoint since that would allow Django to recycle connections in the pool. The cost of that would be some additional delay in the processing for Q / d-Q stuff. But with my solution, everything seems ok now.
Anyway - let me know what you think of my approach. I will get back on adding in the code changes you suggested.@pbrackin Thanks for your reply. Most of the things you mentioned should be good ways to go but
What I did was call connections.close_all() at the end of my message handler.
This should be
close_old_connectionsfor safety. Refer to slackapi/bolt-python#509 for more details.Let me know if my above suggestions work for you. Once it is effective, we are happy to enhance the RTM v2 client to enable developers to pass callback functions for this use case.
We are refitting to use the v2 RTM client and still observe some stale Django connection issues. Ie, new DB connections are made for each message that gets processed (thread), but they are apparently not being released. I have tried using the close_old_connections & also ensure_connection, but seems like it doesn't really help. I saw you did the changes for the slack-bolt thing here. Were there going to be similar changes to the v2 RTM client? Do you have any recommendations in the interim?
Originally posted by @pbrackin in slackapi/bolt-python#280 (comment)