Large dataclips can still cause runs to be lost

[josephjclark]

Despite a lot of improvements to memory handling and payload processing, I think we're still losing lost runs with large dataclips.

I'm looking at impact water which is regularly building 9.8mb dataclips and lost runs.

I'm not sure sure exactly where the error lies - but I think when processing this 10mb dataclip to send to lightning, either the connection is actually timing out or some process (lightning side or worker side) is exceeded a memory limit and giving in.

The worst thing is that this is dying silently and causing the websocket to disconnect, resulting in a lost run

A few things jump to mind:

• Claude points out the state objects are double encoded in to JSOn strings. bloating the size A 9mb dataclip in memory might acutally serialize to 10mb or more in the event sent to lightning because of the nested JSON escape characters
• We are not getting good sentry reports when these events timeout - that needs fixing

This is the underlying cause behind #1072 and #1179. Both those issues describe symptoms of this one underlying thing. They both describe a case where an event fails to send and then bad things happen. This issue is the reason teh events fail.

[josephjclark]

Here's what's acutally going on here:

• The worker claims to support dataclips up to 10mb
• In lightning, cowboy will process data frames up to 11mb
• When sending the final dataclip, we (wrongly I think) stringify it, bloating the size of the object by a significant %
• This means the worker might be sending dataclips which are more than 11mb on the wire
• When this happens, cowboy disconnects the socket entirely. The worker will reconnect but the step complete event will be forever lost

I think the cheapest safest fix right now is to remove that bloat, which I'll be looking at soon