Dropping websocket messages?

[nicksellen]

I was writing in a chat on karrot.world and noticed the message I sent wasn't appearing in the chat. It was there if I refreshed.

I dug a little deeper and here's the story:

• the message was being sent and saved
• it was not receiving the websocket message response it should be
• dev.karrot.world was also demonstrating this problem
• I assumed it was something to do with our recent switch to uvicorn, which it could still be, but so far looks like not
• I could not reproduce locally with uvicorn and lots of workers
• reducing to 1 worker seems to make it go away
• I found this comment in 2021 that was potentially related new connected channels may fail to receive message from group_send() django/channels#1244 (comment)
• there are a bunch of issues for channels_redis that seem related redis-py >=4.4.0 breaking changes django/channels_redis#348 RedisChannelLayer.receive() drops events due to asyncio.shield() in redis-py 4.5.4 django/channels_redis#366 - they point to the redis-py making changes which cause those, so we could pin the version a bit lower (<4.4, we have 4.5.5, which was upgraded from 4.3.4, 3 months ago in a383c8459)

I noticed channels redis has a beta RedisPubSubChannelLayer that can be used, and I wondered if this might magically work (as I could switch that out in prod to try), and so far so good... it might be that it uses a sufficiently differently mechanism and avoids the issue. Or, potentially the issue reappears, as it was not reliably always occuring.

Anyway, this is a dirty fix on the server for now...

[nicksellen]

This seems to be resolved now, and is deployed already for a few days.