btmon server mode hangs

[mattster98]

Been trying to troubleshoot frequent hangs of btmon in server mode. btmon runs from one to a few hours before it stops outputting to influxdb. I can't see a specific pattern and the log output isn't helpful from what I can see.

I tried getting more details by doing a python trace:

sudo python -m trace --trace /usr/local/mtools/bin/btmon.py --config-file=/usr/local/mtools/etc/btmon.cfg --debug

(tail of trace output attached)

I have three GEM panels - one sending directly to btmon via its wifi interface. The other two are serial attached to a dashbox, which is forwarding to btmon. All hitting the same btmon port running in a single instance. The only suspicion I have is that the wifi-connected GEM has a spotty wifi signal from time to time. I'm not sure how that would lead to btmon hanging eventually, but it's a thought if it gets stuck mid-packet or something and can't recover. I haven't had any luck improving the wifi signal.

Any assistance in troubleshooting this is greatly appreciated - I don't see anything so far that tell me where to go next.

btmon_trace.txt

[mattster98]

One note - the first trace output I provided is unique in that it appears to be in some sort of retry loop. I killed it before it completely stopped retrying. I have a trace output of the complete hang as well, attached here.
btmon_trace2.txt

[mattster98]

Update - been trying to align wifi performance and these issues and still not conclusive. Looking at the data, and netstat it does seem to be isolated to the Wifi GEM, so I tried breaking it out into a separate instance, and putting it back into server mode with the separate btmon instance in client mode connecting to it. So far (~6 hours) so good. Both instances are up and running with good data. Will follow up after 24 hours as that would be a record for it running continuously.