http2 websockets leaks while using open ssl for ping and pong

[kerdemdemir]

I am listening ~100 web sockets from binance.com to keep up with live transactions(every single buy and sell order).

I have been having memory leak problems for a long time and have to restart my application after 3-4 hours. Because I am running out of my 32gb ram.

Now I want to listening order book data as well(supply and demand book) but my applications just run out of memory after 5 mins.

I use many different tools to find where is leak. The best was gprefd. I found out there were too many allocations from OpenSSL and Arsd.

I used gdb and put breakpoints to malloc to confirm that and the stack trace always showed OpenSSLSocket's send and receive.

I changed the "int pingFrequency = 5000;" to a very big number and now I can run my application for 6-8 hours without running out of memory. But the memory leak still continues because I just controlled the pings that we send but also my server(binance.com) sends me pong approximately every 3 minutes. And pongs uses the same OpenSSLSocket class and its leaking functions send&recieve.

I can attach the gdb stack outputs and gpref results if it is requested.

Thanks for the awesome Arsd library, I removed all vibe.d stuff and replaced it with Arsd :)

[adamdruppe]

I might have forgotten to call some openssl cleanup function... post what you have and i can see.

[kerdemdemir]

So stack looks like:

Breakpoint 1 at 0x7aa0186ad640: malloc. (48 locations)
(gdb) c
Continuing.
[Switching to Thread 0x7a9ffa6006c0 (LWP 767438)]

Thread 16 "dmarketcalls" hit Breakpoint 1.2, 0x00007aa019038cc0 in tc_malloc () from /lib/x86_64-linux-gnu/libtcmalloc.so.4
(gdb) bt
#0 0x00007aa019038cc0 in tc_malloc () from /lib/x86_64-linux-gnu/libtcmalloc.so.4
#1 0x00007aa018c237c5 in CRYPTO_zalloc () from /lib/x86_64-linux-gnu/libcrypto.so.3
#2 0x00007aa01973e463 in ?? () from /lib/x86_64-linux-gnu/libssl.so.3
#3 0x00007aa01977113c in ?? () from /lib/x86_64-linux-gnu/libssl.so.3
#4 0x00007aa019772361 in ?? () from /lib/x86_64-linux-gnu/libssl.so.3
#5 0x00007aa019752b4b in SSL_write () from /lib/x86_64-linux-gnu/libssl.so.3
#6 0x00006011327da290 in arsd.http2.OpenSSL.opDispatch!("SSL_write").opDispatch!(arsd.http2.SSL*, const(void), uint).opDispatch(arsd.http2.SSL, const(void), uint) (__param_2=6,
__param_1=0x7a9ffa5fd610, __param_0=0x60113c3fc000) at source/http2.d:3686
#7 0x00006011327d41bf in arsd.http2.OpenSslSocket.send(scope const(void)[], std.socket.SocketFlags) (this=0x7aa0117278d0, flags=, buf=...) at source/http2.d:4004
#8 0x00006011327d41fb in arsd.http2.OpenSslSocket.send(scope const(void)[]) (this=0x7aa0117278d0, buf=...) at source/http2.d:4024
#9 0x00006011327d6c12 in arsd.http2.WebSocket.llsend(ubyte[]) (this=0x7aa011726a00, d=...) at source/http2.d:4961
#10 0x00006011327d8e8e in arsd.http2.WebSocketFrame.send(scope void(ubyte[]) delegate) (this=..., llsend=...) at source/http2.d:5875
#11 0x00006011327d7499 in arsd.http2.WebSocket.pong(in ubyte[]) (this=0x7aa011726a00, data=...) at source/http2.d:5201
#12 0x00006011327d7caf in arsd.http2.WebSocket.processOnce() (this=0x7aa011726a00, __HID215=0x7a9ffa5fd920) at source/http2.d:5362
#13 0x00006011327d8066 in arsd.http2.WebSocket.readyToRead(arsd.http2.WebSocket) (sock=0x7aa011726a00) at source/http2.d:5472
#14 0x00006011327d84b2 in arsd.http2.WebSocket.eventLoop(shared(bool)) (localLoopExited=0x7aa00e9f18a0) at source/http2.d:5583
#15 0x00006011326c14b5 in Client.SocketHelper.ArsdMultiThreadSocket.ArsdMultiThreadSocket.InitCurrencies().captureFuction(int).__lambda2() (__capture=0x7aa00cc3e930)
at source/Client/SocketHelpers/ArsdMultiThreadSocket.d:71
#16 0x00006011326c14e1 in Client.SocketHelper.ArsdMultiThreadSocket.ArsdMultiThreadSocket.InitCurrencies().__lambda5() (__capture=0x7aa00cc778c0)
at source/Client/SocketHelpers/ArsdMultiThreadSocket.d:80
#17 0x00006011329d727d in core.thread.context.Callable.opCall() ()
#18 0x00006011329d6d5b in thread_entryPoint ()
#19 0x00007aa01869ca94 in start_thread (arg=) at ./nptl/pthread_create.c:447
#20 0x00007aa018729c3c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78

I put also break point to free which triggers time to time but way more less than mallocs:

Thread 24 "dmarketcalls" hit Breakpoint 2.2, 0x00007aa019036bc0 in tc_free () from /lib/x86_64-linux-gnu/libtcmalloc.so.4
(gdb) bt
#0 0x00007aa019036bc0 in tc_free () from /lib/x86_64-linux-gnu/libtcmalloc.so.4
#1 0x00007aa01973e8cd in ?? () from /lib/x86_64-linux-gnu/libssl.so.3
#2 0x00007aa019771b23 in ?? () from /lib/x86_64-linux-gnu/libssl.so.3
#3 0x00007aa019772361 in ?? () from /lib/x86_64-linux-gnu/libssl.so.3
#4 0x00007aa019752b4b in SSL_write () from /lib/x86_64-linux-gnu/libssl.so.3
#5 0x00006011327da290 in arsd.http2.OpenSSL.opDispatch!("SSL_write").opDispatch!(arsd.http2.SSL*, const(void), uint).opDispatch(arsd.http2.SSL, const(void), uint) (__param_2=6,
__param_1=0x7a9fff5fd610, __param_0=0x601141942000) at source/http2.d:3686
#6 0x00006011327d41bf in arsd.http2.OpenSslSocket.send(scope const(void)[], std.socket.SocketFlags) (this=0x7a9fda9b1c30, flags=, buf=...) at source/http2.d:4004
#7 0x00006011327d41fb in arsd.http2.OpenSslSocket.send(scope const(void)[]) (this=0x7a9fda9b1c30, buf=...) at source/http2.d:4024
#8 0x00006011327d6c12 in arsd.http2.WebSocket.llsend(ubyte[]) (this=0x7a9fda99b600, d=...) at source/http2.d:4961
#9 0x00006011327d8e8e in arsd.http2.WebSocketFrame.send(scope void(ubyte[]) delegate) (this=..., llsend=...) at source/http2.d:5875
#10 0x00006011327d7499 in arsd.http2.WebSocket.pong(in ubyte[]) (this=0x7a9fda99b600, data=...) at source/http2.d:5201
#11 0x00006011327d7caf in arsd.http2.WebSocket.processOnce() (this=0x7a9fda99b600, __HID215=0x7a9fff5fd920) at source/http2.d:5362
#12 0x00006011327d8066 in arsd.http2.WebSocket.readyToRead(arsd.http2.WebSocket) (sock=0x7a9fda99b600) at source/http2.d:5472
#13 0x00006011327d84b2 in arsd.http2.WebSocket.eventLoop(shared(bool)) (localLoopExited=0x7aa00e9f18a0) at source/http2.d:5583
#14 0x00006011326c14b5 in Client.SocketHelper.ArsdMultiThreadSocket.ArsdMultiThreadSocket.InitCurrencies().captureFuction(int).__lambda2() (__capture=0x7aa00cc45280)
at source/Client/SocketHelpers/ArsdMultiThreadSocket.d:71
#15 0x00006011326c14e1 in Client.SocketHelper.ArsdMultiThreadSocket.ArsdMultiThreadSocket.InitCurrencies().__lambda5() (__capture=0x7aa00cc778c0)
at source/Client/SocketHelpers/ArsdMultiThreadSocket.d:80
#16 0x00006011329d727d in core.thread.context.Callable.opCall() ()
#17 0x00006011329d6d5b in thread_entryPoint ()
#18 0x00007aa01869ca94 in start_thread (arg=) at ./nptl/pthread_create.c:447
#19 0x00007aa018729c3c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78

You can see a glimbs of malloc to free ratio here.

When my memory goes super high above 16gb I used https://gperftools.github.io/gperftools/heapprofile.html few times and so most of the mallocs were coming from pings. That was before I solve the issue with ping with my hack.
Now I bet I would see pong which I can not stop because my server documentation "https://developers.binance.com/docs/binance-spot-api-docs/web-socket-api" says if I dont respond pongs with in 10 minutes period they will shutdown the connection. So I can not unfortunately hack pongs.

Best Regards
Erdem

[adamdruppe]

huh so it is right inside the openssl functions. i know they keep some pieces around to chunk it right but.... huh

might have to check the docs again and see what im missing