multidict 4.5 memory leak

[byashimov]

It seems like the recent version of multidict brakes aiohttp somehow.

Server:

from aiohttp import web

async def hello(request):
    return web.json_response(await request.json())

app = web.Application()
app.add_routes([web.post('/', hello)])
web.run_app(app)

Client:

import asyncio
import aiohttp

async def foo(times):
    data = {'foo': 1}
    async with aiohttp.ClientSession() as session:
        for x in range(times):
            resp = await session.post('http://localhost:8080', json=data)
            if not x % 100:
                print(await resp.json())

loop = asyncio.get_event_loop()
loop.run_until_complete(foo(100000))
loop.close()

Just after 100000 cycles this simple server takes 171M of ram and doesn't release it.
4.4.2 works like a charm.

python 3.6.4
aiohttp 3.4.4 but should affect other versions multidict>=4.0,<5.0

[webknjaz]

cc @iemelyanov

[aio-libs-bot]

GitMate.io thinks the contributor most likely able to help you is @asvetlov.

Possibly related issues are aio-libs/aiohttp#1656 (Memory leak), aio-libs/aiohttp#271 (Memory leak in request), aio-libs/aiohttp#133 (Memory leak when doing https request), aio-libs/aiohttp#1029 (Memory leak when doing https request), and aio-libs/aiohttp#1756 (Memory leak with aiohttp.request).

[webknjaz]

@byashimov if you think that it's a regression in multidict, you should report the problem there and post an example not using aiohttp.

[byashimov]

@webknjaz multidict downgrade fixes the issue. Please confirm it to downgrade the package version in setup file. I'll try to reproduce it for multidict issue tracker.

[webknjaz]

Hm.. could smth like https://twitter.com/VictorStinner/status/1062282676462333952 or cprofile help?

[webknjaz]

@byashimov the latest version is also compiled with the newest Cython

[skrawcz]

I can confirm, I am seeing this same memory leak using 4.5.0 of multidict. Using 4.4.2 is fine.

[webknjaz]

Can anyone rebuild that with Cython v28.5.0 to check whether it's related to Cython or to code changes?

[byashimov]

@webknjaz it says:

4.5.0 (2018-11-19)
Multidict views ported from Cython to C extension (#275)

According to the changelog this is the only difference between 4.4.2 and 4.5.

[asvetlov]

Guys, thank for the report.
I'm working on the fix.

[asvetlov]

Fixed by #306

[webknjaz]

@byashimov it's because I did not include dependency upgrade when doing the release
Ref #304

[delfick]

I don't think #306 fixed the issue

[webknjaz]

@asvetlov keeping this open since the issue was reported about the client side.

[aio-libs-bot]

GitMate.io thinks possibly related issues are #105 (istr causes memory leak), #274 (4.4.0a1 failed ), #280 (Can't install multidict 4.4.0 in docker alpine image), #286 (4.4.1 traceback with python 3.5.2), and #62 (Multidict preserves order- document it somewhere).