Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[REGRESSION]: Out of memory errors on 0.15.0 #1968

Closed
celeryclub opened this issue Apr 24, 2020 · 9 comments
Closed

[REGRESSION]: Out of memory errors on 0.15.0 #1968

celeryclub opened this issue Apr 24, 2020 · 9 comments

Comments

@celeryclub
Copy link
Contributor

Context:

  • GOOD Playwright Version: 0.14.0
  • BAD Playwright Version: 0.15.0
  • Operating System: Docker on linux using qawolf/playwright-ci:v0.9.0

Describe the bug

Hi team. I started seeing a lot of Out of memory errors in Chromium after upgrading to 0.15.0. I have a test that can make it fail consistently, but it's very large and I haven't yet narrowed down what part of it is causing the issue.

Here's the error I'm getting. ```sh [docker-2] 2020-04-24T19:25:54.438Z pw:browser:err [0424/192554.150296:FATAL:memory.cc(40)] Out of memory. size=262144 [docker-2] 2020-04-24T19:25:54.438Z pw:browser:err #0 0x564ff73ae919 base::debug::CollectStackTrace() [docker-2] 2020-04-24T19:25:54.438Z pw:browser:err #1 0x564ff7313723 base::debug::StackTrace::StackTrace() [docker-2] 2020-04-24T19:25:54.438Z pw:browser:err #2 0x564ff7324080 logging::LogMessage::~LogMessage() [docker-2] 2020-04-24T19:25:54.438Z pw:browser:err #3 0x564ff733ff79 base::internal::OnNoMemoryInternal() [docker-2] 2020-04-24T19:25:54.438Z pw:browser:err #4 0x564ff845f921 viz::bitmap_allocation::AllocateSharedBitmap() [docker-2] 2020-04-24T19:25:54.438Z pw:browser:err #5 0x564ff862f753 cc::BitmapRasterBufferProvider::AcquireBufferForRaster() [docker-2] 2020-04-24T19:25:54.438Z pw:browser:err #6 0x564ff85c3cbc cc::TileManager::CreateRasterTask() [docker-2] 2020-04-24T19:25:54.438Z pw:browser:err #7 0x564ff85c0e9b cc::TileManager::AssignGpuMemoryToTiles() [docker-2] 2020-04-24T19:25:54.438Z pw:browser:err #8 0x564ff85bf319 cc::TileManager::CheckIfMoreTilesNeedToBePrepared() [docker-2] 2020-04-24T19:25:54.438Z pw:browser:err #9 0x564ff735b5ab base::TaskAnnotator::RunTask() [docker-2] 2020-04-24T19:25:54.438Z pw:browser:err #10 0x564ff736b648 base::sequence_manager::internal::ThreadControllerWithMessagePumpImpl::DoWorkImpl() [docker-2] 2020-04-24T19:25:54.438Z pw:browser:err #11 0x564ff736b3e8 base::sequence_manager::internal::ThreadControllerWithMessagePumpImpl::DoWork() [docker-2] 2020-04-24T19:25:54.438Z pw:browser:err #12 0x564ff7327b8a base::MessagePumpDefault::Run() [docker-2] 2020-04-24T19:25:54.438Z pw:browser:err #13 0x564ff736bbf5 base::sequence_manager::internal::ThreadControllerWithMessagePumpImpl::Run() [docker-2] 2020-04-24T19:25:54.438Z pw:browser:err #14 0x564ff7344fbe base::RunLoop::Run() [docker-2] 2020-04-24T19:25:54.438Z pw:browser:err #15 0x564ff6bb18f5 blink::scheduler::WorkerThread::SimpleThreadImpl::Run() [docker-2] 2020-04-24T19:25:54.438Z pw:browser:err #16 0x564ff73bf3be base::(anonymous namespace)::ThreadFunc() [docker-2] 2020-04-24T19:25:54.438Z pw:browser:err #17 0x7fb0b54026db start_thread [docker-2] 2020-04-24T19:25:54.438Z pw:browser:err #18 0x7fb0aee7888f clone [docker-2] 2020-04-24T19:25:54.438Z pw:browser:err Task trace: [docker-2] 2020-04-24T19:25:54.438Z pw:browser:err #0 0x564ff85c9f9e cc::UniqueNotifier::Schedule() [docker-2] 2020-04-24T19:25:54.439Z pw:browser:err #1 0x564ff85c6b30 cc::(anonymous namespace)::DidFinishRunningAllTilesTask::RunOnWorkerThread() [docker-2] 2020-04-24T19:25:54.439Z pw:browser:err [docker-2] 2020-04-24T19:25:54.439Z pw:browser:err Received signal 6 [docker-2] 2020-04-24T19:25:54.446Z pw:browser:err #0 0x564ff73ae919 base::debug::CollectStackTrace() [docker-2] 2020-04-24T19:25:54.457Z pw:browser:err #1 0x564ff7313723 base::debug::StackTrace::StackTrace() [docker-2] 2020-04-24T19:25:54.464Z pw:browser:err #2 0x564ff73ae4b5 base::debug::(anonymous namespace)::StackDumpSignalHandler() [docker-2] 2020-04-24T19:25:54.464Z pw:browser:err #3 0x7fb0b540d890 (/lib/x86_64-linux-gnu/libpthread-2.27.so+0x1288f) [docker-2] 2020-04-24T19:25:54.464Z pw:browser:err #4 0x7fb0aed95e97 gsignal [docker-2] 2020-04-24T19:25:54.464Z pw:browser:err #5 0x7fb0aed97801 abort [docker-2] 2020-04-24T19:25:54.475Z pw:browser:err #6 0x564ff73ad415 base::debug::BreakDebugger() [docker-2] 2020-04-24T19:25:54.483Z pw:browser:err #7 0x564ff7324520 logging::LogMessage::~LogMessage() [docker-2] 2020-04-24T19:25:54.495Z pw:browser:err #8 0x564ff733ff79 base::internal::OnNoMemoryInternal() [docker-2] 2020-04-24T19:25:54.505Z pw:browser:err #9 0x564ff845f921 viz::bitmap_allocation::AllocateSharedBitmap() [docker-2] 2020-04-24T19:25:54.525Z pw:browser:err #10 0x564ff862f753 cc::BitmapRasterBufferProvider::AcquireBufferForRaster() [docker-2] 2020-04-24T19:25:54.539Z pw:browser:err #11 0x564ff85c3cbc cc::TileManager::CreateRasterTask() [docker-2] 2020-04-24T19:25:54.546Z pw:browser:err #12 0x564ff85c0e9b cc::TileManager::AssignGpuMemoryToTiles() [docker-2] 2020-04-24T19:25:54.559Z pw:browser:err #13 0x564ff85bf319 cc::TileManager::CheckIfMoreTilesNeedToBePrepared() [docker-2] 2020-04-24T19:25:54.568Z pw:browser:err #14 0x564ff735b5ab base::TaskAnnotator::RunTask() [docker-2] 2020-04-24T19:25:54.577Z pw:browser:err #15 0x564ff736b648 base::sequence_manager::internal::ThreadControllerWithMessagePumpImpl::DoWorkImpl() [docker-2] 2020-04-24T19:25:54.587Z pw:browser:err #16 0x564ff736b3e8 base::sequence_manager::internal::ThreadControllerWithMessagePumpImpl::DoWork() [docker-2] 2020-04-24T19:25:54.595Z pw:browser:err #17 0x564ff7327b8a base::MessagePumpDefault::Run() [docker-2] 2020-04-24T19:25:54.607Z pw:browser:err #18 0x564ff736bbf5 base::sequence_manager::internal::ThreadControllerWithMessagePumpImpl::Run() [docker-2] 2020-04-24T19:25:54.615Z pw:browser:err #19 0x564ff7344fbe base::RunLoop::Run() [docker-2] 2020-04-24T19:25:54.623Z pw:browser:err #20 0x564ff6bb18f5 blink::scheduler::WorkerThread::SimpleThreadImpl::Run() [docker-2] 2020-04-24T19:25:54.633Z pw:browser:err #21 0x564ff73bf3be base::(anonymous namespace)::ThreadFunc() [docker-2] 2020-04-24T19:25:54.633Z pw:browser:err #22 0x7fb0b54026db start_thread [docker-2] 2020-04-24T19:25:54.633Z pw:browser:err #23 0x7fb0aee7888f clone [docker-2] 2020-04-24T19:25:54.633Z pw:browser:err r8: 0000000000000000 r9: 00007fb0a39a78e0 r10: 0000000000000008 r11: 0000000000000246 [docker-2] 2020-04-24T19:25:54.633Z pw:browser:err r12: 00007fb0a39a8bb0 r13: 00007fb0a39a7b40 r14: 00007fb0a39a8bc0 r15: aaaaaaaaaaaaaaaa [docker-2] 2020-04-24T19:25:54.633Z pw:browser:err di: 0000000000000002 si: 00007fb0a39a78e0 bp: 00007fb0a39a7b30 bx: 00007fb0a39a8374 [docker-2] 2020-04-24T19:25:54.633Z pw:browser:err dx: 0000000000000000 ax: 0000000000000000 cx: ffffffffffffffff sp: 00007fb0a39a78e0 [docker-2] 2020-04-24T19:25:54.633Z pw:browser:err ip: 00007fb0aed95e97 efl: 0000000000000246 cgf: 6946000000000033 erf: 0000000000000000 [docker-2] 2020-04-24T19:25:54.633Z pw:browser:err trp: 0000000000000000 msk: 0000000000000000 cr2: 0000000000000000 [docker-2] 2020-04-24T19:25:54.633Z pw:browser:err [end of stack trace] [docker-2] 2020-04-24T19:25:54.633Z pw:browser:err Calling _exit(1). Core file will not be generated. ```

I'm wondering if this is caused by #1956 since I don't see any other Chrome-specific changes between the two releases, but I honestly don't know.

Thanks for looking.
@pavelfeldman
Copy link
Member

Viz is a compositor infrastructure in Chromium. According to the trace, it runs out of memory while allocating gpu? memory for the tiles rasterization. Your screen is divided into a grid of tiles that can repaint independently. If you can reliably reproduce it on 0.15 and not at 0.14 it does point at a change in Chrome. Having a repro would be of a great help.

@celeryclub
Copy link
Contributor Author

celeryclub commented Apr 25, 2020 •
edited
Loading

Well this is awkward. It turns out this was 100% my fault. I actually had a memory leak in my code. I'm doing some weird stuff to get custom methods working (#1655) and it turns out one of my functions created an infinite loop. Classic. Not sure why it wasn't causing problems in 0.140, but I don't think it really matters.

Sorry for the noise and thanks for your commitment to taking care of issues!

Edit: see below.

@celeryclub
Copy link
Contributor Author

celeryclub commented Apr 27, 2020 •
edited
Loading

After a lot more digging, I found a way to reproduce this crash without any of my sketchy creative code. I created a repo to demonstrate: https://github.com/celeryclub/playwright-crash-repro.

I originally saw this crash on a Linux host, but this repro works on macOS too. Here's my VM config.
VMware Fusion v11.5.1
Linux 4.14.154-boot2docker
16384 MB RAM

Just to clarify, this repro doesn't crash 100% of the time. It's more like 10-15%. And it turns out this also happens in 0.14.0, but not as often - maybe like 7% of the time. I included some logs in the repo that show the results of my runs.
Logs for 0.14.0
Logs for 0.15.0

One pattern I've notice while watching these runs is that the crash always seems to happen when opening a new tab.

@celeryclub celeryclub reopened this Apr 27, 2020
@pavelfeldman
Copy link
Member

Could you try passing docker run --shm-size=1gb?

@celeryclub
Copy link
Contributor Author

Thanks @pavelfeldman. That does appear to fix it. Do you know why --shm-size=1gb would work but --disable-dev-shm-usage wouldn't? Seems like they should do about the same thing. I guess it might be dependent on my host system.

@pavelfeldman
Copy link
Member

I have no idea - I did not look into why it was not regarded in this viz allocation.

@aslushnikov should we update our docker guide? with that, closing this issue?

@celeryclub
Copy link
Contributor Author

Thanks @pavelfeldman. For future debuggers, docker run --ipc=host seems like the most solid fix I've found for this crash and others too. Reference: cypress-io/cypress#350 (comment).

I'd love to see this added to the docker guide. 👍🏻

@celeryclub celeryclub mentioned this issue Apr 29, 2020
Merged
@pavelfeldman pavelfeldman mentioned this issue May 11, 2020
Merged
@jknooks583
Copy link

jknooks583 commented Oct 19, 2020 •
edited
Loading

Would this work for firefox as well or just Chrome?
Using --shm-size=1gb ?

@celeryclub
Copy link
Contributor Author

Either that or --ipc=host would most likely help with FF too. Give it a try and see what happens!

@mokshjawa mokshjawa mentioned this issue Aug 20, 2021
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@celeryclub @pavelfeldman @jknooks583