Visual Studio becomes slower over time with F# solutions #4718
Comments
I would imagine the cause is high memory usage of devenv.exe. I'm very sceptical that the problem relates to the time which VS has been installed, as there is little transient state persisted between invocations of VS |
|
@dsyme, thanks for your quick response. Perhaps this is something: I may try to hook a profiler to VS next time it happens, but I have no idea what specifically I should monitor.
Dragging was noticeable before I updated to 15.6.2 and disappeared after updating to 15.6.4. After a while (several days) it came back, and yesterday I updated from 15.6.4 > 15.6.6 and this morning I was happy since everything was fast again. This cycle, that it is (much) faster just after updating was noticeable before, but I didn't make the link with the updates.
Typically around 1.2GB, while dragging often around 2GB or higher. Presently, it does not drag, and has Peak at 1.9GB, steady around 0.95GB.
Limited to 3. Earlier, @cartermp made some suggestions to use these settings on code fixes:
The relevant bits, I think, are:
No type providers at any level in the solution. Since it has happened to this particular solution for a long time (always to become speedy after an update, where I usually thought they improved performance, only to found out that after a while it was gone), I recently read, and commented, on this: #4691. I do have several "big" match-cases. I also have quite a few statically resolvable member constraints, some cross-project. As I understand, these may run into exponential performance or memory usage if they suffer from similar problems as the resolution of generic types in C#. But most of these suggest that it would be slow always, and not over time. Hence I was wondering whether I should look into the temp bits that are shattered throughout my system over time? |
|
@abelbraaksma re this:
I believe I suggested turning all of the code fixes off and in-memory cross-project references off as well. That said, if you're using 15.7 Preview 3 then you can probably keep the code fixes settings at their defaults (simplify name off, the rest on) because perf work for unused opens is implemented and they now run in priority order. |
|
@cartermp, yes you did ;). I'm on Preview 2 (main RTM instance on 15.6.6). It may prove helpful to update the Preview, considering the shared code base, I'll try that and will update this issue if the problem persists. Since both you and @dsyme use VS with F# and large solutions, assuming sometimes for hours on end, I wonder whether you ever encountered this behavior? Unless of course you are continuously updating (since you are developing the compiler itself), which seems to eradicate the issue for at least some time. |
|
Just an update. My VS has gotten sluggish again since I installed the update this morning. memory is 1.2GB working set, peak was at 2.2GB. processor is doing zilch, when I'm doing zilch, so that was the wrong assessment earlier (nothing spinning, maybe something hanging?). But when I type anything, I see devenv.exe going to 12% (that is the equivalent of 100% on 3 processors). Maybe I should do some profiling on the typing specifically. |
That would be great
I've only noticed it because of |
|
Ok, it continues to pester me and it makes it rather hard to work on my projects. Sometimes typing becomes so slow as 1-2 seconds per keystroke and type-info tooltips have delays of over a minute, making it hard to get the much sought-after type information of function calls. So, today I attached a profiler (it's ANTS 9.5). There are two important moments in the trace. Up until about 2:30, typing and info-tips were quite OK. Then the delay kicked in, which was roughly the last minute of the profiling session. What I can see is very deeply nested recursive calls in an async area. They don't seem to be tail-recursive, which may make their asyncronicity dubious. In processor time I can see 108%, while with 24 processor cores over two processors that should ideally be closer 2400% when true async kicks in (I know not all can be async, but this part seems to be written with that in mind and it appears to block itself). I'm not familiar enough with this part of the code, and it is very well possible that I draw the wrong conclusions, but this call-chart seems less-than optimal as it currently stands. In the profiling session, the timings used sampling, so they're bit rough. I don't know if it is really useful, if I can do something more useful to help further get to the bottom of this, let me know. Here's a screenshot of a 10-second part:
|
|
Hmm, is there a way I can share the profiling session? 4 minutes of profiling lead to a 61MB zipfile. Here's a smaller portion (just a few seconds) as XML file, maybe it helps a bit already. |
|
@abelbraaksma Have you had a chance to work in this part of your codebase with VS 2017 preview 5? We've observed a significantly better experience with the Visual F# codebase with this release, largely because a significant amount of memory is now being stored in virtual memory. Based on what you're saying here (and I've seen similar traces; it's very difficult to see causal information), memory usage reaches a point where VS enters a "loop of doom" and then there is recourse other than to restart VS. As for the profiling session, we can do that over email just so that no sensitive information is made public. phcart/dsyme/wismith at microsoft dot com |
|
Possibly more relevant part of the moment typing got stuck and extremely slow: |
@cartermp, let me check, not sure what version I'm on right now (considering how often updates are coming). Yes, "loop of doom" sounds about right. My problem is, however, that it happens a little bit too often. And on the same project, I can't remember I had it this severe in the 2015 times, so I know it can be done ;). |
|
Oh, I'm on preview 3. I assume you mean 15.7.0 Preview 5, to be more precise. I'll download/install that now, see if it makes a difference on this project. |
|
The next time you're in this state, can you please start a second instance of VS and use Help -> Send Feedback -> Report a problem, and switch from "Screenshot" to "Record", which will record a profile with ETW event tracing of CPU time, context switches, allocations, etc for us to look at. Once you've done that, if you can link the Developer Community item, we can go find it and look at the recorded trace. |
|
@Pilchie, I didn't know I could capture ETW so easily. I wanted to know how to do that for a while now, esp. for performance related issues. Thanks! Since this happens usually multiple times a day, often within half an hour of working with a large solution, it shouldn't be too hard (though after restart, re-install it speeds up for a few days, no idea why, and then it slows down again). |
|
@abelbraaksma Hopefully the trace will tell us! |
|
@Pilchie, I made a first recording in this corresponding issue: https://developercommunity.visualstudio.com/content/problem/245320/coloring-typing-tooltips-and-intellisense-slow-in.html. The relevant part is in the beginning where I try typing BTW, it doesn't look like I can update an existing issue with a new recording. Which means I may end up with a bunch of issues related to the same. |
|
I took a quick look at this trace. It appears that the UI thread (11432) isn't doing anything, but it's waiting for thread 13536, which is the F# reactor thread, which is doing plenty, but stacks are hard to read to figure out what's actually taking the time. One thing that is pretty obvious is that the amount of garbage is a problem, we spend more than 10% of the CPU time of that thread in the GC - here are the GCStats. Note how high VM is, causing the GC to run quite frequently @sharwell - I did run into a bunch of |
|
@Pilchie, thanks very much for taking the time to look into this, as it's becoming a real problem for the daily usage of VS + F#. I've created a new recording, which has a more significant delay in it which may help pinpoint the relevant issue easier (if anything is easy when it comes to performance of complex systems). The relevant part is between 2:10 and 2:45, it took roughly 30 seconds before the tooltip came back after a copy/paste action (though it is not always a copy/paste action that triggers it, just typing a single letter can have the same effect). Note that it effect all editing experiences: type-info tooltips, quick-info tooltips, dropdown lists, info-tips in dropdown lists (often empty, which is the result of another issue that has been addressed recently by @dsyme, I believe) and auto-completion (ctrl-space sometimes seems to do nothing, but then if you continue typing elsewhere, your previous ctrl-space gets completed like half a minute later). F12 and go-to definition are also affected, as are error squigglies (they remain very long, and sometimes stall forever, until a restart). Perhaps noteworthy: the project is large, and contains a relevant bit of SRTP and inline functions and methods. But in and of itself that's not too special, I think. The total size is roughly 120k lines, not including unit tests. The project is already split up in smaller chunks, but this only helps so much. One single part of the project (the Common library) seems to have the biggest effect. It translates to rouggly 6.3MB in release build dll (if that's any metric). The relevant bits of the project translate to 12MB of compiled dll files. I don't know if that's very much out of the ordinary. Also perhaps relevant: the CPU seems to have a hard time taking advantage multi-threading. It's only 4% of all cores that is doing something (out of 24 cores), which indicates that, if this process is supposed to be highly parallel, it isn't in my case. The new recording is here: https://developercommunity.visualstudio.com/content/problem/245786/slow-f-editing-experience-up-to-a-minute-until-typ.html. |
|
Right now it's generate lot's of garbage but not yet know how / why / where it generated. Could you create an ETW capture (you can try to filter if it becomes too big) and feed it to a perfview stack / allocation flamegraph? These tools should allow viewing the current stack that's leads to gcallocationtick* ETW events: microsoft/perfview#440 You can also use these tools to create to investigate ETW events or even create flamegraphs: On Linux, using CoreCLR it also possible to capture allocation stacks, by hooking into the PAL EventXplatEnabledGCAllocationTick* symbols: Would it be possible to create a standalone "benchmark" application to this usecase (eg.: loading some opensource app, do N number of code completion using an API)? Later this could be used for F# QA. |
|
@zpodlovics: there's quite some info, terminology and tools in your post that I am not familiar with. I assume you also have access to the current (private to Microsoft) ETW recordings I made? I'm on Windows and not very familiar anymore with Linux. The situation with the typing speed, coloring slowing down and downright slow background recompiling still happens, but it is quite difficult to record at "the right moment", as sometimes it goes away after a few minutes of lesser activity. There does some to be some relation to the size of the file that I am editing, or the location in the call-graph of the compiled projects. If Project A depends on B depends on C depends on D, then typing quickly becomes excessively slow if I edit in a big file in project A, but remains reasonably fast when I edit in any file in D. In B and C it is usually something in-between (the deepest hierarchy is about 12 projects, I think, total solution sizes vary). Also, in my case, project A is by far the largest and contains quite a few SRTP, which may or may not have an influence. As a consequence, many months ago I have split the project into a dozen solutions with each of them a subset of the projects. But when I need edits that cross the whole realm of projects and I need most of them in one solution, the current slowness makes it quite unbearable. If I didn't mention it already, this same project had far less performance issues while editing in VS2015, but that was some years back so it's hard to compare truly. I'll try to read into those links you gave me and the technologies involved and will attempt to make a more sensible capture or "flamegraph".
This would certainly be interesting, but this project is not open-source and I am not certain how to get this editing experience into an open-source minimal repro situation. Though it is worth a try. |
|
@abelbraaksma One thing that was enabled in VS2017 that might be a factor here is live cross-project IntelliSense. In VS 2015, if you made a change in D in your example, you would have to build before you could see that change in A. In VS2017, we made it so that the IDE works "live" by processing the source of those other projects, but that has caused performance problems for some customer solution configurations. You might try reverting to the old behavior by unchecking the option at Tools->Options->Text Editor->F#->Performance: |
|
@Pilchie, yes, thanks. I'm aware of that setting and I have it on 3 (it was in an earlier comment, but it starts becoming a long thread, so here's a copy of my current settings):
Whenever I set it back to 200 it quickly freezes completely, so there's definitely an influence there. I will try to work a while with the cross-project references off, but iirc, this removes quite a bit of useful features.
Let me correct myself here: it becomes slow when I edit a file that is referenced most often, i.e. a common functions library that every other project in your solution references (i.e., D in this example). It is fast when a file belongs to a project that is itself not referenced (i.e., the main program, or lead dll).
@Pilchie: is there a direct relation between compile performance and editor behavior, or is it more complex than that? The solution takes anywhere from 30s to 2 min to build (on one system), depending on whether or not everything is out of date. If that's exactly what the in-memory process is doing, then a single change will typically trigger at least 30s of background compilation. And the current implementation of this feature erases any knowledge that it already knew, which means for a tooltip to show up again, you need to wait until this process finishes (and there's no way to see its status, I think). Which may lead to a way to improve the user experience without having to solve all performance problems: cache the current state until the internal in-memory compiler has finished and only switch when all processes are done. That way, typing the dot to see dropdown list and the tooltips and everything would come from the previously compiled set, until the new compiled set is ready. I'm not sure how the internals work, but this all sounds like we could really use some kind of incremental compiler, that "sees" that you are only adding one function and only compiles that function. Or that 'sees" that a signature hasn't changed, so that it does not need to recompile anything (because internals are irrelevant for the tooltips and dropdowns). One other thing that would already help quite a bit is: find and compile only on significant changes. That is, changes in whitespace, in comments, in resource files, in removing unused variables or usings, etc (I'm sure this list can be expanded). At the moment, just hitting the spacebar on a white line will trigger recompilation. |
|
@abelbraaksma This:
Is definitely the in-memory cross-project references toggle. What this will do is return VS2017 to a VS2015 (and earlier) -like experience where cross-project features do not work and you need to rebuild a project to see changes in dependent projects. However, this will significantly reduce memory consumption and likely make your experience far more tolerable when working mostly within a single project. The real solution to this is, bluntly, some hard-ass design and engineering work that must be done in the F# language service. |
|
Thanks, that makes sense. An alternative approach would be to put some often-referenced projects into a local github repo, and reference the dlls instead of using project references. But I've been on that path and it wasn't a very happy experience.
That sounds like an enormous undertaking, so you are generally saying that current design or architecture of the language service internals needs some serious revamping? Patchwork won't cut it? |
|
It's not necessarily an enormous undertaking, it's just going to be difficult and will require a proper design that takes F# semantics into account. I don't think patchwork can cut it here. That said, there are numerous opportunities in performance that are essentially: profile -> see hotspot -> fix hotspot -> repeat We've been doing this recently and will continue to do so for a while. |
|
@abelbraaksma Re-labeling, since you did send us a trace which counts as a reproduction. Unfortunately, we're locking down on things to do with VS 15.8, and going after the issue(s) you're encountering are not in there. This is because we do not have the time to look at this and thoroughly attack the problem before VS 15.8 locks down, and we have a numerous things to ensure are landed and stable before then:
However, we do have some things slated for the longer-term that ought to help. Specifically, I really want us to work on a compiler server and hosting "F# out of process". That last bit is in quotes because what it entails is not straightforward, but the basic gist of it is that we want to give people more memory to work with (and have it be tunable). A compiler server should also help a lot, as it is effectively a metadata cache kept in a server process so that projects in large solutions need not recompute this information. It has resulted in significant memory reductions for C#, and I see no reason for us not to do it either. But until 15.8 is locked down and stable, I cannot offer any ideas about timeline for that. |
|
@cartermp: I'm already very glad that you take this report so seriously and are trying to pin it down somehow. Performance improvements in complex systems (or any system for that matter) are often pretty tough to do and cost a lot of time. I can totally sympathize with this being addressed after the current pending and planned improvements are addressed. That said, we should probably be very careful with any additions to the compiler service in terms of features that may (significantly) add to the processor's or memory workload, to prevent the issue from becoming too difficult to tackle. |
|
I just encountered an exception in the newest VS RTM, which may or may not be related so I am linking it here. #5033. Also, the most recent version seems to have made things worse, not better. Possibly as a result of performance issues, I find that:
|
|
@abelbraaksma That's troubling to hear, largely because 15.7 has been a big perf improvement for many people, and subsequent patch releases have just been bug fixes 😕. I would have expected a notable improvement compared with VS 15.6. At least that's how it is for our own codebase. Unfortunately, what you're describing could be a number of things:
However, if you've adjusted your settings as specified earlier in this issue, then I suspect much of the bottleneck is the compiler itself. But that's really only guessing. Perhaps we can spend some time looking at your codebase (under contractual NDA) directly? |
|
@abelbraaksma I don't have access to your private ETW recordings at Microsoft as I do not work there. I had to check every possible way earlier that I can do for code profiling (especially on Linux) for .NET Core. Did you manage to install a recent perfview tool and did you loaded your recorded ETW there? |
Yes, these settings are still the same.
@cartermp, Yes, that is certainly possible and in fact be very welcome. How would we go about doing just that? Though the DU itself isn't large (50 cases in total, nesting 5 deep max), they are bound with "and" and they have some important inlining (I've toned that down quite a bit, but not sure if SRTP could be the main cause of this) Another thing that just comes to mind: I add a module-init after compilation (to fix some issues with assembly loading). This is a custom compile step with Fody and IL weaving, so I wouldn't think it is run during on-the-fly compiling while editing. But if it is, then that can certainly delay things. I will try for a while with that switched off.
@zpodlovics, no, I have only made a recording using the build-in tools of Visual Studio so far. I was still planning on going over those things you mentioned, haven't done that yet. |
|
@abelbraaksma Any success with PerfView? I just checked the latest windows version from here (https://github.com/Microsoft/perfview/releases/tag/P2.0.15), there is a new FlameGraph tab available. |
|
@zpodlovics, I should definitely give that a try soon, as this is getting worse and worse it seems (but feeling and fact may not be the same). @cartermp, I just recorded the "typing experience", which feels like WP51 on an 8088 machine. I know it doesn't help resolving this issue, but it may give an idea of how it impacts every moment of working with VS + F# (in same project, in C#, this problem doesn't exist). This is smooth typing, I mean, I type that text fluently in about 1-2 seconds, and the keyboard buffer catches up in fits and starts. This is just a comment of one line, imagine typing multiple lines ;).
|
|
@abelbraaksma what VS version are you using? Is it a .NET SDK or legacy style project? I noticed this behavior over super long period of times (over a day?) in FSharp.Compiler.Private, but it doesn't continuously happen. Also, whenever you type something, F# will try to do a full type check on the file you are currently in, even if its a comment. But, we fixed a bug where every time you typed something in a .NET sdk style, it would invalidate the whole project! |
@TIHan, I never fully grasp what "legacy" means, to me that sounds like pre-.NET or something, but I assume it's .NET Framework 4? This is a project started years ago in F# 2.x, later 3.x, the for VS2015 updated to 4.0 and more recently to 4.1. The editing-slowness started at some point with VS2017, though I can't recall what change triggered it. And it uses .NET Framework 4.7.1.
It is not uncommon for me to keep my system running overnight and things remain open. But it also happens just after working a couple of minutes, though it depends a lot on what I am doing.
This explains a lot! Some files are large and contain quite a bit of SRTP and/or The file in the example was itself not big, just around 1000 lines I think. But I wonder whether, if you invalidate file A that any file that depends on it for its inferred types, will also be invalidated? Because that may not be the whole project, but if one file has many dependencies down the line, it can wreak havoc on performance. |
|
@abelbraaksma Based on this, and other information you've given, my educated guess is that the code in this file (and others) is pounding the compiler into submission. Heavy use of SRTP would make me think as much. Other documents won't be invalidated, but it could also be likely that things are recomputed unnecessarily if the typechecking requires lots of information from other places (and with in-memory cross-project references turned on, from other projects as well). This is a general problem that has always existed in the tools, and only a heuristic to try and recompute only what is necessary can be done. That said, typing into a comment shouldn't invalidate the document at all. That definitely seems like a bug. |
|
@abelbraaksma Thank-you for both reports on the developer community, they were extremely helpful and I've identified the problems: The first trace had two instances of VS that I looked at, and the second had one. All three showed signs of high GC usage in all three generations. The slowness over time can be mostly explained by what we call "Large Object" allocations; objects that have a size over 85k. These get put into a special heap and only collected when absolutely needed. When memory usage is low, we don't spend anytime collecting them - however, as this heap grows over time, we need to pay down our taxes, and collect them during a "Gen2" collection. This can take a long time, and leads to visible delays if there's a large amount of data. On top of this, in some of the traces, both Gen0 and Gen1 were also taking up a significant amount of time putting pressure on all three generations. Based on the traces, I've identified the following issues exist for your solution, one of which was already a duplicate: #5922 I'm not that familiar with the F# code base so I'm unsure if these paths that I filed are expected to be hit this many times, or if this might be an indication that we're missing caches and reparsing the world over and over again. I'll leave that for the F# folks to figure that. If possible can you record a trace of your modification to the comment and file a new bug? As short as possible would be great. |
|
Thanks a lot @davkean for this analysis and the great explanation. |
|
For reference I am including here links to the two traces that I reported at developercommunity: |
|
For those that want an overview on Large Object Heap (LOH) and its impact on the above, see https://docs.microsoft.com/en-us/dotnet/standard/garbage-collection/large-object-heap. It's a great read. |
|
From reading about LOH, there is a very important point to make here. LOH objects do get collected during Gen2, however compaction does not happen on the LOH. What this means is that the LOH can get fragmented. If LOH is heavily fragmented, any new allocations on LOH will have to happen at the end of the heap causing the LOH to expand, thus consuming more memory. Based on this, while we have allocations to LOH, it would be good to get evidence of how bad the fragmentation is. I'm going to guess it's not good since we love to allocate on the LOH. |
|
First off thanks @davkean for all your assistance with the profiling, massive help! Is this being considered parent issue of all the child issues @davkean has referenced? I can start looking into these issues, looking for ways to improve memory usage, improve caching, and if possible, improve algo (most of the algo logic is ok, they just use wasteful patterns). So I am not duplicating any work already in progress, and provided you want the help, please let me know what is being worked on, and what you feel is best for me to focus on to start with? |
|
This is a parent issue for some of the issues, but not all of them. Some are from a trace coming from a Microsoft product. Please do feel free to look into things 🙂 |
|
@abelbraaksma I think I'm going to close this issue in favor of #6096, which is an all-up tracking issue for concrete problems found in analyzing Visual Studio performance. Some of what is tracked there is derived directly from the trace you provided, but it also contains other specific problems that we plan to address. Please do continue to submit traces - we can analyze them like what was done for the first one you submitted. Given that a lot of progress on the top performance issues has already been made, it may be the case that the root cause behind something you're observing is already fixed, but I'm sure that there's still plenty left to do 🙂 |
|
@cartermp, thanks for this, and yes, the new collection-issue makes sense, much clearer than using this long thread for it. I've been following the discussions, reports and fixes closely and it appears that a lot of progress is being made here 🥇 💯 . These days I'm a bit pressed for time, but I'll certainly try them out. Btw, I don't notice much, if any, performance gains in the more recent VS 2017 versions (just yesterday, waiting for type inference and coloring took again about 1-2 min), is it possible that all these performance improvements are only submitted to the VS 2019 branches and releases? Or maybe I should try VS 2017 previews again? |
|
This is all in VS 2019. VS 2017 is now locked down for LTS, so any new work has to happen for VS 2019. |
|
Fair enough. Downloading it already (but I was warned by @TIHan's comment in another thread that there's a performance bug with the current preview version, I'll have to wait it out until Preview 3 ;) |
This was reported somewhere within other issues, but I am reporting it here since it seems to continue to happen and because I am curious whether other people have similar experiences.
Repro steps
Another way:
Another way:
(I don't think the compiler itself becomes slower, but the interaction with VS seems to slow the compiler as well)
Expected behavior
On a fresh install or after a new update of Visual Studio, searching succeeds in X seconds. Expected after several days, weeks that it is still X seconds, but but it becomes X+Y seconds where Y appears to increase.
Concretely, today I installed VS 15.6.6, and searching throughout my solution succeeds in under 1 second. So fast that I doubted the search results were correct.
Yesterday, before the update, search took ~5 - 10s seconds. Same words, same Find Options.
Actual behavior
Visual Studio starts to drag after a while.
Known workarounds
Reinstall, it will be fast again, at least for a few days.
Related information
Applies to VS 2017 versions, not seen as such in VS 2015.
I can see the CPU taking up a lot of time and it appears as if single threads are spinning (that is, CPU is only 5% for devenv.exe, but with 24 processors, that is equal to at least two threads being very busy, which apparently aren't written multi-threaded.
I am curious if this happens only to my specific instance or project. I just recently learned that some match-expressions can get unwieldy (take up GBs of memory by the compiler) and I have several large match-expressions to which this may apply (I will go over them, to see if this helps). Perhaps there are other challenges in F# compiling that are 2n complex.
However, I would assume that if that is at the root of this, it would mean that it should be slow always, and not just after time. It is very well possible that this slowing down is caused by something outside of the realm of F# (i.e., if the JIT temp dirs grow, or the sympols dirs), but I am unsure where to even begin looking.
The text was updated successfully, but these errors were encountered: