IDE Performance Results

In a similar vein to Build Performance Results this page aims to measure the performance of Xamarin.Android across different versions of Visual Studio. The goal is to get a higher-level picture than what we can achieve from profiling in MSBuild.

Using an internal Microsoft tool, we are able to get timing information out of Visual Studio. This timing gives a better overall picture of the devloop, as well as the exact experience developers get inside the IDE.

In each timing, I am merely pushing the "play" button (F5). We have timings for:

• Preparation Time: Build + Deploy, the time it took for the Xamarin.Android MSBuild targets
• Launch Time: app started + Mono debugging session started
• Total Time: Preparation Time + Launch Time

From the table, what are first/second/third?

• First: start a debugging session after an initial checkout, after nuget restore.
• Second: stop, new debugging session with no changes.
• Third: stop, modify a XAML file, start debugging again.

I used the same projects from Build Performance Results with a physical Google Pixel 3 device connected via USB.

This is a comparison between:

• Visual Studio 15.9.5, Xamarin.Android 9.1.5
• Visual Studio 16 Preview 2, Xamarin.Android 9.1.103.6

Hello Forms

Session	VS Version	Preparation Time	Launch Time	Total Time
First	15.9	00:57.1	00:02.4	00:59.5
Second (no changes)	15.9	00:04.9	00:01.7	00:06.6
Third (xaml change)	15.9	00:05.8	00:01.7	00:07.6
First	16.0 P2	00:31.0	00:03.3	00:34.3
Second (no changes)	16.0 P2	00:03.9	00:02.5	00:06.4
Third (xaml change)	16.0 P2	00:04.7	00:02.6	00:07.3

SmartHotel360

Session	VS Version	Preparation Time	Launch Time	Total Time
First	15.9	02:17.7	00:02.6	02:20.3
Second (no changes)	15.9	00:06.1	00:01.8	00:07.9
Third (xaml change)	15.9	00:10.4	00:02.0	00:12.4
First	16.0 P2	01:02.8	00:03.2	01:06.0
Second (no changes)	16.0 P2	00:06.5	00:02.8	00:09.3
Third (xaml change)	16.0 P2	00:08.6	00:02.7	00:11.2

Xamarin.Forms Control Gallery

Session	VS Version	Preparation Time	Launch Time	Total Time
First	15.9	03:12.5	00:09.6	03:22.0
Second (no changes)	15.9	00:32.8	00:02.4	00:35.2
Third (xaml change)	15.9	00:33.9	00:02.4	00:36.3
First	16.0 P2	02:05.4	00:03.4	02:08.9
Second (no changes)	16.0 P2	00:27.4	00:02.5	00:29.9
Third (xaml change)	16.0 P2	00:32.5	00:02.7	00:35.2

Conclusion

The results seem to support what we are seeing when profiling MSBuild command-line, such as Build Performance Results. There is an element of the timings being better than command-line, since MSBuild.exe is reused and other build-related performance caching you get in an IDE, but not command-line.

We think there might be a ~1 second performance regression in Launch Time in 16.0 P2. @jonathanpeppers will follow up with the IDE team to figure out what is happening there.