-
-
Notifications
You must be signed in to change notification settings - Fork 3.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add tracing spans to schedules, stages, systems #789
Conversation
Opened a couple of PRs to see if the modifications to add the fields to the span name are upstreamable:
|
Ok I really dig this 😄 I'll play around with this a bit before merging, but it seems like a reasonable thing to add (especially given that its optional). My big open questions / followups (that don't need to be resolved in this pr):
|
Alrighty I adapted this to the latest changes on master and reverted back to the crates.io version of tracing-chrome because we can't publish git dependencies. |
Also made the example's defaults exclude wgpu and include info so users don't need to set that manually |
I agree on all points. I think it would be good to start with a simple subscriber that probably could hook into the diagnostics to register measurements when spans are received. I do wish print diagnostics were ordered somehow though. Either descending by time spent or alphabetically by diagnostics name. And yeah it would be super nice to have it built in to the editor later. A fun thing in the YouTube video was changing the filters at run time so you can refocus the events and spans to profile just what you want. Should help to enable exploratory profiling while minimising performance impact. |
The
tracing
crate allows for gathering data on how much time is spent within a scope, as well as enabling structured logging.tracing
has a structure similar to thelog
crate where instrumentation using spans or events (events being structured logs within a hierarchy of spans, so much more advanced and useful logs) are made using thetracing
crate and then there are many other crates that can subscribe to the events to output data in a format useful for analysis and/or visualisation.If you want to learn more I highly recommend watching this video: https://www.youtube.com/watch?v=JjItsfqFIdo
My motivation was focused on profiling, as I wanted to know how much time was being spent in each system in order to understand where game loop time was being spent. I have been a bit slow in making this PR as I wanted to find some useful ways of analysing and visualising the trace data.
A few good options:
tracing-flame
- captures a trace to a file that can then be used with theinferno-flamegraph
crate to generate flamegraphs (aggregate time spent within spans) or flamecharts (a timeline of spans for the duration of the capture). These are good but I found clicking around between views in an SVG to be a bit cumbersome.tracing-chrome
- captures a trace to a file that can be opened in Chrome/Chromium by visiting chrome://tracing and loading the file. This shows a timeline of events that are quick to navigate using the tools (pan / zoom / highlight range / etc). The dependency on Chrome doesn't feel great, but this looks potentially useful and standalone/possible to integrate: https://github.com/catapult-project/catapult/tree/master/tracingtracing-tracy
- streams the trace to the Tracy frame profiler. This is advanced real-time profiling designed for use with games. I had stability problems on macOS but perhaps on Linux / Windows that are better supported, this is the best option. It can display the timeline of events, statistical aggregates of how much time is being spent in spans, log messages, everything. Here's a screenshot:Unfortunately, it seems the default behaviour of all of these three crates is to not include any of the additional fields added to spans in the visualisation.
tracing
spans must have a static str name and only additional fields can be dynamic. This means that with this current implementation, you can only visualise that a schedule/stage/system has run, and not see what it is. I have made a modification to tracing-chrome here that includes the additional fields in the span's name for the visualisation to be able to see what span is what and similarly for tracing-tracy here.To make use of
tracing-chrome
in my project, I did the following:Cargo.toml
src/trace.rs
src/main.rs
You can also enable
bevy/wgpu_trace
but be warned, it is very chatty to the point of impacting performance so I would suggest not using it, or using it sparingly or when you need to.Note also that in the example code, I added an
EnvFilter
. This allows filtering by settingRUST_LOG=trace,wgpu=warn
similar to how theenv_logger
crate works withlog
, but in addition it allows filtering of log events and spans based on some more advanced rules. See here for more information.For sure some of this information should go into some documentation somewhere and I will add that to the PR once we've decided whether this is wanted and/or in what form.