Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

dotnet-trace on containers #3480

Open
Jongy opened this issue Oct 21, 2022 · 4 comments · May be fixed by #3539
Open

dotnet-trace on containers #3480

Jongy opened this issue Oct 21, 2022 · 4 comments · May be fixed by #3539
Labels
dotnet-trace enhancement New feature or request
Milestone
8.0.0

Comments

@Jongy
Copy link

Jongy commented Oct 21, 2022 •
edited
Loading

Background and Motivation

This is for Linux, when using dotnet-trace in PID connect mode, e.g dotnet-trace collect --process-id XXX ....

dotnet-trace attempts to connect to the transport pipe in /tmp of the mount namespace it's currently running in. If the target PID runs in a container (or if dotnet-trace itself runs in a container), then its mount namespace is different than the mount namespace of dotnet-trace, and we'll not find its pipe.

This is raised in #810 and the proposed solution (at least from my skimming) is to mount the /tmp such that it's accessible on the host.

Proposed Feature

Another solution that I'm proposing is to access the /tmp of the container, via /proc/pid/root/tmp, in case the target process runs in a different mount namespace (easily checked by comparing the /proc/pid/ns/mnt file of dotnet-trace to the one of the process).

We're using this "trick" in our gProfiler (see usage here), by changing the TMPDIR variable which is respected by dotnet-trace, and it's proved itself to work, so I figured we could place this logic in upstream and have it fixed for everyone :)

I searched through the code a bit and found PAL_GetTransportName which appears to be the relevant function for this change. It also seems to handle the same concept for MacOS (using GetApplicationContainerFolder). I figure that'd be the place to put the matching Linux support.

If you think this could be a good addition, I'm happy to implement this change.

Usage Examples

Profiling/tracing a process running in a container.
@Jongy Jongy added the enhancement New feature or request label Oct 21, 2022
@Jongy Jongy mentioned this issue Oct 21, 2022
Open
@tommcdon tommcdon added this to the 8.0.0 milestone Oct 25, 2022
@tommcdon
Copy link
Member

@Jongy thank you for writing up the proposal and offer to implement improving dotnet-trace scenarios for correctly handling container namespaces for the mounted directories. Adding @noahfalk for thoughts.

@noahfalk
Copy link
Member

If I am following correctly the goal is to adjust where dotnet-trace searches for socket based on the filesystem mount of the target process being traced? That sounds useful to me and we'd be glad to get a PR 👍. Thanks!

dotnet-trace (and all the other dotnet-*) tools use this code to enumerate available processes:

diagnostics/src/Microsoft.Diagnostics.NETCore.Client/DiagnosticsClient/DiagnosticsClient.cs

Line 310 in 32e474a

foreach (var port in Directory.GetFiles(PidIpcEndpoint.IpcRootPath))

and this code to connect to get the address for a specific process by PID:

diagnostics/src/Microsoft.Diagnostics.NETCore.Client/DiagnosticsIpc/IpcTransport.cs

Line 300 in 3128b54

defaultAddress = Directory.GetFiles(IpcRootPath, $"dotnet-diagnostic-{pid}-*-socket") // Try best match.

The PAL_GetTransportName function is slightly different, that one gets used by the debugger to determine the path to the debugging pipe. For somewhat arbitrary historical reasons the debugger pipe is distinct from the diagnostics pipe that all other tools use.

@Jongy
Copy link
Author

Jongy commented Oct 26, 2022

If I am following correctly the goal is to adjust where dotnet-trace searches for socket based on the filesystem mount of the target process being traced? That sounds useful to me and we'd be glad to get a PR +1. Thanks!

Yes.

and this code to connect to get the address for a specific process by PID:

Oh wow, I was way off in my search 😅

Okay, that's all the information I need to know. Might take me a while to reach it but I'll try making the modification and if I prove it works, I will send a PR :) Thanks for the replies guys!

@Jongy Jongy linked a pull request Dec 5, 2022 that will close this issue
Draft
@Jongy
Copy link
Author

Jongy commented Dec 5, 2022

I pushed a PoC PR in #3539 , would love your comments :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
dotnet-trace enhancement New feature or request
Projects
None yet
Milestone
8.0.0
Development

Successfully merging a pull request may close this issue.

Search for the diagnostics socket in the mount NS of the target process Jongy/diagnostics
3 participants
@noahfalk @Jongy @tommcdon