Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Exec cleanup improvements #12542

Merged
merged 5 commits into from
Nov 23, 2023
Merged

Exec cleanup improvements #12542

merged 5 commits into from
Nov 23, 2023

Conversation

tomponline
Copy link
Member

@tomponline tomponline commented Nov 22, 2023

Following on from @cjwatson PR #12530 that disabled websocket level ping messages when using Unix sockets, this PR re-enables them for the purposes of trying to identify whether the same issues exist when using TCP sockets and trying to fix the issue more generally.

We will test this PR with the Launchpad folks before moving this to latest/stable.

This PR:

  1. Closes websockets on the client as they finish reading/writing rather than keeping them open until all sockets have finished.
  2. Consumes ping messages being sent from the LXD server over the control and stdin websocket channels. This wasn't happening before this PR because there wasn't an expectation of receiving messages from the LXD server on the control and stdin channels (only data from the client to the server) and so the client was not performing a read on those channels. Unfortunately this meant that the default ping handling mechanism inside the gorilla websocket package was not being called and messages were not being consumed.

Fixes #10034

@tomponline tomponline self-assigned this Nov 22, 2023
@tomponline tomponline force-pushed the tp-exec branch 3 times, most recently from 5812caf to 1b1f768 Compare November 22, 2023 12:40
@tomponline tomponline marked this pull request as ready for review November 23, 2023 09:29
@tomponline tomponline requested a review from xnox November 23, 2023 09:36
Copy link
Contributor

@xnox xnox left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does interactive case needs to consume pings as well?


<-t.C
for {
err := conn.WriteControl(websocket.PingMessage, []byte("keepalive"), time.Now().Add(5*time.Second))
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

are pings here written always, for any connection type? (both interactive and non-interactive)

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes the (s *execWs) Connect is called for both non-interactive and interactive channels so it happens for all of them.

@@ -1249,10 +1253,16 @@ func (r *ProtocolLXD) ExecInstance(instanceName string, exec api.InstanceExecPos
return nil, err
}

go func() {
_, _, _ = conn.ReadMessage() // Consume pings from server.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

and over here pings are only consumed for non-interactive sessions ?!

expanding context, I cannot immediately understand if "interactive" branch of code just above needs to consume pings or not.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Only the control channel (for both interactive and non-interactive) and the stdin channel for non-interactive sessions requires a conn.ReadMessage() call (which internally then loops reading websocket frames and consumes the ping messages).

The other channels are already consuming ping messages because their direction is to read from the websocket and write to the stdout/stderr or the PTY.

I have observed this by setting up a custom ping handler on the client that logs when a ping was received and I was initially only seeing it for the stdout & stderr (both interactive and non-interactive), and not for the control and stdin for non-interactive sessions.

@tomponline
Copy link
Member Author

Does interactive case needs to consume pings as well?

Yes it does, and it is happening already.

@tomponline
Copy link
Member Author

@xnox I am not sure if this will fix the issue on riscv LP when it restores pings over unix sockets - that will be tested and remains to be seen. But it is certainly the case that at this time pings are not being consumed for stdin (non-interactive) and control (non-interactive and interactive) so it needs fixing anyway.

@tomponline tomponline merged commit a4a8f23 into canonical:main Nov 23, 2023
26 checks passed
@tomponline tomponline deleted the tp-exec branch November 23, 2023 16:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

"lxc exec" frequently runs into I/O timeouts
3 participants