Workstation SSH connection failed or timed out (Unexpected HTTP response status code 407)

[ovavadim]

Hello!

I use "Cloud Workstations" plugin for JetBrains Gateway (Windows 10).

My machine uses Proxy server. JetBrains Gateway is configured for this proxy, connection checking works successfully. Gateway also sees my Google workstation project and the workstation. But when I try to launch through Gateway, it says "Workstation SSH connection failed or timed out".

In the logs of Gateway it looks like:
Caused by: jdk.internal.net.http.websocket.CheckFailedException: Unexpected HTTP response status code 407
at java.net.http/jdk.internal.net.http.websocket.OpeningHandshake.checkFailed(OpeningHandshake.java:343)
at java.net.http/jdk.internal.net.http.websocket.OpeningHandshake.handleResponse(OpeningHandshake.java:252)
at java.net.http/jdk.internal.net.http.websocket.OpeningHandshake.resultFrom(OpeningHandshake.java:222)

(see the log file attached)

A 407 status code means "Proxy Authentication Required". But JetBrains Gateway works well with my proxy. I guess, it's a problem of "Cloud Workstation" plugin.

How can I separately configure it for a correct work with my proxy, if it really doesn't inherit proxy settings from JetBrains Gateway?

Have a nice day!

Version Information

JetBrains Gateway version: 2024.1.2

Cloud Workstations plugin version: 24.3.1-api-version-231

idea.log

[ruomengz]

Thank you for the detailed report! Looks like this might be related to the proxy settings, the http requests in the log are successful.

To help debugging:

1. Could you please try to connect to your cloud workstation host using gcloud? By creating a tcp tunnel on WORKSTATION_PORT 22 , or SSH to your workstation host?
2. Just to confirm, did you set up your proxy through JetBrains HTTP proxy settings panel?

[ovavadim]

@ruomengz, thanks for your answer!

When I run:

gcloud workstations ssh \
  --project=... \
  --cluster=... \
  --config=... \
  --region=... \
  w-...-...

then I see

And PuTTY is being open and it's forever empty (black screen only).

When I run
gcloud workstations start-tcp-tunnel --project=... --region=... --cluster=... --config=... w-... 22
then I see

and this "Listening" lasts forever.

This is how I set up the proxy:

[ruomengz]

Hey @ovavadim , sorry for the late reply.

Looks like you are able to create the tcp tunnel from gcloud, while we are trying to reproduce the proxy issue, can you try this workaround?

1. Start TCP tunnel using gcloud

gcloud workstations start-tcp-tunnel --project=... --region=... --cluster=... --config=... w-... 22  --local-host-port=:${local_port_number}

2. Connect to the localhost using JetBrains Gateway native SSH support. Use user as Username.
   

Another question, do you also configure your proxy on Windows system?

[ovavadim]

Hello, @ruomengz

Well, firstly I do this:
gcloud workstations start-tcp-tunnel --project=cc-... --region=europe-west3 --cluster=... --config=... w-...-... 22 --local-host-port=localhost:22222

Then I see there "Listening on port [22222]".

Then I go to JetBrains Gateway to SSH connection, I do this:

Then I click "Check Connection and Continue" and I see "Checking connection..." label, which never disappears.

At this moment I see in CMD:

So, the same problem as I showed in previous message in SSH.

About configuring the proxy on Windows:

As you can see, it's configured in control panel by the employer.

So, what should I do?

Have a nice day!
Vadim.

[ruomengz]

Thanks @ovavadim!
Could you try to set up your proxy for gcloud following proxy configuration? And could you confirm that other gcloud workstations commands like list or start run successfully?

Sorry for the-back-and-forth, it is hard for us to reproduce the issue with proxy setup.

[ovavadim]

Hello, @ruomengz!

Well, I've tested before proxy configuring:

H:\>gcloud workstations start --project=... --region=... --cluster=... --config=... w-...-....
Starting workstation: [w-ga2onuf-lycp9f5t]
Waiting for operation [projects/.../locations/.../operations/operation-1723732900353-61fb9d7369659-d8faf47c-c179c45b] to complete
...done.
Started workstation [w-...-...].

Then this:

H:\>gcloud workstations list --project=...
ERROR: (gcloud.workstations.list) PERMISSION_DENIED: Permission 'workstations.workstations.list' denied on 'projects/.../locations/-/workstationClusters/-/workstationConfigs/-/workstations'. This command is authenticated as ...(username)... which is the active account specified by the [core/account] property

Then I configured proxy using gcloud config.

Then I tries the command gcloud workstations start ... again and it still work similarly.

Then I've tried again ssh:

gcloud workstations ssh --project=... --cluster=... --config=... --region=... w-...

Picking local unused port [61449].
Listening on port [61449].
ERROR: Error connecting to workstation: [Errno 11001] getaddrinfo failed

So, the same SSH problem, as it was before proxy configuring.
I think, gcloud workstations start doesn't use SSH.

Anyway, what does this [Errno 11001] getaddrinfo failed mean?

Have a nice day and thank you!
Vadim

[ruomengz]

Thank you for testing, looks like gcloud has the same behavior as the IDE does for creating the TCP tunnel. Is it possible that your proxy server does not support websockets properly? Are you able to connect without a proxy?

[ovavadim]

@ruomengz
Well, I see, that proxy configuring via gcloud config doesn't influence. But windows proxy is impossible to turn off (company rules), it will be always turned on, so I can't check the behaviour without proxy server.

Could you, maybe, specify, what exactly [Errno 11001] getaddrinfo failed means? What exactly does our proxy not support properly (I mean, which port, which operation etc). Probably something is simply blocked within a company, but then I need to know, what, in your opinion, is exactly blocked in order to cause this error?

[ruomengz]

I found a similar issue (updated link). Would you be able to try some workarounds in that issue?

[ovavadim]

@ruomengz
I'm sorry, but I don't have access there, so I can't open and see, what is inside.

[ruomengz]

So sorry about that, here is the updated link.

[ovavadim]

@ruomengz
Thank you for the links :)

Now I did the workarounds, see the data below.

H:\>gcloud info --run-diagnostics
Network diagnostic detects and fixes local network connection issues.
Checking network connection...done.
Reachability Check passed.
Network diagnostic passed (1/1 checks passed).

Property diagnostic detects issues that may be caused by properties.
Checking hidden properties...done.
Hidden Property Check passed.
Property diagnostic passed (1/1 checks passed).

gcloud workstations ssh --project=... --cluster=... --config=... --region=... w... --local-host-port=localhost:22222 --verbosity=debug
DEBUG: Running [gcloud.workstations.ssh] with arguments: [--cluster: "...", --config: "...", --local-host-port: "<googlecloudsdk.calliope.arg_parsers.HostPort object at 0x000002771D109490>", --project: "...", --region: "...", --verbosity: "debug", WORKSTATION: "w-..."]
DEBUG: Starting new HTTPS connection (1): workstations.googleapis.com:443
DEBUG: https://workstations.googleapis.com:443 "GET /v1/projects/.../locations/.../workstationClusters/.../workstationConfigs/...?alt=json HTTP/1.1" 200 None
DEBUG: Starting new HTTPS connection (1): workstations.googleapis.com:443
DEBUG: https://workstations.googleapis.com:443 "GET /v1/projects/.../locations/.../workstationClusters/.../workstationConfigs/.../workstations/w-...?alt=json HTTP/1.1" 200 None
DEBUG: Starting new HTTPS connection (1): workstations.googleapis.com:443
DEBUG: https://workstations.googleapis.com:443 "POST /v1/projects/.../locations/.../workstationClusters/.../workstationConfigs/.../workstations/w-...:generateAccessToken?alt=json HTTP/1.1" 200 None
Listening on port [22222].
DEBUG: Running command [C:\Apps\google-cloud-sdk\bin\sdk\putty.exe -t -P 22222 user@localhost].
DEBUG: Executing command: ['C:\\Apps\\google-cloud-sdk\\bin\\sdk\\putty.exe', '-t', '-P', '22222', 'user@localhost']
ERROR: Error connecting to workstation: [Errno 11001] getaddrinfo failed

Well, all the links say the same: the application can't resolve the IP address of the host.
So, I have a question: which address (DNS name) exactly can't be resolved?

Then I used nslookup command and that's what I see:

H:\>nslookup https://workstations.googleapis.com/
Server:  (corporative dns server name)
Address:  ...

*** (corporative dns server name) can't find https://workstations.googleapis.com/: Non-existent domain

H:\>nslookup https://oauth2.googleapis.com/
Server:  (corporative dns server name)
Address:  ...

*** (corporative dns server name) can't find https://oauth2.googleapis.com/: Non-existent domain

H:\>nslookup https://cloudresourcemanager.googleapis.com/
Server:  (corporative dns server name)
Address:  ...

*** (corporative dns server name) can't find https://cloudresourcemanager.googleapis.com/: Non-existent domain

H:\>nslookup https://openidconnect.googleapis.com/
Server:  (corporative dns server name)
Address:  ...

*** (corporative dns server name) can't find https://openidconnect.googleapis.com/: Non-existent domain

But maybe it tries to resolve another DNS name, which is not in the list of these four?

[ruomengz]

Thank you for the debugging logs, @ovavadim ! The team is aware of the issue and still investigating.

[ovavadim]

@ruomengz, hello!

I have an update here:

The slash sign in the end really matters.

Hope, this can help to understand the problem. Anyway, deleting the last slash signs in JetBrains Gateway API overriding (Settings -> Tools -> Cloud workstations - > Advanced) didn't help.

[ovavadim]

Hello, @ruomengz!

I have an interesting update.

It's related to the proxy. Google endpoint (....cloudworkstations.dev), which is in front of workstations, can only be reached via the proxy (there are some reasons for this).

When I use gcloud and set the proxy settings:

gcloud config set proxy/type http
gcloud config set proxy/address ...
gcloud config set proxy/port ...
gcloud config set proxy/username ...
gcloud config set proxy/password ...

then
gcloud workstations ssh --project=... --cluster=... --config=... --region=... w-...-...
doesn't work and still shows the error
ERROR: Error connecting to workstation: [Errno 11001] getaddrinfo failed

But when I use the command:
set HTTP_PROXY=http://<user>:<password>@<proxy_name>:<proxy_port>
then
gcloud workstations ssh --project=... --cluster=... --config=... --region=... w-...-...
works!

So, what's the difference between

gcloud config set proxy/type http
gcloud config set proxy/address ...
gcloud config set proxy/port ...
gcloud config set proxy/username ...
gcloud config set proxy/password ...

and
set HTTP_PROXY=http://<user>:<password>@<proxy_name>:<proxy_port>
?

When I set proxy settings in JetBrains Gateway, this still doesn't help anyhow.
So, we then return to my first question: how to set up proxy settings in JetBrains Gateway in order to make it work in "Google workstations" plugin?

Even more news:
If I firstly run
set HTTP_PROXY=http://<user>:<password>@<proxy_name>:<proxy_port>
then run
gcloud workstations start-tcp-tunnel --project=... --cluster=... --config=... --region=... w-...-... 22 --local-host-port=127.0.0.1:2222
then go to JetBrains Gateway and then start SSH connection (NOT with "Cloud workstations" plugin) with username "user", server "localhost", port 2222, then it works, I see IntelliJ loaded!

But the command
set HTTP_PROXY=http://<user>:<password>@<proxy_name>:<proxy_port>
influences only the command line and only before the closing the command line. When you run CMD again, you need to repeat set HTTP_PROXY=http://<user>:<password>@<proxy_name>:<proxy_port> again.

So, why this

gcloud config set proxy/type http
gcloud config set proxy/address ...
gcloud config set proxy/port ...
gcloud config set proxy/username ...
gcloud config set proxy/password ...

doesn't work while this
set HTTP_PROXY=http://<user>:<password>@<proxy_name>:<proxy_port>
works?