Jetty 12 - Flaky test org.eclipse.jetty.websocket.tests.proxy.WebSocketProxyTest.testEcho

[joakime]

Jetty version(s)
Jetty 12

Java version/vendor (use: java -version)
17 & 19

OS type/version
Linux

Description
New flaky test showing up ...

Expected: is "hello world!"
     but: was null
	at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
	at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:6)
	at org.eclipse.jetty.websocket.tests.proxy.WebSocketProxyTest.testEcho(WebSocketProxyTest.java:117)

Running org.eclipse.jetty.websocket.tests.proxy.WebSocketProxyTest.testEcho()
2023-03-29 15:58:27.823:INFO :oejs.Server:main: jetty-12.0.0-SNAPSHOT; built: 2023-03-29T15:28:30.456Z; git: a818a5369903bba82eaebc7e5c646e47999cd3fb; jvm 17.0.6+10
2023-03-29 15:58:27.825:INFO :oejsh.ContextHandler:main: Started oejsh.ContextHandler@3af7d855{ROOT,/,b=null,a=AVAILABLE,h=oejws.WebSocketUpgradeHandler@77049094{STARTED}}
2023-03-29 15:58:27.825:INFO :oejs.AbstractConnector:main: Started ServerConnector@f88bfbe{HTTP/1.1, (http/1.1)}{0.0.0.0:42983}
2023-03-29 15:58:27.825:INFO :oejs.Server:main: Started oejs.Server@1b7a52dd{STARTING}[12.0.0-SNAPSHOT,sto=0] @29760ms
2023-03-29 15:58:32.838:INFO :oejwc.WebSocketClient:main: Shutdown WebSocketClient@6ab9df83[coreClient=oejwcc.WebSocketCoreClient@440e3ce6{STARTED},openSessions.size=1]
2023-03-29 15:58:32.842:INFO :oejs.Server:main: Stopped oejs.Server@1b7a52dd{STOPPING}[12.0.0-SNAPSHOT,sto=0]
2023-03-29 15:58:32.843:INFO :oejs.AbstractConnector:main: Stopped ServerConnector@f88bfbe{HTTP/1.1, (http/1.1)}{0.0.0.0:0}

[sbordet]

The issue was that WebSocketUpgradeHandler was marked as invocationType==NON_BLOCKING.
As such, the NIO subsystem was invoking HttpConnection.onFillable() with mode PC (and therefore from the selector thread), which eventually ends up calling EndPoint.upgrade(), which eventually calls the "connect" event handler, in this case WebSocketProxy.ClientToProxy.onWebSocketConnect() that uses a blocking APIs, namely CompletableFuture.get().

With the selector thread blocked there, the proxy was connecting to the same server, and if the server had only 1 selector, it would now be deadlocked.

This test is easily reproducible using taskset -c 0 to run the JVM because then the JVM assumes that there is only 1 CPU (so the server allocates just one selector).

[sbordet]

Fixed by #9553.