WebXRManager: support more than 2 views

[CodyJasonBennett]

Related issue: #19387

Description

I was chatting with @BryanChrisBrown about WebXRManager's limitation to two XRCameras (also noted in three-types/three-ts-types#169) and found @brianpeiris's previous work sufficient enough to support multiview as described in kainino0x/holoplay-webxr#9 (comment).

I tested this with a few demos and a (webxr) emulator equipped with multiview to try:

r140 dev (baseline)

Demo: https://webxr-multicam-4zetdnsmg-codyb.vercel.app/
Code: https://github.com/CodyJasonBennett/webxr-multicam/blob/r140dev-baseline/index.html

fix/webxr-multicamera (PR)

Demo: https://webxr-multicam.vercel.app/
Code: https://github.com/CodyJasonBennett/webxr-multicam/blob/main/index.html

The only notable changes since have been with RGB depreciations, internal encoding changes, and foveation & reference space features so this worked OOTB.

[cabanier]

Awesome work!
Does this also work if the scene has reflections?

[CodyJasonBennett]

@cabanier, I haven't noticed anything wrong with rendering. Is there a specific example I can look at?

[cabanier]

@cabanier, I haven't noticed anything wrong with rendering. Is there a specific example I can look at?

try https://threejs.org/examples/?q=vr#webxr_vr_sandbox

[CodyJasonBennett]

I'd have to ask @BryanChrisBrown for what this would look like on a real device, but looks fine with multiple views.

Demo: https://webxr-multicam-2v9u5dip5-codyb.vercel.app
Code: https://github.com/CodyJasonBennett/webxr-multicam/blob/reflections/index.html

[CodyJasonBennett]

I have a branch with a build to test without an emulator: https://rawcdn.githack.com/CodyJasonBennett/three.js/d31e899eebf6a9282668672abd49e9bcd6ce6f07/examples/index.html

[BryanChrisBrown]

Confirming refraction & reflection work on the Looking Glass as well, we can probably crank the default view resolution a little bit in the config, I’ll submit a PR to HoloPlay WebXR for that though.

IMG_0316.mov

[cabanier]

Can you point me to all the changes that were made to support multiview? Your PR just mentions the perspective camera.
Even though MV is great, we can't make it the default because it doesn't support multisampled output.

[BryanChrisBrown]

ac56591

So the changes to support adding multiple cameras are here, the request for multiple cameras comes from WebXR’s Device API.

So the HoloPlay WebXR library is essentially acting as the Device in this case and requesting n number of views.

I think there may be a miscommunication with the term multi-view here, I think we’re using it in “this supports multiple views” where as multi-view is an actual render optimization right?

[cabanier]

ah. I see :-)
We're still planning on doing gl based multiview. Would that interfere with this change?

[CodyJasonBennett]

Sorry for the confusing language, I updated the title. This PR implements support for multiple XRViews, I do not implement OVR_multiview2, but would be interested in what that would entail (although for another PR).

We're still planning on doing gl based multiview. Would that interfere with this change?

Not if you're setting the number of views based on pose, seems that's all we'd need. Although it looks like there's a MAX_VIEWS_OVR which I'm not sure if you'd prefer to read from (or how that would interact with devices like the looking glass where it may not be using all of its hardware capabilities).

I'd be happy to put together a demo with both if you think that would help to greenlight the change. I experimented with a basic XR renderer before which I can A/B with (will have to port to three, syntax is slightly different):

[BryanChrisBrown]

Having a 4 view limitation would definitely hurt us here since the Looking Glass requires a minimum of 45 views, going over 45 isn’t required on the Looking Glass Portrait device, but the larger Looking Glass 4K & 8K device do benefit from a slightly larger number of views.

So it doesn’t look like going the multi-view route would work for our case given the current four view limitation, is this something that could be amended or is it a hardware limitation for graphics processors?

[CodyJasonBennett]

I suppose see #24048 for an implementation.

[CodyJasonBennett]

Are there any outstanding issues with this PR? AFAIK OCULUS_multiview (see #24048) would be limited to headsets like the Quest, but it looks like the code path for other WebXR devices would remain intact.

[mrdoob]

All good. Thanks!