Add an example that uses WebXR's estimated AR lighting in feature. #20876
Conversation
| // approximately how frequently the event would fire anyway). | ||
| setInterval( () => { | ||
|
|
||
| this.updateReflection(); |
There was a problem hiding this comment.
Does updating the environment have a cost in the event we aren't using it? And is there cost on the JS side, or only on the implementation side?
There was a problem hiding this comment.
Good point! There is a cost to updating it, which shouldn't be frame-droppingly severe but if your app wasn't using it (like you suggest below) then it's also wasted effort. We could turn the environment map into a getter attrib and only start updating it if you query it as a start, but that doesn't provide an easy way to stop using it, and it could accidentally get triggered by things like iterating over the object properties.
There was a problem hiding this comment.
Might be worth just setting an input parameter to use it or not.
|
|
||
| } | ||
|
|
||
| onXRFrame( time, xrFrame ) { |
There was a problem hiding this comment.
Does this mean the ARCore updates the directional light and SH on every frame but only updates the environment roughly every second? That seems odd; I would have expected all of this to be in the slow update function.
There was a problem hiding this comment.
ARCore probably only updates those values once in a while, but updating these values is pretty low-overhead (it's just changing which values will be set for a couple of uniforms, which happens per-frame anyway in most cases.) As a result on the API side we didn't consider it a burden to just treat the values as if they're new each frame. We could add an event to indicate concretely when changes happen if needed, but I'd like to see some real-world uses before determining if that's useful or not.
The environment maps are a different story, since not only do they involve creating a texture but also may also involve a behind-the-scenes format conversion. As such they're much heavier weight, relatively, to produce and we structured the API to encourage a sparser query.
examples/webxr_ar_lighting.html
Outdated
| scene.remove(defaultLight); | ||
|
|
||
| // The estimated lighting also provides an environment cubemap, which we can apply here. | ||
| scene.environment = xrLight.environment; |
There was a problem hiding this comment.
Agreed that it's nice this is optional. Since high-res reflection with PMREM is so important for PBR, I'm thinking I might skip the estimated environment and use a static PMREM'ed one, but maybe a non-HDR like a jpg, so that most of the intense lighting comes from the directional and SH. I'm not a big fan of SH, but it's probably better than the alternatives.
There was a problem hiding this comment.
I'd be really interested in seeing the results you can get out of that! There's no question that the low res environment map has limits. Everything tends to look rougher than intended above a certain point. That said if you have a perfectly crisp reflection that doesn't match your environment that will probably stand out too.
I wonder if there's a middle ground where you tone map a high res generic environment with the colors coming out of the low res one. 🤔 Gonna guess that falls into the "More trouble than it's worth" category, though.
There was a problem hiding this comment.
I'm hoping the SH will tend to provide that tone map. As for the perfectly crisp reflection; no one will notice except on a spherical mirror like you have here. Most real objects are complex enough that you really can't tell what's reflected anyway. All that matters is how sharp the features are, which we need resolution to represent.
|
I'm trying this on Chrome Beta (88.0.4324.51) and Chrome Dev (89.0.4356.5) but I only see the spheres reflecting the equirect. Does this need a flag? |
|
Sorry for the very long delay following up! Rebased this change and made a couple of small adjustments based on the feedback above:
And @mrdoob: Yes, you do currently need a flag ("XR Incubations" in about:flags) but it's expected to ship enabled by default fairly soon. There is an origin trial available which we could apply to this example, but I'm not sure if it's something you want to bother with? |
|
@toji Awesome feature, on Android it crashes Chrome 88 but it seems to work at least for light intensity on Canary 90 (should colours be estimated as well?). Do any current MR headsets support it? |
|
Thanks! |
Sorry, that was a known issue that has now been fixed. |
This PR adds an example and utility class for showing how to take advantage of estimated lighting in AR.
The
XREstimatedLightclass can be added to the scene as if it were a regular light and any time an AR session with the appropriate permissions is started it will begin updating the lighting estimates automatically. It also has anenvironmentproperty that automatically updates with an estimated cube map of the user's environment, but this is not automatically applied to the scene. We could easily make it do so, but I'm not sure that's the right choice for developers.Internally the "light" is a group that contains both a
LightProbeand aDirectionalLight, as well as the aformentionedenvironmentcube map. I didn't see a need to protect these lights from being inspected or modified by users, since they may conceivably want to set them to non-AR-provided defaults at some point or evaluate the AR-provided values for some other purpose.For scenarios where developers want to switch between a default lighting setup and AR lighting when appropriate the
XREstimatedLightalso provides aestimationstartandestimationendevent to let developers know when the lights values are actively being updated. The example shows how to swap between lighting configurations when these events occur. It may also be beneficial to provide an event that indicates whenever theenvironmentcube map is updates, in case developers want to do some processing on it (such as generating more accurate roughness mipmaps). I haven't added that yet in the interest of keeping API surface limited, but I'd like to know what developers need.CC: @elalish, since I think he would like to use this in
<model-viewer>and I'd appreciate feedback on if this would work for that use case.