can't read camera World pose in AR mode

[riebling]

The technique for getting camera pose described here:
https://github.com/aframevr/aframe/blob/master/docs/components/camera.md#reading-world-position-or-rotation-of-the-camera
doesn't seem to work when in AR mode. (It used to work in all modes before A-Frame version 1.0) The reported world coordinates don't change with changes to device rotation (or translation, I think also), even though the camera's translation and rotation relative to the 3d scene elements clearly do. The technique for getting position and rotation of the camera (non-"world") on that page still works.

• A-Frame Version: 1.0 and higher
• Platform / Device: Chrome (w. sufficient config to support AR mode) on Android, Firefox-iOS (WebXR branch) on iOS

Maybe it's intentionally disabled in efforts to more fully support WebXR. Maybe AR mode implies that world coordinates are undefined?

[dmarcos]

It should work. Provide link to simple example to reproduce.

[riebling]

Sorry need to reopen. Was using cached code that worked around the issue.
Should be able to reproduce it at this URL:

https://xr.andrew.cmu.edu/x/refactor/

Code snippet:

AFRAME.registerComponent('pose-listener', {
    // if we want to make throttling settable at init time over mqtt,
    // create a Component variable here & use instead of globals.updateMillis
    init: function () {
        // Set up the tick throttling.
        this.tick = AFRAME.utils.throttleTick(this.tick, globals.updateMillis, this);
    },

    tick: (function (t, dt) {
        //      var newRotation = this.el.object3D.quaternion;
        //      var newPosition = this.el.object3D.position;
        const newPosition = new THREE.Vector3();
        const newRotation = new THREE.Quaternion();
        this.el.object3D.getWorldQuaternion(newRotation);
        this.el.object3D.getWorldPosition(newPosition);

        const rotationCoords = newRotation.x + ' ' + newRotation.y + ' ' + newRotation.z + ' ' + newRotation.w;
        const positionCoords = newPosition.x + ' ' + newPosition.y + ' ' + newPosition.z;

        const newPose = rotationCoords + " " + positionCoords;

        var worldCoords = coordsToText(newPosition);
        var localCoords = coordsToText(this.el.object3D.position);
        debugRaw("getWorldPosition :"+worldCoords+"\n"+
                 "object3D.position:"+localCoords);

        if (this.lastPose !== newPose) {
            this.el.emit('poseChanged', Object.assign(newPosition, newRotation));
            this.lastPose = newPose;

            // DEBUG
            //debugConixText(newPosition);
        }
    })
});

[riebling]

More info, possibly related problem.
In addition to camera's getWorldPosition() and getWorldQuaternion() not changing when in AR mode on mobile devices (as of A-Frame 1.0.x), when on iOS devices, the values of the camera's this.el.object3D.position and .quaternion are inverted with respect to the corresponding values in this.el.object3d.matrixWorld. On all(?) other devices, the position and quaternion match in sign and value what's in matrixWorld.

The end result being that camera reported position (in e.g. pose-listener Component for getting camera world pose change events) is the opposite of what it should be on iOS devices. This may be a result of WebXR code or WebXR polyfill in the Firefox-webxr (Fennec) viewer... since this is the code that is unique to iOS

[NNskelly]

I'm seeing what looks like the same lack of tracking on a Samsung S9 in a bigger 8th Wall project I'm naively PoCing over to vanilla a-Frame 1.0.3/4 sans-8th-Wall- the A-Scene and basic functionality works when 8W is stripped off, and in default static/locked-view mode things interact, but in AR I completely lose my view-groundplane-intersection reticle, meaning either

• the AR camera does not start where my a-camera is placed (when I pre-place content on the reticle in static-view mode, it ends up well above my head in AR...) OR
• I've completely lost my bearings in AR, OR
• the exposed a-camera parameters really are not actually tracking with the user in AR view.

Not sure if I'll have the open R&D time to look much further into it, tho, beyond noting that it doesn't "just work" (didn't expect it to per se, but it's a lot closer to working than I expected, and this seems like one of the main points of failure) and VR exhibits similar symptoms to AR.

Would it be reasonable to suspect this is a tangent to mrdoob/three.js#15347 and the root cause is that within an AR session, even the THREE camera is not actually continuously updated (as I've also observed in an independent test) and therefore some other mechanism must be used to dig out the underlying viewer reference frame?

[dmarcos]

@riebling probably better to contact 8th wall folks.

@riebling Your example is still pretty complicated to investigate. I see a lot of 3rd components pulled in.

Below there's a simple example reporting world position and rotation of the camera. Tested on desktop, Rift, Quest, phone;

Code: https://glitch.com/edit/#!/instinctive-vaulted-regnosaurus?path=index.html:27:85
Run: https://instinctive-vaulted-regnosaurus.glitch.me/

If you can reproduce the issue based on the code above I'm happy to look a it. Otherwise it's probably caused by ar.js / 8th wall integrations or other 3rd party components that you might be using. I recommend reporting the issue in the ar.js or 8th wall repos.

[NNskelly]

Ok; I've stripped my project down to only A-Frame 1.0.4, A-Frame Extras (likely not needed), 1 script making use of the cursor component on the camera to trace an object along the groundplane, and a basic (nearly empty) component registration script.

https://dev.nextnowagency.com/camera_ray_test/

Expected behavior:

1. in initial/static state, camera is facing into oblivion; no reticle displayed
2. in AR/VR, when looking downwards, cursor raycaster hits groundplane andupdates reticle position in placementReticle.js tick; camera position/rotation/heading & reticle transform confirmed in debug console logging

Observed behavior:

1. as expected
2. In both AR and VR modes, while console logging shows the camera position and quaternion changing, the getWorldDirection call itself mostly returns 0 0 1. Raycast never hits groundplane, reticle is never updated. After entering/exiting AR then entering/exiting VR, I notice the world direction readout (Source Heading) has in fact changed to a plausible non-initial value. Scrolling back through the log, it appears the world direction readout changed once upon exiting AR, reset likely upon entering VR, and changed again, once, upon exiting VR. There is a chance that these value changes are to frames of real data; there is, however, also a chance that they are related to the debugger tripping on the session-end promise rejection and any interpolation logic not liking to make up for a multiple-second timestop. Or it may be that the debugger tripping actually broke a race condition allowing the world-direction getter to return data that would otherwise have been reset every frame.
   I'll probably poke at this demo a little more to see whether it's just getWorldDirection that's affected, or, per the OP implication, all getWorld* calls.

[NNskelly]

New, even more minimal demo 100% confirming issues in AR/VR with most Object3D .getWorld* calls on the a-camera (position, quaternion, direction; did not test scale as I don't think VR/AR updates this). I'd almost suspect this to be purely at the THREE level, but it would be worth ruling out e.g. a timing race in exposing the THREE level up to the A-Frame level.

https://dev.nextnowagency.com/camera_world_test/

Expected behavior:

1. in initial/static state, all logging shows constant values for both local and world position/quaternion/heading
2. in AR and VR, both local and world readouts update to identical values (since the camera is loose in the root scene)

Observed behavior:

1. in initial/static state, logged position stays static but logged quaternion and heading values change dynamically, apparently in response to device motion. World and local quaternion and position values are, at least, in sync.
2. in AR and VR identically, local position and quaternion values update apparently in response to device motion. World position is static 0 0 0, world quaternion is static 0 0 0 1, world direction is static 0 0 1.

edit: corrected affected calls claim; also tagging @dmarcos

[dmarcos]

Is the example I provided not working for you? https://instinctive-vaulted-regnosaurus.glitch.me/

[NNskelly]

Right now glitch appears to be stalling (both code and demo), but that might just be ambient office network issues on my end atm.

I did take a peek at the code before setting up my demos, tho, and at a quick glance it looked like what was set to be reported were just the basic object3d properties, not the world getters. In my tests, the transform properties are just fine, it's the getters that return static defaults in VR/AR. Will let you know if I can actually get at your demo again later.

[NNskelly]

Ok; @dmarcos I remixed your example just adding the logging for the getWorld* call outputs. Seeing the same issue as in my demos: as soon as AR/VR is entered, they become default-looking values as opposed to following the actual transform properties.
So while object3d.matrixWorld may be functional, anything based on the getter functions doesn't look to be.

https://sparkling-moored-basket.glitch.me/

[dmarcos]

I don't understand the problem. In your link all values change on 2D mode (using keyboard) and when in VR mode (Firefox + Oculus Rift). Not sure what you mean by default-looking values or getter functions in the context of your example. FWIW, your code is slow. Don't allocate variables in the tick method and reuse:

https://glitch.com/edit/#!/lace-fern-ghost?path=index.html:71:39

[NNskelly]

What I'm seeing in the debugger is the above: the (your) printouts of the transform are showing convincing values in AR mode; the (my) printouts of the getWorld* returns are zero or unit vectors/quaternions completely unrelated to the direct transform readouts.
This is captured from a remote debug session of my remix of your sample running in latest chrome canary (mainline 80 doesn't support AR yet, and last I checked beta 81 is bugged) on a Samsung Galaxy S9. I'm guessing the screen preview shows rotate-your-device because it's not smart enough to capture the AR mode fullscreen takeover.
I am aware that allocating variables on tick is grotesquely inefficient, but code optimality did not seem relevant to demonstrating the issue at hand.

I'm not sure I understand your claim that it's working as intended. Both OP's report and my report deal with Android web browser behavior, not Oculus Rift. I concede it is possible that this is functionality which works on a mature desktop VR platform but not on mobile, but it seems odd that, to your point, the world matrix would be correct but the subsequently computed world transform getter functions (Object3D.getWorldPosition, Object3D.getWorldQuaternion, Object3D.getWorldDirection) would fail.

[NNskelly]

Also FWIW it looks like the OP's reference link also explicitly deals with the getWorldPosition and getWorldQuaternion calls, not the world matrix used in your sample.
If everything is watertight between the A-Frame layer and the THREE layer, and it's explicitly the THREE camera object3d that is misbehaving with THREE calls on mobile, perhaps this issue just needs to be tossed over to the THREE github, but I seem to recall seeing your name around there as well.

[dmarcos]

@NNskelly I see in your logs that the code provided in the example above is working as expected. You're getting world position and rotation. Is that correct? Below the component complete code if you are having issues with glitch. Can you confirm that the code is working as expected (use code as is, don't mix it to minimize noise)?

AFRAME.registerComponent('track-camera', {
  tick: (function () {
    const position = new THREE.Vector3();
    const rotation = new THREE.Quaternion();
    const euler = new THREE.Euler();
    const matrix = new THREE.Matrix4();
    return function () {
      this.el.object3D.updateMatrixWorld();
      position.setFromMatrixPosition(this.el.object3D.matrixWorld); // Vector3
      rotation.setFromRotationMatrix(matrix.extractRotation(this.el.object3D.matrixWorld));      
      console.log("Position: " + position.x + " " + position.y + " " + position.z);
      console.log("Rotation: " + rotation.x + " " + rotation.y + " " + rotation.z);
     }
   })()
});

[dmarcos]

@NNskelly FYI, getWorld* methods don't work in immersive mode. More info on mrdoob/three.js#18448

You can use the provided code

[NNskelly]

The lines labelled "Position" and "Rotation" are from your demo at https://instinctive-vaulted-regnosaurus.glitch.me/ using data derived from this.el.object3D.matrixWorld . I have not done a deep dive to see if those values are per se correct, but as they appear to update in correspondence to my moving the phone, I assume them to be correct.

The lines in my screenshot labeled World Position, World Quaternion and World Heading are coming from lines I added in my remix https://sparkling-moored-basket.glitch.me/ .

        let d = new THREE.Vector3();
        this.el.object3D.getWorldPosition(d);
        console.log("World Position: "+d.x+" "+d.y+" "+d.z);
        let q = new THREE.Quaternion();
        this.el.object3D.getWorldQuaternion(q);
        console.log("World Quaternion: "+q.x+" "+q.y+" "+q.z+" "+q.w);
        this.el.object3D.getWorldDirection(d);
        console.log("World Heading: "+d.x+" "+d.y+" "+d.z);

The added lines retrieve the world transform parameters via the Object3D .getWorldPosition, .getWorldQuaternion and .getWorldDirection calls. The OP observed and cited in their reference link that .getWorldPosition and .getWorldQuaternion were not returning valid data in VR mode, and my earliest observations showed that .getWorldDirection also was not returning valid data. All 3 seem to fail specifically in VR/AR mode on mobile, although I have not deep-tested their behavior on mobile outside VR/AR (it's a bit harder to use the stock A-Frame keyboard controls on mobile).

And by "fail" I mean that the World Position from getWorldPosition, listed as 0 0 0, does not match the Position from the worldMatrix listed as or about 0.03 1.13 0.02 and the World Quaternion from getWorldQuaternion, listed as 0 0 0 1, would not yield Eulers from the worldMatrix listed as or about Rotation -0.38 0.01 -0.05. Neither would that listed Rotation be consistent with a World Direction (positive Z axis) returned from getWorldDirection as 0 0 1

I concede that it is technically possible to use alternate methods to acquire a valid world position for the camera, but the issue report, as twice stated and variously demonstrated, is that in VR/AR mode, at least on mobile, at least when querying the el.object3d of the A-Frame camera, the specific calls getWorldPosition, getWorldQuaternion, and getWorldDirection (all documented on https://threejs.org/docs/#api/en/core/Object3D as being valid and intended ways to get the world coordinates of an object) do not return data consistent with the effective functioning position of the queried object.

I have not proven that this issue is specific to A-Frame's particular use of the THREE camera, indeed I suspect it isn't.
But yes, ok, if it is a known THREE issue that getWorld* does not work in VR/AR, then that is exactly the issue.

[edusuko]

I have been solving a similar problem. I needed to get a point in front of camera using aframe API. But the challege was when the experience were on VR mode(fullscreen) and playing on movile or headset. In this context the management of the current camera is absolutely controlled by WebXR. With WebXR THREE applies headset pose to the object3D internally.

You only can use the matrixWorld of the three camera to access the camera world reference data, other properties or methods are not correct. In the case of aframe you must access to the object3D of the aframe camera entity and manage its matrixWorld. It is the only method to get correct information of the position/rotation/scale of the camera that it is move by the sensors of a movile or of a AR/VR goggles when the play is on VR/AR mode.

I use to get the in front of Camera Position With WebXR Headset Pose:

const distanceFromCamera = 250; // the depth in the screen, what ever you need
const inFrontOfCameraPosition = new AFRAME.THREE.Vector3( 0, 0, -distanceFromCamera );

const threeSceneCamera = <THREE.PerspectiveCamera>AFRAME.scenes[0].camera; inFrontOfCameraPosition.applyMatrix4( threeSceneCamera.matrixWorld );

return { x: inFrontOfCameraPosition.x, y: inFrontOfCameraPosition.y, z: inFrontOfCameraPosition.z };

[dmarcos]

FYI, if you are not using a camera rig you can retrieve the world position of the camera from the camera entity in both VR and desktop:

document.querySelector('a-scene').camera.el.getAttribute('position');
document.querySelector('a-scene').camera.el.getAttribute('rotation');

The matrixWorld that should be used is the one in the camera entity object3D:

document.querySelector('a-scene').camera.el.object3D.matrixWorld

[edusuko]

AFRAME.scenes[0].camera
and
document.querySelector('a-scene').camera.el.object3D
is the same.

In my case, I am using a-cursor in mobile but in a headset (for example Meta Quest2) not.

The only way that resolve my necesity in VR or AR is to use directly the camera matrixWorld.

[dmarcos]

AFRAME.scenes[0].camera and document.querySelector('a-scene').camera.el.object3D should be different.

Former is the THREE.PerspectiveCamera and latter is the entity that contains the camera (THREE.PerspectiveCamera is a child). A-Frame applies the WebXR pose to the camera entity, not the THREE camera. In most scenarios using the camera.matrixWorld will be equivalent but keep in mind that you're tapping into the internals and that could break in the future. That's why I recommend using cameraEl.object3D.matrixWorld instead.

[edusuko]

I have finally used this option that get the position in front of the Aframe camera at a defined distace. It works in all the situations that I have tested, even using look-controls or in vr mode with Quest 2 or in ar mode with a mobile.

const getInFrontOfCameraPosition = ( distanceFromCamera) => {

const aSceneCameraEntity = <Entity>document.querySelector('a-scene').camera.el;

const inFrontOfCameraPosition = new AFRAME.THREE.Vector3( 0, 0, -distanceFromCamera );
inFrontOfCameraPosition.applyMatrix4( aSceneCameraEntity.object3D.matrixWorld );

return { x: inFrontOfCameraPosition.x, y: inFrontOfCameraPosition.y, z: inFrontOfCameraPosition.z };

};

Thank you marcos by your tips.

[edusuko]

I have the doubt, what would be better or faster to get the camera:

option 1: const aSceneCameraEntity = document.querySelector('a-scene').camera.el;
option 2: const aSceneCameraEntity = AFRAME.scenes[0].camera.el;