From a05c73aa9324bada695d8d59f08d4743b39314d6 Mon Sep 17 00:00:00 2001 From: Maksims Mihejevs Date: Wed, 29 Jul 2020 14:49:03 +0300 Subject: [PATCH 1/2] user manuals for webxr hands --- content/en/user-manual/xr/index.md | 2 +- content/en/user-manual/xr/input-sources.md | 47 ++++++++++++++++++++-- content/en/user-manual/xr/using-webxr.md | 6 +-- 3 files changed, 47 insertions(+), 8 deletions(-) diff --git a/content/en/user-manual/xr/index.md b/content/en/user-manual/xr/index.md index 42292ea5930..ce254bb2495 100644 --- a/content/en/user-manual/xr/index.md +++ b/content/en/user-manual/xr/index.md @@ -31,7 +31,7 @@ button.element.on('click', function () { // check if XR is supported and VR is available if (app.xr.supported && app.xr.isAvailable(pc.XRTYPE_VR)) { // start VR using a camera component - entity.camera.startXr(pc.XRTYPE_VR, pc.XRSPACE_LOCAL); + entity.camera.startXr(pc.XRTYPE_VR, pc.XRSPACE_LOCALFLOOR); } }); ``` diff --git a/content/en/user-manual/xr/input-sources.md b/content/en/user-manual/xr/input-sources.md index de3e01a0449..e963877b768 100644 --- a/content/en/user-manual/xr/input-sources.md +++ b/content/en/user-manual/xr/input-sources.md @@ -19,7 +19,7 @@ for (var i = 0; i < inputSources.length; i++) { } ``` -Input sources can be added and removed dynamically. This can be done by connecting physical devices, or some input sources have a lifespan only during their primary action (for example, a touchscreen in an AR-type session on mobile). You can subscribe to `add` and `remove` events: +Input sources can be added and removed dynamically. This can be done by connecting physical devices or by switching input devices by underlying platform, and some input sources have a lifespan only during their primary action (for example, a touchscreen in an AR-type session on mobile). You can subscribe to `add` and `remove` events: ```javascript app.xr.input.on('add', function (inputSource) { @@ -33,7 +33,7 @@ app.xr.input.on('add', function (inputSource) { ## Primary Action (select) -Each input source can have a primary action `select`. On gaze-based input sources, it is the touch of a screen/button. On handheld devices, it is the primary button/trigger. There are also `selectstart` and `selectend` events which you can subscribe to as follows: +Each input source can have a primary action `select`. On gaze-based input sources, it is the touch of a screen/button. On handheld devices, it is the primary button/trigger. For tracked hands it is emulated by PlayCanvas engine when thumb and index tips are touching. There are also `selectstart` and `selectend` events which you can subscribe to as follows: ```javascript inputSource.on('select', function () { @@ -51,11 +51,12 @@ app.xr.input.on('select', function (inputSource) { ## Ray -Each input source has a ray which has an origin where it points from, and a direction in which it is pointing. A ray is transformed in XR camera parent's space. Some examples of input sources might be, but not limited to: +Each input source has a ray which has an origin where it points from, and a direction in which it is pointing. A ray is transformed to world space. Some examples of input sources might be, but not limited to: * Gaze-based input, such as a mobile device which is inserted into a Google Cardboard™ style device. It will have an input source with `targetRayMode` set to `pc.XRTARGETRAY_GAZE`, and will originate from the viewer's position and point straight where the user is facing. * Screen-based input. This might be available on mobile devices in Augmented Reality session types, where the user can interact with the virtual world by touchscreen. * Handheld devices, like the Oculus Touch™, will have a ray originating from the tip of the handheld device and the direction is based on the rotation of device. + * Tracked Hands have an emulated by PlayCanvas engine ray that originates from point between thumb and index tips, and pointing forward. Here is an example illustrating how to check whether a ray has intersected with the bounding-box of a mesh: @@ -70,7 +71,7 @@ if (meshInstance.aabb.intersectsRay(ray)) { ## Grip -Some input sources are associated with a physical handheld device, such as an Oculus Touch™, and can have position and rotation. Their position and rotation are calculated in XR camera parent's space. +Some input sources are associated with a physical handheld device, such as an Oculus Touch™, and can have position and rotation. Their position and rotation are provided in world space. ```javascript if (inputSource.grip) { @@ -94,6 +95,40 @@ if (gamepad) { } ``` +## Hands + +If platform supports [WebXR Hand Input][7], then an input source might have an associated hand data, which is exposed as [XrHand][8], and provides convenient information in form of [XrFinger][9] and [XrJoint][10] for application developer to use, such as wrist, fingers, each joint, tips and events for detecting when hands loose/restore tracking. + +Creating basic hand model: + +```javascript +var joints = [ ]; +var hand = inputSource.hand; + +if (hand) { + for(var i = 0; i < hand.joints.length; i++) { + var entity = new pc.Entity(); + entity.joint = hand.joints[i]; + entity.addComponent('model', { type: 'box' }); + parent.addChild(entity); + joints.push(entity); + } +} +``` + +And synchronising it on every update: + +```javascript +for(var i = 0; i < joints.length; i++) { + var entity = joints[i]; + var joint = entity.joint; + var radius = joint.getRadius() * 2; + entity.setLocalScale(radius, radius, radius); + entity.setPosition(joint.getPosition()); + entity.setRotation(joint.getRotation()); +} +``` + ## Profiles Each input source might have a list of strings describing a type of input source, which is described in a [profile registry][6]. Based on this, you can figure out what type of model to render for handheld device or what capabilities it might have. Additionally, the profile registry lists gamepad mapping details, such as buttons and axes. @@ -111,3 +146,7 @@ if (inputSource.profiles.indexOf('oculus-touch-v2') !== -1) { [4]: https://www.w3.org/TR/webxr-gamepads-module-1/ [5]: https://w3c.github.io/gamepad/ [6]: https://github.com/immersive-web/webxr-input-profiles/tree/master/packages/registry +[7]: https://immersive-web.github.io/webxr-hand-input/ +[8]: /api/pc.XrHand.html +[9]: /api/pc.XrFinger.html +[10]: /api/pc.XrJoint.html diff --git a/content/en/user-manual/xr/using-webxr.md b/content/en/user-manual/xr/using-webxr.md index d37922facf3..330922de7bf 100644 --- a/content/en/user-manual/xr/using-webxr.md +++ b/content/en/user-manual/xr/using-webxr.md @@ -20,7 +20,7 @@ if (app.xr.supported) { The API for entering XR is on the Camera Component or [XrManager][2] on the Application. To start VR presenting you should use the `startXr` method on a CameraComponent and provide the type of session and reference space type: ```javascript -entity.camera.startXr(pc.XRTYPE_VR, pc.XRSPACE_LOCAL); +entity.camera.startXr(pc.XRTYPE_VR, pc.XRSPACE_LOCALFLOOR); ``` It is an asynchronous operation and is only possible to start on a user interaction, such as a button click, mouse click or touch. To know when a session is started, you can subscribe to the `start` event: @@ -90,11 +90,11 @@ app.xr.on('available:' + pc.XRTYPE_VR, function (available) { ## Camera Position and Orientation in XR -When you are presenting in XR, the position and orientation of the camera are overwritten by data from the XR session. If you want to implement additional movement and rotation of camera, you should add a parent entity to your camera and apply your translation to this entity. +When you are presenting in XR, the position and orientation of the camera are overwritten by data from the XR session. If you want to implement additional movement and rotation of camera, you should add a parent entity to your camera and apply your manipulations to that entity. ![Camera Offset][1] -Input source ray, as well as position and rotation, are in reference space. So if the camera position is manipulated by a parent transformation, then the input source ray, position and rotation should be transformed accordingly. +Input source ray, as well as position and rotation of grip and hands, provided in world space. ## Why can't I enable XR mode automatically? From 2176d61a6138d7352e663dd57fa54f61af3fc0c9 Mon Sep 17 00:00:00 2001 From: Maksims Mihejevs Date: Tue, 4 Aug 2020 00:55:30 +0300 Subject: [PATCH 2/2] reflect recent API changes for XrJoint --- content/en/user-manual/xr/input-sources.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/en/user-manual/xr/input-sources.md b/content/en/user-manual/xr/input-sources.md index e963877b768..f25084fb539 100644 --- a/content/en/user-manual/xr/input-sources.md +++ b/content/en/user-manual/xr/input-sources.md @@ -122,7 +122,7 @@ And synchronising it on every update: for(var i = 0; i < joints.length; i++) { var entity = joints[i]; var joint = entity.joint; - var radius = joint.getRadius() * 2; + var radius = joint.radius * 2; entity.setLocalScale(radius, radius, radius); entity.setPosition(joint.getPosition()); entity.setRotation(joint.getRotation());