VR and AR in code

A WebGL 3D piece goes into VR, AR or both with one line: an xr export. The runtime does the rest. It shows the Enter VR and Enter AR buttons where the device can do them, starts the session, puts the camera on the viewer's head, draws both eyes, and hands your code the controllers, the hands and, in AR, the room. This page is for writing that code by hand. For what a viewer sees, see VR pieces and AR pieces.

Opting in

Add the export at the top level of the module, beside the default export:

export const xr = 'vr';

Only the WebGL 3D engine can go into VR or AR. Any other value is ignored, with a warning in the console.

When the piece starts, the runtime asks the browser which of the declared modes it can enter, and puts a button for each at the bottom of the frame: Enter VR, Enter AR. On a device that can do neither, no button shows and the piece plays on the screen as usual, so it must still work there with mouse, touch and keys. The buttons show in the studio, on share links, in embeds and on the VR and AR page; the embed code already allows them. They don't show in thumbnails or video renders.

What the runtime does for you

• Starts the session from the button, with the floor at y = 0 where the device knows where the floor is.
• Moves the camera into xr.rig for the session. The headset moves the camera; your code moves the rig.
• Draws scene with camera for each eye. A render() your sketch returns, and with it any post-processing, is used on the screen only.
• Reads every controller's buttons and sticks once a frame, before update().
• Puts the camera back as it was, and the renderer at the page's size, when the viewer leaves.

So never add VRButton, ARButton or XRButton, call navigator.xr or renderer.xr.setSession, or start an animation loop of your own.

Units and the viewer

Build in metres, with the floor at y = 0. The viewer stands in xr.rig, eyes about 1.6 m above it, looking down -z.

• Place the rig where they start: xr.rig.position.set(0, 0, 3).
• Move or turn the rig to move them: walking with a stick, teleporting, riding something.
• Never set the camera's position or rotation for VR. Read where the viewer is and where they look with camera.getWorldPosition(v) and camera.getWorldDirection(v).
• Set the screen camera up as usual, near the same viewpoint (camera.position.set(0, 1.6, 3)). It's restored when they leave.

On a phone in AR there may be no floor to stand on: the space then starts where the phone is, so stand things on a surface it finds rather than at fixed heights.

The xr object

Controllers

Each of xr.controllers has:

grip, ray and the rig are already in the scene, so add things to them directly.

Events

const stop = xr.on('select', ({ controller, mode }) => { … });
stop(); // no longer listening

Every listener gets { type, controller, mode }, with controller null for the events that aren't about one. enter fires on the first frame in the session, when the camera is already where the viewer's head is.

A VR piece

Orbs to pop with the trigger in VR, or with a click or tap on a screen, and the left stick to walk.

import * as THREE from 'three';

export const xr = 'vr';

export default function sketch({ scene, camera, canvas, size, xr }) {
  scene.background = new THREE.Color('#05060a');
  scene.add(new THREE.HemisphereLight('#bcd4ff', '#101018', 1.2));

  // The viewer starts 3 m back from the middle, and so does the screen camera.
  xr.rig.position.set(0, 0, 3);
  camera.position.set(0, 1.6, 3);
  camera.lookAt(0, 1.4, 0);

  const orbs = new THREE.Group();
  scene.add(orbs);
  for (let i = 0; i < 9; i++) {
    const orb = new THREE.Mesh(
      new THREE.SphereGeometry(0.15, 24, 16),
      new THREE.MeshStandardMaterial({ color: '#ffb86b', emissive: '#ff7a1a', emissiveIntensity: 0.8 }),
    );
    orb.position.set((i - 4) * 0.45, 1.2 + (i % 3) * 0.3, 0);
    orbs.add(orb);
  }

  const pop = (hit, controller) => {
    if (!hit) return;
    orbs.remove(hit.object);
    controller?.pulse(0.6, 40);
  };

  // In VR: the trigger, or a pinch.
  xr.on('select', ({ controller }) => pop(controller.intersect(orbs)[0], controller));

  // On a screen: a click or a tap.
  const ray = new THREE.Raycaster();
  canvas.addEventListener('pointerdown', (e) => {
    const at = new THREE.Vector2((e.clientX / size.width) * 2 - 1, -(e.clientY / size.height) * 2 + 1);
    ray.setFromCamera(at, camera);
    pop(ray.intersectObjects(orbs.children)[0]);
  });

  return {
    update({ delta }) {
      // Walk with the left stick, at walking pace.
      const left = xr.left;
      if (left) {
        xr.rig.position.x += left.stick.x * delta * 1.5;
        xr.rig.position.z -= left.stick.y * delta * 1.5;
      }
    },
  };
}

In the room

AR adds the real room. Everything above holds in AR too.

The backdrop

In AR the runtime leaves out scene.background and the fog, and hides xr.backdrop, a group already in the scene. Put whatever stands in for a room on a screen and in VR (a floor, walls, a table top, a sky) in xr.backdrop, so the piece looks complete there and the real room shows in AR. Keep what the viewer plays with out of it.

Additive blending adds light over the room, as it does on a screen, so glows, sparks and trails work as they are.

Surfaces

xr.surfaces lists what the headset or phone has found of the room. Each surface has:

xr.surface('table') returns the largest surface with that label, and xr.surface((s) => …) the largest that passes a test, else null. A phone's planes are often all 'other', so tell them apart by orientation and height.

Surfaces arrive a moment after the session starts, change as the viewer looks round, and may never come: no room set up, permission refused, or a phone that only finds flat planes. Start with a sensible placement in front of the viewer on enter, listen for surfaces, and move onto one when it turns up. Never wait for one before showing anything.

Rays against the room

xr.room is the group of the surfaces' meshes, in the scene and hidden. Rays still find them: controller.intersect(xr.room) gives the real wall or table a controller points at, with its surface in hit.object.userData.surface, and a Raycaster against xr.room lets things bounce, roll and land on the real room. Set xr.room.visible = true only to show what was found.

controller.surface is where that controller's ray, or a finger on a phone, meets a real surface this frame, as { point, normal } in the world. xr.gaze is the same for the middle of the view, a phone's crosshair. Both change every frame, so clone() what you keep.

Pinning

xr.pin(object) keeps an object where it is in the real room while AR runs, even as the device re-tracks. Call it after placing the object, and xr.unpin(object) before moving it again. It returns false outside AR or on a device without anchors. Pins are dropped when the session ends.

Occlusion

Set xr.occlusion = true before the viewer goes in, and real hands, people and furniture hide what's behind them, on headsets that can (Quest 3). It uses the headset's depth where it has it, and the scanned room otherwise. Leave it off for things that sit on or in a wall or table, which it would cut into.

On a phone

There are no controllers on a phone. A tap on the screen is a select from a controller whose targetRayMode is 'screen', with its ray running from the phone through the finger, so the same xr.on('select', …) handles taps, triggers and pinches.

An AR piece

A lamp that starts in front of the viewer, moves onto a table when one is found, and goes wherever they point and pull the trigger, or tap.

import * as THREE from 'three';

export const xr = ['vr', 'ar'];

export default function sketch({ scene, camera, xr }) {
  scene.add(new THREE.HemisphereLight('#ffffff', '#404050', 1.5));

  // A floor for screens and VR; hidden in AR, where the real floor shows.
  const floor = new THREE.Mesh(new THREE.CircleGeometry(4, 48), new THREE.MeshStandardMaterial({ color: '#1a1c24' }));
  floor.rotation.x = -Math.PI / 2;
  xr.backdrop.add(floor);

  const lamp = new THREE.Mesh(
    new THREE.IcosahedronGeometry(0.1, 2),
    new THREE.MeshStandardMaterial({ color: '#77ccff', emissive: '#2299ff' }),
  );
  const home = new THREE.Vector3(0, 1, -1);
  lamp.position.copy(home);
  scene.add(lamp);

  let moved = false;
  const placeOn = (point) => {
    xr.unpin(lamp);
    lamp.position.copy(point);
    lamp.position.y += 0.1;
    xr.pin(lamp);
  };

  // Start in front of the viewer, below eye height.
  const head = new THREE.Vector3();
  const look = new THREE.Vector3();
  xr.on('enter', ({ mode }) => {
    moved = false;
    if (mode !== 'ar') return;
    camera.getWorldPosition(head);
    camera.getWorldDirection(look).setY(0).normalize();
    placeOn(head.clone().addScaledVector(look, 0.8).setY(head.y - 0.5));
  });

  // Onto a table, when one turns up.
  xr.on('surfaces', () => {
    const table = xr.surface('table');
    if (xr.mode === 'ar' && table && !moved) placeOn(table.center);
  });

  // Point at a flat surface and pull the trigger, or tap it on a phone.
  xr.on('select', ({ controller }) => {
    const hit = controller.surface;
    if (xr.mode === 'ar' && hit && hit.normal.y > 0.8) {
      moved = true;
      placeOn(hit.point);
    }
  });

  xr.on('exit', () => lamp.position.copy(home));
}

What doesn't reach the headset

• The page's overlays: a timeline's text and image layers, the pointer and the gui panel don't show in VR or AR. Put scores, names and menus on planes in the world, such as a CanvasTexture panel 1 to 2 m away, or on a controller's grip.
• Post-processing: an EffectComposer and its bloom show on the screen only. Make the look stand on its own with emissive materials, additive blending and fog.
• Lighting for the runtime's models: the controller and hand models are lit like the rest of the scene, so keep a light in it, even a HemisphereLight, or they show black.

Performance and comfort

Headsets draw two eyes at 72 to 90 frames a second on a mobile chip. Keep draw calls low with instancing and merged geometry, particles in the low thousands, at most one light casting shadows, simple MeshStandardMaterial or MeshBasicMaterial, and light fragment shaders. Rendering into render targets in update() (GPUComputationRenderer, Reflector, CubeCamera) tends to break in VR.

Only the headset should turn the view. Never roll or shake the rig, keep the horizon level, move at walking pace, and turn the rig in 30° snaps on the right stick. Put things to reach between 0.5 and 3 m away, between chest and eye height; in AR, within 2 m, and a tabletop piece 0.3 to 0.8 m across.

Related

• VR pieces
• AR pieces
• Engines
• The runtime API