The runtime API

The runtime is what runs around a piece's code: it sizes the canvas, drives the frame loop, and gives the code a small set of tools. This page documents those tools, for anyone writing or changing a piece's code by hand.

In the WebGL 3D engine they arrive on the object the sketch function receives (ctx.gui, ctx.sfx and so on). In Canvas 2D they're globals. A GLSL shader gets its controls from annotated uniforms instead, and has none of the rest; see Engines.

Controls

gui is compatible with lil-gui. The controls you add show in the studio's parameter panel rather than over the piece, and they're what timelines, states and the embed API move.

const params = {
  speed: 1,
  count: 800,
  glow: true,
  mode: 'waves',
  tint: '#ff7a45',
  title: 'Hello',
  distance: 8,
  shuffle: () => reseed(),
};

gui.add(params, 'speed', 0, 4, 0.01).name('Speed');
gui.add(params, 'count', 100, 5000, 1).name('Count').onChange(rebuild);
gui.add(params, 'glow').name('Glow');
gui.add(params, 'mode', ['waves', 'rings', 'grid']).name('Mode');
gui.addColor(params, 'tint').name('Tint');
gui.add(params, 'title').name('Title');
gui.add(params, 'shuffle').name('Shuffle');

const view = gui.addFolder('Camera');
view.add(params, 'distance', 2, 20).name('Distance');

What gui.add(object, property, …) makes depends on the property's value:

gui.addColor(object, property) makes a colour picker. The property can hold a '#rrggbb' string, a number such as 0xff7a45, a THREE.Color, an array or an { r, g, b } object; pass a third argument of 255 when the array or object counts to 255 rather than 1. gui.addFolder(title) returns a folder with the same methods.

Controller methods

Each add returns a controller, and every method returns it again, so they chain.

The gui and its folders also take onChange and onFinishChange (called with { object, property, value, controller }), controllersRecursive(), reset(), save(), load(data) and destroy(). title(), open(), close(), show() and hide() are accepted and do nothing, since the panel is the studio's.

Control keys

Every control has a key: its property name, after its folders' names and dots when it's in a folder, such as speed or Camera.distance. A second control on the same key gets #2 after it. Keys are what keyframe tracks, states, variations and the embed API's set() name, and what the values a person sets are saved under. Keep property names when you edit, so their tweaks carry over.

Values a person has set in the panel are applied the moment each control is added, so add the controls before you build anything from their values. A button's changes to other controls are kept like any tweak.

Outside VYBE.ART, in a downloaded HTML file, the piece shows a small panel of its own in the corner. Press H to hide it.

Pointer and size

ctx.pointer follows the mouse, a pen or a finger, and is kept current:

ctx.size is { width, height, pixelRatio }, in CSS pixels, also kept current; ctx.width and ctx.height are the same numbers. The pixel ratio is capped at 2 (1.5 for GLSL). Lay everything out from the size in one place and do it again in resize(), so nothing stretches when the frame changes shape.

A piece can also listen for pointerdown, pointermove, pointerup and pointercancel on ctx.canvas, using e.pointerId to tell fingers apart. Pointer lock is allowed, for first-person games.

The clock

update({ time, delta, frame }) gets seconds since the start, seconds since the last frame (never more than 0.1) and the frame count. Both times stop while the piece is paused. In p5, use frameCount, millis() and deltaTime as usual.

When the studio renders a video, it steps the piece's clock a frame at a time, so the video comes out at an exact frame rate however long each frame takes. performance.now(), Date.now(), requestAnimationFrame and timers all follow that clock, so a piece that reads any of them renders correctly.

Images

textures.get(id) returns one of the piece's images: one Perlin painted with Textures on (an id starting tex_), or one of your uploaded images (img_).

// WebGL 3D: a THREE.Texture with SRGBColorSpace, decoded before the sketch starts.
const logo = ctx.textures.get('img_AbC123xYz789');
const aspect = logo.image.width / logo.image.height;
const plane = new THREE.Mesh(
  new THREE.PlaneGeometry(aspect, 1),
  new THREE.MeshBasicMaterial({ map: logo, transparent: true }),
);

// Tiled and crisp, for pixel art.
const floor = ctx.textures.get('tex_AbC123xYz789', { repeat: 8, pixelated: true });

In WebGL 3D, repeat is a number or [x, y], and pixelated turns smoothing off. The same id and options return the same texture; clone() it for one of its own. In Canvas 2D, textures.get(id) returns a p5.Image to draw with image().

An id the piece doesn't have gives a plain grey stand-in and a warning in the console, so the piece still runs.

Assets

Your uploaded fonts, 3D models and video come through assets. Images come through textures and audio through sfx, above and below.

export default async function sketch({ THREE, scene, assets }) {
  // Centred on the origin, scaled so its longest side is 2 units.
  const { scene: robot, animations, size } = await assets.model('model_AbC123xYz789', { size: 2 });
  robot.position.y = size.y / 2; // stand it on the ground
  scene.add(robot);

  const mixer = new THREE.AnimationMixer(robot);
  if (animations[0]) mixer.clipAction(animations[0]).play();

  const video = new THREE.VideoTexture(assets.video('video_AbC123xYz789'));
  video.colorSpace = THREE.SRGBColorSpace;

  return { update: ({ delta }) => mixer.update(delta) };
}

Pass size to assets.model every time: models are made at any scale and anywhere. Each call builds a new scene from the file, and a model that can't be loaded comes back as an empty group so the piece runs without it. A video's videoWidth and videoHeight are 0 until it has loaded. In Canvas 2D, assets.video(id) returns a p5.MediaElement to draw with image(video, x, y, w, h). In a rendered video the runtime moves each video to the frame's moment itself, so don't seek or time it.

How a piece finds its assets

The code names each file by its id, written as a quoted string: 'img_…', 'font_…', 'model_…', 'audio_…' or 'video_…', with twelve letters and digits after the underscore. The runtime loads the files whose ids appear in the code. Perlin writes the ids in when you attach assets to a prompt; when you edit by hand, keep each id exactly as it is, in quotes.

Sound

sfx plays the piece's sounds by id: sound effects Perlin made with Sound FX on (sfx_…), and your uploaded or generated audio (audio_…).

const hit = sfx.play('sfx_AbC123xYz789', { volume: 0.8, rate: 1.2, pan: -0.3 });
const music = sfx.loop('audio_AbC123xYz789', { volume: 0.5 });

music.setVolume(0.2); // duck it
hit.stop();
sfx.volume = 0.7; // the master level, 0 to 1

play and loop return a voice with stop(), setVolume(v), setRate(r), setPan(p), playing and duration (seconds, once it has loaded). Up to 32 one-off sounds play at once.

Browsers only allow sound after the viewer's first click, tap or key press. A loop asked for before then waits and starts on that first gesture; a one-off play before then is dropped, since it would play late. Make the piece complete in silence, and start its sound from an interaction. Sound pauses with the piece.

// Audio-reactive: pulse with the bass.
return {
  update() {
    const { bass, beat } = sfx.levels();
    orb.scale.setScalar(1 + bass * 0.6);
    if (beat) flash = 1;
  },
};

Saving files

vybe.save hands the viewer a file the piece makes: a picture of their drawing, a recorded loop, a preset, a model. It works in WebGL 3D and Canvas 2D, in the studio, on share links and in embeds.

// The piece as it looks on its next frame, as a PNG (or .jpg, .webp by the name).
params.save = () => vybe.save('my-drawing.png');
gui.add(params, 'save').name('Save image');

// Anything else: a canvas or p5.Graphics, a Blob, an ArrayBuffer, a string or an object.
vybe.save(new Blob(chunks, { type: 'audio/webm' }), 'loop.webm');
vybe.save(JSON.stringify(preset), 'preset.json');
vybe.save(glbArrayBuffer, 'sculpture.glb');

vybe.save(name) saves the piece as it looks, with any timeline layers over it. vybe.save(what, name) saves something else: a string is saved as text and an object as JSON. Both return a promise of true once the file is saved, or false when it wasn't, with the reason in the console.

• Only right after a click, tap or key press. Call it straight from one: a gui button, a button drawn on the canvas, or a key. A save on a timer or at the start is refused.
• Only these kinds: images (PNG, JPEG, WebP, GIF, AVIF), sound (WAV, MP3, OGG, WebM, M4A, AAC, FLAC, MIDI), video (WebM, MP4), 3D models (GLB, glTF, OBJ, STL) and text (TXT, CSV, JSON). A file is saved under the extension for its kind.
• At most 200 MB, and three saves within five seconds.
• With no name, the file is named after the piece.

Download links (<a download>) and p5's own save functions go through vybe.save too. The gui panel only shows in the studio, so when people opening a share link should save, draw a button for it on the canvas as well.

Events and states

When a piece has states (set up in the Motion panel's States tab), its code can talk to them.

if (score >= 10 && !won) {
  won = true;
  vybe.emit('won', { score });
}

A piece loaded onto another piece can vybe.emit too, and sets off the other piece's Piece event transitions. See States and interactions and The embed API.

Errors and the console

console.log, warn and error show in the Console tab of the studio's code drawer, up to the first 150 messages of a run. An error that stops the piece is reported with the line it happened on, and a loop that runs for more than 3 seconds is stopped and reported the same way, so a stray while (true) can't freeze the page.

Related

• Engines
• VR and AR in code
• The embed API
• Editing the code