How a Hand Tracking Synth Works in the Browser

Short answer: a hand tracking synth reads 21 landmarks per hand from each webcam frame, turns finger positions into musical parameters, and drives oscillators with the Web Audio API. Everything happens in the browser tab. This page explains each stage so you can judge the latency, the privacy claim and the design decisions.

1. Hand landmarks from the camera

HandSynth uses MediaPipe's Hand Landmarker, a two-stage model: a palm detector finds hands in the frame and a landmark model returns 21 three-dimensional points per hand (wrist, four joints per finger) plus a left or right label. The model is compiled to WebAssembly and runs on the GPU through WebGL when the browser allows it, otherwise on the CPU. We read the video with requestVideoFrameCallback, so the tracker runs exactly once per camera frame and never quantises movement to a slower timer, which is what made some early gesture instruments feel stepped.

2. From landmarks to features

Raw landmarks jitter by a pixel or two. A one-pole smoothing filter removes the shake while keeping fast gestures responsive. From the smoothed points HandSynth derives a small set of features per hand:

• Extended fingers: a finger counts as raised when its tip is clearly farther from the wrist than its middle joint. This is rotation-independent, so it works with the hand tilted.
• Thumb: compared against the base of the pinky, which separates "thumb out" from "thumb across the palm".
• Tilt: the angle of the wrist-to-middle-finger line. Twenty degrees inward flips major and minor.
• Pinch: thumb-to-index distance divided by hand size, so it is the same for a small hand close up and a large hand far away.
• Position: the palm centre, used for volume (height), pan (left to right) and theremin pitch.

3. Musical mapping

Chord mode maps the finger count on the left hand to a scale degree and looks up the diatonic triad in the chosen key; the right hand chooses the inversion or adds a seventh and controls volume and filter brightness. Theremin mode maps horizontal position across two octaves to a fractional MIDI note, then optionally snaps it to the nearest note of the chosen scale with a short glide, which is how the instrument stays in tune without frets.

4. Sound

Each note is a voice of two detuned oscillators through a low-pass filter and an envelope, mixed into a master filter, a convolution reverb and a compressor. Voices are created on demand and every node is disconnected when its oscillator stops, so a twenty-minute session uses the same memory as the first minute; early gesture synths leaked oscillators and glitched after a while. When a hand leaves the frame, the voices are released within 160 ms instead of hanging, which is the fix for the stuck-chord bug people met in the first demos.

5. Recording and MIDI

Recording taps the master bus with MediaRecorder and then decodes the result and writes a WAV file locally. MIDI out uses the Web MIDI API in Chrome and Edge: chords go out as note-on and note-off messages, the theremin as note-on plus pitch bend, and volume as CC1.

6. Why nothing leaves your device

The site is static files served from a CDN. There is no upload endpoint. The camera stream is attached to a hidden video element, read into the tracker, and dropped. The only network traffic after the first load is the cookie-free page-view ping described in the privacy policy. You can verify this in the browser's network panel while playing.

7. Fallbacks

Mouse, touch and keyboard feed the same mapping layer, which is why the sound is identical in every input mode and why the instrument still works in a dark room. If you want to compare this approach with other browser instruments, the instruments hub lists them with their inputs and features.