Gesture Synth on Mobile: iPhone and Android Setup

Short answer: the gesture synth works on iPhone and Android in the normal browser. Hold the phone in landscape, prop it up, and play with one hand while the front camera watches. If tracking is slow, the touch mode turns the screen into the instrument. This page covers what to expect and how to set it up.

iPhone and iPad

1. Open handsynth.com in Safari (Chrome on iOS uses the same engine).
2. Turn the phone to landscape and prop it against something at chest height.
3. Flip the silent switch off or the synth will be muted.
4. Press Begin and allow the camera. The first load fetches the tracking model once.
5. Use one hand at arm's length; the chord hand works alone and the synth voices the chords for you.

iPhone 12 and newer track at a playable 20 to 30 fps. Older phones may drop lower; the fps readout on the stage tells you. Audio starts only after a tap, which is why Begin must be pressed even if the camera is already allowed.

Android

Chrome on Android runs the tracker on the GPU, so most phones from 2020 onward are smooth. Landscape works best. If the camera opens but the picture is black, another app is holding it; close the camera app and retry. Samsung Internet and Firefox work too, with CPU tracking.

Touch mode

Choose Mouse / touch as the input, or let HandSynth switch to it automatically when the camera is refused. Then:

• Chord mode: the stage is split into seven columns from I on the left to vii on the right. Touch and slide; higher on the screen is louder. Touch with two fingers for the minor flip.
• Theremin mode: slide left and right for pitch, up and down for volume, two fingers for vibrato.

Touch mode is also the right choice on a train or in a dark room, where the camera has nothing to see.

Battery, heat and data

Hand tracking uses the GPU continuously, so expect the phone to warm up after ten minutes and the battery to drop faster than video playback. The tracking model and WebAssembly code total about 20 MB on first load and are cached afterwards; the site sends no video anywhere, so play does not use mobile data.

Headphones

Wired headphones give the lowest latency. Bluetooth adds 100 to 250 ms, which is noticeable when playing to the beat. The phone speaker is fine for chords; for the theremin, headphones make the pitch easier to hear.

Problems with the camera or sound? The troubleshooting guide has fixes by symptom. For the controls themselves, read how to play.