gesture control of music and visuals

Also this week was the first test of the full-body gesture recognition and translation system, downtown at our warehouse space.
With the audio system, video projection, lighting and laptops to run the audio and video software set up, we strapped the sensor system on dancer Forest and ran a short improvised performance where her movements drove both the music and visual compositions.

The sensor rig tracks rotation at the hip and knee joints and elbow (potentiometers), rotational acceleration of the head/body (head-mounted gyroscope), hand movement (accelerometers) and finger flex (flax sensor). These signals run though a wireless A-D system and are fed to a Max patch running on a MacBok Pro for initial processing (filtering, scaling, etc.). MIDI continuous controllers are then fed to several softsynths (mostly Rob Papen Albinos, in this case) hosted in Ableton Live (which is also controlling the overall tempo of the performance and driving arpeggiators and a backing rhythm part) and also routed to a hybrid Director/Flash piece (running on a Windows-based ThinkPad). The latter generates real time animations based on the movement gestures, which are scan-converted and fed to the projector. Gestures also control LED lighting fixtures via DMX.

More information and images can be found here.