camera-based instrument in concert

We asked Adelle and Matt about their camera-based musical instruments. Here’s what they said.

What’s a camera-based musical instrument?  Basically, it’s two cameras facing upwards, about a foot below a player’s hands.  When they move their hand, it’s converted into different types of notes, sound effects and volumes, to create an expressive performance.

Three fireflies under construction

We made 3 of these.  The form was shaped like a musical soundwave: we prototyped it on the laser cutter, and eventually got it CNC milled.

This is part of the CES Intel Keynote pre-show performance. It was the opening of the show, to show off the instruments’ nuance and control before the concert gets too big. The middle instrument was piano sounds; the one on the right was synths and electronics; the one on the left was chords and atmosphere. The show opened dark: the performer, Kevin Doucette, used his hands to bring up the lights on the instrument as well as the synthesisers, then waved his hands to switch keys on a virtual keyboard.   Kevin played the Killers’ “Are we human or are we dancers?”.

In this instance, the performer is wearing gloves with sensors in them, and is using finger bends to trigger notes.  Yes, it looks like a theramin – but it’s way cooler and has blinkenlights.  But seriously, the LEDs are there to show the musician where they are on the instrument and the types of notes that they’re playing (ed: but they’re still cool).

We built this instrument to use the cameras (they’re good at doing fast hand tracking and depth); we added the LEDs because if you have an invisible instrument you don’t know where you are, and the LEDs give feedback to train your hand in space.

Firefly generations

Here’s the lasercut and CNC versions side by side: here, we’re doing LED tests.

(insides of the camera-based instrument)

Here are the insides: the frame, the LED controller and the acrylic housing around them.  The cameras are Realsense.  There are two programs (developed by Nerdmatics) running on linux in the back end, and TouchDesigner to control the lighting.

Firefly guts

Here are the guts of the instrument

Realsense cameras

Here are the cameras

Camera teardown

And the camera teardown

Come talk to us about this project!