Fireflies: camera-based musical instruments

We asked Adelle and Matt about their camera-based musical instruments. Here’s what they said.

What’s a camera-based musical instrument?  Basically, it’s two cameras facing upwards, about a foot below a player’s hands.  When they move their hand, it’s converted into different types of notes, sound effects and volumes, to create an expressive performance.

We made 3 of these.  The form was shaped like a musical soundwave: we prototyped it on the laser cutter, and eventually got it CNC milled.

This is part of the CES Intel Keynote pre-show performance. It was the opening of the show, to show off the instruments’ nuance and control before the concert gets too big. The middle instrument was piano sounds; the one on the right was synths and electronics; the one on the left was chords and atmosphere. The show opened dark: the performer, Kevin Doucette, used his hands to bring up the lights on the instrument as well as the synthesisers, then waved his hands to switch keys on a virtual keyboard.   Kevin played the Killers’ “Are we human or are we dancers?”.

In this instance, the performer is wearing gloves with sensors in them, and is using finger bends to trigger notes.  Yes, it looks like a theramin – but it’s way cooler and has blinkenlights.  But seriously, the LEDs are there to show the musician where they are on the instrument and the types of notes that they’re playing (ed: but they’re still cool).

We built this instrument to use the cameras (they’re good at doing fast hand tracking and depth); we added the LEDs because if you have an invisible instrument you don’t know where you are, and the LEDs give feedback to train your hand in space.

Here’s the lasercut and CNC versions side by side: here, we’re doing LED tests.

(insides of the camera-based instrument)

Here are the insides: the frame, the LED controller and the acrylic housing around them.  The cameras are Realsense.  There are two programs (developed by Nerdmatics) running on linux in the back end, and TouchDesigner to control the lighting.

Here are the guts of the instrument

Here are the cameras

And the camera teardown

Come talk to us about this project!