October 07, 2020

Sound Branches

fall-2020itpmachine-learning-for-the-webp5jsphysical-computingsound

Demo video

How I made it

Photo of the previous version of the instrument, using servo motors making sounds in water cups — Last week’s fingertap player

Last week I experimented with Fingertap Player, an instrument that replays fingertap by hitting objects with different materials. There were two problems with the previous version: I found out that combining force sensitive resistors(FSRs) were not very good at capturing rhythms, and servo motors were not strong enough to produce fast sounds.

So I instead took a different approach – I focused on making percussive sounds with servos instead of capturing rhythm precisely.

I also changed how I control the instrument: instead of using FSR, I used voice commands to control the instrument. This was possible thanks to the Teachable Machine and ml5.js library that I learned from Yining Shi’s Machine Learning for The Web.

Code

link to p5.js code

// sketch.js
// forked from https://github.com/yining1023/machine-learning-for-the-web/tree/master/week4-soundClassifier
// one, two, three
const mySoundModelURL = 'https://teachablemachine.withgoogle.com/models/uqP44iRhB/';
let mySoundModel;
let resultDiv;
let serial;// variable to hold an instance of the serialport library
let portName = '/dev/tty.usbmodem14101';// fill in your serial port name here
let outByte = 0;// for outgoing data

function preload() {
  mySoundModel = ml5.soundClassifier(mySoundModelURL+ 'model.json');
}

function setup() {
  connectionDiv = createElement('p',  '');
  resultDiv = createElement('h2',  'Loading...');
  serial = new p5.SerialPort();    // make a new instance of the serialport library
  serial.on('error', serialError); // callback for errors
  serial.open(portName);           // open a serial port
  mySoundModel.classify(gotResults);
}

function serialError(err) {
  console.log('Something went wrong with the serial port. ' + err);
  connectionDiv.html("Serial port is not connected yet...");
}

function gotResults(err, results) {
  if (err) console.log(err);
  if (results) {
    console.log(results);
    if (results[0].confidence < 0.7) return;
    resultDiv.html("You're saying: " + results[0].label);
    if (results[0].label === 'One') {
      outByte = 1;
    } else if (results[0].label === 'Two') {
      outByte = 2;
    } else if (results[0].label === 'Three') {
      outByte = 3;
    } else {
      outByte = 0;
    }
    // send it out the serial port:
    console.log('outByte: ', outByte)
    serial.write(outByte);
  }
}

I modified Yining’s sample code and applied my own voice trained model made with Teachable Machine to the instrument’s controls. The sample code uses p5.serialcontrol to let the p5 code in the browser communicate with arduino’s serial ports.

As a result, the voice control responds to three words – one, two and three. The ‘one’ command makes the servo turn 0 degrees, ‘two’ command 90 degrees, and ‘three’ command 180 degrees respectively.

Servo motors

Photo of materials used for the pendulum: scrap metal pieces, acrylic beads — Materials used for the pendulum: scrap metal pieces, acrylic beads

Photo of small metal pieces attached to servo motor wings with wire — Hanging metal pieces with wires to the servo wings

The biggest problem from last week was that the servos were too weak to produce any audible sounds. After several attempts I finally found out the problem – the USB cables. After changing the USB cables it worked nicely. However, the speed was still not sufficient to produce rhythmical movements, so I had to just settle with producing movements. (Because of these reasons, I am planning to use actuators next time when I make an instrument again.)

I wanted to make a wind-chime like instrument, but instead of making bell sounds, I wanted to make the pendulum hit nearby bottles to produce different sounds. I attached metal scraps and acrylic beads to wires and made them hang from the servo motors’ wings.

Bottles and Cans

Photo of soup can, aluminum beer can and glass beer bottle — Soup can, aluminum beer can and glass beer bottle

I used a glass beer bottle, aluminum can and soup can to produce sounds. These are all products that I consumed the last week – I explored some local beer in NYC and ate noodles with soup for breakfast. I also used an acrylic stand to adjust the height of soup can.

Hiding the wires and parts

Photo of arduino instrument, wires and breadboards hidden to make it look like an instrument — Hiding the arduino, wires and servo in the stand – to make it look like an instrument

I had do find a way to hide the wires and breadboard, and making the servos hang from a high position.

I made a makeshift stand with a paper cup from cupcake packaging, wood block and readymade wood frames. I hid the breadboard inside the wood frame and connected the wires by drilling a hole in the wood block. I used a paper cup because it resembled a speaker/bell and hid the servo motors.