This weekend I spent a lot of time working on solving the issue of how to control and manage the music clips that we want to use in our project. Our requirements are pretty straight forward, which is not to say easy to address.
Requirements for Audio Controls
We need a solution that can handle playback of looped samples and dynamic control of at least two effects to be applied on the sample (such as tempo and pitch). Ideally we would like the solution to be scalable (so we can add multiple sounds) and be able to support quantization and other techniques to ensure that the resulting sound is of good quality.
Since we are a creating a prototype to run off of a single computer we do not need this solution to be easily portable (e.g. it does not need to be easy to run on different computers).
Initial Assumptions
Due to the expertise of the team members we are using a combination of Arduino and Processing to do the heavy lifting in the areas of input sensing and data handling. After researching all of the options available in processing to manage sound, we have decided to use Ableton Live instead. Processing’s role will be relegated to interpreting the data from the Arduino to control Ableton Live via OSC.
Below I provide a more in-depth overview of my research and the solution that I have chosen. I have also posted an updated version of my sketch along with a link to the file I have created in Ableton Live for this application. Please note that you will need to set-up OSCulator in order for the sketch to work properly.
Latest Version of the Sketch
Note that I am only sharing the code for the sketch because no updates were made to the look, feel and interaction of the applet. All updates are related to enabling the sketch to communicate with Ableton via OSC.
/* IPC Media Controller Project, October, 2009
* VIRTUAL MATRIX SKETCH
*
* This sketch is the first draft of the processing portion of our media controller project
* in its current state, this sketch only focuses on reading input from serial ports,
* processing this input to determine location on the virtual matrix, then provide these
* coordinates to other objects (such as the music generation object that we will create in the future)
*
*/
import processing.serial.*;
Serial arduino;
import oscP5.*;
import netP5.*;
OscP5 oscComm;
NetAddress myRemoteLoc;
boolean isStarted = false;
// Matrix-Related Variables
Matrix matrix;
final int x = 0; final int y = 1; // variables to use with matrix size array
int [] cellSize = {50, 50};
int [] screenPad = {25,25}; // define padding between grid and screen border
int [] screenSize = new int [2]; // define screen size, note that we only add volSize to width, since volume knob will be place to right of screen
// Volume-Control Related Variables
int [] volSize = {0,0};
void setup() {
// initialize the matrix object
matrix = new Matrix(screenPad[x], screenPad[y], cellSize[x], cellSize[y]);
// instantiate the serial variable
arduino = new Serial(this, Serial.list()[0], 9600);
arduino.bufferUntil('.');
// set frame rate
frameRate(25);
// start osc communication, listening for incoming messages at port 12000
oscComm = new OscP5(this,12000);
// set destination of our OSC messages (set to port 8000, which is the OSCulator port)
myRemoteLoc = new NetAddress("10.0.1.3",8000);
// set screen size related variables
screenSize[y] = int(matrix.getMatrixWidth() + (screenPad[y] * 2));
screenSize[x] = int(matrix.getMatrixHeight() + volSize[x] + (screenPad[Y]*2));
size(screenSize[x], screenSize[y]); // draw the window 100 pixels wider and 50 pixels taller
}
void draw() {
matrix.isCellActiveMouse();
matrix.isCellActiveSerial();
matrix.drawMatrix();
matrix.sendOscMessage(oscComm, myRemoteLoc);
}
void serialEvent(Serial arduino) {
matrix.readSerialInput(arduino);
}
void oscEvent(OscMessage theOscMessage) {
/* print the address pattern and the typetag of the received OscMessage */
print("### received an osc message.");
print(" addrpattern: "+theOscMessage.addrPattern());
println(" typetag: "+theOscMessage.typetag());
}
/* CLASS MATRIX
*
* this class holds a virtual matrix that will mimic the real world matrix.
* It contains functions that read input from a serial port or mouse, then use that
* input to determine the location of the object or mouse on the grid
*
*/
class Matrix {
// general variables used accross class
final int x = 0; final int y = 1; // variables to use with matrix size array
final int mouseControl = 0; final int serialControl = 1;
// matrix and cell related variables
final int [] cellNumber = {5, 5}; // number of cells on the horizontal axis of the matrix
final float [] cellSize = new float [2]; // width and height of each cell of the matrix
int [] matrixLoc = new int [2]; // location of the overall matrix
final float [] matrixSize = new float [2]; // the total height and width of the matrix
float [] xCellLoc = new float [cellNumber[x]]; // location of each cell on the grid
float [] yCellLoc = new float [cellNumber[y]]; // location of each cell on the grid
Boolean [][] cellState = new Boolean [cellNumber[x]][cellNumber[y]]; // holds whether the mouse or serial object is hovering over a cell
color activeColor = color (255,0,0); // holds color of active cells
color inactiveColor = color (255); // holds color of inactive cells
int [] previousState = {0,0}; // holds prevous state of the cell
// variables for reading serial input
int mainControl = mouseControl;
float [] serialLoc = {0,0}; // holds Y reading from the serial port
// Matrix Object Constructor
Matrix (int XLoc, int YLoc, int cellWidth, int cellHeight) {
matrixLoc[x] = XLoc; // set X and Y location of the virtual matrix
matrixLoc[y] = YLoc;
cellSize[x] = cellWidth; // set the size of each cell on the grid of the virtual matrix
cellSize[y] = cellHeight;
matrixSize[x] = cellNumber[x] * cellSize[x]; // calculate width of the matrix
matrixSize[y] = cellNumber[y] * cellSize[y]; // calculate height of the matrix
// sets the location of each cell on the grid
for (int xCounter = 0; xCounter < xCellLoc.length; xCounter++) {
xCellLoc[xCounter] = xCounter * cellSize[x]; }
for (int yCounter = 0; yCounter < yCellLoc.length; yCounter++){
yCellLoc[yCounter] = yCounter * cellSize[y]; }
// sets the status of each cell to false
for (int xCounter = 0; xCounter < cellState.length; xCounter++) {
for (int yCounter = 0; yCounter < cellState[xCounter].length; yCounter++) {
cellState[xCounter][yCounter] = false; }
}
} // close the constructor
// function that returns the height of the matrix
float getMatrixHeight(){
return matrixSize[x];
}
// function that returns the width of the matrix
float getMatrixWidth(){
return matrixSize[y];
}
// function that draws the matrix on the screen
void drawMatrix() {
for (int xCounter = 0; xCounter < xCellLoc.length; xCounter++){ // loop through each element in the xCellLoc array
for (int yCounter = 0; yCounter < yCellLoc.length; yCounter++){ // loop through each element in the yCellLoc array
if (cellState[xCounter][yCounter] == true) { fill(activeColor); } // if the cellState is true then change the color of the cell
else { fill(inactiveColor);} // if the cellState is false then don't change the color of the cell
rect(xCellLoc[xCounter]+matrixLoc[x], yCellLoc[yCounter]+matrixLoc[y], cellSize[x], cellSize[y]); // draw rectangle
}
}
} // close drawMatrix() function
// function that reads the input from the serial port
void readSerialInput (Serial Arduino) {
if (!isStarted) { // if this is the first time we are establishing a connection
isStarted = true; // set isStarted to true
arduino.write("n"); // respond to arduino to request more data
} else { // if this is NOT the first time we have received data from the arduino
String bufferString = arduino.readString(); // read the buffer into the bufferString variable
if (bufferString != null) { // if bufferString holds data then process the data
bufferString = bufferString.substring(0, bufferString.length() - 1); // trim the string
String[] serialValues = splitTokens(bufferString, " "); // separate the two values from the string and save them in the serialValues variable
serialLoc[x] = float(serialValues[x]); // assign value to serialLoc[x]
serialLoc[y] = float(serialValues[y]); // assign value to serialLoc[y]
}
arduino.write("n"); // respond to arduino to request more data
}
} // close readSerialInput() function
// returns an array with the unfiltered x and y locations from the serial monitor (may need to filter data based on range of serial input and requirements of music objects)
int[] getSerialXY() {
return int(serialLoc);
}
// TO BE CREATED
// function for user to set whether main input is serial or mouse based
void setMainControl(int tControlType) {
mainControl = tControlType;
}
// function that sends OSC messages with input values
void sendOscMessage(OscP5 tOscComm, NetAddress tMyRemoteLoc) {
float messageX = 0;
float messageY = 0;
// open new OSC messages of type x and type y
OscMessage myOscXMessage = new OscMessage("/controlGrid/x");
OscMessage myOscYMessage = new OscMessage("/controlGrid/y");
// determine whether readings that are sent to OSC will originate from serial device or mouse
if (mainControl == serialControl) {
messageX = map(serialLoc[x], 0, width, 0, 1);
messageY = map(serialLoc[y], 0, height, 0, 1);
} else if (mainControl == mouseControl) {
messageX = map(mouseX, 0, width, 0.075, 0.125);
messageY = map(mouseY, 0, height, 0.3, 0.7);
}
myOscXMessage.add("x "); /* add an int to the osc message */
myOscYMessage.add("y "); /* add an int to the osc message */
myOscXMessage.add(messageX); /* add an int to the osc message */
myOscYMessage.add(messageY); /* add a float to the osc message */
tOscComm.send(myOscXMessage, tMyRemoteLoc);
tOscComm.send(myOscYMessage, tMyRemoteLoc);
print("X: " + messageX + " ");
print("Y: " + messageY + " ");
println();
}
// returns an array with the unfiltered x and y locations from the mouse-based interface (may need to filter data based on requirements of music object)
int [] getMmouseXY() {
int [] mouseXY = {mouseX, mouseY};
return mouseXY;
}
// check if a cell on virtual Matrix is active based on the mouse location
void isCellActiveMouse () {
int XLocMouse = mouseX - matrixLoc[x]; // adjust variable to account for location of Matrix within window
int YLocMouse = mouseY - matrixLoc[y]; // adjust variable to account for location of Matrix within window
isCellActive(XLocMouse, YLocMouse); // call the function to check if the cell is active based on current location of mouse
}
// check if a cell on virtual Matrix is active based on the current physical location/state of an external object
void isCellActiveSerial () {
int xLocSerial = int(map(serialLoc[x], 0, 1024, 0, matrixSize[x])); // adjust variable to account for location of Matrix within window
int yLocSerial = int(map(serialLoc[y], 0, 1024, 0, matrixSize[y])); // adjust variable to account for location of Matrix within window
isCellActive(xLocSerial, yLocSerial); // call the function to check if the cell is active based on current location of mouse
}
// function that checks whether a specific cell is Active
void isCellActive (int tXloc, int tYloc) {
int xLoc = tXloc; // set the location of the X coordinate where the mouse or serial object is located
int yLoc = tYloc; // set the location of the Y coordinate where the mouse or serial object is located
for (int xCounter = 0; xCounter < xCellLoc.length; xCounter++){ // loop through each element in the xCellLoc array
for (int yCounter = 0; yCounter < yCellLoc.length; yCounter++){ // loop through each element in the yCellLoc array
// check out what are the mouse or serial object is intersecting
if ( (xLoc > xCellLoc[xCounter] && xLoc < (xCellLoc[xCounter] + cellSize[x])) &&
(yLoc > xCellLoc[yCounter] && yLoc < (yCellLoc[yCounter] + cellSize[y])) ) {
cellState[previousState[x]][previousState[y]] = false; // set previous grid element to false
cellState[xCounter][yCounter] = true; // set current element to active status
previousState[x] = xCounter; // set x number of previous active cell
previousState[y] = yCounter; // set y number of previous active cell
}
}
}
}
}
Making Some Noise
When we started working on this project we assumed that we would be able to use one of Processing existing sound libraries to play and modulate an audio loop. However, after doing extensive research into Minim, ESS, and Sonia, I realized that none of these tools offered the feature set that we needed for this project.
The next solution that I investigated was Max/MSP. This programming language/environment is definitely capable of providing the functionality that we are looking for. However, no one on our team has the expertise to use it nor the time to learn it for this project.
using OSC to communicate with an external music application that can provide the features we are looking for. I was happy to find out that there is a simple library called oscP5 that makes it easy to communicate from a sketch using OSC. Equally important, I also found an application called OSCulator that routes and translates OSC and MIDI messages.
Having figured out how to get the sketch to communicate via OSC and MIDI we set out to find the right application. This was an easy task in large part because both Michael and I are familiar with Ableton.
I am happy to report that we already have Ableton up and running with the virtual matrix application developed in processing, though that is not to say the sketch is finished. We still need to add start and stop buttons to the interface, along with a volume control (not to mention other improvements and ideas that have not yet been considered).
In the next day or so I will share with you more updates, including details about how the physical elements of the interface are shaping up.
