The Design of Interaction with Music for Mobile Devices

Lately a large number of music applications have become available for portable phones and mobile gaming system. A large number of music-related applications have been released for these devices - including software for music listening, playing, producing and sequencing. These offerings range from virtual pianos, guitars, drums, harmonicas, sequencers and many other instruments to innovative interfaces that allow inexperienced users (or professionals) to create music in new ways.

iPhone Music Applications
As would be expected, most of the music-related applications for the iPhone provide listening or informational functionality. There are some pretty interesting applications in these genres (Pandora and Shazam being two of my favorites), however, the focus of this post is on mobile applications that enable users to create music, both directly and indirectly. So here are a few of my favorites:

BeatMaker
This is an example of applications that mimics existing music instruments. This one provides access to a drum machine and a sequencer. It has a pretty cool interfaces that works well for the most part. They've done a good job at integrating the sequencer and drum machine on to this small touchscreen interface. This application provides a decent set of features, some highlights include ability to loop sounds, to record and playback tracks, and to add new sounds for the drums. The drum is really easy to use featuring large virtual drum pads (the sequencer is a little less user friendly).

Priced at almost $20, this application is definitely not for everyone. It is easy to use considering the the functionality it offers. However, it is more complex than more abstract applications such as Ocarina or Bloom. You do not need experience with drum sets or sequencers to play with beatmaker - I had pretty close to none.



Pro Remote
This is the most high-end example of music interaction application on the iPhone. One distinction about this application is that it does not create music, it is a remote control for Pro-Tools running on a Mac. Nonetheless, it is quite impressive. The video below only shows one small feature of this application. Check out this other video, which highlights some other interfaces provided by this application.



Smule Ocarina
This is an example of a simpler but in some ways more innovative music interface designed specifically for the iPhone. Ocarina is an application that turns your iPhone into an wind instrument whose output can be broadcast to a worldwide network of users. The instrument itself is pretty interesting. You create sound by blowing on to the microphone, and you control it by using four different virtual buttons on the touchscreen display. The application is simple, designed to allow inexperienced to quickly and easily make sounds. That said I quickly grew bored of it.

The networking feature is pretty cool. You can broadcast your music continuously on to the network (even when you are logged off). You can also listen to the broadcast of others via a pretty cool interface that shows the location where the broadcast originates and includes a visual representation of the music - some element of this visual representation can even be customized by the person who makes the song.



Bloom
This application provides another example of a simple and innovative interface designed specifically for the iPhone. Similar to Ocarina, this app was designed to be easy to learn and fun to use for people who are not musicians. Created by Brian Eno and Peter Chilvers, this music instrument enables users to produce melodies simply by touching different parts of the touchscreen. The melodies progress in a loop-like fashion and they slowly melt away and disappear after 30 seconds or so. The instrument will also start playing when left idle for over 40 seconds or so. You can choose from a nine different moods to change the sound of the harmonies. Personally, I prefer this app to Ocarina - though it also grew old pretty quickly.



RJDJ Album
Unlike the other applications featured on this list, this one is primarily used for music consumption rather than creation, which is not to say that it does not create music. The concept behind this RJDJ application is that it generates music on-the-fly using live sounds that are captured in real time mashed up with pre-recorded elements. The live sounds surface in the music as samples, and influence the progression of other elements in the song (if you can call it that).

Based on RJDJ's website they are promoting a musical genre called "reactive", where the sounds that people hear are produced that very moment by digital devices. This is definitely a genre in its infance (I had the music world that will see further development as more artists experience in this




Nintendo DS

KORG DS-10 (w/ optional Straw TalkBox)
The iPhone (and other cell phones) are not the only mobile devices that have more advanced and innovative music applications. A few months ago Nintendo released an emulator for Korg's DS-10 synthesizer. This application turns a Nintendo DS into a mini Korg DS-10. Check out this video - it is definitely more advanced than anything I've seen for the iPhone.

Since I don't own a Nintendo DS-2 - I only have an iPhone - I wonder when an app like this will come out on the iPhone? Is it just a question of time? to what extent does the low-average prices of iPhone apps inhibit the release of software of this complexity?



Another cool thing about Korg DS-10 app on the Nintendo DS is that a hacker in Japan has created a simple way to turn this device into a talkbox, using a simple straw. Check out the video below, just beware because it is in Japanese.



For Nintendo DS there are also a number of open source homebrew music applications - here is 24 examples courtesy of synthopia. Most of these applications are for more experienced users (unlike Ocarina and Bloom on the iPhone). Most of them are designed to provide advanced features and to run these apps users have to learn how to install homebrew software on their Nintendo DS. This requires a higher level technical know-how that tends to be limited to heavier user of the device.

[some content sourced via PSFK]

Touchscreen Design Insights from UIQ Technology

I really like SlideShare. Since I've started my curriculum, I've found numerous interesting presentations regarding interaction and experience design here. That said, this post is not about SlideShare (though that could be an interesting topic). The topic of inquiry is a great primer on the design of touchscreen interfaces for mobile devices that I came across on SlideShare. It provides insightful design tips along with an overview of the current state of the technology. It was written by UIQ Design, a Swedish firm that has a mobile design practice.

Here is a summary of the takeaways that were of special interest to me:

• Overview of the two dominant styles of touchscreen interaction: "double tap" and "drill down". The first style works by requiring two "taps" to initiate an action. The first "tap" on an object puts it in focus, then a second "tap" on the object itself, or an options menu, is provides access to a set of available actions (think tablet PCs). On the other hand, the "drill down" style requires that user tap objects only once to immediately access all available actions (think iPhone).
• Insights regarding how design principles associated to traditional mobile device interfaces need to be adapted to support finger-based touchscreen interactions. (1) The need to provide quick and short interactions can best be satisfied by offering direct access to applications and data (think one-click access on iPhone); (2) To deliver an experience that is easy to use and learn, objects, and the actions they afford, should be made obvious; Lastly, (3) designers need to address the challenge of the small screen while needing to support large hit areas that support finger touches.
• Description of two most common touchscreen technologies on portable devices: Resistive and Capacitive. The former is a technology that is used on the screen of palms and other devices that rely on a stylus for interaction. The benefits provided by these screens include costs coupled with their support for handwriting recognition technologies. Capacitive technology is better suited for finger-touch based devices. It is highly durable and can support multi-touch interfaces. (Some details in this area are definitely lost by the absence of the talking notes).
• Impact of screen resolution and size on the design of mobile applications. Two common approaches are used by designers to design interfaces that work across devices with different specs: "design by pixels" and "design by size". The names are pretty self-explanatory - the first is exemplified by when designers create interfaces that are always fully viewable on the screen of the device. Unfortunately, this often causes fonts to become unreadable on smaller devices. The other approach, "design by size", is where the icons and objects in the interface do not change size regardless of device size. In smaller devices the user is always able to read the fonts, though he may often have to scroll in order to access all options. The suggestion from UIQ is for designers to leverage a mix of both of these approaches.

UIQ Presentation on Touchscreen Design


[via TouchUsability]

Gesture and Touch Based Interface: G-Speak from Oblong

Check out this new gesture- and touch-based interface that was developed by Oblong technologies. I came across this video on the Interaction Design Blog - as they rightfully state, this is the closest thing I've seen to the interface from Minority Report. The demo itself is pretty awesome. It showcases several cool features from this system, here is a quick overview of these elements:

Three-dimensional navigation that enables users to access content via an interface that can organize data within a three-dimensional environment. This enables users to "directly manipulate" data objects and quickly access to information. This is achieved because users are able to leverage their familiarity with physical space to navigate this virtual world.

This is not to say that users can just rely on their physical-world metaphors to successfully engage with this system. After all, computer-based metaphors always have an aspect of "magic" because the physical actions we carry out have meaning beyond their direct physical impact. Therefore, users will need to learn how to use their hands to pull, turn, push, select and act on objects within this virtual space.

Integration of physical and virtual spaces to enable users to interact seamlessly across multiple computers and screens within a given environment. This is a really cool feature that can enable an user to easily move files between computers and other devices, such as mobile phones. Better yet, users are still be able to manipulate the files using the same interface across any of the connect devices. Compare this to Microsoft's surface which enables users to drag a file on the screen so that it is added to a mobile device. However, in order to handle the file once it is on the mobile device, then the user needs to leverage the mobile-device specific interface.

One additional aspect of this physical-virtual integration is that it allows users to change the physical configuration of the system in order to impact the virtual configuration. This is illustrated in the video when the user rotates the "screen table". The data being displayed on the table rotates along with the physical movement of the table, except for the pointer which is being manipulated by the user. This pointer does not move because the users hold his hand stable to hold the pointer in place. Sounds like a small simple element but it adds a lot of possibilities to the interactions that can be supported by this system.

Collaborative interaction is enabled by the system's multi-user support. Therefore, several users can work simultaneously with the system. It is not clear whether different users can work on different types of tasks simultaneously within the same environment. One additional questions is if multi-task support is provided then what are the requirements and how would the behavior of the system change in order to support this type of usage.

The data interaction and visualization opportunities provided by systems of this type will likely enable humans to interact with increasingly complex and large data sets. I would love to see this system in action first hand. Until that is possible I will have to be content with this vide. Hope you enjoy it as well.



[via Interaction Design Blog]

Reading List: Wired, November 2008

Here is an overview regarding two interesting articles from this month's wired magazine. I've provided a link for only one of the articles as I could not find the other one online. Enjoy.

Mainstream Physical Computing Interfaces
In the Gadgets column, Steve Levy talks about three products that have succeeded in integrating the digital and physical world in a compelling and easy to use manner - the Wii, the iPhone and Guitar Hero. Though all of these gadgets were launched before 2008, it was only this year that they reached maturity, achieved more widespread adoption and "dominated the zeitgeist." I have to agree with Steve's assessment since these are three of my favorite gadgets. Link to column, The Melding of Physical and Digital Realms.

Open-Source Hardware
Clive Thompson's article titled "Build it. Share it. Profit. Can Open Source Hardware Work." about open-source hardware was very interesting. Now that production cost for hardware technology is now sufficiently cheap new companies with open-source hardware business models have begun to appear and flourish. Improvements in technology design have also made it easier for DYI gadget builders to create cool prototypes without a PhD in advanced robotics. I have been tinkering with the idea of building some home brewed gadgets for a while - this will be one of a step of my self-imposed curriculum.

The Death of Rollovers

I just realized today how rollovers don't work on touchscreen displays - I know this probably seems obvious to those with experience designing applications for touchscreen displays. I had this realization when I was visiting the Guggenheim website on an iPhone. The home page for the Guggenheim organization features a pictures of their museums and the names of the city appear only on rollover. Therefore, I was only able to see the name of the city where a museum was located only when I clicked on the picture of that specific museum. As touchscreen displays continue to become more commonly available, designers will not be able to rely on rollovers to display useful information about the behavior of a button.