Late Music

Introduction Latency issues are an inherent challenge in realtime online collaboration of music (ROCM). That is, the time it takes for each side to receive and respond to what is heard over the Internet exceeds tolerable limits for response times necessary in the performance of traditional musical techniques and styles. In addition to physics-based speed-of-light type of latency, delays can also be caused by network congestion and the machines themselves. Most organizations engaged in the development of ROCM systems strive to devise ways of making various delays imperceptible to participants. Two major developers are examined in this paper: McGill? University in Canada and Digital Worlds Institute in Florida. These two organizations presented ROCM with fullscreen video, which added to the somewhat unstable performance, however, they proved the possibility of an almost imperceptible delay. The problem is that almost in sync just doesn’t work for most methods of musicmaking.

By contrast, NINJAM is an audio-only ROCM system that manages network latency by increasing it! NINJAM software creates a buffer of what is received and holds it before sending it back to the user. This way the audio is heard in a 'timely' fashion more readily interpreted by the human brain. This is germaine to the ROCM system’s functionality. Participants are advised to expect a one-measure delay in hearing what they’ve just played. Thus the playing field is somewhat leveled as everyone must adjust their approach to playing, hearing, and even thinking about the composition and performance of music.

As profiled in the following pages, the author’s personal experience and resultant music created in the NINJAM’s ROCM system is inspiring and serves as the focus of this paper: that is to suggest that actively playing with latency in NINJAM’s ROCM is quite enjoyable and more fun to play and listen to than the results from systems where participants struggle to overcome latency issues. It is also proposed that the music created with cooperative latency gives birth to a brand new genre of music: a genre of music henceforth dubbed by the author as, Late Music.
What is Late Music? Late Music is a term coined by the author for the purposes of this research paper to describe the music that is created in a ROCM system that actively incorporates latency as essential to participation. That is, users play with the latency as opposed to programmers trying to make it unnoticeable. They play with it almost as if it were another instrument. As is the case with John Cage’s Chance Music, where intentionality is to be avoided, Late Music compels participants to become more open-minded when they do not immediately hear what they expected. Roland Barthes might say that Late Music is another nail in the coffin of the author, because even though it might sound similar to established genres as musicians bring along their influences, it will be a hybrid of those styles, rather than deriving its source from a single one. Late Music doesn’t necessarily have to sound like free jazz or something abstract, but it can be completely indeterministic, and players must be willing to anticipate immediate changes to establish structures and “go with the flow”. This might include being able to keep the piece going when there is what sounds like an incomplete phrase, a shortened verse, or sporadic interruptions. But the defining quality of Late Music is that it is fun - fun to play and fun to hear as participants giggle and cavort their way through a perceptual and auditory adventure.

Current Activity: McGill? University

McGill? University in Montreal is at the forefront of online collaboration technologies with the ongoing development of its Immersive Systems division of the Centre for Interdisciplinary Research and Music Media and Technology (CIRMMT). The pertinent research projects include Ultra-Videoconferencing, Shared Spaces, and the Center for Intelligent Machines’ research lab Shared Reality and Intelligent Environments. Just three years ago, musicians at McGill? were the first to jam together online with musicians at Stanford University in California, using McGill?’s low-latency Ultra-Videoconferencing system. It was the first successful demonstration of its kind and featured multi-channel audio and full-screen bi-directional video. While the entire McGill? side ran from a single machine, the system latency introduced a perceptible audio echo which was mostly removed by transmitting audio-only. Even without the video, and some degree of latency, McGill? considered the event a major success in demonstrating the feasibility of highly interactive events over the Internet. A few months later, musicians from Stanford University and McGill? played jazz together with surround sound and full-screen video, over the Internet, using McGill?’s Ultra-Videoconferencing system. The system maintained a delay below 50 milliseconds, which, according to McGill?, feels to the musicians almost like being on the same stage. Other latency-causing problems occurred, however, including an intervening router that caused periodic delays that were almost double. It was eventually determined by those involved in these particular demonstrations that McGill?’s Ultra-Videoconferencing did not meet the stringent demands of ROCM’s high-resolution multichannel audio and thus would need to be further developed to those standards. Video of McGill? demonstrations confirm the confusion and discomfort of the players and audience alike as they all try to cope with the fractional lags in time. The players are all exemplary but Mother Nature does not care about musicianship. If the McGill? participants were to incorporate Late Music techniques it is likely that they and their audience would have a much more relaxed and enjoyable shared experience. They would marvel together at the exceptional musical passages that no one premeditated but were simply the result of working through unpredictable circumstances. Oops, did you play a wrong note? When it is finally heard some five to ten seconds later it could quite possibly be a luminous fusion of legendary sound inspiring everyone to carry those serendipitous connections into their own lives. Underlying technologies to correct latency in ROCM systems continues to develop at a rapid pace, but because of the limitations caused by the nature of physics, ROCM begs to be approached differently and opened to alternative methods which incorporate the delayed response time.

Current Activity: Digital Worlds Institute

Enter the research project, In Common: TIME at the University of Florida’s Digital Worlds Institute, a live online distributed collaboration linking five continents with partners at the Korean Advanced Institute of Science and Technology (KAIST), the Red Universitaria Nacional (REUNA) of Santiago, Chile, the Australian Cooperative Research Centre for Interaction Design (ACID) at the Queensland University of Technology in Brisbane, the Digital Knowledge Exchange (DKE) of Doncaster, England, the New World School of the Arts (NWSA) in Miami and the UF Colleges of Fine Arts and Engineering featuring students from the PK Yonge Laboratory School, and the Millhopper Montessori School in Gainesville, Florida. In Common: TIME featured performers ranging in age from 11 to 68, including musical artists from the five diverse world cultures. The purpose of the SIGGRAPH showcase was to illustrate how the emerging technology known as the Access Grid allows multiple people in multiple locations to come together in a shared virtual environment in real time. UF Digital Worlds partnered with academic researchers and engineers from Internet2, the National Center for Supercomputing Applications (NCSA), the Access Grid community, Purdue University, Boston University, industrial sponsor InSORS and SIGGRAPH. The author was an enthralled member of the audience for several of the Access Grid and In Common: TIME (ICT) demonstrations which were presented at the Siggraph 2005 Emerging Technologies exhibit. An impressive production, it was extremely professional from the carefully scripted dialogue to the brilliant engineers. The ICT method of dealing with latency was to use something called a “Netronome”, like a beat clock, it automatically calculates network latencies supposedly putting all of the participants “in sync”. This writer was asked to give feedback to the National Center for Supercomputing Applications (NCSA), and reported that while the participants and overall presentation were commendable, the effects of the latency made the musical performance “not yet ready for prime time.” It was an hour of “waiting for the other shoe to drop”, so to speak, like being forced to listen to performers with a poor sense of time. On the contrary, all of the personnel were top-notch professionals, so the latency itself was not the problem. It was the way in which the latency was handled.

Current Activity: NINJAM Expanding the existing time-space continuum of music and shifting the focus to actively doing what works rather than trying to fool Mother Nature, NINJAM comes to ROCM’s rescue. NINJAM, made by Cockos Incorporated, is a program to allow people to make real music together via the Internet. Every participant can hear every other participant. Each user can also tweak their personal mix to his or her liking. NINJAM is cross-platform, with clients available for Mac OS X and Windows.

HYPERLINK "http://www.ninjam.com/images/ninjamosx.jpg" NINJAM uses compressed audio which allows it to work with any instrument or combination of instruments. You can sing, play a real piano, play a real saxophone, play a real guitar with whatever effects and guitar amplifier you want, anything. If your computer can record it, then you can jam with it.

Since the inherent latency of the Internet prevents true realtime synchronization of the jam, and playing with latency is weird (and often uncomfortable), NINJAM provides a solution by making latency (and the weirdness) much longer. Latency in NINJAM is measured in measures, and that's what makes it interesting. They call it “faketime”. The NINJAM client records and streams synchronized intervals of music between participants. Just as the interval finishes recording, it begins playing on everyone else's client. So when you play through an interval, you're playing along with the previous interval of everybody else, and they're playing along with your previous interval. If this sounds pretty bizarre, it sort of is, until you get used to it, then it becomes pretty natural. In many ways, it can be more forgiving than a normal jam, because mistakes propagate differently. NINJAM is designed with an emphasis on musical experimentation and expression.

HYPERLINK "http://www.ninjam.com/images/ninjam_win32.gif" NINJAM uses OGG Vorbis audio compression to compress audio, then streams it to a NINJAM server, which can then stream it to the other people in your jam. This architecture requires a server with adequate bandwidth, but has no firewall or NAT issues. OGG Vorbis is utilized for its great low bitrate characteristics and performance. Each user receives a copy of other users’ audio streams, allowing for each user to adjust the mix to their liking, as well as remix later. This uses more bandwidth than having a server encode a single stream, but has numerous benefits (including lower server CPU use and the client having the full multichannel data for later use). NINJAM can also save all of the original uncompressed source material, for doing full quality remixes after the jam. There is also the opportunity to listen to mp3 recordings of your jams when they are done on NINJAM’s test servers, affectionately known as the Jam Farm. In addition, there is a NINJAM forum which further helps to build the NINJAM community. People can also simply listen and chat with NINJAMmers on the Shoutcast servers as well. The NINJAM chat feature is extremely helpful and helps to facilitate communication between users.

Following is a partial transcript of my first NINJAM:

*** Topic is: brennan & cryptomail hosting <synthany@71.131.53.x> anybody home? *** mike-guitar@66.166.238.x has joined the server <mike-guitar@66.166.238.x> hey <synthany@71.131.53.x> hiya <synthany@71.131.53.x> my first time <mike-guitar@66.166.238.x> I've only been here a few times myself <mike-guitar@66.166.238.x> more fun when other play of course :) <mike-guitar@66.166.238.x> you could sing hehe <synthany@71.131.53.x> yah - will try <mike-guitar@66.166.238.x> sometimes the lag makes it hard to play but when you get a groove it works pretty well <mike-guitar@66.166.238.x> I've heard some crazy stuff here, experimental^2 <mike-guitar@66.166.238.x> mostly when people get tired and start playing too fast heh <mike-guitar@66.166.238.x> ok, what key are you in? <anon@71.51.205.x> im open <anon@71.51.205.x> how far offf> <anon@71.51.205.x> lol <mike-guitar@66.166.238.x> I might be in like C :) <mike-guitar@66.166.238.x> but as long as both guitars are in tune everythings ok :) <mike-guitar@66.166.238.x> I'l just play some bass line and you can go from there eh? <mike-guitar@66.166.238.x> oh drums help, one sec <synthany@71.131.53.x> got a mic hooked up <mike-guitar@66.166.238.x> synth.. got a mic? <synthany@71.131.53.x> yah <mike-guitar@66.166.238.x> only problem with a mic is it feeds back and it sounds like an off-off beat :) <synthany@71.131.53.x> ah - lemme try a synth <mike-guitar@66.166.238.x> cool <mike-guitar@66.166.238.x> I can hear myself from 10 seconds ago haha <anon@71.51.205.x> am i still gettin thru <synthany@71.131.53.x> yah <mike-guitar@66.166.238.x> guitar effects processor bass <alex@203.79.73.x> cool <Dustin@64.223.44.x> serious lag here, let me try and reconnect <Dustin@64.223.44.x> lag agian <alex@203.79.73.x> what key are we laying in fellas? <alex@203.79.73.x> playing* <anon@71.51.205.x> im open <anon@71.51.205.x> g right ? <Dustin@64.223.44.x> thought fsharp minor <anon@71.51.205.x> dont asl me im self taught <anon@71.51.205.x> ok how about this <anon@71.51.205.x> d c7 g <mike-guitar@66.166.238.x> I hear something that want to come out <alex@203.79.73.x> well go by the metronome <alex@203.79.73.x> !vote bpm 100 *** voting system? leading candidate: 1/3 votes for 100 BPM each vote expires in 120s? *** njircgw has left the server <mike-guitar@66.166.238.x> !vote bpm 100 *** voting system? leading candidate: 2/3 votes for 100 BPM each vote expires in 120s? *** njircgw has joined the server <mike-guitar@66.166.238.x> I felt... something :) <alex@203.79.73.x> are u guys going my the metronome? *** neimad@62.234.30.x has joined the server <alex@203.79.73.x> dustin is goin by the metronome <alex@203.79.73.x> how about Am <alex@203.79.73.x> Am C G <Dustin@64.223.44.x> other chords? <anon@71.51.205.x> am d g <Dustin@64.223.44.x> ok *** njircgw has left the server <neimad@62.234.30.x> :} *** njircgw has joined the server <Dustin@64.223.44.x> cool <alex@203.79.73.x> that was good <alex@203.79.73.x> :P <alex@203.79.73.x> new chords? <Dustin@64.223.44.x> ok <anon@71.51.205.x> what we doing chords <anon@71.51.205.x> 80 bpm <Dustin@64.223.44.x> cool <synthany@71.131.53.x> where are you guys located? <Dustin@64.223.44.x> rhode island <mike-guitar@66.166.238.x> it's hard to make that off beat with latency :) <synthany@71.131.53.x> i'm in santa cruz <Dustin@64.223.44.x> oh <anon@71.51.205.x> missouri <mike-guitar@66.166.238.x> california here *** anon@71.51.205.x has left the server <Dustin@64.223.44.x> mabye i should switch insturments (forever) to guitar <mike-guitar@66.166.238.x> woah <Dustin@64.223.44.x> no i said that because bass players should play "in the pocket" <Dustin@64.223.44.x> well, some people say <synthany@71.131.53.x> poop in the pocket <Dustin@64.223.44.x> well, its 3:27 am here, going to bed, mabye you are neimad can take over <synthany@71.131.53.x> night <Dustin@64.223.44.x> night, nice jamming everyone <alex@203.79.73.x> cya <Dustin@64.223.44.x> bye <alex@203.79.73.x> ta for the jamm :D <synthany@71.131.53.x> yah - back atcha <Dustin@64.223.44.x> cool <Dustin@64.223.44.x> bye *** Dustin@64.223.44.x has left the server <mike-guitar@66.166.238.x> see ya <mike-guitar@66.166.238.x> cool <alex@203.79.73.x> :D *** zormal@85.76.250.x has joined the server *** zormal@85.76.250.x has left the server <alex@203.79.73.x> brb <anon@71.51.205.x> you guys do this alot? <mike-guitar@66.166.238.x> I've only been here a few times <synthany@71.131.53.x> my first time <anon@71.51.205.x> all i had to play wirt was a buncha geezers playomng bluegrass <mike-guitar@66.166.238.x> haha <anon@71.51.205.x> really warped my soul dont laugh <synthany@71.131.53.x> yah, this is a pretty cool jam for my first <synthany@71.131.53.x> that's sweet <alex@203.79.73.x> synthany keep the bassline going <alex@203.79.73.x> :D <alex@203.79.73.x> do the one u did before <alex@203.79.73.x> thats good <alex@203.79.73.x> just keep going <alex@203.79.73.x> that was nice ! :D <mike-guitar@66.166.238.x> cool! <alex@203.79.73.x> mike is that u on the wah lead ? <mike-guitar@66.166.238.x> yeah <synthany@71.131.53.x> beauty <alex@203.79.73.x> awesome ! <alex@203.79.73.x> : ) <mike-guitar@66.166.238.x> thanks! <mike-guitar@66.166.238.x> it all sounded really good <alex@203.79.73.x> that was cool <alex@203.79.73.x> yea

Conclusion The most obvious detail to notice about the previous transcript is the exuberance of the users. Everyone is thoroughly entertained and the stranger it gets, the more excited they become. They seem to care about each other’s well-being and delight in listening for the foundation of delay that is their common bond. Of course, NINJAM is audio-only and by no means on the same scale as fullscreen video productions like In Common: TIME or any of the McGill? demonstrations. However, people’s overall experience at the McGill? and Digital Worlds events pale by comparison to the NINJAMmers elation and deep satisfaction as evidenced in the transcript. Based on personal experience and this writer’s informal discussion with other audience members present, both the McGill? and Digital Worlds events are a significant development in highly interactive realtime Internet collaborations, but at this stage, the uncomfortable way in which the participants handle latency makes it too difficult to endure on a regular basis. Maybe those large universities should take a cue from the carefree NINJAMmers: cooperate with ROCM’s latency instead of competing with it. NINJAM-type ROCM engineers might collaborate with the technology available at McGill? University to develop fully immersive systems with fullscreen video using the approach that is currently working so well for the creation of Last Music?.

References

Cheshire, Stuart. 1996-2001. It's the Latency, Stupid

HYPERLINK "http://www.stuartcheshire.org/rants/Latency.html" http://www.stuartcheshire.org/rants/Latency.html

McGill? University Ultra-Videoconferencing, Shared Reality Environment and Shared Spaces

HYPERLINK "http://www.music.mcgill.ca/cirmmt/" http://www.music.mcgill.ca/cirmmt/ HYPERLINK "http://www.cim.mcgill.ca/mikewoz/research/" http://www.cim.mcgill.ca/mikewoz/research/ HYPERLINK "http://www.cim.mcgill.ca/sre/" http://www.cim.mcgill.ca/sre/ HYPERLINK "http://www.cim.mcgill.ca/sre/projects/rtnm/" http://www.cim.mcgill.ca/sre/projects/rtnm/ HYPERLINK "http://www.cim.mcgill.ca/sre/projects/" http://www.cim.mcgill.ca/sre/projects/ HYPERLINK "http://www.canarie.mcgill.ca/project_aap_desc.html" http://www.canarie.mcgill.ca/project_aap_desc.html

Digital Worlds Institute

HYPERLINK "http://www.digitalworlds.ufl.edu/projects/CommonTime/Default.htm" http://www.digitalworlds.ufl.edu/projects/CommonTime/Default.htm

NINJAM

HYPERLINK "http://ninjam.org/" http://NINJAM.org/

Cockos Incorporated

HYPERLINK "http://www.cockos.com/" http://www.cockos.com/

Chance Music

HYPERLINK "http://www.scu.edu.au/schools/edu/student_pages/2000/ajohnson/chance.htm" http://www.scu.edu.au/schools/edu/student_pages/2000/ajohnson/chance.htm Latency in telecommunications is directly associated with the speed of light, making it near impossible to practice traditional music techniques. See HYPERLINK "http://www.stuartcheshire.org/rants/Latency.html" http://www.stuartcheshire.org/rants/Latency.html Indeterministic or chance music also referred to as aleatoric music. From answers.com: Using or consisting of sounds to be chosen by the performer or left to chance; indeterminate: An object placed inside the piano added an aleatory element to the piece. Shoutcast is Nullsoft's Free Winamp-based distributed streaming audio system.

Payne – Late Music: Integrating Latency Into ROCM

PAGE

PAGE 6