CYBERJAMMER - Concept Design – CMPS 80K – Winter 2007

Version: 04.30.07

Name: Cynthia Payne - SpecularOfferings

Title: CYBERJAMMER

Genre: Multiplayer Online

Platform: Computer, console, or location-based immersive

Team size: TBD

Description

CYBERJAMMER is an augmented reality online environment designed specifically for real time music collaboration where the player’s own body drives the movements of a 3D avatar they design for themselves. CYBERJAMMER enables people to experiment with different body images and movements while jamming music live online as a 3D character. An optimal experience is achieved through life-sized projections that a similar to the dimensions of the participant, but CYBERJAMMER is scalable in that participants can choose the level of immersivity they will engage.

· The 3D avatar’s image will be projected life-sized, along with life-sized images of the other avatars.

Each avatar has unique power to create visuals and play virtual instruments included with the basic application.

· The movements of each player will trigger the movements of the personal avatar they have created. In order to facilitate this, players wear lightweight sensors at wrists, elbows, waist, knees, and ankles.

· The music originates from the physical instrument and equipment utilization and resources of each player. Additionally, there are several Embedded Virtual Instruments (EVIs - see below), which can be accessed via SonicArray (see below), and a mouseplay* interface called, SPINNER, which allows the manipulation of the incoming audio stream. *Mouseplay is using mouse-like device to facilitate access to the interface (see SPINNER).

Access and Visualization

Community Philosophy: Public First

Everyone starts out in an open jam. The underlying philosophy of the CYBERJAMMER community is that the conditions remain fluid – people are free to come and go between several different jams going on simultaneously in different rooms. They must listen closely and be sharply aware of others in order to play well, and be rated such that they can generate visual displays and be eligible for private jams.

Private jams are available only to people who have played together in an open jam at least three times, and where all the participants have achieved a specific rating. (Although the CYBERJAMMER software is not open source, if people want to play only privately there is other free and open source software that people can use with there own servers.)

– Community is created by playing well with one another and by engaging in chat screen dialog, or other communications in the CYBERJAMMER FORUM, where people can meet up with others, arrange jams, and post links to previous jams.

– There are several unconnected jam rooms to choose from and the rooms have black backgrounds.

– Participants will need an instrument to play or a microphone, and have their sound equipment online when they enter the CYBERJAMMER.

– At random intervals a player might be rewarded with the ability to cause their musical output to trigger some kind of visual display, something like the iTunes visualizer (see Reward Structures).

– Players are free to come and go in the jam rooms.

– Each BE (see below) has a unique power to create visuals and use the virtual instruments imbedded in the CYBERJAMMER.

BodyEmulator (BE)

The BodyEmulator is part of the CYBERJAMMER software interface that allows the participant to design the body that will respond to their own body movements. The BE consists of an array of choices for different body styles as noted in this document.

BE Types

Note: In this version all character types are players that can play any style of music with any other musical character type.

– Avatars enable players to visual cues between players in a unique and experimental manner.

– Character types are provided as an opportunity for players to try on different bodies.

– Players choose from a wide variety of character types and once they decide on one, the Vision Tracking System (VTS) maps the player’s movements to the avatar.

– The avatars will have limited mobility such that they do not require collision detection, but that they can still move around in a shared space.

– The look of choices for avatars may resemble or be a combination of musical archetypes representative of various genres such as:

· Rock icons (metal, folkrock, punk, rockNroll, progressive, grunge, alternative, country-rock, and others)

o 50’s: Chuck Berry, Elvis Presley, Little Richard, Patsy Cline

o 60’s: Janis Joplin, John Lennon (Beatles), Ringo Starr, Carole King,

o 70’s: Jimi Hendrix, Jimmy Page

o 80’s: Joan Jett, Cyndi Lauper, Donna Summer, Stevie Nicks

o 90’s: Bjork, Tori Amos,

o 00’s: ….

· Country: Johnny Cash, Dolly Parton

· Jazz: John Coltrane, Dizzy Gillespie,

· Blues: Billy Holiday, John Hammond,

· Classical: Bach, Beethovan, Brahms,

· HipHop: RunDMC, Destiny’s Child

· R&B/Soul: Aretha Brown, James Brown, Rick James…

These are just a handful of possible character types.

Embedded Virtual Instruments (EVI)

EVIs are virtual instruments embedded in different areas of the CYBERJAMMER playspace. EVIs mostly consist of drum sounds and drum patterns that can be engaged to provide rhythm for participants. EVIs could also be interfaces such as the SPINNER, which allows a player to 'scratch' the audio of an incoming stream (see SPINNER below).

Players access the EVIs by physically walking in order to move their avatar into a position where the instruments are located and can be engaged. The player wears sensors to move the avatar into the place where the instrument can be played. Some of the instruments are in the wall, and some are in the air. Some will require a controller and body, while others just the controller. The player will be able to trigger abstract visualizations based on their movements, and audio output of their individual instrument. So the offline player is rewarded for playing well by having the power to trigger visuals and move her character around in order to play virtual instruments.

SPINNER

The idea with SPINNER is that even non-traditional musicians can participate in a musical jam. The SPINNER allows players to manipulate the sound of an incoming audio stream and send it back out to the other participants in the jam. It also "spins" visual images, that is, the round disc in the middle will load a movie and play it while spinning and scratching any audio stream that is routed in through the computer's soundcard.

SonicArray

The goal of SonicArray is to enable participants to “play” sounds via body movements. The generated sounds

can be used during live performance, for musical composition, as part of a public art installation, or just for sonified fun.

The staged performance was our first consideration. In order to facilitate simultaneous development and testing, we

set up the “stage” right next to the computer by laying down a black t-shirt on the table and shining a light on it.

The camera was positioned from behind such that the direction of movement mirrored what we saw in the video

monitor of the Cyclops screen. By pointing the camera downward at the lighted “stage”, we simulated a theatrical

environment as well as creating a highly sensitive surface for tracking changes in movement and color.

The SonicArray patch consists of 8 possible zones, each of which can be associated with a given audio sample.

When there is activity in a particular zone the associated sound plays. Various types of controls are available for

each zone. For the purposes of this phase of development, 4 zones were set up (1, 3, 5, 7) each with a threshold

setting of about 128. This means that the amount of activity in each zone was measured and set to about half of

the maximum amount of sensitivity (255).

Reward Structures

Ratings (AI+peer):

The challenge in open public collaborations like CYBERJAMMER as we see it is to motivate people to contribute in ways that are playful while also encouraging high artistic merit, i.e. “to not simply draw penises”, so to speak.

In order to determine the worthiness of a jam an algorithm will analyze and rate the jams at measured time intervals. This analysis will be based on criteria such as sustained interaction, rhythm, chord structure, keynotes, and other basic guidelines for good musicianship.

During any given jam, each player will have one chance to rate at least one other player in the session.

The two ratings (algorithmic plus peer) will be combined and become each player’s overall rating, and labeled "rating" on the interface.

A higher rating will activate a button to allow players to “upload to MySpace”, and they can automatically post an mp3 of the last five minutes of the jam to their MySpace page.

Communications

In-session communications with other players is via text chat screen and speech via microphone. When not in-session discussion, arranging sessions, and sharing links to session content is facilitated through registering with the CYBERJAMMER Forum.

Conflict

Because everyone must start out in the open jams, there is conflict inherent in being able to play well, and in playing well with others to produce meaningful content together. Varying levels of skill creates conflict when one player is able to do things another player cannot. Other conflict is possible when players consider theirs and others' body image, and the kinds of movement and displays they can create.

Equipment

Base Equipment

CYBERJAMMER hardware interface including a soundcard and software:

(Embedded Virtual Instruments (BodyEmulator, and EVIs: SPINNER, and SonicArray)

Musical instrument(s) of choice

Computer

Soundcard

Speakers

Motion tracking device such as The BodyPad

Video camera

Projector and projection surface or monitor screen

Possible Scenarios

A CYBERJAMMER participant, we'll call her Julia begins by strapping the motion sensors around her wrists, elbows, waist, knees and ankles. She then plugs her guitar into the CYBERJAMMER interface, and chooses the appropriate sound device on the computer.

Julia opens the BodyEmulator (BE) and chooses the image that will be projected according to her own body movements. She locks in the BE and then tests the movement response, sound and connectivity by pushing the "preview" button on the software interface. This will allow her to make sure the equipment and functionality are running smoothly before actually beginning a session.

The CYBERJAMMER interface shows Julia choices for joining a number of separate live sessions already in-progress, or entering an empty room to start a new session. Julia can press the "audition" button on the CYBERJAMMER interface to get a 5-second audio-only sample of the music currently being played in any particular room. Julia likes the sound of that music she hears in the red room and pushes the "join" button.

Once inside the session, Julia sees her own BE, and the BEs of a guitar player and a vocalist with a microphone. She strums her guitar and her BE mirrors the movements she makes. Julia plays a chord that is complimentary to what the vocalist and guitarist are playing, and the algorithm has determined that this is good musicianship and rewards Julia with a splash of color on the floor around Julia's feet. The guitar player likes what Julia has played and decides to give her an early rating of 10 on a scale from 1-10 (10 being high).

Now Julia kicks up her leg and the algorithm rating combined with the high 10 peer rating gives Julia the ability to produce a visual display. The vocalist also likes what Julia is doing and gives her a 10 rating. Now Julia has access to the "post to MySpace" button. She can now publish the last five-minutes of the jam to her MySpace page.

Another player logs on to this jam and is playing something that is completely different from what Julia and the others have been playing. Since this is the third time the original three have played together, and Julia has now achieved a specific rating already attained by the other two such that they can all enter a private jam by selecting "private" on the CYBERJAMMER interface. This effectively leaves the disruptive player in the room and places the original three into another private room. This means now that no one can enter the room where the original three have been placed.

Possible Vision Tracking Systems

The tracking system might be similar to the one used for Fugitive by Simon Penny

(UC Irvine ACE Program), pictured here, using the Traces Vision System,

(motion tracking with video), and projections of life-sized images on 30-ft. projection surfaces.

For the console version, participants

might wear something like The BodyPad,

sensor devices which allow the mapping

of movements to be mirrored by the

avatar they have designed for themselves.

The graphics might look

something like the characters
in the video game "Guitar Hero"