INTRODUCTIONA violinist in Warsaw, a cellist in Oslo and a viola player in Helsinki are connected via the Internet by means of live media (audio/video) streaming technology. Due to the imposed restrictions of net protocol and streaming technology, considerable irregular and unpredictable delays will necessarily occur. In the worst case, destructive net congestion may cause complete drop-outs of audio-visual contact.
How does a traditional ensemble, a string trio, generally considered to be of such musical purity and where each musician is so closely linked to the others in both the physical and the mental sense; how can such an ensemble cope with this situation? Each player is remotely located (even in different time zones!). What will happen if the three musicians are unable to synchronously hear or sense each other during the performance?
As for the music itself, is it possible to create a composition for live ensemble without a common time-line, a music existing in more than one time dimension? Can music survive under such conditions?
The System
LIVE-MEDIA STREAMING TECHNOLOGYReal-time media transmission over the Internet is currently very popular worldwide. Video conferencing, Internet telephone, Internet TV & radio stations, etc. are hot topics at the moment. However, real-time audio/video signals present special needs for network transmission. Audio/video is time-critical and it needs Quality-of-Service (QoS) transmission. The main problems in Internet real-time audio/video are latency (network delay) and maintaining the bandwidth.
The current basic Internet architecture unfortunately allows absolutely no control over either of these factors. On the contrary, over the Internet, latency may vary and it is extremely difficult to estimate its value. Especially from the audio point of view, latency is extremely harmful, since it generates disturbingly audible breaks. Bandwidth sets limits to the sound quality. Internet telephone and videoconferencing provide point to point connections which avoid latency by compromising seriously on the sound quality - a limitation more acceptable when only speech is involved.
All streaming technology uses very sophisticated compression methods. The compression process is called "encoding" and is the first step in deploying streaming media.
Reference: Rahkila, Martti; Huopaniemi, Jyri. 1997. Real-time Internet Audio - Problems and Solutions. AES 102nd International Convention, Munich, Germany, March 22-25, 1997.
REALMEDIA
One of the most widespread media broadcasting tools is RealMedia (RealAudio and RealVideo) from RealNetworks. Compared to other methods of streaming, it is capable of transmitting high-quality audio and video suitable for netcasting live concerts and live events.
The system embraces three components: encoding of the live media, transferring it to the Internet, receiving it from the Internet. Reception of the media stream requires the playback program called RealPlayer, downloadable (free of charge) from the company's web page. A net-audience can see and listen to netcasted live concerts and their recorded archives, on demand, by using one of the popular browsers with RealPlayer installed as a plug-in.
BUFFERING
RealMedia uses the so-called 'buffering' technique as an essential part of its real-time media delivery. The playback software (RealPlayer) starts by downloading into the user's computer memory a small portion of the media before commencing playback. Once a pre-allocated amount of media has been buffered in this way, playback of that preloaded material begins.
As the user listens to the first section, RealPlayer continues background downloading of the next section of the media stream into the memory buffer. This advance buffering of the audio/video material can largely overcome the latency problems caused by network congestion or server sluggishness.
DELAY
However, buffering means delay. It takes about 10 seconds to buffer live media data. Even after streaming starts, it takes a further few seconds (depending on many different factors) to reach the receiving end, i.e. the user. Ultimately, music/video transmitted from one location (or site) will be seen and heard at the receiving end around 12 - 15 seconds later. This delay time is not constant and is never predictable. In the event of a serious download delay, a drop-out of whole sections of the data may occur and playback will cease; the video picture freezes and the music stops!
These are most crucial problems when composing music for a remotely located ensemble.
Nonetheless, the key to this particular composition project lies in the interesting delay phenomena and accidental, unpredictable drop-outs of sound inherent in our present technology. These factors inspire the composer to adopt a completely new approach to the task of creating "music".
Scenario
TOPOGRAPHY
The violinist is located in Warsaw, the viola player in Helsinki and the cellist in Oslo. The geographic distances between these cities are Helsinki to Warsaw 950 km, Warsaw to Oslo 1050 km and Oslo to Helsinki 800 km.
In terms of Internet communication, there are many routers and gateways that channel music/video data between these three cities. There is no way to trace or determine which will actually be used within any single transmission. Connection between Warsaw and the two Nordic cities is via Stockholm using satellite Delay times are always undetermined.
At each site the resident musician can hear the two remote musicians separated and isolated in the left and right channels of a stereo image, and can also see a video picture of them, projected on two adjacent screens. However, as mentioned earlier, the sounds and pictures are delayed (typically 10 - 15 seconds).
REMOTE DSP
An additional level is added to the musical experience with the DSP (digital signal processing) programs, written in MAX MSP, a programming language developed at IRCAM (Institut de Recherche et Coordination Acoustique / Musique). This DSP is used to electronically modify the acoustic string sounds during performance. The modification is controlled live and remotely by the composer in Helsinki, using the special remote TCP/IP (Internet) remote control application, called TimbuktuPro.
With the aid of this fantastic program created by Farallon Computer, the composer can take complete control of the remote Macintosh computers in Warsaw and Oslo as if they were right there in front of him. The composer thereby acts as an added fourth musician to the ensemble.
TECHNICAL SET-UP
Communication
REAL-TIME COMMUNICATION BETWEEN THREE MUSICIANS AT THREE REMOTE PLACES USING THE REALMEDIA SYSTEM
The unavoidable delay times, intrinsic to the live-streaming process, limit real-time communication/interaction to just a few alternatives:
Catch-ball: One performer throws a piece of information (audio/video) to another. The recipient reacts and throws it back either the same or processed, or possibly even something completely new.
A Loop: One player continuously throws out packets of information. Another player receives, reacts and throws back either the same, or processed or completely new information. This creates a constant loop of communication.
Simultaneity: Each participant simply throws out information simultaneously, regardless of content. No genuine interaction occurs.
The information can comprise intellectual content, volume levels, dynamic gestures and so on.
The above communication patterns can also be applied to three musicians located in three remote places, resulting in further levels of complexity.
