Blairatta Policeme

work jazz trio, mobile chamber quartet, and audio enhanced performance space

Artistic Overview

Work description

Work on the project began in early March 2008. The composer made two trips to Toronto, one towards the start of the project, and one at the end. The work was performed at the Lower Ossington Theatre on June 3, 2008, and was recorded live by the CBC. The work will be aired in the spring. A video documentation of the performance is included with this report.

The principal collaboration with the ensemble was in the development of a method for synchronizing mobile performers in space to a musical score with metered music (at specific tempi). As the performance space itself was used to capture, transform and reemit the players’ sounds, the score actually had to instruct players to visit specific locations at specific times. Marching band (drum corps) music and techniques were studied, and several sketches (and spatial layouts) were experimented with.

Additionally, members of the ensemble worked with the composer to develop and refine the Wiimote-based tracking system, and to sort out other details, such as wireless audio logistics. Finally, several audio effects modules were developed around each of the (four) mobile musicians in the ensemble (trmpt.,perc., vln, cla.).

Note that no sampling of the instruments was required, as all the electronic sounds used in the piece, resulted from live signal processing using the effects modules mentioned above.

Results

As stated in the original project description, the concept of “writing for the space” was of chief concern in the realization of this project. Prior to this project, the composer had worked with virtual audioscenes to accomplish location-specific signal processing (effects), and with movement tracking systems; however, the composer had little experience in working with mobile ensembles, and none whatsoever, in coordinating movement and music in space. An approach was developed to deal with synchronizing players and music in space. This approach is based on practice in Drum Corps and marching band music, however, the notation used in the score is novel, and will be used and refined in subsequent works.

Of key interest in the realization of this project is the nature of the piece that was composed and performed. As mentioned in the project description, the “performance situation” provided a space, in which, performer and audience member could move about. Because the performance space itself emitted sound at different locations, the resulting music operated as more of a field, as opposed to beacon. Unlike traditional performance venues, where a concentrated ensemble in one specific location (stage) delivers music to listeners in another (audience), this performance was best experienced by an active listener, who moved about the space among the mobile musicians. As in sculpture or architecture, the encounter of the work depended on the observer’s point of view in space. Thus, the CBC had its work cut out for itself in the recording the piece.

In short, the project was invaluable in providing an opportunity to work intensively with the ensemble, in order to, not just realize a new piece of this kind, but to develop an solid approach to the underlying techniques (score/movement etc.), and an aesthetic understanding of how this kind of work can operate. Both ensemble and composer are very pleased and keen on forging ahead in this new direction.

Technical Overview

summary

The project entailed technical innovation in the areas of musician tracking on stage, and in the area of score design, to keep mobile musicians in the right place at the right musical time. An example of the notation from the score is featured in the documentation video.

A system for tracking up to four musicians in discrete sensing regions of a performance space has been developed. The system is based on InfraRed (IR) camera tracking of IR emitting LEDs, which are arranged in unique patterns and worn by the performer(s) as ID tracking tags. The system can identify up to four tags (only one at a time) in any of the four sensing regions on stage; and it can track the position of a given tag (locally) within a region. The system was coupled to the virtual audioscape, so that the musicians’ virtual representations could be moved about the audioscene, and location-specific effects processing could occur (see documentation video or included technical doc.).

Musician Tracking System

Implementation

At the heart of the implementation is a (IR-camera equipped) Nintendo Wii controller which is mounted above a region in which performers wearing tags are identified and localized. The size of the sensing region is given by the distance to the Wii controller above it, however, if the Wii is too high (far), the pattern of the LEDs in a given tag will loose definition—or not even be seen at all. Given the Wii’s camera 45 degree view, the diameter of the region the wii can “see” is 76.5% of its distance from the wii: e.g. a downward-pointed Wii, mounted 4 meters above the floor will “see” an area on the floor approximately 3 meters in diameter. It is worth noting that using the Wiimote Sensor Bar, the limit of reliable IR-LED light strength was at this distance.

the player is recognized and localized within region of approximately 1.5 meters diameter

Problems, experimentation, solutions

With little practical experience with the Wii controller in such an implementation, several issues had to be addressed and performance had to be evaluated.

Bluetooth range

We discovered the maximum reliable distance we could locate a Wii controller from the computer it was paired to. With direct “line-of-sight” relative placement of the devices, a maximum distance of 5 to 6 meters was established. We noticed that if there were obstacles in between the devices, performance was impaired, resulting in timing errors for real-time data.

top view: Wii trackers are suspended 4.1 meters above each zone (1.5m dia.) on stage; all of which are paired to a computer on a table (dist. between Wiis and computer== 5.3m) in the center of the stage.

Tags

As mentioned above, identification tags consisting of IR lights arranged in patterns were used. After experimenting with different IR light sources (varying in intensity and emission characteristics). IR-LEDs were chosen due to efficiency and light weight, i.e., suitability for mobile applications. Having tested several different IR-LEDs, we opted to use the Wii Sensor Bar because the light sources on it have a very wide throw, and can thus be seen by the Wii with greater independence of the bar’s relative orientation.

Head-mounted Wii Sensor Bars (containing IR light emitters at each end, separated by aprox. 9”) were combined to make patterns corresponding to one of four possible ID tags. The patterns were simple: the number of lights determined the identity.

two head-mounted ID-TAGS with IR-LED light emitters

The overall size of the pattern (spacing between IR light emitters) had to be sufficiently wide so that, at a distance from the Wii of 4 meters, the pattern’s light sources would not merge (share the same pixels in the Wii’s 1024 x 768 view). A minium spacing value of one head-width (i.e. 6” proved to be suitable).

performer wearing head-mounted ID tag

Interpreting data from the Wii blob tracker

The Wii controller provides “blob tracking”; up to four separate features in the image field are recognized, and their positions are reported each time they change. The implementation and behavoir of the Wii’s blob tracking algorithm had to be analyzed and understood in order to use its output data at a higher level for ID tracking. De-bouncing, time-outs and other techniques were employed to make end performance more robust.

Logistics

The Bluetooth pairing of the Wii controllers was managed using a utility program, which assigned a device number to each Wii, and streamed its data (using OSC messaging) to our post-processing tracking client. Rechargeable batteries were used and Wii’s were attached by cords to pulleys so that the Wii’s could be easily lowered and recharged at night. A night’s charge was enough to run the Wii all day long (from 9am to midnight).

9v batteries were used to drive the Wii Sensor Bars. When the battery drained to under 7 volts, it had to be replaced, otherwise the Wii would not see the LED the battery powered. Batteries had to be changed after 5 or 6 hours of continuous operation.

Implementation

Four regions (i.e. four suspended Wii controllers) were located on stage. Four performers, wearing unique ID tags, moved among the regions.

simulation overview of stage w/ performers in sensitivity regions

photo of stage: center of zone 4; diameter drawn in surrounding dotted line.

When soloists enter into a given zone, their particular sound (captured via wireless microphone) is routed to a local virtual signal processor (echo unit), which outputs locally to a loudspeaker adjacent to the zone.

Thus, as the playing soloists move through the zones, their particular sounds are captured and echoed locally (at the zone they are in). In this way, players can leave “traces” of their playing at specific locations in the performance space.