9 Audiovisual Interactions in the Digital Medium: Pattern Playback

It was not until the 1990s that participating visitors in interactive art projects were able to engage in joint manipulation of acoustic and visual information. One of the methods used to achieve image sonification was based on the principles of pattern playback. In 1997, Toshio Iwai created Piano as image media, an installation in which visitors use a trackball to draw shapes and patterns that are then projected onto a screen and interpreted as musical notation. The individual pixels of the patterns first move slowly line by line toward a real piano, accelerating from a particular threshold onward as they approach the keyboard, which then independently plays the corresponding note. The pixels now appear to traverse the keyboard, only—this time on a vertical projection screen—to stream out of the piano, changing into colored, geometric objects as they flow.

In his work audiovisual environment suite, Golan Levin also experimented with directly drawing the sounds, using a standard interface consisting of a mouse and a monitor. He is interested in the idea of a painterly metaphor for interfaces: This metaphor is based on the idea of an inexhaustible, extremely variable, dynamic, audiovisual substance which can be freely ‘painted,’ manipulated, and deleted in a free-form, non-diagrammatic context.^[16]

In the first application of the Yellowtail series, the sonification of the shapes drawn with the mouse and set into motion by the system is still achieved by means of an axis that repeatedly sweeps from the bottom to the top of the image, triggering a sound as soon as it makes contact with a pixel (the horizontal position determines the tone, the brightness determines the volume). In his subsequent project, Loom, Levin dispenses with this axis and generates the sound directly from the shape drawn by the user, mapping the time axis straight onto it. Thus, for example, a thicker line generates a louder note, while a change in direction increases the brightness of its timbre. The movement dynamics of the drawing are recorded and then played back repeatedly.