10 Audiovisual Interactions in the Digital Medium: Interactive Widgets

Another means for the visual manipulation of sound is its symbolic representation through objects activated within the framework of an interactive process. Golan Levin calls such objects interactive widgets.^[17]

Between 1992 and 1994, Toshio Iwai developed a system called Music Insects, in which visitors use a mouse to create drawings on a monitor. He assigned musical notes to the lines and shapes based on the colors in which they were drawn. Then he chose various insects to represent different musical instruments and programmed them to run across the screen. As soon as an insect makes contact with a drawing, the corresponding note is sounded, while white and gray color tones change the direction in which the insects move.^[18] The groundbreaking thing about this work is that it turns away from a linearly understood notation toward a system of notation organized in space. A similar direction is taken in Small Fish, created by Kiyoshi Furukawa together with Wolfgang Münch and Masaki Fujihata in 1998/1999. The fifteen different variants of this system almost all work on the basic principle that one or several pick-ups, usually in the form of simple dots, move across the screen and activate notes and change direction when they collide with each other, with sounding graphical elements, or with the boundaries of the window frame. The user can move the elements around in order to manipulate the composition.

Golan Levin criticizes many of these systems for the very limited freedom the user enjoys to influence the acoustic output, which is partly a consequence of the fact, he says, that not the sound object itself but only its environment or direction of movement can be altered interactively.^[19] With their Manual Input Workstation, he and Zachary Lieberman succeeded in developing a fully intuitive system in which the visitor can both create and manipulate shapes and notes at the same time by using hand gestures in a kind of shadow play. The use of human gestures means there is no equipmental level that creates a distance between input and output. Form and color are immediately generated by the hands.