2 Digital Code — The Shared Basis of Sound and Image

Sounds and images, which in terms of media technology are separate phenomena, are represented in the digital media by a shared binary code and described mathematically by means of numbers. This results in a fundamental transformability, which in contrast to analog transformation allows an algorithmic translation of auditory and visual parameters.

Computer music (music that consists of digital sounds and is generated on a computer) had already become common practice in live contexts in the early 1990s. By 1997, the first visualization programs as well as adequate processing power had become available, and these allowed artists to work with image sequences in real time on small, portable personal computers. Based on the principles of live improvisation in electronic music, visual artists began to manipulate and later also generate computer graphics in live settings. A new kind of real-time performance emerged. The audio or control data from the musicians’ controllers were transferred to the image-generating system and used as actuators for visual impulses. Cécile Babiole, who from the outset of her artistic career worked with the transposition and manipulation of images through sounds (and vice versa), is exemplary for her early exploration of these techniques.^[1] As early as 1999, audiences became acquainted with her work Reality Dub Bus at the Phonotaktik festival in Vienna, where Babiole converted a public bus into a rolling performance space. The audience sat in a completely screened-off area of the bus and listened to a live remix of the images and sounds recorded by cameras and microphones (and processed by musician Fred Bigot a.k.a. Electronicat and Babiole) while the bus was being driven.^[2]

Any number of interrelations between audio and video can be produced using digital means, with sounds frequently controlling the images. The sound is registered using various methods of analysis and mathematically translated into numerical values. Values for volume, pitch, timbre, sound duration, as well as heights and depths, which are broken down into a series of wave bands, are first compiled and then fed into the image-generating system. The originally auditory parameters are translated into visual parameters at the software level. The brightness, speed, size, transparency, position, and rotation of two-dimensional forms and three-dimensional bodies are just some of the parameters that can be influenced. In principle, any value can be translated into a value recognized by the respective other system without being affected by signal loss.

At the turn of the millennium, a separate machine was used for each medium, for example one laptop for the generation of sounds, another for the production of image sequences, and additional computers for control protocols or data exchange between the generating systems. Today, both laptops and the available software applications are so high-performance that all media can be generated synchronously on a single computer. It is therefore becoming increasingly easy for just one person to control the sound and image levels simultaneously. Artistic personalities have emerged who see themselves neither exclusively as musicians nor as pure visualists. The Japanese Ryoichi Kurokawa refers to himself as an audiovisual artist. For his performances, such as Parallel Head (2008) and Rheo (2009), he develops his fragile and complex image and sound worlds in a reciprocal process.