4 Semiotic Aspects

In his works about sound in film, and above all in Audio-Vision, Michel Chion dealt intensively with the semiotic aspects of sound-image interrelations, established the central aspects and coined the appropriate terminology. He has thus replaced what in film music often constituted an imprecisely deployed differentiation between counterpoint and underscoring and instead distinguishes between unemphatic sound and emphatic sound.

Chion describes music or sound effects whose mood fits the atmosphere of the setting as emphatic sound. Such sound can range from a kiss to soft string music, and from a murder to industrial sounds, motorbikes and electric guitar music, robots and servo sounds. On the other hand, Chion defines music or sound effects that express an indifference or dissonance to the film plot as unemphatic sound underlying an image montage within a time sequence. A love scene on the couch underlined by horror sounds coming from the television (e.g., in John McNaughton’s Mad Dog and Glory, USA 1993) can be perceived as an unemphatic sound just as much as the cheerful sound of children laughing or birds singing put together with images of a bleak cemetery. A sinister note enters the supposedly idyllic, kitsch front garden when the emphatically sentimental string music is enriched with screeching electric guitars or dissonant music that becomes increasingly sinister (e.g., in David Lynch’s Blue Velvet, USA 1986).^[22] Indeed, unemphatic sound is an audiovisual relationship that we frequently come across, yet it is seldom used as a real counterpoint in the sense of the equal and but autonomous design of image and sound. Exemplary for such an autonomous command of both voices is Slátan Dudow’s Kuhle Wampe (GER 1931/1932) with music by Hanns Eisler, which forms a contrasting counterpart to the image content, and also John Smith’s Film Blight (UK 1996), in which the montage and sound design are perceived as two complementary layers of a composition.

The term diegetic sound is often also defined as source sound and used to describe exactly that. Nondiegetic sound on the other hand is drawn from an external sound source, which is why one also speaks of extra-diegetic sound.^[23] A brass band crossing the screen delivers the original sound, the diegetic sound, even if the sound is moving out of the edge of the screen into the off. When at the beginning of Apocalypse Now Martin Sheen looks through the blinds of his hotel window to the street, we do not see a brass band but we hear it as soon as he steps away from the window back into the room. The music of the brass band is then a non-diegetic sound that has the same effect as an atmo. Film music, such as the music of The Doors is usually clearly recognizable as a non-diegetic sound. The visible stamping of a dinosaur’s feet in Jurassic Park (dir. Steven Spielberg, USA 1993) on the other hand provides a diegetic sound even if we are in this case dealing with Foley Sound. Dinosaur sounds are obviously not reproducible but only representatively assembled, but at the same time have a logical-diegetic causal relationship to what is happening in the image. Diegetic and non-diegetic initially do not have anything to do with on and off screen; diegetic sounds can originate from documentary, authentic original sounds as well as fictional, artificial Foley. These image and sound relationships we know as diegetic and non-diegetic have also been redefined by Chion. For non-diegetic film music (the film score), he uses the term acousmatic sound and also transfers this to the voice-over, which he names the acousmêtre.^[24]

The perceptive phenomenon of the synchresis on the other hand only backs up the evidence that the sounds belong to the pictures. As a result one can also set up a matrix for the categorization of the images and sounds that differentiates between syntopic (belonging to the film space) and synchronal (belonging to film time). There could however be sounds within the scene that can initially be categorized as neither one nor the other. The beginning of Apocalypse Now again serves as an example: while we see Martin Sheen lying on the bed, his narrative voice-over talks about the jungle in a kind of inner monologue. In addition we hear the sound of birds and the chirping of crickets (copied over each other several times by the sound designer Walter Murch), which replaces the previous street atmo and the music. The sound backdrop of the jungle suggests a narrative place and time that have nothing to do with the actual place of action, the film set for that scene, but instead come non-diegetically from the off — from the protagonist’s memory. Thus the sound level could be perceived as a mental internal sound and then again a place of action that can be categorized as the figurative body of the person remembering, in other words both syntopic and synchronal.

In this context, Chion introduces the terms internal and external logic. He perceives external logic in an audiovisual montage to be the logic that enables the continuous flow of sound to contain sudden effects that are not generated by the image. This functions through knowledge of the settings and environments. For example the visual information “city street, cut, inside a hotelroom” suffices to convincingly be able to employ non-visible police cars, helicopters, trains, cars, tram bells, etc. as a backdrop. The visual information loudspeaker in a room also suffices as an explanation for any form of music or spoken information that might be audible in the room. On the other hand, Chion perceives internal logic as the logic that is motivated by the flow of sound generated by a narrative itself: a protagonist presses a switch for example that generates the sound of an industrial crane being driven, although one doesn’t see it in the darkness of the industrial hall. Or he speaks into a telephone and we hear the answer of an imaginary dialogue partner who remains invisible.^[25]

Furthermore, according to Chion, added value is created by the relationship between the image to the sound and the sound to the image. He uses the term to describe the expressive or informative value with which the sound supplements the image or vice versa.^[26] The fact that a weapon appears dangerous relies above all on its metallic, heavy resonance and, when a shot is fired, on the force and dynamic as well as the bass elements in its sound. Apart from the information added using such psychoacoustic effects, further meaning is engrained in the images mostly through verbal statements. The queue at an airport might be loaded with a very different meaning if a terror warning is heard through the loudspeakers than if the passengers are asked to kindly go to the departure gate. The same applies to information heard on a car radio in a road movie. If one hears in the radio that there is a serial killer loose in the area, who has been getting into cars under the guise of a hitchhiker, it gives the images a very different quality than if a speaker introduces the next country song.

It is precisely this phenomenon of added value that makes it clear how much a film’s construction of meaning depends on the link between sound and image.

There is thus a broad scope of creative potential in the complex interaction and interplay of the technical, perception-oriented, and semiotic aspects. However, in order to achieve a harmonious emotional, rhythmic, dramaturgical, and narrative effect on the viewer within the film experience, it is paradoxically necessary to keep the image and sound levels separate in a strategic and calculated manner before selectively bringing them together in the montage and mix. The ability to immerse oneself in a film is intensified by the sound universes that wash around the viewer in the cinema.