2 Procedural aspects of Audiovisual Montage

If the montage determines and limits the shots and their rhythmic, dramaturgical, and narrative sequence on a timeline, then this not only applies to the images, but also to the sounds. The editors move along this axis, juggling with different kinds of sound that can be differentiated into reproduced and representative sounds with regard to their relationship to the objects or persons shown. The reproduction describes an equal relationship between the original and the image (or the recorded sound, author’s note), expressed as the formula A is like B.^[4] Representation on the other hand stands for a relationship between the original and the image that does not exclude the possibility of a transformation, for example A sounds like B.^[5] According to this, reproduced sounds would for example be original sounds, wild sounds, respeaks, atmos, and to some extent primary sound, while ADRs, Foleys, FX, SFX, music, and ALTs can be inserted as representative sounds.

Depending on the film genre, different levels of importance are placed on reproduced and representative sounds. In the field of documentaries, the reproduced sounds are dominant because they make references to authentic situations and are thus more credible. In the case of feature films however, the illusionary aspect requires additional sounds, particularly in the case of science fiction. In recent years however, an increased use of representative sounds can also be observed in non-fiction, for example when Foley sound is mixed in for dramaturgical reasons. For Black Box BRD by Andres Veiel (GER 2001), the sound designer Arpad Bondy for example synthetically created a cool rustling sound as an atmo for the Deutsche Bank, which he mixed together using the sound of a balloon from which the air was escaping and the trickling of a rainstick, an authentic botanical sound body. For Touch the Sound (GER/UK 2004) by Thomas Riedelsheimer, the sound designer Christoph von Schönburg also used original sounds taken from one sequence (a factory building near Cologne) in another place in the film (a street in Manhattan with a building facade full of air conditioners) as representative sounds in a manipulated form.

Prior to the sound mix, the montage determines the temporal relationship between the individual elements on a visual and audio level. Their cuts and transitions are usually not identical and off-sounds and on-sounds overlap, in front of or behind the images.

This discrepancy is due to the fact that sounds require a different duration and presence on the timeline than the images in the frame. While sounds, above all speech and music, can only unfold their meaning in an auditive space when they are given enough time, images can communicate information in the shortest of frames, can take a jump through time and space, and work with a higher editing speed than the dialogue and rhythm of the soundtrack is able to.

This different way of dealing with image and sound is particularly visible in music videos, in which the audio and visual tracks are for the most part independent of one another.

Frame cuts, the characteristic assigned to video clip aesthetics, the act of operating with short takes composed of only a few frames, does not have an equivalent on the audio level. On the contrary: the piece of music, for which the video is produced, is played through without a break.^[6] Despite this there is a perpetual solidarity, as Michel Chion calls it, between the two components, created by synchronization points at which the edit and the beat come together.^[7]

In the case of the dialogue, the editor has the scope of discretion for the cutting point. The montage defines the psychophysical context of the dialogue by deciding who is shown when on and off screen or monitor, and if action or reaction is shown in the sequence. Because it is possible to hear the complete dialogue on the audio track however one need not see everything. The sound creates its own sensation of space. The retina creates a neurophysiologic limitation for the field of vision; the movement of the eyelids causes caesuras in the act of seeing. Our ears on the other hand cannot be closed and we hear more in the space we inhabit than we can see. We can hear something behind us but we are not able to see something behind us while looking forward. The montage can use this to produce moments of tension, as in the case of an orientation reflex, with the eye-follows-ear effect, when the montage often later delivers on screen what has already been heard in the off. Fritz Lang’s first sound film M — Eine Stadt sucht einen Mörder (Murderers among Us , GER 1931) operates, in particular in the exposition sequence in which the mother, daughter, and child-murderer are introduced, in what was a dramaturgically highly efficient manner for his time with the elements of on and off in order to build up tension.

The sound frequently takes on a homogenizing function (or a unification, as Chion calls it) within a scene or even during alternating cross cuts between parallel sequences of events.^[8] Here, homogenizing does not mean harmonizing but rather the unifying of heterogeneous visual passages through the continuity of the soundtrack. In the case of Apocalypse Now by Francis Ford Coppola (USA 1979) for example, the sound designer Walter Murch added music by The Doors to quite different scenes.^[9] It can be heard both during the exposition, where Martin Sheen is introduced through flash forwards (short, motivic lookaheads that are not further developed until later in the film), and later when a water buffalo is slaughtered in a sacrificial ritual in a temple district towards the end of the film, where Marlon Brando as the tyrannical ruler over Sheen is brutally killed, simultaneously in terms of time and as a parallel in terms of content.

All image-sound montage uses editing and layering techniques in fundamentally different ways. While it is common practice both in constructivist and in continuity editing to cut the perspectives, time periods, or different locations so that they sequentially overlap, these disparate levels are usually linked by the sound, which generally develops in a constant, linear way, whether we are dealing with music or off-narrators or the continuous sound of the scene in a narrative. At the same time, the sound in film is always multilayered and as a result depth-staggered. In contrast, usually only one picture is visible at any one time. Here it is rare to combine the different processes, in other words to link a multilayered soundtrack (music, atmos, effects, Foleys, dialogue, voice-over) with several interlocked images (e.g., in the case of Peter Greenaway’s Prospero’s Books, UK/NL/FRA/JP 1991) or jump-cut image montages in combination with jump-cut a-b sound (e.g., in the case of Jacques Tatis’ Playtime, FRA/IT 1967). One example of an experimental approach to the possibilities of editing and layering is Claus Blume’s Kniespiel III (GER 1990), in which he transfers the musical layering of the rhythmic components of a beat to the approach to the visual elements in the video editor. Often, a congruency is created between the a-b editing on the level of the sound design and the sequential jump-cut procedure on the level of the visual edit. At the same time the development of the music in the film score or the sequence of the narrative voice-over set over disparate image-sound design sequences remain linear. Precisely this relationship between sound and image is frequently fulfilled by the bracketing or the homogenization of wild editing sequences (as can almost always be found in the films of Francis Ford Coppola), while long, planned sequences accompanied by wild changes in sound can be found more frequently in experimental films (e.g., films by Jean-Luc Godard).

The linking of sound otherwise always heard in a 360-degree radius to the fixed perspectives of the two-dimensional film with its conventionalized avoidance of crossing the line, provides another creative opportunity in the montage. Tableaus that visually establish the relationship between the audience and the stage or the square of the image in front of them in an almost static manner can be expanded in the sound studio to include things that are next to or even behind the film setting. In his mono films, Andrej Tarkowsky frequently made use of these possibilities that today can in particular be found in surround cinema, for example in the films of David Lynch or Peter Greenaway.

These instances should make it clear that, as in the case of the montages made with the recipient in mind, the audiovisual connections are merged to form coherent whole in the film experience — because montage is above all a category of cohesion.