Introduction

Introduction

Sound and image have been media which have been cross-integrated for centuries with a range of methods. My investigation will delve into the theory behind the multi-modal within filmic theory and how this has manifested overtime into live musical performances. Similarities can be drawn between cinema and music since the moment it became a multi-modal experience. Although the information received are entirely different and can be viewed in a range of ways, the viewer as a “standard biological audio/video processor” takes in audiovisual content channelling them towards a form of emotional response, whether it demonstrates moving image predominantly or contrarily accompaniment of visual stimuli to music. As stated in The Reality of Illusion - An ecological approach to cognitive film theory, “The motion picture can be thought of as a program. And it is more precisely a program than either a language or a mere set of stimuli. It is a very complex set of instructions utilising images, actions and sounds, a string of commands to attend to this now, in this light, from this angle, at this distance, and so forth”. The essence of my research entailed having a number of approaches to cinema in mind while researching contemporary music performance.

In film, sound can be used as a variable aspect allowing for ambiguous interpretation of any visual stimulus. Charlie Batten of BFI wrote that  “visuals are what the viewer tends to mostly focus on and the sound subconsciously alters how the visuals are perceived”. In my investigation I intend to gauge the different emotional responses moving images can have over music, informed by the practice of the multi-modal in cinema. Certain theories within Michel Chion’s Audio Vision - Sound on Screen explore the interplay between sound and image within cinema. Audiovisual trajectory is a key theory explored in Chion’s studies, where he demonstrates a number of methods to give a piece a sense of continuity. It ranges from sound being used in an exact nature to give a definitive result to ways in which sound designers and filmmakers can create illusive effects when using sound and image in conjunction. With relation to music performance, visual content has been utilised both with digital precision and more loose narratives. The existence of media such as music videos and visuals stems from the predominance of visual over audio processing. Dr. Haig Kouyoumdjian of Psychology Today argues that people are more responsive to visual stimuli, supported by the 9900% increase in visual information on the internet alone. Thus, for performing musicians and their audiences, visual cues would in a way bridge a void and create more authentic responses. A study in sight over sound judgments in Virtuosi dance experimented into musicians and their ability to anticipate winning dancers with muted clips. The study concluded that paired with music, participants were able to identify the winning pieces and, In vice versa, it found that visual stimulus “could confound listeners' ability to make judgments of the quality of the music being performed”. Such data represents the complex interplay between all of the senses and justifies that within performative acts, stimulating audiences with multimodality is a sure way to boost their engagement in an act. This translates to live music because often a large audience cannot engage with a performer: To accompany the music and essentially fill a temporal gap, large spectacles of light, projection and gestural movement are applied to assist in dramatising a performance.