Statement of Intent

Statement of intent

Multi modal connection between sound and image with manifestations in music and film

The use of sound in conjunction with visual media has been used in a variety of ways for centuries as a means to add emphasis to performative acts. In film, sound can be used as a variable aspect allowing for ambiguous interpretation of any visual stimulus. Charlie Batten of BFI wrote that  “visuals are what the viewer tends to mostly focus on and the sound subconsciously alters how the visuals are perceived”. In my investigation I intend to gauge the different emotional responses sound can have particularly over music, but informed by cinematic sound designers. Furthermore, in recognition of the complexities of the brain as a processing centre for such things, I wish to look further afield at people who study connections (or lack of) between the senses and how this can be channelled in creative responses.

When regarding the pairing moving image with music, it is arguable that such practice is human nature. Dr. Haig Kouyoumdjian of Psychology Today argues that the brain is primarily an image processor. Thus, for performing musicians and their audiences, visual cues would in a way bridge a void and create more authentic responses. A study in sight over sound judgments in Virtuosi dance experimented into musicians and their ability to anticipate winning dancers with muted clips. The study concluded that paired with music, participants were able to identify the winning pieces and, In vice versa, it found that visual stimulus “could confound listeners' ability to make judgments of the quality of the music being performed”. Such data represents the complex interplay between all of the senses and justifies that within performative acts, stimulating audiences with multimodality is a sure way to boost their engagement in an act.

Incorporation of such stimulus is on the rise and ever expanding. The internet alone saw a 9900% increase in visualised information. In the field of experiential design for performance, an ongoing show at the Design museum represents the visual work surrounding Kraftwerk and the Chemical brothers. The motion graphics which feature at their shows could be an effective way of discovering how practitioners pair visuals and music. Such shows incorporate bespoke visual design and I want to discover if the lights and projections are made in time to the music or whether it allows for the viewers to make connections between the two using multi modal connection. This will branch into my practical research because I want to find a way to create a set of visuals which can accompany different music. With this I hope to show that with the cognitive connections sight and vision have, visual stimulus unrelated to the music can generate a range of emotional responses from audiences. 

My aim to continue nurturing my knowledge of this topic is to gather more research and data. From observing different approaches I hope this can inform my own understanding of the complex cognitive connections between sight and hearing. This should aid in my investigation because I will find more effective ways to demonstrate this with my practical and investigative piece.