Multimodality in Cinema

 Multimodality in Cinema

Since silent cinema was formed, the concept of multimodality in film has grown exponentially. Starting quite simply with the application of music to assist in the overall tone and emphasis of on screen emotions and ranging to nowadays, where sound design is an integral part of the way we perceive it. Ranging from minute details to broad soundscapes, filmmakers choose to utilise sound as a way to engage audiences. Despite a number of agendas behind film surfacing overtime, Joseph Anderson in The Reality of Illusion stated that by the mid eighties,”a few courageous film theorists suggested that cognitive science might be a more productive path than the then pervasive psychoanalytic/marxist approach to film study”. In relation to my exploration of audiovisual illusion, one can see that the amalgamation of media being used on the same timeline can create a number of effects. This is all due to our cognition of different sensory stimuli being used simultaneously. Filmmakers and theorists have tapped into this, enabling them to use sound as a tool to affect the way we perceive moving images.

A blog about the psychology of sound and image notes the work of Swedish researchers Johnny Wingstedt, Sture Brandstrcm and Jan Berg. They discuss leitmotifs within film, specifically in Jaws (1975) wherein the infamous repetition of two notes being repeated among other tension building chords announces the arrival of the “eponymous” shark without even seeing it. This demonstrates how through an audio motif, occurrences offscreen can be hinted at. Not only this but within Jaws “ the researchers demonstrated how the now-famous notes come not only to signal “here is the shark” but to cause a rising sense of discomfort and danger in the audience.” Such methods are now common particularly in the genre of horror because by using this technique, events can be foreshadowed and tension can be built due to the properties of sound being as good at foreshadowing as visual cues are.

In my observations, I have seen how contemporary films demonstrate similar motifs to signal unseen or offscreen entities. Ari Aster’s Hereditary (2018) demonstrates a similar technique to Jaws, using a spoken word motif to signal one of the characters who dies early on in the film. With the clicking noise the character makes with her mouth being established as a motif early on in the film, manifestations of the sound as the film progresses creates an ever more foreboding atmosphere and sense of terror because it takes on a spectral aspect where, with the knowledge of the character in question being deceased, the sound will unnerve audiences with every new iteration. In relation to my exploration of music, a similarity can be drawn through the multimodality listening to music alone perveys to the rest of the senses. Similar to the anticipation of visual cues when hearing a certain sound. The chords within music can create neural pathways with the other senses. For example, in 2016, a University study in the US was undertaken with the intention of explaining the peculiar spinal tingle some receive from audio. DTI Brain scans were carried out on twenty participants, all of whom were listening to their favourite music. A DTI scan tests neural pathways between different sectors of the brain. The results shown were that the half of the participants who previously stated that they feel these musical “chills” had stronger neural connections: “The brains of people who felt the chills had more nerve fibres running from the auditory cortex, needed for basic hearing ability, to two other regions, namely the anterior insular cortex, involved in processing feelings, and the medial prefrontal cortex, which is thought to monitor emotions and assign values to them.” Within such studies, one can notice the potential to blend sound and image in multiple ways, varying in which medium takes more prominence. Where there are ways in which to harness this to collect conclusive data, the question of the complexities of cognition are extremely broad. As a comment in the article questions, “To what extent is a neurological description an explanation of first person singular emotional experience at the mentalistic level? Incommensurable I'd say. The musical experience remains mystically and wonderfully positive and inexplicable.” This addresses cognition in a holistic sense. With the pathways to understanding it being extremely complex, it is important to consider specifically how the theorem I am studying can explore a handful of controlled approaches to our cognition when accompanying visuals to music. Another study into the science of musical “chills”. The study concluded that around half a group of people experience this and that it stems from anticipation of beats within a tune. In a practical sense, I believe that accompanying visuals to the complex forms and individual beats or leitmotifs can boost engagement because music alone triggers our neural multimodality. Triggering other sensory stimuli in conjunction with this would only heighten the experience. 

Parallels between music and film manifest in its trajectory. Imagining sound and image running on a timeline simultaneously gives it a progressive sense. However, film contains a narrative for audiences to grasp onto. When music takes prominence, the performer will always eventually land at visual accompaniment because applying a seamless filmic narrative will direct an audience’s attention to the visuals while complimenting the music in a similar way to film and soundtrack. Within Audiovision, Chion credits this, saying that sound effects “projects onto them” (visuals) “meaning they do not have at all by themselves”. This can be harnessed in my practice by applying animation to a chosen soundscape. The result will be that “Sounds we didn’t especially hear when there was only sound emerge from the image like dialogue balloons in comics.” Through selecting theories from Chion and other cognitive research, I will discover how a number of approaches affect the way we perceive music. There are pieces of music which will compliment each approach. For example, using Skream’s electronic track Settled can test how we can perceive visuals when combined with clear leitmotifs. As it progresses, the repetition of chords will resonate progressively with each reiteration (much like the motifs within Jaws and Hereditary). I believe that combining timed visual cues will boost the stimulus by providing more information and interweaving the senses.