CONTEXTUAL REVIEW

My exploration will delve into cognitive responses to sound and music with a focus on gestural movements and animations. For a controlled contextual review, I will first examine instances of images triggering experience, followed by sound. By looking at these in their own respective modalities, connections will be drawn between the two. 

A principle within the MIT press of Experience: Culture, Cognition and the Common Sense frequently refers to projection; a causal event wherein a viewer/listener/experiencer can foresee events which happen next. The way individuals process information is all unique, but somewhat governed by the fact that ‘our experience of coherence and meaning is produced by activities of modelling - processes of sifting impressions, cascades of calculations, and attributions of significance carved from determinations of causality, which are in turn embedded in and generated by environments we have come to know’.

In a time where artistic multimodality may have more frequently manifested in still images, some interesting thoughts and responses were emerging. In the early 1900’s, Italian Futurist painter Gino Severini was depicting experience based on his own perception of particular events. Plastic analogies (a term Severini coined) were his methodology in mapping out ‘imagistic composites that combined two or more symbolic forms into a single image’(p57). The fact that movement, motion and even sound were being depicted in still images from 1914 is astounding and represents the importance people place on exploring cognition and how artistic practices are an outlet to record, analyse and discover.

In 1943, responses of using artistry as a way to gauge cognition and experience were emerging in the form of simple animations. Fritz Heider and Marianne Simmel experimented with endowing simple shapes with motion: “The projected image shows an outline of a rectangle with an opening on its right side; near this opening the shape of a triangle arrives. Soon a smaller triangle and a circular disk also enter the frame from above. The rectangle remains static, but the disk and triangles become quite literally animated, and each moves in a manner that comes to seem characteristic. That is, they are given movements that model different characters.’(p13) The aim was to research how viewers may perceive these shapes with attributes which go beyond their two dimensional limitations. A spectrum of titles to the research emerged, from ‘Perceptual of Geometrical Figures to Warring Triangles’.(p14) These titles alone aptly represent what the researchers were aiming to achieve. Representing a film ‘devoid of content’ yielding a range of fully fledged stories. The essay I refer to comes from the MIT press book, Experience, and the Heider-Simmel film is noted to have stock in contemporary studies of neuroscience, psychology, and the ‘aesthetics of ethics’(p14). The knowledge that artistic practices can be as informative to the studies of cognition and experience as scientific methods can, alongside the epistemology that 'Cognition implies a degree of plasticity for human psychophysical processes; in this sense, it can be as active and adaptable as consciousness itself'(p8) provides a good framework for my investigation as I elaborate into my own fusion of character animation and abstract representations of sound to stimulate genuine responses from viewers from a variety of sensory modalities. Furthermore, this research continues to be relevant to modern cognitive studies at MIT, with Josh Tenenbaum using the film within studies of Visual and Environmental Studies in 2014. ‘Modelling is a practice common to both artists and scientists, and as a customary activity of humans in general; our modelling behaviours distinguish us as a species. We model ideas, we model clay, we model with pictures, texts, clothing. What were Fritz Heider and Marianne Simmel modelling when they made their enduring film? The short answer is projection. But on a larger scale, they were modelling the complexity in the human psyche’. Here we see this film within a modern context; exploring the malleable nature of cognition as an artistic tool, these films and studies become a heuristic approach to cognitive science.

There are even links between the experience of film and the principles of animation. The 12 principles of animation have much correlation with how one may evoke genuine experience in a viewer.

Look at multimodal approaches THROUGH THE LENS of Italian Futurism and Plastic Analogies

Reading - EXPERIENCE MIT Press

Key Words:

Projection - Cognitive ability to move temporally and create understandings of what is to come.

Kinetic - of or relating to the motion of material bodies and the forces and energy associated therewith

heuristic - allowing for someone to discover someting for themselves

'Cognition implies a degree of plasticity for human psychophysical processes; in this sense, it can be as active and adaptable as consciousness itself.' p8

'This book exists to complicate that model. [nervous system as a series of inputs and outputs being sent to the central processor or brain] Experience is our heuristic, and we intend both to provoke experiences in these pages, and to examine how culture and technology deeply weave the texture of human consciousness.' p15

'David Mather explores in the practice of Italian Futurists, bridging the cosmos and our own afferent/efferent nerve signals [input/output]; the Berliners' abstractions invited projection, proving that humans could construct meaning and experience from minimal perceptual cues.' p22

'Art demonstrates to science the broad political and cultural context forming what sense we make of experience' p26

Adorno's model replaces notions of  art as representation or 'expression'. Art is not the semiotic conveyance of a fixed message-It is a range of marks, indices, rituals, and materials that humans make, arguably uniquely, to simulate perception. We do these things to draw upon prior layers of experience, in order to produce, paradoxically, the genuinely new'. p26

Gives relevance to the artistic research of experience in the modern world without extensive scientific data collection. p28

Starting to map kinetic energy on a flat surface using artistic mediums. p61

'

Physics of Sound - an aesthetic exploration

 

Youtube.com. 2022. Sound: Crash Course Physics #18. [online] Available at: <https://www.youtube.com/watch?v=qV4lR9EWGlY> [Accessed 7 February 2022].

A short video on the physics of sound has helped me to understand the way it travels and how it may look as a semi-tangible entity. The magnets in speakers essentially force the sound out, affecting the gaps between particles as it travels. The oscillator above demonstrates this variation in distance.

Seeing the animated oscillator awakened me to the possibility of using this motion in conjunction with the squash and stretch animation principle. The two looks share attributes that I believe would be effective onscreen for representing sound. I will use this and the physiocromies moving forward to reflect my growing understanding of sound and my experimentation into bringing it onto the screen.

Developing my Aesthetic

 The following videos demonstrate the next stages in my testing/production processes.

Below, architectural reference shots I took from a letting agent in West London provide excellent birds eye shots for some of the urban establishing and wide shots within the film. Rather than if I had used flat buildings, the scene will contain a parallax, giving it depth and perspective.

The outline for a 10 second scene has been completed with roughly 8 hours of work. With this knowledge, I can begin to establish the timings of final production processes by using this scene as an indicator. When colouring and further realisation is completed, the timeline will take shape and will hopefully contain a time for contingency to ensure any mishaps in the process can be addressed.

As previously mentioned, when building up from the animatic, more realistic emotions and reactions need to be applied. careful consideration of the arcs of character and sound will need to be present throughout the film.

These animations are tests of the Cruz Diez research I undertook. By compositing the scene in after effects and keying out the sound movement, I can layer the textures I am currently imagining to reflect the synesthetic movement of sound.

Anim 1: Reflects the physics of soundwave travel (see previous post). The oscillator/ the concept of sound affecting the gaps between particles inspired this.

;

Anim 2: Using Cruz Diez's physiocromie concept, the sound can be manipulated by different colour spectrums. As it travels, it can pick up tone and light change. Using opacity and other elements may help to make the sounds sit in the frames in a more visually pleasing way.

   

Character Animation & Final Production Testing

 I have timed the protagonist to react in time with the music because he is physically interacting with the sound. Sound hasn't been animated yet, but through the lip sync tool in TV Paint, I can map out which sounds are when, and what reactions they trigger.

Considering this timing, I am aiding the animated performance, creating actions and reactions. So far the scene shows the character's reaction, but you'll hear the sound and this is what will trigger the character to move around. Developing from the animatic, I've added more expressive movements to emphasise the action of the sound and the reaction of the character. I have gone from the character passively onlooking to conveying a sense of his frustration, and when the sound comes in, his startled nature is far better communicated than in the animatic. This careful consideration of emotive characters and realistic reactions will aid the themes and believability of the story

This took about 4 hours to make...

The scene will eventually indicate how long it will take to animate a finished complex scene within the film to aid the production. I will need to complete the background, sound representation and colour before this is realised.

Sound Animation Tests

 

The first animation reflects my attempted recreations of Cruz Dies' Physiocromies. My ideas were to shift colours according to the colour script in the film. The beginning sounds will issue blue tones from the Government tannoys but I wanted to experiment transitioning between colours, thus creating new colours from the way our eyes see the animation. As sounds fly around screen, the world will be lit up and characters clothing etc.

The second and third animations show a rough straight ahead animation of the sounds flying around screen. I was experimenting with different colours and possible fluid trails the sounds will leave. The movement works but I want to explore a little more how this sound may look through shape, movement, opacity, light and colour.

Developing this idea is convincing me this is a good method in which to represent sound. The shapes and movements they take need to still be established. Furthermore, I'm working on a scene containing detailed background, character and physical sound. From getting this scene to a polished level (using the line drawing techniques previously mentioned), I hope to discern how long a finished animation will take to make, giving me a relatively accurate idea of how long the production will take.

I have also looked into creating keyframes from audio in After Effects. My idea is to manipulate ideas about light. For example, the sound will be brightly coloured and lit. I want to create a reaction in the character where they blink and see bright lights (much like if you look into a powerful light. This will engage the theme of synesthesia and hopefully bridge the sound and characters together (as if the sound is a character itself).