Literature Review on Western Audiovisual Synesthesia and Art Experiment Research ()
1. Introduction
Synesthesia is a subjective, involuntary perceptual phenomenon, in which a stimulation from one sense can induce perception experiences from another or even multiple senses (Pearce, 2007). This term was derived from the Greek words “syn” (together) and “aisthesis” (perception) (Cytowič, 1995; van Campen, 1999; Karlsen, 2001). Among the more than 70 known synesthetic phenomena, audiovisual synesthesia is more common (Berman, 1999; Safran & Sanda, 2015; Day, n.d.). Throughout history, this has not only given rise to philosophical debates on perception, and new directions in psychology and neurology, but also brought limitless possibilities for artistic creation.
Although a number of studies have scrutinized the development of Western audiovisual synesthesia art, most of these did not involve postmodern and contemporary arts. With the popularity of human-computer interaction, audiovisual synesthesia and its art experiments are advancing with the times, and even gradually jumping out of the original concept, which do not apply to the previous context. The present review therefore focused on the concept of audiovisual synesthesia and related art experiments, combined with the social, technological and ideological changes in the context of the times, in order to explore how audiovisual synesthesia art has grown under the comprehensive influence, and to find the new context for further interdisciplinary research on audiovisual synesthesia.
2. Early Studies before the 18th Century
2.1. Ancient Greece
The study of audiovisual synesthesia dates back to the period of ancient Greece, when philosophers determined whether “color in music is a physical quality that could be quantified” (van Campen, 2007: p. 45). Furthermore, Pythagoras has proven the mathematical principle that pitch is determined by the length of the string. In “On the Soul”, Aristotle proposed the concept of “koine aesthesis” (common sensibles) (Aristotle & Shields, 2016: p. 260). It is influenced by “the sensus communis in its own nature”, which can “perceive the common qualities in the different exterior senses” (Owens, 1982; van Campen, 2007: p. 152). During the same period, he summarized the five human senses in “On Sense and the Sensible”, namely, sight, hearing, smell, taste and touch (Aristotle & Beare, 2010: p. 2; Zhu, 2017). In an empirical test, Aristotle and his students arranged the measured colors in a scale, from black to white, according to brightness. The lowest and highest notes were linked to white and black, respectively, while the other notes were linked to the corresponding grey values, from low to high (van Campen, 2007: p. 45). Although the results were inconclusive, and could not explain the phenomenon of synesthesia, Aristotle begun to examine synesthesia from the perspective of human perception (Suzuki, 1952: p. 14).
The philosophers during this period demonstrated a quantifiable system for linking notes to colors based on mathematical and physical thoughts, which has greatly influenced the directions and methods of later researches.
2.2. The Middle Ages
Thereafter, the Aristotelian worldview became the mainstream in the West for over a thousand years. In the Middle Ages, Aristotle’s metaphysics profoundly influenced various religious systems, especially Christianity. In this worldview, the harmoniously ordered universe was a system with inherent and obvious similarities between different elements, including color and sound.
According to the mediaeval theologians’ understanding of the Aristotelian worldview, light was a vibration in the fifth element, “ether”, in addition to the terrestrial elements of earth, water, air and fire (Betancourt, 2013: p. 18). Colors in the spectrum were produced when complete and pure sunlight is “broken” by a prism. In this imaginary priori model of universal order, the vibrations of ether resembled those of sound. Therefore, sound and light were inferred to be correlated, and even similar, to some extent (Betancourt, 2013: p. 18).
2.3. The Renaissance
The comparison between painting and music actually stems from another that is correlated to painting. The Italian word “paragone”, which means comparison, was a debate on the forms of art during the Renaissance period, and the most intense among these was the opposition between painting and sculpture. Painters considered painting to be superior to sculpture, which was full of materiality, and compared painting positively with other art forms (Junod, 2018: p. 39). For instance, Leonardo da Vinci called painting and music “sorelle” (Italian for sister). However, he saw painting as a “cosa mentalale” (something of the mind or spirit). Therefore, it was still privileged than music (Junod, 2018: pp. 39, 41).
Meanwhile, some artists began to test the hypothesis of music-color correspondence through art experiments. The Italian stylistic painter, Giuseppe Arcimboldo, conducted the first known experiment in 1590. Based on the theories and experiments of Pythagoras and Aristotle, he deduced a system of correspondence between the intervals in music and grey values in color, and created the instrument “gravicembalo” using slips of paper of different colors placed on the keyboard (van Campen, 2007: p. 46).
2.4. Modern Science
While arts flourished during the Renaissance period, natural sciences were also thrown off the shackles of medieval theology. “Optics”, which was published by Isaac Newton in 1704, provided further impetus to the study of music-color correspondence. Newton proposed the concept of “refracting light”, whereby the spectrum was produced by the “refraction” of sunlight through a prism (Betancourt, 2013: p. 13). However, influenced by medieval theology, he considered that audiovisual harmony is correlated to the frequency of light and sound waves, thereby mechanically matching seven colors of the rainbow to seven musical notes.
From the ancient Greece and Middle Ages to the Renaissance period and the period of empirical sciences represented by Newton, scholars in early studies argued on the physical, quantifiable, empirical fact-based connection between music and color. Artists have also compared painting to music, which continued throughout the development of Western art.
3. Development of Audiovisual Arts after the 18th Century
3.1. Color Music during the 18th and 19th Centuries
The concept of color music was born out of the early worldview, following the view that there was a physical link between music and color. However, after Archimbutto, it was not until the 18th century that color music reached its critical period of development (Junod, 2018: p. 37).
Louis Bertrand Castel, a French mathematician and theologian, invented the “clavecin oculaire” in 1730 (another statement is between 1720 and 1754), in which the corresponding colored slip of paper was revealed from the lid of the instrument when a specific key was struck. Compared to Newton’s seven pairs of note-color correspondences, Castel extended these to 12 pairs. Although his experiment had limitations, in which the colored slips could only be lit by candlelight, this was a landmark that refined the theory of color music (van Campen, 2007: p. 47).
During the Industrial Revolution, which began in the middle of the 18th century, the invention of the steam engine offered important technical support for color music. British painter and inventor, Alexander Wallace Rimington, followed Newton’s footsteps, and patented his “color organ” in 1893. This was subsequently published in 1895, and provided a major boost to the development of color music. The color organ had a compartment that had 14 colored lights with controlled hue, brightness and saturation. Rimington sought to demonstrate the inherent relationship between music and color, thereby establishing the relationship of “common knowledge”. However, his color organ could not really play music (Betancourt, 2013: p. 20).
The contribution of color music to the history of arts is indelible, but the fact is that the inability to achieve simultaneous audiovisual effects has become a common problem with the products of color music experiments. The fundamental reason for this is that the nature of the equivalent music-color relationship developed by early scholars, and the proponents of the color music were not valid. Rational methods, such as measurement and reasoning, were used in previous experiments. However, these were still subjective and arbitrary, without uniform and scientific experimental standards.
3.2. Development, Turning Point and Transition of Theory and Practice
During the time color music was in the critical period, the comparison between painting and music also had further developments. Some artists continued Da Vinci’s ideas from the Renaissance period, and the concept of “arti sorelle” (sister arts) was rapidly developed in the 18th century. Meanwhile, the ideological emancipation led by the Enlightenment also reshaped the understanding of art. In his “Essay on the Origin of Languages” in 1755, French philosopher, Jean-Jacques Rousseau, explicitly criticized the analogy between color and sound, arguing that the former corresponded to space, while the latter corresponded to time, and that these two should be separated (Rousseau & Scott, 1998: p. 325; Junod, 2018: p. 51). Nevertheless, Rousseau also endorsed the integration of vision into sound. However, the trend at that time was still towards the interconnection of the “three sisters”: music, painting and poetry. In the wake of Rousseau, it was not until 1766 that the German writer and literary theorist, Gotthold Ephraim Lessing, discussed the inherent difference and opposition between words and images in “Laocoon: Or, The Limits of Poetry and Painting” (Junod, 2018: p. 39). It can be noted that scholars throughout the 18th century were attempting to classify art.
The rise of Romanticism became the turning point of this “new paragone”. Unlike Rousseau and Lessing, some Romantic artists emphasized the unity of art. In a conversation with Frédéric Chopin, French painter, Eugène Delacroix, referred musical harmony as including a logical relationship among chords, and called these, “auditory tints” (Junod, 2018: p. 47). Romanticism praised music as a form of art that was both expressive and spiritual. Therefore, this was in a leading, dominant position alongside painting. As the Romantic movement developed, the quest for the unity of painting and music pointed in a new direction for audiovisual art, which was largely continued by the later Symbolism.
However, comparison remained an important issue for all art movements. The comparative studies in the 19th century significantly contributed to the development of Western aesthetics (Junod, 2018: p. 41). There are four basic approaches to comparing art forms. The first is parallelism, in which the diversity of artistic expression methods determines the differences. However, in essence, all arts have the same function to express the spiritual world of human beings. One form of art can thereby be used to explain another. The second is convergence. Theatre in the 20th century was considered to have been “born of the fusion of music, painting and poetry” (Junod, 2018: p. 44). The third is divergence, which appears to be less accepted by some artists as the opposite of unity and a symbol of “degradation”. The fourth is succession, which provided an important basis for evolutionary theory. Music came to be regarded as the highest form of art, which had a profound influence in the 19th century and later periods. These four comparative approaches are compatible and complementary, linking painting and music at different levels, and to different degrees. This connection can be perceived at the physical level through light and sound vibrations, or at the psychological and spiritual level through sensory resonance. Color music gradually transitioned to visual music.
3.3. Visual Music and Abstraction in Modern Art
As a product of the times, visual music was inherited from color music, but was developed into a new concept that was closely correlated to abstraction. During the period from the late 19th century to the middle of the 20th century, the industrial revolution, development of science, philosophical debates, economic crisis, world war, and other global and social issues challenged the people’s old worldviews. In particular, the theories of Sigmund Freud, an Austrian neurologist and the father of psychoanalysis, opened up new directions for research on the human mind (Mann, n.d.). Artists began to challenge traditional ideas, emphasizing human personality and the spiritual world. Based on previous debates on the forms of art, painting and music were both considered as pioneering arts due to its immateriality. Meanwhile, the special insubstantiality of music enabled it to represent the ideas of “expressivity” and “purity”. The trend in Western art became clear, since “painting [should] take on something of quality, characterizing the essence of music-abstraction” (Junod, 2018: p. 55).
Compared to color music, the emphasis on the spiritual world makes visual music pursue the arbitrary relationship between color and sound, which is separated from the material world. One-to-one correspondence is one of the aesthetic parameters, but this is not a scientific fact (Betancourt, 2013: p. 23). Thus, although visual music is closer to audiovisual synesthesia in the psychological perspective, this is not sufficient to present scientific claims.
From the aspect of artistic expression, modern painters began to explore the connection between vision and music. The French post-impressionist, Paul Gauguin, declared that “painting in color is entering a musical phase” in his “Racontars de Rapin” in 1902 (Junod, 2018: p. 55). Czech painter, František Kupka, one of the founders of the early abstraction movement, considered that artistic creation is based on the cooperation of all senses, and created “Disks of Newton” from 1911 to 1912, as a tribute to Newton (Junod, 2018: p. 135). In his series “Rythme Colore” in 1913, French Orphic painter, Leopold Survage, attempted to introduce the idea of time into visual art (space), and included his concept of “colored rhythm”. He saw film as an ideal abstract audiovisual medium, and as a tool that can convey this effect to the audience (Betancourt, 2013: p. 45).
Russian composer, Alexander Scriabin, also applied visual music in other artistic fields. As a suspected synesthete, he concluded that “a powerful psychological resonator for the listener would be created when a color is perceived with the correct matching sound”, but “not all music was suited for color additions” (van Campen, 2007: pp. 50, 52). Based on his own experiences and theories, Scriabin composed the piano concerto, “Prometheus, Poem of Fire”. In his plan, the musical performance was combined with lighting effects. Instead of blindly pursuing synchronization, the lighting colors only changed when the key musical tones changed (van Campen, 2007: p. 53). Compared to color music, Scriabin gave more consideration to the thematic logic and integrity of artwork, and the sensory experience of the audience.
Wassily Kandinsky, a Russian painter, and the pioneer and representative figure of abstraction, made a significant contribution to the development of visual music. He considered music to be above painting in the hierarchy of art forms, since representational painting still presented the material world. However, indeed, his career as a painter was from representation to abstraction. Kandinsky, who is widely recognized as a synesthetic, studied synesthesia and the relationship between painting and music. In 1909, his play, “The Yellow Sound” (Der Gelbe Klang), attempted to enhance audiovisual synesthesia by alternating dissonant sounds, similar to Scriabin’s ideas (van Campen, 2007: p. 57; Betancourt, 2013: p. 20). The following year, “On the Spiritual in Art” (Über das geistige in der Kunst) systematically documented Kandinsky’s idea of audiovisual synesthesia. He considered that color has both physical and psychological effects. The colors that people visually see are momentary and superficial, but the impression “may be the starting point of a whole chain of related sensations” (Kandinsky & Sadler, 2008: p. 57). As experience accumulates, people build up their perception of color. These physical impressions are the basis of the psychological effects that produce the corresponding mental vibrations. Therefore, Kandinsky affirmed the feasibility of defining “by association the effects of color upon other senses than that of sight”, and described this as, “a phenomenon of transposition of experience from one sense modality to another” (van Campen, 2007: p. 56; Kandinsky & Sadler, 2008: p. 59). However, he was not sure whether the psychological effect was produced directly or through associations between the senses (Kandinsky & Rebay, 1946: p. 41). As the understanding of color and synesthesia deepened, Kandinsky eventually explored his own abstract style, and created his first abstract watercolor painting in 1911. His subsequent series of paintings, “Compositions,” were his “first steps on the way to abstraction” (wassilykandinsky.net, n.d.). For him, the “idea of abstract painting became the ideal existence of synesthetic aesthetics” (Jastrumskytė, 2017). In addition, Kandinsky also encouraged and emphasized synesthetic experience to his students during his tenure at the Bauhaus (van Campen, 2007: p. 56).
Dutch De Stijl painter, Piet Mondrian, experimented with other dimensions of sound to alter the static nature of painting. Mondrian, who is famous for his geometric paintings, actually lacked the visualization of rhythm in his early works. Inspired by jazz, he applied pop music in his paintings, and published “Jazz and Neoplasticism” (De Jazz en de neoplastiek) in 1927 to illustrate the rhythmic artistry of jazz (van Campen, 2007: p. 58; Karlsen, 2001). In subsequent works, he visualized rhythm through eye movement. Although Mondrian was not a synesthetic, he achieved “the synesthesia of movement in visual grids” in a unique way (van Campen, 2007: p. 59).
Modern art, which has been in its infancy since the middle of the 19th century, ushered in the birth of abstraction and visual music in debates on the forms of arts. Color music and visual music together established a form of interconnection between sound and image through a visual medium. Although a number of modern artists considered that music is superior to painting, or the idea of painting is derived from music, in reality, the vision still dominates the sense ratio.
3.4. Audiovisual Installation Art in Postmodern Art
At the same time as the development of modern art, the invention of electricity in the second industrial revolution brought human society into the era of instant communication, where the transmission of information that relied on electric speeds made it possible for human senses to perceive each other with instant consistency. After the Second World War, artists began to reflect on and criticize the so-called “modernity”, resulting in postmodern art, which is distinct from modern art. At the same time, post-war economic recovery provided the material basis for more experimental artistic creations. Cultural institutions, such as museums and art galleries, also made art increasingly commercial. Therefore, although electricity has already been widely used, artists embarked on new experiments with audiovisual art.
In the 1960s, discussions on human identity, physical perception, and sensory interaction attracted various artists, and the body became an important theme in postmodern art (Attarian & Marasy, 2018). Before this, phenomenology developed the concept of body as the subject. French philosopher, Maurice Merleau-Ponty, argued that it is the organs of the body that form an organic whole together (Yang, 2010). Unlike some performance artists who were keen to communicate with the human body in real time, American installation artist, Bruce Nauman, preferred to explore the human body and perceptual experience through a wide range of media forms. Therefore, he exceled in neon sculpture and video installations. In order to create the installation video, “Violin Tuned D.E.A.D.”, Naumann retuned the four strings of the violin, and repeatedly played it with his back to the camera, making the violin in the video different from that in the audience’s common knowledge (Schäfer, 2013; Henricks, 2018). The video screen and loudspeakers built a simple audiovisual environment without placing the audience in a familiar audiovisual experience. Nauman illustrated his ideas through such abstract audiovisual installation videos that “the body is certainly not just one object of perception among others,” but “the point of reference for perception in general” (Schäfer, 2013).
The emphasis on the concepts in postmodern art led artists to abandon traditional linear thinking. Coupled with the blurring of disciplinary boundaries, audiovisual art began to enter a new historical process. Meanwhile, artists noticed the imbalance in sense ratio, and attempted to pursue the balance through technology. However, electrical technology was still unable to meet such demands.
3.5. Technology-Based Audiovisual Art in Contemporary Art
The relatively stable social environment of the post-war period facilitated further technological innovation. Following the revolutions of steam power and electronic technology, the invention of computers in the latter half of the 20th century brought about the Digital Revolution. Similar to the two previous technological revolutions, there was a certain hysteresis in the application of these emerging technologies in the field of art. As a result, at the outset, artists mostly used computers as an auxiliary synthetic technique, and not as a primary creative medium, on a large scale. The maturation of digital technology at the late 20th century not only expanded the scope and form of new media art, but also developed original cinema and television art.
American new media artist, Bill Viola, was passionate about exploring perceptual phenomena, self-awareness and spiritually synesthetic aesthetics through video (Hoberman, Kuspit, & Viola, 1987: p. 9). Influenced by experimental psychology, his understanding of synesthesia was more conceptual and emotional. In addition, the study conducted by Canadian communication theorist, Marshall McLuhan, led Viola to argue that “media is not a vehicle for reproduction,” but “a mechanism for exposing the fundamental correlation of life” (Gelan, 2021). In his video installation, “The Crossing,” in 1996, two giant screens simultaneously played a pair of videos that showed water pouring overhead on one side, and fire burning underfoot on the other side (Young, 2015). The combination of bizarre images and calm sounds provided the audience with a spiritual synesthetic experience. Meanwhile, Viola also used various contemporary media production methods, and placed videos in enclosed spaces to achieve an immersive effect.
The immersive environment based on the concept of space has become an important form of synesthetic experience for artists such as Viola, but this cannot be achieved by cinema and television technologies alone. The impact of rapid technological development since the Digital Revolution, which continues to the present day, has not only reshaped the social framework, but also brought a qualitative leap for audiovisual art. On one hand, as McLuhan argued that the medium is the extension of a human, technology makes it possible to truly balance the sense ratio. In addition, innovations in communication technology have further changed the people’s understanding of senses and perception. On the other hand, contemporary psychology and neurology have laid the scientific and experimental foundations for synesthesia art. As a result, synesthesia is presently no longer just a concept at the sensory level, but has evolved into a medium for processing and connecting information among different senses. The traditional model of communication structured by the medium as a communication-related tool is evolving, with human senses gradually integrating and even becoming the “medium”. The concept of human-computer interaction has grown into a developing trend.
The corresponding art form was also born. Patrice Bouvier and his team (Bouvier et al., 2006) from Université de Marne la Vallée in France systematically summarized the construction of immersive environment based on virtual reality systems, stating that the goal of virtual reality systems is to provide an artificial environment that can be perceived and interacted in real time through sensory and motor interfaces. Italian interdisciplinary artist, Maurizio Martinucci (also known as TeZ), was dedicated to studying synesthetic art-science, and views his work as “the materialization of this study” (Udvardyova, 2016). He attached great importance to the concept of space and its significance for sensory experience. In TeZ’s installation, “Optofonica Capsule” (TeZ, 2008), a capsule-shaped helmet that used special materials, as well as sensors and a special floor, allowed the sound to be transformed into hyper-perceivable vibration directly through the bone system of human body as a medium for information communication, in an attempt to create an immersive audiovisual perceptual space and to achieve the integration of human and technology. Austrian artist, Kurt Henschläger, also applied the concept of synesthesia in an immersive environment. In 2008, he created the immersive audiovisual space “ZEE” (Henschläger, 2008). The abstract sea built by artificial fog, stroboscope, and surround sound blurred the audience’s original perceptual system, establishing a new “mind-scape” through the momentary impressions produced in the space, thereby physiologically influencing the sensory experience. German artists, Martin Backes (2013) and Daniel Franke, developed an audiovisual installation controlled by gestures to achieve interactive effects, where the body became the input, and the hearing and vision were also the medium of processing generated sound and image. Additionally, Romanian artist, Alex Halka (2019), applied a similar thinking of interaction. In his audiovisual installation “Aural Skin”, in order to reveal “a visceral, emotional manner”, body motion was captured by more advanced sensors, which allowed audience to receive more interactive, immersive audiovisual feedback.
The interdisciplinary research of audiovisual art and computers has strengthened international exchanges. The Association for Computing Machinery (ACM) have organized more than 100 kinds of worldwide conferences and workshops on computer-related topics, including the International Multimedia Conference. The 26th ACM Multimedia Conference, which was held in Seoul, Korea, focused on audio-visual scene understanding, exploring related technologies and the possibilities of audio-visual signal processing (Hilton et al., 2018).
Art exhibitions and experiences on audiovisual synesthesia have also been on the rise. In October 2021, five pairs of visual artists and musicians exhibited their collaboration of five music-based visual works in the exhibition “Synesthesia” curated by Wönzimer Gallery (n.d.) in the USA. In June 2022, the art institution Factory Obscura (n.d.) launched SYNESTHESIA, a year-long program in the USA, offering free immersive audiovisual art experience to the public.
4. Discussion
The previous research findings on audiovisual synesthesia art have covered various dimensions of the artistic field, such as the macro art history, and the micro, specific art forms. With the development of the times and the necessity of interdisciplinarity, the research and experiments are not limited to the field of art, and have been linked to other fields including neurology, psychology, and sociology.
Due to the limitations of previous contexts with the developing application of computer technologies and the incomplete change of thought, however, audiovisual art in postmodern and contemporary periods is frequently studied and summarized as an independent finding related to synesthesia, not as part of the history of audiovisual synesthesia art. Additionally, some research only separately involved different dimensions under the same research project of audiovisual synesthesia. Therefore, in the new research context of human-computer interaction, it is necessary to truly establish a systematic and comprehensive audiovisual synesthesia framework of both theory and practice involving the disciplines of art, philosophy, psychology, neurology, media studies, and other dimensions. This would not only provide a feasible methodology for the study of other types of synesthesia, but also open up a new path for the field of human-computer interaction, in which human senses are used as the medium for processing information.
5. Conclusion
Audiovisual synesthesia and related artistic experiments have gone through different periods, from the preliminary exploration of tone-color and the debate on the forms of arts, to the pursuit of mechanical synchronization of music color and the emphasis on immaterial visual music, and to audiovisual art based on new media and computer technology. This process has gradually balanced the emphasis on visual effects in the arts. At the same time, audiovisual synesthesia has begun to evolve beyond the initial sensory concept, into a way of processing and interacting with audiovisual information.
The development of Western audiovisual synesthesia art driven by social development, technological change and ideological innovation is demonstrated in this review. Within the context of the times, each period of audiovisual synesthesia art has echoed relevant changes in other fields on an aesthetic level, while at the same time generating a new wave of thought through the unique means of expression of art. The new context will deepen future research on audiovisual synesthesia, and its relationship with the social, technological and ideological fields will be further refined. Art will subsequently have the potential to reshape the imbalanced sense ratio, and offer a truly perceptible aesthetic experience. Based on the review results, the following research direction will be to conduct interactive art experiments, in an attempt to build the interdisciplinary framework of audiovisual synesthesia.
Acknowledgements
This study was supported in part by Beijing Normal University-Hong Kong Baptist University United International College.
NOTES
*First author.
#Corresponding author.