Science-Art Installation Experiment “Melodic Pigments: Exploring New Synesthesia”

Science-Art Installation Experiment by Yasmin Vega (Tokyo University of the Arts) and Masahiko Hara (Tokyo Institute of Technology/Institute of Science Tokyo)

Synesthesia is a perceptual phenomenon in which the stimulation of one sensory or cognitive pathway leads to involuntary experiences in another sensory or cognitive pathway. An example of synesthesia is experiencing “colors” in response to sensory information. This phenomenon can manifest as perceiving and feeling “colors” when hearing certain sounds, reading letters, or even tasting or smelling something.

In this installation, we aim to train a machine using machine learning based on databases that map the correlation between emotions and colors in response to certain sounds. The machine will then demonstrate the colors it imagines and feels when it hears a sound, thereby expressing its own form of new synesthesia in machines.

Through these explorations, we seek to deepen our understanding of synesthesia in machines and its emergent functions, moving beyond mere sympathy or empathy between humans and machines. Additionally, we aim to explore methodologies for visualizing the internal cognitive processes of machines.

If you would like to attend, please register with events@khk.rwth-aachen.de.

—

Credits header photo: Masahiko Hara