Is Artificial Intelligence (AI) capable of modeling and replicating human sensory associations? This question was explored at the KHK c:o/re on April 28, 2025, as part of the science-art installation experiment “Melodic Pigments: Exploring New Synesthesia” created by the Japanese media artist Yasmin Vega (Tokyo University of the Arts) in cooperation with KHK c:o/re alumni fellow Masahiko Hara (Institute of Science Tokyo).

The goal of the installation experiment is to explore the relationship between sound and color through the phenomenon of synesthesia, in which one sensory perception involuntarily triggers another. The primary focus is on chromesthesia, a form of auditory synesthesia in which sounds evoke the perception of colors.

During the performance, Yasmin Vega played electronic music using live programming, while an AI model trained on sound-color associations predicted and visualized corresponding colors in real time. The AI model used in this project was trained on subjective data reflecting Yasmin Vega’s personal associations between sound and color. Throughout the performance, the AI processed incoming sounds every three seconds and determined the corresponding colors based on the pre-trained data, thereby creating a fluid interplay between auditory and visual elements. The colors chosen by the AI were shown on a series of hanging light tubes and as morphing pattern on a computer screen.

The installation experiment seeks to explore the AI’s ability to model and replicate human sensory associations. The focus is on exploring internal visualization processes and the sensory capabilities of machines. Yasmin Vega’s performance demonstrated that the AI was able to recognize and replicate human perceptions while generating color-sound associations that matched the artist’s expectations.

This installation experiment builds on previous works on the integration of artistic strategies into science and technology that KHK c:o/re alumni fellow Masahiko Hara has engaged in during his fellowship. In January 2024, he presented the art installation “Unfelt Threshold” developed in collaboration with the artist Aoi Suwa that explored the perceptual capabilities of machines in response to unpredictable visual stimuli. Through these experiments, Masahiko Hara aims to open up new perspectives at the intersection of materials science and nanotechnology within scientific engineering. The interplay between science and artistic practice reflects a central research interest of the KHK c:o/re, which investigates how artistic methods can contribute to epistemic questions within an expanded framework of science and technology through performances such as “Melodic Pigments”.

Interview with Yasmin Vega

How do you perceive sound and color?

When I listen to a high-pitched sound, I imagine yellow. When I listen to a dark sound, like dark bass, I also imagine a dark color, like dark green or like deep purple. The volume of the sound also changes how I perceive it. If I listen to a loud sound, I imagine red. And even if I listen to the same melody, if it’s played with different instruments, I also imagine a different color. 

What surprised you most about the visualizations of the AI?

What surprised me the most was how often the AI’s visualization matched the color that I actually imagine when I’m performing. And it almost felt like the AI could understand my personal sense of color. From this whole performance, I realized that how useful it can be to work with AI when it comes to expressing something that’s really personal and hard to explain to others or share with others. 

What do you think about AI and art working together? Where do you see challenges?

I wanted to use the AI just as a tool in my artworks. I felt that if I collected the training data by myself and developed the model by myself, then using AI is just like using the tool. So, the final artwork was really my work. And I didn’t feel like just writing the prompt and letting the AI to generate the image is really artwork. That’s why I originally wanted to control the AI as much as possible, but now I feel a little bit more relaxed about it. Now I’m looking for the unpredictable results that come when I can’t freely control the AI. I can say that I’m not so cautious anymore.

One challenge I see is the amount of the data. This time I only used 300 samples to train the AI model. There are so many sounds in the world, so it’s basically impossible to cover all of it. But improving the model’s accuracy doesn’t automatically mean that the artwork itself gets better. So, I think the creative value comes from something more than just how smart or how accurate the AI is.

Photos and video by Jana Hambitzer

ALIN OLTEANU AND THE C:O/RE TEAM

The Käte Hamburger Kolleg: Cultures of Research (c:o/re) celebrated itself, as it completed the first 4-year cycle of funding and is now successfully entering a second funding cycle. The center is funded by the German Federal Ministry of Education and Research (BMBF) within its framework program for the humanities and social sciences “Shaping the Future”. On March 25-27, 2025, we were delighted to get together for a conference targeted on the specific but encompassing theme of this center, namely Cultures of Research, which, we dare say, has recently become a more prominent academic topic due to the center’s efforts.

Who are we? All of us – c:o/re team members and fellows, both current and alumni, with a scientific advisory board that has steered the center’s activities. Almost all c:o/re fellows, who have carried out research here over four years, were present. This enabled a fascinating, for us, intersectional and inter-paradigmatic academic dialogue, the kind that makes the object of Cultures of Research. Chaired by the c:o/re team, fellows and scientific advisory board members have presented their research in approximately 40 talks. It was a most enjoyable opportunity for us to discuss, in hindsight, what emerged from four years of sustained academic work, having started from scratch, and how we see the center evolving in the future.

Alin Olteanu

Alin Olteanu is an Associate Professor of Semiotics at Shanghai International Studies University. Until July 2024, he worked as a a postdoctoral researcher and publications coordinator at the KHK c:o/re.

Many of us, team members and alumni fellows, deem the conference not just useful, but necessary. c:o/re has become an important dimension in the work of several of us, intellectually and institutionally. As such, gathering altogether is as important as the regular meeting of many themed academic associations. c:o/re has opened new career opportunities and perspectives for several of us. The center was formative and instrumental in the professional development of many, not just fostering the next step on a linear trajectory, such as from postdoc to tenure, but also enabling shifts in research focus, such as from engineering to science and technology studies. A small minority of alumni fellows has even found long-term academic placement at RWTH Aachen University. Even for such colleagues, who never fully left the center, the conference was needed, to reconnect with others. Many remark that it was particularly interesting to have the chance to dialog with the scientific advisory board in a collective, transparent and friendly setting.

c:o/re directors Professors Gabriele Gramelsberger and Stefan Böschen started off the conference, welcoming what was a heterogenous but familiar gathering. They shared their views on the first four years of this center, the main research topics that channel its work and how these evolved. This ushered in the first keynote, “Historicizing Epistemology” by Hans-Jörg Rheinberger, a fitting way to start off a Cultures of Research conference, setting the optics for further conversation. 

The conference was structured thematically in eight panels under three main c:o/re study foci, as follows. To address the theme of Change of research practices, we organized the panels Dealing with Complexity and Digitalization of Science. The theme Organizational transformations in science was addressed through panels on Lifelikeness, “Expanded STS” & Euregio, Freedom of Research, Art and Research. The Historical and intercultural comparison of varieties of science was organized into the panels Historicizing Science and Varieties of Science. This thematic organization results from a dialectics that is both top-down and bottom-up, to follow the research center’s rationale and mission, which have been channeled, in time, through the research it produced, one step at a time.

Being part of the c:o/re team, we feel privileged to be in a position to listen to the various studies that have emerged from this research center, observing how they have shaped the center and how its entailed research topics have changed over time. To illustrate, for someone who has been a part of this four-year effort throughout, it was fascinating to listen to dialogues among the scientific advisory board with and across four generations of fellows, who seldomly knew each other. This was not just a meeting of individual scholars, but of academic groups that have crystallized during their respective fellowships, having each developed their research subculture. In this exercise, we saw first-hand the importance of institutional academic funding structured in this Käte Hamburger Kolleg format. Until now, we have worked with these scholars individually and in well-focused formats, such as thematically organized fellow cohorts.

Our festive conference opened the doors to intersectional dialogue, releasing the, however interdisciplinary, strictly focused work of individuals and clusters within c:o/re into a productive and creative chaos. As some fellows attest, while at first glimpse the range of topics brought together under the roof of the center, as seen in this conference, may seem unrelated, they epistemologically connect very well. It is such facilitating of interdisciplinary research that positioned some fellows to discover that the issues they tackle are of interest beyond the disciplinary confines within which they each operate.

We see c:o/re having enabled new and unexpected quo vadis reflections on Cultures of Research, something we can observe regarding the topic of “Expanded STS”, a c:o/re coinage that is drawing growing attention, as an anticipating consideration on scientific and technological futures. Actually, we contend that the conference panel dedicated to Expanded STS demonstrated how much STS is shaped by ‘othering’ and internal demarcation between disciplines (especially the sociology and philosophy of science). However, at the same time, our conversations reveal not only that a multitude of approaches co-exist, dealing with these boundaries differently and more productively, but also that a growing scholarly community is willing to explore new interdisciplinary avenues for cooperation.

We do not want to give the wrong impression that the research carried out at c:o/re is free of contradicting or even controversies – far from it. The conference has seen plenty of contradictory arguments and contestations among the speakers, in a way that accounts for two important matters for any research institute, namely that (1) this center is a platform for free academic debate and that (2) the approaches it hosts are epistemically compatible (that two positions on a topic are contradictory implies that they are mutually relevant). Actually, the one claim on which we found total agreement is that Freedom of Research is currently one of the most important issues for the academe, as well as society broadly. All fellows, team and scientific advisory board members see the urgent need of freely (!) discussing the freedom of scholars in the current context when sociotechnical shifts have consequences for the freedom of speech and expression.

Of course, discussion on what freedom in research is, how it is practiced and how it should be supported institutionally was fiery, encompassing a broad variety of perspectives. Overall, there is agreement that this is how an exercise in academic freedom looks like: we are free and enabled institutionally to contradict each other. We note that the Cultures of Research conference took place shortly after a new US administration started exercising pressure on scientists and universities. Political pressure on academia will undoubtedly constitute a main concern for c:o/re in its second cycle of funding, shaping its future development, as we hope and anticipate that it will shape the future development of philosophical and social inquiry on technology in general.

Unless otherwise noted, photos by Christian van’t Hoen.

The program with all speakers and titles of the conference can be found in this document.

STEFAN BÖSCHEN, MASAHIKO HARA, PETER MANTELLO AND ALIN OLTEANU

The Käte Hamburger Kolleg: Cultures of Research (c:o/re) is a partner on the project Emotional AI in the Japanese and German Workplace: Exploring Cultural Diversity in AI Ethics, led by Professor Peter Mantello of Ritsumeikan Asia Pacific University (APU) and funded by the Japan Society for the Promotion of Science. The project has been ongoing for about a year, and in January 2025, c:o/re director Stefan Böschen and Alin Olteanu visited Professor Mantello and conducted a first field study in Japan for this project.

As the title claims, this project examines the developments in Emotional AI from a comparative perspective between Japan and Germany. Such research is, of course, conducted in the context of increasingly rapid developments in the field of artificial intelligence (AI). So far, these developments have occurred in waves. Phases of great innovative momentum alternated with phases in which this topic appeared dormant. The recent technological platform of large language models such as ChatGPT suggests that this field is now entering a phase of lasting and disruptive development. The great geopolitical competition between world regions is exemplified in the sudden appearance of China’s DeepSeek. These developments raise questions about the specific technological development under the respective cultural-institutional framework conditions. With its AI Act, the European Union has issued the strictest, risk-based regulation of AI, striking a balance between protection against technology and industrial development, while the United States and Japan appear, at least at the present moment, reluctant to lay down a concrete regulatory policy, preferring instead a market state approach.

Our project offers the opportunity to reflect not only on the specific challenges for science studies, but importantly, in the context of the increasingly quantified workplace, where insights can be fed into the varieties of science discussions pioneered at c:o/re. The following itinerary documents the journey of c:o/re director Stefan Böschen and former team member Alin Olteanu through Japan, together with Principal Investigator Peter Mantello, along with Co-Investigator Hiroshi Miyashita of Chuo University, and the local assistance of Masahiko Hara, an alumni fellow of c:o/re who graciously gave us his time to accompany us from Tokyo to Beppu. The following blog post outlines some of the places and people we met along the way, as well as revealing insights we achieved on this whirlwind one week journey.

Tokyo, Monday, January 13^th – AI Workshop  

Tokyo, Tuesday, January 14^th – German Institute for Japanese Studies and TeamLabs

In the morning of the 14^th of January, we visited the German Institute for Japanese Studies (DIJ) and held discussions with the Deputy Director of Sociology, Barbara Holhus, exploring the possibility of various short and longer-term collaboration avenues between DIJ, c:o/re and other Japanese universities. Having identified various research avenues of common interest, we agreed to meet regularly in the future. The trio of DIJ-c:o/re-APU brings a set of complementary competences to collaborate on the comparative study of research cultures and their technological evolution.

Kyoto, Wednesday, January 15^th – Kyoto University’s Disaster Prevention Research Institute and School of Informatics

On the 15^th of January, we ventured to Kyoto University to meet researchers at the Disaster Prevention Research Institute (DPRI). Here we learned from various faculty members of the institute about the latest of current research developments in natural hazard reduction and integrated strategies utilizing the latest modeling software for disaster loss reduction. We were certainly surprised at the institute’s multidisciplinary embrace of artistic interventions in this regard, as a presentation by Naoko Tosa, a resident artist at DPRI, described her fascinating and thought-provoking designs in clothing fabric which weave digital sensors and actuators that are activated by cellular emergency warnings. Afterwards, we were invited to visit her studio, where we had a chance to get a closer look at the technical and conceptual aspects of her creative process in designing ‘disaster couture’. 

The day concluded with a visit to the Department of Informatics at Kyoto University, where we met Professor Jaward Haqbeen, an Afghan AI researcher whose work focuses on the use of generative AI in language acquisition in developing countries such as Afghanistan and Nepal. We have discussed plans for future collaboration, particularly on matters of education, language learning and technological literacies.

Osaka, Thursday, January 16^th – NTT’s Brain Lab Osaka University

Beppu, Friday, January 17^th – Ritsumeikan Asia Pacific University

On Friday the 17^th, the team along with Professor Masahiko Hara (Institute of Science, Tokyo) gave talks at Professor Mantello’s home institution, Ritsumeikan Asia Pacific University in Beppu, located on the east coast of Kyushu Island in the south of Japan. Targeting a primarily undergraduate audience, Stefan Böschen gave an informative lecture on the importance of science and technology studies. Alin Olteanu engaged students with a talk on the semiotics of Digital Nomadism and Masahiko Hara on his experimental artistic interventions into intelligent interfaces that read human emotions.

Outlook

Like most countries, Japan and Germany share fundamental values such as freedom, democracy, and the rule of law. They also agree that accountability, transparency, human rights and privacy should be built into AI. Yet where the EU/Germany wants a top-down government-led approach to mitigate AI harms, Japan (at least at the moment) prefers an industry/sector approach to give the technology a good chance to grow. At the government level, the annual Japan/Germany ICT Policy dialog forum promotes the need for common rules on AI. Importantly, Japanese labor law is influenced by the German legal contexts. But while German law on AI in the workplace is becoming increasingly precise and restrictive, current Japanese law is vague and ambiguous. Thus, it is important to refer to the development of both EU and German law for comparisons of Emotional AI in the workplace. As Co-Investigator Hiroshi Miyashita argues, “Japan is well-known for importing foreign laws, but a patchwork of copy/paste does not work well in a Japan”.

Concomitantly, our collaborative efforts to date suggest that Japan could offer a third way by heuristically exploring the space of AI development that seeks to create harmonious human-machine relationships, with a focus on AI that preserves human dignity. Keeping this in mind, the journey goes on, and we look forward to further opportunities to collaborate with the project of Professor Mantello, as well as with the DIJ and other stakeholders from both the private and public sector. It is interesting to note that the Japanese government is fueling the evolutionary dynamic in the field by creating far-reaching exchange opportunities for incoming researchers.

Photo Credits: Peter Mantello