Here’s inviting you to step into the emotional world of poetry, to co-create and render it with physical, visual, and aural elements of self-expression. Find another player, and see how your individual choices affect not only the overall mood and interpretation of the poem but also the final choices made by the personas in the poem. This innovative poetry reading format hopes to make Chinese poetry more accessible, encourages active engagement with the process of interpretation, and sparks conversation about the themes of the selected poem. 

A collaboration between Tusitala and Matter.Less.

• Internal prototype
• 9 March 2024
• Web development, user interface design, voice AI

Concept

This is a work-in-progress where two people read a poem together using the familiar karaoke user interface. An extension of an earlier project that combines sound with poetry (see “Chinatown”), the objective of this project is to explore different ways of reading Chinese literary works. 

We adapted the short novel “白鸟” (White Bird) by the Singaporean poet and novelist 英培安 (Yeng Pway Ngon), who received the Cultural Medallion in 2003, into a spoken word poem. The poem has a slight fantastical element, features a natural dialogue between two characters, and explores interesting themes of escape and joblessness.

Process

During the concept phase, we discussed questions such as the optimal number of participants, the intended takeaways for the audience, whether to prioritize interpretation or themes, and the degree to which we should structure the experience. 

The user experience research phase focused on various aspects of role-playing. For example, an equal workload distribution between participants, how to slow down the process for deeper engagement, and ways to enhance interpretation through elements like actions and music.

After months of discussing the concept, poem selection, and objectives, the user interface design began in November, and web development started in January. 

For the web app, audio chunks are transmitted in real-time from the microphone to Google’s Speech-to-Text API via WebSocket. The interactive background is generated using Three.js and GLSL shader. Additionally, the questions phase of the project utilizes WebSocket directly, bypassing the need for a database.

We tested it with close to 10 pairs at an event, and came away with these “how might we”s:

1. HMW help users understand the poem better?
2. HMW guide the users better so that they feel less confused when navigating through the prototype?
3. HMW engage users better / make prototype feel less repetitive?

Next Steps

As we continue to refine the code, we are exploring what is most interesting to us. On top of participants “reading” the poem, do we want to incorporate an additional element of the machine “reading” us back (i.e., using AI to deduce participants’ mood, age, etc. via vocal quality, etc.)? Or would it be better to focus on the analysis aspect: how do two people who experience the same poem together still take away different interpretations? 

The project can be expanded by using different poems or languages, incorporating more performative elements, and/or integrating voice AI, hand gestures, etc. We hope to pitch this project to programming venues and/or festivals, especially those interested in Chinese language projects and/or poetry.

Project Team

Tusitala is a Singapore-based digital storytelling studio producing creative projects that reimagine the future of reading where art meets tech.

Matter.Less is a multidisciplinary arts collective and incubation space where young artists with different quirks and diverse creative practices experiment, collide and synthesise.