The Arts x Tech Lab 2021 is the National Arts Council (NAC)’s inaugural innovation lab that fosters collaboration between the arts and technology sectors. The Lab is designed to empower creatives, artists and technologists to engage in innovative experiments and collaborations. It also aims to build capabilities, co-drive innovation and support changemakers from within the arts and technology sectors.

Concept

What might a digital experience of an illustrated story look like, using immersive technologies?

To explore this question, we prototyped an interactive installation that uses Microsoft HoloLens 2, projection mapping, and immersive audio to present scenes from Singapore’s history in a panoramic format.

By considering how extended reality technology can be used to present multiple narratives against a single backdrop, we aim to devise new ways of storytelling that help cultural institutions make better use of unusual, underused spaces (e.g. corridors, stairwells, areas with curved or slanted walls) and transform these into sites for imaginative, reflective encounters.

Process

Inspired by panoramic graphic novels and immersive art exhibitions in museums, Tusitala set out to create a storytelling experience that brought XR headsets together with the concept of multiple layers in history. We gathered the team, comprising technology partners (Helloholo and d&b audiotechnik) and artists (illustrator Xiao Yan, writer Ng Yi-Sheng, and sound designer Ning).

After we presented the idea of having a large mural that can host multiple stories, writer Ng Yi-Sheng proposed the Southeast Asian concept of three worlds – sky, earth and sea – to present Singapore’s urban legends and folk histories. The team decided to focus on the story of Colonel William Farquhar ascending Bukit Larangan, which was said to be haunted. The poetic monologue form, comprising five stanzas, set the structure and flow for the illustrations and user experience. [Read the poem here] 

Xiao Yan created a storyboard, and the Tusitala team broke it down into different layers, including the mural, the Hololens characters and then atmospheric effects, the projection mapping, and then the narration. This helped us gain clarity on the sequence of the experience, and what each team member would have to do.

Ning selected an actor (Matthew Koenig) who could do a Scottish accent, to read out the monologue as Farquhar. This narration is played through the Hololens, and syncs with the immersive Soundscape design which visitors can hear in the environment.

Once the art was ready, it was developed by HelloHolo into a prototype. All the different elements, including projection mapping using Lightform, were brought together at the d&b Soundscape lab, where we did usability tests and tech runs. The prototype continues to be refined for the upcoming showcase in mid-February.

Project Team

• Tusitala
• HelloHolo (XR tech partner)
• d&b audiotechnik (audio tech partner)
• Ng Xiao Yan (illustrator)
• Ng Yi-Sheng (writer)
• Guo Ningru (sound designer)

Narration: Matthew Koenig

Prototype walkthrough

Watch this on YouTube

Showcase

“The Colonel and the Hantu”
15 – 18 February 2022
National Museum of Singapore, Gallery Theatre, B1
10am – 7pm | Free Admission
(Note: Tusitala booth opens only at 2-4pm, 5-7pm)

View Story (Download PDF Brochure) 

#ArtsxTechLab

BLOG

8 things to note before starting your first mixed reality arts project