In our XR exploration we’ve featured Tusitala’s past AR and VR projects (Part 1: “Getting into the Mix of A/V/MR!”) and taken a quick look at some XR projects in Singapore last year (Part 2: “A quick review of local XR projects).

With the growing interest in such technologies, what’s next for Tusitala?

We’re looking forward to launching our Mixed Reality (MR) project, which was conceptualised as part of the National Arts Council’s Arts x Tech Lab 2021. 

The Arts x Tech Lab 2021 is the National Arts Council (NAC)’s inaugural innovation lab that fosters collaboration between the arts and technology sectors. The Lab is designed to empower creatives, artists and technologists to engage in innovative experiments and collaborations. It also aims to build capabilities, co-drive innovation and support changemakers from within the arts and technology sectors.

As one of the Lab’s ten teams, Tusitala worked on prototyping an interactive installation that uses Microsoft HoloLens 2, projection mapping, and immersive audio to present scenes from Singapore’s history in a panoramic format. The idea is to present multiple narratives against a single backdrop, using extended reality (XR) technology. 

We believe that future art and digital projects will gradually incorporate more extended realities, blending physical and virtual worlds in more seamless ways.

Dabbling with Microsoft’s HoloLens 2 

Firstly, what is the HoloLens 2? It is a pair of mixed reality smart-glasses developed and manufactured by Microsoft, which uses multiple sensors, advanced optics, and holographic processing. In simple terms, it is like AR on your glasses.

In collaboration with tech partners HelloHolo (part of Microsoft’s Mixed Reality Partner Program) and d&b audiotechnik, the Tusitala team began conceptualizing the project in April 2021 and gathered collaborators such as Ng Yi-Sheng (writer), Xiao Yan (illustrator) and Ning (sound designer) in Sep 2021 to start work on a prototype.

It was an interesting challenge for all of us – though various extended technologies are increasingly part of museums’ exhibitions and education efforts worldwide, MR in particular is quite recent and has not been tried much among Singapore’s arts scene. 

“We have seen various museums around the world applying such technologies in their exhibitions. Thus this provides our team an opportunity to showcase our technology in the local arts scene. Working with the illustrators allows us to better understand the requirements and possible ways of implementation in Singapore’s museums.” – Kai Yi, our project partner from HelloHolo.

Tusitala’s Publisher, Christine Chong further adds,

“Our project aims to consider how literary works can be presented in ways that hold more mass appeal, and to reshape the act of reading as a social experience – one that allows both shared enjoyment and dialogue.”

How the idea came about

This concept grew out of our interest in several different things, especially graphic novels and historical panoramas. 

Graphic novels are probably the most accessible and widely consumed form of literary arts, given their visual appeal, and they have long been used to communicate complex histories.

We were quite inspired by comic artist Joe Sacco’s The Great War, an accordion book that depicts a WWI battle in an over 7 metre long panorama, as well as the interactive comic The Ocean is Broken by artist Sutu. Together with MR art installations such as The Unreal Garden (LBX Immersive), these got us thinking about how MR’s unique capabilities could be used in the creation of new works. 

The National Museum of Singapore (NMS) is one of NAC’s partners for the Arts x Tech Lab, and one of the challenge statements NAC invited prospective applicants to respond to was: “How can new digital media and technology be employed to blend artistic experiences and heritage?”

That distinction between ‘artistic’ and ‘heritage’ was interesting to us. We wondered why someone would happily go to an art museum but hesitate to visit a history museum, for instance. Most people would probably agree that history can be entertaining, but of course typical history exhibits aren’t always seen that way – it depends on how the stories are told.

Collaboration with partners for the Arts x Tech Lab 2021 incubation.

Pushing the XR Immersive Experience

As freelance producer Tim Powell said at the MuseumNext XR Summit’s July 2021 conference,  XR technologies (which is the umbrella term for Augmented, Virtual and Mixed Reality interactions) provide us an immersive experience that surrounds a person, making them feel part of an alternative environment. It uses immersive technology which is those that extend or replace reality for the user, and these include AR and VR, which could be used to create real objects in a virtual world, or virtual objects in a real world.

“The key strength of MR is that you can interact with both the physical and virtual world at the same time, and in far more sophisticated ways than AR. Head, eye, and hand tracking (with all ten fingers recognised) are possible on HoloLens 2, and the device also allows voice commands. Users can thus experience a story through more natural movements and interactions. With history content, in particular, this heightened immersion could bridge the psychological distance people often feel between them and events of the past,” – Evelyn Ang, project manager.

Through this experimental process, it is our hope that more local arts groups, museums and organisations would be able to tell their stories more effectively by creating experiences that lead to a very personal experience, unique to each visitor.

“If someone would like to commission a similar project that’s focused on just a specific region of Singapore, that would be fun to work on,” – writer Ng Yi-Sheng.

From L-R: Our partners, Kai Yi from HelloHolo (XR tech partner), Ning (sound designer), Yi-Sheng (writer), and Xiao Yan (illustrator).

We hope this project can pave the way for cultural institutions to make better use of unusual, underused spaces (e.g. corridors, stairwells, areas with curved or slanted walls) and transform these into sites for imaginative, reflective encounters.

Of course, the technology arguably has weaknesses that creators have to creatively use or counter through design. MR and VR experiences are often thought of as more solitary than social, for example – you get to interact with the content, but not other people. 

Can technology enable communal experiences, though? Perhaps that’s another question for another experiment for the future.

---

Catch Tusitala’s upcoming MR prototype installation as part of the Arts x Tech Lab Showcase from 15 – 18 Feb 2022.

 “The Colonel and the Hantu”
15 – 18 February 2022
National Museum of Singapore, Gallery Theatre, B1
10am – 7pm | Free Admission
(Note: Tusitala booth opens only at 2-4pm, 5-7pm)

More info

#ArtsxTechLab

---

Read Part 1: Getting into the Mix of A/V/MR!

Read Part 2: A Quick View of Local XR Projects