With the growing interest in such technologies, what should you know before starting your first mixed reality project?

From April 2021 – February 2022, Tusitala worked on prototyping an interactive installation as part of the National Arts Council’s Arts x Tech Lab 2021. We used Microsoft HoloLens 2, projection mapping, and immersive audio to present scenes from Singapore’s history in a panoramic format. 

Here’s a handy list of 8 tips from what we’ve learnt with the insights from our project:

1) Immersive technology ≠ an immersive experience

A successful XR arts project should evoke a sense of wonder, and make the participant feel like they have been transported to another world. Focus on the atmosphere you want to create and the emotions it should evoke.

This requires attention to both details as well as the big picture – it helps to step back every now and then to consider how a project instinctually feels to the audience.

2) Does your project really need MR? 

MR is a newer technology, which means there are fewer developers and tools available, the headsets are more expensive than VR, and the process is more complex. That being said, MR allows for heightened immersion, because users can physically engage with both the real world and the digital layer simultaneously.

If your experience benefits from a first-person perspective and having users move around a physical space, then MR might be right for you.

3) Create a shared understanding of XR 

Extended Reality or XR productions require close collaboration among people who have different areas of expertise, and the medium itself is still in its infancy.

As such, when starting a project it is critical to communicate the challenges and possibilities of the technology to all team members. Doing so sets the project up for success by clearly establishing a project’s goals.

Nothing beats first-hand experience: have everyone on your team try out different apps on the headsets, before a project begins. 

4) Visualise well to work effectively

Keep discussions concrete using visuals and samples, whether they be sketches, reference images and videos, diagrams, or audio clips.

Within cross-disciplinary teams, clear communication is needed to be able to fully leverage on the creativity of all members as well as efficiently plan, adapt, and integrate workflows. 

Working notes on installation layers

5) Prioritise the physical user experience

MR allows interactions between the digital and physical worlds through head-, eye- and hand-tracking, as well as improved simulation of real-world physics and depth perception.

Creators must thus consider the physicality of the experience: how to guide the user’s eye, where he or she will be facing, their path through the mapped space, pacing, and more.

Details such as the scaling of objects and specific hand gestures all matter.

Well-designed projects balance the need to guide users with giving them enough freedom and control to explore at their own pace. 

Collaboration with partners for the Arts x Tech Lab 2021 incubation. Photo taken at the d&b audiotechnik lab.

6) Level up with sound

Sound plays a key role in creating an emotionally resonant experience and transporting the user to a fictional world, which is critical to produce something truly immersive.

Spatial sound on MR headsets like the HoloLens 2 allows for multi-directionality and thus lends a heightened sense of realism.

External systems by immersive audio innovators such as d&b audiotechnik, Klang, and Meyer can also be used alongside MR to shape the acoustic environment in more nuanced ways. 

7) Iterate and test, repeatedly

Early testing helps creators check blindspots that they might miss in storyboards during the planning process, which can also include the impact of technical limitations like the HoloLens’ Field of View or environmental lighting.

Even testing of low-fidelity physical prototypes can help, though what, how, and why to test depends on the project.

Test frequently and with as many different audience profiles as possible, to refine your project’s user experience by considering factors such as a person’s age, comfort level with tech, and even physical build.

8) Don’t forget practical concerns

MR devices are expensive, with headsets like Microsoft HoloLens 2 and Varjo XR-3 priced between S$5,000 to $8,000 per units.

This places limits on a project’s audience reach, and also requires considering logistical factors such as the charged time needed for MR devices, visitor flow through a space, as well as manpower required for facilitation, device maintenance and hygiene.

MR projects require significant resources that might be barriers to adoption, but this might be solved if the project is backed by big sponsors!

---

Related Blog Posts

Mixed Reality – Blurring the lines between Augmented and Virtual