Spatial Journalism - Part I

Having produced VR films for Red Bull during my time there, I'm pretty excited at the prospect of delving back into spatial storytelling. The accompanying shift from 8-camera stitching to single or double lens cameras (or even better the revolutionary NERF) will empower journalists to seamlessly capture environments on the opposite side of the globe with publishers distributing them globally within XR articles as expandable scenes - it's nothing short of mind-blowing.

In my time producing 360 films I've found invaluable insights from sources like the Oculus Story Studio blog, unfortunately it was discontinued and nowadays I highly recommend delving into the NYT R&D articles delving into the frontiers of spatial journalism.

I'm assuming that people interested in reading will continue to use tablets as the easiest medium to consume text, within the headset there is an intention to experience events rather than read them. Each platform remains useful, but with a different context and audience expectation.

This is the first of a series of articles and interviews I'll be publishing as a part of my research on the implications of Vision Pro for visual storytelling.

Platforms

I have no hands-on experience with the Vision Pro, but from my limited research on Apple's documentation and past experience using/producing for VR headsets and IOS, here's my summary take on current platform scenarios:

• Audiences continue to browse publisher websites within Safari on Vision Pro, articles are moderately adapted to WebXR to have expandable scenes or retrievable objects (Apple Vision Pro Will Support WebXR for Immersive Web Experiences).
• Owned apps from larger publishers or are adapted from iOS to Vision Pro to include additional features for XR content consumption
• Smaller publishers use white-label platform technology to build their owned apps. Using integrations with tools like Unreal Engine or 8th Wall to simplify the content production pipeline.
• All publishers are likely to distribute stories on an XR adapted social media platform (Youtube XR, Instagram XR).

Still figuring out how to prepare for the transition, but for now I’ll be looking at the crossovers from WebGL to WebXR and experimenting with Unreal and 8th Wall to enhance a few stories for webXR.

Storytelling opportunities

Just a few that I’m picking up on that particularly interest me, happy to discuss further with anyone who is interested.

Augmenting context: enriched data visualization and immersive 3D analogies.

Revisiting from Matt Conlen's research paper on cinematic visualization, imagine all of these techniques made available in a 3D space. This scale analogy could be made with us standing closer to the action much more powerfully than on a flat screen, as could videos of in-situ narrators explaining complex events. These scenes will help us grasp the magnitude of events or data not only in physical installations.

Location-based news

Imagine receiving news tailored to your surroundings as you explore the world. Although the Vision Pro seems designed for use at home or at the office, I doubt XR storytelling will be confined to a static environment (perhaps a Vision Air?) and will quickly become responsive to the world around us. As we walk through a city, the XR device presents location-based news that adds historical or journalistic context to our exploration (like a city tour on steroids).

Immersive sound design

We all agree that sound plays a pivotal role in creating truly immersive experiences so I won't advocate on its behalf. I am however particularly interested in audio as an increasingly important interface, and the use of audio cues to guide viewer attention and emotions. Even more exciting is the prospect of combining voiceover with augmented visuals (a.k.a augmented podcasts) to create stories that don't fully disrupt our reality but rather augment it while commuting or walking the dog.

Transporting readers to other locations

This one is the most understood and commonly used in VR experiences, but worth mentioning as I am very excited to see it used for augmented journalism along with LIDAR captured models of scenes or NERFs. Could we expand a nuclear reactor to understand how it functions? Or be transported to the middle Beijing during a flood?

Interaction

A couple thoughts on interaction in the new medium.

Goodby scrollytelling?

I'd venture that traditional "scrollytelling" might be replaced by a more intuitive left and right gesture. Could this reinforce the use of cinematic scene-by-scene techniques, fading in and out of scenes? Or comic-style boxes that expand on gaze intent? I’d also expect behaviours on visual stories to transition from mostly passive observation to more dynamic exploration.

Evolving maps, charts and UI

This one is more obvious, but also has larger consequences. I doubt that 2d maps and charts will be of interest in a 3d space, hopefully Mapbox and other platforms are preparing for that transition to immersive open-source maps and we’ll need to look at new ways of communicating data (or making our loving time-tested charts more interesting in an immersive space) - it's all very exciting.

Dynamic user mechanics

Keeping the best for last! We’ll have many new ways to play with story materials and make them more interesting, more accessible, and more effective. The narrative can extend beyond passive observation to active engagement without the friction of a 2d browser, for example:

• Object manipulation: readers can pull objects out of the article to magnify and interact with them, unlocking hidden layers of information. Let's imagine a COVID particle examination, or a new species discovery.
• Familiar mechanics: We can replicate real-life actions that readers would normally perform to make the narrative even more authentic, for example asking them to move or find things, look over and under, turn a light on or off…
• Escaping the article: I'd like to see unconstrained exploration at the end of XR stories, letting readers break free from the confines of the author guided experience to take a closer look at details and form a connection, an opinion.

The world of XR storytelling is uncharted territory where creativity and technology are converging more than ever (across so many layers of innovation) to redefine storytelling and journalism. It'll be a challenge to adapt, but I’m looking forward to captivating audiences (and my close ones) in new ways that extend beyond the limitations of our traditional screens.