Tech-Narrative

Mixed Reality Story-telling􏰀: Story-setting and Story-sharing 

Alok b. Nandi

ABSTRACT: Mixed Reality Story-telling is explored by immersing real actors in virtual settings, while breaking the narrative design process into story-setting with story flows: in this paper, the author analyses the specific architecture of virtual and actual spaces allowing the interactors to live a narrative or poetic experience. 

KEYWORDS: narrative, point of view, projection, immersion, mixed and augmented reality

INTRODUCTION

The Transfiction [3] concept and system has been designed for mixing synthetic and natural images in real time and allows one to interact with virtual objects and props, without the need to wear glasses or gloves. It is a screen-based immersive system with gesture tracking to give users a control of the events displayed.

A mise-en-scène of narrative spaces in the screen allows the exploration of new spatio-temporal articulation. Fragmentation of spaces and senses of these mediated spaces are apprehended. A specific instance of experiencing an immersive projection and a deambulation into artificial spaces and hyper-travelling into these is designed and analysed.

From demonstration to participation, the nature of the relationship between perception and action underlies the ability to perform tasks. The sense of presence [of “being there”] is analyzed in an hybrid action spaces. Narrative modalities are investigated through body tracking, position detection and image recognition systems.

This paper, focused on one specific type of installation, gives a description of the system and the roles of the protagonists in the setting. The concept of hyper- cinematographic grammar is introduced and then the projection of the protagonist’s image is related to its fusion with the narrative. Present in time and presence in space are prerequisite for using the system for exploring a panorama (360° image) from one point of view.

The system serves as basis for one approach to interactive cinema and questions the boundary between game, role- playing and narrative.

MIXED-REALITY MISE-EN-SCENE

The initial hypothesis in Transfiction is simple: people see themselves on a screen, like in a mirror, but the visual environment (visual background) is removed in real-time. In parallel, algorithms analyse and interpret the body movements to trigger events in a narrative graph. The augmented (or magic mirror) effect, which creates the illusion of immersion, is provided thanks to image segmentation and then re-composition of real-time video images with visual scenes served from a media assets database: this ensures coherent and meaningful sequence composition. The interaction aspect is provided thanks to basic body/gesture analysis that allows users to use their hands and body to virtually ‘touch’ or ‘grasp’ objects and participate through their body attitudes in interactive scenarios.

The next phase, after the “magic mirror”, lead to a new paradigm: the user (also known as the interactor) does not see her/himself, but some visual clues allow her/him to understand that s/he pilots the course of events on the screen. The interactor is projected into the space visualised on the screen, hence cognitively and narratively. The screen-space can be designed in full freedom, the constraints being focused on the coherency of the visuals and the structure of the narrative or the poetic. Experience has shown that there is no need to manage the eventual glitches of video image segmentation in the diegetic visual sequence.

This freedom in the spatio-temporal articulation allows a “mise-en-scène” through multiplicity, juxtaposition, simultaneity, contiguity, as space is seen in a bergsonian way:

“Space, by definition, is outside us … space appears to us to subsist even when we leave it undivided, we know that it can wait and that a new effort of our imagination may decompose it when we choose. As, moreover, it never ceases to be space, it always implies juxtaposition, and, consequently, possible division. Abstract space is, indeed, at bottom, nothing but the mental diagram of infinite divisibility.” [1]

Space as the contraction of time and time as the expansion of space allow us to build interactive scenarios. However, to expand on the decomposition process, this is not only left to imagination, but also to physical action and interaction, in front of the screen-space.

HYPERCINEMATOGRAPHIC GRAMMAR

The word “narrative” refers to a set of events happening during a certain period of time and providing aesthetic, dramaturgical and emotional elements, objects and attitudes. In some ways, it is comparable to the script of a film. However, film scripts offer a linear reading where each piece is closely linked to the previous one: the meaning emerges from the whole story. In digital media (computation allows dynamic combination), modules of scenarios/scripts maintain their own individual meaning. Moreover, modules can be interchanged, and the spectator/interactor will have different experiences according to the paths of exposure through the set of modules. Two levels of design are considered when extending modular narratives towards Transfiction systems: the development process (author-centric orientation) and the reception process (user-centric orientation). These ensure that there is an overall coherency and that meaning is coming out of the experience, or emotion.

Similarly to movie editing, Transfiction scenes are composed of (independent) sequences assembled in a specific order: the narrative experience, along with the related meaning, takes place thanks to this sequencing, with the consequence that a cinematic grammar is shaped.

The information about the sequences and their possible relations and succession are kept in a “narrative graph”. This narrative graph allows, on a reception view point, to have the user jumping from space to space, resulting in a kind of hyper-voyage.

PROJECTION / FUSION

The Transfiction spatial and narrative system proposes the user to find her/his way in the narrative (the hyper-voyage forms are: linear path, labyrinthic path, game, puzzle solving). This action of finding is not only cognitive and requires a “body language”. The spatial relationships among places and things are the key features of the interface and the information overlays enrich the experience of the interactor.

The user has to “speak with his/her hands” as well as with body gestures and movements and to explore the space-screen. As Dove suggests, the interface is the membrane allowing the interactor to touch and feel the story:

“An interface has a recipe for the suspension of disbelief embedded in its operations that is similar to the recipes for the suspension of disbelief present in genre fiction. These codes are what we are willing to accept or believe to enter the world of a story. If interface design and genre fiction codes mutually support the vision of a story, an immersive world opens up. (…) The interface is the osmotic membrane – the means by which we fuse with the virtual space.” [2]

The membrane is the screen revealing the story and may be perceived like an haptic device. One can touch the storyspace and not only is s/he projected in the narrative but her/his gestures influence the course of the storylines.

PRESENT / PRESENCE

Touching the membrane-interface and moving into it has both spatial and temporal effects. The actual time is in balance with the suspension of disbelief, whilst the transportation into the fictional space is challenged by the presence in the actual/present time.

In his essay “The Specious Present”, Francisco Varela proposed a theory of the present as an “embodied now”. His research in the “depth of now” referred to Saint Augustine or even Aristotle and allowed him to go beyond the “notion of time that is ticked away by the clock, and that time is an arrow, and divided in equal seconds …” [5].

The quest for understanding consciousness (William James, Husserl, the Buddhist tradition) pushed Varela to bring together what “belongs to experience – what you might call the first person, which is something we have direct access to – with the scientific side, what you might call the third person, the external approach”. [5]

This hybrid quality (lived quality and objective quality) of time (which he studied in the discipline of “neuro- phenomenology”), is of interest in a Mixed Reality setting where the actual experience of the user is in juxtaposition with a virtual space and a diegetic universe, through a screen. The real-life time of the interactor, in the duration of the interactive experience, is in contraction with the diegetic time of the actor within the storyspace.

From present to presence, the distance is short and the sense of presence is here looked as the engagement in representations:

“This idea of suspension of disbelief is the literary- theoretical equivalent of the VR concept of immersion. It describes the attitude by which the reader brackets out the knowledge that the fictional world is the product of language, in order to imagine it as an autonomous reality populated by solid objects and embodied individuals.” [4]

Exploration in the Transfiction installations focuses on the subjective experience of presence within narratives and on the resulting perception. Users enter Mixed Reality worlds and realise that only customised representations allow them to act in the environment: body and space mutations, avatars or virtual ubiquity all reinforce the experience offered to the users while relying on a cinema- like grammar. The delicate moment is the triggering of the suspension of disbelief, which then allows to forget minor technical glitches and to focus on the narrative. Numerous installations using the Transfiction system have confirmed the perceived transportation in the fictional worlds.

PANORAMA / POINT OF VIEW

A specific Transfiction apparatus and setting allow the user-reader-viewer to become an interactor in 360° panoramic images.

Having the right arm and hand up results in the image turning on the vertical axis of the viewer and showing parts of the image on the right side and the same applies for the left hand and the image on the left. The interactor is represented by an iconic element having the tendency to remain at the center of the image, and moving when hands are moved. This iconic element has two states: passive, meaning that it only shows where is the interactor in the panorama – like a cursor-arrow on a screen-, and active, suggesting to the user that s/he can stop there for some time and it will lead her/him in another place. A combination of having the right and left hands up allows the image to stabilize and not to rotate anymore.

The active state of the icon triggers a video sequence ending by an image which is in fact another 360° panorama wherein the interactor can travel. Another way to jump from panorama to panorama is the fade-out to black and the apparition of another panoramic image. This is similar to a quicktime VR but here the image is under the control of hands without any devices.

Spatialized sound is attached to the panorama and according to the movement, a direct audio clue shows the effectiveness of the immersion and the body gesture allows to move within the diegetic universe.

All these elements contribute to the development of a narrative or a game. Several approaches have been explored (cross-media, immersive, multi-screens, on-site and on-line) to see how collaboration takes place: collaboration between the interactor and the diegetic universe; collaboration between several interactors in a multi-screen setting, on-site or on-line.

Different combinations have been explored in different settings with specific content. These have resulted in an emergent taxonomy of gestures and movements which can be used to further build dramaturgic elements in a mixed-reality setting.

STORY-TELLING / PLAYGROUND

The detailed description of the setting here above with 360° panoramas and video sequences as well as audio clues and sound ambiance proposes a base for application of “interactive cinema” with a technology conceptually stabilised.

However, there are some issues related to usability: some interactors have difficulties in mastering the system and research is planned at the level of usage. On a technical viewpoint, algorithms are optimised and hardware increase of performance allows a smooth experience for the interactor, in terms of reactivity of the system and display of the images. On a content aspect, the boundary between narrative and play is not yet established. The fact that someone has to move leads towards a game-type of attitude. This needs to be further searched in at least two areas.

Play in the game sense: kids using the system have shown

an easyness of mastering it and a very high degree of immersion; moreover, they do not have the inhibition of many adults. Play in the theatrical sense: this allows to tackle content with more depth and themes which are more profound.

In parallel, installations which are requestioning the perception of body and space have shown strong interest and are set up in different venues, such as “utHOPEia”.

CONCLUSION

Immersion in story-spaces allowed to enlarge the narrative envelop from the paradigm of story-telling to “story-sharing in a story setting”. By sharing, one means a dialogic relation between the user-reader-interactor and the diegetic material. By setting, one insists on the need to have a physical space design which ensure that the narrative or game experience in the virtual setting is optimal. The Mixed Reality paradigm and its related technological development should allow story-tellers, dramaturgists and designers to explore the potential of new genres at the intersection of the virtual and the actual.

REFERENCES

1. Bergson, Henry. Matter and Memory, trans. N.M. Paul and W.S. Palmer, Zone Books, New York, 1988, p. 206.

2. Dove, Toni. The Space Between: Telepresence, Re- animation and the Re-casting of the Invisible. in: Rieser, Martin and Zapp, Andrea (eds.). New Screen Media: Cinema, Art, Narrative. London: British Film Institute; 2002.

3. Nandi, Alok and Marichal, Xavier. Transfiction, chapter 2, pp. 13-33, in M. Eskelinen and R. Koskimaa (eds), CyberText – Yearbook 2000, Publications of the Research Centre for Contemporary Culture 68, 2001.

4. Ryan, Marie-Laure. Cyberspace, Virtuality, and the Text, in: Cyberspace Textuality: Computer Technology and Literary Theory (Ed., Marie-Laure Ryan), Pp. 78-107. Bloomington: Indiana University Press, 1999.

5. Varela, Francisco. The Deep Now, interview by Arjen Mulder, in: V2_Organisation, Machine Times, NAI Publishers, Rotterdam 2000.