An integrated framework for the interaction and 3D visualization of cultural heritage

Abstract

In this study, the aim is to design and develop a 3D acquisition, visualization, and interaction framework to preserve cultural heritage and provide new ways to enable museum visitors and cultural audiences to virtually interact with cultural objects. Indeed, cultural assets are nowadays at higher risk and most cultural institutions prohibit visitors from physically manipulating their collections. The main motivation behind our framework is to enable end-user interaction with high valuable cultural objects while addressing cost-effectiveness concerns as well as minimizing the time required to digitize and generate 3D models of cultural heritage objects. The design idea of our framework is to allow interaction with the protected assets' 3D representation using a real-world 3D screen equipped with a depth sensor namely the leap motion controller. Our framework is an end-to-end solution that optimizes all the stages of the 3D acquisition, pre-processing, visualization, and interaction pipeline while providing contributions to its stages. It achieves good quality results thanks to the use of machine learning in the acquisition and modeling stages. Indeed, we adapted a prior preprocessing work that performs super-resolution and motion interpolation on the acquired data. The preprocessed data is then used for the generation of the 3D models using photogrammetry, which optimizes the quality of the resulting 3D models. The created 3D models are then adapted for the visualization and interaction stages. A novel visualization and interaction paradigm is introduced to enable a real-world experience for museum visitors through a 3D screen called "the Looking Glass". The interaction with the 3D content is achieved through a motion sensor used to design our new interaction component of the framework. We propose two new interaction systems suitable for various user profiles focusing on their experience in dealing with motion sensors. The end-to-end framework tested in a museum environment was evaluated by cultural heritage curators and multimedia experts and found to provide an alternate reality tool for asset exhibition and a cost-effective alternative for asset exchange between cultural institutions. For the evaluation, we compared the end-user experience of our framework using various setups where users are visualizing the content through 2D screens and through the Looking glass while enabling and disabling motion interaction. The results of the evaluation suggest that the looking glass paired with the Leap motion sensor using our framework as a backend enables an alternate reality experience for museum visitors and new ways of interacting with cultural content, sharing of cultural knowledge, cultural education, and much more. 2023, The Author(s), under exclusive licence to Springer Science+Business Media, LLC, part of Springer Nature.