After the hype around the metaverse faded, something interesting happened: the applications stayed – in gaming, VR training, simulation and collaborative work. And with them come recordings. Entire collections of them.
What was missing was a structured framework for analysing and indexing them. That is exactly what our new paper in Big Data and Cognitive Computing describes, now published open access.
What it is about
Metaverse Recordings (MVRs) are a medium type of their own. They are captured from the audio-video output of the underlying game software, may additionally contain 3D models and textures – and their structure differs fundamentally from other media. For Multimedia Information Retrieval (MMIR) that means existing analysis methods have to be adapted.
The paper introduces the Processing Framework for Metaverse Recordings (PFMR), describing four phases:
- MVR Structural Analysis – capture the structure of the recording
- MVR Feature Extraction – extract features
- MVR Data Mining – find patterns and relationships
- MVR Feature Fusion – merge the results
The contribution is not a new feature-extraction method but the question of how heterogeneous analysis components can be composed and orchestrated flexibly – and thereby stay adaptable per domain.
How it was evaluated
The evaluation is based on a prototypical implementation and the 256 Metaverse Records Dataset. It shows that metaverse-specific features can indeed be located with the framework: avatar detection, dialog mining and toxicity classification. The domain-specific adaptability of the PFMR enables effective search in metaverse recordings.
Transferability is interesting too: the authors expect the approaches to apply to related areas of temporal multimodal analysis – social media analytics, autonomous driving or robotics.
The paper
Steinert, P.; Wagenpfeil, S.; Frommholz, I.; Hemmje, M.L. “A Reference Model for the Analysis and Indexing of Metaverse Recordings for Information Retrieval”. Big Data Cogn. Comput. 2026, 10(3), 85. Published 8 March 2026, 25 pages, open access (CC BY).
https://doi.org/10.3390/bdcc10030085
The paper belongs to the special issue “Advances in Artificial Intelligence for Computer Vision, Augmented Reality and Metaverse”.