The recording of metaverse experiences supports various use cases in collaboration, VR training, and more. Such Metaverse Recordings can be created as multimedia and time series data during the 3D rendering process of the audio–video stream for the user. To search in a collection of recordings, Multimedia Information Retrieval methods can be used. Also, querying and accessing Metaverse Recordings based on the recorded time series data is possible. The presentation of human-perceivable results of time-series-based Metaverse Recordings is a challenge. This paper demonstrates an approach to generating human-perceivable media from time-series-based Metaverse Recordings with the help of generative artificial intelligence. Our findings show the general feasibility of the approach and outline the current limitations and remaining challenges.

Read the full paper

The post Published: Artificial-Intelligence-Based Image Generation from Scene Graphs for Metaverse Recording Retrieval appeared first on Patrick Steinert.

At the heart of integrating these worlds lies the challenge of efficiently indexing, retrieving, and making sense of a deluge of multimedia content—ranging from images, videos, to 3D models and beyond. Enter the realm of Multimedia Information Retrieval (MMIR), a sophisticated field dedicated to the art and science of finding and organizing multimedia data.

The research explored here, as my Ph.D. project, ventures into this nascent domain, proposing innovative frameworks for bridging the Metaverse with MMIR. Their work unveils two primary narratives: one, how we can leverage MMIR to navigate the vast expanses of the Metaverse, and two, how the Metaverse itself can generate new forms of multimedia for MMIR to organize and retrieve.

In the first scenario, imagine you’re an educator in the Metaverse, looking to build an interactive, virtual classroom. Through the integration of MMIR, you can seamlessly pull educational content—be it historical artifacts in 3D, immersive documentaries, or interactive simulations—right into your virtual space, enriching the learning experience like never before.

The second scenario flips the perspective, showcasing the Metaverse as a prolific generator of multimedia content. From virtual tours and events to user-generated content and beyond, every action and interaction within the Metaverse creates data ripe for MMIR’s picking. This opens up a new frontier for content creators and researchers alike, offering fresh avenues for creativity, analytics, and even virtual heritage preservation.

Navigating these possibilities, the research present sophisticated models and architectures, such as the Generic MMIR Integration Architecture for Metaverse Playout (GMIA4MP) and the Process Framework for Metaverse Recordings (PFMR). These frameworks lay the groundwork for seamless interaction between the Metaverse and MMIR systems, ensuring content is not only accessible but meaningful and contextual.

To bring these concepts to life, let’s visualize a diagram illustrating the flow from multimedia creation in the Metaverse, through its processing by MMIR systems, to its ultimate retrieval and utilization by end-users. This visualization underscores the cyclical nature of creation and discovery in this integrated ecosystem.

In essence, this research lights the path toward a future where the Metaverse and MMIR coalesce, creating a symbiotic relationship that enhances how we create, discover, and interact with multimedia content. It’s a journey not just of technological innovation, but of reimagining the very fabric of our digital experiences.

Let’s create an image to encapsulate this vibrant future: Picture a vast, sprawling virtual landscape, brimming with diverse multimedia content—3D models, videos, images, and interactive elements. Within this digital realm, avatars of researchers, educators, and creators move and interact, bringing to life a dynamic ecosystem where the exchange of multimedia content is fluid, intuitive, and boundlessly creative. This visualization, rooted in the essence of the research, will capture the imagination, inviting readers to envision the endless possibilities at the intersection of the Metaverse and MMIR.

The post Integration of Metaverse and Multimedia Information Retrieval appeared first on Patrick Steinert.

I’m thrilled to announce the availability of the 256-MetaverseRecords Dataset, a dataset for experiments with machine learning technology for metaverse recordings. This dataset represents a significant step forward in the exploration of the integration of virtual worlds in Multimedia Information Retrieval.

The dataset was created to explore the use of meatverse virtual worlds and evlauate performance of feature exraction methods on Metaverse Recordings. The dataset contains 256 video records of user sessions in virtual worlds, mostly based on screen recordings.

The dataset is part of my work to achieve a doctor’s degree. The recordings were made during a student workshop with 13 participants. Each student was tasked to create 20 videos with a duration of 2-5 minutes, resulting in a rich collection of data. This variety ensures a robust dataset that encapsulates a wide range of virtual interactions, making it an invaluable tool for researchers.

Each student was required to annotate their videos based on a provided schema. These annotations include information about the actions occurring within each recording. This structured approach to data collection ensures that the dataset is not only vast but also well-organized, allowing for efficient data retrieval and analysis.

Together with the recordings, several annotations are available. These annotations serve as a starting point for self-developed processing and experiments. Researchers and developers can use these annotations to train machine learning models, test new algorithms, or even develop new annotation techniques tailored to the unique characteristics of the metaverse.

The 256-MetaverseRecords Dataset is not just a collection of videos; it’s a gateway to understanding the complex dynamics of virtual worlds. The potential applications of this dataset are vast. For instance, it could be used to improve user experience in virtual reality (VR) games by enhancing the realism of the environment or to develop new educational tools that leverage the immersive nature of VR.

Furthermore, this dataset could play a crucial role in advancing the field of computer vision, especially in understanding and interpreting the rich, dynamic content of virtual environments. It offers a unique challenge to traditional feature extraction methods, pushing the boundaries of what’s possible in machine learning and AI.

In conclusion, the 256-MetaverseRecords Dataset is an invaluable resource for anyone interested in the intersection of virtual reality and machine learning. It opens up new avenues for research and development, promising to contribute significantly to our understanding and utilization of virtual worlds.

The dataset has its dedicated website and is available at: https://patricksteinert.de/256-metaverse-records-dataset/

The post 256 Metaverse Records Dataset appeared first on Patrick Steinert.

IEEE MetroXRAINE is an interdisciplinary conference on the fields of metrology, Extended Reality, Artificial Intelligence, and Neural Engineering. I tried an EEG-based brain interface for a game. It is an awesome experience, and I’m excited to see more of this technology in the future. But for sure, I need to concentrate more on my research!

The post Award Winning Graphical Abstract and Paper Presentation at IEEE MetroXRAINE 2023 appeared first on Patrick Steinert.

While talking about your thoughts, models and experiences is cool, the bigger stages still make me nervous—however, practice and experience (aka training) help a lot.

AI is a hot topic, not only generative AI, but more and more of the business in media and elsewhere is driven by data. Customer experiences are customized by AI, manual effort vanishes through AI, and even content is created by AI. In Multimedia research, we say “Multimedia is everywhere”, but I think we can state “AI is everywhere”.

The post Big Data Week & AWS User Group Bonn appeared first on Patrick Steinert.

In 2022, it was expected that the metaverse would be a fully realized and integrated part of our daily lives. People would be able to work, socialize, and even go on vacations entirely within virtual worlds. However, as we now know, the reality of the metaverse in 2022 has not quite lived up to these expectations.

Metaverse Challenges

One of the main challenges in realizing the metaverse has been the technical limitations of virtual reality and augmented reality technologies. While these technologies have made tremendous strides in recent years, they still have a way to go before they can provide the level of immersion and seamless integration that is necessary for the metaverse to truly become a part of our daily lives.

Another challenge has been the lack of a clear business model for the metaverse. While there have been some successful virtual worlds, such as Second Life and World of Warcraft, these have primarily been used for leisure and entertainment rather than as a replacement for the real world. Without a clear way for companies and individuals to make money within the metaverse, it has been difficult for it to gain mainstream adoption.

Despite these challenges, the metaverse is still an exciting and promising concept. Many companies and individuals are continuing to work on developing the technologies and business models necessary to bring the metaverse to fruition. It is possible that in the coming years, we will see more and more people using virtual and augmented reality technologies for everyday tasks and activities.

The Future of the Metaverse

In conclusion, while the metaverse of 2022 has not yet lived up to expectations, it is still a concept with a lot of potential. As technology continues to advance and new business models emerge, we may yet see the metaverse become a fully realized part of our lives.

The post Metaverse 2023: Expectation vs. Reality appeared first on Patrick Steinert.

What is the Metaverse

It isn’t easy to describe the metaverse precisely, because it is under heavy development. The term was coined in Neal Stephenson’s novel Snow Crash. Over time, it went through some iterations until the current climax, decided to rebrand to meta and go all in into the metaverse. Avi Bar-Zeev visualized the different stages of the metaverse perfectly.

As the metaverse is under development, the width of the definition is broader. Mystakidis defined the metaverse as follows:

“The Metaverse is the post-reality universe, a perpetual and persistent multiuser environment merging physical reality with digital virtuality. It is based on the convergence of technologies that enable multisensory interactions with virtual environments, digital objects and people such as virtual reality (VR) and augmented reality (AR). Hence, the Metaverse is an interconnected web of social, networked immersive environments in persistent multiuser platforms. It enables seamless embodied user communication in real-time and dynamic interactions with digital artifacts. Its first iteration was a web of virtual worlds where avatars were able to teleport among them. The contemporary iteration of the Metaverse features social, immersive VR platforms compatible with massive multiplayer online video games, open game worlds and AR collaborative spaces.” [1]

Following this definition, for the verification, if metaverse is multimedia, these aspects are relevant

• “physical reality with digital virtuality”
• “multisensory interactions with virtual environments, digital objects and people”
• “real-time and dynamic interactions with digital artefacts”
• “immersive VR platforms compatible with massively multiplayer online video games”

In summary, the metaverse is a not-only digital virtuality with interactions, similar to a video game. Next, the definition of Multimedia will be discussed.

Definition of Multimedia

There is no distinct definition of the term multimedia. Some definitions have been formulated and they have a common ground, which is enough for the verification. In the book “Multimedia making it work”, Vaughan [2] summarized multimedia as “any combination of text, art, sound, animation, and video delivered to you by computer or other electronic or digitally manipulated means.” [2, p. 1] He mentioned examples:

The metaverse is merging physical reality with digital virtuality. Digital virtuality involves video, audio and haptic feedback. Within the metaverse, this media can have different forms. A virtual TV shows a video or in a virtual art gallery pictures are presented and audio can be the explanation of a guide or ambient noise. According to Vaughans definition, the metaverse is a combination of at least video and sound, delivered to you by a computer.

As the metaverse is a multiuser platform, other people are physically or virtually present, moving around and talking to each other, like in a video conference. People can also interact with digital objects or even real objects (with IoT). Therefore, the metaverse allows multisensory interactivity, multitasking and parallelism, which makes it multimedia according to Klimas description. The interaction has no beginning, order and end. It is a persistent world, where people can join and leave all the time, it is categorized as non-linear multimedia.

Finally, the two criteria from Weidenmann should be examined. Since the metaverse is a complex virtual world, in some scenarios it involves multicoded information. If a user is in a virtual museum, she can read a description of a historic situation, along with images, pictorial timelines and ambient sound of this situation. In this case, the metaverse has multicoded information. If a user is staying in a garden, listening to bird sounds, there is no multicoded information. Some could argue, that the presentation of a garden as video along with ambient noise is multicoded information, because it gives the user the information, that the garden looks and sound like this. Anyway, it clearly shows, that Weidenmanns second criterion, multimodality, is fulfilled.

With this matching of the properties of the metaverse and the characteristics of multimedia, it is verified, that the metaverse is multimedia.

References

[1] Mystakidis, S. (2022). Metaverse. Encyclopedia, 2(1), 486-497.

[2] Vaughan, T. (2011). Multimedia: Making it work. McGraw-Hill, Inc.

[3] Issing, L. J., & Klimsa, P. (2002). Informationen und Lernen mit Multimedia und Internet. Weinheim.

[4] Weidenmann, B. (2001): Lernen mit Medien. In: A. Krapp & B. Weidenmann (Hrsg.): Pädagogische Psychologie (S. 415–466), Weinheim: PVU. 4. Aufl.

[5] Wikipedia (2022), Multimodality, https://en.wikipedia.org/wiki/Multimodality, last accessed 11th Nov. 2022.

The post Is Metaverse multimedia? appeared first on Patrick Steinert.

The academic definition of the Metaverse

As the metaverse is under development, the width of the definition is broader. Mystakidis defined the metaverse as follows:

“The Metaverse is the post-reality universe, a perpetual and persistent multiuser environment merging physical reality with digital virtuality. It is based on the convergence of technologies that enable multisensory interactions with virtual environments, digital objects and people such as virtual reality (VR) and augmented reality (AR). Hence, the Metaverse is an interconnected web of social, networked immersive environments in persistent multiuser platforms. It enables seamless embodied user communication in real-time and dynamic interactions with digital artifacts. Its first iteration was a web of virtual worlds where avatars were able to teleport among them. The contemporary iteration of the Metaverse features social, immersive VR platforms compatible with massive multiplayer online video games, open game worlds and AR collaborative spaces.” [1]

With this rich description, what are the origins of the metaverse?

Origin and History

The term metaverse was first coined by Neil Stephenson in the book Snow Crash. Since then, the idea has taken several iterations. Avi Bar-Zeev visualized the different stages of the metaverse perfectly. Early implementations of virtual worlds were created, in the 1980s, such as Habitat, the first high-profile commercial application of virtual world technology. It was the first to use the term “avatar”, the inhabitants of the virtual world, in the virtual world. Another milestone was Worlds, Inc., the first virtual World with full three-dimensional graphics. It enabled users to socialize in 3D spaces.

In recent years, the software Second Life was most close to a real incarnation of the metaverse. Second Life, developed by Linden Labs, was hype around 2008. Many studies have been done on social topics and Second Life, but it was not a breakthrough into the mainstream. While it still is in use (as Linden Labs states), it could be told, that there was a kind of metaverse winter.

In 2012, Google announced its product Google Glass, a head-mounted device, with a display and a camera. It was the first kind of Augmented Reality (AR) device, with broad public attention. Since Google Glass, AR and VR devices developed in small iterations. The goal is to create devices which humans can wear all the time and anywhere, but until now this vision was not achieved.

Since 2020, maybe fueled by the COVID-19 pandemic, the metaverse had another spring. Many implementations emerged. One to mention is Decentraland, an implementation of a metaverse based on web3 technologies. The data is stored decentralized on a distributed ledger called the blockchain. It is a public and decentralized database. Decentralizing and auditing the data and the ledger is done by smart contracts. Users can buy virtual land, paid with their own cryptocurrency Mana, and the purchase is stored as a smart contract.

Finally, the company Meta started a metaverse called Horizon Worlds. While investing a lot of money in the metaverse, in comparison to other existing virtual worlds, it is not the top-notch virtual world.

Use Cases

After all this, what will we do in the metaverse? Or should the question be: what not? In general, a virtual world could include any activities of the real world, just some more like flying through the sky and space. If the metaverse augments reality, people will still do the same things, just in another way. On a high level, human activities can be categorized as Work, Duty and Leisure. Work includes all kinds of jobs, such as collaborative work (e.g. Brainstorming, modelling, virtual conferences), office work (e.g. Documents, Spreadsheets), shopping assistance in a virtual shop, learning (e.g. E-Learning, training use of machinery), or construction worker (e.g. controlling a robot, designing virtual buildings). Duties cover doing errands, managing financials, cleanup, taking care of government business (possible in Seoul) etc. While leisure has maybe the most explored use cases, like gaming, sports, entertainment, virtual travel and relaxation.

The described use cases can all be done alone or together in a group of arbitrary size.

Metaverse and web3

Metaverse is also used with the term web3. The term web3, or web 3.0 was used for a machine-readable internet, where websites provide semantic information, processable by computers. The number 3 is regarded as the version of the internet. While there is no manufacturer or standards organization, it is defined by the Zeitgeist. As the semantic web is existing, it is not regarded as the next version of the internet, such as the social web – Web 2.0. The next wave of technologies arose around decentralization, most blockchain-based apps. Many people from that party proclaim these technologies as the next internet, therefore they took over the term web3. While web3 is a bundle of technologies based on blockchain, the metaverse is not necessarily based on blockchain nor on a decentralized infrastructure. Second Life or Metas Horizon Worlds is an example of a centralized, no blockchain-included Metaverse. While Decentraland is an example of such a decentralized Blockchain Metaverse. In conclusion, the metaverse is independent of web3, but it can be built with web3 technologies.

Beyond decentralization, other blockchain-based technologies could be part of the metaverse implementation. Smart contracts for digital ownership, like property in the virtual world or 3D models. Cryptocurrencies can be used for the payment of digital goods.

Standards

Almost everybody in the space of the metaverse agrees, that there is no single company that can build the metaverse alone. Also, the call is to create an open metaverse, like the internet itself. Therefore, interoperability is needed. In 2022 the Metaverse Standards Forum was founded. It rapidly grew from 37 founding in June 2022, to over 1500 in just two months. As this shows the interest of the internet industry and academics in the metaverse, it also shows the desire for standards.

As the metaverse forum is still forming, some focus domains are declared, including but not limited to:

• Interactive 3D assets and photorealistic rendering
• Human interface and interaction paradigms including AR, VR and XR
• User-created content
• Avatars, identity management and privacy
• Financial transactions
• IoT and digital twins
• Geospatial systems

The need for technical standards is there and especially ethical standards should be applied to these virtual worlds before everything ends up as it went with the social media networks.

Critics and Challenges for the metaverse

The overall challenge is how to transition from a complex but completely independent immersive environment set to a massive integrated network of 3D virtual worlds or the metaverse, thus creating parallel contexts for human interaction and culture. On a high level, this includes technical challenges, social acceptance, and the establishment of a permanent society in the metaverse.

Technology

Technology-wise, a lot of different metaverse implementations exist. They don’t follow a single pattern and are not connected. Architectural aspects vary from centralized versus decentralized approaches, and different access technologies (Web, VR, AR). Standards may change this, but there is also a mix of interests in the form of commercial claiming, social network market leader tendencies and more. Therefore, it is open to how the situation develops.

Another technical aspect is the progress of immersion and immersive hardware. At the moment, the AR/VR technology is mainly bulky headsets and clunky devices (gloves or vests), which are seen as a burden, more than an evolution. Miniaturization of this hardware will progress, and it can be expected that (commercial) interests will drive money into this field, leading to more research.

Overall Security

A large metaverse with lots of users is likely a target of hackers. Therefore, security is highly relevant but complex to achieve. The internet is around for decades, and still not a very secure place. The metaverse is similarly complex and has the unfair disadvantage, that it needs to stand against highly skilled hackers and criminals. The complexity demands security models on different layers and will be a research and development topic in the future.

Next to security, privacy will be a challenge. Design for high privacy is good for human rights. In contrast, high privacy attracts criminals. Trust, Asset Ownership.

Authentication

Authentication is a twofold challenge. Of course, users should be authenticated to the metaverse to buy things or do transactions, but also to hold people accountable for their actions. This can be a challenging design decision because the metaverse is a virtual space where users can create and assume various identities, their avatars. In the game world, nicknames and virtual identities are deeply anchored. Preserving this contradicts the demand for authentication. As a result, ensuring the authenticity of users in the metaverse can be difficult.

Authority

One of the major critical questions is about the governance of the metaverse: How will make the rules and laws for the metaverse? What the world learned from chats and social media and the dark net, such virtual worlds need governance and rules to be a mostly legal place. But the metaverse, the collection of multiple virtual worlds, does not seem to be a thing where traditional governments apply well. The idea of a decentralized government fits this challenge because traditional borders don’t apply. While on the internet, at least companies or physical servers can be located in a country, for the metaverse it will be different. Still, servers are involved there, but if the design of the metaverse is to be independent of a single entity, servers can be anywhere. A community-based approach with democratic structures, written in a smart contract would suit this idea. But the challenge is to build it in detail.

The connecting question is about the enforcement of the rules. Technically they can be built in the virtual worlds. But for many cases, it is hardly possible to be enforced technically. A solution could be based on the separation of powers, but who will be the powers?

As described, a lot of questions are unanswered. But with the current hype on top of the long-term demand, it is obvious that some of these questions will soon be answered and the look into the future is worthwhile.

Forecasts and Outlook

The metaverse or virtual worlds have been around for ages. The current hype is fueled by the Facebook rebrand and Web3 topics. Besides that, the technologies are mature and companies and people are investing energy into the metaverse idea. So it can be expected, that we see more of this kind of things in the future. What will it look like? This is an open question. Even Neil Stephenson put into question if the metaverse will be AR, VR or just 3D? Immersion is what counts most.

Another question is if there will be one metaverse? Tony Parisi put the answer in the first of seven rules of the metaverse: “Rule #1: There is only one Metaverse.”. The following explanation shows, that it is a collection of multiple virtual worlds. It is like the internet today, it is one internet, a collection of multiple websites and (social) networks. This explanation leaves enough room for ideas of the metaverse.

As with most technologies, it is all about use cases. The question to ask is which use cases work better in interactive, immersive storytelling? One clear answer to this is games. A lot of games have immersive virtual worlds and people use them actively. Therefore, it has been predictable that social interactivity would enter such worlds. One example of this is MovieNite in the game Fortnite. Other existing use cases are serious games and e-learning. Training in virtual spaces has been explored intensively and has proven value. The use case of presenting multimedia in a metaverse opens new ways to collaboratively enjoy leisure time.

What use case is next? Since the metaverse can’t predict the future (yet), it is all about guessing. Shopping is a highly rated use case. It is not about wearing a nice 3D jacket on your clunky Horizon Worlds Avatar. Photorealistic avatars already exist and can be a better manikin for accurately fitting 3D models of real-world clothes. Scan your body once for an accurate model and you can test fit clothes. The same principles work with furniture. 3D models of furniture exists anyway, so why should’t them be displayed in a virtual immersive world?

Further use cases are in the collaborative space of design, music and more. It is no wonder that Deloitte or Gartner predict a growing multi-billion dollar market. Just the future will tell, where it will end up.

The post A primer on the Metaverse appeared first on Patrick Steinert.