The metaverse had its peak of inflated expectations in 2022 and has since dropped off the Gartner curve. In one field, though, it stayed and keeps growing: healthcare. Telemedicine, telerehabilitation, training.
Together with colleagues I have published a paper on the CareSync framework, which addresses exactly that. It appeared in Electronics and received the Editor’s Choice award.
The problem
Current telemedical systems work mostly video-based. What is missing is continuous capture of the patient’s state and a structured way of handling the resulting volumes of data. Without affective, physiological and structured longitudinal data, personalisation remains piecemeal.
The approach
CareSync is an emotion-aware, data-driven architecture for personalised interventions and Multimedia Information Retrieval in the Healthcare Metaverse. It builds on the SenseCare KM-EP platform and extends its telerehabilitation subsystem T-Rehab with a real-time pipeline. It brings together:
- contactless physiological sensing – heart rate via remote photoplethysmography (rPPG), respiration rate via chest movement, stress heuristics via RMSSD
- markerless motion analysis – face and body tracking with MediaPipe, hand and wrist tracking with Leap Motion, including joint angles, range of motion, symmetry and repetition counting
- indexing and retrieval – telemetry and vital data are turned into structured session summaries and co-occurrence graphs that allow exact filter queries and graph similarity search
One detail I like: the sessions remain available without video – as a 3D avatar replay.
The evaluation
Five rPPG methods (GREEN, CHROM, POS, PCA, ICA) were examined against a synchronised ECG reference, on a cohort of N = 26 participants from the rPPG-10 benchmark dataset, in 30-second windows. GREEN performed best, with a mean absolute error of 17.74 BPM.
The authors frame this carefully themselves: the results are intended as a methodological benchmark, not as evidence of clinical accuracy. That honesty is what makes the work worth reading.
The paper
Hadjar, H.; Steinert, P.; Vu, B.; Hemmje, M. “Multimodal Healthcare in the Metaverse: Integrating Multimedia Information Retrieval, Physiological Sensing, and Markerless Kinematic Analytics Within the CareSync Framework”. Electronics 2026, 15(19), 4407. Published 24 September 2026, 22 pages, open access (CC BY).