stu and ANYCOLOR Develop Real-Time Mixed Reality System for VTuber Performances
stu Inc. (Shibuya, Tokyo; CEO: Takayasu Kuroda) and ANYCOLOR Inc. (Minato, Tokyo; CEO: Riku Tazumi) have jointly developed a specialized mixed reality (MR) system for VTuber performances. The technology was successfully demonstrated at the “ANYCOLOR/Nijisanji New Technology Demonstration Event” held January 8-9, 2025.
The demonstration featured Nijisanji talent Hayato Kagami performing with host Hyakka Ryoran in two formats: interactive audience talk sessions and musical performances. Attendees experienced the event through Meta Quest 3 headsets and Xperia devices.
Transforming VTuber Live Experiences
Traditional VTuber performances have relied on LED screens displaying pre-rendered content. The new system transforms this experience by allowing audiences to view performances from their personal perspective through headsets or smartphones.
A key innovation is the system’s low-latency data transmission, enabling real-time interaction between VTubers, hosts, and audience members. Rather than isolated audio through earphones, sound comes through venue speakers—creating a communal experience similar to traditional concerts.
Technical Innovations
The system consists of two core components developed specifically for VTuber performances: a specialized transmission system and an MR compositing system that creates the impression of VTubers existing in the physical space.
Advanced Data Transmission
The technical challenge involved transmitting character motion data and lighting effect signals with minimal latency to multiple devices simultaneously. The team developed proprietary data compression technology specifically optimized for VTuber performances, which minimizes video delay and synchronization issues that would otherwise disrupt the immersive experience.
Precision Visual Integration
A significant technical hurdle was generating high-precision real-time composite images that accurately represent spatial relationships between virtual characters and physical stage elements. Conventional AR/MR technology struggles with this level of integration, particularly when incorporating real-time lighting effects.
The team developed a new compositing method specifically for live event environments that maintains accurate depth relationships while synchronizing characters with venue audio and lighting systems—creating a seamless blend of virtual and physical elements.
Optimized Network Infrastructure
To support the substantial data requirements, the team completely redesigned standard venue networking infrastructure. Conventional wireless setups aren’t designed to handle simultaneous real-time data transmission to numerous devices.
The system implements specialized network equipment optimized for MR performances with streamlined data flow architecture. This maintains consistent performance quality while reducing the technical complexity for operators.
Future Applications
This technology creates opportunities beyond traditional performances. The system can be applied to gaming, exhibitions, theme parks, retail environments, and other entertainment sectors.
Moving forward, both companies will continue refining these technologies to support innovation across Japan’s character-based content industry.
Demo Event
*Actual view of the venue as seen by attendees, captured through Quest3
ANYCOLOR/Nijisanji New Technology Demonstration Event ver.2025.01
Host: Hyakka Ryoran
Performer: Hayato Kagami
Technical Development: stu
Motion Capture Support: GUNCY’S
Lighting and Visual Direction: huez / Shibucreation.inc
More info (Japanese): https://www.anycolor.co.jp/news/lcc0onaguaf
■About ANYCOLOR Inc.
ANYCOLOR Inc. is an entertainment company founded on May 2, 2017. Their mission statement, “creating magical new experiences,” guides their work as pioneers in the expanding entertainment economy. As a forward-thinking startup, they develop innovative content that brings a touch of magic to people’s daily lives around the world.
URL: https://www.anycolor.co.jp/contact
■About stu Inc.
stu is a creative technology company on a mission to reinvent entertainment. At the intersection of edge tech and creativity, our teams of developers, designers and planners create captivating, next-generation content and immersive experiences powered by AI, 5G, and XR technologies. From metaverse projects to our all-in-one content production studio, we’re shaping the future of entertainment. Our integrated approach spans from writers rooms and pre-visualization through to marketing. We collaborate with industry experts and government agencies to bring bold concepts to life and establish new benchmarks. Established in 2017, stu continues to grow rapidly, attracting talent and forming partnerships with global brands. Our diverse portfolio includes video production, live events, interactive installations, and digital platforms… all designed to engage audiences in innovative ways.
Join us in shaping tomorrow’s entertainment.
Email: contact@stu.inc
Official Website: https://stu.inc
Official X (formerly Twitter): https://twitter.com/stuinc_